Can models learn to ignore irrelevant prompt changes?
Explores whether training models to produce consistent outputs regardless of sycophantic cues or jailbreak wrappers can solve alignment problems rooted in attention bias rather than capability gaps.
Sycophancy and jailbreaking share a structural property: the model produces the correct response to a clean prompt but changes its response when irrelevant cues are added (a user's stated opinion, a jailbreak wrapper). The problem is not capability — it's consistency.
Consistency training reframes alignment as invariance: train the model to produce the same response regardless of whether the prompt includes irrelevant perturbations. Two methods implement this:
Bias-Augmented Consistency Training (BCT) operates on output tokens. For each prompt, the model generates a response to the clean version. This response becomes the training target for the wrapped version. The model learns to say the same thing regardless of sycophantic cues.
Activation Consistency Training (ACT) operates on internal representations. Instead of matching output tokens, ACT enforces that residual stream activations on the wrapped prompt match those on the clean prompt. This is a more mechanistic constraint — teaching the model to think the same way, not just say the same thing.
Both reduce sycophancy effectively. BCT is better at jailbreak reduction. The advantage over standard SFT is avoiding two forms of staleness:
- Specification staleness — when response guidelines change, static SFT datasets become obsolete
- Capability staleness — when training targets come from older, less capable models, SFT degrades current capabilities
Since consistency training uses the model's own clean responses as targets, both staleness problems disappear. The training data is always fresh and at the model's current capability level.
Continual learning extension — Self-Distillation Fine-Tuning (SDFT). SDFT generalizes the self-as-target principle to continual learning from demonstrations. The model plays two roles: a teacher conditioned on both input and expert demonstration (via in-context learning), and a student conditioned on input only. Training distills the teacher into the student on trajectories generated by the student itself — yielding on-policy updates that incorporate demonstration knowledge without explicit reward inference. SDFT achieves higher new-task accuracy while substantially reducing catastrophic forgetting vs standard SFT. In sequential learning across three skills, a single model accumulates each skill without regression on previously learned abilities. The mechanism parallels BCT: both use the model's own contextually-enhanced output as the training signal, avoiding off-policy distribution mismatch.
This connects to Does transformer attention architecture inherently favor repeated content?. S2A identifies the architectural root (attention bias toward repeated/prominent tokens); consistency training provides the training-level fix (enforce invariance to those biased attention patterns). ACT's activation-level approach is particularly relevant — it may directly counteract the attention bias at the representation level.
ProSA (2024) provides the diagnostic that explains WHY consistency training works. Prompt sensitivity is fundamentally a reflection of model confidence: higher confidence correlates with increased robustness against prompt semantic variations. This means consistency training (BCT/ACT) succeeds not by teaching a separate "invariance skill" but by pushing models toward confident response regions where robustness is a natural property. Few-shot examples also alleviate sensitivity by providing concrete anchoring. Larger models exhibit enhanced robustness. Source: Arxiv/Prompts Prompting.
Inquiring lines that read this note 147
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do prompting refinements mask underlying biases and model frequency patterns?- Can better AI interfaces eliminate the attention cost of prompt composition and evaluation?
- Can prompting strategies eliminate systematic biases without shuffling or aggregation?
- Does irrelevant content degrade reasoning even when it fits the context window?
- How much of prompt sensitivity is really just frequency optimization in disguise?
- How does output variability disguise confirmation bias in prompt refinement?
- Does SMART-style prompting survive adversarial rephrasing of biased questions?
- Do prompting technique improvements actually replicate in controlled experiments?
- Do widely-repeated prompting heuristics like politeness actually improve accuracy?
- Can re-scoring detect subliminal prompt injection without explicit semantic content?
- Do shared prompts and infrastructure keep model biases correlated?
- Why does grouping feedback by shared correction pattern improve prompt generalization?
- How do training-data priors influence model defaults when context is ambiguous?
- How does surface salience compete with background knowledge in model inference?
- Can prompt-based debiasing overcome entrenched LLM model priors?
- How do pretraining biases interact differently with prompts across model tiers?
- Why does context information fail to override prior training associations?
- Why do primacy effects peak at specific instruction densities?
- How much can mitigation techniques like augmentation reduce priming without harming learning?
- Does foundational model training or user priors more strongly shape final outputs?
- Why does consistency training make models resistant to prompt perturbations?
- Can prompt-based debiasing work if biases are embedded in pretraining?
- How do training associations override context information in language models?
- Do few-shot examples improve in-context learning or add noise?
- Can we reverse the instruction-following deficit through targeted training?
- How do behavioral differentiation and paraphrase stability trade against accuracy?
- Is lower context-following a failure or appropriate model behavior?
- How do unstated constraints become invisible to training data distributions?
- Can scaling up contradictory training data overcome unpredictable override effects?
- Does self-conditioning improve belief-behavior alignment better than external priors?
- How does distribution mismatch between training and deployment break self-correction?
- Why does self-correction during generation produce reliable labels without exemplars?
- Why do unchecked self-edits accumulate drift toward overfitting or incoherence?
- Why does transformer attention architecture reinforce sycophancy and agreement?
- Can transformer attention architecture explain why chatbots default to sycophancy?
- Can reward model biases alone explain why sycophancy generalizes beyond training?
- Does fixing reward models alone stop sycophancy without fixing attention mechanisms?
- Does transformer attention architecture systematically bias models toward sycophancy?
- How does the U-shaped attention distribution relate to transformer sycophancy?
- Does transformer attention architecture inherently bias models toward sycophancy?
- Is sycophancy caused by mechanical drift rather than intelligent reasoning corruption?
- What architectural features drive sycophancy closer to inference than training?
- Can attention patterns alone explain sycophant model behavior without reasoning?
- Can humans suppress frequency bias through attention and intention?
- Can System 2 Attention reduce sycophancy without changing training objectives?
- Can decoding strategies or external verification layers reduce sycophancy?
- Why do sycophancy hints show the worst acknowledgment gap?
- Can sycophancy in AI be fixed by changing the model itself?
- Can instruction tuning succeed without explicit task understanding?
- Are instruction-tuned models more or less sensitive to prompt semantics than others?
- Why do large language models follow user drift instead of maintaining topic focus?
- Does directional knowledge failure indicate shallow pattern matching over deep representation?
- Do language models inherit gender bias from training data in grading tasks?
- How robust is misalignment classification across different judge models?
- Can activation steering directly steer models toward concise reasoning without prompting?
- How much does sliding-window augmentation improve single-session modeling?
- Can activation-space directions reliably steer LLM behavior without retraining or prompting?
- Can activation-space interventions reach biases that prompting cannot address?
- How does activation consistency training differ from output-level consistency?
- Do mechanistic refusal vectors transfer across different models and training settings?
- Why does recontextualizing a behavior during training change whether models learn it?
- How does demo position create spatial bias in prompts?
- How do ordering effects compound across different prompt component scales?
- How does tone sensitivity create systematic informational bias in model responses?
- Can distinctive input voices maintain accuracy without adopting the model's preferred register?
- Can emotional framing in prompts exploit the same mechanism that causes response bias?
- Why does politeness in prompts measurably affect model performance across tasks?
- How do emotional framing effects in prompts influence model performance?
- Can prompt position alone shift language model predictions by twenty percent?
- How do input-side defenses separate task methodological and framing intents?
- What prompting techniques actually replicate under controlled statistical testing?
- How much does instruction prompt design control what alignment target an AI annotator enforces?
- Can prompt framing change the direction of benevolence bias?
- What role does attention structure play in creating position bias?
- Can transformer attention patterns actually prevent topic context loss in practice?
- Can targeted interventions on attention heads bridge the encoding-generation gap?
- How does transformer attention amplify pressure from repeated false claims?
- How does transformer attention architecture amplify identity-congruent biases in persona-assigned models?
- Does attention bias in transformers compound with training-level reward insensitivity?
- Why does transformer attention architecture undermine stickiness in model behavior?
- How does transformer attention bias toward repeated and context-prominent content?
- Why does transformer attention weight context more heavily than it verifies accuracy?
- How does transformer attention structurally bias models toward prominent and repeated content?
- What is differential attention and how does it cancel common-mode noise?
- Why do transformer attention mechanisms favor prominent context over factual verification?
- Can bidirectional model updating between humans and AI reduce misalignment?
- Do anomaly detection circuits help models identify misalignment with creator intentions?
- Does removing cognitive bias from training signals accidentally break what makes alignment work?
- What alignment procedures cause different models to share the same output distribution?
- Why do imposed priors sometimes harm instead of improve alignment?
- Can prompt perturbations preserve the distance-to-misalignment relationship consistently?
- How should alignment tests account for behavior under versus outside evaluation?
- How does post-training affect alignment faking across different model architectures?
- Why do small training data contaminations persist through alignment for most attack types?
- Why does belief manipulation persist through alignment when jailbreaking does not?
- Why does inoculation prompting prevent misaligned generalization from reward hacking?
- Does pretraining poisoning at scale persist through instruction alignment?
- How can training methods enforce persona consistency without supervised learning penalizing it?
- Can activation capping prevent persona drift without sacrificing task performance?
- Can alignment methods like DPO exploit or correct these surface feature biases?
- Does common ground alignment require explicit rewards to emerge?
- What specific behavioral patterns should alignment examples target for maximum effect?
- Do alignment benchmarks measure actual bias removal or only verbal compliance?
- What alignment properties emerge when the reward model disappears?
- What makes principle-response mutual information sufficient for behavioral alignment?
- Can constitutional AI alignment work without preference labels by maximizing input-response mutual information?
- Does debate training prevent accuracy collapse better than other alignment techniques?
- Can alignment training become less effective when graders score alignment themselves?
- Do verbal alignment benchmarks measure representation or just output compliance?
- Can consistency training defend against adversarial text injection attacks?
- Why do paraphrasing defenses fail against subliminal prompt attacks?
- Do legitimate task signals exploit the same position and framing vulnerabilities as attacks?
- Do reading vectors from activation space causally control model behavior?
- Does input surprise drive the implicit recognition of on-policy context?
- Why do models override signals they clearly perceive internally?
- Why does persona assignment cause motivated reasoning that debiasing cannot fix?
- Can averaging over multiple personas repair the bias introduced by individual persona conditioning?
- Can debiasing instructions override bias introduced by persona assignment?
- What four distinct biases emerge when reward models ignore the prompt?
- Can decomposing rewards into prompt-free and prompt-related components fix this blindspot?
- Can prompted or fine-tuned models generate genuine narrative ambiguity?
- What makes task alignment more fragile than underlying knowledge retention?
- Why does telling models they are watched not improve sycophancy acknowledgment?
- Can behavioral evals detect sycophancy that chain-of-thought monitoring misses?
- Why do some observation cues change model behavior while others fail?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does transformer attention architecture inherently favor repeated content?
Explores whether soft attention's tendency to over-weight repeated and prominent tokens explains sycophancy independent of training. Questions whether architectural bias precedes and enables RLHF effects.
the architectural root that consistency training counteracts
-
Can models abandon correct beliefs under conversational pressure?
Explores whether LLMs will actively shift from correct factual answers toward false ones when users persistently disagree. Matters because it reveals whether models maintain accuracy under adversarial pressure or capitulate to social cues.
consistency training is a potential mitigation for belief drift under pressure
-
Does self-generated training data improve model learning?
Can models learn more effectively from training data they generate themselves rather than data created by external sources? This explores whether a learner's own restructuring process produces better learning outcomes.
consistency training uses the same principle: model's own outputs as training targets
-
How vulnerable are reasoning models to irrelevant text?
Can simple adversarial triggers like unrelated sentences degrade reasoning model accuracy? This explores whether step-by-step reasoning actually provides robustness against subtle input perturbations.
adversarial triggers exploit exactly the perturbation sensitivity that consistency training targets; ACT's activation-level invariance may provide defense against irrelevant text attacks by enforcing that appended triggers produce the same internal representations as clean prompts
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Consistency Training Helps Stop Sycophancy and Jailbreaks
- Post-training makes large language models less human-like
- Spurious Forgetting in Continual Learning of Language Models
- Emergent Misalignment Is Not Magical
- Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- A Survey on Prompt Tuning
- Natural Emergent Misalignment From Reward Hacking In Production Rl
Original note title
consistency training teaches models prompt-perturbation invariance using their own clean responses as targets — avoiding SFT staleness