How does LLM alignment affect representation across dialects?
When we align language models to specific preferences through RLHF or DPO, do these procedures inadvertently create disparities across English dialects and global opinions? Understanding alignment's unintended effects on representation matters for equitable AI.
Alignment evaluations focus on instruction-following, reasoning, and truthfulness — but human preferences are not universal, and aligning to a specific preference set has unintended effects. Studying alignment's impact along three axes of global representation — English dialects, multilingualism, and opinions from and about countries worldwide — this work finds that current alignment procedures create disparities between English dialects and global opinions (while, notably, improving capabilities in several languages). Because developers have high control over alignment variables (who gives feedback, which prompts are in-domain) — unlike the diffuse pretraining distribution — these disparities are design choices, not inevitabilities.
The keeper and the recommendation: alignment is not a one-size-fits-all solution — different groups are affected differently — so transparency must cover the entire alignment pipeline (annotator demographics, in-domain task choices), not just the final model. The InstructGPT paper reported annotator demographics; most preference datasets since have not.
This is the representation-equity face of the vault's diverse-preferences thread. It is the empirical companion to the impossibility/aggregation arguments in Can a single reward model represent diverse human preferences? and Can aggregate reward models satisfy genuinely disagreeing users? — showing the predicted disparities actually appear across dialects and global opinions, and connecting to Do LLMs represent low-resource cultures through dominant cultural proxies?.
Inquiring lines that use this note as a source 6
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- How does constitutional alignment compare to RLHF in removing human annotation costs?
- Can alignment procedures be redesigned to serve multiple preference groups?
- Can AI-assisted alignment eventually solve fairness at scale?
- Does a single LLM judge capture diverse human preferences in alignment training?
- What biases might an LLM judge introduce into an on-policy alignment process?
- How much does preference data freshness matter compared to data source in DPO?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can a single reward model represent diverse human preferences?
Standard RLHF assumes one shared preference signal. But what happens when human values genuinely conflict? This question explores whether aggregating preferences into one model fundamentally fails at fairness.
the theory; this shows the predicted disparities empirically across dialects and global opinions
-
Can aggregate reward models satisfy genuinely disagreeing users?
When users have conflicting preferences, do aggregate reward models face an impossible choice between satisfying majorities or sampling proportionally? What does this reveal about RLHF deployment?
same one-size-fits-all failure, here measured on global representation
-
Do LLMs represent low-resource cultures through dominant cultural proxies?
Explores whether language models internally represent cultures from data-poor regions by routing through high-resource cultural proxies rather than learning independent representations, and what this reveals about cultural bias in model architecture.
alignment-stage disparities compound pretraining-stage cultural flattening
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Unintended Impacts of LLM Alignment on Global Representation
- Learning Pluralistic User Preferences through Reinforcement Learning Fine-tuned Summaries
- ALIGN: Prompt-based Attribute Alignment for Reliable, Responsible, and Personalized LLM-based Decision-Making
- Direct Language Model Alignment from Online AI Feedback
- MaxMin-RLHF: Alignment with Diverse Human Preferences
- Can LLM be a Personalized Judge?
- Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity
- Advancing LLM Reasoning Generalists with Preference Trees
Original note title
alignment is not one-size-fits-all — RLHF and DPO create disparities across English dialects and global opinions