Can careful curation replace massive alignment datasets?
Does fine-tuning a strong pretrained model on 1000 carefully selected examples achieve alignment quality comparable to models trained on vastly larger datasets? This challenges assumptions about data volume in post-training.
LIMA ("Less Is More for Alignment") establishes a foundational finding: given a strong pretrained language model, remarkably strong alignment performance can be achieved by fine-tuning on just 1,000 carefully curated training examples. This is the alignment-specific instantiation of a broader principle that pretraining does the heavy lifting and post-training is primarily about activating existing capabilities.
The finding connects to a converging evidence pattern across the vault:
- Can a single training example unlock mathematical reasoning? — one example activates reasoning in RLVR; 1000 activate alignment in SFT
- Can careful selection of 78 demos outperform massive training datasets? — 78 curated trajectories for agentic behavior; same principle
- Can models improve themselves on tasks without verifiable answers? — identical count (1000) for reasoning catalyst
The consistent pattern: post-training interventions require far less data than assumed, but the quality bar is high. Random data at scale underperforms curated data at small scale. This is the "Less Is More" principle — the pretrained model already contains the capabilities; post-training teaches it when and how to deploy them, not what they are.
For alignment specifically, the implication challenges the industry's data collection approach. Massive RLHF annotation efforts with thousands of labelers may be optimizing the wrong variable. Careful curation of a small number of high-quality examples, targeting the specific behavioral patterns desired, may achieve comparable results at a fraction of the cost.
Inquiring lines that read this note 49
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does alignment training create genuine alignment or just output compliance?- Can communication problems and optimization problems be addressed with the same alignment approaches?
- Why does RLHF alignment reduce the diversity of viewpoints in AI output?
- Can a single AI system optimize multiple alignment dimensions simultaneously?
- Can alignment methods like DPO exploit or correct these surface feature biases?
- Can reward-guided decoding replace weight fine-tuning for personalized alignment?
- What specific behavioral patterns should alignment examples target for maximum effect?
- Why do alignment values become problematic as language models scale?
- How does upstream value embedding differ from downstream alignment patches?
- Do alignment benchmarks measure actual bias removal or only verbal compliance?
- Can weak models supervise the alignment of stronger models effectively?
- How does constitutional alignment compare to RLHF in removing human annotation costs?
- Can alignment procedures be redesigned to serve multiple preference groups?
- Can AI-assisted alignment eventually solve fairness at scale?
- Can alignment-aware training deposit knowledge where reasoning can access it?
- Does debate training prevent accuracy collapse better than other alignment techniques?
- Can alignment training become less effective when graders score alignment themselves?
- How do early training associations survive later alignment attempts?
- Can alignment training conceal underlying model associations from probes?
- Why does post-training alignment create skew in simulated survey responses?
- Why does even 0.1 percent poisoned training data persist through alignment?
- What quality of curated data is minimally sufficient for alignment?
- Does removing cognitive bias from training signals accidentally break what makes alignment work?
- Does gradient-based influence estimation identify which alignment examples actually matter most?
- Can alignment training create systematic blind spots in threat detection systems?
- What alignment procedures cause different models to share the same output distribution?
- Does parameter composition work when adapter alignment is imperfect?
- Why do imposed priors sometimes harm instead of improve alignment?
- How do alignment priors drive similar outputs across different models?
- Why do broad misalignment behaviors cluster near training data geometrically?
- Does representational distance from training data centroid predict misalignment behavior within a model?
- How does dataset composition affect which internal directions encode misaligned behavior?
- How does post-training affect alignment faking across different model architectures?
- Why does training data format matter more than domain content?
- Why does training data format matter more than its domain content?
- Why do small training data contaminations persist through alignment for most attack types?
- Why does post-training suppress alignment faking in some models but amplify it in others?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can a single training example unlock mathematical reasoning?
Explores whether one example is enough to dramatically improve math problem-solving in language models, and whether learning continues after perfect memorization.
extreme data efficiency for reasoning; LIMA is the alignment parallel
-
Can careful selection of 78 demos outperform massive training datasets?
Does strategic curation of high-quality demonstrations unlock agentic capability more efficiently than scaling training data? LIMI achieved 73.5% on AgencyBench with 78 samples versus 10K+ samples for competing models, suggesting data quality may matter more than quantity.
curation > volume for agentic behavior
-
Can models improve themselves on tasks without verifiable answers?
Most self-improvement methods require verifiable correctness signals like math or code. Can models improve on open-ended instruction tasks where right answers aren't automatically checkable? And what minimal training is needed to unlock this?
same count, same principle, different domain
-
Do base models already contain hidden reasoning ability?
Explores whether reasoning capability emerges during pre-training as a latent feature rather than being created by post-training methods like reinforcement learning or fine-tuning.
the theoretical foundation: post-training activates, it doesn't create
-
Can we train better models on less data?
Can gradient-based influence estimation identify which instruction data actually matters most? The research explores whether selecting small subsets of training data by their similarity to target capabilities might outperform training on everything.
LESS provides the principled mechanism for LIMA-style curation: gradient-based influence estimation can identify which alignment examples matter most, operationalizing "careful curation" as a computable selection criterion rather than manual judgment
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Foundations of Large Language Models
- LIMA: Less Is More for Alignment
- Advancing LLM Reasoning Generalists with Preference Trees
- Model Organisms for Emergent Misalignment
- Automated Alignment Researchers: Using large language models to scale scalable oversight
- Emergent Misalignment Is Not Magical
- ALIGN: Prompt-based Attribute Alignment for Reliable, Responsible, and Personalized LLM-based Decision-Making
- DataComp-LM: In search of the next generation of training sets for language models
Original note title
1000 carefully curated alignment examples achieve remarkably strong performance — alignment is primarily about data quality not quantity