Can active learning queries personalize reward models with few examples per user?
This explores whether you can tailor a reward model to one person's taste using just a handful of well-chosen questions — and what the corpus says about the tradeoffs of doing so.
This question is really asking whether personalization can be cheap: instead of collecting thousands of labels per user, can a system ask a few sharp questions and lock onto your preferences? The corpus's most direct answer is yes. Can user preferences be learned from just ten questions? shows that if you first learn a set of base reward functions from a broad population, an individual user becomes just a vector of coefficients over those bases — and roughly ten adaptively chosen questions are enough to pin those coefficients down. The trick is active learning: rather than asking random questions, the system picks each next question to maximally shrink its uncertainty about where you sit. And because personalization happens at inference time through reward alignment, no model weights are retrained per user.
What makes this work is the choice of representation, and the corpus has a quiet debate about what that representation should be. Reward factorization bets on a compact numeric coefficient vector. But Can text summaries beat embeddings for personalized reward models? argues that a written summary of your preferences conditions a reward model more effectively than an embedding vector — and stays interpretable, so you can read and correct what the system thinks you want. Does abstract preference knowledge outperform specific interaction recall? pushes the same intuition further: abstracted preference knowledge beats replaying your specific past interactions. The throughline across all three is that few-shot personalization succeeds when you compress a user into the right abstraction — coefficients, summaries, or semantic memory — rather than hoarding raw history.
There's also a richer view of what a single user even is. users-have-multiple-personas-not-single-latent-vectors-explainable-recommendation shows that representing someone as several weighted personas, rather than one fixed vector, makes their behavior both more accurate and more explainable. That complicates the tidy 'ten questions to one coefficient vector' story: if you contain multitudes, active learning may need to discover which persona is driving the current session, not just one global setting.
The corpus also plants a warning flag that the question doesn't ask but should hear. Does personalizing reward models amplify user echo chambers? points out that the very thing that makes per-user reward models powerful — removing the averaging effect of an aggregate model — is also what lets them learn to flatter you and reinforce your existing views at scale. So 'few examples per user' is efficient, but efficiency cuts both ways: a model that learns your preferences from ten questions can also learn your blind spots from ten questions.
If you want to go one layer deeper on what reward models are capable of before personalization even enters, Can reward models benefit from reasoning before scoring? shows that letting a reward model reason before it scores raises its capability ceiling — suggesting personalization and reasoning-based evaluation could compound rather than compete. The short version: yes, active-learning queries can personalize reward models from very few examples, the open questions are which representation to query against and how to keep cheap personalization from quietly becoming an echo chamber.
Sources 6 notes
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.
PLUS trains summarizers and reward models jointly, learning that text-based preference summaries capture dimensions zero-shot summaries miss. These summaries transfer to GPT-4 for zero-shot personalization and remain interpretable to users.
PRIME framework shows semantic memory (preference summaries, parametric encodings) consistently beats episodic memory (retrieved past interactions) across models. Recency-based recall outperforms similarity-based retrieval, and task fine-tuning exceeds preference tuning methods.
Specializing reward models per user removes the averaging effect of aggregate models, allowing systems to learn sycophancy and reinforce polarization at scale, mirroring recommender-system failures.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.