Can aggregate reward models satisfy genuinely disagreeing users?
When users have conflicting preferences, do aggregate reward models face an impossible choice between satisfying majorities or sampling proportionally? What does this reveal about RLHF deployment?
Does preference data need more raters than examples? Does personalizing reward models amplify user echo chambers? Can user preferences be learned from just ten questions? Are RLHF annotations actually measuring genuine human preferences? Can text summaries beat embeddings for personalized reward models?
PAC bounds for personalized rewards depend on rater count and example count Personalized reward models amplify sycophancy and echo chambers without ethical safeguards Ten adaptive questions suffice to infer personalized reward coefficients RLHF systematically models elicitation artifacts as human values Text summaries condition reward models more effectively than embedding vectors
How does RLHF reward structure incentivize agreement over accuracy? Does user preference for confirmation override model capability for disagreement? How can consistency across measurement conditions identify genuine versus constructed preferences? Does majority voting reliably signal correctness without risking reward hacking? What distinguishes actual social disagreement from distributional uncertainty in LLM outputs? How do guardrails vary their refusal rates based on user demographics? Why do standard preference alignment methods fail at the individual user level? Do high-disagreement items signal contested values or measurement noise? Can reward models be personalized if annotators lack stable preferences? What consistency tests could distinguish constructed from genuine preferences? What preference dimensions do base reward functions typically capture? Can proper scoring rules fix RLVR's degradation on disagreement prediction?See all 34 inquiring lines on this note →