VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences

Personalized Large Language Models (PLLMs) aim to tailor responses to individual users, where a central challenge is preference reasoning: inferring query-relevant preferences from user-related history. Existing benchmarks, however, largely assume that such preference can be retrieved from semantically related history. We study an underexplored but practically important regime, profile-preference conceptual misalignment (PRCM), where observable profile cues and query-specific preferences lie in different concept spaces, making semantic retrieval inconsistent for personalization. We introduce VIBE-BENCH1, a benchmark with two psychology-grounded tasks, 3,504 personas and 12,239 dialogues, including a manually verified gold test set, and requires cross-concept preference reasoning beyond surface semantic overlap. Experiments with several personalization methods show that current PLLMs largely rely on shallow semantic correlations and fail to acquire robust cross-concept mappings. These findings establish PRCM as a distinct failure regime in PLLMs and position VIBE-BENCH as a focused testbed for advancing preference reasoning beyond semantic matching.

Introduction. Large Language Models (LLMs) have enabled general-purpose systems across diverse NLP tasks (Touvron et al., 2023; Zhang et al., 2025a), yet they are predominantly trained under a one-sizefits-all paradigm that limits adaptation to individual users (Wu et al., 2021; Zhao et al., 2025b). To address this limitation, Personalized Large Language Models (PLLMs) leverage user-specific data, e.g., interaction histories, to provide one-size-fits-one responses (Liu et al., 2025). Recently, numerous benchmarks have been proposed to evaluate how well PLLMs align with user preferences (Jiang et al., 2025a; Ong et al., 2025; Au et al., 2025). A central challenge in PLLMs is preference reasoning: inferring, from a user’s historical data, which preferences should govern the response to the current query (Zhang et al., 2025b). Existing personalization methods largely approach this problem through semantic matching between the query and user history.

Discussion / Conclusion. Following Section 3, our main experiments adopt a weakly supervised setting where models observe only user histories, current queries, and end-task supervision. Models must therefore identify taskrelevant concepts, infer user attributes along these dimensions, and induce the corresponding mappings. We adopt this setting because interaction data are scalable to collect, whereas concept and mapping annotations are costly, domain-specific, and often unavailable. In open-domain personalization, the relevant concept space may be unknown in advance. Under this setting, we evaluate five scalable, general-purpose baselines in Table 2. When the relevant concept ontology and crossconcept knowledge are explicitly available, however, Concept-Aware Reasoning substantially improves performance. We therefore conduct an additional experiment using persona-card labels to construct template-based Chain-of-Thought (CoT) rationales (Wei et al., 2022) that guide models through concept inference and cross-concept mapping before answering (Figure 4).