MatrAIx: Simulating the World with 8.3 Billion Persona Agents

Abstract Human evaluation of Artificial Intelligence (AI) systems and digital products is costly, slow, and difficult to scale. Offline evaluations are more scalable but often abstract away human diversity and interactive behavior. We therefore introduce MatrAIx, a population-scale simulated-user evaluation infrastructure for testing AI systems and digital products with heterogeneous users. MatrAIx has three core components: First, Persona 8B contains 8.3 billion persona records represented through a schema of 1,290 categorical dimensions. Records are either sampled from a dependency graph that preserves correlated attributes or derived from human-authored profiles. We release a quality-filtered coreset of approximately 1 million personas, comprising 599,847 human-grounded and 400,000 synthetic records. Second, the MatrAIx Playground provides four environments in which diverse users evaluate and interact with digital products: Survey, AI Chatbot, Web, and App. Third, MatrAIx provides 1,010 application tasks spanning more than 25 domains, including Commerce, Software, Finance, and Healthcare. We conducted 18,189 evaluation trials across eight representative tasks. Persona agents were powered by three large language models: Claude Opus 4.8, GPT 5.5, and Claude Haiku 4.5.

Introduction. Human evaluation remains essential for understanding how AI systems and digital products perform for real users. However, its time and expense limit the breadth and frequency of studies during development. Offline benchmarks offer a scalable and reproducible alternative. However, they typically measure task outcomes without modeling how diverse users formulate requests, interact with a system (Chang et al., 2025), and judge its results (Santurkar et al., 2023; Kirk et al., 2024). For example, a coding-agent benchmark may test whether the trajectory passes all unit tests (Jimenez et al., 2024; Miserendino et al., 2025; Zan et al., 2025; Zhang et al., 2025). This establishes functional correctness, but it does not capture user needs or preferences. A novice may want explanations, small edits, and frequent confirmation. An expert may instead prefer terse responses, broader refactoring, and greater autonomy. Some users provide detailed specifications and inspect every change. Others begin with underspecified goals and expect the agent to ask clarifying questions.