CloChat: Understanding How People Customize, Interact, and Experience Personas in Large Language Models

Large language models (LLMs) have facilitated significant strides in generating conversational agents, enabling seamless, contextually relevant dialogues across diverse topics. However, the existing LLM-driven conversational agents have fixed personalities and functionalities, limiting their adaptability to individual user needs. Creating personalized agent personas with distinct expertise or traits can address this issue. Nonetheless, we lack knowledge of how people customize and interact with agent personas. In this research, we investigated how users customize agent personas and their impact on interaction quality, diversity, and dynamics. To this end, we developed CloChat, an interface supporting easy and accurate customization of agent personas in LLMs. We conducted a study comparing how participants interact with CloChat and ChatGPT. The results indicate that participants formed emotional bonds with the customized agents, engaged in more dynamic dialogues, and showed interest in sustaining interactions. These findings contribute to design implications for future systems with conversational agents using LLMs.

Introduction. Large language models (LLMs) have revolutionized the fields of natural language processing (NLP) and conversational agent (CA) [72]. Models such as OpenAI’s GPT series and Google’s BERT have shown remarkable proficiency in generating text that is both coherent and contextually relevant, finding applications in sectors including healthcare [30, 93], education [95], and commerce [58]. Notably, LLM-based conversational agents like ChatGPT [3] and Google’s Bard [2] have demonstrated an impressive ability to engage in naturalistic dialogues across various contexts [80]. These models have garnered global recognition and interest from both academic and industrial sectors, becoming widely used by the general public for everyday applications. However, despite their increasing popularity and vast potential, most existing LLM-based conversational agents are typically generic, limiting their adaptability to the diverse preferences and needs of users [13].

Discussion / Conclusion. Our user study was aimed at investigating the impact of agent persona customization on user experience during interactions with LLM-based conversational agents, as opposed to conventional generic conversational agents (RQ1). We discovered that the customization of agent personas significantly boosts user engagement, trust, and emotional connection, offering a noticeable improvement in maintaining user satisfaction and engagement compared to ChatGPT. In addressing RQ2, we delved into the ways users customize their agent personas and the resultant effects on their interactions. We observed that conversations involving customized agent personas tend to be richer and more diverse. Users often align the traits of agent personas in terms of both visual elements and real-world inspirations, which additionally brings to light ethical considerations regarding agent persona customization. In extending our discussions on these findings, we explore relevant topics and present practical implications for the design of user interfaces employing LLM-based conversational agents.