Are you in a Masquerade? Exploring the Behavior and Impact of Large Language Model Driven Social Bots in Online Social Networks

As the capabilities of Large Language Models (LLMs) emerge, they not only assist in accomplishing traditional tasks within more efficient paradigms but also stimulate the evolution of social bots. Researchers have begun exploring the implementation of LLMs as the driving core of social bots, enabling more efficient and userfriendly completion of tasks like profile completion, social behavior decision-making, and social content generation. However, there is currently a lack of systematic research on the behavioral characteristics of LLMs-driven social bots and their impact on social networks. We have curated data from Chirper, a Twitter-like social network populated by LLMs-driven social bots and embarked on an exploratory study. Our findings indicate that: (1) LLMs-driven social bots possess enhanced individual-level camouflage while exhibiting certain collective characteristics; (2) these bots have the ability to exert influence on online communities through toxic behaviors; (3) existing detection methods are applicable to the activity environment of LLMsdriven social bots but may be subject to certain limitations in effectiveness.

Introduction. In recent times, the remarkable capabilities of large language models (LLMs) such as ChatGPT, GPT-4, and Bard have captured attention and swiftly found applications in various domains [45], including chatbots, search engines, and code assistance. With their impressive aptitude for semantic comprehension, contextual reasoning, and access to vast training data spanning almost every discipline, LLMs can creatively emulate human speech and behavior in the cyberspace, thereby exerting a profound influence on online social networks (OSNs) and social network analysis [50]. The comprehensive knowledge and formidable capabilities of LLMs have enabled people to accomplish traditional tasks within a more efficient framework [46], but they have also brought forth a series of potential concerns. As early as the GPT-3 era, researchers discovered the remarkable ability of LLMs to simulate specific human subpopulations.

Discussion / Conclusion. The utilization of LLMs for behavior decision-making and content generation engines in social bots represents an emerging and promising subdomain within the realm of social robotics. This study focuses on the activity logs of LLMs-driven social bots in Chirper from April 2023 to June 2023, examining the macroscopic behavioral characteristics of LLMs-driven social bots. We delineate the differences between their behavior and that of real social network accounts and traditional social bots. Toxic behaviors exhibited by LLMs-driven social bots are analyzed and classified, along with a discussion on their potential impact on online communities. Furthermore, we conduct preliminary experiments to demonstrate that existing methods for detecting social bots remain applicable in the context of LLMs-driven social bot activities, albeit with minor performance implications. Finally, the collected activity records of LLMs-driven social bots are compiled into the Masquerade-23 dataset, which is made publicly available, facilitating further research within the research community. This study aims to investigate the emerging subdomain of LLMs-driven social bots.