SocraSynth: Multi-LLM Reasoning with Conditional Statistics

Large language models (LLMs), while promising, face criticisms for biases, hallucinations, and a lack of reasoning capability. This paper introduces SocraSynth, a multi-LLM agent reasoning platform developed to mitigate these issues. SocraSynth utilizes conditional statistics and systematic context enhancement through continuous arguments, alongside adjustable debate contentiousness levels. The platform typically involves a human moderator and two LLM agents representing opposing viewpoints on a given subject. SocraSynth operates in two main phases: knowledge generation and reasoning evaluation. In the knowledge generation phase, the moderator defines the debate topic and contentiousness level, prompting the agents to formulate supporting arguments for their respective stances. The reasoning evaluation phase then employs Socratic reasoning and formal logic principles to appraise the quality of the arguments presented. The dialogue concludes with the moderator adjusting the contentiousness from confrontational to collaborative, gathering final, conciliatory remarks to aid in human reasoning and decision-making.

Introduction. Revolutionary advancements in large language models (LLMs) [10, 32, 42–44] and, more broadly, foundation models (FMs) [7] have paved the way for multi-agent systems to achieve remarkable progress in knowledge acquisition and natural language understanding [55]. As highlighted by [10, 11, 33], models such as GPT-4 exhibit information processing qualities surpassing human capabilities, including: 1) deep and extensive knowledge, 2) interdisciplinary assimilation and fusion of knowledge, and 3) multimodal and multilingual expertise. While promising, LLMs face criticisms for biases, hallucinations, and a lack of reasoning capability [23]. To mitigate this disparity, we introduce SocraSynth, a pioneering platform blending the principles of “Socratic Synthesis” and “Socratic Symposium.” It fosters collaboration between humans and LLM agents, enabling the formation of deep questions and transcending typical human constraints in reasoning, validation, and assessment. A typical SocraSynth ensemble comprises a human moderator paired with two LLM agents, each espousing divergent perspectives.

Discussion / Conclusion. Reflecting on LLM developments, we developed SocraSynth, a platform designed to utilize the extensive knowledge of LLMs. This innovative multi-agent system reveals insights beyond the scope of traditional human cognition by leveraging LLMs’ vast knowledge and interdisciplinary reasoning capabilities. SocraSynth facilitates enhanced debates and reasoning through the novel use of contentiousness, which modulates debate tone, language, and emphasis, combined with conditional statistics and Socratic methods to mitigate biases and hallucinations. In contrast to other works, SocraSynth minimizes human intervention in modeling reasoning, aligning with the perspective of several AI experts on the limitations of heuristic approaches like chain of thoughts. SocraSynth underscores the importance of human moderation and evaluation, particularly in introducing adversarial conditions and contentiousness to reduce biases and hallucinations, but does not explicitly model reasoning outside LLMs.