Apollo's Oracle: Retrieval-Augmented Reasoning in Multi-Agent Debates

Multi-agent debate systems are designed to derive accurate and consistent conclusions through adversarial interactions among agents. However, these systems often encounter challenges due to cognitive constraints, manifesting as (1) agents’ obstinate adherence to incorrect viewpoints and (2) their propensity to abandon correct viewpoints. These issues are primarily responsible for the ineffectiveness of such debates. Addressing the challenge of cognitive constraints, we introduce a novel framework, the Multi-Agent Debate with Retrieval Augmented (MADRA). MADRA incorporates retrieval of prior knowledge into the debate process, effectively breaking cognitive constraints and enhancing the agents’ reasoning capabilities. Furthermore, we have developed a self-selection module within this framework, enabling agents to autonomously select pertinent evidence, thereby minimizing the impact of irrelevant or noisy data. We have comprehensively tested and analyzed MADRA across six diverse datasets. The experimental results demonstrate that our approach significantly enhances performance across various tasks, proving the effectiveness of our proposed method.1

Introduction. Large Language Models (LLMs) have demonstrated remarkable performance in a variety of Natural Language Processing (NLP) tasks (OpenAI, 2023; Touvron et al., 2023; Pu et al., 2023). This success is largely attributed to their emergent capabilities (Wei et al., 2022a), drawing significant academic and industry attention. However, LLMs face critical challenges, notably the issue of hallucinations (Huang et al., 2023), which impede their ability to deliver advanced human-centric services. To alleviate this issue, extensive research has been

Discussion / Conclusion. In this paper, we propose a multi-agent debate framework based on retrieving augmented. This framework effectively alleviates cognitive constraints in the debate process by introducing externally retrieved prior knowledge. In addition, we build an external retrieval evidence pool to allow the agent to choose evidence that is helpful to itself, effectively avoiding the impact of noisy evidence. We conducted sufficient experiments on six datasets with different tasks to demonstrate the effectiveness of our method. In the future, we will further explore real-time knowledge retrieval and effective knowledge selection methods during debates.