LLMs Resolve Knowledge Conflicts with Multi-Agent Reasoning Framework

Large Language Models often struggle when their internal knowledge conflicts with external information provided in prompts, or when multiple external contexts contradict each other. Current solutions typically prioritize one source, assuming its reliability, rather than actively resolving the discrepancies. This new research introduces MACR, a framework designed to explicitly address these knowledge conflicts. MACR employs an adaptive knowledge assessment method, using a modified semantic entropy to gauge an LLM's confidence in its answers. Based on this confidence, it either externalizes the model's internal knowledge or retrieves external information. A multi-agent reasoning framework then takes over, with specialized agents that identify rules, analyze conflicts, and resolve inconsistencies across all available knowledge sources. Empirical results indicate that MACR significantly outperforms current state-of-the-art baselines, offering more interpretable resolutions to explicit knowledge conflicts within LLMs.