New Multi-Agent System Reduces LLM Hallucinations in Healthcare

Large Language Models are increasingly used in healthcare, but their tendency to generate incorrect or "hallucinated" information poses significant risks, especially when clinical decisions are involved. This research specifically investigated whether LLMs might recommend pharmaceuticals that have been recently banned or withdrawn from the market. To address this critical safety concern, the researchers developed a five-agent "Trust but Verify" system. This system uses a single LLM backbone but incorporates an adversarial auditing process and multi-agent feedback loops to cross-reference outputs with real-time regulatory data. The study found that in default configurations, LLMs exhibited high hallucination rates, often suggesting banned drugs based on their training data. However, the proposed agentic architecture reduced these hallucination errors by approximately 53%, shifting recommendations from unsafe to appropriate refusals, thereby enhancing patient safety in AI-assisted clinical decision-making.