LLMs Predict Failure Risk, Struggle with Optimal Collaboration Protocol Selection.

Research into multi-agent Large Language Model (LLM) systems reveals that while these models can significantly improve reasoning capabilities by employing more computation, optimizing this computational expenditure remains a challenge. The study tested various collaboration protocols—direct solving, iterative self-correction, planner-executor-reviewer, and multi-agent deliberation—on complex math problems and other benchmarks. It found that LLMs are quite adept at predicting when a direct solving attempt is likely to fail, achieving a high AUROC score for failure prediction. However, the same LLMs show much weaker performance in predicting which specific collaboration protocol would yield the best outcome for a given problem. This inability to accurately route problems to the most cost-effective collaboration strategy means that current systems often either under-escalate, missing opportunities for improvement, or over-escalate, incurring unnecessary computational costs. The findings suggest that while confidence scores can guide initial escalation decisions, the nuanced selection of a protocol based on its specific value remains an open problem for cost-aware routing.