Multi-Agent LLM Framework Improves Community Note Evaluation on X

A new research initiative addresses the challenges of slow and low-ratio human-based fact-checking on social media platforms, particularly concerning community notes. The team first compiled ComRate, an extensive dataset comprising 2.5 million community notes and over 209 million associated ratings from the platform X (formerly Twitter). This dataset serves as a foundation for developing more efficient evaluation methods. Building on this, the researchers propose MultiCom, a novel framework that uses persona-guided multi-agent large language models (LLMs) to evaluate community notes. MultiCom simulates a diverse population of raters by clustering real contributors in a matrix-factorized rater space. These persona agents are then prompted to generate structured and explainable assessments of community notes, adhering to the official rating schema and providing details like confidence levels, agreement signals, and underlying reasons. An advanced, out-of-fold calibrated aggregation algorithm processes these agent-generated judgments, combining raw votes with diagnostic reason signals to produce highly reliable predictions. Extensive evaluations demonstrate that MultiCom significantly surpasses existing methods, achieving an average accuracy of 84.7% (with a balanced accuracy of 68.3% and macro-F1 of 60.1%) on the evaluation set. This marks a substantial improvement in the automated assessment of community-based fact-checks.