New Framework Improves LLM Fact-Checking with Selective Abstention

A new framework called Evidence Chain Evaluation (ECE) has been developed to enhance the reliability of fact-checking performed by large language models. While LLMs can achieve high accuracy in fact-checking, they often struggle with overconfidence, issuing definitive verdicts even when the supporting evidence is insufficient or contradictory. ECE addresses this by enabling the system to selectively abstain from making a binary true/false judgment, instead returning an "uncertain" verdict. The ECE system operates as a tool-using agent, actively gathering evidence through web searches, scholarly databases, and executable checks. It then processes this information to produce a structured verdict that includes a confidence score and metadata about the sources. On a dedicated benchmark, ECE demonstrated high accuracy on claims it chose to answer, while effectively deferring cases with lower-reliability evidence, thus functioning as a safety mechanism for epistemically weak information.