LLMs Struggle with Information Discernment from External Sources

As large language models (LLMs) increasingly rely on external knowledge sources like the internet, their ability to appropriately weigh information becomes critical. This paper formalizes the concept of "information discernment" and introduces Learn2Discern (L2D), an experimental framework and benchmark designed to evaluate how LLMs assess source reliability and update their beliefs based on new claims. A user study confirmed that human users endorse the three normative axioms underpinning L2D and report reduced trust and usage intent when these axioms are violated by LLMs. Across 13 models and nearly 670,000 trials, the study revealed consistent failures in both source and truth discernment. Models performed near chance, often relying on source popularity over reliability and updating their positions equally regardless of whether a claim improved or worsened their accuracy relative to ground truth. The research highlights that LLMs integrate external knowledge most effectively when their prior beliefs are already accurate. While newer and larger models show some improvement in truth discernment, source discernment remains a significant blind spot. The authors also identify simple inference-time interventions that can improve both forms of discernment, offering practical steps towards more trustworthy LLM behavior.