LLMs Struggle with Evidence Calibration in Scientific Briefings

Researchers have introduced CalBrief, a novel diagnostic benchmark designed to assess how well large language models (LLMs) can produce scientific briefings that accurately reflect the strength and scope of supporting evidence. This benchmark provides 16 scientific evidence packages and 96 human-verified takeaways, using an auditable framework to pinpoint where LLM briefing capabilities falter. The study revealed that while structured organization can improve an LLM's reasoning about roles and gaps in evidence, an explicit policy for strength calibration often leads to overly conservative assessments. This conservatism is largely attributed to expanding the label space for evidence strength, rather than issues with gap/scope signal injection or the pipeline policy itself. The findings suggest that current LLMs struggle with nuanced evidence interpretation, highlighting a tension between judging strength and organizing evidence.