LLMs Rank Hypotheses Better with Intrinsic Confidence Scores

As large language models (LLMs) become more adept at generating scientific hypotheses, the challenge shifts to reliably evaluating their quality. Current methods often involve using LLMs as judges or relying on semantic similarity, which can inadvertently favor familiar ideas over genuinely novel ones. This research proposes an alternative: a logit-based energy scoring method that assesses hypotheses based on an LLM's intrinsic confidence. The new method was benchmarked against prompted listwise ranking across 1,323 scientific papers from 12 disciplines, each with a correct hypothesis and fifteen incorrect alternatives. The intrinsic scoring approach achieved a Hit@1 rate of 33.0%, significantly outperforming the 16.6% of prompted ranking. A specific configuration, a 1-billion-parameter model using this logit-based scoring, reached 53.1% accuracy. This study suggests that leveraging an LLM's internal confidence mechanisms holds considerable promise for evaluating scientific hypotheses more effectively. It also opens avenues for further research into confidence-based methods to enhance the trustworthiness of AI-enabled scientific discovery workflows.