Learnijoy
ResearchAI Engineering & DevToolsAI Research

Self-Consistency Hurts Small LLMs on Hard Science Problems

Utkarsh Bahuguna· August 13, 2026 View original

Key takeaways

  • Self-consistency via majority vote can decrease accuracy for small LLMs on hard science problems.
  • Model confidence does not reliably track correctness on these challenging tasks.
  • The technique may not be a universal solution for improving LLM performance.
  • Rigorous benchmarking is crucial before deploying self-consistency methods.

Who benefits

AI DevelopmentScientific ResearchEducation TechnologyQuality AssuranceContent Moderation

Summary

A study found that self-consistency via majority vote, a common inference-time technique, actually reduces accuracy for small instruction-tuned LLMs on a majority of hard science problems from the GPQA Diamond benchmark. This is because confidence does not reliably track correctness for these models on such tasks.

A new study reveals a significant drawback of self-consistency (SC) via majority vote, a popular method to improve LLM output quality by sampling multiple chains of thought and selecting the plurality answer. On the challenging GPQA Diamond benchmark, which consists of graduate-level science questions, this technique was found to *reduce* per-problem accuracy for a majority of problems when applied to small instruction-tuned models like Qwen2.5-7B and Llama-3-8B. The research, which pre-registered and confirmed its hypotheses, showed that for Qwen2.5-7B, SC hurt 56.6% of problems, and for Llama-3-8B, it hurt 65.7%. The core reason identified is that these models' confidence in their answers does not correlate with correctness on these difficult problems. Even in the highest-agreement bins, the plurality answer was correct only about half the time for Qwen, and for Llama, the highest-agreement bin was less accurate than the lowest. This suggests that for certain types of hard problems and smaller models, simply generating more answers and voting does not improve reliability.

Why it matters

Professionals deploying or developing AI systems need to be aware that common techniques like self-consistency are not universally beneficial and can degrade performance on challenging tasks, especially with smaller models.

How to implement this in your domain

  1. 1Rethink the default application of self-consistency (majority vote) for small LLMs on complex, knowledge-intensive tasks.
  2. 2Benchmark self-consistency methods rigorously on your specific problem domains, especially for hard science or reasoning tasks.
  3. 3Explore alternative or more sophisticated verification methods beyond simple majority voting if using small LLMs for critical applications.
  4. 4Consider the trade-off between inference compute spent on self-consistency and the actual accuracy gains (or losses).
  5. 5Investigate if reasoning-native models behave differently, as suggested by the research.

Original post by Utkarsh Bahuguna

"arXiv:2608.11403v1 Announce Type: new Abstract: Self-consistency (SC) via majority vote is a widely used way to spend inference-time compute: sample N chains of thought, return the plurality answer. On the full GPQA Diamond benchmark (198 graduate-level science questions), majori…"

View on X

Originally posted by Utkarsh Bahuguna on X · view source

Want to go deeper?

Turn these trends into skills with Learnijoy's hands-on AI & tech courses.

Explore courses

More in AI Engineering & DevTools

AI Engineering & DevTools

Zapier vs. Tray: Enterprise Automation Platform Comparison for 2026

This post compares Zapier and Tray.io, evaluating which platform is better suited for enterprise automation needs by balancing power and ease of use. It argues that the best tools scale for complex requirements while remaining intuitive for all users.

Ryan KaneJul 28, 2027
AI ResearchAI Engineering & DevTools

Task-Vector Interference in Merged LLMs Driven by Orientation, Not Magnitude.

This research reveals that interference in merged language models, often attributed to magnitude, is primarily driven by the orientation of task-vectors. It demonstrates that erasing interference along specific directions causally removes its effects, while magnitude-based interventions are insufficient and inconsistent.

Chencheng ZhuAug 13, 2026
AI Engineering & DevToolsAI Research

New Method Detects Gradual GNSS Spoofing in Autonomous Driving.

This paper proposes a causal high-order liquid evidence framework to detect gradual GNSS spoofing attacks in autonomous driving. By modeling the evolution of GNSS-motion inconsistency with multiple evidence streams and adaptive liquid encoders, the method achieves high F1-scores in detecting subtle spoofing.

Muhammad Ayub Sabir, Junbiao Pang, Fatima AshrafAug 13, 2026