LLMs Extract Scientific Data Reliably with Self-Prompting and Consensus

Scientific data extraction from research papers is a time-consuming and labor-intensive process. This study investigates how frontier, browser-based large language models (LLMs) can be leveraged to automate this task, particularly for highly contextualized information. The researchers developed and tested four escalating workflows, finding that while LLMs perform well with expert-curated prompts, they can also generate effective prompts themselves. A key finding was that autonomous discovery of literature remains challenging for LLMs, often leading to missed references or hallucinations. However, LLMs proved capable of creating new datasets from published guidelines that closely matched human expert judgments, though a human-in-the-loop is still necessary for final validation. The proposed framework outlines an auditable division of labor: human experts set the evidence standards, LLMs perform repeated extractions and cross-checking, and researchers intervene to resolve any discrepancies. This approach offers a practical path to scale scientific data curation while maintaining expert oversight and ensuring reproducibility.