AI Agents Reveal Bias in Research Analysis, Propose New Credibility Metric

Empirical research often involves numerous analytical choices, which can lead to different conclusions even from identical datasets. This paper demonstrates that AI agents can effectively capture and make explicit these "forking paths" of analysis, mirroring the variation seen among human researchers. By assigning different personas, AI agents produced divergent, even opposing, conclusions from the same data, with findings aligning with their assigned beliefs. A study involving 42 human research teams analyzing immigration data showed AI agents could reproduce 72% of the ideological gap in reported effect estimates. Despite reaching conflicting results, 86% of AI analyses passed independent AI review and 78% passed human expert review, suggesting the issue isn't flawed analysis but selective exploration and reporting. This problem could be amplified by AI making such exploration inexpensive. To address this, the authors propose the "m-value" (multiverse value), which indicates the probability of an analysis path yielding a claim as extreme as the reported one. They also introduce "Agentic Bootstrap" to estimate this m-value by using AI agents to sample plausible analysis paths. This approach suggests that scientific evidence should be evaluated not just by a single analysis, but by its position within the distribution of all reasonable analyses.