Preregistration Protocol Mitigates p-Hacking in LLM Research.

The increasing use of Large Language Models (LLMs) for data generation, classification, and annotation in research introduces a significant risk of "p-hacking." This occurs when researchers iteratively adjust prompts, decoding parameters, or output formats until a statistically significant or desired result is achieved, compromising the validity of findings. To counter this, a novel protocol has been proposed: preregistering the experiment and specifying a set of eligible future LLMs. The core idea is that researchers finalize their experimental procedure on current models and then publicly commit to an analysis plan, along with a list of models that will be considered for the confirmatory analysis. The actual confirmatory analysis is then run on the *first eligible LLM released after* the preregistration. Because this future model does not exist at the time of commitment, it cannot be "hacked against." Furthermore, configurations that successfully p-hack one model often do not transfer effectively to a different, newer model. The protocol was evaluated on two tasks with known true values, demonstrating that it blocked the transfer of p-hacks in over 70% of cases across various models and configurations. This method offers a robust way to enhance the scientific rigor and trustworthiness of LLM-based research.