Scaled Point-in-Time LLMs Reduce Lookahead Bias in Research

This research addresses a critical issue in using large language models (LLMs) for historical analysis: lookahead bias, where models trained on unrestricted internet data inadvertently incorporate future information. To counter this, the authors developed "point-in-time" language models, which are strictly trained only on text available up to a specific calendar date, ensuring temporal validity for applications like financial backtesting and causal inference. The study demonstrates that by scaling these chronologically filtered models, the performance disparity with unconstrained LLMs can be substantially reduced. They trained decoder-only transformers with up to 4 billion parameters on 1 trillion tokens from FineWeb, filtered by date. These models, spanning monthly checkpoints from 2013-2024, achieved performance comparable to leading open-weight models like Gemma-3-4B and LLaMA-7B on various benchmarks, albeit with a remaining gap on some tasks. Instruction fine-tuning further improved usability, and the complete pipeline is released to foster reproducible research in this area.