Peer Review Scores Fail to Predict AI Research Impact

Over the past decade, a few foundational methodological contributions, such as the Transformer and large-scale pre-training, have profoundly reshaped AI research. This study investigated whether the peer review scores and acceptance decisions at ICLR conferences could predict which papers would become these "catalysts" – those that measurably redirect future research. Analyzing over 36,000 papers from ICLR 2017-2025, the researchers compared four disruptiveness measures, including a new direction-aware Embedding Disruptiveness Measure (EDM). They also defined a five-type taxonomy for catalyst papers, such as "topic initiators" and "topic bridges." The findings revealed a striking disconnect: peer review scores were essentially orthogonal to a paper's future disruptiveness. Accepted and rejected papers showed indistinguishable mean EDM scores, indicating that the current review process does not effectively identify trajectory-changing research at the submission stage. EDM, however, proved effective at identifying highly cited papers.