Interpreting AI Weather Models for Atmospheric Chemistry Predictions

Weather forecasting foundation models (FMs) are increasingly being adapted to predict air quality, offering rapid global pollution forecasts with lower computational demands compared to traditional chemical transport models. These FMs are typically trained on reanalysis data and generate forecasts through an autoregressive process, without explicitly encoding physical or chemical governing processes. Consequently, high forecast accuracy alone doesn't reveal whether the model has genuinely learned underlying physical mechanisms or is simply exploiting statistical patterns within its training data. This research presents the first in-depth study aimed at understanding what an FM fine-tuned for atmospheric chemistry has actually learned, specifically examining Microsoft's Aurora model. The methodology involves imposing controlled chemical perturbations on its forecasts and comparing them against established photochemical relationships. Additionally, the internal representations that drive these forecasts are scrutinized. The findings indicate that Aurora can capture a first-order ozone response to reactive nitrogen, but it fails to enforce the explicit chemical constraints that a process-based model would. It produces chemically inconsistent combinations of related species and tends to smooth out localized emission features, such as wildfire plumes, towards background levels. Internally, the model's representations largely retain the meteorological organization inherited during its pretraining phase, showing little structure specifically related to chemistry. Using sparse autoencoders, the study identified internal components that causally influence chemical forecasts, but these components do not cleanly map to individual atmospheric processes. This work establishes a framework for evaluating whether AI forecasting systems truly learn atmospheric chemistry from reanalysis data, emphasizing that composition forecasts should be judged by their internal mechanisms, not just benchmark skill, especially as these models increasingly inform environmental policy.