Deep Learning Predictors Show Mixed Results for Scientific Data Compression

The research explores the potential of deep neural networks (DNNs) to improve error-bounded lossy compression, a vital technique for managing the vast amounts of scientific data generated by simulations and instruments. Traditional compression methods rely on accurate predictors to minimize residuals, which are then easier to compress. This study questions whether advanced machine learning models could serve as superior predictors. To address this, the researchers developed a framework integrating spatial and temporal deep learning models into a standard error-bounded compression pipeline, specifically leveraging existing highly accurate weather forecasting foundation models in the climate domain. They evaluated three distinct ML predictors—VAEformer-based codec (CRA5), Graph Neural Network forecaster (GraphCast), and Vision-Transformer forecaster (Aurora)—against the state-of-the-art compressor SZ3.1. All comparisons used identical quantization and entropy-coding backends. Analyzing approximately 1.7 TB of ERA5 climate data, the study yielded a surprising outcome. Although ML predictors generated more accurate predictions and significantly improved reconstruction quality (up to 91%) and compression ratios (up to 9.6x) for highly predictable variables, they failed to improve the overall dataset-level compression ratio. The findings suggest that prediction accuracy alone is insufficient; the spatial structure of the resulting residuals plays a decisive role in the efficiency of entropy coding, which ML models currently do not optimize effectively for general compression.