Advancing Climate Science Data Efficiency by Dual-stage Extreme Compression with an Efficient VAEformer

The growing threat of climate change highlights the urgent need for deeper understanding and precise weather prediction. Despite advancements in atmospheric research, the ever-growing volume of weather data challenges efficient storage and broadcasting, slowing efforts to mitigate climate change. To this end, we present Aeolus, a pioneering deep learning-based framework for efficient atmospheric data compression. Aeolus achieves a remarkable 470 times compression ratio that reduces the 400 TB ERA5 dataset to just 0.85 TB, significantly lowering storage requirements and improving transmission efficiency. This outperforms classical methods such as JPEG2000, which achieves compression ratios typically below 10 times on atmospheric datasets. Furthermore, Aeolus ensures practical compression and decompression speeds exceeding 1 GB/s, providing a substantial computational cost benefit compared to traditional techniques. Extensive comparative experiments validate the high numerical accuracy of the compressed data, with a mean absolute error of 0.17°K for temperature, consistent climatology, and comparable power spectral density. Such a level of precision enables efficient data-driven climate research, including accurate weather forecasting and extreme event reconstructions. Overall, our findings highlight the significant transformative impact of Aeolus on climate science, expanding access to large-scale atmospheric data across disciplines and empowering researchers with the data-driven insights needed to combat climate change.