Forecasting Long-term Spatial-temporal Dynamics with Generative Transformer Networks

Recent advances in deep learning have aimed to address the limitations of traditional numerical simulations, which, although precise, are computationally intensive and often impractical for real-time applications. Current models, however, may have challenge in obtaining high predictive accuracy and long-term stability while obeying physical principles for spatiotemporal prediction problems. We introduce DynamicGPT, a Vision Transformer-based generative model specifically designed for spatiotemporal prediction. This model operates without explicit physical constraints, preserving critical spatial features and effectively capturing dependencies across varying time scales. The model integrates a multi-scale embedding network to preserve critical spatial features and a tailored temporal modeling network to effectively capture dependencies across varying time scales. This combination enables DynamicGPT to maintain predictive accuracy and stability over long-term forecasts, as validated by its performance in diverse real-world scenarios—including composite material stress and crack analysis, global sea surface temperature prediction, and 3D reaction-diffusion simulations—demonstrating its capability to handle out-of-distribution data, extended time horizons, and complex 3D structures. Importantly, DynamicGPT can adhere to physical laws, excels in partial differential equation parameter estimation, and optimizes its architecture for reduced computational load. This work positions DynamicGPT as a scalable, data-driven alternative bridging traditional simulations and modern AI, paving the way for advancement in real-time spatiotemporal modeling.