A novel approach to quantify out-of-distribution uncertainty in Neural and Universal Differential Equations

Dynamical systems play a central role across the quantitative sciences, offering a powerful mathematical framework to describe, analyze, and predict the evolution of complex processes over time. In Systems Biology, dynamical systems provide a foundation for modeling and predicting the intricate behaviors of biological systems. Recent advances in data-driven approaches, such as Neural Ordinary Differential Equations (NODEs) and Universal Differential Equations (UDEs), have enabled the development of models that are either fully or partially data-driven. Integrating data-driven components into dynamical systems amplifies the challenge of generalization beyond training data, highlighting the need for robust methods to quantify uncertainty in scenarios not encountered during training. In this work, we investigate the reliability of uncertainty quantification (UQ) based on ensembles of models in the reconstruction of dynamical systems. We show that standard ensembles (i.e., models trained independently with different random initializations) risk producing overconfident predictions in previously unseen scenarios, as the models in the ensemble tend to exhibit similar behaviors. To address this issue, we propose a novel ensemble construction method for NODEs and UDEs that fosters diversity in the reconstructed vector field across models within specific regions of the state space, while maintaining explicit control over the fit on the training set. We evaluate our method on synthetic test cases derived from three models commonly used as benchmarks for data-driven reconstruction of dynamical systems: the Lotka-Volterra model, the damped harmonic oscillator, and the Lorenz system. Our results demonstrate that the proposed method enhances the reliability of UQ in previously unseen scenarios with respect to standard ensembles, providing a more robust framework for the emerging field of fully or partially data-driven reconstruction of dynamical systems.