Information Geometry of Gaussian Processes and Its Applications to Transfer Learning

Gaussian Processes (GPs) are widely used in machine learning due to their ability to model complex functions while providing uncertainty quantification. However, their infinite-dimensional nature poses fundamental challenges in constructing meaningful geometric structures --- particularly in defining the Kullback-Leibler (KL) divergence in a principled way. In this paper, we propose a novel information-geometric framework that applies to a general class of GPs, not limited to posterior models or those with shared priors. We begin by defining an averaged KL divergence that avoids divergence to infinity and does not depend on a specific choice of input points. This formulation is justified by interpreting GPs as joint distributions over inputs and function values. To accommodate this perspective, we introduce a broader class of models called Integrated Gaussian Processes (IGPs), which relax the GP consistency condition. The IGP space forms a dually flat manifold, within which the GP space constitutes an m-flat submanifold. As an application of this framework, we consider transfer learning by projecting a target GP onto the convex hull of source GPs using an e-projection. To circumvent the difficulty of manipulating infinite-dimensional parameters, we adopt a geometric optimization algorithm based on the generalized Pythagorean theorem. This framework establishes a mathematically principled and computationally feasible foundation for transfer learning, and provides tools for broader geometrically grounded applications of GPs.