Orthogonal Basis Construction for Consistent Sparse Gaussian Process Likelihoods

Many variations of sparse Gaussian process models rely on approximations to the kernel matrix via usage of orthogonal basis functions to speed up likelihood function evaluation. We show that such methods can lead to an inconsistent likelihood function if the basis functions are not orthonormal with respect to the measure describing the distribution of the inputs, and propose a method for construction of basis functions that asymptotically orthonormal with respect to the input measure. This yields an approach that is consistent with the full Gaussian process marginal likelihood, and which exhibits complexity cubic in the dimension of the kernel approximation.