Estimation of a Countably Infinite-Dimensional Transition Probability Matrix Using a Stick-Breaking Prior

We consider the problem of estimation of the transition probability matrix (TPM) of a Markov chain where the state space is countably infinite. At present, there is no methodology in the literature for direct estimation of infinite-dimensional TPMs. Standard approaches, including maximum likelihood estimation and Bayesian methods based on Dirichlet priors, are inherently finite-state and, when applied in this setting, assign zero probability to all unobserved transitions. To derive a method for estimation of infinite-dimensional TPMs, we propose a Bayesian nonparametric approach that employs a random measure prior. Specifically, we utilize hierarchical and generalized hierarchical stick-breaking processes as priors on the rows of the TPM. The generalized hierarchical stick-breaking prior, in particular, allow for both positive and negative correlations among any pair of transition probabilities, thereby enabling more realistic modeling of the dependence across the state transitions. We develop a blocked Gibbs sampling algorithm for posterior computation under a generalized hierarchical stick-breaking prior that is fast, highly efficient, and well suited to large-scale problems. The proposed method is evaluated through extensive simulations as well as applications to real-world datasets on historical trading volume for the United States Oil Fund (USO), and daily precipitation records from Heathrow, London. The empirical results demonstrate strong and stable predictive performance in all scenarios considered.