Generalization in neural posterior estimation: Case studies with the racing diffusion model

Evidence accumulation models (EAMs) are popular tools for explaining speeded decisions because their parameters capture different aspects of the decision process. While researchers increasingly use Bayesian methods for estimating EAM parameters, computational costs and the necessity of tractable model likelihoods still pose obstacles for traditional methods such as Markov chain Monte Carlo (MCMC). Neural posterior estimation (NPE) addresses these obstacles by training a neural network to approximate the joint posterior distribution from simulated data, which only requires a generative model and amortizes computational costs at inference time. However, the quality of the approximation can suffer when the test data at inference time differs from the training data. Both test and training data stem from data-generating contexts and, for EAMs, these are influenced by the experimental design. Using the Racing Diffusion Model as a test case, we investigated how well NPE can generalize across changes in two types of experimental design contexts: number of trials and data-generating prior contexts that affect the error rate. Using MCMC as a reference, we showed in two simulation studies that NPE generalized well across changes in trial numbers but less well across error rates. We applied NPE to empirical datasets from reasoning tasks and found that the patterns observed in our simulation studies held. We discuss challenges and future directions to assess and improve the generalization for neural estimators for EAMs.