Toward a Principled Bayesian Workflow in Semantics and Pragmatics: A Tutorial

When researchers study semantic phenomena using probabilistic modeling, they must validate these models to ensure they accurately formalize the underlying theory and predictions. Probabilistic models have gained increasing popularity in semantics and pragmatics in recent years, yet the question of how to validate these models has not been addressed in the literature. Here, we present R and Stan-based tutorials on (a) five model validation methods based on the principled Bayesian workflow in cognitive science (Betancourt, 2018; Gelman et al., 2020; Schad et al., 2021) and (b) on model comparisons in a Bayesian hypothesis testing framework by means of the Bayes factor (Jeffreys, 1939; Kass & Raftery, 1995). We apply these methods to two models from the RSA family from the influential article by van Tiel et al., 2021. In tutorial I, we demonstrate how to implement and interpret (1) prior predictive checks, (2) computational faithfulness, (3) model sensitivity, (4) parameter recovery, and (5) posterior predictive checks. In tutorial II, we demonstrate how to implement and interpret a Bayes factor model comparison to quantify evidence for the competing probabilistic models. The tutorial includes path diagrams for an easier understanding of the methods, troubleshooting checklists to address undesired outcomes, and R and Stan code to make it easier to apply the methods.