Applying Bayesian checks of cancellation axioms for interval scaling in limited samples

We explore the extent to which one can differentiate item responses compatible with a claim of interval scaling under the Rasch model from responses that are incompatible with this claim at sample sizes frequently encountered in educational and psychological research. We apply Domingue (2014)’s Bayesian method for evaluating an item response dataset’s adherence to the cancellation axioms of additive conjoint measurement under the Rasch model to item responses simulated from both Rasch and non-Rasch IRT models, and compare the extent to which the axiom of double cancellation holds in the data. We develop and evaluate procedures for bootstrapping null distributions of violation rates to improve the interpretability of results. Findings demonstrate the importance of tailoring the application of Domingue’s methods to specific characteristics of one’s data. At a sample size of 250, the method under investigation is not well-powered to detect the violations of interval scaling that we simulate, but the procedure works quite consistently at N = 1000. Implications, especially for researchers eager to assess the scale properties of their own instruments, are discussed.