Analyzing Binary Judgments: A Comparison of ANOVA, Signal Detection Theory, and Generalized Linear Mixed Models in the Context of the Illusory Truth Effect

Binary judgments are widely used in psychological research to understand decision-making processes. Common analysis approaches include analysis of variance (ANOVA) of choice proportions, signal detection theory (SDT) of choice frequencies, and generalized linear mixed models (GLMM) of dichotomous judgments. For substantive researchers, it may be unclear how these three methods differ. Therefore, we explain how the methods relate to each other and highlight their advantages and limitations. Given that a systematic comparison using empirical datasets remains lacking, it is difficult to judge the practical relevance of choosing a specific method. To address this gap, we compared ANOVA, SDT, and GLMMs using 20 openly available datasets featuring the illusory truth effect, a robust phenomenon frequently examined with binary judgments. GLMMs with a moderately complex random-effects structure produced more stable and conservative effect size estimates than the other methods. Moreover, GLMMs can resolve assumption violations of ANOVA and enable a direct interpretation in terms of SDT while being less sensitive to missing data. Overall, GLMMs are a theoretically sound and practically robust method and thus superior for analyzing binary judgments in social and cognitive psychology.