The Statistical Costs of Two-Step Signal Detection Analyses: A Case for a Maximum Likelihood Mixed-Effects Approach

Signal detection theory (SDT) is a highly influential modeling framework for studying decisions under uncertainty and is widely used to disentangle discriminability and response bias in a variety of research areas. A common approach to SDT analysis, the two-step approach, is based on independent individual estimates, which often require correction methods for individual relative response frequencies of 0 and 1. Across three simulation studies, we demonstrate that the two-step approach and associated correction methods produce substantial distortions in both single- and multilevel estimates of discriminability, response criterion and differences therein. These distortions depend in complex ways on the true population means, true population differences, the numbers of trials and participants, and dependencies among individual parameters. Crucially, they also lead to inflated Type I error rates for tests of condition differences in both SDT parameters. To prevent these issues, we propose a principled alternative, maximum likelihood mixed-effects SDT, which uses generalized linear mixed models to estimate individual-level parameters based on a joint population distribution. Unlike the two-step approach, mixed-effects SDT models yield effectively unbiased parameter recovery, maintain nominal Type I error rates across a broad range of simulated conditions, and provide higher statistical power for detecting differences in discriminability and response criterion. To facilitate the adoption of maximum likelihood mixed-effects SDT, we provide an easy-to-use R package called mesdt, which implements fitting, hypothesis testing and post-processing methods for this approach, as well as an accompanying tutorial.