A Psychometrics of Individual Differences in the Truth Effect

The truth effect, the tendency to judge repeated information as more truthful, is a robust cognitive phenomenon. Yet efforts to study individual differences in this effect have returned inconsistent results. We argue that these inconsistencies largely reflect methodological limitations. Specifically, since correlations are attenuated by low reliability, conflicting findings likely stem from inconsistent measurement reliability across studies. In this structured, large-scale analysis of truth-effect datasets, we assess the psychometric properties of truth-effect paradigms. Rather than focusing solely on reliability -- which depends on the number of replicate trials -- we use signal-to-noise ratios, a recently proposed metric that captures psychometric goodness independent of trial size. Together, signal-to-noise ratios and trial size determine reliability, and both vary considerably across studies. Because increasing trial size alone is not always sufficient to ensure adequate reliability, we identify design characteristics that influence psychometric goodness in truth-effect experiments, such as stimulus material, initial task, sample composition, and study setting. Based on these findings, we offer practical recommendations for designing truth-effect experiments that afford the necessary psychometric goodness to measure individual differences. We show that, when psychometric goodness is high, localizing individual differences and their correlations with other psychological constructs is feasible with a manageable number of trials. These findings provide a basis for future research on individual differences in the truth effect and further underscore the importance of psychometric considerations in cognitive experiments.