Evaluating the Utility of Within-Session Retest Correlations as Reliability Estimates

Despite the theoretical arguments and recent empirical evidence indicating retest correlations to be particularly suitable reliability estimates, usual retest intervals (e.g., two weeks; one month) will underestimate the reliability of measures over the within-session interval in which inter-test relationships are regularly estimated. Consequently, we argue that within-session retest correlations may be particularly useful for estimating the reliability of measures over this interval. Specifically, we show evidence that within-session retest correlations: (1) provide the common upper-limit of internal consistency and retest reliability estimates, and consequently (2) are considerably less susceptible to producing disattenuated correlations greater than 1 in magnitude; (3) better track variation in scale validity correlations than common reliability estimators such as coefficient alpha; and (4) can be estimated for single-item measures. We illustrate this using data collected from several samples on two personality inventories, the Inventory of Individual Differences in the Lexicon (IIDL; Wood, Nye, & Saucier, 2010) and Big Five Inventory-2 (BFI-2; Soto & John, 2017). Results suggest that reliability is typically underestimated and thus overadjusted for in meta-analytic and structural equation modeling contexts, and that greater use of these estimates is thus likely to slow the momentum toward broader constructs and provide sturdier psychometric ground for the measurement of narrower constructs or nuances. More generally, we detail how within-session retest correlations draw better attention to the crucial role of the time interval separating measurements toward determining the appropriate estimates to use when adjusting for the measurement unreliability affecting zero-order correlations.