But r Won’t Do That: The Limits of Standardized Covariance

When we want to know if a scale gives reliable scores, or know if someone is accurate about something, or if a test shows stability over time, we often use the correlation coefficient r. Here we show this is wrong. The correlation coefficient is entirely a between-persons metric and cannot show us that people are accurate or a scale is reliable or a trait is stable. All those processes are within-person processes. Here we show using a Monte Carlo simulation study with 10,000 repetitions that the correlation coefficient r consistently provides the wrong answer when people systematically lie, when only a subset of people lie, and when people systematically over and underestimate. In short, in study one, we show the correlation coefficient should not be used to infer if a measure is reliable or a prediction or memory is accurate or a trait is stable. In Study 2, we use the same data setups to evaluate alternate metrics of accuracy percentages, repeatability coefficients, Intraclass Coefficients, Change Scores, and a Univariate Latent Change Score model. In no instance was one metric able to give the ‘right’ answer. Instead, we show that if researchers wish to measure reliability, the ICC should be used but thresholds should become more conservative. If researchers wish to measure accuracy, they should use simple measures of accuracy (e.g., is a prediction or memory accurate either directly or within a tolerance bound). If researchers wish to see if a trait is stable and unchanging, they should use simple change score models while also looking at the variance of the change score. In no scenario is the correlation coefficient r the right metric.