Assessing Reliable and Valid Measurement as a Prerequisite for Informative Replications in Psychology

For a replication to be informative, measurement should be reliable and valid in bothoriginal and replication studies. Recent studies have identified problems with measurementand measurement reporting in both original and replication studies, although theassociation between measurement and replicability remains unclear. We investigated thereliability and measurement reporting of 77 measures within 56 Many Labs replicationsand related original articles (Ebersole et al., 2016, 2020; Klein et al., 2014, 2018) andstudied their association with replicability. We found that not all measures weresufficiently reliable across contexts, and that only few measures were accompanied withreliability and validity evidence. Furthermore, “questionable measurement practices”(QMPs) in replication studies were associated with lower replicability, while original studyreliability was not. These results corroborate existing findings that construct validity inpublished research is often unknown, which may complicate replication research. We offersuggestions on improving measurement practices and argue that reported measurementinformation should inform the decision to replicate.