Risk of Bias in Clinical Psychology Meta-Analyses (2000-2020): An Overview

One of the biggest limitations of meta-analyses is that the information they provide can be affected by the biases of the included primary studies. To address this, evaluations of primary study risk of bias (RoB) can be performed and incorporated into the meta-analysis. However, research on this topic in clinical psychology is scarce. In this study, we examined this issue using a sample of clinical psychology meta-analyses that included RoB assessments. First, we evaluated meta-analysts’ assessment practices. Second, we summarized the RoB ratings of the primary studies included in the meta-analyses. Lastly, we examined the relationship between RoB ratings and effect sizes. We found some suboptimal practices in the assessment procedures, such as only half of the studies reporting that the assessment was conducted in duplicate. Regarding RoB ratings, the domains with the highest ratings were random sequence generation, blinding of outcome assessment, and incomplete outcome data, with about half of the primary studies rated as low RoB. The lowest ratings were found for allocation concealment and, especially, blinding of participants and personnel. Importantly, we found a positive association between the publication year of the primary studies and a lower RoB in most domains. Lastly, performing our own re-analysis, we found an association between RoB and effect sizes, which contrasts with the results of the analyses reported in the meta-analyses that combined those studies. We recommend caution when interpreting a lack of modulation of effect sizes in meta-analyses, as they may not have sufficient statistical power for moderator analyses.