Improving peer review of systematic reviews and related review types by involving librarians and information specialists as methodological peer reviewers: a randomized controlled trial

Objective: To evaluate the impact of adding librarians and information specialists (LIS) as methodological peer reviewers to the formal journal peer review process on the quality of search reporting and risk of bias in systematic review searches in the medical literature.Methods: A pragmatic two-group parallel randomized controlled trial of systematic reviews and related evidence synthesis manuscripts submitted to The BMJ, BMJ Open, and BMJ Medicine and sent out for peer review from January 3, 2023 to September 1, 2023. Randomization (allocation ratio, 1:1) was stratified by journal and used permuted blocks (block size = 4). All manuscripts followed usual journal practice for peer review, but those in the intervention group had an additional (LIS) peer reviewer invited. The primary outcomes are the differences in first revision manuscripts between intervention and control groups in the quality of reporting and risk of bias. Quality of reporting was measured using four pre-specified PRISMA-S items. Risk of bias was measured using ROBIS Domain 2. Assessments were done in duplicate and assessors were blinded to group allocation. We performed intention to treat and per protocol analyses. As secondary outcomes, we analyzed differences between groups for each individual PRISMA-S and ROBIS Domain 2 item. We also assessed differences in the proportion of manuscripts rejected as the first decision between the intervention and control groups. The study protocol was registered (https://doi.org/10.17605/OSF.IO/W4CK2) and the protocol published (https://dx.doi.org/10.1186/s13063-021-05738-z).Results: Of 2,670 manuscripts sent to peer review during study enrollment, 400 met inclusion criteria and were randomized (62 The BMJ, 334 BMJ Open, 4 BMJ Medicine). By study close, 76 first revisions were submitted in the intervention group and 90 in the control group. Differences in the proportion of adequately reported searches (4.4% difference, 95% CI: -2.0%, 10.7%) and risk of bias in searches (0.5% difference, 95% CI: -13.7%, 14.6%) showed no statistically significant differences between groups. Inviting an LIS peer reviewer showed a difference (16.0%, 95% CI: 0.96%, 31.0%) on adequately reporting one individual item, PRISMA-S Item 13 (dates of search). By four months post-study, 98 intervention and 70 control group manuscripts had been rejected after peer review (13.8% difference, 95% CI: 3.9%, 23.8%). Conclusions: Inviting LIS peer reviewers did not impact adequate reporting or risk of bias of searches in first revision manuscripts of biomedical systematic reviews and related review types, though LIS peer reviewers may have contributed to a higher rate of rejection after peer review.