What we do in the shadows: Methodologists use a range of synthesis methods when meta-analysis of all results is not possible but describe challenges in planning and selecting methods
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Introduction
Systematic review authors commonly encounter situations where the data required for meta-analysis are incompletely reported (e.g. when effect estimates are reported without a measure of precision). In this circumstance, many systematic review authors use a method other than meta-analysis (e.g. vote counting), but rarely describe those methods or the rationale for selecting them. We aimed to investigate what methods authors consider when meta-analysis of all study results is not possible, and what factors influence their decisions.
Methods
We interviewed 12 experienced systematic review authors, editors and methodologists, presenting four scenarios in which it was not possible to combine all results using meta-analysis. Scenarios varied in the number and size of included studies, available data, and risk of bias. Participants discussed the methods they considered to summarise, synthesise and present the results; whether they would synthesise available results; and how they would draw overall conclusions.
Results
Factors that informed decisions included participants’ overall purpose in conducting synthesis, existing beliefs about study results and synthesis methods, trust in the available data, and the decision-making needs of end users. Participants differed in which synthesis methods to use, whether they would use multiple synthesis methods, and which studies they would analyse with each method.
Conclusions
We identified several synthesis methods considered when meta-analysis of all results is not possible, and factors that influence the selection of methods, neither of which are routinely reported. More complete reporting of these methods and the factors informing decisions would allow readers to better understand the decisions made.
Highlights
What is already known
Systematic review authors commonly encounter situations where the data required for meta-analysis are incompletely reported (e.g. when effect estimates are reported without a measure of precision). Many systematic review authors use a method other than meta-analysis (e.g. vote counting), but rarely describe those methods or the rationale for selecting them.
What is new
Interviews revealed that systematic review authors consider a range of methods when meta-analysis of all results is not possible, and multiple factors influence the choice of methods. These methods and the decision-making process are not routinely reported in published reviews.
Potential impact for Research Synthesis Methods readers
Authors of systematic reviews should report synthesis methods and the rationale for their methods decisions. This would enable readers to understand the decisions made and support interpretation, appraisal and reproduction of the synthesis.
