Selective reporting of outcomes and results in interrupted time series studies of health interventions: a methodological study

Objective

Selective reporting bias occurs when authors report outcomes or results based on the P value, magnitude or direction of the results. Selective reporting has not been examined in interrupted time series (ITS) studies. Therefore, we investigated (1) discrepancies in outcome reporting, (2) completeness of results reporting, and (3) evidence of selective reporting bias in ITS studies.

Design

We systematically searched for published peer-reviewed protocols of ITS studies of health interventions in 22 databases, and corresponding results reports addressing the protocol’s primary ITS research questions. For each primary research question, we identified outcomes that were reported in the protocol and the corresponding results report(s), and all results reported for those outcomes.

Main outcome measure

We defined a discrepancy as any outcome that was added, omitted, or had its primacy reclassified (e.g., from primary to secondary) in the results reports compared to the protocol. Each result was classified as fully reported if it was reported with both an effect estimate and a measure of precision (such as a confidence interval). Each result was also classified as favourable or not favourable to the interruption, based on its statistical significance and direction of effect.

Results

Our search for ITS protocols returned 4,590 abstracts. After excluding ongoing studies, protocols without published results for the primary research question, and records excluded for other reasons, we identified 44 ITS protocols (published 2010-2022) with 46 corresponding results reports. Among outcomes assessed for discrepancies, 52% (202/388) had a discrepancy, affecting 74% (31/42) of studies. Non-reporting of outcomes was prevalent, with 24% (132/553) of outcomes defined in the protocol not reported in the results report, and 60% (25/42) of protocols having at least one outcome not reported in the result reports. Only 28% (56/202) of discrepancies were justified by authors in the results reports. The association between a result favouring the interruption (based on statistical significance and direction of effect) and the result being fully reported was uncertain (OR=1.06 [95% CI 0.74 to 1.53]).

Conclusion

Non-reporting of outcomes and discrepancies in outcome reporting were prevalent. Pre-specifying outcomes in protocols and registries helps mitigate selective reporting. However, outcomes should be described in sufficient detail for readers to detect any changes.

SUMMARY

Section 1: What is already known on this topic

Selective outcome and result reporting bias ― occurs when the authors report a subset of the measured outcomes or results, based on whether the P value, magnitude of effect estimate or direction of effect supports the authors’ hypothesis. Selective reporting bias has not been examined in interrupted time series (ITS) studies.

Section 2: What this study adds

Our study highlights several issues with outcome reporting among ITS studies. We found that non-reporting of outcomes and discrepancies between outcomes reported in study protocols and outcomes reported in the final reports were prevalent. Moreover, outcomes were often too broadly defined in the protocol to allow an accurate assessment of selective reporting. We introduced a template for the elements that could be reported when describing an outcome in the context of ITS data.

Section 3: How might these results change the focus of research or clinical practice?

Our findings call for greater transparency in outcome reporting among investigators of ITS studies. To mitigate selective reporting bias, authors are encouraged to pre-specify outcomes in a protocol and describe outcomes in sufficient detail in any publication (protocol or results reports) to enable readers to detect any change in outcome reporting. Reporting may also be improved with guidance and stricter enforcement of outcome reporting by journal editors, peer reviewers and funders.