Trustworthiness and Transparency Features Were Less Frequent in Randomized Trials Presenting Large Effects in Abstracts

  1. Jonathan F. Henssler
  2. Joana Reis-Pardal
  3. Lina Koppel
  4. John P.A. Ioannidis
Read the full article See related articles

Discuss this preprint

Start a discussion What are Sciety discussions?

Listed in

This article is not in any list yet, why not save it to one of your lists.
Log in to save this article

Abstract

OBJECTIVES

Large effect sizes (ESs), especially when prominently presented in trial abstracts, draw large attention, but it is important to understand whether they are trustworthy. We aimed to assess indicators of transparency and trustworthiness in randomized controlled trials (RCTs) reporting some large ES in their abstract, in comparison with RCTs presenting only non-large ESs in their abstract.

STUDY DESIGN AND SETTING

We included RCTs indexed in MEDLINE between January 1, 2024 and March 18, 2025, presenting at least one standardized mean differences of absolute value 0.8 or higher (large ES) versus those presenting only smaller absolute standardized mean differences in their abstract. Trial characteristics and methodological features were extracted systematically in large ES and non-large ES trials. Primary outcome was pre-specified protocol registration, secondary outcomes were having no protocol and public availability or repository placement of raw data.

RESULTS

We evaluated 152 trials with large ESs in their abstract and 175 trials with only non-large ESs in their abstract. Large ES trials had suggestively lower rates of pre-registered protocols (45% versus 61%, p=0.0054) and significantly higher rates of no protocol registration (26% versus 13%, p=0.0028) than non-large ES trials. There was no difference in raw data public availability or repository placement (6% versus 7%). Large ES trials were also less likely to be multicenter (p=0.0042), to have high-income country of corresponding author (p=0.0001), to be conducted in high-income country site(s) (p=0.0003), to have a published statistical analysis plan (p=0.0216), and to result from between-group comparisons (p<0.0001). Large effects were significantly more likely to involve non-drug/non-psychological interventions (p=0.0001).

CONCLUSIONS

RCTs presenting large ESs in their abstracts are more likely to lack transparency and trustworthiness features and may operate with higher risk of lack of credibility.

Highlights

  • This meta-research study assessed the association of effect sizes with features of transparency and trustworthiness of RCTs

  • Large effect trials have fewer registered and available protocols

  • Studies with large effects sizes are more likely to have a smaller sample size, single center recruitment and provenance from countries without strong clinical research tradition

  • Data sharing is poor across all trials regardless of effect size

What is new?

Key findings

  • This meta-research study indicated that RCTs presenting large effect sizes in their abstracts are more likely to lack features of transparency and trustworthiness.

What this adds to what is known?

  • Large effect sizes in abstracts of trials draw attention and may seem as compelling evidence for a study’s claim. Yet there have been divergent findings concerning the association between effect size and credibility of a trial. Our study showed that large effect sizes are consistently associated with a range of methodological shortcomings.

What is the implication and what should change now?

  • RCTs presenting large effect sizes may operate with higher risk of lack of credibility.

  • Rather than indicating a convincing finding, extreme results should be met with increased attention and careful scrutiny.

Article activity feed

  1. Version published to 10.1101/2025.10.20.25338369 on medRxiv