The proportion of randomized controlled trials that inform clinical practice

  1. Nora Hutchinson
  2. Hannah Moyer
  3. Deborah A Zarin
  4. Jonathan Kimmelman

  • Curated by eLife

    eLife logo

    Evaluation Summary:

    This article constructs a four-step assessment of the informativeness of a clinical trial that measures its feasibility, reporting, importance, and risk of bias. This is a potentially highly relevant methodology for the class of trials for which it is defined, namely 'clinically directed randomized controlled trials'. It could also be translated and validated in other areas, using data from a wider set of sources beyond the trial registry clinicaltrials.gov. However, the extended longitudinal nature of the assessment and its potential subjectivity limit this tool's utility to being a retrospective diagnostic rather than as a prospective diagnostic and/or fix for at-risk designs.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. All three Reviewers agreed to share their names with the authors.)

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

Prior studies suggest that clinical trials are often hampered by problems in design, conduct, and reporting that limit their uptake in clinical practice. We have described ‘informativeness’ as the ability of a trial to guide clinical, policy, or research decisions. Little is known about the proportion of initiated trials that inform clinical practice. We created a cohort of randomized interventional clinical trials in three disease areas (ischemic heart disease, diabetes mellitus, and lung cancer) that were initiated between January 1, 2009 and December 31, 2010 using ClinicalTrials.gov . We restricted inclusion to trials aimed at answering a clinical question related to the treatment or prevention of disease. Our primary outcome was the proportion of clinical trials fulfilling four conditions of informativeness: importance of the clinical question, trial design, feasibility, and reporting of results. Our study included 125 clinical trials. The proportion meeting four conditions for informativeness was 26.4% (95% CI 18.9–35.0). Sixty-seven percent of participants were enrolled in informative trials. The proportion of informative trials did not differ significantly between our three disease areas. Our results suggest that the majority of randomized interventional trials designed to guide clinical practice possess features that may compromise their ability to do so. This highlights opportunities to improve the scientific vetting of clinical research.

Article activity feed

  1. Version published to 10.7554/elife.79491 on eLife
  2. eLife

    Evaluation Summary:

    This article constructs a four-step assessment of the informativeness of a clinical trial that measures its feasibility, reporting, importance, and risk of bias. This is a potentially highly relevant methodology for the class of trials for which it is defined, namely 'clinically directed randomized controlled trials'. It could also be translated and validated in other areas, using data from a wider set of sources beyond the trial registry clinicaltrials.gov. However, the extended longitudinal nature of the assessment and its potential subjectivity limit this tool's utility to being a retrospective diagnostic rather than as a prospective diagnostic and/or fix for at-risk designs.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. All three Reviewers agreed to share their names with the authors.)

  3. eLife

    Reviewer #1 (Public Review):

    This article creates a formal definition of the 'informativeness' of a randomized clinical trial. This definition rests upon four characteristics: feasibility, reporting, importance, and risk of bias. The authors have conducted a retrospective review of trials from three disease areas and reported the application of their definition to these trials. Their primary finding is that about one quarter of the trials deemed to be eligible for assessment satisfied all four criteria, or, equivalently, about three quarters failed one or more of their criteria. Notably, industry-sponsored studies were much more likely to be informative than non-industry-sponsored studies. It would be interesting to see a version of Figure 3 that categorizes by industry/non-industry to see the differences in fall-off between the four criterion.

    As the authors point out, the key limitations to this work are its inherent retrospective nature and subjectiveness of application, making any sort of prospective application of this idea all but impossible. Rather, this approach is useful as a 'thermometer' for the overall health of the type of trials satisfying the eligibility criteria of this metric. A secondary and inherent limitation of this measure is the sequential nature of the four criteria: only among the trials that have been determined to be feasible (the first criterion measured) can one measure reporting, importance, and lack of bias. And only among those trials that are both feasible and reported properly can one measure their importance and lack of bias, and so forth. Thus, except for feasibility, one cannot determine the proportion of all trials that were properly reported, were importance, or evinced lack of bias.

  4. eLife

    Reviewer #2 (Public Review):

    The authors present a systematic review of 125 trials (in three disease areas: ischemic heart disease, diabetes mellitus and lung cancer) available on clinicaltrials.gov, with the goal of estimating how often clinical trials result in a meaningful impact on clinical practice (or policy or research decisions). This is a very interesting and important question which, if not approached carefully, could lead to results that are misleading and/or difficult to interpret.

    To help reduce the potential for misleading results, the authors employed sensible criteria for inclusion of trials in this analysis (with trials being independently evaluated by multiple authors to determine whether they should be included). Once trials were selected, they had to be classified as "informative" or not. While such classification is, by definition, subjective, the authors attempted to make this process as objective as possible. They proposed a definition of "informative" based on four factors: feasibility (of achieving the target enrollment/completing the trial in a timely fashion), reporting (of results; either on clinicaltrials.gov or in a publication), importance (of the clinical question being addressed) and the quality of the design. As with the evaluation of trial inclusion, the authors independently evaluated each trial to determine informativeness.

    The authors provide a thorough discussion of key issues that could affect the interpretability of a trial and include a nice discussion of the limitations of their research. To me, the major limitation of this analysis (which the authors acknowledge) is that "clinically interesting/informative" is subjective. It is possible that their criteria will miss informative trials (or classify truly non-informative trials as informative). For example, while perhaps uncommon, a trial could be classified as non-informative due to poor design selection, but could end up being truly informative due to overwhelmingly positive results. Also, the "importance" component of their classification criteria could lead to truly important "niche" trials being misclassified as non-informative.

  5. eLife

    Reviewer #3 (Public Review):

    The paper describes an ingenious and painstakingly reported method of evaluating the informativeness of clinical trials. The authors have checked all the marks of robust, well-designed and transparently reported research: the study is registered, deviations from the protocol are clearly laid out, the method is reported with transparency and all the necessary details, code and data are shared, independent raters were used etc. The result is a methodology of assessing informativeness of clinical trials, which I look forward to use in my own content area.

    My only reserve, which I submit more for discussion than for other changes, is the reliance on clinicaltrials.gov. Sadly, and despite tremendous efforts from the developers of clinicaltrials.gov (one of the founders is an author of this paper and I am well-aware of her unrelenting work to improve reporting of information on clinicaltrials.gov), this remains a resource where many trials are registered and reported in a patchy, incomplete or downright superficial and sloppy manner. For outcome reporting, the authors compensate this limitation by searching for and subsequently checking primary publications. However, for the feasibility surrogate this could be a problem. Also, for risk of bias, for the trials the authors had to rate themselves (i.e., ratings were not available in a high-quality systematic review), what did the authors use, the publication or the record from the trial registry?

    In general, it seems like a problem for this sophisticated methodology might be the scarcity of publicly available information that is necessary to rate the proposed surrogates. Though the amount of work involved is already tremendous, the validity of the methodology would be improved by extracting information from a larger and more diverse pool of sources of information (e.g., protocols, regulatory documents, sponsor documents).

    In that sense, maybe it would be interesting for the authors to comment on how their methodology would be improved by having access to clinical trial protocols and statistical analysis plans. Of course, one would also need to know what was prospective and what was changed in those protocols, i.e., having protocols and statistical analysis plans prospectively registered and publicly available. Having access to these documents would open interesting possibilities to assessing changes in primary outcomes, though as the authors say that evaluation would also require making a judgement as to whether the change was justified. Relatedly, perhaps registered reports could be a potential candidate for clinical trials that would also support a more accurate assessment of informativeness, per the authors' method, provided the protocol is made openly available.

    Still related to protocols, were FDA documents consulted for pivotal trials, which again could give an indication of the protocol approved by the FDA and subsequent changes to it?

  6. eLife

    Author Response

    Reviewer 1

    This article creates a formal definition of the 'informativeness' of a randomized clinical trial. This definition rests upon four characteristics: feasibility, reporting, importance, and risk of bias. The authors have conducted a retrospective review of trials from three disease areas and reported the application of their definition to these trials. Their primary finding is that about one quarter of the trials deemed to be eligible for assessment satisfied all four criteria, or, equivalently, about three quarters failed one or more of their criteria. Notably, industry‐sponsored studies were much more likely to be informative than nonindustry‐sponsored studies. It would be interesting to see a version of Figure 3 that categorizes by industry/non‐industry to see the differences in fall‐off between the four criterion.

    Thank you for this suggestion. We have added an additional figure to the supplement, eFigure 1 ‐ The Cumulative Proportion of Trials Meeting Four Conditions of Informativeness by Sponsor.

    We have also indicated the following in the legend of Table 1: (as related to study sponsor)

    Lines 332 – 334 “Included within the designation “Other” are 7 trials that received funding from the U.S. National Institutes of Health (NIH) or other U.S. Federal agencies, and 60 trials that are nonindustry and non‐NIH/U.S. Federal agency funded.”

    As the authors point out, the key limitations to this work are its inherent retrospective nature and subjectiveness of application, making any sort of prospective application of this idea all but impossible. Rather, this approach is useful as a 'thermometer' for the overall health of the type of trials satisfying the eligibility criteria of this metric. A secondary and inherent limitation of this measure is the sequential nature of the four criteria: only among the trials that have been determined to be feasible (the first criterion measured) can one measure reporting, importance, and lack of bias. And only among those trials that are both feasible and reported properly can one measure their importance and lack of bias, and so forth. Thus, except for feasibility, one cannot determine the proportion of all trials that were properly reported, were importance, or evinced lack of bias.

    “Thermometer” is an apt metaphor. Please see response to Essential Revisions # 4 regarding the retrospective nature of our assessment.

    The sequential nature of our assessment is indeed a limitation for readers wanting to know the fraction of trials fulfilling each of the four criteria. This reflects a compromise between our aspirations and our labor capacity. However, we emphasize that our pre‐specified primary outcome was the fraction of trials fulfilling all informativeness criteria. We have also elaborated upon the following in our limitations section:

    Line 521 – 533 “Third, we used a longitudinal and sequential approach, since some of the conditions were only relevant once others had been met. For example, incorporation into a clinical synthesizing document can only occur once results have been reported. Our sequential approach enabled us to address our primary outcome with an economy of resources. However, our study does not enable an assessment of the proportion of trials fulfilling three of the four criteria in isolation from each other. In addition, changes in research practices or policy occurring over the last decade might produce different estimates for the proportion of randomized trials that are informative.”

    Reviewer 2

    The authors present a systematic review of 125 trials (in three disease areas: ischemic heart disease, diabetes mellitus and lung cancer) available on clinicaltrials.gov, with the goal of estimating how often clinical trials result in a meaningful impact on clinical practice (or policy or research decisions). This is a very interesting and important question which, if not approached carefully, could lead to results that are misleading and/or difficult to interpret.

    Thank you!

    To help reduce the potential for misleading results, the authors employed sensible criteria for inclusion of trials in this analysis (with trials being independently evaluated by multiple authors to determine whether they should be included). Once trials were selected, they had to be classified as "informative" or not. While such classification is, by definition, subjective, the authors attempted to make this process as objective as possible. They proposed a definition of "informative" based on four factors: feasibility (of achieving the target enrollment/completing the trial in a timely fashion), reporting (of results; either on clinicaltrials.gov or in a publication), importance (of the clinical question being addressed) and the quality of the design. As with the evaluation of trial inclusion, the authors independently evaluated each trial to determine informativeness.

    The authors provide a thorough discussion of key issues that could affect the interpretability of a trial and include a nice discussion of the limitations of their research. To me, the major limitation of this analysis (which the authors acknowledge) is that "clinically interesting/informative" is subjective. It is possible that their criteria will miss informative trials (or classify truly non‐informative trials as informative). For example, while perhaps uncommon, a trial could be classified as non‐informative due to poor design selection, but could end up being truly informative due to overwhelmingly positive results. Also, the "importance" component of their classification criteria could lead to truly important "niche" trials being misclassified as non‐informative.

    Thank you for this assessment. We agree that our measures of informativeness are imperfect. We hope that we have been forthright in the limitations of our approach. We believe that our first study limitation (lines 498 ‐ 513) outlines the issues highlighted above.

    Reviewer 3

    The paper describes an ingenious and painstakingly reported method of evaluating the informativeness of clinical trials. The authors have checked all the marks of robust, welldesigned and transparently reported research: the study is registered, deviations from the protocol are clearly laid out, the method is reported with transparency and all the necessary details, code and data are shared, independent raters were used etc. The result is a methodology of assessing informativeness of clinical trials, which I look forward to use in my own content area.

    Thank you!

    My only reserve, which I submit more for discussion than for other changes, is the reliance on clinicaltrials.gov. Sadly, and despite tremendous efforts from the developers of clinicaltrials.gov (one of the founders is an author of this paper and I am well‐aware of her unrelenting work to improve reporting of information on clinicaltrials.gov), this remains a resource where many trials are registered and reported in a patchy, incomplete or downright superficial and sloppy manner. For outcome reporting, the authors compensate this limitation by searching for and subsequently checking primary publications. However, for the feasibility surrogate this could be a problem. Also, for risk of bias, for the trials the authors had to rate themselves (i.e., ratings were not available in a high‐quality systematic review), what did the authors use, the publication or the record from the trial registry?

    Thank you. We agree that the sources of data that we relied on for this assessment are imperfect.

    We added the following to our limitations section:

    Lines 534 ‐ 535 “Fourth, our evaluation is limited by the accuracy of information contained in the ClinicalTrials.gov registration record...”

    We also added the following sentence to clarify the sources of our risk of bias assessment in eMethods 9:

    Lines 996 – 998 “Information from both the primary study publication and the ClinicalTrials.gov registration record were used in our risk of bias assessments.”

    In general, it seems like a problem for this sophisticated methodology might be the scarcity of publicly available information that is necessary to rate the proposed surrogates. Though the amount of work involved is already tremendous, the validity of the methodology would be improved by extracting information from a larger and more diverse pool of sources of information (e.g., protocols, regulatory documents, sponsor documents).

    In that sense, maybe it would be interesting for the authors to comment on how their methodology would be improved by having access to clinical trial protocols and statistical analysis plans. Of course, one would also need to know what was prospective and what was changed in those protocols, i.e., having protocols and statistical analysis plans prospectively registered and publicly available. Having access to these documents would open interesting possibilities to assessing changes in primary outcomes, though as the authors say that evaluation would also require making a judgement as to whether the change was justified. Relatedly, perhaps registered reports could be a potential candidate for clinical trials that would also support a more accurate assessment of informativeness, per the authors' method, provided the protocol is made openly available.

    Still related to protocols, were FDA documents consulted for pivotal trials, which again could give an indication of the protocol approved by the FDA and subsequent changes to it?

    We appreciate this comment and suggestion! And thanks for acknowledging the work it took to derive our estimates. Please see our full response to Essential Revision # 2 above.

  11. Version published to 10.1101/2022.05.12.22275021v1 on medRxiv