A New Paper Framework to Increase Reproducibility: Example Relating to Web Pharmacovigilance During COVID-19 in Italy

  1. Alessandro Rovetta
  2. Lucia Castaldo

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

Reproducibility and transparency represent some of the main problems of scientific publishing. Currently, the editorial requests of academic journals and peer reviewers can divert the authors’ attention from an accurate description of the methods adopted, thus compromising these fundamental scientific aspects. This paves the way for the voluntary falsification of data to obtain striking results. Furthermore, the excessive expansion of introduction and discussion sections increases the likelihood of introducing evaluation bias. Since peer reviewers are generally unpaid for their work, they are not required to reproduce the analysis of the studies they review but only to assess methodological accuracy, reproducibility, and plausibility. Therefore, this paper aims to emphasize that the methods and results sections are the central parts of quantitative analysis. In this regard, we firmly believe that the peer review process should, whenever possible, reproduce the analysis from scratch. Consequently, authors must be required to provide a simple and straightforward tutorial to reproduce the analysis as it was conceived both methodologically and chronologically. Ideas, insights, and discussions among the authors must also be reported. This complete description can be provided as integrative material published with the main manuscript, which is nothing more than a summary of methods and results. Such a procedure would represent the first step to improving the quality of scientific publications, waiting for unscientific concepts such as “publish or perish” to be eradicated from the academic world. In this manuscript, we provide a framework that can serve as a fully reproducible and transparent example of analysis. The aim is to investigate the Italian netizens’ web interest in paracetamol, ibuprofen, and nimesulide from 2015 to 2022, searching for causal associations with the fever symptom and COVID-19. The infodemiological tool “Google Trends” has been used to collect the data. Correlational analysis showed plausible causal associations between paracetamol, ibuprofen, and fever due to seasonal flu and COVID-19 and, although to a minor extent, COVID-19 vaccines side effects. Paracetamol was the most historically searched substance. However, the trend of ibuprofen has caught up with that of paracetamol in 2022. Interest in paracetamol, ibuprofen, and nimesulide increased substantially during the COVID-19 pandemic period. We conclude that web pharmacovigilance via Google Trends can provide relevant evidence for monitoring drug intake in relation to epidemiologically significant events such as epidemics and mass vaccination campaigns.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2022.05.03.22274607: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Ethicsnot detected.
    Sex as a biological variablenot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.

    Table 2: Resources

    No key resources detected.

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    This section often consists of three or four main sections, such as i) a summary of the paper’s main findings, ii) a comparison with other literature, iii) limitations (and strengths), and iv) conclusions. In this regard, the authors of this paper consider sections i), iii), and, eventually, iv) to be appropriate. In particular, comparison with other literature should be optional for the same reasons set out in the Introduction section plus one: unless systematic reviews or meta-analyses are performed, such comparisons are likely to be undermined by authors’ biases and are statistically unreliable since no combining techniques (e.g., Fisher formula for P-values) can be employed. Indeed, a scientist must always admit that biases are part of any analysis as they represent a cognitive feature of our human nature [17]. These can be limited or – more or less – managed by quantitative analyses but certainly not eliminated. Hence, it is extremely difficult to conduct a neutral comparison without exploiting the methods of systematic reviews and meta-analyses [18]. For these reasons, we suggest leaving any comparisons optional according to the authors’ aims rather than forcing them to disclose their biases when not necessary. Furthermore, we suggest presenting the results with caution, avoiding overstatements, and trying to describe in detail what was found rather than convince the reader of the study’s relevance. Finally, we believe that this is only a first step towards the qualitat...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2022.05.03.22274607v1 on medRxiv