SeroTracker‐RoB : A decision rule‐based algorithm for reproducible risk of bias assessment of seroprevalence studies
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (ScreenIT)
Abstract
Risk of bias (RoB) assessments are a core element of evidence synthesis but can be time consuming and subjective. We aimed to develop a decision rule‐based algorithm for RoB assessment of seroprevalence studies. We developed the SeroTracker‐RoB algorithm. The algorithm derives seven objective and two subjective critical appraisal items from the Joanna Briggs Institute Critical Appraisal Checklist for Prevalence studies and implements decision rules that determine study risk of bias based on the items. Decision rules were validated using the SeroTracker seroprevalence study database, which included non‐algorithmic RoB judgments from two reviewers. We quantified efficiency as the mean difference in time for the algorithmic and non‐algorithmic assessments of 80 randomly selected articles, coverage as the proportion of studies where the decision rules yielded an assessment, and reliability using intraclass correlations comparing algorithmic and non‐algorithmic assessments for 2070 articles. A set of decision rules with 61 branches was developed using responses to the nine critical appraisal items. The algorithmic approach was faster than non‐algorithmic assessment (mean reduction 2.32 min [SD 1.09] per article), classified 100% ( n = 2070) of studies, and had good reliability compared to non‐algorithmic assessment (ICC 0.77, 95% CI 0.74–0.80). We built the SeroTracker‐RoB Excel Tool, which embeds this algorithm for use by other researchers. The SeroTracker‐RoB decision‐rule based algorithm was faster than non‐algorithmic assessment with complete coverage and good reliability. This algorithm enabled rapid, transparent, and reproducible RoB evaluations of seroprevalence studies and may support evidence synthesis efforts during future disease outbreaks. This decision rule‐based approach could be applied to other types of prevalence studies.
Article activity feed
-
-
-
-
SciScore for 10.1101/2021.11.17.21266471: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Ethics not detected. Sex as a biological variable not detected. Randomization Reliability of the ordinal risk of bias ratings was assessed using a two-way random-effects average-measures intraclass correlation measuring absolute agreement that compared the SeroTracker-ROB decision rules to the manually derived ratings agreed upon by two independent reviewers. Blinding not detected. Power Analysis not detected. Cell Line Authentication Authentication: Part 1: Completion of the modified JBI checklist: The JBI checklist was selected as the foundation for the approach as it is a validated and commonly used critical appraisal tool for prevalence studies. Table 2: Resources
Software and Algorithms Sentences Resources SciScore for 10.1101/2021.11.17.21266471: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Ethics not detected. Sex as a biological variable not detected. Randomization Reliability of the ordinal risk of bias ratings was assessed using a two-way random-effects average-measures intraclass correlation measuring absolute agreement that compared the SeroTracker-ROB decision rules to the manually derived ratings agreed upon by two independent reviewers. Blinding not detected. Power Analysis not detected. Cell Line Authentication Authentication: Part 1: Completion of the modified JBI checklist: The JBI checklist was selected as the foundation for the approach as it is a validated and commonly used critical appraisal tool for prevalence studies. Table 2: Resources
Software and Algorithms Sentences Resources Analyses were conducted using STATA 14 (College Station, TX: StataCorp LP). STATAsuggested: (Stata, RRID:SCR_012763)StataCorpsuggested: (Stata, RRID:SCR_012763)Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:Weighted averages overcome some limitations of simple summary scores, as they may better reflect the relevance of each item; however, items cannot always be considered independently, thereby introducing complexity in the derivation of a weighted score. Considering combinations of items may provide a more nuanced perspective on bias. For example, statistical adjustment for antibody test sensitivity and specificity may introduce less bias when sensitivity and specificity are very high, as opposed to low. Some studies have trained multi-task machine learning algorithms to identify text relevant to risk of bias in randomized controlled trials and predict a risk of bias assessments using the text, with reasonable accuracy compared to human reviewers.16,17 However, key limitations of the algorithms include training on small datasets, imperfect reliability with manual reviewers, inaccessible software to employ these algorithms, and limited interpretability and transparency of the “black-box” decision making of the models. A decision rule approach, on the other hand, provides a transparent and interpretable model for automation of study risk of bias decisions.18 The tree structure is an ideal approach for capturing interactions between features in the data and clustering data points into distinct groups that are fairly easy to understand given that they can be visualized. The decision rules developed as part of the SeroTracker-ROB approach may be of value to other investigators condu...
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
Results from scite Reference Check: We found no unreliable references.
-