Performance of national COVID-19 ‘symptom checkers’: a comparative case simulation study

  1. Fatma Mansab
  2. Sohail Bhatti
  3. Daniel Goyal

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

Identifying those individuals requiring medical care is a basic tenet of the pandemic response. Here, we examine the COVID-19 community triage pathways employed by four nations, specifically comparing the safety and efficacy of national online ‘symptom checkers’ used within the triage pathway.

Methods

A simulation study was conducted on current, nationwide, patient-led symptom checkers from four countries (Singapore, Japan, USA and UK). 52 cases were simulated to approximate typical COVID-19 presentations (mild, moderate, severe and critical) and COVID-19 mimickers (eg, sepsis and bacterial pneumonia). The same simulations were applied to each of the four country’s symptom checkers, and the recommendations to refer on for medical care or to stay home were recorded and compared.

Results

The symptom checkers from Singapore and Japan advised onward healthcare contact for the majority of simulations (88% and 77%, respectively). The USA and UK symptom checkers triaged 38% and 44% of cases to healthcare contact, respectively. Both the US and UK symptom checkers consistently failed to identify severe COVID-19, bacterial pneumonia and sepsis, triaging such cases to stay home.

Conclusion

Our results suggest that whilst ‘symptom checkers’ may be of use to the healthcare COVID-19 response, there is the potential for such patient-led assessment tools to worsen outcomes by delaying appropriate clinical assessment. The key features of the well-performing symptom checkers are discussed.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2020.04.28.20084079: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board Statementnot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.
    Cell Line Authenticationnot detected.

    Table 2: Resources

    Experimental Models: Cell Lines
    SentencesResources
    In particular, an unremitting, persistent fever warrants further assessment in regard to COVID-19 [30], but also in relation to sepsis [29].
    COVID-19
    suggested: None

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Strengths and Limitations: This case simulation study was conducted using 52 standardised simulated cases. The cases were designed to test specific COVID-19 related scenarios, and as such were symptom-based without the need for subjective interpretation. Nonetheless, there remains a risk of bias, particularly when facing subjective questions. The majority of simulations were though more quantitative, for example duration, age and symptoms, and unlikely to be affected meaningfully by any bias. The UK data is pooled from all four nations (England, Wales, Scotland and Northern Ireland). England (making up 90% of the total UK population) uses the same ‘111’ COVID-19 patientled triage system analysed here, whereas Wales, Scotland and Northern Ireland have implemented their own individual patient-led triage systems. It was beyond the scope of this initial investigation to examine each triage system separately. A similar situation applies to the US, where some individual states have implemented their own triage systems.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1136/bmjhci-2020-100187
  3. Version published to 10.1101/2020.04.28.20084079v3 on medRxiv
  4. Version published to 10.1101/2020.04.28.20084079v2 on medRxiv
  5. Version published to 10.1101/2020.04.28.20084079v1 on medRxiv