A fair efficacy formula for assessing the effectiveness of contact tracing applications

  1. Adam Fowler

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Mobile contact tracing apps have been developed by many countries in response to the COVID-19 pandemic. Trials have focussed on unobserved population trials or staged scenarios aimed to simulate real life. No efficacy measure has been developed that assesses the fundamental ability of any proximity detection protocol to accurately detect, measure, and therefore assess the epidemiological risk that a mobile phone owner has been placed at. This paper provides a fair efficacy formula that can be applied to any mobile contact tracing app, using any technology, allowing it’s likely epidemiological effectiveness to be assessed. This paper defines such a formula and provides results for several simulated protocols as well as one real life protocol tested according to the standard methodology set out in this paper. The results presented show that protocols that use time windows greater than 30 seconds or that bucket their distance analogue (E.g. RSSI for Bluetooth) provide poor estimates of risk, showing an efficacy rating of less than 6%. The fair efficacy formula is shown in this paper to be able to be used to calculate the ‘Efficacy of contact tracing’ variable value as used in two papers on using mobile applications for contact tracing [6]. The output from the formulae in this paper, therefore, can be used to directly assess the impact of technology on the spread of a disease outbreak. This formula can be used by nations developing contact tracing applications to assess the efficacy of their applications. This will allow them to reassure their populations and increase the uptake of contact tracing mobile apps, hopefully having an effect on slowing the spread of COVID-19 and future epidemics.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2020.11.07.20227447: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board Statementnot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    No key resources detected.

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    In this section we discuss the limitations of our testing and formula, future work required, and implications of our work on contact tracing applications. 15.1 Conclusion: This paper provides a basis for fairly measuring and comparing distance estimation protocols without bias to a particular technology or method. When used with the Oxford Risk Model the formula presented should provide a consistent and realistic contact tracing efficacy score. We now have a standardised way to measure the efficacy of proximity detection protocols used within contact tracing mobile apps. Governments can use this data to reassure citizens and encourage uptake of the apps. This will in turn increase the efficacy of contact tracing apps, and save lives, and prevent a return to national lockdowns. This work is also relevant to any outbreak of disease, not just COVID 19. This is especially true of the emergence of a new disease in future - keeping contact tracing apps available so they can be used at any time, perhaps with a general symptom reporting tool built in to spot new outbreaks or changes in diseases, is a good preventative steps governments can take to prevent future interference with normal life as has happened with COVID-19. We found in the development of our own Herald protocol that the fair efficacy formula was effective in allowing us to create, test, and improve our own protocol from an efficacy score of 13% to one of over 40% within 5 weeks with a small development team. [18] We ha...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2020.11.07.20227447v1 on medRxiv