How do the UK public interpret COVID-19 test results? Comparing the impact of official information about results and reliability used in the UK, USA and New Zealand: a randomised controlled trial
This article has been Reviewed by the following groups
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
- Evaluated articles (ScreenIT)
Abstract
To assess the effects of different official information on public interpretation of a personal COVID-19 PCR test result.
Design
A 5×2 factorial, randomised, between-subjects experiment, comparing four wordings of information about the test result and a control arm of no additional information; for both positive and negative test results.
Setting
Online experiment using recruitment platform Respondi.
Participants
UK participants (n=1744, after a pilot of n=1657) quota-sampled to be proportional to the UK national population on age and sex.
Interventions
Participants were given a hypothetical COVID-19 PCR test result for ‘John’ who was presented as having a 50% chance of having COVID-19 based on symptoms alone. Participants were randomised to receive either a positive or negative result for ‘John’, then randomised again to receive either no more information, or text information on the interpretation of COVID-19 test results copied in September 2020 from the public websites of the UK’s National Health Service, the USA’s Centers for Disease Control, New Zealand’s Ministry of Health or a modified version of the UK’s wording. Information identifying the source of the wording was removed.
Main outcome measures
Participants were asked ‘What is your best guess as to the percent chance that John actually had COVID-19 at the time of his test, given his result?’; questions about their feelings of trustworthiness in the result, their perceptions of the quality of the underlying evidence and what action they felt ‘John’ should take in the light of his result.
Results
Of those presented with a positive COVID-19 test result for ‘John’, the mean estimate of the probability that he had the virus was 73% (71.5%–74.5%); for those presented with a negative result, 38% (36.7%–40.0%). There was no main effect of information (wording) on these means. However, those participants given the official information from the UK website, which did not mention the possibility of false negatives or false positives, were more likely to give a categorical (100% or 0%) answer (UK: 68/343, 19.8% (15.9%–24.4%); control group: 42/356, 11.8% (8.8%–15.6%)); the reverse was true for those viewing the New Zealand (NZ) wording, which highlighted the uncertainties most explicitly (20/345: 5.8% (3.7%–8.8%)). Aggregated across test result (positive/negative), there was a main effect of wording (p<0.001) on beliefs about how ‘John’ should behave, with those seeing the NZ wording marginally more likely to agree that ‘John’ should continue to self-isolate than those viewing the control or the UK wording. The proportion of participants who felt that a symptomatic individual who tests negative definitely should not self-isolate was highest among those viewing the UK wording (31/178, 17.4% (12.5%–23.7%)), and lowest among those viewing the NZ wording (6/159, 3.8% (1.6%–8.2%)). Although the NZ wording was rated harder to understand, participants reacted to the uncertainties given in the text in the expected direction: there was a small main effect of wording on trust in the result (p=0.048), with people perceiving the test result as marginally less trustworthy after having read the NZ wording compared with the UK wording. Positive results were generally viewed as more trustworthy and as having higher quality of evidence than negative results (both p<0.001).
Conclusions
The public’s default assessment of the face value of both the positive and negative test results (control group) indicate an awareness that test results are not perfectly accurate. Compared with other messaging tested, participants shown the UK’s 2020 wording about the interpretation of the test results appeared to interpret the results as more definitive than is warranted. Wording that acknowledges uncertainty can help people to have a more nuanced and realistic understanding of what a COVID-19 test result means, which supports decision making and behavioural response.
Preregistration and data repository
Preregistration of pilot at osf.io/8n62f , preregistration of main experiment at osf.io/7rcj4 , data and code available online ( osf.io/pvhba ).
Article activity feed
-
-
SciScore for 10.1101/2020.12.04.20243840: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement IRB: The study was conducted with ethical oversight from the University of Cambridge Psychology Research Ethics Committee (PRE.2020.034 with amendment 15th September 2020)
Consent: All participants gave informed consent before participation and were randomised using the ‘randomise’ function in Qualtrics.Randomization John takes the COVID-19 (swab) test to see whether he currently has the virus, and it is”… with the final word of the sentence being positive or negative, depending on whether they had been randomized into the positive or negative ‘test result’ condition. Blinding not detected. Power Analysis Power calculation: Based on the effect sizes achieved in the … SciScore for 10.1101/2020.12.04.20243840: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement IRB: The study was conducted with ethical oversight from the University of Cambridge Psychology Research Ethics Committee (PRE.2020.034 with amendment 15th September 2020)
Consent: All participants gave informed consent before participation and were randomised using the ‘randomise’ function in Qualtrics.Randomization John takes the COVID-19 (swab) test to see whether he currently has the virus, and it is”… with the final word of the sentence being positive or negative, depending on whether they had been randomized into the positive or negative ‘test result’ condition. Blinding not detected. Power Analysis Power calculation: Based on the effect sizes achieved in the pilot, we calculated the number of participants required to achieve 90% power to detect an effect size of η p2 =. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: Thank you for sharing your data.
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:The US’s Centers of Disease Control have taken the middle ground, merely adding the caveat that a recipient of a negative test result ‘probably’ doesn’t have COVID-19. These three pieces of information are not comparable in their aims and audience – they are not all information provided to test recipients – but we here compare their effects on the public’s interpretation of test results as a way of gauging how such information affects people’s natural understanding of such a test result. This study shows that with no additional information about a test result (control group), the UK public demonstrate conservative estimates of the reliability of the results, and have a lower confidence in a negative test result than a positive test result (which is reasonable given that specificity is larger than sensitivity for swab tests, which also holds for ‘quick’ lateral flow tests). The public’s estimate of the reliability and trustworthiness of the results correlates with their reaction to behavioural advice, where again they tend to show a precautionary approach to a negative result, although this may have been because the test recipient had been described as symptomatic and hence ideally the majority of participants would have felt he should self-isolate. The free text that participants gave to explain why they thought the test recipient should or should not self-isolate show how a large number chose to be ‘better safe than sorry’ (although fewer amongst those who were given informa...
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We found bar graphs of continuous data. We recommend replacing bar graphs with more informative graphics, as many different datasets can lead to the same bar graph. The actual data may suggest different conclusions from the summary statistics. For more information, please see Weissgerber et al (2015).
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
-
SciScore for 10.1101/2020.12.04.20243840: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement METHODS The study was conducted with ethical oversight from the University of Cambridge Psychology Research Ethics Committee (PRE.2020.034 with amendment 15th September 2020) Randomization John takes the COVID-19 ( swab ) test to see whether he currently has the virus , and it is”… with the final word of the sentence being positive or negative , depending on whether they had been randomized into the positive or negative ‘test result’ condition. Blinding not detected. Power Analysis Power calculation Based on the effect sizes achieved in the pilot , we calculated the number of participants required to achieve 90 % power to detect an effect size of η p2 =. Sex as a … SciScore for 10.1101/2020.12.04.20243840: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement METHODS The study was conducted with ethical oversight from the University of Cambridge Psychology Research Ethics Committee (PRE.2020.034 with amendment 15th September 2020) Randomization John takes the COVID-19 ( swab ) test to see whether he currently has the virus , and it is”… with the final word of the sentence being positive or negative , depending on whether they had been randomized into the positive or negative ‘test result’ condition. Blinding not detected. Power Analysis Power calculation Based on the effect sizes achieved in the pilot , we calculated the number of participants required to achieve 90 % power to detect an effect size of η p2 =. Sex as a biological variable Table 1: Characteristics of the participants Participant characteristics Gender: Age: Numeracy level* Education level Male Female 18-24 25-34 35-44 45-54 55-64 65+ 1 2 3 4 5 6 7 8 No formal education above age 16 Professional or technical qualifications above age 16 Number of participants (percentage) 834 (48%) 910 (52 Table 2: Resources
No key resources detected.
Results from OddPub: Thank you for sharing your data.
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
The US’s Centers of Disease Control have taken the middle ground, merely adding the caveat that a recipient of a negative test result ‘probably’ doesn’t have COVID-19. These three pieces of information are not comparable in their aims and audience – they are not all information provided to test recipients – but we here compare their effects on the public’s interpretation of test results as a way of gauging how such information affects people’s natural understanding of such a test result. This study shows that with no additional information about a test result (control group), the UK public demonstrate conservative estimates of the reliability of the results, and have a lower confidence in a negative test result than a positive test result (which is reasonable given that specificity is larger than sensitivity for swab tests, which also holds for ‘quick’ lateral flow tests). The public’s estimate of the reliability and trustworthiness of the results correlates with their reaction to behavioural advice, where again they tend to show a precautionary approach to a negative result, although this may have been because the test recipient had been described as symptomatic and hence ideally the majority of participants would have felt he should self-isolate. The free text that participants gave to explain why they thought the test recipient should or should not self-isolate show how a large number chose to be ‘better safe than sorry’ (although fewer amongst those who were given informa...
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We found bar graphs of continuous data. We recommend replacing bar graphs with more informative graphics, as many different datasets can lead to the same bar graph. The actual data may suggest different conclusions from the summary statistics. For more information, please see Weissgerber et al (2015).
Results from JetFighter: We did not find any issues relating to colormaps.
About SciScore
SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.
-
