Uncovering Survivorship Bias in Longitudinal Mental Health Surveys During the COVID-19 Pandemic

  1. M. Czeisler
  2. J. Wiley
  3. C. Czeisler
  4. S. Rajaratnam
  5. M. Howard

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

Aims

Markedly elevated adverse mental health symptoms were widely observed early in the coronavirus disease 2019 (COVID-19) pandemic. Unlike the U.S., where cross-sectional data indicate anxiety and depression symptoms have remained elevated, such symptoms reportedly declined in the U.K., according to analysis of repeated measures from a largescale longitudinal study. However, nearly 40% of U.K. respondents (those who did not complete multiple follow-up surveys) were excluded from analysis, suggesting that survivorship bias might partially explain this discrepancy. We therefore sought to assess survivorship bias among participants in our longitudinal survey study as part of The COVID-19 Outbreak Public Evaluation (COPE) Initiative.

Methods

Survivorship bias was assessed 4,039 U.S. respondents who completed surveys including the assessment of mental health as part of The COPE Initiative in April 2020 and were invited to complete follow-up surveys. Participants completed validated screening instruments for symptoms of anxiety, depression, and insomnia. Survivorship bias was assessed for (1) demographic differences in follow-up survey participation, (2) differences in initial adverse mental health symptom prevalences adjusted for demographic factors, and (3) differences in follow-up survey participation based on mental health experiences adjusted for demographic factors.

Results

Adjusting for demographics, individuals who completed only one or two out of four surveys had higher prevalences of anxiety and depression symptoms in April 2020 (e.g., one-survey versus four-survey, anxiety symptoms, adjusted prevalence ratio [aPR]: 1.30, 95% confidence interval [CI]: 1.08-1.55, P =0.0045; depression symptoms, aPR: 1.43, 95% CI: 1.17-1.75, P =0.00052). Moreover, individuals who experienced incident anxiety or depression symptoms had higher odds of not completing follow-up surveys (adjusted odds ratio [aOR]: 1.68, 95% CI: 1.22-2.31, P =0.0015, aOR: 1.56, 95% CI: 1.15-2.12, P =0.0046, respectively).

Conclusions

Our findings revealed significant survivorship bias among longitudinal survey respondents, indicating that restricting analytic samples to only respondents who provide repeated assessments in longitudinal survey studies could lead to overly optimistic interpretations of mental health trends over time. Cross-sectional or planned missing data designs may provide more accurate estimates of population-level adverse mental health symptom prevalences than longitudinal surveys.

Article activity feed

  1. Version published to 10.1101/2021.01.28.21250694v4 on medRxiv
  2. ScreenIT

    SciScore for 10.1101/2021.01.28.21250694: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board StatementConsent: Study approval and informed consent: The Monash University Human Research Ethics Committee approved the study protocol.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Analyses were conducted in R (version 4.0.2; The R Foundation)29 with the R survey package (version 3.29)30-32 and Python (version 3.7.8).
    Python
    suggested: (IPython, RRID:SCR_001658)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Understanding strengths and limitations of study approaches should inform the design and interpretation of findings.34 Longitudinal studies have advantages, including increased power to detect causal pathways and mediating factors, reduced reliance on recall bias, and establishment of the order in which events and outcomes occur. However, survivorship bias in longitudinal mental health surveys suggest that longitudinal samples may be non-representative of population-level mental health. While unable to determine causation, cross-sectional studies can more rapidly generate data, and our data provide further evidence that cross-sectional data may be more reliable for the assessment of population-level prevalences of adverse mental health symptoms at a given timepoint.35 For future study designs, researchers could consider implementing a planned missing data design36 to benefit from the strengths of these study designs while minimizing associated biases. Strengths of this analysis include four timepoints to assess response bias, high initial response (61.7%) and retention (39.6% of respondents completed three of four surveys) rates, utilization of clinically validated screening instruments, and implementation of quota sampling and survey weighting to improve sample representativeness by national estimates for gender, age, and race/ethnicity. Moreover, multiple types of survivorship bias were assessed, including differential demographic attrition and demographic-adjusted assessme...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2021.01.28.21250694v3 on medRxiv
  4. Version published to 10.1101/2021.01.28.21250694v2 on medRxiv
  5. Version published to 10.1101/2021.01.28.21250694v1 on medRxiv
  6. Version published to 10.1017/s204579602100038x