Country-Specific Estimates of Misclassification Rates of Computer-Coded Verbal Autopsy Algorithms

Background

Computer-coded verbal autopsy (CCVA) algorithms are commonly used to determine individual causes of death (COD) and population-level cause-specific mortality fractions (CSMF), but frequent COD misclassification leads to biased CSMF estimates. The VA-calibration framework [1,2] reduces bias by estimating misclassification rates from limited CHAMPS data, but it overlooks country-level variation in these rates, reducing the accuracy of CSMF estimates.

Methods

Utilizing CHAMPS data and the framework from [3], we estimate VA misclassification rates for three widely used CCVA algorithms (EAVA, InSilicoVA, InterVA), two age groups (neonates 0-27 days and children 1-59 months), and eight countries (Bangladesh, Ethiopia, Kenya, Mali, Mozambique, Sierra Leone, South Africa, other). We then use the Mozambique-specific rates to calibrate VA-only data from the COMSA project in Mozambique.

Findings

We report three key findings. First, the country-specific model better fits CHAMPS misclassification rates than the homogeneous model, reducing average absolute loss by 34-38% for neonates and 13-24% for children. Second, CCVA algorithms show consistent misclassification patterns, systematically over- or underestimating certain causes. Third, calibrating COMSA data increases neonatal CSMF for sepsis/meningitis/infection and decreases it for intrapartum-related events (IPRE) and prematurity; among children, CSMF increases for malaria and decreases for pneumonia.

Interpretation

We generate VA misclassification rate estimates across two age groups, three CCVA algorithms, and eight countries. These publicly available estimates enable calibration of VA-only data from any country without needing access to CHAMPS data. The analysis also highlights systematic algorithm biases, providing direction for future improvements.

Research in context

Evidence before this study: Computer-coded verbal autopsy (CCVA) algorithms are routinely used to determine individual causes of death (COD) and population-level cause-specific mortality fractions (CSMF). However, these algorithms frequently misclassify the COD, leading to biased CSMF estimates. A recently developed framework, VA-calibration [1,2], corrects this bias by accounting for CCVA misclassification rates, estimated using COD inferred from minimally invasive tissue sampling (MITS) from the Child Health and Mortality Prevention Surveillance (CHAMPS) Network. However, there is significant variation in the VA misclassification rates across countries. Currently, VA-calibration does not account for this country-level variation, which diminishes CSMF estimation accuracy in a target population.

Added value of this study: We utilize the CHAMPS data and a recent Bayesian approach [3], and estimate country-specific CCVA misclassification rates. We compare the misclassification rates observed in CHAMPS with their estimates from homogeneous and county-specific models for three widely used CCVA algorithms (EAVA, InSilicoVA, InterVA), two age groups (neonates aged 0-27 days and children aged 1-59 months), and eight countries (Bangladesh, Ethiopia, Kenya, Mali, Mozambique, Sierra Leone, South Africa, and other). To demonstrate their practical application, we apply Mozambique-specific misclassification rates to VA-only data from the Countrywide Mortality Surveillance for Action (COMSA) project in Mozambique and produce national-level calibrated CMSF estimates for neonates and children.

We report three main findings. First, the country-specific estimates of the VA misclassification rates are more concordant with their observed rates in CHAMPS compared to their estimates from the homogeneous model, reducing average absolute loss by 34-38% for neonates and 13-24% for children. Second, each algorithm exhibits a systematic pattern of misclassification, consistently over- or underpredicting certain causes. Third, consistent with previous findings, calibrating COMSA data using Mozambique-specific misclassification estimates leads to notable shifts from uncalibrated CSMF estimates across the three algorithms. Among neonates, estimates generally increase for sepsis/meningitis/infection, while those for intrapartum-related events (IPRE) and prematurity decrease. The estimates among children rise for malaria and decline for pneumonia.

Implications of all the available evidence: We produce an inventory of VA misclassification rates resolved by two age groups, three CCVA algorithms, and country. These estimates will be made publicly available, serving as a vital resource for calibrating VA-only data from any country. The systematic biases in the algorithms quantified by the analysis provide valuable insights into the algorithms’ functioning, providing opportunities for their future improvements.

Accurate mortality data are fundamental to designing effective public health policies and achieving the Sustainable Development Goals. This research is thus highly relevant to global health, particularly for children under age five in low- and middle-income countries where vital registration systems remain incomplete. With the growing reliance on computer-coded algorithms, amid rapid AI advancements in cause-of-death determination, and their known risk of misclassification, our proposed integration of VA-calibration into the verbal autopsy workflow offers a crucial advancement in improving the accuracy of AI-powered mortality surveillance.

Key Messages

• We improve VA-calibration by using CHAMPS data and a country-specific Bayesian model to account for systematic and cross-country variation in CCVA misclassification rates.

• We provide uncertainty-quantified, country-specific misclassification estimates across two age groups, three CCVA algorithms, and eight country categories (including an ‘other’ group for countries outside CHAMPS), enabling VA-calibration for any country without requiring access to CHAMPS data.

• We showcase their utility by using Mozambique-specific misclassification estimate to calibrate VA-only data from Mozambique’s COMSA project, refining CSMF estimates among neonates and children.