Comparative evaluation of methodologies for estimating the effectiveness of nonpharmaceutical interventions in the context of COVID-19: a simulation study

Numerous studies assessing the effectiveness of nonpharmaceutical interventions (NPIs) against COVID-19 have produced conflicting results, partly due to methodological differences. This study aims to clarify these discrepancies by comparing 2 frequently used approaches in terms of parameter bias and CI coverage of NPI effectiveness parameters. We compared 2-step approaches, where NPI effects are regressed on by-products of the first analysis, such as the effective reproduction number ${\mathcal{R}}_t$, with more integrated models that jointly estimate NPI effects and transmission rates in a single-step approach. We simulated datasets with mechanistic and agent-based models and analyzed them with both mechanistic models and a 2-step regression procedure. In the latter, ${\mathcal{R}}_t$ was estimated first and then used as the outcome in a linear regression with NPI variables as predictors. Mechanistic models consistently outperformed 2-step regressions, exhibiting minimal bias (0%-5%) and accurate CI coverage. Conversely, the 2-step regression showed bias up to 25%, with significantly lower-than-nominal CI coverage, reflecting challenges in uncertainty propagation. We identified additional challenges in the 2-step regression method, such high depletion of susceptibles and time lags in observational data. Our findings suggest caution when using 2-step regression methods for estimating NPI effectiveness.