From metric to action: An evaluation framework to translate infectious disease forecasts into policy decisions

Like weather, energy and other areas of society, forecasts and decisions for infectious disease outbreaks are often critical. The difficult and consequential decisions are made under considerable time and societal pressure and in the face of great uncertainty. Ideally, such decisions can be supported by a probabilistic forecast – the prediction of the future value of an epidemiological quantity, together with uncertainty. Forecasting is challenged by noisy, incomplete, and delayed data alongside non-linear and evolving dynamics. Currently, performance metrics for infectious disease forecasts generally focus on the forecaster’s perspective; improving calibration and sharpness of forecasts. There is a lack of principled protocols to measure a forecast’s “value” – its ability to provide actionable insights for decision-makers. We develop a framework for epidemic decision-making, focusing on three aspects: i) translating forecasts and popular forecast evaluation metrics into interpretable, public-health-relevant quantities; ii) refocusing evaluations on decision-makers and epidemic events for more actionable outputs; iii) linking predictability of an epidemic to the value of forecasts for decision-making. By melding concepts from weather forecasting, information theory, and decision theory, our framework bridges conceptual gaps between forecasters and decision-makers. We illustrate the framework with an application to forecasts of weekly incident COVID-19 cases, where ensemble models often, but not necessarily, provided the most value for different decision-makers, epidemic events, and across space and time. The evaluation framework connects statistical concepts with decision-theoretic principles to promote more actionable and reliable infectious disease forecasts for enhancing public health responses.