Weighting the weeks that matter: entropic tilting for epidemic forecast evaluation

Read the full article See related articles

Listed in

This article is not in any list yet, why not save it to one of your lists.
Log in to save this article

Abstract

The performance of infectious disease forecasting models is commonly calculated under the implicit assumption that forecast accuracy is equally important across time. This approach may obscure or under-weight differences in model performance during the epidemic periods that matter most for public-health decisions. Here, we introduce the weighted prequential score (WPS), a forecast evaluation framework that can emphasize epidemic periods of interest. The WPS is a weighted average of scores over time, where weights are constructed through entropic tilting which up-weights priority periods while keeping the WPS as close as possible to the arithmetic average. We demonstrate the WPS on forecasts of incident hospitalizations submitted to the CDC FluSight challenge, focusing on two priority periods: weeks of rapid growth in incidence and weeks near the seasonal peak. Across season-location-horizon pairs, doubling the weight assigned to priority periods changes the top-ranked model in 10%–26% of eligible settings, while the top-ranked model remains unchanged regardless of the amount of weight assigned in 37%-68% of settings. We demonstrate that for suitable weights, the models favored by the WPS yield an average gain in accuracy during priority periods that is at least as large as the average accuracy lost during the remaining weeks. Because weights can be defined using observed outcomes, the WPS is not a proper scoring rule. Hence, we recommend using the WPS as a retrospective evaluation diagnostic. The WPS can guide public-health practitioners in selecting models that perform well during periods most relevant to their decisions, while encouraging forecasting teams to improve accuracy during these critical weeks.

Author summary

In the context of public health decisions, the importance of accurate infectious disease forecasts may vary over the course of an epidemic. For example, accurate forecasts of incident hospitalizations around the seasonal peak may be particularly important for hospital staffing decisions. We introduce a forecast scoring rule that allows users to emphasize accuracy during the epidemic phases most relevant to their decisions. We demonstrate our approach on forecasts of influenza hospitalization incidence, focusing on periods of rapid growth and weeks surrounding the seasonal peak. We find our approach leads to model rankings that prioritize accuracy during these periods with minimal losses in performance across the entire epidemic.

Article activity feed