Performance evaluation of RespiCast ensemble forecasts for primary care syndromic indicators of viral respiratory disease in Europe during the 2023/24 winter season

During the 2023/24 winter season, the European Centre for Disease Prevention and Control (ECDC) launched RespiCast, the first European Respiratory Diseases Forecasting Hub, to provide probabilistic forecasts for influenza-like illness (ILI) and acute respiratory infection (ARI) incidence across 27 European countries. Spanning 20 forecasting rounds, RespiCast collected one-to four-week-ahead forecasts from 14 models for ILI and 10 models for ARI, contributed by different international teams, and combined them into an ensemble. Our analysis shows that the ensemble consistently outperformed the baseline model (a naive, uninformed model that projects the last observed value forward) in over 90% of countries for both ILI and ARI incidence when evaluated using the Weighted Interval Score (WIS), with the ensemble outperforming individual models in most forecasting rounds. Analysis of ensemble prediction coverage (proportion of times that the observed values fall within the specified prediction intervals) indicated that forecast prediction intervals were reliable in the short term, though an overconfidence trend (i.e. prediction intervals that are too narrow) was observed at longer forecasting horizons. The performance of the ensemble declined in certain weeks, likely due to reduced participation from modeling teams, rapid shifts in epidemic dynamics during specific phases, higher data noise, and reporting delays. Forecast scores varied across countries, with some exhibiting consistently higher errors than others. Overall, the findings highlight the strengths of ensemble forecasting in improving the accuracy and reliability of epidemiological forecasts while also identifying areas for improvement, such as managing overconfidence and addressing variability in performance across countries and over time.