Interim evaluation of Google AI forecasting for COVID-19 compared with statistical forecasting by human intelligence in the first week
This article has been Reviewed by the following groups
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
- Evaluated articles (ScreenIT)
Abstract
Background
Since June, Google (Alphabet Inc.) has provided forecasting for COVID-19 outbreak by artificial intelligence (AI) in the USA. In Japan, they provided similar services from November, 2020.
Object
We compared Google AI forecasting with a statistical model by human intelligence.
Method
We regressed the number of patients whose onset date was day t on the number of patients whose past onset date was 14 days prior, with information about traditional surveillance data for common pediatric infectious diseases including influenza, and prescription surveillance 7 days prior. We predicted the number of onset patients for 7 days, prospectively. Finally, we compared the result with Googles AI-produced forecast. We used the discrepancy rate to evaluate the precision of prediction: the sum of absolute differences between data and prediction divided by the aggregate of data.
Results
We found Google prediction significantly negative correlated with the actual observed data, but our model slightly correlated but not significant. Moreover, discrepancy rate of Google prediction was 27.7% for the first week. The discrepancy rate of our model was only 3.47%.
Discussion and Conclusion
Results show Googles prediction has negatively correlated and greater difference with the data than our results. Nevertheless, it is noteworthy that this result is tentative: the epidemic curve showing newly onset patients was not fixed.
Article activity feed
-
SciScore for 10.1101/2020.12.16.20248358: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
Software and Algorithms Sentences Resources By contrast, Google provided their prediction for 4 weeks from November 19, 2020. Googlesuggested: (Google, RRID:SCR_017097)Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:The present study has some limitations. First, although we evaluate two predictions in the first week, evaluations might be different for a longer time or a different phase of the outbreak. Therefore, …
SciScore for 10.1101/2020.12.16.20248358: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
Software and Algorithms Sentences Resources By contrast, Google provided their prediction for 4 weeks from November 19, 2020. Googlesuggested: (Google, RRID:SCR_017097)Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:The present study has some limitations. First, although we evaluate two predictions in the first week, evaluations might be different for a longer time or a different phase of the outbreak. Therefore, we must continue to evaluate the two predictions over a longer time. Secondly, the epidemic curve predicted by our model was not fixed until one month later. Therefore, evaluation for the prediction of our model might change up to one month. The obtained result should be regarded as tentative. By contrast, the evaluation of Google’s prediction will not change.
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
-
