Monitoring global trends in Covid-19 vaccination intention and confidence: a social media-based deep learning study

  1. Xinyu Zhou
  2. Alex de Figueiredo
  3. Qin Xu
  4. Leesa Lin
  5. Per E Kummervold
  6. Heidi Larson
  7. Mark Jit
  8. Zhiyuan Hou

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Background

This study developed deep learning models to monitor global intention and confidence of Covid-19 vaccination in real time.

Methods

We collected 6.73 million English tweets regarding Covid-19 vaccination globally from January 2020 to February 2021. Fine-tuned Transformer-based deep learning models were used to classify tweets in real time as they relate to Covid-19 vaccination intention and confidence. Temporal and spatial trends were performed to map the global prevalence of Covid-19 vaccination intention and confidence, and public engagement on social media was analyzed.

Findings

Globally, the proportion of tweets indicating intent to accept Covid-19 vaccination declined from 64.49% on March to 39.54% on September 2020, and then began to recover, reaching 52.56% in early 2021. This recovery in vaccine acceptance was largely driven by the US and European region, whereas other regions experienced the declining trends in 2020. Intent to accept and confidence of Covid-19 vaccination were relatively high in South-East Asia, Eastern Mediterranean, and Western Pacific regions, but low in American, European, and African regions. 12.71% tweets expressed misinformation or rumors in South Korea, 14.04% expressed distrust in government in the US, and 16.16% expressed Covid-19 vaccine being unsafe in Greece, ranking first globally. Negative tweets, especially misinformation or rumors, were more engaged by twitters with fewer followers than positive tweets.

Interpretation

This global real-time surveillance study highlights the importance of deep learning based social media monitoring to detect emerging trends of Covid-19 vaccination intention and confidence to inform timely interventions.

Funding

National Natural Science Foundation of China.

Research in context

Evidence before this study

With COVID-19 vaccine rollout, each country should investigate its vaccination intention in local contexts to ensure massive vaccination. We searched PubMed for all articles/preprints until April 9, 2021 with the keywords “(“Covid-19 vaccines”[Mesh] OR Covid-19 vaccin*[TI]) AND (confidence[TI] OR hesitancy[TI] OR acceptance[TI] OR intention[TI])”. We identified more than 100 studies, most of which are country-level cross-sectional surveys, and the largest global survey of Covid-19 vaccine acceptance only covered 32 countries to date. However, how Covid-19 vaccination intention changes over time remain unknown, and many countries are not covered in previous surveys yet. A few studies assessed public sentiments towards Covid-19 vaccination using social media data, but only targeting limited geographical areas. There is a lack of real-time surveillance, and no study to date has globally monitored Covid-19 vaccination intention in real time.

Added value of this study

To our knowledge, this is the largest global monitoring study of Covid-19 vaccination intention and confidence with social media data in over 100 countries from the beginning of the pandemic to February 2021. This study developed deep learning models by fine-tuning a Bidirectional Encoder Representation from Transformer (BERT)-based model with 8000 manually-classified tweets, which can be used to monitor Covid-19 vaccination beliefs using social media data in real time. It achieves temporal and spatial analyses of the evolving beliefs to Covid-19 vaccines across the world, and also an insight for many countries not yet covered in previous surveys. This study highlights that the intention to accept Covid-19 vaccination have experienced a declining trend since the beginning of the pandemic in all world regions, with some regions recovering recently, though not to their original levels. This recovery was largely driven by the US and European region (EUR), whereas other regions experienced the declining trends in 2020. Intention to accept and confidence of Covid-19 vaccination were relatively high in South-East Asia region (SEAR), Eastern Mediterranean region (EMR), and Western Pacific region (WPR), but low in American region (AMR), EUR, and African region (AFR). Many AFR countries worried more about vaccine effectiveness, while EUR, AMR, and WPR concerned more about vaccine safety (the most concerns with 16.16% in Greece). Online misinformation or rumors were widespread in AMR, EUR, and South Korea (12.71%, ranks first globally), and distrust in government was more prevalent in AMR (14.04% in the US, ranks first globally). Our findings can be used as a reference point for survey data on a single country in the future, and inform timely and specific interventions for each country to address Covid-19 vaccine hesitancy.

Implications of all the available evidence

This global real-time surveillance study highlights the importance of deep learning based social media monitoring as a quick and effective method for detecting emerging trends of Covid-19 vaccination intention and confidence to inform timely interventions, especially in settings with limited sources and urgent timelines. Future research should build multilingual deep learning models and monitor Covid-19 vaccination intention and confidence in real time with data from multiple social media platforms.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2021.04.17.21255642: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Development of deep learning model: Bidirectional Encoder Representation from Transformer (BERT), proposed by Google in 2018, is a 12/24-layer deep neural network with 110/340 million parameters that outperformed all models before it on the General Language Understanding Evaluation benchmark, with 7.0% absolute improvement over the prior state of the art model.13,14 By pre-training multi-layer bidirectional Transformer encoder with BooksCorpus (800 million words) and English Wikipedia (2,500 million words) to better “understand” a language, BERT can be applied to classify textual data by fine-tuning (aka. adjusting parameters) with a limited manually-classified downstream training set.
    Google
    suggested: (Google, RRID:SCR_017097)
    Data was analyzed with Python.
    Python
    suggested: (IPython, RRID:SCR_001658)

    Results from OddPub: Thank you for sharing your code and data.

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Strengths and limitations: The extensiveness, timeliness and accessibility of social media data, as well as the efficiency and accuracy of the pretrained deep learning models we used, provide a quick and effective method for detecting vaccination intention and confidence, especially in settings with limited sources and urgent timelines. This study could monitor populations who may not be represented in traditional surveys, and real-time social media data can eliminate reporting biases when speaking to a researcher18. Our findings can also be used as a reference point for survey data on a single country’s Covid-19 vaccination intention in the future. This study has several limitations. First, although tweet data reflect the opinions of all Twitter users, such users may not be representative of the general population of the country. They are generally younger and more literate35. There may also be response biases since users may express their opinions differently online compared to in person. These biases are shared among all social media studies. Second, to ensure high performance of deep learning models, we did not train multilingual models. Our model only analyzed English tweets, reducing the representativeness for non-English speaking countries. Multilingual deep learning models are needed in other languages. Despite these limitations, our data and methods provide a feasible approach to gaining valuable insight into an urgent area of research. Where traditional surveys are ...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2021.04.17.21255642v1 on medRxiv