The Accuracy of Survey Measures on Political Activity on Facebook. Evidence from a data donation study

Self-reported survey measures of social media use are commonly used across disciplines, yet the validity of findings in this area depends heavily on the accuracy of these reports. In political communication research, measurement error in reported online political behavior can distort estimates of media effects and lead to misleading conclusions about online political engagement. To address this concern, our study leveraged donated digital trace data from 758 Hungarian Facebook users and compared it with responses to standard survey questions provided by the same individuals. In addition, in a pre-registered experiment, we varied the ordering of the survey items. Our findings reveal moderate correlations between self-reported and actual digital behavior. General items showed closer alignment with observed behavior than self-reports of specific political activities. Participants typically underestimated their general Facebook activities and overestimated their political engagement on Facebook. Importantly, some of this discrepancy may stem not from respondents’ inaccuracy but from difficulties in classifying digital actions as politically relevant. Survey measures were more reliable in the general-specific questionnaire design and among individuals with high political interest. We discuss potential avenues for improving survey-based measures and emphasize the need for further research to address the methodological challenges associated with the use of digital trace data.