Type I Error Inflation in Unexpected Event During Survey Designs

The `Unexpected Event During Survey Design' (UESD) leverages quasi-random timing of interviews relative to external events for causal identification. We evaluate the UESD's Type I (false positive) error rate using large-scale survey data covering 40 countries between 2002 and 2024. Based on over 45 million placebo tests, our analysis reveals that standard inferential approaches yield false positives at rates more than double the conventional 5\% threshold, indicating miscalibrated p-values and potentially erroneous conclusions. We show that this inflation primarily arises from two channels: changes in the composition of survey respondents over the fieldwork period and temporal trends in outcome variables. We further document significant heterogeneity in error inflation across countries, surveys, and outcomes. To address this, we propose an adjustment procedure tailored to context-specific false positive rates and provide open-source software for its implementation. We demonstrate our approach by applying it to two of the most highly cited UESD studies.