On the Robustness of Statistical Results to Data Exclusion in t-Tests

Data exclusion is common in psychology and can potentially impact statistical results and conclusions. We developed a simulation-based methodology, along with a Shiny app, to assess the robustness of statistically significant results under different mechanisms of data exclusion within the framework of Null Hypothesis Significance Testing. Additionally, we applied our method to 77 t-tests in a sample of 16 articles published in the Journal of Personality and Social Psychology in 2014. We found that statistical conclusions (i.e., rejection of H0 given α=.05) are robust to randomized data exclusion (RDE) if the reported two-sided p-value ≤ .01, and to selecting the best of ten subsets after selective randomized data exclusion (SRDE) if the reported two-sided p-value ≤ .001. A small portion of articles reported sample sizes per group before (1.3%) and after data exclusion (2.6%) among 76 independent samples t-tests, complicating direct robustness analyses. Of the statistically significant t-test results reported in the articles, 71% was robust to RDE and 56% was robust to SRDE. Besides detailed study preregistration to prevent potential bias from data exclusion, we recommend that authors explicitly report the reasons for exclusion and provide detailed information about the sample sizes before and after data exclusion.