The Tip of the Iceberg? Insights into the Prevalence of Publication Bias in Two Probability-Based Academic Panels

Publication bias is the prioritized and selective reporting of scientifically significant results and often stems from the assumption that statistically significant research findings are rated as more important, credible, valuable, and publishable than nonsignificant or null findings. Previous work shows that this assumption makes researchers less likely to publish nonsignificant or null findings, a phenomenon termed file drawer bias. While prior research investigated the file drawer bias primarily at the study level (e.g., what share of studies has been published and which has not), we examine the prevalence of unpublished research on both the study and the hypothesis level (e.g. which hypotheses are published and which remain unpublished). We did so by comparing the submissions of 178 studies conducted in two probability-based German panel infrastructures (SOEP-IS and GESIS Panel) with their subsequent publications. We found substantial evidence for publication bias: after an average of more than six years of data availability, 43.8% of studies remained unpublished, 44.4% published at least one peer-reviewed article, and 11.8% published gray literature exclusively. Less than one third of submitted hypotheses were published, while the vast majority of published studies with hypotheses (>95%) included newly added (ad-hoc) hypotheses, and more than half of these studies published exclusively ad-hoc hypotheses. Taken together, over 80% of all published hypotheses were ad-hoc, and over 70% of hypotheses were supported. These findings highlight persistent incentives toward selective reporting and ad-hoc hypothesis formulation (potential HARKing). Our results raise concerns about ongoing publication bias in the social, behavioral, and economic sciences.