False Positive Psychology? An Empirical Investigation of the False Positive Risk in Psychological Science Before and After the Credibility Revolution

Over the past decade, concerns about selective reporting and questionable research practicesraised doubts about the credibility of psychological science. Using z-curve, a method that detectsand corrects for selection bias, we estimated the false positive risk in this journal's literaturethrough 2024. Automated analyses of all tests and hand-coded focal tests (2003–2018) revealedstrong selection bias, though it has declined since 2015 due to increased sample sizes andstatistical power. The estimated false positive risk was 15% (95% CI: 6–32%) overall and 20%(95% CI: 7–52%) for focal tests. Social and cognitive psychology showed similar results andpreregistration had small effects. A stricter alpha (.01) can limit the false positive risk below 5%.While many findings appear credible, modest statistical power remains a concern, highlightingthe need for larger replication samples to avoid false negatives.