A Comparative Evaluation of Multiple Hypothesis Testing Adjustment Methods

This paper provides empirical guidance to researchers when choosing an adjustment method during multiple hypothesis testing. It is shown that methods vary greatly in their false positive rate, not just among each other but also when the p values of the tests stem from a different distribution. It is recommended that researchers carefully choose their adjustment methods, as the choice significantly affects the interpretation of findings. When the aim of the adjustment is to control Family-Wise Error, strict and simple methods like the Bonferroni correction or Union-Intersection tests offer great practical applicability. When controlling False-Discovery Rate, more powerful methods like the Benjamini-Hochberg or Simes-Hochberg methods are more appropriate.