Error rates in Q _ST –F _ST comparisons depend on genetic architecture and estimation procedures

Genetic and phenotypic variation among populations is one of the fundamental subjects of evolutionary genetics. One question that arises often in data on natural populations is whether differentiation among populations on a particular trait might be caused in part by natural selection. For the past several decades, researchers have used Q _ST – F _ST approaches to compare the amount of trait differentiation among populations on one or more traits (measured by the statistic Q _ST ) with differentiation on genome-wide genetic variants (measured by F _ST ). Theory says that under neutrality, F _ST and Q _ST should be approximately equal in expectation, so Q _ST values much larger than F _ST are consistent with local adaptation driving subpopulations’ trait values apart, and Q _ST values much smaller than F _ST are consistent with stabilizing selection on similar optima. At the same time, investigators have differed in their definitions of genome-wide F _ST (such as “ratio of averages” vs. “average of ratios” versions of F _ST ) and in their definitions of the variance components in Q _ST . Here, we show that these details matter. Different versions of F _ST and Q _ST have different interpretations in terms of coalescence time, and comparing incompatible statistics can lead to elevated type I error rates, with some choices leading to type I error rates near one when the nominal rate is 5%. We conduct simulations under varying genetic architectures and forms of population structure and show how they affect the distribution of Q _ST . When many loci influence the trait, our simulations support procedures grounded in a coalescent-based framework for neutral phenotytpic differentiation.