Calculating and interpreting F _ST in the genomics era

The relative genetic distance between populations is commonly measured using the fixation index ( F _ST ). Traditionally inferred from allele frequency differences, the question arises how F _ST can be estimated and interpreted when analysing genomic datasets with low sample sizes. Here, we advocate an elegant solution first put forward by Hudson et al. (1992): F _ST = ( D _xy – π _xy )/ D _xy , where D _xy and π _xy denote mean sequence dissimilarity between and within populations, respectively. This multi-locus F _ST -metric can be derived from allele frequency data, but also from sequence alignment data alone, even when sample sizes are low and/or unequal. As with other F _ST -metrices, the numerator denotes net divergence ( D _a ), which is equivalent to the f ² -statistic and Nei’s D (for realistic estimates of D _xy and π _xy ). In terms of demographic inference, net divergence measures the difference in increase of D _xy and π _xy since the population split, owing to a reduction of coalescence times within populations as a result of genetic drift. Because different combinations of ΔD _xy and Δπ _xy can produce identical F _ST -estimates, no universal relationship exists between F _ST and population split time. Still, in case of recent population splits, when novel mutations are negligible, F _ST -estimates can be accurately converted into coalescent units ( τ . i.e., split time in multiples of 2 N _e ). This then allows to quantify gene tree discordance, without the need for multispecies coalescent based analyses, using the formula: P _discordance = ⅔·(1 – F _ST ). To facilitate the use of the Hudson F _ST -metric, we implemented new utilities in the R package SambaR.