The influence of incompatibilities and heterosis on hybrid population genetics

A long-standing question in evolutionary biology is: Under what circumstances can speciation occur despite hybridisation or because of hybridisation? Some models of hybrid incompatibilities predict that speciation can occur even in the presence of gene flow and strong selection against hybrids, an outcome also influenced by genetic contributions from parental species and genetic architecture. On the other hand, empirical work has shown that heterosis can counteract the effect of incompatibilities, hindering the speciation process. Theoretical models that simultaneously consider the positive and negative impacts of hybridisation on fitness remain scarce, raising questions about the effects of hybrid incompatibilities in the presence of heterosis. To address this question, we study how (Bateson)-Dobzhansky-Muller incompatibilities (BDMIs) interact with overdominant mutations in a two-locus population genetics model of an isolated hybrid population. We find that the strength of overdominance relative to incompatibilities determines the long-term genetic composition of the hybrid population. We show that high recombination exposes incompatibilities to selection and reduces the frequency of derived alleles in the hybrid population, limiting the strength of BDMIs that can be maintained by the balance with overdominance. We also show how neutral variation is affected by the strength of selection and the recombination rate between incompatible loci, generating patterns that include an increase in local variation resembling associative overdominance, or a reduction resembling background selection. Such variation of neutral variation, particularly at intermediate distances from BDMI loci, can generate peaks or troughs of diversity that are explained by the recombination rate between BDMI loci, and initial proportions of admixture between parental populations. Our work demonstrates how the genomic conflict caused by the interplay of overdominance and hybrid incompatibilities, recombination, and parental contributions, shape the genome of an isolated hybrid population.