Evolution under Stochastic Transmission: Mutation-Rate Modifiers

In evolutionary models of large populations, it is common to analyze the effects of cyclic or random variation in the parameters that describe selection. It is less common, however, to study how stochasticity in the genetic transmission process itself affects evolutionary outcomes. Suppose that a gene locus has alleles A and a under constant selection. This locus is linked to a modifier locus with alleles M ₁ and M ₂ , which control the mutation rate from A to a . The Reduction Principle states that, near a mutation–selection balance where M ₁ is fixed with mutation rate u ₁ , a rare allele M ₂ can invade if its associated rate u ₂ is lower than u ₁ . This result, valid for both haploids and diploids, assumes constant mutation rates through time. We extend this framework by allowing the mutation rate associated with M ₂ to fluctuate randomly across generations, denoted as u _{2, t} . In this stochastic setting, the condition for invasion by a new modifier allele depends not only on the resident mutation rate u ₁ and the mean mutation rate u ₂ associated with the invading allele, but also on the temporal distribution of u _{2, t} , the strength of selection at the A/a locus, and the recombination rate between M ₁ /M ₂ and A/a . The analysis shows how stochasticity and recombination in transmission do not simply modify the magnitude of evolutionary change predicted under deterministic assumptions. Instead, through their interaction with selection and linkage, they can generate conditions under which the direction of modifier evolution is qualitatively reversed relative to the deterministic Reduction Principle .