Reciprocal sign epistasis and sign epistasis may not be found in multi-peaked landscapes

Fitness landscapes is a mathematical framework for rigorously analyzing evolutionary dynamics including study of epistasis, the main obstacle to predicting phenotype from genotype. In 2011, Poelwijk et al . formulated a theorem asserting that in a multi-peaked fitness landscape, “at least two mutations exhibit reciprocal sign epistasis” (Poelwijk et al ., J. Theor. Biol ., 272:141). However, their proof disregarded neutral mutations – a critical limitation that significantly diminishes both the theoretical value and practical applicability of the theorem.

In this study, we reexamine the relationship between reciprocal sign epistasis (RSE) and multi-peaked fitness landscapes under the more general and realistic condition that neutral mutations are present. Our analysis reveals that when neutral mutations are allowed, the original theorem no longer holds. Instead, in cases when “strictly neutral” mutations could occur, the presence of RSE is guaranteed only if considering composite mutations – that is, groups of one or more mutations considered collectively in all possible combinations. Furthermore, when “near-neutral” mutations could occur – cases where the effect of a mutation is indistinguishable from zero – we cannot ensure statistically significant detection of RSE in the multi-peaked fitness landscapes even if considering composite mutations.

Our findings have profound theoretical and practical implementations. First, – contrary to one of the consequences of Poelwijk et al .’s theorem – for each mutation in the considered multi-peaked landscape, the compensating mutation(s) do not have to be adjacent in the course of evolution: the compensation can occur after some neutral mutations take place. Second, in real-world scenarios where fitness values are measured with some uncertainty, there may be a fundamental impossibility to detect RSE in a statistically significant manner in multi-peaked fitness landscapes (both at the level of single and composite mutations).