Habitat complexity alters the strength of sexual selection on brain size in a livebearing fish

Animals often reproduce in complex environments, which should generate selection for both enhanced detectability in signaling traits and improved cognitive processing abilities. However, the extent to which signaling and cognitive traits have evolved to overcome the challenges of interacting in complex habitats remains understudied. We examined whether habitat complexity influences sexual selection in the pygmy halfbeak, Dermogenys collettei , a small livebearing freshwater fish. Using free-swimming arenas, we created low- and high-complexity environments and observed mating behaviors in mixed-sex groups. While the opportunity for sexual selection did not differ significantly between environments for either sex, we observed positive selection gradients for both male and female brain size in open arenas, but not in complex habitats. Interestingly, no clear selection on male or female ornamentation was detected in either environment. Our findings suggest that habitat complexity may limit selection pressures on cognitive traits, such as brain size, without significantly affecting visual ornamentation’s role in mating success.