Leveraging weighted quartet distributions for enhanced species tree inference from genome-wide data

Species tree estimation from genes sampled from throughout the whole genome is challeng-ing in the presence of gene tree discordance, often caused by incomplete lineage sorting (ILS), where alleles can coexist in populations for periods that may span several speciation events. Quartet-based summary methods for estimating species trees from a collection of gene trees are becoming popular due to their high accuracy and theoretical guarantees of robustness to arbitrarily high amounts of ILS. ASTRAL, the most widely used quartet-based method, aims to infer species trees by maximizing the number of quartets in the gene trees that are consistent with the species tree. An alternative approach (as in wQFM) is to infer quartets for all subsets of four species and amalgamate them into a coherent species tree. While summary methods can be highly sensitive to gene tree estimation errors–especially when gene trees are derived from short alignments–quartet amalgamation offers an advantage by potentially bypassing the need for gene tree estimation. However, greatly understudied is the choice of weighted quar-tet inference method and downstream effects on species tree estimations under realistic model conditions. In this study, we investigated a broad range of methods for generating weighted quartets and critically assessed their impact on species tree inference. Our results on a collec-tion of simulated and empirical datasets suggest that amalgamating quartets weighted based on gene tree frequencies (GTF) typically produces more accurate trees than leading quartet-based methods like ASTRAL and SVDquartets. Further enhancements in GTF-based weighted quar-tet estimation were achieved by accounting for gene tree uncertainty, through the utilization of a distribution of trees for each gene (instead of a single tree), by employing traditional nonpara-metric bootstrapping methods or Bayesian MCMC sampling. Our study provides evidence that the careful generation and amalgamation of weighted quartets, as implemented in methods like wQFM, can lead to significantly more accurate trees compared to widely employed methods like ASTRAL, especially in the face of gene tree estimation errors.