Identity-by-descent (IBD) segment outlier detection in endogamous populations using pedigree cohorts

Genomic segments that are inherited from a common ancestor are referred to as identical-by-descent (IBD). Because these segments are inherited, they not only allow us to study diseases, population characteristics, and the sharing of rare variants, but also understand hidden familial relationships within populations. Over the past two decades, various IBD finding algorithms have been developed using hidden Markov models (HMMs), hashing and extension, and Burrows-Wheeler Transform (BWT) approaches. In this study, we investigate the utility of pedigree information in IBD outlier detection methods for endogamous populations. With the increasing prevalence of computationally efficient sequencing technology and proper documentation of pedigree structures, we expect complete pedigree information to become readily available for more populations. While IBD segments have been used to reconstruct pedigrees, because we now have access to the pedigree, it is a natural question to ask if including pedigree information would substantially improve IBD segment finding for the purpose of studying inheritance. We propose an IBD pruning algorithm for reducing the number of false positives in IBD segments detected by existing software. While existing software already identify IBD segments with high success rates, our algorithm analyzes the familial relationships between cohorts of individuals who are initially hypothesized to share IBD segments to remove outliers. Our algorithm is inspired by a k-Nearest Neighbors (kNN) approach with a novel distance metric for pedigrees with loops. We apply our method to simulated genomic data under an Amish pedigree, but it could be applied to pedigrees from other human populations as well as domesticated animals such as dogs and cattle.