Statistical assessment of sample generalizability to a target population

A major issue that has arisen in recent years is the degree to which research results in the social and medical sciences can be replicated. This “replication crisis” has called into question a number of heretofore accepted results in the scientific literature and raised concerns among researchers about the degree to which what has been assumed as fact is indeed such. Underlying the issue of replicability is the extent to which the sample upon which study results obtained is generalizable to the target population the researchers are interested in making inferences about. The purpose of this manuscript is to describe several approaches for assessing generalizability of a sample to a target population using a variety of statistical tools. These methods involve comparing the sample used in the study with the target population of interest on a set of variables that are available for both groups prior to the conduct of the study. Such pre-study covariates can be compared between the groups individually and can also be used to create a single score that can be compared. This score is calculated using methods found in the propensity score literature with a variety of comparison methods proposed for comparing them. The current manuscript describes these methods that have been proposed in the literature and then demonstrates how to carry them out using packages available in the R software environment.