Evaluating Network Replicability Across Local, Mesoscale, and Global Structures

Debates on the replicability and generalizability of network models in psychology have led some researchers to conclude that they have limited utility and replicability. The majority of these debates have concentrated on whether individual edges and (rank-order) centrality measures replicate in a few empirical datasets. Some simulation studies have evaluated whether edges can be accurately recovered but few have examined the extent to which networks are replicable. This study aims to fill that gap by employing a Monte Carlo simulation to evaluate the extent to which psychometric networks replicate and at what level of analysis (edges, centrality, community, network, prediction). Several network estimation methods were applied to evaluate regularized and non-regularized approaches. Across the simulated conditions, number of data categories (continuous, polytomous, dichotomous) and sample size (500, 1000, 2500, 5000) were the two most influential factors on replicability. Across the network estimation methods, EBICglasso was consistently at or near the best performing in terms of replicability. The simulation results were supported by 100 split half analyses on three empirical examples. Based on our results, we provide sample size recommendations for minimum replicability at each level of analysis. We conclude with comparisons to previous empirical findings and call for work to determine what "replicable" means in the context of psychometric networks.