Rethinking Repeatability in Observational Social Science
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Sociology has remained relatively insulated from debates about the ‘replication crisis.’ Heeding calls to consider replication more deeply, we introduce a distinction between two types of research reforms that have emerged in the wake of the crisis: specification-restricting reforms and specification-expanding reforms. Specification-restricting reforms—the more popular of the two—aim to increase the repeatability of research findings by controlling false positives. We show how these reforms’ internal logic breaks down outside of randomized experiments; in observational contexts, they risk enshrining fragile or misspecified models. We further argue that the premise of these reforms is flawed. Replication rates cannot be reduced to the purported prevalence of false positive findings. In their place, we propose a replication framework centered on specification-expanding reforms, stronger incentives for confirmatory research, and meta-analysis. This approach equips sociology to assess the repeatability of findings and build a more cumulative discipline.
