Predicting Replicability Challenge: Rounds 1 and 2

Assessing the credibility of research claims is a central and continuous part of the scientific process, but current assessment strategies often require substantial time and effort. If automated assessment methods could deliver comparable performance at a fraction of the cost, they would enable researchers, funders, and policymakers to direct attention toward high-confidence claims and improve strategic allocation of resources toward investigating claims that are important but uncertain. To date these methods are promising but unproven. The Center for Open Science launched the Predicting Replicability Challenge in 2025 as a public competition to advance the goal of automated assessment of research claims. Across three rounds, participating teams are provided access to a training set of replication outcomes and are tasked with scoring a held out set of published claims based on their likelihood of being successfully replicated in a new sample of data. Participating teams demonstrated improved predictive performance across the first two rounds of the Challenge. No models in the first round outperformed the baseline Brier score of .25, while nearly all did so in the second round, with similar improvements found for ROC-AUC (Round 1 median = .53, Round 2 median = .66) and accuracy (Round 1 median = .51, Round 2 median = .63). Gains in predictive performance have been driven by improved calibration (predictions matching observed replication rates more closely) and discrimination (greater separation between the predictions for replicated and non-replicated claims). Improvements in the second round have brought the top models in line with or ahead of prior efforts to predict these claims, depending on the comparison set.