repliCATS-SCORE: Elicited human predictions of social and behavioural science replicability
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
The repliCATS-SCORE dataset consists of five human elicitation studies predicting the replicability of published research in the social and behavioural sciences, conducted by the repliCATS project (Collaborative Assessments for Trustworthy Science) as part of DARPA’s Systematizing Confidence in Open Research and Evidence or SCORE program (2019-2022). The human elicitations were conducted using a structured group deliberation technique – the IDEA protocol – for a total of 4000 published research claims and an additional 25 claim pilot study. The dataset for each study includes participant-level judgements across two elicitation rounds, mathematically aggregated group judgements or “confidence scores” calculated using a suite of 38 aggregation methods, as well as participant free-text reasoning. The repliCATS-SCORE dataset can be re-used in support of studies interrogating scholarly judgements of replicability, plausibility and overall credibility, post-publication peer review and critical evaluation training, developing novel mathematical aggregation models that forecast replication outcomes, and comparative studies.
