Monitoring of Open Science practices: a survey of 10 major medical journals

Objective:

To evaluate open science policies of leading general medical journals and the extent to which open science practices are implemented and detectable using automated tools.

Design:

Cross-sectional audit of journal policies and retrospective observational study of journal articles, with diagnostic accuracy validation of automated screening tools against manual data extraction.

Setting: Annals of Internal Medicine, BMC Medicine, The BMJ, CMAJ, JAMA, JAMA Network Open, The Lancet, Nature Medicine, New England Journal of Medicine, and PLOS Medicine.

Participants:Research articles published between 2020 and 2023 and randomised controlled trials (RCTs) published before and after policy changes regarding registration and data-sharing.

Main outcome measures: Journal policies were assessed using the TOP2025 framework, which rates journals’ transparency and openness standards using the TOP Factor (maximum score 27). At the article level, thirteen core open science practices were examined, including trial registration, protocol sharing, and intention to share data. Nine validated automated tools were applied to detect these practices and compared with manual extraction of 312 articles performed in duplicate. Changes in transparency practices following policy updates were analyzed.

Results:

Transparency policies varied considerably. TOP Factor scores ranged from 1 (NEJM) to 13 (PLOS Medicine), with many journals’ policies applying primarily to clinical trials rather than all research articles. Only one journal (BMC Medicine) proposed registered reports. At the article level, adoption of open science practices was highest in RCTs: trial registration (99% [95%–100%] vs 68% [55%–79%] in meta-analyses vs 15% [8%–27%] in other research), protocol sharing (95% [90%–98%] vs 67% [54%–78%] vs 19% [11%–32%]), and intention to share data (78% [67%–87%] vs 64% [49%–76%] vs 69% [55%–80%]). Performance across tools was very variable (F1 scores 0.16–1.00). Among all 15,624 research articles, tools detected registration in 20%, protocol sharing in 42%, and intention to share data in 48%, but these were generally underestimated. Policy changes such as mandatory registration or data-sharing requirements were associated with measurable improvements in the corresponding open science practices.

Conclusions:

Leading medical journals show only partial alignment with TOP2025, with stronger transparency observed for RCTs than for other study types. Triangulation with manual and automated assessments in articles confirms that key open science practices remain suboptimal especially for non-RCTs, highlighting the need for more robust journal policies. Policy changes are associated with observable improvements.

Registration: OSF: https://doi.org/10.17605/OSF.IO/F2VW9