Reproducibility and replicability of qualitative research: an integrative review of concepts, barriers and enablers

  1. Nicki Lisa Cole
  2. Sven Ulpts
  3. Agata Bochynska
  4. Eva Kormann
  5. Matthew Good
  6. Barbara Leitner
  7. Tony Ross-Hellauer

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

The integrative review presented here examines how reproducibility and replicability are conceptualized and discussed in relation to qualitative research, and which factors and practices enable or undermine them. Both peer-reviewed and grey English-language literature that address reproducibility and/or Open Science in relation to qualitative research were eligible for inclusion. Initial searches were conducted in Scopus, Web of Science, Dimensions, PubMed, APA PsychInfo, and JSTOR, and followed with snowball sampling from included literature. Studies were screened and both quantitative and qualitative data were extracted using the SyRF online platform, with 248 papers included. We found that conceptualizations that stem from quantitative standpoints are overwhelmingly framed as inappropriate practices and epistemic criteria for (most) qualitative research. When conceptualized in alternative ways that are adapted to the epistemic conditions, aims and practices of qualitative research, they can be both applicable and appropriate. Key barriers include the ontological and epistemological misalignment of reproducibility, replicability and Open Science and qualitative research, and ethical and practical concerns surrounding data sharing and reuse. Key enablers include practices that respond to ethical and practical concerns around data sharing and reuse (anonymization, ethical consent practices, context documentation, and ethical access management), adapting expectations and norms of openness, and established qualitative practices including documentation, reflexivity, and considering positionality. We conclude that reproducibility, replicability and Open Science practices must be adapted to the aims and epistemic conditions of qualitative research for them to be applicable and feasible, and that they will not always be both for all qualitative research.

Article activity feed

  1. PREreview

    This Zenodo record is a permanently preserved version of a Structured PREreview. You can view the complete PREreview at https://prereview.org/reviews/15767535.

    Does the introduction explain the objective of the research presented in the preprint? Yes The introduction explicitly that this paper is an integrative review aimed at synthesizing how reproducibility and replicability are discussed in qualitative research, as well as identifying their barriers and enablers. It sets the stage by discussing reproducibility crisis in science in quantitative fields e.g. health and biomedical sciences which has driven policy change thereby enhancing transparency, open science and replication. The authors note that these policy reforms are often based on positivist epistemologies, which may not apply in qualitative research paradigms. The authors, at the introduction, note the limited understanding of how open science practices intersect with the epistemological foundations of qualitative research. The objective is achieved through two thoughtfully formulated research questions.
    Are the methods well-suited for this research? Highly appropriate The methodology here is highly appropriate because the authors have employed integrative review. This methodology allows synthesis of diverse theoretical, empirical and grey literature across different disciplines. The systematic search, transparent preregistration, and mixed coding approaches align with the objective of the study, and explore complex yet contested concepts such as reproducibility and replicability in qualitative research.
    Are the conclusions supported by the data? Highly supported The conclusions are highly supported because they are made from a large diverse sample of 248 sources across disciplines. The conclusions that traditional idea of reproducibility, and replication often conflict with qualitative epistemologies though can be adapted are consistently grounded on the data they present. They are well aligned with the objectives of the study.
    Are the data presentations, including visualizations, well-suited to represent the data? Highly appropriate and clear The PRISMA diagrams, bar charts, and thematic tables present the scope of the study clearly, its frequencies and key patterns. They have been used well to summarize complex data, yet with clarity. They rationate with the narrative, and enhance easy interpretation of the results. As such, they allow readers to grasp pattern, distribution, and Key themes at a glance.
    How clearly do the authors discuss, explain, and interpret their findings and potential next steps for the research? Very clearly The authors have logically organized their findings around their research questions. they have used consistent terminology, and have explained key concepts with examples drawn from the data. They have distinguished between differing viewpoints, effectively summarized patterns, and linked their results to the broader debate. They are concluding directly from the evidence and have highlighted practical implication and the directions of future research.
    Is the preprint likely to advance academic knowledge? Highly likely It addresses a significant knowledge gap in literature by systematically examining how reproducibility and replicability are understood in qualitative research. Synthesis of a whole 248 diverse literature sources gives a comprehensive framework which challenges dominant positivist assumptions, thus advocating for epistemic inclusivity.
    Would it benefit from language editing? No The language here is clear, well-structured, and academically sound. There is precise use of terminology. Sentences are coherent. The arguements are also logically ordered. It meets high standards of scholarly work.
    Would you recommend this preprint to others? Yes, it's of high quality This preprint explores reproducibility and replicability in qualitative research. It addresses major gap in the current discourse. It is conceptually rich, and practically relevant.
    Is it ready for attention from an editor, publisher or broader audience? Yes, as it is It is well-structured, methodologically sound, and clearly written. It addresses a timely, yet underexplored topic with scientific rigor and depth.

    Competing interests

    The author declares that they have no competing interests.

  2. PREreview

    This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/15375661.

    This review is the result of a virtual, collaborative live review discussion on April 16th, 2025, organized and hosted by PREreview as part of the PREreview Champions Program 2025. The discussion was joined by 9 people: 2 facilitators and 7 cohort members of the PREreview 2025 Champions Program. We thank all participants who contributed to the discussion and made it possible for us to provide feedback on this preprint. The authors of this review have dedicated additional asynchronous time over the course of two weeks to help compose this final report using the notes from the Live Review. 

    Summary

    This study mainly addresses how reproducibility and replication are conceptualized and discussed within qualitative research and what factors and practices allow or hinder their application in qualitative research. These questions, clearly articulated on page 5 of the paper, reflect a critical and timely inquiry into how traditionally quantitative-oriented open science concepts intersect with qualitative research methodologies. The authors employ an integrative review method ideal for synthesizing empirical and theoretical literature to explore these themes. Drawing from 248 peer-reviewed articles and grey literature —including work from multiple disciplines such as health sciences and social sciences, which broadens the scope and relevance of the findings—and utilizing tools like Systematic Review Facility (SyRF) for data extraction and analysis, the sources were identified through searches in databases such as Scopus, Web of Science, Dimensions, PubMed, APA PsychInfo, and JSTOR, followed by snowball sampling from included references.  The main findings show that reproducibility and replication don't work the same in qualitative research as in quantitative research. Trying to copy a qualitative study exactly is often not possible. This is because qualitative research is more subjective, often depends on the context, and involves the researcher's role and perspective. The study also points out significant challenges, like ethical issues with sharing data (e.g., protecting participant privacy). However, the authors also highlight things that can help, like keeping good records of how research is done, being open about the research process, and adapting open science tools to fit qualitative research rather than rejecting reproducibility outright, the study argues for rethinking it in qualitative terms—emphasizing transparency, traceability, and positionality. In the end, the paper argues for a more flexible and fair version of open science that respects and supports the unique ways qualitative research works while also acknowledging, as a limitation, the exclusive focus on English-language sources, which may overlook relevant perspectives from non-English-speaking contexts.

    List of major concerns and feedback

    1. The review only includes English-language sources, which may exclude contributions from non-English-speaking communities.. It would strengthen the manuscript to explicitly address how this limitation may affect the global applicability of the findings, particularly in regions with different qualitative traditions. 

    2. The review presents a well-justified and appropriate integrative methodology, offering valuable insight into open science practices in qualitative research. However, the decision to conduct full-text screening, data extraction, and coding with a single reviewer raises concerns about potential bias and consistency. To strengthen the study's reliability, the authors are encouraged to clarify whether any strategies—such as double coding, consensus checks, or reflexive journaling—were used during the process. If no such measures were taken, a brief justification would help provide transparency and context for this methodological choice.

    3. While the authors aim to adapt reproducibility and replicability concepts to qualitative research, the justification and framework for this adaptation remain underdeveloped. Readers would benefit from clearer guidance on how these concepts should be applied in various qualitative fields. For instance, you can include specific examples, such as how reproducibility might be approached in ethnography or thematic analysis, which would make the recommendations more practical and actionable.

    4. The review may be biased toward the Western world, particularly Europe and North America, as it does not include articles representing different ethical or epistemological traditions, and while this is not necessarily inadequate, it may be important to acknowledge that the scope of the conclusions is not global.  

    List of minor concerns and feedback

    1. The authors describe the coding process at a high level. However, one could improve by briefly explaining how disagreements in coding were addressed, whether through discussion, consensus meetings, or cross-checking. Consider specifying whether any disagreement resolution methods were used and, if so, providing a brief description to enhance transparency and reproducibility.

    2. The contrast between qualitative and quantitative research seems oversimplified. Revising this framing could better reflect the complexity of both approaches. A more nuanced discussion acknowledging the diversity and overlapping elements within both paradigms would improve the theoretical framing.

    3. There is sample bias towards medical/health sciences (16.9%) and psychology (13.3%), fields where Open Science norms are more entrenched, while humanities and critical social sciences are underrepresented. It may be helpful to acknowledge this imbalance in the discussion or limitations sections and briefly reflect on how it could influence the findings or their generalizability.

    Concluding remarks

    This paper is valuable in contributing to the conversation about open science in qualitative research. It uses strong methods and gives a fair look at both the challenges and opportunities in making qualitative research more open and transparent. 

    We thank the authors of the preprint for posting their work openly for feedback. We also thank all participants of the Live Review call for their time and for engaging in the lively discussion that generated this review.

    Competing interests

    The authors declare that they have no competing interests.

  3. Version published to 10.31222/osf.io/n5zkw on OSF Preprints