Validation of Large Language Models in Healthcare: a Scoping Review

BACKGROUND. While the advancements of Large Language Models (LLMs) in healthcare hold promise, concerns remain about their potential impact on issues such as data privacy, health inequities, and risks of hallucinations, among others. Validation is therefore considered an essential step before implementation in practice. However, selecting appropriate validation methods is challenging due to the unstructured nature of LLM output, the diversity of tasks (e.g., summarization, question answering), and the lack of an overview of available methodology. OBJECTIVE. This scoping review aims to map existing validation methods for LLM applications in healthcare and to provide methodological considerations to guide stakeholders in selecting context-appropriate validation strategies. METHOD. We included studies published from 2005 onwards that focused on validation methods for LLM tasks relevant to healthcare: general methods, and methods for summarization, question answering, information extraction, and conversational agents. Searches were conducted in PubMed and Scopus in November 2024. We excluded studies where validation was secondary to an application, as well as those focused on non-text-based modalities, information retrieval, machine translation, and benchmark papers that only presented datasets without validation methods. RESULTS. From 3534 unique hits, 266 relevant papers were identified after screening and snowballing. Validation methods were organized into five categories (general, summarization, question answering, information extraction, and conversational agents) and organized by evaluation type: human evaluation, automatic evaluation with human reference, and fully automatic evaluation. DISCUSSION. This review emphasizes the need for frameworks that go beyond generic metrics to validate LLMs in healthcare. The findings provide a foundation for developing structured validation practices tailored to healthcare applications.