Small but Specialized: A Domain-Adapted Retrieval-Augmented LLM Outperforms Frontier Generalists in Pediatric and Adolescent Gynecology
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Pediatric and adolescent gynecology (PAG) is a highly specialized field where timely, accurate clinical guidance can prevent serious harm to a vulnerable population, yet most general-purpose large language models (LLMs) lack reliable, citation-grounded knowledge of this domain.
Objective
To develop and evaluate a domain-specialized AI model that provides safe, evidence-anchored answers to PAG clinical questions, and to test whether a small, specialized model can outperform large general-purpose models.
Methods
We fine-tuned the Mistral-7B-Instruct LLM using QLoRA on a curated question–answer corpus derived from a medical textbook, including selected chapters focused on pediatric and adolescent gynecology and gynecologic care for girls under 18 years old, and combined it with a retrieval-augmented generation (RAG) layer using BGE embeddings and FAISS over 250-word textbook chunks with chapter-level citation. Performance was assessed on 182 held-out questions from chapters never seen during training (no data leakage), using a comprehensive ten-metric evaluation suite spanning lexical overlap (ROUGE-1/2/L, BLEU), character-level fidelity (chrF++), token-level paraphrase robustness (METEOR), contextual embedding similarity (BERTScore), learned semantic similarity (BLEURT, SAS), and an LLM-as-judge clinical rubric (G-Eval), plus a faithfulness proxy.
Results
PAG-Health-LLM achieved BERTScore 0.909, ROUGE-L 0.413, METEOR 0.526, chrF++ 0.489, and BLEURT 0.448, statistically and clinically outperforming GPT-4o-mini, LLaMA-3.3-70B, and Qwen-3-32B across all eight reference-based metrics (all p-values < 0.001; Cohen’s d = 0.46–1.70 across 24 external pairwise comparisons; 21 large, 3 medium effects). Compared with the unmodified base model, our system delivered relative gains of +64 % ROUGE-1, +374% relative BLEU improvement, and +82 % faithfulness (all p-value < 0.001), improving 88–97 % of individual test questions. Ablation analysis showed that retrieval was the dominant performance lever, with fine-tuning adding consistent further gains in stylistic and faithfulness quality. On two evaluation axes that are not reference-anchored, performance was statistically equivalent to frontier generalists on Semantic Answer Similarity (SAS: p > 0.3 vs both GPT-4o-mini and LLaMA-3.3-70B) and slightly lower on an LLM-as-judge G-Eval rubric, reflecting a deliberate trade-off in favor of concise, citation-faithful clinical responses over verbose freestanding exposition.
Conclusion
A small, domain-specialized retrieval-augmented LLM can safely and substantially outperform far larger generalist models on a sensitive pediatric clinical domain, demonstrating that specialization plus citation-grounded retrieval, not scale alone, is a practical path to deployable clinical AI.
