Limits of Single-Pass Retrieval-Augmented Generation for AI-Powered Cancer Care Navigation: A Comparison of Retrieval Strategies
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
People affected by cancer often face difficulty finding relevant clinical, psychological, and practical support services. AI-powered navigation assistants may improve access to these resources, but their retrieval performance must be reliable.
Objective
To develop a single-pass retrieval-augmented generation assistant for cancer-care navigation and compare the retrieval strategies, including their robustness to reworded questions.
Methods
We created a database of 853 cancer-support resources reviewed by librarians, clinicians, researchers, and patient partners. We evaluated the system using 100 questions derived from questions submitted by patients. We compared keyword-based, semantic, and hybrid retrieval using Precision@K, Hit@K, and nDCG@K. The best-performing configuration was then tested using semantically equivalent rewordings of the original questions.
Results
Keyword-based retrieval performed poorly, achieving a P@1 of 25.0% and Hit@5 of 43.0%. Semantic retrieval improved these results to 58.0% and 86.0%, respectively. The best hybrid configuration achieved a P@1 of 64.0%, Hit@5 of 90.0%, and nDCG@5 of 51.0%. Performance remained relatively stable when the questions were reworded, with a P@1 of 61.0%, Hit@5 of 88.0%, and nDCG@5 of 46.1%.
Conclusions
Hybrid retrieval performed best and remained relatively stable when questions were reworded. However, its limited ability to rank a relevant resource first highlights the limitations of single-pass retrieval for patient-facing cancer navigation. Future work will explore metadata filtering and a multi-agent architecture to improve retrieval reliability.
