Implementing a context-augmented large language model to guide precision cancer medicine
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
BACKGROUND
The rapid expansion of molecularly informed therapies in oncology, coupled with evolving regulatory FDA approvals, poses a challenge for oncologists seeking to integrate precision cancer medicine into patient care. Large Language Models (LLMs) have demonstrated potential for clinical applications, but their reliance on general knowledge limits their ability to provide up-to-date and niche treatment recommendations. To address this challenge, we developed a Retrieval-Augmented Generation (RAG)-LLM approach that integrates with a precision oncology knowledge resource, and evaluated whether this approach improved accuracy in biomarker-driven treatment recommendations relative to alternative frameworks.
METHODS
We developed a RAG-LLM workflow that integrates Molecular Oncology Almanac (MOAlmanac) and evaluated this approach relative to alternative frameworks (i.e. LLM-only) in making biomarker-driven treatment recommendations using both unstructured and structured data. We evaluated LLM performance by calculating exact and partial match accuracies across 234 therapy-biomarker relationships. Finally, we assessed real-world applicability of the workflow by testing it on actual queries from practicing oncologists.
RESULTS
While LLM-only achieved 54–69% accuracy in biomarker-driven treatment recommendations, RAG-LLM achieved 73–85% accuracy with an unstructured database and 91–99% accuracy with a structured database. In addition to accuracy, structured context augmentation substantially increased precision (49.0% to 79.9%) and F1-score (55.7% to 84.4%) compared to unstructured data augmentation. In queries recommended by practicing oncologists, RAG-LLM achieved 75–94% accuracy and high precision (91.67%).
CONCLUSIONS
These findings demonstrate the effectiveness of the RAG-LLM framework in recommending FDA-approved precision oncology therapies based on individualized clinical information, and highlight the importance of integrating a well-curated, structured knowledge base in this process. While our RAG-LLM approach significantly improved accuracy compared to standard LLMs, further efforts will enhance the generation of reliable responses for ambiguous or unsupported clinical scenarios.
