Generating Learning Guides for Medical Education with Large Language Models and Statistical Analysis of Test Results

Background : The Progress Test Medizin (PTM) is a formative test for medical students issued twice a year by the Charité-Universitätsmedizin Berlin. We used data from the PTM to automatically generate personalized feedback designed to identify the knowledge gaps of test participants as exactly as possible with the help of large language models and statistical analysis. Methods : We have developed a seven-step approach to fulfil the purpose of this study. Firstly, a large language model (ChatGPT 4.0) identified keywords in the form of MeSH terms from all 200 questions of one PTM run. These keywords were checked against the list of medical terms included in the MeSH thesaurus published by the National Library of Medicine (NLM). Meanwhile, answer patterns of PTM questions were also analysed to find whether, given a pair of questions (A,B), knowing the answer to A has a large effect on improving the chance of knowing the answer to B. If that is the case, we say that question A is a precursor question to question B. With this information, we obtained series of questions related to specific MeSH terms and used them to develop a framework that allowed us to assess the performance of PTM participants and compose personalized feedback structured around a curated list of medical topics. A clustering procedure was also applied to construct benchmarking sets. Results : We simulated the generation of personalized feedback for 1,401 PTM participants out of their test results, thereby producing specific information about their knowledge regarding a number of topics ranging from 34 to 243. Substantial knowledge gaps were found in 14.67% to 21.76% of rated learning topics, depending on the benchmarking set considered. Conclusion : We designed and tested a method to generate student feedback covering up to 243 medical topics defined by MeSH terms. The feedback received by students in the later stages of their studies was more detailed, as they tend to face more questions matching their knowledge level.