Improving clinical expertise in large language models using electronic medical records

Electronic medical records (EMRs) are essential in clinical practice. Although current medical large language models (LLMs) excel in tasks like US Medical Licensing Examination, they struggle with real-world clinical applications due to insufficient large-scale EMR data in their training, hindering their clinical expertise. To address this limitation, we proposed EMR-LLM, an LLM for clinical practice using EMRs. Firstly, we continually pre-trained a general LLM on medical corpora to enhance its domain knowledge. Then, we designed three categories of instruction tasks using EMRs: structure understanding, numerical understanding, and downstream tasks. Finally, we introduced an ability-boosting instruction-tuning method, which mimics human learning, progressing from simple to complex tasks while introducing a data replay strategy to retain learned knowledge. Experimental results demonstrated that EMR-LLM outperformed strong competitors on six EMR tasks, nine medical benchmarks, and three open-domain benchmarks. Moreover, in discharge summary generation, EMR-LLM achieved performance levels close to those of expert clinicians.