Constructing a Smart-Assistant for Improving the Outpatient Service Quality in Real-time: a Prospective Single-center Cohort Study

Background The quality of consultation and outpatient electronic medical records (EMRs) varies among physicians. We aimed to construct an intelligent system (SMART-ASSISTANT) to assist physicians in history taking and the composing of EMRs for patients presenting with the chief complaint of abdominal pain. Methods Anonymized EMRs of 1249 cases, free-text-structured EMRs pairs of 119 cases, and a hot words dictionary were used to train the SMART-ASSISTANT. The SMART-ASSISTANT is constructed with four components: audio transcription, structured EMRs generation, EMRs quality control, and assisted diagnosis. The functions were validated through the simulated set, the retrospective set, and a multi-reader multi-case (MRMC) study. A prospective cohort study including 62 participants was conducted to evaluate the utility of SMART-ASSISTANT to transcribe the consultation audio into standardized EMRs text. Results SMART-ASSISTANT outperformed GPT-4 in identifying symptoms, characteristics, onset characteristics, and disease progression significantly (100.00 vs 19.20%, P <0.001; 92.00 vs 85.80%, P <0.001; 85.60 vs 45.60%, P <0.001; 89.60 vs 45.40%, P <0.001). Physicians’ rating of the completeness and diagnostic correlation of EMRs in the AI-assisted set were significantly superior to those in the human-generated set (4.27 vs 3.92, P <0.001; 2.53 vs 2.33, P <0.001). In prospective cohort study, the mean semantic textual similarity (STS) of audio transcription reached 0.9253. Physicians rated no significant difference between AI and human-generated EMRs in normativity (3.23 vs 3.25, P  = 0.806), readability (3.22 vs 3.38, P  = 0.068), and logicality (3.19 vs 3.32, P  = 0.122). AI-generated EMRs demonstrated significantly superior performance in terms of integrity (3.50 vs 3.21, P  = 0.008) than human-generated EMRs. Conclusions The quality of the outpatient EMRs have been significantly improved by the utilization of SMART-ASSISTANT. The system has the potential to confer benefits on both patients and healthcare organizations through assisted consultation and automated EMRs generation.