Comparative Evaluation of ChatGPT-4 and Ophthalmologist-in-training in the Triage of Patient Messages Sent to the Eye Clinic via the Electronic Medical Record

Objective: To test the ability of ChatGPT-4 (GPT-4) to effectively triage patient messages sent to the general eye clinic at our institution. Methods: Patient messages sent to the general eye clinic via MyChart were de-identified and then triaged by an ophthalmologist-in-training as well as GPT-4 with two main objectives. Both ophthalmologists and GPT-4 were asked to direct patients to either general or specialty eye clinics, urgently or non- urgently, depending on the severity of the condition. Main Outcomes: GPT-4’s ability to accurately direct patient messages to 1.) a general or speciality eye clinic and 2) determine the time frame within which the patient needed to be seen (triage acuity). Accuracy was determined by comparing percent agreement with recommendations given by GPT-4 with those given by Ophthalmologist. Results: The study included 139 patients (and messages). Percent agreement between the ophthalmologist and GPT-4 was 64.7% for general/specialty clinic recommendation and 60.4% for triage acuity. GPT-4 recommended a triage acuity equal to or sooner than ophthalmologist-in-training for 93.5% of cases and recommended a less urgent triage acuity in 6.5% of cases. Conclusions: Our study indicates an AI system, such as GPT-4 should complement rather than replace physician judgment in triaging ophthalmic complaints. These systems may assist providers and reduce the workload of ophthalmologists and ophthalmic technicians as GPT-4 becomes more adept in triaging ophthalmic issues.