Clinician Evaluation of Artificial Intelligence Summaries of Pediatric CVICU Progress Notes

Effective communication in critical care units, such as the Cardiovascular Intensive Care Unit (CVICU), is vital for patient safety; however, clinical notes from multiple professionals are often lengthy and complex. This study evaluated the Mistral large language model for summarizing Cardiovascular Intensive Care Unit progress notes using the Illness severity, Patient summary, Action list, Situation awareness and contingency planning, and Synthesis by receiver (I-PASS) framework, a standardized mnemonic for patient handoffs in healthcare. A total of 385 patients were included in the cohort, and all the progress notes associated with each patient were combined into a single document and summarized by the model. The readability was assessed using multiple metrics, including Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning-Fog Index, Simple Measure of Gobbledygook Index (SMOG), Automated Readability Index, and Dale-Chall Score. The readability metrics showed that the summaries generated with the Mistral Large Language Model (LLM) were much more difficult to read than the original notes, requiring a higher reading level. In a small clinician review, junior residents rated the summaries overall more favorably than senior residents, who often identified missing clinical details. Although Mistral condensed the documentation, this reduced readability and some loss of context may limit its usefulness for clinical handoffs. As a preliminary study with a small clinician-reviewed sample, these findings are descriptive and will require validation in larger clinical settings.