A language model framework for sequence modeling of EHR audit logs to characterize clinician-EHR interactions
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Objective
Electronic health record (EHR) audit logs capture clinician-EHR interaction patterns, but most audit log research relies on aggregated measures (e.g., total time). We investigated how audit logs could be modeled using large language model (LLM) architectures to learn underlying workflow sequences during clinician-EHR interactions.
Materials and Methods
Using >295 million EHR-based audit log actions from inpatient settings spanning 2019 to 2024, we fine-tuned Llama-3-8B under two encodings: (1) symbolic field-based tokens and (2) semantic natural language audit log action descriptions. A first-order Markov model, which used only the immediate prior action, served as baseline minimal context comparator. Model representation was assessed using next-action prediction accuracy in an early-period test set and two temporally distinct out-of-sample (OOS) periods.
Results
In the early period test set, the semantic LLM achieved the highest accuracy (0.7418, 95%CI [0.7415–0.7420]) compared to the symbolic LLM (0.3838, 95% CI: 0.3836–0.3840) and Markov baseline (0.4553, 95%CI [0.4551–0.4555]). The semantic approach was also robust to temporal drift in EHR interaction patterns seen in the two OOS periods (semantic LLM vs Markov accuracy, OOS-1: 0.6509 vs 0.3169; OOS-2: 0.6232 vs 0.2648).
Discussion and Conclusions
Semantic LLM relying on audit log action descriptions yielded the highest next-action prediction accuracy, and demonstrated robustness to temporal drift, suggesting that longer sequential context and semantic action descriptions may improve audit log-based sequence modeling. These findings support further development of semantic sequence models for audit log research, including task identification, automated workflow characterization, and safety-focused analyses of clinician-EHR interaction patterns.
