PharmaAgent: a real-world evaluation of high- and low-autonomy agentic systems for inpatient vancomycin dosing

Vancomycin has a narrow therapeutic window requiring individualized dosing and repeated monitoring, a process many health systems delegate from physicians to pharmacists. We silently deployed PharmaAgent, LLM-based vancomycin dosing agentic systems, at Stanford Hospital for two months and compared high- and low-autonomy configurations, each with and without institutional guidelines. Across 403 dosing events in 316 patients, mean absolute percentage error against the treating pharmacist’s order was 21.8% with high autonomy versus 37.8% with low autonomy, and recommendations were within 25% of the order in 73.4% versus 50.9% of events (P < 0.001); a second pharmacist was within 25% of the treating pharmacist’s order in 90.2% of events. Pharmacist review of 196 out-of-tolerance recommendations found a safety concern in 65.8%, more often with low autonomy (odds ratio 2.37). A higher-autonomy agent that self-debugged its steps proved more robust than stricter engineering, but neither configuration is ready to dose patients without pharmacist oversight.