A Mixed-Methods Evaluation of Clinician Experiences and Adoption Patterns of an EHR-integrated Generative AI-based Clinical Decision Support in Kenya
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Objective
To quantify the adoption pattern of an LLM-based clinical decision support system across private primary health facilities in Kenya (operated by Penda Health), and explore factors influencing clinician uptake and overall experience.
Methods
A mixed-methods study combining quantitative analysis of CDSS metadata from all consultations that took place between 1 st February and 1 st October 2024, augmented by qualitative data from 42 staff members (26 clinical officers, 10 facility managers, four nurses, one quality assurance manager, and one business analyst). Data collection included journey mapping interviews (n=7), user-experience interviews (n=25), focus groups (n=2), and system utilization metrics. Quantitative data were summarized using descriptive statistics, and qualitative data analysed using thematic analysis (drawing on established theories of technology adoption and change management).
Results
In total, there were 258,106 clinical episodes across all Penda Health facilities over the 8-month observation period, of which 56,050 (21.7%) were augmented by use of the ‘AI Consult’. ‘AI Consult’ use, aggregated across the 16 facilities, increased from 4% to 47% over 8 months. Feedback on the CDSS guidance was infrequent (only being provided in 31% of cases), but when it was, it was overwhelmingly positive (99.5%). The qualitative investigation identified five key themes associated with clinicians’ experiences with the AI Consult tool: (1) there are several value propositions to an ‘AI Consult’ style tool, (2) Clinicians’ confidence in the AI consult grew with time, (3) clinicians’ application of the ‘AI consult’ is influenced by case complexity, (4) responses from the AI consult are largely believed and valued by clinicians but several improvements are recommended, and (5) clinicians find the ‘AI consult’ easy to use but identified several pain points that warrant attention.
Discussion
Successful GenAI/LLM-enhanced CDSS implementation in resource-constrained settings requires: (1) robust technological infrastructure, (2) localisation to reflect clinical guidelines, (3) structured change management with clinical champions, and (4) seamless workflow integration. Future product development exercises should specifically consider alternatives to active solicitation of CDSS input, as it is liable to overconfidence-related underutilization.
Key Questions Box (BMJ Digital Health and AI)
What is already known?
-
AI-powered clinical decision support systems face implementation challenges, especially in low-resource contexts.
-
Studies from high-income countries suggest that workflow integration and perceived usefulness are key drivers of adoption.
-
Limited research exists on the implementation of GenAI in resource-constrained healthcare environments.
What are the new findings?
-
It is possible to achieve non-trivial adoption of GenAI-based CDSS (4% to 47% over 8 months) when supported by structured change management.
-
There are several value propositions for a GenAI-based CDSS, including improved quality of care and serving as a learning aid.
-
Several factors influenced utilization/engagement, from perceived case complexity to ease of use and trust in the system.
-
Clinicians’ behaviour and interactions with GenAI tools (as demonstrated by varying prompt use) mature over time based on perceived usefulness.
What do the new findings imply?
-
Sustainable AI implementation in low-resource settings requires addressing both technical infrastructure and human factors.
-
Relying on active solicitation of inputs from an AI system is likely to suffer from overconfidence bias, which limits its overall utility, and instead, we should explore other methods of integrating CDSS into the relevant clinical workflows.
