Evaluating User Experiences with an AI Chatbot for Health-Related Social Needs: A Cross-Sectional Mixed Methods Study
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Unmet Health-related social needs (HRSNs), such as food insecurity and housing, significantly impact health outcomes and wellbeing. Although screening tools are widely adopted to identify the needs, sustainable linkage to resources remains challenging. Conversational agents (chatbots) offer potential solutions for tailored and personalized feedback, real-time navigation, yet their usability and trustworthiness among populations with high needs require further exploration.
Methods
We conducted a mixed-methods study to evaluate user experiences with the DAPHNE© chatbot, which is designed to identify unmet HRSNs and provide personalized resource recommendations. Quantitative and qualitative data were collected from 128 caregivers with at least one dependent child. Online study design combined scenario/ task-based and free form chatbot use, to guide the engagement. Study measures included usability (SUS), task load (NASA-TLX), satisfaction (NPS), and trust. Qualitative analysis involved user feedback and user-chatbot conversation transcripts. We used regression and pairwise analyses to explore associations between demographic characteristics, self-reported unmet HRSNs and user experience outcomes (usability, satisfaction, task load and trust).
Results
Most participants were female (68%), aged 30-49 years (71%), and White (44%) or Black/African American (36%) and Hispanic/Latino (27%), relied on Medicaid/Medicare (83%), and cared for a child with, or had, special healthcare needs (78%). Participants reported high usability (SUS= 84.7, SD=12.4), low task load (NASA-TLX= 6.8, SD=2.8), high satisfaction (NPS= 8.0, SD=2.4), and high trust (Mean= 4.1, SD=0.8). Nearly all participants (98%) reported unmet HRSNs, including food insecurity (76%) and financial limitations (75%). Free-form chatbot conversation sessions averaged 3 minutes and ∼20 turns, with greater use of assistive buttons over typing. Furthermore, DAPHNE (using retrieval-augmented generation to ground every recommendation in a live social-care API) achieved 99 % intent accuracy in 1,523 message turns. Dialogues focused on financial, housing, and food needs. 94% of participants found the tool was helpful, while requesting design features like saved histories, voice interaction, and richer local resource details. Regression analysis showed usability and trust were broadly consistent across most demographic groups, though participants with higher education and lower income showed modest decrements in usability. Several HRSNs, including transportation and utility disruptions, were associated with higher trust and satisfaction, suggesting the assistant may hold particular value for users facing structural these barriers.
Discussion
The DAPHNE chatbot demonstrates potential as a useful tool for addressing HRSNs, with strong usability and trust among diverse populations. Future designs should focus on longitudinal impact assessments and effectiveness to enhance accessibility and address practical implementation challenges.
