Evaluating User Experiences with an AI Chatbot for Health-Related Social Needs: A Cross-Sectional Mixed Methods Study
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Health-related social needs (HRSNs), such as food security and housing, significantly impact health outcomes and wellbeing. Although screening tools are widely adopted to identify the needs, sustainable linkage to resources remains challenging. Conversational agents (chatbots) offer potential solutions for tailored and personalized feedback, real-time navigation, yet their usability and trustworthiness among populations with high needs require further exploration.
Methods
We conducted a mixed-methods study to evaluate user experiences with the DAPHNE© chatbot, which is designed to identify unmet HRSNs and provide personalized resource recommendations. Quantitative and qualitative data were collected from 128 adults having child(ren). Online study design combined scenario/ task-based chatbot use and free form, to guide the engagement. Study measures included usability, task load, satisfaction, and trust. Qualitative analysis involved user feedback and user-chatbot conversation transcripts. We used regression analyses to explore associations between demographic characteristics, self-reported unmet HRSNs and user experience outcomes (usability, satisfaction, task load and trust).
Results
Most participants were female (68%), aged 30-49 years (71%), and White (44%) or Black/African American (36%) and Hispanic/Latino (27%), relied on Medicaid/Medicare (83%), and cared for a child with special needs (78%). Participants reported high usability (SUS= 84.7, SD=12.4), low task load (NASA-TLX= 6.8, SD=2.8), strong satisfaction (NPS= 8.0, SD=2.4), and high trust (Mean= 4.1, SD=0.8). Nearly all participants (98%) reported unmet HRSNs, notably food insecurity (76%) and financial limitations (75%). Free-form conversation sessions averaged 3 min and ∼20 turns, with greater amount use of assistive buttons than typing. Dialogues centered on financial, housing, and nutrition needs, and 94% of participants reported that the tool helpful finding resources, while requesting design features like saved histories, voice interaction, and richer local resource details. Regression models revealed limited but informative associations. Usability and trust were broadly consistent across most demographic groups, though participants with higher education and lower income showed modest decrements in usability. Several HRSNs, including transportation and utility disruptions, were associated with higher trust and satisfaction, suggesting the assistant may hold particular value for users facing structural barriers.
Discussion
The DAPHNE chatbot demonstrates potential as a useful tool for addressing HRSNs, with strong usability and trust among diverse populations. Future designs should focus on longitudinal impact assessments and effectiveness to enhance accessibility and address practical implementation challenges.
