Development and Preliminary Evaluation of a Brief Health Chatbot Acceptability Scale (CAS): A Pilot Study

Read the full article

Listed in

This article is not in any list yet, why not save it to one of your lists.
Log in to save this article

Abstract

Background

Conversational agents (chatbots) are increasingly used to deliver health information to middle-aged and older adults, a population in which the determinants of technology acceptance are known to diverge from those of younger users. However, acceptability is inconsistently defined and typically measured with ad hoc, unvalidated instruments. The psychometrically sound chatbot measures that exist assess usability rather than acceptability or were validated in student samples.

Objective

To develop and conduct an initial psychometric evaluation of a brief, 10-item health chatbot acceptance measure grounded in the Technology Acceptance Model (TAM) and the Theory of Planned Behavior (TPB).

Methods

Twenty adults (mean age = 52.8 years, range 28–81; 85% aged 40 or older; 75% female) enrolled in a sleep-application pilot usability study completed the 10 candidate items, the System Usability Scale (SUS), and sleep measures (PROMIS Sleep Disturbance 8a; Pittsburgh Sleep Quality Index). We examined item performance, internal consistency (Cronbach’s alpha), dimensionality (exploratory factor analysis with oblique rotation), and convergent validity (correlations with the SUS).

Results

A two-factor structure emerged and was invariant across rotation methods, resulting in an Information Quality factor and an Acceptance factor (interfactor r = .39). All items loaded ≥ .60 on their primary factor; mean communality was .83. Subscale reliabilities were high (Information Quality α = .96; Acceptance α = .94; full scale α = .93). Both subscales correlated with the SUS in the expected direction (r = .69 and r = .59), providing preliminary support for convergent validity.

Conclusions

The measure shows a coherent and theory-consistent two-factor structure and shows preliminary evidence of reliability and convergent validity. Given the small pilot sample, these results must be viewed as provisional but identify the measure as a candidate for formal validation in a larger, independent sample.

Article activity feed