Beyond the Scribe: Using Nominal Group Technique to Co-Design an Evaluation Framework for Patient-Held AI

The explosive growth of ambient AI scribes in healthcare has focused almost exclusively on relieving clinician documentation burdens and enhancing clinician efficiency and reducing burnout. While most research has focused on the clinician experience, these same types of tools, such as a patient-held AI Personal Health Assistant, can be put in the hands of patients to facilitate communication with healthcare providers, to provide greater autonomy and activation for patients in managing their own healthcare, and to facilitate the task of receiving and retaining complex new information. However, to ensure these tools serve as more than just administrative aids, we partnered with a large technology company to conceptually co-design how such a tool might be studied in real-world settings in a way that reflected outcomes of greatest importance to patients and families. We used the Nominal Group Technique NGT and engaged 33 patients and caregivers and 10 multidisciplinary clinicians in structured virtual sessions to identify the outcomes on which subsequent studies should hinge.

Our findings reveal a significant misalignment between current industry trends and patient priorities. While the market emphasizes the scribe function for billing, stakeholders overwhelmingly prioritized the AI-PHA’s role in longitudinal care management and post-visit coordination. The most critical outcome identified was the impact on communication and respect, suggesting that patients view AI as a mechanism to improve the human-to-human dynamic and ensure a shared understanding of the care plan. Our findings further highlighted that the primary hurdle for the use of patient-held AI is not technical utility but clinician trust and the need for formal consent processes.

Importantly, as this technology is new and integration into the clinical experience is still being explored, patients cannot assume they will have access to the transcript of their own encounter when that record is created and held by the health system. This gap highlights the potential value of a patient-held tool, preserving discussions from their visit and follow-up steps in clearer detail than they may receive in their clinical note or traditional visit summary.

From a system perspective, this transition requires a strategic shift from measuring workflow-focused efficiency to measuring patient-side advocacy and engagement. By centering the evaluation of AI on these patient-defined metrics and fundamentally designing subsequent AI investigations using a partnership model, health systems can move beyond simple automation toward a model of care delivery that empowers patients to navigate complex clinical journeys with greater autonomy and clarity.