RESPECT: A Conversational AI System for Informed Consent with Accuracy, Safety, and Stakeholder-Centered Evaluation

Informed consent (IC) is a cornerstone of clinical research. While IC typically includes both written materials and, particularly in clinical trials, an oral discussion between the investigator and participant, in practice both components tend to be templated and standardized, limiting opportunities for meaningful, individualized dialogue. While Large Language Models (LLMs) offer possibilities for enhancing the accessibility of IC, realizing this potential requires ensuring accurate, safe, and appropriate responses before research deployment. We developed RESPECT (RESearch Participant Engagement and Consent Tool), an LLM consent assistant utilizing Retrieval-Augmented Generation (RAG) to ground responses in IC source documents. We evaluated accuracy through leave-one-out cross-validation and question rephrasing analysis, demonstrating high accuracy in information retrieval for the RAG system. We introduced a novel safety evaluation framework measuring two dimensions: appropriate refusal (how often the system refuses questions it should not answer) and utility (how often it answers questions it should answer). This approach generalizes simple refusal rates by plotting a Refusal-Utility Curve (RUC) analogous to ROC-AUC curves, revealing that RESPECT demonstrated significantly higher appropriate refusal rates compared to GPT-4, but at the cost of reduced utility in answering legitimate questions. We conducted stakeholder evaluations with research staff to assess accuracy, comprehensiveness, and satisfaction.