Appropriateness of Recommendations Obtained From On-Line Large Language Models using Medical Questionnaires as Input

Individuals seeking prevention and management advice for various chronic and rare diseases often consult multiple information sources, including clinicians and online platforms. Recently, the reliance on traditional internet sources has shifted towards chat-based artificial intelligence models, such as large language models (LLMs). These models are designed to respond to user queries; however, the qualitative appropriateness of their responses, particularly in the context of rare diseases, has not been extensively evaluated. While previous studies have assessed LLMs in specific domains, such as cardiovascular diseases, a comprehensive analysis of their suitability for addressing a broader range of medical topics, including rare diseases, is lacking. In this study, we evaluated the response quality of several LLMs using publicly available questionnaires addressing diverse aspects of disease knowledge. These included instruments such as the DKQ-R (Diabetes Knowledge Questionnaire), Rare Disease Knowledge Questionnaire, Heart Disease Knowledge Question- naire (KC), MSKQ-A and MSKQ-B (Musculoskeletal Knowledge Questionnaires), Leuven Knowledge Questionnaire for Coronary Heart Disease (CHD), and the Questionnaire of Knowledge and Perception Towards COVID-19. Each question from these tools was posed to the online interfaces of selected LLMs, and their responses were systematically recorded. To assess the quality of the generated answers, an expert clinician graded each response as either “appropriate” or “in- appropriate.” Our findings provide valuable insights into the strengths and limitations of LLMs in delivering accurate, rel- evant, and contextually appropriate medical information, with a particular emphasis on rare diseases. This work highlights the potential role of LLMs in supplementing medical knowledge dissemination while underscoring the need for further refinement and domain-specific training to improve their applicability in healthcare contexts.