Evaluation of Gender Bias in the Evaluation of Synthetic Cardiovascular Disease Cases with Open Source LLMs
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Objective
To systematically evaluate gender bias in open-source large language models (LLMs) for cardiovascular diagnostic decision-making using controlled synthetic case vignettes.
Methods
We generated 500 synthetic cardiovascular cases with randomly assigned gender (male/female, equal distribution) and age (45-80 years), keeping all other clinical variables identical. Two structured prompts simulated sequential cardiovascular evaluation stages: initial chest discomfort presentation and post-stress-test evaluation. Three open-source LLMs were evaluated via local Ollama API: Gemma-2b, Phi, and TinyLLaMA. Primary outcomes included coronary artery disease (CAD) likelihood ratings (low/intermediate/high), diagnostic certainty (low/intermediate/high), and test usefulness scores (1-10 scale). Statistical analysis included chi-square tests, Mann-Whitney U tests, and logistic/linear regression with multiple comparison adjustments. Power analysis determined minimum detectable effects of 12.5% for individual models and 7.2% for pooled data.
Results
Evaluation of 1,500 model responses (500 cases × 3 models) revealed minimal gender-related differences. Only one statistically significant finding emerged: Gemma-2b assigned higher diagnostic certainty to female patients in initial presentations (58% vs. 48%, p=0.031, adjusted p=0.092). No other gender-based differences reached significance after multiple-comparison adjustment. Effect sizes were consistently small across all comparisons (Cohen’s h: 0.01-0.18; Cliff’s delta: -0.11 to 0.12). Substantial inter-model variability was observed, with Gemma-2b and Phi demonstrating assertive diagnostic patterns while TinyLLaMA showed conservative tendencies. Parsing quality exceeded 95% for all models.
Conclusions
Open-source LLMs demonstrated largely gender-neutral outputs in controlled cardiovascular scenarios, contrasting with documented biases in human clinicians and commercial LLMs. The isolated gender effect in Gemma-2b was modest and clinically insignificant. More concerning was substantial inter-model variability in diagnostic confidence and test recommendations, highlighting the critical importance of rigorous model benchmarking before clinical deployment. These preliminary findings suggest that open-source LLMs may offer advantages for equitable healthcare applications, but broader validation across diverse clinical contexts and real-world constraints remains essential.
