Unmet Needs in Acute Hepatic Porphyria Diagnosis: A Comparative Big Data Analysis of an AI-based Human-in-the-Loop Screening Versus Standard of Care

Background Acute Hepatic Porphyria (AHP) is a rare genetic disease characterized by unpredictable life-threatening attacks. There is no reliable biochemical screening test for patients outside of an attack and diagnosis is delayed on average by 15 (!) years. AI screening systems can assist in detecting AHP patients, but validating such systems is challenging, due to the limited number of suspected candidates and/or success of recalling such candidates for testing. At the same time no study to date has highlighted human oversight of AI screening tools, while governing bodies and medical device regulations call for it to be allowed for clinical use. Our primary goal was to demonstrate the feasibility of an AI-based Human-in-the-Loop screening (HAI) approach and quantifying the added value by comparing the rate and number of clinically plausible cases found through it with the current Standard of Care (SOC). Methods This retrospective cohort study included data collected from 899,862 electronic health records (EHR) of patients who were treated at the University Hospital Salzburg (SALK) between December 2007 and December 2021. For our HAI approach we used an AI-tool for disease screening (Dx EHRs v2022.11) provided by Symptoma GmbH, that has been validated for Pompe Disease in a previous study. All historically suspected and diagnosed AHP cases retrieved from the collected data served as the reference standard representing the SOC. All suspected AHP cases were first triaged by generalist physicians (GP) without a specialization for AHP representing the *Humans in the Loop*. Specialized physicians (SP) determined the clinical plausibility of cases by reviewing the complete EHRs of the triaged cases. The primary outcome were the rates of clinically plausible cases (=precision) in the HAI and SOC cohorts and its sub-cohorts. Additionally, we investigated the differences in phenotypes in those cohorts. Historically diagnosed AHP cases were reviewed by SP for the reliability of their diagnosis. Findings Of a total of 899,862 EHRs, 191 EHRs were triaged into the HAI cohort and 107 filtered into the SOC cohort. 74 (38.74%) and 28 (27.72%) cases were deemed clinically plausible, for HAI and SOC respectively. Of those 74 clinically plausible cases in HAI, 46 were de-novo cases missed by SOC. The sub-analysis on the phenotypical features indicated that psychological and psychosomatic symptoms (Restlessness, Confusion, Anxiety, Depression, Mood swings, Palpitations) are significantly underrepresented within historically suspected AHP cases. As well were some common and subtle symptoms (Pain, Nausea, Vomiting, Fatigue). Among 16 historically diagnosed cases, four were reclassified as misdiagnosed, and seven lacked conclusive evaluation by current diagnostic standards. Notably, two new AHP cases were identified *incidentally* during the study, with a Poisson probability of 8.34% for this event to happen, suggesting this occurrence was unlikely to be random. Interpretation AHP is incredibly hard to diagnose and even already made AHP diagnoses are unreliable. Additionally, certain phenotypes are especially challenging to identify via the current standard of care. HAI managed to reach a higher precision compared to the SOC and found an additional 46 clinically plausible de-novo cases. Both showing feasibility and added value of HAI. *Incidentally* newly diagnosed AHP patients strongly suggest an increase in awareness through the AI screening project. All our findings suggest that HAI is a viable approach addressing the challenge of early diagnosis of AHP and its adherent issues. Prospective studies in a setting as real-time decision support at the point of care are warranted as a next step to implementing HAI as part of the new standard of care.