Comorbidities of dementia in electronic health records: validation of a natural language processing app and comparison with data linkage

Awareness of comorbidities in people with dementia is vital context for dementia research and clinical management. Natural Language Processing (NLP) applied to free text in the electronic health record (EHR) may help ascertain comorbidity presence. Our aim was to assess the validity of the MedCAT Comorbidity NLP app in a cohort of people with dementia.

We used text from the EHR of South London and Maudsley NHS Mental Health Trust via Clinical Records Interactive Search (CRIS), linked to structured data from local primary care (Lambeth DataNet) and hospital admission data in a cohort of 3,543 people with dementia. We evaluated MedCAT Comorbidity, examining internal validity via manual review of text and external validity by cross-validation against linked primary care and hospital data; and comparison of the predictive value of comorbidity measures for mortality.

Internal precision was highest for atrial fibrillation (100%) and lowest for diabetes (78%). Cross-validation with linked data varied substantially across conditions, with precision from 96% to 59% and recall 77% to 4%. The median number of comorbidities identified was two using any single source, or three when combined. MedCAT Comorbidity demonstrated predictive ability for mortality (HR 1.11 per condition), although the model fit was better using comorbidity from hospital admission data.

In people with dementia, the MedCAT Comorbidity app showed high precision for most conditions during internal validation. The variability of precision and recall on external validity assessment partly reflects incomplete documentation. Results from the MedCAT comorbidity app can assist researchers in understanding comorbidity of cohorts, particularly when primary care or hospital data is not available. Our findings highlight the need to develop quality assurance protocols that include internal and external validity checks before deploying general data extraction apps for specific clinical subpopulations.

Author Summary

People with dementia commonly have other long-term health conditions, and structured electronic health record data may not capture the whole picture. Natural language processing (NLP) may offer additional information but there is a shortage of validation studies. This study explored performance of the MedCAT NLP comorbidity app on full-text records regarding 3,543 people with dementia that had external linkage to structured data.

There were three steps of validation: firstly, internal precision against the original clinical text; secondly, compared against primary care record and hospital admission diagnosis codes; thirdly, examining the ability of comorbidity ascertainment to predict mortality. The results indicated some potential strengths and limitations of the NLP approach, which may vary by comorbidity condition. With improvements, the MedCAT comorbidity app could expand its utility in dementia research and extend into clinical scenarios. Other NLP apps may benefit from a similar validation process in clinical subgroups before deployment.