CMR-CLIP: Contrastive Language Image Pretraining for a Cardiac Magnetic Resonance Image Embedding with Zero-shot Capabilities

Self-supervised learning is crucial for clinical imaging applications, given the lack of explicit labels in healthcare. However, conventional approaches that rely on precise vision-language alignment are not always feasible in complex clinical imaging modalities, such as cardiac magnetic resonance (CMR). CMR provides a comprehensive visualization of cardiac anatomy, physiology, and microstructure. The interpreting physician is required to synthesize information from complex sequences of images representing different tissue traits and different spatial locations of the heart in the context of the clinical history, resulting in potentially weak alignment between the study images and diagnostic report pair. To overcome these challenges, we propose CMRCLIP, a vision language model which treats CMR images as videos to jointly learn embeddings between the CMR images and associated cardiologists' or radiologists' reports. We train our model on a large CMR dataset consisting of 13,787 studies done performed at a single healthcare institution and evaluate the model both on an internal (N = 669) and external dataset (N = 428) with significantly different distribution of disease and CMR vendors. We show that the proposed CMRCLIP achieved remarkable performance in real-world clinical tasks, such as CMR image retrieval and diagnostic report retrieval in our internal held out test set. Furthermore, the learned representations were found to be helpful for downstream applications in unseen external CMR data, as shown in the public Automated Cardiac Disease Classification dataset. Our work could potentially expedite accurate interpretation of complex imaging features within the CMR study and lead to more consistent and effective diagnosis and treatment.