A Hybrid Vision Transformer and U-Net Framework for Automated Cardiac MRI Segmentation and Abnormality Detection

Abstract

Cardiovascular diseases remain a leading cause of mortality worldwide, and accurate segmentation of cardiac structures from MRI is critical for clinical diagnosis. We propose a hybrid framework that integrates a Vision Transformer encoder with a U-Net decoder for automated cardiac MRI segmentation and abnormality detection. The encoder divides input images into non-overlapping patches and processes them through multi-head self-attention layers, thereby capturing long-range dependencies and global anatomical context. These encoded features are then passed to the U-Net decoder, where skip connections preserve high-resolution spatial details from corresponding encoder levels. This design enables the decoder to combine global semantic information with local structural cues for precise pixel-wise segmentation of the left ventricle, right ventricle, and myocardium. A composite loss function, balancing Dice loss and cross-entropy loss, is employed to handle class imbalance and ensure robust pixel discrimination. Following segmentation, quantitative biomarkers such as ventricular volumes, ejection fraction, and wall thickness are extracted from the predicted masks. These features are subsequently fed into a fully connected neural network for binary classification of examinations as normal or abnormal. Furthermore, attention maps derived from the transformer layers provide interpretability by highlighting image regions that most influence the diagnostic decision. The proposed method is evaluated on short-axis cine MRI data with rigorous patient-level data partitioning and extensive augmentation. Our contributions include a novel hybrid architecture that effectively merges global and local feature representations, a clinically relevant abnormality detection pipeline, and an interpretability mechanism that aligns with expert annotations. This work demonstrates significant potential for improving automated cardiac assessment in clinical workflows.