Cross modal reliable pixel contrastive learning for incomplete modal brain tumor segmentation

Abstract In clinical practice, Magnetic Resonance Imaging (MRI) often lacks specific modalities, inevitably leading to a degradation in predictive performance. Different modes are currently treated as independent and non-interfering during training for modal feature extraction, yet there are rich semantic relations between pixels across different modalities. This paper proposes the Cross Modal Reliable Pixel Contrastive Learning (CMR-PCL) algorithm for incomplete modal brain tumor segmentation to compensate for the information deficit among the modalities. Specifically, we propose a label inaccuracies-guided sampling strategy for each modality and then preserve the reliable region to reduce the likelihood of noise sampling. Next, we enforce reliable pixel embeddings belonging to the same semantic class to be more similar than those from different classes. CMR-PCL uses a standard training strategy and requires no specific architectural choices so that it can be easily incorporated into existing incomplete modal brain tumor segmentation. Remarkably, extensive experiments on the BraTS2020, BraTS2018, and BraTS2015 datasets show that CMR-PCL can improve the performance of state-of-the-art algorithms.