CRFTrans: A recursive vision transformer reformulating the mean-field inference in conditional random field for medical image segmentation.

Transformers have recently revolutionised medical image segmentation, achieving state-of-the-art (SOTA) performance in many clinical applications. However, prevailing architectures relying on cascaded Transformer layers (e.g., 12-layer Vision Transformer) face practical limitations, leading to high computational cost and parameter redundancy. In this work, we present a new perspective that interprets Transformer operations through the paradigm of probabilistic inference. Specifically, we reveal a structural correspondence between Transformer blocks and mean-field inference in fully connected Conditional Random Fields (CRFs). Based on this insight, we propose CRFTrans, a recursive Transformer layer that reformulates the mean-field inference process using learnable attention and feed-forward operations. Unlike traditional CRFs that rely on fixed Gaussian kernels, CRFTrans leverages self-attention to model adaptive pairwise relationships in a high-dimensional feature space, enabling more expressive and data-driven contextual reasoning. We replace cascaded Transformer layers with CRFTrans in multiple state-of-the-art segmentation models and evaluate on five public datasets. Results show comparable or improved performance with significantly fewer parameters, lower memory usage, and faster training. CRFTrans provides a mathematically grounded, lightweight foundation for resource-constrained clinical deployments. The code is available in this GitHub link https://github.com/naisops/CRFTrans/tree/main.