A multi-scale cross-modal fusion method for zero-shot surface defect visual detection in indexable inserts

Abstract Indexable inserts feature a wide variety of types and undergo frequent updates, posing enormous challenges to automated visual inspection. Traditional deep learning methods require extensive data collection and retraining for each new type of insert, resulting in high training costs and overly stringent requirements for quality control personnel, which hinders their practical application. To address these issues, we propose a novel zero-shot anomaly detection method in this paper, specifically, a multi-scale multi-modal segmentation contrastive language-image pre-training (CLIP) model (termed M2S-CLIP). First, we designed a multi-scale segment anything model (SAM)-CLIP distillation learning adapter (MS-SAM adapter) that combines the semantic understanding capabilities of the CLIP with the fine-grained segmentation knowledge of the SAM, thereby enhancing the model’s ability to detect fine details. Thereafter, we introduce a learnable textual prompt template based on prompt learning, which enhances the multi-modal large model’s understanding of industrial scenarios. Subsequently, a multi-scale cross-modal fusion module (M2P-Fuse) is designed to extract visual features at multiple scales while dynamically guiding textual features with visual cues. Finally, a bottleneck-structured aligner is developed to achieve precise image-text alignment. Experimental results demonstrate that M2S-CLIP achieves an image-level AUROC of 95.2% and a pixel-level AUROC of 93.0% on our self-built dataset, significantly outperforming existing methods, while cross-domain tests on MVTec AD and VisA verify its strong generalization capability.