HyFormer-Net: A synergistic CNN-Transformer with interpretable multi-scale fusion for breast lesion segmentation and classification in ultrasound images

Background: Breast ultrasound diagnosis faces challenges from speckle noise, operator dependency, and indistinct lesion boundaries. Existing deep learning methods lack hierarchical multi-scale integration, quantitative interpretability validation, and cross-dataset generalization analysis, hindering clinical adoption. Methods: We propose HyFormer-Net, a hybrid CNN-Transformer framework integrating EfficientNet-B3 and Swin Transformer via multi-scale hierarchical fusion blocks at four encoder stages. The attention-gated decoder enables dual-pipeline interpretability: intrinsic attention validation (quantitative IoU verification) and Grad-CAM classification reasoning. We evaluated performance on BUSI dataset and conducted systematic cross-dataset validation on BUS-UCLM with progressive fine-tuning. Results: HyFormer-Net achieved 76.1% Dice score and 93.2% classification accuracy on BUSI, with clinically critical 92.1% Malignant Recall. Ensemble modeling reached 90.2% Dice and 100% Malignant Recall. Ablation studies revealed multi-scale fusion contributed +16.8% Dice improvement. Internal attention maps achieved mean IoU 0.86 with ground truth. Cross-dataset evaluation showed zero-shot failure (5.8% Dice), but 10% target-domain fine-tuning (68 images) recovered 92.5% performance. Conclusions: HyFormer-Net demonstrates clinically viable performance with quantitatively validated interpretability. The 10% fine-tuning requirement provides actionable deployment guidelines for real-world clinical translation across imaging centers.