YOLO-SAM an end-to-end framework for efficient real time object detection and segmentation

Although effective and practical YOLO methods have dominated the field of object detection, they rely on predefined and trained object categories, which limits their broad application. To overcome this limitation, YOLO-World enhances YOLO's open-vocabulary detection capabilities through modeling visual language and pretraining on large-scale datasets. Therefore, this manuscript proposes an improved object detection and segmentation model based on YOLO-World-S to improve detection efficiency and accuracy. The computational complexity and memory usage are reduced by introducing large-kernel separable convolutions into the RepVL-PAN of YOLO-World-S. The Neck incorporates a dynamic sparse attention mechanism (PSBRA module) to reduce the computational cost of traditional multi-head self-attention, while facilitating the integration of YOLO-World-S with EfficientSAM. In addition, the loss function is reconstructed to effectively resolve conflicts on shared features or optimization objectives between object detection and segmentation tasks. This method achieves mAPs (mean average precisions) of 58.8% and 308 FPSs (frames per second) on the COCO dataset, improving both accuracy and speed compared with those of the original baseline model.