ATSG: Adaptive Token Linking With Segment Anything Model Guidance for Weakly Supervised Remote Sensing Image Semantic Segmentation

Semantic segmentation of remote sensing images is vital for applications such as urban planning and disaster monitoring. However, the high cost of pixel-level annotations often results in limited labeled data, necessitating weakly supervised learning approaches. Unlike natural images, remote sensing images typically contain numerous small objects with substantial intra-class variations and high inter-class similarities, which poses challenges for generating high-quality pseudo label. Additionally, while Vision Transformer (ViT) have been integrated into Weakly Supervised Semantic Segmentation (WSSS) for their global modeling capabilities, they are prone to over-smoothing in dense scenes, which impedes model learning. To address these challenges, we propose two novel modules: the Adaptive Token Linking Module (Adalink) and the Segment Anything Model (SAM)-Guided Boundary Refiner module(SGBR). First, Adalink employs a dynamic aggregation mechanism to analyze semantic diversity across the middle layer of ViT, adaptively selecting feature layer and constructing token correlation graphs. It leverages a self-supervised encoder to extract hierarchical token relationships, which to supervise pseudo label generation, thereby reducing erroneous activations in cluttered scenes and improving pseudo label quality. Second, SGBR utilizes the zero-shot segmentation capability of the SAM to refine segmentation results by incorporating object and boundary priors from the output images, significantly enhancing the completeness of small object segmentation and overall accuracy. Extensive experiments on the ISPRS Potsdam, ISPRS Vaihingen, and iSAID datasets demonstrate that our method achieves state-of-the-art (SOTA) performance and exhibits strong practical value in processing complex remote sensing scenes. Code will be available at: https://github.com/zhangyifan25/ATSG.