Wujie Zhou, Jin Xie, Caie Xu, Yuanyuan Liu, Yunchao Wang
Foundation models such as the segment anything model (SAM) have remarkable generalization capabilities in natural image segmentation. However, their application to remote sensing (RS) semantic segmentation faces significant challenges. The single-modal architecture of SAM cannot effectively utilize multi-modal RS data, its vision Transformer encoder lacks sensitivity to multi-scale spatial structures that are characteristic of RS imagery, and its dependence on manual prompts hinders large-scale automation. To address these challenges, we propose a geometry-aware diffusion-guided SAM framework (GeoSAM) that transforms SAM into fully automated multi-modal semantic segmentation through three synergistic innovations. First, we introduce a multi-scale geometric-aware adaptation module (GeoAdapter) that hierarchically integrates RGB images with normalized digital surface model data within the encoder. GeoAdapter incorporates a novel class-prior generator that combines geometric convolution, prototype similarity matching, and statistical modeling to produce structure-aware guidance for semantic–geometric feature fusion. Second, we present a diffusion prompt module that pioneers the use of conditional diffusion models for automatic prompt generation in RS applications, eliminating manual interactions through semantic-feature-guided denoising diffusion implicit model sampling. Third, we propose a prompt-level supervision strategy that mitigates training–inference distribution discrepancies through constraints on semantic consistency and structural alignment, ensuring robust prompt generation across different phases. Extensive experiments demonstrate state-of-the-art performance: 90.92% mean accuracy (mAcc) and 82.71% mean intersection over union (mIoU) on Vaihingen, and 85.28% mAcc and 75.49% mIoU on Potsdam, with improvements of 16.52% mAcc and 16.29% mIoU over original SAM on Vaihingen. The source code is available at https://github.com/110-011/GeoSAM.