Cognitive Spectrum Sharing in Space-Air-Ground Integrated Networks: Optimizing Beamforming and AAV Trajectories With Two-Layer RSMA

The realization of space-air-ground integrated networks (SAGIN) is currently impeded by significant spectrum scarcity and complex cross-layer interference. To tackle these challenges, we propose a novel cognitive hierarchical spectrum sharing framework, wherein the satellite network operates as the primary network and high-altitude platforms (HAPs) and autonomous aerial vehicles (AAVs) collaboratively function as the secondary network. Based on this architecture, we propose a two-layer rate-splitting multiple access scheme that orchestrates the synergy between the wide-area coverage of HAPs and the flexible mobility of AAVs. This scheme robustly manages both intra-layer and inter-layer interference, thereby satisfying the heterogeneous quality-of-service requirements of terrestrial users. To maximize the system sum rate, a joint optimization problem is formulated for transmit beamforming and AAV trajectories, constrained by strict satellite interference temperature limits. To address this non-convex problem, we propose a deep iterative beamforming algorithm that leverages successive convex approximation and deep deterministic policy gradient techniques. Simulation results indicate that the proposed scheme achieves up to 25% higher throughput than conventional zero-forcing and non-orthogonal multiple access schemes for SAGIN.