Energy-Efficient Transmission in STAR-RIS Assisted Secure ISAC Networks With RSMA: An MoE-RBPPO Approach

Simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS) represents an innovative technology with the potential to enhance global coverage in future wireless networks. This paper investigates secure down link transmission and rate-splitting multiple access (RSMA) in a STAR-RIS-assisted integrated sensing and communication (ISAC) system, focusing on the communication and sensing needs of future networks. We propose an optimization problem to maximize system secrecy energy efficiency (SEE) by jointly op timizing beamforming vectors, the common-stream secrecy rate for users, and the amplitude and phase of the STAR-RIS under various operational constraints. To address the challenges inherent in solving this non-convex optimization problem, primarily stemming from complex channel dynamics and the coupling of mixed variables, we introduce a novel mixture of experts rollback proximal policy optimization (MoE-RBPPO) approach. Additionally, we utilize maximum ratio transmission and zero forcing techniques for beamforming design. Simulation results indicate that the proposed MoE-RBPPO method significantly outperforms several baseline approaches in terms of SEE. For example, it achieves a SEE gain of up to 4.5% over MoE-PPO and up to 9.2% over PPO, with these gains observed under varying radar cross section values of the target. Meanwhile, the overall simulation findings further highlight the inherent advantages of STAR-RIS-enhanced ISAC systems and the benefits of RSMA over space division multiple access.