Learning When and Where to Handover: A Hierarchical Reinforcement Learning Framework for Dense LEO Satellite Constellations

The rapid development of low Earth orbit (LEO) satellite constellations has enabled global broadband coverage and low-latency communication, but also brings new challenges to connection continuity and network stability. Due to the ultra-dense deployment and high orbital velocity of LEO satellites, users frequently experience mandatory handovers and are prone to triggering “ping-pong” effects in overlapping coverage areas. To address these issues, we propose a hierarchical reinforcement learning (HRL)-based inter-satellite handover framework that decouples decision-making into two subproblems: when to initiate a handover at the temporal level and which satellite to handover to at the spatial level. To achieve long-term planning, the temporal agent utilizes the proximal policy optimization (PPO) algorithm to determine the handover timing by predicting future satellite load and interference levels. For short-term adaptability, the spatial agent employs the deep Q-network (DQN) to select the target satellite based on a utility function incorporating satellite load, signal quality, and service duration. We implement the proposed method in a Starlink constellation environment. Simulation results show that the proposed HRL-based scheme outperforms traditional and non-hierarchical RL methods in terms of user handover frequency and handover success rate. Furthermore, we validate that the proposed approach significantly improves transmission rate and load balancing, demonstrating its effectiveness in highly dynamic LEO satellite environments.