5G-AKA-HPQC: Hybrid Postquantum Cryptography Protocol for Quantum-Resilient 5G Primary Authentication With Forward Secrecy

5G serves as a catalyst for transformative digital innovation by enabling convergence with various services in our daily lives. The success of this paradigm shift undeniably hinges on robust security measures, with primary authentication— securing access to the 5G network—being paramount. Two protocols, 5G Authentication and Key Agreement (5G-AKA) and the Extensible Authentication Protocol for Authentication and Key Agreement Prime (EAP-AKA’), have been standardized for this purpose, with the former designed for 3rd Generation Partnership Project (3GPP) devices and the latter for non-3GPP devices. However, recent studies have exposed vulnerabilities in the 5G-AKA protocol, rendering it susceptible to security breaches, including linkability attacks. Furthermore, the advent of quantum computing poses significant quantum threats, underscoring the urgent need for the adoption of quantum-resistant cryptographic mechanisms. Although post-quantum cryptography (PQC) is being standardized, the lack of real-world deployment limits its proven robustness. In contrast, conventional cryptographic schemes have demonstrated reliability over decades of practical application. To address this gap, the Internet Engineering Task Force (IETF) has initiated the standardization of hybrid PQC algorithms (HPQC), combining classical and quantum-resistant techniques. Consequently, ensuring forward secrecy and resilience to quantum threats in the 5G-AKA protocol is critical. To address these security challenges, we propose the 5G-AKA-HPQC protocol. Our protocol is designed to maintain compatibility with existing standards while enhancing security by combining keys negotiated via the Elliptic Curve Integrated Encryption Scheme (ECIES) with those derived from a PQC-Key Encapsulation Mechanism (KEM). To rigorously and comprehensively validate the security of 5G-AKA-HPQC, we employ formal verification tools such as SVO Logic and ProVerif. The results confirm the protocol’s security and correctness. Furthermore, performance evaluations highlight the computational and communication overheads inherent to 5G-AKA-HPQC. With only average of 56.5 millisecond(+112.32%) on a total authentication time, proposed protocol provides its advantages. Also, on a environment of multiple UE registration, compared to single UE registration, the difference of increased rate of average authentication time is only 0.01%. The negligible difference 0.01% indicates that the addition of PQC does not lead to increased overhead in multi-UE registration environments, demonstrating its scalability and practical feasibility. In conclusion, our research provides significant insights into the design of secure, quantum-safe authentication protocols and lays the groundwork for the future standardization of secure authentication and key agreement protocols for mobile telecommunications.