A comprehensive survey on intrusion detection systems with advances in machine learning, deep learning and emerging cybersecurity challenges

Abstract Recent advancements in technology, particularly in artificial intelligence (AI) and machine learning (ML), have significantly enhanced connectivity and efficiency in various domains. However, these developments have also led to increased threats and concerns regarding cyberattacks which have significantly challenged governments, businesses, and societies. An Intrusion Detection System (IDS) is a critical component of a cybersecurity architecture that is designed to monitor and analyze network traffic for signs of malicious activity and plays an essential role in detecting and responding to security threats. The increasing complexity of security attacks has led researchers to use machine learning (ML) and deep learning (DL) methods to enhance IDS performance. These cutting-edge systems are designed to detect emerging cyberthreats. The current study investigates and compares various types of IDS and AI-based techniques that use ML and DL algorithms to protect data against malicious threats. It also provides an overview of the architecture and operation of IDS and examines popular datasets such as CIC-IDS2017, KDDCup99 and UNSW-NB15 used to evaluate IDS models. These datasets offer a diverse range of attack scenarios and normal traffic and help create a more reliable and effective IDS which improves the overall network security. The study concludes by emphasizing constant progress in IDS technologies to combat evolving cyberthreats and enhance network security.