Evaluating ASCERT: generative AI for cyber-range scenario generation

Abstract In this paper, we worked in collaboration with the ASCERT (AI-based scenario management for cyber-range training) project and its generative AI prototype that generates dynamic and interactive cyber-range exercise scenarios. We evaluate the model by focusing on two objectives: (i) its ability to replicate real-world cyber attacks, and (ii) its consistency across multiple simulations that uses same inputs. To assess realism, we examine how well the model reproduces three well-documented cyber incidents namely Colonial Pipeline, Equifax, and SolarWinds, when it is provided with relevant source material for training. We then analyze repeatability by comparing outputs across fixed-input simulation runs. As the evaluation results indicate, overall the model generated varied and context-appropriate scenarios. Moreover, it introduced an interactivity feature that allows users to choose responses and observe consequences in real time. However, the consistency in repeated runs was limited: simulations are not reliably repeatable, although what was interesting is that the variability reflects the unpredictability of real attacks. These findings suggest that, while ASCERT already supports scenario variety and meaningful user interaction, it requires targeted refinements to improve stability and repeatability. With such improvements, the ASCERT model has strong potential to contribute to scalable and adaptive cybersecurity education and training.