Evaluating Adversarial Robustness of AI Intrusion Detection Systems Using Automated Traffic Generation

Artificial intelligence (AI) has substantially advanced intrusion detection systems (IDS) by enabling scalable and adaptive analysis of network flows, yet these models remain vulnerable to adversarial manipulation. This study develops a comprehensive and reproducible framework for evaluating the adversarial robustness of machine learning-based IDS under realistic black-box threat conditions. Using CICIDS2017 as the primary benchmark, four representative IDS models, Random Forest, Logistic Regression, Multilayer Perceptron, and CNN1D, are trained and assessed under clean conditions and multiple adversarial scenarios, including surrogate-based FGSM and PGD perturbations, as well as HopSkipJump (HSJA) and Zeroth-Order Optimization (ZOO) black-box attacks. The results reveal substantial differences in robustness across model families: Random Forest remains consistently stable across all attacks, whereas the MLP exhibits severe performance degradation under PGD transfer. Logistic Regression and CNN1D show mixed susceptibility depending on attack strength and feature sensitivity. To evaluate generalization under distribution shift, the models are further tested on the CICIDS2018 Friday slice, showing limited cross-dataset transferability of adversarial examples. These findings reveal significant differences between white-box and operational black-box vulnerability and demonstrate that adversarial robustness depends strongly on model architecture and dataset alignment. The proposed evaluation methodology provides a practical basis for integrating adversarial stress testing into IDS development and deployment workflows.