Optimizing XGBoost for Intrusion Detection Using a Hybrid Firefly-PSO Algorithm

Intrusion detection systems (IDS) play a critical role in safeguarding networks against cyberattacks. Machine learning algorithms, particularly XGBoost, have been widely adopted in IDS for their robustness and efficiency. However, the performance of XGBoost can be significantly improved through hyperparameter optimization. This study proposes a hybrid Firefly-Particle Swarm Optimization (Firefly-PSO) algorithm for tuning XGBoost's hyperparameters to enhance intrusion detection performance. The hybrid algorithm combines the global search ability of PSO and the local search efficiency of the Firefly Algorithm. The proposed model is evaluated on three benchmark datasets: NSL-KDD, CICIDS2017, and UNSW-NB15. Experimental results show that the Firefly-PSO XGBoost model outperforms traditional optimization techniques such as Grid Search and Random Search in terms of accuracy, precision, recall, F1-score, and computational efficiency. Additionally, the model demonstrates excellent generalization capability and minimal false positive and negative rates, making it a robust solution for real-world IDS deployment. The findings of this study highlight the potential of hybrid optimization algorithms for improving machine learning-based IDS.