Evaluating the capability of machine learning models to detect unknown cyberattacks at digital substations

Abstract

The article addresses the problem of evaluating the capability of machine learning models to detect unknown cyberattacks in network traffic of digital substations. The relevance of the study is determined by the fact that digital substations are part of critical infrastructure, and their operation depends on secure network interaction between automation devices, monitoring systems and industrial communication protocols. The experimental study is based on the processed IEC 104 part of the SANDI-2024 dataset, which was designed for training and evaluating intrusion detection systems for electrical substations. The paper proposes a leave-one-attack-out evaluation scenario, in which one attack type is completely excluded from the training set and used only during testing. This approach makes it possible to evaluate the ability of models to detect attack types that were not represented in the training data. The study compares supervised machine learning models and anomaly-based approaches. The supervised group includes Logistic Regression, K-Nearest Neighbors, Linear SVM, Gaussian Naive Bayes, Random Forest, Extra Trees and HistGradientBoosting. The anomaly-based group includes Isolation Forest, One-Class SVM and Local Outlier Factor. The experimental results show that in the standard known-attack scenario, the models achieve nearly perfect classification metrics. However, in the leave-one-attack-out scenario, model performance depends significantly on the type of unseen attack and on the decision threshold. Extra Trees achieved the best average result among supervised models, while Local Outlier Factor demonstrated the best balance among anomaly-based models. Additional threshold analysis showed that the detection of the most difficult unseen attack can be significantly improved by adjusting the classification threshold. The obtained results confirm the importance of using unseen attack detection scenarios for a more realistic evaluation of IDS models in digital substations.