Towards Robust Cybersecurity Realm: An Exhaustive Evaluation of AI-Driven Approaches for Enhanced Insider Threat Detection

Abstract

Today, insider threats pose a significant risk to an organization's cybersecurity posture, often proving difficult to detect and causing substantial damage not only to an organization's financial resources but also to its reputation, mission, personnel, infrastructure, information, equipment, networks, or systems. Despite their critical importance, many organizations tend to primarily focus on external threats, unintentionally neglecting those that come from within. This study aims to explore the effectiveness of artificial intelligence in detecting insider threats in the cybersecurity landscape. It focuses on evaluating different algorithms and their ability to identify unusual behaviour patterns that indicate potential insider threats. To achieve this goal, the study involves developing a Python-based machine learning program in Jupyter Notebook to assess the performance of various anomaly-based and classification-based models such as One-Class Support Vector Machine (OCSVM), Isolation Forest (iForest), Support Vector Machine (SVM), Random Forest (RF), Adaptive Boosting (AdaBoost), Logistic Regression (LR), Extreme Gradient Boosting (XGBoost), and Neural Network (NN). Additionally, the study will conduct a comprehensive examination and comparative analysis of three sophisticated techniques: SelectKBest, Principal Component Analysis, and Synthetic Minority Over-sampling to enhance and optimize the performance of the selected models. This will ultimately lead to identifying the most efficient, anomaly, and classification-based detection models that deliver outstanding performance results, as well as identifying the best techniques to optimize their performance. For anomaly-based detection, the study's results revealed that the iForest algorithm demonstrated superior performance over OCSVM, achieving remarkable metrics of 90% Precision, 93% Recall, 92% F1-Score, and 93% Accuracy. For the classification-based models, a variety of combinations produced impressive results. The integration of the SMOTE technique and SelectKBest proved to be effective in reducing the occurrence of false positives. For instance, the RF-SMOTE-SelectKBest model showcased a remarkable 100% Recall and 99% Accuracy. The SVM-SMOTE-SelectKBest model maintained consistent performance metrics, recording 97% in Precision, Recall, F1-Score, and 99% Accuracy. The AdaBoost-SMOTE-SelectKBest model achieved 99% Accuracy. The XGBoost-SMOTE-SelectKBest model delivered 95% Precision, 95% Recall, 95% F1-Score, and 99% Accuracy. The NN-SMOTE-SelectKBest model exhibited exceptional performance, achieving 99% Accuracy, 97% Precision, and 95% Recall. The results of this study provide important insights into the ability of AI to efficiently identify insider threats, as well as in helping to select appropriate methods to enhance the effectiveness of insider threat detection.