Real-Time Anomaly Detection in TLS Encrypted Network Traffic Using Machine Learning and Zeek Integration

Abstract

The rapid growth of encrypted network traffic has posed significant challenges for network security practitioners in detecting and mitigating malicious activities. Due to encryption, traditional anomaly detection methods that rely on inspecting packet contents are rendered ineffective. This addresses the critical need to detect anomalies within encrypted traffic without compromising privacy through decryption. This study proposes a machine learning-based approach for anomaly detection in encrypted network traffic. The research methodology encompasses dataset creation, feature extraction, model implementation, and evaluation. The CTU-13 dataset is enhanced through an automated labelling process. A comprehensive set of features, including connection-level, IP header, and TCP header features, is extracted to capture relevant information from TLS-encrypted network traffic for distinguishing between normal and malicious traffic. Multiple machine learning models, namely Random Forest, Weighted Scoring Model, and Isolation Forest are implemented and evaluated. The Random Forest model emerges as the best-performing model, achieving high accuracy in detecting anomalies within encrypted network traffic. The proposed system demonstrates its ability to analyse and process encrypted traffic in real-time by leveraging the capabilities of Zeek, a powerful network security monitoring tool. Integrating Zeek's real-time processing functionality with the Random Forest model, as the best performing model, enables prompt detection and response to malicious activities.