AI Powered Log Analysis and Threat Detection System for Windows

Abstract

The increasing volume and complexity of cyber threats demand advanced, automated methods for analyzing Windows event logs. Traditional rule-based systems often fail to detect novel attacks, prompting the exploration of deep learning techniques. This research develops and evaluates an anomaly detection system by fine tuning a BERT (Bidirectional Encoder Representations from Transformers) model on the windows system security logs. The methodology involved processing the ATLASv2 dataset, a collection of 20.5 million realistic Windows Security Logs containing both benign and malicious activity. A baseline model was implemented using the Hugging Face transformers library and trained on a representative sample of 100,000 log events, accelerated by a GPU. Evaluation of this baseline model on an unseen validation set demonstrated strong performance, achieving 96.98% overall accuracy and a 94.55% precision rate. The key finding was a recall of 79.10%, indicating a weakness in detecting rare malicious events due to the natural class imbalance of the dataset. To address this, a new, perfectly balanced dataset was created using oversampling, which dramatically improved the model's F1-Score to 95.33%. Following this data-centric improvement, a comprehensive hyperparameter tuning phase was conducted, employing Grid Search, Random Search, and Bayesian Optimization. This optimization successfully identified a BEST model with a high F1-Score of 96.60%. This research successfully validates a complete framework for applying and optimizing advanced AI models for log analysis. The next phase will focus on implementing a functional prototype with a user interface and expanding the comparative analysis to include other traditional ML models to further strengthen the research findings