Seventeenth International Conference on Technology, Knowledge, and Society

Abstract

While leveraging cloud computing for large-scale applications allows seamless scaling, many companies struggle following up with the amount of data generated in terms of efficient processing and anomaly detection. With the rapid growth of web attacks, anomaly detection becomes a necessary part of the management of modern large-scale distributed web applications. As the record of user behavior, weblogs certainly become the research object related to anomaly detection. Many anomaly detection methods based on automated log analysis have been proposed. However, not in the context of big data applications where normal and anomalous behavior models need to be constructed prior to prediction attempts. To address this problem, Big Data Analytics and Machine Learning (ML) algorithms in overcoming the challenges of data processing, pattern detection, and anomaly prediction in high-dimensional data are utilized. Integrating CRISP-DM methodology, we propose PCA and a combination of unsupervised ML algorithms: Random Isolation Forest and Global Homogenous Outlier Search for pattern detection and construction of labeled dataset, that is the initial model of behaviors. Next, a supervised ML algorithm one-class SVM is trained to generalize the behavior model in order to predict user behavior anomalies. Results show that One-Class SVM is the most efficient supervised algorithm in generalizing the behavior patterns and improves the patterns detected by unsupervised models with a prediction accuracy of 99% and outlier class recall of 80%. We conclude the use of unsupervised learning as a baseline improves the model aging, and the one-class approach in supervised learning contributes to better pattern recognition.