Distributed Explainable Ensemble Anomaly Detection for Cloud-Native Applications Using Azure AI and SQL Analytics

Abstract

In this study, we propose an ensemble anomaly detection architecture based on explainability and distributed computing. Using a combination of Azure Machine Learning and distributed SQL-based analytics, our method attempts to handle large amounts of telemetry data while remaining fully interpretable throughout the process. Our proposed framework involves the combination of three models, including Isolation Forest, LightGBM, and a time-series detector using CNN, and utilizes the outputs from these three in coordination by means of Azure Synapse Analytics, which allows us to adjust the weightings dynamically. To maintain the explainability of our pipeline, we used game-theoretic Shapley values to attribute anomalies to particular features and even sub-models in near real-time.