MCG: MARGINAL–CONDITIONAL GRAPH LEARNING FOR JOINT ANOMALY DETECTION AND ROOT CAUSE LOCALIZATION IN DISTRIBUTED CLOUD SYSTEMS
Keywords:
AIOps, Anomaly detection, Root cause analysis, Graph neural networks, Multivariate time series, Microservices, Cloud computingAbstract
Distributed cloud systems expose large numbers of interdependent performance metrics, and operators must both detect anomalies and identify the component that caused them. Deep multivariate detectors provide per-metric anomaly evidence, yet this evidence mixes the faulty component with the many metrics that merely inherit its effects, whereas most root cause analysis (RCA) techniques require an accurate call topology or costly causal discovery at incident time. This paper proposes MCG, a Marginal–Conditional Graph network that unifies anomaly detection and topology-free root cause localization. MCG learns a sparse directed dependency graph among metrics and couples two predictive heads: a marginal head that forecasts each metric from history only, and a conditional head that additionally propagates the innovations, i.e., the unexpected deviations, of the learned parents of each metric at the current time step. The marginal residual drives detection and metric-level diagnosis, while the conditional residual isolates the part of a deviation that upstream metrics cannot explain, which is the statistical signature of a fault intervention. Experiments on the public SMD server-monitoring dataset and on a simulated microservice benchmark with 120 injected faults compare MCG with seven detectors and five RCA methods. MCG attains the fourth-best average detection rank across the two evaluated datasets, competitive dimension-level diagnosis accuracy on SMD (HitRate@150% of 0.587), and an AC@1 of 0.733 in root-cause service localization without any topology knowledge, the best result among topology-free methods.References
[1] Dang Y, Lin Q, Huang P. AIOps: Real-world challenges and research innovations. In: Proc. IEEE/ACM 41st Int. Conf. Softw. Eng.: Companion, 2019: 4-5.
[2] Notaro P, Cardoso J, Gerndt M. A survey of AIOps methods for failure management. ACM Trans. Intell. Syst. Technol., 2021, 12(6): 81.
[3] Soldani J, Brogi A. Anomaly detection and failure root cause analysis in (micro) service-based cloud applications: A survey. ACM Comput. Surv., 2022, 55(3): 1-39.
[4] Su Y, Zhao Y, Niu C, et al. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In: Proc. 25th ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., 2019: 2828-2837.
[5] Audibert J, Michiardi P, Guyard F, et al. USAD: Unsupervised anomaly detection on multivariate time series. In: Proc. 26th ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., 2020: 3395-3404.
[6] Zhao H, Wang Y, Duan J, et al. Multivariate time-series anomaly detection via graph attention network. In: Proc. IEEE Int. Conf. Data Min. (ICDM), 2020: 841-850.
[7] Deng A, Hooi B. Graph neural network-based anomaly detection in multivariate time series. In: Proc. AAAI Conf. Artif. Intell., 2021, 35(5): 4027-4035.
[8] Tuli S, Casale G, Jennings N R. TranAD: Deep transformer networks for anomaly detection in multivariate time series data. Proc. VLDB Endow., 2022, 15(6): 1201-1214.
[9] Xu J, Wu H, Wang J, et al. Anomaly Transformer: Time series anomaly detection with association discrepancy. In: Proc. Int. Conf. Learn. Represent. (ICLR), 2022.
[10] Li Z, Zhao Y, Han J, et al. Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In: Proc. 27th ACM SIGKDD Conf. Knowl. Discov. Data Min., 2021: 3220-3230.
[11] Kim M, Sumbaly R, Shah S. Root cause detection in a service-oriented architecture. In: Proc. ACM SIGMETRICS, 2013: 93-104.
[12] Wang P, Xu J, Ma M, et al. CloudRanger: Root cause identification for cloud native systems. In: Proc. 18th IEEE/ACM Int. Symp. Cluster, Cloud Grid Comput. (CCGRID), 2018: 492-502.
[13] Wu L, Tordsson J, Elmroth E, et al. MicroRCA: Root cause localization of performance issues in microservices. In: Proc. IEEE/IFIP Netw. Oper. Manage. Symp. (NOMS), 2020: 1-9.
[14] Ma M, Xu J, Wang Y, et al. AutoMAP: Diagnose your microservice-based web applications automatically. In: Proc. Web Conf. (WWW), 2020: 246-258.
[15] Spirtes P, Glymour C, Scheines R. Causation, Prediction, and Search. 2nd ed. Cambridge, MA, USA: MIT Press, 2000.
[16] Lin J, Chen P, Zheng Z. Microscope: Pinpoint performance issues with causal graphs in micro-service environments. In: Proc. Int. Conf. Service-Oriented Comput. (ICSOC), 2018: 3-20.
[17] Meng Y, Zhang S, Sun Y, et al. Localizing failure root causes in a microservice through causality inference. In: Proc. IEEE/ACM 28th Int. Symp. Qual. Service (IWQoS), 2020: 1-10.
[18] Li M, Li Z, Yin K, et al. Causal inference-based root cause analysis for online service systems with intervention recognition. In: Proc. 28th ACM SIGKDD Conf. Knowl. Discov. Data Min., 2022: 3230-3240.
[19] Ikram A, Chakraborty S, Mitra S, et al. Root cause analysis of failures in microservices through causal discovery. In: Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2022, 35: 31158-31170.
[20] Liu F T, Ting K M, Zhou Z H. Isolation forest. In: Proc. 8th IEEE Int. Conf. Data Min. (ICDM), 2008: 413-422.
[21] Hundman K, Constantinou V, Laporte C, et al. Detecting spacecraft anomalies using LSTMs and nonparametric dynamic thresholding. In: Proc. 24th ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., 2018: 387-395.
[22] Abdulaal A, Liu Z, Lancewicki T. Practical approach to asynchronous multivariate time series anomaly detection and localization. In: Proc. 27th ACM SIGKDD Conf. Knowl. Discov. Data Min., 2021: 2485-2494.
[23] Chen Z, Chen D, Zhang X, et al. Learning graph structures with transformer for multivariate time-series anomaly detection in IoT. IEEE Internet Things J., 2022, 9(12): 9179-9189.
[24] Kim S, Choi K, Choi H S, et al. Towards a rigorous evaluation of time-series anomaly detection. In: Proc. AAAI Conf. Artif. Intell., 2022, 36(7): 7194-7201.
[25] Garg A, Zhang W, Samaran J, et al. An evaluation of anomaly detection and diagnosis in multivariate time series. IEEE Trans. Neural Netw. Learn. Syst., 2022, 33(6): 2508-2517.
[26] Liu D, He C, Peng X, et al. MicroHECL: High-efficient root cause localization in large-scale microservice systems. In: Proc. IEEE/ACM 43rd Int. Conf. Softw. Eng.: Softw. Eng. Pract. (ICSE-SEIP), 2021: 338-347.
[27] Shan H, Chen Y, Liu H, et al. ε-Diagnosis: Unsupervised and real-time diagnosis of small-window long-tail latency in large-scale microservice platforms. In: Proc. World Wide Web Conf. (WWW), 2019: 3215-3222.
[28] Wu Z, Pan S, Long G, et al. Connecting the dots: Multivariate time series forecasting with graph neural networks. In: Proc. 26th ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., 2020: 753-763.
[29] Veličković P, Cucurull G, Casanova A, et al. Graph attention networks. In: Proc. Int. Conf. Learn. Represent. (ICLR), 2018.
[30] Kingma D P, Ba J. Adam: A method for stochastic optimization. In: Proc. Int. Conf. Learn. Represent. (ICLR), 2015.