DEPENDENCY-AWARE ROOT CAUSE ANALYSIS AND AUTOMATED FAULT RECOVERY FOR CLOUD-NATIVE MICROSERVICES
Keywords:
Microservices, Root cause analysis, Distributed tracing, AIOps, Self-healing, Automated fault recoveryAbstract
Failures in cloud-native microservice systems propagate along service dependencies, so the services that raise alarms are frequently victims rather than culprits, and a diagnosis alone does not restore service. This paper presents DARE, a dependency-aware framework that unifies root cause analysis (RCA) and automated recovery. DARE converts heterogeneous metrics, distributed traces, and logs into history-calibrated evidence that deflates routinely noisy channels; it separates intrinsic from inherited anomalies through exclusive span latency, error-origin attribution, propagation-aware residuals, and latent-state inference for uninstrumented services; and it ranks root-cause hypotheses with a symptom-gated likelihood-ratio model over a two-layer (call and co-location) dependency graph, which also yields a fault-type belief. A decision-theoretic planner, DARE-Heal, turns this belief into recovery actions by maximizing the success probability per unit cost, where the cost includes the blast radius of each action, and refines its belief after every unsuccessful attempt. On the public GAIA MicroSS dataset (382 labeled incidents, 28.7 million spans), the RCD Sock-Shop dataset, and a multimodal benchmark of 200 cascading-failure incidents, DARE attains top-1 accuracies of 0.817, 1.000, and 0.930 and matches or exceeds eight published baselines, including CIRCA and RCD, on every dataset. In closed-loop experiments, DARE-Heal recovers 86.7% of the GAIA incidents within five actions and reduces the mean time to recovery by 29.1% (GAIA) and 78.9% (synthetic) relative to restart-based remediation.References
[1] Soldani J, Brogi A. Anomaly detection and failure root cause analysis in (micro)service-based cloud applications: A survey. ACM Comput. Surv., 2022, 55(3): 1-39.
[2] Luo S, Xu H, Lu C, et al. Characterizing microservice dependency and performance: Alibaba trace analysis. In: Proc. ACM Symp. Cloud Computing (SoCC), 2021: 412-426.
[3] Gan Y, Zhang Y, Hu K, et al. Seer: Leveraging big data to navigate the complexity of performance debugging in cloud microservices. In: Proc. ASPLOS, 2019.
[4] Gan Y, Liang M, Dev S, et al. Sage: Practical and scalable ML-driven performance debugging in microservices. In: Proc. ASPLOS, 2021.
[5] Wang P, Xu J, Ma M, et al. CloudRanger: Root cause identification for cloud native systems. In: Proc. IEEE/ACM CCGRID, 2018: 492-502.
[6] Meng Y, Zhang S, Sun Y, et al. Localizing failure root causes in a microservice through causality inference. In: Proc. IEEE/ACM IWQoS, 2020: 1-10.
[7] Ma M, Xu J, Wang Y, et al. AutoMAP: Diagnose your microservice-based web applications automatically. In: Proc. Web Conf. (WWW), 2020: 246-258.
[8] Wu L, Tordsson J, Elmroth E, et al. MicroRCA: Root cause localization of performance issues in microservices. In: Proc. IEEE/IFIP NOMS, 2020.
[9] Ikram A, Chakraborty S, Mitra S, et al. Root cause analysis of failures in microservices through causal discovery. In: Proc. NeurIPS, 2022.
[10] Li M, Li Z, Yin K, et al. Causal inference-based root cause analysis for online service systems with intervention recognition. In: Proc. ACM SIGKDD, 2022.
[11] Spirtes P, Glymour C. An algorithm for fast recovery of sparse causal graphs. Soc. Sci. Comput. Rev., 1991, 9(1): 62-72.
[12] Sigelman B H, Barroso L A, Burrows M, et al. Dapper, a large-scale distributed systems tracing infrastructure. Google, Tech. Rep. dapper-2010-1, 2010.
[13] Li Z, Chen J, Jiao R, et al. Practical root cause localization for microservice systems via trace analysis. In: Proc. IEEE/ACM IWQoS, 2021.
[14] Yu G, Chen P, Chen H, et al. MicroRank: End-to-end latency issue localization with extended spectrum analysis in microservice environments. In: Proc. Web Conf. (WWW), 2021: 3087-3098.
[15] Liu D, He C, Peng X, et al. MicroHECL: High-efficient root cause localization in large-scale microservice systems. In: Proc. ICSE-SEIP, 2021: 338-347.
[16] Heckerman D, Breese J S, Rommelse K. Decision-theoretic troubleshooting. Commun. ACM, 1995, 38(3): 49-57.
[17] Shan H, Chen Y, Liu H, et al. ε-Diagnosis: Unsupervised and real-time diagnosis of small-window long-tail latency in large-scale microservice platforms. In: Proc. Web Conf. (WWW), 2019.
[18] Li Z, Zhao N, Li M, et al. Actionable and interpretable fault localization for recurring failures in online service systems. In: Proc. ESEC/FSE, 2022: 996-1008.
[19] Bahl P, Chandra R, Greenberg A, et al. Towards highly reliable enterprise network services via inference of multi-level dependencies. In: Proc. ACM SIGCOMM, 2007: 13-24.
[20] Kephart J O, Chess D M. The vision of autonomic computing. Computer, 2003, 36(1): 41-50.
[21] Candea G, Kawamoto S, Fujiki Y, et al. Microreboot—A technique for cheap recovery. In: Proc. USENIX OSDI, 2004.
[22] Weaveworks. Sock Shop: A microservices demo application. Available: https://github.com/microservices-demo/microservices-demo.
[23] CloudWise. GAIA: Generic AIOps Atlas dataset (MicroSS), version 1.10, 2022. Available: https://github.com/CloudWise-OpenSource/GAIA-DataSet.