ADAHEAL: AN ADAPTIVE FRAMEWORK FOR FAULT DETECTION AND AUTOMATED RECOVERY IN LARGE-SCALE CLOUD-NATIVE MICROSERVICES

Authors

  • Xiang He (Corresponding Author) University of Illinois Urbana-Champaign, Urbana 61801, IL, USA.
  • Lin Zhou University of Florida, Gainesville 32611, FL, USA.

Keywords:

Microservices, AIOps, Anomaly detection, Root cause analysis, Self-healing systems, Thompson sampling

Abstract

Cloud-native microservice systems fail in ways that are frequent and hard to contain: their telemetry drifts with every deployment and traffic surge, faults propagate along call graphs and surface far from their origin, and the effectiveness of remediation actions changes as the system evolves. This paper presents AdaHeal, an adaptive closed-loop framework that unifies fault detection, root-cause localization, fault-type identification, and automated recovery. AdaHeal detects anomalies with a dual-axis masked reconstruction network (DAMR) that reconstructs every telemetry cell only from its temporal and cross-metric context and keeps alarms calibrated under benign drift through score-gated online recalibration. It localizes faults with calibrated propagation-aware ranking (CPR), which scores each service by the part of its deviation that its dependencies cannot explain under the flow-conservation rules of synchronous RPC and assigns latent evidence to stateful backends without request-level metrics. Recovery is planned by a risk-aware discounted Thompson sampling engine (RA-DTS) that selects (service, action) pairs under an attempt budget and revises its beliefs after every failed attempt. On the public SMD and PSM datasets, DAMR reaches a point-adjusted F1 of 0.893 on SMD (best of nine detectors) and the highest AUPR among the deep sequence models on PSM (0.440). On microservice benchmarks built from the Online Boutique, Sock Shop, and a 100-service Alibaba-style topology, CPR raises top-1 localization accuracy to 0.964, 0.903, and 0.922 (best baseline: 0.858, 0.694, 0.892) at 60 ms per incident for 400 services. In closed-loop experiments with shifting action effectiveness, AdaHeal shortens the mean time to recovery by 35.7% relative to a static detection–localization–runbook pipeline.

References

[1] Luo S, Xu H, Lu C, et al. Characterizing microservice dependency and performance: Alibaba trace analysis. In: Proc. ACM Symp. Cloud Computing (SoCC), 2021: 412-426.

[2] Gan Y, Zhang Y, Cheng D, et al. An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems. In: Proc. 24th Int. Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS). Providence, RI, USA, 2019.

[3] Zhou X, Peng X, Xie T, et al. Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study. IEEE Trans. Softw. Eng., 2021, 47(2): 243-260.

[4] Soldani J, Brogi A. Anomaly detection and failure root cause analysis in (micro)service-based cloud applications: A survey. ACM Comput. Surv., 2022, 55(3): 1-39.

[5] Pham L, Ha H, Zhang H. Root cause analysis for microservices based on causal inference: How far are we? In: Proc. 39th IEEE/ACM Int. Conf. Automated Software Engineering (ASE), 2024: 706-715.

[6] Wu H, Hu T, Liu Y, et al. TimesNet: Temporal 2D-variation modeling for general time series analysis. In: Proc. Int. Conf. Learning Representations (ICLR), 2023.

[7] Liu Y, Hu T, Zhang H, et al. iTransformer: Inverted transformers are effective for time series forecasting. In: Proc. Int. Conf. Learning Representations (ICLR), 2024.

[8] Wang S, Wu H, Shi X, et al. TimeMixer: Decomposable multiscale mixing for time series forecasting. In: Proc. Int. Conf. Learning Representations (ICLR), 2024.

[9] Pham L, Ha H, Zhang H. BARO: Robust root cause analysis for microservices via multivariate Bayesian online change point detection. Proc. ACM Softw. Eng., 2024, 1(FSE): 2214-2237.

[10] Li M, Li Z, Yin K, et al. Causal inference-based root cause analysis for online service systems with intervention recognition. In: Proc. 28th ACM SIGKDD Conf. Knowledge Discovery and Data Mining (KDD), 2022.

[11] Wu L, Tordsson J, Elmroth E, et al. MicroRCA: Root cause localization of performance issues in microservices. In: Proc. IEEE/IFIP Network Operations and Management Symp. (NOMS), 2020.

[12] Lee C, Yang T, Chen Z, et al. Eadro: An end-to-end troubleshooting framework for microservices on multi-source data. In: Proc. 45th IEEE/ACM Int. Conf. Software Engineering (ICSE), 2023.

[13] Sun Y, Shi B, Mao M, et al. ART: A unified unsupervised framework for incident management in microservice systems. In: Proc. 39th IEEE/ACM Int. Conf. Automated Software Engineering (ASE), 2024.

[14] Ahmed T, Ghosh S, Bansal C, et al. Recommending root-cause and mitigation steps for cloud incidents using large language models. In: Proc. 45th IEEE/ACM Int. Conf. Software Engineering (ICSE), 2023.

[15] Kephart J O, Chess D M. The vision of autonomic computing. Computer, 2003, 36(1): 41-50.

[16] Su Y, Zhao Y, Niu C, et al. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In: Proc. 25th ACM SIGKDD Int. Conf. Knowledge Discovery & Data Mining (KDD), 2019: 2828-2837.

[17] Audibert J, Michiardi P, Guyard F, et al. USAD: Unsupervised anomaly detection on multivariate time series. In: Proc. 26th ACM SIGKDD Int. Conf. Knowledge Discovery & Data Mining (KDD), 2020: 3395-3404.

[18] Abdulaal A, Liu Z, Lancewicki T. Practical approach to asynchronous multivariate time series anomaly detection and localization. In: Proc. 27th ACM SIGKDD Conf. Knowledge Discovery & Data Mining (KDD), 2021: 2485-2494.

[19] Liu F T, Ting K M, Zhou Z H. Isolation forest. In: Proc. 8th IEEE Int. Conf. Data Mining (ICDM), 2008: 413-422.

[20] Nie Y, Nguyen N H, Sinthong P, et al. A time series is worth 64 words: Long-term forecasting with transformers. In: Proc. Int. Conf. Learning Representations (ICLR), 2023.

[21] Zeng A, Chen M, Zhang L, et al. Are transformers effective for time series forecasting? In: Proc. AAAI Conf. Artificial Intelligence, 2023.

[22] Zhou Q, Pei C, Sun F, et al. KAN-AD: Time series anomaly detection with Kolmogorov–Arnold networks. arXiv:2411.00278, 2024.

[23] Kim S, Choi K, Choi H S, et al. Towards a rigorous evaluation of time-series anomaly detection. In: Proc. AAAI Conf. Artificial Intelligence, 2022, 36(7): 7194-7201.

[24] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners. In: Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2022: 16000-16009.

[25] Shan H, Chen Y, Liu H, et al. ε-Diagnosis: Unsupervised and real-time diagnosis of small-window long-tail latency in large-scale microservice platforms. In: Proc. World Wide Web Conf. (WWW), 2019.

[26] Yu G, Chen P, Li Y, et al. Nezha: Interpretable fine-grained root causes analysis for microservices on multi-modal observability data. In: Proc. 31st ACM Joint European Software Engineering Conf. and Symp. Foundations of Software Engineering (ESEC/FSE), 2023: 553-565.

[27] Chen Y, Xie H, Ma M, et al. Automatic root cause analysis via large language models for cloud incidents. In: Proc. European Conf. Computer Systems (EuroSys). Athens, Greece, 2024.

[28] Qiu H, Banerjee S S, Jha S, et al. FIRM: An intelligent fine-grained resource management framework for SLO-oriented microservices. In: Proc. 14th USENIX Symp. Operating Systems Design and Implementation (OSDI), 2020: 805-825.

[29] Li L, Chu W, Langford J, et al. A contextual-bandit approach to personalized news article recommendation. In: Proc. 19th Int. Conf. World Wide Web (WWW), 2010: 661-670.

[30] Agrawal S, Goyal N. Thompson sampling for contextual bandits with linear payoffs. In: Proc. 30th Int. Conf. Machine Learning (ICML), PMLR, 2013, 28(3): 127-135.

[31] Garivier A, Moulines E. On upper-confidence bound policies for switching bandit problems. In: Proc. 22nd Int. Conf. Algorithmic Learning Theory (ALT), LNCS 6925, 2011: 174-188.

Downloads

Published

2024-12-31

Issue

Section

Research Article

DOI:

How to Cite

Xiang He, Lin Zhou. Adaheal: An Adaptive Framework For Fault Detection And Automated Recovery In Large-Scale Cloud-Native Microservices. AI and Data Science Journal. 2024, 1(1): 132-141. DOI: https://doi.org/10.61784/adsj4045.