CAPA: A PREDICTION-DRIVEN AUTOSCALING FRAMEWORK FOR SLO-AWARE CLOUD-NATIVE MACHINE LEARNING INFERENCE
Keywords:
Cloud-native systems, Autoscaling, Machine-learning inference, Service-level objectives, Workload forecasting, Uncertainty calibrationAbstract
Cloud-native machine-learning inference must reconcile sub-second demand changes with non-negligible replica startup delay. Reactive horizontal scaling therefore tends to provision after a burst has already accumulated a queue, whereas aggressive prediction wastes resources when a workload is stable. This paper presents CAPA, a confidence-aware predictive autoscaler that couples a lightweight rolling ridge forecaster with an online one-sided residual-quantile bound and an SLO-debt feedback signal. The upper demand estimate covers startup delay; the debt signal lowers the utilization target when recent requests approach or exceed a latency objective; and an explicit queue-drain term converts backlog into temporary capacity. A reproducible evaluation combines 250 measured CPU inference latencies and seven measured process cold starts for MobileNetV3-Small with three fixed-seed, trace-calibrated stress profiles. Across eight paired runs, CAPA reduces the estimated SLO violation rate relative to a reactive HPA-style baseline by 17.83% on bursty traffic and 11.37% under regime shifts, while increasing mean replicas by 2.59% and 1.79%, respectively. The improvement is statistically significant in both nonstationary profiles. On a regular periodic workload, CAPA uses 3.82% fewer replicas but raises violations by 14.55%, exposing a clear boundary condition rather than hiding a negative result. The code, raw measurements, generated traces, and statistical outputs are packaged with the paper.References
[1] Dean J, Barroso L A. The tail at scale. Communications of the ACM, 2013, 56(2): 74-80. DOI: 10.1145/2408776.2408794.
[2] Verma A, Pedrosa L, Korupolu M R, et al. Large-scale cluster management at Google with Borg. In: Proceedings of EuroSys, 2015, 18: 1-17. DOI: 10.1145/2741948.2741964.
[3] Burns B, Grant B, Oppenheimer D, et al. Borg, Omega, and Kubernetes. ACM Queue, 2016, 14(1): 70-93. DOI: 10.1145/2890784.
[4] Crankshaw D, Wang X, Zhou G, et al. Clipper: A low-latency online prediction serving system. In: Proceedings of NSDI, 2017: 613-627.
[5] Olston C, Li F, Harmsen J, et al. TensorFlow-Serving: Flexible, high-performance ML serving. In: Workshop on ML Systems at NIPS, 2017.
[6] Gujarati A, Elnikety S, He Y, et al. Swayam: Distributed autoscaling to meet SLAs of machine learning inference services with resource efficiency. In: Proceedings of Middleware, 2017: 109-120. DOI: 10.1145/3135974.3135993.
[7] Zhang C, Yu M, Wang W, et al. MArk: Exploiting cloud services for cost-effective, SLO-aware machine learning inference serving. In: Proceedings of USENIX ATC, 2019: 1049-1062.
[8] Crankshaw D, Sela G E, Mo X, et al. InferLine: Latency-aware provisioning and scaling for prediction serving pipelines. In: Proceedings of ACM SoCC, 2020: 477-491. DOI: 10.1145/3419111.3421285.
[9] Gujarati A, Karimi R, Alzayat S, et al. Serving DNNs like Clockwork: Performance predictability from the bottom up. In: Proceedings of OSDI, 2020: 443-462.
[10] Romero F, Li Q, Yadwadkar N J, et al. INFaaS: Automated model-less inference serving. In: Proceedings of USENIX ATC, 2021: 397-411.
[11] Gunasekaran J R, Mishra C S, Thinakaran P, et al. Cocktail: A multidimensional optimization for model serving in cloud. In: Proceedings of NSDI, 2022: 1041-1057.
[12] Yu G I, Jeong J S, Kim G W, et al. Orca: A distributed serving system for transformer-based generative models. In: Proceedings of OSDI, 2022: 521-538.
[13] Li Z. AlpaServe: Statistical multiplexing with model parallelism for deep learning serving. In: Proceedings of OSDI, 2023: 663-679.
[14] Zhang H, Tang Y, Khandelwal A, et al. SHEPHERD: Serving DNNs in the wild. In: Proceedings of NSDI, 2023: 787-808.
[15] Kwon W, Li Z, Zhuang S, et al. Efficient memory management for large language model serving with PagedAttention. In: Proceedings of SOSP, 2023: 611-626. DOI: 10.1145/3600006.3613165.
[16] Lorido-Botran T, Miguel-Alonso J, Lozano J A. A review of auto-scaling techniques for elastic applications in cloud environments. Journal of Grid Computing, 2014, 12(4): 559-592. DOI: 10.1007/s10723-014-9314-7.
[17] Cortez E, Bonde A, Muzio A, et al. Resource Central: Understanding and predicting workloads for improved resource management in large cloud platforms. In: Proceedings of SOSP, 2017: 153-167. DOI: 10.1145/3132747.3132772.
[18] Rzadca K. Autopilot: Workload autoscaling at Google. In: Proceedings of EuroSys, 2020, 16: 1-16. DOI: 10.1145/3342195.3387524.
[19] Qiu H, Banerjee S S, Jha S, et al. FIRM: An intelligent fine-grained resource management framework for SLO-oriented microservices. In: Proceedings of OSDI, 2020: 805-825.
[20] Zhang Y, Hua W, Zhou Z, et al. Sinan: ML-based and QoS-aware resource management for cloud microservices. In: Proceedings of ASPLOS, 2021: 167-181. DOI: 10.1145/3445814.3446693.
[21] Lee Y, Scolari A, Chun B G, et al. PRETZEL: Opening the black box of machine learning prediction serving systems. In: Proceedings of OSDI, 2018: 611-626.
[22] Shen H, Chen L, Jin Y, et al. Nexus: A GPU cluster engine for accelerating DNN-based video analysis. In: Proceedings of SOSP, 2019: 322-337. DOI: 10.1145/3341301.3359658.
[23] Xiao W. Gandiva: Introspective cluster scheduling for deep learning. In: Proceedings of OSDI, 2018: 595-610.
[24] Delimitrou C, Kozyrakis C. Quasar: Resource-efficient and QoS-aware cluster management. In: Proceedings of ASPLOS, 2014: 127-144. DOI: 10.1145/2541940.2541941.
[25] Holt C C. Forecasting seasonals and trends by exponentially weighted moving averages. International Journal of Forecasting, 2004, 20(1): 5-10. DOI: 10.1016/j.ijforecast.2003.09.015.
[26] Hyndman R J, Athanasopoulos G. Forecasting: Principles and Practice. 3rd ed. Melbourne, Australia: OTexts, 2021.
[27] Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. 2nd ed. New York, NY, USA: Springer, 2009. DOI: 10.1007/978-0-387-84858-7.
[28] Gibbs I, Candès E. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 2021, 34: 1660-1672.
[29] Shahrad M. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In: Proceedings of USENIX ATC, 2020: 205-218.