PACO: PREDICTIVE AUTO-CONFIGURATION FOR SLO-CONSTRAINED LARGE LANGUAGE MODEL INFERENCE SERVING
Keywords:
Large language models, Inference serving, Auto-configuration, Autoscaling, Service-level objectives, Performance predictionAbstract
Serving large language models (LLMs) is expensive, and production workloads fluctuate by an order of magnitude within hours while requests must meet tight latency service-level objectives (SLOs) on time-to-first-token (TTFT) and time-per-output-token (TPOT). Operators today either provision statically for peak load, wasting most of the fleet off-peak, or rely on reactive autoscalers that reconfigure only after violations are observed. This paper presents PACO, a predictive auto-configuration framework that periodically selects the cheapest serving configuration expected to satisfy tail-latency SLOs. PACO combines (i) a lightweight workload forecaster, (ii) a gradient-boosted performance model that predicts windowed P95 TTFT/TPOT from workload features and a candidate configuration, learned from offline profiling, (iii) an SLO-constrained minimum-cost search over the joint space of replica count and chunked-prefill token budget, and (iv) a feedback guard that bounds the impact of prediction errors. We evaluate PACO with an iteration-level cluster simulator driven by production LLM inference traces released by Microsoft Azure. On a diurnal conversation workload, PACO reduces SLO-violating windows from 27.6% to 3.4% at 2% lower cost than a reactive autoscaler, cuts cost by 60.5% relative to static peak provisioning, and stays within 27% of a clairvoyant oracle’s cost. Ablations show the forecaster halves violations and the joint tuning of the token budget is essential for long-context workloads.References
[1] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[2] Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020, 159: 1877-1901.
[3] Patel P, Choukse E, Zhang C, et al. Splitwise: Efficient generative LLM inference using phase splitting. Proc. 51st ACM/IEEE Int. Symposium on Computer Architecture (ISCA), 2024: 118-132.
[4] Stojkovic J, Zhang C, Goiri Í, et al. DynamoLLM: Designing LLM inference clusters for performance and energy efficiency. Proc. IEEE Int. Symposium on High-Performance Computer Architecture (HPCA), 2025: 1348-1362.
[5] Yu G I, Jeong J S, Kim G W, et al. Orca: A distributed serving system for transformer-based generative models. Proc. 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022: 521-538.
[6] Agrawal A, Kedia N, Panwar A, et al. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. Proc. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024: 117-134.
[7] Alipourfard O, Liu H H, Chen J, et al. CherryPick: Adaptively unearthing the best cloud configurations for big data analytics. Proc. 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2017.
[8] Venkataraman S, Yang Z, Franklin M, et al. Ernest: Efficient performance prediction for large-scale advanced analytics. Proc. 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2016.
[9] Kwon W, Li Z, Zhuang S, et al. Efficient memory management for large language model serving with PagedAttention. Proc. 29th ACM Symposium on Operating Systems Principles (SOSP), 2023: 611-626.
[10] Holt C C. Forecasting seasonals and trends by exponentially weighted moving averages. International Journal of Forecasting, 2004, 20(1): 5-10.
[11] Friedman J H. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 2001, 29: 1189-1232.
[12] Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011, 12: 2825-2830.
[13] Agrawal A, Kedia N, Mohan J, et al. Vidur: A large-scale simulation framework for LLM inference. Proceedings of Machine Learning and Systems (MLSys), 2024, 6: 351-366.
[14] Touvron H, Lavril T, Izacard G, et al. LLaMA: Open and efficient foundation language models. LLaMA: Open and Efficient Foundation Language Models, 2023.
[15] Zhong Y, Liu S, Chen J, et al. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. Proc. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024.
[16] Zhang C, Yu M, Wang W, et al. MArk: Exploiting cloud services for cost-effective, SLO-aware machine learning inference serving. Proc. USENIX Annual Technical Conference (ATC), 2019: 1049-1062.
[17] Li Z, Zheng L, Zhong Y, et al. AlpaServe: Statistical multiplexing with model parallelism for deep learning serving. Proc. 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2023.
[18] Crankshaw D, Wang X, Zhou G, et al. Clipper: A low-latency online prediction serving system. Proc. 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2017: 613-627.
[19] Gujarati A, Karimi R, Alzayat S, et al. Serving DNNs like clockwork: Performance predictability from the bottom up. Proc. 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2020.
[20] Romero F, Li Q, Yadwadkar N J, et al. INFaaS: Automated model-less inference serving. Proc. USENIX Annual Technical Conference (ATC), 2021.