XSLA: AN SLA-AWARE HETEROGENEOUS XPU SCHEDULING FRAMEWORK FOR EDGE AI INFERENCE
Keywords:
Edge computing, Inference serving, Heterogeneous accelerators, Service-level agreements, Real-time scheduling, Energy efficiencyAbstract
Edge AI deployments increasingly aggregate several classes of accelerator — embedded GPUs, neural processing units, FPGAs, dataflow ASICs and ARM CPUs — into a single "XPU" pool, while the requests they serve arrive with heterogeneous service-level agreements (SLAs). Using published measurements of 15,625 architectures on six real accelerator platforms, we show that one fixed model varies in latency by up to 51.9x across XPU classes, that 30.1% of architectures cannot execute on the Edge TPU at all, and that no accelerator is simultaneously best for latency, energy and accuracy. Which accelerator to use and which model variant to run are therefore one decision, not two. We present XSLA, which co-selects variant and accelerator per request. XSLA (i) shrinks the candidate set with a per-accelerator Pareto test we prove is lossless, (ii) chooses among deadline-feasible candidates by a drift-plus-penalty rule whose quality weight adapts to measured backlog, and (iii) estimates backlog per SLA class so latency-critical traffic is never queued behind best-effort work. In a measurement-driven simulator with 6 XPU classes, 3 tasks and 24 real variants, XSLA holds SLA violation at 0.0% up to 9,000 req/s, where an accuracy-greedy model-less policy misses 47.0% of deadlines and fixed-model EDF misses 99.0%. It raises deadline-constrained accuracy yield by 14.4–81.4% over the strongest baseline while using 37.4–58.4% less energy per request, lifting pool utilisation from 0.26 to 0.99.References
[1] Zhou Z. Edge intelligence: Paving the last mile of artificial intelligence with edge computing. Proceedings of the IEEE, 2019, 107(8): 1738-1762.
[2] Kang Y. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. In: Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2017: 615-629.
[3] Romero F. INFaaS: Automated model-less inference serving. In: Proceedings of USENIX ATC, 2021: 397-411.
[4] Narayanan D. Heterogeneity-aware cluster scheduling policies for deep learning workloads. In: Proceedings of USENIX OSDI, 2020: 481-498.
[5] Li C. HW-NAS-Bench: Hardware-aware neural architecture search benchmark. In: Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[6] Dong X, Yang Y. NAS-Bench-201: Extending the scope of reproducible neural architecture search. In: Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[7] Crankshaw D. Clipper: A low-latency online prediction serving system. In: Proceedings of USENIX NSDI, 2017: 613-627.
[8] Gujarati A. Serving DNNs like clockwork: Performance predictability from the bottom up. In: Proceedings of USENIX OSDI, 2020: 443-462.
[9] Shen H. Nexus: A GPU cluster engine for accelerating DNN-based video analysis. In: Proceedings of ACM SOSP, 2019: 322-337.
[10] Zhang C. MArk: Exploiting cloud services for cost-effective, SLO-aware machine learning inference serving. In: Proceedings of USENIX ATC, 2019: 1049-1062.
[11] Zhang H. SHEPHERD: Serving DNNs in the wild. In: Proceedings of USENIX NSDI, 2023: 787-808.
[12] Matsubara Y, Levorato M, Restuccia F. Split computing and early exiting for deep learning applications: Survey and research challenges. ACM Computing Surveys, 2023, 55(5): 1-30.
[13] Ran X. DeepDecision: A mobile deep learning framework for edge video analytics. In: Proceedings of IEEE INFOCOM, 2018: 1421-1429.
[14] Nigade V. Jellyfish: Timely inference serving for dynamic edge networks. In: Proceedings of IEEE RTSS, 2022: 277-290.
[15] Jeong J S. Band: Coordinated multi-DNN inference on heterogeneous mobile processors. In: Proceedings of ACM MobiSys, 2022: 235-247.
[16] Jia F. CoDL: Efficient CPU-GPU co-execution for deep learning inference on mobile devices. In: Proceedings of ACM MobiSys, 2022: 209-221.
[17] Ling N. RT-mDL: Supporting real-time mixed deep learning tasks on edge platforms. In: Proceedings of ACM SenSys, 2021: 1-14.
[18] Kim S. DREAM: A dynamic scheduler for dynamic real-time multi-model ML workloads. In: Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2023: 73-86.
[19] Fan H. Sparse-DySta: Sparsity-aware dynamic and static scheduling for sparse multi-DNN workloads. In: Proceedings of IEEE/ACM MICRO, 2023: 353-366.
[20] Weng Q. MLaaS in the wild: Workload analysis and scheduling in large-scale heterogeneous GPU clusters. In: Proceedings of USENIX NSDI, 2022: 945-960.
[21] Liu C L, Layland J W. Scheduling algorithms for multiprogramming in a hard-real-time environment. Journal of the ACM, 1973, 20(1): 46-61.
[22] Topcuoglu H, Hariri S, Wu M-Y. Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Transactions on Parallel and Distributed Systems, 2002, 13(3): 260-274.
[23] Neely M J. Stochastic Network Optimization with Application to Communication and Queueing Systems. Morgan & Claypool, 2010.
[24] Chen Y-H. Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE Journal of Solid-State Circuits, 2017, 52(1): 127-138.
[25] Zhang L L. nn-Meter: Towards accurate latency prediction of deep-learning model inference on diverse edge devices. In: Proceedings of ACM MobiSys, 2021: 81-93.
[26] Reddi V J. MLPerf inference benchmark. In: Proceedings of ACM/IEEE ISCA, 2020: 446-459.
[27] Sandler M. MobileNetV2: Inverted residuals and linear bottlenecks. In: Proceedings of IEEE CVPR, 2018: 4510-4520.
[28] Tan M, Le Q V. EfficientNet: Rethinking model scaling for convolutional neural networks. In: Proceedings of the International Conference on Machine Learning (ICML), 2019: 6105-6114.
[29] Cai H. Once-for-All: Train one network and specialize it for efficient deployment. In: Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[30] Shahrad M. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In: Proceedings of USENIX ATC, 2020: 205-218.