PRISM: PREDICTION-GUIDED RISK-AWARE DEEP REINFORCEMENT LEARNING FOR ENERGY-AWARE DYNAMIC RESOURCE SCHEDULING IN DISTRIBUTED CLOUD DATA CENTERS
Keywords:
Cloud computing, Energy efficiency, Virtual machine consolidation, Deep reinforcement learning, Quantile forecasting, Geo-distributed data centersAbstract
Geo-distributed cloud data centers spend a large share of their electricity on lightly loaded servers, while aggressive consolidation causes overloads, service-level agreement (SLA) violations and migration storms. This paper presents PRISM, a prediction-guided, risk-aware deep reinforcement learning (DRL) framework for energy-aware dynamic resource scheduling in distributed cloud data centers. PRISM couples three components. A lightweight multi-horizon quantile forecaster (MQF) predicts the mean, median and 90th percentile of the CPU demand of every virtual machine (VM), including newly arrived VMs without history. A risk-aware host model aggregates these quantiles into a chance-constrained capacity test that drives overload detection and placement feasibility at a tunable risk level. A residual Q-network, whose action value is an analytic energy-cost prior plus a learned correction, scores a compact set of candidate hosts for arrival dispatch, overload relief and consolidation across all data centers and is trained with Double DQN. Trace-driven experiments with 11,746 real PlanetLab VM traces on three heterogeneous data centers with SPECpower server models, time-of-use tariffs and different PUE values show that, on unseen test days and relative to the strongest baseline, PRISM reduces energy consumption by 12.5%, electricity cost by 5.2%, carbon emissions by 14.1%, the SLA violation time per active host by 44.6%, the energy–SLA-violation product by 90.1% and the number of migrations by 84.6%, while requiring 16.3 ms of computation per five-minute scheduling interval.References
[1] Masanet E, Shehabi A, Lei N, et al. Recalibrating global data center energy-use estimates. Science, 2020, 367(6481): 984-986.
[2] Barroso L A, Hölzle U. The case for energy-proportional computing. Computer, 2007, 40(12): 33-37.
[3] Clark C, Fraser K, Hand S, et al. Live migration of virtual machines. In: Proceedings of the 2nd USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2005: 273-286.
[4] Beloglazov A, Buyya R. Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in cloud data centers. Concurrency and Computation: Practice and Experience, 2012, 24(13): 1397-1420.
[5] Beloglazov A, Abawajy J, Buyya R. Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing. Future Generation Computer Systems, 2012, 28(5): 755-768.
[6] Verma A, Ahuja P, Neogi A. pMapper: Power and migration cost aware application placement in virtualized systems. In: Proceedings of ACM/IFIP/USENIX Middleware, LNCS 5346, 2008: 243-264.
[7] Qureshi A, Weber R, Balakrishnan H, et al. Cutting the electric bill for internet-scale systems. In: Proceedings of ACM SIGCOMM, 2009: 123-134.
[8] Rao L, Liu X, Xie L, et al. Minimizing electricity cost: Optimization of distributed internet data centers in a multi-electricity-market environment. In: Proceedings of IEEE INFOCOM, 2010: 1145-1153.
[9] Liu Z, Lin M, Wierman A, et al. Greening geographical load balancing. In: Proceedings of ACM SIGMETRICS, 2011: 233-244.
[10] Farahnakian F, Pahikkala T, Liljeberg P, et al. Energy-aware VM consolidation in cloud data centers using utilization prediction model. IEEE Transactions on Cloud Computing, 2019, 7(2): 524-536.
[11] Calheiros R N, Masoumi E, Ranjan R, et al. Workload prediction using ARIMA model and its impact on cloud applications’ QoS. IEEE Transactions on Cloud Computing, 2015, 3(4): 449-458.
[12] Kumar J, Singh A K. Workload prediction in cloud using artificial neural network and adaptive differential evolution. Future Generation Computer Systems, 2018, 81: 41-52.
[13] Mao H, Alizadeh M, Menache I, et al. Resource management with deep reinforcement learning. In: Proceedings of the 15th ACM Workshop on Hot Topics in Networks (HotNets), 2016: 50-56.
[14] Liu N, Li Z, Xu J, et al. A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning. In: Proceedings of the IEEE 37th International Conference on Distributed Computing Systems (ICDCS), 2017: 372-382.
[15] Cheng M, Li J, Nazarian S. DRL-cloud: Deep reinforcement learning-based resource provisioning and task scheduling for cloud service providers. In: Proceedings of the 23rd Asia and South Pacific Design Automation Conference (ASP-DAC), 2018: 129-134.
[16] Wang B, Liu F, Lin W. Energy-efficient VM scheduling based on deep reinforcement learning. Future Generation Computer Systems, 2021, 125: 616-628.
[17] Zeng J, Ding D, Kang K, et al. Adaptive DRL-based virtual machine consolidation in energy-efficient cloud data center. IEEE Transactions on Parallel and Distributed Systems, 2022, 33(11): 2991-3002.
[18] Shaw R, Howley E, Barrett E. Applying reinforcement learning towards automating energy efficient virtual machine consolidation in cloud data centers. Information Systems, 2022, 107: Art. 101722.
[19] Calheiros R N, Ranjan R, Beloglazov A, et al. CloudSim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Software: Practice and Experience, 2011, 41(1): 23-50.
[20] Park K, Pai V S. CoMon: A mostly-scalable monitoring system for PlanetLab. ACM SIGOPS Operating Systems Review, 2006, 40(1): 65-74.
[21] Cortez E, Bonde A, Muzio A, et al. Resource Central: Understanding and predicting workloads for improved resource management in large cloud platforms. In: Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP), 2017: 153-167.
[22] Khan T, Tian W, Ilager S, et al. Workload forecasting and energy state estimation in cloud data centres: ML-centric approach. Future Generation Computer Systems, 2022, 128: 320-332.
[23] Koenker R, Bassett G. Regression quantiles. Econometrica, 1978, 46(1): 33-50.
[24] Farahnakian F, Liljeberg P, Plosila J. Energy-efficient virtual machines consolidation in cloud data centers using reinforcement learning. In: Proceedings of the 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP), 2014: 500-507.
[25] Hester T, Vecerik M, Pietquin O, et al. Deep Q-learning from demonstrations. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI), 2018: 3223-3230.
[26] Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning. Nature, 2015, 518(7540): 529-533.
[27] van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double Q-learning. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI), 2016: 2094-2100.