Entropy-Regularized Uncertainty-Aware Multi-Objective Optimization for Serverless Data Pipelines using Hybrid Reinforcement Learning

Serverless data pipelines have a very dynamic, non-stationary workload, bursty request patterns, cold-start delays, and resource variability are the order of the day, and traditional one-dimensional or statical optimization strategies do not apply. The currently used methods mainly aim at minimizing either cost or latency without paying much attention to predictive uncertainty and its effects on the stability of the decision making which results in suboptimal and unstable scheduling in real world situations. In a bid to fill this gap, this paper introduces a new multi-objective framework called ERUO (Entropy-Regularized Uncertainty-Aware Optimization) that concurrently optimizes the latency, cost, probability of cold-start, and uncertainty in a single reinforcement learning framework. ERUO proposes to use an entropy-based regularization on predictive uncertainty distributions to allow the system to discourage unstable regimes of decisions and enhance resistance to workload variability. The structure combines the use of quantile-based uncertainty modelling, gradient-boosted prediction, LSTM-based workload forecasting and PPO-based policy optimizer. As shown by experimental assessment on real-world cloud traces (Alibaba and Google cluster datasets), ERUO can make latency, cost, and cold-start instances reductions of up to 23.6, 18.4 and 27.1 over baseline approaches, as well as converge faster and can make improved decisions. The importance of this work is that uncertainty entropy has been formally coupled with the control optimization, which offers an efficient and scalable solution to managing next-generation serverless data pipelines.