DRL Optimizes Edge-Cloud Networks with Multi-Timescale Latent Actions

Hierarchical edge-cloud computing (HECC) systems often suffer from load imbalance, leading to high latency and inefficient resource use, especially with dynamic task arrivals and heterogeneous resources. The challenge lies in jointly optimizing service placement, computational delegation, and power control (JSCP), which is a complex, mixed-integer, and NP-hard problem due to tightly coupled discrete and continuous variables. To address this, the research proposes decomposing the JSCP problem into long-term system configuration and short-term resource allocation subproblems, leveraging the inherent differences in decision dynamics. Based on this decomposition, a novel two-timescale multi-layer deep reinforcement learning framework with a latent action space (2T-MDRL-LA) is introduced. This framework jointly optimizes various aspects, including service placement, user association, computational delegation, task offloading, and user transmit power. A variational autoencoder is used to create a latent action representation, effectively compressing the high-dimensional combinatorial action space. Simulations demonstrate that 2T-MDRL-LA adapts well to dynamic network conditions, achieving near-optimal performance, up to a 20.8% reduction in average end-to-end latency, and a 13% improvement in resource utilization compared to schemes without computational delegation, while converging 50% faster than conventional PPO.