Two Time-Scale DRL for Service Caching and Task Offloading in Cross-Domain Marine Networks

With increasing computational demands and limited network resources in marine environments, efficient service caching and task offloading have become critical. In such environments, Autonomous Underwater Vehicles (AUVs) rely on Uncrewed Surface Vehicles (USVs) as relays, forming a cross-domain network comprising underwater acoustic and above-water RF links. However, the heterogeneity in bandwidth, latency, and bit error rates introduces challenges for reachability analysis and delay estimation. This paper addresses the joint optimization of caching, task offloading, and resource allocation in a cross-domain marine network composed of offshore base stations, USVs, and AUVs. To tackle the inherent heterogeneity in network links and decision timescales, we formulate the problem as a two-time-scale Hierarchical Markov Decision Process (H-MDP) and propose a Two Time-Scale Deep Reinforcement Learning (T2S-DRL) approach that integrates a hybrid policy network and a lightweight structure-aware action masking mechanism. The large time-scale agent optimizes caching decisions, while the short time-scale agent focuses on offloading and resource allocation. Extensive simulations show that our approach significantly reduces task execution delay and energy consumption, validating its effectiveness.