Preference-Aware Task Routing for Edge-Cloud Hierarchical Large Language Model Inference

The hierarchical edge-cloud architecture is regarded as a promising solution for deploying large models at the edge, as it effectively alleviates resource and latency constraints on edge devices. However, it often employs static routing that overlooks the heterogeneity of task requests. Therefore, we propose Lyra, a preference-aware online routing framework to bridge this gap. Lyra synergizes edge SLMs and cloud LLMs, employing Lyapunov drift-plus-penalty theory to dynamically optimize the inference quality-latency trade-off without requiring future traffic knowledge. Experimental results demonstrate that Lyra can effectively adapt to dynamic workloads, thereby enhancing system throughput to meet diverse user demands.