AutoCut: Multi-Objective Offloading Service for Heterogeneous DNN in Internet of Things

Deep Neural Networks (DNNs) play a crucial role in the smart Internet of Things (IoT), with widespread applications in inference tasks like interactive games, intelligent driving, and augmented reality. Along with these promising applications, various task-offloading methods were proposed to improve the utilization of system resources, given that DNN model inference typically requires substantial computational power. However, existing offloading methods focus primarily on a specific model, and research addressing heterogeneous DNN models (with different structures and layers) remains limited in practical IoT environments. Directly integrating these methods would require frequent re-initialization to adapt to changes in the search space during the offloading of mixed heterogeneous DNN inference tasks, resulting in insufficient flexibility and the waste of computational resources. Thus, we propose AutoCut, a global heterogeneous model offloading service based on a customized multi-objective differential evolution algorithm, to find low-latency and energyefficient offloading partitions. AutoCut utilizes a group-layer granularity partitioning that avoids frequent changes in the search space when continuously offloading heterogeneous DNN inference tasks, thereby improving search efficiency. Experiments with six popular models show that AutoCut significantly improves inference performance regarding latency and energy efficiency.