Scheduling framework and algorithms for large-scale divisible load processing with multi-dimensional QoS constraints

In this paper, we propose a scheduling framework and related algorithms for processing large-scale, computation-intensive divisible loads. The framework is organized into a two-level tree architecture. Based on this framework, admission test and load partitioning and distribution algorithms are designed to ensure that the multi-dimensional QoS requirements, i.e. processing deadline, security and reliability, of admitted loads can be satisfied. We take a novel approach to incorporate resource reservation and time step-size adaptive scheduling schemes into the optimal solution that makes computation nodes finish computing at the same time instant. We provide an implementation of the framework atop of a distributed communication middleware extended with QoS-aware resource management facilities. Prototype implementation and preliminary experimental results demonstrate the engineering feasibility and good performance of the proposed framework and algorithms.