Shadow Detection and Removal Based on Multi-task Generative Adversarial Networks

The existence of shadows is difficult to avoid in images. Also, it will affect object recognition and image understanding. But on the other hand, shadow can provide information about the light source and object shape. Therefore, accurate shadow detection and removal can contribute to many computer vision tasks. However, even the same object, its shadow will vary greatly under different lighting conditions. Thus it is quite challenging to detect and remove shadows from images. Recent research always treated these two tasks independently, but they are closely related to each other actually. Therefore, we propose a multi-task adversarial generative networks (mtGAN) that can detect and remove shadows simultaneously. In order to enhance shadow detection and shadow removal mutually, a cross-stitch unit is proposed to learn the optimal ways to fuse and constrain features between multi-tasks. Also, the combination weight of multi-task loss functions are learned according to the uncertainty distribution of each task, which is not set empirically as usual. Based on these multi-task learning strategies, the proposed mtGAN can jointly achieve shadow detection and removal tasks better than existing methods. In experiments, the effectiveness of the proposed mtGAN is shown.