A Deep Joint Network for Multispectral Demosaicking Based on Pseudo-Panchromatic Images

Single-sensor multispectral cameras generally utilize a multispectral filter array (MSFA) to sample spatial-spectral information for a reduced capturing time. However, in this situation, each pixel in an MSFA image only contains information from a single channel. Thus, demosaicking is necessary to reconstruct a full-resolution multispectral image from the raw MSFA image. In this paper, we propose a novel end-to-end deep learning framework based on pseudo-panchromatic images (PPIs), which consists of two networks, namely the Deep PPI Generation Network (DPG-Net) and Deep Demosaic Network (DDM-Net). Among them, we first pre-train DPG-Net to reconstruct a full-resolution panchromatic image from the raw MSFA image and then jointly train both networks to recover a full-resolution multispectral image, followed by fine-tuning both networks with fewer restrictions. Experimental results reveal that the proposed method outperforms state-of-the-art traditional and deep learning demosaicking methods both qualitatively and quantitatively.