MD³Net: Integrating Model-Driven and Data-Driven Approaches for Pansharpening

Pansharpening is a special image fusion task of reconstructing a high-resolution multispectral (HRMS) image by integrating a panchromatic (PAN) image of high spatial resolution and a low-resolution multispectral (LRMS) image. To handle such an ill-posed multimodal fusion task, in this article, we propose a novel pansharpening method, referred to as model-driven and data-driven network (MD3Net), which combines model-driven and data-driven approaches. The architecture design of MD3Net is inspired from the traditional model constructed based on domain knowledge and thus making its network topology explainable and its input-output predictable. To further explore the powerful learning ability of deep-learning-based approaches, we introduce the deep prior into the MD3Net as its implicit regularization, thus improving its data adaptability and representation capability. Comprehensive experiments conducted on both reduced and full resolution of several acknowledged datasets have qualitatively and quantitatively verified the superiority of our network compared with a benchmark consisting of several state-of-the-art approaches. The code can be downloaded from https://github.com/YinsongYan/M3DNet.