Hierarchical spatio-spectral fusion for hyperspectral image super resolution via sparse representation and pre-trained deep model

Fusing a hyperspectral image (HSI) with a high resolution multispectral image (MSI) has been a highly attractive and effective approach for improving the spatial resolution of HSIs. However, most existing spatio-spectral fusion methods directly accomplish such a fusion process on the desired image scale. This limits the accuracy of the resulting model, especially when large magnification factors are involved. In this paper, a new dual pyramid model is proposed to implement hyperspectral image super resolution, based on a novel hierarchical spatial and spectral fusion method, in an effort to estimate the high resolution image progressively. In particular, an input low resolution HSI is upscaled progressively using a pre-trained deep Laplacian pyramid network, while the corresponding high resolution MSI is down sampled with multiple pyramid layers, forming a dual pyramid model. At each pyramid layer, the HSI and the MSI within the current layer are fused via sparse representation. Systematic qualitative and quantitative evaluations against benchmark datasets demonstrate that this new approach outperforms a number of spatio-spectral fusion based super resolution techniques, achieving outstanding performance over large scale factors.