U-Shape Spectral-Transformer for Robust Fusion Based Hyperspectral Super-Resolution

Fusing a high-spatial-resolution (HR) multi-spectral image (MSI) with a low-spatial resolution (LR) hyperspectral image (HSI) provides an effective way for HSI super-resolution (SR). Although recent deep neural network-based methods have shown pleasing fusion performance, most of them assume both input images for fusion to be clean without any noise corruption. When random noise exists in real applications, their performance drops greatly. To mitigate this problem, we present a U-shape spectral transformer for robust fusion-based HSI SR, which mainly contributes in the following three aspects. 1) A two-stage network is established to end-to-end denoise both input images and fuse them for SR. 2) A U-shape spectral transformer is constructed to simultaneously exploit the multi-scale spatial information and the long-range correlation in spectral domain, which enables sufficiently fusing the supplementary spatial-spectral information in both input images for accurate HSI SR. 3) A mutual information maximization based loss is composed with the conventional reconstruction loss to more accurately supervise the training process, thus further enhance the performance. Experimental results on two datasets demonstrate the efficacy of the proposed method in terms of HSI SR under different levels of noise corruption.