Hyperspectral–multispectral image fusion enhancement based on deep learning

Hyperspectral image (HSI) contains rich spatial and spectral information, which is beneficial for identifying different materials. HSI has been applied in many fields, including land-cover classification and target detection. However, due to limited photonic energy, there are trade-offs between spatial resolution, bandwidth, swath width, and signal-noise-ratio. One outcome is that the spatial resolution of HSI is often moderate, which may lead to the spectral mixture of different materials in each pixel. In addition, for some Earth Observation applications such as urban mapping and fine mineral exploration, it is required to have a high spatial resolution image. Compared with HSI, multispectral imagery (MSI) often has wider spectral bandwidth and higher spatial resolution. HSI-MSI fusion for HSI resolution enhancement aims at fusing MSI with HSI, and generating HSI of higher resolution is an important technology. Having enormous capacity in feature extraction and representing mapping function, deep learning has shown great potential in HSI resolution enhancement. Two issues exist when we apply deep learning to HSI-MSI fusion: (1) how to jointly extract spectral–spatial deep features from HSI and MSI, (2) and how to fuse the extracted features and then generate high-resolution HSI. In this chapter, we first review the recent advances in HSI resolution enhancement technologies, particularly in HSI-MSI fusion technology, and then present our solution for HSI-MSI fusion based on a two-branch convolutional neural network.