DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion

Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, Luc Van Gool

科研成果: 书/报告/会议事项章节会议稿件同行评审

112 引用 (Scopus)

摘要

Multi-modality image fusion aims to combine different modalities to produce fused images that retain the complementary features of each modality, such as functional highlights and texture details. To leverage strong generative priors and address challenges such as unstable training and lack of interpretability for GAN-based generative methods, we propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM). The fusion task is formulated as a conditional generation problem under the DDPM sampling framework, which is further divided into an unconditional generation subproblem and a maximum likelihood subproblem. The latter is modeled in a hierarchical Bayesian manner with latent variables and inferred by the expectation-maximization (EM) algorithm. By integrating the inference solution into the diffusion sampling iteration, our method can generate high-quality fused images with natural image generative priors and cross-modality information from source images. Note that all we required is an unconditional pre-trained generative model, and no fine-tuning is needed. Our extensive experiments indicate that our approach yields promising fusion results in infrared-visible image fusion and medical image fusion. The code is available at https://github.com/Zhaozixiang1228/MMIF-DDFM.

源语言英语
主期刊名Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
出版商Institute of Electrical and Electronics Engineers Inc.
8048-8059
页数12
ISBN(电子版)9798350307184
DOI
出版状态已出版 - 2023
活动2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023 - Paris, 法国
期限: 2 10月 20236 10月 2023

出版系列

姓名Proceedings of the IEEE International Conference on Computer Vision
ISSN(印刷版)1550-5499

会议

会议2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
国家/地区法国
Paris
时期2/10/236/10/23

指纹

探究 'DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion' 的科研主题。它们共同构成独一无二的指纹。

引用此