Image harmonization in complex degradation scenes

Image harmonization aims to synthesize photo-realistic images given composite images constructed by combining background and foreground source images. Most existing methods assume that the only difference between the two sources is illumination. However, in degradation scenarios, the source images inevitably exhibit complex distribution inconsistencies, including noise and resolution variations. Previous works have struggled to handle such inconsistencies despite their impressive performance. In this paper, we propose an image Harmonization Conditional Diffusion Model (H-CDM) to address the challenge in complex degradation scenes. Our model eliminates the intricate distribution inconsistencies by iteratively transforming the distributions from one source to another in a tractable manner. To achieve this, we first introduce a degradation-aware network with a degradation map estimation module (DEM). Given the degraded composite image, the degradation map can be generated by this estimation module and serves as a prior in our model. Further, we design a multi-scale masked deep supervision (MDS) loss to enhance the generation of photo-realistic harmonized images with fewer sampling steps. To verify the effectiveness of our H-CDM, we compare our H-CDM with other comparison methods on the newly constructed D-iHarmony4 dataset, where we achieve promising performance across the three metrics, mean squared error (MSE), peak signal-to-noise ratio (PSNR), and foreground MSE (fMSE). To evaluate the robustness of our H-CDM, we also conduct a mean opinion score (MOS) test. The results demonstrate that our H-CDM produces more favorable and photo-realistic images. Our H-CDM can be applied in image editing to generate user-friendly composite images, especially when one of the source images is degraded due to being captured under non-ideal photography conditions. Our code and dataset can be accessed at https://github.com/guanguanboy/HCDM.