View Synthesis with Multi-scale Cost Aggregation and Confidence Prior

This paper presents a learning-based novel view synthesis (NVS) approach from wide-baseline image pairs. Inspired by prior work, we first predict a depth probability volume which represents the scene structure as a set of depth probability layers (DPLs) within a reference view frustum. To reduce geometric uncertainty in ambiguous regions between input images, a multi-scale cost aggregation network is proposed to generate the DPLs for both input views without supervision. Furthermore, to mitigate the depth discretizaiton artifacts in distant views, we calculate the disparity map of the target view by passing the warped DPLs onto the target view to a CNN-based fusion network. Finally the predicted view could be obtained by incorporating the disparity map, warped input images and the confidence prior together. The proposed method improves the performance on challenging scenarios such as texture-less or non-textured regions, occlusion boundaries, non-Lambertian surfaces, and distant viewpoints. Experimental results show that our method achieves state-of-the-art view interpolation and extrapolation results on RealEstate10K mini dataset.