Synthetic aperture photography using a moving camera-IMU system

Occlusion poses as a critical challenge in computer vision for a long time. Recently, the technique of synthetic aperture photography using a camera array has been regarded as a promising way to address the problem of occluded object imaging. Unfortunately, the expensive cost of a standard camera array system with the required calibration procedure still limits the widespread popularity of this technique. To make synthetic aperture photography practical, in this paper, we propose a novel approach based on a mobile camera-IMU system moving on a planar surface. The contributions of our work include: (1) with geometric property assumptions, we mathematically prove that the transformation matrices for synthetic aperture image generation can be factorized into an infinite homography and a shift; (2) the IMU data are utilized to estimate the homography by selecting the plane at infinity as the reference plane, which is scene independent and does not require any calibration; (3) compared to the conventional camera array based approaches, the proposed method effectively improves the solution for synthetic aperture images capturing and generation by using a more flexible, convenient and inexpensive system. Extensive experimental results with qualitative and quantitative evaluations have demonstrated the see-through-occlusion performance with satisfactory accuracy of the proposed approach in different scenarios.