Ultimate reconstruction: Understand your bones from orthogonal views

3D image reconstruction is a common basis of medical image analysis, which requires a sequence of 2D slices/tomograms obtained from the relative motion to provide enough 3D information. When considering only the task to localize exception objects, a pair of two-view perspective 2D images may also be able to provide enough 3D information, which, however, has not been well studied. In this paper, we proposed the concept of Ultimate Reconstruction (UR) that reconstructs a 3D image from only a pair of two-view perspective 2D images. We resort techniques of generative adversarial network (GAN) to deal with this task, where we propose the Sense-consistency GAN (SGAN) with the sense-consistency constraint to learning the potential coarse-to-fine sense information during training the generative model. Experiments on the KiTS19 dataset with 300 subjects demonstrate that our SGAN achieves MAE/SSIM / PSNR values of 11.16% / 66.50%/23.82 when using only two 2D perspective images. It supports the possibility of UR and indicates that SGAN is promising to deal with UR.