Atlas registration and ensemble deep convolutional neural network-based prostate segmentation using magnetic resonance imaging

Automatic segmentation of prostate in magnetic resonance (MR) images has been more and more applied to the diagnosis of prostate disease and various clinical applications. However, due to the inhomogeneous and varying anatomical appearance around prostate boundary, the segmentation of prostate MR images faces great challenges. Since deep learning shows superior performance in computer vision, we propose a coarse-to-fine segmentation strategy by using deep neural networks to tackle the segmentation problem of the endorectal coil prostate images and non-endorectal coil prostate images separately. First, we present a registration-based coarse segmentation to the pre-processed prostate MR images to get the potential boundary region. Second, we train deep neural networks as pixel-based classifier to predict whether the pixel in the potential boundary region is prostate pixel or not. To improve the discriminability of the algorithm, we further introduce ensemble learning for fine segmentation. Finally, a boundary refinement is used to eliminate the outlier and smooth the boundary. The proposed method has been extensively evaluated on the PROMIS12 challenge dataset and PROSTATEx17 challenge dataset. Experimental results show superior segmentation performance (0.910 ± 0.036 in dice ratio, 1.583 ± 0.441 in average boundary distance and 4.579 ± 1.791 in Hausdorff distance), which demonstrates the effectiveness of the proposed algorithm.