Multi-needle digitization in ultrasound image using max-margin mask R-CNN

Digitalizing all the needles in ultrasound (US) images is a crucial step of treatment planning for US-guided high-dose-rate (HDR) prostate brachytherapy. However, current computer-aided technologies are broadly focused on single-needle digitization, while manual digitization of all needles is labor intensive and time consuming. In this paper, we proposed a deep learning-based workflow for fast automatic multi-needle digitization, including needle shaft detection and needle tip detection. The major workflow is composed of two components: a large margin mask R-CNN model (LMMask R-CNN), which adopts the lager margin loss to reformulate Mask R-CNN for needle shaft localization, and a needle-based density-based spatial clustering of application with noise (DBSCAN) algorithm which integrates priors to model a needle in an iteration for a needle shaft refinement and tip detections. Besides, we use the skipping connection in neural network architecture to improve the supervision in hidden layers. Our workflow was evaluated on 23 patients who underwent US-guided HDR prostrate brachytherapy with 339 needles being tested in total. Our method detected 98% of the needles with 0.0911±0.0427 mm shaft error and 0.3303±0.3625 mm tip error. Compared with only using mask R-CNN and only using LMMask R-CNN, the proposed method gains a significant improvement of accuracy on both shaft and tip localization. The proposed method automatically digitizes needles per patient with in a second. It streamlines the workflow of US-guided HDR prostate brachytherapy and paves the way for the development of real-time treatment planning system that is expected to further elevate the quality and outcome of HDR prostate brachytherapy.