DAN-NucNet: A dual attention based framework for nuclei segmentation in cancer histology images under wild clinical conditions[Formula presented]

Nuclei segmentation plays an essential role in histology analysis. The nuclei segmentation in histology images is challenging in variable conditions (clinical wild), such as poor staining quality, stain variability, tissue variability, and conditions having higher morphological variability. Recently, some deep learning models have been proposed for nuclei segmentation. However, these models rarely solve the problems mentioned above simultaneously. Most of the information in Hematoxylin and Eosin (H&E) stained histology images is in its channel, and the remaining information is in the spatial domain. We observed that most problems could be solved by considering channel and spatial features simultaneously, e.g., the spatial and channel features provide the solution to the morphological variability and staining variability, respectively. Therefore, we propose a novel spatial-channel attention-based modified UNet architecture with ResNet blocks in encoder layers. The UNet baseline preserves coarse and fine features, thus proving the solution to the tissue variability. The proposed method significantly improves the segmentation performance compared to the state-of-the-art methods on three different benchmark datasets. We demonstrate that the proposed model is generalized for 20 cancer sites, more than any reported literature. The proposed model is less complex than most state-of-the-art models. The impact of the proposed model is that it will help improve further procedures such as nuclei instance segmentation, nuclei classification, and cancer grading.