Encoding Deep Residual Features into Fisher Vector for Skin Lesion Classification

Computer-aided skin lesion classification using dermoscopy is essential for early detection of melanoma, which is the most effective means to reduce the mortality rate. Although many deep learning models have been designed for this task, skin lesion classification remains challenging due to the small sample size, inter-class similarity, intra-class inconsistency, and class imbalance. In this paper, we propose a hybrid deep residual network and Fisher vector (ResNet-FV) algorithm for skin lesion classification, aiming to boost the performances of ResNet using the Fisher vector encoding scheme. The proposed algorithm has been evaluated on the 2018 Skin Lesion Analysis Towards Melanoma Detection Challenge (ISIC-skin 2018) dataset and achieved a balanced multi-class accuracy of 0.798, outperforming several existing solutions. Clinical relevance- We propose a computer-aided diagnosis algorithm called ResNet-FV which achieves superior performance when comparing to several existing solutions and hence has the potential to be applied to large-scale skin cancer screening.