Towards the classification of cancer subtypes by using cascade deep forest model in gene expression data

The classification of cancer subtypes is of great importance in cancer disease diagnosis and therapy. Many supervised learning methods have been applied to classification of cancer subtypes in the past few years, especially of deep learning based methods. Recently, a deep forest model has been proposed as an alternative of deep neural networks to learn hyper-representations by using cascade ensemble decision trees, and it has been proved that deep forest model has competitive or even better performance than deep neural networks. However, the original deep forest may face under-fitting and ensemble diversity problems when dealing with small sample size, and high-dimension biology data. It is important to improve the deep forest model to work better on small-scale biology data. In this paper, we propose a deep learning model to follow the mission of cancer subtype classification on small-scale biology data sets, which can be viewed as modification of original deep forest model. Our model distinguishes from the original deep forest model with two main contributions: First, a named multi-class-scanning method is proposed to train multiple simple binary classifiers to encourage diversity of ensemble. Meanwhile, the fitting quality of each classifier is considered in representations learning. Second, we propose a boosting strategy to emphasize more important features in cascade forests of representations learning, thus to propagate the benefits of discriminative features among layers to improve the overall classification performance. Systematical experiments on both microarray and RNA-seq data sets demonstrate that our method consistently outperforms the most state-of-the-art classification methods in application of cancer subtype classifications.