Research on RBM Accelerating Learning Algorithm with Weight Momentum

Fei Li, Xiao Guang Gao, Kai Fang Wan

科研成果: 期刊稿件文章同行评审

6 引用 (Scopus)

摘要

Momentum algorithms can accelerate the training speed of restricted Boltzmann machine theoretically. Through a simulation study on existing momentum algorithms, it is found that existing momentum algorithms for training restricted Boltzmann machine have a poor accelerating effect and they began to lose acceleration performance. In the latter part of training process. Focusing on this problem, firstly, this paper gives a theoretical analysis of the algorithms based on Gibbs sampling convergence theorem. It is proved that the acceleration effect of existing momentum algorithms is at the expense of enlarging network weights. Then, a further investigation on network weights shows that the network weights contain a lot of information of the true gradient direction which can be used to train the network. According to this, a weight momentum algorithm is proposed based on the weight of the network. Finally, simulation results demonstrate that the proposed algorithm has a better acceleration effect and has the accelerating ability even in the end of the training process. Therefore the proposed algorithm can well make up for the weaknesses of existing momentum algorithms.

源语言英语
页(从-至)1142-1159
页数18
期刊Zidonghua Xuebao/Acta Automatica Sinica
43
7
DOI
出版状态已出版 - 7月 2017

指纹

探究 'Research on RBM Accelerating Learning Algorithm with Weight Momentum' 的科研主题。它们共同构成独一无二的指纹。

引用此