TY - JOUR
T1 - Training restricted boltzmann machine using gradient fixing based algorithm
AU - Li, Fei
AU - Gao, Xiaoguang
AU - Wan, Kaifang
N1 - Publisher Copyright:
© 2018 Chinese Institute of Electronics. All rights reserved.
PY - 2018/7/10
Y1 - 2018/7/10
N2 - Most of the algorithms for training restricted Boltzmann machines (RBM) are based on Gibbs sampling. When the sampling algorithm is used to calculate the gradient, the sampling gradient is the approximate value of the true gradient and there is a big error between the sampling gradient and the true gradient, which seriously affects the training effect of the network. Aiming at this problem, this paper analysed the numerical error and orientation error between the approximate gradient and the true gradient. Their influence on the performance of network training is given then. An gradient fixing model was established. It was designed to adjust the numerical value and orientation of the approximate gradient and reduce the error. We also designed gradient fixing based Gibbs sampling training algorithm (GFGS) and gradient fixing based parallel tempering algorithm (GFPT), and the comparison experiment of the novel algorithms and the existing algorithms is given. It has been demonstrated that the new algorithms can effectively tackle the issue of gradient error, and can achieve higher training accuracy at a reasonable expense of computational runtime.
AB - Most of the algorithms for training restricted Boltzmann machines (RBM) are based on Gibbs sampling. When the sampling algorithm is used to calculate the gradient, the sampling gradient is the approximate value of the true gradient and there is a big error between the sampling gradient and the true gradient, which seriously affects the training effect of the network. Aiming at this problem, this paper analysed the numerical error and orientation error between the approximate gradient and the true gradient. Their influence on the performance of network training is given then. An gradient fixing model was established. It was designed to adjust the numerical value and orientation of the approximate gradient and reduce the error. We also designed gradient fixing based Gibbs sampling training algorithm (GFGS) and gradient fixing based parallel tempering algorithm (GFPT), and the comparison experiment of the novel algorithms and the existing algorithms is given. It has been demonstrated that the new algorithms can effectively tackle the issue of gradient error, and can achieve higher training accuracy at a reasonable expense of computational runtime.
KW - Deep Learning
KW - Gibbs sampling training algorithm (GFGS)
KW - Gradient fixing
KW - Gradient fixing based parallel tempering algorithm (GFPT)
KW - Restricted Boltzmann machine (RBM)
UR - http://www.scopus.com/inward/record.url?scp=85051354897&partnerID=8YFLogxK
U2 - 10.1049/cje.2018.05.007
DO - 10.1049/cje.2018.05.007
M3 - 文章
AN - SCOPUS:85051354897
SN - 1022-4653
VL - 27
SP - 694
EP - 703
JO - Chinese Journal of Electronics
JF - Chinese Journal of Electronics
IS - 4
ER -