Sparse Optimization Mechanism and Application of Recurrent Neural Network via Fractional-Order Gradient Descent Learning

Fractional calculus, leveraging its hereditary and infinite memory properties, offers promising applications in information processing and control systems. Recurrent neural networks (RNNs) have also attracted widespread attention in areas such as time series prediction and temporal system modeling. This paper proposes a theoretical result of RNNs based on a fractional-order (FO) gradient descent method and a sparsity mechanism, which not only enhances the stability and sparsity of the network, but also improves its generalization capability. Specifically, the Caputo derivative is firstly employed to define the fractional-order gradient of the error function, which is applied in the backpropagation training of RNNs. Secondly, the smoothing group L1/2 regularization (SGL1/2) is introduced to successfully overcome the oscillation problem of the error function caused by the traditional group L1/2 regularization (GL1/2), and the network architecture is optimized in terms of the redundant nodes and redundant weights of remaining nodes tend to zero, which further improves the sparsity of the network. Finally, numerical simulation results verify the correctness and effectiveness of the proposed algorithm.