TOEPLITZ NEURAL NETWORK FOR SEQUENCE MODELING

Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, Yiran Zhong

科研成果: 会议稿件论文同行评审

16 引用 (Scopus)

摘要

Sequence modeling has important applications in natural language processing and computer vision. Recently, the transformer-based models have shown strong performance on various sequence modeling tasks, which rely on attention to capture pairwise token relations, and position embedding to inject positional information. While showing good performance, the transformer models are inefficient to scale to long input sequences, mainly due to the quadratic space-time complexity of attention. To overcome this inefficiency, we propose to model sequences with a relative position encoded Toeplitz matrix and use a Toeplitz matrix-vector production trick to reduce the space-time complexity of the sequence modeling to log linear. A lightweight sub-network called relative position encoder is proposed to generate relative position coefficients with a fixed budget of parameters, enabling the proposed Toeplitz neural network to deal with varying sequence lengths. In addition, despite being trained on 512-token sequences, our model can extrapolate input sequence length up to 14K tokens in inference with consistent performance. Extensive experiments on autoregressive and bidirectional language modeling, image modeling, and the challenging Long-Range Arena benchmark show that our method achieves better performance than its competitors in most downstream tasks while being significantly faster. The code is available at https://github.com/OpenNLPLab/Tnn.

源语言英语
出版状态已出版 - 2023
活动11th International Conference on Learning Representations, ICLR 2023 - Kigali, 卢旺达
期限: 1 5月 20235 5月 2023

会议

会议11th International Conference on Learning Representations, ICLR 2023
国家/地区卢旺达
Kigali
时期1/05/235/05/23

指纹

探究 'TOEPLITZ NEURAL NETWORK FOR SEQUENCE MODELING' 的科研主题。它们共同构成独一无二的指纹。

引用此