Ideal-LLM: Integrating Dual Encoders and Language-Adapted LLM for Multilingual Speech-to-Text

Hongfei Xue; Wei Ren; Xuelong Geng; Kun Wei; Longhao Li; Qijie Shao; Linju Yang; Kai Diao; Lei Xie

doi:10.1007/978-981-95-5382-2_5

Ideal-LLM: Integrating Dual Encoders and Language-Adapted LLM for Multilingual Speech-to-Text

Hongfei Xue
, Wei Ren
, Xuelong Geng
, Kun Wei
, Longhao Li
, Qijie Shao
, Linju Yang
, Kai Diao
, Lei Xie

School of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Integrating audio encoders with LLMs has enabled models to process audio, enhancing speech-to-text tasks including automatic speech recognition (ASR) and automatic speech translation (AST). However, these methods often overlook language adaptation in multilingual settings, relying on multilingual data without adequately addressing language differences. To address this gap, we propose the Ideal-LLM model, which employs dual multilingual encoders to enrich language features and uses a language-adapted connector to target each language. By leveraging the complementary strengths of Whisper and MMS encoders, our approach ensures richer multilingual representations. Additionally, the connector enhances modal transformation via a weight selector tailored for each language. Experimental results demonstrate that Ideal-LLM improves ASR performance, achieving a 32.6% relative reduction in word error rates compared to the standard speech encoder integrated with LLMs and yields an average BLEU score of 36.78 for AST.

Original language	English
Title of host publication	Man-Machine Speech Communication - 20th National Conference, NCMMSC 2025, Proceedings
Editors	Jia Jia, Zhiyong Wu, Lijian Gao, Gongping Huang, Ya Li
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	47-58
Number of pages	12
ISBN (Print)	9789819553815
DOIs	https://doi.org/10.1007/978-981-95-5382-2_5
State	Published - 2026
Event	20th National Conference on Man-Machine Speech Communication, NCMMSC 2025 - Zhenjiang, China Duration: 16 Oct 2025 → 19 Oct 2025

Publication series

Name	Communications in Computer and Information Science
Volume	2662 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	20th National Conference on Man-Machine Speech Communication, NCMMSC 2025
Country/Territory	China
City	Zhenjiang
Period	16/10/25 → 19/10/25

Keywords

Dual Encoders
Large Language Models
Multilingual Speech-to-Text

Access to Document

10.1007/978-981-95-5382-2_5

Cite this

Xue, H., Ren, W., Geng, X., Wei, K., Li, L., Shao, Q., Yang, L., Diao, K., & Xie, L. (2026). Ideal-LLM: Integrating Dual Encoders and Language-Adapted LLM for Multilingual Speech-to-Text. In J. Jia, Z. Wu, L. Gao, G. Huang, & Y. Li (Eds.), Man-Machine Speech Communication - 20th National Conference, NCMMSC 2025, Proceedings (pp. 47-58). (Communications in Computer and Information Science; Vol. 2662 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-981-95-5382-2_5

Xue, Hongfei ; Ren, Wei ; Geng, Xuelong et al. / Ideal-LLM : Integrating Dual Encoders and Language-Adapted LLM for Multilingual Speech-to-Text. Man-Machine Speech Communication - 20th National Conference, NCMMSC 2025, Proceedings. editor / Jia Jia ; Zhiyong Wu ; Lijian Gao ; Gongping Huang ; Ya Li. Springer Science and Business Media Deutschland GmbH, 2026. pp. 47-58 (Communications in Computer and Information Science).

@inproceedings{cdb7bf22f8494f8ba9f52f1eebeb462c,

title = "Ideal-LLM: Integrating Dual Encoders and Language-Adapted LLM for Multilingual Speech-to-Text",

abstract = "Integrating audio encoders with LLMs has enabled models to process audio, enhancing speech-to-text tasks including automatic speech recognition (ASR) and automatic speech translation (AST). However, these methods often overlook language adaptation in multilingual settings, relying on multilingual data without adequately addressing language differences. To address this gap, we propose the Ideal-LLM model, which employs dual multilingual encoders to enrich language features and uses a language-adapted connector to target each language. By leveraging the complementary strengths of Whisper and MMS encoders, our approach ensures richer multilingual representations. Additionally, the connector enhances modal transformation via a weight selector tailored for each language. Experimental results demonstrate that Ideal-LLM improves ASR performance, achieving a 32.6\% relative reduction in word error rates compared to the standard speech encoder integrated with LLMs and yields an average BLEU score of 36.78 for AST.",

keywords = "Dual Encoders, Large Language Models, Multilingual Speech-to-Text",

author = "Hongfei Xue and Wei Ren and Xuelong Geng and Kun Wei and Longhao Li and Qijie Shao and Linju Yang and Kai Diao and Lei Xie",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2026.; 20th National Conference on Man-Machine Speech Communication, NCMMSC 2025 ; Conference date: 16-10-2025 Through 19-10-2025",

year = "2026",

doi = "10.1007/978-981-95-5382-2\_5",

language = "英语",

isbn = "9789819553815",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "47--58",

editor = "Jia Jia and Zhiyong Wu and Lijian Gao and Gongping Huang and Ya Li",

booktitle = "Man-Machine Speech Communication - 20th National Conference, NCMMSC 2025, Proceedings",

}

Xue, H, Ren, W, Geng, X, Wei, K, Li, L, Shao, Q, Yang, L, Diao, K & Xie, L 2026, Ideal-LLM: Integrating Dual Encoders and Language-Adapted LLM for Multilingual Speech-to-Text. in J Jia, Z Wu, L Gao, G Huang & Y Li (eds), Man-Machine Speech Communication - 20th National Conference, NCMMSC 2025, Proceedings. Communications in Computer and Information Science, vol. 2662 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 47-58, 20th National Conference on Man-Machine Speech Communication, NCMMSC 2025, Zhenjiang, China, 16/10/25. https://doi.org/10.1007/978-981-95-5382-2_5

Ideal-LLM: Integrating Dual Encoders and Language-Adapted LLM for Multilingual Speech-to-Text. / Xue, Hongfei; Ren, Wei; Geng, Xuelong et al.
Man-Machine Speech Communication - 20th National Conference, NCMMSC 2025, Proceedings. ed. / Jia Jia; Zhiyong Wu; Lijian Gao; Gongping Huang; Ya Li. Springer Science and Business Media Deutschland GmbH, 2026. p. 47-58 (Communications in Computer and Information Science; Vol. 2662 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Ideal-LLM

T2 - 20th National Conference on Man-Machine Speech Communication, NCMMSC 2025

AU - Xue, Hongfei

AU - Ren, Wei

AU - Geng, Xuelong

AU - Wei, Kun

AU - Li, Longhao

AU - Shao, Qijie

AU - Yang, Linju

AU - Diao, Kai

AU - Xie, Lei

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2026.

PY - 2026

Y1 - 2026

N2 - Integrating audio encoders with LLMs has enabled models to process audio, enhancing speech-to-text tasks including automatic speech recognition (ASR) and automatic speech translation (AST). However, these methods often overlook language adaptation in multilingual settings, relying on multilingual data without adequately addressing language differences. To address this gap, we propose the Ideal-LLM model, which employs dual multilingual encoders to enrich language features and uses a language-adapted connector to target each language. By leveraging the complementary strengths of Whisper and MMS encoders, our approach ensures richer multilingual representations. Additionally, the connector enhances modal transformation via a weight selector tailored for each language. Experimental results demonstrate that Ideal-LLM improves ASR performance, achieving a 32.6% relative reduction in word error rates compared to the standard speech encoder integrated with LLMs and yields an average BLEU score of 36.78 for AST.

AB - Integrating audio encoders with LLMs has enabled models to process audio, enhancing speech-to-text tasks including automatic speech recognition (ASR) and automatic speech translation (AST). However, these methods often overlook language adaptation in multilingual settings, relying on multilingual data without adequately addressing language differences. To address this gap, we propose the Ideal-LLM model, which employs dual multilingual encoders to enrich language features and uses a language-adapted connector to target each language. By leveraging the complementary strengths of Whisper and MMS encoders, our approach ensures richer multilingual representations. Additionally, the connector enhances modal transformation via a weight selector tailored for each language. Experimental results demonstrate that Ideal-LLM improves ASR performance, achieving a 32.6% relative reduction in word error rates compared to the standard speech encoder integrated with LLMs and yields an average BLEU score of 36.78 for AST.

KW - Dual Encoders

KW - Large Language Models

KW - Multilingual Speech-to-Text

UR - https://www.scopus.com/pages/publications/105027943300

U2 - 10.1007/978-981-95-5382-2_5

DO - 10.1007/978-981-95-5382-2_5

M3 - 会议稿件

AN - SCOPUS:105027943300

SN - 9789819553815

T3 - Communications in Computer and Information Science

SP - 47

EP - 58

BT - Man-Machine Speech Communication - 20th National Conference, NCMMSC 2025, Proceedings

A2 - Jia, Jia

A2 - Wu, Zhiyong

A2 - Gao, Lijian

A2 - Huang, Gongping

A2 - Li, Ya

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 16 October 2025 through 19 October 2025

ER -

Xue H, Ren W, Geng X, Wei K, Li L, Shao Q et al. Ideal-LLM: Integrating Dual Encoders and Language-Adapted LLM for Multilingual Speech-to-Text. In Jia J, Wu Z, Gao L, Huang G, Li Y, editors, Man-Machine Speech Communication - 20th National Conference, NCMMSC 2025, Proceedings. Springer Science and Business Media Deutschland GmbH. 2026. p. 47-58. (Communications in Computer and Information Science). doi: 10.1007/978-981-95-5382-2_5