E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models

Hongfei Xue, Yuhao Liang, Bingshen Mu, Shiliang Zhang, Mengzhe Chen, Qian Chen, Lei Xie

科研成果: 书/报告/会议事项章节会议稿件同行评审

1 引用 (Scopus)

摘要

This study focuses on emotion-sensitive spoken dialogue in human-machine speech interaction. With the advancement of Large Language Models (LLMs), dialogue systems can handle multimodal data, including audio. Recent models have enhanced the understanding of complex audio signals through the integration of various audio events. However, they are unable to generate appropriate responses based on emotional speech. To address this, we introduce the Emotional chat Model (E-chat), a novel spoken dialogue system capable of comprehending and responding to emotions conveyed from speech. This model leverages an emotion embedding extracted by a speech encoder, combined with LLMs, enabling it to respond according to different emotional contexts. Additionally, we introduce the E-chat200 dataset, designed explicitly for emotion-sensitive spoken dialogue. In various evaluation metrics, E-chat consistently outperforms baseline model, demonstrating its potential in emotional comprehension and human-machine interaction.

源语言英语
主期刊名2024 14th International Symposium on Chinese Spoken Language Processing, ISCSLP 2024
编辑Yanmin Qian, Qin Jin, Zhijian Ou, Zhenhua Ling, Zhiyong Wu, Ya Li, Lei Xie, Jianhua Tao
出版商Institute of Electrical and Electronics Engineers Inc.
586-590
页数5
ISBN(电子版)9798331516826
DOI
出版状态已出版 - 2024
活动14th International Symposium on Chinese Spoken Language Processing, ISCSLP 2024 - Beijing, 中国
期限: 7 11月 202410 11月 2024

出版系列

姓名2024 14th International Symposium on Chinese Spoken Language Processing, ISCSLP 2024

会议

会议14th International Symposium on Chinese Spoken Language Processing, ISCSLP 2024
国家/地区中国
Beijing
时期7/11/2410/11/24

指纹

探究 'E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models' 的科研主题。它们共同构成独一无二的指纹。

引用此