A comparison of expressive speech synthesis approaches based on neural network

Liumeng Xue; Xiaolian Zhu; Xiaochun An; Lei Xie

doi:10.1145/3267935.3267947

A comparison of expressive speech synthesis approaches based on neural network

Liumeng Xue
, Xiaolian Zhu
, Xiaochun An
, Lei Xie

School of Computer Science

Northwestern Polytechnical University Xian
Hebei University of Economics and Business

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

Adaptability and controllability in changing speaking styles and speaker characteristics are the advantages of deep neural networks (DNNs) based statistical parametric speech synthesis (SPSS). This paper presents a comprehensive study on the use of DNNs for expressive speech synthesis with a small set of emotional speech data. Specifically, we study three typical model adaptation approaches: (1) retraining a neural model by emotion-specific data (retrain), (2) augmenting the network input using emotion-specific codes (code) and (3) using emotion-dependent output layers with shared hidden layers (multi-head). Long-short term memory (LSTM) networks are used as the acoustic models. Objective and subjective evaluations have demonstrated that the multi-head approach consistently outperforms the other two approaches with more natural emotion delivered in the synthesized speech.

Original language	English
Title of host publication	ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018
Publisher	Association for Computing Machinery, Inc
Pages	15-20
Number of pages	6
ISBN (Electronic)	9781450359856
DOIs	https://doi.org/10.1145/3267935.3267947
State	Published - 19 Oct 2018
Event	Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data Workshop, ASMMC-MMAC 2018 - Seoul, Korea, Republic of Duration: 26 Oct 2018 → …

Publication series

Name	ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018

Conference

Conference	Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data Workshop, ASMMC-MMAC 2018
Country/Territory	Korea, Republic of
City	Seoul
Period	26/10/18 → …

Keywords

Code
Expressive speech synthesis
Multi-head network
Neural networks
Retrain
Statistical parametric speech synthesis
Text-to-speech

Access to Document

10.1145/3267935.3267947

Cite this

Xue, L., Zhu, X., An, X., & Xie, L. (2018). A comparison of expressive speech synthesis approaches based on neural network. In ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018 (pp. 15-20). (ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018). Association for Computing Machinery, Inc. https://doi.org/10.1145/3267935.3267947

Xue, Liumeng ; Zhu, Xiaolian ; An, Xiaochun et al. / A comparison of expressive speech synthesis approaches based on neural network. ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018. Association for Computing Machinery, Inc, 2018. pp. 15-20 (ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018).

@inproceedings{b5adcc09aa994e67b0911b068ad85f24,

title = "A comparison of expressive speech synthesis approaches based on neural network",

abstract = "Adaptability and controllability in changing speaking styles and speaker characteristics are the advantages of deep neural networks (DNNs) based statistical parametric speech synthesis (SPSS). This paper presents a comprehensive study on the use of DNNs for expressive speech synthesis with a small set of emotional speech data. Specifically, we study three typical model adaptation approaches: (1) retraining a neural model by emotion-specific data (retrain), (2) augmenting the network input using emotion-specific codes (code) and (3) using emotion-dependent output layers with shared hidden layers (multi-head). Long-short term memory (LSTM) networks are used as the acoustic models. Objective and subjective evaluations have demonstrated that the multi-head approach consistently outperforms the other two approaches with more natural emotion delivered in the synthesized speech.",

keywords = "Code, Expressive speech synthesis, Multi-head network, Neural networks, Retrain, Statistical parametric speech synthesis, Text-to-speech",

author = "Liumeng Xue and Xiaolian Zhu and Xiaochun An and Lei Xie",

note = "Publisher Copyright: {\textcopyright} 2018 Association for Computing Machinery.; Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data Workshop, ASMMC-MMAC 2018 ; Conference date: 26-10-2018",

year = "2018",

month = oct,

day = "19",

doi = "10.1145/3267935.3267947",

language = "英语",

series = "ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018",

publisher = "Association for Computing Machinery, Inc",

pages = "15--20",

booktitle = "ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018",

}

Xue, L, Zhu, X, An, X & Xie, L 2018, A comparison of expressive speech synthesis approaches based on neural network. in ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018. ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018, Association for Computing Machinery, Inc, pp. 15-20, Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data Workshop, ASMMC-MMAC 2018, Seoul, Korea, Republic of, 26/10/18. https://doi.org/10.1145/3267935.3267947

A comparison of expressive speech synthesis approaches based on neural network. / Xue, Liumeng; Zhu, Xiaolian; An, Xiaochun et al.
ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018. Association for Computing Machinery, Inc, 2018. p. 15-20 (ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A comparison of expressive speech synthesis approaches based on neural network

AU - Xue, Liumeng

AU - Zhu, Xiaolian

AU - An, Xiaochun

AU - Xie, Lei

PY - 2018/10/19

Y1 - 2018/10/19

N2 - Adaptability and controllability in changing speaking styles and speaker characteristics are the advantages of deep neural networks (DNNs) based statistical parametric speech synthesis (SPSS). This paper presents a comprehensive study on the use of DNNs for expressive speech synthesis with a small set of emotional speech data. Specifically, we study three typical model adaptation approaches: (1) retraining a neural model by emotion-specific data (retrain), (2) augmenting the network input using emotion-specific codes (code) and (3) using emotion-dependent output layers with shared hidden layers (multi-head). Long-short term memory (LSTM) networks are used as the acoustic models. Objective and subjective evaluations have demonstrated that the multi-head approach consistently outperforms the other two approaches with more natural emotion delivered in the synthesized speech.

AB - Adaptability and controllability in changing speaking styles and speaker characteristics are the advantages of deep neural networks (DNNs) based statistical parametric speech synthesis (SPSS). This paper presents a comprehensive study on the use of DNNs for expressive speech synthesis with a small set of emotional speech data. Specifically, we study three typical model adaptation approaches: (1) retraining a neural model by emotion-specific data (retrain), (2) augmenting the network input using emotion-specific codes (code) and (3) using emotion-dependent output layers with shared hidden layers (multi-head). Long-short term memory (LSTM) networks are used as the acoustic models. Objective and subjective evaluations have demonstrated that the multi-head approach consistently outperforms the other two approaches with more natural emotion delivered in the synthesized speech.

KW - Code

KW - Expressive speech synthesis

KW - Multi-head network

KW - Neural networks

KW - Retrain

KW - Statistical parametric speech synthesis

KW - Text-to-speech

UR - https://www.scopus.com/pages/publications/85061709851

U2 - 10.1145/3267935.3267947

DO - 10.1145/3267935.3267947

M3 - 会议稿件

AN - SCOPUS:85061709851

T3 - ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018

SP - 15

EP - 20

BT - ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018

PB - Association for Computing Machinery, Inc

T2 - Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data Workshop, ASMMC-MMAC 2018

Y2 - 26 October 2018

ER -

Xue L, Zhu X, An X, Xie L. A comparison of expressive speech synthesis approaches based on neural network. In ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018. Association for Computing Machinery, Inc. 2018. p. 15-20. (ASMMC-MMAC 2018 - Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and 1st Multi-Modal Affective Computing of Large-Scale Multimedia Data, Co-located with MM 2018). doi: 10.1145/3267935.3267947

A comparison of expressive speech synthesis approaches based on neural network

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this