Dualsep: A Light-Weight Dual-Encoder Convolutional Recurrent Network For Real-Time In-Car Speech Separation

Ziqian Wang, Jiayao Sun, Zihan Zhang, Xingchen Li, Jie Liu, Lei Xie

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

Advancements in deep learning and voice-activated technologies have driven the development of human-vehicle interaction. Distributed microphone arrays are widely used in incar scenarios because they can accurately capture the voices of passengers from different speech zones. However, the increase in the number of audio channels, coupled with the limited computational resources and low latency requirements of in-car systems, presents challenges for in-car multi-channel speech separation. To migrate the problems, we propose a lightweight framework that cascades digital signal processing (DSP) and neural networks (NN). We utilize fixed beamforming (BF) to reduce computational costs and independent vector analysis (IVA) to provide spatial prior. We employ dual encoders for dual-branch modeling, with spatial encoder capturing spatial cues and spectral encoder preserving spectral information, facilitating spatial-spectral fusion. Our proposed system supports both streaming and non-streaming modes. Experimental results demonstrate the superiority of the proposed system across various metrics. With only 0.83 M parameters and 0.39 real-time factor (RTF) on an Intel Core i7 (2.6 GHz) CPU, it effectively separates speech into distinct speech zones.

Original languageEnglish
Title of host publicationProceedings of 2024 IEEE Spoken Language Technology Workshop, SLT 2024
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages286-293
Number of pages8
ISBN (Electronic)9798350392258
DOIs
StatePublished - 2024
Event2024 IEEE Spoken Language Technology Workshop, SLT 2024 - Macao, China
Duration: 2 Dec 20245 Dec 2024

Publication series

NameProceedings of 2024 IEEE Spoken Language Technology Workshop, SLT 2024

Conference

Conference2024 IEEE Spoken Language Technology Workshop, SLT 2024
Country/TerritoryChina
CityMacao
Period2/12/245/12/24

Keywords

  • deep learning
  • in-car communication
  • microphone arrays
  • speech separation

Fingerprint

Dive into the research topics of 'Dualsep: A Light-Weight Dual-Encoder Convolutional Recurrent Network For Real-Time In-Car Speech Separation'. Together they form a unique fingerprint.

Cite this