Skip to main navigation Skip to search Skip to main content

DC-AVSR: A dynamic collaborative approach for robust audio-visual speech recognition under multimodal distortions

  • Xinyue Zhang
  • , Yakun Zhang
  • , Xingyu Zhang
  • , Changyan Zheng
  • , Liang Xie
  • , Jiangbin Zheng
  • , Erwei Yin
  • Northwestern Polytechnical University Xian
  • Tianjin Artificial Intelligence Innovation Center (TAIIC)
  • Academy of Military Medical Science China
  • High-tech Institute

Research output: Contribution to journal › Article › peer-review

Abstract

Audio-Visual Speech Recognition (AVSR) achieves strong performance in controlled conditions, but its accuracy drops sharply when audio or video streams are degraded by noise, blur, or occlusion. Existing fusion strategies often rely on fixed weighting or attention mechanisms, which cannot reliably adapt to per-sample modality quality and are easily biased toward corrupted inputs. To address this limitation, we propose DC-AVSR, a dynamic collaborative framework for robust multimodal speech recognition. The framework comprises two key modules: a Dynamic Contribution Learning Network (DCLN) that leverages prediction loss as a weak supervisory signal for sample-level reliability estimation, and a Collaborative Compensatory Update (CCU) mechanism that encourages learners to correct each other's errors through error-driven sample reweighting. Experiments on LRS2 and LRS3 show that DC-AVSR maintains competitive accuracy under clean inputs and achieves significant improvements under diverse distortion scenarios. These results demonstrate the effectiveness of loss-driven adaptive fusion and collaborative learning for building reliable audio-visual speech recognition systems in real-world environments.

Original languageEnglish
Article number134694
JournalNeurocomputing
Volume702
DOIs
StatePublished - 14 Nov 2026

Keywords

  • Audio-visual speech recognition
  • Collaborative learning
  • Multimodal learning
  • Robustness
  • Weak supervision

Fingerprint

Dive into the research topics of 'DC-AVSR: A dynamic collaborative approach for robust audio-visual speech recognition under multimodal distortions'. Together they form a unique fingerprint.

Cite this