Abstract
Audio-Visual Speech Recognition (AVSR) achieves strong performance in controlled conditions, but its accuracy drops sharply when audio or video streams are degraded by noise, blur, or occlusion. Existing fusion strategies often rely on fixed weighting or attention mechanisms, which cannot reliably adapt to per-sample modality quality and are easily biased toward corrupted inputs. To address this limitation, we propose DC-AVSR, a dynamic collaborative framework for robust multimodal speech recognition. The framework comprises two key modules: a Dynamic Contribution Learning Network (DCLN) that leverages prediction loss as a weak supervisory signal for sample-level reliability estimation, and a Collaborative Compensatory Update (CCU) mechanism that encourages learners to correct each other's errors through error-driven sample reweighting. Experiments on LRS2 and LRS3 show that DC-AVSR maintains competitive accuracy under clean inputs and achieves significant improvements under diverse distortion scenarios. These results demonstrate the effectiveness of loss-driven adaptive fusion and collaborative learning for building reliable audio-visual speech recognition systems in real-world environments.
| Original language | English |
|---|---|
| Article number | 134694 |
| Journal | Neurocomputing |
| Volume | 702 |
| DOIs | |
| State | Published - 14 Nov 2026 |
Keywords
- Audio-visual speech recognition
- Collaborative learning
- Multimodal learning
- Robustness
- Weak supervision
Fingerprint
Dive into the research topics of 'DC-AVSR: A dynamic collaborative approach for robust audio-visual speech recognition under multimodal distortions'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver