DCIM-AVSR: Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module

doi:10.1109/ICASSP49660.2025.10890272

	DCIM-AVSR: Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module
	Xinyu Wang 1; Haotian Jiang1 ; Haolin Huang 1; Yu Fang 1; Mengjie Xu1 ; Qian Wang 1,2
	2025-04-11
会议录名称	ICASSP 2025 - 2025 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP)
ISSN	1520-6149
发表状态	已发表
DOI	10.1109/ICASSP49660.2025.10890272
摘要	Speech recognition is the technology that enables machines to interpret and process human speech, converting spoken language into text or commands. This technology is essential for applications such as virtual assistants, transcription services, and communication tools. The Audio-Visual Speech Recognition (AVSR) model enhances traditional speech recognition, particularly in noisy environments, by incorporating visual modalities like lip movements and facial expressions. While traditional AVSR models trained on large-scale datasets with numerous parameters can achieve remarkable accuracy, often surpassing human performance, they also come with high training costs and deployment challenges. To address these issues, we introduce an efficient AVSR model that reduces the number of parameters through the integration of a Dual Conformer Interaction Module (DCIM). In addition, we propose a pre-training method that optimizes model performance by fine-tuning. Unlike conventional models that require the system to independently learn the hierarchical relationship between audio and visual modalities, our approach incorporates this distinction directly into the model architecture. This design enhances both efficiency and performance, resulting in a more practical and effective solution for AVSR tasks.
会议地点	Hyderabad, India
会议日期	6-11 April 2025
URL	查看原文
来源库	IEEE
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/496897
专题	生物医学工程学院_硕士生生物医学工程学院_硕士生
作者单位	1.School of Biomedical Engineering & State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University, Shanghai, China 2.Shanghai Clinical Research and Trial Center, Shanghai, China
第一作者单位	上海科技大学
第一作者的第一单位	上海科技大学
推荐引用方式 GB/T 7714	Xinyu Wang,Haotian Jiang,Haolin Huang,et al. DCIM-AVSR: Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module[C],2025.