| |||||||
ShanghaiTech University Knowledge Management System
| SEE: Semantically Aligned EEG-to-Text Translation | |
| 2024-09-14 | |
| 状态 | 已发表 |
| 摘要 | Decoding neurophysiological signals into language is of great research interest within brain-computer interface (BCI) applications. Electroencephalography (EEG), known for its non-invasiveness, ease of use, and cost-effectiveness, has been a popular method in this field. However, current EEG-to-Text decoding approaches face challenges due to the huge domain gap between EEG recordings and raw texts, inherent data bias, and small closed vocabularies. In this paper, we propose SEE: Semantically Aligned EEG-to-Text Translation, a novel method aimed at improving EEG-to-Text decoding by seamlessly integrating two modules into a pre-trained BART language model. These two modules include (1) a Cross-Modal Codebook that learns cross-modal representations to enhance feature consolidation and mitigate domain gap, and (2) a Semantic Matching Module that fully utilizes pre-trained text representations to align multi-modal features extracted from EEG-Text pairs while considering noise caused by false negatives, i.e., data from different EEG-Text pairs that have similar semantic meanings. Experimental results on the Zurich Cognitive Language Processing Corpus (ZuCo) demonstrate the effectiveness of SEE, which enhances the feasibility of accurate EEG-to-Text decoding. |
| 关键词 | EEG-to-Text self-supervised learning multi- modality |
| 语种 | 英语 |
| DOI | arXiv:2409.16312 |
| 相关网址 | 查看原文 |
| 出处 | Arxiv |
| 收录类别 | PPRN.PPRN |
| WOS记录号 | PPRN:98872468 |
| WOS类目 | Biology ; Computer Science, Artificial Intelligence ; Engineering, Electrical& Electronic |
| 文献类型 | 预印本 |
| 条目标识符 | https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/433538 |
| 专题 | 生物医学工程学院 生物医学工程学院_公共科研平台_智能医学科研平台 生物医学工程学院_PI研究组_王乾组 生物医学工程学院_PI研究组_张寒组 生物医学工程学院_硕士生 生物医学工程学院_博士生 |
| 通讯作者 | Zhang, Han |
| 作者单位 | 1.ShanghaiTech Univ, Sch Biomed Engn, Shanghai, Peoples R China 2.ShanghaiTech Univ, State Key Lab Adv Med Mat & Devices, Shanghai, Peoples R China 3.Shanghai Clin Res & Trial Ctr, Shanghai, Peoples R China |
| 推荐引用方式 GB/T 7714 | Tao, Yitian,Liang, Yan,Wang, Luoyu,et al. SEE: Semantically Aligned EEG-to-Text Translation. 2024. |
| 条目包含的文件 | ||||||
| 条目无相关文件。 | ||||||
修改评论
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。