A brain-to-text framework of decoding natural tonal sentences

doi:10.1101/2024.03.16.585337

	A brain-to-text framework of decoding natural tonal sentences
	Zhang, Daohan 1,2,3; Wang, Zhenjie4 ; Qian, Youkun 1,2,3; Zhao, Zehao 1,2,3; Liu, Yan 1,2,3; Hao, Xiaotao 1,2,3; Li, Wanxin 1,2,3; Lu, Shuo 5; Zhu, Honglin 6; Chen, Luyao 7; Xu, Kunyu 8; Li, Yuanning4,9 ; Lu, Junfeng 1,2,3,8,10
	2024-03-31
状态	已发表
摘要	Speech brain-computer interfaces (BCIs) directly translate brain activity into speech sound and text, yet decoding tonal languages like Mandarin Chinese poses a significant, unexplored challenge. Despite successful cases in non-tonal languages, the complexities of Mandarin, with its distinct syllabic structures and pivotal lexical information conveyed through tonal nuances, present challenges in BCI decoding. Here we designed a brain-to-text framework to decode Mandarin tonal sentences from invasive neural recordings. Our modular approach dissects speech onset, base syllables, and lexical tones, integrating them with contextual information through Bayesian likelihood and the Viterbi decoder. The results demonstrate accurate tone and syllable decoding under variances in continuous naturalistic speech production, surpassing previous intracranial Mandarin tonal syllable decoders in decoding accuracy. We also verified the robustness of our decoding framework and showed that the model hyperparameters can be generalized across participants of varied gender, age, education backgrounds, pronunciation behaviors, and coverage of electrodes. Our pilot study shed lights on the feasibility of more generalizable brain-to-text decoding of natural tonal sentences from patients with high heterogeneities.
关键词	Electrocorticography (ECoG) Brain-Computer Interface (BCI) Tonal language Natural speech Neural Networks
语种	英语
DOI	10.1101/2024.03.16.585337
相关网址	查看原文
出处	bioRxiv
收录类别	PPRN.PPRN
WOS记录号	PPRN:88160760
WOS类目	Neurosciences
资助项目	STl Major Projects["22&ZD299","2023ZKZD13","22PJ1410500","32371146","2022ZD0212300"]
文献类型	预印本
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/372943
专题	生物医学工程学院信息科学与技术学院_硕士生信息科学与技术学院_博士生生物医学工程学院_PI研究组_李远宁
通讯作者	Li, Yuanning; Lu, Junfeng
作者单位	1.Fudan Univ, Huashan Hosp, Shanghai Med Coll, Dept Neurosurg, Shanghai 200040, Peoples R China 2.Shanghai Key Lab Brain Funct Restorat & Neural Regenerat, Shanghai 200040, Peoples R China 3.Fudan Univ, Huashan Hosp, Shanghai Med Coll, Natl Ctr Neurol Disorders, Shanghai 200040, Peoples R China 4.ShanghaiTech Univ, Sch Biomed Engn, Shanghai 201210, Peoples R China 5.Sun Yat Sen Univ, Dept Chinese Language & Literature, Guangzhou 510080, Peoples R China 6.Kings Coll London, Fac Life Sci & Med, London SE1 1UL, England 7.Beijing Normal Univ, Sch Int Chinese Language Educ, Beijing 100875, Peoples R China 8.Fudan Univ, Inst Modern Languages & Linguist, Shanghai 200433, Peoples R China 9.ShanghaiTech Univ, State Key Lab Adv Med Mat & Devices, Shanghai 201210, Peoples R China 10.Fudan Univ, Huashan Hosp, MOE Frontiers, Ctr Brain Sci, Shanghai 200040, Peoples R China
推荐引用方式 GB/T 7714	Zhang, Daohan,Wang, Zhenjie,Qian, Youkun,et al. A brain-to-text framework of decoding natural tonal sentences. 2024.