TUMSyn: A Text-Guided Generalist Model for Customized Multimodal MR Image Synthesis

doi:10.1007/978-3-031-73471-7_13

	TUMSyn: A Text-Guided Generalist Model for Customized Multimodal MR Image Synthesis
	Wang, Yulin1,2,3,4 ; Xiong, Honglin3,4 ; Xie, Yi3,4 ; Liu, Jiameng3,4 ; Wang, Qian3,4,5 ; Liu, Qian 1,2,6; Shen, Dinggang3,4,5
	2025
会议录名称	FOUNDATION MODELS FOR GENERAL MEDICAL AI, MEDAGI 2024 (IF:0.402[JCR-2005],0.000[5-Year])
ISSN	0302-9743
卷号	15184
发表状态	已发表
DOI	10.1007/978-3-031-73471-7_13
摘要	Multimodal magnetic resonance (MR) imaging has revolutionized our understanding of the human brain. However, various limitations in clinical scanning hinder the data acquisition process. Current medical image synthesis techniques, often designed for specific tasks or modalities, exhibit diminished performance when confronted with heterogeneous-source MRI data. Here we introduce a Text-guided Universal MR image Synthesis (TUMSyn) generalist model to generate text-specified multimodal brain MRI sequences from any real-acquired sequences. By leveraging demographic data and imaging parameters as text prompts, TUMSyn achieves diverse cross-sequence synthesis tasks using a unified model. To enhance the efficacy of text features in steering synthesis, we pre-train a text encoder by using contrastive learning strategy to align and fuse image and text semantic information. Developed and evaluated on a multi-center dataset of over 20K brain MR image-text pairs with 7 structural MR contrasts, spanning almost entire age spectrum and various physical conditions, TUMSyn demonstrates comparable or exceeding performance compared to task-specific methods in both supervised and zero-shot settings, and the synthesized images exhibit accurate anatomical morphology suitable for various downstream clinical-related tasks. In summary, by incorporating text metadata into the image synthesis, the accuracy, versatility, and generalizability position TUMSyn as a powerful augmentative tool for conventional MRI systems, offering rapid and cost-effective acquisition of multi-sequence MR images for clinical and research applications.
关键词	Foundation Model Multimodal MRI MRI Synthesis Super-resolution
会议名称	2nd International Workshop on Foundation Models for General Medical AI
出版地	GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND
会议地点	null,Marrakesh,MOROCCO
会议日期	OCT 06, 2024
URL	查看原文
收录类别	CPCI-S
语种	英语
资助项目	National Natural Science Foundation of China["62131015","U23A20295","62250710165","2022ZD0209000"] ; Shanghai Municipal Central Guided Local Science and Technology Development Fund[YDZX20233100001001] ; Key R&D Program of Guangdong Province, China["2023B0303040001","2021B0101420006"] ; Science and Technology special fund of Hainan Province[KJRC2023B06]
WOS研究方向	Computer Science ; Radiology, Nuclear Medicine & Medical Imaging
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications ; Computer Science, Theory & Methods ; Radiology, Nuclear Medicine & Medical Imaging
WOS记录号	WOS:001426955400013
出版者	SPRINGER INTERNATIONAL PUBLISHING AG
EISSN	1611-3349
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/503603
专题	生物医学工程学院生命科学与技术学院_本科生信息科学与技术学院_博士生生物医学工程学院_PI研究组_沈定刚组生物医学工程学院_PI研究组_王乾组
通讯作者	Liu, Qian; Shen, Dinggang
作者单位	1.Hainan Univ, Sch Biomed Engn, Haikou 570228, Hainan, Peoples R China 2.Hainan Univ, State Key Lab Digital Med Engn, Haikou 570228, Hainan, Peoples R China 3.ShanghaiTech Univ, Sch Biomed Engn, Shanghai 201210, Peoples R China 4.ShanghaiTech Univ, State Key Lab Adv Med Mat & Devices, Shanghai 201210, Peoples R China 5.Shanghai Clin Res & Trial Ctr, Shanghai 201210, Peoples R China 6.Hainan Univ, Hlth Inst 1, Key Lab Biomed Engn Hainan Prov, Haikou 570228, Hainan, Peoples R China
第一作者单位	生物医学工程学院; 上海科技大学
通讯作者单位	生物医学工程学院; 上海科技大学
推荐引用方式 GB/T 7714	Wang, Yulin,Xiong, Honglin,Xie, Yi,et al. TUMSyn: A Text-Guided Generalist Model for Customized Multimodal MR Image Synthesis[C]. GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND:SPRINGER INTERNATIONAL PUBLISHING AG,2025.