Merlin: Empowering Multimodal LLMs with Foresight Minds

doi:10.1007/978-3-031-73235-5_24

	Merlin: Empowering Multimodal LLMs with Foresight Minds
	Yu, En 1; Zhao, Liang 2; Wei, Yana3 ; Yang, Jinrong 3; Wu, Dongming 4; Kong, Lingyu 5; Wei, Haoran 2; Wang, Tiancai 2; Ge, Zheng 2; Zhang, Xiangyu 2; Tao, Wenbing 1
	2025
会议录名称	COMPUTER VISION-ECCV 2024, PT IV (IF:0.402[JCR-2005],0.000[5-Year])
ISSN	0302-9743
卷号	15062
发表状态	已发表
DOI	10.1007/978-3-031-73235-5_24
摘要	Humans can foresee the future based on present observations, a skill we term as foresight minds. However, this capability remains under-explored within existing MLLMs, hindering their capacity to understand intentions behind subjects. To address this, we integrate the future modeling into MLLMs. By utilizing the trajectory, a highly structured representation, as a learning objective, we aim to equip the model to understand spatiotemporal dynamics. Inspired by the learning paradigm of LLMs, we first propose Foresight Pre-Training (FPT) that jointly learns various tasks centered on trajectories, enabling MLLMs to predict entire trajectories from a given initial observation. Then, we propose Foresight Instruction-Tuning (FIT) that requires MLLMs to reason about potential future events based on predicted trajectories. Aided by FPT and FIT, we build an unified MLLM named Merlin that supports complex future reasoning. Experiments show Merlin's foresight minds with impressive performance on both future reasoning and visual comprehension tasks. Project page: https://ahnsun.github.io/merlin.
关键词	Multimodal Large Language Model Future Reasoning
会议名称	18th European Conference on Computer Vision (ECCV)
出版地	GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND
会议地点	null,Milan,ITALY
会议日期	SEP 29-OCT 04, 2024
URL	查看原文
收录类别	CPCI-S
语种	英语
资助项目	National Natural Science Foundation of China["62176096","61991412"] ; Research on AI Terminal Computing Power[R2411B0R]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods
WOS记录号	WOS:001346385600024
出版者	SPRINGER INTERNATIONAL PUBLISHING AG
EISSN	1611-3349
引用统计	正在获取...
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/458340
专题	信息科学与技术学院_硕士生
通讯作者	Tao, Wenbing
作者单位	1.Huazhong Univ Sci & Technol, Wuhan, Peoples R China 2.MEGVII Technol, Beijing, Peoples R China 3.ShanghaiTech Univ, Shanghai, Peoples R China 4.Beijing Inst Technol, Beijing, Peoples R China 5.Univ Chinese Acad Sci, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Yu, En,Zhao, Liang,Wei, Yana,et al. Merlin: Empowering Multimodal LLMs with Foresight Minds[C]. GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND:SPRINGER INTERNATIONAL PUBLISHING AG,2025.