ShanghaiTech University Knowledge Management System
Merlin:Empowering Multimodal LLMs with Foresight Minds | |
2024-07-03 | |
状态 | 已发表 |
摘要 | Humans can foresee the future based on present observations, a skill we term as foresight minds. . However, this capability remains under-explored within existing MLLMs, hindering their capacity to understand intentions behind subjects. To address this, we integrate the future modeling into MLLMs. By utilizing the trajectory, , a highly structured representation, as a learning objective, we aim to equip the model to understand spatiotemporal dynamics. Inspired by the learning paradigm of LLMs, we first propose Foresight Pre-Training (FPT) that jointly learns various tasks centered on trajectories, enabling MLLMs to predict entire trajectories from a given initial observation. Then, we propose Foresight Instruction-Tuning (FIT) that requires MLLMs to reason about potential future events based on predicted trajectories. Aided by FPT and FIT, we build an unified MLLM named Merlin that supports complex future reasoning. Experiments show Merlin’s foresight minds with impressive performance on both future reasoning and visual comprehension tasks. |
关键词 | Multimodal Large Language Model Future Reasoning |
DOI | arXiv:2312.00589 |
相关网址 | 查看原文 |
出处 | Arxiv |
WOS记录号 | PPRN:86357261 |
WOS类目 | Computer Science, Software Engineering |
文献类型 | 预印本 |
条目标识符 | https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/404293 |
专题 | 信息科学与技术学院_硕士生 |
通讯作者 | Tao, Wenbing |
作者单位 | 1.Huazhong Univ Sci & Technol, Wuhan, Peoples R China 2.MEGVII Technol, Beijing, Peoples R China 3.ShanghaiTech Univ, Shanghai, Peoples R China 4.Beijing Inst Technol, Beijing, Peoples R China 5.Univ Chinese Acad Sci, Beijing, Peoples R China |
推荐引用方式 GB/T 7714 | Yu, En,Zhao, Liang,Wei, Yana,et al. Merlin:Empowering Multimodal LLMs with Foresight Minds. 2024. |
条目包含的文件 | ||||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 |
个性服务 |
查看访问统计 |
谷歌学术 |
谷歌学术中相似的文章 |
[Yu, En]的文章 |
[Zhao, Liang]的文章 |
[Wei, Yana]的文章 |
百度学术 |
百度学术中相似的文章 |
[Yu, En]的文章 |
[Zhao, Liang]的文章 |
[Wei, Yana]的文章 |
必应学术 |
必应学术中相似的文章 |
[Yu, En]的文章 |
[Zhao, Liang]的文章 |
[Wei, Yana]的文章 |
相关权益政策 |
暂无数据 |
收藏/分享 |
修改评论
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。