<i>SPHINX</i>: A Mixer of Weights, Visual Embeddings and Image Scales for Multi-modal Large Language Models

doi:10.1007/978-3-031-73033-7_3

	SPHINX: A Mixer of Weights, Visual Embeddings and Image Scales for Multi-modal Large Language Models
	Lin, Ziyi 1,2; Liu, Dongyang 2; Zhang, Renrui 1,2; Gao, Peng 2; Qiu, Longtian2,3 ; Xiao, Han 2; Qiu, Han 2; Shao, Wenqi 2; Chen, Keqin 2; Han, Jiaming 1,2; Huang, Siyuan 2; Zhang, Yichi 2; He, Xuming3 ; Qiao, Yu 2; Li, Hongsheng 1,2,4
	2025
会议录名称	COMPUTER VISION - ECCV 2024, PT LXII (IF:0.402[JCR-2005],0.000[5-Year])
ISSN	0302-9743
卷号	15120
发表状态	已发表
DOI	10.1007/978-3-031-73033-7_3
摘要	We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, visual embeddings and image scales. First, for stronger vision-language alignment, we unfreeze the large language model (LLM) during pre-training, and introduce a weight mix strategy between LLMs trained by real-world and synthetic data. By directly integrating the weights from two domains, the mixed LLM can efficiently incorporate diverse semantics with favorable robustness. Then, we propose to extract comprehensive visual embeddings from various network architectures, pre-training paradigms, and information granularity, providing language models with more robust image representations. We further propose an efficient strategy aiming to better capture fine-grained appearances of high-resolution images. With a mixing of different scales and high-resolution sub-images, SPHINX attains exceptional visual parsing and reasoning performance on existing evaluation benchmarks. Based on our proposed joint mixing, SPHINX exhibits superior multi-modal understanding capabilities on a wide range of applications, with highlighted fine-grained visual recognition abilities such as region-level understanding, caption grounding, document layout detection, and human pose estimation. We hope our work may cast a light on the exploration of joint mixing in future MLLM research. Code is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.
会议名称	18th European Conference on Computer Vision (ECCV)
出版地	GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND
会议地点	null,Milan,ITALY
会议日期	SEP 29-OCT 04, 2024
URL	查看原文
收录类别	CPCI-S
语种	英语
资助项目	National Key R&D Program of China["2022ZD0161100","2022ZD0160102"] ; National Natural Science Foundation of China[62206272] ; Smart Traffic Fund[PSRI/76/2311/PR] ; RGC General Research Fund[14204021]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications ; Computer Science, Theory & Methods
WOS记录号	WOS:001353685900003
出版者	SPRINGER INTERNATIONAL PUBLISHING AG
EISSN	1611-3349
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/458342
专题	信息科学与技术学院_博士生信息科学与技术学院_PI研究组_何旭明组
通讯作者	Gao, Peng
作者单位	1.Chinese Univ Hong Kong, Multimedia Lab, Ma Liu Shui, Hong Kong, Peoples R China 2.Shanghai AI Lab, Shanghai, Peoples R China 3.ShanghaiTech Univ, Shanghai, Peoples R China 4.Ctr Perceptual & Interact Intelligence Ltd, Hong Kong, Peoples R China
推荐引用方式 GB/T 7714	Lin, Ziyi,Liu, Dongyang,Zhang, Renrui,et al. SPHINX: A Mixer of Weights, Visual Embeddings and Image Scales for Multi-modal Large Language Models[C]. GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND:SPRINGER INTERNATIONAL PUBLISHING AG,2025.