SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding

doi:arXiv:2502.10810

	SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding
	Yang, Zhenyu 1,2,3; Hu, Yuhang 4; Du, Zemin5 ; Xue, Dizhan 1,2; Qian, Shengsheng 1,2; Wu, Jiahong 3; Yang, Fan 3; Dong, Weiming 1,2; Xu, Changsheng 1,2,6
	2025-02-15
状态	已发表
摘要	Despite the significant advancements of Large Vision-Language Models (LVLMs) on established benchmarks, there remains a notable gap in suitable evaluation regarding their applicability in the emerging domain of long-context streaming video understanding. Current benchmarks for video understanding typically emphasize isolated single-instance text inputs and fail to evaluate the capacity to sustain temporal reasoning throughout the entire duration of video streams. To address these limitations, we introduce SVBench, a pioneering benchmark with temporal multi-turn question-answering chains specifically designed to thoroughly assess the capabilities of streaming video understanding of current LVLMs. We design a semi-automated annotation pipeline to obtain 49,979 Question-Answer (QA) pairs of 1,353 streaming videos, which includes generating QA chains that represent a series of consecutive multi-turn dialogues over video segments and constructing temporal linkages between successive QA chains. Our experimental results, obtained from 14 models in dialogue and streaming evaluations, reveal that while the closed-source GPT-4o outperforms others, most open-source LVLMs struggle with long-context streaming video understanding. We also construct a StreamingChat model, which significantly outperforms open-source LVLMs on our SVBench and achieves comparable performance on diverse vision-language benchmarks. We expect SVBench to advance the research of streaming video understanding by providing a comprehensive and in-depth analysis of current LVLMs. Our benchmark and model can be accessed at https://yzy-bupt.github.io/SVBench.
语种	英语
DOI	arXiv:2502.10810
相关网址	查看原文
出处	Arxiv
收录类别	PPRN.PPRN
WOS记录号	PPRN:121697535
WOS类目	Computer Science, Software Engineering
资助项目	National Natural Science Foundation of China["62036012","62276257"] ; Beijing Natural Science Foundation[JQ23018] ; National Key Research and Development Program of China[2023YFC3310700]
文献类型	预印本
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/514091
专题	信息科学与技术学院_硕士生
通讯作者	Qian, Shengsheng
作者单位	1.Chinese Acad Sci, Inst Automat, Beijing, Peoples R China 2.Univ Chinese Acad Sci, Beijing, Peoples R China 3.Kuaishou Technol, Beijing, Peoples R China 4.Zhengzhou Univ, Zhengzhou, Peoples R China 5.ShanghaiTech Univ, Shanghai, Peoples R China 6.Peng Cheng Lab, Shenzhen, Peoples R China
推荐引用方式 GB/T 7714	Yang, Zhenyu,Hu, Yuhang,Du, Zemin,et al. SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding. 2025.