SportsCap: Monocular 3D Human Motion Capture and Fine-Grained Understanding in Challenging Sports Videos

doi:10.1007/s11263-021-01486-4

	SportsCap: Monocular 3D Human Motion Capture and Fine-Grained Understanding in Challenging Sports Videos
	Chen, Xin1,2,3 ; Pang, Anqi1,2,3 ; Yang, Wei 4; Ma, Yuexin1 ; Xu, Lan1 ; Yu, Jingyi1
	2021-10
发表期刊	INTERNATIONAL JOURNAL OF COMPUTER VISION
ISSN	0920-5691
EISSN	1573-1405
卷号	129 期号:10 页码:2846-2864
DOI	10.1007/s11263-021-01486-4
摘要	Markerless motion capture and understanding of professional non-daily human movements is an important yet unsolved task, which suffers from complex motion patterns and severe self-occlusion, especially for the monocular setting. In this paper, we propose SportsCap-the first approach for simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input. Our approach utilizes the semantic and temporally structured sub-motion prior in the embedding space for motion capture and understanding in a data-driven multi-task manner. To enable robust capture under complex motion patterns, we propose an effective motion embedding module to recover both the implicit motion embedding and explicit 3D motion details via a corresponding mapping function as well as a sub-motion classifier. Based on such hybrid motion information, we introduce a multi-stream spatial-temporal graph convolutional network to predict the fine-grained semantic action attributes, and adopt a semantic attribute mapping block to assemble various correlated action attributes into a high-level action label for the overall detailed understanding of the whole sequence, so as to enable various applications like action assessment or motion scoring. Comprehensive experiments on both public and our proposed datasets show that with a challenging monocular sports video input, our novel approach not only significantly improves the accuracy of 3D human motion capture, but also recovers accurate fine-grained semantic action attribute.
关键词	Human modeling 3D motion capture Motion understanding
URL	查看原文
收录类别	SCIE ; EI
语种	英语
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:000681171700003
出版者	SPRINGER
原始文献类型	Article; Early Access
引用统计
文献类型	期刊论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/127834
专题	信息科学与技术学院_PI研究组_马月昕信息科学与技术学院_PI研究组_虞晶怡组信息科学与技术学院_博士生信息科学与技术学院_PI研究组_许岚组
通讯作者	Xu, Lan; Yu, Jingyi
作者单位	1.ShanghaiTech Univ, Sch Informat Sci & Technol, Shanghai Engn Res Ctr Intelligent Vis & Imaging, Shanghai, Peoples R China; 2.Chinese Acad Sci, Shanghai Inst Microsyst & Informat Technol, Shanghai, Peoples R China; 3.Univ Chinese Acad Sci, Shanghai, Peoples R China; 4.Huazhong Univ Sci & Technol, Sch Comp Sci & Technol, Wuhan, Peoples R China
第一作者单位	信息科学与技术学院
通讯作者单位	信息科学与技术学院
第一作者的第一单位	信息科学与技术学院
推荐引用方式 GB/T 7714	Chen, Xin,Pang, Anqi,Yang, Wei,et al. SportsCap: Monocular 3D Human Motion Capture and Fine-Grained Understanding in Challenging Sports Videos[J]. INTERNATIONAL JOURNAL OF COMPUTER VISION,2021,129(10):2846-2864.
APA	Chen, Xin,Pang, Anqi,Yang, Wei,Ma, Yuexin,Xu, Lan,&Yu, Jingyi.(2021).SportsCap: Monocular 3D Human Motion Capture and Fine-Grained Understanding in Challenging Sports Videos.INTERNATIONAL JOURNAL OF COMPUTER VISION,129(10),2846-2864.
MLA	Chen, Xin,et al."SportsCap: Monocular 3D Human Motion Capture and Fine-Grained Understanding in Challenging Sports Videos".INTERNATIONAL JOURNAL OF COMPUTER VISION 129.10(2021):2846-2864.