I'M HOI: Inertia-Aware Monocular Capture of 3D Human-Object Interactions

doi:10.1109/CVPR52733.2024.00076

	I'M HOI: Inertia-Aware Monocular Capture of 3D Human-Object Interactions
	Chengfeng Zhao1 ; Juze Zhang1,2,3 ; Jiashen Du1 ; Ziwei Shan1 ; Junye Wang1 ; Jingyi Yu1 ; Jingya Wang1 ; Lan Xu1
	2024
会议录名称	2024 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)
ISSN	1063-6919
页码	729-741
发表状态	已发表
DOI	10.1109/CVPR52733.2024.00076
摘要	We are living in a world surrounded by diverse and “smart” devices with rich modalities of sensing ability. Conveniently capturing the interactions between us humans and these objects remains far-reaching. In this paper, we present I'm-HOI, a monocular scheme to faithfully capture the 3D motions of both the human and object in a novel setting: using a minimal amount of RGB camera and object-mounted Inertial Measurement Unit (IMU). It combines general motion inference and category-aware refinement. For the former, we introduce a holistic human-object tracking method to fuse the IMU signals and the RGB stream and progressively recover the human motions and subsequently the companion object motions. For the latter, we tailor a category-aware motion diffusion model, which is conditioned on both the raw IMU observations and the results from the previous stage under over-parameterization representation. It significantly refines the initial results and generates vivid body, hand, and object motions. Moreover, we contribute a large dataset with ground truth human and object motions, dense RGB inputs, and rich object-mounted IMU measurements. Exten-sive experiments demonstrate the effectiveness of I'm-HOI under a hybrid capture setting. Our dataset and code will be released to the community.
关键词	3D modeling Holmium alloys Human engineering Motion tracking Object detection Sensor data fusion Three dimensional computer graphics 3D motion Human motions Human-object interaction Inertial measurements units Motion capture Object motion RGB cameras Sensing abilities Sensor fusion Smart devices
会议地点	Seattle, WA, USA
会议日期	16-22 June 2024
URL	查看原文
收录类别	EI
语种	英语
出版者	IEEE Computer Society
EI入藏号	20250917942119
EI主题词	Motion capture
EI分类号	101.5 Ergonomics and Human Factors Engineering ; 1106.2 Data Handling and Data Processing ; 1106.3.1 Image Processing ; 1106.8 Computer Vision ; 1201.12 Modeling and Simulation ; 202.7.2 Rare Earth Materials ; 902.1 Engineering Graphics
原始文献类型	Conference article (CA)
来源库	IEEE
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/424436
专题	信息科学与技术学院_PI研究组_许岚组信息科学与技术学院_PI研究组_虞晶怡组信息科学与技术学院_硕士生信息科学与技术学院_本科生信息科学与技术学院_博士生信息科学与技术学院_PI研究组_汪婧雅组
作者单位	1.ShanghaiTech University 2.Shanghai Advanced Research Institute, Chinese Academy of Sciences 3.University of Chinese Academy of Sciences
第一作者单位	上海科技大学
第一作者的第一单位	上海科技大学
推荐引用方式 GB/T 7714	Chengfeng Zhao,Juze Zhang,Jiashen Du,et al. I'M HOI: Inertia-Aware Monocular Capture of 3D Human-Object Interactions[C]:IEEE Computer Society,2024:729-741.