| |||||||
ShanghaiTech University Knowledge Management System
I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions | |
2024-03-30 | |
状态 | 已发表 |
摘要 | We are living in a world surrounded by diverse and "smart" devices with rich modalities of sensing ability. Conveniently capturing the interactions between us humans and these objects remains far-reaching. In this paper, we present I'm-HOI, a monocular scheme to faithfully capture the 3D motions of both the human and object in a novel setting: using a minimal amount of RGB camera and object-mounted Inertial Measurement Unit (IMU). It combines general motion inference and category-aware refinement. For the former, we introduce a holistic human-object tracking method to fuse the IMU signals and the RGB stream and progressively recover the human motions and subsequently the companion object motions. For the latter, we tailor a category-aware motion diffusion model, which is conditioned on both the raw IMU observations and the results from the previous stage under over-parameterization representation. It significantly refines the initial results and generates vivid body, hand, and object motions. Moreover, we contribute a large dataset with ground truth human and object motions, dense RGB inputs, and rich object-mounted IMU measurements. Extensive experiments demonstrate the effectiveness of I'm-HOI under a hybrid capture setting. Our dataset and code will be released to the community. |
DOI | arXiv:2312.08869 |
相关网址 | 查看原文 |
出处 | Arxiv |
WOS记录号 | PPRN:86592444 |
WOS类目 | Computer Science, Software Engineering |
资助项目 | National Key R&D Program of China[ |
文献类型 | 预印本 |
条目标识符 | https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/372936 |
专题 | 信息科学与技术学院_PI研究组_许岚组 信息科学与技术学院_PI研究组_虞晶怡组 信息科学与技术学院_硕士生 信息科学与技术学院_本科生 信息科学与技术学院_博士生 信息科学与技术学院_PI研究组_汪婧雅组 |
通讯作者 | Xu, Lan |
作者单位 | 1.ShanghaiTech Univ, Shanghai, Peoples R China 2.Chinese Acad Sci, Shanghai Adv Res Inst, Shanghai, Peoples R China 3.Univ Chinese Acad Sci, Beijing, Peoples R China |
推荐引用方式 GB/T 7714 | Zhao, Chengfeng,Zhang, Juze,Du, Jiashen,et al. I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions. 2024. |
条目包含的文件 | ||||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 |
修改评论
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。