Monocular Human-Object Reconstruction in the Wild

doi:10.1145/3664647.3681452

	Monocular Human-Object Reconstruction in the Wild
	Huo, Chaofan; Shi, Ye; Wang, Jingya
	2024-10-28
会议录名称	MM 2024 - PROCEEDINGS OF THE 32ND ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA
页码	5547-5555
DOI	10.1145/3664647.3681452
摘要	Learning the prior knowledge of the 3D human-object spatial relation is crucial for reconstructing human-object interaction from images and understanding how humans interact with objects in 3D space. Previous works learn this prior from the latest-released human-object interaction dataset collected in controlled environments. However, due to the domain divergence, these methods are limited by the data that the prior learned from and fail to generalize to real-world data with high diversity. To overcome this limitation, we present a 2D-supervised method that learns the 3D human-object spatial relation prior purely from 2D images in the wild. Our method utilizes a flow-based neural network to learn the prior distribution of the 2D human-object keypoint layout and viewports for each image in the dataset. The effectiveness of the prior learned from 2D images is demonstrated on the human-object reconstruction task by applying the prior to tune the relative pose between the human and the object during the post-optimization stage. To validate and benchmark our method on in-the-wild images, we collect the WildHOI dataset from the YouTube website, which consists of various interactions with 8 objects in real-world scenarios. We conduct the experiments on the indoor BEHAVE dataset and the outdoor WildHOI dataset. The results show that our method achieves almost comparable performance with fully 3D supervised methods on the BEHAVE dataset, even if we have only utilized the 2D layout information, and outperforms previous methods in terms of generality and interaction diversity on in-the-wild images. The code and the dataset are available at https://huochf.github.io/WildHOI/ for research purposes. © 2024 Owner/Author.
会议录编者/会议主办者	ACM SIGMM
关键词	3D modeling Neural networks Three dimensional computer graphics 2D images 3D computer vision 3D spaces Human-object interaction Human-object interaction reconstruction Learn+ Object reconstruction Prior-knowledge Spatial relations Supervised methods
会议名称	32nd ACM International Conference on Multimedia, MM 2024
会议地点	Melbourne, VIC, Australia
会议日期	October 28, 2024 - November 1, 2024
URL	查看原文
收录类别	EI
语种	英语
出版者	Association for Computing Machinery, Inc
EI入藏号	20244817416979
EI主题词	3D reconstruction
EI分类号	101.1 ; 1101 ; 1106.2 ; 1106.8 ; 1201.12 ; 902.1 Engineering Graphics
原始文献类型	Conference article (CA)
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/455178
专题	信息科学与技术学院_硕士生信息科学与技术学院_PI研究组_汪婧雅组信息科学与技术学院_PI研究组_石野组
通讯作者	Wang, Jingya
作者单位	ShanghaiTech University, Shanghai, China
第一作者单位	上海科技大学
通讯作者单位	上海科技大学
第一作者的第一单位	上海科技大学
推荐引用方式 GB/T 7714	Huo, Chaofan,Shi, Ye,Wang, Jingya. Monocular Human-Object Reconstruction in the Wild[C]//ACM SIGMM:Association for Computing Machinery, Inc,2024:5547-5555.