EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild

doi:arXiv:2411.14280

	EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild
	Liu, Yumeng 1,2; Long, Xiaoxiao 3; Yang, Zemin2 ; Liu, Yuan 3,4; Habermann, Marc 5; Theobalt, Christian 5; Ma, Yuexin2 ; Wang, Wenping 6
	2024-11-21
状态	已发表
摘要	Our work aims to reconstruct hand-object interactions from a single-view image, which is a fundamental but ill-posed task. Unlike methods that reconstruct from videos, multi-view images, or predefined 3D templates, single-view reconstruction faces significant challenges due to inherent ambiguities and occlusions. These challenges are further amplified by the diverse nature of hand poses and the vast variety of object shapes and sizes. Our key insight is that current foundational models for segmentation, inpainting, and 3D reconstruction robustly generalize to in-the-wild images, which could provide strong visual and geometric priors for reconstructing hand-object interactions. Specifically, given a single image, we first design a novel pipeline to estimate the underlying hand pose and object shape using off-the-shelf large models. Furthermore, with the initial reconstruction, we employ a prior-guided optimization scheme, which optimizes hand pose to comply with 3D physical constraints and the 2D input image content. We perform experiments across several datasets and show that our method consistently outperforms baselines and faithfully reconstructs a diverse set of hand-object interactions.Â
语种	英语
DOI	arXiv:2411.14280
相关网址	查看原文
出处	Arxiv
收录类别	PPRN.PPRN
WOS记录号	PPRN:119320397
WOS类目	Computer Science, Software Engineering
文献类型	预印本
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/467834
专题	信息科学与技术学院_PI研究组_马月昕信息科学与技术学院_硕士生
通讯作者	Long, Xiaoxiao
作者单位	1.HKU, Hong Kong, Peoples R China 2.ShanghaiTech Univ, Shanghai, Peoples R China 3.HKUST, Hong Kong, Peoples R China 4.NTU, Singapore, Singapore 5.Max Planck Inst Informat, Saarbrucken, Germany 6.Texas A&M Univ, College Stn, TX, USA
推荐引用方式 GB/T 7714	Liu, Yumeng,Long, Xiaoxiao,Yang, Zemin,et al. EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild. 2024.