Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

doi:10.1145/3474085.3475442

	Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions
	Sun, Guoxing1 ; Chen, Xin 2,3; Chen, Yizhang1 ; Pang, Anqi1,2,3 ; Lin, Pei1 ; Jiang, Yuheng1 ; Xu, Lan1 ; Yu, Jingyi1 ; Wang, Jingya1
	2021-10-17
会议录名称	MM 2021 - PROCEEDINGS OF THE 29TH ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA
页码	4651-4660
DOI	10.1145/3474085.3475442
摘要	4D reconstruction of human-object interaction is critical for immersive VR/AR experience and human activity understanding. Recent advances still fail to recover fine geometry and texture results from sparse RGB inputs, especially under challenging human-object interactions scenarios. In this paper, we propose a neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of both human and objects under challenging interaction scenarios in arbitrary novel views, from only sparse RGB streams. To deal with complex occlusions raised by human-object interactions, we adopt a layer-wise scene decoupling strategy and perform volumetric reconstruction and neural rendering of the human and object. Specifically, for geometry reconstruction, we propose an interaction-aware human-object capture scheme that jointly considers the human reconstruction and object reconstruction with their correlations. Occlusion-aware human reconstruction and robust human-aware object tracking are proposed for consistent 4D human-object dynamic reconstruction. For neural texture rendering, we propose a layer-wise human-object rendering scheme, which combines direction-aware neural blending weight learning and spatial-temporal texture completion to provide high-resolution and photo-realistic texture results in the occluded scenarios. Extensive experiments demonstrate the effectiveness of our approach to achieve high-quality geometry and texture reconstruction in free viewpoints for challenging human-object interactions. © 2021 ACM.
会议录编者/会议主办者	ACM SIGMM
关键词	Blending Geometry Multilayer neural networks Rendering (computer graphics) Dynamic reconstruction Free viewpoint Geometry reconstruction High quality Human object interaction Implicit reconstruction Layer wise Neural rendering Performance Photo realistic
会议名称	29th ACM International Conference on Multimedia, MM 2021
会议地点	Virtual, Online, China
会议日期	October 20, 2021 - October 24, 2021
URL	查看原文
收录类别	EI
语种	英语
出版者	Association for Computing Machinery, Inc
EI入藏号	20214711200139
EI主题词	Textures
EI分类号	723.2 Data Processing and Image Processing ; 723.5 Computer Applications ; 802.3 Chemical Operations ; 921 Mathematics
原始文献类型	Conference article (CA)
引用统计
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/133567
专题	信息科学与技术学院_PI研究组_汪婧雅组信息科学与技术学院_PI研究组_虞晶怡组信息科学与技术学院_硕士生信息科学与技术学院_本科生信息科学与技术学院_博士生信息科学与技术学院_PI研究组_许岚组
通讯作者	Wang, Jingya
作者单位	1.ShanghaiTech University, School of Information Science and Technology, Shanghai Engineering Research Center of Intelligent Vision and Imaging, China; 2.Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, China; 3.University of Chinese Academy of Sciences, China
第一作者单位	信息科学与技术学院
通讯作者单位	信息科学与技术学院
第一作者的第一单位	信息科学与技术学院
推荐引用方式 GB/T 7714	Sun, Guoxing,Chen, Xin,Chen, Yizhang,et al. Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions[C]//ACM SIGMM:Association for Computing Machinery, Inc,2021:4651-4660.