ShanghaiTech University Knowledge Management System
| Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions | |
| 2021-10-17 | |
| Source Publication | MM 2021 - PROCEEDINGS OF THE 29TH ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA
 |
| Pages | 4651-4660 |
| DOI | 10.1145/3474085.3475442 |
| Abstract | 4D reconstruction of human-object interaction is critical for immersive VR/AR experience and human activity understanding. Recent advances still fail to recover fine geometry and texture results from sparse RGB inputs, especially under challenging human-object interactions scenarios. In this paper, we propose a neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of both human and objects under challenging interaction scenarios in arbitrary novel views, from only sparse RGB streams. To deal with complex occlusions raised by human-object interactions, we adopt a layer-wise scene decoupling strategy and perform volumetric reconstruction and neural rendering of the human and object. Specifically, for geometry reconstruction, we propose an interaction-aware human-object capture scheme that jointly considers the human reconstruction and object reconstruction with their correlations. Occlusion-aware human reconstruction and robust human-aware object tracking are proposed for consistent 4D human-object dynamic reconstruction. For neural texture rendering, we propose a layer-wise human-object rendering scheme, which combines direction-aware neural blending weight learning and spatial-temporal texture completion to provide high-resolution and photo-realistic texture results in the occluded scenarios. Extensive experiments demonstrate the effectiveness of our approach to achieve high-quality geometry and texture reconstruction in free viewpoints for challenging human-object interactions. © 2021 ACM. |
| Author of Source | ACM SIGMM |
| Keyword | Blending Geometry Multilayer neural networks Rendering (computer graphics) Dynamic reconstruction Free viewpoint Geometry reconstruction High quality Human object interaction Implicit reconstruction Layer wise Neural rendering Performance Photo realistic |
| Conference Name | 29th ACM International Conference on Multimedia, MM 2021 |
| Conference Place | Virtual, Online, China |
| Conference Date | October 20, 2021 - October 24, 2021 |
| URL | 查看原文 |
| Indexed By | EI |
| Language | 英语 |
| Publisher | Association for Computing Machinery, Inc |
| EI Accession Number | 20214711200139 |
| EI Keywords | Textures |
| EI Classification Number | 723.2 Data Processing and Image Processing ; 723.5 Computer Applications ; 802.3 Chemical Operations ; 921 Mathematics |
| Original Document Type | Conference article (CA) |
| Citation statistics | |
| Document Type | 会议论文 |
| Identifier | https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/133567 |
| Collection | 信息科学与技术学院_PI研究组_许岚组 信息科学与技术学院_PI研究组_虞晶怡组 信息科学与技术学院_硕士生 信息科学与技术学院_本科生 信息科学与技术学院_博士生 信息科学与技术学院_PI研究组_汪婧雅组 |
| Corresponding Author | Wang, Jingya |
| Affiliation | 1.ShanghaiTech University, School of Information Science and Technology, Shanghai Engineering Research Center of Intelligent Vision and Imaging, China; 2.Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, China; 3.University of Chinese Academy of Sciences, China |
| First Author Affilication | School of Information Science and Technology |
| Corresponding Author Affilication | School of Information Science and Technology |
| First Signature Affilication | School of Information Science and Technology |
| Recommended Citation GB/T 7714 | Sun, Guoxing,Chen, Xin,Chen, Yizhang,et al. Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions[C]//ACM SIGMM:Association for Computing Machinery, Inc,2021:4651-4660. |
| Files in This Item: | Download All | |||||
| File Name/Size | DocType | Version | Access | License | ||
Edit Comment
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.