Instant-NVR: Instant Neural Volumetric Rendering for Human-object Interactions from Monocular RGBD Stream

doi:10.1109/CVPR52729.2023.00065

	Instant-NVR: Instant Neural Volumetric Rendering for Human-object Interactions from Monocular RGBD Stream
	Yuheng Jiang1 ; Kaixin Yao1 ; Zhuo Su 2; Zhehao Shen1 ; Haimin Luo1 ; Lan Xu1
	2023-06-17
会议录名称	2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)
ISSN	1063-6919
发表状态	已发表
DOI	10.1109/CVPR52729.2023.00065
摘要	Convenient 4D modeling of human-object interactions is essential for numerous applications. However, monocular tracking and rendering of complex interaction scenarios remain challenging. In this paper, we propose Instant-NVR, a neural approach for instant volumetric human-object tracking and rendering using a single RGBD camera. It bridges traditional non-rigid tracking with recent instant radiance field techniques via a multi-thread tracking-rendering mechanism. In the tracking front-end, we adopt a robust human-object capture scheme to provide sufficient motion priors. We further introduce a separated instant neural representation with a novel hybrid deformation module for the interacting scene. We also provide an on-the-fly reconstruction scheme of the dynamic/static radiance fields via efficient motion-prior searching. Moreover, we introduce an online key frame selection scheme and a rendering-aware refinement strategy to significantly improve the appearance details for online novel-view synthesis. Extensive experiments demonstrate the effectiveness and efficiency of our approach for the instant generation of human-object radiance fields on the fly, notably achieving real-time photo-realistic novel view synthesis under complex human-object interactions. Project page: https://nowheretrix.github.io/Instant-NVR/.
关键词	Humans: Face body pose gesture movement
会议地点	Vancouver, BC, Canada
会议日期	17-24 June 2023
URL	查看原文
来源库	IEEE
引用统计	正在获取...
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/333412
专题	信息科学与技术学院_博士生信息科学与技术学院_本科生信息科学与技术学院_PI研究组_许岚组
作者单位	1.ShanghaiTech University 2.Pico IDL, ByteDance
第一作者单位	上海科技大学
第一作者的第一单位	上海科技大学
推荐引用方式 GB/T 7714	Yuheng Jiang,Kaixin Yao,Zhuo Su,et al. Instant-NVR: Instant Neural Volumetric Rendering for Human-object Interactions from Monocular RGBD Stream[C],2023.