Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos

doi:10.1109/CVPR52729.2023.00016

	Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos
	Liao Wang1 ; Qiang Hu1 ; Qihan He1 ; Ziyu Wang1 ; Jingyi Yu1 ; Tinne Tuytelaars 2; Lan Xu1 ; Minye Wu 2
	2023
会议录名称	THE IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION(CVPR)
ISSN	1063-6919
卷号	2023-June
页码	76-87
发表状态	已发表
DOI	10.1109/CVPR52729.2023.00016
摘要	The success of the Neural Radiance Fields (NeRFs) for modeling and free-view rendering static objects has in-spired numerous attempts on dynamic scenes. Current techniques that utilize neural rendering for facilitating free-view videos (FVVs) are restricted to either offline rendering or are capable of processing only brief sequences with minimal motion. In this paper, we present a novel technique, Residual Radiance Field or ReRF, as a highly com-pact neural representation to achieve real-time FVV ren-dering on long-duration dynamic scenes. ReRF explicitly models the residual information between adjacent times-tamps in the spatial-temporal feature space, with a global coordinate-based tiny MLP as the feature decoder. Specif-ically, ReRF employs a compact motion grid along with a residual feature grid to exploit inter-frame feature similar-ities. We show such a strategy can handle large motions without sacrificing quality. We further present a sequential training scheme to maintain the smoothness and the spar-sity of the motion/residual grids. Based on ReRF, we design a special FVV codec that achieves three orders of magni-tudes compression rate and provides a companion ReRF player to support online streaming of long-duration FVVs of dynamic scenes. Extensive experiments demonstrate the effectiveness of ReRF for compactly representing dynamic radiance fields, enabling an unprecedented free-viewpoint viewing experience in speed and quality. © 2023 IEEE.
会议录编者/会议主办者	Amazon Science ; Ant Research ; Cruise ; et al. ; Google ; Lambda
关键词	Humans: Face body pose gesture movement
会议名称	2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
会议地点	Vancouver, BC, Canada
会议日期	17-24 June 2023
URL	查看原文
收录类别	EI
语种	英语
出版者	IEEE Computer Society
EI入藏号	20234114868051
原始文献类型	Conference article (CA)
来源库	IEEE
引用统计	正在获取...
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/287891
专题	信息科学与技术学院_博士生信息科学与技术学院_PI研究组_虞晶怡组信息科学与技术学院_硕士生信息科学与技术学院_PI研究组_许岚组
作者单位	1.ShanghaiTech University 2.KU Leuven
第一作者单位	上海科技大学
第一作者的第一单位	上海科技大学
推荐引用方式 GB/T 7714	Liao Wang,Qiang Hu,Qihan He,et al. Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos[C]//Amazon Science, Ant Research, Cruise, et al., Google, Lambda:IEEE Computer Society,2023:76-87.