IButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering

doi:10.1145/3474085.3475412

	IButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering
	Wang, Liao1 ; Wang, Ziyu1 ; Lin, Pei1 ; Jiang, Yuheng1 ; Suo, Xin1 ; Wu, Minye1 ; Xu, Lan1 ; Yu, Jingyi2
	2021-10-17
会议录名称	MM 2021 - PROCEEDINGS OF THE 29TH ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA
页码	4641-4650
DOI	10.1145/3474085.3475412
摘要	Generating "bullet-time"effects of human free-viewpoint videos is critical for immersive visual effects and VR/AR experience. Recent neural advances still lack the controllable and interactive bullet-time design ability for human free-viewpoint rendering, especially under the real-time, dynamic and general setting for our trajectory-aware task. To fill this gap, in this paper we propose a neural interactive bullet-time generator (iButter) for photo-realistic human free-viewpoint rendering from dense RGB streams, which enables flexible and interactive design for human bullet-time visual effects. Our iButter approach consists of a real-time preview and design stage as well as a trajectory-aware refinement stage. During preview, we propose an interactive bullet-time design approach by extending the NeRF rendering to a real-time and dynamic setting and getting rid of the tedious per-scene training. To this end, our bullet-time design stage utilizes a hybrid training set, light-weight network design and an efficient silhouette-based sampling strategy. During refinement, we introduce an efficient trajectory-aware scheme within 20 minutes, which jointly encodes the spatial, temporal consistency and semantic cues along the designed trajectory, achieving photo-realistic bullet-time viewing experience of human activities. Extensive experiments demonstrate the effectiveness of our approach for convenient interactive bullet-time design and photo-realistic human free-viewpoint video generation. © 2021 ACM.
会议录编者/会议主办者	ACM SIGMM
关键词	Rendering (computer graphics) Semantics Bullet time Design stage Free viewpoint renderings Free viewpoint video Neural rendering Neural representations Novel view synthesis Photo realistic Time design Visual effects
会议名称	29th ACM International Conference on Multimedia, MM 2021
会议地点	Virtual, Online, China
会议日期	October 20, 2021 - October 24, 2021
URL	查看原文
收录类别	EI
语种	英语
出版者	Association for Computing Machinery, Inc
EI入藏号	20214711200059
EI主题词	Trajectories
EI分类号	723.2 Data Processing and Image Processing ; 723.5 Computer Applications
原始文献类型	Conference article (CA)
引用统计	正在获取...
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/133507
专题	信息科学与技术学院_硕士生信息科学与技术学院_PI研究组_虞晶怡组信息科学与技术学院_博士生信息科学与技术学院_PI研究组_许岚组
作者单位	1.ShanghaiTech University, Shanghai, China; 2.Shanghai Engineering Research Center of Intelligent Vision and Imaging, School of Information Science and Technology, ShanghaiTech University, Shanghai, China
第一作者单位	上海科技大学
第一作者的第一单位	上海科技大学
推荐引用方式 GB/T 7714	Wang, Liao,Wang, Ziyu,Lin, Pei,et al. IButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering[C]//ACM SIGMM:Association for Computing Machinery, Inc,2021:4641-4650.