BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video

doi:arXiv:2502.08297

	BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video
	Hong, Yu1,2 ; Wu, Yize1 ; Shen, Zhehao1,2 ; Guo, Chengcheng 1,2; Jiang, Yuheng1 ; Zhang, Yingliang 3; Yu, Jingyi1 ; Xu, Lan 1
	2025-02-12
状态	已发表
摘要	Volumetric video enables immersive experiences by capturing dynamic 3D scenes, enabling diverse applications for virtual reality, education, and telepresence. However, traditional methods struggle with fixed lighting conditions, while neural approaches face trade-offs in efficiency, quality, or adaptability for relightable scenarios. To address these limitations, we present BEAM, a novel pipeline that bridges 4D Gaussian representations with physically-based rendering (PBR) to produce high-quality, relightable volumetric videos from multi-view RGB footage. BEAM recovers detailed geometry and PBR properties via a series of available Gaussian-based techniques. It first combines Gaussian-based performance tracking with geometry-aware rasterization in a coarse-to-fine optimization framework to recover spatially and temporally consistent geometries. We further enhance Gaussian attributes by incorporating PBR properties step by step. We generate roughness via a multi-view-conditioned diffusion model, and then derive AO and base color using a 2D-to-3D strategy, incorporating a tailored Gaussian-based ray tracer for efficient visibility computation. Once recovered, these dynamic, relightable assets integrate seamlessly into traditional CG pipelines, supporting real-time rendering with deferred shading and offline rendering with ray tracing. By offering realistic, lifelike visualizations under diverse lighting conditions, BEAM opens new possibilities for interactive entertainment, storytelling, and creative visualization.
语种	英语
DOI	arXiv:2502.08297
相关网址	查看原文
出处	Arxiv
收录类别	PPRN.PPRN
WOS记录号	PPRN:121541708
WOS类目	Computer Science, Software Engineering
文献类型	预印本
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/514096
专题	信息科学与技术学院_本科生信息科学与技术学院_PI研究组_虞晶怡组信息科学与技术学院_硕士生信息科学与技术学院_博士生
通讯作者	Hong, Yu
作者单位	1.ShanghaiTech Univ, Shanghai, Peoples R China 2.NeuDim, Shanghai, Peoples R China 3.DGene, Shanghai, Peoples R China
推荐引用方式 GB/T 7714	Hong, Yu,Wu, Yize,Shen, Zhehao,et al. BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video. 2025.