ShanghaiTech University Knowledge Management System
| Instant Facial Gaussians Translator for Relightable and Interactable Facial Rendering | |
| 2024-09-30 | |
| 状态 | 已发表 |
| 摘要 | We propose GauFace, a novel Gaussian Splatting representation, tailored for efficient animation and rendering of physically-based facial assets. Leveraging strong geometric priors and constrained optimization, GauFace ensures a neat and structured Gaussian representation, delivering high fidelity and real-time facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform. Then, we introduce TransGS, a diffusion transformer that instantly translates physically-based facial assets into the corresponding GauFace representations. Specifically, we adopt a patch-based pipeline to handle the vast number of Gaussians effectively. We also introduce a novel pixel-aligned sampling scheme with UV positional encoding to ensure the throughput and rendering quality of GauFace assets generated by our TransGS. Once trained, TransGS can instantly translate facial assets with lighting conditions to GauFace representation, With the rich conditioning modalities, it also enables editing and animation capabilities reminiscent of traditional CG pipelines. We conduct extensive evaluations and user studies, compared to traditional offline and online renderers, as well as recent neural rendering methods, which demonstrate the superior performance of our approach for facial asset rendering. We also showcase diverse immersive applications of facial assets using our TransGS approach and GauFace representation, across various platforms like PCs, phones and even VR headsets. |
| 语种 | 英语 |
| DOI | arXiv:2409.07441 |
| 相关网址 | 查看原文 |
| 出处 | Arxiv |
| 收录类别 | PPRN.PPRN |
| WOS记录号 | PPRN:91839866 |
| WOS类目 | Computer Science, Software Engineering |
| 文献类型 | 预印本 |
| 条目标识符 | https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/433529 |
| 专题 | 信息科学与技术学院_PI研究组_许岚组 信息科学与技术学院_PI研究组_虞晶怡组 信息科学与技术学院_硕士生 信息科学与技术学院_本科生 信息科学与技术学院_博士生 |
| 通讯作者 | Komura, Taku |
| 作者单位 | 1.Univ Hong Kong, Hong Kong, Peoples R China 2.Deemos Technol Co Ltd, Shanghai, Peoples R China 3.ShanghaiTech Univ, Shanghai, Peoples R China 4.Adobe Res, Seattle, WA, USA |
| 推荐引用方式 GB/T 7714 | Qin, Dafei,Lin, Hongyang,Zhang, Qixuan,et al. Instant Facial Gaussians Translator for Relightable and Interactable Facial Rendering. 2024. |
| 条目包含的文件 | ||||||
| 条目无相关文件。 | ||||||
修改评论
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。