Video-driven Neural Physically-based Facial Asset for Production

doi:10.1145/3550454.3555445

	Video-driven Neural Physically-based Facial Asset for Production
	Zhang, Longwen1,2 ; Zeng, Chuxiao 1,2; Zhang, Qixuan1,2 ; Lin, Hongyang1,2 ; Cao, Ruixiang1,2 ; Yang, Wei 3; Xu, Lan1,2,4 ; Yu, Jingyi1,2,4
	2022-12-01
发表期刊	ACM TRANSACTIONS ON GRAPHICS
ISSN	0730-0301
EISSN	1557-7368
卷号	41 期号:6
发表状态	已发表
DOI	10.1145/3550454.3555445
摘要	Production-level workflows for producing convincing 3D dynamic human faces have long relied on an assortment of labor-intensive tools for geometry and texture generation, motion capture and rigging, and expression synthesis. Recent neural approaches automate individual components but the corresponding latent representations cannot provide artists with explicit controls as in conventional tools. In this paper, we present a new learning-based, video-driven approach for generating dynamic facial geometries with high-quality physically-based assets. For data collection, we construct a hybrid multiview-photometric capture stage, coupling with ultra-fast video cameras to obtain raw 3D facial assets. We then set out to model the facial expression, geometry and physically-based textures using separate VAEs where we impose a global MLP based expression mapping across the latent spaces of respective networks, to preserve characteristics across respective attributes. We also model the delta information as wrinkle maps for the physically-based textures, achieving high-quality 4K dynamic textures. We demonstrate our approach in high-fidelity performer-specific facial capture and cross-identity facial motion retargeting. In addition, our multi-VAE-based neural asset, along with the fast adaptation schemes, can also be deployed to handle in-the-wild videos. Besides, we motivate the utility of our explicit facial disentangling strategy by providing various promising physically-based editing results with high realism. Comprehensive experiments show that our technique provides higher accuracy and visual fidelity than previous video-driven facial reconstruction and animation methods.
关键词	Physically-Based Face Rendering Facial Modeling Digital Human Video-Driven Animation
URL	查看原文
收录类别	SCI ; EI ; CPCI-S
语种	英语
资助项目	Shanghai YangFan Program[21YF1429500] ; Shanghai Local College Capacity Building Program[22010502800] ; NSFC programs["61976138","61977047"] ; National Key Research and Development Program[2018YFB2100500] ; STCSM[2015F0203000-06] ; SHMEC[2019-01-07-00-01-E00003]
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:000891651900028
出版者	ASSOC COMPUTING MACHINERY
引用统计
文献类型	期刊论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/266605
专题	信息科学与技术学院_本科生信息科学与技术学院_PI研究组_虞晶怡组信息科学与技术学院_硕士生信息科学与技术学院_博士生信息科学与技术学院_PI研究组_许岚组
通讯作者	Xu, Lan; Yu, Jingyi
作者单位	1.ShanghaiTech Univ, Shanghai, Peoples R China 2.Deemos Technol Co Ltd, Shanghai, Peoples R China 3.Huazhong Univ Sci & Technol, Wuhan, Peoples R China 4.Shanghai Engn Res Ctr Intelligent Vis & Imaging, Shanghai, Peoples R China
第一作者单位	上海科技大学
通讯作者单位	上海科技大学
第一作者的第一单位	上海科技大学
推荐引用方式 GB/T 7714	Zhang, Longwen,Zeng, Chuxiao,Zhang, Qixuan,et al. Video-driven Neural Physically-based Facial Asset for Production[J]. ACM TRANSACTIONS ON GRAPHICS,2022,41(6).
APA	Zhang, Longwen.,Zeng, Chuxiao.,Zhang, Qixuan.,Lin, Hongyang.,Cao, Ruixiang.,...&Yu, Jingyi.(2022).Video-driven Neural Physically-based Facial Asset for Production.ACM TRANSACTIONS ON GRAPHICS,41(6).
MLA	Zhang, Longwen,et al."Video-driven Neural Physically-based Facial Asset for Production".ACM TRANSACTIONS ON GRAPHICS 41.6(2022).