Free-view Face Relighting Using a Hybrid Parametric Neural Model on a SMALL-OLAT Dataset

doi:10.1007/s11263-022-01730-5

	Free-view Face Relighting Using a Hybrid Parametric Neural Model on a SMALL-OLAT Dataset
	Wang, Youjia1 ; He, Kai1,3 ; Zhou, Taotao1 ; Yao, Kaixin1 ; Li, Nianyi 2; Xu, Lan1,4 ; Yu, Jingyi1,4
	2023
发表期刊	INTERNATIONAL JOURNAL OF COMPUTER VISION (IF:11.6[JCR-2023],14.5[5-Year])
ISSN	0920-5691
EISSN	1573-1405
卷号	131 期号:4 页码:1002-1021
发表状态	已发表
DOI	10.1007/s11263-022-01730-5
摘要	The development of neural relighting techniques has by far outpaced the rate of their corresponding training data (e.g., OLAT) generation. For example, high-quality relighting from a single portrait image still requires supervision from comprehensive datasets covering broad diversities in gender, race, complexion, and facial geometry. We present a hybrid parametric neural relighting (PN-Relighting) framework for single portrait relighting, using a much smaller OLAT dataset or SMOLAT. At the core of PN-Relighting, we employ parametric 3D faces coupled with appearance inference and implicit material modelling to enrich SMOLAT for handling in-the-wild images. Specifically, we tailor an appearance inference module to generate detailed geometry and albedo on top of the parametric face and develop a neural rendering module to first construct an implicit material representation from SMOLAT and then conduct self-supervised training on in-the-wild image datasets. Comprehensive experiments show that PN-Relighting produces comparable high-quality relighting to TotalRelighting (Pandey et al., 2021), but with a smaller dataset. It further improves shape estimation and naturally supports free-viewpoint rendering and partial skin material editing. PN-Relighting also serves as a data augmenter to produce rich OLAT datasets beyond the original capture.
关键词	3D Reconstruction Relighting Neural rendering
URL	查看原文
收录类别	SCI ; EI ; SCOPUS
语种	英语
资助项目	Shanghai YangFan Program[21YF1429500] ; Shanghai Local college capacity building program[22010502800] ; NSFC programs["61976138","61977047"] ; National Key Research and Development Program[2018YFB2100500] ; STCSM[2015F0203-000-06] ; SHMEC[2019-01-07-00-01-E00003]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:000910844500002
出版者	SPRINGER
EI入藏号	20230213369321
EI主题词	Image reconstruction
EI分类号	723.2 Data Processing and Image Processing ; 723.5 Computer Applications
原始文献类型	Journal article (JA)
Scopus 记录号	2-s2.0-85145838515
来源库	Scopus
引用统计	正在获取...
文献类型	期刊论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/274937
专题	信息科学与技术学院_PI研究组_许岚组信息科学与技术学院_PI研究组_虞晶怡组信息科学与技术学院_硕士生信息科学与技术学院_本科生信息科学与技术学院_博士生
通讯作者	Wang, Youjia
作者单位	1.ShanghaiTech Univ, Sch Informat Sci & Technol, Shanghai, Peoples R China 2.Clemson Univ, Clemson, SC USA 3.Deemos Technol Co Ltd, Shanghai, Peoples R China 4.Shanghai Frontiers Sci Ctr Human Ctr Artificial In, Shanghai, Peoples R China
第一作者单位	信息科学与技术学院
通讯作者单位	信息科学与技术学院
第一作者的第一单位	信息科学与技术学院
推荐引用方式 GB/T 7714	Wang, Youjia,He, Kai,Zhou, Taotao,et al. Free-view Face Relighting Using a Hybrid Parametric Neural Model on a SMALL-OLAT Dataset[J]. INTERNATIONAL JOURNAL OF COMPUTER VISION,2023,131(4):1002-1021.
APA	Wang, Youjia.,He, Kai.,Zhou, Taotao.,Yao, Kaixin.,Li, Nianyi.,...&Yu, Jingyi.(2023).Free-view Face Relighting Using a Hybrid Parametric Neural Model on a SMALL-OLAT Dataset.INTERNATIONAL JOURNAL OF COMPUTER VISION,131(4),1002-1021.
MLA	Wang, Youjia,et al."Free-view Face Relighting Using a Hybrid Parametric Neural Model on a SMALL-OLAT Dataset".INTERNATIONAL JOURNAL OF COMPUTER VISION 131.4(2023):1002-1021.