Liquid Warping GAN with Attention: A Unified Framework for Human Image Synthesis

doi:10.1109/TPAMI.2021.3078270

	Liquid Warping GAN with Attention: A Unified Framework for Human Image Synthesis
	Wen Liu1 ; Zhixin Piao 1; Zhi Tu1 ; Wenhan Luo 2; Lin Ma 3; Shenghua Gao4
	2022-09-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
EISSN	1939-3539
卷号	44 期号:9
发表状态	已发表
DOI	10.1109/TPAMI.2021.3078270
摘要	We tackle human image synthesis, including motion imitation, appearance transfer, and novel view synthesis, within a unified framework. The model, once being trained, can be used to handle all these tasks. We propose to use a 3D body mesh recovery module to disentangle the pose and shape. It can not only model the joint location and rotation but also characterize the personalized body shape. To preserve the source information, such as texture, style, color, and face identity, we propose an Attentional Liquid Warping GAN with Attentional Liquid Warping Block (AttLWB) that propagates the source information in both image and feature spaces to the synthesized reference. Specifically, the source features are extracted by a denoising convolutional auto-encoder for characterizing the source identity well. Our proposed method can support a more flexible warping from multiple sources. To further improve the generalization ability of the unseen source images, a one/few-shot adversarial learning is applied in a self-supervised way to generate high-resolution 512 x 512 and 1024 x 1024 results. Also, we build a new dataset, namely iPER, for the evaluation of these three tasks. Extensive experiments demonstrate the effectiveness of our methods in terms of preserving face identity, shape consistency, and clothes details. IEEE
关键词	Liquids Textures Adversarial learning Generalization ability Image synthesis Motion imitations Multiple source Novel view synthesis Recovery modules Unified framework
URL	查看原文
收录类别	EI ; SCIE ; SCI
语种	英语
资助项目	National Key R&D Program of China[2018AAA0100704] ; NSFC[61932020] ; Science and Technology Commission of Shanghai Municipality[20ZR1436000]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000836666600047
出版者	IEEE Computer Society
EI入藏号	20212010370763
EI主题词	Image enhancement
原始文献类型	Article in Press
来源库	IEEE
引用统计
文献类型	期刊论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/135720
专题	信息科学与技术学院信息科学与技术学院_PI研究组_高盛华组信息科学与技术学院_本科生信息科学与技术学院_博士生
作者单位	1.School of Information Science and Technology, ShanghaiTech University, Shanghai, China 2.Tencent, Shenzhen, China 3.Meituan, Beijing, China 4.Engineering Research Center of Intelligent Vision and Imaging, ShanghaiTech University, Shanghai, China
第一作者单位	信息科学与技术学院
第一作者的第一单位	信息科学与技术学院
推荐引用方式 GB/T 7714	Wen Liu,Zhixin Piao,Zhi Tu,et al. Liquid Warping GAN with Attention: A Unified Framework for Human Image Synthesis[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2022,44(9).
APA	Wen Liu,Zhixin Piao,Zhi Tu,Wenhan Luo,Lin Ma,&Shenghua Gao.(2022).Liquid Warping GAN with Attention: A Unified Framework for Human Image Synthesis.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,44(9).
MLA	Wen Liu,et al."Liquid Warping GAN with Attention: A Unified Framework for Human Image Synthesis".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 44.9(2022).