Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates

doi:10.1109/ICCV48922.2021.01089

	Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates
	Qian, Shenhan1 ; Tu, Zhi1 ; Zhi, Yihao1 ; Liu, Wen1 ; Gao, Shenghua1,2,3
	2021
会议录名称	PROCEEDINGS OF THE IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), 2021
发表状态	已发表
DOI	10.1109/ICCV48922.2021.01089
摘要	Co-speech gesture generation is to synthesize a gesture sequence that not only looks real but also matches with the input speech audio. Our method generates the movements of a complete upper body, including arms, hands, and the head. Although recent data-driven methods achieve great success, challenges still exist, such as limited variety, poor fidelity, and lack of objective metrics. Motivated by the fact that the speech cannot fully determine the gesture, we design a method that learns a set of gesture template vectors to model the latent conditions, which relieve the ambiguity. For our method, the template vector determines the general appearance of a generated gesture sequence, while the speech audio drives subtle movements of the body, both indispensable for synthesizing a realistic gesture sequence. Due to the intractability of an objective metric for gesture-speech synchronization, we adopt the lip-sync error as a proxy metric to tune and evaluate the synchronization ability of our model. Extensive experiments show the superiority of our method in both objective and subjective evaluations on fidelity and synchronization. (1)
会议名称	18th IEEE/CVF International Conference on Computer Vision (ICCV)
出版地	345 E 47TH ST, NEW YORK, NY 10017 USA
会议地点	null,null,ELECTR NETWORK
会议日期	OCT 11-17, 2021
URL	查看原文
收录类别	SCI ; EI ; CPCI ; CPCI-S
语种	英语
资助项目	National Key R&D Program of China[2018AAA0100704] ; NSFC[61932020] ; Science and Technology Commission of Shanghai Municipality[20ZR1436000]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods
WOS记录号	WOS:000798743201023
出版者	IEEE
EI入藏号	unknown
引用统计
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/178467
专题	信息科学与技术学院_硕士生信息科学与技术学院_PI研究组_高盛华组信息科学与技术学院_本科生信息科学与技术学院_博士生
通讯作者	Gao, Shenghua
作者单位	1.ShanghaiTech Univ, Shanghai, Peoples R China 2.Shanghai Engn Res Ctr Intelligent Vis & Imaging, Shanghai, Peoples R China 3.Shanghai Engn Res Ctr Energy Efficient & Custom A, Shanghai, Peoples R China
第一作者单位	上海科技大学
通讯作者单位	上海科技大学
第一作者的第一单位	上海科技大学
推荐引用方式 GB/T 7714	Qian, Shenhan,Tu, Zhi,Zhi, Yihao,et al. Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:IEEE,2021.