LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation
2023-10-06
Source Publication2023 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV)
ISSN1550-5499
Status已发表
DOI10.1109/ICCV51070.2023.01902
AbstractGestures are non-verbal but important behaviors accompanying people’s speech. While previous methods are able to generate speech rhythm-synchronized gestures, the semantic context of the speech is generally lacking in the gesticulations. Although semantic gestures do not occur very regularly in human speech, they are indeed the key for the audience to understand the speech context in a more immersive environment. Hence, we introduce LivelySpeaker, a framework that realizes semantics-aware co-speech gesture generation and offers several control handles. In particular, our method decouples the task into two stages: script-based gesture generation and audio-guided rhythm refinement. Specifically, the script-based gesture generation leverages the pre-trained CLIP text embeddings as the guidance for generating gestures that are highly semantically aligned with the script. Then, we devise a simple but effective diffusion-based gesture generation backbone simply using pure MLPs, that is conditioned on only audio signals and learns to gesticulate with realistic motions. We utilize such powerful prior to rhyme the script-guided gestures with the audio signals, notably in a zero-shot setting. Our novel two-stage generation framework also enables several applications, such as changing the gesticulation style, editing the co-speech gestures via textual prompting, and controlling the semantic awareness and rhythm alignment with guided diffusion. Extensive experiments demonstrate the advantages of the proposed framework over competing methods. In addition, our core diffusion-based generative model also achieves state-of-the-art performance on two benchmarks. The code and model will be released to facilitate future research.
KeywordInterpolation Computer vision Codes Semantics Benchmark testing Rhythm Generators
Conference PlaceParis, France
Conference Date1-6 Oct. 2023
URL查看原文
Source DataIEEE
Citation statistics
Document Type会议论文
Identifierhttps://kms.shanghaitech.edu.cn/handle/2MSLDSTB/354918
Collection信息科学与技术学院_硕士生
信息科学与技术学院_PI研究组_高盛华组
Affiliation
1.ShanghaiTech University
2.Tencent AI Lab
3.Intellindust
4.INRIA
First Author AffilicationShanghaiTech University
First Signature AffilicationShanghaiTech University
Recommended Citation
GB/T 7714
Yihao Zhi,Xiaodong Cun,Xuelin Chen,et al. LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation[C],2023.
Files in This Item: Download All
File Name/Size DocType Version Access License
Related Services
Usage statistics
Scholar Google
Similar articles in Scholar Google
[Yihao Zhi]'s Articles
[Xiaodong Cun]'s Articles
[Xuelin Chen]'s Articles
Baidu academic
Similar articles in Baidu academic
[Yihao Zhi]'s Articles
[Xiaodong Cun]'s Articles
[Xuelin Chen]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Yihao Zhi]'s Articles
[Xiaodong Cun]'s Articles
[Xuelin Chen]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: 10.1109@ICCV51070.2023.01902.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.