LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation
2023-10-06
会议录名称2023 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV)
ISSN1550-5499
发表状态已发表
DOI10.1109/ICCV51070.2023.01902
摘要Gestures are non-verbal but important behaviors accompanying people’s speech. While previous methods are able to generate speech rhythm-synchronized gestures, the semantic context of the speech is generally lacking in the gesticulations. Although semantic gestures do not occur very regularly in human speech, they are indeed the key for the audience to understand the speech context in a more immersive environment. Hence, we introduce LivelySpeaker, a framework that realizes semantics-aware co-speech gesture generation and offers several control handles. In particular, our method decouples the task into two stages: script-based gesture generation and audio-guided rhythm refinement. Specifically, the script-based gesture generation leverages the pre-trained CLIP text embeddings as the guidance for generating gestures that are highly semantically aligned with the script. Then, we devise a simple but effective diffusion-based gesture generation backbone simply using pure MLPs, that is conditioned on only audio signals and learns to gesticulate with realistic motions. We utilize such powerful prior to rhyme the script-guided gestures with the audio signals, notably in a zero-shot setting. Our novel two-stage generation framework also enables several applications, such as changing the gesticulation style, editing the co-speech gestures via textual prompting, and controlling the semantic awareness and rhythm alignment with guided diffusion. Extensive experiments demonstrate the advantages of the proposed framework over competing methods. In addition, our core diffusion-based generative model also achieves state-of-the-art performance on two benchmarks. The code and model will be released to facilitate future research.
关键词Interpolation Computer vision Codes Semantics Benchmark testing Rhythm Generators
会议地点Paris, France
会议日期1-6 Oct. 2023
URL查看原文
来源库IEEE
引用统计
正在获取...
文献类型会议论文
条目标识符https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/354918
专题信息科学与技术学院_硕士生
信息科学与技术学院_PI研究组_高盛华组
作者单位
1.ShanghaiTech University
2.Tencent AI Lab
3.Intellindust
4.INRIA
第一作者单位上海科技大学
第一作者的第一单位上海科技大学
推荐引用方式
GB/T 7714
Yihao Zhi,Xiaodong Cun,Xuelin Chen,et al. LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation[C],2023.
条目包含的文件 下载所有文件
文件名称/大小 文献类型 版本类型 开放类型 使用许可
个性服务
查看访问统计
谷歌学术
谷歌学术中相似的文章
[Yihao Zhi]的文章
[Xiaodong Cun]的文章
[Xuelin Chen]的文章
百度学术
百度学术中相似的文章
[Yihao Zhi]的文章
[Xiaodong Cun]的文章
[Xuelin Chen]的文章
必应学术
必应学术中相似的文章
[Yihao Zhi]的文章
[Xiaodong Cun]的文章
[Xuelin Chen]的文章
相关权益政策
暂无数据
收藏/分享
文件名: 10.1109@ICCV51070.2023.01902.pdf
格式: Adobe PDF
所有评论 (0)
暂无评论
 

除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。