Sophia-in-Audition: Virtual Production with a Robot Performer

doi:10.1145/3664647.3685509

	Sophia-in-Audition: Virtual Production with a Robot Performer
	Zhou, Taotao1,2 ; Xu, Teng1,2 ; Zhang, Dong1,2 ; Jiao, Yuyang1 ; Xu, Peijun1 ; He, Yaoyu1 ; Xu, Lan1 ; Yu, Jingyi1
	2024-10-28
会议录名称	MM 2024 - PROCEEDINGS OF THE 32ND ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA
页码	11184-11193
DOI	10.1145/3664647.3685509
摘要	We present Sophia-in-Audition (SiA), a new frontier in virtual production, by employing the humanoid robot Sophia within an UltraStage environment composed of a controllable lighting dome coupled with multiple cameras. We demonstrate Sophia's capability to replicate iconic film segments, follow real performers, and perform a variety of motions and expressions, showcasing her versatility as a virtual actor. Key to this process is the integration of facial motion transfer algorithms and the UltraStage's controllable lighting and multi-camera setup, enabling dynamic performances that align with the director's vision. Our comprehensive user studies indicate positive audience reception towards Sophia's performances, highlighting her potential to reduce the uncanny valley effect in virtual acting. Additionally, the immersive lighting in dynamic clips was highly rated for its naturalness and its ability to mirror professional film standards. The paper presents a first-of-its-kind multi-view robot performance video dataset with dynamic lighting, offering valuable insights for future enhancements in humanoid robotic performers and virtual production techniques. This research contributes significantly to the field by presenting a unique virtual production setup, developing tools for sophisticated performance control, and providing a comprehensive dataset and user study analysis for diverse applications. © 2024 Owner/Author.
会议录编者/会议主办者	ACM SIGMM
关键词	Audition Intelligent robots Multipurpose robots Rendering (computer graphics) Virtual environments Computational illuminations Humanoid robot Iconics Multi-view capture Multi-views Multiple cameras Neural rendering User study Virtual actors Virtual production
会议名称	32nd ACM International Conference on Multimedia, MM 2024
会议地点	Melbourne, VIC, Australia
会议日期	October 28, 2024 - November 1, 2024
URL	查看原文
收录类别	EI
语种	英语
出版者	Association for Computing Machinery, Inc
EI入藏号	20244817416789
EI主题词	Anthropomorphic robots
EI分类号	101.5 ; 101.6.1 ; 1106.3.1 ; 1106.5 ; 1107.1 ; 731.5 Robotics ; 731.6 Robot Applications ; 942.1.2
原始文献类型	Conference article (CA)
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/455180
专题	信息科学与技术学院_本科生信息科学与技术学院_PI研究组_虞晶怡组信息科学与技术学院_硕士生信息科学与技术学院_博士生信息科学与技术学院_PI研究组_许岚组
通讯作者	Yu, Jingyi
作者单位	1.ShanghaiTech University, Shanghai, China; 2.LumiAni Technology, Shanghai, China
第一作者单位	上海科技大学
通讯作者单位	上海科技大学
第一作者的第一单位	上海科技大学
推荐引用方式 GB/T 7714	Zhou, Taotao,Xu, Teng,Zhang, Dong,et al. Sophia-in-Audition: Virtual Production with a Robot Performer[C]//ACM SIGMM:Association for Computing Machinery, Inc,2024:11184-11193.