Can language understand depth?

doi:10.1145/3503161.3549201

	Can language understand depth?
	Zhang, Renrui 1; Zeng, Ziyao2 ; Guo, Ziyu 1; Li, Yafeng 1
	2022-10-10
会议录名称	MM 2022 - PROCEEDINGS OF THE 30TH ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA
页码	6868-6874
DOI	10.1145/3503161.3549201
摘要	Besides image classification, Contrastive Language-Image Pre-Training (CLIP) has accomplished extraordinary success for a wide range of vision tasks, including object-level and 3D space understanding. However, it's still challenging to transfer semantic knowledge learned from CLIP into more intricate tasks of quantified targets, such as depth estimation with geometric information. In this paper, we propose to apply CLIP for zero-shot monocular depth estimation, named DepthCLIP. We found that the patches of input image could respond to a certain semantic distance token and then be projected to a quantified depth bin for coarse estimation. Without any training, our DepthCLIP surpasses existing unsupervised methods and even approaches the early fully-supervised networks. To our best knowledge, we are the first to conduct zero-shot adaptation from the semantic language knowledge to quantified downstream tasks and perform zero-shot monocular depth estimation. We hope our work could cast a light on the future research. The code is available at https://github.com/Adonis-galaxy/DepthCLIP. © 2022 ACM.
会议录编者/会议主办者	ACM SIGMM
关键词	Transfer learning Zero-shot learning 3D spaces Contrastive language-image pre-training Depth Estimation Geometric information Images classification Monocular depth estimation Pre-training Semantics knowledge Transfer learning Zero-shot transfer learning
会议名称	30th ACM International Conference on Multimedia, MM 2022
会议地点	Lisboa, Portugal
会议日期	October 10, 2022 - October 14, 2022
URL	查看原文
收录类别	EI
语种	英语
出版者	Association for Computing Machinery, Inc
EI入藏号	20231313810636
EI主题词	Semantics
EI分类号	723.4 Artificial Intelligence
原始文献类型	Conference article (CA)
引用统计	正在获取...
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/294847
专题	信息科学与技术学院_本科生
作者单位	1.Peking University, Beijing, China; 2.ShanghaiTech University, Shanghai, China
推荐引用方式 GB/T 7714	Zhang, Renrui,Zeng, Ziyao,Guo, Ziyu,et al. Can language understand depth?[C]//ACM SIGMM:Association for Computing Machinery, Inc,2022:6868-6874.