ConvNets vs. Transformers: Whose Visual Representations are More Transferable?

doi:10.1109/ICCVW54120.2021.00252

	ConvNets vs. Transformers: Whose Visual Representations are More Transferable?
	Zhou, Hong-Yu 1; Lu, Chixiang 1; Yang, Sibei2 ; Yu, Yizhou 1
	2021
会议录名称	2021 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION WORKSHOPS (ICCVW 2021)
ISSN	2473-9936
发表状态	已发表
DOI	10.1109/ICCVW54120.2021.00252
摘要	Vision transformers have attracted much attention from computer vision researchers as they are not restricted to the spatial inductive bias of ConvNets. However; although Transformer-based backbones have achieved much progress on ImageNet classification, it is still unclear whether the learned representations are as transferable as or even more transferable than ConvNets' features. To address this point, we systematically investigate the transfer learning ability of ConvNets and vision transformers in 15 single-task and multi-task performance evaluations. We observe consistent advantages of Transformer-based backbones on 13 downstream tasks (out of 15), including but not limited to line-grained classification, scene recognition (classification, segmentation and depth estimation), open-domain classification, face recognition, etc. More specifically, we find that two ViT models heavily rely on whole network fine-tuning to achieve performance gains while Swin Transformer does not have such a requirement. Moreover, vision transformers behave more robustly in multi-task learning, i.e., bringing more improvements when managing mutually beneficial tasks and reducing performance losses when tackling irrelevant tasks. We hope our discoveries can facilitate the exploration and exploitation of vision transformers in the future.
会议名称	IEEE/CVF International Conference on Computer Vision (ICCVW)
出版地	10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA
会议地点	null,null,ELECTR NETWORK
会议日期	OCT 11-17, 2021
URL	查看原文
收录类别	CPCI-S ; EI ; CPCI
语种	英语
WOS研究方向	Computer Science ; Imaging Science & Photographic Technology
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications ; Imaging Science & Photographic Technology
WOS记录号	WOS:000739651102035
出版者	IEEE COMPUTER SOC
引用统计
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/157686
专题	信息科学与技术学院_PI研究组_杨思蓓组
通讯作者	Zhou, Hong-Yu
作者单位	1.Univ Hong Kong, Hong Kong, Peoples R China 2.ShanghaiTech Univ, Shanghai, Peoples R China
推荐引用方式 GB/T 7714	Zhou, Hong-Yu,Lu, Chixiang,Yang, Sibei,et al. ConvNets vs. Transformers: Whose Visual Representations are More Transferable?[C]. 10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA:IEEE COMPUTER SOC,2021.