A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective

doi:10.1109/TPAMI.2024.3445463

	A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective
	Chaoqi Chen 1; Yushuang Wu 2,3; Qiyuan Dai4 ; Hong-Yu Zhou 1; Mutian Xu 2; Sibei Yang4,5 ; Xiaoguang Han 2,3; Yizhou Yu 1
	2024-12-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE (IF:20.8[JCR-2023],22.2[5-Year])
ISSN	0162-8828
EISSN	1939-3539
卷号	46 期号:12 页码:1-20
发表状态	已发表
DOI	10.1109/TPAMI.2024.3445463
摘要	Graph Neural Networks (GNNs) have gained momentum in graph representation learning and boosted the state of the art in a variety of areas, such as data mining (e.g., social network analysis and recommender systems), computer vision (e.g., object detection and point cloud learning), and natural language processing (e.g., relation extraction and sequence learning), to name a few. With the emergence of Transformers in natural language processing and computer vision, graph Transformers embed a graph structure into the Transformer architecture to overcome the limitations of local neighborhood aggregation while avoiding strict structural inductive biases. In this paper, we present a comprehensive review of GNNs and graph Transformers in computer vision from a task-oriented perspective. Specifically, we divide their applications in computer vision into five categories according to the modality of input data, i.e., 2D natural images, videos, 3D data, vision + language, and medical images. In each category, we further divide the applications according to a set of vision tasks. Such a task-oriented taxonomy allows us to examine how each task is tackled by different GNN-based approaches and how well these approaches perform. Based on the necessary preliminaries, we provide the definitions and challenges of the tasks, in-depth coverage of the representative approaches, as well as discussions regarding insights, limitations, and future directions.
关键词	Task analysis Computer vision Three-dimensional displays Transformers Point cloud compression Visualization Videos Computer vision graph transformers graph neural networks medical image analysis point clouds and meshes vision and language
URL	查看原文
收录类别	EI ; SCI
语种	英语
资助项目	Guangdong Provincial Outstanding Youth Project[2023B1515020055] ; NSFC["62172348","61931024"] ; Shenzhen General Project[JCYJ20220530143604010]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:001364431200175
出版者	IEEE COMPUTER SOC
来源库	IEEE
引用统计	正在获取...
文献类型	期刊论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/414193
专题	信息科学与技术学院信息科学与技术学院_博士生信息科学与技术学院_PI研究组_杨思蓓组
作者单位	1.Department of Computer Science, The University of Hong Kong, Hong Kong 2.School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 3.Future Network of Intelligence Institute, CUHK-Shenzhen 4.School of Information Science and Technology, ShanghaiTech University, Shanghai 5.Shanghai Engineering Research Center of Intelligent Vision and Imaging
推荐引用方式 GB/T 7714	Chaoqi Chen,Yushuang Wu,Qiyuan Dai,et al. A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2024,46(12):1-20.
APA	Chaoqi Chen.,Yushuang Wu.,Qiyuan Dai.,Hong-Yu Zhou.,Mutian Xu.,...&Yizhou Yu.(2024).A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,46(12),1-20.
MLA	Chaoqi Chen,et al."A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 46.12(2024):1-20.