Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training
2023
会议录名称IJCAI INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE
ISSN1045-0823
卷号2023-August
页码791-799
发表状态已发表
摘要

Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for both 2D and 3D computer vision. However, existing MAE-style methods can only learn from the data of a single modality, i.e., either images or point clouds, which neglect the implicit semantic and geometric correlation between 2D and 3D. In this paper, we explore how the 2D modality can benefit 3D masked autoencoding, and propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training. Joint-MAE randomly masks an input 3D point cloud and its projected 2D images, and then reconstructs the masked information of the two modalities. For better cross-modal interaction, we construct our JointMAE by two hierarchical 2D-3D embedding modules, a joint encoder, and a joint decoder with modal-shared and model-specific decoders. On top of this, we further introduce two cross-modal strategies to boost the 3D representation learning, which are local-aligned attention mechanisms for 2D-3D semantic cues, and a cross-reconstruction loss for 2D-3D geometric constraints. By our pre-training paradigm, Joint-MAE achieves superior performance on multiple downstream tasks, e.g., 92.4% accuracy for linear SVM on ModelNet40 and 86.07% accuracy on the hardest split of ScanObjectNN. © 2023 International Joint Conferences on Artificial Intelligence. All rights reserved.

会议录编者/会议主办者International Joint Conferences on Artifical Intelligence (IJCAI)
关键词Artificial intelligence Image reconstruction Learning systems Semantics Three dimensional computer graphics 2D images 3D computer vision 3D point cloud Auto encoders Geometric correlations Implicit semantics Learn+ Performance Point-clouds Pre-training
会议名称32nd International Joint Conference on Artificial Intelligence, IJCAI 2023
会议地点Macao, China
会议日期August 19, 2023 - August 25, 2023
收录类别EI
语种英语
出版者International Joint Conferences on Artificial Intelligence
EI入藏号20233714713473
EI主题词Decoding
EI分类号723.2 Data Processing and Image Processing ; 723.4 Artificial Intelligence ; 723.5 Computer Applications
原始文献类型Conference article (CA)
文献类型会议论文
条目标识符https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/348743
专题信息科学与技术学院_硕士生
通讯作者Li, Xianzhi
作者单位
1.Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong
2.CUHK MMLab, Hong Kong
3.Huazhong University of Science and Technology, China
4.Institute of Medical Intelligence and XR, The Chinese University of Hong Kong, Hong Kong
5.ShanghaiTech University, China
推荐引用方式
GB/T 7714
Guo, Ziyu,Zhang, Renrui,Qiu, Longtian,et al. Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training[C]//International Joint Conferences on Artifical Intelligence (IJCAI):International Joint Conferences on Artificial Intelligence,2023:791-799.
条目包含的文件
文件名称/大小 文献类型 版本类型 开放类型 使用许可
个性服务
查看访问统计
谷歌学术
谷歌学术中相似的文章
[Guo, Ziyu]的文章
[Zhang, Renrui]的文章
[Qiu, Longtian]的文章
百度学术
百度学术中相似的文章
[Guo, Ziyu]的文章
[Zhang, Renrui]的文章
[Qiu, Longtian]的文章
必应学术
必应学术中相似的文章
[Guo, Ziyu]的文章
[Zhang, Renrui]的文章
[Qiu, Longtian]的文章
相关权益政策
暂无数据
收藏/分享
所有评论 (0)
暂无评论
 

除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。