ATLAS: Adapter-Based Multi-Modal Continual Learning with a Two-Stage Learning Strategy

doi:arXiv:2410.10923

KMS

	ATLAS: Adapter-Based Multi-Modal Continual Learning with a Two-Stage Learning Strategy
	Li, Hong 1,4; Tan, Zhiquan 2; Li, Xingyu 3; Huang, Weiran 1
	2024-10-14
状态	已发表
摘要	While vision-and-language models significantly advance in many fields, the challenge of continual learning is unsolved. Parameter-efficient modules like adapters and prompts present a promising way to alleviate catastrophic forgetting. However, existing works usually learn individual adapters for each task, which may result in redundant knowledge among adapters. Moreover, they continue to use the original pre-trained model to initialize the downstream model, leading to negligible changes in the model's generalization compared to the original model. In addition, there is still a lack of research investigating the consequences of integrating a multi-modal model into the updating procedure for both uni-modal and multi-modal tasks and the subsequent impacts it has on downstream tasks. In this paper, we propose an adapter-based two-stage learning paradigm, a multi-modal continual learning scheme that consists of experience-based learning and novel knowledge expansion, which helps the model fully use experience knowledge and compensate for novel knowledge. Extensive experiments demonstrate that our method is proficient for continual learning. It expands the distribution of representation upstream while also minimizing the negative impact of forgetting previous tasks. Additionally, it enhances the generalization capability for downstream tasks. Furthermore, we incorporate both multi-modal and uni-modal tasks into upstream continual learning. We observe that learning from upstream tasks can help with downstream tasks.
语种	英语
DOI	arXiv:2410.10923
相关网址	查看原文
出处	Arxiv
收录类别	PPRN.PPRN
WOS记录号	PPRN:113088837
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Software Engineering
文献类型	预印本
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/446058
专题	上海科技大学
通讯作者	Huang, Weiran
作者单位	1.Shanghai Jiao Tong Univ, Qing Yuan Res Inst, MIFA Lab, SEIEE, Shanghai, Peoples R China 2.Tsinghua Univ, Dept Math Sci, Beijing, Peoples R China 3.Lin Gang Lab, Shanghai, Peoples R China 4.ShanghaiTech Univ, Shanghai, Peoples R China
推荐引用方式 GB/T 7714	Li, Hong,Tan, Zhiquan,Li, Xingyu,et al. ATLAS: Adapter-Based Multi-Modal Continual Learning with a Two-Stage Learning Strategy. 2024.