SIMPLE: Specialized Model-Sample Matching for Domain Generalization

	SIMPLE: Specialized Model-Sample Matching for Domain Generalization
	Li, Ziyue; Ren, Kan; Jiang, Xinyang; Shen, Yifei; Zhang, Haipeng; Li, Dongsheng
	2023-02
会议录名称	THE 11TH INTERNATIONAL CONFERENCE ON LEARNING REPRESENTATIONS
发表状态	正式接收
摘要	In domain generalization (DG), most existing methods aspire to fine-tune a specific pretrained model through novel DG algorithms. In this paper, we propose an alternative direction, i.e., to efficiently leverage a pool of pretrained models without fine-tuning. Through extensive empirical and theoretical evidence, we demonstrate that (1) pretrained models have possessed generalization to some extent while there is no single best pretrained model across all distribution shifts, and (2) out-of-distribution (OOD) generalization error depends on the fitness between the pretrained model and unseen test distributions. This analysis motivates us to incorporate diverse pretrained models and to dispatch the best matched models for each OOD sample by means of recommendation techniques. To this end, we propose SIMPLE, a specialized model-sample matching method for domain generalization. First, the predictions of pretrained models are adapted to the target domain by a linear label space transformation. A matching network aware of model specialty is then proposed to dynamically recommend proper pretrained models to predict each test sample. The experiments on DomainBed show that our method achieves significant performance improvements (up to 12.2% for individual dataset and 3.9% on average) compared to state-of-the-art (SOTA) methods and further achieves 6.1% gain via enlarging the pretrained model pool. Moreover, our method is highly efficient and achieves more than 1000× training speedup compared to the conventional DG methods with fine-tuning a pretrained model. Code and supplemental materials are available at https://seqml.github.io/simple. © 2023 11th International Conference on Learning Representations, ICLR 2023. All rights reserved.
会议录编者/会议主办者	Baidu ; DeepMind ; et al. ; Google Research ; Huawei ; Meta AI
关键词	Generalisation Generalization algorithms Generalization Error Matched models Matchings Model samples Novel domain Shift-and Simple++ Without fine-tuning
会议名称	11th International Conference on Learning Representations, ICLR 2023
会议地点	Kigali, Rwanda
会议日期	May 1, 2023 - May 5, 2023
收录类别	EI
语种	英语
出版者	International Conference on Learning Representations, ICLR
EI入藏号	20243116790514
EI主题词	Linear transformations
EI分类号	921.3 Mathematical Transformations
原始文献类型	Conference article (CA)
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/286568
专题	信息科学与技术学院_PI研究组_张海鹏组
通讯作者	Ren, Kan; Jiang, Xinyang
推荐引用方式 GB/T 7714	Li, Ziyue,Ren, Kan,Jiang, Xinyang,et al. SIMPLE: Specialized Model-Sample Matching for Domain Generalization[C]//Baidu, DeepMind, et al., Google Research, Huawei, Meta AI:International Conference on Learning Representations, ICLR,2023.