Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method

doi:arXiv:2409.19610

	Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
	Pan, Bikang1 ; Huang, Wei 2; Shi, Ye1
	2024-09-29
状态	已发表
摘要	Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.
语种	英语
DOI	arXiv:2409.19610
相关网址	查看原文
出处	Arxiv
收录类别	PPRN.PPRN
WOS记录号	PPRN:100734430
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications ; Computer Science, Software Engineering
文献类型	预印本
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/433526
专题	信息科学与技术学院_博士生信息科学与技术学院_PI研究组_石野组
通讯作者	Huang, Wei; Shi, Ye
作者单位	1.ShanghaiTech Univ, Shanghai, Peoples R China 2.RIKEN Ctr Adv Intelligence Project, Chuo, Japan
推荐引用方式 GB/T 7714	Pan, Bikang,Huang, Wei,Shi, Ye. Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method. 2024.