Federated Low-Rank Adaptation for Large Language Model Fine-Tuning Over Wireless Networks

doi:10.1109/GLOBECOM52923.2024.10901572

	Federated Low-Rank Adaptation for Large Language Model Fine-Tuning Over Wireless Networks
	Zixin Wang 1; Yong Zhou2 ; Yuanming Shi2 ; Khaled. B. Letaief 1
	2024-12-12
会议录名称	GLOBECOM 2024 - 2024 IEEE GLOBAL COMMUNICATIONS CONFERENCE
ISSN	1930-529X
页码	3063-3068
发表状态	已发表
DOI	10.1109/GLOBECOM52923.2024.10901572
摘要	Low-rank adaptation (LoRA) is an emerging fine-tuning method for personalized large language models (LLMs) due to its capability of achieving comparable learning performance to full fine-tuning by training a much smaller number of parameters. Federated fine-tuning (FedFT) combines LoRA with federated learning (FL) to enable collaborative fine-tuning of a global model with edge devices, leveraging distributed data while ensuring privacy. However, limited radio resources and computation capabilities of edge devices pose critical challenges on deploying FedFT over wireless networks. In this paper, we propose a split FedFT framework to separately deploy the computationally-intensive encoder of a pre-trained model at the edge server while reserving the embedding and the task modules at the edge devices, where the information exchanges between these modules are carried out over wireless networks. By exploiting the low-rank property of LoRA, the proposed FedFT framework reduces communication overhead by aggregating the gradient of the task module with respect to the output of a low-rank matrix. To enhance learning performance under stringent resource constraints, we formulate a joint device scheduling and bandwidth allocation problem while considering average transmission delay. By applying the Lyapunov technique, we decompose the formulated long-term mixed-integer programming (MIP) problem into sequential subproblems, followed by developing an online algorithm for effective device scheduling and bandwidth allocation. Simulation results demonstrate the effectiveness of our proposed online algorithm in enhancing learning performance.
关键词	Edge computing Frequency allocation Integer programming Learning to rank Mixed-integer linear programming Problem oriented languages Resource allocation Critical challenges Device scheduling Distributed data Fine tuning Fine-tuning methods Global models Language model Learning performance On-line algorithms Radio resources
会议名称	2024 IEEE Global Communications Conference, GLOBECOM 2024
会议地点	Cape Town, South Africa
会议日期	8-12 Dec. 2024
URL	查看原文
收录类别	EI
语种	英语
出版者	Institute of Electrical and Electronics Engineers Inc.
EI入藏号	20251318125097
EI主题词	Bandwidth
EISSN	2576-6813
EI分类号	716.1 Information Theory and Signal Processing ; 716.3 Radio Systems and Equipment ; 912.2 Management ; 1101.2 Machine Learning ; 1105 Computer Networks ; 1106.1 Computer Programming ; 1106.1.1 Computer Programming Languages ; 1201.7 Optimization Techniques
原始文献类型	Conference article (CA)
来源库	IEEE
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/500280
专题	信息科学与技术学院信息科学与技术学院_PI研究组_石远明组信息科学与技术学院_PI研究组_周勇组
作者单位	1.Department of ECE, Hong Kong University of Science and Technology, Hong Kong, China 2.School of Information Science and Technology, ShanghaiTech University, Shanghai, China
推荐引用方式 GB/T 7714	Zixin Wang,Yong Zhou,Yuanming Shi,et al. Federated Low-Rank Adaptation for Large Language Model Fine-Tuning Over Wireless Networks[C]:Institute of Electrical and Electronics Engineers Inc.,2024:3063-3068.