Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning

doi:10.1109/TSP.2022.3214122

	Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning
	Fang, Wenzhi1 ; Yu, Ziyi1 ; Jiang, Yuning 2; Shi, Yuanming1 ; Jones, Colin N.2; Zhou, Yong1
	2022
发表期刊	IEEE TRANSACTIONS ON SIGNAL PROCESSING
ISSN	1053-587X
EISSN	1941-0476
卷号	70 页码:1-16
发表状态	已发表
DOI	10.1109/TSP.2022.3214122
摘要	Federated learning (FL), as an emerging edge artificial intelligence paradigm, enables many edge devices to collaboratively train a global model without sharing their private data. To enhance the training efficiency of FL, various algorithms have been proposed, ranging from first-order to second-order methods. However, these algorithms cannot be applied in scenarios where the gradient information is not available, e.g., federated black-box attack and federated hyperparameter tuning. To address this issue, in this paper we propose a derivative-free federated zeroth-order optimization (FedZO) algorithm featured by performing multiple local updates based on stochastic gradient estimators in each communication round and enabling partial device participation. Under non-convex settings, we derive the convergence performance of the FedZO algorithm on non-independent and identically distributed data and characterize the impact of the numbers of local iterates and participating edge devices on the convergence. To enable communication-efficient FedZO over wireless networks, we further propose an over-the-air computation (AirComp) assisted FedZO algorithm. With an appropriate transceiver design, we show that the convergence of AirComp-assisted FedZO can still be preserved under certain signal-to-noise ratio conditions. Simulation results demonstrate the effectiveness of the FedZO algorithm and validate the theoretical observations. IEEE
关键词	Computational efficiency Learning algorithms Radio transceivers Signal processing Signal to noise ratio Stochastic models Stochastic systems Atmospheric modeling Convergence Federated learning Optimisations Order optimizations Over the airs Over-the-air computation Performances evaluation Signal processing algorithms Zeroth-order optimization
URL	查看原文
收录类别	EI ; SCI ; SCIE
语种	英语
资助项目	National Natural Science Foundation of China (NSFC)["U20A20159","62001294"] ; Swiss National Science Foundation through the RISK Project (Risk Aware Data-Driven Demand Response)[200021175627]
WOS研究方向	Engineering
WOS类目	Engineering, Electrical & Electronic
WOS记录号	WOS:000880643100003
出版者	Institute of Electrical and Electronics Engineers Inc.
EI入藏号	20224212976301
EI主题词	Random processes
EI分类号	716.1 Information Theory and Signal Processing ; 716.3 Radio Systems and Equipment ; 723.4.2 Machine Learning ; 731.1 Control Systems ; 922.1 Probability Theory ; 961 Systems Science
原始文献类型	Article in Press
引用统计
文献类型	期刊论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/241100
专题	信息科学与技术学院_硕士生信息科学与技术学院_PI研究组_石远明组信息科学与技术学院_PI研究组_周勇组信息科学与技术学院_本科生
通讯作者	Zhou, Yong
作者单位	1.ShanghaiTech Univ, Sch Informat Sci & Technol, Shanghai 201210, Peoples R China 2.Ecole Polytech Fed Lausanne, Automat Control Lab, CH-1015 Lausanne, Switzerland
第一作者单位	信息科学与技术学院
通讯作者单位	信息科学与技术学院
第一作者的第一单位	信息科学与技术学院
推荐引用方式 GB/T 7714	Fang, Wenzhi,Yu, Ziyi,Jiang, Yuning,et al. Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning[J]. IEEE TRANSACTIONS ON SIGNAL PROCESSING,2022,70:1-16.
APA	Fang, Wenzhi,Yu, Ziyi,Jiang, Yuning,Shi, Yuanming,Jones, Colin N.,&Zhou, Yong.(2022).Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning.IEEE TRANSACTIONS ON SIGNAL PROCESSING,70,1-16.
MLA	Fang, Wenzhi,et al."Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning".IEEE TRANSACTIONS ON SIGNAL PROCESSING 70(2022):1-16.