Decomposing Temporal Equilibrium Strategy for Coordinated Distributed Multi-Agent Reinforcement Learning

doi:10.1609/aaai.v38i16.29713

	Decomposing Temporal Equilibrium Strategy for Coordinated Distributed Multi-Agent Reinforcement Learning
	Zhu, Chenyang 1; Si, Wen 1; Zhu, Jinyu 1; Jiang, Zhihao2
	2024
会议录名称	THIRTY-EIGHTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, VOL 38 NO 16
ISSN	2159-5399
卷号	38
期号	16
页码	17618-17627
发表状态	已发表
DOI	10.1609/aaai.v38i16.29713
摘要	The increasing demands for system complexity and robustness have prompted the integration of temporal logic into Multi-Agent Reinforcement Learning (MARL) to address tasks with non-Markovian properties. However, incorporating non-Markovian properties introduces additional computational complexities, as agents are required to integrate historical data into their decision-making process. Also, optimizing strategies within a multi-agent environment presents significant challenges due to the exponential growth of the state space with the number of agents. In this study, we introduce an innovative hierarchical MARL framework that synthesizes temporal equilibrium strategies through parity games and subsequently encodes them as individual reward machines for MARL coordination. More specifically, we reduce the strategy synthesis problem into an emptiness problem concerning parity games with optimized states and transitions. Following this synthesis step, the temporal equilibrium strategy is decomposed into individual reward machines for decentralized MARL. Theoretical proofs are provided to verify the consistency of the Nash equilibrium between the parallel composition of decomposed strategies and the original strategy. Empirical evidence confirms the efficacy of the proposed synthesis technique, showcasing its ability to reduce state space compared to the state-of-the-art tool. Furthermore, our study highlights the superior performance of the distributed MARL paradigm over centralized approaches when deploying decomposed strategies.
会议录编者/会议主办者	Association for the Advancement of Artificial Intelligence
关键词	Fertilizers Game theory Multi agent systems Reinforcement learning Decision-making process Equilibria strategies Historical data Multi-agent environment Multi-agent reinforcement learning Non-Markovian property Parity games State-space System robustness Systems complexity
会议名称	38th AAAI Conference on Artificial Intelligence (AAAI) / 36th Conference on Innovative Applications of Artificial Intelligence / 14th Symposium on Educational Advances in Artificial Intelligence
出版地	2275 E BAYSHORE RD, STE 160, PALO ALTO, CA 94303 USA
会议地点	null,Vancouver,CANADA
会议日期	FEB 20-27, 2024
URL	查看原文
收录类别	EI ; CPCI-S
语种	英语
资助项目	National Natural Science Foundation of China[62202067] ; Natural Science Foundation of the Higher Education Institutions of Jiangsu Province[22KJB520012] ; Postgraduate Research and Practice Innovation Project of Jiangsu Province[SJCX231485]
WOS研究方向	Computer Science ; Education & Educational Research
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications ; Computer Science, Theory & Methods ; Education, Scientific Disciplines
WOS记录号	WOS:001239323500038
出版者	ASSOC ADVANCEMENT ARTIFICIAL INTELLIGENCE
EI入藏号	20241515880743
EI主题词	Decision making
EISSN	2374-3468
EI分类号	723.4 Artificial Intelligence ; 804 Chemical Products Generally ; 821.2 Agricultural Chemicals ; 912.2 Management ; 922.1 Probability Theory
原始文献类型	Conference article (CA)
引用统计	正在获取...
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/354894
专题	信息科学与技术学院信息科学与技术学院_PI研究组_江智浩组
通讯作者	Zhu, Chenyang
作者单位	1.Changzhou Univ, Sch Comp Sci & Aritificial Intelligence, Changzhou, Jiangsu, Peoples R China 2.ShanghaiTech Univ, Sch Informat Sci & Technol, Shanghai, Peoples R China
推荐引用方式 GB/T 7714	Zhu, Chenyang,Si, Wen,Zhu, Jinyu,et al. Decomposing Temporal Equilibrium Strategy for Coordinated Distributed Multi-Agent Reinforcement Learning[C]//Association for the Advancement of Artificial Intelligence. 2275 E BAYSHORE RD, STE 160, PALO ALTO, CA 94303 USA:ASSOC ADVANCEMENT ARTIFICIAL INTELLIGENCE,2024:17618-17627.