Generalized Supervised Attention for Text Generation

	Generalized Supervised Attention for Text Generation
	Liu, Yixian1,2,3,4 ; Zhang, Liwen1,2,3,4 ; Zhang, Xinyu ; Jiang, Yong 7; Zhang, Yue 5,6; Tu, Kewei1,2,3,4
	2021
会议录名称	FINDINGS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, ACL-IJCNLP 2021
摘要	The attention-based encoder-decoder framework is widely used in many natural language generation tasks. The attention mechanism builds alignments between target words and source items that facilitate text generation. Previous work proposes supervised attention that uses human knowledge to guide the attention mechanism to learn better alignments. However, well-designed supervision built from ideal alignments can be costly or even infeasible. In this paper, we build a Generalized Supervised Attention method (GSA) based on quasi alignments, which specify candidate sets of alignments and are much easier to obtain than ideal alignments. We design a Summation Cross-Entropy (SCE) loss and a Supervised Multiple Attention (SMA) structure to accommodate quasi alignments. Experiments on three text generation tasks demonstrate that GSA improves generation performance and is robust against errors in attention supervision.
会议名称	Joint Conference of 59th Annual Meeting of the Association-for-Computational-Linguistics (ACL) / 11th International Joint Conference on Natural Language Processing (IJCNLP) / 6th Workshop on Representation Learning for NLP (RepL4NLP)
出版地	209 N EIGHTH STREET, STROUDSBURG, PA 18360 USA
会议地点	null,null,ELECTR NETWORK
会议日期	AUG 01-06, 2021
URL	查看原文
收录类别	EI ; CPCI-S
语种	英语
资助项目	National Natural Science Foundation of China[61976139]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods
WOS记录号	WOS:001181734704035
出版者	ASSOC COMPUTATIONAL LINGUISTICS-ACL
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/126612
专题	信息科学与技术学院_博士生信息科学与技术学院_PI研究组_屠可伟组
作者单位	1.School of Information Science and Technology, ShanghaiTech University 2.Shanghai Engineering Research Center of Intelligent Vision and Imaging 3.Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences 4.University of Chinese Academy of Sciences 5.Institute of Advanced Technology, Westlake Institute for Advanced Study, China 6.School of Engineering, Westlake University, Hangzhou, China 7.DAMO Academy, Alibaba Group
第一作者单位	信息科学与技术学院
第一作者的第一单位	信息科学与技术学院
推荐引用方式 GB/T 7714	Liu, Yixian,Zhang, Liwen,Zhang, Xinyu,et al. Generalized Supervised Attention for Text Generation[C]. 209 N EIGHTH STREET, STROUDSBURG, PA 18360 USA:ASSOC COMPUTATIONAL LINGUISTICS-ACL,2021.