Learning Low-Rank Tensor Cores with Probabilistic 0-Regularized Rank Selection for Model Compression

	Learning Low-Rank Tensor Cores with Probabilistic 0-Regularized Rank Selection for Model Compression
	Cao, Tianxiao1 ; Sun, Lu1 ; Nguyen, Canh Hao 2; Mamitsuka, Hiroshi 2
	2024
会议录名称	IJCAI INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE
ISSN	1045-0823
页码	3780-3788
发表状态	已发表
摘要	Compressing deep neural networks is of great importance for real-world applications on resource-constrained devices. Tensor decomposition is one promising answer that retains the functionality and most of the expressive power of the original deep models by replacing the weights with their decomposed cores. Decomposition with optimal ranks can achieve a good compression-accuracy tradeoff, but it is expensive to optimize due to its discrete and combinatorial nature. A common practice is to set all ranks equal and tune one hyperparameter, but it may significantly harm the flexibility and generalization. In this paper, we propose a novel automatic rank selection method for deep model compression that allows learning model weights and decomposition ranks simultaneously. We propose to penalize the 0 (quasi-)norm of the slices of decomposed tensor cores during model training. To avoid combinatorial optimization, we develop a probabilistic formulation and apply an approximate Bernoulli gate to each of the slices of tensor cores, which can be implemented in an end-to-end and scalable framework via gradient descent. It enables the automatic rank selection to be incorporated with arbitrary tensor decompositions and neural network layers such as linear layers, convolutional layers, and embedding layers. Comprehensive experiments on various tasks, including image classification, text sentiment classification, and neural machine translation, demonstrate the superior effectiveness of the proposed method over baselines. © 2024 International Joint Conferences on Artificial Intelligence. All rights reserved.
会议录编者/会议主办者	International Joint Conferences on Artifical Intelligence (IJCAI)
会议名称	33rd International Joint Conference on Artificial Intelligence, IJCAI 2024
出版地	ALBERT-LUDWIGS UNIV FREIBURG GEORGES-KOHLER-ALLEE, INST INFORMATIK, GEB 052, FREIBURG, D-79110, GERMANY
会议地点	Jeju, Korea, Republic of
会议日期	August 3, 2024 - August 9, 2024
URL	查看原文
收录类别	EI ; CPCI-S
语种	英语
资助项目	MEXT KAKENHI["21H05027","22H03645"]
WOS研究方向	Computer Science ; Mathematics
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods ; Mathematics, Applied
WOS记录号	WOS:001347142803099
出版者	International Joint Conferences on Artificial Intelligence
EI入藏号	20243817075178
原始文献类型	Conference article (CA)
文献类型	会议论文
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/430538
专题	信息科学与技术学院_本科生信息科学与技术学院_PI研究组_孙露组
作者单位	1.ShanghaiTech University, China 2.Kyoto University, Japan
第一作者单位	上海科技大学
第一作者的第一单位	上海科技大学
推荐引用方式 GB/T 7714	Cao, Tianxiao,Sun, Lu,Nguyen, Canh Hao,et al. Learning Low-Rank Tensor Cores with Probabilistic 0-Regularized Rank Selection for Model Compression[C]//International Joint Conferences on Artifical Intelligence (IJCAI). ALBERT-LUDWIGS UNIV FREIBURG GEORGES-KOHLER-ALLEE, INST INFORMATIK, GEB 052, FREIBURG, D-79110, GERMANY:International Joint Conferences on Artificial Intelligence,2024:3780-3788.