上海科技大学知识管理系统

Deep neural networks have shown excellent performance in various domains, but the large number of parameters and computational inefficiency pose significant challenges in practice. Existing sparse learning methods, such as pruning and regularization, play a crucial role in reducing model size and improving generalization. However, they are limited to the single-level grouping structure and ignore the correlation between consecutive layers, leading to insufficient sparsity and performance degradation. To address these challenges, we propose a novel sparsity regularizer that promotes structured sparsity based on the multi-level grouping structure. It encourages inter-group cooperation and intra-group competition at the first-level, and promotes inter-group competition and intra-group cooperation at the second-level. The multi-level grouping nature can flexibly model the correlation between consecutive layers or convolutional kernels by carefully defining the groups based on specific neural architectures. Moreover, we introduce a more general form that unifies a family of convex and non-covnex sparse regularizers and prove its equivalence to multiplicative weight decomposition, which helps us develop a simple but efficient optimization algorithm. Extensive experiments on real-world datasets show that the proposed method can generate more compact and efficient models compared to cutting-edge methods.