1: \begin{abstract}
2:
3: Deep neural networks (DNNs) have achieved significant success in a variety of real world applications, i.e., image classification.
4: However, tons of parameters in the networks restrict the efficiency of neural networks due to the large model size and the intensive computation.
5: To address this issue, various approximation techniques have been investigated, which seek for a light weighted network with little performance degradation in exchange of smaller model size or faster inference.
6: Both low-rankness and sparsity are appealing properties for the network approximation.
7: In this paper we propose a unified framework to compress the convolutional neural networks (CNNs) by combining these two properties, while taking the nonlinear activation into consideration.
8: Each layer in the network is approximated by the sum of a structured sparse component and a low-rank component, which is formulated as an optimization problem. Then, an extended version of alternating direction method of multipliers (ADMM) with guaranteed convergence is presented to solve the relaxed optimization problem.
9: %The performance of the network is retained by minimizing the reconstruction error of the feature maps after activation of each layer, using the alternating direction method of multipliers (ADMM).
10: Experiments are carried out on \emph{VGG-16}, \emph{AlexNet} and \emph{GoogLeNet} with large image classification datasets.
11: The results outperform previous work in terms of accuracy degradation, compression rate and speedup ratio.
12: The proposed method is able to remarkably compress the model (with up to $4.9\times$ reduction of parameters) at a cost of little loss or without loss on accuracy.
13:
14: %We demonstrate the effectiveness of our proposed approach on \emph{VGG-16}, \emph{AlexNet} and \emph{GoogLeNet}.
15: %The results show that model compression and inference acceleration are realized on all these networks at a cost of little loss or without loss on accuracy.
16:
17: \end{abstract}
18: