1: \begin{abstract}

2: % Definition of TNN

3: We propose \textit{\tensornet ({\tensornetshort}s)}, a generalization of existing neural networks by extending tensor operations on low order operands to those on high order ones. 

4: % Challenge of TNN 

5: The problem of parameter learning is challenging, as it corresponds to hierarchical nonlinear tensor decomposition.

6: % Solution to the problem

7: We propose to solve the learning problem using stochastic gradient descent by deriving nontrivial backpropagation rules in generalized tensor algebra we introduce.

8: % Benefits of TNN

9:  Our proposed {\tensornetshort}s has three advantages over existing neural networks: 

10:  (1) {\tensornetshort}s naturally apply to high order input object and thus preserve the multi-dimensional structure in the input, as there is no need to flatten the data. 

11:  (2) {\tensornetshort}s interpret designs of existing neural network architectures. 

12:  (3) Mapping a neural network to {\tensornetshort}s with the same expressive power results in a {\tensornetshort} of fewer parameters. 

13:  % Compression algorithms

14:  \ouralgo of neural network improves existing \ourbaseline methods as {\tensornetshort}s exploit two other types of {\em invariant structures}, periodicity and modulation, in addition to the low rankness.

15:  % Experimental results

16: Experiments on LeNet-5 (MNIST), ResNet-32 (CIFAR10) and ResNet-50 (ImageNet) demonstrate that our \ouralgo outperforms (5\% test accuracy improvement universally on CIFAR10) the state-of-the-art \ourbaseline methods under the same compression rate, besides achieving orders of magnitude faster convergence rates due to the efficiency of {\tensornetshort}s. 

17: \end{abstract}

18: