1: \begin{abstract}

2: We propose rectified factor networks (RFNs) to 

3: efficiently construct 

4: very sparse, non-linear, high-dimensional representations

5: of the input.

6: RFN models identify rare and small events in the input,

7: have a low interference between code units,

8: have a small reconstruction error, and

9: explain the data covariance structure.

10: RFN learning is a generalized alternating minimization

11: algorithm derived from the posterior regularization method 

12: which enforces

13: non-negative and normalized posterior means.

14: We proof convergence and correctness 

15: of the RFN learning algorithm.

16: 

17: On benchmarks, RFNs are compared to

18: other unsupervised methods like

19: autoencoders, RBMs, factor analysis,

20: ICA, and PCA.

21: In contrast to previous sparse coding methods,

22: RFNs yield sparser codes,

23: capture the data's covariance structure more precisely, 

24: and have a significantly smaller reconstruction error.

25: We test RFNs as pretraining technique for deep networks

26: on different vision datasets, where RFNs were superior to

27: RBMs and autoencoders. 

28: On gene expression data from two

29: pharmaceutical drug discovery studies, RFNs 

30: detected small and rare gene modules that

31: revealed highly relevant new biological insights which 

32: were so far missed by other unsupervised methods.

33: 

34: %On CIFAR datasets

35: %RFNs improved the performance of

36: %different deep convolutional nets.

37: \end{abstract}

38: