1: \begin{abstract}

2: \input{abstract_rev.tex}

3: %

4: %Training of conventional recurrent models suffers from the issues of inaccuracy and instability over time. 

5: %

6: %

7: %Motivated by autapse in neuroscience, in this paper we propose novel {\em Equilibrium Recurrent Models (ERMs)} which introduce neuronal time delayed self-feedback into conventional recurrent models for better training. We hypothesize and validate empirically that such neuronal self-feedback helps stabilize the hidden state transition matrices rapidly by learning discriminative latent features, leading to fast convergence in training and good generalization in testing. We propose a novel efficient inexact Newton method to solve these fixed-point conditions given model parameters for generating the latent features at each hidden state. We prove that our inexact Newton method converges locally with linear rate (under mild conditions). As a result we manage to convert ERMs into residual networks {\em in a principal way} and train them efficiently using backpropagation. We demonstrate the superiority of ERMs over the state-of-the-art on several benchmark data sets. For instance, on the Pixel-MNIST-10 data set our training is $\bf\sim30\times$ faster than RNN, and $\bf\sim10\times$ faster than FastGRNN \cite{NIPS2018_8116} with comparable accuracy.

8: %

9: \end{abstract}

10: