1: \begin{abstract}

2: This study investigates the learnability of memory-less and memory-augmented Recurrent Neural Networks (RNNs) with deterministic and non-deterministic stacks, which are theoretically equivalent to Pushdown Automata in terms of expressivity. However, empirical evaluations reveal that these models often fail to generalize on longer sequences, particularly when learning context-sensitive languages, suggesting they rely on precision rather than mastering symbolic grammar rules. Our experiments examined fully trained models and models with various frozen components: the controller, the memory, and only the classification layer. While all models showed similar performance on training validation, the model with frozen memory achieved state-of-the-art performance on the Penn Treebank (PTB) dataset, reducing the best overall test perplexity from 123.5 to 120.5—a gain of approximately 1.73\%. When tested on context-sensitive languages, models with frozen memory consistently outperformed others on small to medium test sets. Notably, well-trained models experienced up to a 60\% performance drop on longer sequences, whereas models with frozen memory retained close to 90\% of their initial performance. Theoretically, we explain that freezing the memory component enhances stability by anchoring the model's capacity to manage temporal dependencies without constantly adjusting memory states. This approach allows the model to focus on refining other components, leading to more robust convergence to optimal solutions. These findings highlight the importance of designing stable memory architectures and underscore the need to evaluate models on longer sequences to truly understand their learnability behavior and limitations. The study suggests that RNNs may rely more on precision in data processing than on internalizing grammatical rules, emphasizing the need for improvements in model architecture and evaluation methods.

3: \end{abstract}

4: