1: \begin{abstract}

2: This paper details the work of the University of Groningen for the BabyLM Challenge \cite{warstadt-et-al-2023-babylm}.

3: We follow the idea that, like babies, language models should be introduced to simpler concepts first and build off of that knowledge to understand more complex concepts. 

4: We examine this strategy of simple-then-complex through a variety of lenses, namely context size, vocabulary, and overall linguistic complexity of the data. 

5: We find that only one, context size, is truly beneficial to training a language model.

6: However this simple change to context size gives us improvements of 2 points on average on (Super)GLUE tasks, 1 point on MSGS tasks, and 12\% on average on BLiMP tasks. 

7: Our context-limited model outperforms the baseline that was trained on 10$\times$ the amount of data.

8: 

9: % We find that a combination of vocabulary expansion from the character level up to the subword level as well as the incremental increase of context size improves the performance of language models on the BLiMP benchmark by about 9\% over the provided baseline. We also find that such methods can be used with larger models of up to 600M parameters \todo{check the parameter count} (Deberta-Large) and still see further improvements up to 12\% over the baseline, showing that one can still achieve convergence with a large model trained from scratch even in low-resource settings.

10: \end{abstract}

11: