1: \begin{abstract}

2: Pre-trained language models such as BERT have been successful at tackling many natural language processing tasks. However, the unsupervised sub-word tokenization methods commonly used in these models (e.g., byte-pair encoding -- BPE) are sub-optimal at handling morphologically rich languages. Even given a morphological analyzer, naive sequencing of morphemes into a standard BERT architecture is inefficient at capturing morphological compositionality and expressing word-relative syntactic regularities. We address these challenges by proposing a simple yet effective two-tier BERT architecture that leverages a morphological analyzer and explicitly represents morphological compositionality. 

3: Despite the success of BERT, most of its evaluations have been conducted on high-resource languages, obscuring its applicability on low-resource languages. We evaluate our proposed method on the low-resource morphologically rich Kinyarwanda language, naming the proposed model architecture \textit{KinyaBERT}. A robust set of experimental results reveal that \textit{KinyaBERT} outperforms solid baselines by 2\% in F1 score on a named entity recognition task and by 4.3\% in average score of a machine-translated GLUE benchmark. \textit{KinyaBERT} fine-tuning has better convergence and achieves more robust results on multiple tasks even in the presence of translation noise.\footnote{\label{code1}Code and data are released at \url{https://github.com/anzeyimana/kinyabert-acl2022}}

4: 

5: 

6: \end{abstract}

7: