1: \begin{abstract}

2: We find that at sequence length 512 padding tokens represent in excess of $50\%$ 

3: of the Wikipedia dataset used for pretraining BERT (Bidirectional Encoder Representations from Transformers). 

4: Therefore by removing all padding we achieve a 2x speed-up in terms of sequences/sec. 

5: To exploit this characteristic of the dataset, 

6: we develop and contrast two deterministic packing algorithms.

7: Both algorithms rely on the assumption that sequences are interchangeable 

8: and therefore packing can be performed on the histogram of sequence lengths, rather than per sample. 

9: This transformation of the problem leads to algorithms which are fast and have linear complexity in dataset size.

10: The shortest-pack-first histogram-packing (SPFHP) algorithm determines the packing order for the Wikipedia dataset of over $16$M sequences in $0.02$ seconds. 

11: The non-negative least-squares histogram-packing (NNLSHP) algorithm converges in $28.4$ seconds 

12: but produces solutions which are more depth efficient,

13: managing to get near optimal packing by combining a maximum of $3$ sequences in one sample.

14: Using the dataset with multiple sequences per sample requires additional masking in the attention layer and a modification of the MLM loss function.

15: We demonstrate that both of these changes are straightforward to implement and have relatively little impact on the achievable performance gain on modern hardware. 

16: Finally, we pretrain BERT-Large using the packed dataset, 

17: demonstrating no loss of convergence and the desired 2x speed-up. 

18: \end{abstract}

19: