1: \begin{abstract}

2: Multi-task learning with an unbalanced data distribution skews model learning towards high resource tasks, especially when model capacity is fixed and fully shared across all tasks. Sparse scaling architectures, such as BASELayers, provide flexible mechanisms for different tasks to have a variable number of parameters, which can be useful to counterbalance skewed data distributions. We find that that sparse architectures for multilingual machine translation can perform poorly out of the box, and propose two straightforward techniques to mitigate this --- a temperature heating mechanism and dense pre-training. Overall, these methods improve performance on two multilingual translation benchmarks compared to standard BASELayers and Dense scaling baselines, and in combination, more than 2x model convergence speed.

3: % To alleviate this problem, we experiment with two simple techniques that promote cross-lingual transfer. 

4: % In the first method, we slowly increase the upsampling ratios of low resource tasks as compared to previous works which subscribe to a fixed sampling ratio throughout the training process. In the second method, we start with a dense version of the encoder-decoder model and slowly transform it into a sparse MoE model as training progresses. Overall these two methods help improve low resource performance by 0.7 BLEU on WMT-15 and 1.1 BLEU on ML-50 benchmarks. Along with the quantitative improvements, they also help reduce the convergence time by $>$50\% on average.

5: 

6: \end{abstract}

7: