1: \begin{abstract}

2: % Previously existing quantization methods relies on learnable "quantization interval" to strike a balance between gradient accuracy and weight ranges. 

3: % However, due to existence of local minima in the loss landscapes of deep learning models, it is likely that a quantized model would converge to local optima that are too sharp to tolerate the existence of large quantization errors and thus resulting in loss of large weights and poor generalization properties. 

4: Quantization is an effective technique to reduce memory footprint, inference latency, and power consumption of deep learning models. 

5: However, existing quantization methods suffer from accuracy degradation compared to full-precision (FP) models due to the errors introduced by coarse gradient estimation through non-differentiable quantization layers. 

6: The existence of sharp local minima in the loss landscapes of overparameterized models (e.g., Transformers) tends to aggravate such performance penalty in low-bit (2, 4 bits) settings. %~\cite{shen2020q}.

7: In this work, we propose sharpness- and quantization-aware training (SQuAT), which would encourage the model to converge to flatter minima while performing quantization-aware training. 

8: Our proposed method alternates training between sharpness objective and step-size objective, which could potentially let the model learn the most suitable parameter update magnitude to reach convergence near-flat minima.

9: % The flatter minima not only improve generalization but also alleviate accuracy loss caused by the errors introduced in the quantization step.

10: Extensive experiments show that our method can consistently outperform state-of-the-art quantized BERT models under 2, 3, and 4-bit settings on GLUE benchmarks by 1\%, and can sometimes even outperform full precision (32-bit) models.

11: Our experiments on empirical measurement of sharpness also suggest that our method would lead to flatter minima compared to other quantization methods. 

12: \end{abstract}

13: