1: \begin{abstract}

2: Accelerating the sampling speed of diffusion models remains a significant challenge. Recent score distillation methods distill a heavy teacher model into an efficient student generator, which is optimized by calculating the difference between the two score functions on the samples generated by the student model.

3: However, there is a score mismatch issue in the early stage of the distillation process, because existing methods mainly focus on using the endpoint of pre-trained diffusion models as teacher models, overlooking the importance of the convergence trajectory between the student generator and the teacher model.

4: To address this issue, we extend the score distillation process by introducing the entire convergence trajectory of teacher models and propose \textbf{Dis}tribution \textbf{Back}tracking Distillation (\textbf{DisBack}) for distilling student generators. DisBask is composed of two stages: \textit{Degradation Recording} and \textit{Distribution Backtracking}. 

5: \textit{Degradation Recording} is designed for obtaining the convergence trajectory of teacher models, which obtains the degradation path from the trained teacher model to the untrained initial student model. 

6: The degradation path implicitly represents the intermediate distributions of teacher models.

7: Then \textit{Distribution Backtracking} trains a student generator to backtrack the intermediate distributions for approximating the convergence trajectory of teacher models.

8: Extensive experiments show that the DisBack achieves faster and better convergence than the existing distillation method and accomplishes comparable generation performance.

9: Notably, DisBack is easy to implement and can be generalized to existing distillation methods to boost performance. Please find more details of this paper at \url{https://github.com/SYZhang0805/DisBack}.

10: \end{abstract}

11: