1: \begin{abstract}  

2:   Diffusion models have significantly advanced the field of generative modeling. However, training a diffusion model is computationally expensive, creating a pressing need to adapt off-the-shelf diffusion models for downstream generation tasks. Current fine-tuning methods focus on parameter-efficient transfer learning but overlook the fundamental transfer characteristics of diffusion models.

3:   In this paper, we investigate the transferability of diffusion models and observe a monotonous \textit{chain of forgetting} trend of transferability along the reverse process. Based on this observation and novel theoretical insights, we present \textit{Diff-Tuning}, a frustratingly simple transfer approach that leverages the chain of forgetting tendency. Diff-Tuning encourages the fine-tuned model to retain the pre-trained knowledge at the end of the denoising chain close to the generated data while discarding the other noise side.

4:   We conduct comprehensive experiments to evaluate Diff-Tuning, including the transfer of pre-trained Diffusion Transformer models to eight downstream generations and the adaptation of Stable Diffusion to five control conditions with ControlNet.

5:   Diff-Tuning achieves a 26\% improvement over standard fine-tuning and enhances the convergence speed of ControlNet by 24\%. Notably, parameter-efficient transfer learning techniques for diffusion models can also benefit from Diff-Tuning.

6: \end{abstract}

7: