1: \begin{abstract}

2:     Conventional NAS-based pruning algorithms aim to find the sub-network with the best validation performance.

3:     However, validation performance does not successfully represent test performance, i.e., potential performance.

4:     Also, although fine-tuning the pruned network to restore the performance drop is an inevitable process, few studies have handled this issue.

5:     This paper provides a novel Ensemble Knowledge Guidance (EKG) to solve both problems at once.

6:     First, we experimentally prove that the fluctuation of loss landscape can be an effective metric to evaluate the potential performance.

7:     In order to search a sub-network with the smoothest loss landscape at a low cost, we employ EKG as a search reward.

8:     EKG utilized for the following search iteration is composed of the ensemble knowledge of interim sub-networks, i.e., the by-products of the sub-network evaluation.

9:     Next, we reuse EKG to provide a gentle and informative guidance to the pruned network while fine-tuning the pruned network.

10:     Since EKG is implemented as a memory bank in both phases, it requires a negligible cost.

11:     For example, when pruning and training ResNet-50, just 315 GPU hours are required to remove around 45.04\% of FLOPS without any performance degradation, which can operate even on a low-spec workstation.

12:     the implemented code is available at \href{https://github.com/sseung0703/EKG}{here}.

13: L:/    

14:     \renewcommand{\thefootnote}{\fnsymbol{footnote}}

15:     \footnotetext{

16:     \textbf{Acknowledgments:}

17:     This work was supported by IITP grants funded by the Korea government (MSIT) (No.2021-0-02068, AI Innovation Hub and No. 2020-0-01389, Artificial Intelligence Convergence Research Center(Inha University)), and was supported by the NRF grant funded by the Korea government (MSIT) (No. 2022R1A2C2010095 and No. 2022R1A4A1033549).

18:     }

19: \end{abstract}

20: