1: \begin{abstract}

2: We aim at exploiting additional auxiliary labels from an independent (auxiliary) task to \emph{boost the primary task performance} which we focus on, while preserving \emph{a single task inference cost} of the primary task. While most existing auxiliary learning methods are optimization-based relying on loss weights/gradients manipulation, our method is architecture-based with a flexible \emph{asymmetric structure} for the primary and auxiliary tasks, which produces different networks for training and inference. Specifically, starting from two single task networks/branches (each representing a task), we propose a novel method with evolving networks where only primary-to-auxiliary links exist as the cross-task connections after convergence. These connections can be removed during the primary task inference, resulting in a single-task inference cost. We achieve this by formulating a Neural Architecture Search (NAS) problem, where we initialize bi-directional connections in the search space and guide the NAS optimization converging to an architecture with only the single-side primary-to-auxiliary connections. % Such NAS design results in a discovered architecture with only  connections at the best locations while cutting off all the auxiliary-to-primary connections. Thus, all the auxiliary related computations/parameters can also be removed accordingly during the inference. \\

3: Moreover, our method can be incorporated with optimization-based auxiliary learning approaches. Extensive experiments with \emph{six} tasks on NYU v2, CityScapes, and Taskonomy datasets using VGG, ResNet, and ViT backbones validate the promising performance. The codes are available at https://github.com/ethanygao/Aux-NAS.

4: \end{abstract}

5: