1: \begin{abstract}

2:     We consider stochastic unconstrained bilevel optimization problems when only the first-order gradient oracles are available. 

3:     While numerous optimization methods have been proposed for tackling bilevel problems, existing methods either tend to require possibly expensive calculations involving Hessians of lower-level objectives, or lack rigorous finite-time performance guarantees. 

4:     In this work, we propose a Fully First-order Stochastic Approximation (\algname) method, 

5:     %% \jycomment{Algorithm name?}\swcomment{both names look OK}  

6:     and study its non-asymptotic convergence properties. Specifically, we show that \algname~converges to an $\epsilon$-stationary solution of the bilevel problem after $\epsilon^{-7/2}, \epsilon^{-5/2}$, or $\epsilon^{-3/2}$ iterations (each iteration using $O(1)$ samples)  when stochastic noises are in both level objectives, only in the upper-level objective, or not present (deterministic settings), respectively. 

7:     We further show that if we employ momentum-assisted gradient estimators, the iteration complexities can be improved to $\epsilon^{-5/2}, \epsilon^{-4/2}$, or $\epsilon^{-3/2}$, respectively. 

8:     We demonstrate the superior practical performance of the proposed method over existing second-order based approaches on MNIST data-hypercleaning experiments.

9: \end{abstract}

10: