1: \begin{abstract}

2: Hierarchical decision making problems, such as bilevel programs and Stackelberg games,

3: are attracting increasing interest in both the engineering and machine learning communities.

4: Yet, existing solution methods lack either convergence guarantees or computational efficiency, due to the absence of smoothness and convexity.

5: In this work, we bridge this gap by designing a first-order hypergradient-based algorithm 

6: for single-leader multi-follower Stackelberg games

7: and mathematically establishing its convergence using technical tools from nonsmooth analysis.

8: To evaluate the \textit{hypergradient}, namely, the gradient of the upper-level objective,

9: we develop an online scheme that simultaneously computes the lower-level equilibrium and its Jacobian.

10: Crucially, this scheme exploits and preserves the original hierarchical and distributed structure of the problem, which renders it scalable

11: and privacy-preserving.

12: We numerically verify the computational efficiency and scalability of our algorithm on a large-scale hierarchical demand-response model.% and show that it significantly outperforms a popular solution approach in the literature.

13: \end{abstract}

14: