1: \begin{abstract}

2: The first-order optimality conditions of convexly constrained nonconvex-nonconcave min-max optimization problems formulate variational inequality problems, which are equivalent to a system of nonsmooth equations. In this paper, we propose a quasi-Newton subspace trust region (QNSTR) algorithm for the least-square problem defined by the smoothing approximation of the nonsmooth equation. Based on the structure of the least-square problem, we use an adaptive quasi-Newton formula to approximate the Hessian matrix and solve a low-dimensional strongly convex quadratic program with ellipse constraints in a subspace at each step of QNSTR algorithm. According to the structure of the adaptive quasi-Newton formula and the subspace technique, the strongly convex quadratic program at each step can be solved  efficiently.  We prove the global convergence of QNSTR algorithm to an $\epsilon$-first-order stationary point of the min-max optimization problem.

3: Moreover, we present numerical results of QNSTR algorithm with different subspaces for the mixed generative adversarial networks in eye image segmentation using real data to show the efficiency and effectiveness of QNSTR algorithm for solving large scale min-max optimization problems.

4: \end{abstract}

5: