1: \begin{abstract}

2: Alternating direction method of multipliers (ADMM) is a popular optimization tool

3: for the composite and constrained problems in machine learning.

4: However, in many machine learning problems such as black-box learning and bandit feedback,

5: ADMM could fail because the explicit gradients of these problems are difficult

6: or even infeasible to obtain. Zeroth-order (gradient-free) methods can effectively solve these problems due to that the objective function values

7: are only required in the optimization. Recently, though there exist a few zeroth-order ADMM methods,

8: they build on the convexity of objective function.

9: Clearly, these existing zeroth-order methods are limited in many applications.

10: In the paper, thus, we propose a class of fast zeroth-order stochastic ADMM methods

11: (\emph{i.e.}, ZO-SVRG-ADMM and ZO-SAGA-ADMM) for solving nonconvex problems with multiple nonsmooth penalties, based on the

12: coordinate smoothing gradient estimator.

13: Moreover, we prove that both the ZO-SVRG-ADMM and ZO-SAGA-ADMM have convergence rate of $O(1/T)$, where $T$ denotes

14: the number of iterations.

15: In particular, our methods not only reach the best convergence rate of $O(1/T)$ for the nonconvex optimization,

16: but also are able to effectively solve many complex machine learning problems with multiple regularized penalties and constraints.

17: Finally, we conduct the experiments of black-box binary classification and structured adversarial attack on black-box deep neural network

18: to validate the efficiency of our algorithms.

19: \end{abstract}

20: