1: \begin{abstract}

2:   

3: This paper introduces a new method for minimizing matrix-smooth non-convex objectives through the use of novel Compressed Gradient Descent (CGD) algorithms enhanced with a matrix-valued stepsize. 

4: The proposed algorithms are theoretically analyzed first in the single-node and subsequently in the distributed settings. 

5: Our theoretical results reveal that the matrix stepsize in CGD can capture the objective's structure and lead to faster convergence compared to a scalar stepsize. 

6: As a byproduct of our general results, we emphasize the importance of selecting the compression mechanism and the matrix stepsize in a layer-wise manner, taking advantage of model structure. 

7: Moreover, we provide theoretical guarantees for free compression, by designing specific layer-wise compressors for the non-convex matrix smooth objectives. Our findings are supported with empirical evidence.\footnote{This work was supported by the KAUST Baseline Research Funding Scheme.}

8: 

9: \end{abstract}

10: