1: \begin{abstract}

2: 

3: Model heterogeneous federated learning (MHeteroFL) enables FL clients to collaboratively train models with heterogeneous structures in a distributed fashion. However, existing MHeteroFL methods rely on training loss to transfer knowledge between the client model and the server model, resulting in limited knowledge exchange.

4: To address this limitation, we propose the \underline{Fed}erated model heterogeneous \underline{M}atryoshka \underline{R}epresentation \underline{L}earning (\methodname{}) approach for supervised learning tasks. 

5: It adds an auxiliary small \homo model shared by clients with \hetero local models. 

6: (1) The \gen and \pers \reps extracted by the two models' feature extractors are fused by a \pers lightweight \rep projector. This step enables \rep fusion to adapt to local data distribution.

7: (2) The fused \rep is then used to construct Matryoshka representations with multi-dimensional and multi-granular embedded \reps learned by the global \homo model header and the local \hetero model header. This step facilitates multi-perspective \rep learning and improves model learning capability.

8: Theoretical analysis shows that \methodname{} achieves a $\mathcal{O}(1/T)$ non-convex convergence rate.

9: Extensive experiments on benchmark datasets demonstrate its superior  

10: model accuracy with low communication and computational costs compared to seven state-of-the-art baselines. It achieves up to $8.48\%$ and $24.94\%$ accuracy improvement compared with the \sota and the best same-category baseline, respectively.

11: \end{abstract}

12: