1: \begin{abstract}

2: 

3: % 2022/11/10 fyf

4: Multimodal learning (MML) aims to jointly exploit the common priors of different modalities to compensate for their inherent limitations.

5: %

6: However, existing MML methods often optimize a uniform objective for different modalities, leading to the notorious ``modality imbalance'' problem and counterproductive MML performance. 

7: %

8: To address the problem, some existing methods modulate the learning pace based on the fused modality, which is dominated by the better modality and eventually results in a limited improvement on the worse modal. 

9: %

10: To better exploit the features of multimodal, we propose Prototypical Modality Rebalance (PMR) to perform stimulation on the particular slow-learning modality without interference from other modalities.  

11: %

12: Specifically, we introduce the prototypes that represent general features for each class, to build the non-parametric classifiers for uni-modal performance evaluation. Then, we try to accelerate the slow-learning modality by enhancing its clustering toward prototypes. 

13: %

14: Furthermore, to alleviate the suppression from the dominant modality, we introduce a prototype-based entropy regularization term during the early training stage to prevent premature convergence.

15: %

16: Besides, our method only relies on the representations of each modality and without restrictions from model structures and fusion methods, making it with great application potential for various scenarios.

17: %

18: % The source code is available at. 

19: 

20: 

21: \end{abstract}

22: