TY - JOUR
T1 - DBMC
T2 - Dynamic Balance Modality Control in Deep Reinforcement Learning
AU - Shi, Haobin
AU - Wang, Zehan
AU - Liu, Zihang
AU - Wang, Hao
N1 - Publisher Copyright:
© 2016 IEEE.
PY - 2026/11/1
Y1 - 2026/11/1
N2 - Multimodal reinforcement learning has gained attention due to its ability to integrate information from different modalities, enhancing the agent's perception of the environment. However, differences in data distribution, informational content, and significance across modalities lead to imbalances in the training process of the modality encoders. To address this, we propose a Dynamic Balance Modality Control (DBMC) approach, which uses the dormant ratio of modality encoders as a metric to quantify the training progress of each modality encoder and adjusts the gradients to regulate the performance of different modalities. Experiments conducted in the DeepMind Control Suite (DMC) and Robosuite environments show a clear association between the dormant ratio and encoder performance. In the Stand task, when the encoder's dormant ratio is fixed and does not update during training, the encoder with a lower dormant ratio achieves a final cumulative reward that is 44.7% higher than that of the one with a higher dormant ratio. Meanwhile, in the Swingup task, the encoder with a high dormant ratio fails to achieve reward convergence, indicating severe performance degradation. When integrated with DrQ, DBMC improves cumulative rewards by 21.9% and 10.9% on the Stand and Swingup tasks in DMC, respectively, and by 50.0%, 9.6%, and 83.8% on the Lift, Wipe, and Door tasks in Robosuite, respectively. Furthermore, DBMC can be readily integrated into DrQ, SAC-AE, CURL, and DrQv2, and remains effective in tasks involving more modalities.
AB - Multimodal reinforcement learning has gained attention due to its ability to integrate information from different modalities, enhancing the agent's perception of the environment. However, differences in data distribution, informational content, and significance across modalities lead to imbalances in the training process of the modality encoders. To address this, we propose a Dynamic Balance Modality Control (DBMC) approach, which uses the dormant ratio of modality encoders as a metric to quantify the training progress of each modality encoder and adjusts the gradients to regulate the performance of different modalities. Experiments conducted in the DeepMind Control Suite (DMC) and Robosuite environments show a clear association between the dormant ratio and encoder performance. In the Stand task, when the encoder's dormant ratio is fixed and does not update during training, the encoder with a lower dormant ratio achieves a final cumulative reward that is 44.7% higher than that of the one with a higher dormant ratio. Meanwhile, in the Swingup task, the encoder with a high dormant ratio fails to achieve reward convergence, indicating severe performance degradation. When integrated with DrQ, DBMC improves cumulative rewards by 21.9% and 10.9% on the Stand and Swingup tasks in DMC, respectively, and by 50.0%, 9.6%, and 83.8% on the Lift, Wipe, and Door tasks in Robosuite, respectively. Furthermore, DBMC can be readily integrated into DrQ, SAC-AE, CURL, and DrQv2, and remains effective in tasks involving more modalities.
KW - Reinforcement learning
KW - machine learning for robot control
KW - sensor fusion
UR - https://www.scopus.com/pages/publications/105050368459
U2 - 10.1109/LRA.2026.3732832
DO - 10.1109/LRA.2026.3732832
M3 - 文章
AN - SCOPUS:105050368459
SN - 2377-3766
VL - 11
SP - 12807
EP - 12814
JO - IEEE Robotics and Automation Letters
JF - IEEE Robotics and Automation Letters
IS - 11
ER -