TY - GEN
T1 - Adaptive Open Set Recognition with Multi-modal Joint Metric Learning
AU - Fu, Yimin
AU - Liu, Zhunga
AU - Yang, Yanbo
AU - Xu, Linfeng
AU - Lan, Hua
N1 - Publisher Copyright:
© 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.
PY - 2022
Y1 - 2022
N2 - Open set recognition (OSR) aims to simultaneously identify known classes and reject unknown classes. However, existing researches on open set recognition are usually based on single-modal data. Single-modal perception is susceptible to external interference, which may cause incorrect recognition. The multi-modal perception can be employed to improve the OSR performance thanks to the complementarity between different modalities. So we propose a new multi-modal open set recognition (MMOSR) method in this paper. The MMOSR network is constructed with joint metric learning in logit space. By doing this, it can avoid the feature representation gap between different modalities, and effectively estimate the decision boundaries. Moreover, the entropy-based adaptive weight fusion method is developed to combine the multi-modal perception information. The weights of different modalities are automatically determined according to the entropy in the logit space. A bigger entropy will lead to a smaller weight of the corresponding modality. This can effectively prevent the influence of disturbance. Scaling the fusion logits by the single-modal relative reachability further enhances the unknown detection ability. Experiments show that our method can achieve more robust open set recognition performance with multi-modal input compared with other methods.
AB - Open set recognition (OSR) aims to simultaneously identify known classes and reject unknown classes. However, existing researches on open set recognition are usually based on single-modal data. Single-modal perception is susceptible to external interference, which may cause incorrect recognition. The multi-modal perception can be employed to improve the OSR performance thanks to the complementarity between different modalities. So we propose a new multi-modal open set recognition (MMOSR) method in this paper. The MMOSR network is constructed with joint metric learning in logit space. By doing this, it can avoid the feature representation gap between different modalities, and effectively estimate the decision boundaries. Moreover, the entropy-based adaptive weight fusion method is developed to combine the multi-modal perception information. The weights of different modalities are automatically determined according to the entropy in the logit space. A bigger entropy will lead to a smaller weight of the corresponding modality. This can effectively prevent the influence of disturbance. Scaling the fusion logits by the single-modal relative reachability further enhances the unknown detection ability. Experiments show that our method can achieve more robust open set recognition performance with multi-modal input compared with other methods.
KW - Adaptive weight fusion
KW - Joint metric learning
KW - Multi-modal perception
KW - Open set recognition (OSR)
UR - http://www.scopus.com/inward/record.url?scp=85142667820&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-18907-4_49
DO - 10.1007/978-3-031-18907-4_49
M3 - 会议稿件
AN - SCOPUS:85142667820
SN - 9783031189067
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 631
EP - 644
BT - Pattern Recognition and Computer Vision - 5th Chinese Conference, PRCV 2022, Proceedings
A2 - Yu, Shiqi
A2 - Zhang, Jianguo
A2 - Zhang, Zhaoxiang
A2 - Tan, Tieniu
A2 - Yuen, Pong C.
A2 - Guo, Yike
A2 - Han, Junwei
A2 - Lai, Jianhuang
PB - Springer Science and Business Media Deutschland GmbH
T2 - 5th Chinese Conference on Pattern Recognition and Computer Vision, PRCV 2022
Y2 - 4 November 2022 through 7 November 2022
ER -