Disentangled Representation Learning for Cross-Modal Biometric Matching

Hailong Ning, Xiangtao Zheng, Xiaoqiang Lu, Yuan Yuan

科研成果: 期刊稿件文章同行评审

34 引用 (Scopus)

摘要

Cross-modal biometric matching (CMBM) aims to determine the corresponding voice from a face, or identify the corresponding face from a voice. Recently, many CMBM methods have been proposed by forcing the distance between two modal features to be narrowed. However, these methods ignore the alignability between the two modal features. Because the feature is extracted under the supervision of identity information from single modal data, it can only reflect the identity information of single modal data. In order to address this problem, a disentangled representation learning method is proposed to disentangle the alignable latent identity factors and nonalignable the modality-dependent factors for CMBM. The proposed method consists of two main steps: 1) feature extraction and 2) disentangled representation learning. Firstly, an image feature extraction network is adopted to obtain face features, and a voice feature extraction network is applied to learn voice features. Secondly, a disentangled latent variable is explored to disentangle the latent identity factors that are shared across the modalities from the modality-dependent factors. The modality-dependent factors are filtered out, while the latent identity factors from the two modalities are enforced to be narrowed to align the same identity information. Then, the disentangled latent identity factors are considered as pure identity information to bridge the two modalities for cross-modal verification, 1:N matching, and retrieval. Note that the proposed method learns the identity information from the input face images and voice segments with only identity label as supervised information. Extensive experiments on the challenging VoxCeleb dataset demonstrate the proposed method outperforms the state-of-the-art methods.

源语言英语
页(从-至)1763-1774
页数12
期刊IEEE Transactions on Multimedia
24
DOI
出版状态已出版 - 2022

指纹

探究 'Disentangled Representation Learning for Cross-Modal Biometric Matching' 的科研主题。它们共同构成独一无二的指纹。

引用此