An attention-based user profiling model by leveraging multi-modal social media contents

With the popularization of social media, inferring user profiles from the user-generated content has aroused wide attention for its applications in marketing, advertising, recruiting, etc. Most existing works focus on using data from single modality (such as texts and profile photos) and fail to notice that the combination of multi-modal data can supplement with each other and can therefore improve the prediction accuracy. In this paper, we propose AMUP model, namely the Attention-based Multi-modal User Profiling model, which uses different tailored neural networks to extract and fuse semantic information from three modalities, i.e., texts, avatar, and relation network. We propose a dual attention mechanism. The word-level attention network selects informative words from the noisy and prolix texts and the modality-level attention network addresses the problem of imbalanced contribution among different modalities. Experimental results on more than 1.5K users’ real-world data extracted from a popular Q&A social platform show that our proposed model outperforms the single-modality methods and achieves better accuracy when compared with existing approaches that utilize multi-modal data.