Multimodal continuous affect recognition based on LSTM and multiple kernel learning

Jiamei Wei, Ercheng Pei, Dongmei Jiang, Hichem Sahli, Lei Xie, Zhonghua Fu

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

16 Scopus citations

Abstract

In this paper, we propose a Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) and multiple kernel learning (MKL) based multi-modal affect recognition scheme (LSTM-MKL). It takes the LSTM-RNN advantage to model the long range dependencies between successive observations, and uses the MKL power to model the non-linear correlations between the inputs and outputs. For each of the affect dimensions (arousal, valence, expectancy, and power), two LSTM-RNN models are trained, one for each modality. In the recognition phase, the audio and visual features are input to the corresponding learned LSTM models, which in turn produce initial estimates of the affect dimensions. The LSTM outputs are further input into a multi-kernel support vector regression (MK-SVR) for the final recognition. Experimental results carried out on the AVEC2012 database, show that compared to the traditional SVR-LLR (Support Vector Machine - local linear regression) or MK-SVR fusion scheme, the proposed LSTM-MKL fusion scheme obtains higher recognition results, with an correlation coefficient (COR) of 0.354, compared to a COR of 0.124 for SVR-LLR, and 0.168 for MK-SVR, respectively.

Original languageEnglish
Title of host publication2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9786163618238
DOIs
StatePublished - 12 Feb 2014
Event2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014 - Chiang Mai, Thailand
Duration: 9 Dec 201412 Dec 2014

Publication series

Name2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014

Conference

Conference2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014
Country/TerritoryThailand
CityChiang Mai
Period9/12/1412/12/14

Fingerprint

Dive into the research topics of 'Multimodal continuous affect recognition based on LSTM and multiple kernel learning'. Together they form a unique fingerprint.

Cite this