A hybrid neural network hidden Markov model approach for automatic story segmentation

Jia Yu, Lei Xie, Xiong Xiao, Eng Siong Chng

Research output: Contribution to journalArticlepeer-review

13 Scopus citations

Abstract

We propose a hybrid neural network hidden Markov model (NN-HMM) approach for automatic story segmentation. A story is treated as an instance of an underlying topic (a hidden state) and words are generated from the distribution of the topic. The transition from one topic to another indicates a story boundary. Different from the traditional HMM approach, in which the emission probability of each state is calculated from a topic-dependent language model, we use deep neural network (DNN) to directly map the word distribution into topic posterior probabilities. DNN is known to be able to learn meaningful continuous features for words and hence has better discriminative and generalization capability than n-gram models. Specifically, we investigate three neural network structures: a feed-forward neural network, a recurrent neural network with long short-term memory cells (LSTM-RNN) and a modified LSTM-RNN with multi-task learning ability. Experimental results on the TDT2 corpus show that the proposed NN-HMM approach outperforms the traditional HMM approach significantly and achieves state-of-the-art performance in story segmentation.

Original languageEnglish
Pages (from-to)925-936
Number of pages12
JournalJournal of Ambient Intelligence and Humanized Computing
Volume8
Issue number6
DOIs
StatePublished - 1 Nov 2017

Keywords

  • Hidden Markov model
  • Long short-term memory
  • Multi-task learning
  • Neural network
  • Story segmentation
  • Topic modeling

Fingerprint

Dive into the research topics of 'A hybrid neural network hidden Markov model approach for automatic story segmentation'. Together they form a unique fingerprint.

Cite this