An end-to-end neural network approach to story segmentation

This paper proposes an end-to-end story segmentation approach based on long short-term memory (LSTM) - recurrent neural network (RNN). Traditional story segmentation approaches are a two-stage pipeline consisting of feature extraction and segmentation, each of which has its individual objective function. In other words, the objective function used to extract features is different from the true performance measure of story segmentation, which may degrade the segmentation results. In this paper, we combine the two components and optimize them jointly, using an LSTM-RNN. Specifically, one LSTM layer is used to extract sentence vectors, and another LSTM layer is used to predict story boundaries by taking as input of the sentence vectors. Importantly, the whole network is optimized directly to predict story boundaries. We also investigate bi-directional LSTM (BLSTM) that can utilize past and future information in the process of extracting sentence vectors and story boundary prediction. Experimental results on the TDT2 corpus show that the proposed approach achieves state-of-the-art performance in story segmentation.