3D Convolutional Long-Short Term Memory Network for Spatiotemporal Modeling of fMRI Data

Complex spatiotemporal correlation and dependency embedded in functional magnetic resonance imaging (fMRI) data introduce critical challenges in related analytical methodologies. Despite remarkable successes, most of existing approaches only model spatial or temporal dependency alone and the development of a unified spatiotemporal model is still a challenge. Meanwhile, the recent emergence of deep neural networks has provided powerful models for interpreting complex spatiotemporal data. Here, we proposed a novel convolutional long-short term memory network (3DCLN) for spatiotemporal modeling of fMRI data. The proposed model is designed to decode fMRI volumes belonging to different task events by joint training a 3D convolutional neural network (CNN) for spatial dependency modeling and a long short-term memory (LSTM) network for temporal dependency modeling. We also designed a 3D deconvolution scheme for fMRI sequence reconstruction to inspect the feature learning process in the 3DCLN. The experimental results on the motor task-fMRI data from Human Connectome Project (HCP) showed that fMRI volumes can be decoded with a relatively high accuracy (76.38%). More importantly, the proposed 3DCLN can dramatically remove noises and highlights signals of interest in the reconstructed fMRI sequence and hence improve the performance of activation detection, validating the spatiotemporal feature learning in the proposed 3DCLN model.