Prediction of cognitive scores by joint use of movie-watching fMRI connectivity and eye tracking via Attention-CensNet

Jiaxing Gao, Lin Zhao, Tianyang Zhong, Changhe Li, Zhibin He, Yaonei Wei, Shu Zhang, Lei Guo, Tianming Liu, Junwei Han, Xi Jiang, Tuo Zhang

Research output: Contribution to journalArticlepeer-review

6 Scopus citations

Abstract

Background: Brain functional connectivity under the naturalistic paradigm has been shown to be better at predicting individual behaviors than other brain states, such as rest and doing tasks. Nevertheless, the state-of-the-art methods have found it difficult to achieve desirable results from movie-watching paradigm functional magnetic resonance imaging (mfMRI) -induced brain functional connectivity, especially when there are fewer datasets. Incorporating other physical measurements into the prediction method may enhance accuracy. Eye tracking, becoming popular due to its portability and lower expense, can provide abundant behavioral features related to the output of human's cognition, and thus might supplement the mfMRI in observing participants' subconscious behaviors. However, there are very few studies on how to effectively integrate the multimodal information to strengthen the performance by a unified framework. Objective: A fusion approach with mfMRI and eye tracking, based on convolution with edge-node switching in graph neural networks (CensNet), is proposed in this article. Methods: In this graph model, participants are designated as nodes, mfMRI derived functional connectivity as node features, and different eye-tracking features are used to compute similarity between participants to construct heterogeneous graph edges. By taking multiple graphs as different channels, we introduce squeeze-and-excitation attention module to CensNet (A-CensNet) to integrate graph embeddings from multiple channels into one. Results: The proposed model outperforms those using a single modality and single channel, and state-of-the-art methods. Conclusions: The results indicate that brain functional activities and eye behaviors might complement each other in interpreting trait-like phenotypes.

Original languageEnglish
Article numberkkad011
JournalPsychoradiology
Volume3
DOIs
StatePublished - 2023

Keywords

  • attention
  • CensNet
  • eye movement
  • functional connectivity
  • naturalistic stimulus

Fingerprint

Dive into the research topics of 'Prediction of cognitive scores by joint use of movie-watching fMRI connectivity and eye tracking via Attention-CensNet'. Together they form a unique fingerprint.

Cite this