TY - GEN
T1 - Prediction of Cognitive Scores by Joint Use of Movie-Watching fMRI Connectivity and Eye Tracking via Attention-CensNet
AU - Gao, Jiaxing
AU - Zhao, Lin
AU - Zhong, Tianyang
AU - Li, Changhe
AU - He, Zhibin
AU - Wei, Yaonai
AU - Zhang, Shu
AU - Guo, Lei
AU - Liu, Tianming
AU - Han, Junwei
AU - Zhang, Tuo
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.
PY - 2023
Y1 - 2023
N2 - Brain functional connectivity under the naturalistic paradigm has been demonstrated to be better at predicting individual behaviors than other brain states, such as rest and task. Nevertheless, the state-of-the-art methods are difficult to achieve desirable results from movie-watching paradigm fMRI(mfMRI) induced brain functional connectivity, especially when the datasets are small, because it is difficult to quantify how much useful dynamic information can be extracted from a single mfMRI modality to describe the state of the brain. Eye tracking, becoming popular due to its portability and less expense, can provide abundant behavioral features related to the output of human’s cognition, and thus might supplement the mfMRI in observing subjects’ subconscious behaviors. However, there are very few works on how to effectively integrate the multimodal information to strengthen the performance by unified framework. To this end, an effective fusion approach with mfMRI and eye tracking, based on Convolution with Edge-Node Switching in Graph Neural Networks (CensNet), is proposed in this article, with subjects taken as nodes, mfMRI derived functional connectivity as node feature, different eye tracking features used to compute similarity between subjects to construct heterogeneous graph edges. By taking multiple graphs as different channels, we introduce squeeze-and-excitation attention module to CensNet (A-CensNet) to integrate graph embeddings from multiple channels into one. The experiments demonstrate the proposed model outperforms the one using single modality, single channel and state-of-the-art methods. The results suggest that brain functional activities and eye behaviors might complement each other in interpreting trait-like phenotypes. Our code will make public later.
AB - Brain functional connectivity under the naturalistic paradigm has been demonstrated to be better at predicting individual behaviors than other brain states, such as rest and task. Nevertheless, the state-of-the-art methods are difficult to achieve desirable results from movie-watching paradigm fMRI(mfMRI) induced brain functional connectivity, especially when the datasets are small, because it is difficult to quantify how much useful dynamic information can be extracted from a single mfMRI modality to describe the state of the brain. Eye tracking, becoming popular due to its portability and less expense, can provide abundant behavioral features related to the output of human’s cognition, and thus might supplement the mfMRI in observing subjects’ subconscious behaviors. However, there are very few works on how to effectively integrate the multimodal information to strengthen the performance by unified framework. To this end, an effective fusion approach with mfMRI and eye tracking, based on Convolution with Edge-Node Switching in Graph Neural Networks (CensNet), is proposed in this article, with subjects taken as nodes, mfMRI derived functional connectivity as node feature, different eye tracking features used to compute similarity between subjects to construct heterogeneous graph edges. By taking multiple graphs as different channels, we introduce squeeze-and-excitation attention module to CensNet (A-CensNet) to integrate graph embeddings from multiple channels into one. The experiments demonstrate the proposed model outperforms the one using single modality, single channel and state-of-the-art methods. The results suggest that brain functional activities and eye behaviors might complement each other in interpreting trait-like phenotypes. Our code will make public later.
KW - Attention
KW - CensNet
KW - Eye Movement
KW - Functional Connectivity
KW - Naturalistic Stimulus
UR - http://www.scopus.com/inward/record.url?scp=85174690727&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-43895-0_27
DO - 10.1007/978-3-031-43895-0_27
M3 - 会议稿件
AN - SCOPUS:85174690727
SN - 9783031438943
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 287
EP - 296
BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 - 26th International Conference, Proceedings
A2 - Greenspan, Hayit
A2 - Greenspan, Hayit
A2 - Madabhushi, Anant
A2 - Mousavi, Parvin
A2 - Salcudean, Septimiu
A2 - Duncan, James
A2 - Syeda-Mahmood, Tanveer
A2 - Taylor, Russell
PB - Springer Science and Business Media Deutschland GmbH
T2 - 26th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2023
Y2 - 8 October 2023 through 12 October 2023
ER -