Bridging Natruralistic Stimuli, Eye Movement And Brain Activity Via Cca And Locality Preserving Projection

Naturalistic stimuli provide more life-like multimodal inputs and promise to investigate brain activity and cognition. Many studies have succeeded in bridging multimedia features and semantics derived from brain responses. In this naturalistic stimulus paradigm, eye movement was widely used as a behavior measurement, which was also demonstrated to closely related to cognitive processes of individuals. However, there lacks an integrative work to bridge these three modalities ranging from low-level multimedia feature, high-level brain semantics, to human behavior. In this work, we adopted the canonical correlation analysis (CCA) method with the locality preserving projection (LPP) to correlate functional MRI based brain responses to multimedia features under the guidance of eye movement types. Canonical correlation was estimated between brain responses and multimedia features while the LPP constraint is used to preserve the temporal relationship by maximizing the discrimination between time points which have different eye movement types. Brain regions identified by our method exhibit a gradient on the cortex, and both selected brain regions and multimedia features have higher distinguishing ability among eye movement types than other unselected ones, suggestive of the validity of the method.