FMRI-Guided Time-Symmetric Joint Model for Visual Attention Prediction

Visual attention prediction is linked to brain activity, cognition, and behavior. Despite the availability of brain activity features, previous studies have not fully utilized them, resulting in saliency maps predicted by models primarily based on image features that do not accurately reflect visual attention in the human brain. This inspires us to use functional Magnetic Resonance Imaging (fMRI) signals as a 'brain observer' to supervise the training of developing models that integrate top-down image attention-dependent cues and supervise information from saliency maps generated from gaze movement patterns under natural stimuli. Hence, this paper presents an FMRI-Guided Time-Symmetric Joint Model to predict saliency maps from movie clips, which captures the dynamic aspects of human brain cognition and attention, enabling the combination of image features with brain features. Furthermore, we generalize the model to the MS-COCO challenge, evaluating its performance on non-movie data. Our model outperforms other brain-feature-free methods in focusing on visual attention regions of humans in both movie and non-movie datasets. Additionally, incorporating brain features improves model performance, indicating their ability to bridge the semantic gap between human cognition and visual images, allowing for more accurate capture of visual attention regions.