Dybrainformer: Decoding Dynamic Brain Semantics with Hierarchical Transformer for Brainmultimedia Association

Exploring the association between high-level semantic brain responses and multimedia features is crucial for understanding the human semantic processing mechanism. However, a significant 'semantic gap' persists between abstract brain representations captured by functional Magnetic Resonance Imaging (fMRI) and concrete multimedia features, remaining both unclear and challenging to quantify. To address this, we introduce DyBrainFormer, a novel Transformer-based Brain Dynamics Decoder for Brain-Multimedia Association. Inspired by the topological structure and dynamic properties of the human brain, DyBrainFormer uniquely integrates Graph Convolutional Networks (GCNs) and Hierarchical Temporal Transformer (HTT). It first encodes each sequenced dynamic brain graph using GCNs to capture spatial dependencies and derive brain temporal node attention. Subsequently, these temporal graph representations are fed into the HTT, which excels at modeling complex dynamic changes and long-range temporal dependencies within brain networks. The learned temporal weights from HTT serve as interpretable semantic descriptors, forming a quantifiable bridge that links high-level brain semantics to dynamic multimedia features. Evaluated on the Healthy Brain Network naturalistic fMRI dataset, DyBrainFormer effectively learns distinguishable brain dynamics, achieving ∼ 83% classification accuracy in differentiating between children and adolescents. Our analysis further identifies distinct age-related patterns in semantic processing, demonstrating that children emphasize perceptual features while adolescents focus on higher-level conceptual elements. This work provides important references for bridging the semantic gap by establishing a robust and interpretable link between high-level semantic features and multimedia features, offering a novel perspective to uncover the human semantic understanding mechanism.