Distinguishing sleepiness from mental fatigue in sustained monitoring tasks to enhance the reliability of fatigue detection based on multimodal fusion

In monitoring tasks involving sustained interaction with display systems, fatigue is a primary factor diminishing efficiency. Traditional models confuse sleepiness with mental fatigue, which compromises the reliability of assessments. We propose an explainable multimodal framework that models these two subtypes separately and integrates them into a comprehensive fatigue assessment. To validate our methodology, we invited 20 pilots to participate in a 90-minute continuous monitoring experiment, during which we collected multimodal data including their eye movements, electroencephalogram (EEG), electrocardiogram (ECG), and video. First, we derive explicit representation functions for sleepiness and mental fatigue using symbolic regression on facial and behavioral cues, enabling continuous subtype related labeling beyond intermittent questionnaires. Second, we identify compact physiological marker subsets via a cascaded feature selection method that combines mRMR prescreening with a heuristic search, yielding key feature sets while substantially reducing dimensionality. Finally, dynamic weighted coupling analysis based on information entropy revealed the nonlinear superposition effects between sleepiness and mental fatigue. Using 30 s windows under the current cohort and evaluation setting, the resulting comprehensive classifier achieves 94.8% accuracy. Following external validation and domain-specific adaptations, the methodology developed in this study holds broad application prospects across numerous automation scenarios involving monotonous human–machine interaction tasks.