Deep Learning and Dynamical Modeling Frameworkfor EEG-Based Cognitive State Evolution inBrain-Computer Interfaces

Brain-computer interfaces (BCIs) require reliable cognitive state monitoring to ensure safe and effective operation. While deeplearning approaches have shown promise for EEG-based state detection, they lack temporal consistency and mechanisticinterpretability, limiting clinical applicability. We present a novel unified framework integrating Long Short-Term Memory (LSTM)networks with a three-state compartmental Ordinary Differential Equation (ODE) model for interpretable cognitive state evolutionin BCIs. The framework employs a probabilistic coupling mechanism where LSTM classification probabilities dynamicallymodulate ODE transition rates between Active, Passive, and Fatigued cognitive states. We evaluated the framework on theOpenNeuro ds004148 dataset (60 participants, 61 channels, 500 Hz) using stratified temporal cross-validation. The integratedLSTM-ODE framework achieved 85.57% accuracy (95% CI: 85.19–85.93%) with F1-score of 0.854, outperforming traditionalmachine learning baselines including Random Forest (82.29%) and XGBoost (80.92%). Explainability analysis revealedanterior frontal EEG channels (AF7, AF3, Fp1) as primary contributors with gradient-based, permutation, and SHAP importancemethods. Learned ODE parameters showed physiologically meaningful dynamics: the Passive-to-Fatigued transition ratedominated (time constant 3.33 seconds), indicating rapid vigilance decrement, while full recovery was substantially slower (timeconstant 100 seconds). The fatigue accumulation ratio of 0.91 indicates strongly asymmetric dynamics where fatigue-directedtransitions dominate over recovery. This framework establishes a principled approach to combining deep learning patternrecognition with mechanistic dynamical modeling, providing clinically meaningful transition rate interpretations for proactive BCIsafety interventions