Paper deep dive
MUPA$^{2}$E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment
Stefanos Gkikas, Eric Nichols, Christian Arzate Cruz, Randy Gomez
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/22/2026, 3:25:27 AM
Summary
The paper introduces MUPA2E, a unified multimodal framework for emotion assessment that processes facial video and EEG signals through a single shared asymmetric-attention backbone. Evaluated on the DMER dataset, the model achieves 70.07% test accuracy with zero-padded recordings, but this performance is attributed to duration-related cues. When controlling for recording duration by cropping to 20 seconds, the accuracy drops to 62.71%, highlighting the importance of rigorous evaluation protocols in affective computing.
Entities (6)
Relation Signals (6)
MUPA2E → evaluatedon → DMER Dataset
confidence 98% · The framework is evaluated on the DMER dataset
MUPA2E → processes → facial video
confidence 95% · MUPA2E... processes facial video and electroencephalography (EEG)
MUPA2E → processes → EEG
confidence 95% · MUPA2E... processes facial video and electroencephalography (EEG)
MUPA2E → uses → asymmetric attention
confidence 95% · MUPA2E... processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone.
MUPA2E → achievesaccuracy → 70.07%
confidence 90% · merged fusion at stride 30 achieves the highest validation performance and a test accuracy of 70.07%
MUPA2E → achievesaccuracy → 62.71%
confidence 90% · cropping all recordings to a common duration of 20 seconds yielded a test accuracy of 62.71%
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion. This paper presents MUPA\textsuperscript{2}E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone. Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion. The framework is evaluated on the DMER dataset under a stratified subject-independent protocol, comparing unimodal video, unimodal EEG, and fused video--EEG configurations with per-channel and merged EEG projections. Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride~$30$ achieves the highest validation performance and a test accuracy of $70.07\%$. Further analysis revealed that recording duration is unevenly distributed across the affective classes, making the padding pattern a potential classification cue. Controlling for this factor by cropping all recordings to a common duration of $20$ seconds yielded a test accuracy of $62.71\%$, providing a stricter duration-controlled assessment of the framework in which differences in recording length are removed as a potential classification cue. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in affective datasets.
Tags
Links
- Source: https://arxiv.org/abs/2608.15999v1
- Canonical: https://arxiv.org/abs/2608.15999v1
Trouble viewing inline? Open PDF directly →
Full Text
41,186 characters extracted from source content.
Expand or collapse full text
MUPA2E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment Stefanos Gkikas Affiliation: Honda Research Institute Japan, Wako City, Japan Eric Nichols Affiliation: E-mail stefanos.gkikas, e.nichols, christian.arzate, r.gomez@jp.honda-ri.com Christian Arzate Cruz Randy Gomez Abstract Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion. This paper presents MUPA2E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone. Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion. The framework is evaluated on the DMER dataset under a stratified subject-independent protocol, comparing unimodal video, unimodal EEG, and fused video–EEG configurations with per-channel and merged EEG projections. Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride 3030 achieves the highest validation performance and a test accuracy of 70.07%70.07\%. Further analysis revealed that recording duration is unevenly distributed across the affective classes, making the padding pattern a potential classification cue. Controlling for this factor by cropping all recordings to a common duration of 2020 seconds yielded a test accuracy of 62.71%62.71\%, providing a stricter duration-controlled assessment of the framework in which differences in recording length are removed as a potential classification cue. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in affective datasets. Keywords: Electroencephalogram, emotion assessment, affective computing, attention 1 Introduction Automatic recognition of human emotional states is a core challenge in affective computing, underpinning systems designed to sense, interpret, and respond to users’ affective states [25]. Emotional experience is most commonly represented along two independent dimensions: valence, reflecting the positive-to-negative quality of an affective state, and arousal, reflecting its level of activation [26]. Standardized self-report instruments, such as the Positive and Negative Affect Schedule, capture positive and negative affect and have been widely adopted for ground-truth data collection in affective computing studies [31]. The Self-Assessment Manikin provides a direct measure of valence, arousal, and dominance and has been applied extensively for affective labeling in laboratory settings [2]. These self-report methods depend on explicit user participation, motivating the development of automatic recognition systems that passively infer emotional states from physiological and behavioral signals, with applications ranging from human-computer interaction to social robotics platforms that generate emotion-aware co-speech gestures [22] to body-based emotion recognition systems [3]. Physiological biosignals have been studied extensively as passive channels for monitoring stress-related affective responses without relying on explicit self-report [6]. Deep learning has driven substantial progress in stress detection from heterogeneous data sources, including physiological signals, facial expressions, speech, gestures, and other behavioral data [18]. Among physiological modalities, EEG provides direct access to central nervous system correlates of emotional states, encoding neural dynamics that peripheral biosignals, such as electrodermal activity or heart rate, cannot capture directly [1]. Early approaches represented EEG using hand-crafted spectral features, with differential entropy computed across frequency bands, establishing a widely adopted representation for emotion classification [4]. Subsequent work identified critical frequency bands and electrode channels as primary carriers of emotion-relevant information, motivating spatially and spectrally selective architectures [33]. Graph convolutional networks advanced this direction by modeling multichannel EEG features as graphs and learning dynamic inter-channel relationships for emotion recognition [28]. Convolutional transformer architectures have further improved EEG decoding by jointly modeling local temporal structure and global sequential dependencies [29]. Multimodal affective datasets pairing EEG recordings, peripheral physiological signals, affective stimuli, and subjective ratings have provided important benchmarks for EEG-based emotion recognition research [16]. Facial data provide a complementary and non-intrusive channel for affective sensing, with large-scale in-the-wild databases demonstrating the feasibility of recognizing facial expressions, valence, and arousal from unconstrained facial images [21]. Deep learning architectures for facial expression recognition have advanced substantially, with spatiotemporal models learning to capture both spatial appearance and temporal dynamics across sequences of frames [20]. Continuous valence and arousal estimation from facial video in naturalistic conditions has further confirmed the viability of video as a scalable affective sensing modality [17]. The face encodes affect-related cues through observable dynamics, including eye activity, mouth movements, head motion, and facial appearance changes, which have been studied for stress and anxiety recognition from video [5]. Joint analysis of EEG and facial expressions has demonstrated the added value of combining neural and behavioral information for continuous emotion detection [27]. EEG and facial video are complementary modalities for emotion recognition: neural signals reflect internal cortical processing, while facial behavior encodes its external expression; combining the two has been shown to provide non-redundant discriminative information [27]. Feature-level fusion of EEG trials and facial video sequences has improved upon unimodal baselines in multi-class emotion classification [23]. Multimodal learning frameworks have been extended to affective categories requiring the joint representation of co-occurring emotional components [19]. Recent work combining facial video and physiological biosignals illustrates this pattern, using modality-specific facial and physiological representations before applying fusion strategies [30]. This design introduces choices that are difficult to generalize and prevents both signals from being processed within a unified computational framework. In this work, we propose MUPA2E, a framework for multimodal emotion assessment that processes facial video and EEG through a single shared asymmetric attention backbone. Three configurations are evaluated within a common architecture: unimodal facial video, unimodal EEG as a raw multichannel waveform, and multimodal fusion of both, all assessed using a stratified, subject-independent evaluation protocol. 2 Related Work Automatic emotion recognition from physiological and behavioral signals has been studied across multiple modalities, including EEG recordings, peripheral physiological signals, and facial video paired with subjective affective ratings [16]. EEG occupies a distinct position among physiological modalities, capturing central nervous system correlates of affective states with high temporal resolution and encoding neural dynamics that peripheral biosignals cannot directly reflect [1]. EEG-based emotion recognition pipelines have combined hand-crafted spectral representations with deep neural architectures, with differential entropy computed across the delta, theta, alpha, beta, and gamma frequency bands establishing an early and widely adopted feature extraction approach [4]. Deep networks trained on spectral EEG representations further demonstrated the effectiveness of frequency-band and channel-specific information for EEG-based emotion recognition across subjects [33]. Graph convolutional networks subsequently improved on static feature approaches by modeling dynamic inter-channel relationships in EEG-based emotion recognition [28], while convolutional transformer architectures have further advanced the modeling of local and global structure in EEG decoding [29]. More recently, multi-scale temporal modeling combined with a dynamic fusion strategy has been employed for EEG-based emotion recognition, on the dataset also used in this work [10]. Facial video provides a behavioral complement to EEG-based approaches, encoding affect-related cues through facial muscle activity, eye movements, head motion, and dynamic changes in appearance [5]. Graph attention networks operating on differential facial action units have further demonstrated that the relational structure between facial muscle activations carries discriminative information for affective state recognition [15]. Visual affect prediction from facial video has also been studied in the wild, with deep architectures that combine convolutional and recurrent layers demonstrating continuous estimation of valence and arousal from spontaneous facial behavior [17]. Joint analysis of EEG and facial expressions has further demonstrated the value of combining neural and facial information for continuous emotion detection [27], motivating the investigation of architectures that process both channels within a unified framework. Multimodal fusion of EEG and facial video has been explored to capture complementary aspects of affective responses, motivated by the idea that neural and behavioral signals reflect different components of emotional processing. Feature-level fusion of EEG trials and facial video sequences has produced improvements over unimodal baselines in multi-class emotion classification [23], while broader multimodal affect-recognition research has extended learning frameworks to mixed-emotion settings that require the joint representation of co-occurring affective components [19]. Existing fusion pipelines apply modality-specific preprocessing and feature extraction before combining representations at the feature or decision level, introducing design choices that are difficult to generalize across tasks and modality combinations. A unified architecture that processes EEG and facial video through a shared backbone, without modality-specific inductive biases in the core model, has not been systematically explored for valence-based emotion classification. Modality-agnostic architectures have been investigated in related human state assessment tasks, processing facial videos alongside functional near-infrared spectroscopy data within a single transformer framework [13], and recognizing human states from diverse modalities using a single model [8, 9, 7]. These studies indicate that a shared backbone can accommodate behavioral and neural inputs without modality-specific feature extractors, although they target pain rather than valence. 3 Methodology 3.1 Signal Representation The framework maps facial video and EEG recordings into token sequences, which are processed by a shared asymmetric-attention backbone, enabling evaluation of unimodal and multimodal configurations under a common architecture. Each trial consists of multiple stored segments, which are temporally sorted and concatenated into a complete recording; trials with fewer segments than the maximum are zero-padded to a fixed length. Facial video For a video of LvL_v retained frames, each a 224×224224× 224 RGB image, the temporal dimension is folded into the channel dimension, a step referred to as axis folding: v∈ℝB×H×W×3Lv,X_v ^B× H× W× 3L_v, (1) where B is the batch size and H=W=224H=W=224. Projecting a one-dimensional biosignal into a visual representation prior to classification has been shown to outperform processing the raw waveform directly in stress detection from electrodermal activity [11]. Geometric information is incorporated by encoding each spatial position ∈[−1,1]2p∈[-1,1]^2 with Fourier features, using K=6K=6 frequency bands and a maximum frequency fmax=10f_ =10. Since the input has D=2D=2 spatial axes, the encoding adds D(2K+1)=26D(2K+1)=26 positional features per token: γ()=[sin(πs1),cos(πs1),…,sin(πsK),cos(πsK),], splitγ(p)= [& (π s_1p),\ (π s_1p),\ …,\\ & (π s_Kp),\ (π s_Kp),\ p ], split (2) where skk=1K\s_k\_k=1^K spans [1,fmax/2][1,f_ /2]. The spatial axes are flattened into N=H×W=50176N=H× W=50176 tokens, with data channels and positional features concatenated per token to form the token matrix: v∈ℝB×N×Cv′,Cv′=3Lv+26.T_v ^B× N× C _v, C _v=3L_v+26. (3) The token sequence is partitioned into S=32S=32 contiguous spatial segments of length ns=N/S=1568n_s=N/S=1568. EEG waveform For unimodal EEG processing, the raw multichannel waveform is used directly without hand-crafted feature extraction. All temporal segments of a trial are concatenated along the time axis, yielding: e∈ℝB×Ce×Le,X_e ^B× C_e× L_e, (4) where CeC_e is the number of selected EEG channels and LeL_e is the total concatenated waveform length. Each normalized time position t∈[−1,1]t∈[-1,1] is encoded using Fourier features with K=6K=6 bands (D=1D=1), adding 2K+1=132K+1=13 positional features per token. The time axis is flattened into LeL_e tokens, with waveform channels and positional features concatenated at each step to form the token matrix: e∈ℝB×Le×Ce′,Ce′=Ce+13.T_e ^B× L_e× C _e, C _e=C_e+13. (5) The token sequence is partitioned into S=32S=32 contiguous temporal segments of length ns=⌈Le/S⌉n_s= L_e/S , with the final segment zero-padded if necessary. 3.2 Multimodal Channel Fusion In the multimodal configuration, the EEG waveform is projected into the 2D spatial domain and fused with the video tensor along the channel dimension, enabling joint processing of both modalities by a shared 2D backbone. Before projection, the EEG waveform is standardized via z-score normalization followed by min-max rescaling to [0,1][0,1]. In the merged configuration, this normalization is applied jointly over the full Ce×LeC_e× L_e amplitude grid; in the per-channel configuration, it is applied independently to each channel prior to stacking. Two projection variants are evaluated. Merged All CeC_e channels are treated jointly as a Ce×LeC_e× L_e amplitude grid and projected to a single image via bilinear interpolation: emerged∈ℝB×H×W×1.X_e^merged ^B× H× W× 1. (6) Per-channel Each channel is projected independently to a 224×224224× 224 image via bilinear interpolation, and the results are stacked along the channel dimension: eper∈ℝB×H×W×Ce.X_e^per ^B× H× W× C_e. (7) In both cases, the EEG image is concatenated with the axis-folded video tensor along the channel dimension: m=[v∥e]∈ℝB×H×W×(3Lv+ne),X_m= [X_v\ \|\ X_e ] ^B× H× W×(3L_v+n_e), (8) where ne=1n_e=1 for merged and ne=Cen_e=C_e for per-channel. The fused tensor is tokenized identically to the unimodal video case: flattened into N=50176N=50176 spatial tokens, augmented with 26 Fourier positional features, and partitioned into S=32S=32 segments of ns=1568n_s=1568 tokens each. 3.3 Asymmetric Attention All configurations share a common asymmetric-attention backbone of depth 11, comprising a single cross-attention block with a feed-forward sublayer, followed by R=8R=8 self-attention rounds, each with a feed-forward sublayer. A single cross-attention module operating over a shared latent state has previously been employed for physiological signal classification, yielding competitive performance at a low parameter count [12]. A latent state (0)∈ℝd0e^(0) ^d_0, with d0=128d_0=128, is shared across all S=32S=32 segments and instantiated at runtime from a set of M0=64M_0=64 learnable latent vectors ℓmm=1M0\ _m\_m=1^M_0: ℓinit=1M0∑m=1M0ℓm∈ℝd0. _init= 1M_0 _m=1^M_0 _m ^d_0. (9) Cross-attention Each segment state aggregates information exclusively from its corresponding token subset: s=(0)+Attn((0),~s),e_s=e^(0)+Attn\! (e^(0),\ T_s ), (10) where (0)∈ℝB×1×d0e^(0) ^B× 1× d_0 provides the queries and ~s∈ℝB×ns×C′ T_s ^B× n_s× C provides the keys and values. The operation is asymmetric: the query is a single vector while the key-value side spans ns≫1n_s 1 tokens. All S segments are processed in parallel. Cross-attention uses 88 heads, each with a head dimension of 1616. Self-attention After cross-attention, all segment states are stacked to form ∈ℝB×S×d0E ^B× S× d_0 and processed through R=8R=8 self-attention rounds, enabling global information exchange across all segments: ←+Attn(,).E +Attn\! (E,\ E ). (11) Self-attention also uses 88 heads, each with a head dimension of 1616. All attention and feed-forward sublayers use pre-layer normalization and residual connections, with attention and feed-forward dropout of 0.100.10 applied uniformly. After the final round, segment states are averaged over S and passed through a linear classification head to predict binary valence (positive vs. negative). 3.4 Augmentation & Regularization Several augmentation strategies were applied independently to each modality. Video frames were augmented using TrivialAugment [24], AugMix [14], additive noise, center cropping, and spatial masking. A shared random seed was applied across all frames of a video sample to ensure temporal consistency. EEG waveforms were augmented per channel using additive noise and temporal masking, applied prior to 2D projection in the multimodal configuration. Regularization included label smoothing, attention dropout, and feed-forward dropout. Table 1 summarizes the complete augmentation, regularization, and training configuration used across all experiments. Table 1: Augmentation, regularization, and training configuration. Method/Parameter Value Video augmentations AugMix p∈[0.20,0.40]p∈[0.20,0.40] TrivialAugment p∈[0.20,0.40]p∈[0.20,0.40] Center Crop p∈[0.20,0.40]p∈[0.20,0.40], 200×200200×200 Noise p∈[0.20,0.40]p∈[0.20,0.40], σ=100σ=100 Masking-1 p∈[0.20,0.80]p∈[0.20,0.80], 3 blocks Masking-2 p∈[0.20,0.40]p∈[0.20,0.40], 20 blocks EEG augmentations Add Noise p∈[0.10,0.80]p∈[0.10,0.80] Temporal Masking p∈[0.10,0.80]p∈[0.10,0.80], size ∈[0.15,0.30]∈[0.15,0.30] Regularization Label Smoothing 0.150.15 Att-Dropout 0.100.10 F-Dropout 0.100.10 Training Optimizer AdamW Learning rate 10−410^-4 LR decay cosine Weight decay 0.050.05 Epochs 200200 Warmup epochs 2020 Cooldown epochs 1010 Batch size 3232 • Att-Dropout/F-Dropout: dropout probability in attention/feed-forward sublayers Masking-1/2: Cutout on video frames, square 32×3232×32; value after | = number of blocks Temporal Masking: applied per EEG channel prior to 2D projection; size = fraction of total waveform length masked at the beginning, end, or center of the waveform Notes: x1x_1–x2x_2 means we sample p∼(x1,x2)p (x_1,x_2) per sample and apply the transform with probability p. Figure 1: Overview of MUPA2E. Facial video is axis-folded into a 224×224×3L224× 224× 3L tensor; the EEG waveform (C×LC× L) is projected to a 224×224×1224× 224× 1 grid via bilinear interpolation (z=α×βz=α×β). The two representations are channel-concatenated, flattened into 5017650176 Fourier-augmented spatial tokens, and segmented into 3232 groups of 15681568 tokens, processed by the asymmetric attention backbone. 4 Experimental Evaluation & Results This section presents the experimental evaluation of a binary classification task that distinguishes positive from negative affective states. Validation performance is reported using macro-averaged accuracy, precision, and F1 score. Test performance is reported using macro-averaged accuracy. 4.1 Data Collection This study uses the DMER dataset [32], comprising 7373 participants aged 1818 to 3535 (mean 23.0623.06, SD 3.373.37); 77 of the original 8080 participants were excluded due to physiological signal quality issues. Participants viewed 3232 short video clips drawn from the Stanford film library, selected through a rule-based filtering procedure and expert evaluation to elicit positive, negative, and mixed emotional states: 88 clips for positive, 88 for negative, and 1616 for mixed. This work focuses on binary valence classification, distinguishing positive from negative affective states; trials corresponding to mixed emotional content are excluded from all experiments. EEG was recorded using the DSI-24 wireless dry-electrode system at 300300 Hz across 2121 channels following the international 1010-2020 system. The time-aligned preprocessed EEG signals provided by DMER are used directly without additional processing; preprocessing steps applied by the dataset authors include independent component analysis, bandpass filtering from 11 to 5050 Hz, 5050 Hz notch filtering, baseline removal, and re-referencing to Pz with ear electrodes excluded, yielding 1818 channels per trial. Participants are partitioned into training, validation, and testing sets at the subject level, ensuring no participant appears in more than one split. To avoid performance inflation due to subject-difficulty imbalance, a stratified split protocol is used. Leave-one-subject-out cross-validation is conducted to estimate per-subject classification difficulty, and subjects are ranked by their combined z-score and assigned to four difficulty groups. The final partition comprises 4848 training, 1212 validation, and 1313 testing subjects, with each set drawing from all four groups in proportion. The exact subject-level partition is reported in Table 2 to support reproducibility and direct comparison with future work. Table 2: Subject-level split by difficulty group. Subjects are ranked by combined z-score and assigned to four quartile-based groups (Q1 = hardest, Q4 = easiest). Split Difficulty Group Q1 – Hard Q2 – Med-Hard Q3 – Med-Easy Q4 – Easy Training (48) 8, 12, 20, 25, 36, 37, 43, 45, 56, 60, 66, 71 14, 30, 33, 35, 39, 50, 51, 53, 55, 65, 72, 73 5, 6, 9, 10, 15, 23, 32, 52, 54, 59, 70, 78 11, 24, 26, 34, 40, 42, 49, 61, 63, 64, 68, 76 Validation (12) 19, 48, 58 22, 41, 79 29, 47, 77 62, 67, 69 Testing (13) 28, 74, 75, 80 21, 38, 57 1, 2, 7 18, 44, 46 • Q1: z<−0.36z<-0.36; Q2: −0.36≤z<−0.08-0.36≤ z<-0.08; Q3: −0.08≤z<0.17-0.08≤ z<0.17; Q4: z≥0.17z≥ 0.17. 4.2 Unimodal Table 3 reports the validation performance of the unimodal video and EEG configurations. For the video modality, the average score varies only slightly across temporal stride settings, spanning 0.620.62 percentage points from 68.7268.72 at stride 1010 to 69.3469.34 at stride 1515. Strides 2020 and 3030 achieve average scores of 68.8168.81 and 69.3269.32, respectively. Accuracy, precision, and F1 remain closely aligned across all video settings, indicating that no temporal stride produces a clear and consistent advantage. The best video configuration uses a stride of 1515, achieving an average score of 69.3469.34. The unimodal EEG configuration achieves the highest unimodal performance, with an average score of 69.8169.81. This result exceeds the best video configuration by 0.470.47 percentage points in average score and by 0.520.52 percentage points in accuracy. The EEG accuracy, precision, and F1 scores are 69.7969.79, 69.8769.87, and 69.7669.76, respectively, showing balanced performance across the reported validation metrics. Table 3: Unimodal performance across stride settings. Average: arithmetic mean of Accuracy, Precision, and F1, used as the primary performance criterion throughout. Bold marks the highest Average; underline marks the second-highest. Modality Stride Performance Accuracy Precision F1 Average Video 10 68.75 69.05 68.36 68.72 Video 15 69.27 69.63 69.13 69.34 Video 20 68.75 69.05 68.63 68.81 Video 30 69.27 69.53 69.17 69.32 EEG 1 69.79 69.87 69.76 69.81 4.3 Multimodal Table 4 reports the validation performance of the multimodal fusion configurations across temporal stride settings. For per-channel fusion, the average score ranges from 68.8168.81 at stride 1010 to 69.3769.37 at stride 2020, with strides 1515 and 3030 yielding 69.3269.32 and 68.8468.84, respectively. Performance peaks at stride 2020 and then declines at stride 3030. The best per-channel result, with an average score of 69.3769.37, marginally exceeds the best unimodal video result by 0.030.03 percentage points but remains 0.440.44 percentage points below the unimodal EEG average of 69.8169.81. Merged fusion follows a different pattern. The average scores at strides 1010, 1515, and 2020 remain clustered between 68.8168.81 and 69.3469.34, consistent with the range observed for per-channel fusion and unimodal video. However, a stride of 3030 yields a clear improvement. At this setting, merged fusion achieves an accuracy of 70.3170.31, a precision of 72.6872.68, and an F1 score of 69.5269.52, yielding an average score of 70.8470.84. This is the highest validation result across all unimodal and multimodal configurations. The precision at a merged stride of 3030 also exceeds accuracy and F1 by a wider margin than in the other fusion settings. Overall, the merged stride 3030 configuration surpasses the best per-channel result by 1.471.47 percentage points in average score and the unimodal EEG configuration by 1.031.03 percentage points. Table 4: Multimodal fusion performance using Videos and EEG across stride settings. Fusion Stride Performance Accuracy Precision F1 Average Per-channel 10 68.75 69.05 68.63 68.81 Per-channel 15 69.27 69.53 69.17 69.32 Per-channel 20 69.27 69.75 69.08 69.37 Per-channel 30 68.85 69.05 68.63 68.84 Merged 10 68.75 69.05 68.63 68.81 Merged 15 69.27 69.53 69.17 69.32 Merged 20 69.27 69.63 69.13 69.34 Merged 30 70.31 72.68 69.52 70.84 • Stride refers to the video temporal stride; the EEG waveform stride is fixed at 1 in all multimodal experiments Figure 2: Average validation performance (arithmetic mean of Accuracy, Precision, and F1) as a function of video temporal stride for unimodal (a) and multimodal (b) configurations. Unimodal EEG at stride 1 is shown as a dashed reference in both panels. Table 5: Test accuracy, computational cost, and inference efficiency of the best validation-selected merged-fusion configuration. Fusion Stride Computational Cost Inference Cost Accuracy Params (M) GFLOPs Latency (ms) GPU↓ Samples/s GPU↑ Merged 30 1.89 3.13 6.55 152.65 70.07 • Computational and inference cost measured on a single unified sample (Video & EEG) at inference time on an NVIDIA A100 GPU. 4.4 The Effect of Recording Duration The experiments reported above initially used the original DMER recordings, preserving their full duration and zero-padding shorter trials as described in Section 3. After completing these experiments, we examined the distribution of trial durations and identified a potential confounding factor: 62.16%62.16\% of the positive trials last 3030 seconds, compared with only 25.04%25.04\% of the negative trials. Consequently, the amount of padding may itself provide class information. Indeed, a simple duration-based rule achieves 68.56%68.56\% accuracy, close to the 70.07%70.07\% obtained by the proposed model. We therefore performed an additional controlled experiment using the best validation-selected merged-fusion configuration, cropping all recordings to the first 2020 seconds for both modalities. This follow-up experiment removes recording duration as an available cue while retaining the original evaluation for completeness. Under the same training setup, the cropped configuration achieved 62.71%62.71\% accuracy, suggesting that recording duration contributed to performance on the original samples. To further assess subject-independent generalization under the duration-controlled setting, we evaluated the cropped configuration using leave-one-subject-out (LOSO) validation. The framework achieved a mean accuracy of 76.37%76.37\%, providing complementary evidence of its performance across participants. Finally, [10] reported 65.22%65.22\% accuracy using the same stratified hold-out protocol and 2020-second cropped EEG signals, exceeding our cropped multimodal result by 2.512.51 percentage points. Their multi-scale temporal windowing strategy may partly explain this difference, consistent with previous findings on physiological-signal analysis [12]. Table 6: Accuracy comparison across sample preprocessing and validation protocols. Study Modality Validation Protocol Samples Accuracy Ours Video+EEG Hold-out Original 70.07 Ours Video+EEG Hold-out Cropped 62.71 Ours Video+EEG LOSO Cropped 76.37 [10] EEG Hold-out Cropped 65.22 • Hold-out: is the same stratified hold-out protocol described in this study Cropped: is the same cropped approach of 20 sec described in this study 4.5 Overall Analysis & Discussion Across the unimodal configurations, EEG provides the strongest reference, achieving an average score of 69.81%69.81\%, while video performance remains relatively stable across stride settings. In the multimodal experiments, per-channel fusion remains below the EEG reference, whereas merged fusion at stride 3030 achieves the highest validation performance, with an average score of 70.84%70.84\%. The corresponding configuration reaches 70.07%70.07\% accuracy on the held-out test set while remaining compact, with 1.891.89M parameters and 3.133.13 GFLOPs. Further analysis showed that recording duration is unevenly distributed between the positive and negative classes, introducing a potential duration-related cue in the original padded samples. When all recordings were cropped to 2020 seconds, and the same merged-fusion configuration was retrained, test accuracy decreased to 62.71%62.71\%. This suggests that recording duration contributed to the performance obtained with the original samples and provides a more controlled estimate when this cue is removed. The cropped configuration also achieved 76.37%76.37\% mean accuracy under LOSO evaluation, although this result is not directly comparable with the stratified hold-out protocol because of the different training and evaluation partitions. 5 Conclusion This paper presented MUPA2E, a unified perception framework for multimodal emotion assessment in which facial video and EEG are processed through a single shared asymmetric-attention backbone. The framework supports video-only, EEG-only, and fused video–EEG configurations without requiring separate modality-specific feature-extraction backbones. Under the original padded-sample setting, EEG provides the strongest unimodal performance, while merged multimodal fusion at stride 3030 achieves the highest validation performance and a test accuracy of 70.07%70.07\%. Additional experiments with all recordings cropped to 2020 seconds reduced the hold-out accuracy to 62.71%62.71\%, suggesting that recording duration contributed to the original performance. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in emotion-recognition datasets. Acknowledgments The authors used large language model (LLM)-based tools for language editing and improvement. All scientific content, results, and conclusions are solely the work of the authors. References [1] S. M. Alarcão and M. J. Fonseca (2019) Emotions recognition using EEG signals: a survey. IEEE Transactions on Affective Computing 10 (3), p. 374–393. External Links: Document Cited by: §1, §2. [2] M. M. Bradley and P. J. Lang (1994) Measuring emotion: the self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25 (1), p. 49–59. External Links: Document Cited by: §1. [3] C. A. Cruz, S. Gkikas, and H. Asadi (2026) Efficient and interpretable body-based emotion recognition with lightweight temporal convolutional networks. Note: https://arxiv.org/abs/2607.20820Accepted at the 2026 14th International Conference on Affective Computing and Intelligent Interaction (ACII 2026) External Links: 2607.20820 Cited by: §1. [4] R. Duan, J. Zhu, and B. Lu (2013) Differential entropy feature for EEG-based emotion classification. In 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER), p. 81–84. External Links: Document Cited by: §1, §2. [5] G. Giannakakis, M. Pediaditis, D. Manousos, E. Kazantzaki, F. Chiarugi, P. G. Simos, K. Marias, and M. Tsiknakis (2017) Stress and anxiety detection using facial cues from videos. Biomedical Signal Processing and Control 31, p. 89–101. External Links: Document Cited by: §1, §2. [6] G. Giannakakis, D. Grigoriadis, K. Giannakaki, O. Simantiraki, A. Roniotis, and M. Tsiknakis (2022) Review on psychological stress detection using biosignals. IEEE Transactions on Affective Computing 13 (1), p. 440–460. External Links: Document Cited by: §1. [7] S. Gkikas, C. A. Cruz, V. Becchetti, M. U. Khan, A. Giuseppi, and R. F. Rojas (2026) A unified tokenization framework for pain recognition using heterogeneous 3d modalities. Note: Accepted at the 28th ACM International Conference on Multimodal Interaction (ICMI 2026)https://arxiv.org/abs/2607.19716 Cited by: §2. [8] S. Gkikas, C. A. Cruz, Y. Fang, L. Cao, M. U. Khan, T. Kassiotis, G. Giannakakis, R. F. Rojas, and R. Gomez (2026) A lightweight transformer for pain recognition from brain activity. Note: https://arxiv.org/abs/2604.16491Accepted at the 2026 International Conference on Electronic Engineering, Information Technology & Education (EEITE 2026) External Links: 2604.16491 Cited by: §2. [9] S. Gkikas, C. A. Cruz, C. Joseph, G. Giannakakis, and R. F. Rojas (2026) Towards a unified modality-agnostic multimodal framework for cognitive workload assessment. Note: Accepted at the 2026 14th International Conference on Affective Computing and Intelligent Interaction (ACII 2026) Cited by: §2. [10] S. Gkikas, Y. Guo, G. Li, R. F. Rojas, G. Giannakakis, and R. Gomez (2026) A multi-scale temporal framework with dynamic fusion for eeg-based emotion recognition. Note: https://arxiv.org/abs/2608.09088Accepted at the 2026 9th International Conference on Pattern Recognition and Artificial Intelligence (PRAI 2026) External Links: 2608.09088 Cited by: §2, §4.4, Table 6. [11] S. Gkikas, T. Kassiotis, Y. Guo, G. Li, E. Nichols, H. Asadi, N. Smyrnis, and G. Giannakakis (2026) Beyond the raw waveform: fusing visual representations of eda for stress detection. Note: Accepted at the 2026 9th International Conference on Pattern Recognition and Artificial Intelligence (PRAI 2026) Cited by: §3.1. [12] S. Gkikas, I. Kyprakis, and M. Tsiknakis (2025) Efficient pain recognition via respiration signals: a single cross-attention transformer multi-window fusion pipeline. In Companion Proceedings of the 27th International Conference on Multimodal Interaction, ICMI Companion ’25, New York, NY, USA, p. 70–79. External Links: Document Cited by: §3.3, §4.4. [13] S. Gkikas and M. Tsiknakis (2024) Twins-painvit: towards a modality-agnostic vision transformer framework for multimodal automatic pain assessment using facial videos and fnirs. In 2024 12th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), p. 13–21. External Links: Document Cited by: §2. [14] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan (2019) Augmix: a simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781. Cited by: §3.4. [15] T. Kassiotis, S. Gkikas, N. Smyrnis, and G. Giannakakis (2026) Explainable graph attention network for stress recognition (stressgat) via differential action units. Note: https://arxiv.org/abs/2607.20819Accepted at the 2026 14th International Conference on Affective Computing and Intelligent Interaction (ACII 2026) Cited by: §2. [16] S. Koelstra, C. Mühl, M. Soleymani, J. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras (2012) DEAP: a database for emotion analysis using physiological signals. IEEE Transactions on Affective Computing 3 (1), p. 18–31. External Links: Document Cited by: §1, §2. [17] D. Kollias, P. Tzirakis, M. A. Nicolaou, A. Papaioannou, G. Zhao, B. Schuller, I. Kotsia, and S. Zafeiriou (2019) Deep affect prediction in-the-wild: Aff-Wild database and challenge, deep architectures, and beyond. International Journal of Computer Vision 127 (6–7), p. 907–929. External Links: Document Cited by: §1, §2. [18] M. Kyrou, I. Kompatsiaris, and P. C. Petrantonakis (2025) Deep learning approaches for stress detection: a survey. IEEE Transactions on Affective Computing 16 (2), p. 499–517. External Links: Document Cited by: §1. [19] M. Li, Y. Liu, F. Liu, H. Sheng, Y. Fan, Y. Wei, M. Luo, W. Zhang, and W. Wang (2026) Memory-guided prototypical co-occurrence learning for mixed emotion recognition. External Links: 2602.20530, Document Cited by: §1, §2. [20] S. Li and W. Deng (2022) Deep facial expression recognition: a survey. IEEE Transactions on Affective Computing 13 (3), p. 1195–1215. External Links: Document Cited by: §1. [21] A. Mollahosseini, B. Hasani, and M. H. Mahoor (2019) AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing 10 (1), p. 18–31. External Links: Document Cited by: §1. [22] E. C. Montiel-Vazquez, C. A. Cruz, S. Gkikas, T. Kassiotis, G. Giannakakis, and R. Gomez (2026) Efficient emotion-aware iconic gesture prediction for robot co-speech. External Links: 2604.11417 Cited by: §1. [23] F. Muhammad, M. Hussain, and H. Aboalsamh (2023) A bimodal emotion recognition approach through the fusion of electroencephalography and facial sequences. Diagnostics 13 (5), p. 977. External Links: Document Cited by: §1, §2. [24] S. G. Müller and F. Hutter (2021) TrivialAugment: tuning-free yet state-of-the-art data augmentation. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), p. 754–762. External Links: Document Cited by: §3.4. [25] R. W. Picard (1997) Affective computing. MIT Press, Cambridge, MA. External Links: Document Cited by: §1. [26] J. A. Russell (1980) A circumplex model of affect. Journal of Personality and Social Psychology 39 (6), p. 1161–1178. External Links: Document Cited by: §1. [27] M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic (2016) Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Transactions on Affective Computing 7 (1), p. 17–28. External Links: Document Cited by: §1, §1, §2. [28] T. Song, W. Zheng, P. Song, and Z. Cui (2020) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing 11 (3), p. 532–541. External Links: Document Cited by: §1, §2. [29] Y. Song, Q. Zheng, B. Liu, and X. Gao (2023) EEG conformer: convolutional transformer for EEG decoding and visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering 31, p. 710–719. External Links: Document Cited by: §1, §2. [30] P. Valergaki, V. C. Nicodemou, I. Oikonomidis, A. Argyros, and A. Roussos (2026) Combining facial videos and biosignals for stress estimation during driving. External Links: 2601.04376 Cited by: §1. [31] D. Watson, L. A. Clark, and A. Tellegen (1988) Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of Personality and Social Psychology 54 (6), p. 1063–1070. External Links: Document Cited by: §1. [32] P. Yang, N. Liu, X. Liu, Y. Shu, W. Ji, Z. Ren, J. Sheng, M. Yu, R. Yi, D. Zhang, and Y. Liu (2024) A multimodal dataset for mixed emotion recognition. Scientific Data 11 (1), p. 847. External Links: Document, ISSN 2052-4463 Cited by: §4.1. [33] W. Zheng and B. Lu (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development 7 (3), p. 162–175. External Links: Document Cited by: §1, §2.