Paper deep dive
HSEmotion Team at the 11th ABAW Challenge: Multi-Task Learning and Ambivalence/Hesitancy Video Recognition
Aleksei Bakin, Andrey V. Savchenko
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/15/2026, 7:38:45 AM
Summary
This paper presents the HSEmotion Team's approach for the 11th ABAW Challenge, focusing on multi-task learning (MTL) for facial affect analysis and ambivalence/hesitancy (A/H) video recognition. For MTL on the s-Aff-Wild2 dataset, the team employs frozen lightweight facial extractors (MT-EmotiDDAMFN and MT-EmotiEffNet-B0) with separate task heads and systematic post-processing, including temporal smoothing, expression bias tuning, AU threshold optimization, and backbone blending. For A/H recognition on the BAH dataset, they extend an audiovisual pipeline to video-level classification by late-fusing face, HuBERT audio, and RoBERTa text features with temporal aggregation. Their method achieves competitive performance without fine-tuning heavy backbones, demonstrating that systematic calibration and lightweight multimodal fusion can rival heavier end-to-end approaches while improving efficiency.
Entities (12)
Relation Signals (11)
HSEmotion Team → participatedin → ABAW-11
confidence 97% · This article presents our results for the 11th Affective Behavior Analysis in-the-Wild (ABAW) competition.
HSEmotion Team → uses → MT-EmotiDDAMFN
confidence 95% · we use frozen lightweight facial extractors, MT-EmotiDDAMFN and MT-EmotiEffNet-B0
HSEmotion Team → uses → MT-EmotiEffNet-B0
confidence 95% · we use frozen lightweight facial extractors, MT-EmotiDDAMFN and MT-EmotiEffNet-B0
ABAW-11 → evaluates → s-Aff-Wild2
confidence 94% · For multi-task learning with simultaneous prediction of valence, arousal, facial expressions, and action units on s-Aff-Wild2 dataset
ABAW-11 → evaluates → BAH dataset
confidence 93% · For ambivalence/hesitancy video recognition on the expanded BAH dataset
HSEmotion Team → outperforms → ConvNeXt
confidence 93% · On the official validation set, our ensemble significantly exceeds the performance of the ConvNeXt baseline.
HSEmotion Team → uses → HuBERT
confidence 92% · late fusion of face, HuBERT audio, and RoBERTa text classifiers
HSEmotion Team → uses → RoBERTa
confidence 92% · late fusion of face, HuBERT audio, and RoBERTa text classifiers
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This article presents our results for the 11th Affective Behavior Analysis in-the-Wild (ABAW) competition. For multi-task learning with simultaneous prediction of valence, arousal, facial expressions, and action units on s-Aff-Wild2 dataset, we use frozen lightweight facial extractors, MT-EmotiDDAMFN and MT-EmotiEffNet-B0, with separate heads and systematic post-processing: temporal Gaussian smoothing, per-class expression bias, AffectNet blending, per-AU threshold tuning, and weighted backbone fusion. On the official validation set, our ensemble significantly exceeds the performance of the ConvNeXt baseline. For ambivalence/hesitancy video recognition on the expanded BAH dataset, we extend the audiovisual pipeline to video-level Macro F1 by late fusion of face, HuBERT audio, and RoBERTa text classifiers, temporal aggregation, and a global-text gate. Frame-level Weighted F1 on validation set rises from 0.74 in ABAW-8 to 0.79, while the best public-test video-level Macro F1 reaches 0.73. In both tasks, competitive performance is achieved without fine-tuning heavy backbones. These results indicate that systematic prediction calibration and lightweight multimodal fusion can rival substantially heavier end-to-end approaches while offering improved efficiency and deployment flexibility.
Tags
Links
- Source: https://arxiv.org/abs/2607.12774v1
- Canonical: https://arxiv.org/abs/2607.12774v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
37,170 characters extracted from source content.
Expand or collapse full text
11institutetext: Central University, Moscow, Russia 11email: bakin95a@gmail.com 22institutetext: Sber AI Lab, Moscow, Russia 33institutetext: HSE University, Laboratory of Algorithms and Technologies for Network Analysis, Nizhny Novgorod, Russia 33email: avsavchenko@hse.ru HSEmotion Team at the 11th ABAW Challenge: Multi-Task Learning and Ambivalence/Hesitancy Video Recognition Aleksei Bakin Andrey V. Savchenko Abstract This article presents our results for the 11th Affective Behavior Analysis in-the-Wild (ABAW) competition. For multi-task learning with simultaneous prediction of valence, arousal, facial expressions, and action units on s-Aff-Wild2 dataset, we use frozen lightweight facial extractors, MT-EmotiDDAMFN and MT-EmotiEffNet-B0, with separate heads and systematic post-processing: temporal Gaussian smoothing, per-class expression bias, AffectNet blending, per-AU threshold tuning, and weighted backbone fusion. On the official validation set, our ensemble significantly exceeds the performance of the ConvNeXt baseline. For ambivalence/hesitancy video recognition on the expanded BAH dataset, we extend the audiovisual pipeline to video-level Macro F1 by late fusion of face, HuBERT audio, and RoBERTa text classifiers, temporal aggregation, and a global-text gate. Frame-level Weighted F1 on validation set rises from 0.74 in ABAW-8 to 0.79, while the best public-test video-level Macro F1 reaches 0.73. In both tasks, competitive performance is achieved without fine-tuning heavy backbones. These results indicate that systematic prediction calibration and lightweight multimodal fusion can rival substantially heavier end-to-end approaches while offering improved efficiency and deployment flexibility. 1 Introduction Robust affect analysis in unconstrained environments is essential for human-computer interaction, mental health monitoring, and socially aware AI. The Affective Behavior Analysis in-the-wild (ABAW) workshop series [kollias2020analysing, kollias2021affect, kollias2023abaw2] has become a major benchmark for this problem, with tasks built on Aff-Wild [zafeiriou2017aff, kollias2019deep] and Aff-Wild2 [kollias2019expression, kollias2022abaw]. Recent editions increasingly emphasize multimodal cues, temporal reasoning, and reliable evaluation under missing labels and class imbalance. At the same time, top-ranking solutions often rely on large ensembles, self-supervised vision transformers, and heavy fine-tuning [qiu2024learning, zhang2022multi], which limits their use on mobile or privacy-sensitive platforms [kharchevnikova2018neural, savchenko2022cvprw]. The 11th ABAW competition [kollias2026abaw11] features two challenges. The first is multi-task learning (MTL) [kollias2019face, kollias2021distribution], in which each facial frame must be annotated with valence, arousal, one of eight expressions, and twelve action units. This task continues the s-Aff-Wild2 line studied in ABAW-3 [kollias2022abaw], ABAW-4 [kollias2023abaw] and ABAW-7. The second challenge is Ambivalence/Hesitancy (A/H) recognition on the BAH dataset [kollias2025bah]. A/H was introduced in ABAW-8 [kollias2025abaw8] with frame-level Weighted F1; ABAW-10 [kollias2026abaw10] continued the task on the same BAH split with video-level Macro F1. ABAW-11 returns to A/H on an expanded release of 1,4271,427 videos and keeps video-level Macro F1. Our approach to both challenges follows the same principle: keep strong pre-trained extractors frozen, train only lightweight task-specific classifiers, and invest accuracy gains into post-processing and fusion. For MTL, we reimplement the complete pipeline from ABAW-7 in PyTorch, enabling reproducible training and simplified deployment [savchenko2024leveraging]. We combine MT-EmotiDDAMFN with MT-EmotiEffNet-B0 [savchenko2023hse, zhang2023dual]. For A/H, we combine MT-EmotiEffNet-B0 face descriptors with HuBERT-large audio and RoBERTa-go_emotions text embeddings, apply late fusion of modality-specific MLPs, aggregate frame scores inside each video, and optionally filter predictions with a global transcript classifier inherited from ABAW-8 [savchenko2025abaw8]. This design allows facial, audio, and text features to be computed locally, while only compact predictions are combined downstream. Thus, unlike previous ABAW submissions that primarily improve representation learning through larger pretrained encoders or additional self-supervised pretraining, we instead investigate how far frozen lightweight representations can be pushed through systematic prediction calibration and modality-specific post-processing. Our contributions are fourfold: • We demonstrate that frozen lightweight affect recognition backbones combined with lightweight task-specific heads remain highly competitive for both multi-task facial affect analysis and multimodal ambivalence recognition. • We show that carefully designed validation-time calibration, including temporal smoothing, expression calibration, AU threshold optimization, and backbone fusion, provides larger gains than replacing the backbone with substantially heavier architectures. • We present a unified lightweight framework applicable to two different ABAW challenges, achieving state-of-the-art validation/public-test performance while avoiding end-to-end fine-tuning. • We release the complete PyTorch implementation to facilitate reproducible affective computing research. The rest of the paper is organized as follows. Section 2 reviews related work on MTL and A/H recognition. Section 3 presents our pipelines for both challenges. Section 4 reports validation and public-test results. Section 5 concludes the paper. 2 Related Works 2.1 Multi-Task Learning in ABAW Competitions The MTL challenge requires simultaneous prediction of valence, arousal, eight expressions, and twelve action units on s-Aff-Wild2 [kollias2022abaw]. Because ABAW-3 participants were not required to train on this split [kollias2022abaw], we focus on ABAW-4 [kollias2023abaw] and ABAW-7 [kollias20247th], which share the official combined metric PMTL=PVA+PEXPR+PAUP_MTL=P_VA+P_EXPR+P_AU and are closest to the ABAW-11 protocol. Early s-Aff-Wild2 solutions mainly explored task interaction and calibration without large backbones. The two-aspect information interaction model [sun2022two] models relations between facial sign vehicles and emotional messages. S-MFAR [gera2022facial] combines ResNet [he2016resnet] features with expression-specific thresholds estimated via semi-supervised learning. A hybrid CNN–Transformer [mtl2022fifth] fuses ResNet-18 [he2016resnet] and a spatial transformer and finished fifth on the validation leaderboard. Cross-attentive AU graphs [nguyen2022affective] capture dependencies among action units, while SMMEmotionNet ensembles [mtl2022six] aggregate facial embeddings from multiple extractors. Among lightweight approaches, MT-EmotiEffNet [savchenko2023hse]—an EfficientNet-B0 [tan2019efficientnet] model pre-trained for joint VA/EXPR learning—reached third place without MAE fine-tuning. The top two validation submissions relied on masked-autoencoder (MAE) [he2022masked] facial priors: an EMMA ensemble of MAE ViT [he2022masked, dosovitskiy2021vit] and CNN encoders [li2023affective] took second place, and an MAE+Transformer ensemble with temporal modeling [zhang2022multi, he2022masked] won the challenge. ABAW-7 moved evaluation to the private test split and shifted the field toward heavier representation learning. The winning Netease Fuxi AI Lab pipeline [liu2024affective] uses a progressive strategy: task-specific heads are trained first on a self-supervised MAE [he2022masked] extractor, then jointly refined with cross-task feature fusion and a temporal convergence module that models expression dynamics across frames. HFUT-MAC1 [shen2024facial] ranked third with a multi-architecture design that blends MAE [he2022masked], ResNet [he2016resnet], POSTER, and OpenFace AU descriptors through a Transformer encoder and affine feature alignment. SCU ACers [li2024affective] placed fourth using frozen DINOv2 [oquab2023dinov2] features, task-adaptive query decoding, and an AU-assisted graph convolutional network that transfers AU structure to EXPR and VA prediction. Our ABAW-7 entry [savchenko2024leveraging] took second place with a contrasting philosophy: frozen MT-EmotiEffNet and MT-EmotiDDAMFN extractors, lightweight PyTorch heads, Gaussian temporal smoothing, per-AU threshold tuning, and weighted backbone blending, without fine-tuning large transformers. For ABAW-11, organizers provide a frozen ConvNeXt baseline [liu2022convnext, kollias2026abaw11] as a modern convolutional reference point on the same validation protocol. 2.2 Ambivalence and Hesitancy Recognition Ambivalence/hesitancy (A/H) recognition was introduced in ABAW-8 [kollias2025abaw8] on the BAH dataset [kollias2025bah], which provides frame-level labels, speech transcripts, and cropped faces from behavioural Q&A videos. The first edition evaluated frame-level Weighted F1 on the BAH test split [kollias2025abaw8]. The organizer baseline is a multimodal model with visual, audio, and transcript features fused through temporal convolution and co-attention [richet2024text, kollias2025abaw8]. Top frame-level solutions combined pretrained unimodal encoders with simple fusion. The HSEmotion pipeline [savchenko2025abaw8] extracts facial descriptors from EmotiEffLib [savchenko2023emotieffnets], acoustic embeddings from HuBERT [hsu2021hubert] and wav2vec 2.0 [baevski2020wav2vec], and text embeddings from RoBERTa [liu2019roberta] trained on GoEmotions [demszky2020goemotions]; non-aligned audio and text are interpolated to frame rate, and lightweight MLP classifiers are fused by blending with temporal smoothing. This audiovisual–text design won the challenge. The runner-up, HCAI-VIS (Semantic Matters) [hallmen2025semantic], pursued a complementary semantic route: ViT [dosovitskiy2021vit] visual features with chunk-wise temporal pooling, emotional-speech Wav2Vec 2.0 [baevski2020wav2vec] features, and BERT text representations fused through MLP/LSTM modules, confirming that transcripts are often the strongest single cue for hesitancy. ABAW-10 [kollias2026abaw10] reformulated the task as video-level Macro F1 on an expanded BAH release, requiring a single binary decision per video rather than per-frame labels. The winning VisPBF solution (BROTHER) [pereira2026brother] trains separate expert networks on visual, audio, and contextual cues, ensembles them with diversity constraints, and adds sequence modeling for temporal aggregation. Fennec (ConflictAwareAH) [bekhouche2026conflictaware] extracts VideoMAE [tong2022videomae], HuBERT [hsu2021hubert], and RoBERTa–GoEmotions [liu2019roberta, demszky2020goemotions] embeddings and explicitly models cross-modal conflict via pairwise embedding differences before attention pooling. LEYA [ryumina2026leya] ranked third with a four-stream design—scene VideoMAE [tong2022videomae], face EfficientNet-B0 [tan2019efficientnet], EmotionWav2Vec 2.0 with Mamba temporal encoding, and fine-tuned text transformers—fused through prototype-augmented Transformer modules and ensembled for the final video decision. ABAW-11 continues video-level Macro F1 on an expanded BAH corpus of 1,4271,427 videos [kollias2025bah, kollias2026abaw11]. The official zero-shot Video-LLaVA baseline [lin2024videollava, kollias2026abaw11] remains weak on the public split. 3 Methods We address two tasks of the ABAW-11 challenge. Below we formulate each task and describe the proposed pipeline. 3.1 MTL Challenge The MTL challenge requires recognizing the emotions of each video frame X(t)X(t), t=t1,t2,…,tNt=t_1,t_2,…,t_N, where 1≤t1<t2<…<tN1≤ t_1<t_2<…<t_N are the observed frame indices from s-Aff-Wild2 [kollias2022abaw]. Human affect can be represented both continuously and discretely. In the former case, the most typical emotional space is the two-factor Russell’s circumplex model of affect [russell1980circumplex] with VA-based encoding. Discrete representations include the set of basic expressions of Paul Ekman and the Facial Action Coding System (FACS) [ekman1978facial] with specific facial action units (AUs). For the MTL competition, it is necessary to assign X(t)X(t) to three emotional representations: 1. Valence V(t)∈[−1,1]V(t)∈[-1,1] and arousal A(t)∈[−1,1]A(t)∈[-1,1] (multi-output regression task). 2. Facial expression c(t)∈1,…,CEXPRc(t)∈\1,…,C_EXPR\, where CEXPR=8C_EXPR=8 is the total number of basic emotions: Neutral, Anger, Disgust, Fear, Happiness, Sadness, Surprise, and Other (multi-class classification). 3. AUs (t)=[AU1(t),…,AUCAU(t)]AU(t)=[AU_1(t),…,AU_C_AU(t)], where CAU=12C_AU=12 is the total number of AUs and AUi(t)∈0,1AU_i(t)∈\0,1\ (multi-label classification). The official performance measure is PMTL=PVA+PEXPR+PAUP_MTL=P_VA+P_EXPR+P_AU, where PVA=(CCCV+CCCA)/2P_VA=(C_V+C_A)/2 is the mean Concordance Correlation Coefficient (C) [lawrence1989concordance] of valence and arousal; PEXPRP_EXPR is the macro-averaged F1-score across all eight expression categories; and PAUP_AU is the average F1-score across all twelve AUs. This paper proposes a pipeline for the ABAW-11 MTL challenge (Fig. 1). Compared to our ABAW-7 solution [savchenko2024leveraging], we migrate the training code to PyTorch, keep the backbone weights frozen, and add systematic post-processing comprising: (i) temporal Gaussian smoothing of VA and EXPR predictions; (i) per-class expression logit bias; (i) per-AU threshold tuning; (iv) weighted blending of the two backbones; and (v) AffectNet blending of the fused EXPR predictions. s-Aff-Wild2 frame X(t)X(t) Frozen MT-Emoti- DDAMFN Frozen MT-Emoti- EffNet-B0 VA head Temporal smoothing EXPR head Temporal smoothing, Expression bias AU head Threshold tuning Backbone blending V^,A V, A Backbone blending, AffectNet blend c c Backbone blending AU AU Figure 1: Proposed MTL pipeline: two frozen extractors, three task-specific heads, separate post-processing per task, per-task weighted backbone blending, and AffectNet blending for EXPR after backbone blending. The main part of the pipeline is the feature extractor backbones based on lightweight neural network architectures [savchenko2022cvprw, savchenko2023hse, savchenko2024leveraging]. We use MT-EmotiDDAMFN (Dual-Direction Attention Mixed Feature Network) [zhang2023dual] and MT-EmotiEffNet-B0 (EfficientNet-B0) [tan2019efficientnet]. Both models were firstly pre-trained to recognize faces from the VGGFace2 dataset [cao2018vggface2] and next fine-tuned on AffectNet [mollahosseini2017affectnet] to simultaneously classify static facial expressions and predict VA using a multi-task loss [savchenko2023hse], which is essentially a sum of weighted categorical cross-entropy for facial expressions and C for valence and arousal. During MTL training on s-Aff-Wild2, the backbone weights remain frozen. We hypothesize that AffectNet-pretrained facial representations already capture most low-level affective information required by ABAW. Consequently, optimizing lightweight prediction heads and prediction calibration becomes more beneficial than updating backbone parameters, particularly under relatively limited challenge training data. MT-EmotiDDAMFN produces a 522522-dimensional descriptor (512512-dimensional embedding x plus 1010-dimensional scores s), while MT-EmotiEffNet-B0 produces a 12901290-dimensional descriptor (12801280-dimensional embedding x plus 1010-dimensional scores s). In both cases, s contains eight logits of AffectNet expressions plus valence and arousal predictions at the output of the last layer. Next, we train lightweight PyTorch heads on the official cropped_aligned subset of s-Aff-Wild2. The EXPR head is a linear layer with softmax outputs EXPR∈[0,1]8p_EXPR∈[0,1]^8. The VA head uses only the ten AffectNet scores s (similarly to the slice layer in [savchenko2023hse]) and returns VA∈[−1,1]2p_VA∈[-1,1]^2 with hyperbolic tangent activations. The AU head is a feed-forward network with one hidden layer (128128 units, ReLU) and 1212 logistic sigmoid outputs AU∈[0,1]12p_AU∈[0,1]^12. The inputs of the EXPR and AU heads are the concatenation [,][x,s]. Each head is trained separately with early stopping on its validation metric (macro F1 for EXPR and AU, mean C for VA). As sequential emotions should be smooth, we apply task-specific post-processing inside each video, fuse the two backbones with task-specific weights, and finally apply AffectNet blending to the fused EXPR stream (Fig. 1). The steps are: 1. Temporal smoothing. Dynamic changes of facial expressions and VA may be significantly different, so we use separate parameters σEXPR _EXPR, σVA _VA, δEXPR _EXPR, and δVA _VA. In practice, smoothing decreases the quality of AU detection, so AU predictions are not smoothed. For VA and EXPR, frame-wise predictions of each backbone are smoothed inside each video with a Gaussian kernel: ¯(t)=∑ti∈δ(t)exp(−(ti−t)2σ)(ti)∑ti∈δ(t)exp(−(ti−t)2σ), p(t)= Σ _t_i _δ(t) \! (- (t_i-t)^2σ )p(t_i)Σ _t_i _δ(t) \! (- (t_i-t)^2σ ), (1) where δ(t)N_δ(t) contains frames from the same video within δ frames of t. The parameters σ and δ are selected on the validation set for each backbone and task separately. 2. Expression bias. For each backbone, a per-class logit bias vector ∈ℝ8b ^8 for EXPR is estimated by coordinate search on the validation set. The biased EXPR label of backbone b is defined as c^MTL(b)(t)=argmaxk(log(p¯EXPR,k(b)(t)+ϵ)+bk), c^(b)_MTL(t)= _k ( ( p^(b)_EXPR,k(t)+ε)+b_k ), (2) where ¯EXPR(b)(t) p^(b)_EXPR(t) are the smoothed EXPR probabilities of backbone b and ϵε is a small constant for numerical stability. 3. AU threshold tuning. Each AU is predicted independently by comparing the head score with a class-specific threshold: AUi(t)=1,if pAU,i(t)≥tAU,i,0,otherwise,i=1,…,12.AU_i(t)= cases1,&if p_AU,i(t)≥ t_AU,i,\\ 0,&otherwise, cases i=1,…,12. (3) We consider a fixed threshold tAU,i=0.5t_AU,i=0.5 for all AUs and the best thresholds tAU,i∗t^*_AU,i that maximize the mean F1-score over twelve AUs on the validation set. 4. Backbone blending. Frame-wise predictions of the two backbones are fused with task-specific weights wVAw_VA, wEXPRw_EXPR, and wAUw_AU chosen on the validation set: VA(blend)(t) ^(blend)_VA(t) =wVA⋅¯VA(1)(t)+(1−wVA)⋅¯VA(2)(t), =w_VA· p^(1)_VA(t)+(1-w_VA)· p^(2)_VA(t), (4) EXPR(blend)(t) ^(blend)_EXPR(t) =wEXPR⋅EXPR,bias(1)(t)+(1−wEXPR)⋅EXPR,bias(2)(t), =w_EXPR·p^(1)_EXPR,bias(t)+(1-w_EXPR)·p^(2)_EXPR,bias(t), (5) AU(blend)(t) ^(blend)_AU(t) =wAU⋅AU(1)(t)+(1−wAU)⋅AU(2)(t), =w_AU·p^(1)_AU(t)+(1-w_AU)·p^(2)_AU(t), (6) where EXPR,bias(b)(t)p^(b)_EXPR,bias(t) is the one-hot (or soft) EXPR distribution induced by c^MTL(b)(t) c^(b)_MTL(t), and AU(b)(t)p^(b)_AU(t) are raw AU scores before thresholding. Our best configuration is wVA=1.0w_VA=1.0, wEXPR=0.4w_EXPR=0.4, and wAU=0.5w_AU=0.5, meaning that DDAMFN dominates VA estimation, while EXPR and AU predictions are mixed almost equally. 5. AffectNet blending. After backbone fusion, let pkAffectNet(t)p^AffectNet_k(t) be the softmax probability of the k-th AffectNet expression computed from the frozen scores (t)s(t) of MT-EmotiDDAMFN. The most confident AffectNet label among the first seven basic expressions is k^(t)=argmaxk<7pkAffectNet(t). k(t)= _k<7p^AffectNet_k(t). (7) The fused MTL label is c^MTL(t)=argmaxkpEXPR,k(blend)(t) c_MTL(t)= _kp^(blend)_EXPR,k(t). The final EXPR label is c^EXPR(t)=k^(t),if pk^(t)AffectNet(t)>τblend,c^MTL(t),otherwise, c_EXPR(t)= cases k(t),&if p^AffectNet_ k(t)(t)> _blend,\\ c_MTL(t),&otherwise, cases (8) where τblend=0.88 _blend=0.88 for the final ensemble. For single-backbone ablations, we use τblend=0.85 _blend=0.85 for MT-EmotiDDAMFN and τblend=0.90 _blend=0.90 for MT-EmotiEffNet-B0. The final VA prediction is VA(blend)(t)p^(blend)_VA(t). The EXPR output is c^EXPR(t) c_EXPR(t) from Eq. (8). For AU detection, thresholds tAU,i∗t^*_AU,i are tuned on AU(blend)(t)p^(blend)_AU(t) and applied according to Eq. (3). 3.2 A/H Video Recognition Challenge The A/H challenge is a binary video-level classification problem. For each video V from the BAH dataset [kollias2025bah], the goal is to predict whether ambivalence or hesitancy is present (y=1y=1) or absent (y=0y=0). The expanded ABAW-11 release contains 1,4271,427 videos (10.610.6 hours, 300300 participants) with frame- and video-level expert annotations, speech transcripts, and cropped-aligned faces. The official metric is video-level Macro F1 across both classes; we also report average precision (AP) of the positive class. We extend ABAW-8 audiovisual pipeline [savchenko2025abaw8] from frame-level to video-level prediction. The pipeline extracts frozen multimodal descriptors, trains lightweight frame classifiers, fuses their outputs, aggregates scores inside each video, and optionally applies a global text gate. BAH video V, frame t Frozen MT-Emoti- EffNet-B0 Frozen HuBERT- large Frozen RoBERTa- GoEmotions Interp to frames Interp to frames MLP (pos-weighted) MLP (pos-weighted) MLP (pos-weighted) pface(t)p^face(t) paudio(t)p^audio(t) ptext(t)p^text(t) Late fusion ptp_t Temporal smoothing δ Mean or max agg →s(v)→ s(v) Global text classifier y^(v) y(v) Global text logistic regression Figure 2: Proposed A/H pipeline: frozen face, audio, and text descriptors; frame-level MLPs; late fusion; temporal aggregation; optional global-text gate for video-level Macro F1. 3.2.1 Multimodal feature extraction Visual modality. For each frame, we use MT-EmotiEffNet-B0 [savchenko2023hse], pre-trained on VGGFace2 [cao2018vggface2] and fine-tuned on AffectNet, to extract a 12801280-dimensional facial embedding face(t)x^face(t) and ten expression/VA scores (t)s(t) from the official cropped-aligned faces. Audio modality. The audio track of each video is converted to 1616 kHz mono wav and processed by HuBERT-large [hsu2021hubert], producing 10241024-dimensional hidden states audio(τ)h^audio(τ) along the acoustic timeline. Text modality. The official transcript of each video is encoded with RoBERTa-base trained on GoEmotions [demszky2020goemotions, liu2019roberta], yielding a sequence of 768768-dimensional token/sentence embeddings text(τ)e^text(τ). Since audio and text are not aligned with face frames, we linearly interpolate audioh^audio and texte^text to the visual frame indices of each video, obtaining audio(t)h^audio(t) and text(t)e^text(t). 3.2.2 Frame-level classifiers and late fusion For each modality we train a separate feed-forward network (one hidden layer with 6464 ReLU units and a single logistic output) on frame-level BAH annotations. Training uses binary cross-entropy with logits and a positive-class weight w+=Nneg/Nposw_+=N_neg/N_pos to compensate for class imbalance; all frames labeled as A/H are kept, while negative frames are subsampled with step 11. For the final submission, each modality classifier is fine-tuned on the concatenation of the official training and validation splits. This yields frame-wise probabilities pface(t)p^face(t), paudio(t)p^audio(t), and ptext(t)p^text(t). The multimodal frame score is obtained by late fusion: pt=w1ptface+w2ptaudio+w3pttext,w1+w2+w3=1.p_t=w_1\,p^face_t+w_2\,p^audio_t+w_3\,p^text_t, w_1+w_2+w_3=1. (9) We also study early fusion, where [face(t),audio(t),text(t)][x^face(t),h^audio(t),e^text(t)] is concatenated and fed into one MLP, but the late-fusion ensemble performs better in our experiments. 3.2.3 Video-level decision ABAW-11 requires one binary label per video. Starting from fused frame probabilities, we optionally apply temporal smoothing inside each video with half-width δ: p~t=1|δ(t)|∑ti∈δ(t)pti, p_t= 1|N_δ(t)| _t_i _δ(t)p_t_i, (10) where δ(t)N_δ(t) contains frame indices from the same video within δ frames of t. The video score is then computed by aggregating frame probabilities: s(v)=Agg(p~t:t∈ℱv),s(v)=Agg (\ p_t:t _v\ ), (11) where ℱvF_v is the set of labeled frames in video v, and AggAgg is either the arithmetic mean or the maximum. Following ABAW-8 solution [savchenko2025abaw8], we additionally train a global text classifier at the video level. Pooled transcript embeddings (mean and standard deviation over tokens) are standardized and passed to logistic regression, producing a video-level text probability ptextglobal(v)p^global_text(v). A hard gate filters videos for which the transcript alone does not suggest A/H: s^(v)=0,if ptextglobal(v)<τglobal,s(v),otherwise. s(v)= cases0,&if p^global_text(v)< _global,\\ s(v),&otherwise. cases (12) The final video label is y^(v)=1,if s^(v)≥τ,0,otherwise. y(v)= cases1,&if s(v)≥τ,\\ 0,&otherwise. cases (13) Fusion weights (w1,w2,w3)(w_1,w_2,w_3), smoothing width δ, aggregation rule AggAgg, thresholds τ and τglobal _global, gate mode, and global-text hyperparameters are selected by grid search on the labeled public test split (525525 videos). The global text classifier is trained on the concatenation of training and validation splits (902902 videos). 4 Experiments 4.1 MTL Challenge We evaluate the ABAW-11 MTL pipeline (Sec. 3.1) on the official training and validation splits of s-Aff-Wild2 [kollias2022abaw]. Due to missing labels, 142,333142,333 training frames provide only 103,917103,917 valence/arousal values, 90,64590,645 expression labels, and 103,316103,316 AU annotations. The validation set contains 26,87626,876 faces with complete VA and AU labels, but only 15,44015,440 expression labels. 4.1.1 Comparison with published methods Table 1: Comparison with MTL scores on s-Aff-Wild2 from prior ABAW challenge publications. Method PVAP_VA PEXPRP_EXPR PAUP_AU PMTLP_MTL Organizer baseline [kollias2023abaw] 0.12 0.10 0.12 0.34 HSEmotion [savchenko2024leveraging] 0.56 0.42 0.52 1.49 AIWELL-UOC [cabacas2024enhancing] 0.37 0.28 0.47 1.11 SCU ACers [li2024affective] 0.37 0.30 0.49 1.16 HFUT-MAC1 [shen2024facial] 0.38 0.30 0.50 1.18 HSEmotion [savchenko2024leveraging] 0.41 0.33 0.51 1.25 Netease Fuxi AI Lab [liu2024affective] 0.54 0.43 0.56 1.53 ABAW-11 baseline [liu2022convnext, kollias2026abaw11] – – – 0.45 Ours 0.56 0.46 0.54 1.56 Table 1 summarizes MTL scores reported in prior ABAW publications on s-Aff-Wild2. Ablation of our ABAW-11 pipeline on the labeled validation split is given in Tables 2 and 3. On the s-Aff-Wild2 validation split, our ensemble reaches PMTL=1.56P_MTL=1.56, improving over the ABAW-7 validation ensemble (1.491.49, same split protocol) and over the official ABAW-11 ConvNeXt baseline (0.450.45). 4.1.2 Ablation of the proposed pipeline Table 2 isolates the contribution of each component of the proposed ABAW-11 MTL pipeline. Frame-level MT-EmotiDDAMFN and MT-EmotiEffNet-B0 heads reach PMTL=1.31P_MTL=1.31 and 1.271.27, respectively. Post-processing improves DDAMFN to 1.531.53 and EffNet-B0 to 1.491.49; weighted backbone blending further raises the score to 1.561.56. Table 2: Ablation of the proposed ABAW-11 MTL pipeline on the validation set. Configuration PVAP_VA PEXPRP_EXPR PAUP_AU PMTLP_MTL MT-EmotiDDAMFN + heads 0.48 0.33 0.50 1.31 MT-EmotiEffNet-B0 + heads 0.44 0.34 0.49 1.27 MT-EmotiDDAMFN + post-processing 0.56 0.44 0.52 1.53 MT-EmotiEffNet-B0 + post-processing 0.52 0.45 0.52 1.49 Ensemble (DDAMFN + EffNet) 0.56 0.46 0.54 1.56 Table 3 details the incremental post-processing steps for MT-EmotiDDAMFN on a single backbone. VA smoothing provides the main VA gain; temporal EXPR smoothing and per-class bias improve expression recognition; AffectNet blending adds a further EXPR gain in the single-backbone setting; per-AU threshold tuning adds the final AU boost. In the final ensemble, AffectNet blending is applied after weighted backbone blending (Fig. 1). A grid search over σ and δ (Eq. 1) shows that smoothing helps VA and EXPR (up to +0.08+0.08 and +0.11+0.11 for DDAMFN) but not AU detection. Interestingly, temporal smoothing contributes nearly all improvements in valence-arousal prediction, whereas expression calibration mainly benefits categorical recognition. AU detection benefits almost exclusively from threshold optimization, indicating that calibration requirements differ substantially across affective tasks. Table 3: Incremental post-processing ablation for MT-EmotiDDAMFN. Step PVAP_VA PEXPRP_EXPR PAUP_AU PMTLP_MTL Frame-level heads 0.48 0.33 0.50 1.31 + VA smoothing 0.56 0.33 0.50 1.39 + EXPR smoothing 0.56 0.38 0.50 1.44 + EXPR bias 0.56 0.43 0.50 1.49 + AffectNet blending 0.56 0.44 0.50 1.50 + AU threshold tuning 0.56 0.44 0.52 1.53 The final evaluation protocol applies the validation-selected post-processing without further adaptation and ensemble weights are applied to 51,15951,159 official test frames using heads trained on the training split only. 4.2 A/H Video Recognition Challenge We evaluate the ABAW-11 A/H pipeline (Sec. 3.2) on the official BAH splits [kollias2025bah, kollias2026abaw11]: 778778 training videos, 124124 validation videos, and 525525 public test videos. 4.2.1 Comparison with published methods Table 4: Reported video-level Macro F1 on BAH from prior ABAW challenge publications. Method Macro F1 ABAW-10 baseline [kollias2026abaw10] 0.343 Lenovo PCIE [kollias2026abaw10] 0.675 LEYA [ryumina2026leya] 0.714 Fennec [bekhouche2026conflictaware] 0.715 VisPBF [pereira2026brother] 0.727 ABAW-11 baseline (Video-LLaVA zero-shot) [kollias2026abaw11] 0.283 Ours 0.731 Table 4 lists video-level Macro F1 scores reported in prior BAH challenge publications. After the competition concludes, we will update the table with the final test-set results. Component and fusion ablations of our ABAW-11 pipeline are in Tables 5 and 6. On the ABAW-11 public test set, the proposed pipeline reaches video Macro F1 0.730.73, well above the official Video-LLaVA baseline (0.2830.283). 4.2.2 Ablation of the proposed pipeline Table 5 reports frame-level diagnostics on the ABAW-11 validation split (79,53879,538 labeled frames). Text is the strongest unimodal cue (Weighted F1 0.770.77), and late fusion outperforms early fusion (0.790.79 vs. 0.760.76). Table 5: Frame-level ablation of the proposed A/H pipeline on ABAW-11 validation (τ=0.5τ=0.5 unless noted). Configuration Weighted F1 Macro F1 Face MLP only 0.71 0.52 Audio MLP only 0.67 0.53 Text MLP only 0.77 0.59 Early fusion MLP (τ=0.4τ=0.4) 0.76 0.60 Late fusion (τ=0.55τ=0.55) 0.79 0.59 Table 6 summarizes video-level ablations on ABAW-11 validation and the public test split. On validation, internal baselines (always-negative majority class, RandomForest, early fusion) are outperformed by late fusion; on the public split, max aggregation with the global-text gate yields the best Macro F1 (0.7310.731). The best public configuration uses max aggregation, late fusion weights (0.20,0.45,0.35)(0.20,0.45,0.35), smoothing δ=40δ=40, gate threshold τglobal=0.41 _global=0.41, and τ=0.35τ=0.35. A transcript-only global classifier reaches a comparable video Macro F1 (0.7340.734) with higher AP (0.870.87 vs. 0.820.82), confirming that spoken language is a very strong cue; the hard gate restricts audiovisual fusion to videos whose transcript already suggests A/H. Table 6: Video-level ablation of the proposed A/H pipeline on ABAW-11 validation (124124 videos) and the public test split (525525 videos). The validation majority baseline always predicts no A/H. Configuration Split Macro F1 AP Majority class val 0.28 0.60 RandomForest on concatenated features val 0.67 0.82 Early fusion MLP val 0.68 0.85 Late fusion, mean aggregation val 0.72 0.85 Late fusion, max aggregation val 0.71 0.83 Late fusion, mean public 0.69 0.79 Late fusion, mean + hard global-text gate public 0.71 0.80 Late fusion, max public 0.69 0.82 Late fusion, max + hard global-text gate public 0.73 0.82 Global text classifier only public 0.73 0.87 Frame MLPs are fine-tuned on all 902902 training and validation videos, the global text classifier is trained on the same 902902 videos, and fusion and gate hyperparameters are selected by grid search on the public test split. The primary submission applies the max+gate configuration to the private test set (152152 videos). 4.3 Discussion Our experiments reveal three consistent observations. First, temporal calibration contributes substantially more than replacing lightweight frozen backbones with larger architectures, suggesting that temporal consistency rather than feature quality is the dominant error source on s-Aff-Wild2. Second, transcript information remains the strongest modality for ambivalence recognition, while audiovisual cues mainly improve borderline examples. Finally, both challenges indicate that competitive affect recognition can be achieved without end-to-end fine-tuning, reducing computational cost while simplifying deployment. 5 Conclusion This paper advocates a compute-on-device, fuse-locally design for in-the-wild affect analysis: strong facial, audio, and text encoders remain frozen, while accuracy is recovered through lightweight heads, validation-time post-processing, and modality-specific fusion rather than end-to-end fine-tuning of large backbones. For privacy-sensitive settings, raw video and transcripts need not leave the device; only compact frame-level scores or pooled embeddings are combined downstream (Figs. 1 and 2). On s-Aff-Wild2 validation, post-processing and dual-backbone fusion raised PMTLP_MTL from 1.311.31 to 1.561.56 (Tables 2), outperforming both the official ConvNeXt baseline (0.450.45) and our ABAW-7 validation ensemble (1.491.49). The gain comes mainly from temporal smoothing, expression calibration, and per-AU thresholding applied after frozen feature extraction, suggesting that much of the remaining MTL error on wild faces reflects temporal inconsistency and calibration rather than backbone capacity alone. For A/H, the same frozen-extractor principle extends to video-level decisions: late fusion with a transcript gate reached Macro F1 0.7310.731 on the public development split (Table 6), while a transcript-only global classifier reached 0.7340.734 with higher AP. Hesitancy is therefore largely lexical, but audiovisual fusion remains useful when the transcript is ambiguous. This supports a two-stage deployment pattern: a fast text screen at the video level, with multimodal fusion reserved for borderline cases. The source code of training scripts, evaluation code, hyperparameter configurations, and inference pipelines for both challenges is publicly available at111https://github.com/bakinalexey/abaw-11-mtl-bah-recognition. Overall, our results suggest that careful prediction calibration and multimodal fusion can compensate for much of the performance typically attributed to increasingly large backbone models. This observation encourages future research on efficient affective computing systems that prioritize robustness, interpretability, and deployment efficiency over backbone scale. The proposed framework is particularly suitable for privacy-sensitive applications such as mobile affect recognition, digital health, and human-computer interaction, where transmitting raw audiovisual data is undesirable. References