Paper deep dive
From Surface to Depth: Towards Cognitive Appraisal Reasoning in Multimodal Emotion Understanding
Jia Li, Yichao He, Yangchen Yu, Qiankun Li, Xinyi Li, Baiyi Ye, Zhenzhen Hu, Richang Hong, Erik Cambria
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/10/2026, 2:41:25 AM
Summary
This paper introduces a novel paradigm for multimodal emotion understanding that shifts from surface-level cue-label associations to cognitive appraisal reasoning. The authors present CogEmo-40K, a large-scale instruction-tuning dataset constructed via a perception-to-appraisal pipeline; CogEmo-MoE, a compact sparse multimodal large language model (MLLM) using interleaved Mixture-of-Experts (MoE) blocks for appraisal-specific adaptation; and CogEmo-Bench, a benchmark featuring the Appraisal Evidence Quality Score (AEQS) to evaluate cognitive-affective understanding across six appraisal dimensions. Experiments demonstrate that CogEmo-MoE outperforms existing models in both appraisal reasoning and emotion recognition, showing strong cross-domain generalization.
Entities (17)
Relation Signals (15)
CogEmo-Bench → usesmetric → Appraisal Evidence Quality Score
confidence 97% · CogEmo-Bench introduces an Appraisal Evidence Quality Score (AEQS) to assess cognitive-affective understanding
CogEmo-MoE → trainedon → CogEmo-40K
confidence 96% · Finally, we use CogEmo-40K as cognition-level supervision to specialize the sparse experts
CogEmo-MoE → uses → CLIP
confidence 95% · It adopts Qwen-1.8B as the language backbone and CLIP/CLAP as the visual/audio encoders.
CogEmo-MoE → uses → CLAP
confidence 95% · It adopts Qwen-1.8B as the language backbone and CLIP/CLAP as the visual/audio encoders.
CogEmo-MoE → uses → Qwen-1.8B
confidence 95% · It adopts Qwen-1.8B as the language backbone
CogEmo-MoE → usesarchitecture → Mixture-of-Experts
confidence 95% · CogEmo-MoE is a compact sparse MLLM that introduces interleaved MoE blocks
CogEmo-40K → derivedfrom → MER2025
confidence 94% · we annotate the same 115,595 MER2025 (Lian et al., 2025b) source samples
CogEmo-MoE → outperforms → GPT-5.2
confidence 92% · CogEmo-MoE: 70.60 vs. 56.30 AEQS for GPT-5.2
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent multimodal large language models (MLLMs) increasingly incorporate explainable reasoning for emotion understanding. However, reasoning based mainly on observable affective cues can reduce emotion understanding to superficial cue-label associations, giving rise to the Clever Hans effect. Such shortcuts become unreliable when affective cues are implicit, conflicting across modalities, linguistically misleading, or obscured by redundant details. In contrast, human emotions are shaped by how individuals interpret and evaluate surrounding events beyond observable cues. Inspired by appraisal theories of emotion, we formulate multimodal emotion understanding as a progression from perception to cognitive appraisal, and introduce a dataset, a model, and a benchmark to support this novel paradigm. CogEmo-40K is a large-scale instruction-tuning dataset constructed through a perception-to-appraisal pipeline to elicit evidence-grounded reasoning across six cognitive appraisal dimensions underlying emotion. CogEmo-MoE is a compact sparse MLLM that introduces interleaved MoE blocks for appraisal-specific adaptation, enabling effective appraisal reasoning at a substantially smaller scale than typical emotion MLLMs. CogEmo-Bench introduces an Appraisal Evidence Quality Score (AEQS) to assess cognitive-affective understanding across six complementary appraisal dimensions, addressing the limitation of conventional emotion metrics that evaluate what emotion is predicted but not why it arises. Extensive experiments show that our paradigm not only leads CogEmo-Bench, but also exhibits strong cross-domain generalization. Our findings suggest that perception-to-appraisal reasoning can move beyond surface-level cue-label associations toward more reliable multimodal emotion understanding and closer cognitive alignment between MLLMs and humans.
Tags
Links
- Source: https://arxiv.org/abs/2610.11918v1
- Canonical: https://arxiv.org/abs/2610.11918v1
Trouble viewing inline? Open PDF directly →
Full Text
110,780 characters extracted from source content.
Expand or collapse full text
FROM SURFACE TO DEPTH: TOWARDS COGNITIVE APPRAISAL REASONING IN MULTIMODAL EMOTION UNDERSTANDING Jia Li 1 , Yichao He 1 , Yangchen Yu 1 , Qiankun Li 2 , Xinyi Li 1 , Baiyi Ye 1 Zhenzhen Hu 1 , Richang Hong 1 , Erik Cambria 2 1 Hefei University of Technology 2 Nanyang Technological University jiali@hfut.edu.cn, cambria@ntu.edu.sg ABSTRACT Recent multimodal large language models (MLLMs) increasingly incorporate ex- plainable reasoning for emotion understanding. However, reasoning based mainly on observable affective cues can reduce emotion understanding to superficial cue-label associations, giving rise to the Clever Hans effect 1 . Such shortcuts become unreliable when affective cues are implicit, conflicting across modali- ties, linguistically misleading, or obscured by redundant details. In contrast, hu- man emotions are shaped by how individuals interpret and evaluate surround- ing events beyond observable cues. Inspired by appraisal theories of emotion, we formulate multimodal emotion understanding as a progression from percep- tion to cognitive appraisal, and introduce a dataset, a model, and a benchmark to support this novel paradigm.❶ CogEmo-40K is a large-scale instruction- tuning dataset constructed through a perception-to-appraisal pipeline to elicit evidence-grounded reasoning across six cognitive appraisal dimensions under- lying emotion.❷ CogEmo-MoE is a compact sparse MLLM that introduces interleaved MoE blocks for appraisal-specific adaptation, enabling effective ap- praisal reasoning at a substantially smaller scale than typical emotion MLLMs. ❸ CogEmo-Bench introduces an Appraisal Evidence Quality Score (AEQS) to assess cognitive-affective understanding across six complementary appraisal di- mensions, addressing the limitation of conventional emotion metrics that evaluate what emotion is predicted but not why it arises. Extensive experiments show that our paradigm not only leads CogEmo-Bench (CogEmo-MoE: 70.60 vs. 56.30 AEQS for GPT-5.2; 71.43% vs. 63.10% accuracy and 57.17% vs. 52.19% M-F1 for Qwen3-Omni), but also exhibits strong cross-domain generalization. Our find- ings suggest that the perception-to-appraisal reasoning can move beyond surface- level cue-label associations toward more reliable multimodal emotion understand- ing and closer cognitive alignment between MLLMs and humans. Project Page: https://github.com/MSA-LMC/CogEmo 1INTRODUCTION Emotion understanding is not simply a matter of recognizing observable affective expressions. A prominent perspective in emotion research holds that emotional responses depend not only on what happens, but also on how individuals interpret and evaluate what happens to them (Ellsworth & Scherer, 2003; Moors et al., 2013). The same event may lead to different emotional responses depending on whether it supports or obstructs personal goals, whether the situation is perceived as manageable, who is considered responsible, and how future consequences are anticipated (Roseman, 1991; Yeo & Ong, 2024; Liang et al., 2026). From this perspective, observable behavior provides only part of the information needed for emotion understanding: perception reveals what is happen- ing, whereas cognitive appraisal concerns what the situation means to the individual (Ong et al., 2015; Skerry & Saxe, 2015; Teotia et al., 2026). 1 Clever Hans Effect: A system exhibits seemingly intelligent behavior but actually relies on hidden cues, data biases, or superficial associations rather than genuinely understanding the mechanisms underlying the task. 1 arXiv:2610.11918v1 [cs.M] 8 Oct 2026 2020 66 22 22 5555 44 1515 3535 2020 4444 4848 4242 4040 7171 4444 4949 5757 4242 Goal Congruence Coping Potential Accountability Futur e Expectancy Arousal Action Tendency AEQS Accuracy Macr o-F1 59.7759.77 77.7677.76 67.7567.75 65.3365.33 81.2881.28 71.7271.72 70.6070.60 71.4371.43 57.1757.17 Emotion-LLaMAAffectGPTQwen3-Omni GPT-5.6-LunaGemini-3.6-FlashOurs (a) Comparison of Representative MLLMs on CogEmo-Bench 3535 3535 3030 4545 5050 5050 5050 5050 3030 5959 5757 4646 5353 6868 6262 7070 6868 4848 MER2023 MER2024 MELD IEMOCAP CMU-MOSI CMU-MOSEI CH-SIMS CH-SIMS v2 OV -MERD+ 74.0174.01 72.6672.66 57.3557.35 57.5357.53 79.1579.15 69.7569.75 83.0183.01 77.9477.94 58.1658.16 LLaVAEmoVITMAFW MERR-FineMER-CaptionOurs (b) Cross-Domain Generalization Results on MERBench Figure 1: Performance comparison of representative MLLMs on cognitive appraisal reasoning and cross-domain emotion understanding. In recent years, multimodal large language models (MLLMs) have advanced emotion recognition toward more flexible and interpretable multimodal emotion understanding by generating natural- language explanations alongside emotion predictions (Lian et al., 2025a; Cheng et al., 2024b; Zhao et al., 2025; Zhang et al., 2026b). However, plausible explanations do not necessarily imply gen- uine reasoning about why an emotion arises (Luo et al., 2026). Models often still rely on statistical associations between observable facial, acoustic, or linguistic cues and emotion categories (Jiang et al., 2025; Yeo & Jaidka, 2025; Chen et al., 2026), while accurate perception alone does not en- sure that perceptual evidence is explicitly organized into cognition-level appraisal reasoning (Ong et al., 2015; Houlihan et al., 2023; Yeo & Jaidka, 2025). Such cognitive misalignment becomes par- ticularly problematic when multimodal evidence is ambiguous, conflicting, or context-dependent, and may contribute to inappropriate empathic responses (Wang et al., 2022) or unreliable emotion- informed decisions in mental-health settings (Zhang et al., 2023). Yet existing multimodal emotion datasets largely provide emotion labels or perception-level descriptions (Lian et al., 2026; Cheng et al., 2024b; Lian et al., 2025a), while conventional metrics mainly evaluate final predictions, leav- ing cognition-level reasoning insufficiently supervised and assessed. From a psychological perspective, we model cognitive appraisal as a key intermediate process in emotion understanding, rather than treating emotion as a direct cue-to-label mapping. Specifically, we characterize appraisal reasoning through six complementary dimensions: goal congruence, cop- ing potential, accountability, future expectancy, arousal, and action tendency (Roseman, 1991; Yeo & Ong, 2024), providing an interpretable structure for reasoning about why an emotion arises. How- ever, existing emotion-oriented MLLMs provide limited support for cognition-specific specializa- tion, and scaling model capacity alone is inefficient for targeted cognitive adaptation. The main contributions of this paper are summarized as follows: 1 ⃝ We introduce a perception-to-appraisal paradigm for multimodal emotion understanding, shifting from direct cue-to-label prediction toward reasoning about how perceptual evidence is cognitively interpreted before emotion arises. To support this paradigm, we construct CogEmo- 40K, a large-scale human-aligned instruction-tuning dataset for cognition-grounded emotion reasoning. 2 ⃝ We propose CogEmo-MoE, a lightweight sparse MLLM for cognitive appraisal reasoning with perception-preserving adaptation. Instead of scaling up or broadly fine-tuning the backbone, we preserve established multimodal perception and specialize only sparse experts for appraisal- specific adaptation, enabling strong appraisal reasoning within a compact model. 3 ⃝ We develop CogEmo-Bench, a fine-grained benchmark with an Appraisal Evidence Quality Score (AEQS) for evaluating cognitive appraisal reasoning and emotion recognition. Across diverse open- and closed-source MLLMs, CogEmo-MoE improves AEQS over GPT-5.2 from 56.30→70.60 and accuracy over Qwen3-Omni from 63.10%→71.43%, revealing both the strengths and limitations of current models in cognitive emotion understanding. 2 MER2025 Raw Data Description-Level Perception (Stage1) Appraisal Reasoning (Stage 2) Human Verification & Alignment Perception -Level Description (Stage1) Scene Facial Audio Text CogEmo-40K MER2025 Raw Data ... Appraisal Reasoning (Stage2) Human Verification & Alignment Question: Please infer the most likely emotion category from the multimodal input. 115595 samples without labels Audio, Video, Text Generation constraints: 1.No emotional words 2.No psychological inference 3.Only observable descriptions Omni-LLM Description-Level Dataset GPT-5.4 Final Basic Emotions： Cognitive Appraisal-Guided Emotion Reasoning Candidate labels: happiness, sadness, anger, fear, anxiety, disgust, contempt, and neutral. Answer: Based on the multimodal input...<cognitivereasoning>... Emotion is... Cognitive Appraisal Reasoning Emotion Inference from Cognitive Appraisals Congruence Coping Accountability Future Arousal Action tendency Annotators Step2:Emotion Label Verification Step1:Human Cognitive Appraisal Alignment Raw Data Action tendency Arousal Future Accountability Coping Congruence Affective Computing Experts ≥ 4 Dimensions Consistent (Pass) ＜4 Dimensions Consistent (Fail) Fail Pass Samples for Correction CogEmo-40K Expert Correction Cognitive Appraisal Dimensions · Environment · Verbatim transcription · People Interactions · Visible actions · Visible events · Gaze · Eyebrows · Mouth · Jaw · Muscle Activity · Head Pose · Lip Corners · Speech Presence · Speech Rate · Pitch Variation · Articulation Clarity · Prosody Dynamics ... ...... CogEmo-40K CogEmo-Bench-Eval Figure 2: Illustration of CogEmo-40K Dataset Construction. 2COGEMO-40K: COGNITION-GROUNDED MULTIMODAL EMOTION REASONING DATASET 2.1DATASET OVERVIEW Existing multimodal emotion datasets mainly provide emotion labels or perception-level descrip- tions, offering limited supervision for the cognitive appraisal processes underlying emotional re- sponses. To address this gap, we construct CogEmo-40K, a cognition-grounded multimodal emo- tion reasoning dataset. Following the screening protocol of AffectGPT (Lian et al., 2025a), we an- notate the same 115,595 MER2025 (Lian et al., 2025b) source samples used by MERCaption (Lian et al., 2025a) through a two-stage perception-to-appraisal pipeline, followed by human-guided align- ment and refinement to obtain approximately 40K high-quality instances. A held-out subset forms CogEmo-Bench-Eval, providing over 6,000 human-verified dimension-level appraisal references across six appraisal dimensions. The overall construction pipeline is illustrated in Figure 2. Each instance explicitly follows a multimodal observation → cognitive appraisal → emotion state rea- soning process, providing supervision not only for what emotion is expressed, but also for how emotional responses emerge from interpreted situations. Detailed dataset statistics are provided in Appendix B.4. 2.2TWO-STAGE PERCEPTION-TO-APPRAISAL DATA GENERATION As illustrated in Figure 2, we decompose annotation generation into perception and appraisal to reduce the entanglement between perceptual description and cognitive interpretation, while discour- aging shortcut cue-label associations. Stage 1: Perception-Level Description. The first stage converts multimodal inputs into emotion- agnostic descriptions of observable evidence across four sources: Scene, Face, Speaker Audio, and Transcript. Emotion labels and latent psychological interpretations are explicitly excluded to prevent appraisal leakage, yielding an evidence-grounded representation for subsequent cognitive reasoning. Stage 2: Cognition-Level Appraisal Reasoning. The second stage interprets the perceptual evi- dence through six appraisal dimensions motivated by cognitive appraisal theory: Goal Congruence, Coping, Accountability, Future Expectancy, Arousal, and Action Tendency. Rather than directly in- ferring an emotion label from observable cues, the model is required to explain how the perceived 3 (b)FromSurfacetoDepth:CognitiveAppraisalReasoning&CogEmo-MoE(Ours) (a)PreviousParadigm:Perception-LevelAffectiveCueReasoning&ExistingAudio-VisualEmotionLLMs Step 1: Identify affective cues Step 1: Identify affective cues Step 2: Cognitive Appraisal Reasoning ... Text Cues 1. furrowed brow 2.Extreme wording 3. ... Audio Cues 1. inhale sound 2.controlled voice 3. ... Facial Cues 1. furrowed brow 2.raised upper lip 3. ... Aversive evaluation Strong goal conflict Blame on another person Steady posture Controlled speech Accusatory “he” Confident control Facial tension Charged wording Strong activation ... ... Text Cues 1. furrowed brow 2.Extreme wording 3. ... Audio Cues 1. inhale sound 2.controlled voice 3. ... Facial Cues 1. furrowed brow 2.raised upper lip 3. ... Furrowed brow + Inhale Anxiety / Frustration Extreme wording Fear/Anxiety/Sadness Neutral face Uncertain inference ...... Surface Cue–Based Implicit Emotion Mapping Cognitive Appraisal Reasoning with Deeper Understanding Standard LoRAFine-tuning Audio / Video Encoders Projectors LoRA LoRAattached to the LLM backbone CogEmo-MoE （ Ours ） Audio / Video Encoders Projectors SparseLLM FFN Copy weight MoE Copy FFN weights to initialize experts Dense LLM Figure 3: Comparison of surface cue-based emotion reasoning and our cognitive appraisal-based CogEmo-MoE framework. situation is evaluated along these dimensions before deriving the emotional response. Each appraisal is grounded in supporting evidence from perception-level description, yielding an explicit reasoning trajectory from multimodal observation to cognitive interpretation and finally to emotion state. 2.3HUMAN-GUIDED COGNITIVE ALIGNMENT Reliable cognition-grounded supervision requires consistency across the full perception–appraisal– emotion chain. We therefore conduct human-guided alignment at perceptual, cognitive, and emo- tional levels. Perception-Level Quality Assessment. We randomly sample 1,000 generated descriptions and ask five researchers with backgrounds in affective computing to evaluate their Accuracy, Completeness, and Objectivity on a five-point scale, following Appendix B.3. The average score for each criterion exceeds 4.0, indicating that the generated descriptions provide reliable and largely emotion-agnostic perceptual evidence. Cognitive Appraisal and Emotion Alignment. We recruit 300 trained volunteers to assess whether the generated cognitive appraisal reasoning is supported by the original multimodal evidence and aligned with plausible human interpretations. For each sample, volunteers compare the generated appraisals with human interpretations across the six cognitive appraisal dimensions, while also veri- fying whether the final emotion label is consistent with both the multimodal context and the inferred cognitive appraisal reasoning. Rather than directly discarding inconsistent, uncertain, or ambigu- ous cases, we adopt a human-in-the-loop refinement process to revise the cognitive reasoning and emotion labels based on human feedback, followed by further expert review. This process preserves challenging samples while improving the consistency between cognitive appraisals and emotion la- bels. 3COGEMO-MOE: MODEL DESIGN 3.1ARCHITECTURE OVERVIEW Existing multimodal emotion models often rely on large backbones for affective understanding. We instead explore whether compact models can acquire effective cognitive appraisal through targeted specialization, and introduce CogEmo-MoE, a lightweight sparse model built upon MoE-LLaVA. It adopts Qwen-1.8B as the language backbone and CLIP/CLAP as the visual/audio encoders. As shown in Fig. 3, sparse experts initialized from the corresponding FFN weights are specialized for cognitive appraisal reasoning. Given video V , audio A, and instruction I , multimodal features are projected into the language embedding space and combined with the instruction embeddings: Y = LM MoE ([P v (f CLIP (V ));P a (f CLAP (A));E(I)]),(1) where P v (·) and P a (·) are modality-specific projectors, E(I) is the instruction embedding, and LM MoE denotes the MoE-augmented language backbone for cognition-specific reasoning. 4 3.2SPARSE COGNITIVE EXPERT ADAPTATION Cognitive appraisal reasoning builds on earlier multimodal perception. We therefore interleave sparse MoE modules across the Transformer backbone. Each MoE layer contains multiple experts and a lightweight router, which computes the selection probability for expert e given hidden state h: p(e|h) = Softmax(W r h),(2) where W r denotes the routing parameters. Only the top-k experts are activated for each input: MoE(h) = X i∈TopK(p) p i E i (h),(3) where E i (·) denotes the i-th expert and p i is its routing weight. 3.3PROGRESSIVE PERCEPTION-TO-COGNITION TRAINING CogEmo-MoE follows a progressive three-stage training strategy. All stages are optimized with the same autoregressive objective, while differing in supervision granularity and trainable modules. Given the stage-specific datasetD s , the training objective is L s =− X (X,I,Y )∈D s X |Y| t=1 logp θ s (y t | y <t ,X,I), s∈1, 2, 3,(4) where X , I , and Y denote the multimodal input, instruction, and stage-specific target, respectively, and θ s are the trainable parameters. The training data used are summarized in Table 1. MERR- Coarse is a coarse-grained face–audio emotion dataset from Emotion-LLaMA (Cheng et al., 2024a), whereas our CogEmo-Desc comprises fine-grained perception-level descriptions generated during the automatic annotation process. Stage 1: Multimodal Alignment. Using MERR-Coarse, we optimize only the modality-specific projectors, while keeping the pretrained encoders and language backbone frozen, to align visual and audio representations with the language embedding space. Stage 2: Perception Enhancement. Starting from the aligned model, we introduce CogEmo- Desc as fine-grained description-level supervision to strengthen the model’s perception of facial and acoustic details. During this stage, both the language backbone and modality-specific projectors are unfrozen and jointly optimized. Stage 3: Cognitive Reasoning Adaptation. Finally, we use CogEmo-40K as cognition- level supervision to specialize the sparse experts for appraisal-based emotion reasoning on top of the multimodal perception acquired in the pre- vious stages. During this stage, the language backbone is frozen and only the sparse expert modules are optimized, concentrating adapta- tion on cognition-specific reasoning. Table 1: Training data used in the three stages of CogEmo-MoE. UsageSource#Samples Stage IMERR-Coarse28,618 Stage IICogEmo-Desc114,587 Stage IIICogEmo-40K38,532 4COGEMO-BENCH: COGNITION-GROUNDED EMOTION UNDERSTANDING BENCHMARK 4.1COGEMO-BENCH OVERVIEW The introduced CogEmo-Bench jointly evaluates cognitive appraisal reasoning and conventional emotion recognition. An Appraisal Extractor normalizes free-form MLLM responses into the six appraisal dimensions defined in Sec. 2.2, without judging correctness or adding evidence. The extracted appraisals are compared with human-verified references using the Appraisal Evidence Quality Score (AEQS), while final emotion predictions are evaluated with standard classification metrics. 5 4.2EVALUATION METRICS Conventional emotion metrics evaluate final predictions but do not reveal whether the underlying cognitive evidence is well aligned with human interpretation. We therefore introduce a new metric to evaluate appraisal evidence across the six cognitive dimensions. Appraisal Evidence Quality Score (AEQS). For sample i and appraisal dimension d, let ˆe i,d and e i,d denote the extracted model evidence and human-verified reference, respectively. Using a fixed embedding encoder φ(·), the evidence similarity is defined as s i,d = cos(φ(ˆe i,d ),φ(e i,d )).(5) If only one of ˆe i,d and e i,d is empty, we assign s i,d = 0; if both are empty, the sample is ex- cluded from the corresponding dimension. LetS d denote the valid sample set for dimension d. The dimension-level score is S d = 1 |S d | X i∈S d s i,d .(6) AEQS is then computed by averaging across the six appraisal dimensions: AEQS = α D X D d=1 S d ,(7) where D denotes the number of appraisal dimensions and α is a scaling factor; we set D = 6 and α = 100 in this work. Higher AEQS indicates stronger semantic alignment between model- generated appraisal evidence and human-verified cognitive references. Emotion Recognition. We additionally report standard Accuracy and Macro-F1 for final emotion prediction, with detailed computation provided in Appendix D.3. 5RESULTS AND OBSERVATIONS Main results. Detailed implementation information can be found in Appendix C. Table 2 com- pares representative MLLMs on both cognitive appraisal reasoning and emotion recognition. The results show substantial differences across models in AEQS and final emotion performance. Some models achieve competitive emotion classification despite relatively weak appraisal reasoning, while larger general-purpose and proprietary models generally perform better on both aspects. CogEmo- MoE achieves a favorable balance between appraisal reasoning and emotion recognition despite its compact model scale. Specifically, it improves AEQS over GPT-5.2 from 56.30 → 70.60, while outperforming Qwen3-Omni in accuracy from 63.10% → 71.43% and Macro-F1 from 52.19%→57.17%. Additional model-size comparisons are provided in Appendix C.2. Obs. 1: Emotion recognition does not necessarily imply cognitive understanding. VideoChat- Flash (Li et al., 2025b) achieves strong emotion prediction despite limited explicit appraisal rea- soning, while emotion-specialized models such as AffectGPT (Lian et al., 2025a) and Emotion- LLaMA (Cheng et al., 2024a) also show limited transfer to cognition-level understanding. These results indicate that emotion recognition and surface cue-to-label mapping cannot substitute for cognitive appraisal reasoning about why emotions arise. By contrast, cognition-oriented supervi- sion enables our CogEmo-MoE to improve both appraisal reasoning and emotion discrimination within a compact model. Obs. 2: True emotion intelligence requires adaptive reasoning across different emotion spaces. Strong performance within a fixed label space does not necessarily indicate robust emotion under- standing. A capable emotion model should adapt its judgments to task-specific emotion taxonomies rather than rely on surface-level cue-to-label mappings. Although open-vocabulary approaches such as AffectGPT (Lian et al., 2025a) improve output flexibility, evaluation protocols that count a predic- tion as correct whenever the target emotion appears among multiple generated candidates may over- estimate performance without establishing that the model selected the most appropriate emotion. In contrast, as shown in Fig. 1(b) and detailed in Appendix E.2, our results show that cognition-oriented reasoning transfers across different emotion spaces and consistently improves emotion recognition tasks, suggesting that cognitive appraisal provides a more generalizable basis for emotion under- standing. 6 Table 2: Comparison of different MLLMs on CogEmo-Bench. A, V, and T denote audio, visual, and textual modalities, respectively. Cong., Cop., Account., Fut., Aro., and Act. denote Goal con- gruence, Coping potential, Accountability, Future expectancy, Arousal, and Action tendency. ModelAVTCong.Cop.Account.Fut.Aro.Act.AEQSAcc.M-F1 Open-source MLLMs OneLLM (Han et al., 2024)✓ ×✓31.478.0210.109.5930.906.8116.1530.16 26.03 SECap (Xu et al., 2024)✓ ×✓ 11.127.424.593.9742.264.2312.2630.85 21.25 Qwen2-Audio (Chu et al., 2024)✓ ×✓39.2718.1622.2624.85 30.55 15.6025.1243.55 32.68 SALMONN (Tang et al., 2024)✓ ×✓ 42.8718.2020.1824.10 27.17 10.6723.8743.69 33.43 Otter (Li et al., 2025a)×✓26.093.838.903.8915.653.5810.3236.61 24.69 LLaMA-VID (Li et al., 2024)×✓47.7316.3520.3321.62 39.77 13.7926.6036.71 31.84 VideoChat-Flash (Li et al., 2025b) ×✓6.821.221.261.754.320.782.6955.36 39.11 Chat-UniVi (Jin et al., 2024)×✓49.9622.8026.6627.92 46.63 19.7332.2834.03 29.86 mPLUG-Owl3 (Ye et al., 2025)×✓ 47.7818.5323.1127.49 58.21 17.3032.0745.54 34.72 Qwen3.8-Max (Team, 2026)×✓51.3044.3747.1242.56 67.71 43.9449.5057.04 46.29 Kimi-K3 (Kimi Team, 2026)×✓52.0243.5845.2340.54 68.38 42.4248.7046.13 39.04 PandaGPT (Su et al., 2023)✓50.9714.6519.9221.22 32.92 13.5525.5425.79 18.78 Emotion-LLaMA (Cheng et al., 2024a)✓27.7410.423.716.2464.499.0420.2743.75 25.90 AffectGPT (Lian et al., 2025a)✓44.4624.2625.0523.58 59.20 20.3932.8342.76 31.90 Qwen3-Omni (Xu et al., 2025)✓ 49.5039.2143.7437.79 69.69 36.7646.1263.10 52.19 Closed-source MLLMs GPT-5.2 (OpenAI, 2025)×✓55.8956.7853.3648.37 70.34 53.0756.3046.33 36.92 GPT-5.6-Luna (OpenAI, 2026b) ×✓53.5746.4841.2339.18 70.41 38.7648.2751.59 43.46 Grok-4.3 (xAI, 2026)×✓51.6736.8040.6737.82 69.46 36.8845.5546.83 39.79 Gemini-3.6-Flash (Google, 2026)✓40.7926.4230.4424.36 72.33 23.4536.3059.13 45.99 Our CogEmo Models CogEmo-MoE✓ ×✓58.6877.0467.4465.04 81.03 71.8170.1763.59 50.18 CogEmo-MoE×✓59.5677.5666.8864.59 80.32 71.2670.0362.70 50.07 CogEmo-MoE✓59.7777.7667.7565.33 81.28 71.7270.6071.43 57.17 Cong. Cop. Acc. Fut. Aro. Act. Anger conflict 28% control 51% external 31% present 21% elevated 32% confront 59% Cong. Cop. Acc. Fut. Aro. Act. Anxiety conflict 13% uncertain 33% external 45% concern 19% elevated 34% avoid 53% Cong. Cop. Acc. Fut. Aro. Act. Contempt conflict 31% high 53% external 33% present 23% restrained 58% distance 48% Cong. Cop. Acc. Fut. Aro. Act. Disgust aversion 31% manageable 34% external 53% present 25% moderate 39% avoid 56% Cong. Cop. Acc. Fut. Aro. Act. Fear threat 9% low 24% external 61% concern 25% high 31% escape 40% Cong. Cop. Acc. Fut. Aro. Act. Happiness alignment 47% confident 50% cause 27% present 28% mobility 57% engage 53% Cong. Cop. Acc. Fut. Aro. Act. Neutral alignment 19% adequate 58% source 20% present 21% low 69% maintain 30% Cong. Cop. Acc. Fut. Aro. Act. Sadness loss 26% limited 20% situational 31% present 21% restrained 37% withdraw 61% Figure 4: Emotion-Specific Cognitive Language Profiles in CogEmo-40K. Radar plots show six- dimensional appraisal patterns, with radial extent and label size indicating phrase prevalence. Cognitive Language Profiles. We further analyze the structured outputs of the fixed Appraisal Extractor by aggregating recurring phrase-level patterns for each emotion and appraisal dimension. Figure 4 visualizes the resulting emotion-specific profiles across six cognitive dimensions, where radial extent and label size indicate the prevalence of the most frequent phrase within each emotion category. The surrounding terms summarize the corresponding appraisal semantics. These percent- ages reflect phrase prevalence rather than appraisal intensity. The original top-3 phrases and their occurrence rates are reported in Appendix B.4.3. 7 Obs. 3: Emotion categories emerge from structured configurations of cognitive appraisals rather than isolated affective cues. As shown in Figure 4, emotions exhibit distinct joint appraisal profiles even when individual dimensions overlap. For example, anger is characterized by con- flict, maintained control, external attribution, elevated arousal, and confrontation, whereas sadness is associated with loss, limited coping, situational attribution, restrained arousal, and withdrawal. Likewise, fear emphasizes threat, low coping, high arousal, and escape, while anxiety is more as- sociated with conflict, uncertain coping, elevated arousal, and avoidance. These patterns suggest that emotion differentiation is better characterized by configurations of cognitive appraisals than by isolated affective cues. MERCaptionPlus Silhouette score -0.090 CH index 711.1 CogEmo40K Contempt Anxiety Fear Silhouette score 0.081 CH index 3724.2 Anger Anxiety Contempt Disgust Fear Happiness Neutral Sadness Accuracy Macro-F1 Weighted-F1 0% 20% 40% 60% 80% 100% 5-fold mean score Per-class accuracyOverall metrics (a) Emotion-label structure in t-SNE space (b) Emotion-label probing performance AngerAnxietyContemptDisgustFearHappinessNeutralSadness CogEmo40KMERCaptionPlusPerformance gapMean ± 1 SD Figure 5: Emotion Alignment of Cognition-Grounded Supervision. We compare CogEmo-40K and MERCaption+ on their intersected samples under the same eight-way emotion partition. Emotion Alignment of Cognition-Grounded Supervision. On the samples shared by CogEmo- 40K and MERCaption+ (Lian et al., 2025a), we compare how the two forms of supervision encode emotion-related information. As shown in Figure 5(a), cognition-grounded reasoning yields clearer emotion-label structure than open-ended descriptions. For Figure 5(b), we conduct five-fold cross- validation with the same XLM-RoBERTa classification framework for each supervision source. Af- ter removing explicit emotion-conclusion sentences, the classifier predicts eight emotion categories from each source’s text. CogEmo-40K achieves stronger per-class and overall performance, sug- gesting that its appraisal reasoning encodes information more closely aligned with the final emotion labels. Detailed settings are provided in Appendix E.1. Effect of MoE Placement.We investigate where the sparse expert layers should be in- troduced. We compare four placement strate- gies, including First-Half, Second-Half, Inter- val, and All. The Interval strategy achieves the best overall performance, raising accuracy from 64.29%→ 71.43% over First-Half placement and supporting our design of inserting MoE lay- ers at alternating Transformer blocks for more balanced cognition-specific adaptation. Dense vs. Sparse Experts. We further com- pare the dense baseline with different sparse routing strategies. Simply increasing the num- ber of experts with Top-1 routing does not im- prove performance. In contrast, Top-2 rout- ing improves both emotion recognition and cognitive appraisal quality, including accuracy from 65.97%→ 71.43% and Macro-F1 from 52.28%→ 57.17% under the four-expert set- ting, suggesting that collaborative expert activa- tion is more effective for cognition-level emo- tion reasoning. Table 3: Effect of different MoE placement strate- gies on CogEmo-Bench-Eval. Architecture AEQSAcc.M-F1 First-Half68.7664.2950.80 Second-Half70.1068.4554.57 Interval70.6071.4357.17 All69.7670.0455.49 Table 4: Comparison of dense and sparse expert configurations on CogEmo-Bench-Eval. Configuration AEQSAcc.M-F1 Dense Model Baseline70.4867.2653.27 Sparse MoE 4E / Top-170.0265.9752.28 4E / Top-270.6071.4357.17 8E / Top-169.8064.6851.40 8E / Top-269.7670.0455.49 8 6RELATED WORK 6.1FROM MULTIMODAL PERCEPTION TO EMOTION REASONING MLLMs align visual, acoustic, and temporal evidence with language, providing a foundation for open-ended multimodal interpretation (Han et al., 2024; Li et al., 2024; Chu et al., 2024). In af- fective computing, earlier systems mapped fused multimodal representations to predefined emotion categories (Zhang et al., 2024; Zadeh et al., 2017; Tsai et al., 2019), while contextual models demon- strated the importance of scenes and conversational history beyond isolated expressions (Kosti et al., 2017; Lee et al., 2019; Poria et al., 2017). Emotion-oriented MLLMs further generate explanations alongside predictions (Cheng et al., 2024a; Lian et al., 2025a; Zhao et al., 2025), supported by explanation-rich datasets and unified benchmarks (Cheng et al., 2024a; Lian et al., 2025a; 2026; Zhang et al., 2026a). Nevertheless, current supervision and evaluation still emphasize final labels, observable cues, or textual plausibility. Because fluent rationales can be unstable and benchmark formats can admit shortcuts (Jiang et al., 2025; Salido et al., 2025), explanation generation alone does not establish that a model captures how an emotion arises. 6.2COGNITIVE APPRAISAL AS AN INTERMEDIATE REPRESENTATION Appraisal theories locate emotion not in observable cues alone, but in how an individual evaluates an event with respect to goals, coping potential, responsibility, expected outcomes, action readi- ness and so on (Roseman, 1991; Frijda, 1987; Ellsworth & Scherer, 2003; Moors et al., 2013). Empirical and computational studies further link appraisal patterns and abstract event representa- tions to emotion judgments (Yeo & Ong, 2024; Ong et al., 2015; Skerry & Saxe, 2015; Houlihan et al., 2023). Although appraisal has informed text-based emotion modeling and Theory-of-Mind evaluation (Hofmann et al., 2020; Troiano et al., 2023; Yeo & Jaidka, 2025), it has yet to be inte- grated systematically with multimodal perception and free-form reasoning. We operationalize ap- praisal as an explicit intermediate representation between observation and emotion: CogEmo-40K provides perception-to-appraisal supervision, CogEmo-MoE models this transition, and CogEmo- Bench evaluates it. 7CONCLUSION We introduce a perception-to-appraisal paradigm for multimodal emotion understanding. CogEmo- 40K supplies cognition-grounded supervision, CogEmo-MoE applies sparse appraisal-specific adap- tation, and CogEmo-Bench uses AEQS to evaluate appraisal evidence alongside emotion recogni- tion. On CogEmo-Bench, CogEmo-MoE achieves 70.60 AEQS, 71.43% accuracy, and 57.17% Macro-F1. Analyses of appraisal language reveal distinct patterns across emotion categories, while cross-domain experiments suggest that cognition-grounded supervision can transfer beyond the training domain. These results support appraisal reasoning as an intermediate representation be- yond surface-level cue-to-label emotion recognition. ACKNOWLEDGMENTS We gratefully acknowledge the psychology researchers and domain experts who provided valu- able consultation on cognitive appraisal theory, annotation design, and human cognitive alignment throughout this project. We also thank the affective computing researchers, annotators, and volun- teers who contributed to the verification and refinement of CogEmo-40K. AI USE STATEMENT Generative AI tools were used to improve the clarity, grammar, and readability of the manuscript. All AI-assisted edits were reviewed and revised by the authors, who take full responsibility for the final content. 9 REPRODUCIBILITY STATEMENT We provide detailed descriptions of the data construction, model training, and evaluation proce- dures to facilitate reproducibility. The complete two-stage data-generation prompts and human- verification protocols for CogEmo-40K are provided in Appendix B and Appendix B.3. Model architecture, three-stage training configurations, optimization settings, and training prompts for CogEmo-MoE are reported in Appendix C. For CogEmo-Bench, we provide the evaluation prompts, Appraisal Extractor, metric computation procedure, and model-specific evaluation settings in Ap- pendix D. Additional experimental details, including the 5-fold cognition–emotion alignment anal- ysis and cross-domain MERBench evaluation, are provided in Appendix E.1 and Appendix E.2, respectively. These details specify the data sources, model configurations, training hyperparame- ters, prompts, and evaluation protocols used throughout the experiments. ETHICS STATEMENT CogEmo-40K is constructed from the existing MER2025 multimodal dataset rather than from newly collected audiovisual recordings. Human involvement in our study is limited to annotation, verifica- tion, and refinement of automatically generated perceptual descriptions, cognitive appraisal reason- ing, and emotion labels. Because emotion and cognitive appraisal are inherently context-, culture-, and individual-dependent, the resulting annotations should be viewed as evidence-grounded inter- pretations rather than definitive descriptions of an individual’s internal psychological state. We therefore caution against using model-generated appraisal reasoning for clinical diagnosis, psycho- logical profiling, or other high-stakes decisions without appropriate human oversight. Potential biases inherited from the source data, including linguistic, cultural, and domain biases, are further discussed in the limitations section. REFERENCES AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2236–2246, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1208. URL https://aclanthology.org/P18-1208/. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359, 2008. Jingyi Chen, Zhimeng Guo, Jiyun Chun, Pichao Wang, Andrew Perrault, and Micha Elsner. Do audio llms really listen, or just transcribe? measuring lexical vs. acoustic emotion cues reliance. In Proceedings of the 19th Conference of the European Chapter of the Association for Computa- tional Linguistics (Volume 1: Long Papers), p. 5848–5877, 2026. Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander G. Hauptmann. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning. In Advances in Neural Information Processing Systems, volume 37, p. 110805–110853, 2024a. Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. Emotion-LLaMA: Multimodal Emotion Recognition and Reason- 10 ing with Instruction Tuning. Advances in Neural Information Processing Systems, 37:110805– 110853, 2024b. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. IEEE, 2023. Phoebe C Ellsworth and Klaus R Scherer. Appraisal processes in emotion. 2003. Nico H Frijda. Emotion, cognitive structure, and action tendency. Cognition and emotion, 1(2): 115–143, 1987. Google.Introducing gemini 3.6 flash, 3.5 flash-lite, and 3.5 flash cyber, July 2026.URL https://blog.google/innovation-and-ai/models-and-research/ gemini-models/gemini-3-6-flash-3-5-flash-lite-3-5-flash-cyber/. Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 26574– 26585. IEEE, 2024. Jan Hofmann, Enrica Troiano, Kai Sassenberg, and Roman Klinger. Appraisal theories for emo- tion classification in text. In Proceedings of the 28th international conference on computational linguistics, p. 125–138, 2020. Sean Dae Houlihan, Max Kleiman-Weiner, Luke B Hewitt, Joshua B Tenenbaum, and Rebecca Saxe. Emotion prediction as computation over a generative theory of mind. Philosophical transactions. Series A, Mathematical, physical, and engineering sciences, 381(2251):20220047, 2023. Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint arXiv:2502.09621, 2025. Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 13700–13710. IEEE, 2024. Kimi Team. Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. Ronak Kosti, Jose M Alvarez, Adria Recasens, and Agata Lapedriza. Emotion recognition in con- text. In 2017 IEEE conference on computer vision and pattern recognition (CVPR), p. 1960– 1968. IEEE, 2017. Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn. Context-aware emotion recognition networks. In 2019 IEEE/CVF international conference on computer vision (ICCV), p. 10142–10151. IEEE, 2019. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tun- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(9):7543–7557, 2025a. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. Science China Information Sciences, 68(10):200102, 2025b. Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European conference on computer vision, p. 323–340. Springer, 2024. 11 Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mngyu Xu, Kexin Wang, Ke Xu, Yu He, Ying Li, Jinming Zhao, et al. Mer 2023: Multi-label learning, modality robustness, and semi-supervised learning. In Proceedings of the 31st ACM international conference on multimedia, p. 9610– 9614, 2023. Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, Jinming Zhao, Ziyang Ma, Xie Chen, et al. Mer 2024: Semi-supervised learning, noise robustness, and open-vocabulary multimodal emotion recognition. In Proceedings of the 2nd International Work- shop on Multimodal and Responsible Affective Computing, p. 41–48, 2024. Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. In Proceedings of the 42nd International Conference on Machine Learning, 2025a. Zheng Lian, Rui Liu, Kele Xu, Bin Liu, Xuefei Liu, Yazhou Zhang, Xin Liu, Yong Li, Zebang Cheng, Haolin Zuo, et al. Mer 2025: When affective computing meets large language models. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 13837–13842, 2025b. Zheng Lian, Haiyang Sun, Licai Sun, Haoyu Chen, Lan Chen, Hao Gu, Zhuofan Wen, Shun Chen, Zhang Siyuan, Hailiang Yao, et al. Ov-mer: Towards open-vocabulary multimodal emotion recog- nition. In Proceedings of the 42nd International Conference on Machine Learning, 2025c. Zheng Lian, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, and Jianhua Tao. Mer- bench: A unified evaluation benchmark for multimodal emotion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026. Qiao Liang, Ying Shen, Yao Liu, Tiantian Chen, and Lin Zhang. Why do emotions change? appraisal-guided reasoning for emotion–cause triplet extraction in conversations. In Proceed- ings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 11750–11772, 2026. Bin Lin, Zhenyu Tang, Yang Ye, Jinfa Huang, Junwu Zhang, Yatian Pang, Peng Jin, Munan Ning, Jiebo Luo, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. IEEE Transactions on Multimedia, 2026. Yihe Liu, Ziqi Yuan, Huisheng Mao, Zhiyun Liang, Wanqiuyue Yang, Yuanzhe Qiu, Tie Cheng, Xi- aoteng Li, Hua Xu, and Kai Gao. Make acoustic and visual cues matter: Ch-sims v2. 0 dataset and av-mixup consistent module. In Proceedings of the 2022 international conference on multimodal interaction, p. 247–258, 2022a. Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. Mafw: A large-scale, multi-modal, compound affective database for dynamic facial ex- pression recognition in the wild. In Proceedings of the 30th ACM international conference on multimedia, p. 24–32, 2022b. Meng Luo, Bobo Li, Shanqing Xu, Shize Zhang, Qiuchan Chen, Menglu Han, Wenhao Chen, Yanxiang Huang, Hao (Scofield) Fei, Mong-Li Lee, and Wynne Hsu.Unveil- ing the cognitive compass:Theory-of-mind–guided multimodal emotion reasoning.In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (eds.), In- ternational Conference on Learning Representations, volume 2026, p. 85428–85498, 2026. URL https://proceedings.iclr.c/paper_files/paper/2026/file/ 89ab43b626b480347dbaccda7be8aedb-Paper-Conference.pdf. Agnes Moors, Phoebe C Ellsworth, Klaus R Scherer, and Nico H Frijda. Appraisal theories of emotion: State of the art and future development. Emotion review, 5(2):119–124, 2013. Desmond C Ong, Jamil Zaki, and Noah D Goodman. Affective cognition: Exploring lay theories of emotion. Cognition, 143:141–162, 2015. OpenAI.Introducing gpt-5.2, December 2025.URL https://openai.com/index/ introducing-gpt-5-2/. 12 OpenAI.Introducing gpt-5.4, mar 2026a.URL https://openai.com/index/ introducing-gpt-5-4/. OpenAI. Gpt-5.6: Frontier intelligence that scales with your ambition, July 2026b. URL https: //openai.com/index/gpt-5-6/. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis- Philippe Morency. Context-dependent sentiment analysis in user-generated videos. In Proceed- ings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers), p. 873–883, 2017. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Anna Korhonen, David Traum, and Llu ́ ıs M ` arquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 527–536, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1050. URL https: //aclanthology.org/P19-1050/. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen. ai/blog?id=qwen3.5. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. PmLR, 2021. Ira J Roseman. Appraisal determinants of discrete emotions. Cognition & Emotion, 5(3):161–200, 1991. Eva S ́ anchez Salido, Julio Gonzalo, and Guillermo Marco. None of the others: a general technique to distinguish reasoning from memorization in multiple-choice llm evaluation benchmarks. arXiv preprint arXiv:2502.12896, 2025. Amy E Skerry and Rebecca Saxe. Neural representations of emotion are organized around abstract event features. Current biology, 25(15):1945–1954, 2015. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. In Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the Era of Interactive Assistants!, p. 11–23, 2023. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. In International Conference on Learning Representations, volume 2024, p. 16607–16629, 2024. Qwen Team. Qwen3. 8: A new bar for coding and cowork. Qwen, August, 2026. Jayant Teotia, Xiaowei Wang, Xulang Zhang, Rui Mao, and Erik Cambria. Evaluating multimodal large language model narrative interpretation through the lens of appraisal theory. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), volume 11, p. 11417–11426, 2026. Enrica Troiano, Laura Ana Maria Oberl ̈ ander, and Roman Klinger. Dimensional modeling of emo- tions in text with appraisal theories: Corpus creation, annotation reliability, and prediction. Com- putational Linguistics, 49(1):1–72, 2023. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Anna Korhonen, David Traum, and Llu ́ ıs M ` arquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 6558–6569, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1656. URL https: //aclanthology.org/P19-1656/. 13 Lanrui Wang, Jiangnan Li, Zheng Lin, Fandong Meng, Chenxu Yang, Weiping Wang, and Jie Zhou. Empathetic dialogue generation via sensitive emotion recognition and sensible knowledge selec- tion. In Findings of the Association for Computational Linguistics: EMNLP 2022, p. 4634–4645, 2022. xAI. Grok 4.3, 2026. URL https://docs.x.ai/developers/models/grok-4.3. Hongxia Xie, Chu-Jun Peng, Yu-Wen Tseng, Hung-Jen Chen, Chan-Feng Hsu, Hong-Han Shuai, and Wen-Huang Cheng. Emovit: Revolutionizing emotion insights with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 26586– 26595. IEEE, 2024. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jingren Zhou, and Junyang Lin. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765, 2025. Yaoxun Xu, Hangting Chen, Jianwei Yu, Qiaochu Huang, Zhiyong Wu, Shi-Xiong Zhang, Guangzhi Li, Yi Luo, and Rongzhi Gu. Secap: Speech emotion captioning with large language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, p. 19323–19331, 2024. Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. In International Conference on Learning Representations, volume 2025, p. 98891–98913, 2025. Gerard C Yeo and Desmond C Ong. Associations between cognitive appraisals and emotions: A meta-analytic review. Psychological Bulletin, 150(12):1440, 2024. Gerard Christopher Yeo and Kokil Jaidka. Beyond context to cognitive appraisal: Emotion reasoning as a theory of mind benchmark for large language models. In Findings of the Association for Computational Linguistics: ACL 2025, p. 26517–26525, 2025. Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 3718–3727, 2020. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Ten- sor fusion network for multimodal sentiment analysis. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, p. 1103–1114, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1115. URL https://aclanthology. org/D17-1115/. Fan Zhang, Zebang Cheng, Chong Deng, Haoxuan Li, Zheng Lian, Qian Chen, Huadai Liu, Wen Wang, YiFan Zhang, Renrui Zhang, et al. Mme-emotion: A holistic evaluation benchmark for emotional intelligence in multimodal large language models. In International Conference on Learning Representations, volume 2026, p. 48767–48807, 2026a. Hongrui Zhang, Daiqing Wu, Yangyang Li, Kuien Liu, Yuhui Wang, Yu Zhou, and Sicheng Zhao. Multimodal emotion recognition with large language models. In Diego Calvanese (ed.), Pro- ceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26, p. 8088–8096. International Joint Conferences on Artificial Intelligence Organization, 8 2026b. doi: 10.24963/ijcai.2026/897. URL https://doi.org/10.24963/ijcai.2026/897. Survey Track. Shiqing Zhang, Yijiao Yang, Chen Chen, Xingnan Zhang, Qingming Leng, and Xiaoming Zhao. Deep learning-based multimodal emotion recognition from audio, visual, and text modalities: A systematic review of recent advancements and future prospects. Expert Systems with Applications, 237:121692, 2024. 14 Tianlin Zhang, Kailai Yang, Shaoxiong Ji, and Sophia Ananiadou. Emotion fusion for mental illness detection from social media: A survey. Information Fusion, 92:231–246, 2023. Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable omni-multimodal emotion recog- nition with reinforcement learning. arXiv preprint arXiv:2503.05379, 2025. 15 APPENDIX CONTENTS Appendix A. Limitations and Social Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Appendix B. Details of CogEmo-40K Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Appendix C. Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Appendix D. Details of CogEmo-Bench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Appendix E. Additional Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 ALIMITATIONS AND SOCIAL IMPACT While our work advances multimodal emotion understanding from surface-level affective cue recog- nition toward cognition-grounded appraisal reasoning, several directions remain open for future ex- ploration: • First, CogEmo-40K adopts six complementary appraisal dimensions inspired by estab- lished cognitive appraisal theories. These dimensions capture important aspects of emotion formation but are not intended to exhaust the full space of cognitive emotion processes. Fu- ture work may explore richer appraisal dimensions and alternative theoretical formulations. • Second, CogEmo-40K is constructed from existing multimodal video data and therefore remains limited by its current domain, language, cultural, and emotion-space coverage. Extending cognition-grounded emotion understanding to more diverse multilingual, cross- cultural, and open-world settings is an important future direction. • Third, CogEmo-Bench relies on a fixed Appraisal Extractor and semantic embedding model for normalizing and comparing free-form reasoning outputs. Although the same evaluation protocol is applied consistently to all models, future work may further investi- gate evaluator-independent cognition-level metrics. Overall, cognition-grounded emotion understanding offers a promising direction for building more reliable and generalizable affective systems. However, model-generated appraisal interpretations should be treated as evidence-grounded reasoning rather than definitive assessments of an individ- ual’s internal psychological state, especially in high-stakes applications. We hope this work en- courages further research toward more robust, flexible, and human-aligned emotional intelligence in MLLMs. 16 BDETAILS OF COGEMO-40K DATASET B.1SOURCE DATASET: MER2025 The raw data for CogEmo-40K are derived from MER2025 (Lian et al., 2025b), the third edition of the MER challenge, which focuses on integrating large language models with multimodal emotion understanding. MER2025 includes a large-scale collection of multimodal samples without emo- tion annotations, providing video, audio, facial information, and subtitle text. Following the data screening protocol of AffectGPT (Lian et al., 2025a), we adopt the same MER2025 source samples used to construct MERCaption (Lian et al., 2025a) as the raw data for CogEmo-40K. Importantly, we do not use the automatically generated descriptions provided by MERCaption. Instead, we treat these source samples as unlabeled multimodal inputs and construct new cognition-grounded rea- soning annotations through the two-stage perception-to-appraisal pipeline described below. The generated annotations are further refined through multi-round filtering and human-guided alignment (Section B.3), forming the final CogEmo-40K dataset. B.2TWO-STAGE DATA GENERATION PROMPTS We adopt a two-stage prompt pipeline to generate CogEmo-40K: Stage 1 produces structured, emotion-free perceptual descriptions from the multimodal inputs; Stage 2 takes these perceptual descriptions as input and performs cognitive appraisal reasoning to derive cognition-grounded emo- tion interpretations. Stage 1 uses Qwen3-Omni-30B-A3B-Instruct (Xu et al., 2025), while Stage 2 uses GPT-5.4 (OpenAI, 2026a). The complete prompts for both stages of data generation are provided below. Stage 1: Perception Generation [Instruction] You are given a multimodal input (video + audio). Your task is to describe the scene, facial behavior, and audio signals in a structured and strictly objective way. [Goal] 1. Provide a brief description of the overall scene. 2. Provide a HIGH-DENSITY, structured description of facial movements. 3. Provide a HIGH-DENSITY, structured description of audio features. [Strict Constraints] • DO NOT use any emotion words (e.g., happy, sad, angry, fear). • Only describe observable visual or acoustic information. [Scene Description Rules] • Describe environment, people, and interactions • Include visible actions only • Keep it brief (1–2 sentences) • Use neutral and factual language [Facial Description Rules] You MUST describe ALL facial feature categories below. For each category include: (1) state; (2) degree (slight / moderate / strong / stable); (3) dynamics (no change / gradual change / fluctuating). Required categories: • Gaze: direction, stability, shifts • Eyes: openness, eyelid tension, blinking pattern, asymmetry • Eyebrows: position, movement, contraction or relaxation, symmetry • Mouth: opening degree, lip tension, corner direction, lip compression • Jaw: tension level, clenching, movement • Facial muscles: overall tension, rigidity, relaxation • Left–Right asymmetry: differences in facial movement or tension • Head pose: orientation, tilt, movement • Temporal dynamics: changes over time (start→ middle→ end) • Overall facial mobility: level of movement intensity (high / medium / low) You MUST additionally describe fine-grained facial action components using observable anatomical or geometric descriptors, including but not limited to: • Lip corners: upturned / downturned / stretched / retracted • Cheek region: raised / tightened / relaxed / bulging 17 • Eye aperture: widened / narrowed / squinting / partially closed • Eyelid behavior: lifted / lowered / tightened / compressed • Nasolabial region: deepened / relaxed • Teeth visibility: visible / partially visible / not visible • Orbital tension: increased / reduced / stable • Facial symmetry changes during motion: symmetric / asymmetric activation Use only visible structural descriptions; do NOT label emotions or interpret meaning. These features must be embedded into Mouth / Eyes / Facial muscles sections. [Audio Description Rules] Describe audio features from the perspective of the ACTIVE SPEAKER (the person who is speaking or producing vocal sound in the video). For each category include: (1) state; (2) degree (slight / moderate / strong); (3) dynamics (no change / gradual change / fluctuating). Required categories: • Speech presence: continuous speech / intermittent speech / no speech • Spoken content (verbatim transcription if audible): exact spoken words as heard, without interpretation, summarization, or paraphrasing • Voice intensity: soft / moderate / loud / variable • Speech rate: slow / normal / fast / changing • Pitch variation: flat / slightly varied / highly varied • Prosody dynamics: stable / gradual modulation / fluctuating • Articulation clarity: clear / moderately clear / unclear [Writing Requirements] • Write in 1 unified paragraph for each modality (Scene / Facial / Audio) • Preserve temporal order of changes • Embed all attributes naturally into continuous flow • Keep descriptions strictly visual or acoustic and measurable Stage 2: Cognitive Appraisal (CoT) You are an expert in multimodal emotion analysis based on cognitive appraisal theory. Your task is to infer the emotional state from scene, facial behavior, audio, and spoken content. Requirements: 1. You MUST integrate evidence and appraisal inference together. Do NOT separate “evidence” and “appraisal” into different sections. Each appraisal dimension must be inferred together with its supporting evidence in the SAME sentence. 2. You MUST produce ONE single coherent paragraph. No bullet points, no section headers, and no line breaks. 3. For each appraisal dimension, you MUST follow this format: first briefly describe the supporting evidence, then explicitly state the appraisal result in parentheses. 4. You MUST include ALL six appraisal dimensions in the following order: Goal Congruence, Coping, Accountability, Future, Arousal, Action Tendency. 5. The paragraph MUST end with a brief summary of the overall appraisal pattern and a final emotion label selected from: happiness, sadness, anger, fear, anxiety, disgust, contempt, neutral. B.3HUMAN VERIFICATION AND QUALITY FILTERING To ensure that CogEmo-40K provides reliable perceptual evidence and human-aligned cognitive ap- praisal labels, we conduct staged human quality control after automatic generation. Stage 1 audits description-level perception outputs; Stage 2 verifies cognition-level appraisal judgments and emo- tion labels. All annotators are trained with the standardized rubrics below and inspect each sample against the original video and audio. Stage 1: Description-level quality assessment. This stage evaluates the objective multimodal descriptions generated by Qwen3-Omni-30B-A3B (Xu et al., 2025), covering Scene, Facial, and Audio. Each sample is rated on a 1–5 scale along three criteria: Accuracy (whether the descrip- tion matches the video and contains no hallucination), Completeness (whether Scene, Facial, and 18 Audio are covered), and Objectivity (whether the description stays within observable evidence with- out emotion or psychological inference). We randomly sample 1,000 descriptions and ask five re- searchers with backgrounds in affective computing to score each sample holistically under the rubric in Table 5, with priority given to Accuracy, then Completeness. Fluent wording alone does not justify a high score; annotators must strictly separate observable description from emotion or psychological inference. Table 5: Stage 1 description-level human quality rubric (1–5). ScoreLevelCriteria 5Excellent Fully accurate; covers Scene, Facial, and Audio; includes fine-grained cues (e.g., gaze, mouth dynamics, prosody); fully objective with no subjective inference; clear and unambiguous. Directly usable as high-quality data. 4Good Mostly accurate with only minor deviations; three modalities largely covered; slightly coarser or fewer details; remains largely objective. Usable with limited information loss. 3Fair Noticeable missing information (e.g., one modality absent); locally inaccurate or vague; occasional mild subjective phrasing. Marginally usable but unstable in quality. 2Poor Multiple inaccuracies; clear modality omissions; subjective inference or non-compliant description. Not recommended for use. 1Bad Severely incorrect or inconsistent with the video; obvious hallucination; lacks usable information. Should be discarded. In addition to the overall rubric, annotators apply the mandatory score caps in Table 6. Records are stored as (id, score, notes), where notes are required whenever the score is≤ 3. When a sample cannot be judged reliably, annotators document the reason in notes. Table 6: Mandatory score caps for Stage 1 description assessment. CapTriggering conditions ≤ 3 Emotion words appear (e.g., angry / happy / nervous); subjective judgments occur (e.g., “looks angry”); any one of Scene / Facial / Audio is missing. ≤ 2 Multiple modalities are missing; clear inconsistency with the video; description is severely vague or non-specific. Under this protocol, the average scores of Accuracy, Completeness, and Objectivity all exceed 4.0, indicating that Stage 1 descriptions provide reliable observable evidence for subsequent appraisal reasoning. Stage 2: Cognition-level human verification. This stage verifies cognition-level reasoning across the six appraisal dimensions and the final emotion label. Given the substantial annotation cost, we randomly select approximately one-third of the 115,595 automatically generated cognition instances for human verification, without preferentially selecting samples based on their difficulty. Before annotation, well-trained annotators receive standardized instructions and review the defi- nitions of the six appraisal dimensions in Table 7. For each sample, annotators first inspect the original video and audio, considering scene context, facial behavior, body action, vocal cues, and spoken content. They then verify whether the generated reasoning for each appraisal dimension is supported by the multimodal evidence and consistent with the corresponding dimension definition. The final emotion label is additionally checked for consistency with both the multimodal evidence and the verified appraisal reasoning. An instance is directly retained when its emotion label agrees with human judgment and at least four of the six appraisal dimensions are judged to be consistent. We adopt four dimensions as a practical majority-based threshold: it requires broad agreement across the appraisal structure while allowing limited disagreement that may arise from the inherent subjectivity and individual variation of cognitive appraisal. 19 Failed instances are not discarded: human annotations are used to correct emotion labels and ap- praisal reasoning, regenerate evidence-grounded reasoning explanations, and undergo additional review by five affective computing researchers. This correction-based filtering yields the final CogEmo-40K of approximately 40K high-quality instruction-tuning instances. Table 7: Definitions of the six cognitive appraisal dimensions used in Stage 2 human verification. DimensionDefinition Goal CongruenceEvaluates how the perceived situation relates to the subject’s goals, needs, or ex- pectations, including whether it supports, obstructs, or has little relevance to them. Coping PotentialReflects the subject’s perceived ability to manage, control, adapt to, or respond effectively to the situation. AccountabilityCaptures how responsibility or causal attribution is assigned, including whether the situation is attributed to the subject, another person, or no clearly identifiable agent. Future ExpectancyDescribes how the subject interprets the temporal development and expected con- sequences of the situation, including anticipation, continuation, or the absence of a salient future-oriented expectation. ArousalReflects the perceived level of physiological or behavioral activation associated with the situation, as supported by observable multimodal cues such as vocal in- tensity, facial dynamics, or body movement. Action TendencyCaptures the behavioral orientation implied by the appraisal, such as engaging with, avoiding, confronting, or otherwise responding to the situation. B.4DATASET STATISTICS AND COMPARISON 0200400 (a) EmoVIT 0 10000 20000 30000 frequency 0200400 (b) MERR-Fine 0200400 (c) MERR-Coarse 0200400 (d) MAFW 0200400 (e) OV-MERD 0 10 20 30 frequency 0200400 (f) MER-Caption 0200400 (g) CogEmo-40K Figure 6: Distribution of description lengths (word count) across descriptive emotion datasets. B.4.1TEXT LENGTH COMPARISON We compare CogEmo-40K with several representative multimodal emotion datasets, including EmoVIT (Xie et al., 2024), MERR-Fine (Cheng et al., 2024a), MERR-Coarse (Cheng et al., 2024a), MAFW (Liu et al., 2022b), OV-MERD (Lian et al., 2025c), and MERCaption (Lian et al., 2025a), in terms of text length measured by word count. As shown in Figure 6, CogEmo-40K contains substantially longer reasoning annotations than MERR and MAFW, while remaining more concise 20 than MERCaption. This reflects the different supervision focus of CogEmo-40K, which emphasizes structured cognitive appraisal reasoning rather than purely descriptive emotion-related content. B.4.2EMOTION DISTRIBUTION Happiness Sadness Anger Fear Anxiety Disgust Contempt Neutral (a) CogEmo-40K training set 0 5,000 10,000 15,000 Number of samples 12.3% 4,727 13.8% 5,332 12.1% 4,654 4.4% 1,710 7.6% 2,914 6.1% 2,339 6.7% 2,575 37.1% 14,281 Happiness Sadness Anger Fear Anxiety Disgust Contempt Neutral (b) CogEmo-Bench-Eval 0 100 200 300 400 500 Number of samples 8.2% 83 12.3% 124 7.9% 80 7.0% 71 5.5% 55 9.6% 97 5.7% 57 43.8% 441 Figure 7: Emotion category distributions of CogEmo-40K and CogEmo-Bench-Eval. (a) CogEmo- 40K training set (N=38,532). (b) CogEmo-Bench-Eval (N=1,008). Bars indicate the number of samples in each emotion category, with percentages annotated above. As shown in Figure 7, both CogEmo-40K and CogEmo-Bench-Eval exhibit a distribution consistent with naturalistic emotion data, where neutral constitutes the dominant category. Beyond this over- all trend, CogEmo-Bench-Eval assigns relatively larger proportions to fear and disgust, which are less frequent and often more challenging to distinguish. This design preserves the naturally imbal- anced emotion distribution while providing stronger coverage of difficult low-frequency categories, making the benchmark more demanding for fine-grained cognition-grounded emotion reasoning. B.4.3COGNITIVE LANGUAGE PROFILES To further characterize the cognition-level supervision in CogEmo-40K, we analyze recurring lan- guage patterns across the six cognitive appraisal dimensions for each emotion category. Since the reasoning annotations are expressed in free-form natural language, we first apply the same fixed Appraisal Extractor used in CogEmo-Bench to normalize each annotation into the six predefined appraisal dimensions. The extractor only reorganizes explicitly expressed content and does not judge correctness or introduce additional evidence. We then group the 38,532 reasoning annotations by emotion category and appraisal dimension, and report the three most frequent semantic phrases and their occurrence rates. Percentages indicate the proportion of samples within the corresponding emotion category containing each phrase. Ta- ble 8 provides the full phrase-level statistics for all eight emotion categories across the six appraisal dimensions. These statistics reveal distinct appraisal tendencies across emotions. For example, anger frequently co-occurs with goal conflict and externally directed attribution, whereas fear is more strongly as- sociated with perceived threat, limited coping capacity, and withdrawal-related tendencies. The compact cognitive profiles presented in the main paper are semantic summaries of these recurring phrase-level patterns rather than direct copies of the raw phrases reported here. 21 Table 8: Full Cognitive Language Profiles for Eight Emotion Categories. Part I: Goal Congruence, Coping, and Accountability EmotionGoal CongruenceCopingAccountability Angersituation conflicts (28%) situation involves (6%) unfavorable evaluation (3%) able to (51%) clear articulation (43%) feels able (42%) another person (31%) the other (24%) source of (22%) Anxietysituation conflicts (13%) situation involves (13%) unfavorable evaluation (2%) clear articulation (33%) to manage (24%) able to (20%) source of (45%) the source (43%) attributed to (25%) Contemptsituation conflicts (31%) unfavorable evaluation (13%) situation involves (10%) clear articulation (53%) able to (51%) feels able (38%) source of (33%) the source (33%) another person (31%) Disgustsituation conflicts (31%) situation involves (8%) unfavorable evaluation (6%) clear articulation (34%) to manage (28%) able to (25%) the source (53%) source of (53%) another person (30%) Fearsituation conflicts (9%) situation involves (2%) perceived ability (24%) to manage (21%) low perceived (17%) source of (61%) the source (58%) attributed to (30%) Happinessaligns with (47%) situation supports (35%) situation aligns (31%) able to (50%) feels able (41%) clear articulation (37%) the cause (27%) another person (15%) the other (15%) Neutralaligns with (19%) situation aligns (14%) personal goals (7%) clear articulation (58%) able to (55%) to manage (41%) source of (20%) the source (19%) another person (16%) Sadnesssituation conflicts (26%) situation involves (10%) unfavorable evaluation (4%) perceived ability (20%) relaxed jaw (18%) clear articulation (16%) the cause (31%) the source (22%) source of (21%) Part I: Future, Arousal, and Action Tendency EmotionFutureArousalAction Tendency Angerthe present (21%) a present (17%) distant possibility (13%) physiological activation (32%) facial tension (31%) point to (28%) inclination to (59%) behavioral inclination (48%) a behavioral (47%) Anxietyconcern about (19%) points to (18%) the present (15%) activation rather (34%) facial mobility (33%) low facial (32%) inclination to (53%) behavioral inclination (34%) a behavioral (33%) Contemptthe present (23%) a present (21%) future threat (13%) facial mobility (58%) low facial (55%) activation rather (54%) inclination to (48%) behavioral inclination (27%) a behavioral (27%) Disgustthe present (25%) tied to (18%) a present (17%) point to (39%) facial mobility (36%) low facial (31%) inclination to (56%) behavioral inclination (48%) a behavioral (47%) Fearconcern about (25%) what may (21%) about what (21%) physiological activation (31%) point to (17%) facial tension (16%) inclination to (40%) behavioral inclination (36%) a behavioral (33%) Happinessthe present (28%) points to (17%) a present (12%) facial mobility (57%) point to (41%) low facial (41%) inclination to (53%) inclination toward (43%) behavioral inclination (39%) Neutralpoints to (21%) the present (20%) a present (13%) low facial (69%) facial mobility (69%) point to (46%) inclination to (30%) orientation toward (25%) toward the (24%) Sadnessthe present (21%) future threat (16%) a present (14%) activation rather (37%) point to (35%) facial mobility (29%) inclination to (61%) behavioral inclination (47%) a behavioral (46%) 22 CIMPLEMENTATION DETAILS C.1DETAILS OF COGEMO-MOE MODEL CogEmo-MoE extends the MoE-LLaVA (Lin et al., 2026) framework to support joint video, audio, and text modeling, while maintaining a compact model scale of approximately 3.1B parameters. It adopts a Qwen-1.8B (Bai et al., 2023) language model with CLIP ViT-L/14 (Radford et al., 2021) and CLAP (Elizalde et al., 2023) as the visual and audio encoders, respectively, and uses two-layer GELU MLP projectors for modality alignment. The model is trained progressively for multimodal alignment, perception enhancement, and cognitive appraisal specialization using DeepSpeed ZeRO- 2 with bfloat16, and a maximum sequence length of 2048 tokens. All experiments are conducted on two NVIDIA A100 80GB GPUs. Detailed configurations are summarized in Table 9. Table 9: Training configuration of CogEmo-MoE across three stages. ConfigurationStage 1Stage 2Stage 3 Trainable modulesProjectorsFull modelMoE FFN + router MoE enabled–Sparse (4 experts, top-2) Capacity factor–1.5 Router aux. loss λ–0.01 Epochs416 Learning rate1× 10 −3 2× 10 −5 1× 10 −5 Warmup ratio0.030.030.05 LR schedulercosinecosinecosine Batch size (per device)884 Gradient accumulation121 Max sequence length204820482048 Precisionbf16bf16bf16 DeepSpeed strategyDeepSpeed ZeRO-2DeepSpeed ZeRO-2DeepSpeed ZeRO-2 C.1.1TRAINING PROMPTS FOR EACH STAGE Each training stage uses a task-specific user instruction and multimodal inputs consisting of video frames and audio. The placeholders <frames> and <audio> indicate the positions where visual and acoustic features encoded by CLIP (Radford et al., 2021) and CLAP (Elizalde et al., 2023) are inserted, respectively. The supervision targets differ across stages: Stage 1 uses coarse multimodal descriptions from MERR-Coarse (Cheng et al., 2024a), Stage 2 uses perception-level descriptions from CogEmo-Desc, and Stage 3 uses cognition-level appraisal reasoning from CogEmo-40K. Stage 1: Multimodal Alignment. The user instruction elicits a concise, free-form description of how the subject looks and sounds, providing a lightweight alignment signal for the multimodal projectors. Stage 1: Multimodal Alignment (user instruction) Here is the video: <frames> Here is the audio: <audio> Describe how the person looks and sounds in the video. Stage 2: Perception enhancement. The instruction shifts toward structured, high-density per- ceptual description of the scene, facial behavior, and acoustic characteristics. The target response follows the emotion-free perception format produced by the dataset Stage 1 generation prompt. Stage 2: Perception Enhancement (user instruction) Here is the video: <frames> Here is the audio: <audio> 23 Please briefly describe the video scene, and provide a detailed description of the main sub- ject’s facial details and the speaker’s acoustic characteristics. Stage 3: Cognitive appraisal tuning. The instruction asks the model to perform cognitive ap- praisal reasoning over all available modalities and output a final discrete emotion label. The target response follows the appraisal chain-of-thought format produced by the dataset Stage 2 generation prompt. Stage 3: Cognitive Appraisal Tuning (user instruction) Here is the video: <frames> Here is the audio: <audio> Please infer the most likely emotion category from multimodal cues by reasoning through six cognitive appraisal dimensions: Congruence, Coping, Accountability, Future, Arousal, and Action Tendency. The final emotion should be selected from happiness, sadness, anger, fear, anxiety, disgust, contempt, and neutral. C.2MODEL SCALE AND PERFORMANCE COMPARISON Table 10 compares model scale and performance on CogEmo-Bench. While most multimodal emo- tion LLMs use backbones of at least 7B parameters, the 3.1B CogEmo-MoE achieves the best AEQS, accuracy, and Macro-F1, outperforming the strongest baselines by 14.30, 8.33, and 4.98 points, respectively. This demonstrates that its performance gains arise from effective cognitive modeling rather than increased model scale. Table 10: Performance–Scale Comparison on CogEmo-Bench. ModelParametersAEQSAcc.Macro-F1 Open-source MLLMs OneLLM (Han et al., 2024)7B16.1530.1626.03 SECap (Xu et al., 2024)7B12.2630.8521.25 Qwen2-Audio (Chu et al., 2024)7B25.1243.5532.68 SALMONN (Tang et al., 2024)7B23.8743.6933.43 Otter (Li et al., 2025a)7B10.3236.6124.69 LLaMA-VID (Li et al., 2024)7B26.6036.7131.84 VideoChat-Flash (Li et al., 2025b)7B2.6955.3639.11 Chat-UniVi (Jin et al., 2024)7B32.2834.0329.86 mPLUG-Owl3 (Ye et al., 2025)7B32.0745.5434.72 Qwen3.8-Max (Team, 2026)2.4T49.5057.0446.29 Kimi-K3 (Kimi Team, 2026)2.8T (104B active)48.7046.1339.04 PandaGPT (Su et al., 2023)7B25.5425.7918.78 Emotion-LLaMA (Cheng et al., 2024a)7B20.2743.7525.90 AffectGPT (Lian et al., 2025a)7B32.8342.7631.90 Qwen3-Omni (Xu et al., 2025)30B (3B active)46.1263.1052.19 Closed-source MLLMs GPT-5.2 (OpenAI, 2025)—56.3046.3336.92 GPT-5.6-Luna (OpenAI, 2026b)—48.2751.5943.46 Grok-4.3 (xAI, 2026)—45.5546.8339.79 Gemini-3.6-Flash (Google, 2026)-.36.3059.1345.99 Our Model CogEmo-MoE3.1B70.6071.4357.17 24 DDETAILS OF COGEMO-BENCH CogEmo-Bench is a held-out evaluation set providing over 6,000 human-verified dimension-level appraisal references across six cognitive appraisal dimensions, under the same eight-class emotion space as CogEmo-40K. It is used to assess both cognitive appraisal evidence quality and final emo- tion recognition under a unified protocol. Its label distribution is shown beside the training set in Figure 7. Consistent with that training panel, neutral remains dominant (43.8%), while the remaining seven categories range from 5.5% (anxiety) to 12.3% (sadness). Relative to CogEmo-40K, disgust and fear occupy a larger share of the evaluation split, which helps stress-test appraisal reasoning under less frequent negative emotions. The remainder of this section details the evaluated models, prompting templates, appraisal extractor, and metric computation. D.1MODEL SETUP We evaluate all models reported in Table 2 on the CogEmo-Bench test set. Each model is loaded from its official release and evaluated with the decoding configuration recommended by the original implementation. Table 11 summarizes the model category, supported modalities, and the corre- sponding evaluation-prompt template. For open-source models, we follow the default checkpoints and preprocessing pipelines released by the original authors. For closed-source models, we access the latest available API endpoints at the time of evaluation. CogEmo-MoE is evaluated under three modality settings (A+T, V+T, and A+V+T) to match the ablations in the main benchmark table. General-purpose MLLMs (Gen.) are prompted for cognitive appraisal reasoning with modality- specific templates; emotion-specialized MLLMs (Spec.) follow their original prompting protocols. Table 11: Evaluation setup for models on CogEmo-Bench. Gen. and Spec. denote general- purpose MLLMs and emotion-specialized models, respectively. A, V, and T denote audio, visual, and textual modalities. Evaluation prompt templates are provided in Appendix D.2. ModelTypeAVTPrompt Template Open-source MLLMs OneLLM (Han et al., 2024)Gen.✓ ×✓AT (Appendix D.2) SECap (Xu et al., 2024)Spec.✓ ×✓SECap (Appendix D.2) Qwen2-Audio (Chu et al., 2024)Gen.✓ ×✓AT (Appendix D.2) SALMONN (Tang et al., 2024)Gen.✓ ×✓AT (Appendix D.2) Otter (Li et al., 2025a)Gen. ×✓VT (Appendix D.2) LLaMA-VID (Li et al., 2024)Gen. ×✓VT (Appendix D.2) VideoChat-Flash (Li et al., 2025b)Gen. ×✓VT (Appendix D.2) Chat-UniVi (Jin et al., 2024)Gen. ×✓VT (Appendix D.2) mPLUG-Owl3 (Ye et al., 2025)Gen. ×✓VT (Appendix D.2) Qwen3.8-Max (Team, 2026)Gen. ×✓VT (Appendix D.2) Kimi-K3 (Kimi Team, 2026)Gen. ×✓VT (Appendix D.2) PandaGPT (Su et al., 2023)Gen.✓AVT (Appendix D.2) Emotion-LLaMA (Cheng et al., 2024a) Spec.✓Emotion-LLaMA (Appendix D.2) AffectGPT (Lian et al., 2025a)Spec.✓ AffectGPT (Appendix D.2) Qwen3-Omni (Xu et al., 2025)Gen.✓AVT (Appendix D.2) Closed-source MLLMs GPT-5.2 (OpenAI, 2025)Gen. ×✓VT (Appendix D.2) GPT-5.6-Luna (OpenAI, 2026b)Gen. ×✓ VT (Appendix D.2) Grok-4.3 (xAI, 2026)Gen. ×✓VT (Appendix D.2) Gemini-3.6-Flash (Google, 2026)Gen.✓AVT (Appendix D.2) Our CogEmo Models CogEmo-MoE (A+T)Spec.✓ ×✓ AT (Appendix D.2) CogEmo-MoE (V+T)Spec. ×✓VT (Appendix D.2) CogEmo-MoE (A+V+T)Spec.✓ AVT (Appendix D.2) 25 D.2EVALUATION PROMPT OF DIFFERENT MODELS All models are evaluated on the same CogEmo-Bench instances using the input modalities sup- ported by each model. Specifically, video frames, audio waveforms, and subtitle transcripts are pro- vided whenever the corresponding modality is supported, while unsupported modalities are omitted. General-purpose MLLMs are prompted to perform cognitive appraisal reasoning and produce a final emotion prediction, whereas emotion-specialized models follow their original prompting protocols to preserve their intended inference behavior. Accordingly, we use two types of evaluation prompts: model-specific prompts for emotion- specialized models and modality-specific appraisal prompts for general-purpose MLLMs, as de- tailed below. Emotion-specialized models. AffectGPT (user instruction) Please infer the person’s emotional state and provide your reasoning process. Emotion-LLaMA (user instruction) [reason] Please analyze all clues and reason out the emotional label. SECap (user instruction) Please describe the speaker’s emotional state in one sentence. SECap follows its original protocol of using a concise emotion-description prompt. The original implementation provides multiple Chinese prompt paraphrases with equivalent meaning; we use the single English sentence above for consistency across models. General-purpose MLLMs. Depending on the supported modalities (Table 11), we use one of the following three prompt templates. A+V+T prompt (AVT) Given the multimodal input, analyze the person’s emotional state and explain why the person may feel this way. Ground your reasoning in the available visual, audio, textual, and con- textual evidence. Finally, identify the most likely emotion from: happiness, sadness, anger, fear, anxiety, disgust, contempt, neutral. A+T prompt (AT) Given the multimodal input, analyze the person’s emotional state and explain why the person may feel this way. Ground your reasoning in the available audio, textual, and contextual evidence. Finally, identify the most likely emotion from: happiness, sadness, anger, fear, anxiety, disgust, contempt, neutral. V+T prompt (VT) Given the multimodal input, analyze the person’s emotional state and explain why the person may feel this way. Ground your reasoning in the available visual, textual, and contextual evidence. Finally, identify the most likely emotion from: happiness, sadness, anger, fear, anxiety, disgust, contempt, neutral. Appraisal Extractor. After each model generates a free-form response, we apply a fixed extrac- tion prompt implemented with Qwen3.5-9B (Qwen Team, 2026) to obtain six appraisal-specific evidence units and the final emotion label. The extractor only reorganizes information explicitly present in the original response; it does not add new evidence or modify the model’s reasoning. If a 26 dimension is absent in the response, its extracted evidence is left empty. All evaluated models share the same extractor and prompt to ensure a consistent evaluation protocol. Appraisal Extractor (user instruction) You are an appraisal extractor. Given a model-generated emotion reasoning response, ex- tract the supporting evidence and appraisal judgment for each of the following dimensions if explicitly present: Goal Congruence, Coping, Accountability, Future, Arousal, and Action Tendency. Also extract the final emotion label. Do not add new evidence or modify the original reasoning. If a dimension is missing, leave it empty. D.3METRIC COMPUTATION DETAILS CogEmo-Bench jointly evaluates cognitive appraisal reasoning and final emotion recognition. The former is measured by the Appraisal Evidence Quality Score (AEQS), while the latter is evaluated using Accuracy and Macro-F1. The formal definitions are introduced in Section 4.2; here we provide the corresponding implementation details. Evaluation set. CogEmo-Bench-Eval is a human-verified multimodal evaluation set under the same eight-class emotion space as CogEmo-40K:happiness, sadness, anger, fear, anxiety, disgust, contempt, neutral. Each instance provides human-verified cognitive appraisal references across six dimensions together with a final emotion label. Response normalization. For each instance, the evaluated model first generates a free-form re- sponse using the corresponding evaluation prompt described in Appendix D.2. The response is then processed by the same fixed Appraisal Extractor (Appendix D.2), which reorganizes the explicitly expressed content into six appraisal dimensions and a final emotion prediction. The extractor does not judge correctness or introduce additional evidence. Appraisal information not expressed in the original response is retained as empty. Appraisal Evidence Quality Score. For each sample i and appraisal dimension d, we compare the extracted model evidence ˆe i,d with the corresponding human-verified reference e i,d . Both are en- coded using the same fixed Qwen3-Embedding model, and their semantic similarity is computed by cosine similarity as defined in Eq. equation 5. When only one side contains appraisal evidence, the similarity score is set to zero, penalizing ei- ther missing or unsupported evidence. When both sides are empty, the sample is excluded from the corresponding dimension because no evidence is available for semantic comparison. We first average the valid sample-level similarities within each appraisal dimension and then average the six dimension-level scores to obtain AEQS according to Eq. equation 7. For readability, the final AEQS is multiplied by 100 when reported. Emotion Recognition. For final emotion prediction, we report Accuracy and Macro-F1. Accuracy measures the proportion of correctly predicted emotion labels, while Macro-F1 equally weights the class-wise F1 scores across all emotion categories: Acc. = 1 N X N i=1 I(ˆy i = y i ),Macro-F1 = 1 C X C c=1 2P c R c P c + R c .(8) where ˆy i and y i denote the predicted and ground-truth emotion labels for sample i, respectively; P c and R c denote the precision and recall for class c; N is the number of evaluation samples; and C = 8 is the number of emotion categories. Unified evaluation. All evaluated models share the same Appraisal Extractor, embedding model, emotion label space, and metric computation procedure. This unified pipeline minimizes evaluation differences introduced by heterogeneous response formats and enables consistent comparison of both appraisal reasoning and emotion recognition across models. 27 EADDITIONAL EXPERIMENTAL RESULTS E.1COGNITIVE REASONING–EMOTION ALIGNMENT To examine whether the cognitive appraisal reasoning in CogEmo-40K is aligned with the corre- sponding final emotion, we conduct a 5-fold cross-validation experiment on the intersection subset of CogEmo-40K and MERCaption+ (11,770 samples) (Lian et al., 2025a). Specifically, we use the cognitive reasoning annotations as representations and formulate emotion prediction as an 8- class classification task. This experiment is intended as a diagnostic analysis of the relationship between intermediate cognitive reasoning and the resulting emotion label, rather than an evaluation of a standalone text classifier. As shown in Table 12, CogEmo-40K reasoning representations exhibit strong emotion separability, suggesting that the generated appraisal chains encode emotion-relevant cognitive information that is closely aligned with the final emotion labels. Table 12: 5-fold cross-validation results on the text-reason branch of the CogEmo-40K & MER- Caption+ (Lian et al., 2025a) intersection subset. Accuracy, Weighted-F1, and Macro-F1 are reported in %. The best average is shown in bold. MetricFold 1Fold 2Fold 3Fold 4Fold 5Avg±Std Accuracy92.4492.5791.5992.4492.4092.29±0.35 Weighted-F192.4392.5791.5092.4892.3192.26±0.39 Macro-F191.4491.9990.2791.7291.2391.33±0.59 Training Details. The 5-fold setup follows a fixed, pre-generated split (seed 42, 80%/20% per fold) on the intersection subset described above, ensuring full reproducibility. We list the key train- ing configurations below: 1. Backbone: XLM-RoBERTa-base. 2. Optimizer: AdamW (HuggingFace Trainer default). 3. Initial learning rate: 2× 10 −5 with a linear decay schedule and no warmup. 4. Weight decay: 0.01. 5. Batch size: 32 per device. 6. Epochs: 15. 7. Maximum sequence length: 512. 8. Precision: Full precision (FP32); no gradient accumulation. 9. Hardware: Single NVIDIA L40 (48GB). 10. Data preprocessing: A rule-based label-leakage stripping is applied to remove final ex- plicit emotion-conclusion sentences (e.g., “therefore the emotion is anger”), forcing the model to rely on the actual reasoning content. In addition to the classification results, we perform a feature-space analysis by extracting mean- pooled XLM-RoBERTa embeddings of the reasoning text, applying PCA, and computing the Sil- houette score on 5,000 sampled instances. The resulting cluster separability further supports the quality of the cognitive reasoning annotations. E.2CROSS-DOMAIN GENERALIZATION EXPERIMENTS To evaluate the generalization ability of cognitive appraisal-guided training, we conduct a con- trolled comparison between our automatically generated cognition-grounded supervision and ex- isting emotion description supervision. Specifically, we use the same 115,595 source samples as MERCaption (Lian et al., 2025a) and generate cognitive appraisal annotations under our perception- to-appraisal paradigm without human refinement. In contrast, MERCaption provides automatically generated free-form affective descriptions on the same source set. This setup controls for both data source and data scale, allowing us to more directly examine the effect of the supervision paradigm. All models are trained under the same AffectGPT (Lian et al., 2025a) framework and evaluated following the MERBench protocol (Lian et al., 2025a), covering nine emotion datasets. The corre- sponding zero-shot results are reported in Table 13 for Part I (MER2023–CMU-MOSI) and Table 14 28 for Part I (CMU-MOSEI–OV-MERD+), covering both General Instruction and Emotion Descrip- tion tasks. A detailed description of MERBench is provided in Appendix E.3. Table 13: Cross-domain generalization results on General Instruction and Emotion Description datasets (Part I: MER2023–CMU-MOSI). All models are trained on AffectGPT and evaluated on MERBench. Top: General Instruction models. Bottom: Emotion Description models. CogEmo- 40K is highlighted in blue. DatasetFiltering MER2023 MER2024 MELD IEMOCAP CMU-MOSI MiniGPT4×11.5612.9118.8916.0653.57 MiniGPT4✓17.5716.6522.6030.1852.58 VideoChat×24.8722.4221.5632.9150.13 VideoChat✓27.7024.7327.6639.4645.45 LLaVA×42.2141.5432.9749.9654.48 LLaVA✓41.5642.3032.6146.2152.82 WavCaps×5.757.714.354.9945.59 WavCaps✓23.7226.9723.3927.3054.67 EmoVIT–39.3150.2432.3648.2453.40 MAFW–52.6755.9940.8557.6066.11 MERR-Coarse–35.3436.6029.3736.9465.10 MERR-Fine–69.0072.8447.3854.4966.21 MERCaption–72.1274.2154.6956.4175.10 CogEmo-40K (Ours)–74.0172.6657.3557.5379.15 Table 14: Cross-domain generalization results on General Instruction and Emotion Description datasets (Part I: CMU-MOSEI–OV-MERD+). All models are trained on AffectGPT and evaluated on MERBench. Top: General Instruction models. Bottom: Emotion Description models. CogEmo- 40K is highlighted in blue. Mean is computed across all nine datasets. DatasetFiltering CMU-MOSEI CH-SIMS CH-SIMS v2 OV-MERD+ Mean MiniGPT4×45.9857.6655.1613.8631.74 MiniGPT4✓56.5052.3651.1920.1635.53 VideoChat×56.1750.0751.7124.5637.16 VideoChat✓56.8643.6847.0526.0937.63 LLaVA×56.4252.0454.8035.7546.69 LLaVA✓57.7252.7853.4436.9646.27 WavCaps×22.7653.0445.684.9521.65 WavCaps✓49.5458.1255.9321.5737.91 EmoVIT–61.5369.7266.5338.0951.05 MAFW–62.2775.2070.0242.7558.16 MERR-Coarse–63.2775.1273.7633.1849.85 MERR-Fine–60.0379.9078.5452.5664.55 MERCaption–70.9780.2178.0658.4268.91 CogEmo-40K (Ours)–69.7583.0177.9458.1669.95 Discussion. Compared with the strongest emotion description baseline, MERCaption (Lian et al., 2025a), CogEmo-40K improves performance on most benchmarks, including MER2023 (Lian et al., 2023), MER2024 (Lian et al., 2024), MELD (Poria et al., 2019), IEMOCAP (Busso et al., 2008), CMU-MOSI (Zadeh et al., 2017), CMU-MOSEI (Bagher Zadeh et al., 2018), CH-SIMS (Yu et al., 2020), CH-SIMS v2 (Liu et al., 2022a)and OV-MERD+ (Lian et al., 2025c). Notably, CogEmo-40K improves MER2023 from 72.12 → 74.01 (+1.89 points) and MELD from 54.69 → 57.35 (+2.66 points), demonstrating that structured cognitive appraisal reasoning provides more transferable supervision than free-form affective descriptions. On the challenging CMU-MOSEI dataset, CogEmo-40K reaches 69.75 versus 70.97 for MERCaption, while on CH-SIMS v2 it trails by only 0.12 points. Overall, these results confirm that explicitly modeling the perception-to- appraisal pathway yields stronger cross-domain generalization. 29 E.3MERBENCH EVALUATION PROTOCOL We adopt MERBench (Lian et al., 2025a) to evaluate the cross-domain generalization of different supervision paradigms. As summarized in Table 15, MERBench provides a unified evaluation pro- tocol covering nine representative multimodal emotion datasets and three complementary affective tasks: fine-grained emotion recognition, basic emotion recognition, and sentiment analysis. Table 15: Statistics of the nine datasets in MERBench. MERBench covers fine-grained emotion recognition, basic emotion recognition, and sentiment analysis. TaskDataset# Samples# Classes Fine-grainedOV-MERD+532Unified Basic MER20234116 MER20241,1696 IEMOCAP1,2414 MELD2,6107 Sentiment CMU-MOSI6863 CMU-MOSEI4,6593 CH-SIMS4573 CH-SIMS v21,0343 Fine-grained emotion recognition. This setting evaluates open-vocabulary emotion understand- ing using OV-MERD+ (Lian et al., 2025c), where emotion predictions are not restricted to a fixed closed-set taxonomy. Compared with conventional categorical emotion datasets, OV-MERD+ con- tains more diverse and fine-grained affective expressions, making it suitable for evaluating whether learned representations can generalize to open-ended emotion semantics. Basic emotion recognition. This setting includes MER2023 (Lian et al., 2023), MER2024 (Lian et al., 2024), IEMOCAP (Busso et al., 2008), and MELD (Poria et al., 2019). MER2023 and MER2024 are large-scale Chinese multimodal emotion datasets collected primarily from movies and television programs. IEMOCAP contains acted dyadic interactions and is evaluated under the commonly used four-class setting, while MELD consists of multi-party conversational clips from television dialogues and contains seven basic emotion categories. Sentiment analysis. This setting includes CMU-MOSI (Zadeh et al., 2017), CMU- MOSEI (Bagher Zadeh et al., 2018), CH-SIMS (Yu et al., 2020), and CH-SIMS v2 (Liu et al., 2022a). CMU-MOSI (Zadeh et al., 2017) and CMU-MOSEI contain multimodal opinion videos with sentiment annotations, with CMU-MOSEI providing a substantially larger-scale extension. CH-SIMS and CH-SIMS v2 consist of Chinese multimodal sentiment samples and provide com- plementary evaluation in a different language and cultural context. Following the original MERBench protocol, task-specific metrics are used for the three settings. Fine-grained emotion recognition adopts a set-based evaluation after normalizing semantically equivalent emotion expressions; basic emotion recognition evaluates whether the predicted emo- tion matches the ground-truth basic category; and sentiment analysis reports class-imbalance-aware classification performance after mapping continuous sentiment annotations into discrete polarity cat- egories. Although MERBench is built on conventional emotion recognition and sentiment datasets, the evaluated models produce free-form emotion analyses that are mapped to benchmark label spaces for scoring. Our perception-to-appraisal supervision further structures these analyses through cog- nitive dimensions, allowing us to test whether cognition-grounded emotion understanding transfers to standard downstream benchmarks. All compared models in our cross-domain experiments follow the same MERBench preprocessing, prompting, and metric computation protocol to ensure consistent evaluation. E.4CASE STUDY: QUALITATIVE COMPARISON OF COGNITIVE APPRAISAL REASONING To complement the quantitative results in Table 2, we present a qualitative case study comparing the reasoning trajectories of our CogEmo-MoE with the best-performing baseline (GPT-5.2). We se- lected three samples from CogEmo-Bench that highlight typical failure modes of existing MLLMs: (1) emotion confusion between contempt and anger, (2) surface-level textual misleading, and (3) 30 over-interpretation of neutral contexts. For each sample, we show the ground-truth cognitive ap- praisal reasoning, the output of our model, and the output of GPT-5.2. E.4.1CASE 1: DISTINGUISHING CONTEMPT FROM ANGER (a) Frame 1(b) Frame 2(c) Frame 3 Figure 8: Selected video frames for Case 1 (Contempt vs. Anger). The frames show the controlled downward gaze, slightly downturned mouth corners, and stable head pose characteristic of contempt, as opposed to high-arousal anger cues. Ground Truth Emotion: Contempt Context: A man is criticized as “the weakest link” in a military confrontation. Ground-Truth Cognitive Appraisal: “The dismissive verbal content calling the others merely ‘three hundred defeated soldiers,’ together with the downturned lip corners and steady off-camera address, suggests a de- valuing, unfavorable evaluation of the target; meanwhile, his clear articulation, moderate voice intensity, stable gaze, and relaxed jaw imply he feels able to handle the situation and speak from a position of control; furthermore, because the criticism is directed outward at the off-camera group through the insulting label ‘defeated soldiers,’ the cause of concern is assigned to others rather than himself. Overall, the pattern is one of negative evalua- tion, confident control, blame toward others, present-focused engagement, and restrained activation, therefore the most likely emotion is contempt.” Our Model (CogEmo-MoE) Output: “The man’s steady slight downward gaze, mildly downturned mouth corners, and the crit- ical verbal content (‘you are the weakest link’) convey an unfavorable evaluation of the target; meanwhile, his controlled posture, clear articulation, and stable low-mobility face with moderate vocal intensity suggest he remains composed enough to address the issue di- rectly rather than being overwhelmed; furthermore, because the utterance explicitly points to ‘you’ as the source of the problem and his facial expression stays directed outward rather than self-reproachful, responsibility is assigned externally. Overall, the pattern is one of controlled negative evaluation with external blame, present-focused criticism, low arousal, and direct engagement, therefore the most likely emotion is contempt.” Prediction: Contempt✓ Baseline Model (GPT-5.2) Output: “Across the sampled frames, the person appears tense and increasingly confrontational. His brows are drawn and his gaze is fixed off to the side as if reacting to someone nearby; his mouth stays slightly open as though speaking defensively. The final frame shows a sharp escalation: eyes widened, jaw set, and teeth visible in what looks like a raised voice or shouted remark—classic cues of agitation and aggression... Together, the facial tension, sudden wide-eyed glare, and the insulting/critical text point to an angry response rather than sadness or fear.” Prediction: Anger× (Ground Truth: Contempt) Analysis: The baseline model interprets the raised voice and confrontational facial cues as anger, focusing on observable affective signals of agitation. In contrast, our model explicitly reasons about Accountability (“responsibility is assigned externally”) and Coping (“remains composed enough to address the issue directly”), which are the key appraisal dimensions distinguishing contempt (characterized by moral superiority and external blame with low arousal) from anger (characterized by high arousal and goal obstruction). This demonstrates that cognitive appraisal reasoning enables our model to differentiate emotionally similar but cognitively distinct negative states. 31 E.4.2CASE 2: OVERCOMING SURFACE-LEVEL TEXTUAL MISLEADING (a) Frame 1(b) Frame 2(c) Frame 3 Figure 9: Selected video frames for Case 2 (Fear vs. Sadness). The frames illustrate the tightened jaw, narrowed eyes, and tense mouth while recounting being comforted, indicating a lingering threat appraisal rather than past-oriented sadness. Ground Truth Emotion: Fear Context: A woman testifies in court, recounting that someone comforted her. Ground-Truth Cognitive Appraisal: “Her tightened jaw, narrowed eyes, raised yet unstable brows, and downturned but tense mouth while recounting that she felt okay because another person was there in a for- mal hearing-like setting suggest an unpleasant and threatening evaluation of the situation; meanwhile, although she continues speaking clearly and remains physically composed, the growing facial tension and reliance in her words on another person’s reassurance imply only limited personal control over the threatening situation; furthermore, because her state- ment frames safety as depending on someone else’s presence and comfort rather than her own action, the source of regulation is centered outside herself... Overall, the pattern is of negatively evaluated threat, limited control, externally centered responsibility, anticipation of danger, high activation, and a withdrawal-oriented response, therefore the emotion is fear.” Our Model (CogEmo-MoE) Output: “The speaker’s statement that someone comforted her, together with the otherwise subdued and serious interaction, points to a negatively toned situation rather than a pleasant one; meanwhile, the relaxed face, steady gaze, moderate voice, and lack of forceful movement suggest the person is not actively exerting strong control over the situation; furthermore, the spoken content centers on another person’s comforting behavior as the relevant cause, which places causation outside the speaker... Overall, the pattern is of a negative, externally caused, ongoing situation with limited coping, low arousal, and withdrawal, therefore the emotion is fear.” Prediction: Fear✓ Baseline Model (GPT-5.2) Output: “The woman appears emotionally distressed and vulnerable. Her furrowed brows, raised inner eyebrows, tense eyes, pursed/downturned lips, and lowered gaze suggest worry and lingering sadness rather than happiness or neutrality. The subtitles indicate that Mu Zi comforted her and that she felt things would be okay because Mu Zi was there, implying she had previously been upset or anxious and is now only partially reassured. Her expres- sion remains strained while speaking, suggesting emotional pain and concern beneath the relief... Most likely emotion: sadness.” Prediction: Sadness× (Ground Truth: Fear) Analysis: The baseline model appears to rely on the textual cues “comforted” and “felt better” and predicts sadness, interpreting the situation as grief or loss. Our model instead describes an ongoing situation and limited coping, which are more consistent with the fear label in this example. The case illustrates how appraisal-based descriptions can help distinguish interpretations when the textual cues are ambiguous. 32 E.4.3CASE 3: CORRECTLY IDENTIFYING NEUTRAL AFFECT (a) Frame 1(b) Frame 2(c) Frame 3 Figure 10: Selected video frames for Case 3 (Neutral vs. Anxiety). The woman’s relaxed facial expression, stable gaze, and slight smile indicate low arousal and adequate coping, supporting a neutral appraisal rather than anxiety. Ground Truth Emotion: Neutral Context: A woman politely apologizes and explains that someone has offended a noble. Ground-Truth Cognitive Appraisal: “The woman’s polite wording asking for an audience, along with her relaxed face, slight mouth upturn, and steady moderate speech toward the man, suggests the interaction is not strongly adverse and may even be socially formal; her clear articulation, stable head pose, direct gaze, and controlled vocal delivery imply she feels able to manage the situation and speak deliberately; meanwhile, because she explicitly says she is sorry and that Wang Shang has offended Mi Yuan, the source of the issue is framed as lying with external parties rather than herself... Overall, the pattern is mildly positive, controlled, externally attributed, present-focused, low-arousal, and socially engaging, therefore the most likely emotion is neutral.” Our Model (CogEmo-MoE) Output: “The woman’s polite wording asking for an audience, along with her relaxed face, slight mouth upturn, and steady moderate speech toward the man, suggests the interaction is not strongly adverse and may even be socially formal; her clear articulation, stable head pose, direct gaze, and controlled vocal delivery imply she feels able to manage the situation... Overall, the pattern is mildly positive, controlled, externally attributed, present-focused, low-arousal, and socially engaging, therefore the most likely emotion is neutral.” Prediction: Neutral✓ Baseline Model (GPT-5.2) Output: “The woman appears emotionally distressed and concerned. Her raised, knitted eyebrows, slightly widened eyes, tense mouth, and pleading gaze toward the man suggest worry about the interpersonal conflict. The text indicates that Wang Shang has offended Mi Yuan, and she apologizes (‘Sorry to bother you’), implying she may feel anxious about the conse- quences... Most likely emotion: anxiety.” Prediction: Anxiety× (Ground Truth: Neutral) Analysis: The baseline model over-interprets the apologetic and formal social context as emo- tional distress, projecting anxiety onto a neutral interaction. Our model correctly recognizes that the woman’s Coping (“able to manage the situation”) and Arousal (low activation, “relaxed face”) do not support an anxiety interpretation. This case illustrates how appraisal reasoning may help the model avoid assigning anxiety to a neutral conversational exchange. E.4.4SUMMARY OF CASE STUDY Across these three representative cases, we observe consistent patterns in how our model outper- forms the baseline: 33 Table 16: Appraisal dimensions correctly identified by CogEmo-MoE in each case study. DimensionCase 1Case 2Case 3 (Contempt vs. Anger)(Fear vs. Sadness)(Neutral vs. Anxiety) Goal CongruenceNegative evaluationNegative threatNot strongly adverse Coping PotentialHigh (controlled)Low (limited)High (able) AccountabilityExternal blameExternal causeExternal parties Future ExpectancyPresent-focusedAnticipatory threatPresent-focused ArousalLow (restrained)High (activation)Low (relaxed) Action TendencyConfrontationalWithdrawalSocially engaging These qualitative examples illustrate that cognitive appraisal reasoning provides critical discrimina- tive information that is not accessible through surface-level affective cue matching alone. By explic- itly modeling how individuals evaluate situations along dimensions such as accountability, coping potential, and future expectancy, our model achieves more reliable emotion understanding, partic- ularly in cases involving subtle emotional distinctions, misleading textual content, or contextually appropriate neutral affect. We acknowledge that these three cases are qualitatively selected for illustrative purposes; the statis- tical superiority of our model is rigorously validated in the main experiments (Table 2). 34