Paper deep dive

Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenes

Hao Li, Jinye Zhang, Bobo Li, Mong-Li Lee, Wynne Hsu, Zheng Wang, Hao Fei, Min Zhang

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 151

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 10/10/2026, 1:29:26 AM

Summary

The paper introduces TRACE, a cognition-oriented framework for analyzing affective episodes in real-world social scenes through three stages: Condition, Affect, and Effect. It presents TRACE-Bench, a multimodal benchmark with 3,746 QA pairs over 646 videos, evaluating five tasks from grounded affect recognition to full-chain reconstruction. The study reveals a significant performance gap between human and model capabilities, particularly in distinguishing displayed behavior from genuine feeling and avoiding factual drift. To address these issues, the authors propose TRACER, a structured reasoning method that grounds inferences in explicit factual observations and cognitive appraisals, outperforming existing baselines.

Entities (10)

TRACE · framework · 98%TRACE-Bench · benchmark · 98%TRACER · method · 97%Affective Blueprint · concept · 95%Cognitive Appraisal Theory · theory · 92%EASI · theory · 92%Emotion Regulation Theory · theory · 92%Multimodal Large Language Models · technology · 90%IEMOCAP · dataset · 85%MELD · dataset · 85%

Relation Signals (9)

TRACE-Bench → contains → 3,746 structured question-answer pairs

confidence 95% · TRACE-Bench... covering 3,746 structured question-answer pairs over 646 videos.

TRACE → defines → Affective Blueprint

confidence 95% · We introduce an Affective Blueprint that represents an affective episode through three causally connected stages: Condition→Affect→Effect.

TRACE-Bench → evaluates → Multimodal Large Language Models

confidence 93% · TRACE-Bench evaluates multimodal models in real-world social scenes through five tasks

TRACE-Bench → identifies → surface reading

confidence 90% · The first is surface reading: a model reads affect from observable behavior without considering what the situation means to the person involved.

TRACE-Bench → identifies → factual drift

confidence 90% · The second is factual drift: when reconstructing an affective chain, a model treats interpretations it forms during reasoning... as established facts

TRACER → outperforms → Multimodal Large Language Models

confidence 90% · Experiments show that TRACER outperforms all evaluated model baselines on each of the five tasks.

Emotion Regulation Theory → underpins → TRACE

confidence 88% · Emotion Regulation Theory... Emotion regulation can modify affective experience and expression... modeling an affective episode therefore requires recovering... strategic regulation of display

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Affective computing has progressed from categorical emotion recognition to open-ended affective analysis with large multimodal models. Yet affective science describes emotion as an unfolding process shaped by appraisal, regulation, and social interpretation, which remains underexplored computationally. We propose TRACE, a cognition-oriented framework that formalizes an affective episode through three interrelated stages: Condition, Affect, and Effect, integrating observable cues with cognitive factors such as internal stance and regulation of emotional display. Based on this formulation, TRACE-Bench evaluates multimodal models in real-world social scenes through five tasks spanning grounded affect recognition, regulation decoding, cause reasoning, effect reasoning, and full-chain reconstruction, with 3,746 structured question-answer pairs over 646 videos. A matched human-model comparison reveals a substantial performance gap, while affect-specialized models also generally lag behind general-purpose MLLMs. Model outputs show recurring failures, including treating displayed behavior as genuine feeling and fabricating unsupported events during long-chain generation. We further propose TRACER, a cognition-grounded structured reasoning method that couples each inference with explicit premises from factual observations, cognitive appraisals, and established upstream conclusions, forming a traceable graph of intermediate and target conclusions. TRACER outperforms all evaluated model baselines on each of the five tasks. Project page: this https URL

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

147,723 characters extracted from source content.

Expand or collapse full text

LI et al.: COGNITION-ORIENTED EMOTION TRACING1 Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenes Hao Li, Jinye Zhang, Bobo Li, Mong-Li Lee, Wynne Hsu, Zheng Wang, Senior Member, IEEE , Hao Fei * , Senior Member, IEEE , and Min Zhang, Senior Member, IEEE Abstract—Affective computing has progressed from traditional categorical emotion recognition with small specialized models to open-ended affective analysis with large models, lifting recognition performance to unprecedented levels. However, theories in affective science describe emotion as an unfolding process involving causally interacting components, with appraisal, regulation, and social interpretation shaping its emergence, expression, and consequences. Computational modeling and evaluation of this full process remain largely unexplored in the field. To this end, this paper proposes the TRACE framework, which formalizes an affective episode through three interrelated stages, Condition, Affect, and Effect, incorporating both observable cues and cognitive factors such as the subject’s internal stance toward the situation and strategic regulation of emotional display. Based on this formalization, we present TRACE-Bench, a benchmark for evaluating multimodal models on affective understanding in real-world social scenes, which comprises five tasks with widening structural spans, from grounded affect recognition, regulation decoding, and cause reasoning to effect reasoning and full-chain reconstruction, covering 3,746 structured question-answer pairs over 646 videos. A matched human–model comparison reveals a substantial gap to human performance. Across the full benchmark, the evaluated affect-specialized models also generally lag behind general-purpose MLLMs. Analysis of model outputs identifies recurring failure patterns, including treating displayed behavior as genuine feeling and fabricating plausible but unsupported events during long-chain generation. Motivated by these findings, we propose TRACER, a cognition-grounded structured reasoning method. Each derivation step couples a natural-language inference with explicit premise citations drawn from factual observations, cognitive appraisals, and established upstream conclusions. The linked steps form a traceable graph of intermediate and target conclusions. Experiments show that TRACER outperforms all evaluated model baselines on each of the five tasks. Project page: https://cogaffc.github.io/TRACE. Index Terms—affective computing, cognitive-affective reasoning, multimodal large language models, benchmark ✦ 1 INTRODUCTION A FFECTIVE computing has progressively expanded its analytical scope, evolving from the isolated recognition of surface signals to the contextual interpretation of human affect [1]. Early paradigms formalized emotion as a closed classification problem, relying on small specialized models to predict categorical states or continuous dimensions from facial, acoustic, and linguistic cues [2], [3], [4]. As the field advanced, this scope broadened to incorporate conversa- tional context [5], [6] and explicit causal triggers [7], [8]. More recently, the advent of multimodal large language models has propelled affective analysis beyond rigid la- bels, enabling open-ended emotion descriptions [9], [10], causal explanations [11], [12], and generalized affective reasoning [13], [14], [15], lifting recognition performance to unprecedented levels. Nevertheless, despite this genera- tive shift, the prevailing computational paradigm remains fundamentally observational: it maps perceptible signals to affective outcomes or provides descriptive accounts of emotional events. As surface-level recognition approaches •Hao Li, Jinye Zhang, and Zheng Wang are with the School of Computer Science, National Engineering Research Center for Multimedia Software, Wuhan University, Wuhan 430072, China. •Bobo Li is with the National University of Singapore, Singapore. •Mong-Li Lee and Wynne Hsu are with the School of Computing, National University of Singapore, Singapore. •Hao Fei is with the Department of Computer Science, University of Oxford, Wolfson Building, Parks Road, Oxford OX1 3QD, U.K. •Min Zhang is with the Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen), Shenzhen, China. * Corresponding author: Hao Fei (haofei7419@gmail.com). human performance [16], the ceiling of this paradigm be- comes apparent. By treating affect as a direct readout of observable behavior, it leaves cognitive mechanisms such as subjective appraisal, internal drivers, and strategic regulation largely unexplored. In cognitive psychology and affective science [17], [18], [19], affect has been studied not as an isolated output signal but as a dynamic process with complex cognitive involvement. Appraisal theories offer a causal perspective on emotion elicitation: events elicit different emotional responses depending on how they relate to a person’s beliefs, desires, expectations, and coping potential [20], [21]. The resulting affect may then be strategically managed before it becomes publicly visible [22], and its display may further change what other people think, feel, and do [23]. In other words, an affective episode can be understood as a chain shaped by cognition: antecedent→emotion→ consequence. Capturing this full chain, rather than a single link on it, is a precondition for the next generation of affective applications: an empathetic dialogue agent needs to know where an emotion comes from in order to respond appropriately, a social robot needs to anticipate what its expression will bring about, and a mental-health support system needs to distinguish what a person displays from what the person feels. Computational research has started to adopt this cognitive view: textual commonsense studies infer the intents, reactions, and likely consequences behind events [24], [25], [26], and emotional-intelligence and Theory- of-Mind oriented benchmarks bring mental-state inference This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. arXiv:2610.11410v1 [cs.AI] 8 Oct 2026 LI et al.: COGNITION-ORIENTED EMOTION TRACING2 So, those gift cards, they work in here? Question:Whydoestheemployeefeelsurpriseandanxiety? Answer:Becausethecustomersayshewillstealtheshoes. Question:Whatemotiondoestheemployeeappeartoexpress? Answer:Surprise (Leaves the store with the shoes) Wait,what?Whatareyou talkingabout? Theygotano-chasepolicy. Theycan'tstopme... Sir, those shoes. Can I get you a different size? (3)Anaffectiveepisodefromacausalandcognitiveperspective: Idon'tknow.I'mjustgoing totakethis. Internal driver: He believed it was a routine purchase. External reality: The customer says he will steal the shoes. What caused his affect? (T3) Object: Customer Category: Surprise, Anxiety Polarity: Negative Intensity: Medium What is the employee's affect?(T1) Tactic: Substitution Goal: Stay professional and non-confrontational. Target state: Happiness How did he regulate it? (T2) Mental effect:His no-pursuit belief is reinforced. Affective effect:Feels more confident. Physical effect: Leaves the store with the shoebox. What’s the effect on the customer?(T4) How does the employee’s complete affective episode unfold? (T5) (1)Existingemotionrecognitiontask: (2)Existingemotion-causeunderstandingtask: Fig. 1. Existing affective benchmarks often evaluate local targets such as affective states or causes. TRACE-Bench evaluates the full cognitive-affective chain: from the conditions that give rise to an emotion, through the regulation that reshapes its expression, to the consequences it produces for the subject or other participants. into evaluation [27], [28], [29], [30]. Yet each of these efforts covers only one section of the chain and remains scattered across modalities and task formats: the antecedent of affect is often reduced to an external trigger, omitting the internal stance that makes it matter; the regulation between felt and displayed affect is rarely modeled; and the downstream consequences of affect on others lack systematic evaluation. The result is a set of fragmented computational treatments that lack a unified definition of affective episodes from a formal cognitive perspective. To address this gap, we introduce TRACE, short for Trac- ing Emotions from Causes to Consequences, to study cognitive- affective processes in real-world social scenes with complex interpersonal dynamics. TRACE asks how a subject-centered affective episode can be formally represented, systematically evaluated, and computationally reconstructed from multi- modal social interactions. We begin by defining the affective episode itself. Affective Blueprint. We introduce an Affective Blueprint that represents an affective episode through three causally connected stages: Condition→Affect→Effect. Condition captures the antecedent of affect, including the external reality and the subject’s internal driver that together make the situation emotionally significant. Affect records the felt state and its observable manifestation, along with the regulatory process by which the subject may reshape what is displayed, for example suppressing sadness to maintain composure. Effect records the downstream consequences, mental, affective, and physical, that the episode produces for the subject or other participants. Each transition in this chain is shaped by cognition, whether through the subject’s interpretation of the triggering event, strategic regulation of display, or the observer’s inference from what is shown. Section 3.1 presents the theoretical grounding, and Section 3.2 gives the full formal definition. TRACE-Bench. Based on this representation, TRACE-Bench formulates affective process understanding as five comple- mentary, diagnostic tasks whose structural span widens along the chain, from a single stage, to cross-stage causal relations, to the complete episode.1Grounded Affect Recognition evaluates whether a predicted affective state is supported by the subject’s concrete manifestations. 2 Reg- ulation Decoding evaluates whether a model can recover the cognitive regulation that mediates between the subject’s felt state and outward manifestation, including why and how the displayed affect is reshaped.3Affective Cause Reasoning and 4 Affective Effect Reasoning trace an observed affect backward to its external reality and internal driver, and forward to its mental, affective, and physical consequences. 5Full Chain Reconstruction jointly recovers Condition, Affect, and Effect. It evaluates whether the reconstructed episode is supported by the observed evidence and remains cognitively and causally correct. The five tasks share the same Blueprint and annotation framework, allowing different reasoning capabilities to be evaluated from a unified process perspective. Under this task design, we carefully curate and annotate 3,746 structured question-answer pairs from 646 video clips, spanning diverse social scenes, interpersonal relations, and configurations of the Affective Blueprint. TRACER. Evaluation on TRACE-Bench exposes two recur- ring failure patterns that motivate our method. The first is surface reading: a model reads affect from observable behavior without considering what the situation means to the person involved. In Fig. 1, the employee maintains a friendly manner when the customer announces the theft. Taken in isolation, his manner suggests a pleasant exchange. From his perspective, however, the announcement threatens his job, and maintaining his composure is a way of coping with that threat. Such a cognitive interpretation is needed LI et al.: COGNITION-ORIENTED EMOTION TRACING3 to understand what gives rise to his affective state, what he actually feels, and why he regulates its expression, yet current models mostly remain at the level of observable behavior. The second is factual drift: when reconstructing an affective chain, a model treats interpretations it forms during reasoning, even when incorrect, as established facts and builds further conclusions on them, introducing plausible events that are not grounded in facts observable in the video. To address these failures, we propose TRACER, which structures reasoning around explicit factual observations and cognitive analysis of what those observations mean to the characters involved. Specifically, we first establish what occurs in the video, then analyze its significance from each character’s perspective along four cognitive appraisal dimensions: relevance, implication, coping potential, and normative significance. With this factual and cognitive basis, we derive the requested constructs through a task-specific reasoning topology grounded in the Blueprint. The topology specifies the dependencies supporting each inference, which the model makes explicit by citing the relevant observa- tions, appraisal readings, or established conclusions as premises. Evaluation on TRACE-Bench shows that TRACER outperforms all evaluated model baselines on each of the five tasks. Diagnostic experiments show reduced factual fabrication under TRACER and improved State attribution with supplied appraisal records. This work makes three contributions. •We introduce the Affective Blueprint to formalize affective episodes in complex social scenes as three interrelated stages, Condition, Affect, and Effect, with cognition shap- ing the relations among them. •We introduce TRACE-Bench, a multimodal benchmark with 3,746 structured QA pairs over 646 videos and five tasks that evaluate affective chain understanding at widening structural spans, from single-stage analysis through cross-stage reasoning to full-chain reconstruction. •We propose TRACER, a Blueprint-guided structured rea- soning method that grounds affective inference in factual observations and character-specific cognitive appraisals. 2 RELATED WORK 2.1 Affective Recognition and Extraction Affective computing has long relied on benchmarks that make affect measurable through localized targets. Repre- sentative datasets such as IEMOCAP [2], MELD [5], and CMU-MOSI [4] standardize categorical or sentiment-oriented prediction from speech, facial behavior, and language. Di- mensional settings further evaluate valence and arousal signals [31], as in the ABAW series [3]. EMOTIC [32] combines person and scene context to recognize categorical and dimensional emotions in images. Recent multimodal resources extend the input space to richer and larger settings, including face-audio integration [33] and privacy-aware de-identified observation [34]. MERBench [35] provides a unified evaluation of feature selection, multimodal fusion, and robustness for emotion recognition. These benchmarks are important because they establish whether a model can recognize affective evidence, yet their evaluation target is still centered on the current state, polarity, or expressed signal. A related line moves from recognition to extraction, asking which part of the context explains an affective state. Emotion- Cause Pair Extraction [7], RECCON [8], MECPE [36], and SemEval-2024 Task 3 [37] recover the utterance, event, or pair associated with an emotion in dialogue. ECEM [11] adds free-form causal explanations, and MTMEUR [14] expands the format to progressive question answering over affective cues and triggering factors. Together, recognition and extraction benchmarks make affective evaluation con- crete: a model predicts a state, identifies a cue, or selects an explanatory fragment. The resulting targets are useful and measurable, but they remain local with respect to the affective episode. They do not jointly evaluate the external situation, the subject’s internal driver, the relation between felt state and manifestation, and the effects that follow in the surrounding interaction. 2.2 Generative Affective Understanding The rise of MLLMs has shifted affective evaluation from closed-form prediction toward open-ended output. Recent surveys of affective computing in the LLM era note that heterogeneous affective understanding and generation tasks are increasingly cast as instruction-following or sequence- generation problems [16]. This matters for benchmark design because the expected output is no longer limited to a class label; it may also be an explanation, caption, judgment, or response. EMER [9] makes this shift explicit by asking models not only to predict emotions, but also to provide the evidence and reasoning behind the prediction, thereby using explanations to reduce label ambiguity and support more fine-grained affective labels. AffectGPT [10] further develops this direction with descriptive emotion annotations, MER-Caption, and MER-UniBench, aligning multimodal emotion understanding with the free-form output style of MLLMs. Emotion-LLaMA [13] focuses on instruction- tuned multimodal emotion recognition and reasoning with audio, visual, and textual inputs, while EmoLLMs [38] studies instruction-following LLMs for broader affective analysis across classification and regression tasks. In parallel, emotional intelligence benchmarks such as EmoBench-M [28] and MME-Emotion [15] broaden evaluation from single tasks to multi-scenario understanding and reasoning, including the causes of an emotion. Their tasks are nevertheless scored independently, and none represents how a felt state is regulated in display or what it brings about in others. TRACE-Bench instead organizes its tasks around one subject- centered episode, so that Condition, State, Manifestation, Regulation, and Effect are evaluated as components of one affective episode rather than as separate abilities. 2.3 Cognitive and Social Reasoning A third line of work studies the social and cognitive reason- ing needed to interpret human behavior beyond surface sig- nals. Textual commonsense work, including Event2Mind [24], ATOMIC [25], and CICERO [26], studies intent, reaction, motivation, and likely subsequent events. In video and mul- timodal settings, Social-IQ [39] evaluates question answering for artificial social intelligence, MovieGraphs [40] represents movie clips with graphs of characters, relationships, inter- actions, attributes, and motivations, and DramaQA [41] LI et al.: COGNITION-ORIENTED EMOTION TRACING4 uses character-centered hierarchical QA to evaluate video story understanding. Multimodal intent benchmarks such as MIntRec2.0 [42] further examine how language, vision, and audio support intent recognition in multi-party con- versations. Theory-of-mind benchmarks make the latent cognitive target more explicit: MMToM-QA [43] evaluates belief and goal inference in household activities, MuMA- ToM [44] extends the setting to multi-agent interactions and beliefs about others’ goals, and MoMentS [45] evaluates ToM abilities in realistic narrative videos. EmoBench [27] and ToMBench [46] likewise place affective or social understand- ing within broader abilities such as emotional intelligence, intent attribution, and mental-state inference. These bench- marks clarify why affective computing in complex social scenes cannot be reduced to recognizing visible expressions. Their primary targets include intentions, beliefs, relations, story states, and general social judgments. TRACE-Bench centers these capabilities on a subject-centered affective episode, evaluating whether the triggering condition, the felt affective state, its manifestation, possible regulation, and the resulting mental, affective, and physical effects are mutually consistent under the Affective Blueprint. 3 PRELIMINARIES This section draws on established theories from cognitive science and social psychology to analyze the internal com- position of an affective process, and formalizes this analysis into a representation with concrete stages and attributes. Sec- tion 3.1 provides the theoretical grounding, and Section 3.2 translates it into the Affective Blueprint. 3.1 Theoretical Grounding THEORY 1: Cognitive Appraisal Theory [17], [18], [47] A situation acquires affective significance through appraisal of the relation between the external situation and the subject’s internal stance. Affective responses depend on both what happens and the psychological state from which the subject encounters it. Appraisal theory explains this dependency: the significance of an event emerges through its evaluation in relation to the subject’s current beliefs, goals, concerns, and coping possibilities. The same external event can therefore produce different affective responses when subjects bring different internal states to it. As illustrated in Fig. 2, a poor exam result may evoke sadness for a student who treats it as evidence of personal failure, while eliciting determination when it is understood as an opportunity for growth. Modeling an affective episode therefore requires recovering both the external situation and the subject-specific internal conditions. THEORY 2: Emotion Regulation Theory [22], [48] Emotion regulation can modify affective experience and expression, allowing outward manifestation to diverge from the felt state. Even after an emotion has emerged, its outward manifes- tation may differ from what the subject feels. The mapping Belief: This exam will define my worth ! Belief: It will be a chance to grow Reality: bad results Emotion: Sadness Intensity: High He’s a rival, I won’t show my weakness ! Her smile is not true ① ② ③ ① Regulation: Substitution Target Emotion: Happiness Manifestation: Forced smile Mental: She is not really happy Affective: I’m worry about her Physical: I comfort her it’s OK Emotion: Sadness Intensity: High Emotion: Determination Manifestation: Upright posture Fig. 2. Theoretical grounding of affect as a causally structured process. The example illustrates how internal stance shapes affective meaning, how regulation can separate felt affect from display, and how displayed affect can influence others. from felt affect to observable expression can be shaped by emotion regulation. A subject may suppress, amplify, sub- stitute, or fabricate an affective display in response to social goals, relationship concerns, or situational expectations [49], [50]. The lower-left part of Fig. 2 illustrates this relation. Although the student remains sad, the presence of a rival motivates her to conceal the sadness and display a forced smile. This illustrates that an observable manifestation cannot always be interpreted as a direct readout of the underlying affective state. Therefore, understanding an affective process requires considering whether and how emotion regulation shapes the transition from what is felt to what is shown. THEORY 3: Emotions as Social Information (EASI) [23] Emotions provide social information that can influence what others understand, feel, and do during interaction. Affective experiences extend beyond individual expres- sion and can shape the subsequent course of social interac- tion. The Emotions as Social Information (EASI) framework explains that a subject’s affective response provides socially relevant information that can influence how others under- stand the person and situation, how they feel in response, and how they subsequently behave [23], [51]. Observers interpret available affective cues together with the situational context, updating their beliefs and forming affective responses that can further guide their behavior toward the subject [52]. Affect can therefore propagate across individuals, producing cognitive, affective, and behavioral changes in others. The lower-right part of Fig. 2 illustrates this interpersonal dynamic. In the example, another student forms the belief that she is concealing sadness (mental effect), feels concern for her (affective effect), and steps forward to offer comfort (physical effect). Therefore, understanding an affective process requires tracing how a subject’s affective response gives rise to downstream mental, affective, and physical consequences in others. Together, the three theories each anchor one span of the chain: appraisal theory explains how affect is triggered LI et al.: COGNITION-ORIENTED EMOTION TRACING5 IntentionDesireBelief Mental Condition ① Condition FacialBodily VerbalVocal Manifestation ② Affect ③ Effect Triggering Cause Internal External Event CharactersEnvironment Physical Condition Mental EffectAffective EffectPhysical Effect Resulting First-order Tactic Regulation Source State Target State Object Category （Coarse&Fine） Polarity State Intensity Second-order Subject T a s k 3 T a s k 4 T a s k 5 T a s k 1 T a s k 2 T a s k 1 Goal Fig. 3. The Affective Blueprint. A subject-centered affective episode is represented through three causally connected stages: Condition, Affect, and Effect. Each stage is decomposed into several components, each represented by a set of fields. (Condition), regulation theory explains how a feeling reaches its public form (Affect), and EASI explains how a display acts on others (Effect). They jointly establish the cognitive and causal account of affect that we formalize below. 3.2 The Affective Blueprint Building on these theoretical foundations, we construct the Affective BlueprintB, our formal representation of a subject- centered affective episode in real-world social interaction. The Blueprint includes three aspects that our formulation treats as jointly necessary: the conditions through which a situation acquires affective significance, the affective response together with its regulation and manifestation, and the downstream consequences that unfold in the interaction. We organize them into three causally connected stages, ConditionC, AffectA, and EffectE : B =⟨C, A, E⟩.(1) Fig. 3 shows the resulting structure. Stage I: Condition. Given a Subject (C sub ) whose affective process is being analyzed, the Condition stage provides the broader context from which the subject’s affective response emerges. Physical Condition describes the external situation through its environment, characters, and events, while Mental Condition captures the subject’s beliefs, desires, and intentions [53]. For a specific affective episode, we further identify the antecedents most directly relevant to the response as its affective cause. Specifically, External Reality (C ext ) captures the external circumstance implicated in the response, while Internal Driver (C int ) captures the subject- specific internal stance that gives this circumstance affective significance. We represent the affective cause as C cau = (C ext ,C sub ,C int ). Stage I: Affect. The Affect stage comprises two first-order components, Affective State and Manifestation, together with a second-order analysis of their relation, Regulation. Affective State (A sta ) records what the subject feels through four fields: object, category, polarity, and intensity. Object specifies the person toward whom the affect is directed. Category identifies the affect at coarse and fine-grained levels, while polarity and intensity characterize its valence and strength. Manifestation (A man ) records the outward expressions corresponding toA sta across facial, bodily, ver- bal, and vocal channels. Facial and bodily channels capture visible expressions and movements, while verbal content captures what the subject says and vocal cues characterize the utterance’s acoustic and prosodic delivery. Regulation (A reg ) provides a second-order analysis of the relation betweenA sta andA man . It records the regulation tactic, its goal, and the source and target affective states. The source statescorresponds to the subject’s felt affective stateA sta , while the target statetdenotes the affective state intended for outward display throughA man . In Fig. 2, for example, the student remains sad but displays a forced smile. Sadness is the source state, the smile is the manifestation, and happiness is the target state. Under Suppression, the target retains the source category even when its outward expression becomes barely perceptible. The tactic characterizes the relation between the source and target states. Lets c andt c denote their respective categories, and letNdenote Neutral. For a shared non-neutral category, r i ∈ lower, comparable, higherdescribes the intensity conveyed by the manifestation relative to the felt intensity. The five tactics are defined as follows: tactic(s, t, r i ) =                      None,s c = t c = N, or s c = t c ̸= N, r i = comparable, Suppression,s c = t c ̸= N, r i = lower, Amplification, s c = t c ̸= N, r i = higher, Substitution,s c ̸= N, t c /∈s c , N, Fabrication,s c = N, t c ̸= N. (2) For a multi-labelA sta ,s c denotes the particular affective category involved in the regulation. We record the target category and encode its intensity relative to the source through the regulation tactic, rather than assigning a separate target-intensity value. The Affect stage is thus represented as A = (A sta ,A man ,A reg ). Stage I: Effect. The Effect stage records the downstream consequences of the Subject’s affective response within the interaction. We distinguish three channels. Mental Effect (E men ) records what a recipient comes to believe, desire, or intend. Affective Effect (E aff ) records the affect experi- enced by the recipient as a consequence. Physical Effect (E phy ) records the recipient’s resulting action or behavioral response. Each channel is associated with its own recipient, so different effects within the same affective episode may involve different individuals. The recipient may also be the LI et al.: COGNITION-ORIENTED EMOTION TRACING6 TABLE 1 Field-level schema of the Affective Blueprint. Each row specifies one field and its admissible value space. Effect fields describe the impact of the subject’s affect. Effect recipients are identified within the corresponding effect fields. ComponentFieldDefinitionValue space I CONDITION What triggers an affect? Cause (C cau ) subjectFocal person in the affective episodeFree-form text external realityObjective event triggering the subject’s affectFree-form text internaldriverSubject’s belief, desire, or intention motivating affectFree-form text I AFFECT What is the subject’s affective state, and how is it expressed and regulated? Affective State (A sta ) objectPerson toward whom the affect is directedFree-form text coarse categoryCategory of the subject’s affective state12 categories finedescriptionContextual nuance or blend of affectFree-form text polarityValence of the affective statePositive, Negative, Neutral, Mixed intensityStrength of the affective stateHigh, Medium, Low, None Manifestation (A man ) facial expressionObservable facial movement or configurationFree-form text bodylanguageObservable posture, gesture, movement, or spatial behavior Free-form text vocalcuesTone, pitch, volume, pace, or rhythmFree-form text verbalcontentSubject’s words or speech actFree-form text Regulation (A reg ) tactictypeRegulatory relation between source and targetSee Eq. (2) goalPurpose of regulating the affective stateFree-form text source affectstate Affective state before regulation12 categories target affectstateAffective state the subject intends to manifest12 categories I EFFECT How does the subject’s affect influence those present? Mental Effect (E men ) mental effectImpact on a recipient’s mental stateFree-form text Affective Effect (E aff ) affectiveeffectImpact on a recipient’s affective stateFree-form text Physical Effect (E phy ) physicaleffectImpact on a recipient’s action or speechFree-form text Subject. For example, a person who remains angry with a colleague may later refuse the colleague’s request for help because of that anger. Here, the Physical Effect is the Subject’s own subsequent behavior. The Effect stage is represented as E = (E men ,E aff ,E phy ). Table 1 summarizes the fields and value forms used to represent these components. 4 TRACE-BENCH: BENCHMARKING AFFECTIVE CHAIN INFERENCE Building on the Affective Blueprint, we develop TRACE- Bench to evaluate affective chain inference in multimodal social videos. We first formulate five tasks covering within- stage reasoning, cross-stage reasoning, and full-chain recon- struction, then describe the data construction process and evaluation protocol. 4.1 Task Formulation Each task takes a multimodal social videov, including its visual content, audio, and aligned subtitles, as the base input. A task instance specifies three parts: • Context (ctx t ) identifies the focal subject and episode. • Known Constructs (K t ) are the Blueprint constructs whose values are provided by the question. • Queried Constructs (Q t ) specify which Blueprint con- structs the model is asked to infer and, for a queried Effect, its channel and recipient. A known construct may be supplied through only a subset of its fields, as specified for each task. The answer template specifies the requested fields and any supporting evidence required by the task. The model generates the answer as b Y t = f (v, ctx t ,K t ,Q t ).(3) The five tasks cover within-stage reasoning (Tasks 1–2), cross- stage reasoning (Tasks 3–4), and full-chain reconstruction (Task 5). The mappings below summarize their given and queried constructs, with the shared video inputvomitted. Construct symbols denote task-level roles, with the supplied or predicted fields specified in the accompanying text and answer template. Square brackets indicate annotation- dependent constructs. Task 1. Grounded Affect Recognition. Givenctx 1 , which specifies the focal subject and episode, the model jointly predicts Affective State A sta and Manifestation A man : (ctx 1 )−→ Q 1 =A sta , A man (4a) No Blueprint constructs are supplied. Task 1 evaluates joint recognition of the subject’s felt state and observable expressions from multimodal evidence. Task 2. Regulation Decoding. Task 2 asks whether and how the observed manifestation is regulated relative to the felt affective state. The task supplies the category under regulation, its intensity, and the affective object as known information aboutA sta . Within the episode specified byctx 2 , the model infers regulation A reg : (ctx 2 ; A sta )−→ Q 2 =A reg (4b) The model reads the display from the video and determines whether it has been reshaped relative to the given state. Its answer identifies the tactic and, when regulation occurs, the goal and target affective state intended for outward LI et al.: COGNITION-ORIENTED EMOTION TRACING7 display. In addition to the queried Regulation construct, Task 2 requires Regulation Evidence (M reg ), comprising the manifestation cues inA man that specifically support the inferred regulation.M reg is scored on regulated instances and is not an additional Blueprint construct. The source state is fixed by the suppliedA sta . Whereas Task 1 jointly identifies what is felt and shown, Task 2 examines the regulatory relation between them. Task 3. Affective Cause Reasoning. Task 3 reasons back- ward from the subject’s Affect to its cause. GivenA sta ,A man , and any available annotation ofA reg , together withctx 3 , the model infers the relevant antecedents in the Condition stage: (ctx 3 ; A sta , A man , [A reg ])−→ Q 3 =C ext , [C int ](4c) The suppliedA sta includes category, intensity, polarity, and object, but excludes the fine-grained description.A man includes the annotated channels. The model must ground the affective outcome in its external antecedent and, when the annotation supports it, infer the Internal Driver that gives the event its affective significance. External Reality is required in every Task 3 instance, while Internal Driver is included only when supported by the annotation. Task 4. Affective Effect Reasoning. Task 4 reasons forward from the subject’s Affect to its Effect. GivenA sta ,A man , and any available annotation ofA reg , together withctx 4 , the model predicts the consequence along the queried Effect channel for the specified recipient. We write this query as E q (r), whereq ∈men, aff, phyidentifies the channel and r the recipient: (ctx 4 ; A sta , A man , [A reg ])−→ Q 4 =E q (r)(4d) Unlike Task 3, the suppliedA sta also includes the fine- grained description.A man andA reg follow the same pro- vision rules as in Task 3. Task 4 infers the downstream consequence for the specified recipient along the queried Effect channel. Task 5. Full Chain Reconstruction. Task 5 jointly recovers the complete Affective Blueprint fromctx 5 , which identifies the focal subject, affective object, and episode anchor. The query specifies the Condition and Affect constructs together with the requested Effect channels and their recipients: (ctx 5 )−→ Q 5 =C ext , [C int ], A sta , A man , [A reg ], E(4e) HereEcollectively denotes the requestedE q (r)entries. The affective object is given inctx 5 and is neither inferred nor scored. No other Blueprint fields are supplied. Question and answer templates appear in Appendix C.4. 4.2 Data Construction Step 1: Source Collection. We construct TRACE-Bench from social video clips drawn from Movie-clips [54], EQ4You [55], and Social-IQ [39]. The raw collections are first screened by an MLLM that summarizes each clip’s basic content and filters out clips that lack sufficient context for a recognizable affective episode, such as isolated reaction shots, exposition- only dialogue, single-person monologues, and highly styl- ized performances. Clips that pass this initial screening are further verified during chain annotation, where additional unsuitable clips are discarded. This two-stage process yields a curated pool of context-rich social interaction clips. Step 2: Chain Annotation. The basic annotation unit is a subject-centered affective chain, namely a subject-centered affective episode anchored at a specific moment in the video. We adopt this granularity because the Affective Blueprint is defined around a single subject’s process from Condition through Affect to Effect. Each chain is stored as a structured JSON object organized by the three Blueprint stages. A video may yield multiple chains across characters or moments, each localized by an episode anchor such as an utterance, facial display, body action, or vocal cue. For each selected clip, an MLLM drafts candidate chains from the video and subtitles. Human annotators then reject invalid proposals, correct inaccurate fields, add missed episodes, and verify the retained content against the subject, anchor, and scene evidence. Appendix C.1 reports field- level rewrite rates for retained candidate chains. During verification, a field is filled only when the video and subtitles provide sufficient evidence for it. A chain is retained as long as it contains sufficient information for at least one valid stage or cross-stage relation. The independent re-annotation protocol and agreement results are reported in Appendix C.2, while guidelines for selected key fields are provided in Appendix C.3. Step 3: QA Annotation. A verified chain may support multiple evaluation tasks, depending on which parts of the Blueprint are available in its annotation. We therefore assign each chain only to the tasks it can support and convert the corresponding annotations into task-specific QA pairs through predefined templates. For each eligible task, the template determines which information is exposed as the episode contextctx t and Known ConstructsK t , which constructs are queried asQ t , and how the relevant chain fields are rewritten into the question and reference answer. The same chain may consequently yield multiple QA pairs under different task templates. Each resulting pair undergoes final human validation against the original chain annotation. Dataset Statistics. TRACE-Bench contains 3,746 structured QA pairs from 646 videos. The videos span settings such as homes and workplaces, covering close personal relationships as well as professional and authority-based interactions. Task 1 uses a 12-category affective vocabulary comprising basic categories grounded in Ekman’s account [56], ad- ditional categories such as guilt and relief, and Neutral. Task 2 covers all five regulation tactics, from intensity modulation and category substitution to None, where the display matches the felt state. Task 4 contains between 286 and 452 queries for each of the three Effect channels. Fig. 4 summarizes task composition and annotation distributions. 4.3 Evaluation Protocol The five tasks require predictions of both categorical labels and open-ended descriptions, which we score at the field level before aggregating them into task scores. Categorical predictions are compared directly with the reference: since an Affective State can contain multiple categories, category sets are scored by set-F1, and single-label fields by exact match. Open-ended descriptions can express equivalent content in different wording, so we use DeepSeek-V4-Flash to rate each prediction against the annotated reference. Semantic LI et al.: COGNITION-ORIENTED EMOTION TRACING8 Video&Subtitle ◎ Movie-clips ◎ EQ4You ◎ Social-IQ Emo-rich Subsets AI Pre-annotate Human Re-annotate Human Validation Task Routing QA Generation Subsets 2 . C h a i n A n n o t a t i o n 3 . Q A A n n o t a t i o n LLM LLM 1 . S o u r c e C o l l e c t i o n Video&Subtitle ◎ Movie-clips ◎ EQ4You ◎ Social-IQ Emo-rich Subsets AI Pre-annotate Human Re-annotate Human Validation Task Routing QA Generation Subsets 2 . C h a i n A n n o t a t i o n 3 . Q A A n n o t a t i o n LLM LLM 1 . S o u r c e C o l l e c t i o n Video&Subtitle ◎ Movie-clips ◎ EQ4You ◎ Social-IQ Emo-rich Subsets AI Pre-annotate Human Re-annotate Human Validation Task Routing QA Generation Subsets 2 . C h a i n A n n o t a t i o n 3 . Q A A n n o t a t i o n LLM LLM 1 . S o u r c e C o l l e c t i o n 0100200300 Number of Videos Home Outdoor Office Institutional School Restaurant Medical Leisure 240 (37.2%) 128 (19.8%) 90 (13.9%) 47 (7.3%) 39 (6.0%) 38 (5.9%) 34 (5.3%) 30 (4.6%) Video Scene 050100150200 Number of Videos Romantic Authority Professional Family Friends Strangers 161 (24.9%) 141 (21.8%) 109 (16.9%) 103 (15.9%) 90 (13.9%) 42 (6.5%) Social Relationship 0100200300 Number of Videos Home Outdoor Office Institutional School Restaurant Medical Leisure 240 (37.2%) 128 (19.8%) 90 (13.9%) 47 (7.3%) 39 (6.0%) 38 (5.9%) 34 (5.3%) 30 (4.6%) Video Scene 050100150200 Number of Videos Romantic Authority Professional Family Friends Strangers 161 (24.9%) 141 (21.8%) 109 (16.9%) 103 (15.9%) 90 (13.9%) 42 (6.5%) Social Relationship Task Composition: 3,746 QA / 646 videos T1 919T2 401T3 795T4 1,123T5 508 24.5% QA / 625 vids10.7% QA / 344 vids21.2% QA / 540 vids30.0% QA / 418 vids13.6% QA / 381 vids Task 1: Affect Category Anger 294 (21.7%) Surprise 205 (15.1%) Sadness 203 (15.0%) Fear 146 (10.8%) Happiness 138 (10.2%) Disgust 118 (8.7%) Guilt 60 (4.4%) Frustration 49 (3.6%) Anxiety 48 (3.5%) Embarrassment 44 (3.2%) Relief 40 (2.9%) Neutral 11 (0.8%) Polarity 80% Negative 734 (79.9%) Positive142 (15.5%) Mixed23 (2.5%) Neutral20 (2.2%) Intensity 46% Medium422 (45.9%) High 415 (45.2%) Low76 (8.3%) None6 (0.7%) Task 2: Regulation Tactic 47% Suppression 189 (47.1%) Substitution 98 (24.4%) None 76 (19.0%) Amplification 29 (7.2%) Fabrication 9 (2.2%) Task 3: Cause Structure 51% External only 409 (51.4%) Internal + ext. 386 (48.6%) External reality appears in all T3 items Task 4: Effect Channel 40% Physical 452 (40.2%) Affective 385 (34.3%) Mental 286 (25.5%) Fig. 4. TRACE-Bench construction pipeline and dataset statistics, with the three-stage construction workflow on the left. adequacy,d 1 , measures whether the prediction captures the intended content for the correct character and episode. A prediction may also overlap with the reference by repeating generic statements or listing competing guesses, and a separate redundancy rating,d 2 , estimates how much of the answer adds no information relevant to the queried field. Each field is rated repeatedly, and thed 1 andd 2 values are averaged separately, giving the field score s(f ) = ̄ d 1 (f ) 1− λ ̄ d 2 (f ) ,(5) where both ratings lie in[0, 1]andλbounds the redundancy discount. We setλ = 0.5, so that redundancy can reduce a field score by at most half while semantic adequacy remains its basis. Appendix B.1 provides the field-matching rules and judging rubric. Field scores are combined along the structure of the Affec- tive Blueprint: fields are first averaged within constructs and then constructs within stages, so that a construct described by several fields, such as Manifestation, does not outweigh the others. Full-chain reconstruction thus gives Condition, Affect, and Effect equal weight. In Task 2, Regulation Evidence is scored alongside the applicable Regulation fields, with equal weight for each. WithN t evaluated items in taskt,F ti the evaluated fields of itemi, andw ti (f )their normalized task- specific weights, the overall score averages item scores within each task and then across tasks: TRACE-Score = 100 5 5 X t=1 1 N t N t X i=1 X f∈F ti w ti (f )s ti (f ).(6) Appendix B.2 specifies task-dependent scoring conditions and the treatment of missing fields, Appendix B.3 the scoring and reporting of unparsed outputs, and Appendix A.7 human solvability and judge calibration. We use video-level cluster bootstrap (10,000 resamples), sampling videos with replacement and retaining all their items, with shared resamples across models and tasks. We apply Holm correction to comparisons against 14 baselines on each of the five tasks. The same resamples yield the 95% confidence interval for the five-task average margin over GPT-5 (thinking). 5 TRACER: COGNITION-GROUNDED STRUC- TURED REASONING TRACE tasks involve heterogeneous Blueprint constructs, including observable manifestations, psychological drivers, affective states, regulation, and downstream effects. Inferring these constructs requires different forms of evidence and attention to their dependencies within an affective episode. This complexity presents two challenges to current models. First, models must interpret the affective meaning behind the observed behaviors and interactions. They must determine whether an event functions as an affective trigger for the subject and whether an outward manifestation expresses the felt state or a regulated display. Reliance on visible cues leads to a surface reading that collapses these alternatives. Second, placing heterogeneous constructs in a single generation process increases factual drift, especially in multi-field, long- chain reconstruction. An upstream interpretation may be promoted into an established fact and then used to generate a plausible downstream event beyond the recorded evidence. The resulting chain remains linguistically coherent while departing from what occurs in the video. To address these challenges, we propose TRACER, which constrains every inference to premises from factual observations, their affective meanings, and relevant Blueprint constructs. 5.1 Method Overview As shown in Fig. 5, TRACER transforms multimodal social video into task-specific predictions through three stages. Factual Observation Extraction converts the video and aligned subtitles into an indexed recordOof observable events, behaviors, and utterances. Cognition-Grounded Affective Analy- sis interprets the relevant observations from the involved characters’ perspectives, producing character-specific ap- praisal readingsA. Blueprint-Guided Structured Derivation combines these observations and appraisal readings with the task’s known and queried constructs to derive the requested outputs through premise-grounded reasoning. LI et al.: COGNITION-ORIENTED EMOTION TRACING9 Step-I: Factual Observation Extraction (a) Task-conditioned Reasoning Topology Step-I: Blueprint-Guided Structured Reasoning Observe The customer is having a chat with his friend in ... Holding up a shoebox, the customer suddenly says he... The customer cites the policy with a slight smirk... The customer walks away with the shoebox. The employee gestures and offers another size with a smile. Step-I: Cognition-Grounded Affective Analysis Observations O Coping Potential Normative Significance RelevanceImplication Appraise R 1 I 2 C 1 N 1 R 1 I 2 C 1 N 1 R 1 I 2 C 1 N 1 R: His service duty and the store's merchandise are at stake. I: The customer intends to exploit the policy and leave without paying. C: He cannot physically intervene N: His retail role calls for courtesy and non-confrontation. R 1 I 2 C 1 N 1 Appraisals A T4: Effect Reasoning T1: Affect Recognition T3: Affective Cause Reasoning T2: Regulation Decoding T5: Full-chain Reconstruction (b) Premise-declared Derivation Customer declares theft, employee offers another size R: Protecting mer- chandise matters to me. C: I cannot physically intervene. Theft exploiting store policy Wants to protect stock Surprise, Anxiety Service smile Substitution to Happiness N:Role requires non-confrontation. Leave with shoes See? they can’t stop me The employee has no authority to stop me. The employee gives a stiff smile, his eyes darting, while offering another size. Fig. 5. Overview of TRACER. Factual observations are interpreted through character-centered appraisal and used in Blueprint-guided derivation. (a) Gray arrows denote intrinsic relations, while black arrows specify inference dependencies and premise requirements. Blue, green, and purple construct nodes denote known, intermediate, and queried constructs, respectively. (b) An abridged shoe-store example illustrates how observations, appraisals, and intermediate conclusions jointly support subsequent inferences. The five tasks share this pipeline and the Affective Blueprint, while the final derivation is organized according to each task’s supplied and queried constructs. All three stages use stage-specific prompts with off-the-shelf MLLMs or LLMs and require no additional training. 5.2 Factual Observation Extraction The first stage establishes the factual record used throughout reasoning. Given a complete videovand aligned subtitless, the observation module (Appendix A.2) produces O = Observe(v,s) =o i n i=1 ,(7) where each observation itemo i records factual content from a local video interval or its aligned subtitles and receives a stable identifier. Together, they form a temporally ordered ac- count of the participants, actions, interactions, facial and bod- ily behavior, vocal delivery, and spoken content in the scene. This stage records perceptible content before assigning its affective meaning. Affective state, internal driver, regulation, and downstream effect are therefore reserved for later reasoning. This boundary gives every subsequent conclusion an independent factual source and keeps an inferred state from becoming evidence for its own manifestation. The model uses the task contextctx t as an anchor to locate relevant observations inO. It then selects and groups observation itemsS t ⊆Ofor appraisal and reasoning. The full record remains available for citation throughout the subsequent stages. 5.3 Cognition-Grounded Affective Analysis An observed event acquires affective meaning through its relation to a character’s goals, expectations, resources, and social position. Inspired by cognitive appraisal theory [18], we analyze the selected items from the position of the character involved. The analyzed characters are specified by the task, namely the subject and, when the query includes an Effect, the target recipient. For characterr, a selected group of itemsS t,k ⊆ S t , and appraisal dimensiond, the analysis produces a (d) r,k = Appraise d (S t,k | r), d∈D,(8) wherea (d) r,k is a complete natural-language proposition stating what the cited facts mean fromr’s position. Its evidence links remain attached to the proposition. The four dimensions in D examine complementary parts of this meaning: Cognitive Appraisal Dimensions •Relevance identifies why the event attracts the charac- ter’s attention through novelty, intrinsic pleasantness, or relation to an active goal or need. • Implication determines the event’s consequences for that goal or need through agency, expected outcome, goal conduciveness, and urgency. •Coping Potential examines the character’s control, available power and resources, and capacity to adapt to the consequences. •Normative Significance evaluates compatibility with the character’s self-concept, personal standards, and recognized social or moral standards. The output of each dimension is an evidence-grounded interpretation rather than a score, level, category, or polarity. We collect the resulting readings asA. In Fig. 5(a),A sub andA rec denote the subject-side and recipient-side readings in this collection. Subject-side readings are produced for every selected group. When the queried construct includes an Effect, recipient-side readings are additionally produced for the group that contains the subject’s described behavior. The same observation can thus receive different meanings LI et al.: COGNITION-ORIENTED EMOTION TRACING10 under the subject and recipient perspectives while each interpretation remains bound to its role and cited facts. These readings enter Blueprint-guided derivation as cognitive premises, not final field values. Core prompt instructions appear in Appendix A.3. 5.4 Blueprint-Guided Structured Derivation Given the factual observationsOand appraisal readingsA established in the previous stages, the final stage derives the Blueprint constructs required by the task. The five tasks share the same affective structure but differ in their known constructsK t and queried constructsQ t , requiring different derivation paths. A task-conditioned reasoning topology specifies which constructs must be established, what infor- mation each inference requires, and their derivation order. Premise-declared derivation then uses concrete premises from the current video to carry out these inferences. Task-conditioned reasoning topology (RT). The Affective Blueprint provides the common construct system and the intrinsic relations among its components. RT organizes the dependencies needed to establish the queried constructs, introducing missing constructs as intermediate conclusions when needed. Fig. 5(a) distinguishes intrinsic Blueprint relations in gray from active inference links in black. The former describe how constructs are related within an affective episode, while the latter specify how available information is used to derive a construct for the current task. In Task 3, for example, supplied Affect constrains the inference of its antecedent Condition, while the underlying affective process still follows Condition→ Affect. Nodes. We retain External, Internal, State, Manifestation, Regulation, and Effect as construct nodes. Their roles depend on the task: a construct may be supplied inK t , introduced as an intermediate conclusion, or queried inQ t . Premise- source nodes represent observationsOand subject-side and recipient-side appraisal collectionsA sub andA rec . Links. A construct-dependency linku→ xrequires informa- tion fromuto inferx. External and Internal jointly support State, reflecting how an external event acquires affective significance in relation to the subject’s internal stance. State and Manifestation jointly support Regulation through their comparison. Effect draws on the supplied or derived Affect constructs, including Regulation when applicable. The topol- ogy also specifies the premise sources used by each inference. External and Internal draw on observations together with subject-side appraisal. Manifestation is derived directly from observations, independently of the inferred State. Regulation uses subject-side appraisal, while Effect uses appraisal from the queried recipient. We represent the resulting topology as T t = (V t ,E t ,≺ t ).(9) whereV t contains the relevant construct and premise-source nodes,E t contains the active inference links, and≺ t places construct premises before conclusions that use them. Sup- plied values in K t remain fixed during derivation. Premise-declared derivation (PDD). Following the topology, PDD selects concrete observations and appraisal readings for the current video, together with supplied constructs or earlier conclusions, and cites them as premises. This design follows the broader principle of making intermediate reasoning structure explicit during inference, as explored in prior work on faithful and structured logical reasoning [57], [58]. For each intermediate or queried construct, the model jointly produces the construct identity, its premise references, and the inferred field values: c j =⟨x j ,P j ,y j ⟩.(10) Herex j identifies the construct,P j records its premise refer- ences, andy j contains its inferred field values. The derivation sequenceD t = (c 1 ,...,c L )follows≺ t . Established nodes become available to later steps with fixed identifiers and field values, while the full observation and appraisal collections remain accessible for premise selection. The final answer valuesD t [Q t ]are extracted from the queried nodes rather than generated separately. These form the task answer b Y t , with Task 2 additionally including Regulation EvidenceM reg . In Fig. 5(b), the customer’s declaration and the em- ployee’s appraisal establish External, while the employee’s concerns about merchandise and non-confrontation support Internal. These intermediate conclusions then support the inferred State. For Task 1, the Condition constructs therefore serve as intermediate premises even though the task queries State and Manifestation. 6 EXPERIMENTS 6.1 Experimental Setup Model groups. We compare TRACER with the three base- line groups listed in Table 2. Open-source generic models cover different scales and multimodal architectures, including LLaVA-OneVision [59], MiniCPM-V-4.5 [61], InternVL3.5 [64], and Qwen3-VL [65]. Affective-specialized models are trained specifically for emotion understanding and include Emotion- LLaMA [13], AffectGPT [10], and Emotion-Qwen [66]. Closed- source models comprise GPT-5 [67], evaluated in thinking and non-thinking modes, and Gemini-3-Pro [68]. These groups allow comparison of model-family strengths and limitations across TRACE tasks. Implementation details. Baseline models receive video frames, available subtitles appended to the prompt, and the task question. Only Qwen3-Omni-30B receives audio. Frame budgets are model-specific: GPT-5 uses eight uniformly sam- pled frames, while Qwen3-VL-32B uses its video processor with a 60-frame budget, both covering the full clip. The question provides known information and a JSON template specifying the requested fields. For TRACER, Qwen3-VL generates factual observation records, and GPT-5 generates character-specific appraisals and subsequent construct-level inferences. Appendix A.1 gives full inference settings. All models follow the evaluation protocol in Section 4.3. 6.2 Main Results on TRACE-Bench Overall performance. Closed-source general-purpose mod- els lead the baselines in Table 2, with GPT-5 (thinking) scoring highest overall. TRACER outperforms every model baseline LI et al.: COGNITION-ORIENTED EMOTION TRACING11 TABLE 2 Main results on TRACE-Bench. Avg. averages T1–T4 and T5 Full. Model scores use the main evaluation set; Human scores use a subset of 50 questions per task. Matched model scores are in Appendix A.7. Per-stage T5 scores are in Section 6.4. Unless otherwise stated, tables use bold for the best model results and underliningfor the second best. * marks cells with substantial parsing failures, and† cells where most outputs could not be parsed. ModelT1 Rec.T2 Reg.T3 CauseT4 EffectT5 FullAvg. Human (n = 50)83.6072.3285.5578.9075.4079.15 Affect Specialized Emotion-LLaMA [13]0.455.22 * 2.26 † 0.31 † 0.20 † 1.69 AffectGPT [10]34.574.64 † 17.99 * 15.79 * 20.87 * 18.77 Emotion-Qwen29.45 * 24.24 * 21.04 * 13.4919.13 * 21.47 Open Source Generic LLaVA-OneVision-7B [59]35.1323.5125.284.115.47 † 18.70 LLaVA-NeXT-Video-32B [60]25.8927.0127.13 * 23.6113.80 * 23.49 MiniCPM-V-4.5 [61]37.3029.1943.0428.2729.4333.45 GLM-4.6V-Flash-9B [62]40.8028.8542.3028.8132.3634.62 LLaVA-OneVision-70B [59]42.7426.7347.3033.0334.6236.88 Qwen3-Omni-30B [63]46.2833.1751.6939.8141.7842.55 InternVL3.5-38B [64]45.3528.0546.1632.1937.0737.76 Qwen3-VL-32B [65]50.9735.4960.4244.3850.1648.28 Closed Source Generic GPT-5 (non-thinking)56.1337.2670.7452.3658.2254.94 Gemini-3-Pro54.3240.5570.7451.4856.0554.63 GPT-5 (thinking)56.61 41.5677.1153.7560.7957.96 TRACER (Ours)61.6749.5384.5559.4763.5963.76 Emotion-Qwen LLaVA-OV-70B Qwen3-VL-32B GPT-5 (non-thinking) Gemini-3-Pro GPT-5 (thinking) TRACER Human average 45 55 65 75 100 T1 T2 T3 T4 (a) Component profile. AffectOpenClosed T1 T2 T3 T4 T5 Mean +3.0+1.6 -4.6 +5.4 +2.6 -8.0 -8.6 +0.0 +8.6 +0.1-1.0+0.9 +0.1-3.2+3.2 14.034.555.8 -8-4048 Relative preference (p) (b) Family preference. Fig. 6. Model profiles by (a) task component and (b) family. Human scores average two participants. on each task (p < 0.05, bootstrap test, Holm-corrected). Its average margin over GPT-5 (thinking) is 5.80 points (bootstrap 95% CI 4.42 to 7.17). The largest gains occur in T2 regulation decoding (7.97 points) and T3 cause diagnosis (7.44 points), followed by T4 effect reasoning and T1 affect recognition. Gains span recognition and reasoning. Full-chain reconstruction shows the smallest improvement. Despite this improvement, T2 remains the lowest-scoring task for both GPT-5 (thinking) and TRACER. For a matched human–model comparison, we evalu- ate models on the same 50 questions per task answered by the two human participants (Appendix A.7). On this subset, humans average 79.15, GPT-5 (thinking) 58.75, and TRACER 63.58, leaving a human–TRACER gap of 15.57 points. TRACER is closest to humans on T3: 84.39 versus 85.55, a 1.16-point gap. The other four tasks show larger gaps. Scale and thinking yield task-selective gains. Within LLaVA-OneVision, scaling from 7B to 70B improves all five tasks, with the largest gains in cause reasoning, effect reasoning, and full-chain reconstruction. However, Qwen3- VL-32B outperforms the 70B model across all tasks, showing that parameter count alone does not explain performance differences across models. Enabling thinking in GPT-5 pri- marily benefits cause diagnosis and regulation decoding, with smaller gains in affect recognition and effect reasoning. For example, T3 improves by 6.37 points, whereas T1 changes by only 0.48 points. Model families favor different tasks.Fig. 6b shows distinct task profiles across the three model families. Affect- specialized models exhibit relative strengths in T1 affect recognition and T2 regulation decoding, but a pronounced weakness in T3 cause reasoning. Open-source general- purpose models have a more balanced profile across in- dividual tasks, yet show a deficit in T5, where Condition, Affect, and Effect must be reconstructed jointly. Closed- source models display a contrasting pattern: their strengths are concentrated in cause reasoning and full-chain recon- struction, while T1 and T2 are comparatively weaker parts of their profile. Scores are centered by family and task averages (supplementary Eq. S1), so these profiles describe within- family task preferences, not absolute rankings across families. Closed-source models can therefore be relatively weaker on T1 and T2 yet score higher than affect-specialized models. 6.3 Evidence Ablation Design. To assess the contribution of video evidence beyond the information supplied in each task, we evaluate GPT- 5 (thinking) and Qwen3-VL-32B with the question and supplied fields alone, then add subtitles, frames, or both. Frame inputs use eight frames for GPT-5 and 60 for Qwen. We also test GPT-5 with subtitles and 16 frames. Results and analysis. Joint subtitle and frame input out- performs question-only input on all five tasks for both models (Fig. 7), showing that the supplied fields do not LI et al.: COGNITION-ORIENTED EMOTION TRACING12 0 40 80 GPT-5 (thinking) T1 32.0 41.4 43.6 55.5 56.8 T2 29.5 34.3 30.3 40.7 40.2 T3 49.7 69.2 59.4 76.5 78.7 T4 45.1 54.3 48.1 56.1 58.9 T5 46.9 56.8 50.8 61.6 62.8 0 40 80 Qwen3-VL-32B 41.4 48.4 46.2 51.0 29.4 31.6 34.3 35.7 39.4 52.2 42.3 60.1 32.6 37.4 42.5 44.4 39.3 47.7 43.8 50.1 Q only+ Subs+ Frames+ Both+ Both (16f) Fig. 7. Evidence ablation. Q includes the question and supplied fields. Frames denotes 8 frames for GPT-5 and 60 for Qwen. Both (16f): GPT-5 with subtitles and 16 frames. (a) coarse category Fear Ang. Sad. Sur. Fru. Anx. Ang. 43% 29% 25% 21% (b) intensity Med. Low High Med. 56% 30% 28% (c) polarity Pos. Mix. Pos. Neg. 47% 47% 23% Fig. 8. T1 confusions from true (left) to predicted (right) labels. Band widths show proportions within each true class. replace episode-specific evidence. Neither channel alone matches joint input on any task, supporting the need for multimodal evidence in affective understanding. Channel preferences nevertheless vary: both models favor subtitles on T3 and T5, but their preferences differ on T1, T2, and T4. For example, Qwen favors frames on T4, whereas GPT-5 favors subtitles. Increasing GPT-5’s frame count from eight to 16 produces small gains on four tasks and a slight decrease on T2. Combining evidence channels therefore provides a more consistent benefit than denser visual sampling. 6.4 Within-Task Performance Breakdown Task averages can mask uneven component performance. Components are scored over applicable instances, with full results in Appendix A.4. Task 1. Grounded Affect Recognition. Table 3 shows a consistent gap between the two first-order components: State scores higher than Manifestation for all six baselines, with larger gaps for the two open-source models than for GPT-5. Enabling thinking improves Manifestation by 1.4 points, while State remains nearly unchanged. Looking more closely at the State predictions reveals where the remaining errors concentrate. Fig. 8 shows recurrent confusions between semantically related categories, particularly fear and anxiety, and anger and frustration. Intensity predictions frequently shift both low- and high-intensity states toward medium, TABLE 3 T1 field-level performance. Shaded rows report construct scores aggregated within each instance. EmoQ* OV-70B Q3-VL GPT5-N Gem-3 GPT5-T TRACER Affective state36.857.360.159.763.359.264.9 Category27.240.138.836.148.141.540.5 Description11.736.848.155.653.757.3 60.0 Polarity57.580.181.469.181.471.377.5 Intensity35.461.164.264.558.263.366.5 Object51.867.767.873.074.7 62.579.9 Manifestation22.128.241.852.645.454.058.4 Facial30.032.947.256.948.559.257.7 Body20.423.136.646.237.347.255.6 Verbal16.031.044.860.155.461.670.6 Vocal23.327.543.052.445.654.2 56.6 NoneSuppr.Subst.Ampl. None Suppr. Subst. Ampl. True tactic 24382513 1162234 534574 10342431 GPT-5 (thinking) NoneSuppr.Subst.Ampl. 8542216 179155 044478 0312841 TRACER 0 25 50 75 100 % Predicted tactic Fig. 9. T2 tactic confusion for GPT-5 (thinking) and TRACER. Rows: reference; columns: prediction; values: row percentages. Fabrication is omitted from both axes. while mixed polarity is often reduced to a single polarity. Thus, higher State scores still mask errors in category, intensity, and mixed polarity. Task 2. Regulation Decoding. For Task 2, the clearest error pattern appears in tactic prediction. Models frequently infer active regulation when the reference tactic is None, with false- alarm rates ranging from 76% to 96% across GPT-5 (thinking), Qwen3-VL, and TRACER. As shown in Fig. 9, Suppression is the most common prediction for these unregulated cases. TRACER identifies Suppression and Amplification more often than GPT-5 (thinking), but also recognizes None less often, indicating a stronger tendency to over-detect regu- lation. This pattern suggests that tactic prediction remains particularly difficult at the boundary between genuinely regulated displays and cases without active regulation. Beyond tactic prediction, thinking substantially improves Target State for GPT-5, from 28.0 to 46.7 (Table 4), suggesting that reasoning helps infer the intended target affect given the underlying felt state. Task 3. Affective Cause Reasoning. All evaluated models across the three families score lower on Internal Driver than External Reality, falling belowy = xin Fig. 10. Thinking improves GPT-5’s Internal Driver score more than External Reality, narrowing this gap. TRACER scores highest on both components, improving more over GPT-5 (thinking) on External Reality. Internal Driver also improves but remains lower-scoring. These results suggest that, despite gains in overall cause reasoning, recovering the subject’s Internal Driver remains harder for most evaluated models. LI et al.: COGNITION-ORIENTED EMOTION TRACING13 TABLE 4 T2 field-level performance. Goal and Regulation Evidence use regulated instances. Target state is scored only when its reference category differs from the source category. EmoQ* OV-70B Q3-VL GPT5-N Gem-3 GPT5-T TRACER Tactic36.746.948.947.449.550.953.8 Target state28.05.626.228.031.646.737.7 Goal13.530.151.154.851.256.266.5 Reg. evidence10.08.718.123.6 18.123.650.3 TABLE 5 T4 effect-channel performance, including self-directed and other-directed effects. Each channel is scored on its applicable instances. EmoQ*OV-70BQ3-VLGPT5-NGem-3GPT5-TTRACER Mental2.138.153.559.458.860.564.5 Affective19.640.749.057.852.656.857.0 Physical15.523.334.643.345.946.958.0 020406080 External Reality 0 20 40 60 80 Internal Driver Internal = External +thinking closed-source open-source affect-specialized TRACER (ours) GPT-5 (thinking) GPT-5 (non-thinking) Gemini-3-Pro Qwen3-VL-32B Qwen3-Omni-30B InternVL3.5-38B LLaVA-OV-70B GLM-4.6V-9B MiniCPM-V-4.5 LLaVA-OV-7B LLaVA-NeXT-32B AffectGPT Emotion-Qwen* Emotion-LLaMA Fig. 10. T3 External Reality versus Internal Driver scores by model. The dashed line marks equal scores. The arrow shows GPT-5’s change with thinking. 66.857.258.0TRACER (ours) 61.257.450.2 GPT-5 (thinking) 59.958.547.0 GPT-5 (non-thinking) 59.453.246.0 Gemini-3-Pro 53.550.238.2 Qwen3-VL-32B 47.748.133.2 Qwen3-Omni-30B 38.741.531.1 LLaVA-OV-70B 38.443.223.0 InternVL3.5-38B 42.343.58.6 GLM-4.6V-9B 31.637.916.6 MiniCPM-V-4.5 25.935.019.0 LLaVA-NeXT-32B 1.76.11.4 LLaVA-OV-7B 12.021.112.7 AffectGPT 2.419.515.4 Emotion-Qwen* 0.00.70.04 Emotion-LLaMA Mental EffectAffective Effect Physical Effect Fig. 11. T4 effect-channel scores on other-directed instances. Dashed lines separate TRACER, closed-source, open-source, and affect- specialized models. TABLE 6 T5 full-chain reconstruction scores by construct. Cond.StateManif.Reg.Effect Emotion-Qwen * 14.0945.6923.2012.0911.18 LLaVA-OV-70B33.4964.1039.2832.0820.39 Qwen3-VL-32B51.5969.6155.8340.6438.21 GPT-5 (non-thinking)66.9170.3462.0142.5543.80 Gemini-3-Pro63.6773.2258.7221.4643.25 GPT-5 (thinking)70.29 70.5463.1441.0947.74 TRACER72.1871.8863.0842.1853.84 Task 4. Affective Effect Reasoning. Physical Effect is the lowest-scoring channel for the general-purpose baselines shown in Table 5, in both the full evaluation and the other-directed subset (Fig. 11). The relative performance on Mental and Affective Effect, however, varies across models. In Fig. 11, closed-source models and Qwen3-VL score higher on Mental than on Affective Effect, whereas the remaining open-source and affect-specialized models show the reverse pattern. TRACER achieves the highest Mental and Physical Effect scores in both evaluation scopes, while its Affective Effect score remains close to GPT-5 (thinking). Its gains are therefore uneven across channels, with the largest improvement over GPT-5 (thinking) on Physical Effect, from 46.9 to 58.0 in Table 5. This improvement also accords with the T5 diagnostic results in Section 6.5, which show fewer unsupported Effect statements under TRACER. Task 5. Full Chain Reconstruction. Full-chain performance varies markedly by component (Table 6). Affective State scores highest across all baselines, but the lowest-scoring component varies: Effect scores lowest for Emotion-Qwen, LLaVA-OneVision-70B, and Qwen3-VL-32B, while Regula- tion scores lowest for the closed-source general-purpose models. For GPT-5, enabling thinking mainly improves Condition and Effect, with only minor changes in the remaining components. TRACER further improves on GPT-5 (thinking), raising the overall full-chain score from 60.79 to 63.59. Its largest component-level gain is on Effect, which increases from 47.74 to 53.84, while State, Manifestation, and Regulation change only slightly. These results suggest that TRACER’s improvement in full-chain reconstruction is partic- ularly pronounced in identifying downstream consequences. 6.5 Diagnostic Evaluation of Failure Modes These experiments diagnose factual drift through fabricated observable content in full-chain reconstruction, and surface reading through confusion between Affective State and Mani- festation. Both use judgments blind to configuration identity. 6.5.1 Factual Drift in Full-Chain Reconstruction Setup and metric. For T5 outputs, we check factual state- ments in External Reality, Manifestation, and Physical Effect against video frames and subtitles. The fabrication rate is the percentage of fabricated statements among all judged state- ments, including unverifiable ones. Empty fields are tracked separately. Appendix A.6.1 details the sample, agentic audit, human verification, and results. Results and analysis. Baseline fabrication is most pro- nounced in downstream Effect statements (Fig. 13). TRACER reduces the aggregate rate from 11.0% to 4.7%, with the LI et al.: COGNITION-ORIENTED EMOTION TRACING14 ...make sure they hate you. You’re not home killing your wife. ...say something before? ctx 1 : The defendant’s response when Kevin explains the defense strategy. Y 1 : State, Manifestation T1: Grounded Affect Recognition Observation & Appraisal O Kevin tells the defendant that he will work hard to make the jury hate him, as long as he was not at home killing his wife. As the defendant listens, .... He briefly smiles, nods, and asks in a gentler voice, “Why didn’t you...” A s R: The explanation matters to the goal of avoiding conviction. I: Unlikability now serves the defense rather than signaling abandonment. C: The jury remains beyond direct control, but following Kevin’s plan offers a workable path. N:Cooperative restraint is appropriate after grasping the strategy. Condition External Reality In the courthouse hallway, Kevin explains that making the jury hate the defendant is part of the defense, provided he was not at home killing his wife. Internal Driver The defendant wants to avoid convic-tionand preserve his standing with the jury. He is motivated to trust counsel if this advances his defense while minimizing reputational damage. Affect Affective State Object:Kevin Category:Relief, Anxiety Polarity:Mixed Intensity:Medium Description:Easing tension with residual vigilance about the jury’s dislike. Manifestation Facial:brief smile Body:relaxed posture, small nod Vocal:gentler voice Verbal:“Why didn’t you say something before?” O SourceCategoryPolarity / Intensity ReferenceRelief / EmbarrassmentMixed / Medium Qwen3-VL-32BFrustration / AnxietyNegative / High GPT-5 (thinking)Anxiety / Frustration / AngerNegative / High ctx 2 : Focus on the colleague as Laura confronts him. K 2 : State (category, intensity) Y 2 : Regulation T2: Regulation Decoding Observation O Laura presses him, urgent and accusatory: “You’re not just a bystander, you’re a contributor. You’re an architect of the entire operation.” He listens, his face tense, his jaw tightening. He then replies louder and more fiercely: “You’re trying to get me to spill state secrets. They’re bombing our state.” His gaze shifts and his expression shows strain while she watches him closely. Affect Affective State Category:Fear Intensity:Medium Manifestation Facial:tense face, tightening jaw Body:holds posture, leans into his reply Vocal:louder, fierce, firm tone Verbal:“You’re trying to get me to spill state secrets. They’re bombing our state.” Regulation & Appraisal Regulation Tactic:Substitution Target:Anger Goal:Reframe fear as controlled, duty-bound firmness to maintain authority and justify refusal to disclose. Evidence:Face tense with a tightening jaw; voice louder and fierce compared to prior turns. A s I: Her questioning is an intentional attempt to force disclosure of protected information during a crisis. C: He still has enough standing to push back forcefully and reframe the exchange. N: Disclosure would violate a binding duty in wartime, so a firm, controlled refusal is required. SourceTacticTarget ReferenceSubstitutionAnger Qwen3-VL-32BSuppressionFear GPT-5 (thinking)SuppressionFear Have you noticed?No, I haven’t noticed. He asked you out already, is that it? ctx 3 : The woman’s embarrassment as June probes her father’s interest in her. K 3 : State, Manifestation, Regulation Y 3 : External Reality, Internal Driver T3: Affective Cause Reasoning Affect Affective State Object:June Category:Embarrassment Polarity:Negative Intensity:Medium Manifestation Facial:pursed lips, downward gaze, stiff smile Body:leans back, hands pause Vocal:faster speech, light laugh Verbal:“No, I haven’t noticed.” Observation & Appraisal A s N:Adult responsibility calls for tact and adult boundaries. N:Private romantic concerns should not burden the child. Remain cautious and noncommittal. C:Can deflect without committing, but cannot speak for June’s father. O June asks whether the woman has noticed her father’s interest, then presses whether he has already asked her out. The woman says, “No, I haven’t noticed,” looks down, and suggests asking him. Cause External Reality During a lesson, June questions the woman about her father’s romantic interest and whether he has asked her out. She insists that he still loves her mother, pressing the adult about a private romantic matter. Internal Driver Preserve adult boundaries, protect the child’s emotional comfort and maintain personal privacy. Answer cautiously and redirect to the father rather than disclose intimate details. SourceExternal RealityInternal Driver Reference The girl explicitly details the father's courtship tactics, revealing she has observed their interactions. She expects the conversation to remain on casual or academic topics, avoiding scrutiny of her personal romantic life. Qwen3-VL-32B [Blonde Girl-Student] directly addresses [Brunette Female- Stepmother] about her father's romantic interest in her, implying that the stepmother has noticed or should notice it. She desires to maintain a neutral and uninvolved position in her stepfather's romantic lifeand fears seeming overly interested or intrusive, so she denies noticing anything. GPT-5 (thinking) [Blonde Girl-Student] stated that her father likes [Brunette Female-Stepmother], described his behavior as evidence, and asked if she had noticed or if he had already asked her out. Not returned. Wait a minute, you knew... I was gonna... ...something bigger with Harper. ctx 4 : Black Shirt’s hesitant explanation and its effect on Pink Shirt. K 4 : State, Manifestation Y 4 : Physical Effect T4: Affective Effect Reasoning Affect Affective State Object:Pink Shirt Category:Guilt Polarity:Negative Intensity:Medium Manifestation Facial:eyes evading, slight frown Body:turns head slightly Vocal:weak, hesitant Verbal:“I was gonna...” Observation & Appraisal A r R: It bears on whether her bandmate trusted her and whether she was kept in the dark. I: Black Shirt did not protect her from a harmful surprise, which pushes her toward asserting a bigger path for the band. C: Pressing an alternative plan can influence the band’s direction, but not fully control the bandmate’s response. N: Withholding the news breaches honesty between bandmates and warrants accountability. O Pink Shirt, her voice trembling, says, “You knew ... and you didn’t tell me.” Black Shirt looks down and answers, “I was gonna,” trailing off. Pink Shirt continues, firm and challenging: “Now who’s the one with the vision for the band?” Black Shirt turns to the counter without answering. Effect Physical Effect She confronts the withheld news, interrupts the hesitant explanation and raises her voice. She gestures while proposing something bigger with Harper, then challenges who has a vision for the band. SourcePhysical Effect (condensed) ReferenceInterrupts the explanation and angrily shifts to their future options. Qwen3-VL-32BStops moving, looks down and turns away in discomfort and hesitation. GPT-5 (thinking)Hands over the blender jar, steps back and stops helping with cooking. Fig. 12. Case studies for T1–T4, with T1 and T2 in the top row and T3 and T4 in the bottom row. Panels show task inputs, selected observations and appraisal readings, construct outputs, and reference and baseline comparisons. Text is condensed from the source records. The T1 example comes from a separate illustrative run. The layouts follow the corresponding task-specific reasoning topologies. largest reduction in Effect, from 17.3% to 3.3%. Supplying ob- servation records alone also reduces fabrication, showing the contribution of explicit evidence. The full method substan- tially reduces unsupported observable content, particularly in the inferred aftermath of the episode. 6.5.2 Surface Reading of Affective Manifestations Setup and metric. We test whether appraisal helps models identify the subject’s felt state when it differs from the outward display. The baseline describes the clip from video frames and subtitles without being alerted to this mismatch. Appraisal CoT asks the model to analyze each character’s situation along the four appraisal dimensions before writing the description. In the appraisal-records setting, readings produced by a separate analysis are supplied alongside the original inputs. Fig. 14 reports the intervention comparisons for GPT-5 and Qwen3-VL, using the same moments across settings within each model. We score State attribution at moments where State and Manifestation differ in category as correct State, Manifestation mistaken for State, or other errors or omissions. Appendix A.6.2 details the prompts, text-based judging, human verification, and control results. Results and analysis. Both models frequently mistake Man- ifestation for State in their baseline descriptions (Fig. 14). Supplying appraisal records improves correct State attribu- tion from 50% to 60% for GPT-5 and from 40% to 51% for Qwen3-VL. For GPT-5, appraisal CoT offers little improve- ment, whereas supplied records yield more correct State attributions with the same share of other errors or omissions. For Qwen3-VL, appraisal CoT raises correct State attribution to 47%, but surface reading remains at 26%. Supplied records reduce surface reading to 2%, while other errors or omissions increase to 47%. Avoiding the displayed category therefore does not always mean identifying the felt state correctly. These results support appraisal records for improving State attribution, although other errors remain. 6.6 Component Ablation Setup. We assess the contributions of evidence provision, appraisal-based reasoning, and structured execution on the same fixed list of 150 items per task (stratified sampling in Appendix A.5). These sets differ in size from the main evaluation. Think denotes test-time reasoning, with minimal effort used in TRACER. Frames and Subs are the raw LI et al.: COGNITION-ORIENTED EMOTION TRACING15 Condition (external reality) Affect (manifestation) Effect (physical) 0 5 10 15 20 Fabricated claims (%) ↓ 17.3 3.3 GPT-5 (thinking) + Observations + RT + RT + PDD (w/o App.) TRACER (RT + PDD) Fig. 13. Fabrication rates by chain stage. RT and PDD correspond to the topology and derivation modules in Section 5.4. RT includes supplied appraisals, removed in the w/o App. configuration. 0255075100% 50% 19% 31% GPT-5 (thinking) 52% 16% 33% + appraisal CoT 60% 7% 33% + appraisal records 40% 27% 33% Qwen3-VL-32B 47% 26% 27% + appraisal CoT 51% 2% 47% + appraisal records Correct state ( ↑) Display mistaken for state ( ↓) Other errors / omissions ( ↓) Fig. 14. Appraisal interventions for State attribution. Display mistaken for state means identifying the outward display as the felt state when they differ. All settings share 58 moments for GPT-5 and 55 for Qwen3-VL. TABLE 7 Component ablation. Filled circles indicate enabled components (open Think: minimal reasoning). RT and PDD denote task-conditioned reasoning topology and premise-declared derivation, respectively. T2 uses the revised regulation task, as in Table 2. ThinkFramesSubsObsAppCoTRTPDDT1T2T3T4T5Avg. GPT-5 (non-thinking)◦● ◦ ◦ ◦ ◦ ◦56.9239.5370.8455.0057.6255.98 GPT-5 (thinking)● ◦ ◦ ◦ ◦ ◦56.8443.4975.3957.7360.8958.87 w/ Appraisal-based CoT● ◦ ◦ ● ◦ ◦55.6247.3980.0158.8661.1560.61 w/ Observations● ◦ ◦ ● ◦ ◦ ◦ ◦50.6346.8477.5247.0759.0256.22 w/ Observations & Appraisals ● ◦ ◦ ● ● ◦ ◦ ◦52.5648.3779.7354.2062.8359.54 w/ RT● ◦ ◦ ● ● ◦ ● ◦53.7751.1280.3855.4561.4260.43 w/ RT◦ ◦ ◦ ● ● ◦ ● ◦56.5249.7674.9753.3959.6858.86 w/ RT + PDD (TRACER)◦ ◦ ◦ ● ● ◦ ● ●61.7152.1283.7659.9363.3964.18 video channels. Obs is the observation inventory used in their place. App supplies appraisal records, whereas CoT prompts the model to generate appraisal-based reasoning within its answer call. RT supplies the reasoning topology of Section 5.4 as guidance on construct dependencies and the use of observation and appraisal premises. PDD implements that topology by generating conclusion nodes in dependency order, with explicit premise references and inferred field values. Configurations using Obs and App share the cor- responding records with TRACER. All configurations are evaluated under the same judge. Effect of supplied material. Replacing raw video input with observation records improves T2 and T3 but lowers T1 and T4, with losses in intensity, facial expression, and the recipient’s affective reaction. Adding appraisal records raises every task over observations alone, most on T4 (7.1 points) and T5 (3.8), and lifts the average above GPT-5 (thinking). Observation records alone therefore produce task- dependent gains and losses, while adding appraisals yields a net improvement over the raw-input baseline. Effect of appraisal reasoning. We test whether prompting GPT-5 to reason through cognitive appraisals within its answer call is sufficient to improve affective understanding. Appraisal-based CoT raises the average score from 58.87 to 60.61, but the gains are task-dependent: T2–T4 improve, T1 declines, and T5 remains nearly unchanged. It still trails RT + PDD by 3.6 points, suggesting that appraisal prompting alone does not recover the full method’s gains. Effect of RT. We examine whether providing the task- conditioned dependency structure as guidance improves reasoning. With the same observation and appraisal records, adding RT yields a modest average gain of 0.89 points and does not improve all tasks. Performance drops further when reasoning effort is reduced to minimal. These results suggest that supplying the dependency structure as guidance alone offers limited benefits. Effect of PDD. PDD turns the topology into an explicit execution procedure. Holding the observation records, ap- praisal records, RT, and reasoning effort fixed, adding PDD improves all five tasks and raises the average score from 58.86 to 64.18. This comparison supports explicitly executing the topology, with intermediate constructs derived in dependency order and carried forward as premises, rather than merely presenting it as guidance. 6.7 Limitations and Future Work TRACE-Bench uses subject-centered affective episodes as its annotation and evaluation unit, linking each episode’s LI et al.: COGNITION-ORIENTED EMOTION TRACING16 causes, affect, and consequences within the surrounding scene. Although a video may contain several annotated episodes, the current tasks do not explicitly evaluate depen- dencies between successive episodes or changes in regulation strategy across them. Future work can extend the annotations and tasks to examine how earlier affective experiences shape later appraisals and regulation over longer interactions. At the method level, TRACER reasons over a fixed obser- vation record extracted before derivation. This representation improves evidence accessibility, but it can omit fine-grained cues that later become important for a particular construct. Our ablations reflect this trade-off: replacing raw inputs with observation records improves some tasks while reducing performance on affect intensity, facial expression, and the recipient’s affective response. A promising extension is to make evidence acquisition adaptive, allowing the reasoning process to revisit relevant video intervals when the current record is insufficient to resolve a construct. Future work could also reduce the computational overhead of observa- tion extraction, appraisal, and derivation while preserving evidence grounding. 7 CONCLUSION We presented TRACE, a framework for studying affective understanding as a cognition-shaped process in complex social video. The Affective Blueprint formalizes a subject- centered affective episode through Condition, Affect, and Effect, and TRACE-Bench operationalizes this formulation through five tasks spanning single-stage analysis, cross- stage reasoning, and full-chain reconstruction. Evaluation across affect-specialized, open-source general-purpose, and closed-source general-purpose MLLMs shows that current performance remains substantially below human perfor- mance, and that stronger emotion recognition does not reliably extend to understanding regulation, causes, effects, or complete affective chains. Error analyses further expose two persistent weaknesses: surface reading of outward behavior and factual drift during chain reconstruction. To provide a baseline for this setting, we developed TRACER, which combines cognition-grounded appraisal with premise- declared derivation. TRACER improves performance across all five tasks, with the largest gains in regulation decoding and cause reasoning, and substantially reduces unsupported content in full-chain reconstruction. These results suggest that, as multimodal models become increasingly capable of recognizing affective signals, the cognitive processes underlying affect deserve greater attention in both modeling and evaluation. REFERENCES [1] Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, “A survey of affect recognition methods: Audio, visual, and spontaneous expressions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 1, p. 39–58, 2009. [2]C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: Interactive emotional dyadic motion capture database,” in Language Resources and Evaluation, vol. 42, no. 4. Springer, 2008, p. 335–359. [3]D. Kollias et al., “ABAW: Valence-arousal estimation, expression recognition, action unit detection & multi-task learning challenges,” in Proceedings of CVPR Workshops, 2023. [4] A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages,” in IEEE Intelligent Systems, vol. 31, no. 6, 2016, p. 82–88. [5]S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “MELD: A multimodal multi-party dataset for emotion recognition in conversations,” in Proceedings of ACL, 2019, p. 527– 536. [6]N. Majumder, S. Poria, D. Hazarika, R. Mihalcea, A. Gelbukh, and E. Cambria, “DialogueRNN: An attentive RNN for emotion detection in conversations,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, p. 6818–6825, 2019. [7]R. Xia and Z. Ding, “Emotion-cause pair extraction: A new task to emotion analysis in texts,” in Proceedings of ACL, 2019, p. 1003– 1012. [8] S. Poria, N. Majumder, D. Hazarika, D. Ghosal, R. Bhardwaj, S. Y. B. Jian, P. Hong, R. Ghosh, A. Roy, N. Chhaya, A. Gelbukh, and R. Mihalcea, “Recognizing emotion cause in conversations,” Cognitive Computation, vol. 13, p. 1317–1332, 2021. [9]Z. Lian, H. Sun, L. Sun, H. Gu, Z. Wen, S. Zhang, S. Chen, M. Xu, K. Xu, K. Chen, L. Chen, S. Liang, Y. Li, J. Yi, B. Liu, and J. Tao, “Explainable multimodal emotion recognition,” arXiv preprint arXiv:2306.15401, 2023. [10] Z. Lian, H. Chen, L. Chen, H. Sun, L. Sun, Y. Ren, Z. Cheng, B. Liu, R. Liu, X. Peng, J. Yi, and J. Tao, “AffectGPT: A new dataset, model, and benchmark for emotion understanding with multimodal large language models,” in Proceedings of ICML, vol. 267, 2025, p. 36 993– 37 014. [11]L. Wang, X. Yang, S. Feng, D. Wang, Y. Zhang, and Z. Zhang, “Gen- erative emotion cause explanation in multimodal conversations,” in Proceedings of ACM ICMR, 2025. [12]Y. Lin, J. Sun, Z.-Q. Cheng, J. Wang, H. Liang, Z. Cheng, Y. Dong, J.-Y. He, X. Peng, and X.-S. Hua, “Why we feel: Breaking boundaries in emotional reasoning with multimodal large language models,” in Proceedings of the IEEE/CVF CVPR Workshops, 2025, p. 5235–5245. [13] Z. Cheng, Z.-Q. Cheng, J.-Y. He, J. Sun, K. Wang, Y. Lin, Z. Lian, X. Peng, and A. G. Hauptmann, “Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning,” in Advances in Neural Information Processing Systems, vol. 37, 2024. [14]J. Hu, H. Shi, C. Dai, Z. Li, P. Song, and M. Wang, “Beyond emotion recognition: A multi-turn multimodal emotion understanding and reasoning benchmark,” in Proceedings of ACM Multimedia, 2025. [15]F. Zhang, Z. Cheng, C. Deng, H. Li, Z. Lian, Q. Chen, H. Liu, W. Wang, Y.-F. Zhang, R. Zhang, Z. Guo, Z. Zhu, H. Wu, H. Wang, Y. Zheng, X. Peng, X. Wu, K. Wang, X. Li, J. Ye, and P.-A. Heng, “MME-Emotion: A holistic evaluation benchmark for emotional intelligence in multimodal large language models,” in International Conference on Learning Representations, 2026. [16] Y. Zhang, X. Yang, X. Xu, Z. Gao, Y. Huang, S. Mu, S. Feng, D. Wang, Y. Zhang, K. Song, and G. Yu, “Affective computing in the era of large language models: A survey from the NLP perspective,” Knowledge-Based Systems, vol. 337, p. 115411, 2026. [Online]. Available: https://doi.org/10.1016/j.knosys.2026.115411 [17]R. S. Lazarus, Emotion and Adaptation.Oxford University Press, 1991. [18] K. R. Scherer, “Appraisal considered as a process of multilevel sequential checking,” in Appraisal Processes in Emotion: Theory, Methods, Research, K. R. Scherer, A. Schorr, and T. Johnstone, Eds. Oxford University Press, 2001, p. 92–120. [19]—, “The dynamic architecture of emotion: Evidence for the component process model,” Cognition and Emotion, vol. 23, no. 7, p. 1307–1351, 2009. [20] A. Moors, P. C. Ellsworth, K. R. Scherer, and N. H. Frijda, “Ap- praisal theories of emotion: State of the art and future development,” Emotion Review, vol. 5, no. 2, p. 119–124, 2013. [21]K. R. Scherer and A. Moors, “The emotion process: Event appraisal and component differentiation,” Annual Review of Psychology, vol. 70, no. 1, p. 719–745, 2019. [22] J. J. Gross, “The emerging field of emotion regulation: An integra- tive review,” Review of General Psychology, vol. 2, no. 3, p. 271–299, 1998. [23]G. A. Van Kleef, “How emotions regulate social life: The emotions as social information (easi) model,” Current Directions in Psychologi- cal Science, vol. 18, no. 3, p. 184–188, 2009. [24]H. Rashkin, M. Sap, E. Allaway, N. A. Smith, and Y. Choi, “Event2Mind: Commonsense inference on events, intents, and reactions,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018, p. 463–473. LI et al.: COGNITION-ORIENTED EMOTION TRACING17 [25]M. Sap, R. Le Bras, E. Allaway, C. Bhagavatula, N. Lourie, H. Rashkin, B. Roof, N. A. Smith, and Y. Choi, “ATOMIC: An atlas of machine commonsense for if-then reasoning,” in Proceedings of AAAI, vol. 33, 2019, p. 3027–3035. [26]D. Ghosal, S. Shen, N. Majumder, R. Mihalcea, and S. Poria, “CICERO: A dataset for contextualized commonsense inference in dialogues,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022, p. 5010–5028. [27]S. Sabour et al., “EmoBench: Evaluating the emotional intelligence of large language models,” in Proceedings of ACL, 2024. [28]H. Hu, Y. Zhou, L. You, H. Xu, Q. Wang, Z. Lian, F. R. Yu, F. Ma, and L. Cui, “EmoBench-M: Benchmarking emotional in- telligence for multimodal large language models,” arXiv preprint arXiv:2502.04424, 2025. [29]M. Luo, B. Li, S. Xu, S. Zhang, Q. Chen, M. Han, W. Chen, Y. Huang, H. Fei, M.-L. Lee, and W. Hsu, “Unveiling the cognitive compass: Theory-of-mind-guided multimodal emotion reasoning,” in International Conference on Learning Representations, 2026. [Online]. Available: https://arxiv.org/abs/2602.00971 [30]S. Bhattacharyya, E. Kuriabov, L. Craig, T. Dilliraj, R. B. Adams Jr., J. Li, and J. Z. Wang, “Large language models show fragile cognitive reasoning about human emotions,” arXiv preprint arXiv:2508.05880, 2025. [Online]. Available: https://arxiv.org/abs/2508.05880 [31]J. A. Russell, “A circumplex model of affect,” Journal of Personality and Social Psychology, vol. 39, no. 6, p. 1161–1178, 1980. [32]R. Kosti, J. M. Alvarez, A. Recasens, and A. Lapedriza, “Context based emotion recognition using EMOTIC dataset,” IEEE Transac- tions on Pattern Analysis and Machine Intelligence, vol. 42, no. 11, p. 2755–2766, 2020. [33]Q. Yang, D. Bai, Y.-X. Peng, and X. Wei, “Omni-emotion: Extending video MLLM with detailed face and audio modeling for multi- modal emotion analysis,” arXiv preprint arXiv:2501.09502, 2025. [34]D. Li, B. Xing, X. Liu, B. Xia, B. Wen, and H. K ̈ alvi ̈ ainen, “DEEMO: De-identity multimodal emotion recognition and reasoning,” in Proceedings of ACM Multimedia, 2025. [35]Z. Lian, L. Sun, Y. Ren, H. Gu, H. Sun, L. Chen, B. Liu, and J. Tao, “MERBench: A unified evaluation benchmark for multimodal emotion recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 48, no. 5, p. 5793–5810, 2026. [36]F. Wang, Z. Ding, R. Xia, Z. Li, and J. Yu, “Multimodal emotion- cause pair extraction in conversations,” IEEE Transactions on Affective Computing, vol. 14, no. 3, p. 1832–1844, 2023. [37]F. Wang, R. Xia, Z. Ding et al., “Semeval-2024 task 3: Multimodal emotion cause analysis in conversations,” in Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), 2024, p. 1968–1977. [38]Z. Liu, K. Yang, T. Zhang, Q. Xie, and S. Ananiadou, “EmoLLMs: A series of emotional large language models and annotation tools for comprehensive affective analysis,” in Proceedings of ACM SIGKDD, 2024. [39] A. Zadeh, M. Chan, P. P. Liang, E. Tong, and L.-P. Morency, “Social-IQ: A question answering benchmark for artificial social intelligence,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, p. 8807–8817. [40] P. Vicol, M. Tapaswi, L. Castrejon, and S. Fidler, “MovieGraphs: Towards understanding human-centric situations from videos,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, p. 8581–8590. [41] S. Choi, K.-W. On, Y.-J. Heo, A. Seo, Y. Jang, M. Lee, and B.-T. Zhang, “DramaQA: Character-centered video story understanding with hierarchical qa,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 2, 2021, p. 1166–1174. [42]H. Zhang, X. Wang, H. Xu, Q. Zhou, K. Gao, J. Su, J. Zhao, W. Li, and Y. Chen, “MIntRec2.0: A large-scale benchmark dataset for multimodal intent recognition and out-of-scope detection in con- versations,” in International Conference on Learning Representations, 2024. [43]C. Jin, Y. Wu, J. Cao, J. Xiang, Y.-L. Kuo, Z. Hu, T. Ullman, A. Torralba, J. Tenenbaum, and T. Shu, “MMToM-QA: Multimodal theory of mind question answering,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, p. 16 077–16 102. [Online]. Available: https://aclanthology.org/2024.acl-long.851/ [44] H. Shi, S. Ye, X. Fang, C. Jin, L. Isik, Y.-L. Kuo, and T. Shu, “MuMA- ToM: Multi-modal multi-agent theory of mind,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 2, p. 1510–1519, 2025. [45] E. Villa-Cueva, S. M. M. Ahmed, R. Chevi, J. C. B. Cruz, K. Elzeky, F. Cristobal, A. F. Aji, S. Wang, R. Mihalcea, and T. Solorio, “MoMentS: A comprehensive multimodal benchmark for theory of mind,” in Findings of the Association for Computational Linguistics: EMNLP 2025, 2025, p. 22 591–22 611. [Online]. Available: https://aclanthology.org/2025.findings-emnlp.1230/ [46]Z. Chen et al., “ToMBench: Benchmarking theory of mind in large language models,” in Proceedings of ACL, 2024. [47]A. Ortony, G. L. Clore, and A. Collins, The Cognitive Structure of Emotions. Cambridge University Press, 1988. [48]J. J. Gross, “Emotion regulation: Current status and future prospects,” Psychological Inquiry, vol. 26, no. 1, p. 1–26, 2015. [49]P. Ekman and W. V. Friesen, “The repertoire of nonverbal behavior: Categories, origins, usage, and coding,” Semiotica, vol. 1, no. 1, p. 49–98, 1969. [50]A. R. Hochschild, “Emotion work, feeling rules, and social struc- ture,” American Journal of Sociology, vol. 85, no. 3, p. 551–575, 1979. [51]G. A. Van Kleef and S. C ˆ ot ́ e, “The social effects of emotions,” Annual Review of Psychology, vol. 73, p. 629–658, 2022. [52]C. M. de Melo, P. J. Carnevale, S. J. Read, and J. Gratch, “Reading people’s minds from emotion expressions in interdependent deci- sion making,” Journal of Personality and Social Psychology, vol. 106, no. 1, p. 73–88, 2014. [53]A. S. Rao and M. P. Georgeff, “BDI agents: From theory to practice,” in Proceedings of the First International Conference on Multi-Agent Systems. San Francisco, CA: MIT Press, 1995, p. 312–319. [54]ChristophSchuhmann, “movie-clips,” Hugging Face dataset. Avail- able:https://huggingface.co/datasets/ChristophSchuhmann/ movie-clips, 2023, accessed: 2026-07-07. [55]EQ4You, “movieclips,” Hugging Face dataset. Available: https: //huggingface.co/datasets/EQ4You/movieclips, 2024, accessed: 2026-07-07. [56]P. Ekman, “An argument for basic emotions,” Cognition and Emotion, vol. 6, no. 3-4, p. 169–200, 1992. [57]J. Xu, H. Fei, L. Pan, Q. Liu, M.-L. Lee, and W. Hsu, “Faithful logical reasoning via symbolic chain-of-thought,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, p. 13 326–13 365. [Online]. Available: https://aclanthology.org/2024.acl-long.720/ [58]J. Xu, H. Fei, M. Luo, Q. Liu, L. Pan, W. Y. Wang, P. Nakov, M.-L. Lee, and W. Hsu, “Aristotle: Mastering logical reasoning with a logic- complete decompose-search-resolve framework,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, p. 3052–3075. [Online]. Available: https://aclanthology.org/2025.acl-long.153/ [59]B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, and C. Li, “LLaVA-OneVision: Easy visual task transfer,” Transactions on Machine Learning Research, 2025. [60]F. Li, R. Zhang, H. Zhang, Y. Zhang, B. Li, W. Li, Z. Ma, and C. Li, “LLaVA-NeXT-Interleave: Tackling multi-image, video, and 3d in large multimodal models,” arXiv preprint arXiv:2407.07895, 2024. [61]Y. Yao et al., “MiniCPM-V 4.5: Cooking efficient MLLMs via archi- tecture, data, and training recipes,” arXiv preprint arXiv:2509.18154, 2025. [62]GLM-V Team et al., “GLM-4.5V and GLM-4.1V-Thinking: Towards versatile multimodal reasoning with scalable reinforcement learn- ing,” arXiv preprint arXiv:2507.01006, 2025. [63]J. Xu, Z. Guo, H. Hu, Y. Chu, X. Wang, J. He, Y. Wang, X. Shi, T. He, X. Zhu et al., “Qwen3-Omni technical report,” arXiv preprint arXiv:2509.17765, 2025. [64]Z. Chen et al., “InternVL3.5: Exploring advanced training and test- time recipes for open-source multimodal models,” arXiv preprint arXiv:2508.18265, 2025. [65]Qwen Team, “Qwen3-VL technical report,” arXiv preprint arXiv:2511.21631, 2025. [66]D. Huang, Q. Li, C. Yan, Z. Cheng, Z. Han, Y. Huang, X. Li, B. Li, X. Wang, Z. Lian et al., “Emotion-qwen: A unified framework for emotion and vision understanding,” arXiv preprint arXiv:2505.06685, 2025. [67]A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram et al., “Openai gpt-5 system card,” arXiv preprint arXiv:2601.03267, 2025. [68]Google DeepMind, “Gemini 3 Pro model card,” https://deepmind. google/models/model-cards/gemini-3-pro/, 2026, model released November 2025. Updated May 2026. LI et al.: TRACE APPENDIX1 Appendix This appendix provides implementation and experimental details (Appendix A), the scoring protocol (Appendix B), and details of data construction and annotation (Appendix C). Figures, tables, and equations here carry the prefix “S”. APPENDIX A IMPLEMENTATION AND EXPERIMENTAL DETAILS A.1 Inference Settings During inference, we provide all evaluated methods with the same task instructions and subtitle transcripts. Alongside these textual inputs, the models receive frames sampled across the complete clip, with the model-specific budgets listed below. Audio is provided only to Qwen3-Omni-30B. We retain the released inference settings for affect-specialized models. GPT-5 uses the provider’s default reasoning setting for the thinking baseline and minimal reasoning effort for the non-thinking baseline. ModelVideo budget Open-source general-purpose LLaVA-OneVision-7B/70B, LLaVA-NeXT-Video-32B 32 frames InternVL3.5-38B, Qwen3-Omni-30B, Qwen3-VL-32B 60 frames GLM-4.6V-Flash-9B8 frames MiniCPM-V-4.55 fps Closed-source general-purpose GPT-5 (thinking/non-thinking), Gemini-3-Pro8 frames TRACER settings. We use Qwen3-VL-32B to extract one observation record per video and reuse that record across tasks, ablations, and diagnostics, following the procedure in Appendix A.2. We then use GPT-5 for appraisal, anchor alignment, and structured derivation, setting its reasoning effort to low for appraisal and minimal for the latter two operations. For T5, we derive Condition, Affect, and Effect through separate calls. TABLE S3 Answer-generation calls and token usage, averaged over five tasks. ConfigurationCalls Input Output Hidden reasoning GPT-5 (thinking)11.7k1.25k1.06k GPT-5 (thinking) + Obs12.2k1.10k0.90k GPT-5 (thinking) + Obs + App12.9k1.13k0.93k TRACER1.4 17.3k0.71k0 Token use. We measure token use on five questions per task and average equally across the five tasks (Table S3). T1–T4 each use one derivation call, while T5 uses three, giving TRACER an average of 1.4 calls per question. Output counts include provider-reported hidden reasoning tokens. The table reports answer generation; observation records are generated once per video on a local GPU, and appraisal uses 9.2k input and 1.0k output tokens per question. These records are reused across configurations. Each TRACER derivation call includes the method specification and observation record, resulting in higher input token use. (a) Observer prompt (abridged) You record what is VISIBLE and AUDIBLE in a video. You are an observer: you do not interpret, you do not name emotions, you do not judge or assign causes — you record what is seen and heard, faithfully, in time order, across the WHOLE clip. [. . . ] Everything visual [. . . ] is yours to observe from the video, and you should trust your eyes for it. Model-generated speaker/action attribution: dialogue_attribution A moment is a short span where one EVENT holds together — one exchange, one action, one reaction. [. . . ] For roughly moment_budget moments total (a guide, not a quota): cover the whole clip with reasonably EVEN attention [. . . ]a camera cut is NOT a moment boundary. – Do NOT name anyone’s emotion or inner state [. . . ] Record the face and the voice — the literal markers — do not label what they mean. – Do NOT assign roles or causality [. . . ] – Do NOT select for relevance to any later question. Record the whole scene as it is. – Do NOT invent. [. . . ] Record only what is actually observable. Output only the JSON object. [. . . ] Characters from QA questions (subject, object, recipient): roster Subtitle transcript (verbatim, with times [. . . ] ): subtitle_text (b) Output format "observations": [ "id": "o1", "time_s_t": [start_seconds, end_seconds], "present_characters": ["[Appearance-Role(Name)]"], "content": "Everything observable in this moment, integrated: [. . . ]" ], "notes": "Optional: [. . . ]" (c) One observation of a record "id": "o2", "time_s_t": [10.0, 20.0], "present_characters": ["[Black Suit-Father]", "[Blonde-Mother]", "[Audience]"], "content": “The camera cuts to a close-up of [Black Suit-Father] as he sits down beside [Blonde-Mother] [. . . ] He turns to her and says, ’Excuse me,’ then ’Hi,’ with a slight smile and a relaxed posture. [. . . ] [Blonde-Mother] turns her head toward him, smiling faintly, then looks back toward the stage. [. . . ] ” Fig. S1. Observation extraction. (a) Abridged observer prompt with simplified input labels; [. . . ] marks omissions and braces denote inserted inputs. (b) Output format. (c) One observation of a record. A.2 Observation Extraction To construct the observation record, we sample each video at 1 fps, up to 256 frames, and provide these frames to Qwen3- VL-32B-Instruct with timestamped subtitles and model- generated speaker/action attribution, using a character roster to align person references. The character roster is compiled from the subject, object, and recipient identifiers provided in the QA questions. Speaker/action attribution is generated by a preceding model stage using the full dialogue. The observation call does not include the full task question. We ask the model to describe 8 to 12 moments depending on clip length, allowing up to 8,192 output tokens. For each moment, it records an identifier, time span, characters, and a description grounded in the frames and subtitles. The prompt and output example in Fig. S1 show how these records are organized, with emotion labels, causal statements, and selection by relevance excluded. LI et al.: TRACE APPENDIX2 TABLE S1 T1 affect recognition. State and Manifestation totals are computed within each item. Affective StateManifestation ModelCat.Desc.Pol.Int.Obj.AllFacialBodyVerbalVocalAll Affect-specialized Emotion-LLaMA1.40.11.11.40.00.80.10.10.10.10.1 AffectGPT21.823.571.550.256.444.827.722.815.631.824.4 Emotion-Qwen27.2 * 11.757.535.451.836.830.020.416.023.322.1 Open-source general-purpose LLaVA-OneVision-7B31.616.374.451.872.849.527.919.114.221.520.7 LLaVA-NeXT-Video-32B27.612.379.544.335.239.915.010.010.312.111.9 MiniCPM-V-4.537.227.265.748.960.848.028.525.127.029.126.6 GLM-4.6V-Flash-9B29.435.977.057.263.052.634.320.434.828.729.0 LLaVA-OneVision-70B40.136.880.1 61.167.757.332.923.131.027.528.2 Qwen3-Omni-30B42.543.676.159.252.554.942.032.140.438.937.7 InternVL3.5-38B44.0 37.077.361.474.759.040.027.530.331.031.7 Qwen3-VL-32B38.848.181.464.267.860.147.236.644.843.041.8 Closed-source general-purpose GPT-5 (non-thinking)36.155.669.164.573.059.756.946.260.152.452.6 Gemini-3-Pro48.153.781.458.274.7 63.348.537.355.445.645.4 GPT-5 (thinking)41.557.371.363.362.559.259.247.261.654.254.0 Ours TRACER40.560.077.566.579.964.957.755.670.656.658.4 TABLE S2 Field-level results for T2 regulation decoding, T3 cause reasoning, and T4 effect reasoning. T2 RegulationT3 CauseT4 Effect ModelTacticTargetGoalEvidenceExternalInternalMentalAffectivePhysical Affect-specialized Emotion-LLaMA10.2 * 5.64.31.52.5 † 0.20.0 † 0.60.2 AffectGPT7.0 † 1.96.21.619.6 * 10.812.4 * 22.112.6 Emotion-Qwen36.7 * 28.013.510.024.4 * 9.12.119.615.5 Open-source general-purpose LLaVA-OneVision-7B45.415.923.25.226.917.92.36.92.9 LLaVA-NeXT-Video-32B47.98.428.94.128.9 * 15.323.133.915.2 MiniCPM-V-4.540.932.740.113.043.836.432.537.417.9 GLM-4.6V-Flash-9B44.65.636.96.443.537.241.843.48.2 LLaVA-OneVision-70B46.95.630.18.749.137.038.140.723.3 Qwen3-Omni-30B47.422.446.517.652.347.947.447.428.6 InternVL3.5-38B47.45.630.38.547.539.936.441.721.5 Qwen3-VL-32B48.926.251.118.160.857.953.549.034.6 Closed-source general-purpose GPT-5 (non-thinking)47.428.054.823.6 72.962.659.457.843.3 Gemini-3-Pro49.531.651.218.171.665.858.852.645.9 GPT-5 (thinking)50.946.756.223.678.372.060.556.846.9 Ours TRACER53.837.766.550.387.176.464.557.058.0 A.3 Appraisal Prompt For each appraisal request, we provide the observation record and specify the character and the group of items to interpret. The model returns one evaluative proposition for each of the four dimensions. During subsequent derivation, it selects which of these propositions to cite as premises. Fig. S2 reproduces the core instructions, omitting the surrounding method description and item-specific input. A.4 Field-Level Results Tables S1–S4 extend the selected comparisons in Section 6.4 of the main paper to all evaluated models. We average each field’s score over the items that query it, whereas task scores are aggregated within each item before averaging across items. An asterisk flags parsing rates below 90% and a dagger rates below 50%; failed outputs score zero in both cases. The parsing rates and the means over parsed outputs are provided in Appendix B.3. A.5 Ablation and Model-Family Comparisons Sampling. The ablation uses a fixed list of 150 items per task. We divide the per-item scores of GPT-5 (thinking), on a 0–1 scale, into five equal-width bands:[0, 0.2),[0.2, 0.4), [0.4, 0.6),[0.6, 0.8), and[0.8, 1]. For each task, the 150 items LI et al.: TRACE APPENDIX3 Appraisal instructions You are the appraiser. Stance: For each request below, put yourself in that person’s own position, with their standing, relationships, concerns, and stakes in this scene, and evaluate the situation at the anchored moment, one reading per requested dimension. What each dimension’s reading must state: • relevance: which goal, need, or attachment of this person the event touches, and how much is at stake for them. •implication: who or what caused it and with what intent, whether it helps or obstructs this person’s goals, and how certain and urgent it is. •coping: whether this person can influence the event (control), whether they have the strength or standing to do so (power), and if not, whether they could live with the outcome (adjustment). •normative: whether it violates or surpasses a standard this person holds or their community enforces, and what the social setting demands of their visible behavior. Grounding: • basis states the concrete facts the reading rests on, including relationship and status facts when they matter, with supporting ids in observation_ids. • reading is one evaluative proposition from this person’s own position, on the requested dimension only. Boundaries: • Do not describe expressive channels for their own sake. • Do not emit emotion labels or answer fields. • Do not answer for any other person or any dimension that was not requested. Fig. S2. Core appraisal instructions. Wording is retained from the recorded prompt, with formatting adjusted. TABLE S4 T5 full-chain reconstruction. Full is the overall T5 score. ModelCond. State Manif.Reg. EffectFull Affect-specialized Emotion-LLaMA0.00 † 1.380.000.000.000.20 AffectGPT31.36 * 47.280.341.329.89 20.87 Emotion-Qwen14.09 * 45.6923.20 12.0911.18 19.13 Open-source general-purpose LLaVA-OneVision-7B1.95 † 15.866.653.173.995.47 LLaVA-NeXT-Video-32B13.80 * 28.5812.35 13.677.71 13.80 MiniCPM-V-4.526.95 58.1428.59 12.4021.22 29.43 GLM-4.6V-Flash-9B33.43 57.8243.43 15.0016.57 32.36 LLaVA-OneVision-70B33.49 64.1039.28 32.0820.39 34.62 Qwen3-Omni-30B43.93 64.1746.62 33.1127.76 41.78 InternVL3.5-38B36.99 64.9342.47 17.0824.28 37.07 Qwen3-VL-32B51.59 69.6155.83 40.6438.21 50.16 Closed-source general-purpose GPT-5 (non-thinking)66.91 70.3462.01 42.5543.80 58.22 Gemini-3-Pro63.67 73.2258.72 21.4643.25 56.05 GPT-5 (thinking)70.2970.5463.14 41.0947.7460.79 Ours TRACER72.18 71.8863.0842.1853.84 63.59 are allocated across these bands in proportion to their shares in the full pool. Model-family normalization. For the three baseline families in Fig. 6b of the main paper, letS g,t be the mean score of familygon taskt, excluding TRACER. The heatmap uses double-centered scores: P g,t = S g,t − ̄ S g − ̄ S t + ̄ S,(S1) where ̄ S g , ̄ S t , and ̄ Sare the family, task, and grand means. Positive values indicate relative strengths. The Mean row reports the uncentered family scores. TABLE S5 T5 fabrication rates by stage (%, lower is better). ConfigurationCond.AffectEffectAllUnver.Mispl. GPT-5 (thinking)3.112.217.311.012.69.5 + Observations1.87.77.16.312.66.5 + RT1.67.43.75.513.05.0 + RT + PDD ‡ 2.47.04.05.013.53.6 TRACER3.06.23.34.712.93.9 Unver.: unverifiable, not ranked. Mispl.: misplaced. ‡ Without appraisal records. A.6 Diagnostic Experiments A.6.1 Factual Drift Assessment. To diagnose factual drift, we audit 460 answers to 92 T5 items under the five configurations in Table S5, checking External Reality, the four Manifestation channels, and Physical Effect against video evidence. For each answer, we provide an agent usingclaude-opus-5with timestamped subtitles without speaker labels and a common frame grid. The agent can read files and extract additional frames as needed, inspecting an average of 23 time points per answer. Observation records help locate relevant moments and align speaker references. Events are verified against video frames and subtitles. We conceal configuration iden- tities with randomized path codes and withhold reference answers, answers from other configurations, and audio. Decision rule. The agent splits each field into atomic factual statements and assigns one of four labels using the rubric in Fig. S3: LabelCriterion S: SupportedEvidence confirms the event and character. L: MisplacedThe statement describes another character or moment. F: FabricatedRelevant frames or subtitles cover the claimed content but do not support it. U: Unverifiable Off-screen events or missing audio or dia- logue prevent verification. Writing n S ,n L ,n F ,n U for the label counts, we report Fabrication rate = n F n S + n L + n F + n U . Across the audited answers, 10,579 statements enter this denominator. We treat empty fields as omissions rather than statements and exclude off-topic answers, which also yield no statements, from the fabrication rate. The single off-topic answer is reported separately (1 of 460). Human verification. Each answer receives one primary agentic assessment. Human reviewers reassess 64 distinct an- swers, including 20 with multiple reassessments in the pilot phase. Pairing each reassessment with its primary assessment gives 105 pairs and 630 field-slot comparisons. Judgments on whether a slot contains fabricated content agree in 570 comparisons (90.5%,κ = 0.59). These comparisons are not independent, since some answers contribute multiple pairs. Paired comparison. Pairing fabrication rates by item, the GPT-5 (thinking) minus TRACER difference is 6.3 percentage points (t = 6.69). The smaller differences from + RT and LI et al.: TRACE APPENDIX4 Audit rubric You audit ONE model answer against the VIDEO (frames you pull yourself plus the subtitle lines). Observation records help locate relevant moments and align speaker references. Events are verified against video frames and subtitles. Start from a uniform frame grid shared across configurations, then inspect additional frames at relevant timestamps as needed. Do not judge quality; do not use any reference answer; you do not know which configuration produced it. Six slots: external_reality (s1), facial_expression (s2), body_language (s3), vocal_cues (s4), verbal_content (s5), physical_effect (s6). Ignore affective_state, internal_driver, goal, mental_effect, affective_effect, and purely mental sentences. Split each slot into atomic factual claims (one event per claim, name the person). An empty,None,<TBI>, or purely vague slot isomitted: truewith no claims. Omission is not fabrication. Windows: s1 = before or at the anchored moment, any actor, event must bear on the subject. s2–s5 = around the anchored moment (anchor observations and their immediate neighbours), the subject only. s6 = at or after the anchored moment (the anchor observations themselves count as inside the window), the recipient named in the answer, who must be present in the record. Windows follow observation order, not exact seconds. Off-topic gate first: a different person or a clearly different moment→ off_topic: true, no slots. Off-topic answers stay in the denominator. Per claim: Q1 do the frames or the subtitles contain this event, action, or line? Restatements count, inferences do not. Present only in the record, and the window is genuinely uncovered (off-screen, no audio/subtitle coverage, subject absent)→ U with unverifiable_reason; if the modality covers the window and the content is not there→ F. Absent everywhere→ F, evidence no match in frames, subtitles or record. Q2 is the actor allowed for this slot and is the event inside the slot’s window? No→ L (quote the line, say which person or position). Yes→ S (quote the subtitle line id or the frame timestamp; set support to frames / subtitles / both and window to anchor / neighbour). Rules: vocal claims (volume, tone, tremor) are S only with a textual cue in the subtitles (exclamation, repetition, stammer, pace from timing), otherwise U with reason no_audio — a record description of the voice is never sufficient; motion claims may be judged from change across adjacent grid frames. Degree words never flip a label; opposite direction is F; a visible reaction in s6 needs a visible basis; unattributed dialogue is attributed via the co-timed observation; when uncertain between S and F, choose F. Before labelling any F or L, view frames from that claim’s time window (pull them yourself if the shipped frame grid do not cover it) and list every timestamp viewed in frames_viewed. All six slots must appear. Return the audit results in JSON format. Fig. S3. Rubric of the unsupported-content audit. Slots s1–s6 are the audited fields. Labels: S supported, L misplaced, F fabricated, U unverifiable. The off-topic denominator refers to the answer-level off-topic rate, not the statement-level fabrication rate. (a) Description prompt You are shown frames from a short video clip in temporal order, together with the dialogue transcript with speaker labels. Describe this clip from an emotional perspective, in 150-250 words of plain prose, covering: 1. What happens between the people: who says or does what to whom, and what triggers what. 2. What each main person feels, and why. 3. How each person’s feelings and reactions affect the others in the scene. Base your description only on this clip. Refer to people using the names or descriptions from the transcript. TRANSCRIPT: transcript (b) Appraisal records, appended to (a) APPRAISAL RECORDS (first-person readings of the main people in this clip at the key moments, produced by a separate analysis; treat them as additional evidence about each person’s situation): - person: [moment] dimension: reading (c) Appraisal CoT, prepended to (a) Before writing, put yourself in the position of each main person in the clip, with their standing, relationships, concerns, and stakes in this scene. For each such person work through four dimensions, one short reading each, grounded in what you see and hear: - relevance: which goal, need, or attachment of this person the events touch, and how much is at stake for them. - implication: who or what caused it and with what intent, whether it helps or obstructs this person’s goals, and how certain and urgent it is. - coping: whether this person can influence the events, whether they have the strength or standing to do so, and if not, whether they could live with the outcome. - normative: whether it violates or surpasses a standard this person holds or their community enforces, and what the social setting demands of their visible behaviour. Write these readings first under the heading APPRAISAL (they do not count toward the word limit), then write the description under the heading DESCRIPTION. Fig. S4. Prompts of the surface-reading diagnostic. (a) Description prompt. (b) Block appended in the appraisal-records setting. (c) Instruction prepended in the appraisal-CoT setting. + RT + PDD (w/o App.) are not statistically significant. Unverifiable rates remain similar across configurations. A.6.2 Surface Reading Assessment. We ask models to describe each clip using the prompts in Fig. S4, without directing them to regulation or a particular moment. From these descriptions, we assess State attribution at moments where State and Manifestation differ in category, using unregulated moments as controls. We give a judge usingclaude-opus-5the description and annotated State and Manifestation categories, but no video or subtitles, and conceal model and configuration identities with randomized path codes. Decision rule. For each character–moment, the judge assigns one outcome: •Correct State: the description identifies the annotated State, including a synonym or a main component of a mixed state. Naming both State and Manifestation also counts as correct. •Surface reading: Manifestation is mistaken for State. This outcome does not apply to unregulated controls. • Other/omitted: another error or no attribution. Human verification. After the Opus judge assigns one classification to each character–moment, a human reviewer independently re-evaluates a subset drawn from the compar- ison and intervention sets. The two judgments agree on 91 of 98 descriptions (92.9%). Table S6 reports regulated moments and unregulated controls separately. Table S7 extends the intervention com- parison to four models. For the open-source models, reduced surface reading is accompanied by an increase in other errors or omissions, so it does not translate fully into correct State attribution. LI et al.: TRACE APPENDIX5 0:01 Male guest “Hey, you guys ever see that show, uh, Trading Spouses?” 0:06 Subject “Oh, God, no.” 0:08 Subject “Who even watches that garbage?” 0:11 Host “It’s my new guilty pleasure.” 0:16 Subject “Oh, God, it’s all so predictable.” 0:18 Subject “We’ve paired a vegan to live with a butcher.” Reference annotation Condition The male guest jokes about a reality TV show. The subject expects the others “to share her scorn.” State Disgust, Medium, Negative. Manifestation “Exaggerated laughter followed by a look of distaste and a sarcastic smirk.” Regulation Substitution, target state Happiness. Effect The host “awkwardly admits that the show is her ‘guilty pleasure’.” Without appraisal records GPT-5 (thinking) “recoils with mock disgust, which sets a lightly teasing tone” Qwen3-VL-32B “the woman’s expression shifts from laughter to thoughtful amusement” ✗ Surface reading With appraisal records Records “He is intentionally steering us toward a trashy TV topic, which obstructs my preference for a more worthwhile exchange” . . . “it publicly invites judgment of anyone present who likes the show.” GPT-5 (thinking) “immediately dismisses it as ‘garbage,’ signaling disdain and a higher standard for conversation.” Qwen3-VL-32B “reacts with immediate disdain” ✓ Correct State Fig. S5. Appraisal intervention on a regulated moment. Top: frames and dialogue of the annotated episode, with the focal utterance in bold. Bottom: the reference annotation and excerpts of the model descriptions without and with appraisal records. Without the records, both models take the laughter at face value. With them, both identify the annotated State. TABLE S6 State attribution with and without regulation (%). Momentn Correct State↑ Surface reading↓ Other/ omitted GPT-5 (thinking) Regulated5850.019.031.0 Unregulated1050.0–50.0 Qwen3-VL-32B Regulated5837.927.634.5 Unregulated1090.0–10.0 AffectGPT Regulated5211.51.986.5 Unregulated1050.0–50.0 Surface reading: Manifestation mistaken for State. Other/omitted: other errors or omissions. –: not applicable. TABLE S7 Effect of appraisal on state attribution (%). Settingn Correct State↑ Surface reading↓ Other/ omitted GPT-5 (thinking) Baseline5850.019.031.0 Appraisal CoT5851.715.532.8 Appraisal records5860.36.932.8 Qwen3-VL-32B Baseline5540.027.332.7 Appraisal CoT5547.325.527.3 Appraisal records5550.91.847.3 InternVL3.5-38B Baseline5538.221.840.0 Appraisal records5543.69.147.3 GLM-4.6V-Flash Baseline5538.218.243.6 Appraisal records5547.33.649.1 Baseline: no appraisal intervention. Fig. S5 illustrates the role of appraisal in interpreting the same observable cue. The supplied records connect the guest’s preference for a more worthwhile conversation with the social cost of criticizing it. Both models then interpret her laughter as softening disdain rather than expressing amusement. A.7 Human Solvability and Judge Calibration Human solvability. We select 50 questions per task from 250 distinct videos using reference annotations and video metadata, without consulting model predictions or evalua- tion results. We stratify by the first annotated affect category for T1, regulation tactic for T2, whether Internal Driver is queried for T3, the queried Effect channel (mental, affective, or physical) for T4, and whether the chain involves regulation for T5. Sampling follows the eligible pool’s stratum propor- tions. Strata expected to contribute fewer than two questions are merged before selection, without additional weighting. Videos used in other paper analyses, including annotation- agreement assessment and case studies, are excluded. The seed, stratum proportions, exclusions, and ordered sample list are frozen before answer collection. Unavailable items are replaced by the next eligible item on the list, with skips and replacements logged. Two human participants independently answer each question using the same inputs as the models. Table S8 compares their mean scores with model scores on the same questions. Their task and component means also appear in Table 2 and Fig. 6a of the main paper. Judge calibration. To compare automated scores with hu- man judgments, we sample 100 open-text fields, stratified by model, field type, and judge score, and ask a blinded human rater to score them from 0 to 4. We multiply the judge scores LI et al.: TRACE APPENDIX6 0 (n = 16) 1 (n = 17) 2 (n = 35) 3 (n = 25) 4 (n = 7) Judge score 0 1 2 3 4 Human rating Mean human rating Equal scores Fig. S6. Human–judge calibration on 100 free-text fields. Points show mean human ratings, with group sizes below the axis. The dashed line indicates equal scores. TABLE S8 Human and model performance on the same 50 questions per task. Human scores average two participants; Avg. averages the five tasks. Model / HumanT1T2T3T4T5Avg. Human83.6072.3285.5578.9075.4079.15 GPT-5 (thinking)57.9741.8775.8057.6160.4858.75 GPT-5 (non-thinking)55.2136.0068.1261.0659.0955.89 Gemini-3-Pro55.5941.5368.0060.1954.1855.90 Qwen3-VL-32B50.5634.2663.6649.0050.6849.63 Qwen3-Omni-30B49.3229.3754.3242.6841.5543.45 InternVL3.5-38B48.0028.6444.0534.9435.1638.16 TRACER (Ours)60.7849.9184.3960.2162.5963.58 by four and round them to the same scale before comparing the two sets of ratings. Fig. S6 shows their relationship, and the table below reports agreement statistics with bootstrap 95% confidence intervals. StatisticEstimate95% CI Spearman’s ρ0.65[0.49, 0.78] Quadratic weighted κ0.55[0.41, 0.67] Agreement within one point84%[76%, 91%] Mean human minus judge score0.65[0.45, 0.83] The positive mean difference indicates that the judge assigns lower scores than the human rater. APPENDIX B SCORING PROTOCOL This appendix supplements the evaluation metrics in Sec- tion 4.3 of the main paper, detailing field scores, task aggregation, and the treatment of unparsed outputs. B.1 Field-Level Scoring For an evaluated fieldf, letˆy f andy f denote the prediction and reference. The field score is s f =      F1 set (ˆy f ,y f ), f ∈F cat , 1[ˆy f = y f ],f ∈F label , ̄ d 1,f (1− 1 2 ̄ d 2,f ), f ∈F text . (S2) HereF cat contains Affective State category fields.F label contains polarity, intensity, regulation tactic, and regulation source and target categories.F text contains open-text fields. Label comparisons normalize case and separators. The indicator1[·]is 1 for a match and 0 otherwise. DeepSeek-V4- Flash rates each open-text field three times for T2 and twice for the other tasks.d 1 andd 2 are averaged separately before computing the field score. Semantic adequacy (d 1 ). The judge assesses agreement with the reference in meaning, including the character, focal episode, and information requested by the field. Equivalent wording receives the same credit. TABLE S9 Semantic adequacy d 1 : higher is better. ScoreCriterion 0.00No meaningful match, an opposite claim, or the wrong character or episode. 0.25Only a broad or generic relation to the reference. 0.50 Partial match, with an important detail or role missing or incorrect. 0.75 Correct central meaning, with a secondary omission or harmless addition. 1.00Correct content, characters, and episode, regardless of wording. Redundancy (d 2 ). The judge estimates how much of the answer adds no information relevant to the queried field. This includes repetition within the answer, irrelevant restate- ments of the question, generic filler, and competing guesses. Agreement with the reference and useful supporting details are not redundancy. Answer length itself is not penalized, and factual errors are assessed through d 1 . TABLE S10 Redundancy d 2 : higher is worse. ScoreCriterion 0.00Focused answer with almost no unnecessary content. 0.25Minor repetition, filler, or unnecessary hedging. 0.50About half the answer is uninformative or lists compet- ing guesses. 0.75 Most of the answer is repetition, generic content, or competing guesses. 1.00Almost entirely redundant or noncommittal. The judge assigns each rating on a continuous[0, 1]scale rather than calculating it from word counts. The bars in(S2) denote the mean of the repeated ratings for each dimension. B.2 Task-Level Aggregation Field scores are combined according to the task’s requested outputs, including Regulation Evidence in Task 2. In Task 5, the given affective object is excluded from the State score. The equations below instantiate the hierarchy in (6) of the main paper when all components are annotated: S T1 = 1 2 (S State + S Manifestation ), S T2 = 1 4 (S Tactic + S Target + S Goal + S Evidence ), S T3 = 1 2 (S External Reality + S Internal Driver ), S T4 = S requested Effect , LI et al.: TRACE APPENDIX7 S T5 = 1 3 (S Condition + S Affect + S Effect ), S Affect = 1 3 (S State + S Manifestation + S Regulation ).(S3) Applicable fields. The reference determines which compo- nents enter each mean. Two cases must be distinguished: •Not annotated in the reference. Exclude the component and renormalize the remaining weights at that level. • Missing from the prediction. Keep the component’s weight and assign it zero. Item scores are averaged within each task, and the five task means contribute equally to the overall score, as in (6) of the main paper. T2 regulation. The source state is supplied in the question, so the task evaluates the regulation rather than recognizing that state again: •Target. Score it only when the reference target category differs from the supplied source category, that is, for Substitution and Fabrication. Two predicted categories do not match a single reference category. • No regulation. Score tactic alone. HereS Evidence scores Regulation Evidence (M reg ), the man- ifestation cues supporting the regulation judgment. It is a task-specific output, not a separate Blueprint construct. For regulated items, it receives equal weight with the other applicable fields. Neither the evidence score nor the goal score enters the mean for items without regulation. B.3 Parsing and Score Reporting An output that cannot be parsed receives an item score of zero. Table S11 distinguishes the resulting all-item mean from the mean over parsable outputs alone. All reported scores keep parsing failures at zero. The table gives both values for settings below 90% parsing success, with the reported value in bold. TABLE S11 Output parsing rates below 90%. Bold values are reported in Table 2 of the main paper. ModelTaskPars. (%)AllPars. only Emotion-LLaMA * T2615.228.61 Emotion-LLaMA † T3372.266.11 Emotion-LLaMA † T449.50.310.63 Emotion-LLaMA † T570.202.81 AffectGPT † T2174.6427.38 AffectGPT * T37917.9922.84 AffectGPT * T48315.7918.94 AffectGPT * T58720.8723.96 Emotion-Qwen * T18129.4536.58 Emotion-Qwen * T28124.2430.09 Emotion-Qwen * T38221.0425.74 Emotion-Qwen * T57719.1324.80 LLaVA-OneVision-7B † T5295.4718.76 LLaVA-NeXT-Video-32B * T38427.1332.19 LLaVA-NeXT-Video-32B * T55413.8025.77 Pars.: share of parsable outputs. All: unparsable outputs score zero. Pars. only: mean over the parsable outputs. * At least half parsable. † Fewer than half parsable. 0255075100 Fields rewritten (%) Subject 20.04 Characters 16.92 Environment 4.10 Event 54.41 External reality 75.24 Belief 64.65 Desire 63.22 Intention 65.72 Condition Coarse category 75.85 Fine description 80.75 Intensity 27.99 Object 23.98 Polarity 13.90 Affective State Facial expression 65.86 Body language 72.55 Vocal cues 64.97 Verbal content 73.17 Manifestation Source state 46.08 Tactic 37.79 Target state 45.45 Goal 41.89 Regulation Mental 82.07 Affective 75.65 Physical 79.81 Effect Fig. S7. Field-level rewrite rates after human review of retained MLLM- proposed chains. APPENDIX C DATA CONSTRUCTION AND ANNOTATION C.1 Annotation and Human Revision Paired records. To quantify human edits, we pair 1,125 retained MLLM-proposed chains with their raw human records from the annotation archive, before AI polishing. For each field, we restrict the comparison to chains where that field is present in both records. Edit criterion. We count a field as edited when its value differs after ignoring formatting and standardizing empty- value markers. The resulting rates in Fig. S7 therefore include wording changes as well as changes in meaning. C.2 Annotation Consistency Dataset construction involved eight trained annotators. Two annotators independently re-annotated 50 chains excluded from guideline development, using frames and subtitles. We compare these two annotations (A and B) field by field, including only chains for which both provide an applicable answer. Fig. S9 reports the results. Closed fields. We compute Cohen’sκ, with linear weights for the ordered intensity levels (Low, Medium, and High). Category, Tactic, and Intensity each have 50 eligible chains; Polarity has 32 and Target has 29. For Category, we compare the first listed emotion category. For Target, we compare only the emotion category, taking it to match the felt category for Suppression and Amplification. Open-text fields. For each field, we pair the non-empty texts from A and B and randomize their order. A model judge (claude-opus-5), blind to annotator identity, assesses the pairs using a shared rubric: the same content (1), partly overlapping (0.5), or different or contradictory (0). Wording and length may differ without changing the content; adding or omitting a substantive element constitutes partial agree- ment. We report strict agreement, the proportion of eligible pairs rated 1, rather than the mean rating. This measures LI et al.: TRACE APPENDIX8 (a) A response can span consecutive actions Affect Bandmate · Guilt “I was gonna...” Response Interrupts the explanation “Okay, okay...” Continuation Turns to future plans “We can do something bigger...” then One Physical Effect (b) Distinguishing responses to different events Affect Mel · Guilt “I love your suit.” Affective Effect Barrett · Happiness “Oh, thank you.” New event Mia · Dismissal “You’re fired.” Response Barrett · Surprise Different cause Fig. S8. Effect boundaries. (a) Consecutive actions belong to one Physical Effect. (b) Barrett’s later surprise follows the dismissal, not Mel’s preceding Affect. Checks mark supported links, and the cross marks the incorrect attribution. Fig. S9. Agreement between two independent re-annotations. Closed fields: Cohen’sκ(linear weights for Intensity). Open fields: proportion of pairs judged to state the same content. Only κ corrects for chance agreement; the two measures are not directly comparable. agreement on content where both annotations provide text, not on whether a field should be present. Eligible pair counts are 50 each for External Reality, Internal Driver, fine-grained affective description, and Body Language; 49 each for Facial Expression and Vocal Cues; 48 for Verbal Content; 31 each for Regulation Goal and Evidence; and 45, 33, and 46 for Mental, Affective, and Physical Effect, respectively. Counts fall below 50 when a chain lacks the field or either annotator leaves it unfilled. C.3 Field Annotation Guidance The following criteria supplement the annotation procedure in the main paper. Intensity. State intensity is judged from the situation, the person’s concerns, and their response. A calm Manifestation or successful Regulation does not imply low intensity, and high intensity does not require loss of control. The levels are: •Low: mild feeling with little influence on attention or ongoing activity. •Medium: feeling that changes attention or response without dominating the episode. •High: strong feeling that dominates the person’s concerns or creates an urgent need to respond. Mixed polarity. Positive and negative feelings coexist in the same person at the same moment, such as relief with embar- rassment. A mismatch between State and Manifestation, or a change across moments, is not sufficient. Effect. When annotating an Effect, we include connected utterances or actions that respond to the subject’s affective episode, even when they extend beyond the first response. We exclude reactions to a new event, as illustrated by the contrasting cases in Fig. S8. LI et al.: TRACE APPENDIX9 C.4 Task and Answer Formats Following Section 4.1 of the main paper, each question combines episode context (ctx t ), known constructs (K t ), and a JSON template for queried constructs (Q t ). Task 2 stores Regulation Evidence (M reg ) in theevidencefield of regulation_pathway; this is not an additional Blueprint construct. Fig. S10 gives the question and answer formats for all five tasks. Fig. S10. Task and answer templates. Braces indicate item-specific content. Field names follow Table 1 of the main paper. Item identifiers are omitted. Manifestation channels, Regulation, Internal Driver, and Effect channels appear only when annotated. T1 Grounded Affect Recognition Task: Infer the subject’s affective state and observable affective manifestation. Focal Context: Focus on subject. scene and trigger, based on the subject’s observed annotated channels, infer the subject’s internal affective state. Known Structured Fields: - subject: subject Answer Template: "subject": "subject", "affective_state": "object": "string", "coarse_category": ["string"], "fine_description": "string", "polarity": "string", "intensity": "string" , "affective_manifestation": "facial_expression": "string", "body_language": "string", "vocal_cues": "string", "verbal_content": "string" T2 Regulation Decoding Task: Given the subject’s affective state (category and intensity) at the focal moment, decide whether and how the subject regulates its outward manifestation. Report the regulation tactic, the goal, the target affective state as a category, and the evidence observed in the video. Focal Context: Focus on subject. Focal window: m:s-m:s. Known Structured Fields: - source_subject: subject - affective_state.object: object - affective_state.coarse_category: category - affective_state.intensity: intensity Answer Template: "subject": "subject", "regulation_pathway": "tactic_type": "string", "goal": "string", "target_affect_state": "string", "evidence": "string" T3 Affective Cause Reasoning Task: Given the subject’s affective state and manifestation, infer the affective cause requested by the answer template. Focal Context: Analyze the affective cause for subject’s affect toward object. Known Structured Fields: - source_subject: subject - affective_state.object: object - affective_state.coarse_category: category - affective_state.polarity: polarity - affective_state.intensity: intensity - affective_manifestation.facial_expression: text - affective_manifestation.body_language: text - affective_manifestation.vocal_cues: text - affective_manifestation.verbal_content: text - regulation_pathway.tactic_type: tactic - regulation_pathway.goal: text - regulation_pathway.target_affect_state: category - regulation_pathway.evidence: text T3 Affective Cause Reasoning (continued) Answer Template: "subject": "subject", "affective_cause": "internal_driver": "string", "external_reality": "string" T4 Affective Effect Reasoning Task: Given the source full affect stage, infer the requested downstream effect on the specified recipient. Focal Context: Known source affective episode: subject’s full affect toward object. Requested downstream effect: effect type on recipient. Known Structured Fields: - source_subject: subject - source_affective_state.object: object - source_affective_state.coarse_category: category - source_affective_state.fine_description: text - source_affective_state.polarity: polarity - source_affective_state.intensity: intensity - source_affective_manifestation.facial_expression: text - source_affective_manifestation.body_language: text - source_affective_manifestation.vocal_cues: text - source_affective_manifestation.verbal_content: text - source_regulation_pathway.tactic_type: tactic - source_regulation_pathway.goal: text - source_regulation_pathway.target_affect_state: category - source_regulation_pathway.evidence: text - effect_type: effect type - effect_recipient: recipient Answer Template: "subject": "subject", "object": "object", "effect type": "string" T5 Full Chain Reconstruction Task: Reconstruct the complete condition-affect-effect chain for the focal affective moment. Focal Context: Reconstruct the affective chain for subject’s affect related to object in the environment environment Around this focal moment: When name says “line”. Known Structured Fields: - subject: subject - affect_object: object - focal_anchor: Around this focal moment: When name says "line". - given_affective_fields: none; reconstruct the full chain from the focal anchor and multimodal evidence Answer Template: "1_condition": "subject": "subject", "causal_analysis": "internal_driver": "<TBI>", "external_reality": "<TBI>" , "2_affect": "affective_state": "object": "object", "coarse_category": "<TBI>", "fine_description": "<TBI>", "intensity": "<TBI>", "polarity": "<TBI>" , "affective_manifestation": "facial_expression": "<TBI>", "body_language": "<TBI>", "vocal_cues": "<TBI>", "verbal_content": "<TBI>" , "regulation_pathway": "tactic_type": "<TBI>", "goal": "<TBI>", "source_affect_state": "<TBI>", "target_affect_state": "<TBI>" , "3_effect": "mental_effect": "<TBI> on recipient", "affective_effect": "<TBI> on recipient", "physical_effect": "<TBI> on recipient"

Instant research discovery