Paper deep dive
ReactMotion: Generating Reactive Listener Motions from Speaker Utterance
Cheng Luo, Bizhu Wu, Bing Li, Jianfeng Ren, Ruibin Bai, Rong Qu, Linlin Shen, Bernard Ghanem
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/22/2026, 5:17:50 AM
Summary
ReactMotion is a novel framework for generating reactive listener body motions from speaker utterances (text, audio, emotion). It introduces ReactMotionNet, a large-scale dataset with 151,328 pairs featuring one-to-many mappings and tiered appropriateness labels (Gold, Silver, Negative). The framework uses a unified Seq2Seq model with modality-specific tokenizers and preference-based learning to produce diverse, naturalistic listener responses.
Entities (5)
Relation Signals (3)
ReactMotion → trainedon → ReactMotionNet
confidence 95% · Building on this dataset design, we develop preference-oriented evaluation protocols... We further propose ReactMotion
ReactMotion → usesarchitecture → T5-base
confidence 95% · we adopt T5-base as the Seq2Seq backbone
ReactMotionNet → derivedfrom → HumanML3D
confidence 90% · we curate motions from the more dynamic and commonly used HumanML3D dataset
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions.
Tags
Links
- Source: https://arxiv.org/abs/2603.15083v1
- Canonical: https://arxiv.org/abs/2603.15083v1
Trouble viewing inline? Open PDF directly →
Full Text
106,647 characters extracted from source content.
Expand or collapse full text
ReactMotion: Generating Reactive Listener Motions from Speaker Utterance Cheng Luo 1† , Bizhu Wu 2,4,5† , Bing Li 1∗ , Jianfeng Ren 4 , Ruibin Bai 4 , Rong Qu 5 , Linlin Shen 2,3∗ , and Bernard Ghanem 1 1 King Abdullah University of Science and Technology 2 Computer Vision Institute, School of Artificial Intelligence, Shenzhen University 3 Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University 4 School of Computer Science, University of Nottingham Ningbo China 5 School of Computer Science, University of Nottingham, United Kingdom Project page: https://reactmotion.github.io Abstract. In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate nat- uralistic listener body motions that appropriately respond to a speaker’s utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMo- tionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropri- ateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground- truth motion. Building on this dataset design, we develop preference- oriented evaluation protocols tailored to evaluate reactive appropriate- ness, where conventional motion metrics focusing on input–motion align- ment ignore. We further propose ReactMotion, a unified generative frame- work that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and di- verse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, gen- erating more natural, diverse, and appropriate listener motions. Keywords: Dyadic interaction· Interactional AI systems 1 Introduction Modeling dyadic human communication is crucial for virtual agents [33], dig- ital humans [50, 104], and social robots [71]. While prior work has advanced speech-to-speech dialogue [15], language-based interfaces [1,28], and listener fa- cial reactions [57, 70], reactive listener body motions remain largely overlooked † Equal contribution. ∗ Corresponding authors. arXiv:2603.15083v1 [cs.CV] 16 Mar 2026 2C. Luo et al MovingandgreetingWavinghands Ecstatic React Motion I'm so excited you're here! I was hoping you'd show up. Emotion: Input: speakerutterance Output: reactivelistenermotion Fig. 1: Illustration of the proposed new task: Reactive Listener Motion Generation from Speech Utterance. Given a speaker’s utterance, i.e., transcript and/or audio (op- tionally supplemented with emotion), a generative model such as our ReactMotion generates a corresponding responsive body-motion sequence for the listener. despite being central to face-to-face interaction. Listeners often convey engage- ment and understanding through posture and subtle gestures, and generating such feedback is important for natural dyadic communication. We introduce a new task, Reactive Listener Motion Generation from Speech Utterance, which aims to generate naturalistic listener body motions that ap- propriately respond to a speaker’s utterance given its audio and/or transcript. Unlike text-to-motion [21, 62, 75, 76, 96] or audio-driven motion generation [88] that primarily realize the input content, our setting models conversational reac- tions where speaker cues are indirect and the output is inherently one-to-many. This task poses three challenges. (i) The same utterance can elicit multiple valid listener reactions [57,70]. Such non-deterministic listener behaviour poses a significant challenge for modeling the listener’s motion responses. (i) There is no publicly available large-scale dataset with multiple listener-reactive body motions per utterance, to the best of our knowledge. (i) Reactive appropriateness is difficult to evaluate. Metrics based on a single ground truth or motion diversity are insufficient to measure the appropriateness of a listener’s reaction. To address these challenges, we introduce ReactMotionNet, a curated dataset with 151,328 (speaker utterance, listener motion) pairs. Unlike prior motion datasets that typically provide a single target per condition, we associate each utterance with multiple candidate reactions and annotate them into three preference tiers, Gold, Silver, and Negative. This tiered design captures one-to-many ambiguity and enables preference-style supervision and eval- uation [11, 13, 102]. Moreover, we propose a scalable pipeline that re-purposes existing motion data into dyadic speaker-listener pairs for dataset construction, which avoids relying on expensive speaker–listener motion capture. To evaluate reactive appropriateness, we introduce a tier-aware ranking protocol. We train a multimodal judge network to score and rank candidate re- actions under the same speaker input and report win rates against the Gold, Silver, or Negative tiers. This relative evaluation goes beyond single- reference similarity and better reflects that multiple reactions can be ap- propriate for the same utterance. Finally, we propose ReactMotion, a unified generative framework that jointly models speaker transcript, emotion, and audio Abbreviated paper title3 to generate listener motions. We leverage the tiered annotations with preference- based objectives that learn from relative comparisons within each utterance group for the training. Contributions. (i) To the best of our knowledge, we introduce the first task of reactive listener body motion generation from speaker speech in dyadic in- teraction. (i) We present ReactMotionNet, a new dataset with multi-tier (Gold/Silver/Negative) reactive listener motions and a tier-aware evaluation protocol for reactive appropriateness, enabling research on nonverbal listener response behavior. (i) We propose ReactMotion, a unified multimodal gen- erative model that processes multiple speaker cues and generates high-quality listener body motions in response to the speaker. 2 Related Work Human Motion Generation. Human motion generation can be conditioned on diverse modalities, including text [8, 30, 42, 48, 52, 63, 78, 84, 95, 98], action classes [60,64,74], and audio signals such as music [37,39,40,90] or speech [38, 45,85]). Among these, text- and audio-driven motion generation are most related to our setting. Text-based approaches generate motions from explicit action de- scriptions [4,10,18,26,34,61,79,80,97,101,105], while audio-driven methods syn- thesize gestures aligned with temporally synchronized acoustic signals [7,53,99]. Representative modeling paradigms include transformer-based latent models (e.g., [43, 60, 100]), discrete motion tokenization with autoregressive modeling (e.g., [3,9,91,96]), and diffusion-based frameworks (e.g., [2,22,44,76]). Beyond single-person generation, recent works [24, 41, 53, 55, 73, 81] extend motion synthesis to multi-person scenarios. These approaches typically generate multi-person motions by conditioning on explicit textual descriptions of joint actions or on the audio streams of both individuals. In contrast, our problem setting differs in that the target motion is not directly specified by explicit action instructions or synchronized signals. Instead, the model must infer the implicit interaction intention from the speaker’s utterance, including transcript, audio, and emotion cues, and produce a socially appropriate reactive motion for the listener. This requires reasoning over cross-speaker dynamics rather than direct condition-to-motion mapping. Human Reaction Generation. Human reaction generation is crucial for AI in- teraction systems. Spoken language modeling has progressed from cascaded ASR → LLM → TTS pipelines to end-to-end and full-duplex speech-to-speech mod- els [15,66,77,94], while facial reaction generation has advanced from conditional GANs [27] to uncertainty-aware and diffusion-based methods [49,51,57,70,103]. Audio-visual face-to-face dialogue modeling has been explored [14,57,59,103]. In 3D human body modeling, most methods synthesize reactor motion con- ditioned on actor motion [12, 17, 46, 47, 87]. For instance, InterFormer [12] uses temporal-spatial attention in Transformers, and ReGenNet [87] and ReMoS [17] employ diffusion models for full-body motion. Recently, HERO [93] generates 4C. Luo et al 3D reactive motion directly from RGB videos, incorporating the actor’s facial expressions to capture emotional cues. Differently, our method generates 3D re- actor motion from the speaker’s utterance, which includes transcript, audio, and optional emotion annotations. Transcript provides a lightweight, user-friendly modality, audio offers rich vocal cues, and emotion labels explicitly indicate mood, facilitating more effective interaction modeling. 3D Human Body Interaction Datasets. Recent datasets have facilitated research on multi-person dynamics and interaction-aware 3D motion. Several works [20,25,41,86,92] provide paired human motions, modeling interaction as symmetric kinematic coupling, where one participant’s motion is predicted from the other’s. While effective for spatial coordination, this ignores linguistic and affective signals that drive conversation. Other datasets [31, 32, 35, 56, 67, 68, 93] supply silent RGB videos with 3D reactive motions, offering richer context but still lacking speech semantics and emotional cues, which are central to communicative intent. Some datasets [24,36, 55,73] include both audio and motion for human interactions, but the movements of their motions primarily focus on the upper body, such as arms, and are limited to one-to-one speaker-listener pairs. In contrast, our dataset provides a one-to-many mapping between speaker utterances and listener reactive motions. Each utterance has multiple responses labeled gold, silver, and neg for appropriate, partially appropriate, and irrelevant reactions, making it better suited for practical applications. Plus, motions are more dynamic, such as jumping, enabling more diverse body reactions. 3 Task Definition In this paper, we study Reactive Listener Motion Generation in dyadic interac- tion, which consists of a speaker and a listener. Given a speaker utterance C s , the goal is to generate appropriate reactive body motion of the listener, denoted as R l . Formally, the objective is to learn the conditional distribution: p θ R l | C s , C s ∈ n A s , T s , (A s ,T s ), (A s ,E s ), (T s ,E s ), (A s ,T s ,E s ) o . (1) Here, A s denotes the speaker audio, T s is the corresponding textual transcript, E s represents the speaker emotion, and θ denotes the model parameters. As shown in Eqn. 1, C s may consist of different modalities of the speaker utter- ance or their combinations. At inference time, diverse listener reactions can be sampled from p θ (R l | C s ). In contrast to conventional text-to-motion generation, the speaker utterance do not explicitly specify the target listener motion. The mapping from C s to R l is therefore inherently one-to-many, which requires the model to generate motions that are contextually appropriate while maintaining diversity. Abbreviated paper title5 The man plays the violin Audio emotion Person is doing a hand stand. the person is moving his arms like he is arguing with someone. A person is waving hi with his right hand. A person punches with their right hand before they do a counterclockwise spin. Step1:Dyadic Listener Reactive Motion CurationStep2:Inverse Speaker-Condition Inference Step4:Speaker–Listener Candidate Ranking and Preference Tiering Samplecaption-motionpairs fromHumanML3Ddataset Selectdyadicconversation relatedmotion LLMs ✔ ✔ ✖ ✖ Motioncaption A bunch of your old schoolmates just arrived,and they're all looking this way A person is waving hi with his right hand. Speaker utterance LLM Inference Content Speaker emotion TTS Synthetic speaker audio Step3:Data Filtration Synthetic speaker audio Ta r g e t speaker emotion Speechemotion recognition HumeAI Emotion consistency check (keep/discard) Speaker transcript Rank listener candidates by appropriateness for each speaker utterance, and retain the top-ranked ones. Speaker emotion Natural language inference models LLMs + A bee seemed to zip past you just now. That gave me a tiny scare Speaker utterance Someone abruptly steps backward,seemingly surprised or startled by something Listener motion candidates Aperson standing still then suddenly stepping back out of the way Gold Silver Negative Step2:Inverse Speaker-Condition Synthesis Step3:Data Filtering Alllistener motioncaptions Fig. 2: ReactMotionNet dataset construction. We curate dyadic listener motions (Step 1), synthesize speaker conditions via inverse inference and Text-to-Speech (TTS) (Step 2), filter unreliable samples (Step 3), and rank/re-tier speaker–listener pairs into gold/silver/negative preferences (Step 4). 4 ReactMotionNet Dataset To bridge the gap between existing 3D human motion interaction datasets and real-world conversational dynamics, we construct a dataset, ReactMotionNet, featuring one-to-many speaker utterance–listener reaction mappings with graded appropriateness annotations. To construct this dataset, we present a novel data construction pipeline (Fig. 2) that repurposes existing human motion data into speaker–listener motion–response pairs using powerful LLMs [58, 89], thereby avoiding costly data collection. 4.1 Dataset Construction Pipeline Step 1: Dyadic Listener Reactive Motion Curation. Unlike existing audio-driven 3D human interaction datasets, which mainly focus on upper-body movements while standing still, we curate motions from the more dynamic and commonly used HumanML3D dataset [19]. Leveraging the textual captions of motions, we filter out conversation-irrelevant ones (e.g., doing a handstand) using multiple LLM-based verifiers (e.g., ChatGPT-o1 [29], ChatGPT-o3 mini [58]). This step results in a set of motions with reaction-like semantics, which serve as the lis- tener’s reactive motions. Step 2: Inverse Speaker-Condition Synthesis. For each listener motion R l from the last step, we infer multiple plausible speaker utterances that could elicit the observed reaction. Concretely, we input the listener motion’s caption into OpenAI o3-mini [1, 58, 69] to generate potential speaker transcripts T s and as- sociated emotion labels E s . We incorporate emotion into utterance generation, as the speaker’s emotional state influences the listener’s reaction. For example, the same transcript, “Do whatever you want,” can lead to different responses: a 6C. Luo et al Table 1: Dataset statistics. #Pairs is the total number of labeled speaker–listener pairs (i.e. candidate reactions). #Trans., #Audio, and #Emo. denote the numbers of unique transcripts, audio files, and emotion categories, respectively. #Motion is the number of unique motion sequences. #Motion/Utter. reports the average num- ber of candidate motions per speaker utterance. Label counts report the numbers of gold/silver/negative candidates (#G/#S/#N). Split #Pairs Speaker UtteranceListener Reaction #Motion/Utter.Labels (y) #Trans. #Audio #Emo.#Motion(avg.)(#G/#S/#N) Train 137,8796,6316,631461,82220.79 7,527 / 30,862 / 99,490 Val6,790841841401958.07903 / 1,682 / 4,205 Test6,659826826391978.06877 / 1,652 / 4,130 All151,3288,2988,298472,02918.24 9,307 / 34,196 / 107,825 supportive tone may cause the listener to jump happily in place, whereas a frus- trated tone may cause the listener to walk away feeling hurt. Given T s and E s , we synthesize the corresponding speaker audio A s using GPT-4o mini TTS [28]. These steps produce a pool of possible speaker utterances (A s , T s , E s ). Step 3: Data Filtering. We perform a series of procedures to ensure the dataset quality. First, for each speaker utterance, we verify whether the synthesized audio A s faithfully reflects the intended emotion E s . Specifically, we apply an auto- matic speech emotion recognizer (i.e., Hume AI 6 ) to the generated audio and discard any utterance whose predicted emotion is inconsistent with its assigned emotion label. Next, we pair each remaining speaker utterance with the caption of every listener reactive motion R l obtained in Step 1. We then employ Qwen (Qwen3-235B-A22B-Instruct) [89] to assign a dyadic conversation appropriate- ness score to each speaker-utterance and listener motion caption pair. For each speaker utterance, we retain only the top several higher-scoring listener reactive motions, thereby removing inappropriate pairs. Step 4: Speaker–Listener Candidate Ranking and Preference Tiering. Given a pair consisting of a speaker utterance and one of its corresponding listener re- active motions from Step 3, we use multiple agents (i.e., ChatGPT-o1 [29], ChatGPT-o3 mini [58], and Qwen3-235B-A22B-Instruct [89]) to evaluate the pair. They score it according to (1) semantic appropriateness (whether the re- action fits the utterance), and (2) conversational plausibility (whether it sounds like a natural dyadic response). We further use a natural language inference (NLI) model 7 to verify whether the listener motion caption is a logically plau- sible inference from the speaker utterance. We then weighted sum the agents’ scores to obtain a final score, which is used to label the pair as gold, silver, or negative according to predefined thresholds. 6 https://w.hume.ai/expression-measurement 7 https://huggingface.co/MoritzLaurer/deberta-v3-large-zeroshot-v1.1- all-33 Abbreviated paper title7 T5-Encoder Excited T5-To k e n i z e r MiMiNeural Audio Codec Unifiedvocabulary!=! ! ∪! " ∪! # S peakertranscript L istenermotion sequence Speaker emotion S peakeraudio I'm so pumped to try that massive ride. It's much bigger than I imagined! Autoregressive generation overunified vocabulary!(motion-onlyoutput! # ) Cross- attention T5-Decoder MotionVQ-VAEDecoder Shared T5-To k e n E m b e d d i n g <Audio Token i>... <Motion Token i>... Fig. 3: Overview of the ReactMotion framework. We use modality-specific tok- enizers to convert raw data, i.e., the speaker’s utterances (including transcript, audio, and emotion) and the listener’s reactive motions, into discrete special tokens. With these tokenizers, a Seq2Seq model is employed to integrate information across modali- ties and learns to generate the listener’s reactive motions from the speaker’s utterances. 4.2 Dataset Statistics In total, our dataset contains 151,328 labeled (speaker utterance, listener reac- tive motion) pairs, covering 8,298 unique speaker utterances and 2,029 unique listener reactive motions. On average, each speaker’s utterance is paired with 18.24 candidate reactive motions, highlighting the one-to-many nature of lis- tener reactions. Overall, 9,307, 34,196, and 107,825 pairs are labeled as Gold, Silver, and Negative, respectively, reflecting graded appropriateness of candi- date reactions. We split the dataset by speaker utterance with an 8:1:1 ratio for train/val/test, such that speaker utterances are disjoint across splits (i.e., no utterance appears in more than one split). Tab. 1 lists detailed statistics. Our automated construction pipeline further enables straightforward scaling to larger datasets. 5 Methodology We present ReactMotion, a unified framework for Reactive Listener Motion Generation from Speaker Utterance. As illustrated in Fig. 3, we first introduce modality-specific tokenizers that convert raw inputs, i.e., the speaker utterance (including transcript, audio, and emotion) and the listener’s reactive motions, into discrete special tokens. With these tokenizers, we employ a Seq2Seq model to unify information across modalities and learn the conditional distribution of the task (Eqn. 1). To capture the one-to-many nature of dyadic interactions, we further train the model with a group-wise preference-based learning objective, 8C. Luo et al which explicitly allows the generation of multiple appropriate reactions for the same speaker utterance. 5.1 Modality-Specific Tokenization We employ modality-specific tokenizers to convert raw data from different modal- ities into discrete tokens. Audio Tokenization. We use Moshi [15] (its Neural Audio Codec MiMi) to con- vert the audio waveform in the speaker utterance A s into discrete codes. Specif- ically, its audio encoder E aud (·) is employed to extract audio features from A s , which are then quantized using the base codebook C aud . h s a =E aud (A s ), x s a =Q aud (h s a ),(2) where quantizer Q aud (·) maps the features to their nearest entries in the code- book C aud , and outputs the corresponding codebook indices x s a . The resulting indices are treated as discrete audio tokens, allowing the unified model to incor- porate audio information while retaining prosody and paralinguistic cues that are informative for reactive behaviors. Motion Tokenization. We represent the listener’s reactive motion R l as discrete tokens with [96], similar to the audio tokenization process: h l m =E mot (R l ), x l m =Q mot (h l m ).(3) where E mot and Q mot are the motion encoder and quantizer, respectively, and x l m are discrete indices of motion codebook C mot . Also, the predicted listener reactive motion in the form of discrete tokens from the unified model can be mapped back to the raw motion data through: h l m =Q −1 mot (x l m ), R l =D mot (h l m ),(4) where Q −1 mot (·) maps the discrete token indices to the vectors in the codebook, and a VQ-VAE motion decoder [82,96] D mot (·) decodes the vectors back to the raw motion data. 5.2 Unified Seq2Seq Modeling With above modality-specific tokenizers, we can now represent information across modalities into a unified space, and thus enable a Seq2Seq model to generate a listener reactive motion conditioned on the speaker utterance. Specifically, we adopt T5-base [65] as the Seq2Seq backbone and extend its original textual vocabulary V t to include audio and motion vocabulary: V = V t ∪ V m ∪ V a ∪ V s ,(5) Abbreviated paper title9 where V m are the code indices of the motion codebook C mot , represented as <Motion Token i> |V C mot |−1 i=0 , and V a are the code indices of audio codebook C aud , represented as <Audio Token i> |V C aud |−1 i=0 , respectively. V s contains spe- cial tokens such as <Motion Tokens>, </Motion Tokens>, <Audio Tokens>, </Audio Tokens>, <Emotion> and </Emotion>, which wrap the motion, audio, and emotion token sequences. This unified vocabulary allows us to formulate reactive listener motion gener- ation, conditioned on different modalities or their combinations C s , in a general format and achieve them within a single model. Specifically, we first fit discrete codes of the speaker utterance C s and the listen reactive motion R l into fixed prompt templates. Due to page limit, a coarse example task template of using only speaker audio as the condition is shown; detailed one and templates for other conditions are provided in the Appendix A.2. Input: You are modeling a speaker-listener dyadic interaction. Given SPEAKER_AUDIO: [Au- dio Tokens Placeholder], return ONLY a sequence of listener reactive motion tokens. Output: [Motion Tokens Placeholder] Now, the modeling process of generating listener reactive motion can be represented as an auto-regressive one, where each motion token is generated with probability p θ x out t | x in (C s ),x out <t . Here, x in (C s ) are the input token sequences of the task template embedding with input speaker utterance C s , and x out are the output token sequences, i.e., listener reactive motion x l m . 5.3 Group-wise Preference Learning A single speaker utterance C s can correspond to multiple plausible listener re- active motions R l . Directly fine-tuning on such one-to-many pairs may lead the model to collapse to averaged and safe behaviors, e.g., standing still. To mitigate this issue, we train the model using group-wise preference learning. For each speaker utterance C s , we randomly sample its corresponding lis- tener motions from each label to construct a group G,S,N, where G, S, and N denote the sets of motions labeled as Gold, Silver, and Negative, respectively. Each motion R l in the set is represented as a motion token sequence x l m . We compute the predicted score for each motion using the length-normalized condi- tional log-likelihood [5,54,83]: ℓ(x l m | C s ) = 1 |x l m | |x l m | X t=1 logp θ x l m,t | x in (C s ),x l m,<t .(6) We then aggregate the predicted scores of motions with the same label using a smooth log-mean-exp operator: ℓ A (C s ) = log 1 |A(C s )| X x l 2 m ∈A(C s ) exp ℓ(x l m | C s ) , A∈G,S,N. (7) 10C. Luo et al This yields three predicted scores for C s , namely ℓ G , ℓ S , and ℓ N corresponding to the Gold, Silver, and Negative sets. Since Gold motions are preferred over Silver, and Silver over Negative, the model is encouraged to produce ℓ G > ℓ S > ℓ N . We enforce this ordering with a soft-margin ranking loss: L rank = log 1 + exp m− (ℓ G − ℓ S ) + log 1 + exp m− (ℓ S − ℓ N ) + λ gn log 1 + exp m− (ℓ G − ℓ N ) , (8) where m specifies the margin between different labels, and λ gn controls the strength of the Gold≻Negative constraint. Training objective with frequency reweighting. To mitigate the dominance of frequently occurring motion sequences, we apply inverse-frequency weighting based on motion sequence IDs. Let i index a group (corresponding to one speaker utterance) and let r ij denote the motion sequence ID of the j-th candidate in group i. We compute freq(r) as the number of times motion ID r appears in the training set and assign an item weight ̃w ij = 1 √ freq(r ij ) . We then define the group weight as the mean item weight within the group, w i = 1 |C i | P j ̃w ij , where C i denotes the candidate set of group i. Finally, we maximize the aggregated Gold score while applying the ranking loss: L = P i w i − ℓ (i) G + λ rank L (i) rank P i w i .(9) 6 Experiments 6.1 Implementation Details We train ReactMotion for 100,000 iterations using the default AdamW optimizer and a cosine learning-rate schedule. The learning rate is set to 2×10 −5 with 1,000 warmup steps. We use a per-device batch size of 8 with gradient accumulation of 2 steps on a single NVIDIA A100 GPU. We train with six conditioning variants (T, A, T +A, T +E, A+E, T +A+E) and apply modality dropout (p=0.3) to improve robustness (see the Appendix A for more details of the implementation). 6.2 Evaluation Protocol Evaluation metrics. (i) Reactive appropriateness, i.e., how well the generated reactive human motions respond to the speaker’s input, is a core objective of our task. Inspired by preference-based evaluation paradigms [6,11,13,16,72,102], we evaluate reactive appropriateness using group-level win rates Win(g>G), Win(g>S), and Win(g>N). Specifically, we compare the best generated sam- ple g with annotated listener motions labeled as Gold (G), Silver (S), and Neg- ative (N), and compute the win rate against each reference tier. A win against a Abbreviated paper title11 Table 2: Multi-modal judge network reliability under strict modality missingness (Strict-L2). We evaluate six input modes (text T, audio A, emotion E, and their fusions) on the test set, reporting pairwise win rates (Win(G>N), Win(G>S), Win(S>N)) and ranking metrics (MRR(G), nDCG@K) with graded relevance G>S>N. Mode Win(G>N) ↑ Win(G>S) ↑ Win(S>N) ↑ MRR(G) ↑ nDCG@3 ↑ nDCG@5 ↑ nDCG@10 ↑ T0.9920.8730.9830.8290.8640.8780.932 A0.9920.8720.9830.8320.8660.8780.933 T+E0.9930.8760.9820.8260.8570.8760.929 A+E0.9920.8740.9830.8310.8650.8780.933 T+A0.9930.8790.9820.8200.8550.8750.928 T+A+E0.9930.8780.9820.8280.8590.8780.930 higher reference tier (e.g., Silver) indicates that the generated motion is ranked above a higher-quality annotated response, reflecting stronger reactive appro- priateness. To realize this evaluation, we train a multimodal judge network to rank generated reactive body motions conditioned on the same speaker in- put. Details of the judge network are provided in the appendix. We also report Gen@3, the fraction of groups where a generated candidate is ranked within the top-3 amongG,S,N plus generated candidates under the same group. (i) Motion quality is measured by Fréchet Inception Distance (FID) [23] computed in a motion feature space, and (i) Diversity is measured as the average pairwise embedding distance across generated samples, following human motion genera- tion [82,96]. (see the Appendix B.4 for more details of the evaluation metrics). Validation of the multimodal judge network. Since the judge network is central to measuring reactive appropriateness, we validate it on samples with tiered appropriateness annotations (G/S/N). Specifically, we compute the tier- consistency win rates Win(G>S), Win(G>N), and Win(S>N) to test whether the judge assigns higher scores to more appropriate reactions. Higher values in- dicate a more reliable judge. We further report MRR(G), which measures how highly the Gold reaction is ranked, and nDCG@3/nDCG@5/nDCG@10 to assess graded ranking quality among the top-K candidates. Table 2 shows the judge consistently preserves the expected preference order- ing with near-perfect separation, across all six modes and both Test set. Gold almost always beats negatives (Win(G>N)≈ 0.99) and silver also strongly beats negatives (Win(S>N) ≈ 0.98), indicating that the judge reliably distinguishes poor motions from plausible ones. Meanwhile, gold beats silver with a clear margin (Win(G>S) ≈ 0.87–0.88), reflecting sensitivity to fine-grained quality differences beyond simply rejecting negatives. The judge further achieves strong ranking quality (MRR(G) ≈ 0.82–0.84; nDCG@5 ≈ 0.87–0.88; nDCG@10 ≈ 0.93), demonstrating stable and meaningful top-K ordering. Although our multimodal judge network is trained on multiple input modal- ities, i.e., text (T), audio (A), and emotion (E), it supports missing modalities using Strict-L2. Disabled modalities are replaced with information-free inputs (all-padding text, all-padding audio codes, or an unknown emotion token). This enables the judge network to operate with any subset of modalities; even with a 12C. Luo et al single modality, it performs well in evaluation. (see the Appendix B.1 and B.2 for more details of the judge network). 6.3 Quantitative Results Since reactive listener motion generation remains underexplored, we evaluate a set of representative baselines. (a) Random Selection uniformly samples a mo- tion sequence from HumanML3D [19]. (b) Retrieval applies the text–motion matching network from prior HumanML3D T2M work [82,96] to compute text– motion similarity and retrieves the nearest-neighbor listener motion sequence from the training set given the speaker transcript. We also consider stronger cascaded LLM→T2M baselines: given a speaker utterance (and emotion), an LLM [89] first generates a listener-motion caption, which is then passed to a T2M generator to synthesize the final motion. We instantiate the LLM with Qwen3-30B-A3B (30.5B parameters) and a fine-tuned Qwen3-4B-Thinking (4B parameters) trained on our training-set (speaker utterance, listener-motion caption) pairs. The resulting captions are fed into two representative T2M gen- erators, T2M-GPT [96] and MG-MotionLLM [82]. More details of baselines are in the Appendix B.3. Tab. 3 shows that ReactMotion outperforms all baselines in reactive appropri- ateness. Among the cascaded LLM→T2M pipelines, LLM→MG-MotionLLM * is the strongest, improving over Random Selection and Retrieval. However, de- spite using a powerful motion generator, it still performs poorly under strict com- parisons to Silver references (Win(g>S)), indicating that the two-stage caption- then-generate pipeline struggles to produce highly appropriate listener reactions. In contrast, ReactMotion achieves near-perfect Win(g>N) across input modes and substantially improves Win(g>S) and Gen@3. Our full model (T +A+E) yields the best overall Win rates, while maintaining low FID and competitive diversity. Although Retrieval attains the highest diversity by construction, it yields much lower appropriateness and worse realism than our approach. More experimental results are provided in the Appendix D. 6.4 Qualitative Results We visualize representative examples in Fig. 4, comparing our ReactMotion (Ours), a cross-entropy trained variant (CE), and LLM→MG-MotionLLM * with a finetuned Qwen [89] (Qwen3-4B-Thinking) on training set, together with gold and silver reference reactions under the same speaker condition. Overall, ReactMotion produces reactive motions that are both semantically consistent with the speaker content and expressive in intensity. For instance, for the ut- terance “The energy in here feels electric right now” with excited emotion, our model generates larger, more dynamic upper-body and arm movements, which better reflect the high-energy “electric” cue and match the communicative style seen in the gold reaction. In contrast, the silver reaction exhibits a rapid hand-wave but remains relatively low-energy, making it less aligned with the excited condition. The Abbreviated paper title13 Table 3: Quantitative results on the test set. Main evaluation metrics are Win(g>N), Win(g>S), Win(g>G), Gen@3 measuring Reactive Appropriateness. We additionally evaluate motion quality (FID) and diversity. ∗ indicates that the LLM is fine-tuned using training-set speaker utterance and listener motion caption pairs. MethodInput Mod. Win(g>N)↑ Win(g>S)↑ Win(g>G)↑ Gen@3↑ FID↓ Diversity↑ GT-----0.278 6.187 Random Selection-0.2650.1220.0060.099 42.363 9.880 RetrievalT0.3920.2520.1300.206 7.429 8.207 LLM→T2M-GPTT+E0.1380.0380.0160.199 49.920 4.946 LLM→T2M-GPT * T+E0.1710.0270.0170.350 42.589 6.102 LLM→MG-MotionLLM T+E0.7750.2450.0440.345 23.629 5.082 LLM→MG-MotionLLM * T+E0.8830.2740.0470.380 25.723 4.546 ReactMotion (Ours)T0.9930.7740.2580.916 4.706 4.789 ReactMotion (Ours)A0.9920.6140.1640.864 6.221 4.009 ReactMotion (Ours)T+E0.9900.6960.2060.930 5.422 4.475 ReactMotion (Ours)A+E0.9930.7360.323 0.981 6.485 4.162 ReactMotion (Ours)T+A0.9930.6510.2150.931 6.560 4.145 ReactMotion (Ours)T+A+E1.0000.7970.2660.960 4.760 4.804 CE variant tends to regress to generic, weakly-conditioned responses (e.g, a static pose such as crossing arms), indicating limited ability to exploit prefer- ence structure and model the one-to-many nature of reactive behaviors. Finally, the LLM→T2M baseline often generates repetitive motions (e.g, near-constant waving) with limited temporal variation, which appears less suitable for dyadic communication, where reactions typically evolve over time (e.g, hands rising and lowering, pose changes and subtle turns). Moreover, because dyadic reactions can be difficult to describe in natural language, the out-of-domain captions produced by the LLM may be noisy, which can lead MG-MotionLLM to produce degraded outputs, including overly short motion sequences. 6.5 User Study We recruit 59 volunteers and conduct a user study to evaluate the reactive appropriateness of listener motions generated by ReactMotion (Ours) against two baselines (the CE variant and LLM→MG-MotionLLM * ) and the best- in-group Silver reference. In each case, participants watch two motion videos (A/B) conditioned on the same speaker utterance (audio with transcript shown) and select the more appropriate listener reaction. Each participant completes 36 cases covering six speaker conditions (six pairwise comparisons per condition). As shown in Fig. 5, Ours is preferred over the generative baselines, achieving win rates of 67.8% against CE and 72.0% against LLM→MG-MotionLLM * . Ours is also competitive with the Silver reference, receiving 44.1% of votes in Silver vs. Ours, substantially higher than CE (31.9%) and LLM→MG- MotionLLM (31.4%). 14C. Luo et al The energy in here feels electric right now. Emotion:Excited Ours Gold CE LLM→MG- MotionLLM Silver Fig. 4: Qualitative results. We compare gold and silver listener reactions, motions generated by our ReactMotion (Ours), a cross-entropy trained variant (CE), and a cascaded LLM→T2M baseline, all conditioned on the same speaker utterance. We visualize the resulting 3D motion sequences. 6.6 Ablation Studies Modality study. We study the effect of input modalities in Tab. 3. Across settings, multimodal fusion performs best overall. Text is the strongest single cue, giving high alignment and the lowest single-modality FID (e.g., T: Win(g>N)=0.993, Win(g>S)=0.774, FID=4.706). Audio alone is weaker for fine-grained appropri- ateness, but adding emotion substantially improves it (best Win(g>G)=0.323 and Gen@3=0.981). Full fusion (T +A+E) is the most balanced, achieving the best Win(g>N)=1.000, strong Win(g>S)=0.797, and a low FID=4.760. Ablations on group-wise preference learning. Tab. 4 ablates key components of our group-wise preference learning objective. Compared to training with cross- entropy only, our full model substantially improves both reactive appropriate- ness and motion quality (e.g., Win(g>S): 0.741→0.797; Gen@3: 0.938→0.960; FID: 6.555→4.760). Removing inverse-frequency reweighting leads to the largest appropriateness drop, especially against the strongest tier (Win(g>G): 0.266→0.220), highlighting the importance of mitigating the dominance of fre- quent and generic motions. Removing the ranking loss degrades fidelity (FID: 4.760→5.950) while increasing diversity (4.804→5.453), suggesting that the rank- ing constraints help enforce correct relative ordering among tiers. Finally, re- Abbreviated paper title15 Silver vs. OursSilver vs. CESilver vs. LLM→MG- MotionLLM Ours vs. CEOurs vs. LLM→MG- MotionLLM 0 25 50 75 100 User Preference Rate (%) 55.9% 44.1% 68.1% 31.9% 68.6% 31.4% 67.8% 32.2% 72.0% 28.0% SilverOursCELLM→MG-MotionLLM Fig. 5: User study on reactive appropriateness. Table 4: Ablation studies on the test split (all use A+T+E unless noted). w/o denotes training without the corresponding component. The CE baseline trains the same model using only a cross-entropy loss by pairing each speaker input with a single Gold reaction as supervision. MethodWin(g>N)↑ Win(g>S)↑ Win(g>G)↑ Gen@3↑ FID↓ Diversity↑ CE baseline0.9900.7410.2620.938 6.555 5.448 Ours (full)1.0000.7970.266 0.960 4.760 4.804 w/o Inverse-frequency reweighting 0.9790.7040.2200.946 5.177 4.929 w/o L rank 0.9960.7810.260 0.960 5.950 5.453 w/o ℓ G 0.9960.7120.2150.943 6.376 4.493 moving ℓ G consistently harms both appropriateness and quality, indicating that likelihood supervision on Gold reactions remains necessary. 7 Conclusion We introduce Reactive Listener Motion Generation from Speaker Utterance, a new task for modeling listener motion responses in dyadic interactions. To sup- port this task, we present ReactMotionNet, a multi-modal dataset that explic- itly captures the inherent non-determinism of human behavior: for each speaker utterance, we provide multiple candidate listener motions with preference anno- tations, enabling supervision beyond a single “ground-truth” response. Building on this dataset design, we develop preference-oriented evaluation protocols tai- lored to reactive motion generation. Finally, we propose ReactMotion, a unified framework that processes multi-modal speaker cues, substantially outperforms strong baselines in motion quality and reactive appropriateness. We believe this work provides a foundation for future research on modeling dyadic interactions. 16C. Luo et al Outline of the Supplementary Material The supplementary material is organized as follows: • Section A presents the implementation details, including the model config- uration, vocabulary construction, optimization settings, and training hyper- parameters. • Section A.1 presents the model size of ReactMotion. • Section A.2: prompt templates for different speaker-condition settings; • Section B further provides the additional evaluation details, including: • Section B.1: the formulation of the multimodal judge network; • Section B.3: details of the baseline methods. • Section B.4 introduces the evaluation metrics, covering reactive appropri- ateness, motion quality, and diversity. • Section C provides additional statistics and analysis of the ReactMotionNet dataset. • Section D.1 presents the hyperparameter sensitivity analysis, including the full sweep results, representative configurations, and heatmap visualizations. • Section D.2 evaluates the inference efficiency of the proposed method. • Section D.3 reports the protocol and results of the user study. • Section D.4 shows representative failure cases. • Section E discusses the limitations of the current framework. A Implementation Details Tab. 5 summarizes the key implementation details and training hyperparameters used in our experiments. Specifically, ReactMotion is instantiated with a T5-base Seq2Seq backbone, comprising 222.9M backbone parameters and 235.9M train- able parameters after extending the vocabulary. In accordance with the method- ology section, the original textual vocabulary (|V t | = 32,100) is augmented with motion tokens (|V m | = 512), MiMi audio tokens (|V a | = 2,048 per codebook; 8 codebooks), and modality-specific special tokens that mark the boundaries of different modalities, resulting in a unified vocabulary of size 63,338. Notably, the vocabulary includes tokens from all 8 MiMi codebooks for completeness, while in practice we only use tokens from the base codebook during training to accelerate the process. The model takes tokenized speaker utterances as input and autoregressively predicts listener reactive motion tokens, with maximum source and target lengths set to 512 and 256, respectively. We train the model using AdamW with learning rate 2.0× 10 −5 , β 1 = 0.9, β 2 = 0.999, weight decay 0.0, 1,000 warmup steps, per-device batch size 8, gradient accumulation over 2 steps, and 100,000 total optimization steps. To capture the one-to-many mapping from a speaker utterance to plausible listener reactions, training adopts the pro- posed group-wise preference objective with λ rank = 0.25, λ gn = 0.25, and margin m = 0.5. We further apply modality dropout with rate 0.3 to improve robustness to missing modalities, while length-normalized LogSumExp aggregation is used to obtain stable set-level scores during preference optimization. Abbreviated paper title17 Table 5: Implementation details and hyperparameters used in training. SetupValue Seq2Seq backbone modelT5-base [65] Text tokenizerT5-base tokenizer [65] Audio tokenizerMiMi neural audio codec [15] Motion tokenizerVQ-VAE from T2M-GPT [96] Per-device batch size8 Gradient accumulation steps2 Training steps100,000 Warmup steps1,000 OptimizerAdamW Adam β 1 0.9 Adam β 2 0.999 Weight decay0.0 Learning rate2.0× 10 −5 Maximum source length512 Maximum target length256 Text vocabulary size |V t |32,100 Audio codebook size |V a |2,048 Number of MiMi audio codebooks8 Motion VQ-VAE codebook size |V m |512 Total vocabulary size |V|49,002 Backbone parameters222.9M Total trainable parameters after vocabulary expansion235.9M Ranking loss weight λ rank 0.25 Gold-negative loss weight λ gn 0.25 Ranking Margin m0.5 Modality dropout rate0.3 LogSumExp normalizationEnabled A.1 Model Size Table 6: Model Configuration and Parameters of ReactMotion. MetricValue Backbone parameters222.9M Total trainable parameters 235.9M Unified vocabulary size49,002 Table 6 summarizes the model size of ReactMotion. The model is built upon a T5-base backbone with 222.9M parameters and 235.9M trainable parameters after extending the vocabulary to incorporate multimodal tokens. A.2 Prompt Templates To support unified generation under different speaker-condition settings, we con- vert the available speaker cues into a fixed natural-language prompt template. 18C. Luo et al Given a speaker utterance consisting of transcription, audio, and optional emo- tion annotation, we construct the input prompt by selectively enabling the corre- sponding fields. The model is instructed to output only the listener motion-token sequence in a strict format, without any additional natural language. Formally, for a speaker utterance C s , the prompt is constructed as Input: You are modeling a speaker-listener dyadic interaction. Input: - SPEAKER_TRANSCRIPTION: [Speaker Transcription] - SPEAKER_AUDIO: [Speaker Audio] - SPEAKER_EMOTION: <Emotion> [Speaker Emotion] </Emotion> Output: Return ONLY a sequence of listener motion tokens in the exact format: <Motion Tokens> <Motion Token i> ... </Motion Tokens> Do NOT output any other words. In practice, the fields in the prompt are enabled or disabled depending on the chosen condition mode. For example, when transcription is used but audio is not, the SPEAKER_AUDIO field is left empty; when emotion is disabled, the emotion line is omitted entirely. This design allows us to handle text-only, audio-only, text+audio, text+emotion, audio+emotion, and text+audio+emotion settings within a single unified framework. Below we show several concrete examples. Text-only condition (T). Input: You are modeling a speaker-listener dyadic interaction. Input: - SPEAKER_TRANSCRIPTION: [Speaker Transcription] - SPEAKER_AUDIO: Output: Return ONLY a sequence of listener motion tokens in the exact format: <Motion Tokens> <Motion Token i> ... </Motion Tokens> Do NOT output any other words. Text+Emotion condition (T +E). Input: You are modeling a speaker-listener dyadic interaction. Input: - SPEAKER_TRANSCRIPTION: [Speaker Transcription] - SPEAKER_AUDIO: - SPEAKER_EMOTION: <Emotion> [Speaker Emotion] </Emotion> Output: Return ONLY a sequence of listener motion tokens in the exact format: <Motion Tokens> <Motion Token i> ... </Motion Tokens> Do NOT output any other words. Audio-only condition (A). Input: You are modeling a speaker-listener dyadic interaction. Input: - SPEAKER_TRANSCRIPTION: - SPEAKER_AUDIO: [Speaker Audio] Output: Return ONLY a sequence of listener motion tokens in the exact format: Abbreviated paper title19 <Motion Tokens> <Motion Token i> ... </Motion Tokens> Do NOT output any other words. Audio+Emotion condition (A+E). Input: You are modeling a speaker-listener dyadic interaction. Input: - SPEAKER_TRANSCRIPTION: - SPEAKER_AUDIO: [Speaker Audio] - SPEAKER_EMOTION: <Emotion> [Speaker Emotion] </Emotion> Output: Return ONLY a sequence of listener motion tokens in the exact format: <Motion Tokens> <Motion Token i> ... </Motion Tokens> Do NOT output any other words. Text+Audio condition (T +A). Input: You are modeling a speaker-listener dyadic interaction. Input: - SPEAKER_TRANSCRIPTION: [Speaker Transcription] - SPEAKER_AUDIO: [Speaker Audio] Output: Return ONLY a sequence of listener motion tokens in the exact format: <Motion Tokens> <Motion Token i> ... </Motion Tokens> Do NOT output any other words. Text+Audio+Emotion condition (T +A+E). Input: You are modeling a speaker-listener dyadic interaction. Input: - SPEAKER_TRANSCRIPTION: [Speaker Transcription] - SPEAKER_AUDIO: [Speaker Audio] - SPEAKER_EMOTION: <Emotion> [Speaker Emotion] </Emotion> Output: Return ONLY a sequence of listener motion tokens in the exact format: <Motion Tokens> <Motion Token i> ... </Motion Tokens> Do NOT output any other words. Given the constructed prompt x in (C s ), the model auto-regressively predicts the listener motion-token sequence x out as p θ x out t | x in (C s ),x out <t . Here, x in (C s ) denotes the prompt sequence instantiated from the speaker utter- ance C s , and x out denotes the output listener motion-token sequence. B Additional Evaluation Details B.1 Multimodal Judge Network To evaluate the reactive appropriateness of generated listener motions and sup- port best-of-Kselection, we train a multimodal judge network, illustrated in 20C. Luo et al Fig. 6. Given a speaker utteranceC s and a candidate listener motion token sequencex l m , the judge networks ψ outputs a scalar compatibility score s ψ (C s ,x l m )∈ R,(10) where a larger value indicates that the candidate listener motion is more appro- priate for the given speaker utterance. Excited I'm so pumped to try that massive ride. It's much bigger than I imagined! T5-To k e n i z e r MiMiNeural Audio Codec S peakertranscript Speaker emotion S peakeraudio T5-Encoder LinearLayer AttentionPooling z ! Tr a n s f o r m e rLayer LinearLayer z " Fusion Transformer AttentionPooling z # MotionVQ-VAEEncoder MotionVQ-VAEDecoder LearnableEmotion Embedding LearnableMotion Embedding PositionalEncoding Tr a n s f o r m e rEncoder AttentionPooling z $ InfoNCELoss LearnableAudio Embedding AvgPooling " h ! H ! E !%&" ! Modalitytype embedding u ! AttentionPooling z ' AvgPooling H ' + " h ' u ' + E !%&" ' Modalitytype embedding ModeEncoding " h " u " + E !%&" " Modalitytype embedding z $ z # z ' z ! z $ z $ Fig. 6: Architecture of the multimodal judge network. Given a speaker ut- terance and a candidate listener motion, the judge encodes transcript, MiMi audio tokens, and the discrete emotion label with three modality-specific branches, produc- ing modality embeddingsz t ,z a , andz e , as well as hidden summaries used to form fusion tokensu t ,u a , andu e . Modality-type embeddings and a mode embedding are added to these tokens, which are then processed by a fusion transformer and attention pooling to obtain the unified condition embeddingz f . In parallel, the candidate listener motion, represented by VQ-VAE motion tokens, is encoded by a motion transformer and pooled into a motion embeddingz m . The judge computes compatibility between the condition and motion embeddings in a shared normalized scoring space. During training, a group-wise InfoNCE objective is applied to the fused embedding and auxil- iary modality-specific embeddings, enabling reliable scoring under both full and partial speaker utterances. Snowflake and flame icons denote frozen and trainable modules, respectively. Architecture. It contains three branches to encode different modalities: tran- script, audio, and emotion in the speaker utterance C s , a fusion branch that in- Abbreviated paper title21 tegrates the available information inC s while allowing missing modalities, and a motion branch to encode the reactive motion. All branches project their features from dimension dinto a shared scoring space of dimensiond o . By default, all score-space embeddings areℓ 2 -normalized. Text branch. Letx s t denote the tokenized speaker transcript, and letM t ∈ 0, 1 T t denote its padding mask, whereM t (j) = 1indicates that thej-th token is valid andM t (j) = 0indicates padding. We encode the transcript with a T5 encoderE T5 (·)and project the resulting hidden states into the shared hidden space: H t = W t E T5 (x s t ) + b t ,(11) whereH t ∈ R T t ×d ,W t ∈ R d×d T5 , andb t ∈ R d are learnable parameters. We then aggregate the token-level features into a text embedding in the scoring space: ̃z t = AttnPool t (H t ;M t ), z t = L2Norm( ̃z t ),(12) where ̃z t ,z t ∈ R d o ,AttnPool t (·)denotes a masked attention-pooling operator that ignores padded positions according toM t , and L2Norm denotes ℓ 2 normal- ization. Audio branch. Letx s a denote the speaker audio token sequence obtained from the MiMi neural codec tokenizer [15]. Because MiMi audio is represented by multiple codebooks, we first map the discrete tokens into embeddings, add learnable codebook-level embeddings and positional embeddings, and then pro- cess the resulting sequence with a transformer encoder: H a =E a Emb a (x s a ) + E a lvl + E a pos ,(13) whereH a ∈ R T a ×d ,Emb a (·)denotes the learnable audio-token embedding layer, E a lvl is the learnable codebook-level embedding,E a pos is the learnable positional embedding, andE a is the audio transformer encoder. LetM a ∈0, 1 T a denote the audio padding mask, whereM a (j) = 1indicates a valid audio token and M a (j) = 0indicates padding. The token-level audio features are pooled into an audio embedding: ̃z a = AttnPool a (H a ;M a ), z a = L2Norm( ̃z a ),(14) where ̃z a ,z a ∈ R d o . Emotion branch. Let e s denote the discrete speaker emotion label. We map it to a learnable embedding and project it into the shared scoring space: h e = LayerNorm Emb e (e s ) , ̃z e = W e h e + b e , z e = L2Norm( ̃z e ),(15) where Emb e (·)is the learnable emotion embedding table,h e ∈ R d ,W e ∈ R d o ×d , b e ∈ R d o , and ̃z e ,z e ∈ R d o . Fusion branch. To unify all available information, we construct one fusion token for each modality in thed-dimensional hidden space. Let o ⊆ t,a,e 22C. Luo et al denote the active modality set, and letδ k (o)∈0, 1indicate whether modality k ∈t,a,eis available under modeo. For text and audio, we summarize the hidden states by masked mean pooling over valid positions: ̄ h t = P T t j=1 M t (j)H t (j) P T t j=1 M t (j) , δ t (o) = 1, 0,δ t (o) = 0, ̄ h a = P T a j=1 M a (j)H a (j) P T a j=1 M a (j) , δ a (o) = 1, 0,δ a (o) = 0, (16) where H t (j),H a (j) ∈ R d denote thej-th hidden states. For emotion, which is already represented by a single hidden vector, we define ̄ h e = ( h e , δ e (o) = 1, 0, δ e (o) = 0. (17) We then form the modality-specific fusion tokens u t = ̄ h t + E t type , u a = ̄ h a + E a type , u e = ̄ h e + E e type ,(18) where u t ,u a ,u e ∈ R d , andE t type ,E a type , andE e type are learnable type embeddings. To explicitly encode which modalities are active, we further introduce a learn- able mode embeddingE mode (o)∈ R d . The initial fusion-token sequence is X (0) f = u t + E mode (o) u a + E mode (o) u e + E mode (o) ∈ R 3×d .(19) Since some modalities may be absent, we define a modality-presence mask M f = δ t (o), δ a (o), δ e (o) ∈0, 1 3 .(20) The fusion sequence is processed by a transformer encoder with masking: H f =E f X (0) f ;M f , ̃z f = AttnPool f (H f ;M f ), z f = L2Norm( ̃z f ), (21) where H f ∈ R 3×d ,E f denotes the multimodal fusion transformer, and ̃z f ,z f ∈ R d o . Motion branch. Each candidate listener motion is represented as a motion token sequence x l m , obtained using a motion VQ-VAE tokenizer. We map the motion tokens to embeddings, add positional embeddings, and encode them with a motion transformer: H m =E m Emb m (x l m ) +E m pos , ̃z m = AttnPool m (H m ;M m ), z m = L2Norm( ̃z m ), (22) whereH m ∈ R T m ×d ,M m ∈ 0, 1 T m is the motion padding mask,Emb m (·)is the motion-token embedding layer,E m pos is the motion positional embedding,E m is the motion transformer encoder, and ̃z m ,z m ∈ R d o . Abbreviated paper title23 Compatibility scoring. Letφ(·,·)denote the embedding-space compati- bility function. Given a condition embeddingz ∈ R d o and a motion embedding z m ∈ R d o , we define φ(z,z m ) = αz ⊤ z m , α = exp(τ ),(23) whereτis a learnable temperature parameter andα > 0is the corresponding scaling factor. Because all score-space embeddings areℓ 2 -normalized, Eq. (23) is a scaled cosine similarity. The fused compatibility score is defined as s ψ (C s ,x l m ) = φ(z f ,z m ).(24) In addition, we compute auxiliary modality-specific compatibility scores s (k) ψ (C s ,x l m ) = φ(z k ,z m ), k ∈t,a,e,(25) which allow the judge to score candidate motions under partial speaker utter- ances. Group-wise contrastive training. For each speaker utteranceC s i , we construct a candidate set U i =G(C s i )∪S(C s i )∪N (C s i ),(26) whereG(C s i ),S(C s i ), andN (C s i )denote the Gold, Silver, and Negative listener motion sets, respectively. During training, we randomly sample a small number of candidates from each tier and encode them jointly. To improve robustness to incomplete conditions, we randomly vary the active modality setoduring training. This encourages the judge to remain reliable under different condition modes, including single-modality settings such as text- only and audio-only. Let P i ⊆U i denote the positive set associated withC s i ; in our default setting, P i =G(C s i ). Given a condition embeddingz i (which can be the fused embedding z f or an active modality-specific embeddingz t ,z a ,z e ), we optimize the following group-wise InfoNCE objective: L con (z) =− 1 |B| X i∈B log P x∈P i exp φ(z i ,z m (x)) P x∈U i exp φ(z i ,z m (x)) + P b∈B bank exp β φ(z i ,z m (b)) , (27) whereBis the mini-batch,B bank is an auxiliary motion bank providing additional generic negatives,z m (x)denotes the motion embedding of candidatex,z m (b) denotes the embedding of a motion sampled from the bank, andβcontrols the contribution of bank negatives. The motion bank discourages the judge from assigning overly high compatibility scores to generic or template-like motions. We always apply Eq. (27) to the fused embedding z f . For the modality- specific auxiliary losses, we apply it only to the modalities active under the 24C. Luo et al current modeo: L judge = λ f L con (z f ) + X k∈o λ k L con (z k ), k ∈t,a,e,(28) whereλ f ,λ t ,λ a ,λ e are loss weights to balance different loss terms. Validation of the multimodal judge network. Because the judge network is central to our evaluation protocol, we further verify whether its rankings respect the annotated tier ordering G ≻ S ≻ N. For any tierA ∈ G,S,N, we define its mean judge score under conditionC s as ̄s A (C s ) = 1 |A(C s )| X x∈A(C s ) s ψ (C s ,x).(29) We then report Win(G>S), Win(G>N), and Win(S>N), defined as Win(A >B) = 1 |D| X C s ∈D κ ̄s A (C s ), ̄s B (C s ) ,(30) where(A,B)∈(G,S), (G,N ), (S,N ),Ddenotes the evaluation set of speaker utterances, and κ(u,v) = 1, u > v, 0.5, u = v, 0, u < v. (31) We further report MRR(G), defined as MRR(G) = 1 |D| X C s ∈D 1 min x∈G(C s ) rank C s (x) , (32) where all candidates in U (C s )are sorted in descending order ofs ψ (C s ,x), and rank C s (x)denotes the resulting 1-based rank of candidatex. Finally, we report nDCG@3, nDCG@5, and nDCG@10, using graded rel- evance labels2,1, and0for Gold, Silver, and Negative candidates, respectively. These metrics verify whether the learned judge produces rankings aligned with the annotated appropriateness structure. Strict-L2 missing-modality injection. For partial-condition evaluation, we adopt a Strict-L2 missing-modality injection protocol. Given an active modality set o ⊆ t,a,e, every unavailable modality is replaced by a null input before it is processed by its encoder branch. This differs from a weak masking strategy that removes a modality only during fusion while still allowing its encoder to observe the original input. Formally, letδ t (o),δ a (o), andδ e (o)indicate whether text, audio, and emotion are active under modeo, respectively. For text, ifδ t (o) = 0, we replace the transcript with an all-padding sequence and set its padding mask to zero: x s t ← PAD, M t (j) = 0, ∀j.(33) Abbreviated paper title25 For audio, ifδ a (o) = 0, we replace all codec tokens with the audio padding index and mark all time steps as padded: x s a ← PAD a , M a (j) = 0, ∀j.(34) For emotion, ifδ e (o) = 0, we replace the original label with a dedicated unknown symbol: e s ← <unk>.(35) At the fusion stage, the corresponding modality token is additionally masked out through M f . As a result, unavailable modalities contribute no semantic information to the final condition representation. This protocol provides a strict test of whether the judge can reliably score listener motions using only the actually available speaker signals. Unless otherwise specified, all partial-condition reliability experiments are conducted under this Strict-L2 protocol. B.2 Implementation Details of Judge Network Table 7: Hyperparameters for the multimodal judge network. ParameterValue Backbone encoderT5-base Hidden dimension d768 Embedding dimension 512 Transformer heads12 Transformer layers6 Feedforward dimension 3072 Dropout0.1 Temperature0.07 Memory bank size4096 OptimizerAdamW Learning rate5× 10 −5 Weight decay0.01 Batch size16 Epoch50 λ f 1.0 λ t 0.5 λ a 0.5 λ e 0.2 The multimodal judge network is implemented using a transformer-based architecture that evaluates the compatibility between speaker utterances and candidate listener motions. The textual modality is encoded using a pre-trained T5-base encoder, while audio tokens, emotion labels, and motion tokens are 26C. Luo et al Table 8: We evaluate the multi-modal matching judge on validation and test set across six input modes (text T, audio A, emotion E, and their fusions). We report pairwise win rates based on mean score comparisons (Win(G>N), Win(G>S), Win(S>N)) and ranking metrics (MRR(G), nDCG@K with graded relevance G>S>N), where G=G (Gold), S=S (Silver), and N=N (Negative). Mode Split Win(G>N) ↑ Win(G>S) ↑ Win(S>N) ↑ MRR(G) ↑ nDCG@3 ↑ nDCG@5 ↑ nDCG@10 ↑ Val TVal0.9900.8730.9850.8390.8780.8910.939 AVal0.9900.8730.9850.8420.8810.8930.940 T+AVal0.9930.8830.9880.8400.8750.8900.937 T+EVal0.9940.8810.9880.8410.8750.8910.938 A+EVal0.9900.8750.9850.8400.8780.8920.939 T+A+E Val0.9930.8820.9880.8400.8760.8900.937 Test TTest0.9920.8730.9830.8290.8640.8780.932 ATest0.9920.8720.9830.8320.8660.8780.933 T+ATest0.9930.8790.9820.8200.8550.8750.928 T+ETest0.9930.8760.9820.8260.8570.8760.929 A+ETest0.9920.8740.9830.8310.8650.8780.933 T+A+E Test0.9930.8780.9820.8280.8590.8780.930 embedded and processed through transformer encoders to obtain modality rep- resentations. These representations are projected into a shared embedding space where the final compatibility score is computed. Table 7 summarizes the key hyperparameters used for training the judge network. The model adopts a hidden dimension of 768 and projects the rep- resentations into a 512-dimensional embedding space. The transformer encoder uses 12 attention heads and 6 layers with a feedforward dimension of 3072. Train- ing is performed using the AdamW optimizer with a learning rate of 5× 10 −5 , weight decay of 0.01, and batch size of 16. A memory bank of size 4096 is used to provide additional negative samples for contrastive training. B.3 Baseline Methods GT. We use the ground-truth listener motion sequences from the test set as an upper-bound reference. Random Selection. We randomly sample a motion sequence from HumanML3D [19] as a naive baseline. Retrieval. Following standard text–motion matching protocols [19,82], we re- trieve a listener motion by matching the speaker transcription against candidate motions and returning the top-1 nearest neighbor from the training set. Specifi- cally, we use the pretrained text and motion encoders from [19], which are trained with a contrastive objective so that matched text–motion pairs are close in the shared embedding space, while mismatched pairs are separated by a margin. The text encoder maps the input transcription to a semantic feature vector, while the motion encoder first converts a pose sequence into motion snippet codes and then maps them to a motion feature vector. In practice, the text encoder follows the architecture in [19], and the motion encoder is implemented as a bidirectional GRU with hidden size 1,024. Abbreviated paper title27 Cascaded LLM→T2M. We construct cascaded baselines by first prompting an LLM to generate the caption of listener reactive motion conditioned on the speaker transcription and emotion. Then, we feed the generated caption into a text-to-motion (T2M) model to synthesize the final motion. Here, we consider two LLMs, Qwen3-30B-A3B and a fine-tuned Qwen3-4B-Thinking, together with two representative T2M generators, T2M-GPT and MG-MotionLLM. Accordingly, LLM→T2M-GPT denotes the cascade using Qwen3-30B- A3B and T2M-GPT, while LLM→T2M-GPT ∗ uses the fine-tuned Qwen3- 4B-Thinking together with T2M-GPT. Similarly, LLM→MG-MotionLLM denotes the cascade using Qwen3-30B-A3B and MG-MotionLLM, while LLM→MG-MotionLLM ∗ uses the fine-tuned Qwen3-4B-Thinking together with MG-MotionLLM. To keep the main table concise, we report the cascaded baselines under the T +E setting. B.4 Evaluation Metrics We evaluate model performance from three complementary perspectives: (i) re- active appropriateness, (i) motion quality, and (i) diversity. Reactive appropriateness. Reactive appropriateness measures how well the gen- erated listener motions respond to the speaker utterance. For each speaker ut- terance C s , the annotated listener motions are partitioned into three relevance tiers: GoldG(C s ), SilverS(C s ), and NegativeN (C s ). Let b R l (C s ) =ˆx l m,1 ,..., ˆx l m,M (36) denote the set of Mgenerated listener motion sequences for the same condi- tion. To assess relative appropriateness, we use the multimodal judge network introduced in Sec. B.1, which assigns a compatibility score s ψ (C s ,x l m )(37) to a candidate listener motionx l m conditioned on the speaker inputC s . For any candidate set A(C s ), we define its mean judge score as ̄s A (C s ) = 1 |A(C s )| X x l m ∈A(C s ) s ψ (C s ,x l m ). (38) For brevity, we denote the mean scores of the generated set and the three anno- tated tiers by g(C s ) = ̄s b R l (C s ), G(C s ) = ̄s G (C s ), S(C s ) = ̄s S (C s ), N (C s ) = ̄s N (C s ). (39) We then report Win(g>G), Win(g>S), and Win(g>N), defined as Win(g >A) = 1 |D| X C s ∈D κ g(C s ), ̄s A (C s ) , A∈G,S,N,(40) 28C. Luo et al whereDdenotes the evaluation set, and κ(u,v) = 1, u > v, 0.5, u = v, 0, u < v. (41) Intuitively, Win(g>N) measures whether the generated motions are preferred over clearly inappropriate responses, Win(g>S) is a stricter criterion against moderately appropriate responses, and Win(g>G) is the most challenging cri- terion against highly appropriate annotated reactions. Higher values indicate stronger reactive appropriateness. We further report Gen@3, which measures whether at least one generated motion is ranked within the top 3 among all candidates under the same speaker utterance. For each C s , we form the candidate pool C(C s ) =G(C s )∪S(C s )∪N (C s )∪ b R l (C s ),(42) rank all candidates in C(C s )bys ψ (C s ,·)in descending order, and denote the resulting rank of a candidatex l m byrank C s (x l m ). We then compute Gen@3 = 1 |D| X C s ∈D I " min ˆx l m ∈ b R l (C s ) rank C s (ˆx l m )≤ 3 # . (43) This metric is particularly suitable for our task because reactive listener behavior is inherently one-to-many: the same speaker utterance may admit multiple plau- sible listener reactions, and Gen@3 evaluates whether the model can produce at least one highly competitive response within a limited candidate budget. Motion quality. We evaluate motion quality using Fréchet Inception Distance (FID) [23] in a motion feature space. Let f eval (x l m )denote the feature repre- sentation of a motion sequence extracted by a pretrained motion evaluation network. We compute the feature statistics of generated motions and real mo- tions in the test set, and then measure the Fréchet distance between the two Gaussian distributions: FID =∥μ r − μ g ∥ 2 2 + Tr Σ r + Σ g − 2(Σ r Σ g ) 1/2 ,(44) where(μ r ,Σ r )and(μ g ,Σ g )are the mean and covariance of the real and gen- erated motion features, respectively. Lower FID indicates that the generated motions are closer to the distribution of real listener motions, and therefore reflects better overall motion quality. Diversity. Since a single speaker utterance may admit multiple plausible listener reactions, it is also important to evaluate the diversity of generated motions. Following prior work in human motion generation [82,96], we measure diversity in the same motion feature space. Given the set of all generated motions, we Abbreviated paper title29 randomly sample two subsets of equal sizeS d , denoted byˆx l m,1 ,..., ˆx l m,S d and ˆx l′ m,1 ,..., ˆx l′ m,S d , and define diversity as Diversity = 1 S d S d X i=1 f eval (ˆx l m,i )− f eval (ˆx l′ m,i ) 2 . (45) Higher diversity indicates that the generated motions exhibit greater variation and are less likely to collapse to a small set of repetitive motion patterns. Table 9: Full hyperparameter sweep results for group-wise preference training. We vary the ranking margin m, ranking-loss weight λ rank , and Gold-vs-Negative weight λ gn . We report pairwise preference metrics (Win(g>N), Win(g>S), Win(g>G)), together with Gen@3, FID, and Diversity. m λ rank λ gn Win(g>N) ↑ Win(g>S) ↑ Win(g>G) ↑ Gen@3 ↑ FID ↓ Diversity ↑ 0.00 0.00 0.000.99760.78090.25850.9600 5.2638 5.3005 0.00 0.00 0.250.99760.78090.26330.9600 5.2638 5.3005 0.00 0.00 0.500.99760.78090.26150.9600 5.2638 5.3005 0.00 0.00 1.000.99640.78090.26150.9613 5.2638 5.3005 0.00 0.25 0.000.99880.78090.23310.9467 5.9644 4.6993 0.00 0.25 0.250.99520.74820.22400.9467 5.2102 4.8197 0.00 0.25 0.500.99880.72880.21370.9455 5.3426 4.9865 0.00 0.25 1.000.99390.78150.24580.9528 5.3948 4.7384 0.00 0.50 0.000.99270.77600.25480.9443 4.6552 4.7315 0.00 0.50 0.250.99520.77300.24820.9600 5.4479 4.4127 0.00 0.50 0.500.99520.74760.23790.9600 5.9814 4.3124 0.00 0.50 1.000.99390.76940.25120.9576 5.3426 4.5137 0.00 1.00 0.000.99640.75480.23120.9443 6.5379 3.9613 0.00 1.00 0.250.98910.73060.23910.9479 7.0065 3.9543 0.00 1.00 0.500.99640.73910.21250.9540 5.5322 4.4312 0.00 1.00 1.000.98550.67310.19250.9407 6.8036 3.9632 0.50 0.00 0.000.99640.78090.25970.9600 5.2638 5.3005 0.50 0.00 0.250.99760.78090.26390.9613 5.2638 5.3005 0.50 0.00 0.500.99760.78090.26150.9600 5.2638 5.3005 0.50 0.00 1.000.99520.78090.26150.9588 5.2638 5.3005 0.50 0.25 0.000.99390.74940.23490.9407 5.0807 4.8318 0.50 0.25 0.251.00000.79660.26630.9600 4.7596 4.8039 0.50 0.25 0.500.99030.73370.23430.9407 4.8888 4.6845 0.50 0.25 1.000.99520.81840.27780.9552 5.1955 4.8183 0.50 0.50 0.000.99640.82870.30570.9625 5.8396 4.1884 0.50 0.50 0.250.99520.75790.23180.9310 5.3855 4.3443 0.50 0.50 0.500.99520.77360.23850.9625 6.2371 4.3488 0.50 0.50 1.000.99520.67620.19130.9467 6.1306 4.3766 0.50 1.00 0.000.99150.73370.24030.9492 6.7096 3.9289 0.50 1.00 0.250.99150.70820.21490.9443 5.4811 4.1878 0.50 1.00 0.500.96730.61320.19010.9334 6.9334 3.9102 0.50 1.00 1.000.98910.61680.18340.9237 6.5986 3.9541 1.00 0.00 0.000.99760.78090.25970.9600 5.2638 5.3005 30C. Luo et al m λ rank λ gn Win(g>N) ↑ Win(g>S) ↑ Win(g>G) ↑ Gen@3 ↑ FID ↓ Diversity ↑ 1.00 0.00 0.250.99760.78090.26090.9588 5.2638 5.3005 1.00 0.00 0.500.99640.78090.26090.9600 5.2638 5.3005 1.00 0.00 1.000.99760.78090.26270.9588 5.2638 5.3005 1.00 0.25 0.000.99640.80080.28510.9516 6.0285 4.2946 1.00 0.25 0.250.99390.76760.24640.9552 5.1537 4.6242 1.00 0.25 0.500.99390.78210.26820.9516 5.3639 4.5391 1.00 0.25 1.000.99880.81170.27060.9625 5.1943 4.6935 1.00 0.50 0.000.99270.75240.22880.9528 5.3754 4.3702 1.00 0.50 0.250.99520.73610.22880.9455 5.6698 4.2394 1.00 0.50 0.500.99030.71130.20100.9516 5.8942 4.3384 1.00 0.50 1.000.99150.65010.18160.9310 5.6888 4.2328 1.00 1.00 0.000.99520.65620.19730.9310 7.0648 3.9867 1.00 1.00 0.250.98490.59380.17740.9262 7.4283 3.8852 1.00 1.00 0.500.99210.59140.17980.9104 8.6083 3.6349 1.00 1.00 1.000.98310.58470.17310.9237 6.2941 3.9609 2.00 0.00 0.000.99760.78090.25670.9600 5.2638 5.3005 2.00 0.00 0.250.99760.78090.25850.9600 5.2638 5.3005 2.00 0.00 0.500.99640.78090.25790.9600 5.2638 5.3005 2.00 0.00 1.000.99760.78090.26270.9613 5.2638 5.3005 2.00 0.25 0.000.99520.76390.25120.9540 5.6781 4.4907 2.00 0.25 0.250.98910.74330.24520.9588 5.1178 4.7459 2.00 0.25 0.500.99640.78150.26030.9588 5.6664 4.3494 2.00 0.25 1.000.99390.77480.27850.9697 5.7083 4.1561 2.00 0.50 0.000.99390.72640.22280.9516 6.1482 4.1211 2.00 0.50 0.250.99640.64770.18280.9249 6.7075 3.8914 2.00 0.50 0.500.99640.63260.19010.9249 5.4215 4.1601 2.00 0.50 1.000.99090.66100.19070.9370 6.8355 3.7096 2.00 1.00 0.000.99270.64230.19980.9298 7.1093 3.8085 2.00 1.00 0.250.97150.64830.20460.9407 6.8560 3.7436 2.00 1.00 0.500.97520.63620.19070.9298 6.1279 3.8659 2.00 1.00 1.000.96550.56480.15440.9140 6.1125 4.0394 C More Details of ReactMotionNet Dataset ReactMotionNet exhibits three desirable properties for studying reactive listener motion generation. First, it provides large-scale supervision, containing over 151K labeled speaker–listener pairs. Second, it explicitly captures the one-to- many nature of listener behavior by associating each speaker utterance with mul- tiple candidate reactive motions. Third, it provides graded supervision through Gold, Silver, and Negative labels, supporting both generative modeling and preference-aware evaluation. Moreover, the dataset is split by disjoint speaker utterances, enabling a cleaner evaluation of generalization to unseen conversa- tional conditions. In total, ReactMotionNet contains 151,328 labeled speaker–listener pairs, covering 8,298 unique speaker utterances and 2,029 unique listener reactive mo- tions. On average, each speaker utterance is paired with 18.24 candidate reactive Abbreviated paper title31 (a) All (b) Train (c) Val (d) Test Fig. 7: Emotion distributions over the full dataset and across the train/validation/test splits. motions, further highlighting the inherently one-to-many nature of reactive lis- tener behavior. Among all pairs, 9,307, 34,196, and 107,825 are annotated as Gold, Silver, and Negative, respectively, reflecting the graded appropriateness of candidate reactions. We partition the dataset by speaker utterance using an 8:1:1 train/validation/test split, ensuring that utterances are disjoint across splits, i.e., no utterance appears in more than one partition. The dataset covers 47 emotion categories, including admiring, adoring, aes- thetically appreciative, amused, angry, anxious, ashamed, aware, awed, awkward, bored, calm, confused, contemplative, contemptuous, content, craving, desirous, determined, disappointed, disgusted, distressed, doubtful, ecstatic, embarrassed, empathetic (in pain), entranced, envious, excited, fearful, focused, guilty, horri- fied, interested, joyful, loving, nostalgic, pained, proud, relieved, romantic, sad, satisfied, surprised, sympathetic, tired, and triumphant. As shown in Fig. 7, these emotion labels exhibit a broad yet imbalanced distribution across the full dataset and each split, making ReactMotionNet a realistic benchmark for modeling di- verse affective conversational responses. D Additional Experimental Results D.1 Hyperparameter Sensitivity Analysis We study the sensitivity of group-wise preference training to the ranking mar- gin m, the ranking-loss weight λ rank , and the Gold-vs-Negative weight λ gn . We 32C. Luo et al Table 10: Representative hyperparameter configurations selected from the full sweep. We emphasize Gen@3, Win(g>S), and Win(g>G), together with FID and Diversity. Config m λ rank λ gn Win(g>N)↑ Win(g>S)↑ Win(g>G)↑ Gen@3↑ FID↓ Diversity↑ C12.00 0.25 1.00 0.99390.77480.27850.9697 5.7083 4.1561 C20.50 0.50 0.00 0.99640.82870.30570.9625 5.8396 4.1884 C30.00 0.50 0.00 0.99270.77600.25480.9443 4.6552 4.7315 C41.00 0.00 0.25 0.99760.78090.26090.9588 5.2638 5.3005 primarily consider Gen@3, which measures whether generated motions can be ranked among the top plausible candidates under the same candidate budget. We additionally report Win(g>S) and Win(g>G) to assess relative preference quality against medium-quality and high-quality reference candidates, respec- tively. FID and Diversity are further included to characterize motion realism and output diversity. The hyperparameter sweep reveals several consistent patterns. First, intro- ducing a small positive ranking margin is beneficial and more reliable than using no margin. Under λ rank = 0.25 and λ gn = 0.25, increasing m from 0 to 0.5 improves Win(g>S) from 0.7482 to 0.7966, Win(g>G) from 0.2240 to 0.2663, and Gen@3 from 0.9467 to 0.9600, while simultaneously reducing FID from 5.2102 to 4.7596. Although larger margins can further increase Gen@3 in certain cases, such gains are not consistently accompanied by improvements in prefer- ence alignment or motion quality, suggesting that excessively large margins may over-specialize the objective. Second, λ rank is the most sensitive hyperparameter in the sweep. Moderate ranking supervision is beneficial, whereas overly large values tend to degrade both alignment and generation quality. For instance, at m = 0.5 and λ gn = 0.25, increasing λ rank from 0.25 to 0.5 and 0.1 decreases Win(g>S) from 0.7966 to 0.7579 and 0.7082, decreases Win(g>G) from 0.2663 to 0.2318 and 0.2149, and worsens FID from 4.7596 to 5.3855 and 5.4811. This indicates that excessive ranking pressure can bias optimization toward relative ordering at the expense of generative fidelity. Third, λ gn has a secondary but non-negligible effect, with a moderate value yielding the most favorable trade-off. At m = 0.5 and λ rank = 0.25, setting λ gn = 0.25 improves Win(g>S), Win(g>G), and Gen@3 over λ gn = 0, while also reducing FID. By contrast, further increasing λ gn to 1.0 slightly improves pairwise preference scores, but lowers Gen@3 and degrades FID, indicating that stronger Gold-vs-Negative separation does not necessarily translate into better overall generation quality. Accordingly, we use m = 0.5, λ rank = 0.25, and λ gn = 0.25 in all main experiments, as this setting resides in a stable regime of the sweep and yields the most balanced overall performance across preference-oriented and generation- oriented criteria. Abbreviated paper title33 Fig. 8: Hyperparameter sensitivity heatmaps under different ranking margins. We show Gen@3, Win(g>S), and FID as functions of λ rank and λ gn . D.2 Inference Efficiency Table 11 lists the inference efficiency of the proposed ReactMotion. During infer- ence, ReactMotion runs on a single NVIDIA A100 80GB GPU and autoregres- sively generates listener motion tokens conditioned on the speaker’s multimodal inputs. In our evaluation, the model generates 50 listener reactive motions cor- responding to 50 speaker utterances. In total, it produces 1,830 motion tokens in 28.8 seconds, achieving a generation throughput of 63.6 tokens per second and 1.74 motion sequences per second, which corresponds to an average latency of approximately 0.60 seconds per listener motion sequence. The generated motion tokens are then decoded into joint sequences using the VQ-VAE decoder. The decoder processes 39.1 motion sequences per second, introducing minimal computational overhead. As a result, the complete pipeline 34C. Luo et al Table 11: Inference Efficiency on a single NVIDIA A100 (80GB). MetricValue Token generation speed63.6 tokens/s Motion generation speed1.74 turns/s End-to-end generation speed 1.66 turns/s Average latency per sample ∼0.60 s VQ-VAE decoding speed39.12 turns/s achieves an end-to-end throughput of 1.66 motion sequences per second. These results indicate that ReactMotion maintains a favorable balance between model capacity and inference efficiency, enabling near real-time reactive motion gener- ation in conversational scenarios. D.3 More Details of User Study We conducted a user study on the Tencent Questionnaire platform to evaluate the listener motions generated by ReactMotion (Ours) against two generative baselines, namely the CE variant and LLM→MG-MotionLLM * , as well as the best-in-group Silver reference. A total of 59 volunteers (16 female and 43 male), all with relevant backgrounds in machine learning or deep learning, par- ticipated in the study through an online survey. In each trial, participants were presented with a pair of listener-motion videos (A/B) conditioned on the same speaker utterance, with the speaker’s transcript displayed and the correspond- ing audio played. They were asked to choose which video exhibited the more appropriate reactive listener motion. To avoid positional bias, the two compared motions were randomly assigned to the A/B positions. Each participant com- pleted 36 trials, covering six speaker utterances with six pairwise comparisons per condition. For the Silver condition, we selected the best candidate within each speaker-condition group based on its motion caption and rendered motion clip. The results in Fig. 5 reveal three notable findings. First, Ours is consistently preferred over both generative baselines, achieving 67.8% preference against CE and 72.0% against LLM→MG-MotionLLM, which demonstrates the advan- tage of our unified multimodal Seq2Seq formulation over both standard CE training and cascaded generation pipelines. Second, although the Silver refer- ence remains stronger overall, Ours is substantially closer to Silver than either baseline: Ours receives 44.1% of the votes against Silver, whereas CE and LLM→MG-MotionLLM receive only 31.9% and 31.4%, respectively. This in- dicates that the motions generated by Ours are perceptually much closer to high-quality in-group references. Third, these results highlight the effectiveness of the proposed group-wise preference learning objective, which explicitly mod- els the ordering among Gold, Silver, and Negative reactions and leads to more appropriate listener behaviors under human evaluation. At the same time, the remaining gap between Ours and Silver suggests that reactive listener motion Abbreviated paper title35 generation remains challenging, leaving room for further improvement in motion naturalness, contextual precision, and diversity. D.4 Failure Cases While the model effectively generates contextually appropriate listener motions in many scenarios, capturing deeper conversational intent in complex dialogues remains challenging. In ambiguous or long-tail situations where appropriate lis- tener behavior requires deeper intent understanding, the current model may still exhibit limited robustness. This highlights a promising research direction for future work to further enhance intent-aware interaction modeling in dyadic interaction. E Limitations Since we are the first to explore this task, we design a relatively simple yet effective model architecture to maintain training stability and computational ef- ficiency. This design allows us to validate the core idea of our approach without introducing excessive architectural complexity. The proposed approach already achieves promising results, demonstrating its feasibility and effectiveness. Nev- ertheless, there remains a large potential for further improvement. Future work could explore more advanced network architectures and more sophisticated train- ing techniques to further enhance performance. References 1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report (2023) 2. Alexanderson, S., Nagy, R., Beskow, J., Henter, G.E.: Listen, denoise, action! audio-driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG) 42(4), 1–20 (2023) 3. Ao, T., Gao, Q., Lou, Y., Chen, B., Liu, L.: Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings. ACM Transac- tions on Graphics (TOG) 41(6), 1–19 (2022) 4. Barquero, G., Escalera, S., Palmero, C.: Seamless human motion composition with blended positional encodings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024) 5. Bishop, C.M., Nasrabadi, N.M.: Pattern recognition and machine learning, vol. 4. Springer (2006) 6. Bradley, R.A., Terry, M.E.: Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika 39(3/4), 324–345 (1952) 7. Chen, B., Li, Y., Ding, Y.X., Shao, T., Zhou, K.: Enabling synergistic full-body control in prompt-based co-speech motion generation. In: Proceedings of the ACM International Conference on Multimedia (ACM M). p. 6774–6783 (2024) 36C. Luo et al 8. Chen, C., Zhang, J., Lakshmikanth, S.K., Fang, Y., Shao, R., Wetzstein, G., Fei-Fei, L., Adeli, E.: The language of motion: Unifying verbal and non-verbal language of 3d human motion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 6200–6211 (June 2025) 9. Chen, C., Zhang, J., Lakshmikanth, S.K., Fang, Y., Shao, R., Wetzstein, G., Fei- Fei, L., Adeli, E.: The language of motion: Unifying verbal and non-verbal lan- guage of 3d human motion. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 6200–6211 (2025) 10. Chen, X., Jiang, B., Liu, W., Huang, Z., Fu, B., Chen, T., Yu, G.: Executing your commands via motion diffusion in latent space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 18000– 18010 (2023) 11. Chiang, W.L., Zheng, L., Sheng, Y., Angelopoulos, A.N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J.E., et al.: Chatbot arena: An open platform for evaluating llms by human preference. In: The International Conference on Machine Learning (ICML) (2024) 12. Chopin, B., Tang, H., Otberdout, N., Daoudi, M., Sebe, N.: Interaction trans- former for human reaction generation. IEEE Transactions on Multimedia (TMM) 25, 8842–8854 (2023) 13. Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems (NeurIPS) 30 (2017) 14. Chu, X., Liu, R., Huang, Y., Liu, Y., Peng, Y., Zheng, B.: Unils: End-to-end audio- driven avatars for unified listening and speaking. arXiv preprint arXiv:2512.09327 (2025) 15. Défossez, A., et al.: Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037 (2024) 16. Dubois, Y., Galambosi, B., Liang, P., Hashimoto, T.B.: Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475 (2024) 17. Ghosh, A., Dabral, R., Golyanik, V., Theobalt, C., Slusallek, P.: Remos: 3d motion-conditioned reaction synthesis for two-person interactions. In: European Conference on Computer Vision (ECCV). p. 418–437 (2024) 18. Guo, C., Mu, Y., Javed, M.G., Wang, S., Cheng, L.: Momask: Generative masked modeling of 3d human motions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 1900–1910 (2024) 19. Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., Cheng, L.: Generating diverse and natural 3d human motions from text. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 5152– 5161 (2022) 20. Guo, W., Bie, X., Alameda-Pineda, X., Moreno-Noguer, F.: Multi-person extreme motion prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 13053–13064 (2022) 21. Han, B., Peng, H., Dong, M., Ren, Y., Shen, Y., Xu, C.: AMD: autoregressive motion diffusion. In: Wooldridge, M.J., Dy, J.G., Natarajan, S. (eds.) The As- sociation for the Advancement of Artificial Intelligence (AAAI). p. 2022–2030 (2024) 22. He, X., Huang, Q., Zhang, Z., Lin, Z., Wu, Z., Yang, S., Li, M., Chen, Z., Xu, S., Wu, X.: Co-speech gesture video generation via motion-decoupled diffusion model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 2263–2273 (2024) Abbreviated paper title37 23. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS) 30 (2017) 24. Ho, L., Huang, Y., Qin, D., Shi, M., Tse, W., Liu, W., Yamagishi, J., Komura, T.: Interact: A large-scale dataset of dynamic, expressive and interactive activities between two people in daily scenarios. Proceedings of the ACM on Computer Graphics and Interactive Techniques (PACMCGIT) 8(4), 1–27 (2025) 25. Hu, T., Zhu, X., Guo, W., Su, K.: Efficient interaction recognition through positive action representation. Mathematical Problems in Engineering 2013(1), 795360 (2013) 26. Huang, Y., Wan, W., Yang, Y., Callison-Burch, C., Yatskar, M., Liu, L.: Como: Controllable motion generation through language guided pose code editing. In: European Conference on Computer Vision (ECCV). p. 180–196 (2024) 27. Huang, Y., Khan, S.M.: Dyadgan: Generating facial expressions in dyadic inter- actions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017) 28. Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card (2024) 29. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al.: Openai o1 system card (2024) 30. Jeong, M., Hwang, Y., Lee, J., Jung, S., Kim, W.H.: Hgm 3 : Hierarchical generative masked motion modeling with hard token mining. In: International Conference on Learning Representations (ICLR) (2025) 31. Khirodkar, R., Bansal, A., Ma, L., Newcombe, R., Vo, M., Kitani, K.: Ego- humans: An ego-centric 3d multi-human benchmark. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 19807– 19819 (2023) 32. Khirodkar, R., Song, J.T., Cao, J., Luo, Z., Kitani, K.: Harmony4d: A video dataset for in-the-wild close human interactions. Advances in Neural Information Processing Systems (NeurIPS) 37, 107270–107285 (2024) 33. Kim, D.Y., Lee, H.K., Chung, K.: Avatar-mediated experience in the metaverse: The impact of avatar realism on user-avatar relationship. Journal of Retailing and Consumer Services 73, 103382 (2023) 34. Kim, J., Kim, J., Choi, S.: Flame: Free-form language-based motion synthesis & editing. In: The Association for the Advancement of Artificial Intelligence (AAAI). vol. 37, p. 8255–8263 (2023) 35. Ko, W.R., Jang, M., Lee, J., Kim, J.: Air-act2act: Human–human interaction dataset for teaching non-verbal social behaviors to robots. The International Jour- nal of Robotics Research 40(4-5), 691–697 (2021) 36. Lee, G., Deng, Z., Ma, S., Shiratori, T., Srinivasa, S.S., Sheikh, Y.: Talking with hands 16.2 m: A large-scale dataset of synchronized body-finger motion and audio for conversational motion analysis and synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 763–772 (2019) 37. Li, B., Zhao, Y., Zhelun, S., Sheng, L.: Danceformer: Music conditioned 3d dance generation with parametric motion transformer. In: The Association for the Ad- vancement of Artificial Intelligence (AAAI). vol. 36, p. 1272–1279 (2022) 38. Li, J., Kang, D., Pei, W., Zhe, X., Zhang, Y., Bao, L., He, Z.: Audio2gestures: Generating diverse gestures from audio. IEEE Transactions on Visualization and Computer Graphics (TVCG) 30(8), 4752–4766 (2023) 38C. Luo et al 39. Li, R., Dai, Y., Zhang, Y., Li, J., Yang, J., Guo, J., Li, X.: Exploring multi-modal control in music-driven dance generation. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). p. 8281–8285 (2024) 40. Li, R., Zhang, H., Zhang, Y., Zhang, Y., Zhang, Y., Guo, J., Zhang, Y., Li, X., Liu, Y.: Lodge++: High-quality and long dance generation with robust choreog- raphy patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) p. 1–15 (2025) 41. Liang, H., Zhang, W., Li, W., Yu, J., Xu, L.: Intergen: Diffusion-based multi-human motion generation under complex interactions. arXiv preprint arXiv:2304.05684 (2023) 42. Liao, T.H., Zhou, Y., Shen, Y., Huang, C.H.P., Mitra, S., Huang, J.B., Bhat- tacharya, U.: Shape my moves: Text-driven shape-aware synthesis of human mo- tions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 1917–1928 (2025) 43. Liu, H., Zhu, Z., Becherini, G., Peng, Y., Su, M., Zhou, Y., Zhe, X., Iwamoto, N., Zheng, B., Black, M.J.: Emage: Towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 1144–1154 (2024) 44. Liu, P., Song, L., Huang, J., Liu, H., Xu, C.: Gesturelsm: Latent shortcut based co-speech gesture generation with spatial-temporal modeling. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 10929– 10939 (2025) 45. Liu, Y., Cao, Q., Wen, Y., Jiang, H., Ding, C.: Towards variable and coordinated holistic co-speech motion generation. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR). p. 1566–1576 (2024) 46. Liu, Y., Chen, C., Ding, C., Yi, L.: Physreaction: Physically plausible real-time humanoid reaction synthesis via forward dynamics guided 4d imitation. In: Pro- ceedings of the ACM International Conference on Multimedia (ACM M). p. 3771–3780 (2024) 47. Liu, Y., Chen, C., Yi, L.: Interactive humanoid: Online full-body motion reaction synthesis with social affordance canonicalization and forecasting (2023) 48. Lu, S., Wang, J., Lu, Z., Chen, L.H., Dai, W., Dong, J., Dou, Z., Dai, B., Zhang, R.: Scamo: Exploring the scaling law in autoregressive motion generation model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 27872–27882 (2025) 49. Luo, C., Song, S., Yan, S., Yu, Z., Ge, Z.: Reactdiff: Fundamental multiple appro- priate facial reaction diffusion model. In: Proceedings of the ACM International Conference on Multimedia (ACM M). p. 5607–5616 (2025) 50. Luo, C., Wang, J., Li, B., Song, S., Ghanem, B.: Omniresponse: Online multi- modal conversational response generation in dyadic interactions. In: Advances in Neural Information Processing Systems (NeurIPS) (2025) 51. Luo, C., et al.: Reactface: Online multiple appropriate facial reaction generation in dyadic interactions. arXiv preprint (2024), arXiv:2305.15748 52. Meng, Z., Xie, Y., Peng, X., Han, Z., Jiang, H.: Rethinking diffusion for text- driven human motion generation: Redundant representations, evaluation, and masked autoregression. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR). p. 27859–27871 (2025) Abbreviated paper title39 53. Mughal, M.H., Dabral, R., Habibie, I., Donatelli, L., Habermann, M., Theobalt, C.: Convofusion: Multi-modal conversational diffusion for co-speech gesture syn- thesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 1388–1398 (2024) 54. Murray, K., Chiang, D.: Correcting length bias in neural machine translation. In: Proceedings of the Conference on Machine Translation (WMT). p. 212–223 (2018) 55. Ng, E., Romero, J., Bagautdinov, T., Bai, S., Darrell, T., Kanazawa, A., Richard, A.: From audio to photoreal embodiment: Synthesizing humans in conversations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 1001–1010 (2024) 56. Ng, E., Xiang, D., Joo, H., Grauman, K.: You2me: Inferring body pose in ego- centric video via first and second person interactions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 9890–9900 (2020) 57. Ng, E., et al.: Learning to listen: Modeling non-deterministic dyadic facial motion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022) 58. OpenAI: Introducing openai o3 and o4-mini. https://openai.com/index/ openai-o3-mini/ (2025) 59. Park, S., Kim, C., Rha, H., Kim, M., Hong, J., Yeo, J., Ro, Y.: Let’s go real talk: Spoken dialogue model for face-to-face conversation. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). p. 16334–16348 (2024) 60. Petrovich, M., Black, M.J., Varol, G.: Action-conditioned 3d human motion syn- thesis with transformer vae. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV) (2021) 61. Petrovich, M., Black, M.J., Varol, G.: Temos: Generating diverse human motions from textual descriptions. In: European Conference on Computer Vision (ECCV). p. 480–497 (2022) 62. Petrovich, M., Litany, O., Iqbal, U., Black, M.J., Varol, G., Bin Peng, X., Rempe, D.: Multi-track timeline control for text-driven 3d human motion generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 1911–1921 (2024) 63. Pinyoanuntapong, E., Saleem, M.U., Karunratanakul, K., Wang, P., Xue, H., Chen, C., Guo, C., Cao, J., Ren, J., Tulyakov, S.: Controlmm: Controllable masked motion generation (2024) 64. Raab, S., Leibovitch, I., Li, P., Aberman, K., Sorkine-Hornung, O., Cohen-Or, D.: Modi: Unconditional motion synthesis from diverse data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 13873–13883 (2023) 65. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research (JMLR) 21(140), 1–67 (2020) 66. Rubenstein, P.K., et al.: Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925 (2023) 67. Ryoo, M.S., Fuchs, T.J., Xia, L., Aggarwal, J.K., Matthies, L.: Robot-centric ac- tivity prediction from first-person videos: What will they do to me? In: Proceed- ings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI). p. 295–302 (2015) 40C. Luo et al 68. Ryoo, M.S., Matthies, L.: First-person activity recognition: What are they doing to me? In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 2730–2737 (2013) 69. Singh, A., Fry, A., Perelman, A., Tart, A., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., Ananthram, A., et al.: Openai gpt-5 system card (2025) 70. Song, S., et al.: React 2024: the second multiple appropriate facial reaction gen- eration challenge. arXiv preprint (2024), arXiv:2401.05166 71. Spaccatini, F., Corlito, G., Sacchi, S.: New dyads? the effect of social robots’ anthropomorphization on empathy towards human beings. Computers in Human Behavior 146, 107821 (2023) 72. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., Christiano, P.F.: Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems (NeurIPS) 33, 3008–3021 (2020) 73. Sun, M., Xu, C., Jiang, X., Liu, Y., Sun, B., Huang, R.: Beyond talking–generating holistic 3d human dyadic motion for communication. International Journal of Computer Vision 133(5), 2910–2926 (2025) 74. Tevet, G., Gordon, B., Hertz, A., Bermano, A.H., Cohen-Or, D.: Motionclip: Exposing human motion generation to clip space. In: European Conference on Computer Vision (ECCV). p. 358–374 (2022) 75. Tevet, G., Raab, S., Cohan, S., Reda, D., Luo, Z., Peng, X.B., Bermano, A.H., van de Panne, M.: Closd: Closing the loop between simulation and diffusion for multi-task character control. In: International Conference on Learning Represen- tations (ICLR) (2025) 76. Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., Bermano, A.H.: Human motion diffusion model. arXiv preprint arXiv:2209.14916 (2022), also known as MDM; widely used as a diffusion baseline. 77. Veluri, B., Peloquin, B.N., Yu, B., Gong, H., Gollakota, S.: Beyond turn-based interfaces: Synchronous llms as full-duplex dialogue agents. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). p. 21390–21402 (2024) 78. Wang, T., Wu, Z., He, Q., Chu, J., Qian, L., Cheng, Y., Xing, J., Zhao, J., Jin, L.: Stickmotion: Generating 3d human motions by drawing a stickman. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 12370–12379 (2025) 79. Wang, Y., Leng, Z., Li, F.W., Wu, S.C., Liang, X.: Fg-t2m: Fine-grained text- driven human motion generation via diffusion model. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 22035– 22044 (2023) 80. Wang, Y., Li, M., Liu, J., Leng, Z., Li, F.W., Zhang, Z., Liang, X.: Fg-t2m++: Llms-augmented fine-grained text driven human motion generation. International Journal of Computer Vision (IJCV) 133(7), 4277–4293 (2025) 81. Wang, Z., Wang, J., Li, Y., Lin, D., Dai, B.: Intercontrol: Zero-shot human inter- action generation by controlling every joint. In: Advances in Neural Information Processing Systems (NeurIPS) (2024) 82. Wu, B., Xie, J., Shen, K., Kong, Z., Ren, J., Bai, R., Qu, R., Shen, L.: Mg- motionllm: A unified framework for motion comprehension and generation across multiple granularities. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 27849–27858 (2025) 83. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation (2016) Abbreviated paper title41 84. Xiao, L., Lu, S., Pi, H., Fan, K., Pan, L., Zhou, Y., Feng, Z., Zhou, X., Peng, S., Wang, J.: Motionstreamer: Streaming motion generation via diffusion-based autoregressive model in causal latent space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 10086– 10096 (2025) 85. Xu, C., Sun, M., Cheng, Z.Q., Wang, F., Liu, Y., Sun, B., Huang, R., Haupt- mann, A.: Combo: Co-speech holistic 3d human motion generation and efficient customizable adaptation in harmony. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) p. 1–18 (2025) 86. Xu, L., Lv, X., Yan, Y., Jin, X., Wu, S., Xu, C., Liu, Y., Zhou, Y., Rao, F., Sheng, X., et al.: Inter-x: Towards versatile human-human interaction analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 22260–22271 (2024) 87. Xu, L., Zhou, Y., Yan, Y., Jin, X., Zhu, W., Rao, F., Yang, X., Zeng, W.: Regen- net: Towards human action-reaction synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 1759–1769 (2024) 88. Xu, S., Dou, Z., Shi, M., Pan, L., Ho, L., Wang, J., Liu, Y., Lin, C., Ma, Y., Wang, W., et al.: Mospa: Human motion generation driven by spatial audio (2025) 89. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., Qiu, Z.: Qwen3 technical report (2025) 90. Yang, Y., Huang, Z., Xu, C., He, S.: Lagrangian motion fields for long-term mo- tion generation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 48(2), 1171–1184 (2026) 91. Yi, H., Liang, H., Liu, Y., Cao, Q., Wen, Y., Bolkart, T., Tao, D., Black, M.J.: Generating holistic 3d human motion from speech. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 469–480 (2023) 92. Yin, Y., Guo, C., Kaufmann, M., Zarate, J.J., Song, J., Hilliges, O.: Hi4d: 4d in- stance segmentation of close human interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 17016– 17027 (2023) 93. Yu, C., Zhai, W., Yang, Y., Cao, Y., Zha, Z.J.: Hero: Human reaction genera- tion from videos. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 10262–10274 (2025) 94. Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., Qiu, X.: Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities. In: Findings of the Association for Computational Linguistics. p. 15757– 15773 (2023) 95. Zhang, J., Fan, H., Yang, Y.: Energymogen: Compositional human motion gen- eration with energy-based diffusion model in latent space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 17592–17602 (2025) 42C. Luo et al 96. Zhang, J., Zhang, Y., Cun, X., Zhang, Y., Zhao, H., Lu, H., Shen, X., Shan, Y.: Generating human motion from textual descriptions with discrete representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 14730–14740 (2023) 97. Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., Liu, Z.: Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 46(6), 4115–4128 (2024) 98. Zhang, P., Liu, P., Garrido, P., Kim, H., Chaudhuri, B.: Kinmo: Kinematic-aware human motion understanding and generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 11187–11197 (2025) 99. Zhang, X., Li, J., Zhang, J., Dang, Z., Ren, J., Bo, L., Tu, Z.: Semtalk: Holistic co-speech motion generation with frame-level semantic emphasis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 13761–13771 (2025) 100. Zhang, X., Li, J., Zhang, J., Ren, J., Bo, L., Tu, Z.: Echomask: Speech-queried attention-based mask modeling for holistic co-speech motion generation. In: Pro- ceedings of the ACM International Conference on Multimedia (ACM M). p. 10827–10836 (2025) 101. Zhang, Y., Huang, D., Liu, B., Tang, S., Lu, Y., Chen, L., Bai, L., Chu, Q., Yu, N., Ouyang, W.: Motiongpt: Finetuned llms are general-purpose motion generators. In: The Association for the Advancement of Artificial Intelligence (AAAI). vol. 38, p. 7368–7376 (2024) 102. Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.: Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems (NeurIPS) 36, 46595– 46623 (2023) 103. Zhou, M., Bai, Y., Zhang, W., Yao, T., Zhao, T., Mei, T.: Responsive listening head generation: A benchmark dataset and baseline. In: European Conference on Computer Vision (ECCV) (2022) 104. Zhu, Y., Zhang, L., Rong, Z., Hu, T., Liang, S., Ge, Z.: Infp: Audio-driven inter- active head generation in dyadic conversations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025) 105. Zou, Q., Yuan, S., Du, S., Wang, Y., Liu, C., Xu, Y., Chen, J., Ji, X.: Parco: Part-coordinating text-to-motion synthesis. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds.) European Conference on Computer Vision (ECCV). vol. 15114, p. 126–143 (2024)