Paper deep dive

SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models

Tianyu Xie, Jinfa Huang, Yuexiao Ma, Rongfang Luo, Yan Yang, Wang Chen, Yuhui Zeng, Ruize Fang, Yixuan Zou, Xiawu Zheng, Jiebo Luo, Rongrong Ji

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 80

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 3/22/2026, 5:47:41 AM

Summary

SocialOmni is a comprehensive benchmark designed to evaluate the social interactivity of Omni-modal Large Language Models (OLMs) across three dimensions: speaker identification (who), interruption timing control (when), and natural interruption generation (how). It features 2,000 perception samples and 209 interaction-generation instances, revealing a decoupling between perceptual accuracy and conversational competence in current models.

Entities (5)

Omni-modal large language models · model-architecture · 100%SocialOmni · benchmark · 100%Interruption Timing Control · task · 95%Natural Interruption Generation · task · 95%Speaker Identification · task · 95%

Relation Signals (4)

SocialOmni → evaluates → Omni-modal large language models

confidence 100% · We benchmarked 12 leading OLMs, which uncovers significant variance in their social-interaction capabilities

SocialOmni → includestask → Speaker Identification

confidence 100% · SocialOmni operationalizes the evaluation of this conversational interactivity across three core dimensions: (i) speaker separation and identification

SocialOmni → includestask → Interruption Timing Control

confidence 100% · SocialOmni operationalizes the evaluation of this conversational interactivity across three core dimensions: (ii) interruption timing control

SocialOmni → includestask → Natural Interruption Generation

confidence 100% · SocialOmni operationalizes the evaluation of this conversational interactivity across three core dimensions: (iii) natural interruption generation

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Omni-modal large language models (OLMs) redefine human-machine interaction by natively integrating audio, vision, and text. However, existing OLM benchmarks remain anchored to static, accuracy-centric tasks, leaving a critical gap in assessing social interactivity, the fundamental capacity to navigate dynamic cues in natural dialogues. To this end, we propose SocialOmni, a comprehensive benchmark that operationalizes the evaluation of this conversational interactivity across three core dimensions: (i) speaker separation and identification (who is speaking), (ii) interruption timing control (when to interject), and (iii) natural interruption generation (how to phrase the interruption). SocialOmni features 2,000 perception samples and a quality-controlled diagnostic set of 209 interaction-generation instances with strict temporal and contextual constraints, complemented by controlled audio-visual inconsistency scenarios to test model robustness. We benchmarked 12 leading OLMs, which uncovers significant variance in their social-interaction capabilities across models. Furthermore, our analysis reveals a pronounced decoupling between a model's perceptual accuracy and its ability to generate contextually appropriate interruptions, indicating that understanding-centric metrics alone are insufficient to characterize conversational social competence. More encouragingly, these diagnostics from SocialOmni yield actionable signals for bridging the perception-interaction divide in future OLMs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

79,822 characters extracted from source content.

Expand or collapse full text

SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models Tianyu Xie 1,2 , Jinfa Huang 5 , Yuexiao Ma 1,3 , Rongfang Luo 4 Yan Yang 4 , Wang Chen 1,2 , Yuhui Zeng 1,2 , Ruize Fang 1,2 Yixuan Zou 1,2 , Xiawu Zheng 1,2,B , Jiebo Luo 5 , Rongrong Ji 1,2,3 1 Media Analytics and Computing Lab, Xiamen University, Xiamen, China 2 Institute of Artificial Intelligence, Xiamen University, Xiamen, China 3 School of Informatics, Xiamen University, Xiamen, China 4 Sichuan Agricultural University, Yaan, China 5 Department of Computer Science, University of Rochester, Rochester, NY, USA B Corresponding Author ABSTRACT Omni-modal large language models (OLMs) redefine human-machine interaction by natively integrating audio, vision, and text. However, existing OLM benchmarks remain anchored to static, accuracy-centric tasks, leaving a critical gap in assessing social interactivity, the fundamental capacity to navigate dynamic cues in natural dialogues. To this end, we propose SocialOmni, a comprehensive benchmark that operationalizes the evaluation of this conversational interactivity across three core dimensions: (i) speaker separation and identification (who is speaking), (i) interruption timing control (when to interject), and (i) natural interruption generation (how to phrase the interruption). SocialOmni features 2,000 perception samples and a quality-controlled diagnostic set of 209 interaction-generation instances with strict temporal and contextual con- straints, complemented by controlled audio-visual inconsistency scenarios to test model robustness. We benchmarked 12 leading OLMs, which uncovers significant variance in their social-interaction capabilities across models. Furthermore, our analysis reveals a pronounced decoupling between a model’s perceptual accuracy and its ability to generate contextually appropriate interruptions, indicating that understanding-centric metrics alone are insufficient to characterize conversational social competence. More encouragingly, these diagnostics from our SocialOmni yield actionable signals for bridging the perception-interaction divide in future OLMs. Email: Tianyu Xie teery@stu.xmu.edu.cn Project Page: github.com/MAC-AutoML/SocialOmni Data: huggingface.co/datasets/alexisty/SocialOmni 1 Introduction Omni-modal large language models (OLMs) support real-time multimodal conversation by continuously integrating audio, vision, and text within a unified generation loop [7,10,18,25,50,51]. In such settings, success depends not only on producing correct content but also on genuine interaction competence: perceiving and responding to dynamic dialog cues, deciding when to speak, and generating socially coherent responses. As summarized in Table 1, existing OLM benchmarks remain anchored to static, accuracy-centric understanding tasks [14, 22, 26, 57], leaving this interaction capability largely unevaluated. This gap motivates benchmarks that evaluate interaction competence beyond mere answer correctness. However, as shown in Table 1, existing evaluation paradigms remain insufficient to capture the full spectrum 1 arXiv:2603.16859v1 [cs.AI] 17 Mar 2026 Table 1. Positioning of OLM benchmarks under a social-interactivity lens. We compare existing representative benchmarks by whether they explicitly operationalize who (speaker identification), when (turn timing), how (interruption generation), robustness to conflict (audio–visual inconsistency), and temporal granularity of evaluation. (✓: explicitly evaluated by task design; ~ : partially covered via indirect proxies, e.g., QA/localization outcomes;✗: not explicitly evaluated. Granularity:•◦ranging from Global-level to•Frame-level.) BenchmarkTypesWhoWhenHowConflict Temporal Granularity OmniBench [26]Understanding✗•◦ OmniVideoBench [22]Understanding ~~ ✗•◦ WorldSense [14]Understanding ~~ ✗•◦ OmniEval [57]Understanding ~~ ✗•◦ Daily-Omni [61]Understanding ~~ ✗•◦ JointAVBench [3]Understanding ~~ ✗•◦ OmniMMI [47]Interaction ~~~ ✗•◦ Omni-SafetyBench [39]Understanding✗ ~ •◦ SocialOmni (Ours)Interaction✓• of dynamic conversational abilities. Prior work can be broadly categorized into two groups. Answer-centric benchmarks focus on what a model knows by posing static question-answering or retrieval tasks over pre- segmented audio-visual clips [14,22,26,57], measuring propositional accuracy alone. While effective for isolating perceptual and reasoning skills, these benchmarks treat queries independently and thus fail to assess coherent understanding across multi-turn dialogues, neglecting crucial conversational dynamics. In contrast, behavior-centric benchmarks explore how models act within context, probing skills such as multi-speaker perception [3, 20], socially grounded reasoning [6, 38], or daily conversational inference [61]. Although these benchmarks advance beyond answer correctness by targeting interactive behaviors, they typically isolate single facets—e.g., speaker diarization or emotion recognition—without simultaneous evaluation of perception, reasoning, and social appropriateness. Consequently, neither family sufficiently addresses the integrated, multimodal, and social complexities inherent to real-world dialogue, where models must interpret evolving context, understand multimodal cues, and respond coherently and appropriately in real time. This limitation is consequential. In live dialogue, utility critically depends jointly on semantic correctness and social timing: a delayed turn entry, a premature interruption, or an incoherent topic continuation can each substantially degrade user experience even when the propositional content is accurate [44]. If evaluation remains fixated on correctness alone, model selection will inevitably systematically over-reward offline comprehension while under-penalizing such interaction failures [52]. To close this gap, we propose SocialOmni, a benchmark that operationalizes social interactivity evaluation across three core dimensions: SocialOmni: Three Dimensions of Social Interactivity Who (speaker identification): identifying speakers by integrating multimodal information including visual cues, acoustic features, and contextual dialogue history across multiple speakers. When (interruption timing control): determining optimal timing and strategy for interruptions by analyzing dialogue dynamics and turn-taking patterns in real-time. How (natural interruption generation): producing a response fitting the ongoing dialogue context while maintaining coherence with speaker intent and conversation flow. Accordingly, SocialOmni comprehensively tests the end-to-end pipeline from precise audio-visual grounding to turn-entry decision and then to adaptive on-the-fly continuation under strict latency constraints. Beyond defining evaluation targets, these dimensions expose fundamental concrete architectural challenges for current OLMs: Who requires fine-grained audio-visual alignment beyond the temporal granularity of most video encoders; When demands nuanced fusion of prosodic, lexical, and visual turn-taking cues under dynamically shifting salience; and How stresses robust real-time generation of contextually grounded continuations under cross-modal attention and latency constraints. SocialOmni comprises 2,000 perception samples and a quality- controlled diagnostic set of 209 interaction-generation instances across 15 dialogue domains, with systematic 2 controlled audio-visual inconsistency scenarios designed to probe robustness under cross-modal conflict. We evaluate 12 OLMs and observe two recurring patterns. First, models exhibit systematic markedly different error profiles across who–when–how, indicating that substantial gains on one axis do not imply robustness on the others. Second, we observe a pronounced decoupling between perceptual accuracy and interruption- generation quality: models that excel at speaker identification do not always produce natural interruptions. These results show that understanding-centric benchmarks alone are fundamentally insufficient to characterize conversational social competence, motivating dedicated interaction-oriented evaluation. Our contributions are threefold: i)New Omni Models Benchmark. We introduce SocialOmni, a comprehensive benchmark for evaluating audio-visual social interaction understanding along three axes: who, when, and how. i)New Dual-Axis Evaluation Protocol. We propose a protocol that couples frame-level perception diagnosis with multi-judge generation scoring, enabling perception-generation decoupling analysis. i) New Robustness Probes. We design controlled mismatch probes that systematically quantify model robustness and generalization under realistic audio-visual conflict scenarios. 2 Related Work Omni-Modal Large Language Models. Multimodal modeling has rapidly evolved from perception- centric paradigms such as CLIP [40] and instruction-tuned VLMs like Flamingo and LLaVA [1,29] toward omni-modal large language models (OLMs) that natively couple text, vision, and audio within a unified interaction loop [7,10,18,25,48,50,51]. Recent studies on multimodal perception and representation learning also broaden the design space of modern MLLMs [23,55,56]. From a system-design perspective, diverse OLM stacks range from dispatch designs, where a central LLM orchestrates external ASR, VAD, diarization, and visual grounding modules, to native designs that tighten cross-modal coupling inside a single generation loop [7,10,12,13,18,25,49,50,52,54]. Simultaneously, scalable deployment motivates parallel work on adaptation, pruning, quantization, and efficiency optimization for large models [15–17,34–37,59]. Yet even highly capable systems remain largely evaluated under turn-level request–response protocols, leaving open whether they can proactively decide when to enter a conversation, who to address in multi-party settings, and how to realize interruptions in socially coherent ways, as highlighted by classical turn-taking theory [42, 44]. Moreover, common architectural choices such as sparse temporal sampling, coarse cross-modal alignment, and turn-level segmentation systematically mask timing errors surfacing only under fine-grained, real-time conditions, an issue closely related to recent efforts on semantic-boundary-based frame selection, event-anchored sampling, query-oriented token budgeting, and retrieval-augmented long-video comprehension [4,5,32,33], thereby motivating benchmarks explicitly testing interaction competence beyond answer correctness. Answer-Centric Benchmarks for OLMs. Comprehensive broad-coverage omni suites and modality- specific understanding benchmarks [9,14,21,22,24,26,31,45,53,57,60] evaluate what a model knows by posing question-answering or retrieval tasks over pre-segmented multimodal stimuli and measuring factual propositional accuracy of the response. Cross-modal QA suites [26,57] pair audio-visual clips with factual questions and score models on answer correctness under unified metrics, while domain-specific benchmarks such as MMMU [53] and AudioBench [45] probe expert-level comprehension within individual modalities through multiple-choice or open-ended question answering, again using answer accuracy as the sole evaluation signal. Video understanding benchmarks [9,22,24,60] extend this paradigm to temporal reasoning by querying event ordering or causal relations across frames, yet still treat each question as an isolated, single-turn trial. Although these efforts have substantially expanded perceptual and reasoning coverage, they share a fundamental common structural limitation: evaluation is confined to static prompt-response pairs and does not enforce temporal alignment at frame level, turn-entry decisions, or interruption handling within an unfolding dialogue. Consequently, strong answer accuracy does not imply reliable interaction behavior under real-time, multi-party constraints, leaving the behavior-centric dimension largely unaddressed. Behavior-Centric Benchmarks for OLMs. Recent efforts have systematically begun to probe inter- active behavior. Social reasoning benchmarks [20,38] target multi-speaker inference and social attribute understanding, yet do not evaluate turn-entry timing or interruption strategy. Spoken-dialogue and full-duplex 3 c o n s i s t e n t i n c o n s i s t e n t Sports Education Real Daily General Food Travel Fashion Technology Health Art Others 394 336 286 218 152 136 110 96 84 78 70 64 56 50 70 2209 Videos15 Domains 2 Tasks (a) SocialOmni Benchmark Overview Entertainment Task I: Who — Perception Task I: When & How — Generation At timestamp [t],who is speaking: (A) [Person description 1] : Person Confusion (B) [Person description 2] : Person Confusion (C) [Alternative] (D) [Alternative] Video Context: (from 0.s - t.s) Question 1:Should speak at [t]? score continue Video Context: (full) Question 2:Should say Sth? (role-playing) (scoring) (b) Task Design (c) Models Performance omni models Task I Task I Figure 1. Overview of SocialOmni. (a) Benchmark data distribution across 15 subcategories and four domains, with consistent/inconsistent stratification and perception/generation task splits. (b) Overview of the proposed evaluation tasks and metrics. (c) Performance comparison of 12 OLMs on both Task I and Task I. benchmarks [2,19,27,28,43,46] emphasize turn-taking timing and interruption detection, yet predominantly often operate under audio-only stimuli with limited speaker grounding or multimodal conflict control. Mul- timodal interaction benchmarks [3,6,39,47,61] introduce joint audio-visual conversational settings, but frequently lack frame-level temporal supervision and diagnostic control of cross-modal conflict. Although each line of work advances one facet, to the best of our knowledge, no existing benchmark simultaneously operationalizes the integrated triad required for full-duplex multi-party conversation: speaker attribution (who), turn-entry decision (when), and interruption realization (how). In real dialogue, the three are causally entangled, a correct presupposition of who, and the appropriateness of how depends on both. Therefore, evaluating them in isolation can significantly systematically overestimate interactive competence by masking failure cascades. This means that the joint, integrated evaluation of multi-party interaction competence under fine-grained temporal alignment and controlled cross-modal conflict remains an open problem. 3 SocialOmni: Evaluating Omni-Modal Multi-Party Interactivity We propose SocialOmni, a comprehensive benchmark for evaluating the social interactivity of omni-modal large language models (OLM) in multi-party conversational settings. Unlike traditional existing video-understanding benchmarks that treat the model as a passive observer, SocialOmni requires jointly recognizing who is speaking, judging when to take the floor, and deciding how to respond—three tightly coupled fundamental abilities that underpin natural conversation yet have systematically not been assessed in a unified framework. In what follows, we first introduce how the benchmark is constructed and curated (§3.1–3.2), then formalize the unified who–when–how task design (§3.3) and the accompanying evaluation protocol (§3.4). 3.1 Benchmark Construction Evaluating the social interactivity of OLM rigorously requires dialogue videos that span a wide spectrum of conversational types while maintaining high audio-visual quality and appropriate redistribution licenses. We first compile a search-term database targeting diverse multi-party dialogue scenarios, systematically rank terms by the volume of retrievable videos on public platforms with C-BY-compatible licenses, and retain only those yielding sufficient results of high production quality. This procedure produces 15 dialogue subcategories organized into four domains: Entertainment, Sports, Art, and Fashion under the Entertainment domain; Business, Technology, Education, and General under the Professional domain; Daily, Food, Travel, and Health 4 under the Daily Life domain; and Emotion, Real, and Others under the Narrative domain. In total, we crawl over 3,000 raw videos across these subcategories. Eight trained annotators independently review every video and extract segments of 10–30,s containing clear multi-party dialogue. Each clip is assigned to the perception or generation task according to the criteria detailed in §3.3. After stringent filtering for audio clarity, face visibility, and turn-structure quality, 2,209 clips survive with a mean duration of 25.0,s. We then apply Whisper [41] and FunASR [11] to every surviving clip to obtain automatic transcripts, which serve dual purposes: they provide essential raw material for constructing perception answer options and act as reference text for evaluating generation quality. Prompt templates and parsing rules appear in Appendix A.14. 3.2 Statistics and Quality Control Table 1 systematically compares the scale and scope of SocialOmni with prior benchmarks. The benchmark comprises 2,209 evaluation instances divided into two complementary splits: the perception split carefully contains 2,000 multiple-choice questions (1,725 consistent and 275 inconsistent), while the generation split provides 209 open-ended items, each accompanied by multi-reference responses. As shown in Fig. 1(a), the 15 subcategories are deliberately balanced so that no single conversational style dominates: the General category contributes the most clips (394) and Fashion the fewest (70). The generation subset is intentionally kept compact to maintain manageable variance in open-ended judging, yet it preserves full domain coverage. Substantial inter-annotator agreement reaches 94.2% on the perception split and 91.8% on the generation split, confirming high annotation reliability. Full agreement statistics, a size-rationale analysis for the generation split, and complete subcategory definitions appear in Appendices A.1, A.5, and A.2, respectively. 3.3 Task Design SocialOmni frames real-time multi-party interaction as a unified who–when–how problem. Recognizing who is speaking at a given moment is fundamentally a perceptual ability, whereas deciding when to take the floor and how to respond demands genuine generative interaction. We therefore operationalize the benchmark through two complementary tasks that together cover the full arc of a conversational turn. Task I: Who — Perception. This task accurately evaluates the ability to identify the active speaker at timestamptwithin videoVand audioA. Candidate choices are systematically synthesized by permuting two orthogonal axes—speaker identity and textual content—derived automatically from the ASR transcripts. The resulting comprehensive four-way classification includes the ground truth (correct speaker, correct content) alongside three distractors: wrong speaker with correct content, correct speaker with wrong content, and wrong speaker with wrong content. This design effectively decouples errors in visual grounding from errors in speech recognition. Each clip is carefully additionally labeled as consistent (the on-screen person matches the audio source) or inconsistent (the camera shows a different person), enabling fine-grained diagnosis of robust robustness to cross-modal mismatch. Representative examples of both types appear in Fig. 2. Task I: When & How — Generation. Given a video prefixV ≤t with the corresponding audio prefixA ≤t , the model first addresses when to speak, which is a binary turn-taking decision at timestampt. If the answer is affirmative, how to respond by generating a context-appropriate utterance. Clips for this task are selected under stricter criteria: speaker turns must alternate with sufficient clarity for a human observer to pinpoint transition boundaries unambiguously. The annotated boundaries serve as ground truth for the when sub-question, and each clip is paired with multi-reference continuations to support robust evaluation of the how sub-question. All annotations undergo two rounds of adjudication—independent labeling followed by cross-review, to ensure consistency. Further details appear in Appendix A.9. 3.4 Evaluation Metrics We design evaluation metrics for each of the three axes. For who, we use top-1 accuracy and macro-F1; for when, we measure the signed response offset and assign each prediction to one of five timing categories; for how, we adopt an LLM-as-a-judge score. Perception is evaluated independently, while timing and response quality are evaluated jointly: the model first decides when to speak, and only then is its response scored. Perception metric (who). The perception split containsN p = 2,000 clips. Each clip is paired with a query timestamp and four candidate descriptions of who is saying what at that moment; the model selects the correct one, and we report top-1 accuracy with non-parsable outputs counted as incorrect. Because the benchmark 5 Timestamp 3s Audio Context (Sarah voice: Mike, you never listen!) Z o n e 1 Video Context (Mike Right, closed lips) Z o n e 2 Visual: Mike (Right, Closed lips) Audio: Sarah (Left, Voice active) Audio-Vision Inconsistent Visual:Sarah (Left, Speaking) Audio: (Left, Voice active) Ground Truth Text: Mike, you never listen! Next-Turn Generation Turn-entry Decision Yes No Ground Truth: “I‘m busy right now.” OLM Response:Wait, I was just ... OLM Response:None GPT Gemini Qwen LLM Judges Generative Task Z o n e 3 A B C D A: Person Right (Closed lips) B: Person Left (Active voice) C: Person Middle D: Person Right (Mis-attribution) Audio-Vision consistent Task I Task I At timestamp [3s],who is speaking? Figure 2. Illustration of the SocialOmni evaluation pipeline. Given a multi-modal conversation stream (Zone 1), SocialOmni constructs both audio-vision inconsistent and consistent consistent (Zone 2), then evaluates models on speaker perception (Task I) and turn-entry generation (Task I) with LLM-based judging (Zone 3). deliberately includes both consistent clips (N cons = 1,725) and inconsistent clips (N incons = 275), we also report accuracy on each subset separately. Their difference defines the consistency gap ∆ cons ≜ Acc cons − Acc incons , which quantifies the model’s reliance on visual–audio alignment: a large positive gap reveals that the model struggles when the visible face does not match the speaker’s voice. To check for systematic positional bias (e.g., always selecting option A), we additionally report macro-averaged F1 across the four answer positions on parsable outputs. The complete procedure is summarized in Algorithm 1. Turn-taking timing metric (when). The generation split containsN g = 209 clips, each annotated with a ground-truth turn-entry timestampτ ⋆ i and a candidate speakerX i . To simulate real-time reception, we incrementally extend the visible prefix by one second at each step and query the model with “ShouldX i speak now?” We evaluate strides of 0.5 s, 1 s, and 2 s; the 1 s stride provides a favorable trade-off between evaluation cost and temporal precision (Appendix A.6). Letˆτ i denote the first timestamp at which the model answersYES. The signed response offset ∆τ i ≜ ˆτ i − τ ⋆ i captures the deviation from the ideal entry point, where negative values indicate premature interruption and positive values indicate delayed response. Based on thresholds of (1,2,5) s, we assign each clip to one of five timing categories: Interrupted (∆τ i <−1 s) means the model disrupts the ongoing turn; Perfect (−1≤∆τ i ≤2 s) indicates an acceptable entry window; Delayed (2<∆τ i ≤5 s) marks a noticeably late but still relevant response; TooLate (∆τ i >5 s) signals that the conversational window has passed; and NoResponse means the model never answersYES. We collapse these into three summary groups: Early (E) = Interrupted, On-time (O) = Perfect, and Late (L) = Delayed∪TooLate∪NoResponse. The primary when-score is the On-time rate O, the fraction of clips in the Perfect window. Threshold justification is in Appendix A.7. Response quality metric (how). For every clip in which the model decides to speak, it produces a responseˆs i , which we rigorously assess via an LLM-as-a-judge protocol [30,58] with three independent judges: GPT-4o [18], Gemini 2.5 Pro [12], and Qwen3-Omni [50]. Each judge receives the full ASR transcript, the annotated reference continuation, and the model’s response, then assigns a score on a four-level scale 25,50,75,100; this coarse granularity reduces judge hesitation and improves inter-judge agreement [30,58]. The per-clip score is the three-judge mean ̄s i = 1 3 (s (1) i +s (2) i +s (3) i ), and the dataset-level how-score averages ̄s i over all clips with non-empty responses. Two important auxiliary metrics accompany the how-score: response coverageCov=|G|/N g , recording the fraction of clips for which the model produces a valid utterance, and the large-gap rateR gap , measuring the fraction of clips on which at least two judges disagree by≥25 points. The coupled evaluation pipeline for the when and how tasks is given in Algorithm 2. 6 Algorithm 1 Perception Evaluation (who) Require: Clips (V i ,A i ,t i ,O i ,y ⋆ i ) N p i=1 ; model f Ensure: Acc all , Acc cons , Acc incons , ∆ cons , macro-F1 1: for each clip i do 2: Feed video V i , audio A i , timestamp t i , and options O i to f; obtain prediction ˆy i 3: if ˆy i is non-parsable then 4:Mark as incorrect 5: else 6:Compare ˆy i with ground truth y ⋆ i ; update per-subset counters 7: end if 8: end for 9: Compute overall, consistent, and inconsistent accuracy 10: ∆ cons ← Acc cons − Acc incons 11: Compute macro-F1 over the four answer positions 4 Experiments We systematically organize experiments around two questions: (1) Comprehensively where do current omni- modal models stand on the three axes of social interaction? and (2) What capabilities are still missing, and where do models fail collectively? We first introduce the setup (§4.1), then present the main results with a detailed unified leaderboard and capability profiles (§4.2), and thoroughly conduct diagnostic analysis across three layers: perception reliability, timing–response behavior, and failure cases (§4.3). 4.1 Experiment Setup Models. We evaluate twelve omni-modal large language models spanning commercial APIs and open-source systems across diverse benchmarks. Commercial: GPT-4o [18], Gemini 2.5 Pro/Flash [7], Gemini 3 Flash/Pro Preview [12]. Open-source: Qwen3-Omni, Qwen3-Omni-Thinking, Qwen2.5-Omni [50], OmniVinci [51], Baichuan-Omni-1.5 [25], VITA-1.5 [10], and MiniOmni2 [48]. MiniOmni2 lacks a stable generation interface and is evaluated on perception only due to technical limitations. We use the default system prompts and generation settings for all models to ensure fair comparison across different platforms. Inputs and prompting. All models receive raw video (decoded at 30 fps) and audio (native sampling rate) under a unified interface. Ground-truth transcripts are never exposed to evaluated models; they are used solely by judges for response-quality scoring. Prompt templates are fixed across all models (Appendix A.14). No single model dominates all three axes. The leader differs by axis: Qwen3-Omni on who (69.25%), Gemini 3 Pro Preview on when (67.31%), and Gemini 2.5 Flash on how (85.08). Every radar polygon in Figure 3 is visibly lopsided, confirming that a single aggregate score would mask critical axis-specific gaps. Open-source models lag substantially behind commercial systems. The gap is particularly pro- nounced on response quality: the best open-source how score (Qwen2.5-Omni, 66.15) trails the best commercial score (Gemini 2.5 Flash, 85.08) by nearly 19 points. Models such as VITA-1.5 (12.49) and Baichuan-Omni-1.5 (27.27) produce fluent but contextually irrelevant responses. On when, the gap is narrower but consistently favors commercial APIs. On who, the overall picture is mixed: Qwen3-Omni leads all models, while most other open-source systems remain below the commercial median. Perception and generation abilities do not correlate. Rank inversion is striking and clearly visible: Qwen3-Omni-Thinking achieves relatively competitive who yet falls among the lowest on how (18.06), while GPT-4o surprisingly shows low who (36.75%) but strong how (69.64). This decoupling confirms that conversational interactivity must be properly evaluated as a multi-dimensional profile. 7 Algorithm 2 Generation Evaluation (when–how) Require:Clips(V i ,A i ,τ ⋆ i ,X i ) N g i=1 ; modelf; strideδ= 1 s; judgesJ k 3 k=1 ; thresholds (α,β,γ) = (1,2,5) s Ensure: Timing distribution,∆τ, Score how , Cov, R gap 1: for each clip i do 2: for t = δ, 2δ, ..., T i do 3:Show the first t seconds to f and ask “Should X i speak now?” 4:if f answers YES then 5:Record entry time ˆτ i ← t; break 6:end if 7: end for 8: if f never answers YES then 9:Label as NoResponse; continue 10: end if 11: Compute offset ∆τ i ← ˆτ i − τ ⋆ i 12: Assign timing category: Interrupted / Perfect / Delayed / TooLate 13: Ask f to generate a response ˆs i given the first ˆτ i seconds 14: for each judge J k do 15: s (k) i ← J k (transcript, reference, ˆs i ) ∈ 25, 50, 75, 100 16: end for 17: Per-clip score: ̄s i ← 1 3 P k s (k) i 18: end for 19: Aggregate timing distribution and mean offset∆τ 20: Score how ← average ̄s i over clips with responses 21: Cov← fraction of clips with responses 22: R gap ← fraction of clips where judges disagree by ≥ 25 points 4.2 Main Results Who Who-Cons. Who-Incons. Robustness (100-||) When-Acc. When-F1 On-time How 0.25 0.50 0.75 1.00 Model Legend GPT-4o Gemini 2.5 Pro Gemini 2.5 Flash Gemini 3 Flash Gemini 3 Pro Qwen3-Omni Qwen3-Omni-Think Qwen2.5-Omni OmniVinci VITA-1.5 Baichuan-Omni-1.5 Figure 3. Cross-axis capability profiles. Each polygon shows one model over normalized who–when–how dimensions. No single model dominates all axes, revealing distinct strengths and weaknesses. 8 Table 2. SocialOmni main performance across the who–when–how axes. Who is top-1 accuracy on the perception split (2,000 items). When is timing accuracy on the generation split (209 items). How is judge score (/100). ‘–‘ indicates not supported by interface constraints. * MiniOmni2 is evaluated on perception only due to unavailable stable generation interface and significant technical implementation constraints. ModelWho (%)When (%)How (/100) GPT-4o [18]36.7546.8969.64 Gemini 2.5 Pro [7]44.6955.6772.32 Gemini 2.5 Flash [7]47.0361.5085.08 Gemini 3 Flash Preview [12]53.2361.0679.08 Gemini 3 Pro Preview [12]64.9967.3181.77 Qwen3-Omni [50]69.2563.6445.57 Qwen3-Omni-Thinking [50]54.6046.4118.06 Qwen2.5-Omni [50]36.7557.4266.15 OmniVinci [51]35.8641.6355.86 VITA-1.5 [10]36.9543.3712.49 Baichuan-Omni-1.5 [25]25.6546.8827.27 MiniOmni2* [48]16.72– Table 3. Perception-task (who) speaker identification metrics (bootstrap 95% CI, 10,000 resamples, seed=42). ModelAcc.Acc. [95% CI]F1-mF1-m [95% CI] GPT-4o [18]36.75[34.66, 38.89]35.80[33.63, 37.97] Gemini 2.5 Pro [7]44.69[42.52, 46.88]44.53[42.39, 46.67] Gemini 2.5 Flash [7]47.03[44.82, 49.24]46.75[44.56, 48.93] Gemini 3 Flash Preview [12]53.23[51.04, 55.41]53.36[51.18, 55.53] Gemini 3 Pro Preview [12]64.99[62.86, 67.06]65.02[62.93, 67.08] Qwen3-Omni [50]69.25[67.19, 71.23]68.81[66.72, 70.81] Qwen3-Omni-Thinking [50]54.60[52.43, 56.76]53.99[51.65, 56.25] Qwen2.5-Omni [50]36.75[34.66, 38.89]33.38[31.24, 35.49] OmniVinci [51]35.86[32.64, 39.14]31.09[27.75, 34.33] VITA-1.5 [10]36.97[34.86, 39.03]34.43[32.29, 36.50] Baichuan-Omni-1.5 [25]25.65[23.78, 27.61]16.67[15.49, 17.86] 4.3 Diagnostic Analysis The main results reveal what the landscape looks like; we now ask why. We structure the diagnosis into three layers: perception reliability, timing-and-response coupling, and universal failure modes. 4.3.1 Who: Perception Reliability Table 3 supplements who accuracy with macro-F1 and 95% bootstrap confidence intervals. The overall ranking is however broadly preserved, but several models show a notable accuracy-to-F1 drop, indicating uneven performance across answer positions, a hallmark of positional selection bias (e.g., consistently favoring option A). This strongly validates the use of macro-F1 as a complementary reliability metric: models that appear competitive on accuracy alone may be unreliable when class balance is enforced. 4.3.2 When + How: Timing Behavior and Response Quality Interruption vs. delay. Figure 4 decomposes every model’s timing predictions into E/O/L phases, revealing two distinctly opposing failure modes. Aggressive models (notably, e.g., Qwen2.5-Omni, E = 22.5%; VITA-1.5, E = 21.9%) frequently interrupt the ongoing speaker before the turn boundary, demonstrating poor turn-taking awareness. Conservative models (e.g., OmniVinci, L = 54.5%; GPT-4o, L = 45.5%) rarely interrupt but miss the conversational window entirely, sacrificing responsiveness for caution. The best when performers (e.g., Gemini 3 Pro, E = 5.3%, L = 27.4%) achieve low E and low L simultaneously, thereby reflecting a well-calibrated entry strategy that balances timing precision with conversational naturalness. 9 Table 4. Turn-taking timing (when) reliability on the generation task (δ = 0.2s, bootstrap 95% CI). ModelAcc.Acc. [95% CI]PRF1 GPT-4o [18]46.89[40.19, 53.59]70.3728.5740.64 Gemini 2.5 Pro [7]55.67[48.77, 62.56]75.9545.8057.14 Gemini 2.5 Flash [7]61.50[54.50, 68.00]73.4563.8568.31 Gemini 3 Flash Preview [12]61.06[54.33, 67.79]73.2161.6566.94 Gemini 3 Pro Preview [12]67.31[61.06, 73.56]87.3657.1469.09 Qwen3-Omni [50]63.64[56.94, 69.86]76.6461.6568.33 Qwen3-Omni-Thinking [50]46.41[39.71, 53.11]60.6145.1151.72 Qwen2.5-Omni [50]57.42[50.72, 64.11]65.9468.4267.16 OmniVinci [51]41.63[34.93, 48.33]70.3714.2923.75 VITA-1.5 [10]43.37[36.73, 50.51]55.2143.8048.85 Baichuan-Omni-1.5 [25]46.89[40.19, 53.59]59.3252.6355.78 Precision–recall trade-off. Figure 5 strikingly reveals that models with ostensibly similar On-time rates can occupy very different positions in precision–recall space: a high-precision / low-recall model is inherently overly cautious (correct when it speaks but misses valid entry points), while conversely the reverse signals a trigger-happy strategy. This shows that timing behavior is fundamentally a two-dimensional trade-off that the single On-time percentage in Table 2 does not adequately capture. Timing–quality coupling. Comparing systematically Figure 4 with the how column of Table 2, premature entry (high E) does not necessarily always degrade response quality, some models remarkably generate reasonable continuations even when entering slightly early. Conversely, very late entry (high L) robustly consistently correlates with lower how scores, because the model misses the relevant conversational context. Good timing is ultimately a necessary but not sufficient condition for good response quality. Models 0 20 40 60 80 100 Ratio (%) GPT-4o Gemini 2.5 Pro Gemini 2.5 Flash Gemini 3 Flash Gemini 3 Pro Qwen3-Omni Qwen3-Omni-Thinking Qwen2.5-Omni OmniVinci VITA-1.5 Baichuan-Omni-1.5 Early (E)On-time (O)Late (L) Figure 4. Timing-phase decomposition for turn entry. Early/On-time/Late rates expose whether a model tends to interrupt prematurely or miss the optimal conversational window during dialogue. 4.3.3 Failure Cases Beyond critically aggregate metrics, we systematically inspect cases where the majority of evaluated models consistently fail, thereby identifying systemic bottlenecks rather than individual model weaknesses. Perception failures. Two dominant patterns emerge. (i). Cross-modal temporal incoherence: when the camera cuts to a reaction shot while the speaker continues off-screen, most models attribute the utterance to 10 102030405060708090 Recall (%) 50 55 60 65 70 75 80 85 90 Precision (%) Iso-F1 guideline (--) GPT-4o Gemini 2.5 Pro Gemini 2.5 Flash Gemini 3 Flash Gemini 3 Pro Qwen3-Omni Qwen3-Omni-Thinking Qwen2.5-Omni OmniVinci VITA-1.5 Baichuan-Omni-1.5 Figure 5. Precision–recall operating points for when decisions. Iso-F1 guides highlight the fundamental trade-off between cautious and trigger-happy turn-entry strategies in dialogue systems. the visually salient face rather than maintaining speaker–identity binding across frames. This reflects a failure to reconcile “who was speaking before the cut” with “who is visible now,” a deficit in temporal cross-modal coherence rather than in either modality alone. (i). Correct transcription, wrong speaker: models often select the option matching the correct ASR content yet assign it to the wrong on-screen person. The perception pipeline effectively collapses to text matching, bypassing genuine voice–face grounding such as timbre or lip-sync alignment. These two modes together explain the majority of perception errors and are amplified in the inconsistent subset where visual cues are deliberately unreliable. Generation failures. Two distinctly parallel patterns appear on the generation side. (i) Premature interruption: models inadvertently frequently trigger a turn entry at prosodic pauses or hesitations that merely resemble turn-final cues, indicating reliance on shallow silence-gap detection rather than integrating discourse-level signals such as syntactic incompleteness, sustained eye contact, or rising intonation. (i) Contextually incoherent continuation: even when a model times the interruption correctly, the generated content is often generic or tonally mismatched, ignoring the emotional tenor, topic trajectory, and interpersonal dynamics established in the prior context. This directly instantiates the perception–generation decoupling central to SocialOmni: correct perception does not guarantee socially appropriate generation. Overall, these four failure modes confirm that social interactivity must be evaluated as a joint who–when–how profile; strong performance on any single axis does not preclude systemic failure on the others. 5 Conclusion In this paper, we present SocialOmni, a comprehensive benchmark for joint who–when–how evaluation in omni-modal large language models, where Task I targets speaker identification (who) and Task I targets turn timing (when) and response generation (how). Experiments on 12 OLMs systematically show rank decoupling between perceptual accuracy and generation quality, alongside heterogeneous robustness under speaker-camera mismatch. These findings suggest that understanding accuracy alone cannot characterize conversational social competence, underscoring the urgent need for interaction-oriented evaluation. Limitations and Future Work. The generation subset serves as a controlled diagnostic and does not exhaustively cover all dialogue transitions. The Task I evaluation, especially its response-quality component, relies on transcribed model outputs and may underweight visual grounding and prosodic cues. In the future, we will scale SocialOmni to multi-turn interaction trajectories, incorporate human evaluation for pragmatically subtle cases, and extend modality coverage to prosody- and gesture-aware assessments. 11 References [1]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: A visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. URL https://arxiv.org/abs/2204.14198. [2] Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, Ruoming Pang, and Shinji Watanabe. Talking turns: Bench- marking audio foundation models on turn-taking dynamics. arXiv preprint arXiv:2503.01174, 2025. URL https://arxiv.org/abs/2503.01174. [3]Jianghan Chao, Jianzhang Gao, Wenhui Tan, Yuchong Sun, Ruihua Song, and Liyun Ru. JointAVBench: A benchmark for joint audio-visual reasoning evaluation. arXiv preprint arXiv:2512.12772, 2025. URLhttps: //arxiv.org/abs/2512.12772. [4]Wang Chen, Yongdong Luo, Yuhui Zeng, Luojun Lin, Tianyu Xie, Fei Chao, Rongrong Ji, and Xiawu Zheng. Event-anchored frame selection for effective long-video understanding. arXiv preprint arXiv:2603.00983, 2026. [5]Wang Chen, Yuhui Zeng, Yongdong Luo, Tianyu Xie, Luojun Lin, Jiayi Ji, Yan Zhang, and Xiawu Zheng. Wavelet-based frame selection by detecting semantic boundary for long video understanding. arXiv preprint arXiv:2603.00512, 2026. [6]Sanjoy Chowdhury, Karren Dai Yang, Xudong Liu, Fartash Faghri, Pavan Kumar Anasosalu Vasu, Oncel Tuzel, Dinesh Manocha, Chun-Liang Li, and Raviteja Vemulapalli. AMUSE: Audio-visual benchmark and alignment framework for agentic multi-speaker understanding. arXiv preprint arXiv:2512.16250, 2025. URL https://arxiv.org/abs/2512.16250. [7]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06261, 2025. URL https://arxiv.org/abs/2507.06261. [8]Bradley Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1–26, 1979. doi: 10.1214/aos/1176344552. URLhttps://projecteuclid.org/journals/annals-of-statistics/volume-7/iss ue-1/Bootstrap-Methods--Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full. [9]Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24108–24118, 2025. URLhttps://openaccess.thecvf.com/content/CVPR2025/html/Fu_Video-MME_The_First-Ever_Comp rehensive_Evaluation_Benchmark_of_Multi-modal_LLMs_in_CVPR_2025_paper.html. [10] Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957, 2025. URL https://arxiv.org/abs/2501.01957. [11] Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, and Shiliang Zhang. Funasr: A fundamental end-to-end speech recognition toolkit. arXiv preprint arXiv:2305.11013, 2023. URL https://arxiv.org/abs/2305.11013. [12] Google. Gemini 3: Introducing the latest gemini ai model from google.https://blog.google/products-and-p latforms/products/gemini/gemini-3/, 2025. Google Blog, Nov 18, 2025. Accessed: 2026-03-01. [13]Google AI for Developers. Release notes | gemini api | google ai for developers.https://ai.google.de v/gemini-api/docs/changelog, 2026. Documents launch/update records forgemini-3-pro-previewand gemini-3-flash-preview. Accessed: 2026-03-01. [14] Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. WorldSense: Evaluating real- world omnimodal understanding for multimodal llms. arXiv preprint arXiv:2502.04326, 2025. URLhttps: //arxiv.org/abs/2502.04326. [15]Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Fei Chao, and Rongrong Ji. Determining layer-wise sparsity for large language models through a theoretical perspective. arXiv preprint arXiv:2502.14770, 2025. [16] Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Fei Chao, Rongrong Ji, and Liujuan Cao. Discovering important 12 experts for mixture-of-experts models pruning through a theoretical perspective. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [17] Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Yang Liu, Jing Lin, Yiwu Yao, and Rongrong Ji. Dynamic low-rank sparse adaptation for large language models. arXiv preprint arXiv:2502.14816, 2025. [18]Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o System Card. arXiv preprint arXiv:2410.21276, 2024. URL https://arxiv.org/abs/2410.21276. [19] Shixin Jiang, Jiafeng Liang, Jiyuan Wang, Xuan Dong, Heng Chang, Weijiang Yu, Jinhua Du, Ming Liu, and Bing Qin. From specific-mllms to omni-mllms: a survey on mllms aligned with multi-modalities. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8617–8652, 2025. doi: 10.18653/v1/2025.finding s-acl.453. URL https://aclanthology.org/2025.findings-acl.453/. [20]Fanqi Kong, Weiqin Zu, Xinyu Chen, Yaodong Yang, Song-Chun Zhu, and Xue Feng. SIV-Bench: A video benchmark for social interaction understanding and reasoning. arXiv preprint arXiv:2506.05425, 2025. URL https://arxiv.org/abs/2506.05425. [21] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench- 2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092, 2023. URLhttps: //arxiv.org/abs/2311.17092. [22]Caorui Li, Yu Chen, Yiyan Ji, Jin Xu, Zhenyu Cui, Shihao Li, Yuanxing Zhang, Jiafu Tang, Zhen Song, Dingling Zhang, Yinghui He, Haoxian Liu, Yuxuan Wang, Qiufeng Wang, Zhenhe Wu, Jiehui Luo, Zhiyu Pan, Weihao Xie, Chenchen Zhang, Zhaohui Wang, Jiayi Tian, Yanghai Wang, Zhe Cao, Minxin Dai, Kefeng Wang, Runzhe Wen, Ying Ma, Yaning Pan, Sungkyun Chang, Termeh Taheri, Haiwen Xia, Christos Plachouras, Emmanouil Benetos, Yizhi Li, Ge Zhang, Jian Yang, Tianhao Peng, Zili Wang, Minghao Liu, Junran Peng, Zhao-Hui Zhang, and Jiaheng Liu. OmniVideoBench: Towards audio-visual understanding evaluation for omni mllms. arXiv preprint arXiv:2510.10689, 2025. URL https://arxiv.org/abs/2510.10689. [23] Danyang Li, Tianhao Wu, Bin Lin, Zhenyuan Chen, Yang Zhang, Yuxuan Li, Ming-Ming Cheng, and Xiang Li. WOW-seg: A word-free open world segmentation model. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=AyJPSnE1bq. [24] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024. URL https://openaccess.thecvf.com/content/CVPR2024/html/Li_MVBench_A_Comprehensive_Multi-modal_Vid eo_Understanding_Benchmark_CVPR_2024_paper.html. [25] Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. Baichuan-Omni-1.5 Technical Report. arXiv preprint arXiv:2501.15368, 2025. URL https://arxiv.org/abs/2501.15368. [26]Yizhi Li, Ge Zhang, Yi Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhen Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, and Chenghua Lin. OmniBench: Towards the future of universal omni-language models. arXiv preprint arXiv:2409.15272, 2024. URL https://arxiv.org/abs/2409.15272. [27]Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H. Liu, and Hung-yi Lee. Full-duplex-bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities. arXiv preprint arXiv:2503.04721, 2025. URL https://arxiv.org/abs/2503.04721. [28] Zhaojiang Lin, Yong Xu, Kai Sun, Jing Zheng, Yin Huang, Surya Teja Appini, Krish Narang, Renjie Tao, Ishan Kapil Jain, Siddhant Arora, et al. Wearvox: An egocentric multichannel voice assistant benchmark for wearables. arXiv preprint arXiv:2601.02391, 2025. URL https://arxiv.org/abs/2601.02391. [29]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. URL https://arxiv.org/abs/2310.03744. [30] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023. URLhttps://arxiv.org/ab s/2303.16634. 13 [31]Yuanzhan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? European Conference on Computer Vision, 2023. URL https://arxiv.org/abs/2307.06281. [32]Yongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, et al. Video-rag: Visually-aligned retrieval-augmented long video comprehension. arXiv preprint arXiv:2411.13093, 2024. [33]Yongdong Luo, Wang Chen, Xiawu Zheng, Weizhong Huang, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Jiebo Luo, et al. Quota: Query-oriented token assignment via cot query decouple for long video comprehension. arXiv preprint arXiv:2503.08689, 2025. [34]Yuexiao Ma, Taisong Jin, Xiawu Zheng, Yan Wang, Huixia Li, Yongjian Wu, Guannan Jiang, Wei Zhang, and Rongrong Ji. Ompq: Orthogonal mixed precision quantization. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 9029–9037, 2023. [35]Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. arXiv preprint arXiv:2403.12544, 2024. [36]Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Outlier-aware slicing for post-training quantization in vision transformer. In Forty-first International Conference on Machine Learning, 2024. [37] Yuexiao Ma, Xuzhe Zheng, Jing Xu, Xiwei Xu, Feng Ling, Xiawu Zheng, Huafeng Kuang, Huixia Li, Xing Wang, Xuefeng Xiao, et al. Flow caching for autoregressive video generation. arXiv preprint arXiv:2602.10825, 2026. [38]Leena Mathur, Marian Qian, Paul Pu Liang, and Louis-philippe Morency. Social genome: Grounded social reasoning abilities of multimodal models. Conference on Empirical Methods in Natural Language Processing, 2025. URL https://arxiv.org/abs/2502.15109. [39] Leyi Pan, Zheyu Fu, Yunpeng Zhai, Shuchang Tao, Sheng Guan, Shiyu Huang, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Felix Henry, Lijie Wen, and Aiwei Liu. Omni-SafetyBench: A benchmark for safety evaluation of audio-visual large language models. arXiv preprint arXiv:2508.07173, 2025. URL https://arxiv.org/abs/2508.07173. [40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763, 2021. URLhttps://proceedi ngs.mlr.press/v139/radford21a. [41] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. Proceedings of the 40th International Conference on Machine Learning (ICML), 202:28492–28518, 2023. URL https://proceedings.mlr.press/v202/radford23a.html. [42]Harvey Sacks, Emanuel A. Schegloff, and Gail Jefferson. A simplest systematics for the organization of turn-taking for conversation. Language, 50(4):696–735, 1974. doi: 10.2307/412243. [43]Ramaneswaran Selvakumar, Ashish Seth, Nishit Anand, Utkarsh Tyagi, Sonal Kumar, Sreyan Ghosh, and Dinesh Manocha. Multivox: A benchmark for evaluating voice assistants for multimodal interactions. arXiv preprint arXiv:2507.10859, 2025. URL https://arxiv.org/abs/2507.10859. [44] Gabriel Skantze. Turn-taking in conversational systems and human-robot interaction: A review. Computer Speech & Language, 67:101178, 2021. doi: 10.1016/j.csl.2020.101178. URLhttps://doi.org/10.1016/j.csl.2020.101178. [45]Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy Chen. Audiobench: A universal benchmark for audio large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4297–4316, 2025. doi: 10.18653/v1/2025.naacl-long.218. URL https://aclanthology.org/2025.naacl-long.218/. [46] Ke Wang, Houxing Ren, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Voiceassistant-eval: Benchmarking ai assistants across listening, speaking, and viewing. arXiv preprint arXiv:2509.22651, 2025. URLhttps: //arxiv.org/abs/2509.22651. 14 [47]Yuxuan Wang, Yueqian Wang, Borun Chen, Tong Wu, Dongyan Zhao, and Zilong Zheng. OmniMMI: A comprehensive multi-modal interaction benchmark in streaming video contexts. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. doi: 10.1109/CVPR52734.2025.01763. URL https://arxiv.org/abs/2503.22952. [48]Zhifei Xie and Changqiao Wu. Mini-Omni2: Towards open-source GPT-4o with vision, speech and duplex capabilities. arXiv preprint arXiv:2410.11190, 2024. URL https://arxiv.org/abs/2410.11190. [49]Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-Omni Technical Report. arXiv preprint arXiv:2503.20215, 2025. URL https://arxiv.org/abs/2503.20215. [50]Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jingren Zhou, and Junyang Lin. Qwen3-Omni Technical Report. arXiv preprint arXiv:2509.17765, 2025. URL https://arxiv.org/abs/2509.17765. [51] Hanrong Ye, Chao-Han Huck Yang, Arushi Goel, Wei Huang, Ligeng Zhu, Yuanhang Su, Sean Lin, An-Chieh Cheng, Zhen Wan, Jinchuan Tian, et al. OmniVinci: Enhancing architecture and data for omni-modal understanding llm. arXiv preprint arXiv:2510.15870, 2025. URL https://arxiv.org/abs/2510.15870. [52]Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. SALMONN-omni: A codec-free llm for full-duplex speech understanding and generation. arXiv preprint arXiv:2411.18138, 2024. URL https://arxiv.org/abs/2411.18138. [53]Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2023. URL https://arxiv.org/abs/2311.16502. [54] Danyang Zhang, Junhao Song, Ziqian Bi, Yingfang Yuan, Tianyang Wang, Joe Yeong, and Junfeng Hao. Mixture of experts in large language models. arXiv preprint arXiv:2507.11181, 2025. URLhttps://arxiv.org/abs/2507 .11181. [55] Xu Zhang, Danyang Li, Xiaohang Dong, Tianhao Wu, Hualong Yu, Jianye Wang, Qicheng Li, and Xiang Li. Unichange: Unifying change detection with multimodal large language model. arXiv preprint arXiv:2511.02607, 2025. [56] Yang Zhang, Danyang Li, Yuxuan Li, Xin Zhang, Tianyu Xie, Mingming Cheng, and Xiang Li. Crystal: Spontaneous emergence of visual latents in mllms. arXiv preprint arXiv:2602.20980, 2026. [57]Yiman Zhang, Ziheng Luo, Qiangyu Yan, Wei He, Borui Jiang, Xinghao Chen, and Kai Han. OmniEval: A benchmark for evaluating omni-modal models with visual, auditory, and textual inputs. arXiv preprint arXiv:2506.20960, 2025. URL https://arxiv.org/abs/2506.20960. [58] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. URLhttps://arxiv.org/abs/2306.05685. [59]Xiawu Zheng, Yuexiao Ma, Teng Xi, Gang Zhang, Errui Ding, Yuchao Li, Jie Chen, Yonghong Tian, and Rongrong Ji. An information theory-inspired strategy for automatic network pruning. arXiv preprint arXiv:2108.08532, 2021. [60]Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: Benchmarking multi-task long video understanding. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13691–13701, 2025. URLhttps://openaccess.thecvf.com/cont ent/CVPR2025/html/Zhou_MLVU_Benchmarking_Multi-task_Long_Video_Understanding_CVPR_2025_paper.h tml. [61] Ziwei Zhou, Rui Wang, and Zuxuan Wu. Daily-Omni: Towards audio-visual reasoning with temporal alignment across modalities. arXiv preprint arXiv:2505.17862, 2025. URL https://arxiv.org/abs/2505.17862. 15 Appendix Appendix Table of Contents Appendix A Additional Method Details for SocialOmni ................................. 17 A.1Inter-Annotator Agreement ........................................................................ 17 A.2Domain and Subcategory Definitions .............................................................. 17 A.3Option Balance in the Perception MCQ ........................................................... 17 A.4Consistency Labeling and Boundary Cases ........................................................ 17 A.5Generation Subset Size (209 Items) ................................................................ 17 A.6Q1 Step Size and Temporal Granularity ........................................................... 17 A.7Q1 Timing-Label Mapping ......................................................................... 17 A.8Delta-Window Binary Metrics for Q1.............................................................. 18 A.9Multiple References for Generation Calibration ................................................... 18 A.10 Generation Judging Scope and Visual Grounding ................................................. 18 A.11 Judge Configuration, Disagreement, and Tie Statistics............................................ 18 A.12 Modality Ablation Implementation ................................................................ 18 A.13 Reproducibility ..................................................................................... 19 A.14 Prompt Templates and Parsing Rules ............................................................. 19 A.15 Perception Results and Macro Metrics............................................................. 19 A.16 Statistical Definitions .............................................................................. 20 A.17 Human Feedback on a Challenging Subset ........................................................ 21 A.18 Human Feedback Discussion ....................................................................... 22 A.19 Representative Failure Cases ....................................................................... 22 16 A Additional Method Details for SocialOmni A.1 Inter-Annotator Agreement This subsection reports detailed inter-annotator agreement statistics. We use raw percent agreement as the primary IAA metric. Percent agreement is directly interpretable for our annotation types (4-way perception labels and binary consistency labels), and all disagreements are resolved by a senior reviewer via adjudication. We release adjudication flags and, where licensing permits, brief rationales and error categories alongside the final labels. Chance-corrected measures (e.g., Cohen’s κ) are left for future work. A.2 Domain and Subcategory Definitions SocialOmni organizes 15 dialogue subcategories into four domains used for benchmark stratification: en- tertainment, professional, daily life, and narrative. Entertainment covers interactive media formats (talk shows, podcasts). Professional covers task-oriented or formal settings (interviews, debates). Daily life covers naturally occurring everyday conversations (family interactions, street dialogue). Narrative covers scripted conversational scenes from movies and drama clips. Domain labels are used for split balancing and per-domain analysis; exact source metadata and clip-level assignments are released subject to licensing constraints. A.3 Option Balance in the Perception MCQ The correct-option distribution across the 2,000 perception items is A: 569, B: 561, C: 453, D: 417. This imbalance reflects natural speaker prominence and camera-framing biases in real dialogue footage rather than annotation artifacts. We deliberately avoid artificial rebalancing, which would distort real-world statistics and introduce selection bias. To ensure fairness, we report per-option accuracy and detailed confusion matrices, and stratify all analyses by domain, consistency split, and model type. A.4 Consistency Labeling and Boundary Cases Each consistency label is assigned by one annotator, independently verified by a second reviewer, and adjudicated on disagreement. Reviewers must cite visible evidence at timestampt(face visibility, clothing cues, on-screen positioning) to justify the label. Representative boundary cases include: (i) the active speaker is partially visible (small on-screen area); (i) the speaker is visible but identity cues are weak (heavy occlusion); and (i) reaction shots with off-screen speech. Illustrative examples appear in the supplementary figures; label rationales and complete annotation guidelines are released where licensing permits. A.5 Generation Subset Size (209 Items) This subsection explains the size choice for the 209-item generation split. The generation subset is kept relatively small to prioritize comparability over scale. Open-ended dialogue continuation annotation introduces variance along three axes: (1) decision-point selection, (2) reference quality, and (3) judge sensitivity. Scaling without tight control can amplify evaluation noise and hinder fair cross-model comparison. Our protocol fixes prompts, scoring rubrics, and judges, includes multi-reference calibration (§A.9) to make variance explicit. A.6 Q1 Step Size and Temporal Granularity This subsection explains the temporal granularity used for Q1. The perception task uses frame-level timestamps (30 fps) to evaluate who. For streaming Q1, we query at a 1 s step as a compute-stable approximation of real-time turn-entry when benchmarking many models. This trades temporal resolution for evaluation cost. Our timing categories use multi-second windows (e.g., “perfect” spans [−1, 2] s), which reduces sensitivity to small step-size changes in typical settings. We encourage future work to adopt finer steps (e.g., 0.5 s) when compute permits and to report step-size sensitivity explicitly. A.7 Q1 Timing-Label Mapping This subsection defines the timing labels used for Q1 evaluation. Let ∆τ i =ˆτ i − τ ⋆ i denote the response offset for itemi. The timing labelc i is assigned based on the response offset: responses are labeled as Interrupted if ∆τ i <−θ 1 , Perfect if−θ 1 ≤∆τ i ≤ θ 2 , Delayed ifθ 2 <∆τ i ≤ θ 3 , TooLate if ∆τ i > θ 3 , and NoResponse if ˆτ i =∅. The default thresholds are (θ 1 ,θ 2 ,θ 3 ) = (1, 2, 5) s. 17 A.8 Delta-Window Binary Metrics for Q1 This subsection defines the binary metrics used for tolerance-window evaluation in Q1. To align with prior binary turn-taking formulations, we define the tolerance-window target at decision time t as: y (δ) 1,t = 10 < τ X − t≤ δ,(1) whereτ X is the ground-truth turn-entry time of the candidate speaker. We evaluate withδ ∈0.2,0.5,1.0s and report: Prec = TP TP + FP ,Rec = TP TP + FN ,F1 = 2 Prec· Rec Prec + Rec .(2) Undefined ratios are set to 0 for stable aggregation. These metrics are reported alongside offset-based diagnostics. A.9 Multiple References for Generation Calibration This subsection describes the multi-reference calibration used for generation evaluation. Dialogue continuation is inherently multi-solution. For a fixed subset ofK mr = 30 tasks, we collect multiple semantically equivalent reference rewrites from annotators. These references calibrate judge tolerance to diverse valid continuations; we report score variance across references. A.10 Generation Judging Scope and Visual Grounding This subsection clarifies what is covered by generation judging and how visual grounding is examined. The generation task targets interruption continuation quality (appropriateness, coherence, pragmatics), which is primarily determined by dialogue context. Visual cues affect who attribution and when decisions; these are probed via (1) the inconsistency split, (2) modality ablations, and (3) a visually grounded subset. We provide a subset where the candidate response is required to reference a visible event or entity; judges are instructed to penalize hallucinated visual references. Performance on this subset is reported separately, and the corresponding prompts are released. A.11 Judge Configuration, Disagreement, and Tie Statistics This subsection summarizes the generation judges, score interpretation, and disagreement statistics. The three judges are GPT-4o, Gemini 3 Pro, and Qwen3-Omni. Each judge outputs a single score in25,50,75,100 under deterministic decoding (τ dec = 0 where supported) with fixed prompts to minimize prompt and sampling variance. The score 100 denotes a fluent response that is grounded in context and pragmatically appropriate. A score of 75 is used for responses that are generally appropriate but remain somewhat generic or incomplete. A score of 50 indicates partial relevance together with noticeable grounding or coherence problems. A score of 25 is assigned to responses that are irrelevant, contradictory, overly generic, or pragmatically inappropriate. A coarse discrete scale is used to improve stability and reduce judge variance. We report tie rates (fraction of samples with identical aggregated scores across models) and rank-discrimination statistics (e.g., Kendall’sτ between judge and aggregated rankings). A large-gap event is defined as|s (a) − s (b) | ≥20, corresponding to at least one near-step disagreement on the 4-level scale. The threshold of 20 (slightly below the 25-point step) is intended to capture near-step disagreements while reducing sensitivity to minor judge calibration drift. We report large-gap frequency and its association with ambiguous contexts. A.12 Modality Ablation Implementation This subsection describes how the modality ablations are implemented. We preserve the original audio waveform and replace the video stream with a static first frame replicated at the original frame rate, keeping the input container and frame count unchanged while removing visual dynamics. We preserve the original video frames and replace the audio waveform with zeros (silence), removing acoustic cues without altering video timing. 18 Table 5. SocialOmni perception task results (2,000 items). ∆ cons = Acc cons − Acc incons . ModelOverall (%)Cons. (%)Incons. (%)∆ cons (%) GPT-4o [18]36.7538.1428.00+10.1 Gemini 2.5 Pro [7]44.6945.8837.23+8.7 Gemini 2.5 Flash [7]47.0348.5237.59+10.9 Gemini 3 Flash Preview [12]53.2353.6650.55+3.1 Gemini 3 Pro Preview [12]64.9966.2457.04+9.2 Qwen3-Omni [50]69.2569.9764.73+5.2 Qwen3-Omni-Thinking [50]54.5553.7459.64−5.9 Qwen2.5-Omni [50]36.7536.4638.55−2.1 OmniVinci [51]15.1515.0116.00−1.0 VITA-1.5 [10]36.9536.8137.82−1.0 Baichuan-Omni-1.5 [25]25.6525.9723.64+2.3 MiniOmni2* [48]16.7217.574.55+13.0 This design avoids out-of-distribution artifacts (e.g., random noise) while keeping the model interface identical, isolating each modality’s contribution. A.13 Reproducibility This subsection lists the metadata and prompts released for reproduction. Subject to licensing constraints, we release the metadata required to reproduce each sample: video identifiers/URLs where permitted, timestamp t, candidate speakerXfor generation items, aligned transcript segments, consistency labels, and adjudication flags. Evaluation prompts and judge prompts for the generation task are also released. A.14 Prompt Templates and Parsing Rules This subsection summarizes the prompt templates and the corresponding parsing constraints. We use fixed prompt cards with strict output parsing constraints to reduce prompt-induced variance across heterogeneous APIs and open-source checkpoints. For who, only a single option letter inA, B, C, Dis accepted. For when, only unambiguousYES/NOoutputs are accepted after normalization. For how, non-empty continuations are retained for judging; empty responses are counted as no-response. SocialOmni Prompt Cards (who–when–how) Who (Perception). System: You are a precise video-audio reasoning assistant. You must answer ONLY with the option letter (A, B, C, or D). When (Q1 Decision). You are a conversation participant watching a video. Based on what you see, answer: Is it your turn to speak now? YES or NO. How (Generation). You are another participant in this conversation. Watch the video carefully. When the other person finishes speaking and it is your turn, respond naturally in English. Do not interrupt while they are still speaking. A.15 Perception Results and Macro Metrics This subsection reports the detailed perception results and the macro metrics used alongside accuracy. ∆ cons is a useful robustness indicator and should be interpreted jointly with overall accuracy. Gemini 2.5 19 Pro and Gemini 3 Pro differ in overall accuracy (44.69% vs. 64.99%) yet exhibit comparable consistency gaps (+8.7 and +9.2), showing that higher absolute accuracy does not eliminate split-specific brittleness. Qwen3-Omni-Thinking shows a negative gap (−5.9%), i.e., lower accuracy on the consistent split than on the inconsistent split. This pattern is consistent with the possibility that more deliberative reasoning can interfere with immediate cue integration in mismatch-heavy scenes, so the result may reflect more than random variation. For C = 4 perception classes, macro-averaged F1 is computed over the parsable subset P: F1 macro = 1 C C X c=1 2 Pr c Re c Pr c + Re c ,(3) wherePr c andRe c are per-class precision and recall. Non-parsable outputs are excluded fromPbut counted as incorrect for top-1 accuracy (defined in Sec. 3.4). A.16 Statistical Definitions This subsection collects the statistical definitions used in the appendix analyses. We first define the generation aggregation: ̄s i = 1 J J X j=1 s (j) i , J = 3.(4) G =i∈1,...,N gen : ˆy 1,i = 1∧ ˆs i ̸=∅.(5) We then define the cross-task association statistics: r = P m∈M (a m − ̄a)(q m − ̄q) p P m (a m − ̄a) 2 p P m (q m − ̄q) 2 ,(6) p = 1 + P B perm b=1 1 |r (b) |≥|r| 1 + B perm ,(7) CI 0.95 (r) = h Q 0.025 r ⋆(b) , Q 0.975 r ⋆(b) i .(8) Finally, we summarize the evaluation metrics used in the appendix: For who, we report top-1 accuracy on the full set and on consistent/inconsistent splits, with macro precision, recall, and F1 computed over the parsable subsetP; non-parsable outputs are treated as incorrect. For when (Q1), we adopt a primary tolerance windowδ=0.2 s around each annotated boundary and report accuracy, precision, recall, and F1. To examine timing biases, we decompose predictions into Early/On-time/Late (E/O/L) phases over the responded subset R =i : ˆτ i ̸=∅: p E = |i∈R : ˆτ i < τ ⋆ i − δ| N R , p O = |i∈R :|ˆτ i − τ ⋆ i |≤ δ| N R , p L = |i∈R : ˆτ i > τ ⋆ i + δ| N R , (9) with no-response ratep NR = 1− N R /N gen . We sweepδ ∈ 0.2,0.5,1.0s; full results appear in the supplementary material. For how, each judge assignss (k) i ∈25,50,75,100under deterministic decoding; the item-level aggregate isg i = 1 3 P 3 k=1 s (k) i , and the model-level score is ̄g= 1 N speak P i g i , whereN speak is the number of items for which the model decides to speak. Unless stated otherwise, all confidence intervals are 95% bootstrap CIs with B=10,000 replicates [8] using the percentile method [θ ∗ (0.025) ,θ ∗ (0.975) ]. 20 GroupWho (%) When Q1 (%) How Q2 (/100) Full benchmark (model mean)41.8153.3655.75 Full benchmark (best model)69.2566.9985.08 Model mean on selected subset1.2512.500.00 Human on selected subset72.5080.0055.15 Table 6. Human feedback on the selected subset, with full-benchmark model references for scale. 020406080100 Score Who When (Q1) How (Q2) 71.2 67.5 55.1 Full meanBest modelSame-subset model meanHuman Figure 6. Selected-subset human feedback across who/when/how. The pale interval spans the full-benchmark mean to the best reported model; the dark connector links the same-subset model mean and selected-subset human score. A.17 Human Feedback on a Challenging Subset This subsection reports an additional human-feedback analysis without changing the claims in the main paper. We follow the same who/when/how axes as the benchmark protocol. The verification subset contains 200 items for who, 200 items for when (Q1), and 50 judged items for how (Q2). Items were selected from cases on which current models often fail. For this reason, this subset is used to examine failure modes rather than as an IID substitute for the full benchmark. The asymmetric size on how is intentional. Evaluating open-ended dialogue response quality requires careful judgment of empathy, grounding, and social appropriateness, so we use a smaller high-precision set. Each how item requires longer review and stricter consistency checks than binary who/when judgments; annotation effort is thus allocated to depth and quality control. Table 6 reports two references: (i) full-benchmark model anchors and (i) same-subset model/human comparison. Full-benchmark rows serve only as scale anchors and are not used to estimate a human baseline on the full benchmark. Human performance on the selected subset reaches 72.50% on who, 80.00% on when, and 55.15/100 on how. The comparison is descriptive: even on cases that are hard for current models, human performance remains higher.Figure 6 shows the same comparison with consistent color semantics across axes. Table 7 and Figure 7 summarize correlation statistics. The table reports Pearsonrand Spearmanρwith p-values, 95% bootstrap confidence intervals, and sample sizen. The negative item-level association on when (human vs. ensemble) is interpreted as a signal of shallow heuristics: models may rely too much on local acoustic cues (e.g., brief pause-like gaps) and too little on discourse-level completion cues used by human raters. For the when model-level comparison, the aligned sample remains limited (n= 11), so those coefficients should be interpreted as exploratory rather than definitive. 21 SettingAxis n Pearson r p r 95% CI (r) Spearman ρ p ρ 95% CI (ρ) Model-level (full vs subset)Who 120.5249 0.0797 [0.0811, 0.8799]0.53880.0707 [0.0518, 0.9149] Model-level (full vs subset)When 11 -0.4332 0.4663 [-1.0000, 1.0000]-0.20000.7471 [-1.0000, 1.0000] Item-level (human vs ensemble) Who 200 -0.2117 0.1898 [-0.5447, 0.1534]-0.21170.1898 [-0.5447, 0.1534] Item-level (human vs ensemble) When 200 -0.4663 0.0382 [-0.7003, -0.1667]-0.44050.0519 [-0.6692, -0.1667] Table 7. Correlation statistics on the selected challenging subset. The negative item-level association on when reflects a divergence in which models are often misled by deceptive acoustic cues. 1.00.50.00.51.0 Estimate with 95% CI Model-level Who n=12 Model-level When n=5 Item-level Who n=40 Item-level When n=20 B Pearson r 1.00.50.00.51.0 Estimate with 95% CI Spearman Figure 7. Correlation estimates on the selected challenging subset. Each point shows the estimated correlation; horizontal segments show the corresponding 95% confidence interval. The dashed vertical line marks zero association. A.18 Human Feedback Discussion This subsection discusses how the human-feedback results should be read. One finding is the negative item-level association between human judgments and model ensembles on the when axis (Pearsonr=−0.4663, p= 0.0382; Spearmanρ=−0.4405,p= 0.0519). We interpret this pattern as evidence of shallow heuristic over-reliance: in challenging social scenes, current Omni-LLMs tend to trigger turn-taking decisions from the surface acoustic cues (e.g., brief silence-like gaps or local energy drops), while human raters rely on higher-order semantic completion and pragmatic intent. Because the selected subset intentionally includes pseudo-silence cases (pauses inside an unfinished turn), the model decision tendency becomes inversely aligned with human-grounded timing labels. This result is consistent with the view that even competitive systems still lacks stable audiovisual-semantic binding for distinguishing a pause-for-thought from true turn completion. The negative association is therefore consistent with a gap in social-temporal reasoning that standard IID-style evaluation may show less clearly. A.19 Representative Failure Cases This subsection presents representative qualitative examples for the three benchmark axes. We present three failure cases, one for each benchmark axis. Each case links the benchmark target, representative model outputs, and the failure pattern visible in the visual and textual evidence. For who, Figure 8 shows a Level-1 item asking who speaks between 0:05 and 0:07. The correct answer is the man on the right saying “That’s so true.” The figure combines a dominant left-side close-up, a later right-side context frame, and the continuous audio track. Under this combination of visual and audio cues, all checked models fail: GPT-4o predicts option C, Qwen3-Omni predicts option A, and multiple Gemini variants predict option D. The error pattern is that models follow the most salient visible face rather than the audiovisual speaker binding. This case illustrates a saliency-driven speaker-attribution error. For when, Figure 9 presents a Level-2 Q1 item asking whether the woman speaks at the 22nd second. The annotated answer is No. The figure shows adjacent frames from the same ongoing turn together with a pause-like waveform gap, while the ASR remains semantically unfinished: “orange juice......a grapefruit...” GPT-4o is correct, but Gemini 3 Flash, Gemini 3 Pro, Gemini 2.5 Flash, and Gemini 2.5 Pro all answer Yes. The failure pattern is premature triggering: a brief acoustic gap is treated as turn completion even though the utterance is still pragmatically open. This case reflects shallow silence-gap reasoning rather than more complete turn-completion modeling. 22 Figure 8. Who failure case. The visually dominant frame favors the wrong speaker; The correct answer requires cross-modal speaker binding. The case illustrates a saliency-driven attribution error. Figure 9. When failure case. The waveform contains a pause-like gap, but the turn remains unfinished. The case illustrates premature turn triggering from shallow silence-gap cues. Figure 10. How failure case. The dialogue establishes an interpersonal context, but the response remains generic rather than empathetic. The case illustrates context-response decoupling in generation. For how, Figure 10 shows a Level-2 Q2 generation item. The scene is interpersonal, and the transcript centers on collateral, a guarantor, and the speaker’s discomfort with asking family for help. The reference continuation is empathetic: “I understand your feelings about it, Richard.” In contrast, several models produce generic problem-solving replies: GPT-4o (“We need to find another solution.”), Gemini 3 Flash (“But what are we going to do?”), Gemini 3 Pro (“But is there any other way?”), and Gemini 2.5 Pro (“There has to be another way.”), all scored 0 by the LLM judge protocol. Only Gemini 2.5 Flash matches the reference and receives 100. This case shows context-response decoupling: recognizing the topic does not ensure a grounded or socially appropriate continuation. These examples are broadly aligned with the quantitative findings in the main paper and the human-feedback appendix. Across the three axes, the errors include incorrect cross-modal identity binding for who, premature turn triggering for when, and a weakly grounded continuation for how. 23

Instant research discovery