Paper deep dive
TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming
Yibo Hu, Yu Qian, Mao Gu, Yingfan Tao, Yuhao Chen, Yongdong Luo, Zhuoqun Liu, Meiguang Jin, Junfeng Ma
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/24/2026, 5:42:52 AM
Summary
The paper introduces TLive-Omni, an omni-modal understanding model designed for e-commerce live streaming. It integrates image, video, audio, and text inputs into a unified representation space using a Qwen3.5 backbone and an audio encoder from Qwen3-Omni. Key innovations include Per-vGrid for temporal alignment, a three-stage supervised fine-tuning recipe, and Faithful-RFT, a reinforcement fine-tuning method using Group Relative Policy Optimization (GRPO) with task-verifiable rewards. The model is supported by a scenario-oriented capability taxonomy and a compact data production engine, demonstrating strong performance on e-commerce benchmarks and generalization on general benchmarks.
Entities (10)
Relation Signals (9)
TLive-Omni → usesbackbone → Qwen3.5
confidence 95% · TLive-Omni builds on a Qwen3.5 (Qwen Team, 2026a) backbone
TLive-Omni → usesmethod → Faithful-RFT
confidence 95% · we propose Faithful-RFT, a reinforcement fine-tuning stage
TLive-Omni → usesmethod → Per-vGrid
confidence 95% · we introduce Per-vGrid, a timestamped token organization... to facilitate temporal alignment
Faithful-RFT → usesalgorithm → GRPO
confidence 90% · Faithful-RFT uses Group Relative Policy Optimization (GRPO) (Shao et al., 2024)
TLive-Omni → usesaudioencoderfrom → Qwen3-Omni
confidence 90% · integrates the pretrained audio encoder from Qwen3-Omni (Xu et al., 2025b)
TLive-Omni → usesmodelfor → TransNet-V2
confidence 85% · Physical Shot Detection (TransNet V2)
Faithful-RFT → usessoftware → vLLM
confidence 85% · with vLLM (Kwon et al., 2023) providing generation for this dynamic process
TLive-Omni → iscomparedwith → MiniCPM-o 4.5
confidence 80% · Open-source omni models such as MiniCPM-o 4.5... demonstrate the feasibility... yet their data... are not primarily organized around product-centric live streaming understanding.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:E-commerce live streaming requires omni-modal understanding of noisy, temporally extended streams, where product facts are distributed across speech, video frames, product images, overlaid text, and user queries. We present TLive-Omni, an omni-modal understanding model tailored to live-commerce scenarios. It maps image, video, audio, and text inputs into a unified representation space. For long-form live streaming analysis, we introduce Per-vGrid, a timestamped token organization that groups each video grid with its temporally corresponding audio within explicit boundary tokens to facilitate temporal alignment. We design a three-stage supervised training recipe that progressively develops live-commerce understanding, from omni-modal perception to instruction-following responses. We then propose Faithful-RFT, a reinforcement fine-tuning stage that further improves answer faithfulness and expression quality while meeting real-time demands, scoring final responses directly with task-verifiable feedback rather than optimizing for reasoning-style exploration during rollout. Moreover, TLive-Omni is supported by a scenario-oriented atomic capability taxonomy and a compact data production engine that converts live-commerce audio, image, and video streams into training signals for speech recognition, speaker analysis, product visual grounding, text recognition, temporal grounding, video dense caption, and omni-modal QA, etc. For scalable training, a synchronized length-grouped sampler reduces padding while preserving comparable workloads across workers, while a lightweight dynamic sampling strategy regenerates rollout groups with near-zero reward variance to maintain meaningful relative advantages for GRPO. Experiments on e-commerce live streaming benchmarks demonstrate strong performance across live-commerce domain tasks, together with excellent generalization on general benchmarks.
Tags
Links
- Source: https://arxiv.org/abs/2608.20958v1
- Canonical: https://arxiv.org/abs/2608.20958v1
Trouble viewing inline? Open PDF directly →
Full Text
89,566 characters extracted from source content.
Expand or collapse full text
TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming TLive-Omni Team Taobao & Tmall Group of Alibaba @ https://github.com/TaoLiveAIGC/TLive-Omni https://huggingface.co/TaoLiveAIGC/TLive-Omni-4B https://huggingface.co/TaoLiveAIGC/TLive-Omni-9B Abstract E-commerce live streaming requires omni-modal understanding of noisy, tempo- rally extended streams, where product facts are distributed across speech, video frames, product images, overlaid text, and user queries. We present TLive-Omni, an omni-modal understanding model tailored to live-commerce scenarios. It maps im- age, video, audio, and text inputs into a unified representation space. For long-form live streaming analysis, we introduce Per-vGrid, a timestamped token organiza- tion that groups each video grid with its temporally corresponding audio within explicit boundary tokens to facilitate temporal alignment. We design a three-stage supervised training recipe that progressively develops live-commerce understand- ing, from omni-modal perception to instruction-following responses. We then propose Faithful-RFT, a reinforcement fine-tuning stage that further improves an- swer faithfulness and expression quality while meeting real-time demands, scoring final responses directly with task-verifiable feedback rather than optimizing for reasoning-style exploration during rollout. Moreover, TLive-Omni is supported by a scenario-oriented atomic capability taxonomy and a compact data production engine that converts live-commerce audio, image, and video streams into training signals for speech recognition, speaker analysis, product visual grounding, text recognition, temporal grounding, video dense caption, and omni-modal QA, etc. For scalable training, a synchronized length-grouped sampler reduces padding while preserving comparable workloads across workers, while a lightweight dy- namic sampling strategy regenerates rollout groups with near-zero reward variance to maintain meaningful relative advantages for GRPO. Experiments on e-commerce live streaming benchmarks demonstrate strong performance across live-commerce domain tasks, together with excellent generalization on general benchmarks. 1 Introduction E-commerce live streaming poses a focused but challenging setting for omni-modal understanding. Product facts are distributed across host speech, video frames, product images, overlaid text, and user queries, while the supporting evidence may appear at different moments of a long stream. As a result, models must jointly interpret heterogeneous signals rather than process each modality in isolation, align audio and visual evidence to product-centric temporal segments, and express perception-derived answers faithfully for tasks such as automatic speech recognition, optical character recognition, product visual grounding, temporal grounding, and omni-modal question answering. General omni models and e-commerce-oriented systems have made important progress in this direction, but live- commerce understanding remains under-specified. Open-source omni models such as MiniCPM-o 4.5 (Cui et al., 2026), Qwen3-Omni (Xu et al., 2025b), OmniVinci (Ye et al., 2025), and Nemotron 3 Nano Omni (Deshmukh et al., 2026) demonstrate the feasibility of unified omni interaction, yet their data and evaluation are not primarily organized around product-centric live streaming understanding. arXiv:2608.20958v1 [cs.AI] 21 Aug 2026 Valley3 (Chen et al., 2026b) extends toward e-commerce scenarios. However, Valley3 is not primarily organized around fine-grained atomic capabilities for live streaming. We present TLive-Omni, an omni-modal understanding model tailored to e-commerce live streaming. To couple heterogeneous modalities, TLive-Omni builds on a Qwen3.5 (Qwen Team, 2026a) backbone and integrates the pretrained audio encoder from Qwen3-Omni (Xu et al., 2025b) into a unified interface. TLive-Omni supports up to 256K tokens of multimodal context, providing long-context capacity for extended live streaming segments. For audio–video alignment, we introduce Per-vGrid, which groups each video grid with the audio covering the same time interval in a span marked by explicit boundary tokens, together with a textual timestamp computed from the actual sampled frame indices. This explicit grid-level grouping keeps matched visual and audio evidence adjacent and makes their correspondence directly identifiable in the input sequence. Training begins with a three- stage supervised fine-tuning recipe that progressively develops live streaming understanding using both live-commerce and general multimodal supervision. Since perception-centered live-commerce tasks require answers that are both faithful to perceived evidence and timely enough for real-time live streaming, we introduce Faithful-RFT, a reinforcement fine-tuning stage that suppresses explicit think traces and scores final answers directly with task-verifiable rewards, improving answer faithfulness and expression quality for live-commerce understanding tasks. During rollout, a lightweight dynamic strategy resamples response groups with near-zero reward variance to yield higher-variance group- relative feedback, with vLLM (Kwon et al., 2023) providing generation for this dynamic process. To support heterogeneous multimodal training across stages, our synchronized length-grouped sampling organizes mixed-modality batches with more compatible sequence lengths. TLive-Omni is organized around a scenario-oriented capability taxonomy and a compact data produc- tion engine. The taxonomy cover atomic capabilities across audio, image, video, and omni-modal understanding, including speech recognition, speaker analysis, product visual grounding, text recog- nition, temporal grounding, video dense caption, and omni-modal QA, etc. Based on this taxonomy, the data engine maps live-commerce streams into capability-specific supervision, enabling staged training data to support systematic improvement in live-commerce understanding. We further construct an in-house live-commerce evaluation suite to verify live streaming understand- ing capabilities across key dimensions. Experiments on in-house live-commerce, general-purpose multimodal and omni benchmarks indicate that TLive-Omni achieves strong performance on the business-oriented tasks while obtaining leading results on several general-purpose benchmarks, and remains competitive on the rest. 2 Architecture 2.1 Overview TLive-Omni is a text-only output omni-modal understanding model for image, video, audio, and text inputs. Figure 1 summarizes the architecture. It uses a Qwen3.5 backbone (Qwen Team, 2026a) as the language and vision substrate, and grafts the audio transformer (AuT) encoder from Qwen3-Omni (Xu et al., 2025b) into the same embedding space through a lightweight audio aligner. The staged training recipe then aligns the extended audio pathway with the backbone for live streaming understanding. 2.2 Vision-Language Backbone TLive-Omni uses Qwen3.5 (Qwen Team, 2026a) as its vision-language backbone, which provides the language-model substrate and native visual processing pipeline. After spatial merging, each image contributes(h/32)× (w/32)visual tokens, and each sampled video contributes⌈f/2⌉× (h/32)× (w/32)visual tokens, wherefis the number of sampled frames andh,ware the resized height and width, respectively. The native Qwen3.5 vision aligner applies a multi-layer perceptron (MLP) to map the merged visual features to the backbone embedding dimension. 2.3 Audio Encoder In live commerce, host speech carries many product facts that are not visible in video frames. Transcribing speech with an external ASR system and feeding only the resulting text to the model would discard the temporal correspondence between speech and video, as well as paralinguistic cues 2 Figure 1: The architectural overview of TLive-Omni. It is built upon a Qwen3.5 backbone and extended with the AuT audio encoder through a lightweight audio aligner. The inset illustrates the Per-vGrid token organization that groups each video grid with its temporally corresponding audio within explicit boundary tokens. such as speaker identity. TLive-Omni therefore keeps audio as a first-class input modality. The audio encoder is the AuT adopted from Qwen3-Omni (Xu et al., 2025b), trained from scratch on 20 million hours of audio data. It consumes 128-dimensional mel-spectrogram features at 16 kHz, supports variable-length audio, and compresses speech into approximately 13 tokens per second, keeping long recordings computationally feasible within the context budget. A two-layer aligner projects the audio features into the backbone embedding space. 2.4 Multimodal Temporal Alignment Long live streaming understanding requires explicit audio–video correspondence in the input token sequence. We introduce Per-vGrid, which organizes a video and its corresponding audio into a sequence of timestamped video grids. The visual content of a temporal grid and the audio segment covering the same time interval are placed in the same local span. Compared with Qwen3-Omni (Xu et al., 2025b), Per-vGrid additionally prepends an explicit textual timestamp to each grid, makes the grid boundaries explicit, keeps each grid’s video and audio tokens contiguous, and separates neighboring grids at the sequence level. This distinction is illustrated in Figure 1. Per-vGrid further derives each grid’s timestamp and audio span from the realized video sampling process. This distinction becomes visible when frame sampling involves integer rounding. As shown in Figure 1, consider a 119-frame source video at 30 FPS, whose duration is approximately 3.97 seconds. With a requested sampling rate of 2 FPS, the actual sampled frame indices can be [0, 20, 39, 59, 79, 98, 118], giving a realized sampling rate of7/119× 30 ≈ 1.76FPS rather than exactly 2 FPS. Per-vGrid follows these actual sampled frames when assigning timestamps and audio spans. Since temporal patching groups two sampled frames into one grid and pads the final frame when necessary, the seven sampled frames form four grids, with each full grid spanning about 2/1.76≈ 1.13seconds rather than the 1.0 seconds implied by the requested rate. With roughly 13 audio tokens per second, this changes the audio span of a full grid from about 13 tokens to about 14–15 tokens. Thus, when integer frame selection makes the realized sampling rate differ from the requested rate, both the grid timestamp and its audio-token span follow the actual sampled frames, preserving more precise temporal alignment. 3 Supervised Fine-Tuning: Data and Recipe 3.1 Data Construction Constructing supervision for omni-modal live-commerce understanding cannot rely on a single shared strategy: audio, image, and video sources exhibit distinct noise patterns and therefore require modality-specific construction. We process raw e-commerce data and curated general-domain data 3 RawDataPreprocessing / SegmentationModality-Specific Construction&Quality RefinementCurated Outputs Live-Stream / Open-Domain Audio Audio Live-Stream / Open-Domain Image Image Live-Stream / Open-Domain Video Video VAD Silence Detection Audio Normalization ASR Model Ensemble Cross-Model Agreement Voting MLLM-based Temporal SpeakerDiarization Te m p o ra l IoU Consistency Check Audio-LLM Ensemble Multi-Dims Captioning SoundMusic Content Rhythm LLM-based Live-Keyword Extraction Video Back-tracing LLM-based Aggregation & Cleaning High-Quality ASR Data Live-Specific Keyword Lexicon Speaker-Labeled Audio-Visual ASR Data Audio Caption Data Audio QA Data (1) ASR / speaker / caption construction from raw live and open-domain audio Product-Name Retrieval VLM Task Labeling Tex t-Label Cleaning Semantic Clustering Category-Balanced Sampling Generic Detection Data VLM Detector LabelingVLMJudgerCleaning Target Exists? Product Visual Grounding Data Rejected-Sample Data Text Localization/Recognition Markdown & HTML Parsing CaptionSelling-Point Ye s No VLMJudgerCleaning Re-render Validation Aggregation Voting VLMJudgerCleaningAggregation Voting Tex tData MarkdownData HTMLData ImageCaptionData Selling-Point Data (2) Grounding / detection / parsing / content-understanding construction from raw images (3) Dense-caption / QA / temporal-grounding construction from raw video Physical Shot Detection (TransNet V2) Semantic Segmentation (VLM-based) ASR Extraction Visual Perception Decoupled Dense-Cap Construction LLMFusion Native End-to-End QA/TG Generation (Capable VLM) Multi-Task Ope rator （Dense-Cap/QA/TG） Scene-Oriented Resampling Temporal Calibration Quality Refinement VLM Judger (CoT Feedback) Video Dense Caption Data Video QA Data Temporal Grounding Data ImageQASamples Reasoning-AnswerGeneration (CapableVLM) Rule / LLM Answer Judging Reasoning-Answer Consistency Check Reasoning-Enriched Image QA Data No-AudioVideo QA Samples Reasoning-AnswerGeneration (CapableVLM) Rule / LLM Answer Judging Reasoning-Answer Consistency Check Reasoning-Enriched Video QA Data Deterministic Processing Generative /Semantic ModelHybrid ProcessingConditional Decision Curated Data Output Audio-only Time-domain SpeakerDiarization Figure 2: TLive-Omni three-stage SFT data construction framework. Modality-specific generation and quality control transform audio, image, and video sources into task-grounded supervision for the three-stage SFT recipe. through separate audio, image, and video pathways, each with its own filtering and quality control, turning noisy inputs into task-grounded supervision. Figure 2 summarizes the source, construction, filtering, and output stages. Audio pathway. Live-commerce speech poses two practical challenges for audio labeling. Hosts speak quickly and use domain-specific low-frequency terms such as brand names, materials, colors, and model numbers, which general ASR models may fail to recognize. Overlapping speakers and short interjections also make single-pass end-to-end diarization unreliable. After voice-activity detection (VAD) and audio normalization, we use cross-model agreement voting across an ASR model ensemble to obtain ASR pseudo-labels. A large language model (LLM) mines these transcripts for domain- specific low-frequency terms and builds a live-specific keyword lexicon for in-context ASR. For speaker-aware supervision, we cross-validate two independently generated speaker labels for the same segment: an audio-only time-domain diarization model assigns speaker identities from acoustic features alone, while a multimodal large language model (MLLM) predicts speaker identity from the ASR transcript together with visual cues. We compare the time spans from the two streams with a temporal intersection-over-union (IoU) consistency check, retaining high-overlap segments directly. In terms of mismatched segments, we use the corresponding video frames and lip-motion cues to verify the speaker assignment. Broader audio understanding is handled by separating captioning into sound, music, speech content, and speaker rhythm. Each dimension is generated by an audio-LLM ensemble and merged by an LLM into audio caption and audio QA-pair data. Image pathway.The image pathway addresses challenges that are common in e-commerce imagery. Manual bounding-box annotation is expensive because product categories are visually diverse and live-stream backgrounds are cluttered. We construct product visual grounding data with a vision- language model (VLM) Detector–Judger loop. The detector proposes candidate boxes, the judger filters out inaccurate ones as rejected-sample data. For open-source generic-detection data, we cluster raw text labels into standardized categories and apply category-balanced sampling to reduce label noise and long-tail bias. In terms of Markdown and HTML parsing, we re-render the parsed content 4 and compare it with the source image rather than trusting the VLM’s output directly. Caption and selling-point data are filtered in the same source-consistency manner, with a VLM judger removing unsupported descriptions before aggregation. For reasoning-enriched Image QA, a capable VLM generates an answer with a reasoning trace for each selected Image QA sample. We then apply rule- or LLM-based judging to score the answer and keep only samples that pass judging and whose answer remains consistent with the trace. Video pathway.Video annotation is complicated by the mismatch between physical shot boundaries and semantic event boundaries. For dense captioning, TransNet V2 (Sou ˇ cek and Loko ˇ c, 2020) splits each video at physical shot boundaries, generating visually coherent clips. For each clip, a dedicated ASR model extracts the speech, while a VLM describes the visual content. An LLM combines the transcript and visual description into a dense caption. For Video QA and temporal grounding, a VLM segments each video at semantic event boundaries, thereby keeping each clip focused on a self- contained semantic event. A capable VLM then directly generates QA pairs and temporal-grounding annotations from these clips. The dense captions, QA pairs, and temporal-grounding annotations share a common refinement pipeline. Scene-oriented resampling improves coverage of long-tail scenarios, while temporal calibration adjusts clip boundaries to create targets of different durations. A VLM judger then checks factual and logical consistency as well as timestamp alignment, correcting or removing low-quality annotations. For reasoning-enriched Video QA, we use QA samples from no-audio videos. A capable VLM generates an answer with a reasoning trace for each sample. We validate multiple-choice and numerical answers through rule-based matching and free-form answers with an LLM judge, retaining only samples that pass these checks. 3.2 Three-Stage SFT Recipe TLive-Omni follows a three-stage supervised fine-tuning (SFT) recipe that first establishes audio– language alignment, then strengthens audio understanding, and finally performs joint adaptation with audio, image, video, and text supervision. This progression separates modality alignment from capability learning and full multimodal adaptation, allowing each stage to update only the components required by its objective. Stage 1 freezes the language model and audio encoder and trains only the audio aligner on 5M ASR samples, establishing an initial mapping from acoustic representations to the language-model embedding space. Stage 2 introduces a broader audio mixture comprising ASR, audio captioning, and audio QA over 26M audio samples. Training the audio encoder together with its aligner, while keeping the language model frozen, extends the audio pathway beyond transcription to speech content, sound events, music, and speaker-related cues. Stage 3 performs joint multimodal supervised fine-tuning over 14M multimodal samples spanning audio, image, video, and text data. The audio and visual encoders remain frozen, whereas their aligners and the language model are optimized for tasks including speech recognition, speaker analysis, product visual grounding, text recognition, temporal grounding, video dense caption, omni-modal QA, and reasoning-enriched image and video QA. Detailed settings are reported in Appendix D.1. 3.3 Synchronized Length-Grouped Sampling Training on heterogeneous multimodal data poses a practical batching challenge. Audio clips, product images, OCR-heavy pages, short videos, and long video segments vary substantially in token length and computational cost. Random sampling can place examples of disparate lengths in the same global batch, increasing padding and creating workload imbalance across workers. Sequence packing reduces padding by concatenating short samples, but introduces a trade-off between fixed source- sample counts and full token-budget utilization. Suppose the memory budget isNLtokens, enough forNsequences of lengthL. If each step is restricted toNsource samples, packing several short samples into one sequence may produce onlyN ′ < Npacked sequences, leaving part of the available token budget unused. Filling this budget with additional samples instead makes the number of source samples variable across steps, complicating control over the effective sample-level batch size and per-step data mixture. Packing also requires careful handling of block-diagonal attention, position IDs, loss masks, and multimodal metadata to preserve sample boundaries. To address these issues, we propose a synchronized length-grouped sampler that forms fixed-size global batches without merging source samples. During initialization, it partitions samples by modal- ity, sorts each partition by token length, and splits the sorted samples into global batches. This 5 Algorithm 1: Synchronized length-grouped sampling Require: DatasetD, modalitiesM, world size R, local batch size b, seed s, epoch e, rank r 1: Phase I: Registry construction (once per run) 2: Set global batch size B ← Rb 3: for each modality m∈M do 4:Collect indicesI m and sort by token length// Reduce padding 5:Set K m ←⌊|I m |/B⌋ 6:FormQ m ←I m [kB : (k + 1)B] K m −1 k=0 7:DiscardI m [K m B :|I m |]// Keep global batch size fixed 8: end for 9: Make the registryQ m m∈M identical across workers 10: Phase I: Synchronized scheduling (each epoch e) 11: Initialize every worker with seed s + e// Identical random state 12: Every worker builds the same shuffled copy e Q m of eachQ m 13: while at least one e Q m is nonempty do 14:Sample m in proportion to| e Q m |// Number of remaining batches 15:Every worker pops the same global batchG t from e Q m 16:Worker r yieldsG t [rb : (r + 1)b] 17: end while organization reduces padding while preserving a fixed sample count. At each epoch, all workers use the same epoch-dependent seed. They therefore select the same modality and the same global batch at every step. Each selected global batch is partitioned into disjoint local batches, one per worker. Because samples within a global batch have similar lengths, these local batches impose comparable workloads across workers. During training, we enable the sampler with sequence packing disabled. Algorithm 1 gives the detailed procedure. 4 Faithful-RFT The three-stage supervised recipe equips TLive-Omni with multimodal perception and task-solving capabilities, but its likelihood objective does not directly incorporate task-specific feedback on gen- erated responses. In live-commerce applications, these responses must faithfully reflect perceived evidence while remaining timely for real-time understanding of live streams. To meet these require- ments, we introduce Faithful-RFT after the three-stage SFT. Faithful-RFT uses Group Relative Policy Optimization (GRPO) (Shao et al., 2024) with task-verifiable rewards that directly score final responses. This approach improves answer faithfulness and expression quality without explicitly rewarding reasoning length or visible reasoning traces, thereby avoiding unnecessary generation overhead for real-time live streaming. 4.1 Faithful-RFT Framework Initialization and data organization. Faithful-RFT starts from the Stage-3 model without a separate cold-start SFT stage. To preserve modality-specific schemas within a single optimization stage, we organize the data into four streams. We uset i to denote the task associated with an example, so that reward assignment can be formulated as task-conditioned routing rather than as a modality- level rule. The image stream includes representative tasks such as image QA, visual grounding, text recognition, and detailed captioning. The video-with-audio stream covers detailed captioning and Omni QA, while the video-without-audio stream focuses on video QA, shot understanding and temporal grounding. The audio stream includes ASR and audio QA. All streams are mixed within one GRPO stage rather than optimized as separate sequential stages. Grouped rollout and optimization.For each multimodal promptx i , the policy samples a group of Gcandidate responsesy i,g G g=1 . The visual encoder, visual aligner, audio encoder, and audio aligner remain frozen during policy optimization. Following the outcome-supervision formulation of GRPO, all tokens in a response share the group-relative advantage below, whereR i,g is the aggregated scalar 6 reward defined in Eq. 5: b A i,g = R i,g − mean g ′ (R i,g ′ ) std g ′ (R i,g ′ ) + ε s .(1) Letπ θ ,π θ old , andπ ref denote the policy being optimized, the rollout policy, and the frozen reference policy, respectively. In Eq. 2, we use local group notation withx = x i ,y g = y i,g , b A g = b A i,g , and prefix y g,<t . The implementation minimizes the group-level loss L i (θ) = 1 G G X g=1 1 |y g | |y g | X t=1 " − min π θ (y g,t | x,y g,<t ) π θ old (y g,t | x,y g,<t ) b A g , clip π θ (y g,t | x,y g,<t ) π θ old (y g,t | x,y g,<t ) , 1− ε l , 1 + ε h b A g + βd KL g,t # . (2) Hered KL g,t is the token-level KL penalty against the frozen reference policy under the same local notation. Faithful-RFT suppresses unnecessary explicit think traces, rather than encouraging visible reasoning traces or an additional thinking process. The corresponding optimization hyperparameters are reported in Appendix D. Rollout strategy.Rollout responses are generated by a vLLM engine deployed within the training recipe. Faithful-RFT extends the synchronized length-grouped sampler from Section 3.3 with task- aware bucketing and repeated sampling. The sampler assigns each example to a bucket defined by its modality and task identifiert i , so different tasks within the same modality are grouped separately. Within each bucket, examples are ordered by sequence length as in the sampler. This length grouping reduces input-side padding during rollout. For GRPO training, the repeated sampler yields each prompt indexGtimes, and the model generates theGcandidate responses. The repeated sampler also keeps a starvation counter over active buckets and prioritizes a bucket once it has remained unselected beyond the configured threshold. During generation, we further use a lightweight dynamic resampling strategy to keep GRPO updates informative. Since the group-relative advantage in Eq. 1 depends on reward differences among theG responses to the same prompt, a group whose aggregated rewards are all identical provides no relative preference signal. After scoring the rollout responses, the trainer computes the reward variance within each group, retains groups with nonzero variance, and regenerates candidate groups in cases of near-zero variance with adjusted generation settings or rewritten input prompts. This mechanism increases the proportion of groups that can produce meaningful relative advantages. 4.2 Task-Conditioned Reward Function The GRPO objective above requires a scalar rewardR i,g for each candidate response. In Faithful-RFT, this scalar reward is computed through task-conditioned reward routing, because a single multimodal batch can contain tasks with different notions of correctness. We use task-conditioned reward routing to select the reward functions applicable to each example. Each reward functionf j declares an applicable task setT j and is evaluated only whent i ∈ T j . Inapplicable rewards return an invalid sentinel and are removed before reward aggregation. Reward taxonomy.The reward pool is organized by evaluation mechanism and applied condition- ally across the four data streams above. Rule-based rewards handle tasks with deterministic targets or machine-checkable structures, including multiple-choice questions, visual grounding and OCR. Some of these rewards use an LLM only to extract a final answer from a response that may contain an explanation or short thinking trace, and the extracted answer is still scored by a deterministic rule. LLM-judge rewards evaluate open-ended responses other free-form multimodal understanding tasks, for which exact string matching is insufficient. Finally, the final reward uses a lightweight format constraint to suppress unnecessary explicit think tags, without assigning reward to reasoning length, reasoning content, or visible chain-of-thought quality. Reward aggregation.To obtain the scalar reward required by GRPO for each candidate, we exclude inapplicable or invalid reward outputs and renormalize the configured weights over the remaining rewards. Letr i,g,j be the score assigned by reward functionjto candidatey i,g , and letw j be its 7 configured weight. A reward is valid only when it is applicable to the example and returns a finite numerical score rather than the invalid sentinel: v i,g,j = 1[t i ∈T j ∧ valid(r i,g,j )].(3) The configured weights are normalized over the valid rewards for each candidate: ew i,g,j = v i,g,j w j P k v i,g,k w k , P k v i,g,k w k > 0, 0,otherwise. (4) The scalar reward used by GRPO is then R i,g = X j:v i,g,j =1 ew i,g,j r i,g,j .(5) As a result, each example is scored only by reward functions applicable to its task. Together, the three-stage supervised recipe and Faithful-RFT yield two TLive-Omni variants: TLive-Omni-4B and TLive-Omni-9B. 5 Evaluation We evaluate TLive-Omni model along two axes: live-commerce tasks that reflect the target application, and general benchmarks that measure the generalization capabilities. 5.1 Live-Commerce Evaluation We evaluate the core multimodal capabilities required for live-commerce understanding. The suite is built from live-commerce sources and covers speech transcription, speaker-attributed ASR, audio description and question answering, product visual grounding, text localization/recognition/classifica- tion, temporal grounding, dense video caption, video question answering, and shot understanding. Appendix B defines the detailed evaluation metrics used in these tasks. ModelParams Live-Commerce ASR Speaker-Attributed ASR Audio Description Audio QA CER↓cpWER↓Acc.↑ Hal.↓Acc.↑ Closed-source Omni models Gemini 2.5 Flash-16.3017.1465.2126.1976.28 Gemini 2.5 Pro-11.4812.1781.1014.1682.85 Gemini 3 Flash-15.1819.0468.2726.1774.68 Gemini 3 Pro-12.0911.6785.0710.9288.62 Gemini 3.5 Flash-13.0911.9979.9714.3687.99 Qwen3.5-Omni Flash-6.8113.2362.8227.8178.04 Open-source Audio models MiMo-Audio7B12.71–64.2626.0170.97 Fun-Audio-Chat8B14.55–61.3532.2169.71 Step-Audio-R1.132B10.21–75.0820.5069.80 Open-source Omni models OmniVinci9B–39.9047.3666.51 Nemotron 3 Nano Omni 30B-A3B12.1017.6533.0139.7764.90 Ming-Lite-Omni v1.520B-A3B10.06–45.9944.0540.54 MiniCPM-o 2.68B13.88–49.8441.7639.74 MiniCPM-o 4.59B10.7018.8947.5952.4142.47 Qwen2.5-Omni7B7.86–47.9236.9261.38 Qwen3-Omni30B-A3B6.7527.8461.0630.2276.76 Ours TLive-Omni4B6.66 12.8876.1220.9772.60 TLive-Omni9B6.4612.2775.9621.0076.28 Table 1: Live-commerce audio evaluation covering live-commerce ASR, speaker-attributed ASR, audio description and question answering. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold, while the second- best results are in underlined. We compare against open-source baselines from the MiniCPM-family models (OpenBMB, 2025; Cui et al., 2026), Ming-Lite-Omni (Inclusion AI et al., 2025), OmniVinci (Ye et al., 2025), Nemotron 3 8 Nano Omni (Deshmukh et al., 2026), Qwen-Omni-family models (Xu et al., 2025a,b; Qwen Team, 2026b), Step-Audio (StepFun AI, 2026), Fun-Audio-Chat (Tongyi Fun Team et al., 2025), and MiMo-Audio (Zhang et al., 2025c), together with Gemini-family models (Gemini Team Google, 2023; Comanici et al., 2025; Google DeepMind, 2025a,b, 2026b). As shown in Table 1, we evaluate live-commerce ASR using character error rate (CER) and speaker- attributed ASR using concatenated minimum-permutation word error rate (cpWER), which minimizes the total error over speaker assignments. For audio description, Accuracy (Acc.) evaluates whether generated descriptions support correct answers to audio-grounded questions, while Hallucination Rate (Hal.) reports the rate of unsupported content in those descriptions. Audio-QA Accuracy measures audio question answering under rule-based answer matching. On ASR, TLive-Omni-9B achieves the lowest CER, with TLive-Omni-4B close behind. Their cpWER scores are among the lower reported results, showing strong performance on speaker-attributed ASR for live-stream. Table 2 shows product-centric image understanding including product visual grounding and text understanding. We evaluate product visual grounding with Average Precision at an IoU threshold of 0.5 (AP@IoU=0.5) in both live-stream frames (Live) and product images (Prod). For text under- standing, we report text localization F1 score (Loc. F1), normalized edit distance for text recognition (Rec. NED), and text classification accuracy over commerce-oriented semantic labels (Cls. Acc.). Rec. NED is reported as a percentage. The two TLive-Omni variants achieve the highest Prod AP, text localization, and classification scores, as well as the lowest recognition edit distances among the evaluated open-source and closed-source models, while also remaining competitive with Gemini 3.5 Flash (Google DeepMind, 2026b), the strongest closed-source model on Live AP. ModelParams Visual GroundingText Understanding Live AP↑ Prod AP↑ Loc. F1↑ Rec. NED↓ Cls. Acc.↑ Closed-source Omni models Gemini 2.5 Flash-61.0828.8120.5243.2851.21 Gemini 2.5 Pro-51.9832.6331.6027.8261.86 Gemini 3 Flash-80.3865.6761.1116.2569.11 Gemini 3 Pro-73.8058.8368.609.7276.86 Gemini 3.5 Flash-84.1574.8964.4416.6469.76 Qwen3.5-Omni Flash-79.9660.4474.0712.4853.25 Open-source Omni models OmniVinci9B34.868.9350.2532.7757.29 Nemotron 3 Nano Omni 30B-A3B73.0848.6252.9129.4237.86 Ming-Lite-Omni v1.520B-A3B52.4640.7313.2759.1632.94 MiniCPM-o 2.68B3.821.775.7477.5815.92 MiniCPM-o 4.59B23.9053.635.4371.6511.62 Qwen2.5-Omni7B75.6122.8542.6437.7951.25 Qwen3-Omni30B-A3B79.2268.8830.4614.8369.46 Ours TLive-Omni4B82.8591.4586.994.7279.06 TLive-Omni9B82.3389.9687.594.2479.85 Table 2: Live-commerce image evaluation covering product visual grounding and text understanding. A dash denotes an undisclosed parameter count. The Best results among open-source models are marked in bold, while the second-best results are in underlined. Table 3 reports video understanding abilities across temporal grounding, dense video caption, video question answering, and shot understanding. We use mIoU to evaluate the temporal grounding (TG) ability of product or event intervals in live-stream videos. When an example contains multiple intervals, scoring uses interval-level matching rather than a single union IoU over all segments. Dense Caption Accuracy and Hallucination Rate report the correctness and unsupported-content rate of time-aware dense video descriptions, while Video QA Accuracy reports question answering accuracy over video evidence. Shot understanding is evaluated across four structured perspectives: layout, shot size, camera angle, and content category. TLive-Omni-9B achieves the highest TG mIoU, Video QA Accuracy, Dense Caption Accuracy, and the lowest Hallucination Rate, while TLive-Omni-4B obtains 9 the second-best open-source results on the same four metrics. For shot understanding, TLive-Omni-9B ranks first in camera angle and content category among open-source models, while TLive-Omni-4B ranks second in shot size and content category. These results highlight the advantage of TLive-Omni in live video understanding. ModelParams TG Dense Caption Video QAShot Understanding mIoU↑ Acc.↑ Hal.↓ Acc.↑ Layout↑ Shot Size↑ Camera↑ Content↑ Closed-source Omni models Gemini 2.5 Flash-76.5054.6010.9788.2180.0046.8084.2068.60 Gemini 2.5 Pro-76.2241.9516.8892.6285.2050.8076.0070.40 Gemini 3 Flash-77.4332.2120.7689.6476.8045.7078.5071.60 Gemini 3 Pro-77.9037.8020.9984.3680.4043.4075.7074.80 Gemini 3.5 Flash-77.9033.8017.3086.9083.4044.2075.5070.20 Qwen3.5-Omni Flash-62.1032.9420.9187.2884.4048.9085.5066.40 Open-source Omni models OmniVinci9B13.1018.5927.1372.5173.6052.7068.1049.20 Nemotron 3 Nano Omni 30B-A3B23.3917.9616.6282.5679.2034.0080.2058.60 Ming-Lite-Omni v1.520B-A3B14.3413.8139.3364.5174.6041.7072.8051.60 MiniCPM-o 2.68B14.5610.5326.9360.3066.6038.3068.3049.00 MiniCPM-o 4.59B43.2021.0628.6184.6278.2042.8079.2066.60 Qwen2.5-Omni7B30.8316.5136.4475.4874.4038.1081.2068.00 Qwen3-Omni30B-A3B39.2221.4425.8281.6282.2037.4076.1063.60 Ours TLive-Omni4B77.63 69.239.5792.3178.4051.2080.9069.80 TLive-Omni9B81.49 74.63 8.7693.2377.0051.0082.0071.00 Table 3: Live-commerce video evaluation covering temporal grounding, dense video caption, video question answering, and shot understanding. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold, while the second- best results are in underlined. 5.2 General Benchmark Evaluation General benchmarks evaluate whether a model retains broad multimodal capabilities beyond the target live-commerce domain. This evaluation is important because vertical-domain specialization can improve target-domain performance while weakening general reasoning, perception, or cross-modal understanding, thereby narrowing the model’s usable scenarios. We therefore evaluate TLive-Omni across image reasoning and question answering, hallucination/OCR/grounding/spatial reasoning, video understanding and temporal grounding, and omni-modal perception and reasoning. The results show that TLive-Omni maintains strong generalization across these general multimodal benchmarks, improves over Qwen3.5 (Qwen Team, 2026a) 4B and 9B backbones on the majority of these benchmarks while retaining and eliciting broad omni-modal understanding capabilities. All models are evaluated in their instruction-tuned or chat variants, without dedicated reasoning modes. The general benchmark comparisons include Qwen-family models (Bai et al., 2025; Qwen Team, 2026a; Xu et al., 2025a,b; Qwen Team, 2026b), MiniCPM-family models (OpenBMB, 2025; Cui et al., 2026; Yu et al., 2025b), InternVL3.5 (Wang et al., 2025a), NVILA (Liu et al., 2024b), MiMo-VL (Yue et al., 2025), Ming-Lite-Omni (Inclusion AI et al., 2025), InteractiveOmni (Tong et al., 2025), VITA-1.5 (Fu et al., 2025), Valley-family models (Chen et al., 2026b; Valley Team, ByteDance Group, 2025), SAIL-VL2 (Yin et al., 2025), LLaVA-OneVision-family models (Li et al., 2024; An et al., 2026), LLaVA-Video (Zhang et al., 2025e), LongVU (Shen et al., 2024), LongVILA (Chen et al., 2024b), Kangaroo (Liu et al., 2024a), Video-XL-2 (Qin et al., 2025), VideoLLaMA 3 (Zhang et al., 2025a), VideoChat3 (Li et al., 2026), Molmo2 (Clark et al., 2026), Mage-VL (Yang et al., 2026), OmniVinci (Ye et al., 2025), Nemotron 3 Nano Omni (Deshmukh et al., 2026), video-SALMONN 2 (Tang et al., 2025), GPT-4o (OpenAI, 2024), GPT-5 (Singh et al., 2025), and Gemini-family models (Gemini Team Google, 2023; Comanici et al., 2025; Google DeepMind, 2025b, 2026a). The evaluation prompts are reported in Appendix E. Table 4 focuses on image-centric reasoning and question answering. MMMU (Yue et al., 2023) assesses multimodal understanding and reasoning with domain-specific knowledge, while Math- Vista (Lu et al., 2023) evaluates mathematical reasoning in visual contexts. DynaMath (Zou et al., 2024) measures mathematical reasoning robustness under visual and textual variations of the same 10 problem, whereas VLMsAreBlind (Rahmanzadehgervi et al., 2024) evaluates low-level visual per- ception requiring precise spatial information. MMBench (Liu et al., 2023a) and MMStar (Chen et al., 2024a) provide broad coverage of multimodal perception and reasoning. RealWorldQA (xAI, 2024) evaluates spatial and physical understanding of everyday scenes, while SimpleVQA (Cheng et al., 2025b) measures factuality in short-answer visual question answering. Compared with open-source baselines, TLive-Omni achieves the best results on MMBench and RealWorldQA, and ranks second on MMMU, MathVista, DynaMath, VLMsAreBlind, MMStar, and SimpleVQA. ModelParams MMMU MathVista DynaMath VLMsAreBlind MMBench RealWorldQA MMStar SimpleVQA Closed-source models Gemini 2.5 Flash-76.375.369.775.986.675.775.859.2 Gemini 2.5 Pro-80.977.778.578.588.476.078.566.9 Gemini 3 Pro-87.287.985.1–93.783.383.173.2 GPT-4o-70.763.854.4–86.0– GPT-5 (minimal)-74.450.974.053.481.377.365.256.7 Qwen3.5-Omni Flash-76.982.979.3–88.877.575.754.4 Open-source VLM models MiMo-VL-SFT7B64.681.846.978.084.5– SAIL-VL28B55.476.417.8–76.370.7– Valley2.58B62.174.432.7–85.570.567.3– LLaVA-OneVision-28B–85.769.764.8– InternVL3.54B66.677.135.7–80.366.365.0– InternVL3.58B73.4 78.437.7–79.567.569.3– Qwen3-VL4B67.473.765.371.983.970.969.848.0 Qwen3-VL8B69.677.267.774.084.571.570.950.2 Qwen3.54B72.181.069.662.386.372.574.844.6 Qwen3.59B74.282.274.671.887.7 72.976.348.9 Open-source Omni models InteractiveOmni4B61.161.7–78.9–62.6– InteractiveOmni8B66.968.0–81.4–66.8– VITA-1.57B52.166.2–76.7–59.9– Valley38B69.3– OmniVinci9B49.763.5–67.5– Nemotron 3 Nano Omni 30B-A3B55.271.9– Ming-Lite-Omni v1.520B-A3B54.372.0–65.1– MiniCPM-o 2.68B50.471.9–80.5–64.0– MiniCPM-o 4.59B67.6–87.6–73.1– Qwen2.5-Omni7B59.267.9–81.870.364.0– Qwen3-Omni30B-A3B69.175.9–68.5– Ours TLive-Omni4B70.979.972.571.887.077.773.947.6 TLive-Omni9B73.481.973.375.588.976.675.150.0 Table 4: General image benchmark results on MMMU, MathVista, DynaMath, VLMsAreBlind, MMBench, RealWorldQA, MMStar, and SimpleVQA. MMBench results are reported on the EN- DEV-v1.1 split. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold, while the second-best results are in underlined. Table 5 extends image evaluation to hallucination, diagram and chart understanding, OCR perception, visual grounding, embodied reasoning, and spatial reasoning. It includes HallusionBench (Guan et al., 2023) for hallucination, AI2D (Kembhavi et al., 2016) and CharXiv (Wang et al., 2024b) for diagram and chart understanding, OCRBench (Liu et al., 2023b) and C-OCR (Yang et al., 2024) for OCR perception, and RefCOCO (Yu et al., 2016), ERQA (Gemini Robotics Team, 2025), and EmbSpatialBench (Du et al., 2024) for grounding, embodied reasoning, and spatial reasoning. These results indicate that TLive-Omni preserves competitive general image reasoning while showing particular strength on hallucination, OCR-centric perception, and spatial reasoning. Table 6 shifts the comparison to video understanding. MVBench (Li et al., 2023) focuses on temporal understanding, MLVU (Zhou et al., 2024), LongVideoBench (Wu et al., 2024), and LVBench (Wang et al., 2024a) target long-context video reasoning, Video-MME (without subtitles) (Fu et al., 2024) provides broad-coverage video question answering, while MMVU (Zhao et al., 2025) and VideoM- MMU (Hu et al., 2025) measure expert-level knowledge-intensive video understanding across multiple disciplines. TLive-Omni remains competitive, with the 9B model leading open-source models on MLVU, Video-MME, LongVideoBench, and MMVU, and the 4B model leading on VideoMMMU. 11 ModelParams Hallusion AI2D OCRBench C-OCR CharXiv(RQ) RefCOCO ERQA EmbSpatial Closed-source models Gemini 2.5 Flash-59.187.786.474.860.1– Gemini 2.5 Pro-60.990.087.276.862.9–50.373.3 Gemini 3 Pro-68.694.190.479.081.484.170.561.2 GPT-4o-–82.684.3– GPT-5 (minimal)-53.784.178.766.157.8–42.075.1 Qwen3.5-Omni Flash-–89.089.180.864.492.650.082.7 Open-source VLM models MiMo-VL-SFT7B–83.287.6–54.485.7– SAIL-VL28B55.187.791.3–74.0– Valley2.58B56.384.487.0– LLaVA-OneVision-28B–84.378.2–43.378.1 InternVL3.54B44.882.682.2–39.689.438.5– InternVL3.58B54.584.084.0–44.489.7 41.073.2 Qwen3-VL4B57.684.188.176.239.789.041.379.6 Qwen3-VL8B61.185.789.679.946.489.145.878.5 Qwen3.54B76.9 87.185.971.162.987.646.876.6 Qwen3.59B76.088.088.573.467.590.047.378.7 Open-source Omni models InteractiveOmni4B52.283.880.0– InteractiveOmni8B61.384.383.7– VITA-1.57B44.979.373.2– Valley38B55.9– Nemotron 3 Nano Omni 30B-A3B–88.588.3–49.180.6– Ming-Lite-Omni v1.520B-A3B54.684.988.9–87.8– MiniCPM-o 2.68B51.985.889.7– MiniCPM-o 4.59B63.287.687.6– Qwen2.5-Omni7B–83.2–87.7– Qwen3-Omni30B-A3B59.785.286.0–61.1– Ours TLive-Omni4B77.786.686.680.561.387.442.379.3 TLive-Omni9B76.088.690.381.363.190.048.080.4 Table 5: General image benchmark results on HallusionBench, AI2D, OCRBench, C-OCR, CharXiv(RQ), RefCOCO, ERQA, and EmbSpatialBench. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold, while the second-best results are in underlined. Table 7 reports video temporal grounding performance on TimeLens-Bench (Zhang et al., 2025d). It includes Charades-TL, ActivityNet-TL, and QVHighlights-TL, which evaluate temporal grounding across short daily-life videos, longer activity videos, and mixed-domain videos, respectively. TLive- Omni-4B achieves the highest open-source mIoU on all three benchmarks, while TLive-Omni-9B ranks second on Charades-TL and ActivityNet-TL. These results show that TLive-Omni retains strong general video capability across broad video reasoning, long-context understanding, question answering, and fine-grained temporal grounding. Table 8 evaluates joint audio-video perception and reasoning with omni-modal benchmarks. AVUT (Yang et al., 2025) evaluates audio-centric video understanding without text shortcuts, Dai- lyOmni (Zhou et al., 2025) focuses on audio-visual reasoning with temporal alignment, World- Sense (Hong et al., 2025) evaluates real-world omnimodal understanding across visual, audio, and text inputs, and VideoHolmes (Cheng et al., 2025a) focuses on complex video reasoning. Om- niVideoBench (Li et al., 2025a) evaluates synergistic audio-visual understanding with an emphasis on modality complementarity, while FutureOmni (Chen et al., 2026a) measures future-event forecasting. Among open-source models, TLive-Omni-9B achieves the best results on AVUT, WorldSense, Dai- lyOmni, and FutureOmni, and ranks second on VideoHolmes and OmniVideoBench. These results indicate that TLive-Omni preserves strong general omni-modal capability across audio-centric video understanding, audio-visual temporal alignment, real-world omnimodal reasoning, and future-oriented video understanding. 12 ModelParams MVBench MLVU Video-MME LongVideoBench LVBench MMVU VideoMMMU Closed-source models Gemini 2.5 Flash-–77.875.6–62.268.265.2 Gemini 2.5 Pro-65.881.280.6–69.072.279.4 Gemini 3 Pro-74.183.087.776.776.277.587.6 GPT-4o-–71.9– GPT-5 (minimal)-64.678.377.3–68.161.6 Qwen3.5-Omni Flash-70.881.977.0–62.7– Open-source VLM models MiMo-VL-SFT7B–66.9–53.1 SAIL-VL28B–62.758.3– LLaVA-OneVision-28B66.276.671.9 66.955.556.2– LLaVA-Video7B58.670.863.358.244.247.136.1 InternVL3.54B71.270.465.460.843.247.657.6 InternVL3.58B72.170.266.062.146.760.2– MiniCPM-V 4.58B–75.167.963.950.458.957.1 LongVU7B66.965.460.6– LongVILA7B67.1–60.157.1– Mage-VL4B65.168.764.061.341.8– Molmo24B75.163.069.668.0 53.951.250.7 Molmo28B75.960.269.967.552.8– NVILA8B68.170.164.257.7– Kangaroo8B61.161.056.054.839.4– Video-XL28B–74.866.661.048.450.039.9 VideoChat34B–70.1–56.756.457.4 VideoLLaMA 37B69.773.066.259.845.344.134.6 Qwen3-VL4B68.975.369.3–56.250.556.2 Qwen3-VL8B68.778.171.4–58.058.765.3 Qwen3.54B66.675.171.665.155.357.869.8 Qwen3.59B75.7 79.766.967.960.963.770.3 Open-source Omni models InteractiveOmni4B–68.063.357.0– InteractiveOmni8B–71.666.059.1– VITA-1.57B55.4–56.1– Valley38B–55.6–61.2 OmniVinci9B70.6–68.261.3– Nemotron 3 Nano Omni 30B-A3B–70.8– Ming-Lite-Omni v1.520B-A3B69.4–67.159.5– MiniCPM-o 2.68B–63.9– MiniCPM-o 4.59B–76.570.466.0– Qwen2.5-Omni7B70.3–64.3– Qwen3-Omni30B-A3B–75.270.5– Ours TLive-Omni4B69.076.171.366.157.159.973.9 TLive-Omni9B72.580.975.669.960.867.172.8 Table 6: General video benchmark results on MVBench, MLVU, Video-MME, LongVideoBench, LVBench, MMVU, and VideoMMMU. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold, while the second-best results are in underlined. Taken together, the live-commerce and general benchmark evaluations show that TLive-Omni provides consistent multimodal understanding across audio, image, and video inputs. On live-commerce benchmarks, the models achieve strong results in ASR and speaker-attributed ASR, product visual grounding and text understanding, temporal grounding, dense video caption, video question answering, and shot understanding. These results cover perception and reasoning tasks that require temporal, visual, and audio evidence. On general benchmarks, TLive-Omni remains competitive in image- centric reasoning and question answering, hallucination and OCR evaluation, visual and spatial grounding, long-context video understanding, temporal grounding, and omni-modal perception and reasoning. Across the two model sizes, the 9B variant achieves the best open-source results on many of the reported metrics, while the 4B variant also obtains the best or second-best open-source results across multiple benchmarks. This overall pattern indicates that the strong performance on live-commerce tasks is accompanied by broad performance across general multimodal benchmarks rather than remaining limited to the target domain. 13 ModelParamsCharades-TLActivityNet-TLQVHighlights-TL Closed-source models Gemini 2.5 Flash-48.652.564.3 Gemini 2.5 Pro-52.858.170.4 GPT-4o-41.840.452.1 GPT-5 (minimal)-40.542.956.8 Open-source VLM models MiMo-VL-SFT7B39.635.541.5 LLaVA-OneVision-28B53.553.866.4 LLaVA-Video7B15.214.610.4 InternVL3.54B16.014.917.7 InternVL3.58B27.831.331.3 MiniCPM-V 4.58B31.932.346.1 Mage-VL4B50.745.457.4 Molmo24B33.339.858.7 Video-XL-28B38.930.046.2 VideoChat34B56.154.667.0 VideoLLaMA 37B39.829.836.9 Qwen3-VL4B46.448.258.7 Qwen3-VL8B48.346.859.4 Qwen3.54B48.751.655.0 Qwen3.59B52.054.057.2 Ours TLive-Omni4B57.058.269.2 TLive-Omni9B56.3 55.464.1 Table 7: Temporal grounding results on TimeLens-Bench, reported as mIoU on Charades-TL, ActivityNet-TL, and QVHighlights-TL. A dash denotes an undisclosed parameter count. The Best re- sults among open-source models are marked in bold, while the second-best results are inunderlined. ModelParams AVUT WorldSense VideoHolmes DailyOmni OmniVideoBench FutureOmni Closed-source models Gemini 2.5 Flash-65.450.9–55.6 Gemini 3.1 Pro-85.665.5–82.7– Qwen3.5-Omni Flash-81.457.9–81.8– Open-source Omni models video-SALMONN 2+3B66.248.342.267.7– video-SALMONN 2+7B69.550.946.971.8– OmniVinci9B–48.2–66.536.752.8 Nemotron 3 Nano Omni 30B-A3B–55.2–74.5– MiniCPM-o 4.59B78.6 55.764.380.241.156.1 Qwen2.5-Omni7B–45.4–62.436.548.9 Qwen3-Omni30B-A3B74.254.050.471.943.853.4 Ours TLive-Omni4B78.654.057.578.641.657.2 TLive-Omni9B80.056.059.380.543.258.5 Table 8: General Omni benchmark results on AVUT, WorldSense, VideoHolmes, DailyOmni, Om- niVideoBench, and FutureOmni. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold, while the second-best results are in underlined. 5.3 Qualitative Analysis We present qualitative examples of TLive-Omni in both domain-specific and general settings. Figure 3 shows live-commerce examples, while Figure 4 presents general-capability examples. Together, these cases illustrate TLive-Omni’s multimodal understanding across specialized and open-domain scenarios. 14 原神1.jpg 原神2.jpg 原神3.jpg 原神4.jpg 原神5.jpg原神1.jpg 原神3.jpg 原神4.jpg 原神5.jpg (1) Live-CommerceVideo QA (3) Product Visual Grounding (2) Te m p o ralGrounding (4) OCR Prompt CN: 定位"美甲贴纸防水免封层成品甲油膜" ，以JSON格式报告bbox坐标。 EN: Locate "waterproof, no-seal-layer nail art stickers—ready-to-use nail polish film", report the bbox coordinates in JSON format. SourceTLive-Omni Prompt CN:当主播强调这款白色帆布鞋具有“隐 形增高4公分”的效果，主播是如何展示这 款帆布鞋的鞋底的？ EN: When the livestreamer emphasized that these white canvas shoes offer a “4-centimeter invisible height boost,” how did she showcase the shoe’s sole? ModelOutput CN: 视频中，主播手持一双白色的低帮帆布鞋进行展示。当提到"隐形增高4公分"时，她特意将 鞋子翻转过来，清晰地展示了其厚实的黑色橡胶鞋底。通过这种视觉呈现，配合她的讲解，直观 地说明了鞋子通过加厚鞋底实现增高效果的设计特点。 EN: In the video, the host holds up a pair of white low-top canvas shoes for demonstration. When mentioning the "invisible 4 cm height increase," she specifically turns the shoe over to clearly display its thick black rubber sole. Through this visual presentation combined with her narration, it intuitively illustrates the design feature of achieving height increase through a thickened sole. Audio cue: This insole provides a hidden 4-centimeter height boost, and as you can see, it even features a black edge on the lowest part. Prompt CN:精确定位"原神FES西格雯徽章"出现的时间段。 EN: Precisely localize the time intervals when the "Genshin Impact FES Siggwen Badge" appears. GroundTr ut h TLive-Omni 00:00 00:10 00:00 00:10 00:31 00:56 01:5902:03 01:5802:03 GroundTr ut h Prompt CN:识别图片中的所有文本块，提取文本、边界框 [x1, y1, x2, y2]，并分类。标签：产品信息、 价格信息、促销活动、品牌标识、信任背书、服务承诺、交互引导、字幕、其他。仅输出 JSON： ["bbox_2d" : [x1, y1, x2, y2], "text_content" : "text", "label" : "label"]. EN: Identify all text blocks in the image. Extract each block’s text and bounding box [x1, y1, x2, y2], and assign one label: Product, Price, Promotion, Brand, Trust, Service, Interaction, Subtitle, or Other. Return JSON only:... SourceTLive-Omni GroundTr ut h (5) DenseVideoCaption (6) Multi-dimensional Shot Understanding ModelOutput CN:"景别" : ["中景","特写"],"镜头视⻆" : ["平视"],"内容分类" : "真人演示","画面布局": "单 画面布局" EN: "Shot Size": ["Medium Shot","Close-Up"],"Camera": ["Eye Level"],"Content": "Live Action Demonstration","Layout": "Single-Frame Composition" ModelOutput (CN) "00:00 - 00:19": "一名女性模特手持一张写有尺码信息的白色 板子，站在画面中央，另一名女性模特从左侧进入画面，两人 并排站立。口播中提到尺码信息，建议根据身高体重选择合适 尺码，并引导用户点击领券、关注、点赞、收藏，同时提到裙 子价格为1699元。" “00:19 - 00:45”: “手持尺码板的女性模特离开画面，左侧女性 模特开始展示服装，她戴着帽子，身穿黑色上衣、白色开衫和 黑白条纹半身裙，并做出各种姿势和手势。口播中提到开衫价 格为799元，裙子内搭价格为1699元，裙子为特别福利价，申 请回来可享五折，支持七天无理由退换，建议先试穿感受。" "00:45 - 01:58": "女性模特在画面中央来回走动，展示全身服 装，包括正面、侧面和背面，并不断变换姿势。口播中提到产 品为三件套，价格为1699元，支持七天无理由退换，搭配裙子 可提升整体造型感，裙子为拼接设计，有层次感，且为限量款， 建议搭配开衫购买。" "01:58 - 02:05": "女性模特脱下白色开衫，露出里面的黑色无 袖上衣，并继续展示服装。口播提到二号链接可直接购买，同 时介绍工字型背心，强调其上身效果。" 00:00 -00:19 00:19 -00:45 01:58 -02:05 00:45 -01:58 00:30 00:56 Figure 3: Qualitative live-commerce examples of TLive-Omni. (5) Omni-ModalQA (3) Visual Grounding (2) Te m p o ralGrounding (4) OCR Prompt Please provide the bounding box coordinate of the region this sentence describes: man in white shorts on line, report the bbox coordinates in JSON format. Source TLive-Omni Prompt What short phrase does Goofy speak during the brief instance he appears hanging upside down from the top right corner of the screen against a blue background? ModelOutput He says the phrase, "Some lemon." Audio cue: [speaker1] Some honey. [speaker2] Some lemon. Prompt Identify the moment the green-haired woman pulls a large watering can from a bag. GroundTr ut h TLive-Omni 01:08 01:10 GroundTr ut h Prompt Spotting all the text in the image with line-level, and output in JSON format as ['bbox_2d': [x1, y1, x2, y2], 'text_content': 'text', ...]. SourceTLive-Omni GroundTr ut h (1) DenseVideoCaption (6) Multi-dimensional Shot Understanding Prompt CN:"景别": ["近景"], "镜头视⻆": ["平视"], "内容分类": "真人演示", "画面布局": "单画面布局" EN: "Shot Size": ["Close-Up"],"Camera": ["Eye Level"],"Content": "Live Action Demonstration", "Layout": "Single-Frame Composition" ModelOutput (CN) "00:00 - 00:10": "特写镜头显示一个穿着蓝色雨靴的孩子站在 户外游乐设施上，旁边有一只光脚踩在湿滑的表面上；两名儿 童在幼儿园走廊里骑着平衡⻋玩耍，其中一名孩子戴着红色帽 子，另一名戴着蓝色帽子。口播提到将孩子托付给他人可能是 一个充满挑战的过程；一个孩子穿着运动鞋，在草地上踩踏由 塑料桶和绳子组成的障碍物，进行平衡训练。家⻓可以通过国 家注册机构查询幼儿园的资质评级。" "00:10 - 00:20": "一位金发女士坐在室内，面对镜头讲话并伴 有手势，背景是白色窗户和植物。她指出部分服务的质量评估 频率不足，存在评估不及时的情况。" "00:20 - 00:26": "特写镜头展示一只手将一个玩具火⻋头放在 木质轨道上，旁边还有另一个玩具火⻋头和一辆小汽⻋。口播 提示用户可以询问提供者相关问题。" 00:00 -00:10 00:20 -00:26 01:08 01:10 00:10 -00:20 Figure 4: Qualitative general-capability examples of TLive-Omni. 15 6 Conclusion and Limitations We present TLive-Omni, a scenario-oriented omni-modal understanding model for e-commerce live streaming. It maps image, video, audio, and text inputs into a unified representation space, uses Per-vGrid to organize timestamped audio–video evidence, and combines a three-stage supervised training recipe with Faithful-RFT over an atomic live-commerce capability taxonomy. The live- commerce evaluation shows strong performance across speech transcription and speaker-attributed ASR, audio description and question answering, product visual grounding, text localization/recog- nition/classification, temporal grounding, dense video captioning, video question answering, and shot understanding. The general benchmark results further show that TLive-Omni retains broad multimodal capabilities beyond the target domain, with competitive results across image, video and omni-modal evaluations, and improvements over the corresponding Qwen3.5 4B and 9B backbones on multiple benchmarks. Overall, these results suggest that aligning model design, data construction, training objectives, and evaluation protocols with the target deployment scenario is important for building a strong omni-modal model for e-commerce live streaming. Despite these results, TLive-Omni remains focused on understanding rather than generation or full-duplex real-time interaction. Future work will extend evaluation coverage on broader public benchmarks and further improve robustness for longer, noisier, and more diverse live-stream scenarios. Another direction is to strengthen the calibration of temporal evidence under incomplete or ambiguous multimodal inputs, which are common in practical live-stream settings. Contributors Project Lead: Yibo Hu. Contributors: Yu Qian, Mao Gu, Yingfan Tao, Yuhao Chen, Yongdong Luo, Zhuoqun Liu, Yibo Hu, Meiguang Jin, Junfeng Ma. Contact: huyibo871079699@gmail.com 16 Appendix A Related Work A.1 Omni-Modal Large Models Recent omni-modal models have shown that image, video, audio and text can be integrated into a shared language-model interface. Gemini (Gemini Team Google, 2023) and GPT-4 (OpenAI et al., 2023) demonstrate the effectiveness of large-scale multimodal systems, while open omni-modal models such as Qwen2.5-Omni (Xu et al., 2025a), Qwen3-Omni (Xu et al., 2025b), Qwen3.5- Omni (Qwen Team, 2026b), Baichuan-Omni-1.5 (Li et al., 2025b), OmniVinci (Ye et al., 2025), MiniCPM-o 4.5 (Cui et al., 2026), and Nemotron 3 Nano Omni (Deshmukh et al., 2026) make this direction increasingly accessible. These models provide the architectural foundation for unified multimodal interaction, but they are primarily organized around open-domain capabilities rather than the long-form, product-centric, and temporally grounded demands of e-commerce live streaming. A.2 E-commerce and Live-stream Multimodal Understanding E-commerce multimodal research has developed along several complementary paths. The MOON series (Zhang et al., 2025b; Nie et al., 2025; Wu et al., 2026) focuses on product representation learning from multimodal product content, while E-VAds (Liu et al., 2026) introduces a benchmark for evaluating commercial-intent reasoning in e-commerce short videos. Valley3 (Chen et al., 2026b) extends e-commerce modeling toward an omni foundation model, and LiViBench (Wang et al., 2026) highlights the evaluation challenges of interactive live-stream videos. These works motivate our setting, but TLive-Omni targets a different emphasis: a live-commerce understanding model whose architecture, data construction, and capability taxonomy are organized around joint audio, video, image, text and product-grounded evidence. A.3 Reinforcement Learning for Large Language Models GRPO (Shao et al., 2024) and related verifiable-reward methods have made reinforcement learning (RL) post-training a practical strategy for improving reasoning-oriented models. For vision-language models, recent RL studies directly optimize perceptual correctness in visual understanding, hal- lucination mitigation, and temporal grounding (Yu et al., 2025a; Wang et al., 2025c; Yoon et al., 2026; Sharif et al., 2026; Chen et al., 2025; Wang et al., 2025b). GRPO has also been applied to speech recognition (Shivakumar et al., 2025). These directions motivate a broader view of multi- modal post-training: rewards should evaluate whether a response is supported by modality-specific evidence, rather than reasoning alone. Faithful-RFT follows this perception-centered view: it uses task-conditioned reward routing over image, video, and audio streams to score final-answer quality, without rewarding long chain-of-thought or treating reasoning traces as the objective. B Evaluation Metrics We use standard accuracy, AP, F1, and CER definitions unless otherwise noted. For ASR, both the ground-truth transcript and the model-generated transcript are first normalized by removing speaker markers, bracketed tags, punctuation, spaces, and modal particles, and by converting Chinese text to simplified Chinese. CER is then computed as(S + D + I)/N, whereS,D, andIare the numbers of character substitutions, deletions, and insertions, andNis the number of characters in the ground-truth transcript. cpWER. Speaker-attributed ASR is evaluated using concatenated minimum-permutation WER, following the meeting-transcription convention (von Neumann et al., 2023). WER is the word-level error rate, computed from word substitutions, deletions, and insertions relative to the ground-truth transcript. All transcript segments assigned to the same speaker are first concatenated separately for the ground-truth and model predictions. Predicted speakers are then matched to ground-truth speakers using the assignment that minimizes the total word error. If the two sides contain different numbers of speakers, empty streams are added to the smaller side before matching. cpWER is the WER after this optimal speaker pairing. 17 OCR and grounding. Product visual grounding uses AP under an IoU threshold of 0.5 with one-to-one matching between predicted and ground-truth boxes. OCR localization uses an IoU threshold of 0.5 to match predicted and ground-truth text boxes one to one, and computes F1 from the matched boxes. OCR recognition is evaluated on IoU-matched text boxes: predicted and ground-truth boxes are matched one to one usingIoU > 0.5, and the lower-is-better normalized edit distance d edit / max(|p|,|g|)is computed for each matched text pair after normalizing whitespace, whered edit is the edit distance between the predicted text p and the ground-truth text g. Temporal and video metrics.For temporal grounding, mIoU averages interval IoU over samples, withIoU i =|P i ∩ G i |/|P i ∪ G i |for predicted intervalP i and ground-truth intervalG i , where|·| denotes interval length. For multi-interval examples, we first match individual predicted intervals to ground-truth intervals, compute IoU for each matched pair, and then average these IoUs. Audio description and dense video caption are evaluated through a caption-based question-answering protocol. For each audio or video sample, we combine model-assisted question generation and verification with human review to construct multiple-choice questions grounded in modality-specific reference annotations. Audio questions focus on product attributes, prices and promotions, and purchase or interaction instructions, while video questions cover visual and spoken content as well as bidirectional temporal grounding. Together, they probe entities and attributes, actions, scenes, events, and temporal relations. Each question contains one annotation-supported answer and plausible distractors derived from confusable or unsupported content, yielding three to five options including “cannot determine.” At test time, the evaluated model generates a description or caption from the original audio or video. The generated text and preconstructed questions are then passed to a separate evaluator LLM, which answers solely from the generated text without access to the original input. The evaluator’s answers are compared with the ground-truth answers, and accuracy is the fraction that are correct. Hallucination rate is the fraction of incorrect answers among valid questions for which the evaluator LLM selects a concrete answer rather than “cannot determine.” For shot understanding, the model predicts four structured tags for each clip: layout, shot size, camera angle, and content category. We report accuracy separately for each dimension. A single-choice tag must exactly match the ground-truth tag, while for multi-choice tags, a predicted subset of the ground-truth set receives partial credit of 0.5, and wrong or extra tags receive 0. C Additional In-Context ASR Results In-Context ASR evaluates whether a model can use domain-specific keyword prompts to improve transcription of product names, brand names, and other domain terms. Unlike standard ASR, which transcribes audio without textual hints, this setting provides a candidate keyword list before tran- scription. The keyword-list size ranges from 0 to 1000. The zero-keyword setting serves as the no-context baseline, while larger lists test whether additional context improves keyword recognition or introduces interference. We report keyword recall and CER: the former measures whether target keywords are recovered in the transcription, while the latter measures the overall character error rate, capturing whether keyword prompting affects the full transcript beyond the target terms. We compare TLive-Omni with Qwen3-ASR-Flash (Shi et al., 2026; Qwen Team, 2025) and Qwen3-Omni (Xu et al., 2025b). As shown in Table 9, keyword prompting substantially improves recall for both TLive- Omni variants while reducing their CER. TLive-Omni-9B achieves the lowest CER at every nonzero keyword-list size and the highest recall for lists containing 200–1000 keywords. D Training and Faithful-RFT Details D.1 Three-Stage SFT Hyperparameters The three SFT stages introduced in Section 3.2 (audio-language alignment, audio strengthening, and full multimodal SFT) all use the AdamW optimizer withβ 1 = 0.9,β 2 = 0.95, a cosine learning-rate schedule, a weight decay of 0.1, gradient clipping at a maximum norm of 1, ZeRO-3 sharding via DeepSpeed (Rajbhandari et al., 2020; Rasley et al., 2020), and gradient checkpointing. Each stage is trained for a single epoch over its stage-specific data mixture. The global batch size is 1,024 for Stage 1, 2,048 for Stage 2, and 1,024 for Stage 3. The learning rate is1× 10 −4 ,1× 10 −5 , and4× 10 −6 for the three stages, respectively, and the warmup ratio is 0.01, 0.01, and 0.05. 18 KeywordQwen3-ASR-FlashQwen3-OmniTLive-Omni-4BTLive-Omni-9B CountRecall↑CER↓Recall↑CER↓Recall↑CER↓Recall↑CER↓ 051.886.3353.446.5949.126.6948.626.54 5075.695.4085.447.6479.065.4181.005.18 10076.946.5981.567.3477.755.45 80.315.19 20068.005.7378.698.1376.565.5679.505.30 30067.125.8976.947.6576.065.6178.315.36 50060.566.0774.387.4974.94 5.7177.315.42 100056.8110.6870.567.5472.445.7674.315.53 Table 9: In-Context ASR results under different keyword-list sizes. Recall denotes keyword recall, and CER denotes character error rate. For each keyword-list size, best values are shown in bold and second-best values are underlined. D.2 Faithful-RFT Hyperparameters For Faithful-RFT, the number of candidate responses per prompt is set toG = 8. Group-relative rewards are normalized with a smoothing constant ofε s = 10 −4 . The clipped objective usesε l = 0.2 andε h = 0.28for the lower and upper policy-ratio bounds, respectively, together with a KL coefficient of β = 0.1. E Prompts for General Benchmark Evaluation This section reports the exact prompts used to evaluate TLive-Omni on the general-purpose bench- marks introduced in Section 5.2. Prompts are grouped by input modality. E.1 Image Benchmarks Benchmark(s)Prompt Template MMMU, MMBench, MMStar, AI2D question Options: A. option_a / B. option_b / C. option_c / D. option_d Think step by step before answering. The last line of your response should be of the following format: ‘Answer: $LETTER’ (without quotes) where LETTER is one of the options. VLMsAreBlind, RealWorldQA, ERQA, EmbSpatialBench Hint: hint Question: question Options: A. option_a / B. option_b / ... Please select the correct answer from the options above. MathVistahint question Think step by step before answering. The last line of your response should be of the following format: ‘Answer: $ANSWER’ (without quotes) where $ANSWER is your final answer. CharXiv, SimpleVQA question Think step by step before answering. The last line of your response should be of the following format: ‘Answer: $ANSWER’ (without quotes) where $ANSWER is your final answer. 19 Benchmark(s)Prompt Template DynaMath## Question question ## Answer Instruction Please provide an answer to the question outlined above. Your response should adhere to the following JSON format, which includes two keys: ‘solution’ and ‘short answer’. The ‘solution’ key can contain detailed steps needed to solve the question, and the ‘short answer’ key should provide a concise response. Provide the corresponding choice option in the ‘short answer’ key, such as ‘A’, ‘B’, ‘C’, or ‘D’. Example of expected JSON response format: “solution”: “[Detailed step-by-step explanation]”, “short answer”: “[Concise Answer]” HallusionBench question Think step by step before answering. The last line of your response should be of the following format: ‘Answer: Yes/No’ (without quotes). OCRBenchWhat is written in the image? C-OCR Please output only the text content from the image without any additional descriptions or formatting. RefCOCOPlease provide the bounding box coordinate of the region this sentence describes: <ref>sentence</ref> E.2 Video Benchmarks Benchmark(s)Prompt Template MVBench, Video-MME, MLVU, LongVideoBench, LVBench, MMVU question A. option_a / B. option_b / ... Answer with the option’s letter from the given choices directly. VideoMMMU (multiple-choice) question A. option_a / B. option_b / ... Think step by step before answering. End your response with “Answer: X”, where X is the option letter. VideoMMMU (open-ended) question Think step by step before answering. Add “Answer: Your final answer” at the end of your reply. Charades-TL, ActivityNet-TL, QVHighlights-TL Query: “question” Return only one timestamp range in m:s-m:s format. E.3 Omni-Modal Benchmarks Benchmark(s)Prompt Template VideoHolmes, WorldSense, DailyOmni, OmniVideoBench, AVUT, FutureOmni question A. option_a / B. option_b / C. option_c / D. option_d Answer with the option’s letter from the given choices directly. 20 References Xiang An, Yin Xie, Feilong Tang, Yunyao Yan, Huajie Tan, et al. Llava-onevision-2: Towards next-generation perceptual intelligence, 2026. URL https://arxiv.org/abs/2605.25979. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, et al. Qwen3-vl technical report, 2025. URL https://arxiv.org/abs/2511.21631. Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Chao Sun, Rongzhou Zhang, Guanyu Zhou, Lijie Wen, and Xuming Hu. Omnidpo: A preference optimization framework to address omni-modal hallucination, 2025. URL https://arxiv.org/abs/2509.00723. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, et al. Are we on the right way for evaluating large vision-language models?, 2024a. URL https://arxiv.org/abs/2403.20330. Qian Chen, Jinlan Fu, Changsong Li, Min Zhang, See-Kiong Ng, and Xipeng Qiu. FutureOmni: Evaluating future forecasting from omni-modal context for multimodal LLMs, 2026a. URL https://arxiv.org/abs/2601.13836. Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, et al. Longvila: Scaling long- context visual language models for long videos, 2024b. URL https://arxiv.org/abs/2408.10188. Zeyu Chen, Guanghao Zhou, Qixiang Yin, Ziwang Zhao, Huanjin Yao, Pengjiu Xia, Min Yang, Cen Chen, and Minghui Qiu. Valley3: Scaling omni foundation models for e-commerce, 2026b. URL https://arxiv.org/abs/2605.01278. Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-holmes: Can mllm think like holmes for complex video reasoning?, 2025a. URL https://arxiv.org/abs/2505. 21374. Xianfu Cheng, Wei Zhang, Shiwei Zhang, Jian Yang, Xiangyuan Guan, et al. SimpleVQA: Multimodal factuality evaluation for multimodal large language models, 2025b. URL https: //arxiv.org/abs/2502.13059. Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, et al. Molmo2: Open weights and data for vision-language models with video understanding and grounding, 2026. URL https://arxiv.org/abs/2601.10611. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https://arxiv.org/abs/2507.06261. Junbo Cui, Bokai Xu, Chongyi Wang, Tianyu Yu, Weiyue Sun, Yingjing Xu, Tianran Wang, Zhihui He, Wenshuo Ma, Tianchi Cai, Jiancheng Gui, Luoyuan Zhang, Xian Sun, Fuwei Huang, Moye Chen, Zhuo Lin, Hanyu Liu, Qingxin Gui, Qingzhe Han, Yuyang Wen, Huiping Liu, Rongkang Wang, Yaqi Zhang, Hongliang Wei, Chi Chen, You Li, Kechen Fang, Jie Zhou, Yuxuan Li, Guoyang Zeng, Chaojun Xiao, Yankai Lin, Xu Han, Maosong Sun, Zhiyuan Liu, and Yuan Yao. Minicpm-o 4.5: Towards real-time full-duplex omni-modal interaction, 2026. URL https: //arxiv.org/abs/2604.27393. Amala Sanjay Deshmukh, Kateryna Chumachenko, Tuomas Rintamaki, Matthieu Le, Tyler Poon, Danial Mohseni Taheri, Ilia Karmanov, Guilin Liu, Jarno Seppanen, Arushi Goel, et al. Nemotron 3 nano omni: Efficient and open multimodal intelligence, 2026. URL https://arxiv.org/abs/2604. 24954. Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. EmbSpatial-Bench: Bench- marking spatial understanding for embodied tasks with large vision-language models, 2024. URL https://arxiv.org/abs/2406.05756. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, et al. Video-MME: The first- ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis, 2024. URL https://arxiv.org/abs/2405.21075. 21 Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction, 2025. URL https://arxiv.org/abs/2501.01957. Gemini Robotics Team. Gemini Robotics: Bringing AI into the physical world, 2025. URL https: //arxiv.org/abs/2503.20020. Gemini Team Google. Gemini: A family of highly capable multimodal models, 2023. URL https://arxiv.org/abs/2312.11805. Google DeepMind. Gemini 3 Flash model card. Model card, 2025a. URL https://deepmind.google/ models/model-cards/gemini-3-flash/. Google DeepMind. Gemini 3 Pro model card. Model card, 2025b. URL https://deepmind.google/ models/model-cards/gemini-3-pro/. Google DeepMind. Gemini 3.1 Pro model card. Model card, 2026a. URL https://deepmind.google/ models/model-cards/gemini-3-1-pro/. Google DeepMind. Gemini 3.5 Flash model card. Model card, 2026b. URL https://deepmind.google/ models/model-cards/gemini-3-5-flash/. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, et al. HallusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models, 2023. URL https://arxiv.org/abs/2310.14566. Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. WorldSense: Evaluating real-world omnimodal understanding for multimodal LLMs, 2025. URL https://arxiv.org/abs/2502. 04326. Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-MMMU: Evaluating knowledge acquisition from multi-discipline professional videos, 2025. URL https://arxiv.org/abs/2501.13826. Inclusion AI, Biao Gong, Cheng Zou, Chuanyang Zheng, Chunluan Zhou, et al. Ming-omni: A unified multimodal model for perception and generation, 2025. URL https://arxiv.org/abs/2506.09344. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European conference on computer vision, pages 235–251. Springer, 2016. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, pages 611–626, 2023. doi: 10.1145/3600006.3613165. URL https://doi.org/10. 1145/3600006.3613165. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, et al. Llava-onevision: Easy visual task transfer, 2024. URL https://arxiv.org/abs/2408.03326. Caorui Li, Yu Chen, Yiyan Ji, Jin Xu, Zhenyu Cui, et al. OmniVideoBench: Towards audio-visual understanding evaluation for omni MLLMs, 2025a. URL https://arxiv.org/abs/2510.10689. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, et al. MVBench: A comprehensive multi-modal video understanding benchmark, 2023. URL https://arxiv.org/abs/2311.17005. Xinhao Li, Yuhan Zhu, Xiangyu Zeng, Yuhao Dong, Haoning Wu, et al. Videochat3: Fully open video mllm for efficient and generalist video understanding, 2026. URL https://arxiv.org/abs/2607.14935. Yadong Li et al. Baichuan-omni-1.5 technical report, 2025b. URL https://arxiv.org/abs/2501.15368. Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, et al. Kangaroo: A powerful video- language model supporting long-context video input, 2024a. URL https://arxiv.org/abs/2408.15542. 22 Xianjie Liu, Yiman Hu, Liang Wu, Ping Hu, Yixiong Zou, Jian Xu, and Bo Zheng. E-vads: An e-commerce short videos understanding benchmark for mllms, 2026. URL https://arxiv.org/abs/ 2602.08355. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, et al. MMBench: Is your multi-modal model an all-around player?, 2023a. URL https://arxiv.org/abs/2307.06281. Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, et al. OCRBench: On the hidden mystery of OCR in large multimodal models, 2023b. URL https://arxiv.org/abs/2305.07895. Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models, 2024b. URL https://arxiv.org/abs/2412.04468. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, et al. MathVista: Evaluating mathemati- cal reasoning of foundation models in visual contexts, 2023. URL https://arxiv.org/abs/2310.02255. Zhanheng Nie, Chenghan Fu, Daoze Zhang, Junxian Wu, Wanxian Guan, Pengjie Wang, Jian Xu, and Bo Zheng. Moon2.0: Dynamic modality-balanced multimodal representation learning for e-commerce product understanding, 2025. URL https://arxiv.org/abs/2511.12449. OpenAI. Gpt-4o system card, 2024. URL https://arxiv.org/abs/2410.21276. OpenAI, Josh Achiam, et al. Gpt-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774. OpenBMB. MiniCPM-o 2.6: A GPT-4o level MLLM for vision, speech and multimodal live streaming on your phone. Model release page, 2025. URL https://huggingface.co/openbmb/MiniCPM-o-2_6. Minghao Qin, Xiangrui Liu, Zhengyang Liang, Yan Shu, Huaying Yuan, et al. Video-xl-2: Towards very long-video understanding through task-aware kv sparsification, 2025. URL https://arxiv.org/ abs/2506.19225. Qwen Team. Qwen3-ASR-Flash: A speech recognition service built on Qwen3-ASR. Blog post, 2025. URL https://qwen.ai/blog?id=41e4c0f6175f9b004a03a07e42343eaaf48329e7. Qwen Team. Qwen3.5-4B and Qwen3.5-9B. Model cards, 2026a. URL https://huggingface.co/ Qwen/Qwen3.5-4B. Qwen Team. Qwen3.5-omni technical report, 2026b. URL https://arxiv.org/abs/2604.15804. Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind: Failing to translate detailed visual features into words, 2024. URL https://arxiv.org/abs/2407.06581. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: international conference for high performance computing, networking, storage and analysis, pages 1–16. IEEE, 2020. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505–3506, 2020. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. Omar Sharif, Eftekhar Hossain, Nikhil Singh, and Patrick Ng. Disentangling perception and reasoning in multimodal llms via reward design, 2026. URL https://arxiv.org/abs/2601.00215. Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding, 2024.URL https://arxiv.org/abs/2410.17434. 23 Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, et al. Qwen3-ASR technical report, 2026. URL https://arxiv.org/abs/2601.21337. Prashanth Gurunath Shivakumar, Yile Gu, Ankur Gandhe, and Ivan Bulyko. Group relative policy optimization for speech recognition, 2025. URL https://arxiv.org/abs/2509.01939. Aaditya Singh, Adam Fry, Adam Perelman, et al. Openai gpt-5 system card, 2025. URL https: //arxiv.org/abs/2601.03267. Tomáš Sou ˇ cek and Jakub Loko ˇ c. Transnet v2: An effective deep network architecture for fast shot transition detection, 2020. URL https://arxiv.org/abs/2008.04838. StepFun AI. Step-audio-r1.1. Hugging Face model card, 2026. URL https://huggingface.co/stepfun- ai/Step-Audio-R1.1. Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, et al. video-salmonn 2: Caption-enhanced audio-visual large language models, 2025. URL https://arxiv.org/abs/2506. 15220. Wenwen Tong, Hewei Guo, Dongchuan Ran, Jiangnan Chen, Jiefan Lu, et al. Interactiveomni: A unified omni-modal model for audio-visual multi-turn dialogue, 2025. URL https://arxiv.org/abs/ 2510.13747. Tongyi Fun Team, Qian Chen, Luyao Cheng, Chong Deng, Xiangang Li, et al. Fun-audio-chat technical report, 2025. URL https://arxiv.org/abs/2512.20156. Valley Team, ByteDance Group. Valley2.5 technical report, 2025. URL https://raw.githubusercontent. com/bytedance/Valley/refs/heads/main/docs/Valley2_5_Tech_Report.pdf. Technical report. Thilo von Neumann, Christoph Boeddeker, Marc Delcroix, and Reinhold Haeb-Umbach. Meeteval: A toolkit for computation of word error rates for meeting transcription systems. arXiv preprint arXiv:2307.11394, 2023. Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, et al. LVBench: An extreme long video understanding benchmark, 2024a. URL https://arxiv.org/abs/2406.08035. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, et al. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency, 2025a. URL https: //arxiv.org/abs/2508.18265. Xiaodong Wang, Langling Huang, Zhirong Wu, Xu Zhao, Teng Xu, Xuhong Xia, and Peixi Peng. Livibench: An omnimodal benchmark for interactive livestream video understanding, 2026. URL https://arxiv.org/abs/2601.15016. Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, Xiangnan Fang, Zewen He, Zhenbo Luo, Wenxuan Wang, Junqi Lin, Jian Luan, and Qin Jin. Time-r1: Post-training large vision language model for temporal video grounding, 2025b. URL https://arxiv.org/abs/2503.13377. Zhenhailong Wang, Xuehang Guo, Sofia Stoica, Haiyang Xu, Hongru Wang, Hyeonjeong Ha, Xiusi Chen, Yangyi Chen, Ming Yan, Fei Huang, and Heng Ji. Perception-aware policy optimization for multimodal reasoning, 2025c. URL https://arxiv.org/abs/2507.06448. Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, et al. CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs, 2024b. URL https://arxiv.org/abs/2406.18521. Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. LongVideoBench: A benchmark for long-context interleaved video-language understanding, 2024. URL https://arxiv.org/abs/2407.15754. Junxian Wu, Chenghan Fu, Zhanheng Nie, Daoze Zhang, Bowen Wan, Wanxian Guan, Chuan Yu, Jian Xu, and Bo Zheng. Moon3.0: Reasoning-aware multimodal representation learning for e-commerce product understanding, 2026. URL https://arxiv.org/abs/2604.00513. xAI. RealWorldQA. Dataset card, 2024. URL https://huggingface.co/datasets/xai-org/RealworldQA. 24 Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025a. URL https://arxiv.org/abs/2503.20215. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jingren Zhou, and Junyang Lin. Qwen3-omni technical report, 2025b. URL https://arxiv.org/abs/2509.17765. Senqiao Yang, Kaichen Zhang, Zhaoyang Jia, Jinghao Guo, Yifei Shen, et al. Mage-vl: An efficient codec-native streaming multimodal foundation model, 2026. URL https://arxiv.org/abs/2607. 24904. Yudong Yang, Jimin Zhuang, Guangzhi Sun, Changli Tang, Yixuan Li, et al. Audio-centric video understanding benchmark without text shortcut, 2025. URL https://arxiv.org/abs/2503.19951. Zhibo Yang, Jun Tang, Zhaohai Li, Pengfei Wang, Jianqiang Wan, et al. C-OCR: A comprehensive and challenging OCR benchmark for evaluating large multimodal models in literacy, 2024. URL https://arxiv.org/abs/2412.02210. Hanrong Ye, Chao-Han Huck Yang, Arushi Goel, Wei Huang, Ligeng Zhu, Yuanhang Su, Sean Lin, An-Chieh Cheng, Zhen Wan, Jinchuan Tian, Yuming Lou, Dong Yang, Zhijian Liu, Yukang Chen, Ambrish Dantrey, Ehsan Jahangiri, Sreyan Ghosh, Daguang Xu, Ehsan Hosseini-Asl, Danial Mohseni Taheri, Vidya Murali, Sifei Liu, Yao Lu, Oluwatobi Olabiyi, Yu-Chiang Frank Wang, Rafael Valle, Bryan Catanzaro, Andrew Tao, Song Han, Jan Kautz, Hongxu Yin, and Pavlo Molchanov. Omnivinci: Enhancing architecture and data for omni-modal understanding llm, 2025. URL https://arxiv.org/abs/2510.15870. Weijie Yin, Yongjie Ye, Fangxun Shu, Yue Liao, Zijian Kang, et al. Sail-vl2 technical report, 2025. URL https://arxiv.org/abs/2509.14033. Hee Suk Yoon, Eunseop Yoon, Ji Woo Hong, SooHwan Eom, Gwanhyeong Koo, Mark Hasegawa- Johnson, Qi Dai, Chong Luo, and Chang D. Yoo. Pdcr: Perception-decomposed confidence reward for vision-language reasoning, 2026. URL https://arxiv.org/abs/2605.13467. En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Jingyu Wang, and Wenbing Tao. Perception-r1: Pioneering perception policy with reinforcement learning, 2025a. URL https: //arxiv.org/abs/2504.07954. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions, 2016. URL https://arxiv.org/abs/1608.00272. Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, Wenshuo Ma, et al. Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe, 2025b. URL https://arxiv.org/abs/2509. 18154. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, et al. MMMU: A massive multi- discipline multimodal understanding and reasoning benchmark for expert AGI, 2023. URL https://arxiv.org/abs/2311.16502. Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, et al. Mimo-vl technical report, 2025. URL https://arxiv.org/abs/2506.03569. Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025a. URL https://arxiv.org/ abs/2501.13106. Daoze Zhang, Chenghan Fu, Zhanheng Nie, Jianyu Liu, Wanxian Guan, Yuan Gao, Jun Song, Pengjie Wang, Jian Xu, and Bo Zheng. Moon: Generative mllm-based multimodal representation learning for e-commerce product understanding, 2025b. URL https://arxiv.org/abs/2508.11999. 25 Dong Zhang, Gang Wang, Jinlong Xue, Kai Fang, et al. Mimo-audio: Audio language models are few-shot learners, 2025c. URL https://arxiv.org/abs/2512.23808. Jun Zhang, Teng Wang, Yuying Ge, Yixiao Ge, Xinhao Li, Ying Shan, and Limin Wang. TimeLens: Rethinking video temporal grounding with multimodal llms, 2025d. URL https://arxiv.org/abs/ 2512.14698. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, et al. Llava-video: Video instruction tuning with synthetic data, 2025e. URL https://arxiv.org/abs/2410.02713. Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, et al. MMVU: Measuring expert-level multi-discipline video understanding, 2025. URL https://arxiv.org/abs/2501.12380. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, et al. MLVU: Benchmarking multi-task long video understanding, 2024. URL https://arxiv.org/abs/2406.04264. Ziwei Zhou, Rui Wang, Zuxuan Wu, and Yu-Gang Jiang. Daily-Omni: Towards audio-visual reasoning with temporal alignment across modalities, 2025. URL https://arxiv.org/abs/2505.17862. Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models, 2024. URL https://arxiv.org/abs/2411.00836. 26