Paper deep dive
PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation
Yuchen Sun, Qian Yang, Jun Wang, Detai Xin, Guoqiao Yu, Guanglu Wan, Qi Jia
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/10/2026, 2:16:52 AM
Summary
The paper introduces PRISM-Bench, the first audio-centric diagnostic benchmark for Text-to-Audio-Video (T2AV) generation. It addresses the underestimation of audio modality in existing evaluations by factorizing assessment along two axes: audio type (Speech, Music, Sound) and sound-source visibility (On-screen, Off-screen). Built on 900 human-verified samples, it uses an enhanced MLLM-as-a-Judge protocol to evaluate four perceptual dimensions across 35 criteria. Evaluation of models like Seedance 2.0, Kling v3 Omni, and Sora 2 reveals significant gaps between frontier and open-source models, particularly in complex grounding and synchronized audio generation.
Entities (18)
Relation Signals (11)
PRISM-Bench → containsdatasetsize → 900
confidence 95% · Built from a rigorously curated dataset of 900 human-verified samples
PRISM-Bench → evaluates → Text-to-Audio-Video Generation
confidence 95% · PRISM-Bench is the first audio-centric diagnostic benchmark for T2AV generation.
PRISM-Bench → hascriteriacount → 35
confidence 95% · with 35 fine-grained criteria
PRISM-Bench → measures → Prompt Following
confidence 95% · It evaluates generated content across four perceptual dimensions (... and Prompt Following)
PRISM-Bench → measures → Audio-Visual Coherence
confidence 95% · It evaluates generated content across four perceptual dimensions (Audio-Visual Coherence...)
PRISM-Bench → measures → Audio Quality
confidence 95% · It evaluates generated content across four perceptual dimensions (... Audio Quality...)
PRISM-Bench → measures → Audio Expressiveness
confidence 95% · It evaluates generated content across four perceptual dimensions (... Audio Expressiveness...)
PRISM-Bench → stratifiesby → Sound-Source Visibility
confidence 95% · PRISM-Bench factorizes audio evaluation along two orthogonal axes: ... sound-source visibility (On-screen vs. Off-screen)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality. Existing benchmarks either treat audio as an auxiliary component of video quality or assess it in isolation from audiovisual grounding, making it difficult to diagnose where current systems truly succeed or fail in audio generation. We present PRISM-Bench, the first audio-centric diagnostic benchmark for T2AV generation. Built from a rigorously curated dataset of 900 human-verified samples, PRISM-Bench factorizes audio evaluation along two orthogonal axes: audio type (Speech, Music, and Sound) and sound-source visibility (On-screen vs. Off-screen). It evaluates generated content across four perceptual dimensions (Audio-Visual Coherence, Audio Quality, Audio Expressiveness, and Prompt Following) with 35 fine-grained criteria. To ensure reliable assessment, we adopt an enhanced MLLM-as-a-Judge protocol based on blind, side-by-side comparison against ground-truth references, demonstrating strong alignment (over 70% mean agreement) with human raters. Our evaluation of recent T2AV systems highlights a significant performance gap between frontier and open-source models. Furthermore, we demonstrate that current generation paradigms overfit to perceptual fidelity while struggling with complex grounding and control tasks, particularly in generating music and synchronized On-screen audio.
Tags
Links
- Source: https://arxiv.org/abs/2609.04867v2
- Canonical: https://arxiv.org/abs/2609.04867v2
Trouble viewing inline? Open PDF directly →
Full Text
98,297 characters extracted from source content.
Expand or collapse full text
PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation Yuchen Sun ∗ sunyuchen@pjlab.org.cn Shanghai Artificial Intelligence Laboratory Shanghai, China Qian Yang ∗ yangqian77@meituan.com Meituan Beijing, China Jun Wang wangjun85@meituan.com Meituan Beijing, China Detai Xin xindetai@meituan.com Meituan Beijing, China Guoqiao Yu † yuguoqiao@meituan.com Meituan Beijing, China Guanglu Wan † wanguanglu@meituan.com Meituan Beijing, China Qi Jia jiaqi@pjlab.org.cn Shanghai Artificial Intelligence Laboratory Shanghai, China Abstract Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality. Existing bench- marks either treat audio as an auxiliary component of video quality or assess it in isolation from audiovisual grounding, making it dif- ficult to diagnose where current systems truly succeed or fail in audio generation. We present PRISM-Bench, the first audio-centric diagnostic benchmark for T2AV generation. Built from a rigorously curated dataset of 900 human-verified samples, PRISM-Bench fac- torizes audio evaluation along two orthogonal axes: audio type (Speech, Music, and Sound) and sound-source visibility (On-screen vs. Off-screen). It evaluates generated content across four percep- tual dimensions (Audio-Visual Coherence, Audio Quality, Audio Expressiveness, and Prompt Following) with 35 fine-grained criteria. To ensure reliable assessment, we adopt an enhanced MLLM-as- a-Judge protocol based on blind, side-by-side comparison against ground-truth references, demonstrating strong alignment (over 70% mean agreement) with human raters. Our evaluation of recent T2AV systems highlights a significant performance gap between frontier and open-source models. Furthermore, we demonstrate that current generation paradigms overfit to perceptual fidelity while struggling with complex grounding and control tasks, particularly in generating music and synchronized On-screen audio. 1 CCS Concepts • General and reference→Evaluation;• Information systems →Multimedia information systems;• Computing method- ologies→ Computer vision. 1 We provide a public leaderboard for reporting and comparing benchmark results at https://huggingface.co/spaces/prismbench/prismbench-leaderboard, while the under- lying audiovisual data remain restricted due to licensing and redistribution constraints. This work is licensed under a Creative Commons Attribution-NonCommercial- NoDerivatives 4.0 International License. M ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2213-4/2026/11 https://doi.org/10.1145/3767308.3836100 Keywords Audio-Centric Benchmark, Text-to-Audio-Video Generation, Multi- modal Evaluation, MLLM-as-a-Judge ACM Reference Format: Yuchen Sun, Qian Yang, Jun Wang, Detai Xin, Guoqiao Yu, Guanglu Wan, and Qi Jia. 2026. PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation. In Proceedings of the 34th ACM Inter- national Conference on Multimedia (M ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 19 pages. https://doi.org/10.1145/ 3767308.3836100 1 Introduction Text-to-audio-video (T2AV) generation has recently emerged as one of the most active frontiers in multimodal generative modeling. Recent systems now include a growing range of both proprietary and publicly released generators that produce video together with native or jointly modeled audio [2,9,13,18,23,30,35]. Despite this rapid progress, evaluation has lagged noticeably behind. Recent benchmarks have expanded coverage to cross-modal alignment, physical plausibility, multiple generation tasks, and broad audio- video quality [3,14,25,40]. However, their reported taxonomies are organized primarily around tasks, content scenarios, general evaluation dimensions, or physical failure modes, rather than a crossed diagnosis of both audio content type and sound-source visibility. As a result, they provide limited insight into how the same system’s audio behavior changes jointly across speech, music, and sound and across On-screen and Off-screen sources. This limitation is not a lack of fine-grained evaluation per se: several recent benchmarks define detailed dimensions or failure taxonomies [3,11,14,25,40,45]. The missing formulation is an explicit joint stratification over two factors that critically shape generation difficulty: the type of audio content and the visibility of its source. Evaluations spanning speech, music, and environmen- tal sound have reported markedly different performance profiles across these domains [33], while audio tied to visible events poses a different grounding problem from audio whose source remains ∗ Yuchen Sun and Qian Yang contributed equally to this research. This work was done while Yuchen Sun was an intern at Meituan. † Corresponding authors: Guoqiao Yu and Guanglu Wan. arXiv:2609.04867v2 [cs.M] 9 Sep 2026 M ’26, November 10–14, 2026, Rio de Janeiro, BrazilYuchen Sun et al. Figure 1: Grouped radar comparison across the On-screen, Off-screen, and Mixed subsets for the evaluated T2AV systems. Each spoke reports a calibrated mean score for one evaluation-dimension–audio-type pair; higher is better. off screen [36,37]. Without a crossed organization over these fac- tors, aggregate results do not directly reveal where current T2AV systems succeed or fail under specific audio conditions. To address these limitations, we introduce PRISM-Bench (Fig- ure 2), the first audio-centric diagnostic benchmark for T2AV gen- eration. Rather than relying on holistic audio assessment, PRISM- Bench explicitly models audio variation along two orthogonal axes: audio type, comprising Speech, Music, and Sound, and sound-source visibility, distinguishing On-screen from Off-screen audio. PRISM- Bench is built on a rigorously curated dataset of 900 human-verified samples, and evaluates generated content along four perceptual dimensions: Audio-Visual Coherence, Audio Quality, Audio Ex- pressiveness, and Prompt Following. To make these dimensions operational for fine-grained diagnosis, we further instantiate them with 35 criteria that specify what the evaluator should examine under different audio types and visibility conditions. Recent work has explored MLLMs as evaluators for image safety, audio-language responses, and any-to-any multimodal tasks [32, 39,42]. We design an enhanced side-by-side protocol anchored by ground-truth references. In each comparison, the judge is presented with a generated sample and a ground-truth sample in blinded and randomized order, and evaluates each candidate independently rather than through direct preference alone. We validate the frame- work through stability analysis and human-alignment experiments, showing that judge-side variation remains substantially smaller than the performance differences across candidate models, while the scores achieve over 70% mean agreement with human raters. These results support the use of the protocol for large-scale bench- mark evaluation. Using PRISM-Bench, we evaluate Seedance 2.0, Kling v3 Omni, Veo 3.0/3.1, Sora 2, LTX-2, Ovi, and MOVA, and find clear differences across models, audio types, visibility conditions, and evaluation dimensions (Figure 1). Seedance 2.0 ranks first across the On-screen, Off-screen, and Mixed subsets, yet the factorized scores expose resid- ual weaknesses that its aggregate lead would hide: On-screen AV Coherence remains lower than its fidelity- and control-oriented di- mensions, and Off-screen Sound Prompt Following trails its Speech and Music results. Frontier proprietary systems also retain a clear advantage over open-source models. These findings indicate that improved acoustic realism and prompt adherence do not by them- selves resolve fine-grained audiovisual grounding and sound-event control. Our contributions are threefold: •We introduce PRISM-Bench, audio-centric diagnostic benchmark for T2AV generation. By jointly stratifying audio type and sound- source visibility, PRISM-Bench enables structured analysis under diverse grounding conditions. •We construct a rigorously curated dataset of 900 human-verified samples and pair it with a robust MLLM-as-a-Judge evaluation framework covering four perceptual dimensions and 35 fine- grained criteria. •We benchmark recent systems and provide a factorized empirical analysis of their capability profiles. Seedance 2.0 sets a new over- all lead, while the remaining gaps in visible-source coherence and fine-grained Sound control demonstrate that aggregate progress does not eliminate grounding-sensitive failure modes. We publicly release the PRISM-Bench leaderboard and evaluation code at https://huggingface.co/spaces/prismbench/prismbench-leaderboard; the underlying audiovisual data remain restricted because redistri- bution rights vary across sources. 2 Related Work Text-to-Audio-Video Generation Text-to-audio-video (T2AV) generation has progressed from loosely coupled pipelines to in- creasingly unified multimodal generators. Early joint approaches such as M-Diffusion modeled audio and video within a shared diffusion framework [34]. More recent ap- proaches scale joint generation with tighter cross-modal coupling, improving temporal alignment and semantic consistency between visual events and audio tracks. Recent systems such as Sora 2 [30], PRISM-BenchMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil On screen Off screen Mixed Speech Music Sound Speech Music Sound Speech Music Sound Prism Bench (a) Factorized benchmark taxonomy 0250500750100012501500 Caption Length 0 0.5 1 1.5 2 2.5 Density (10 ³) On-screen Off-screen Mixed (b) Caption-length distribution(c) Eval Criteria Word Cloud Figure 2: Overall visualization of PRISM-Bench. Veo [9], Movie Gen [31], Kling v3 Omni [18], and SkyReels V4 [4] in- creasingly target generation of temporally aligned video and audio. Seedance 2.0 adopts a unified multimodal audio-video generation architecture and adds dual-channel audio output for speech, music, and sound effects [2], illustrating how rapidly native audio capa- bilities are advancing. At the same time, LTX-2 [13], Ovi [23], and MOVA [35] release model artifacts for reproducible study, making systematic comparison more feasible. Despite this rapid progress, the audio side of T2AV generation remains notably less mature than the visual side. Current models often exhibit uneven behavior across different audio categories, strong ambient sound genera- tion but weak speech synchronization, or plausible music quality but poor event-level grounding. Conventional automatic measures instead tend to target video distribution or motion [7,20,24,38], audio distribution [16], or learned cross-modal similarity [6,12]; these scores do not by themselves provide a crossed diagnosis by audio type and source visibility. Audio-Visual Evaluation Benchmarks Audio-visual evaluation predates generative T2AV benchmarks and has traditionally focused on perceptual interactions between audio and video streams [27]. However, evaluation for modern generative T2AV remains frag- mented. Unimodal benchmarks such as VBench [15] and Eval- Crafter [21] assess general video quality, whereas AudioCaps [17] and related audio benchmarks focus on audio semantics, captioning, or fidelity. Although informative, these benchmarks do not directly evaluate whether generated audio is temporally coordinated with video, grounded in visible or implied events, and consistent with multimodal prompts. Recent work has moved toward joint audiovisual evaluation. TAVGBench [25] introduces a large-scale text-to-audible-video dataset and AVHScore for audiovisual alignment, but does not jointly diagnose audio types and sound-source visibility conditions. T2AV-Compass [3] combines signal-level metrics with MLLM-based judging over taxonomy-driven prompts, while VABench [14] eval- uates multiple generation settings using comprehensive rubrics. PhyAVBench [40] focuses on physically grounded audiovisual events, MTAVG-Bench [45] targets multi-speaker conversational scenarios, and ALIVE [11] evaluates 264 diverse prompts through blinded, side-by-side human comparisons across six categories. Comple- mentary studies of LLM-as-a-Judge further reveal limitations of automatic evaluation, including presentation-order bias in pairwise judgments and the loss of uncertainty when human evaluation distributions are reduced to single-point scores [5, 44]. Despite these advances, existing T2AV benchmarks do not pro- vide a dedicated crossed diagnosis over sound type and source visibility. Consequently, they offer limited insight into how mod- els behave across speech, music, and sound under On-screen and Off-screen conditions. 3 PRISM-Bench 3.1 Overview PRISM-Bench is an audio-centric diagnostic benchmark for text- to-audio-video (T2AV) generation. It is designed to evaluate audio not as an auxiliary byproduct of video generation, but as a struc- tured modality whose behavior depends on both audio type and grounding condition. To this end, PRISM-Bench consists of two tightly coupled components: a curated benchmark dataset and a visibility-aware evaluation framework. As illustrated in Figure 2(a), at the core of PRISM-Bench is a factorized benchmark taxonomy organized along two axes: audio type and sound-source visibility. Their combination defines three complementary evaluation settings, namely On-screen, Off-screen, and Mixed, which together support more fine-grained diagnosis of model behavior than holistic audio-video evaluation. Built on top of this benchmark taxonomy, the evaluation framework assesses generated samples against ground-truth references along four per- ceptual dimensions: Audio-Visual Coherence, Audio Quality, Audio Expressiveness, and Prompt Following. 3.2 Dataset Curation Pipeline As illustrated in Figure 3, we construct the dataset of PRISM-Bench through a benchmark-oriented pipeline that integrates scenario- aware sampling, multi-stage annotation, and human verification. The goal is to support the two core diagnostic axes of PRISM-Bench: audio type (Speech, Music, and Sound) and sound-source visibility (On-screen versus Off-screen). Rather than collecting clips only for M ’26, November 10–14, 2026, Rio de Janeiro, BrazilYuchen Sun et al. Gemini 2.5 pro Qwen 3 Omni Video-Audio Segments Low-frequency wind noise in background, a slight rustling sound of clothing can be heard one female voice:"During...to that" a male voice:"And your mood?" sec female voice:"What about me?" a male voice:”what do you want?” Two women sit in a ski lift traversing a snowy slope. Their backdrop was sky and snow-capped hillsides， Their posture didn’t change ... female voice:"During ... to that" male voice: "And you know?" female voice:"What about me?"male voice: "What do you?" Audio Caption Visual Caption GPT-5-chat Fusion Human Correction Integrated Caption Two women sit in a ski lift traversing a snowy slope. Accompanied by low- frequency wind noise and a slight rustling sound of clothing, a female voice says:"During ... to that".Then an off-screen male voice asks:"And your mood?". Another female voice says:"What about me?". The male voice says:"what do you want?" Two women sit in anascending ski lift,traversing a snowy slope under blue sky. The woman on the left,wearing a cap and sunglasses, smiles and says: "During...to that" Accompanied by wind noise and clothing rustling, a man off-screen asks:"And your mood?"Then,the woman on the right says:"What about me".The man asks:" what do you want..." Off screen On screen speech sound Pure Audio Tags speech sound Verified Caption & Tags Off screen On screen Multi-Modal Captioning Audio Captioning Figure 3: Data construction and annotation pipeline. generic audio-visual diversity, we explicitly build the dataset to provide broad and balanced coverage of these benchmark-critical conditions. The source pool comprises publicly accessible audiovisual con- tent collected from web sources, together with audiovisual assets commercially licensed or purchased by our institution for internal research and evaluation. Because redistribution rights vary, the clips are retained in a controlled internal environment and are not publicly released. We instead provide a public leaderboard, with the data-access policy documented on the project page. To ensure ecological diversity, PySceneDetect 1 segments source videos into candidate scenes, which we balance across Movie, Dialogue-centric, Documentary, Concert, and Daily-life content while monitoring Speech, Music, Sound, On-screen, and Off-screen coverage. After sampling, all videos are divided into clips of up to 8 sec- onds through the Video-Audio Segments step, which defines the basic evaluation unit of the benchmark. This granularity preserves meaningful audio-visual events while keeping each sample com- pact enough for controlled evaluation. We then apply a multi-stage annotation pipeline that combines multimodal captioning, audio- only calibration, cross-source fusion, and benchmark-level human correction. In the first annotation stage, we apply Multi-Modal Captioning using the stable API model Gemini 2.5 Pro 2 to obtain an initial multimodal description for each clip. Specifically, Gemini produces two complementary captions: a visual caption describing scenes, objects, and actions, and an audio caption describing audible content such as speech, music, and sound events. These captions provide broad multimodal coverage and serve as the basis for subsequent refinement. In the second stage, we perform Audio Captioning by extract- ing the raw audio track from each clip and feeding it to Qwen3- Omni [41]. We use Qwen3-Omni in an audio-only setting, and empirically find that pure-audio input produces more accurate au- dio captions than audio-video input, allowing the model to focus more directly on acoustic evidence. In addition to audio descriptions, Qwen3-Omni also produces auxiliary attributes such as language 1 https://github.com/breakthrough/pyscenedetect 2 https://ai.google.dev/gemini-api/docs/models/gemini-2.5-pro information that are later used for filtering and analysis. This stage is particularly useful for improving audio-centric annotation when multimodal perception alone is insufficient. In the third stage, we conduct GPT-5 Chat Fusion using the API alias gpt-5-chat-latest, which the provider documents as pointing to the dated snapshot gpt-5-chat-latest-2025-08-07 [29]. The model takes as input both the multimodal captions from Gemini and the audio-only descriptions from Qwen3-Omni, and reconciles them into a more consistent clip-level annotation. This stage improves cross-source consistency, reduces hallucinated audio-visual cor- respondences, and refines incomplete or ambiguous descriptions. More importantly, it introduces the structured labels required by PRISM-Bench: the fusion model identifies which audio types are present in each clip among Speech, Music, and Sound, and deter- mines whether the relevant sound source is On-screen or Off-screen. Finally, all 900 benchmark samples undergo Human Correc- tion, where annotators verify benchmark-critical attributes, includ- ing synchronization descriptions, speech content, paralinguistic cues, music and environmental sound events, and the labels of au- dio type and sound-source visibility. This final quality-control step makes the annotations suitable for fine-grained diagnostic evalu- ation. The benchmark contains three visibility-aware sets of 300 audio-video pairs each. The benchmark annotations also exhibit clear regularities. As shown in Figure 2(b), the caption-length distributions differ system- atically across the three benchmark settings, reflecting different descriptive demands under On-screen, Off-screen, and Mixed condi- tions. On-screen samples tend to have longer captions because they require joint description of audible content, visible sources, and their grounding relations, whereas Off-screen samples are generally shorter since they do not require explicit visible-source alignment. Mixed samples lie between these two cases. 3.3 Evaluation Dimensions PRISM-Bench evaluates T2AV outputs along four complementary perceptual dimensions, which together capture grounding, fidelity, expressiveness, and instruction adherence in audio generation. Rather than applying the same criteria uniformly to all samples, the benchmark instantiates each dimension with type-aware criteria PRISM-BenchMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Blind Review Input Judger I AV Coherence Two-Stage MLLM-as-Judge Video Guide Eval Caption: In a dimly lit room, two middle aged men are playing instruments... Speech 9 Muisc 9 1 2 Speech Sound Music Activated Tags Caption Guide Eval Activated Tags PrismBench Ignored Ignored Audio Quality Prompt Following Expressiveness Judger I Speech 8 Muisc 8 1 2 Speech 10 Muisc 8 Activated Tags Stage 1 Stage 2 On screen Off screen Speech 6 Muisc 5 Speech 8 Muisc 7 Speech 9 Muisc 8 Without Caption Blind Review Input Cand 1Cand 2 Speech 9 Muisc 9 Cand 1 Speech 8 Muisc 6 Cand 2 Cand 1Cand 2 Cand 1Cand 2 Figure 4: PRISM-Bench evaluation pipeline. for Speech, Music, and Sound, and activates them conditionally according to the sample’s activated tags and visibility condition. A lexical overview of these criteria is shown in Figure 2(c). This design allows the evaluation to remain both diagnostic and comparable across heterogeneous audio scenarios. Audio-Visual Coherence measures whether generated audio is properly grounded in visible events and sounding sources, following the broader audio-visual alignment and on-screen grounding prob- lems studied in prior work [25,28,36,37]. It is activated only when the relevant audio condition is visually grounded, and therefore applies to On-screen and the On-screen portion of Mixed samples, but remains inactive for purely Off-screen cases. This dimension focuses on temporal alignment and physical plausibility, including whether speech is synchronized with visible speaking, whether musical sounds correspond to visible playing actions, and whether sound events are consistent with visible sources and their timing. Audio Quality evaluates the perceptual fidelity and technical real- ism of the audio signal itself. For Speech, it concerns naturalness, clarity, and vocal realism. For Music, it captures properties such as timbral plausibility, rhythmic stability, and overall musical co- herence. For Sound, it focuses on whether the generated audio is physically believable, acoustically well-formed, and free from obvi- ous synthetic artifacts or degradation. This dimension is intended to measure how realistic the audio sounds, independent of whether it is well grounded or well controlled. Audio Expressiveness assesses whether the generated audio con- veys appropriate emotion, atmosphere, and scene-level support beyond basic fidelity. For Speech, this includes the adequacy of speaking style, emotional tone, and delivery. For Music, it concerns whether the music appropriately conveys mood and expressive character. For Sound, it reflects whether the audio contributes plau- sibly to scene atmosphere and perceptual vividness. Prompt Following measures how faithfully the generated audio satisfies the textual instruction. This dimension is restricted to auditory adherence, so that the benchmark remains aligned with its audio-centric scope. For Speech, it considers verbal content and vocal characteristics; for Music, it concerns style, instrumentation, and mood; and for Sound, it focuses on event accuracy and texture correspondence. By isolating audio-side compliance from unrelated visual factors, this dimension makes the final scores more directly interpretable for T2AV evaluation. 3.4 Evaluation Framework As illustrated in Figure 4, PRISM-Bench adopts a tag-conditioned and visibility-aware evaluation framework for audio-centric T2AV assessment. Given a generated sample and its ground-truth refer- ence, the framework activates only the evaluation targets relevant to the sample’s annotated audio conditions, including both the ap- plicable dimensions and the corresponding type-specific rubrics, and evaluates them under a fixed blind judging protocol. This de- sign enables PRISM-Bench to support not only overall model com- parison, but also fine-grained and interpretable diagnosis across heterogeneous audio scenarios. Tag-Conditioned Evaluation. Using the human-verified activated tags defined in Section 3.2, PRISM-Bench partitions evaluation into three settings: On-screen, Off-screen, and Mixed. These tags specify which audio types are present in a sample and whether the relevant sound sources are visually grounded. During evaluation, only the dimensions and rubrics selected by the activated tags are included in the judge input. As example shown in Figure 4, the sample activates Speech and On-screen, while Sound and Off-screen are ignored. Accordingly, Stage 1 evaluates only the activated criteria for AV Coherence and Audio Quality from the merged clip without caption, whereas Stage 2 evaluates Speech-related Expressiveness and Prompt Following with access to the caption. This conditional activation mechanism keeps the evaluation focused on the relevant audio conditions and avoids introducing irrelevant scoring criteria into the judging process. Blind Side-by-Side Review. To ensure direct comparability under a shared perceptual context, we adopt a blind side-by-side review protocol. As shown in Figure 4, the evaluation model receives two candidate videos as separate inputs. In our implementation, each video begins with a 0.5-second black identifier frame labeled “1” or “2”, which is used only to map the outputs to Candidate 1 and Candi- date 2 and is explicitly ignored during evaluation. The model then evaluates the two candidates under matched viewing conditions M ’26, November 10–14, 2026, Rio de Janeiro, BrazilYuchen Sun et al. without being told which one is the generated sample and which one is the ground-truth reference. This design reduces presentation bias and preserves a controlled comparison setting. Two-Stage Judge Design. We use Gemini-3.1-pro-preview [8] 3 as the judge model and keep its nominal decoding configuration fixed. We set the temperature to 0.0 to remove sampling temper- ature as a varying factor, while recognizing that this setting does not guarantee identical outputs across repeated hosted-API calls: backend execution and numerical effects can still introduce vari- ation even under greedy decoding [1,19,26,43]. To align each request with the evidence required by the target dimensions, the evaluation is divided into two stages, as illustrated in Figure 4. The first stage evaluates Audio-Visual Coherence and Audio Quality from the merged clip together with each sample’s activated tags, which specify the audio categories to be evaluated. The second stage evaluates Audio Expressiveness and Prompt Following with access to the human-verified caption, which serves as the reference prompt and corresponds to the text prompt used to generate the tested video. This two-stage design separates perception-driven evaluation from prompt-conditioned evaluation, thereby improving the clarity and consistency of the judging process. Absolute Scoring with Evidence. Within each stage, the judge scores Candidate 1 and Candidate 2 independently against fixed rubric descriptions, rather than assigning only a relative preference. This is a pointwise, rubric-based design consistent with form-filling and reliability-oriented LLM-evaluation frameworks [10,22]. In addition, the judge produces a brief evidence statement for each active dimension. These evidence traces make the evaluation pro- cess more transparent, support later auditing and error analysis, and provide qualitative support for understanding model strengths and weaknesses beyond the numerical scores alone. 4 Experiments 4.1 T2AV Generation Models We evaluate the following recent text-to-audio-video generation systems on PRISM-Bench. Seedance 2.0 [2] 4 . Seedance 2.0 uses a unified multimodal audio- video generation architecture with native dual-channel audio. We evaluate its text-conditioned audiovisual outputs on the same bench- mark prompts used for the other systems. Kling v3 Omni [18]. We access Kling v3 Omni through the official API 5 and evaluate the native audiovisual outputs returned by the system without additional post-processing. Veo 3.0 and Veo 3.1 [9]. We access both versions through the official Gemini API 6 and evaluate synchronized audiovisual clips generated at 1280× 720 resolution with an 8-second duration. Sora 2 [30]. We access Sora 2 through the official OpenAI video API 7 and evaluate native system outputs generated at 1280×720 resolution with an 8-second duration. LTX-2 [13]. We run LTX-2 locally with the official repository 8 [13] and evaluate outputs generated with the released LTX-2 pipeline. 3 Official Doc: https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview. 4 https://seed.bytedance.com/seedance2_0. 5 https://kling.ai/document-api/apiReference/model/OmniVideo. 6 https://ai.google.dev/gemini-api/docs/video. 7 https://developers.openai.com/api/docs/guides/video-generation. 8 https://github.com/Lightricks/LTX-2. Our setup uses the 19B development model together with the official upsampling pipeline, producing 8-second clips at 704×1280 and 24 FPS with 40 sampling steps and classifier-free guidance scale 4.0. Ovi [23]. We evaluate Ovi locally using the official open-source release 9 . We adopt the official text-to-video generation pipeline and evaluate outputs at 720×1280 resolution with a 10-second duration, using UniPC sampling with 50 steps, audio guidance scale 3.0, and video guidance scale 4.0. MOVA [35]. We evaluate MOVA locally using the public codebase 10 . We follow the official T2VA testing protocol and use a pure white image as the first-frame condition for text-driven generation; in our preliminary experiments, this mode occasionally produced corrupted frames or black-screen failures. We evaluate outputs at 720×1280 resolution with a 10-second duration, using the default official inference path with classifier-free guidance scale 5.0. 4.2 PRISM-Bench Evaluation Results To mitigate context-dependent variation in ground-truth assess- ments, the mean model score for each metric-by-audio-type cell is normalized by the corresponding paired ground-truth mean and rescaled to the global ground-truth mean. The Final score is com- puted as the sum of all active calibrated cells. The evaluation results of representative recent T2AV systems are summarized in Table 1. Following the visibility-aware design of PRISM-Bench, we report results separately on the three benchmark subsets, namely On-screen, Off-screen, and Mixed. Within each subset, performance is evaluated along four perceptual dimensions, and is further broken down by the three audio types. This layout allows the table to present model behavior simultaneously from three complementary perspectives: evaluation dimension, audio type, and visibility condition. As a result, the table supports a more diagnostic view of model capability than holistic evaluation alone. The following discussion provides a detailed characterization of these observations, along with their model-level implications and insights. Frontier Proprietary Systems Maintain a Clear Advantage. Table 1 shows a substantial and consistent gap between fron- tier proprietary systems and current open-source models across all three subsets. Seedance 2.0 ranks first on On-screen, Off-screen, and Mixed with Final scores of 97.73, 78.11, and 99.88, respec- tively; Kling-v3-omni is the runner-up on the first two subsets with 85.99 and 72.76, and on Mixed with 94.61. By comparison, the strongest open-source result is 49.50 on On-screen (LTX-2), 40.16 on Off-screen (LTX-2), and 64.38 on Mixed (Ovi). Thus, the lead of Seedance 2.0 over the strongest open model remains at least 35 points on every subset. The dimension-wise results reveal where this advantage arises. On On-screen, Seedance 2.0 reaches 25.24 in Audio Quality, 25.69 in Audio Expressiveness, and 25.89 in Prompt Following, producing a Final score of 97.73 despite a lower AV Co- herence total of 20.91. These results show that recent proprietary systems have advanced native audio generation substantially, while current open-source alternatives remain far behind across both fidelity- and grounding-sensitive dimensions. 9 https://github.com/character-ai/Ovi. 10 https://github.com/OpenMOSS/MOVA PRISM-BenchMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Table 1: Results across the three Subsets, reported as GT-calibrated means on a 0–10 scale. Within each dimension, Tot. sums Speech (Spe.), Music (Mus.), and Sound (Snd.); Final sums all applicable totals. AV Coherence is marked as ’/’ when Off-screen. Audio QualityAudio ExpressivenessPrompt FollowingAV CoherenceTotal SubsetModelSpe.Mus.Snd. Tot.Spe.Mus.Snd. Tot.Spe.Mus.Snd. Tot.Spe.Mus.Snd. Tot.Spe.Mus.Snd. Final On-screen Ground Truth 8.067.547.5023.108.007.627.3222.948.628.247.9524.817.236.537.1720.9331.9129.9429.9391.78 Seedance 2.0 8.37 8.43 8.44 25.24 8.49 8.98 8.22 25.699.12 8.67 8.11 25.896.64 6.46 7.81 20.9132.62 32.54 32.58 97.73 Kling-v3-omni 8.536.467.3022.298.306.587.2722.14 9.236.627.0322.89 7.754.066.8618.6733.8223.7228.4685.99 Veo 3.1 8.136.897.6822.708.096.877.0722.038.576.436.8521.855.734.777.0317.5430.5224.9628.6484.11 Veo 3.0 7.977.067.4822.517.607.156.6621.428.206.636.7021.546.184.936.9518.0629.9625.7727.7983.52 Sora 2 6.675.485.6817.836.276.015.4117.696.746.796.0119.555.274.285.2114.7624.9522.5622.3169.82 LTX-2 5.224.724.1314.083.774.673.5511.983.794.873.8012.473.753.603.6210.9816.5317.8715.1049.50 Ovi 3.903.024.0210.952.833.513.579.922.784.163.8810.822.622.363.498.4712.1313.0614.9740.15 MOVA 4.313.344.2211.884.593.803.8212.215.713.643.9113.252.631.703.367.6917.2412.4815.3145.04 Off-screen Ground Truth 8.197.867.5623.608.087.657.1622.898.768.287.5624.61/25.0423.7922.2871.10 Seedance 2.0 8.65 8.80 8.37 25.82 8.65 8.74 8.21 25.609.24 9.29 8.16 26.69/26.54 26.83 24.74 78.11 Kling-v3-omni 8.658.257.7424.648.597.957.1323.67 9.308.446.7124.44/26.5424.6321.5972.76 Veo 3.1 8.397.527.4823.388.007.167.0022.168.326.966.2221.49/24.7021.6320.7067.03 Veo 3.0 8.577.287.5823.427.966.907.0821.937.976.805.9520.73/24.5020.9720.6166.08 Sora 2 6.757.166.7820.706.436.986.5619.976.607.397.1721.16/19.7921.5320.5161.83 LTX-2 4.735.164.8814.773.684.574.0912.343.764.964.3213.05/12.1714.6913.2940.16 Ovi 3.434.304.9012.622.773.814.1410.722.814.114.1911.11/9.0212.2213.2234.46 MOVA 4.044.474.0812.583.143.853.5210.513.144.173.8711.18/10.3212.4911.4734.28 Mixed Ground Truth 8.178.207.8824.257.757.427.1422.318.378.087.6324.087.338.077.6523.0531.6231.7730.3093.69 Seedance 2.0 8.22 8.34 8.50 25.06 8.38 8.20 8.21 24.78 9.00 8.77 8.59 26.35 7.32 8.05 8.32 23.6932.91 33.35 33.62 99.88 Kling-v3-omni 8.278.047.9624.278.177.397.7723.328.767.918.0924.766.817.717.7322.2632.0031.0531.5594.61 Veo 3.1 7.878.078.0924.027.566.876.8821.317.396.936.6720.996.227.837.6721.7229.0329.7029.3088.04 Veo 3.0 7.657.787.6523.087.296.416.5020.206.856.376.3119.536.087.427.3020.7927.8727.9727.7683.61 Sora 2 7.777.787.4723.027.787.516.9022.198.428.126.9323.475.587.697.1020.3629.5531.1028.3989.04 LTX-2 5.365.214.6215.193.003.773.2310.003.213.943.3510.493.424.943.9212.2914.9917.8515.1247.96 Ovi 5.476.166.0517.685.235.105.0515.385.595.585.1616.333.306.155.5414.9819.5922.9921.8064.38 MOVA 4.404.214.2712.882.463.002.487.942.783.402.568.752.453.593.249.2812.0914.2012.5538.85 Audio Fidelity and Prompt Adherence Advance Faster than Visible-Source Grounding. Seedance 2.0 exceeds the ground- truth Audio Quality total on On-screen, Off-screen, and Mixed. It also exceeds the corresponding ground-truth Prompt Follow- ing total on all three subsets. These numbers indicate substantial progress not only in perceptual audio realism, but also in audio-side instruction adherence. The remaining weakness is concentrated in visible-source grounding. On On-screen, Seedance 2.0 reaches 20.91 in AV Coherence, essentially matching but not exceeding the ground-truth total of 20.93, and remaining well below its other dimension totals. The same internal gap appears for Kling-v3-omni (18.67 in AV Coherence vs. 22.29 in Audio Quality) and Veo 3.1 (17.54 vs. 22.70). Thus, the updated results refine rather than overturn the original finding: frontier models now produce strong and con- trollable audio, but synchronizing that audio with specific visible sources remains comparatively difficult. On-screen Audio Remains the Most Challenging Grounding Regime. The On-screen subset simultaneously requires realistic audio, prompt consistency, and precise synchronization with visi- ble sound-producing events. Even Seedance 2.0, which exceeds the On-screen ground-truth aggregate by 5.95 points, remains slightly below the ground-truth AV Coherence total (20.91 vs. 20.93). All other evaluated systems fall further behind on this dimension. By comparison, Seedance 2.0 exceeds the ground truth by more than two points in both On-screen Audio Quality and Audio Expres- siveness. This contrast shows that the residual difficulty is not low perceptual realism alone. It lies in binding generated audio to visible speech, instrument playing, and object events. Off-screen evalu- ation removes explicit visible grounding, but shifts the challenge toward plausible acoustic world-building and faithful control of non-visible sound events. Mixed scenes then test whether these abilities coexist, which Seedance 2.0 handles well in aggregate but not uniformly across all type–dimension cells. Music Grounding Improves, but Remains Model-Dependent. Seedance 2.0 changes the earlier picture that Music is uniformly the weakest audio type. Its On-screen subtype totals are nearly balanced at 32.62 for Speech, 32.54 for Music, and 32.58 for Sound. Nevertheless, its Music AV Coherence score (6.46) remains lower than its Sound score (7.81), showing that aggregate balance can conceal a grounding-specific weakness. The weakness is more pro- nounced for other frontier systems. On Kling-v3-omni, On-screen AV Coherence is 7.75 for Speech, 4.06 for Music, and 6.86 for Sound; on Veo 3.1, the corresponding values are 5.73, 4.77, and 7.03. Their Music subtype totals also remain substantially below Speech. The new model therefore narrows the aggregate Music gap, but visible musical-performance grounding remains a useful stress test that distinguishes capability profiles across models. PRISM-Bench Reveals Failure Modes Hidden by Aggregate Scores. A principal value of PRISM-Bench is that it exposes failure modes that aggregate scores alone would obscure, including for the new leader. Seedance 2.0 achieves an On-screen Final score of 97.73, yet its AV Coherence total is 20.91, compared with 25.24 in Audio Quality, 25.69 in Audio Expressiveness, and 25.89 in Prompt Fol- lowing. On Off-screen, its Sound total (24.74) trails Speech (26.54) and Music (26.83), with the largest type gap appearing in Prompt M ’26, November 10–14, 2026, Rio de Janeiro, BrazilYuchen Sun et al. LTX2OviSora 0 20 40 60 80 100 Alignment Rate (%) 91% 93% 68% Aligned Misaligned (a) GT vs Model Agreement PFAVAQEXP 0 20 40 60 80 100 Alignment Rate (%) 77.6% 67.9% 70.1% 75.9% Aligned Misaligned (b) Overall Alignment by Dimension Sora vs Ovi 0 20 40 60 80 100 Alignment Rate (%) 76.7% Sora vs LTX2 77.7% LTX2 vs Ovi 63.4% Aligned Misaligned (c) Pairwise Alignment Overall Figure 5: Alignment analysis of model-based evaluation against human judgments. Following. Kling-v3-omni exhibits a different profile: its On-screen Speech total is 33.82, but its Music total drops to 23.72. These exam- ples demonstrate that a strong overall score does not imply uniform behavior across dimensions, audio types, and visibility conditions. By factorizing evaluation along these axes, PRISM-Bench remains informative even when a new model establishes a large aggregate lead: it identifies whether the remaining bottleneck is visible-source binding, musical-performance synchronization, or fine-grained con- trol of Off-screen sound events. Ground Truth Is Not a Hard Upper Bound. The updated re- sults strengthen the observation that the ground-truth reference is an anchor rather than a hard upper bound. Seedance 2.0 exceeds the calibrated ground-truth Final score on On-screen, Off-screen (78.11 vs. 71.10), and Mixed (99.88 vs. 93.69). The comparison is not uniform across dimensions: on On-screen, Seedance 2.0 exceeds ground truth in Audio Quality, Audio Expressiveness, and Prompt Following, but remains marginally lower in AV Coherence (20.91 vs. 20.93); on Mixed, it exceeds the ground-truth total in all four dimensions. These patterns indicate that the blind, rubric-based protocol does not mechanically favor the reference video. They also clarify how to interpret above-reference performance: generated samples can be judged more perceptually polished or more prompt- adequate than the collected reference without reproducing every reference-specific event. Most importantly, the factorized scores continue to expose residual grounding and control weaknesses after the aggregate score surpasses the reference, so PRISM-Bench does not collapse into a fixed upper-bound test. 4.3 Human Alignment of Prism-Bench To assess the human alignment of PRISM-Bench, we conduct a study on a representative subset spanning the On-screen, Off-screen, and Mixed settings. We sample outputs from Sora 2, LTX-2, and Ovi. Human raters follow the same factorized protocol as the automated evaluator, assessing Audio-Visual Coherence (AV) and Audio Qual- ity (AQ) from videos alone, and Audio Expressiveness (EX) and Prompt Following (PF) with access to the corresponding captions. Mean agreement exceeds 70% across dimensions and pairwise com- parisons. Detailed annotation procedures and study configurations are provided in the Appendix. We define alignment as agreement in preference direction: a comparison is aligned when human raters and the MLLM-based evaluator select the same preferred candidate in a Ground Truth– model or model–model comparison; otherwise, it is misaligned. Figure 5(a) reports Ground Truth–model alignment of 91% for LTX- 2, 93% for Ovi, and 68% for Sora 2. The lower agreement for Sora 2 is consistent with the greater difficulty of distinguishing stronger generated outputs from the reference when their perceptual gap is smaller. Figure 5(b) reports alignment across evaluation dimensions. PF achieves the highest agreement at 77.6%, whereas AV is the most challenging at 67.9%; the remaining dimensions fall between these values. PF and EX are often supported by explicit semantic or stylis- tic cues, while AV requires joint assessment of temporal synchro- nization, visible-source grounding, and scene-dependent plausibil- ity. Mean agreement across dimensions remains above 70%, sup- porting the automated protocol as a scalable evaluation proxy. Figure 5(c) shows agreement of 76.7%–77.7% for clearly separated model pairs and 63.4% for the closer LTX-2 versus Ovi comparison, where preferences are more difficult to distinguish. Taken together, these results support PRISM-Bench as a human-aligned automated proxy for large-scale T2AV evaluation. 5 Conclusion We present PRISM-Bench, the first audio-centric diagnostic bench- mark for T2AV generation. By organizing evaluation along two orthogonal axes, namely audio type (Speech, Music, and Sound) and sound-source visibility (On-screen and Off-screen), PRISM-Bench provides a more fine-grained and interpretable framework than ex- isting holistic benchmarks. Built on a rigorously curated dataset of around 900 human-verified samples, PRISM-Bench further supports reliable evaluation through a visibility-aware, side-by-side MLLM- as-a-Judge protocol. Experiments on representative recent T2AV systems show that current progress is uneven across dimensions and audio conditions. While perceptual audio quality has improved substantially, audiovisual grounding remains a major challenge, especially for on-screen music. We hope that PRISM-Bench will facilitate future research on audio-aware multimodal generation and more diagnostic evaluation of T2AV systems. PRISM-BenchMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Acknowledgments We thank the Meituan LongCat M team for their essential re- source support and the Meituan Visual Intelligence team for their valuable support with data resources. Their contributions greatly facilitated the development and evaluation of PRISM-Bench. References [1] Berk Atıl, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. 2025. Non-Determinism of “Deterministic” LLM System Settings in Hosted Environments. In Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems. Association for Computational Linguistics, Mumbai, India, 135–148. doi:10.18653/v1/2025.eval4nlp-1.12 [2] ByteDance Seed. 2026. Seedance 2.0 Official Launch. Official model release. https://seed.bytedance.com/en/blog/seedance-2-0-official-launch [3]Zhe Cao, Tao Wang, Jiaming Wang, Yanghai Wang, Yuanxing Zhang, Jiahao Wang, Jialu Chen, Miao Deng, Chenxi Liao, Yize Zhang, Yubin Guo, Zhaox- iang Zhang, and Jiaheng Liu. 2026. T2AV-Compass: Towards Unified Evalu- ation for Text-to-Audio-Video Generation. In Proceedings of the 43rd Interna- tional Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 306). Seoul, South Korea. arXiv:2512.21094 [cs.CV] https://openreview.net/ pdf/12d5ffbee9688a5393fa0db49d5b51de5d39f86a.pdf [4]Guibin Chen, Dixuan Lin, Jiangping Yang, Youqiang Zhang, Zhengcong Fei, Debang Li, Sheng Chen, Chaofeng Ao, Nuo Pang, Yiming Wang, et al.2026. SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model. arXiv:2602.21818 [cs.CV] https://arxiv.org/abs/2602.21818 [5] Luyu Chen, Zeyu Zhang, Haoran Tan, Quanyu Dai, Hao Yang, Zhenhua Dong, and Xu Chen. 2025. Distributional LLM-as-a-Judge. In Advances in Neural Information Processing Systems, Vol. 38. https://openreview.net/forum?id=0SRGbRbngJ [6]Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023. CLAP: Learning Audio Concepts From Natural Language Supervision. In 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5. doi:10.1109/ICASSP49357.2023.10095889 [7] Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, and Jia-Bin Huang. 2024. On the Content Bias in Fréchet Video Distance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 7277–7288. https://openaccess.thecvf.com/content/CVPR2024/html/Ge_On_the_ Content_Bias_in_Frechet_Video_Distance_CVPR_2024_paper.html [8] Google AI for Developers. 2026. Gemini 3.1 Pro Preview. Official Gemini API model documentation. https://ai.google.dev/gemini-api/docs/models/gemini- 3.1-pro-preview [9]Google DeepMind. 2025. Veo. Official model page. Describes Veo 3 and its native-audio capabilities. https://deepmind.google/technologies/veo/ [10] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2024. A Survey on LLM-as- a-Judge. arXiv preprint arXiv:2411.15594 (2024). https://arxiv.org/abs/2411.15594 [11]Ying Guo, Qijun Gan, Yifu Zhang, Jinlai Liu, Yifei Hu, Pan Xie, Dongjun Qian, Yu Zhang, Ruiqi Li, Yuqi Zhang, Ruibiao Lu, Xiaofeng Mei, Bo Han, Xiang Yin, Bingyue Peng, and Zehuan Yuan. 2026. ALIVE: Animate Your World with Lifelike Audio-Video Generation. Bytedance ALIVE Team. arXiv:2602.08682 [cs.CV] https://arxiv.org/abs/2602.08682 [12] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2022. AudioCLIP: Extending CLIP to Image, Text and Audio. In 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 976–980. doi:10.1109/ ICASSP43922.2022.9747631 [13]Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, et al.2026. LTX- 2: Efficient Joint Audio-Visual Foundation Model. arXiv preprint arXiv:2601.03233 (2026). https://arxiv.org/abs/2601.03233 [14]Daili Hua, Xizhi Wang, Bohan Zeng, Xinyi Huang, Hao Liang, Junbo Niu, Xinlong Chen, Quanqing Xu, and Wentao Zhang. 2025. VABench: A Comprehensive Benchmark for Audio-Video Generation. arXiv preprint arXiv:2512.09299 (2025). https://arxiv.org/abs/2512.09299 [15]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. 2024. VBench: Comprehensive Benchmark Suite for Video Generative Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 21807–21818. https://openaccess.thecvf.com/content/CVPR2024/html/Huang_ VBench_Comprehensive_Benchmark_Suite_for_Video_Generative_Models_ CVPR_2024_paper.html [16]Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. 2019. Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhance- ment Algorithms. In Interspeech 2019. 2350–2354. doi:10.21437/Interspeech.2019- 2219 [17]Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019. AudioCaps: Generating Captions for Audios in the Wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 119–132. doi:10.18653/v1/N19-1011 [18] Kling AI. 2026. Kling VIDEO 3.0 Model User Guide. Official model documentation. https://app.klingai.com/cn/quickstart/klingai-video-3-model-user-guide [19] Lujun Li, Lama Sleem, Yangjie Xu, Yewei Song, Aolin Jia, Jerome Francois, and Radu State. 2026. The Necessity of Setting Temperature in LLM-as-a-Judge. arXiv preprint arXiv:2603.28304 (2026). https://arxiv.org/abs/2603.28304 [20]Jiahe Liu, Youran Qu, Qi Yan, Xiaohui Zeng, Lele Wang, and Renjie Liao. 2024. Fréchet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos. arXiv preprint arXiv:2407.16124 (2024). https://arxiv.org/abs/2407.16124 [21]Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. 2024. EvalCrafter: Benchmarking and Evaluating Large Video Generation Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 22139–22149. https://openaccess.thecvf.com/content/CVPR2024/html/ Liu_EvalCrafter_Benchmarking_and_Evaluating_Large_Video_Generation_ Models_CVPR_2024_paper.html [22] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Singapore, 2511–2522. doi:10.18653/v1/2023.emnlp-main.153 [23]Chetwin Low, Weimin Wang, and Calder Katyal. 2025. Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation. arXiv preprint arXiv:2510.01284 (2025). https://arxiv.org/abs/2510.01284 [24] Ge Ya Luo, Gian Mario Favero, Zhi Hao Luo, Alexia Jolicoeur-Martineau, and Christopher Pal. 2025. Beyond FVD: An Enhanced Evaluation Metrics for Video Generation Distribution Quality. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=cC3LxGZasH [25]Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, and Yuchao Dai. 2024. TAVGBench: Benchmarking Text to Audible- Video Generation. arXiv preprint arXiv:2404.14381 (2024). https://arxiv.org/abs/ 2404.14381 [26]Alberto Messina and Stefano Scotta. 2026. Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models. Transactions on Machine Learning Research (2026). https://openreview.net/forum?id=bz0he4bARF [27]Xiongkuo Min, Guangtao Zhai, Jiantao Zhou, Mylene C. Q. Farias, and Alan Con- rad Bovik. 2020. Study of Subjective and Objective Quality Assessment of Audio- Visual Signals. IEEE Transactions on Image Processing 29 (2020), 6054–6068. doi:10.1109/TIP.2020.2988148 [28]Shentong Mo, Jing Shi, and Yapeng Tian. 2024. Text-to-Audio Generation Syn- chronized with Videos. arXiv:2403.07938 [cs.SD] https://arxiv.org/abs/2403.07938 [29]OpenAI. 2025. GPT-5 Chat Model. Official OpenAI API model documentation. https://developers.openai.com/api/docs/models/gpt-5-chat-latest [30] OpenAI. 2025. Sora 2 System Card. OpenAI system card. https://openai.com/ index/sora-2-system-card/ [31] Adam Polyak et al.2024. Movie Gen: A Cast of Media Foundation Models. arXiv preprint arXiv:2410.13720 (2024). https://arxiv.org/abs/2410.13720 [32]Shu Pu, Yaochen Wang, Dongping Chen, Yuhang Chen, Guohao Wang, Qi Qin, Zhongyi Zhang, Zhiyuan Zhang, Zetong Zhou, Shuang Gong, Yi Gui, Yao Wan, and Philip S. Yu. 2025. Judge Anything: MLLM as a Judge Across Any Modality. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 5742–5753. doi:10.1145/3711896.3737409 [33]Jielin Qiu, Jianguo Zhang, Zixiang Chen, Liangwei Yang, Ming Zhu, Juntao Tan, Haolin Chen, Wenting Zhao, Rithesh Murthy, Roshan Ram, Akshara Prabhakar, Shelby Heinecke, Caiming Xiong, Silvio Savarese, and Huan Wang. 2026. Au- dioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech. arXiv preprint arXiv:2602.23649 (2026). https://arxiv.org/abs/2602.23649 [34]Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. 2023. M-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10219–10228. https://openaccess.thecvf.com/content/CVPR2023/html/ Ruan_M-Diffusion_Learning_Multi-Modal_Diffusion_Models_for_Joint_ Audio_and_Video_CVPR_2023_paper.html [35] SII-OpenMOSS Team. 2026. MOVA: Towards Scalable and Synchronized Video- Audio Generation. arXiv preprint arXiv:2602.08794 (2026). https://arxiv.org/abs/ 2602.08794 [36] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. 2018. Audio-Visual Event Localization in Unconstrained Videos. In Proceed- ings of the European Conference on Computer Vision. 247–263.https: //w.ecva.net/papers/eccv_2018/papers_ECCV/html/Yapeng_Tian_Audio- Visual_Event_Localization_ECCV_2018_paper.php M ’26, November 10–14, 2026, Rio de Janeiro, BrazilYuchen Sun et al. [37]Efthymios Tzinis, Scott Wisdom, Tal Remez, and John R. Hershey. 2022. Au- dioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation. In Computer Vision – ECCV 2022. 368–385. doi:10. 1007/978-3-031-19836-6_21 [38]Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. 2018. Towards Accurate Generative Models of Video: A New Metric & Challenges. arXiv preprint arXiv:1812.01717 (2018). https://arxiv.org/abs/1812.01717 [39] Zhenting Wang, Shuming Hu, Shiyu Zhao, Xiaowen Lin, Felix Juefei-Xu, Zhuowei Li, Ligong Han, Harihar Subramanyam, Li Chen, Jianfa Chen, Nan Jiang, Lingjuan Lyu, Shiqing Ma, Dimitris N. Metaxas, and Ankit Jain. 2025. MLLM-as-a-Judge for Image Safety without Human Labeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14657– 14666. https://openaccess.thecvf.com/content/CVPR2025/html/Wang_MLLM-as- a-Judge_for_Image_Safety_without_Human_Labeling_CVPR_2025_paper.html [40]Tianxin Xie, Wentao Lei, Kai Jiang, Guanjie Huang, Pengfei Zhang, Chunhui Zhang, Fengji Ma, Haoyu He, Han Zhang, Jiangshan He, Jinting Wang, Linghan Fang, Lufei Gao, Orkesh Ablet, Peihua Zhang, Ruolin Hu, Shengyu Li, Weilin Lin, Xiaoyang Feng, Xinyue Yang, Yan Rong, Yanyun Wang, Zihang Shao, Zelin Zhao, Chenxing Li, Shan Yang, Wenfu Wang, Meng Yu, Dong Yu, and Li Liu. 2025. PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation. arXiv:2512.23994 [cs.SD] https://arxiv.org/abs/2512.23994 [41]Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al.2025. Qwen3-Omni Technical Report. arXiv preprint arXiv:2509.17765 (2025). https://arxiv.org/abs/2509.17765 [42]Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. 2024. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand, 1979–1998. doi:10.18653/v1/2024.acl-long.109 [43]Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. 2025. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. In Advances in Neural Information Processing Systems, Vol. 38. https://proceedings.neurips.c/paper_files/paper/ 2025/hash/f80094a824ba5912d4a2de169c404a40-Abstract-Conference.html [44] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, Vol. 36. 46595–46623. https://openreview.net/forum?id=uccHPGDlao [45] Yanghao Zhou, Haitian Li, Rexar Lin, Heyan Huang, Jinxing Zhou, Changsen Yuan, Tian Lan, Ziqin Zhou, Yudong Li, Jiajun Xu, Jingyun Liao, YiMing Cheng, Xuefeng Chen, Xian-Ling Mao, and Yousheng Feng. 2026. MTAVG-Bench: A Di- agnostic Benchmark for Multi-Talker Dialogue-Centric Audio-Video Generation. In Proceedings of the 64th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). Association for Computational Linguistics, San Diego, California, United States, 19906–19942. doi:10.18653/v1/2026.acl-long.913 PRISM-BenchMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Appendix A Ground-Truth Reliability Analysis The single GT row reported in the main results table should be interpreted as a context-averaged descriptive reference. Because the same GT clip is repeatedly judged when paired with different candidate models, some reference-side fluctuation across pairing contexts is expected. The key question is therefore not whether such fluctuation exists, but whether it remains small relative to the real performance differences among candidate models. To answer this question, we treat GT Std,푑 bias , and푑 signal as the primary evidence, and report푑 signal /푑 bias only as a compact descriptive summary. A.1 Definitions and Subset-Level Summary To assess robustness to contextual anchoring, we report two standardized effect sizes. For any pair of candidate models(푚 1 ,푚 2 ), let s (푚) GT denote the distribution of GT scores obtained when GT is paired against푚under the same judge and prompt configuration, and let s 푚 denote the corresponding distribution of candidate-model totals on the overlapping evaluation items. We define the GT-induced contextual drift as 푑 bias (푚 1 ,푚 2 )= E[s (푚 1 ) GT ]−E[s (푚 2 ) GT ] 푠 pooled ,(1) and the inter-model performance gap as 푑 signal (푚 1 ,푚 2 )= E[s 푚 1 ]−E[s 푚 2 ] 푠 pooled ,(2) where 푠 pooled is the pooled standard deviation of the compared score distributions. For completeness, we also report SNR= 푑 signal 푑 bias ,(3) as a compact descriptive ratio. All subset-level quantities in Table 2 are reported as unweighted averages over all available model pairs. Specifically, Std denotes the average GT standard deviation across comparison contexts,푑 bias and푑 signal denote the average absolute Cohen’s 푑values computed over all available model pairs, and Eval SNR is the corresponding average ratio푑 signal /푑 bias . These statistics jointly characterize both the magnitude of context-induced GT drift and its separation from genuine inter-model performance differences at the subset level. Table 2: Subset-level GT stability and bias–signal separation. SubsetStdd bias d signal Eval SNR On-screen1.4260.2810.5211.85 Off-screen1.0420.1520.6414.21 Mixed1.2370.2330.8143.49 Table 2 shows a clear subset-level pattern. Most importantly, the average model signal remains at least medium in magnitude on all three subsets:푑 signal =0.521 for On-screen, 0.641 for Off-screen, and 0.814 for Mixed. At the same time, the GT-induced contextual drift remains smaller than the model-level signal throughout. The cleanest regime is Off-screen, where푑 bias =0.152 remains small while푑 signal =0.641 is substantially larger. Mixed is similarly well separated, with푑 bias =0.233 versus푑 signal =0.814. On-screen is the hardest regime: its푑 bias rises to 0.281, reflecting greater sensitivity to visible grounding and synchronization, but its average signal still reaches 0.521, so the subset remains informative for comparative diagnosis. The raw GT fluctuations in Table 2 should therefore be read as dispersion indicators rather than as the benchmark decision criterion itself. Their ordering is nevertheless meaningful and consistent with the subset design: On-screen is the most variable, Off-screen the most stable, and Mixed lies between the two. A.2 Dimension-Wise Stability Breakdown Table 3 localizes the main source of instability. The dominant contribution comes from the synchronization-sensitive branch of Audio-Visual Coherence, especially for On-screen speech and music, which reach 2.145 and 2.314, respectively. This indicates that visible grounding is the most context-sensitive component of the evaluation: once the judge must resolve lip-sync, performer–sound consistency, or event-level onset alignment, small contextual shifts propagate more strongly into the final score. By contrast, the Off-screen subset removes Audio-Visual Coherence entirely, and the remaining branches are generally more stable: several entries fall below 1.0, including Off-screen Audio Quality for speech (0.801) and music (0.889), as well as Off-screen Audio Expressiveness for speech (0.881). The Mixed subset behaves as an intermediate regime. It retains an Audio-Visual Coherence branch, but only for the on-screen portion of the active tags, which reduces music-side AV variance relative to pure On-screen while preserving a more difficult speech-side M ’26, November 10–14, 2026, Rio de Janeiro, BrazilYuchen Sun et al. Table 3: Dimension-wise GT stability (average standard deviation) across subsets. “/” indicates that the dimension is inactive for that subset. DimensionSubtype On-screen Off-screen Mixed Audio-Visual Coherence Speech2.145/1.929 Music2.314/0.944 Sound1.718/1.308 Audio Quality Speech1.2830.8011.163 Music1.4650.8890.925 Sound1.2771.0541.057 Audio Expressiveness Speech0.9640.8811.021 Music1.1540.9901.090 Sound1.1771.1551.241 Prompt Following Speech0.9660.8451.041 Music1.2871.2111.405 Sound1.3951.5711.497 grounding problem. In short, the higher subset-level fluctuation of On-screen is driven primarily by synchronization-sensitive dimensions rather than by a uniform instability across the entire rubric. A.3 Interpretation of Bias, Signal, and SNR Our main validity criterion is the joint behavior of푑 bias and푑 signal : contextual drift should remain limited, while model-level separation should remain at least moderate. Under this criterion, the benchmark remains reliable across all three subsets. Off-screen and Mixed provide the clearest margins, as they combine relatively small푑 bias with strong푑 signal . On-screen is more conservative, but its elevated bias is concentrated in the visible grounding branch identified in Table 3, rather than indicating a general collapse of evaluator consistency. SNR is therefore reported as a compact descriptive summary rather than as the sole decision criterion. This is particularly appropriate for synchronization-heavy settings, where the ratio can be depressed by harder contextual conditions or near-tie comparisons even when the evaluator still preserves meaningful model separation. This is exactly the case for the On-screen subset: although its averageSNR=1.85, its average푑 signal =0.521 remains in the medium-effect range, so the evaluator continues to distinguish models by a non-trivial margin. The practical implication is visibility-aware: Off-screen and Mixed provide the cleanest reliability margins, while On-screen remains informative but should be interpreted more cautiously for closely matched systems. B Judge Model Ablation We further conduct a judge-model ablation under a controlled evaluation setup. All judge variants are evaluated on the same Mixed-set protocol, with the same two-stage prompt structure, the same blind two-candidate input format, and the same JSON output schema. We use the Mixed subset because it jointly includes on-screen and off-screen conditions, making it the most comprehensive setting for judge selection. The comparison thus isolates judge behavior rather than differences in prompt engineering or interface design. The purpose is not merely to identify the judge with the lowest variance, but to examine whether a candidate judge can maintain limited contextual drift while still preserving meaningful rubric-aligned discrimination. Table 4 summarizes the resulting reliability statistics. Table 4: Judge-model ablation under a controlled evaluation setup. Std is the average GT standard deviation, and푑 bias ,푑 signal , and Eval SNR are computed in the same way as in Table 2. JudgeStdd bias d signal Eval SNR Gemini-3.1 Pro (Final)0.9150.1410.4293.05 Gemini-3.1 Flash0.5240.0750.2733.64 Gemini-3.0 Flash0.5910.0730.3675.00 Qwen3-Omni-30B-Instruct0.1670.0110.0121.11 Quantitative Comparison and Final Selection. The ablation suggests an important selection principle: an extremely small푑 bias is not necessarily optimal by itself. A desirable judge should keep bias low while still preserving sufficient score variation to separate models with genuinely different performance. This trade-off is illustrated by Qwen3-Omni. It achieves the smallest푑 bias , indicating strong scoring consistency. However, its푑 signal is similarly small, resulting in an Eval SNR of only 1.11. This pattern suggests that, under our evaluation setup, Qwen3-Omni uses the score scale rather conservatively, so its stability advantage does not translate into equally strong discriminative PRISM-BenchMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil power. One possible explanation is that our blind pairwise video-evaluation protocol may not be well aligned with the input format or supervision signals emphasized during its training. We therefore interpret this result not as a general weakness of Qwen3-Omni, but as evidence that judge selection should balance low bias with sufficient sensitivity to meaningful quality differences. The two flash variants show a milder version of the same trade-off. Both Gemini-3.1 Flash and Gemini-3.0 Flash reduce contextual drift to around 0.07 – 0.08, but they also reduce the average model-level signal to 0.273 and 0.367, respectively. These values indicate that the judges are comparatively conservative and may compress meaningful score differences, especially when the compared models are not extremely far apart. By contrast, Gemini-3.1 Pro maintains a small but non-trivial contextual bias (푑 bias =0.141), while preserving the largest average inter-model separation among the tested judges (푑 signal =0.429). This balance is preferable for PRISM-Bench. The selected judge should not drive푑 bias toward zero by becoming insensitive; rather, it should retain enough rubric responsiveness to separate genuinely different candidates while keeping bias materially below signal. Under this criterion, Gemini-3.1 Pro provides the most reasonable operating point. A more concrete indication appears in the pairwise ablation results. Under Gemini-3.1 Pro, the strongest model contrast reaches 푑 signal =0.639, which is already in the medium-to-large range, while the corresponding average푑 bias remains well below that level. This pattern is exactly what the benchmark needs: small contextual drift together with non-compressed model separation. The final judge is therefore chosen not because it minimizes every reliability statistic in isolation, but because it best preserves the intended trade-off between stability and discriminability. C Evaluation Protocol Details The implementation of the visibility-aware judging protocol in PRISM-Benchis described below. All three subsets are evaluated under a common blind side-by-side scaffold, while differing only in how visibility assumptions determine the dimensions that are meaningfully instantiated. Figures 8 and 9 illustrate the shared scaffold and its stage-wise slots, respectively. In particular, the system-level instruction, blind candidate setup, preprocessing rules, score scale, and response schema are held fixed across subsets. The only variation lies in how subset-specific visibility conditions govern dimension activation and tag routing. Moreover, the reference prompt is not a separate prompt family component, but a shared field that is populated only in the caption-conditioned stage. C.1 Subset-Specific Judge Configuration The protocol follows a shared two-stage formulation, illustrated in Figures 8 and 9, so that differences across subsets reflect diagnostic intent rather than prompt-format variation. As shown in Figure 8, all subsets inherit the same global scaffold, including the system-level instruction, blind candidate setup, preprocessing rules, scoring scale, and response schema. Figure 9 further shows how this scaffold is instantiated through two stage-specific prompt slots. Stage A focuses on perceptual grounding without textual conditioning and is used to assess dimensions whose validity depends primarily on direct audio–visual evidence. Stage B introduces the human-verified caption and evaluates dimensions that require semantic interpretation of the generated audio relative to the intended scene description. This separation keeps the judging process structurally consistent while preventing prompt information from leaking into dimensions that should be determined from perceptual evidence alone. Within this common scaffold, subset-specific behavior is determined by the visibility assumption encoded in the active tags. For the On-screen subset, the protocol assumes that the relevant sound-producing source is visually observable, so synchronization and source- grounded correspondence remain valid evaluation targets. Under this setting, Stage A assesses AV Coherence together with Audio Quality, while Stage B evaluates Expressiveness and Prompt Following with caption conditioning. For the Off-screen subset, by contrast, the protocol explicitly removes visible-source correspondence from the decision space: since the causally relevant source is not expected to appear in the frame, the evaluation should not penalize the absence of on-screen alignment. As a result, AV Coherence is deactivated, whereas the remaining dimensions continue to be assessed under an off-screen interpretation. The Mixed subset inherits the same global structure but requires visibility-aware routing within a single request, because on-screen and off-screen audio conditions may coexist in the same sample. In this case, only the on-screen portion of the active tags contributes to AV Coherence, since strict source-grounded alignment is meaningful only when the sound source is visible. By contrast, Audio Quality remains applicable to all active audio types regardless of visibility, and Stage B evaluates Expressiveness and Prompt Following over the full tag set with caption conditioning. The mixed setting therefore does not introduce a new prompt template; instead, it uses the shared scaffold in Figure 9 to selectively activate visibility-sensitive criteria while preserving a unified response format across heterogeneous audio conditions. At the request level, each judge call is organized as a single multimodal message consisting of a text instruction followed by two blinded candidate videos. In the evaluation implementation, the two videos are uploaded separately as Base64-encoded video parts rather than pre-concatenated into one merged clip. Candidate identity is recovered from the 0.5-second black identifier frame at the beginning of each clip, labeled “1” or “2”, and that introductory frame is explicitly ignored during scoring. For the On-screen and Mixed subsets, the implementation issues two judge calls per sample. Stage A receives the active tags and the two candidate videos, and returns scores and evidence for AV Coherence and Audio Quality. Stage B additionally receives the human-verified caption as the reference prompt and returns scores and evidence for Expressiveness and Prompt Following. For the Off-screen subset, the same blind input structure is preserved, but the active dimensions are reduced because AV Coherence is inactive. M ’26, November 10–14, 2026, Rio de Janeiro, BrazilYuchen Sun et al. Inside a vast open-air stadium, a male singer in a black suit with a red shirt underneath is speaking on stage. Holding a microphone, he moves steadily across the center of the stage and, amid an excited, booming live atmosphere, says in English: “Oh my love is also by Ed Sheeran and Steve Mac, and this one as well is for...”. Behind him, the stands are packed to capacity, a sea of people, with sustained cheering and a slight reverberation in the expansive space enhancing the concert mood. Overhead is an arched grandstand structure. As the camera moves, a large LED screen and lighting rig appear on the left side of the stage; the screen shows synchronized close-ups of the singer, with blue spotlights in the background, intensifying the live ambience. The singer is fully engaged, sometimes looking up and sometimes turning sideways, his movements expressive, his left hand occasionally lifting to make emphatic gestures. In a dim, atmospheric jazz club, mid-tempo jazz rings out on stage, blending piano, drums, and electric guitar. A woman in a black sequined top sits at a black grand piano, angled toward the lower right of the frame, singing into the microphone with a bright tone and a moderate pace the original English lyrics: “When the beautiful awake, I see.” Her hair is dark brown with blunt bangs, and her expression is focused. Behind and to her left, a male drummer wearing glasses and a black shirt strikes a drum kit with white drumheads, his movements powerful and strongly rhythmic. On the right side of the frame, a male bassist, also wearing glasses and a black top, holds a light-colored wooden bass, swaying to the music as his fingers deftly pluck the strings. Light falls from above and slightly to the right, casting soft highlights on the instruments and performers. A cheetah wearing a brown collar, with a tawny coat covered in black spots, moves slowly from right to left through golden grass, its mouth slightly open and eyes looking ahead. The background is a blur of grass and the horizon. A male voiceover can be heard off-screen, speaking at a moderate pace in a calm tone, saying: "...her dynasty. " The background music is instrumental with a grand, majestic feel, gradually building as the video progresses. On-screen Set MUSIC On-screen SPEECH On-screen Mixed Set SPEECH On-screen SOUND On-screen Off-screen Set MUSIC Off-screen SPEECH Off-screen Figure 6: Caption examples and subset-aware tag routing. The role of the caption is therefore not merely descriptive. It determines which semantic branch is activated in the caption-conditioned stage and clarifies how the same audio-type tags are interpreted under the On-screen, Off-screen, and Mixed protocols. Representative examples of subset-specific captions and their activated tags are illustrated in Figure 6. C.2 Case Study: Stage-Wise Score Aggregation After both stages succeed, the implementation merges the two JSON responses candidate by candidate. The final record copies AV Coherence and Audio Quality from Stage A, copies Expressiveness and Prompt Following from Stage B, and then verifies that all required dimensions are present before saving the unified evaluation record. In this sense, the final output is not produced by a third scoring pass; it is a deterministic merge of the two stage-wise results. Figure 7 provides a concrete case study of this process. Rather than emphasizing the activation procedure itself, the figure is intended to show how the protocol operates on an individual sample: one caption, its activated tags, the stage-wise score objects returned for the two candidates, and the final merged record. This sample-level view makes the evaluation outcome directly auditable and offers a more informative illustration of judging validity than a purely schematic request-format diagram. D Human Alignment Following common practice in human evaluation, each comparison pair is independently rated by three human annotators. All annotators assign absolute integer scores on a 0–10 scale using the same rubric as the automated judge. For each scoring item, the final human score is reported as a single integer label obtained by majority vote. In the rare case where all three ratings differ, the item is adjudicated through a brief reconciliation step to determine the final integer score. This protocol is intended to reduce individual rater noise while preserving a discrete and interpretable human reference. PRISM-BenchMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil In the driver’s seat of a vehicle with a brown interior, a short-blonde woman in a dark vest sits upright, her expression serious and focused as she calmly gazes ahead. The frame is dominated by the off-road vehicle’s interior, including the windows, brown headliner, the steering wheel and the hand gripping it, and the sun visor on the left. A bottle of mineral water is visible on the right. She says softly,"not very close." She then looks to the left. Outside the window stretches a vast grassland, with two lions resting in the distance. Someone beside her says,"You know, it wouldn't be a good idea for them to start hunting here, because whatever they kill..." Meanwhile, her head slowly turns to the right, her eyes continually tracking the scene outside. The camera remains steady throughout; through the window, two tawny lions lie flat on the grass, resting, suggesting the vehicle may be parked on the savanna of a wildlife reserve. A low, faint ambient background noise and the chirring of insects can be heard continuously. SOUND Off-screen SPEECH On-screen SPEECH Off-screen CaptionGTTags Kling-v3-Omni Veo 3.0 Sora 2 Ovi MOVA LTX-2 On-screen speech is generally well grounded, with the female line broadly aligned to the visible mouth motion, and the off-screen speech is easy to interpret as a separate passenger-side reply. The main deductions come from audio realism rather than structure: both speech tracks sound somewhat synthetic and stiff, while the off-screen sound layer is thinner than the caption and misses the continuous insect chirping. Reason (Summary) AV COHERENCE AUDIO QUALITY EXPRESSIVENESS PROMPT FOLLOWING SOUND: 6 SPEECH: 7 SOUND: 6 SPEECH: 4 SOUND: 8 SPEECH: 9 SOUND: 9 SPEECH: 10 Scores AV COHERENCE AUDIO QUALITY EXPRESSIVENESS PROMPT FOLLOWING SOUND: 8 SPEECH: 7 SOUND: 7 SPEECH: 6 SOUND: 6 SPEECH: 8 SOUND: 5 SPEECH: 9 Scores AV COHERENCE AUDIO QUALITY EXPRESSIVENESS PROMPT FOLLOWING SOUND: 8 SPEECH: 7 SOUND: 7 SPEECH: 4 SOUND: 8 SPEECH: 9 SOUND: 6 SPEECH: 10 Scores The clip preserves the intended split between on- screen speech and off-screen speech reasonably well, and the off-screen reply remains contextually plausible inside the vehicle. Its main weakness lies in off-screen sound: the ambient layer is present but generic, with low-level background noise that broadly fits the scene yet fails to realize the continuous insect chirping required by the prompt. The semantic structure is recovered cleanly: the on-screen speech line is recognizable, the off- screen speech continuation is clear, and the role assignment follows the caption well. However, both speech tracks still sound somewhat dry and synthetic, and although the off-screen sound layer is more complete than in several weaker models, the overall scene ambience remains less natural than the reference. AV COHERENCE AUDIO QUALITY EXPRESSIVENESS PROMPT FOLLOWING SOUND: 8 SPEECH: 1 SOUND: 8 SPEECH: 4 SOUND: 7 SPEECH: 9 SOUND: 5 SPEECH: 10 Scores The separation between on-screen speech and off- screen speech is largely correct, and the off-screen reply is still understandable as a second speaker in the car. The biggest failure is in on-screen speech, where the first female line shows a severe lip-sync mismatch; in addition, the speech quality remains noticeably synthetic, even though the off-screen sound bed is broadly compatible with the vehicle interior. AV COHERENCE AUDIO QUALITY EXPRESSIVENESS PROMPT FOLLOWING SOUND: 3 SPEECH: 2 SOUND: 4 SPEECH: 6 SOUND: 5 SPEECH: 3 SOUND: 6 SPEECH: 2 Scores AV COHERENCE AUDIO QUALITY EXPRESSIVENESS PROMPT FOLLOWING SOUND: 9 SPEECH: 8 SOUND: 9 SPEECH: 5 SOUND: 3 SPEECH: 3 SOUND: 3 SPEECH: 2 Scores This sample shows compounded failures across all three active categories. The on-screen speech has weak lip-sync and poor grounding, the off-screen speech is cut off before completing the intended reply, and the off-screen sound layer is sparse and under-specified, so both coherence and prompt recovery are substantially weakened. The sample retains a rough scene setup, with partially plausible on-screen speech timing and some background audio that does not fully break the vehicle context. However, the on-screen speech b e g i n s u n n a t u r a l l y, t h e o f - s c r e e n s p e e c h continuation is missing, and the off-screen sound layer is too weak and under-detailed to support the full caption, including the expected continuous insect chirping. Figure 7: Suggested case study for stage-wise score aggregation. M ’26, November 10–14, 2026, Rio de Janeiro, BrazilYuchen Sun et al. <system_instruction> <role> You are an expert audio-visual evaluation specialist proficient in multimodal analysis. You possess rigorous analytical skills, and strictly follow established scoring standards to provide objective, absolute assessments for each candidate independently. </role> <instructions> 1. **Plan**: Understand the video content, carefully read <reference_prompt> and <active_tags> to determine the audio types that need assessment. 2. **Evaluate Candidate 1 First**: Strictly follow <scoring_standards> and <dimensions> to complete ALL dimension scores for Candidate 1. Each score must be based solely on Candidate 1's own performance against the absolute scoring standards. 3. **Evaluate Candidate 2 Next**: Then complete ALL dimension scores for Candidate 2 in the same manner, treating it as an entirely independent evaluation. Do NOT revisit or adjust Candidate 1's scores. 4. **Self-Check**: Before outputting, verify that each score is justified by the candidate's own performance — not by comparison with the other candidate. Reference <reference_prompt> to ensure evaluation accuracy. 5. **Format**: Output the results strictly following the JSON structure required by <output_format>. </instructions> <constraints> - Tone: Professional, technical, objective. - Guidelines: This round of evaluation focuses on the technical quality, expressiveness, and prompt adherence of the audio. - Absolute Scoring: This is an independent evaluation, NOT a comparative ranking. Each candidate's score must be determined solely by how well it meets the <scoring_standards>. Do NOT use one candidate's quality to anchor, inflate, or deflate the other's score. Two candidates of identical quality should receive identical scores; two candidates of different quality will naturally differ — but the gap must arise from each candidate's own merits, not from deliberate differentiation. Eval System Prompt <user_prompt> <context> <reference_prompt> REFERENCE_PROMPT </reference_prompt> (activated only in Stage A) <active_tags> ACTIVE_TAGS </active_tags> <candidates> - Candidate 1: Corresponds to the video with the black numbered frame "1" at the beginning. - Candidate 2: Corresponds to the video with the black numbered frame "2" at the beginning. </candidates> </context> <preprocessing_instructions> (1) Identity Recognition: The beginning (0 seconds) of each video file contains a black background frame lasting about 0.5 seconds, displaying a white number ("1" or "2"). Please use this number to distinguish candidate_1 and candidate_2 in the output. (2) Ignore Interference: When evaluating, please completely ignore this **introductory ID frame** and evaluate only the main content. </preprocessing_instructions> <scoring_standards>【Scoring Standards: 0-10 points for each item】 * 0 = Extremely Poor / Complete Failure * 1–2 = Very Poor (Serious and frequent errors) * 3–4 = Weak (Obvious defects, but barely recognizable) * 5–6 = Moderate (Average, with minor flaws) * 7–8 = Good (Mostly natural, meets expectations) * 9–10 = Excellent (Extremely realistic/precise, almost impeccable) </scoring_standards> <dimensions> ... </dimensions> <output_format> ... </output_format> </user_prompt> Eval Framework Prompt Figure 8: Shared prompting scaffold of the judge model. PRISM-BenchMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil <dimensions> [Dimension 1] AV_COHERENCE [Assess the temporal synchronization and physical plausibility between audio and video] Only evaluate items corresponding to <active_tags> 1.SPEECH : If <active_tags> contains "SPEECH_A" (On-screen Speech/Singing): - Voice Feature Match: Do timbre, age, and gender accurately correspond to the person speaking in the frame? - Temporal Boundaries: Are the start and end times of speech completely consistent with mouth movements or actions in the frame? - Lip Sync: Evaluate mouth movements in relation to speech, but first assess lip visibility conditions: * High Visibility (close-up, front-facing, clear articulation): Strictly check if mouth opening/closing matches pronunciation. * Reduced Visibility (humming, whispering, crying/laughing, speaker facing away): Do NOT heavily penalize lip-sync mismatches; focus on whether overall speech timing aligns with visible body language. - Paralinguistic Sync: When laughing, crying, coughing, or clearing the throat, are micro-expressions and body vibrations synchronized? 2.MUSIC : If <active_tags> contains "MUSIC_A" (On-screen Performance): - Rhythmic Alignment: Is the frequency of hand waving or playing consistent with the beat of the music? - Dynamic Match: Do the intensity and emotion of the playing action match the intensity of the music heard? - Instrument Consistency: Does the solo instrument heard appear correctly in the frame? 3.SOUND : If <active_tags> contains "SOUND_A" (On-screen Sound Effect): - Onset Synchronization: Do the sound burst points of transient events (impacts, drops, drum hits, cymbal strikes) align precisely with the frames where the actions occur? Is there significant lag? - Causal Correspondence: Do obvious sound-producing actions in the frame all have corresponding sounds? Does the sound type match the action type? [Dimension 2] AUDIO_QUALITY [Assess the technical quality and realism of the audio itself — based solely on the audio signal, excluding content semantics] Only evaluate items corresponding to <active_tags> 1.SPEECH If <active_tags> contains SPEECH (including on-screen and off-screen): - Pronunciation Accuracy: Is pronunciation clear and accurate? Are tones, consonants, vowels, and liaisons correct? Are polyphones correct? Is there slurring or mumbling? - Human-likeness and Naturalness: Is the intonation natural and fluid? Does it contain natural paralinguistic phenomena? (Breathing sounds, air sounds, pauses) - Continuity and Consistency: Are there unnatural breaks or sudden timbre changes in long sentences? Are the overall tone and speech rate natural and reasonable? - Audio Fidelity: Is the human voice timbre realistic? Is there mechanically stiff synthetic speech or distorted vocals with an electronic feel? 2.MUSIC If <active_tags> contains MUSIC: - Melodic Logic: Is the melodic line fluid and coherent? Is there meaningless random playing, severe off-key notes, or dissonant intervals? - Instrument Timbre Fidelity: Is the timbre of each instrument real and natural? Are there synthetic artifacts or timbre distortion? - Rhythmic Stability: Is the beat stable and consistent? Is there unintentional speeding up/slowing down or rhythmic chaos? - Audio Fidelity: Are there spectral defects, reverb distortion, or unnatural sound effect processing? 3.SOUND If <active_tags> contains SOUND: - Sound Texture Realism: Does the sound effect authentically reproduce the physical texture of the sound source? Are material characteristics (metal, wood, glass, drum skin, cymbal, etc.) accurate? - Physical Plausibility: Do the loudness, timbre, and duration of the sound effect conform to physical common sense? (e.g., drum transient sharpness, cymbal decay tail) - Audio Fidelity: Are there traces of over-compression, spectral loss, or unnatural sound synthesis? </dimensions> Stage A Eval Prompt <dimensions> [Dimension 3] EXPRESSIVENESS [Assess the expressiveness of audio in emotion conveyance, atmosphere creation, and narrative enhancement] Only evaluate items corresponding to <active_tags> 1.SPEECH : If <active_tags> contains "SPEECH_A" (On-screen Speech/Singing): - Emotional Expression Accuracy: Do tone, intonation, and volume changes accurately convey the speaker's emotional state (joy/sadness/anger/surprise, etc.)? - Narrative Contribution: Do the rhythm, pauses, and stress of the speech effectively drive the narrative process or enhance the expressiveness of the video? - Spatial Sense and Realism: Do the loudness and clarity match the perceived distance between the speaker and the viewer's perspective? If <active_tags> contains "SPEECH_B" (Off-screen Speech): - Emotional Expression Accuracy: As off-screen narration/voiceover/dialogue, is the emotional delivery compelling and believable on its own? - Narrative Contribution: Does the off-screen speech effectively complement, explain, or enrich the visual narrative? - Mixing Clarity and Presence: Is the off-screen voice mixed at an appropriate level clear and prominent without being drowned out by other audio elements? 2.MUSIC : If <active_tags> contains "MUSIC_A" (On-screen Performance): - Theme Adaptability: Are the musical style, rhythm, and mood highly compatible with the video theme and content? - Emotional Resonance and Narrative: Does the music successfully enhance the emotional impact of the visuals? Are transitions synchronized with the narrative rhythm? - Spatial Sense and Layering: Do the reverb and stereo field of the music match the spatial sense of the visuals? Is the multi-instrument arrangement layered? If <active_tags> contains "MUSIC_B" (Off-screen Background Music): - Theme Adaptability: Are the musical style, rhythm, and mood highly compatible with the video theme, scene, and overall tone? - Emotional Resonance and Narrative: Does the BGM enhance the emotional impact and pacing of the visuals? Do musical transitions align with scene changes? - Mix Balance and Layering: Is the BGM mixed at an appropriate volume level — supporting the visuals without overpowering speech or key sound effects? 3.SOUND : If <active_tags> contains "SOUND_A" (On-screen Sound Effect): - Atmosphere and Mood: Do sound effects (including percussion hits such as drum strokes and cymbal crashes) enhance the realism and immersion of the scene? - Narrative Assistance: Do sound effects effectively supplement visual information and enhance the coherence of the story? - Spatial Sense and Realism: Do the loudness and directionality of the sound effects accurately reflect the position of sound sources relative to the visual viewpoint? If <active_tags> contains "SOUND_B" (Off-screen Sound Effect): - Atmosphere and Mood: Do off-screen environmental sounds effectively establish or enhance the scene's atmosphere and emotional tone? - Narrative Assistance: Do off-screen sound effects provide meaningful contextual information beyond what is visible? - Environmental Coherence: Do the implied distance and acoustic properties of off-screen sounds feel consistent with the scene's setting? [Dimension 4] PROMPT_FOLLOWING [Assess the degree to which the AUDIO content of each specific type remains faithful to the textual description in <reference_prompt>] Only evaluate items corresponding to <active_tags> 1.SPEECH If <active_tags> contains "SPEECH": - Content & Keywords: Does the audio contain the dialogue, narration, keywords, or lyrics requested in the prompt? - Voice Characteristics: Does the speaker's gender, age, accent, and emotional tone match the text description? 2.MUSIC If <active_tags> contains "MUSIC": - Genre & Style: Does the musical style strictly follow the prompt? (e.g., "Jazz", "Electronic", "Classical"). - Instrumentation & Mood: Are the requested instruments present? Does the mood align with the text? 3.SOUND If <active_tags> contains "SOUND": - Event Accuracy: Do the specific sound events described occur? (This includes percussion events such as drum hits and cymbal strikes when tagged as SOUND.) - Material & Texture: Does the sound texture match the description? (e.g., Mismatched surface materials or impact mass, drum skin type, cymbal size). </dimensions> Stage B Eval Prompt Figure 9: Stage-specific prompt slots in the shared judging scaffold. M ’26, November 10–14, 2026, Rio de Janeiro, BrazilYuchen Sun et al. The human study covers outputs from three representative generation models, Sora 2, LTX-2, and Ovi. We select these models because they span different model families and performance levels in the main benchmark, allowing human alignment to be examined across a broader range of generation quality rather than within a single performance tier. Pair allocation is stratified by valid tag combinations rather than by random subset-level sampling. For each subset, we first enumerate the valid combinations of active tags and then sample four examples for each combination. The On-screen subset contains seven valid tag combinations, resulting in 28 pairs in total. The Off-screen subset follows the same design and also contains seven valid tag combinations, resulting in 28 pairs. The Mixed subset allows on-screen and off-screen tags to co-occur within the same sample and therefore contains 15 valid tag combinations, resulting in 60 pairs. This design ensures coverage of the subset-specific tag space without overrepresenting frequent patterns. Each task is presented as a blind GT-versus-model comparison. The two clips are assigned randomized identities, and each begins with a 0.5-second black frame containing a white numeric identifier. Annotators use that identifier only to map scores to Candidate 1 or Candidate 2; the identifier frame itself is ignored during quality judgment. As in the automatic pipeline, inactive audio types are not scored. The manual protocol mirrors the automated two-stage design. In Stage 1, raters watch the paired clips without reading the prompt and score only AV Coherence and Audio Quality. AV Coherence is restricted to on-screen audio and focuses on timing, causal matching, and visible-source consistency. Audio Quality assesses the realism and technical fidelity of speech, music, and sound regardless of meaning. In Stage 2, raters read the prompt and then score Expressiveness and Prompt Following, again only for the audio types that are present in the current sample. The score interpretation is identical to that used by the judge model: 9–10 for excellent, 7–8 for good, 5–6 for moderate, 3–4 for weak, 1–2 for very poor, and 0 for complete failure. Figure 10 shows the human annotation interface used in the alignment study. The interface presents the English prompt together with two blinded candidate videos displayed side by side, and organizes the four evaluation dimensions into stage-specific scoring panels. It also records sample metadata such as num, subset, and active tag category, while providing a discard option for invalid cases. PRISM-BenchMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Reference Prompt In a broad natural setting, a forest lake comes into view. On the left is a man wearing a black cap, a gray long-sleeved shirt, and jeans, holding a boom pole with a microphone aloft, with monitoring headphones on his ears, focused on capturing sound; next to him, another person, only partially visible, wearing a blue jacket and khaki pants, sits on a rock. In the middle of the frame, two people stand on a gravel surface: one in a red jacket and jeans, the other in a white top and dark pants; they appear to be interacting. In the foreground on the right, a man in a gray jacket with his back to the camera operates a professional video camera, clearly a member of the filming crew. The background is a dense forest of tall, dark green trees beside a body of water, with large rocks scattered across the ground and an overcast sky, creating the feel of an outdoor film shoot. video1video2 AV Coherence Audio Quality * SPEECH 9 * MUSIC / * SOUND 9 Expressiveness Prompt Following AV Coherence Audio Quality * SPEECH 5 * MUSIC / * SOUND 6 Expressiveness Prompt Following Discard this case? true false num 291 set Off screen cate SOUND_B,SPEECH_B › › › › › › › › Figure 10: Human evaluation interface used in the alignment study.