Paper deep dive
Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni models
Rohit Patel, Dieuwke Hupkes, Sloan Strader
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/28/2026, 3:35:31 AM
Summary
The paper introduces the Modality Maturity Index (MMI), a benchmark designed to evaluate the multimodal capabilities of large language models across five modalities (text, image, audio, video, and document) in both inputs and outputs. Unlike existing benchmarks that focus on bimodal understanding, MMI assesses the model's ability to choose correct output modalities and generate content within them. The benchmark consists of 893 self-contained prompts with human-authored rubrics. Evaluation of five frontier models (GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro/Flash, Llama 4 Maverick) reveals poor performance in modality presence, with Modality Presence Scores (MPS) ranging from 15.6 to 34.9. The paper also validates the use of LLM judges against human annotators, finding a 70.8% agreement rate.
Entities (18)
Relation Signals (18)
Modality Maturity Index → coversmodality → Image
confidence 99% · five modalities (text, image, audio, video and document)
Modality Maturity Index → coversmodality → Video
confidence 99% · five modalities (text, image, audio, video and document)
Modality Maturity Index → coversmodality → Document
confidence 99% · five modalities (text, image, audio, video and document)
Modality Maturity Index → coversmodality → Text
confidence 99% · five modalities (text, image, audio, video and document)
Modality Maturity Index → coversmodality → Audio
confidence 99% · five modalities (text, image, audio, video and document)
Modality Maturity Index → usesmetric → MMI Value
confidence 98% · a model's MMI Value expresses the average of the per-modality scores for each prompt.
Modality Maturity Index → usesmetric → Modality Presence Score
confidence 98% · we introduce also a supplementary Modality Presence Score (MPS), a per-prompt F1 over the expected output modalities.
Modality Maturity Index → createdby → Meta Superintelligence Labs
confidence 95% · Rohit Patel Affiliation: Meta Superintelligence Labs... Code: https://github.com/facebookresearch/modality-maturity-index
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Frontier language models are increasingly marketed as omni systems that can perceive and respond across modalities. Existing evaluation frameworks, however, focus almost exclusively on bimodal understanding, typically text plus one other modality. We propose the Modality Maturity Index (MMI), a benchmark designed to evaluate the multimodal capabilities of large language models across five modalities (text, image, audio, video and document) and combinations of up to three modalities in both inputs and outputs. MMI consists of 893 questions, each carefully crafted to require the model to demonstrate its understanding of multiple input modalities and to generate responses that incorporate various output formats. The questions are designed to be self-contained, with clear expectations for the correct modality or mix of modalities required for an accurate response. Every MMI prompt carries human-authored rubric criteria for each output modality expected in the response; a model's MMI Value expresses the average of the per-modality scores for each prompt. Because low scores can reflect either failure to generate a modality (lack of presence) or failure to generate correct content, we introduce also a supplementary Modality Presence Score (MPS), a per-prompt F1 over the expected output modalities. Applying MMI to five frontier multimodal models, we find that the MPS ranges from only 15.6 (Claude Opus 4.6) to 34.9 (GPT-5.4). Given the low availability of returned modalities to even grade, we report MPS as our main result pending model improvements. To assess the viability of judging output correctness with LLM judges and rubrics, we run a separate experiment with custom generation tools. On the assets that generates, we find that an LLM judge applying the rubrics agrees with rubric-blind human annotators (who score the outputs directly and never see the criteria) on 70.8% of judgments.
Tags
Links
- Source: https://arxiv.org/abs/2608.26317v1
- Canonical: https://arxiv.org/abs/2608.26317v1
Trouble viewing inline? Open PDF directly →
Full Text
87,718 characters extracted from source content.
Expand or collapse full text
Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni models Rohit Patel Affiliation: Meta Superintelligence Labs Dieuwke Hupkes Affiliation: Meta Superintelligence Labs Sloan Strader Affiliation: Meta Superintelligence Labs Abstract Frontier language models are increasingly marketed as multimodal, or even omni systems that can perceive and respond across text, image, audio, video and document inputs and outputs. Existing evaluation frameworks, however, focus almost exclusively on bimodal understanding, typically text plus one other modality. Few benchmarks probe how models combine multiple non-text modalities within a single prompt and even fewer evaluate whether models choose the correct output modalities. To address this gap, we propose the Modality Maturity Index (MMI), a benchmark designed to evaluate the multimodal capabilities of large language models across five modalities (text, image, audio, video and document) and combinations of up to three modalities in both inputs and outputs. MMI consists of 893 questions, each carefully crafted to require the model to demonstrate its understanding of multiple input modalities and to generate responses that incorporate various output formats. The questions are designed to be self-contained, with clear expectations for the correct modality or mix of modalities required for an accurate response. Every MMI prompt carries human-authored rubric criteria for each output modality expected in the response; a model’s MMI Value expresses the average of the per-modality scores for each prompt. Because low scores can reflect either failure to generate a modality (lack of presence) or failure to generate correct content, we introduce also a supplementary Modality Presence Score (MPS), a per-prompt F1 over the expected output modalities. Applying MMI to five frontier multimodal models, we find that the MPS ranges from only 15.6 (Claude Opus 4.6) to 34.9 (GPT-5.4). Given the low availability of returned modalities to even grade, we report MPS as our main result pending model improvements. To assess the viability of judging output correctness with LLM judges and rubrics, we do run a separate experiment in which we give models access to tools generating images, audio and video. On the assets that generates, we find that an LLM judge applying the rubrics agrees with rubric-blind human annotators (who score the outputs directly and never see the criteria) on 70.8% of judgments (Scott’s π of 0.41), ranging from 66.5% on image to 74.4% on video. These findings highlight the need for further research into the development of more effective multimodal models as well as benchmarks to properly evaluate true multimodality. †date: May 20, 2026†correspondence: mmipaper@rpeml.com or dieuwkehupkes@meta.com†Code: https://github.com/facebookresearch/modality-maturity-index†Dataset: https://huggingface.co/datasets/facebook/mmi 1 Introduction Recently released frontier LLMs are routinely marketed as being able to reason, plan, perceive and respond across multiple modalities (OpenAI, 2026; Anthropic, 2026; Google DeepMind, 2026; Meta Superintelligence Labs, 2026). However, the release reports of these same models scarcely cover all modalities in their benchmark tables: almost all of them omit one or more modalities, and none report results for benchmarks that explicitly require cross-modal output (see fig. 3). Furthermore, while there is a wealth of bimodal benchmarks assessing image or video understanding and to some extent generation (see, e.g. Zhang et al., 2025b; Zhang et al., 2025a, for overviews), mixed-modality and audio benchmarks (Xie et al., 2025; Li et al., 2024e, e.g.) are rare, and benchmarks that require fully cross-modal processing, combining multiple non-text modalities in both input and output, are virtually non-existent. subfigure [c]0.45 subfigure[c]0.52 Gemini 3 Llama 4 GPT-5.4 Claude 4 Text ✓ ✓ ✓ ✓ Image + Text ✓ ✓ ✓ ✓ Video + Text ✓ ✗ – – Audio / Speech ✓ – – – Cross-modal ✗ ✗ ✗ ✗ Figure 1: Per-model precision, recall and F1 (MPS) on MMI. Figure 2: Modality coverage of benchmarks reported in recent frontier model release notes. Figure 3: Summary model performance and benchmark coverage of frontier models. a) Average per-prompt precision, recall and F1 (the MPS) under lenient detection. F1 is low across the board, with even the best-performing model (GPT-5.4) not surpassing an F1 of 35 on returning the correct modality. b) Modality coverage of the evaluation benchmarks reported in recent frontier model release notes; no release reports cross-modal output benchmarks. In this paper, we propose the Modality Maturity Index (MMI), the first benchmark we are aware of that directly assesses the multimodal capabilities of LLMs across multiple modalities and combinations of modalities, in both inputs and outputs, and in particular the first to evaluate whether models choose the correct output modality to respond to a self-contained user request. MMI consists of 893 questions, carefully crafted to require understanding of various input modalities and to elicit responses that incorporate multiple output formats. It encompasses five different modalities (text, image, audio, video and document) and spans prompts with up to three input and up to three output modalities. The MMI prompts are designed to be self-contained, with clear expectations for the correct modality or mix of modalities. All MMI prompts furthermore come with human-authored rubric criteria, each tied to one of the output modalities expected in the response. Scoring MMI A prompt’s MMI Value is the macro-average of the rubric scores over the output modalities expected in the response: we first average the criterion scores within each expected output modality, then average those per-modality means, so that every expected output modality counts equally regardless of how many criteria were written for it. The MMI Value conflates two different failure modes: failing to return an asset for a particular modality, and returning an asset with the correct modality that does not answer the question correctly. To separate these two cases, we also introduce a second score, the Modality Presence Score (MPS), a per-prompt F1 over the expected output modalities, which measures production alone. The full definitions of both MMI Value and MPS can be found in section 3.4. Testing a set of five multimodal models, namely GPT-5.4 (OpenAI, 2026), Claude Opus 4.6 (Anthropic, 2026), Gemini 3.1 Pro (Google DeepMind, 2026), Gemini 3.1 Flash (Google DeepMind, 2025), and Llama 4 Maverick (Meta AI, 2025), we find that all of them struggle to both ingest modalities and return the right modalities at all. The MPS ranges from only 15.6 (Claude Opus 4.6) to 34.9 (GPT-5.4): even ignoring answer correctness entirely, model performance is staggeringly poor. As computing the MMI Value on the few returned assets would not yield a meaningful extra signal, we report MPS as our main result, pending stronger future models. To validate the rubrics, we instead run a second experiment, where we give three models access to tools they can use to generate images, videos and audio (sections 4.3 and 5.4). This raises modality presence substantially, yielding gradeable answers for 1,499 output modalities (71%, or 76% excluding documents, for which we did not provide a model tool). We find that an LLM judge using the rubrics agrees with a human annotator on 70.8% of the 1,499 judgments this yields (Scott’s π of 0.41), ranging from 66.5% on image to 74.4% on video. Our analysis attributes the gap to judge errors, the difficulty of writing criteria for open-ended requests, and the difference between a rubric-driven judgment of a single modality and an unaided one made with the full response available (section 5.4). Contributions In sum, our main contributions are: • A cross-modal dataset. 893 self-contained prompts spanning five modalities (text, image, audio, video and document), with up to three modalities in both input and output, and human-authored per-modality rubrics for every prompt. To our knowledge, this is the first dataset built to test whether a model chooses the correct output modality, and then assess the quality of generated assets. • An index over that dataset, and a diagnostic for it. The MMI Value grades a response against the prompt’s rubrics. The MPS decomposes it, scoring modality presence as a per-prompt mean F1, complemented by precision, recall, pass rate and input failure rate. Both are implemented in the released harness. • An evaluation of five frontier models, and of the index itself. The MPS ranges from 15.6 to 34.9, with failures predominantly recall-driven: audio is by far the hardest output modality, and most models default to text even when the prompt clearly calls for otherwise. We validate the rubrics directly, reporting 70.8% rubric-blind judge–human agreement. Outline In the rest of this paper, we first review the related work (section 2), then describe MMI (section 3), followed by a brief section on our experimental setup and evaluation (section 4) and our results (section 5). We conclude in section 6. 2 Related work The rapid expansion of LLM evaluation into the multimodal domain has been accompanied by a proliferation of benchmarks, recently surveyed in Zhang et al. (2025b) and Zhang et al. (2025a). We organize the most relevant prior work by the modality combinations they target, and then position MMI with respect to the small number of existing cross-modal benchmarks. Image-text understanding. The bulk of multimodal evaluation effort has gone into vision-language understanding. General-purpose suites such as MME (Fu et al., 2023), MMBench (Liu et al., 2024a), M-Vet (Yu et al., 2024), SEED-Bench (Li et al., 2024b), MMStar (Chen et al., 2024) and MMMU (Yue et al., 2024; Yue et al., 2025) probe a broad set of perception and reasoning skills over (mostly) single-image inputs and short-form text answers. More targeted benchmarks evaluate specific capabilities, e.g. multi-image and visual perception via BLINK (Fu et al., 2024b), mathematical reasoning via MathVista (Lu et al., 2024b), or instruction-following granularity via MIA-Bench (Qian et al., 2025). A more recent line of work targets the egocentric, wearable-device setting, where image quality and framing differ substantially from curated third-person benchmarks: WearVQA (Chang et al., 2025) evaluates VQA on smart-glasses imagery that may be occluded, poorly lit or blurry, and CRAG-M (Wang et al., 2025b) extends this to multi-turn, retrieval-augmented question answering over wearable image inputs. While extensive, all of these benchmarks share a fundamentally bimodal setup: a (potentially complex) visual input is coupled with a textual question and a textual answer. Video-text understanding. Several benchmarks extend the image-text setup to video. Video-MME (Fu et al., 2024a) provides a broad-coverage suite over short, medium and long videos; MVBench (Li et al., 2024d) focuses on temporal reasoning skills; MMBench-Video (Fang et al., 2024) extends the MMBench setup to long-form, multi-shot videos; TempCompass (Liu et al., 2024b) stress-tests fine-grained temporal understanding; EgoSchema (Mangalam et al., 2023) targets long-form egocentric video question answering; and Perception Test (Pătrăucean et al., 2023) measures perceptual abilities across video and audio. As with the image-text suites, these benchmarks evaluate text-only outputs in response to a video (and sometimes audio) input. Audio-text understanding. Audio capabilities have historically been evaluated through narrow speech-only suites, but recent work has produced more general audio-language benchmarks. AudioBench (Wang et al., 2025a), AIR-Bench (Yang et al., 2024) and MMAU (Sakshi et al., 2024) cover varying combinations of speech, music and environmental sound understanding via instruction-following or QA formats, while Dynamic-SUPERB (yu Huang et al., 2024) provides a broad collaborative instruction-tuning benchmark for the speech subset. Outputs in all of these benchmarks are textual. Document understanding. Document inputs - mixing layout, tables, figures and text - are evaluated by DocVQA (Mathew et al., 2021), MP-DocVQA (Tito et al., 2023), SlideVQA (Tanaka et al., 2023) and the long-document benchmark MMLongBench-Doc (Ma et al., 2024). These remain text-output benchmarks over single-document inputs. Multimodal generation. A complementary line of work evaluates models on producing non-text outputs, almost exclusively images. T2I-CompBench (Huang et al., 2023) and GenAI-Bench (Li et al., 2024a) target compositional text-to-visual generation, while MMGenBench (Huang et al., 2024) evaluates LMMs on a text-to-image generation pipeline (asking the model to produce an image-generating caption from a reference image). With few exceptions, generation benchmarks isolate a single output modality and presuppose that the model is a generator, rather than asking whether a model that could generate that modality actually chooses to do so when relevant - a question MMI directly evaluates (see sections 3 and 4). Cross- and any-to-any benchmarks. The benchmarks closest in spirit to MMI are those that consider multiple input and output modalities. OmniBench (Li et al., 2024e) jointly evaluates audio, image and text understanding but produces text outputs only. MME-Unify (Xie et al., 2025) couples a multimodal understanding suite with a single-modality generation suite (image generation) and a small set of generation-then-understanding “mixed” tasks. MixEval-X (Ni et al., 2025) aggregates many existing single-modality benchmarks under a unified protocol but does not test cross-modal output decisions within a single prompt. GIM (Patel et al., 2026) similarly reflects a broader move toward harder, more integrative evaluation tasks, but its difficulty comes from coordinating cognitive operations over accessible knowledge rather than from selecting and producing among multiple output modalities. WorldSense (Hong et al., 2026) and OmniMMI (Wang et al., 2025c) target streaming and world-grounded video+audio understanding, again with text outputs. Outside of academic benchmarks, frontier model release notes (OpenAI, 2026; Anthropic, 2026; Google DeepMind, 2026; Meta AI, 2025) typically report only a subset of the bimodal benchmarks listed above and do not report any results that require choosing the correct output modality. Any-to-any and omni models. A parallel line of work has produced models architected for cross-modal input and output, including AnyMAL (Moon et al., 2023), NExT-GPT (Wu et al., 2024), CoDi (Tang et al., 2023), Macaw-LLM (Lyu et al., 2023), Unified-IO 2 (Lu et al., 2024a) and Chameleon (Chameleon Team, 2024). These models motivate the need for evaluations like MMI: as the space of ‘omni’ models grows, there is a corresponding need for benchmarks that probe whether such systems actually choose the appropriate output modality, rather than only asking whether they can produce a given modality in isolation. Tool use and output-format choice. A loosely related thread is tool-use evaluation, in which the model must decide which tool (or output channel) is appropriate for a given user request, e.g. the Berkeley Function Calling Leaderboard (Yan et al., 2024). Output-modality selection in MMI can be viewed as a multimodal analog: rather than choosing a function call, the model must choose the right combination of asset modalities to deliver in its response. Positioning of MMI. MMI differs from existing benchmarks along two axes. First, it spans five modalities - text, image, audio, video and document - in both inputs and outputs, with up to three modalities composed within a single prompt; we are not aware of any existing benchmark that covers this combinatorial space. Second, MMI evaluates both whether a model that is marketed as multimodal produces the set of output modalities a self-contained user request calls for, and whether what it returns in each of those modalities actually answers the question. Human-authored rubric criteria, tagged per output modality, supply the latter judgment, and the MMI Value averages them across the expected output modalities; the MPS isolates production alone and bounds the MMI Value from above. Table 1 summarizes the modality coverage of the academic benchmarks discussed above and contrasts it with MMI; fig. 3 provides the complementary view of benchmarks reported in frontier model release notes. Inputs Outputs Cross- Benchmark T I A V D T I A V D modal out? MME, MMBench, MMMU (Fu et al., 2023; Liu et al., 2024a; Yue et al., 2024) ✓ ✓ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ Video-MME, MVBench (Fu et al., 2024a; Li et al., 2024d) ✓ ✗ ∼ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ AudioBench, MMAU (Wang et al., 2025a; Sakshi et al., 2024) ✓ ✗ ✓ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ DocVQA, MMLongBench-Doc (Mathew et al., 2021; Ma et al., 2024) ✓ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ T2I-CompBench, GenAI-Bench (Huang et al., 2023; Li et al., 2024a) ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ OmniBench (Li et al., 2024e) ✓ ✓ ✓ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ MME-Unify (Xie et al., 2025) ✓ ✓ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ∼ MixEval-X (Ni et al., 2025) ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✗ ∼ WorldSense, OmniMMI (Hong et al., 2026; Wang et al., 2025c) ✓ ✗ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ MMI (ours) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Table 1: Modality coverage of related academic benchmarks. For each benchmark, we indicate whether it includes Text (T), Image (I), Audio (A), Video (V) and Document (D) on the input and output side, and whether it evaluates the model’s choice of output modality across multiple non-text options (“Cross-modal out?”). ✓ indicates support, ✗ indicates no support, and ∼ indicates partial support (e.g. a separate single-modality generation track that is not jointly evaluated with the understanding side). LLM judges in evaluation. Both halves of our scoring pipeline use an LLM judge, and we position that choice against a now-substantial literature. LLM-as-a-judge has become standard practice in open-ended evaluation since Zheng et al. (2023), with subsequent work documenting both its reliability and its biases (Li et al., 2024c, see e.g.). How much weight the judge carries differs between our two uses of it, and we return to each in section 4. 3 Modality Maturity Index: the benchmark Prompt Input Output What’s the average salary for my employees? Document Text Omelet recipe, can you talk me through the steps as I go? Image Audio Can you help me remove the people from this photo and then turn it into a painting? Please make the sky pink! Text, Image Image Can you tell me what animal this is and pronounce its name slowly? Audio, Video Audio I would like to recreate this look. Can you list out all the products that I will need and write out a very detailed instruction in a .txt file? Text, Audio, Video Document Can you show me how to make soft chocolate chip cookies? A list of ingredients and a tutorial video would be nice. Text Text, Video My brother sent me this image of his car wheel. Can you help me understand how to read and understand tire sizes? Provide a brief explanation and annotate the picture so I can understand what the size markings on the tire are. Text, Image Image, Text Explain the science behind convergent plate tectonics, including a 2D diagram and a video simulation of the plates shifting. Text Image, Text, Video This audio captures a doctor giving important pre-surgery instructions. List some positive things a patient could do before the surgery. Can you also create a video that visually encourages them to stay positive and prepared? Create a printable PDF summary of instructions for patients to refer to later. Audio Document, Text, Video Table 2: Example prompts from the benchmark, showing the input assets provided to the model and the desired output modalities. In total, the MMI benchmark contains 893 prompts, covering five different modalities, each with a set of human-written rubrics for each desired output modality. While the largest subset of the collected prompts has one input and one output modality, almost half of the benchmark comprises prompts that have multiple input or output modalities. All prompts are in English. In this section, we describe the distribution of prompts over (number of) modalities (section 3.1), provide more details on the data collection and validation process (section 3.2), discuss the rubrics used to assess correctness (section 3.3), and define the MMI Value together with the MPS, the modality-presence diagnostic that bounds it from above (section 3.4). 3.1 MMI prompts The five different modalities covered in the MMI prompts are text, audio, image, video and document11 1 The 152 documents in MMI cover six file types: .pdf (63 prompts), .docx (48), .xlsx (12), .pptx (11), .csv (8) and .txt (10). . As can be seen in fig. 7, those modalities are more or less evenly covered in the prompts’ outputs, with a slight overrepresentation of the text modality, which often accompanies other modalities for clarification or explanation. In the prompts’ inputs, this overrepresentation of text as a modality is even more pronounced (see fig. 7). As language is the most natural way for humans to express questions or requests, most prompts with modalities besides text or audio contain text or audio alongside them. In fig. 7, we show how many input and output modalities the MMI prompts have. In the top left corner of that figure, we see that the bulk of the prompts have one input and one output modality. As text remains one of the most common input modalities, 249 of the one-input/one-output prompts have text as input and one of the five modalities as output. The remaining 226 one-input/one-output prompts are distributed over all the other possible input-output modality combinations. From the one-input/one-output modality prompts, we gradually increased first the number of input modalities, and then the number of output modalities up to three different modalities. In total, of the 893 prompts, 475 are one-input/one-output, 264 have multiple inputs and a single output, 76 have a single input and multiple outputs, and 78 have multiple inputs and multiple outputs. A complete picture of coverage across different in- and output modalities can be found in fig. 17. The prompts themselves can be inspected using the viewer implemented in our github repository or on HuggingFace; some representative examples can be found in table 2. subfigure [b]0.35 subfigure[b]0.35 subfigure[b]0.29 Figure 4: Figure 5: Figure 6: Figure 7: Benchmark modality statistics. a) Number of times each modality occurs in the input of a MMI question. b) Number of times each modality occurs in the output of a MMI question. c) Number of in- and output modalities across MMI questions. 3.2 Data collection We created the data for MMI in three stages. First, we piloted the data collection by creating a small set of 50 questions with text as input and a single modality in the output. This initial set was created by an internal Meta content team, from the broad instruction to generate questions that would require different modalities to optimally answer them, where the correct output modality should be clear and unambiguous but not explicitly mentioned (e.g. avoid questions like ‘give me a video with…’). Through several iterations of manual inspection by the authors, we guided the content team to create a varied set of 500 questions with one input and one desired output modality. Using the lessons learned from the first batch of questions, in the next two stages we created questions that have interleaved modalities only in the input, and in both input and output, respectively. Like in the first stage, we created both these sets starting from a small set of questions to understand potential issues with the collection protocol. We then manually reviewed and iteratively refined and built upon them to ensure correctness, diversity in question types and coverage across modalities. Each of the questions in the final set of 893 questions has been reviewed manually by at least two of the authors, checking specifically for clarity and correctness of the desired output modalities.22 2 We did not correct minor issues such as typos that did not affect the clarity of the prompts or expected outputs, as such issues likely also exist in real-world user prompts. Disagreements were resolved via discussion. In the rubric creation round (see section 3.3), about 30 gold modalities were adjusted based on input from the annotators writing the rubrics. The content team consisted of full-time Meta employees compensated as part of their regular employment; all prompts and assets are released. All assets in the dataset are either generated with MetaAI or created by members of the content team using their phones. In some cases, we regenerated images or videos taken by the content team with MetaAI. While the quality of those images and videos is not always comparable to the originals they replaced, we ensured that they would not make answering the question impossible (e.g. because the model is asked to analyze a non-coherent video). 3.3 Rubrics For each prompt, we also collected a set of rubrics for each of the modalities desired to correctly answer the prompt. These rubrics allow to assess whether the content of a returned modality is correct, rather than only whether that modality appeared. They are written to facilitate judging of each output modality in isolation by an LLM judge or human, not requiring access to the question, external knowledge or the other modalities returned by the model.33 3 For some prompts, fully judging correctness would require assessing several output modalities jointly, for instance checking that generated audio and video are aligned. We do not attempt such judgments, as this would require processing even more combinations of modalities than answering the benchmark prompts themselves. We nonetheless supply the judge with the prompt and its input assets for context. All rubrics are written by an external vendor; the annotation guidelines can be found in appendix D.44 4 The guidelines target 4–12 criteria per output modality. In the released set of 7,165 criteria across 1,099 (prompt, modality) groups, the median group has 6 criteria and 71.5% fall in the target band; 10.8% have fewer than three and 6.0% more than twelve. We did not enforce the target during collection. In section 5.4, we report judge–human agreement between judges using the rubrics and humans annotating outputs without a rubric. Example rubric set (desired output modalities: video, image, audio) “I’m learning to crochet. Generate a clip of different patterns to use and how to manuever the fingers. Have some still images of the patterns for reference. Also, how do you pronounce crochet?” 1. (video) Provides a video clip for learning crochet patterns and finger movements. 2. (video) Shows multiple crochet patterns in the video. 3. (video) Demonstrates how to maneuver the fingers, yarn, and hook during crocheting in the video. 4. (image) Provides images of the crochet patterns. 5. (audio) Pronounces the word crochet correctly as kroh-SHAY. 3.4 Scoring MMI The MMI Value is main score of the benchmark: it quantifies the extent to which a model answered the question correctly. The MPS instead focuses only on modality presence, measuring only whether the correct set of output modalities is returned rather than if their content is correct. Both are implemented in our github repository. Below, we discuss them in more detail. 3.4.1 The MMI Value The MMI Value quantifies overall model correctness, assigning equal weight to each modality. To circumvent the need for human annotators, we compute MMI Value using an LLM judge, who grades the answers based on a set of modality-specific rubric criteria (section 3.3), A criterion is graded in isolation, meaning the judge sees the model’s output for that one modality and nothing it produced for the others, and returns a score in [0,1][0,1]. A prompt’s MMI Value is macro average of the scores for each desired output modality; a model’s MMI Value is the average of those per-prompt values. If a desired modality is absent in a model’s output, it’s score is set to 0. This makes the MMI Value little informative for models that return very few assets: a low score does not tell us whether the answers for returned modalities are incorrect or simply missing. We therefore complement it with a second score that isolates the missing-modality component. 3.4.2 The MPS and complementary metrics The MPS quantifies only how often the correct modalities are returned, ignoring entirely whether they are correct or not.55 5 How we determine which modalities a response contains is described in section 4.2. We define MPS as macro-average of the F1 between the set of modalities a model returns for an MMI prompt and the gold set of modalities required for the prompt. For completeness, we also report precision, recall and a separate pass rate which indicates whether all expected modalities are present in the model response.66 6 Unlike recall, pass rate is a binary per-prompt metric: it is 1 if and only if all desired modalities are present, and 0 otherwise. Where recall tells us how often the desired modalities are included in the model’s response, precision indicates how often the modalities returned by a model are in fact desired. In computing precision, we do not penalize undesired text, so points for text are essentially ‘free’. Some models cannot process some of the input modalities in the prompt set (table 3); the provider rejects the call and the prompt is scored zero, because being unable to ingest a modality is itself a deficiency in handling it. We report how often this happens as the input failure rate, in the last column of table 4. 4 Experimental setup We run two experiments. First, in our main evaluation (sections 4.1 and 4.2), we apply MMI to five models marketed as multimodal, under the benchmark’s normal protocol, and report MPS scores. As this experiment returns very few gradeable assets, we run a second experiment, with the sole purpose of validatiing the rubrics (section 4.3). This rubric-validation run is a separate experiment on three different models given generation tools; its purpose is to produce enough gradeable assets to put the rubrics to the test, and it is not a model comparison.77 7 Given the different purposes of these experiments and the different settings, the numbers of these experiments are thus not comparable. 4.1 Main evaluation: models and generation settings The five models we evaluated are GPT-5.4 (OpenAI, 2026), Claude Opus 4.6 (Anthropic, 2026), Gemini 3.1 Pro (Google DeepMind, 2026), Gemini 3.1 Flash (Google DeepMind, 2025) and Llama 4 Maverick (Meta AI, 2025). The first four were run through their respective provider APIs. Llama 4 Maverick is open-weight and has no first-party API of the same kind, so we served the Llama-4-Maverick-17B-128E-Instruct-FP8 checkpoint ourselves on local hardware and queried that deployment instead. All models were run with the model’s default parameters, except for the generation limit, which we set to 32,768 tokens throughout.88 8 Provider calls use retry with transient-error keyword matching (e.g. rate limit, 5x, timeout-related signals), with a maximum of five retries, and a runner-level request time-out of 300 seconds per call. If retries are exhausted or a non-retryable exception occurs, the prompt is recorded as a failure and counts toward the input failure rate of table 4. 4.2 Detecting modalities in model outputs We have three modes of detecting whether a model outputs a particular modality. First, we look directly at the assets a model returns. When a model responds with an asset of the correct modality, we mark that modality as natively generated. Sometimes, models return links to assets of the right modality, which we detect using regexes for platforms that host assets of that specific modality, such as YouTube for videos or Flickr for images. If nothing is detected using either of these two options, we pass the answer to Gemini 3 Flash and ask it to judge whether an asset of a particular modality was generated.99 9 Model ID gemini-3-flash-preview. The same judge model is used for the rubric-validation run in section 4.3. The full prompt can be found in appendix A. These three routes give us two standards for counting a modality as present. Under the strict standard, only natively returned assets count: a modality is present if and only if the model itself produced an asset of that type. Under the lenient standard, a modality counts as present if any of the three routes fires: a returned asset, a link to an asset of that modality, or a fallback judge detection. Unless stated otherwise, every presence number we report is lenient, including the MPS values in table 4 and fig. 3; the sole exception is fig. 12, which reports strict (native) precision. We take lenient as the default because it reflects what a user actually receives: a working link to a video of the right content answers the request, whether or not the model rendered the pixels itself. Note that this also implies that even models that do not have the ability to natively generate specific modalities (e.g. video) can still answer questions that require that modality as output correctly by responding with a link to a pre-existing asset. Whereas for some questions this may not be appropriate (e.g. if the prompt asks to edit a video), for others it is instead the desired mode of responding (e.g. when a prompt asks for a video of an interview with an existing person). Text Image Audio Video Document Model In Out In Out In Out In Out In Out GPT-5.4 ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ∼ ✗ Gemini 3.1 ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ Claude Opus 4.6 ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ∼ ✗ Llama 4 Maverick ✓ ✓ ✓ ✗ ✗ ✗ ✓ ✗ ∼ ✗ Table 3: Natively supported modalities. Modalities natively supported by the different models. ✓ indicates the model supports the modality, ✗ indicates it does not; ∼ indicates that the modality is partly supported, e.g. a model accepts .txt and .pdf documents, but not .docx. 4.3 Rubric-validation run: tool scaffolding Because the models in our main evaluation produce so few assets, their answers are not sufficient to judge if our rubrics grade well. We therefore set up an additional experiment in which we give three models access to tools that help them generate audio, image and video, respectively. The purpose is not to improve model performance or to rank models, but to increase coverage across output modalities so that the rubrics can be exercised. The models that we use for this experiment are Claude Opus 4.8, Gemini 3.5 Flash and GPT-5.4. Each of them is given three tools, named image_gen, audio_gen and video_gen, that are exposed as generic function calls: each takes a single free-text prompt argument and carries an identical, uninformative description, so that the only signal about a tool is its name and the modality that name implies. Behind that interface we route the prompt to a backend generator (Gemini 3.1 Flash Image for images, Gemini 2.5 Flash TTS for audio and Veo 3.1 Fast for video), capture the returned asset, and hand it back to the calling model. Tool use runs for at most three rounds per prompt: in each round we dispatch every tool call the model requested, return the resulting assets, and let it continue, stopping early once it requests no further tools. A round may contain several calls, so a prompt can yield more than three generated assets; in practice the models requested tools on 189–502 of the 893 prompts and issued a median of one to three calls when they did. The calling model is never told which backend it is addressing, what that backend is good at, or how to prompt it, limiting any benefit from recognizing a familiar tool API or from backend-specific instructions in the tool description. We use the same generation settings as in the main evaluation (32,768-token limit, up to 5 retries), with an extended 900-second request time-out to accommodate slower video and audio generation. We then collect all assets and text and ask both human annotators and an LLM judge (Gemini 3 Flash) to provide per-modality judgments on the correctness of the responses. The LLM judge uses the human-written rubrics, grading one criterion at a time with the prompt given in appendix B, while the humans judge the answers without rubric (annotation instructions are provided in appendix E). We use the resulting agreement between humans and judges to quantify how well LLM judges can approximate MMI Value given the rubrics. 5 Results Following the division set out in section 4, sections 5.1, 5.2 and 5.3 report the main evaluation of five models and section 5.4 the rubric-validation run. In table 4, we report a summary table with all aggregate metrics we computed for the five main-evaluation models, sorted by MPS (per-prompt mean F1). We will discuss these metrics as well as further breakdowns along specific axes in the subsequent sections. Model F1 Recall Precision Pass rate Input failure rate GPT-5.4 34.9 33.8 99.8 31.0 30.1% Gemini 3.1 Flash 27.2 25.4 99.4 20.3 8.2% Gemini 3.1 Pro 25.4 23.6 99.7 18.6 8.0% Llama 4 Maverick 16.3 15.1 100 12.0 36.5% Claude Opus 4.6 15.6 14.5 100 11.3 37.0% Table 4: Summary model scores. Per-model sample-level averages of F1 (the MPS), recall, precision, pass rate and input failure rate. Models are sorted by MPS. 5.1 Overall presence scores In fig. 3, we show the overall MPS (F1), precision and recall for all models. The precision of all models is at or near 100, indicating that, apart from text, models rarely generate any modality that was not in the list of expected modalities. The models’ recall is substantially lower: GPT-5.4, Gemini 3.1 Flash, Gemini 3.1 Pro, Llama 4 Maverick and Claude Opus 4.6 obtain recall scores of 33.8, 25.4, 23.6, 15.1 and 14.5 respectively, with no model exceeding 35. Because precision is this high, the MPS stays within two points of recall for every model, so on this set of models the two support the same conclusions. subfigure [b]0.30 subfigure[b]0.21 subfigure[b]0.21 subfigure[b]0.21 Figure 8: F1 Figure 9: Recall Figure 10: Precision Figure 11: Precision native Figure 12: Performance by model and output modality. We show F1, recall and precision per model and output modality. Apart from text, GPT-5.4 is the only evaluated model to natively generate another desired modality (images), though the models do sometimes return links to images and videos. 5.2 Input failure rate and pass rate The input failure rates in table 4 vary substantially across models, from 8.0% for Gemini 3.1 Pro to 37.0% for Claude Opus 4.6. Read against recall, the input failure rate separates two distinct deficiencies: a model that cannot accept the input at all, and one that accepts it and then returns the wrong output modality. Pass rate provides the complementary prompt-level view, asking whether a response contains every expected output modality rather than averaging success across them. It falls from 31.0 for GPT-5.4 to 11.3 for Claude Opus 4.6 and is consistently below recall, showing that partial fulfillment is common: models may return one expected output modality while omitting another. Like recall, pass rate counts an input failure as a failure, so it does not by itself separate requests a model cannot accept from those it accepts but only partly fulfils; that separation comes from reading it against the input failure rate. 5.3 The impact of output modality In fig. 12, we show F1, recall, precision and native precision per output modality. To compute these values, we average across the entire dataset for each modality, rather than averaging per-prompt scores as for the main MPS. E.g. video recall indicates the percentage of times a model returns a video when a video was a desired output modality, whereas video precision indicates the percentage of times a returned video was in fact in the correct set of output modalities. When a model never returns a specific output modality, we set precision to NaN rather than 0; NaN cells are excluded from any cross-modality averages. As none of the models supports all output modalities natively, recall is expectedly low across the board. Notably, however, not all F1 and recall values are zero for unsupported output modalities. As can be seen in fig. 13, image and video credit comes from models returning links to existing assets, whereas GPT-5.4’s document credit comes almost entirely from the fallback judge recovering generated files that asset- and URL-based detection missed. Because so few modalities are natively supported, it is hard to say which modalities are more difficult. As can be seen in fig. 12, GPT-5.4 is the only model that ever natively generates any modality besides text. The per-modality F1, precision and recall therefore mostly reflect the (often small) fraction of cases where a model is able to return the modality at all, rather than a true measure of its difficulty. Audio, not supported as an output modality by any of the models, is the most difficult modality across the board: no model generates even links to audio snippets. In sum, the plots show clearly that there is much room for improvement for any non-text modality, exemplifying the need for more benchmarks such as MMI. Figure 13: Detection method per modality. Models can return assets of the right modality natively or via a URL. As a fallback, we ask a judge to parse the answer and detect whether an asset of the correct modality is returned. 5.4 Rubric validation results The main-evaluation models produce too few assets for their MMI Values to be worth grading (section 3.4.1), so we validate the rubrics on the tool-scaffolded run of section 4.3 instead. The scaffolded models produced an output for 71% of the modalities, or 76% once documents, for which we provided no tool and which are never produced, are set aside, against about a quarter of expected modalities in the main evaluation. For completeness, we record the MMI Values these three runs obtain in appendix F. We compare the two graders on the 1,499 (prompt, model, modality) triples for which both returned a judgment and the model actually produced the modality. The two graders do not natively produce the same quantity: the humans produce a per-modality binary judgement (see appendix E for their annotation guidelines), the judges provide judgements for each rubric criterion, resulting in graded judgements for each modality. To align the two types of judgements, we collapse the judge grades to a binary judgement, assigning a score of 1. if all rubric criteria for a modality are graded 1. and 0. otherwise.1010 10 This is the strictest choice possible, but most aligned with the task of the humans. Empirically, it does not appear to influence the number negatively in a significant way: a sweep over the collapse threshold resulted in a maximum agreement of 73.2% for t=0.8t=0.8 and Scott’s π at 0.43 for t=0.9t=0.9. Because the judge is itself a Gemini model and grades a plurality of assets from its own family, we cannot rule out a self-preference effect (Li et al., 2024c, see e.g.); the agreement figures should therefore not be read as judge-independent. Following Thakur et al. (2025), we compute both percent agreement and Scott’s π between the human and LLM judgements. In fig. 16, we can see that the judge and the human agree on 70.8% of judgments overall, ranging from 66.5% on image to 74.4% on video. Scott’s π is 0.41 overall (last column of fig. 16), lower because it normalizes for chance agreement: as can be inferred from the last row of fig. 16, both the LLM judge and the humans mark between 50 and 60% of outputs correct, putting their chance agreement near 50%. As document is missing entirely from the generated assets, the document rubrics unfortunately remain unvalidated. subfigure [c]0.48 Modality N % Agree π Text 519 70.9 0.34 Image 415 66.5 0.28 Audio 284 73.6 0.45 Video 281 74.4 0.31 Overall 1499 70.8 0.41 subfigure[c]0.35 Figure 14: Figure 15: Figure 16: Judge–human agreement on MMI correctness a) Per modality judge–human agreement (% agreement and Scott’s π) between the LLM judge and human annotators. b) Per modality breakdown of true and false positives and negatives. Document is absent from both panels: we provided no document-generation tool, so no model in this experiment produced a document output and the document rubrics could not be exercised (section 5.4). We conduct a small analysis of the differences between human and LLM judgments, looking at 20 samples in which the human assessment was ‘correct’ and the judge ‘incorrect’, and vice versa. In the former category, cases in which the LLM marks a sample incorrect while the human marks it correct, a few cases are LLM judgment errors (e.g. stating that a piece of text has three paragraphs while in fact it has two plus a title), and a few involve prompts where correctness is somewhat subjective. The rest, rather than being real errors, mostly illustrate the difficulty of writing rubrics for complex requests that precisely describe when an answer is correct, especially when there are multiple different ways of correctly answering the request. The latter category, cases in which the LLM marks a sample correct while the human marks it incorrect, paints a clearer picture. Here too, some cases are judge mistakes (e.g. stating that an audio file is 20 seconds while it is in fact 30). However, most involve prompts that request an edit or adaptation of an input image or video, where the response does not adhere to the original image or video. The judge does receive the prompt’s input assets, so this is not a matter of missing context. 6 Conclusion Prompted by the lack of cross-modal benchmarks for LLMs, in this paper we propose the Modality Maturity Index (MMI), a multimodal benchmark with 893 prompts spanning up to three out of five possible in- and output modalities. The MMI prompts aim to comprise a diverse set of possible user questions, covering different topics and modality combinations in input as well as output. Our main evaluation reports the MPS, a score quantifying how often models return the correct set of modalities. This metric shows that even without considering the correctness of the answers, MMI is challenging for most models, with the MPS (per-prompt mean F1) ranging from 15.6 (Claude Opus 4.6) to 34.9 (GPT-5.4). We analyze the difficulty of various output modalities and study the difference between model precision and recall. We find that low scores are mostly recall driven: many modalities are simply not supported by some of the models, and even if they are supported, they are not always used correctly. The most challenging modality is audio, which none of the evaluated models supports as an output and which is not returned for any prompt that requires it. Image and video receive higher scores, but are still very challenging on the output side even for the highest scoring model. Because these models return so few assets, we validate the rubrics behind the MMI Value in a separate experiment. We find that an LLM judge using the rubrics agrees with a human annotator on 70.8% of judgments (Scott’s π of 0.41), ranging from 66.5% on image to 74.4% on video. Our analysis indicates that the remaining gap comes from judge errors, from the difficulty of specifying correctness for open-ended requests, and from the difference between a rubric-driven judgment of a single modality and an unaided human one. We expect agreement to go up as judges get better. The results on MMI paint a stark picture: despite being marketed as multimodal or even ‘omni’ models, current LLMs remain predominantly text-output in practice. The gap between what is advertised and what is delivered is substantial, not because models confuse modalities, but because they fail to respond with non-text modalities even when the question clearly calls for them. As multimodal interactions become more common in user-facing applications, closing this gap is a natural target for future model development. MMI is created to aid in making such progress and is, beyond providing a snapshot of current capabilities, designed to serve as a living benchmark for tracking multimodal progress over time. Limitations Creating an interleaved multimodal benchmark and appropriately evaluating it comes with many challenges. While we have tried to address those to the best of our abilities, our work knows limitations as well. • The benchmark’s main index is defined but not evaluated. The main numbers we report in our work are MPS values, which express only whether the expected modalities were returned. The evaluated models return too few assets for an MMI Value to be meaningful (section 3.4.1), and while this is a limitations of models perhaps more than a limitation of the benchmark, it does imply that the actual main score of the benchmark is not well tested. We assessed the rubrics through human-judge agreement, but the agreement was far from perfect, leaving some open questions regarding the viability of computing the MMI Value automatically using juges. • We evaluate APIs, not products. All five models we tested are queried through their provider API with bare settings, no system prompt and no tools. While this is where also their claim to omni-modality is made, deployed assistants often wrap these models with image, audio or video generators, which would likely largely increase the MPS as well as MMI Value. The scores therefore characterize the model as exposed by the API, not the assistant a user interacts with. • Answer correctness may subjective. Some prompts have an unambiguous answer, but others are open-ended, and reasonable users may disagree about what a correct answer should involve and – while we tried to avoid this as much as possible – even about what modalities may be acceptable to do so. This adds a degree of subjectivity to the benchmark which is not properly evaluated given the low MPS of the models. Future work Most of the future work we envision is contigent on better models: as MPS scores go up, this will likely allow us to improve both the benchmark as well as the scoring. One potential immediate way to do that would be to evaluate tool-augmented and agentic configurations directly, evaluating complete products more than the models underpinning it. Such models could be evaluated as judges as well as on the benchmark. A new generation of models may also warrant revising the rubrics themselves. Another important future direction involves creating benchmark data beyond English. Acknowledgments We thank Di Lin, Kieran Nash, Tara Fowler, and Manohar Paluri for their support. References Anthropic (2026) Anthropic. Claude Opus 4-6, 2026. https://w.anthropic.com/news/claude-opus-4-6. Chameleon Team (2024) Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. CoRR, abs/2405.09818, 2024. https://doi.org/10.48550/arXiv.2405.09818. Chang et al. (2025) Eun Chang, Zhuangqun Huang, Yiwei Liao, Sagar Ravi Bhavsar, Amogh Param, Tammy Stark, Adel Ahmadyan, Xiao Yang, Jiaqi Wang, Ahsan Abdullah, Giang Nguyen, Akil Iyer, David Hall, Elissa Li, Shane Moon, Nicolas Scheffer, Kirmani Ahmed, Babak Damavandi, Rakesh Wanga, Anuj Kumar, Rohit Patel, and Xin Luna Dong. WearVQA: A visual question answering benchmark for wearables in egocentric authentic real-world scenarios. CoRR, abs/2511.22154, 2025. 10.48550/ARXIV.2511.22154. https://doi.org/10.48550/arXiv.2511.22154. Chen et al. (2024) Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In Advances in Neural Information Processing Systems (NeurIPS), 2024. https://doi.org/10.48550/arXiv.2403.20330. MMStar benchmark. Fang et al. (2024) Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. MMBench-Video: A long-form multi-shot benchmark for holistic video understanding. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024. https://doi.org/10.48550/arXiv.2406.14515. Fu et al. (2023) Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306.13394, 2023. 10.48550/ARXIV.2306.13394. https://doi.org/10.48550/arXiv.2306.13394. Fu et al. (2024a) Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. CoRR, abs/2405.21075, 2024a. 10.48550/ARXIV.2405.21075. https://doi.org/10.48550/arXiv.2405.21075. Fu et al. (2024b) Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive. In European Conference on Computer Vision (ECCV), 2024b. https://doi.org/10.48550/arXiv.2404.12390. Google DeepMind (2025) Google DeepMind. Gemini 3.1 Flash, 2025. https://deepmind.google/models/gemini/flash/. Google DeepMind (2026) Google DeepMind. Gemini 3.1 Pro, 2026. https://deepmind.google/models/gemini/pro/. Hong et al. (2026) Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. WorldSense: Evaluating real-world omnimodal understanding for multimodal LLMs. In International Conference on Learning Representations (ICLR), 2026. https://doi.org/10.48550/arXiv.2502.04326. Huang et al. (2024) Hailang Huang, Yong Wang, Zixuan Huang, Huaqiu Li, Tongwen Huang, Xiangxiang Chu, and Richong Zhang. MMGenBench: Fully automatically evaluating LMMs from the text-to-image generation perspective. CoRR, abs/2411.14062, 2024. 10.48550/ARXIV.2411.14062. https://doi.org/10.48550/arXiv.2411.14062. Huang et al. (2023) Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2I-CompBench: A comprehensive benchmark for open-world compositional text-to-image generation. In Advances in Neural Information Processing Systems (NeurIPS), 2023. https://doi.org/10.48550/arXiv.2307.06350. Li et al. (2024a) Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. GenAI-Bench: Evaluating and improving compositional text-to-visual generation. CoRR, abs/2406.13743, 2024a. 10.48550/ARXIV.2406.13743. https://doi.org/10.48550/arXiv.2406.13743. Li et al. (2024b) Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. SEED-Bench: Benchmarking multimodal LLMs with generative comprehension. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024b. https://doi.org/10.48550/arXiv.2307.16125. Li et al. (2024c) Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. LLMs-as-judges: A comprehensive survey on LLM-based evaluation methods. CoRR, abs/2412.05579, 2024c. 10.48550/ARXIV.2412.05579. https://doi.org/10.48550/arXiv.2412.05579. Li et al. (2024d) Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. MVBench: A comprehensive multi-modal video understanding benchmark. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024d. https://doi.org/10.48550/arXiv.2311.17005. Li et al. (2024e) Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, and Chenghua Lin. Omnibench: Towards the future of universal omni-language models. CoRR, abs/2409.15272, 2024e. 10.48550/ARXIV.2409.15272. https://doi.org/10.48550/arXiv.2409.15272. Liu et al. (2024a) Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? In European Conference on Computer Vision (ECCV), 2024a. https://doi.org/10.48550/arXiv.2307.06281. Liu et al. (2024b) Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. TempCompass: Do video LLMs really understand videos? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 8731–8772, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. 10.18653/v1/2024.findings-acl.517. https://aclanthology.org/2024.findings-acl.517/. Lu et al. (2024a) Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-IO 2: Scaling autoregressive multimodal models with vision, language, audio, and action. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024a. https://doi.org/10.48550/arXiv.2312.17172. Lu et al. (2024b) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024b. https://doi.org/10.48550/arXiv.2310.02255. Lyu et al. (2023) Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-LLM: Multi-modal language modeling with image, audio, video, and text integration. CoRR, abs/2306.09093, 2023. https://doi.org/10.48550/arXiv.2306.09093. Ma et al. (2024) Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun. MMLongBench-Doc: Benchmarking long-context document understanding with visualizations. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024. https://doi.org/10.48550/arXiv.2407.01523. Spotlight. Mangalam et al. (2023) Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. EgoSchema: A diagnostic benchmark for very long-form video language understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2023. https://doi.org/10.48550/arXiv.2308.09126. Mathew et al. (2021) Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA on document images. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2021. https://doi.org/10.1109/WACV48630.2021.00225. Meta AI (2025) Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025. https://ai.meta.com/blog/llama-4-multimodal-intelligence/. Meta Superintelligence Labs (2026) Meta Superintelligence Labs. Introducing muse spark: The first model from meta superintelligence labs, 2026. https://ai.meta.com/blog/introducing-muse-spark-msl/. Blog post. Moon et al. (2023) Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, Kavya Srinet, Babak Damavandi, and Anuj Kumar. AnyMAL: An efficient and scalable any-modality augmented language model. CoRR, abs/2309.16058, 2023. https://doi.org/10.48550/arXiv.2309.16058. Ni et al. (2025) Jinjie Ni, Yifan Song, Deepanway Ghosal, Bo Li, David Junhao Zhang, Xiang Yue, Fuzhao Xue, Zian Zheng, Kaichen Zhang, Mahir Shah, Kabir Jain, Yang You, and Michael Shieh. MixEval-X: Any-to-any evaluations from real-world data mixtures. In International Conference on Learning Representations (ICLR), 2025. https://doi.org/10.48550/arXiv.2410.13754. OpenAI (2026) OpenAI. Introducing GPT-5.4, 2026. https://openai.com/index/introducing-gpt-5-4/. Patel et al. (2026) Rohit Patel, Alexandre Rezende, and Steven McClain. GIM: Evaluating models via tasks that integrate multiple cognitive domains. CoRR, abs/2605.18663, 2026. 10.48550/ARXIV.2605.18663. https://doi.org/10.48550/arXiv.2605.18663. Pătrăucean et al. (2023) Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alexandre Fréchette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. https://doi.org/10.48550/arXiv.2305.13786. Qian et al. (2025) Yusu Qian, Hanrong Ye, Jean-Philippe Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. MIA-Bench: Towards better instruction following evaluation of multimodal LLMs. In International Conference on Learning Representations (ICLR), 2025. https://doi.org/10.48550/arXiv.2407.01509. Sakshi et al. (2024) S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. MMAU: A massive multi-task audio understanding and reasoning benchmark. CoRR, abs/2410.19168, 2024. 10.48550/ARXIV.2410.19168. https://doi.org/10.48550/arXiv.2410.19168. Tanaka et al. (2023) Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. SlideVQA: A dataset for document visual question answering on multiple images. In AAAI Conference on Artificial Intelligence, 2023. https://doi.org/10.48550/arXiv.2301.04883. Tang et al. (2023) Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. In Advances in Neural Information Processing Systems (NeurIPS), 2023. https://doi.org/10.48550/arXiv.2305.11846. Thakur et al. (2025) Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-judges. In Ofir Arviv, Miruna Clinciu, Kaustubh Dhole, Rotem Dror, Sebastian Gehrmann, Eliya Habba, Itay Itzhak, Simon Mille, Yotam Perlitz, Enrico Santus, João Sedoc, Michal Shmueli Scheuer, Gabriel Stanovsky, and Oyvind Tafjord, editors, Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), pages 404–430, Vienna, Austria and virtual meeting, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-261-9. https://aclanthology.org/2025.gem-1.33/. Tito et al. (2023) Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transformers for multi-page DocVQA. Pattern Recognition, 144, 2023. https://doi.org/10.1016/j.patcog.2023.109834. Wang et al. (2025a) Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F. Chen. AudioBench: A universal benchmark for audio large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4297–4316, Albuquerque, New Mexico, April 2025a. Association for Computational Linguistics. ISBN 979-8-89176-189-6. 10.18653/v1/2025.naacl-long.218. https://aclanthology.org/2025.naacl-long.218/. Wang et al. (2025b) Jiaqi Wang, Xiao Yang, Kai Sun, Parth Suresh, Sanat Sharma, Adam Czyzewski, Derek Andersen, Surya Appini, Arkav Banerjee, Sajal Choudhary, Shervin Ghasemlou, Ziqiang Guan, Akil Iyer, Haidar Khan, Lingkun Kong, Roy Luo, Tiffany Ma, Zhen Qiao, David Tran, Wenfang Xu, Skyler Yeatman, Chen Zhou, Gunveer Gujral, Yinglong Xia, Shane Moon, Nicolas Scheffer, Nirav Shah, Eun Chang, Yue Liu, Florian Metze, Tammy Stark, Zhaleh Feizollahi, Andrea Jessee, Mangesh Pujari, Ahmed Aly, Babak Damavandi, Rakesh Wanga, Anuj Kumar, Rohit Patel, Wen-tau Yih, and Xin Luna Dong. CRAG-M: Multi-modal multi-turn comprehensive RAG benchmark. CoRR, abs/2510.26160, 2025b. 10.48550/ARXIV.2510.26160. https://doi.org/10.48550/arXiv.2510.26160. Wang et al. (2025c) Yuxuan Wang, Yueqian Wang, Bo Chen, Tong Wu, Dongyan Zhao, and Zilong Zheng. OmniMMI: A comprehensive multi-modal interaction benchmark in streaming video contexts. In Conference on Computer Vision and Pattern Recognition (CVPR), 2025c. https://doi.org/10.48550/arXiv.2503.22952. Wu et al. (2024) Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. NExT-GPT: Any-to-any multimodal LLM. In International Conference on Machine Learning (ICML), 2024. https://doi.org/10.48550/arXiv.2309.05519. Xie et al. (2025) Wulin Xie, Yifan Zhang, Chaoyou Fu, Yang Shi, Bingyan Nie, Hongkai Chen, Zhang Zhang, Liang Wang, and Tieniu Tan. Mme-unify: A comprehensive benchmark for unified multimodal understanding and generation models. CoRR, abs/2504.03641, 2025. 10.48550/ARXIV.2504.03641. https://doi.org/10.48550/arXiv.2504.03641. Yan et al. (2024) Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard, 2024. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html. Yang et al. (2024) Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. AIR-Bench: Benchmarking large audio-language models via generative comprehension. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024. https://doi.org/10.48550/arXiv.2402.07729. Yu et al. (2024) Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. M-Vet: Evaluating large multimodal models for integrated capabilities. In International Conference on Machine Learning (ICML), 2024. https://doi.org/10.48550/arXiv.2308.02490. yu Huang et al. (2024) Chien yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi-Yuan Hsiao, Chun-Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, Roshan Sharma, Shinji Watanabe, Bhiksha Ramakrishnan, Shady Shehata, and Hung yi Lee. Dynamic-SUPERB: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024. https://doi.org/10.48550/arXiv.2309.09510. Yue et al. (2024) Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024. https://doi.org/10.48550/arXiv.2311.16502. Yue et al. (2025) Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. MMMU-pro: A more robust multi-discipline multimodal understanding benchmark. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15134–15186, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. 10.18653/v1/2025.acl-long.736. https://aclanthology.org/2025.acl-long.736/. Zhang et al. (2025a) Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. LMMs-eval: Reality check on the evaluation of large multimodal models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025, pages 881–916, Albuquerque, New Mexico, April 2025a. Association for Computational Linguistics. ISBN 979-8-89176-195-7. 10.18653/v1/2025.findings-naacl.51. https://aclanthology.org/2025.findings-naacl.51/. Zhang et al. (2025b) Zicheng Zhang, Xiangyu Zhao, Xinyu Fang, Chunyi Li, Xiaohong Liu, Xiongkuo Min, Haodong Duan, Kai Chen, and Guangtao Zhai. Redundancy principles for MLLMs benchmarks. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12492–12504, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8-89176-251-0. 10.18653/v1/2025.acl-long.612. https://aclanthology.org/2025.acl-long.612/. Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023. https://doi.org/10.48550/arXiv.2306.05685. Appendix A Modality Detection Judge Prompt Below, we show both the judge system prompt and judge user prompt template passed to Gemini 3 Flash (gemini-3-flash-preview) to judge whether a model returned an asset of the correct modality. Judge System Prompt You are a strict evaluator for a multimodal AI benchmark. Your job is to examine a model’s API response and determine which output modalities it genuinely produced. The five modalities are: Text, Image, Audio, Video, Document. For each modality, look for concrete evidence the model actually produced it: • Text: Any substantive text output (not just a refusal or error message). • Image: Native image data, base64-encoded image content, complete SVG markup, or a URL pointing to a generated image. • Audio: Native audio data, base64-encoded audio content, or a URL pointing to generated audio. • Video: Native video data, base64-encoded video content, or a URL pointing to generated video. • Document: A generated file such as PDF, CSV, HTML, DOCX, or structured data clearly intended as a downloadable document. Do NOT count a modality if the model only wrote ABOUT it. The model must have produced actual media data or file content. Describing what an image would look like, writing a screenplay for a video, or providing lyrics instead of audio are NOT productions of those modalities. A refusal (‘I can’t generate images’) is NOT a production. Respond with ONLY valid JSON: "detected_modalities": [], "reasoning": "brief explanation" The detected_modalities array should list every modality the model genuinely produced. An empty array means the model produced nothing usable. Valid modality names: Text, Image, Audio, Video, Document. Judge User Prompt Template The model was asked: ‘prompt_text’ The model’s response text was: text_content The model’s structured/raw API response (JSON, truncated) was: raw_response_json Which of the five modalities (Text, Image, Audio, Video, Document) did the model genuinely produce in this response? Appendix B Rubric Judge Prompt Below, we show the system prompt and user prompt template passed to Gemini 3 Flash (gemini-3-flash-preview) when grading a model response against the human-written rubrics of appendix D. The judge is called once per rubric criterion: each call carries a single criterion and only the model-produced payload for the modality that criterion is tagged with, never the model’s output for the other modalities. For text criteria the payload is the response text itself; for the other modalities the generated artifact is attached to the call as media, alongside the prompt’s own input assets as context. Rubric Judge System Prompt You are a strict grader for one rubric criterion in a multimodal model evaluation. You will receive the original prompt, expected output modalities, input-file metadata, and exactly one model-produced modality payload for the rubric being graded. Grade only the supplied rubric against only the supplied model-produced modality payload. Do not infer credit from missing modalities, descriptions of absent media, or any model response content that was not supplied to you. If the supplied payload is insufficient to satisfy the rubric, score it 0.0. Return only valid JSON with this schema: "score": 0.0, "explanation": "brief reason" Scoring: • 1.0 means the criterion is fully satisfied. • 0.0 means the criterion is not satisfied. • Use partial credit between 0.0 and 1.0 when the criterion is partly satisfied. Rubric Judge User Prompt Template # Original prompt prompt_text # Expected output modalities expected_modalities # Input files input_files # Rubric modality being graded modality # Rubric criterion rubric # Supplied model-produced payload payload_summary Evaluate only this rubric criterion using only the supplied modality payload. Return only JSON. Non-text artifacts are appended to the same call as attached media, each group introduced by a label so that the judge cannot confuse the artifact it is grading with the artifact the model was given. Attached Media Labels # Attached below: the model-produced modality artifact. This is the artifact to grade. # Attached below: the artifact(s) the model was given as prompt input. Context only. Never grade these as model output. Appendix C Full modality heatmap Figure 17: Full modality grid. We show the distribution of input and output modalities of the full Modality Maturity Index benchmark. Appendix D Rubric writing guidelines Below we summarize the guidelines followed when authoring the MMI rubrics. Annotators worked from the finalized prompt set: for each prompt they received the prompt text, its input assets, and the gold output modalities, and returned one rubric set, a numbered list of criteria, each tagged with exactly one output modality and each stating one requirement a correct answer must satisfy. Guiding principle A rubric is a grading instrument, not a specification of the answer. It captures only those properties whose presence or absence settles correctness, one per criterion; it does not attempt to describe in full what the asset should look or sound like. Two rules follow. • Write for a blind, single-modality grader. Each criterion is graded against the model’s output for the one modality it is tagged with, and must be decidable from that output alone: without the model’s output for the other modalities, and without the prompt or its input assets. A criterion that can only be settled by comparing two output modalities, or by inspecting an input, must not be written. (The grader is supplied the prompt and inputs as background context, but no criterion may depend on them.) • Settle the answer inside the criterion. Never merely name a topic the response should address; state the fact, value, or property it must contain, so the grader never has to derive the correct answer or fall back on its own knowledge. Bad: Identifies the breed of the dog. Good: Identifies the dog as an Australian Shepherd or Miniature American Shepherd. This also covers prompts that ask the model to edit or transform a supplied asset: the criterion names the property the output must have, stated concretely enough to check without the input (“The map in the output image is viewed from directly above”, not “Preserves the perspective of the original”). Procedure 1. Confirm the output modalities. Decide which modalities a genuinely useful answer would need and compare against the gold list, reading for the user’s underlying goal rather than for keywords; flag the gold list rather than writing against one you disagree with. Two conventions apply: modalities nested inside others are not labeled separately (a PDF containing diagrams is document only, a video with a soundtrack is video only), and text is an output modality only when the prompt calls for textual content in its own right, not for commentary wrapped around a generated asset. 2. Partition the requirements. Assign each requirement to the single modality that carries it. Every gold modality must receive criteria, each criterion is tagged with exactly one modality, and a modality’s criteria should cover everything the prompt asks of that modality and nothing else. Never grade one modality’s criterion against another’s content: if a fact is required in both a diagram and a written explanation, assert it in both the image and the text criteria. 3. Write the criteria. Order each modality’s criteria as: one delivery criterion asserting that the asset exists and is of the right kind, including sub-kind where specified (“Delivers a CSV document file.”), which text normally omits; for non-text assets an integrity criterion where it is a plausible failure mode (the asset opens or plays without corruption, truncation, or rendering errors); then the content criteria, which are the bulk of the rubric; then at most one or two craft criteria on legibility, structure, or pacing, where usefulness genuinely depends on execution. Every criterion must be: • Self-contained and atomic: it settles the expected answer itself and asserts one fact. “Labels sunlight and water and carbon dioxide as inputs” should be three criteria. • Mutually exclusive: no two criteria award credit for the same content, which would silently double-weight it within its modality. • Declarative: a third-person statement about what the output does (“Delivers…”, “Shows…”, “The diagram depicts…”), never a question or an instruction to the model. • Binary-decidable and objectively verifiable: answerable yes or no from the asset itself plus, where relevant, publicly checkable fact. • Timeless: anchor anything time-dependent. Bad: States the current marathon world record. Good: States that the men’s marathon world record as of January 2025 is 2:00:35. • Achievable in the modality: do not ask an image to carry a causal argument only prose can express, or audio to be “well formatted”. • Generation-agnostic: specify properties of the output, never the technique or tool used to produce it. • Tolerant where the answer legitimately varies: enumerate the acceptable set with “such as” or “or” rather than silently picking one. Subjectivity. Open-ended and creative prompts still need objective criteria: grade the observable properties that make the output fit for purpose, not the impression it creates. Replace “The image is visually appealing” with “Uses readable labels and an uncluttered visual layout that makes the inputs, process, and outputs easy to distinguish”. Common mistakes. Referring to the prompt or user from inside a criterion (“as requested”); requiring consistency across output modalities; grading exact wording of speech rather than its content; padding with criteria the prompt does not support; restating delivery further down the list; phrasing a requirement negatively where a positive statement would do. How many. Target 4–12 criteria per output modality. Generating substantial content from scratch warrants more, a small edit to a supplied asset fewer. Scoring averages within a modality before averaging across modalities, so writing more criteria for a modality sharpens how finely it is measured without giving it more weight in the prompt’s score, and a tight set of well-separated criteria scores far more reliably than a long list of overlapping ones. Output format Return one JSON object per prompt, with criteria numbered from 1 across all modalities and modality lower-case and drawn from the prompt’s gold output modalities. All criteria within a modality carry equal weight, so there is no weight field: the score for a modality is the fraction of its criteria satisfied, and the prompt’s score is the mean of those per-modality scores. Rubric Schema "criteria": [ "id": "1", "criterion": "<one requirement>", "modality": "<text|image|audio|video|document>", … ] Worked example “Can you create a list of pros and cons between hybrid and fully electric cars and send it to me in a C S V file?” (spoken audio input, document output). Criterion 1 pins the file type the prompt named, criterion 2 checks structure, and the content criteria enumerate acceptable answers rather than fixing one. 1. Delivers a CSV document file. 2. Formats the file as a valid tabular CSV with a header row and comma-separated rows. 3. Includes at least two pros for hybrid cars, such as better fuel efficiency than gas-only cars, reduced range anxiety, or less reliance on charging stations. 4. Includes at least two cons for hybrid cars, such as continued gasoline use, tailpipe emissions, more complex maintenance, or limited electric-only driving. 5. Includes at least two pros for fully electric cars, such as zero tailpipe emissions, lower routine maintenance, home charging, or lower operating costs. 6. Includes at least two cons for fully electric cars, such as charging time, charger availability, driving range limits, battery degradation, or higher upfront cost. The crochet rubric set in section 3.3 is the multi-modality counterpart: each modality answers a different clause of the prompt, and the audio criterion carries the expected pronunciation itself rather than deferring to the video. Appendix E Human annotation guidelines The instructions below were given to the human annotators whose judgments we compare against the rubric-based LLM judge in section 5.4. They are reproduced with light edits for formatting and to match the annotation task as it was finally delivered. Instructions as given to annotators You will get a set of questions asked to a model and the model answers to those questions. Those questions typically involve multiple modalities in both the input and the output. Your task is to provide a per-modality assessment of whether the model’s answer to the question is correct. The modalities present in the inputs and outputs of the benchmark are: text, audio, image, video and document, where document can be any type of document (txt, pdf, docx, ppt, etc.). Input / expected output Input: 1. A question / prompt involving one or more modalities. 2. The list of output modalities that are required to answer that question satisfactorily (the prompt’s desired output modalities). 3. A model-generated answer to this question. Expected output: 1. For each of the prompt’s desired output modalities, an assessment of the correctness of the model’s response for that output modality: ‘correct’ or ‘incorrect’. 2. For each such assessment, a short free-text comment justifying the verdict. Judge every desired output modality, also if the model did not return anything for that modality. Modalities that are not in the list are not judged. Additional instructions • If the model did not generate a specific output modality that was needed to answer the question, mark that output modality as ‘incorrect’. • Do not provide a fine-grained judgment of correctness: the verdict is binary, ‘correct’ or ‘incorrect’. Always add a short comment stating why, and use it as well to flag doubt or anything else worth recording. • A link to an asset counts as returning that modality. If a model provides a link in its answer but the link is broken, you may mark its response for that modality incorrect. • If a question requires aligning two modalities (e.g. audio and video) and the model correctly produces both but does not align them, you may choose one of the modalities to mark incorrect. • Modalities within other modalities are not marked separately. For example: – If the user asks to return a pdf with instructions, the desired output modality is document only, not also text or image if the document contains texts or images. – If the user asks to return a video and that video contains audio, audio is not a separate modality. • Text that merely accompanies another modality is never judged in itself; text is assessed only when it is one of the desired output modalities, in which case it is judged like any other modality (see the example below). Likewise, modalities the model produced but that were not asked for are not judged separately, though you may mention them in the comment. Knowledge acquisition using AI / LLMs Some questions may require expert knowledge to be answered; you may use an AI model to help acquire that knowledge and indicate you did so for that question. However, you may not: • Pass the model answer to an AI or LLM and ask if it is correct. • Pass the original question directly to an AI or LLM and use its response to judge the correctness of the provided model answer. Example Question Can you show me how to make soft chocolate chip cookies? A list of ingredients and a tutorial video would be nice. Also give me an image of chocolate chip cookies that I can send to my friends to invite them to come eat the cookies. Desired output modalities: text, video, image. Model answer: Sure, here is a video with a tutorial for how to make chocolate chip cookies: https://youtu.be/PFJAuAWxuvI?si=_dqH10g3xEFOeEJz. List of ingredients: • Brown Sugar & White Sugar • Unsalted Butter & Salt • Large Egg • Vanilla Extract • All-Purpose Flour Judgment: • Video: correct. The model returned a video tutorial to make chocolate chip cookies as required. • Text: incorrect. The model did return a list of ingredients, but baking soda and chocolate chips were missing from the list. • Image: incorrect. The model did not return an image. Appendix F MMI Values for the rubric-validation runs The experiment of section 4.3 was run to exercise the rubrics, not to rank models: these values document a different, later set of models run with Gemini generation backends, and are neither a ranking nor comparable to table 4. We record them here so that the run is fully documented and the numbers behind section 5.4 can be checked. table 5 gives, for each of the three tool-scaffolded models, the MMI Value over all 893 prompts, the same score with document criteria removed, and the MMI Value restricted to each modality in turn. The scaffolding raises modality presence enough for the MMI Value to reflect gradeable content, which is not the case for the main-evaluation models (section 3.4.1). No document-generation tool was provided, so the document column is zero by construction. MMI Value MMI Value by modality Model All Excl. doc. Text Image Audio Video Doc. Gemini 3.5 Flash 0.51 0.63 0.69 0.78 0.53 0.47 0.00 Claude Opus 4.8 0.36 0.44 0.51 0.46 0.42 0.32 0.00 GPT-5.4 0.29 0.35 0.59 0.39 0.20 0.22 0.00 Table 5: MMI Values for the three tool-scaffolded models of section 4.3. All is the MMI Value as defined in section 3.4.1: the mean over the 893 prompts of each prompt’s mean across its output modalities of the mean criterion score within that modality. Excl. doc. is the same quantity with document criteria removed, over the prompts that retain at least one non-document output modality. The five modality columns apply that same within-modality mean, averaged over the prompts for which the modality in question is expected; because different combinations of modalities are expected across prompts, the columns do not average to the MMI Value. Document is zero throughout because no document-generation tool was provided.