Paper deep dive

Blending Concepts: Benchmarking Visual Metaphor Generation in Text-to-Image Models

Chuer Chen, Zichen Wang, Yi He, Zhengxi Yu, Nan Cao

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 60

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 9/5/2026, 2:25:30 AM

Summary

This paper introduces VMetaphor-Bench, the first benchmark for evaluating visual metaphor generation in Text-to-Image (T2I) models. The benchmark consists of 1,500 curated visual metaphors organized into three levels and ten categories, paired with conceptual and descriptive prompts. The authors develop a hybrid evaluation framework using an MLLM-as-judge paradigm, combining multiple-choice questions (MCQs) for metaphorical fidelity and dimension-based scoring for perceptual quality. Evaluations of 11 representative T2I models reveal that while proprietary models outperform open-source ones, all models struggle significantly with compositional structuring and cross-domain mapping, highlighting a key frontier for future research.

Entities (10)

VMetaphor-Bench · benchmark · 100%FLUX2-dev · model · 95%GPT-image-1.5 · model · 95%Nano Banana 2 · model · 95%Qwen3.5-27B · model · 95%Seedream 5.0 Lite · model · 95%Compositional Structuring · challenge · 90%Cross-Domain Mapping · challenge · 90%MLLM-as-judge · method · 90%Visual Metaphor · concept · 90%

Relation Signals (7)

VMetaphor-Bench → contains → 1,500 visual metaphors

confidence 100% · It comprises 1,500 visual metaphors curated from real-world creative imagery

GPT-image-1.5 → achievesbestperformance → VMetaphor-Bench

confidence 95% · GPT Image 1.5 achieves the best overall performance with 85.4% MCQ accuracy

VMetaphor-Bench → evaluates → Text-to-Image Models

confidence 95% · VMetaphor-Bench, the first benchmark for evaluating visual metaphor generation in T2I models.

Qwen3.5-27B → servesasjudge → VMetaphor-Bench

confidence 95% · we adopt an open-source MLLM, Qwen3.5-27B, as the default judge model

VMetaphor-Bench → usesevaluationmethod → MLLM-as-judge

confidence 95% · we develop a hybrid framework within an MLLM-as-judge paradigm

Text-to-Image Models → strugglewith → Compositional Structuring

confidence 90% · even the strongest proprietary models struggle with compositional structuring

Text-to-Image Models → strugglewith → Cross-Domain Mapping

confidence 90% · even the strongest proprietary models struggle with ... cross-domain mapping

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Text-to-image (T2I) models have achieved remarkable success at faithfully rendering specified objects and attributes, yet their ability to produce visual metaphors, images that convey abstract ideas by combining elements from two distinct domains, remains largely unexamined. To bridge this gap, we introduce VMetaphor-Bench, the first benchmark for evaluating visual metaphor generation in T2I models. It comprises 1,500 visual metaphors curated from real-world creative imagery, organized into three levels and ten categories, with each sample paired with two prompts of differing specificity. For evaluation, we develop a hybrid framework within an MLLM-as-judge paradigm, combining a multiple-choice question (MCQ) based protocol of 9,594 questions across four levels of metaphorical fidelity with a dimension-based scoring protocol along three perceptual dimensions. Extensive evaluation of 11 representative T2I models reveals that even the strongest proprietary models struggle with compositional structuring and cross-domain mapping, key aspects of metaphorical expression, highlighting visual metaphor generation as an important frontier for future T2I research.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

57,483 characters extracted from source content.

Expand or collapse full text

Blending Concepts: Benchmarking Visual Metaphor Generation in Text-to-Image Models Chuer Chen Affiliation: Tongji University Zichen Wang Affiliation: Tongji University Yi He Affiliation: Tongji University Zhengxi Yu Affiliation: Tongji University Nan Cao †thanks: Corresponding author. Affiliation: Tongji University Affiliation: Shanghai Innovation Institute Abstract Text-to-image (T2I) models have achieved remarkable success at faithfully rendering specified objects and attributes, yet their ability to produce visual metaphors—images that convey abstract ideas by combining elements from two distinct domains—remains largely unexamined. To bridge this gap, we introduce VMetaphor-Bench, the first benchmark for evaluating visual metaphor generation in T2I models. It comprises 1,500 visual metaphors curated from real-world creative imagery, organized into three levels and ten categories, with each sample paired with two prompts of differing specificity. For evaluation, we develop a hybrid framework within an MLLM-as-judge paradigm, combining a multiple-choice question (MCQ) based protocol of 9,594 questions across four levels of metaphorical fidelity with a dimension-based scoring protocol along three perceptual dimensions. Extensive evaluation of 11 representative T2I models reveals that even the strongest proprietary models struggle with compositional structuring and cross-domain mapping, key aspects of metaphorical expression, highlighting visual metaphor generation as an important frontier for future T2I research. 1 Introduction Recent years have witnessed remarkable progress in text-to-image (T2I) generation, driven by advances across three main architectural directions: diffusion models Ho et al. (2020); Rombach et al. (2022), autoregressive models Team (2024); Tian et al. (2024), and unified multimodal architectures Chen et al. (2025b); Deng et al. (2025); Wu et al. (2025b). Powered by large-scale training, a wide range of T2I systems, from leading proprietary models such as GPT-Image-1 OpenAI (2025a) and Gemini 3.1 Flash Image Google (2026) to open-source counterparts Labs (2024); Wu et al. (2025a), now produce photorealistic images that faithfully render specified objects, attributes, and spatial layouts, achieving strong performance on basic semantic prompt following. However, their ability to express abstract concepts and convey meaning beyond what is literally depicted remains far less explored. A particularly important form of such abstract expression is the visual metaphor, where one concept is understood through another via visual analogy Forceville and others (2008). Drawing on conceptual blending theory Fauconnier and Turner (2003), a visual metaphor combines elements from two distinct domains — a source that lends its properties, and a target that receives them — to produce a single image carrying meaning richer than either domain alone. Generating such images is far from straightforward: the model must (i) blend the two given domains coherently using a specific compositional strategy (e.g., fusion or juxtaposition), (i) establish meaningful element-level mappings between the two, and (i) ensure that the resulting image conveys the intended meaning to viewers. As shown in Figure 1, even state-of-the-art T2I models frequently struggle with these tasks, producing images that miss one of the domains, adopt the wrong structural form, blend the two awkwardly, or fail to convey the intended meaning. Yet this challenge has been largely overlooked by existing T2I benchmarks, which remain dominated by literal prompt following. Mainstream benchmarks such as GenEval Ghosh et al. (2023) and T2I-CompBench Huang et al. (2023) evaluate object, attribute, and basic spatial fidelity from short prompts, while more recent reasoning-aware benchmarks Niu et al. (2025); Sun et al. (2025a); Zhao et al. (2025) extend toward factual or physical inference, leaving the cross-domain conceptual blending required by visual metaphors unexamined. On the metaphor side, existing efforts have focused predominantly on the understanding direction Akula et al. (2023); Kundu et al. (2025); Zhang et al. (2026); the only generation evaluation we are aware of, conducted as a small probe within MetaCLUE Akula et al. (2023), is limited to 300 samples assessed via FID, CLIP similarity, and a small user study — none of which can systematically diagnose where and how T2I models fail at metaphorical expression. Together, these limitations leave a clear gap: no existing benchmark systematically evaluates how well T2I models generate visual metaphors across diverse themes and metaphorical structures. Figure 1: Common failure modes in visual metaphor generation from GPT-Image-1.5 OpenAI (2025a), Seedream 5.0 Lite ByteDance Seed (2026), and FLUX.2-dev Labs (2025). Figure 2: Overview of VMetaphor-Bench. Left: hierarchical taxonomy with three levels and ten thematic categories. Middle: representative examples spanning the three levels and the three structure types of visual metaphors. Right: performance of 11 evaluated T2I models on MCQ accuracy and dimension scoring under conceptual prompts. To address this gap, we introduce VMetaphor-Bench, a new benchmark designed for evaluating visual metaphor generation in T2I models. It comprises 1,500 visual metaphors curated from real-world creative imagery, organized into three levels and ten categories, as illustrated in Figure 2. Each sample is annotated with its source and target domains, structure type, metaphorical meaning, element-level mappings, and paired with two prompts of differing specificity: a conceptual prompt that conveys only the metaphorical idea, and a descriptive prompt that details its visual realization. For evaluation, we develop a hybrid framework consisting of an MCQ-based protocol — 9,594 questions targeting four levels of metaphorical fidelity (domain presence, structure type, meaning, and element mapping) — and a dimension-based scoring protocol rating each image on metaphoric efficacy, metaphor logic, and perceptual harmony. Both protocols are realized within an MLLM-as-judge paradigm, enabling fine-grained, interpretable diagnosis of T2I models’ strengths and weaknesses in visual metaphor generation. Using VMetaphor-Bench, we conduct a comprehensive evaluation of 11 representative T2I models, spanning leading proprietary systems, open-source diffusion models, and unified multimodal architectures. Our results reveal that proprietary models such as GPT Image 1.5 and Nano Banana 2 achieve substantially better performance than open-source counterparts. However, even the strongest models struggle with compositional structuring and cross-domain mapping—key aspects of metaphorical expression—highlighting visual metaphor generation as an important frontier for future T2I research. In summary, our contributions are threefold: • We introduce VMetaphor-Bench, the first benchmark (to the best of our knowledge) dedicated to evaluating visual metaphor generation in T2I models, comprising 1,500 high-quality, carefully curated samples organized into a hierarchical taxonomy of three levels and ten categories. • We design a hybrid evaluation framework with an MCQ-based protocol of 9,594 questions across four levels of metaphorical fidelity and a dimension-based protocol along three perceptual dimensions, enabling fine-grained, interpretable assessment. • We conduct a comprehensive evaluation of 11 representative T2I models, providing insights into their capabilities in visual metaphor generation and highlighting key directions for future research. 2 Related Works 2.1 Text-to-Image Generation Models Text-to-image (T2I) generation has advanced rapidly along three main architectural directions. Diffusion models Dhariwal and Nichol (2021); Ho et al. (2020); Rombach et al. (2022) remain the dominant paradigm, with recent progress driven by scaling transformer-based denoisers Esser et al. (2024); Labs (2024); Peebles and Xie (2023). Autoregressive models Sun et al. (2024); Team (2024); Tian et al. (2024) cast image generation as sequential token prediction, offering fine-grained compositional control through explicit modeling of inter-token dependencies. More recently, unified multimodal models Chen et al. (2025b); Deng et al. (2025); Wu et al. (2025b); Xie et al. (2025) integrate visual understanding and generation within a single architecture, leveraging multimodal reasoning to better interpret complex prompts. State-of-the-art systems such as GPT-Image-1 OpenAI (2025a) and Gemini 3.1 Flash Image Google (2026) now produce images of remarkable fidelity and aesthetic quality for everyday scenes. However, generating images that convey abstract semantic relationships, such as visual metaphors, remains a significant challenge across all three paradigms. 2.2 Text-to-Image Generation Benchmarks Early evaluation of text-to-image generation relied on distribution-level metrics such as FID Heusel et al. (2017) and feature-based similarity scores like CLIPScore Hessel et al. (2021), which capture perceptual quality but fail to assess fine-grained semantic correctness. The recent rise of multimodal large language models (MLLMs) Hurst et al. (2024); Team et al. (2023) has enabled a paradigm shift toward MLLM-as-judge evaluation Chen et al. (2024), where a vision-language model directly examines the generated image against the prompt, providing a more flexible and semantically grounded assessment framework. Under this paradigm, a line of benchmarks targets foundational semantic alignment — verifying whether generated images faithfully depict the specified objects Kamath et al. (2025), attributes Ye et al. (2025), and spatial relationships Wang et al. (2025); Wang et al. (2026). Representative works include GenEval Ghosh et al. (2023), T2I-CompBench Huang et al. (2023), and TIFA Hu et al. (2023), which decompose prompts into atomic verification questions to systematically evaluate compositional fidelity. More recently, benchmarks have shifted toward higher-order reasoning capabilities, probing whether T2I models can incorporate world knowledge Niu et al. (2025); Zhang et al. (2025a) and reasoning Chen et al. (2025a); Sun et al. (2025a); Zhao et al. (2025) into their generations, reflecting a broader trend from evaluating object-level fidelity toward scene-level semantic understanding. 2.3 Visual Metaphor Visual metaphor communicates abstract ideas by mapping one conceptual domain onto another through visual imagery Ervas (2021); Forceville and others (2008); Van Mulken et al. (2014). On the understanding side, MetaCLUE Akula et al. (2023) introduces a comprehensive benchmark for metaphor classification, localization, interpretation, along with a small probe on generation; ImageMet Kundu et al. (2025) benchmarks VLMs on metaphor captioning and VQA, and MetaphorStar Zhang et al. (2026) applies visual reinforcement learning to improve metaphor reasoning. On the generation side, several recent methods leverage text-to-image models as the visual backbone: Creative Blends Sun et al. (2025b) fuses disparate visual concepts via commonsense knowledge, The Mind’s Eye Koushik et al. (2025) proposes a source-target-meaning decomposition with self-evaluation rewards, and Beyond Pixels Xu et al. (2026) introduces schema-driven reasoning for metaphor transfer. However, evaluation of these methods has been limited—typically conducted on small-scale ad hoc test sets via FID, CLIP similarity, or user studies, none of which can diagnose where and how T2I models fail at metaphorical expression. While benchmarks exist for metaphor understanding, no counterpart has been established for systematically evaluating the generation quality of visual metaphors. 3 VMetaphor-Bench We present VMetaphor-Bench, the first benchmark, to the best of our knowledge, for evaluating visual metaphor generation in text-to-image models, featuring multi-dimensional assessments of cross-domain conceptual blending. We describe the dataset construction process in Section 3.1 and detail the evaluation framework in Section 3.2. An overview of the complete pipeline is illustrated in Figure 3. Figure 3: Overview of the VMetaphor-Bench pipeline. Left: Dataset Construction — images are collected and annotated through a two-stage pipeline, producing structured annotations and two granularity levels of prompts. Right: Evaluation Framework — generated images are assessed via MCQ-based evaluation (L1–L4 levels) and dimension-based scoring (Efficacy, Logic, Harmony). 3.1 Dataset Construction Image Collection. To construct a benchmark grounded in real-world visual metaphors, we collect images from Pinterest [33], a platform rich in creative visual content such as advertisements, editorial illustrations, and conceptual art. We query the platform using over ten keywords spanning both general and domain-specific terms, e.g. visual metaphor, creative ads. This process yields approximately 5,000 candidate images. Two authors then review the entire collection, filtering out duplicates, images that do not contain a clear visual metaphor, and low-quality or ambiguous samples, resulting in approximately 1,600 images for subsequent annotation. Data Annotation. After collecting images, we employ Gemini 3.1 Pro Google DeepMind (2026) as the annotation backbone for its strong visual reasoning capabilities. Since visual metaphors involve multi-layered semantics Cappa et al. (2025), generating accurate annotations and high-quality prompts simultaneously in a single pass is challenging. We therefore adopt a two-stage annotation pipeline. In the first stage, the model is presented with each image and tasked with extracting structured annotations: • Target domain: the subject being described — what the image is ultimately “about”. • Source domain: the concept used to characterize the target— it lends a property to the target through visual analogy. • Structure type: how the two domains are visually combined. Following Phillips et al. Phillips and McQuarrie (2004), we categorize it as replacement (one domain is absent and inferred from context), fusion (both domains are merged into a single object), or juxtaposition (both domains appear as separate objects in the same scene). • Meaning: the overall message conveyed through the interaction of the two domains. • Element mappings: fine-grained correspondences specifying which source element maps to which target element. In the second stage, the model receives both the image and the extracted annotations, and generates two prompts at different levels of granularity: (1) The descriptive prompt provides a detailed depiction of the visual content and how the metaphor is physically rendered in the image; (2) The conceptual prompt specifies the target and source domains, implicitly conveys the structure type, and states the intended meaning, without describing specific visual details or element-level mappings. Additionally, a style suffix describing the visual style of the original image (e.g., photorealistic, illustration) is extracted and appended to both prompts during generation to ensure stylistic consistency. This two-stage design allows the structured annotations to guide prompt generation, ensuring consistency between the structured annotations and the textual descriptions. Three annotators, after training on visual metaphor theory and annotation guidelines, perform rigorous manual inspection on all images, annotations, and prompts, and cross-validate each other’s decisions. Images where the metaphor is ambiguous or unrecognizable are removed. Inaccurate annotations (e.g., misidentified target or source domains) and imprecise prompts are manually corrected to align with the visual content. This verification process yields our final set of 1,500 high-quality images, each accompanied by structured annotations and two granularity levels of prompts. 3.2 Evaluation Framework Evaluating visual metaphor generation requires assessing both metaphorical fidelity — whether the generated image faithfully captures the intended cross-domain mapping — and expressive quality — how effectively and aesthetically the metaphor is conveyed. To this end, we adopt a hybrid evaluation framework with MLLM-as-judge Zhang et al. (2025b), comprising two complementary approaches: (1) MCQ-based evaluation that measures the correctness of metaphorical elements through multiple-choice questions, and (2) dimension-based scoring that rates the overall quality of the generated metaphor on a 1-5 Likert scale. An overview of the evaluation framework is shown in Figure 3 (Right). MCQ-based Evaluation. To objectively assess the correctness of generated images at each level of the metaphorical structure, we automatically construct a set of multiple-choice questions (MCQs) for all 1,500 samples in the dataset. Specifically, we leverage Gemini 3.1 Pro to generate MCQs based on each sample’s structured annotations along with its descriptive prompt. The generated MCQs assess four hierarchical levels of metaphorical fidelity: • L1 – Domain Presence: Whether the target and source domains are visually recognizable in the image (one question per domain). • L2 – Structure Type: Whether the visual combination strategy matches the ground-truth structure type (replacement, fusion, or juxtaposition). • L3 – Meaning: Whether the image conveys the intended metaphorical meaning. • L4 – Element Mapping: Whether each source element correctly corresponds to its intended target element (one question per mapping pair; scores are averaged). This process yields a total of 9,594 MCQs across all samples, which are manually verified by the same annotators to ensure correctness. Each question consists of four options: one ground-truth answer derived from the annotation, two plausible but incorrect distractors, and a "cannot be determined" option. All option orders are randomized to eliminate positional bias. For each image, we compute per-level accuracy: L1 averages over the two domain questions, L2 is a single binary score, L3 is a single binary score, and L4 averages over all mapping questions. The overall MCQ score is the mean across all four levels. Dimension-based Scoring. While MCQ-based evaluation measures the correctness of metaphorical elements, it does not capture the overall perceptual quality of the generated image. We therefore complement it with dimension-based scoring, where an MLLM evaluator is presented with the generated image along with its structured annotations and asked to rate the image on a 1–5 Likert scale across three independent dimensions: • Metaphoric Efficacy: How clearly the image communicates the intended metaphorical meaning to a viewer without any textual explanation. • Metaphor Logic: Whether the cross-domain mapping between source and target is conceptually coherent and logically sound. • Perceptual Harmony: How naturally and coherently the two domains are visually presented, considering blending, composition, and overall visual cohesion. The overall dimension score is the average across all three dimensions. Together with the MCQ-based evaluation, this hybrid framework provides a comprehensive assessment of both the metaphorical fidelity and expressive quality of generated visual metaphors. Full evaluation prompts are provided in Appendix A.7. Table 1: Evaluation results on VMetaphor-Bench with conceptual prompts. MCQ Accuracy (%) Dimension Score (1–5) Model Domain Structure Meaning Mapping Overall Efficacy Logic Harmony Overall Closed-source T2I Models Seedream 5.0 Lite 90.6 65.6 96.0 66.5 78.5 3.78 3.74 4.08 3.86 Nano Banana 2 93.0 76.3 98.5 75.9 84.8 4.05 4.08 4.21 4.11 GPT Image 1.5 93.6 78.1 97.6 76.5 85.4 4.27 4.23 4.38 4.30 Open-source T2I Models FLUX.1-dev 82.3 54.9 85.6 55.2 68.4 2.90 2.87 3.82 3.19 SD 3.5 Large 83.0 60.1 85.7 58.9 70.8 2.99 3.01 3.75 3.25 Z-Image 84.3 59.6 89.5 60.8 72.4 2.93 3.03 3.67 3.21 Qwen-Image 87.3 66.1 90.5 61.7 74.9 3.39 3.31 3.84 3.52 FLUX.2-dev 88.6 63.7 92.1 63.9 76.0 3.48 3.46 3.93 3.62 Open-source Unified MLLMs OmniGen2 74.8 48.1 77.1 49.1 61.4 2.53 2.44 3.45 2.74 Bagel 79.8 54.8 84.2 54.2 67.0 2.69 2.72 3.56 2.99 Janus-Pro 83.4 59.1 89.1 63.2 73.0 2.90 3.02 3.43 3.12 4 Experiments In this section, we evaluate representative T2I models on VMetaphor-Bench and analyze their behavior under different prompt granularities. We further validate the reliability of our MLLM-as-judge framework through cross-judge consistency and human alignment studies. Additional experimental details and results are provided in the Appendix. 4.1 Setup Models. We evaluate 11 representative text-to-image models spanning three categories: (1) Closed-source T2I Models. GPT Image 1.5 OpenAI (2025a), Nano Banana 2 (Gemini 3.1 Flash Image) Google (2026), and Seedream 5.0 Lite ByteDance Seed (2026). These proprietary models represent the current state of the art in text-to-image generation; (2) Open-source T2I Models. Stable Diffusion 3.5 Large Stability AI (2024), FLUX.1-dev Labs (2024), FLUX.2-dev Labs (2025), Z-Image Team (2025), and Qwen-Image Wu et al. (2025a). These models represent the leading open-source text-to-image models; (3) Open-source Unified MLLMs. OmniGen2 Wu et al. (2025b), Bagel Deng et al. (2025), and Janus-Pro Chen et al. (2025b). These models employ a unified architecture for both image generation and multimodal understanding. All models are evaluated using their default generation configurations. MLLM Evaluator. For both MCQ-based evaluation and dimension-based scoring, we adopt an open-source MLLM, Qwen3.5-27B Qwen Team (2026), as the default judge model for its strong visual reasoning capability and full reproducibility. We additionally report results with GPT-5.4 OpenAI (2025b) in the Appendix A.5 to validate cross-judge consistency. 4.2 Main Results on VMetaphor-Bench We report the main evaluation results with conceptual prompts in Table 1. All metrics are assigned by Qwen3.5-27B as the default judge. Based on the results across 11 models, we highlight several key findings regarding model capabilities and core challenges in visual metaphor generation. Closed-source models lead by a substantial margin. GPT Image 1.5 achieves the best overall performance with 85.4% MCQ accuracy and a dimension score of 4.30, followed by Nano Banana 2 (84.8%, 4.11). In contrast, the best-performing open-source T2I model, FLUX.2-dev, trails by 9.4 percentage points (76.0%, 3.62). While FLUX.2-dev narrows the gap with the weakest closed-source model Seedream 5.0 Lite (78.5%), a clear performance tier remains, suggesting that visual metaphor generation—which demands both semantic comprehension and creative compositional ability—still substantially benefits from the larger-scale training and architectural advantages of proprietary systems. Unified MLLMs show mixed performance: while Janus-Pro (73.0%) is competitive with mid-tier open-source T2I models, OmniGen2 (61.4%) and Bagel (67.0%) rank at the bottom despite their multimodal understanding capabilities. Structure and Mapping are the primary bottleneck. All models exhibit a consistent difficulty hierarchy across MCQ levels: Domain and Meaning accuracies are consistently the two highest among the four levels for every model, indicating that models can generate images containing the correct conceptual elements and conveying the intended metaphorical meaning. However, performance drops sharply on Structure (48.1–78.1%) and Mapping (49.1–76.5%), revealing that the core challenge lies in compositional structuring—models can depict the right elements but struggle to compose them into coherent metaphorical structures. This pattern is particularly evident in Seedream 5.0 Lite, which achieves 90.6% on Domain and 96.0% on Meaning yet drops to 65.6% on Structure and 66.5% on Mapping. Notably, a deeper analysis of structure type errors reveals that replacement is far more challenging than fusion or juxtaposition, a pattern that mirrors the cognitive complexity hierarchy in visual rhetoric theory Phillips and McQuarrie (2004) (see Appendix A.3). Visually harmonious but metaphorically shallow. Across all models, Perceptual Harmony is consistently the highest-scoring dimension (3.43–4.38), while Metaphoric Efficacy and Metaphor Logic generally trail behind. This suggests that although the intended meaning can be inferred (as reflected by high meaning accuracy), the metaphor is not conveyed with sufficient clarity and immediacy through visual structure—models produce aesthetically pleasing images without powerful metaphorical expression. The gap between Harmony and Efficacy widens for weaker models: OmniGen2 scores 3.45 on Harmony but only 2.53 on Efficacy (gap = 0.92), whereas GPT Image 1.5 maintains a gap of only 0.11 (4.38 vs. 4.27). Figure 4: Qualitative comparison of generated images across seven representative models on four visual metaphors. Each row corresponds to one metaphor, with the conceptual prompt shown on the left. The four metaphors cover three structure types: fusion (rows 1–2), replacement (row 3), and juxtaposition (row 4). Blue and purple text highlight the target and source domains, respectively. 4.3 Qualitative Analysis To complement the quantitative results above, Figure 4 presents representative outputs from seven models on four visual metaphors covering different structure types (fusion, replacement, and juxtaposition). Several patterns emerge that are consistent with our quantitative findings. Closed-source models such as GPT Image 1.5 and Nano Banana 2 generally produce coherent compositions that integrate the two domains in meaningful ways. In contrast, weaker open-source models often struggle to compose the two domains correctly. Mid-tier models such as FLUX.2-dev and Z-Image fall in between, occasionally capturing the metaphorical concept but producing visually disharmonious compositions or failing on structurally demanding cases. A closer look at typical failure cases reveals three recurring issues that map directly to our quantitative findings. The most common is incorrect structural composition: models depict both domains but fail to combine them in the structural form implied by the prompt. For example, in row 4, where the prompt suggests a juxtaposition of a cigarette and a handgun-shaped shadow, SD 3.5 Large, Z-Image, and Janus-Pro instead fuse the two into a single physical object—echoing the sharp performance drop on L2 Structure across all models. A second issue is poor visual integration, where the two domains are placed together but blend awkwardly, such as a coffee cup stacked on visibly distinct pillows without any unified form (row 2). Although the resulting images can still appear visually polished, the awkward integration weakens the metaphor’s expressive force, which is consistent with our finding that Metaphoric Efficacy lags behind Perceptual Harmony across most models. In the most severe cases, models omit one of the domains entirely: Z-Image, for instance, fills the claw machine with ordinary toy balls rather than business professionals (row 3), making the metaphor impossible to recover. Together, these failure modes underscore the value of a multi-level evaluation framework for diagnosing the specific weaknesses of current T2I models. 4.4 Effect of Prompt Granularity In addition to the conceptual prompts reported in our main experiments, we also generate and evaluate images using the descriptive prompts paired with each sample. Full results under descriptive prompts are provided in the Appendix A.4. Here we analyze the differences between the two prompt settings on a representative subset of five models spanning the three model categories, summarized in Table 2. We first observe that all models benefit from descriptive prompts on MCQ accuracy, and most models also see improvements on dimension scores, indicating that explicit visual specification reduces the difficulty of metaphor generation. However, the magnitude of improvement varies dramatically across models. GPT Image 1.5 gains only +3.1 percentage points on MCQ overall and shows essentially no change on dimension score (−-0.06), whereas Z-Image improves by +16.6 points and +0.69, respectively. A clear inverse relationship emerges between a model’s baseline capability and the gain it receives: stronger models exhibit smaller gains, while weaker models benefit substantially more. This pattern suggests that capable models such as GPT Image 1.5 and Nano Banana 2 already possess the ability to reason about how a metaphorical idea should be visually composed without explicit specification. In contrast, weaker models lack this inferential capacity and rely heavily on the explicit visual guidance provided by descriptive prompts. This aligns with our earlier observation that visual metaphor generation requires more than literal rendering: it demands a degree of conceptual reasoning, on which current open-source models still fall short. Beyond the per-model gains, descriptive prompts also narrow the capability gap between models. Under conceptual prompts, the dimension score gap between GPT Image 1.5 (4.30) and Z-Image (3.21) is 1.09; under descriptive prompts, this gap narrows to 0.34 (4.24 vs. 3.90). For practitioners working with open-source models, providing fine-grained visual descriptions is therefore an effective strategy to compensate for the limited metaphorical reasoning capability. Table 2: Performance comparison between conceptual and descriptive prompts on representative models. “Conc.” and “Desc.” denote the conceptual and descriptive prompt settings, respectively. Δ denotes the change when switching from conceptual to descriptive prompts. MCQ Overall (%) Dimension Overall Model Conc. Desc. Δ Conc. Desc. Δ GPT Image 1.5 85.4 88.5 +3.1 4.30 4.24 −-0.06 Nano Banana 2 84.8 89.2 +4.4 4.11 4.18 +0.07 FLUX.2-dev 76.0 87.3 +11.3 3.62 4.09 +0.47 Z-Image 72.4 85.9 +13.5 3.21 3.90 +0.69 OmniGen2 61.4 78.0 +16.6 2.74 3.35 +0.61 4.5 Reliability of the Evaluator To validate the reliability of using Qwen3.5-27B as our default evaluator, we conduct two studies: a cross-judge consistency study assessing robustness to the choice of evaluator, and a human alignment study measuring agreement with human judgment. Cross-Judge Consistency. To test whether our conclusions depend on the specific choice of evaluator, we re-run the full evaluation pipeline with GPT-5.4 OpenAI (2025b) as an alternative judge and compare model rankings and scores against those produced by Qwen3.5-27B. As shown in Table 3, the top-5 models receive identical rankings from both judges across both metrics. Over all 11 models, Spearman’s ρ reaches 0.991 for MCQ and 0.936 for dimension scores, confirming that our findings are not dependent on the choice of evaluator. Table 3: Cross-judge comparison. Model rankings by MCQ Overall and Dimension Overall under two judges. MCQ Dimension Model Qwen3.5 GPT-5.4 Qwen3.5 GPT-5.4 GPT Image 1.5 1 1 1 1 Nano Banana 2 2 2 2 2 Seedream 5.0 Lite 3 3 3 3 FLUX.2-dev 4 4 4 4 Qwen-Image 5 5 5 5 Human Alignment Study. We further evaluate how well the two MLLM judges align with human annotators. We randomly sample 100 images each from FLUX.2-dev and Qwen-Image, yielding 200 images in total. Five human annotators, trained on visual metaphor theory and our annotation guidelines, independently evaluate each image using the same protocol applied to the MLLM evaluator. We measure alignment using balanced accuracy Brodersen et al. (2010) for MCQ evaluation and Mean Absolute Error (MAE) for dimension-based scoring, following established practice in recent benchmark studies Wang et al. (2026); Zhao et al. (2025). Results are reported in Table 4. The MCQ alignment is consistently high across all four levels, with both Qwen3.5-27B and GPT-5.4 achieving over 80% balanced accuracy overall. Alignment is highest on the more concrete L1 (Domain) and L3 (Meaning) levels (above 83%), and lower on L2 (Structure) and L4 (Mapping), which require finer-grained judgments about how source and target are visually combined. For dimension-based scoring, Qwen3.5-27B yields a low MAE across all three dimensions (below 0.8 on a 1–5 scale), indicating that its ratings are close to those of human annotators. GPT-5.4 shows slightly larger errors (up to 0.89 on Logic) but remains within a reasonable range (< 1.0). The fact that an open-source evaluator achieves alignment comparable to or better than a leading proprietary model further supports the choice of Qwen3.5-27B as our default judge. Table 4: Alignment between MLLM evaluators (Qwen3.5-27B and GPT-5.4) and human annotators on 200 sampled images. (a) MCQ alignment by balanced accuracy (%); (b) dimension alignment by mean absolute error (MAE) on a 1–5 scale. (a) MCQ alignment (Bal.Acc, %) Judge Domain Structure Meaning Mapping Overall Qwen3.5-27B 87.61 71.25 83.97 76.45 80.24 GPT-5.4 88.79 73.89 84.50 76.51 81.16 (b) Dimension alignment (MAE) Judge Efficacy Logic Harmony Qwen3.5-27B 0.65 0.71 0.64 GPT-5.4 0.71 0.89 0.75 5 Conclusion In this paper, we introduce VMetaphor-Bench, the first benchmark for evaluating visual metaphor generation in text-to-image models. Built upon 1,500 carefully curated visual metaphors with structured annotations, our benchmark provides a hierarchical taxonomy of three levels and ten categories and a hybrid evaluation framework that combines MCQ-based fidelity assessment with dimension-based scoring. Extensive evaluation of 11 representative T2I models reveals that proprietary models substantially outperform open-source counterparts, yet even the strongest models continue to struggle with compositional structuring and cross-domain mapping—key aspects of metaphorical expression. We hope VMetaphor-Bench serves as a foundation for future research on bridging the gap between literal rendering and creative, meaning-rich visual generation. References [1] A. R. Akula, B. Driscoll, P. Narayana, S. Changpinyo, Z. Jia, S. Damle, G. Pruthi, S. Basu, L. Guibas, W. T. Freeman, et al. (2023) Metaclue: towards comprehensive visual metaphors research. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 23201–23211. Cited by: §1, §2.3. [2] K. H. Brodersen, C. S. Ong, K. E. Stephan, and J. M. Buhmann (2010) The balanced accuracy and its posterior distribution. In 2010 20th international conference on pattern recognition, p. 3121–3124. Cited by: §4.5. [3] ByteDance Seed (2026) Seedream 5.0 lite. Note: https://seed.bytedance.com/en/seedream5_0_lite Cited by: Figure 1, Figure 1, §4.1. [4] S. Cappa, A. S. Lippolis, and S. Zoia (2025) Meanings are like onions: a layered approach to metaphor processing. arXiv preprint arXiv:2507.10354. Cited by: §3.1. [5] D. Chen, R. Chen, S. Zhang, Y. Wang, Y. Liu, H. Zhou, Q. Zhang, Y. Wan, P. Zhou, and L. Sun (2024) Mllm-as-a-judge: assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning, Cited by: §2.2. [6] K. Chen, Z. Lin, Z. Xu, Y. Shen, Y. Yao, J. Rimchala, J. Zhang, and L. Huang (2025) R2i-bench: benchmarking reasoning-driven text-to-image generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 12606–12641. Cited by: §2.2. [7] X. Chen, Z. Wu, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, and C. Ruan (2025) Janus-pro: unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811. Cited by: §1, §2.1, §4.1. [8] C. Deng, D. Zhu, K. Li, C. Gou, F. Li, Z. Wang, S. Zhong, W. Yu, X. Nie, Z. Song, et al. (2025) Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683. Cited by: §1, §2.1, §4.1. [9] P. Dhariwal and A. Nichol (2021) Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34, p. 8780–8794. Cited by: §2.1. [10] F. Ervas (2021) Metaphor, ignorance and the sentiment of (ir) rationality. Synthese 198 (7), p. 6789–6813. Cited by: §2.3. [11] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. (2024) Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, Cited by: §2.1. [12] G. Fauconnier and M. Turner (2003) Conceptual blending, form and meaning. Recherches en communication 19, p. 57–86. Cited by: §1. [13] C. Forceville et al. (2008) Metaphor in pictures and multimodal representations. The Cambridge handbook of metaphor and thought, p. 462–482. Cited by: §1, §2.3. [14] D. Ghosh, H. Hajishirzi, and L. Schmidt (2023) Geneval: an object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems 36, p. 52132–52152. Cited by: §1, §2.2. [15] Google DeepMind (2026) Gemini 3.1 pro. Note: https://deepmind.google/models/gemini/pro/ Cited by: §3.1. [16] Google (2026) Gemini 3.1 Flash Image Preview. Note: https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-image-preview Cited by: §1, §2.1, §4.1. [17] J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi (2021) Clipscore: a reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 7514–7528. Cited by: §2.2. [18] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30. Cited by: §2.2. [19] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §1, §2.1. [20] Y. Hu, B. Liu, J. Kasai, Y. Wang, M. Ostendorf, R. Krishna, and N. A. Smith (2023) Tifa: accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 20406–20417. Cited by: §2.2. [21] K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu (2023) T2i-compbench: a comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems 36, p. 78723–78747. Cited by: §1, §2.2. [22] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. (2024) Gpt-4o system card. arXiv preprint arXiv:2410.21276. Cited by: §2.2. [23] A. Kamath, K. Chang, R. Krishna, L. Zettlemoyer, Y. Hu, and M. Ghazvininejad (2025) GenEval 2: addressing benchmark drift in text-to-image evaluation. arXiv preprint arXiv:2512.16853. Cited by: §2.2. [24] G. A. Koushik, F. Nazarieh, K. Birch, S. Qian, and D. Kanojia (2025) The mind’s eye: a multi-faceted reward framework for guiding visual metaphor generation. arXiv preprint arXiv:2508.18569. Cited by: §2.3. [25] M. Kundu, S. Shekhar, and P. Bhattacharyya (2025) Looking beyond the pixels: evaluating visual metaphor understanding in vlms. Findings of the Association for Computational Linguistics: EMNLP, p. 23137–23158. Cited by: §1, §2.3. [26] B. F. Labs (2024) FLUX. Note: https://github.com/black-forest-labs/flux Cited by: §1, §2.1, §4.1. [27] B. F. Labs (2025) FLUX.2: Frontier Visual Intelligence. Note: https://bfl.ai/blog/flux-2 Cited by: Figure 1, Figure 1, §4.1. [28] Y. Niu, M. Ning, M. Zheng, W. Jin, B. Lin, P. Jin, J. Liao, C. Feng, K. Ning, B. Zhu, et al. (2025) Wise: a world knowledge-informed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.07265. Cited by: §1, §2.2. [29] OpenAI (2025) GPT-Image-1.5. Note: https://developers.openai.com/api/docs/models/gpt-image-1.5 Cited by: Figure 1, Figure 1, §1, §2.1, §4.1. [30] OpenAI (2025) Introducing gpt-5.4. Note: https://openai.com/zh-Hans-CN/index/introducing-gpt-5-4/ Cited by: §4.1, §4.5. [31] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4195–4205. Cited by: §2.1. [32] B. J. Phillips and E. F. McQuarrie (2004) Beyond visual metaphor: a new typology of visual rhetoric in advertising. Marketing theory 4 (1-2), p. 113–136. Cited by: §A.3, 3rd item, §4.2. [33] Pinterest. Note: https://w.pinterest.com Cited by: §3.1. [34] Qwen Team (2026) Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §4.1. [35] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10684–10695. Cited by: §1, §2.1. [36] Stability AI (2024) Stable diffusion 3.5 large. Note: https://huggingface.co/stabilityai/stable-diffusion-3.5-large Cited by: §4.1. [37] K. Sun, R. Fang, C. Duan, X. Liu, and X. Liu (2025) T2i-reasonbench: benchmarking reasoning-informed text-to-image generation. arXiv preprint arXiv:2508.17472. Cited by: §1, §2.2. [38] P. Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, P. Luo, and Z. Yuan (2024) Autoregressive model beats diffusion: llama for scalable image generation. arXiv preprint arXiv:2406.06525. Cited by: §2.1. [39] Z. Sun, Z. Zhang, Y. Zhang, M. Lu, D. Lischinski, D. Cohen-Or, and H. Huang (2025) Creative blends of visual concepts. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, p. 1–17. Cited by: §2.3. [40] C. Team (2024) Chameleon: mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818. Cited by: §1, §2.1. [41] G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. (2023) Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: §2.2. [42] Z. Team (2025) Z-image: an efficient image generation foundation model with single-stream diffusion transformer. arXiv preprint arXiv:2511.22699. Cited by: §4.1. [43] K. Tian, Y. Jiang, Z. Yuan, B. Peng, and L. Wang (2024) Visual autoregressive modeling: scalable image generation via next-scale prediction. Advances in neural information processing systems 37, p. 84839–84865. Cited by: §1, §2.1. [44] M. Van Mulken, A. Van Hooft, and U. Nederstigt (2014) Finding the tipping point: visual metaphor and conceptual complexity in advertising. Journal of Advertising 43 (4), p. 333–343. Cited by: §2.3. [45] Z. Wang, J. Xu, Z. Zhang, T. Pang, C. Du, H. Zhao, and Z. Zhao (2025) GenSpace: benchmarking spatially-aware image generation. arXiv preprint arXiv:2505.24870. Cited by: §2.2. [46] Z. Wang, X. Hu, Y. Wang, F. Xiong, M. Zhang, and X. Chu (2026) Everything in its place: benchmarking spatial intelligence of text-to-image models. arXiv preprint arXiv:2601.20354. Cited by: §2.2, §4.5. [47] C. Wu, J. Li, J. Zhou, J. Lin, K. Gao, K. Yan, S. Yin, S. Bai, X. Xu, Y. Chen, Y. Chen, Z. Tang, Z. Zhang, Z. Wang, A. Yang, B. Yu, C. Cheng, D. Liu, D. Li, H. Zhang, H. Meng, H. Wei, J. Ni, K. Chen, K. Cao, L. Peng, L. Qu, M. Wu, P. Wang, S. Yu, T. Wen, W. Feng, X. Xu, Y. Wang, Y. Zhang, Y. Zhu, Y. Wu, Y. Cai, and Z. Liu (2025) Qwen-image technical report. External Links: 2508.02324, Link Cited by: §1, §4.1. [48] C. Wu, P. Zheng, R. Yan, S. Xiao, X. Luo, Y. Wang, W. Li, X. Jiang, Y. Liu, J. Zhou, et al. (2025) Omnigen2: exploration to advanced multimodal generation. arXiv preprint arXiv:2506.18871. Cited by: §1, §2.1, §4.1. [49] J. Xie, Z. Yang, and M. Z. Shou (2025) Show-o2: improved native unified multimodal models. arXiv preprint arXiv:2506.15564. Cited by: §2.1. [50] Y. Xu, Y. Zhang, J. Cao, L. Gao, C. Wang, O. Deussen, T. Lee, and F. Tang (2026) Beyond pixels: visual metaphor transfer via schema-driven agentic reasoning. arXiv preprint arXiv:2602.01335. Cited by: §2.3. [51] J. Ye, D. Jiang, Z. Wang, L. Zhu, Z. Hu, Z. Huang, J. He, Z. Yan, J. Yu, H. Li, et al. (2025) Echo-4o: harnessing the power of gpt-4o synthetic images for improved image generation. arXiv preprint arXiv:2508.09987. Cited by: §2.2. [52] C. Zhang, Y. Niu, and H. Li (2026) MetaphorStar: image metaphor understanding and reasoning with end-to-end visual reinforcement learning. arXiv preprint arXiv:2602.10575. Cited by: §1, §2.3. [53] D. Zhang, C. Jiang, R. Xu, B. Chen, Z. Jin, Y. Lu, J. Zhang, L. Yong, J. Luo, and S. Luo (2025) Worldgenbench: a world-knowledge-integrated benchmark for reasoning-driven text-to-image generation. arXiv preprint arXiv:2505.01490. Cited by: §2.2. [54] Z. Zhang, J. Wang, F. Wen, Y. Guo, X. Zhao, X. Fang, S. Ding, Z. Jia, J. Xiao, Y. Shen, et al. (2025) Large multimodal models evaluation: a survey. Science China Information Sciences 68 (12), p. 221301. Cited by: §3.2. [55] X. Zhao, P. Zhang, K. Tang, X. Zhu, H. Li, W. Chai, Z. Zhang, R. Xia, G. Zhai, J. Yan, et al. (2025) Envisioning beyond the pixels: benchmarking reasoning-informed visual editing. arXiv preprint arXiv:2504.02826. Cited by: §1, §2.2, §4.5. Appendix A Appendix A.1 Benchmark Statistics Table 5 summarizes the dataset statistics. Conceptual prompts are concise (mean 25.6 words), while descriptive prompts provide richer detail (mean 65.0 words); appending the style suffix roughly doubles the length in both cases. We organize the 1,500 images into 10 thematic categories across three levels: Individual (495 images) covers topics related to personal cognition, emotion, health, and growth; External (499 images) encompasses societal, environmental, technological, and interpersonal themes; and Creative (506 images) includes artistic designs and commercial advertisements. The distribution is approximately balanced across levels, with the largest single category being Creative Design (21.07%). Table 5: Left: Prompt length statistics (word count). Right: Distribution of thematic categories. Mean Min Max Conceptual Prompt 25.6 10 56 + Style Suffix 68.4 29 133 Descriptive Prompt 65.0 20 142 + Style Suffix 107.7 38 226 Level Category Count Prop.(%) Individual (495) Cognitive Processes 198 13.20 Mental States 161 10.73 Healthy Living 72 4.80 Personal Development 64 4.27 External (499) Social Issues 243 16.20 Natural Environment 119 7.93 Technological Life 88 5.87 Social Interaction 49 3.27 Creative (506) Creative Design 316 21.07 Advertising 190 12.67 A.2 Experimental Setup Details We conduct all experiments on a single server equipped with 4 × NVIDIA H200 GPUs. Detailed information about each evaluated model and MLLM evaluators is provided in Table 6. Table 6: Detailed information about the evaluated T2I models and MLLM evaluators. Model Year Resolution Source Version / HF Checkpoint Closed-source T2I Models Seedream 5.0 Lite 2026 2K API doubao-seedream-5-0-260128 Nano Banana 2 2026 1024 × 1024 API gemini-3.1-flash-image-preview GPT Image 1.5 2025 1024 × 1024 API gpt-image-1.5-2025-12-16 Open-source T2I Models FLUX.1-dev 2024 1024 × 1024 Checkpoint black-forest-labs/FLUX.1-dev SD 3.5 Large 2024 1024 × 1024 Checkpoint stabilityai/stable-diffusion-3.5-large Z-Image 2025 1024 × 1024 Checkpoint Tongyi-MAI/Z-Image-Turbo Qwen-Image 2025 1328 × 1328 Checkpoint Qwen/Qwen-Image-2512 FLUX.2-dev 2025 1024 × 1024 Checkpoint black-forest-labs/FLUX.2-dev Open-source Unified MLLMs OmniGen2 2025 1024 × 1024 Checkpoint OmniGen2/OmniGen2 Bagel 2025 1024 × 1024 Checkpoint ByteDance-Seed/BAGEL-7B-MoT Janus-Pro 2025 384 × 384 Checkpoint deepseek-ai/Janus-Pro-7B MLLM Evaluators Qwen3.5-27B 2026 – Checkpoint Qwen/Qwen3.5-27B GPT-5.4 2026 – API gpt-5.4-2026-03-05 A.3 Per-Structure Analysis To better understand where models struggle on the L2 Structure level, we examine the structure type predicted by the MCQ evaluation against the ground-truth structure type annotated in our dataset. Each L2 MCQ question asks the judge to identify the structure type of the generated image among fusion, replacement, juxtaposition, and cannot be determined, allowing us to construct a confusion matrix between ground-truth and predicted structure types. Aggregated results across all 11 evaluated models are shown in Figure 5. The confusion matrix reveals a striking asymmetry across structure types. Models achieve reasonable accuracy on fusion (67.9%) and juxtaposition (69.2%), where the two domains are either merged into a single object or placed alongside each other. In contrast, replacement accuracy drops dramatically to only 18.9%. When prompted with replacement metaphors, models tend to either fuse the two domains together (37.8%) or place them as separate objects (36.6%), rather than substituting one for the other. Interestingly, this difficulty hierarchy mirrors the cognitive complexity ordering proposed in visual rhetoric theory [32], where replacement is identified as the most demanding structure type for human viewers because the missing element must be inferred from its absence. In contrast, fusion requires disentangling two co-present elements, while juxtaposition simply pairs them side by side. Our results extend this ordering from comprehension to generation: T2I models exhibit the same complexity gradient when producing visual metaphors. This suggests that the cognitive demands of replacement, which require preserving context while substituting a localized element, pose a fundamental challenge for current models. This finding points to compositional precision, the ability to perform localized, structure-aware modifications, as a key direction for advancing visual metaphor generation. Figure 5: Confusion matrix of predicted vs. ground-truth structure types, aggregated across all models. A.4 Full Results with Descriptive Prompts Table 7 reports the full evaluation results under descriptive prompts, complementing the conceptual prompt results in Table 1. Most models show substantial improvements under descriptive prompts, as the detailed visual descriptions provide more explicit guidance for generation. A notable exception is GPT Image 1.5, whose dimension score remains essentially unchanged (−-0.06), suggesting that the strongest proprietary models can already infer appropriate visual realizations from abstract conceptual prompts and do not require further specification. Despite these magnitude differences, the overall model ranking remains largely consistent across the two prompt types. Table 7: Evaluation results on VMetaphor-Bench with descriptive prompts. MCQ Accuracy (%) Dimension Score (1–5) Model Domain Structure Meaning Mapping Overall Efficacy Logic Harmony Overall Closed-source T2I Models Seedream 5.0 Lite 89.3 69.2 96.5 87.4 86.6 4.02 4.14 4.27 4.15 GPT Image 1.5 89.6 72.9 96.6 90.6 88.5 4.12 4.24 4.34 4.24 Nano Banana 2 90.8 73.6 97.1 91.2 89.2 4.06 4.21 4.28 4.18 Open-source T2I Models SD 3.5 Large 83.0 62.3 93.4 76.0 78.8 3.36 3.45 3.82 3.54 FLUX.1-dev 83.8 64.1 93.5 79.6 80.7 3.45 3.55 3.97 3.66 Z-Image 87.9 67.7 96.4 87.5 85.9 3.76 3.90 4.04 3.90 Qwen-Image 88.6 70.9 95.9 88.5 87.0 3.98 4.03 4.08 4.03 FLUX.2-dev 89.4 70.3 96.4 88.9 87.3 4.00 4.09 4.18 4.09 Open-source Unified MLLMs OmniGen2 82.6 59.9 93.2 75.3 78.0 3.07 3.26 3.71 3.35 Bagel 83.2 62.7 92.6 76.8 79.1 3.28 3.36 3.77 3.47 Janus-Pro 84.0 65.9 94.9 79.1 81.0 3.31 3.52 3.62 3.48 A.5 Cross-Judge Results with GPT-5.4 Table 8 reports the full evaluation results using GPT-5.4 as the judge, complementing the Qwen3.5-27B results in Table 1. Across all 11 models, GPT-5.4 produces highly consistent rankings with Qwen3.5-27B, with Spearman rank correlations of 0.991 for MCQ and 0.936 for dimension scoring. The two judges differ slightly in absolute scores, with GPT-5.4 tending to assign somewhat more lenient dimension scores. However, the relative performance ordering of models is preserved, confirming that the conclusions reported in our main experiments do not depend on the specific choice of evaluator. Table 8: Evaluation results on VMetaphor-Bench with conceptual prompts, judged by GPT-5.4. MCQ Accuracy (%) Dimension Score (1–5) Model Domain Structure Meaning Mapping Overall Efficacy Logic Harmony Overall Closed-source T2I Models Seedream 5.0 Lite 93.3 66.7 96.3 73.0 82.0 3.72 3.85 3.94 3.84 Nano Banana 2 94.3 75.6 98.3 76.9 85.5 4.08 4.29 4.14 4.17 GPT Image 1.5 94.4 77.2 97.6 77.2 85.8 4.13 4.37 4.33 4.27 Open-source T2I Models FLUX.1-dev 91.3 63.8 91.9 62.8 76.4 2.95 3.07 3.70 3.24 SD 3.5 Large 91.5 65.7 91.4 64.6 77.4 2.97 3.14 3.68 3.26 Z-Image 93.1 66.5 94.8 65.6 78.9 3.05 3.25 3.56 3.28 Qwen-Image 91.9 70.3 93.9 67.3 79.6 3.32 3.45 3.73 3.50 FLUX.2-dev 94.0 69.3 94.7 69.0 80.9 3.45 3.62 3.84 3.64 Open-source Unified MLLMs OmniGen2 90.2 62.0 90.9 57.6 73.7 2.51 2.65 3.42 2.86 Bagel 91.7 64.7 92.6 62.1 76.6 2.78 2.92 3.44 3.05 Janus-Pro 92.3 67.1 93.9 67.6 79.4 3.06 3.34 3.49 3.30 A.6 Benchmark Construction Prompts. Figure 6 and Figure 7 show the prompts for the two-stage annotation pipeline: structured annotation extraction (Stage 1) and descriptive/conceptual prompt generation (Stage 2). Figure 8 shows the prompt for MCQ distractor generation. Figure 6: Prompt for structured annotation generation (Stage 1). Figure 7: Prompt for descriptive and conceptual prompt generation (Stage 2). Figure 8: Prompt for MCQ distractor generation. A.7 Evaluation Prompts Figure 9 and Figure 10 show the prompts for MCQ evaluation, which is split into two calls: Q1–Q4 (target, source, structure, and meaning) are answered together, while Q5 (element mapping) is answered separately to prevent domain leakage from its answer options. Specifically, Q5 (element mapping) options may reveal the source domain (e.g., a question stem mentioning ’spherical bomb casing’ would expose ’bomb’ as the source), which would compromise the independence of L1 and L4 evaluations. Figure 11 shows the prompt for dimension scoring with 1–5 Likert-scale rubrics. Figure 9: Prompt for MCQ evaluation (Q1–Q4: target domain, source domain, structure type, and meaning). Figure 10: Prompt for MCQ evaluation (Q5: Element mapping). Figure 11: Prompt for dimension score evaluation. A.8 Interface for Human Annotators We provide screenshots of the annotation interfaces used in our study. Figure 12 shows the interface for manual verification during data curation (Section 3.1), where annotators review and edit metadata fields such as domains, structure type, element mappings, and meaning. Figures 13 and 14 show the two pages of the interface used in the human alignment study (Section 4.5): one for answering MCQ questions alongside the generated image, and another for rating dimension scores on a 1–5 scale. Figure 12: Manual verification interface for data curation Figure 13: Human evaluation interface for the alignment study: MCQ answering Figure 14: Human evaluation interface for the alignment study: dimension rating A.9 Samples We present representative samples in Figure 15. For each category, we select one visual metaphor and display the generated images from all evaluated models under the conceptual prompt setting. These examples illustrate the diversity of metaphorical themes in our benchmark and highlight the varying capabilities of different models in generating visual metaphors. Figure 15: Generated visual metaphors across ten categories from VMetaphor-Bench. A.10 Limitations We acknowledge several limitations. First, the distribution of samples across thematic categories is uneven, reflecting the natural prevalence of visual metaphors in different topics in real-world creative imagery. Second, our evaluation relies on an MLLM-as-judge framework, which, despite strong alignment with human annotators, inherits the fundamental limitations of automatic visual reasoning on the most subjective judgments. Finally, our prompts and source imagery are predominantly English and from Western creative traditions, leaving multilingual prompts and cross-cultural metaphors as future directions. A.11 Broader Impacts This work advances the evaluation of visual metaphor generation, a challenging aspect of text-to-image models that requires coherent cross-domain reasoning and expressive visual composition. By providing a structured benchmark with fine-grained evaluation criteria, it can support the development of more capable and interpretable generative models. Such progress may benefit real-world applications including creative design, education, and advertising, where visual metaphors are widely used to communicate messages effectively and engage audiences. We also acknowledge potential negative implications. Improved visual metaphor generation could be misused to produce persuasive misleading imagery (e.g., manipulative advertising or political propaganda) that exploits the strong emotional and rhetorical effects of visual analogies. We encourage future work to consider safeguards against such misuse, and our benchmark itself is intended for evaluation rather than direct deployment. A.12 Data Licensing and Release To respect the rights of original content creators on Pinterest, we do not redistribute the source images. Our public release consists of: (1) image URLs pointing to the original Pinterest pages, (2) all structured annotations (target/source domains, structure types, meaning, element mappings), (3) descriptive and conceptual prompts, and (4) the 9,594 multiple-choice questions. Users can retrieve images via the URLs for research purposes following Pinterest’s terms of service. The annotations and prompts are released under C-BY-4.0.

Instant research discovery