Paper deep dive
Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing
Shaoan Zhao, Fang Zhao, Xueqiang Guo, Xinpei Su, Huanlin Gao, Qiang Hui, Ting Lu, Fuyuan Shi, Chao Tan, Bikun Yang, Kai Wang, Shiguo Lian
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/26/2026, 5:40:14 AM
Summary
The paper introduces QC-T2I-Bench, a question-centric framework for evaluating text-to-image (T2I) models. It converts prompts into atomic questions organized via Davidsonian Scene Graphs (DSGs) to enable hierarchy-constrained aggregation, dependency-aware scoring, and fine-grained diagnosis. The framework supports reliable model ranking, structural failure analysis, and training-free cost-aware routing, demonstrating a 21.3% reduction in GPU cost while matching the performance of ERNIE-Image.
Entities (7)
Relation Signals (6)
Cost-Aware Router → matchesperformanceof → ERNIE-Image
confidence 95% · our cost-aware router matches ERNIE's 89.51-point estimate
Cost-Aware Router → reducescostby → 21.3%
confidence 95% · with 21.3% less GPU-s/MP
QC-T2I-Bench → uses → Davidsonian Scene Graphs
confidence 92% · organizes their dependencies with Davidsonian Scene Graphs (DSGs)
QC-T2I-Bench → evaluates → FLUX2-dev
confidence 90% · We evaluate multiple open-source T2I models... FLUX.2-dev
QC-T2I-Bench → uses → Hierarchy-Constrained Question Aggregation
confidence 90% · We use hierarchy-constrained question aggregation to exclude downstream questions after a prerequisite fails
QC-T2I-Bench → powers → Cost-Aware Router
confidence 88% · Finally, we reuse the same records for training-free routing; our cost-aware router...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Modern text-to-image (T2I) models often have similar total scores but different strengths, making practical selection difficult. Fine-grained benchmarks decompose prompts into questions, yet often return them to prompt scores and fixed categories, weakening attribution and ignoring complexity. Related requirements are also scored separately or as one total, obscuring basic versus compositional failure. We present QC-T2I-Bench, a question-centric framework that converts open prompts into attributed atomic questions and organizes their dependencies with Davidsonian Scene Graphs (DSGs). We use hierarchy-constrained question aggregation to exclude downstream questions after a prerequisite fails and to prevent simple and complex prompts from receiving the same total weight. We then use the DSG structure to measure joint success within prompts and compare repeated entities across prompts, separating basic realization failures from failures under additional requirements. We evaluate multiple open-source T2I models on English and Chinese prompts. The resulting question-level evidence supports reliable ranking and fine-grained diagnosis: joint completion falls from 80.7\% for components with two capabilities to 37.2\% for those with seven or more. Finally, we reuse the same records for training-free routing; our cost-aware router matches ERNIE's 89.51-point estimate with 21.3\% less GPU-s/MP.
Tags
Links
- Source: https://arxiv.org/abs/2608.24112v1
- Canonical: https://arxiv.org/abs/2608.24112v1
Trouble viewing inline? Open PDF directly →
Full Text
43,941 characters extracted from source content.
Expand or collapse full text
Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing Shaoan Zhao Affiliation: Data Science & Artificial Intelligence Research Institute, China Unicom Affiliation: Unicom Data Intelligence, China Unicom Fang Zhao Affiliation: Data Science & Artificial Intelligence Research Institute, China Unicom Affiliation: Unicom Data Intelligence, China Unicom Xueqiang Guo Affiliation: Data Science & Artificial Intelligence Research Institute, China Unicom Affiliation: Unicom Data Intelligence, China Unicom Xinpei Su Affiliation: Data Science & Artificial Intelligence Research Institute, China Unicom Affiliation: Unicom Data Intelligence, China Unicom Huanlin Gao Affiliation: Data Science & Artificial Intelligence Research Institute, China Unicom Affiliation: Unicom Data Intelligence, China Unicom Qiang Hui Affiliation: Data Science & Artificial Intelligence Research Institute, China Unicom Affiliation: Unicom Data Intelligence, China Unicom Ting Lu Affiliation: Data Science & Artificial Intelligence Research Institute, China Unicom Affiliation: Unicom Data Intelligence, China Unicom Fuyuan Shi Affiliation: Data Science & Artificial Intelligence Research Institute, China Unicom Affiliation: Unicom Data Intelligence, China Unicom Chao Tan Affiliation: Data Science & Artificial Intelligence Research Institute, China Unicom Affiliation: Unicom Data Intelligence, China Unicom Bikun Yang Affiliation: China Unicom Group Co.,Ltd Kai Wang Affiliation: Data Science & Artificial Intelligence Research Institute, China Unicom Affiliation: Unicom Data Intelligence, China Unicom Shiguo Lian Affiliation: Data Science & Artificial Intelligence Research Institute, China Unicom Affiliation: Unicom Data Intelligence, China Unicom Abstract Modern text-to-image (T2I) models often have similar total scores but different strengths, making practical selection difficult. Fine-grained benchmarks decompose prompts into questions, yet often return them to prompt scores and fixed categories, weakening attribution and ignoring complexity. Related requirements are also scored separately or as one total, obscuring basic versus compositional failure. We present QC-T2I-Bench, a question-centric framework that converts open prompts into attributed atomic questions and organizes their dependencies with Davidsonian Scene Graphs (DSGs). We use hierarchy-constrained question aggregation to exclude downstream questions after a prerequisite fails and to prevent simple and complex prompts from receiving the same total weight. We then use the DSG structure to measure joint success within prompts and compare repeated entities across prompts, separating basic realization failures from failures under additional requirements. We evaluate multiple open-source T2I models on English and Chinese prompts. The resulting question-level evidence supports reliable ranking and fine-grained diagnosis: joint completion falls from 80.7% for components with two capabilities to 37.2% for those with seven or more. Finally, we reuse the same records for training-free routing; our cost-aware router matches ERNIE’s 89.51-point estimate with 21.3% less GPU-s/MP. 00footnotetext: *Equal contribution. † author. Introduction Recent text-to-image (T2I) models have greatly improved image quality and prompt following (Esser and others 2024; Black Forest Labs 2024; Wu and others 2025). As models become stronger, choosing between them becomes harder. Models with similar aggregate scores may have very different capability profiles: one may render text accurately, another may handle spatial relations reliably, and yet another may better follow knowledge-intensive prompts. Practical model selection therefore needs more than a leaderboard. It needs evidence about where each model succeeds, why it fails, and which model best fits a particular request. Conventional metrics provide global image- or prompt-level scores rather than evidence about individual requirements (Hessel et al. 2021). Evaluation has therefore moved toward question-generation and visual-question-answering (QG/A) methods that decompose a prompt into local checks (Hu et al. 2023; Cho et al. 2024; Li et al. 2024). This is an important step, but decomposition alone does not guarantee fine-grained evaluation. The central issue is how the resulting evidence is attributed, aggregated, and reused. Many evaluation pipelines still organize this evidence around prompts. They collect prompts under predefined categories, generate local checks, average the checks into a prompt score, and assign that score back to the prompt’s category (Hu et al. 2023; Cho et al. 2024; Wei and others 2025; Li and others 2026). This design creates several recurring problems. A complex prompt may contain requirements from multiple capabilities, making its category score hard to interpret. A new request may not fit the predefined categories. Simple and complex prompts may receive the same total weight even though they contain different numbers of requirements. A missing parent object may cause several dependent checks to fail, exaggerating one error. Finally, evidence about the same requirement is rarely compared across prompts, making it difficult to tell whether a model fails on the basic content or only after additional constraints are added. Table 1 summarizes these gaps. Atomic attribution (A), open-prompt evaluation (O), complexity awareness (C), and cross-prompt evidence (X) are four desired properties. Dependency awareness (D) is a necessary safeguard against repeated penalties caused by one missing prerequisite. Existing benchmarks support different subsets of these properties, but none combines all five. Benchmark A O D C X Arena-T2I Hard ConceptMix CVTG-2K DPG-Bench DSG EvalMuse-40K GenAI-Bench GenEval GenEval 2 LongTextBench OneIG-Bench PhyBench PRISM-Bench Qwen-Image-Bench T2I-CompBench++ T2I-CoReBench TIFA TIIF-Bench WISE QC-T2I-Bench Table 1: Support for five structural properties across T2I benchmarks. , , and denote full, partial, and no support. A: atomic attribution; O: open prompts; D: dependency awareness; C: complexity awareness; X: cross-prompt evidence. We introduce QC-T2I-Bench, which changes the basic unit of evaluation from the prompt to the atomic question. Each record preserves the question’s capability label, visual answer, dependency validity, and prompt context. A two-level taxonomy organizes questions into 21 capabilities under Non-text Entities, Text Content, Attributes, Relations, and High-Level Semantics. Hierarchy-Constrained Question Aggregation (HCQ) removes dependency-invalid evidence and balances capabilities instead of giving every prompt the same total weight. Because prompts use a shared question schema rather than benchmark-specific prompt categories, the same protocol can also be applied to new requests. We further use DSGs to organize the questions in each prompt according to their dependencies (Cho et al. 2024). This lets us test whether a model can satisfy a group of related requirements together, rather than only checking them one by one. We then compare repeated entities across different prompts to distinguish a basic generation failure from a failure caused by additional attributes or relations. The same question records also form a capability profile for each model and can guide generator selection for new requests. We evaluate QC-T2I-Bench on 13 T2I models with bilingual evidence. The framework supports three connected uses: reliable ranking, fine-grained diagnosis, and cost-aware routing. Bootstrap analysis separates most model pairs while preserving uncertainty for the closest systems. The DSG analysis shows where models fail as related requirements accumulate. Finally, a training-free cost-aware router matches the fixed ERNIE point estimate while reducing GPU-s/MP by 21.3%. Our contributions are threefold. (1) We introduce QC-T2I-Bench, which converts open prompts into attributed atomic questions and uses validity-aware Hierarchy-Constrained Question Aggregation (HCQ) to avoid prompt-level normalization. (2) We develop a DSG-based compositional analysis that combines complete-component success, cross-prompt root controls, and topology-matched contrasts to localize failures. (3) We demonstrate reliable ranking, fine-grained diagnosis, and training-free quality–cost routing, including a 21.3% cost reduction at the matched ERNIE point estimate. Related Work Fine-grained evaluation benchmarks. T2I evaluation has progressed from controlled property tests to question-based verification. GenEval (Ghosh et al. 2023) evaluates predefined object properties, while GenEval 2 (Kamath et al. 2025) adds atom-level questions and atomicity analysis. TIFA (Hu et al. 2023) decomposes prompts into VQA pairs, and DSG (Cho et al. 2024) organizes atomic questions through valid dependencies. Other work improves supervision and calibration through dimension-specific concepts or element-level annotations (Wei and others 2025; Han et al. 2024), while VQAScore and GenAI-Bench study holistic alignment and its agreement with human preferences (Lin et al. 2024; Li et al. 2024). More recent benchmarks use hierarchical capability rubrics or dependency-aware checklists (Li and others 2026; Ban et al. 2026). These developments make local verification more reliable, but their evidence is usually summarized within each prompt or a fixed reporting taxonomy. Capability-specific and closed-set benchmarks. A complementary line of work deepens evaluation within selected capabilities. WISE and PhyBench focus on world knowledge and physical rules (Niu and others 2025; Meng and others 2024); T2I-CompBench++, ConceptMix, and DPG-Bench stress compositional or dense instructions (Huang et al. 2023; Wu et al. 2024; Hu et al. 2024); and PRISM-Bench, OneIG-Bench, and T2I-CoReBench expand coverage to bilingual, creative, and reasoning-heavy scenarios (Fang et al. 2025; Chang et al. 2026; Li et al. 2025a). LongTextBench and CVTG-2K further examine long or multilingual instructions and visual text (Geng and others 2025; Du et al. 2025). Table 1 summarizes these structural choices: atomic checking is common, but support for open prompts, dependencies, and prompt complexity is fragmented; none of the compared benchmarks connects evidence across prompts. QC-T2I-Bench combines all five properties in one question-centric framework. Evidence reuse and generator routing. Generator-routing work addresses model selection directly. CATImage learns prompt-conditioned quality–cost decisions, while DiffAgent uses an LLM agent for API selection (Li et al. 2025b; Zhao et al. 2024). OctoT2I, Image-POSER, and GenArtist extend selection to stateful or multi-step orchestration through self-evolving memory, reinforcement learning, or tool planning (Jiang et al. 2026; Mohebbi et al. 2025; Wang et al. 2024). These methods obtain routing signals by learning a policy, building agent state, or repeatedly evaluating intermediate outputs. Our router instead reuses question-level histories already collected for ranking and diagnosis, enabling training-free model selection from the same evaluation evidence. Method QC-T2I-Bench retains the identity, capability coordinates, and dependency context of every atomic judgment. These records support two complementary operations. The scoring path aggregates valid non-text and text evidence into capability coordinates and a hierarchy-constrained question aggregation (HCQ) score. The structural path reuses the recorded DSGs to measure component completion, construct cross-prompt root controls, and test topology-localized outcome coupling. The same records subsequently support model diagnosis and training-free routing. Question construction and capability coordinates. We collect prompts from six public sources spanning knowledge, entities, attributes, relations, text rendering, reasoning, and long multilingual instructions; source counts are provided in the supplement. Each prompt p is converted into c(p)=(qi,ti)i=1npc(p)=\(q_i,t_i)\_i=1^n_p, where npn_p is the number of atomic questions, qiq_i is an independently judgeable question, and tit_i is its secondary capability label. Construction follows four rules: Target Yes, Atomicity, Coverage, and Tag Validity. Together, they require a “yes” answer for a compliant image, one visual requirement per question, coverage of salient constraints, and attribution by the question’s core predicate. We instantiate this fixed contract with Qwen3-235B-A22B (Yang et al. 2025), iterative automated auditing, and a final cleanup pass; the supplement reports source composition and audit details. Evaluation produces one record per question, r=(p,q,m,ℓ,g,t,y,v),r=(p,q,m, ,g,t,y,v), (1) where p and q are the prompt and question, m is the evaluated model, ℓ is the language, g is the first-level reporting group, and t is the secondary capability. For non-text capabilities, y∈0,1y∈\0,1\ is the native binary outcome and v∈0,1v∈\0,1\ indicates dependency validity. Text Content retains the same capability coordinates but uses the transcription statistics defined below for official scoring. The taxonomy contains five first-level groups—Non-text Entities, Text Content, Attributes, Relations, and High-Level Semantics—and 21 secondary capabilities (Figure 1). Each question receives exactly one secondary capability by its core predicate, while one prompt may contribute questions to several groups. Text Content is separated because it uses the dedicated transcription evaluator described below. Figure 1: Two-level taxonomy: five first-level reporting groups organize 21 mutually exclusive secondary capabilities, assigned by each question’s core predicate. Dependency-aware capability scoring. Before adjudication, we organize prompt p into a Davidsonian Scene Graph Gp=(Vp,Ep)G_p=(V_p,E_p) (Cho et al. 2024), where VpV_p is the set of question nodes and EpE_p is the set of directed prerequisite edges. An edge u→q∈Epu\!→\!q∈ E_p indicates that question u is a semantic prerequisite of q. During adjudication, q is scored only when its prerequisites succeed, preventing one missing entity from becoming repeated attribute and relation failures. For model m, language scope Λ , and non-text capability t, let ℓ,tQ_ ,t be the questions in language ℓ , ymℓq∈0,1y_m q∈\0,1\ their adjudicated outcomes, and vmℓq∈0,1v_m q∈\0,1\ their dependency validity. The valid count and score are Nm,Λ,t N_m, ,t =∑ℓ∈Λ∑q∈ℓ,tvmℓq, = _ ∈ _q _ ,tv_m q, (2) Sm,Λ,t S_m, ,t =1Nm,Λ,t∑ℓ∈Λ∑q∈ℓ,tvmℓqymℓq,t≠ttext. = 1N_m, ,t _ ∈ _q _ ,tv_m qy_m q, t≠ t_text. Thus Sm,Λ,tS_m, ,t is the mean over valid questions at one capability coordinate when Nm,Λ,t>0N_m, ,t>0 and is NA otherwise; invalid records enter neither sum. Text-rendering branch. Rendered text remains a semantic capability coordinate, but its evidence type differs: a binary VQA judgment can accept semantically related text while missing character-level errors. We therefore use Qwen3-VL-Instruct-30B to extract text blocks and optimally match them to targets within each image i. Let DmℓiD_m i be the target-conditioned UTF-16 edit distance and LℓiL_ i the target length, and let ℐmℓtextI^text_m contain images with an available transcription record. We micro-average over all available text records in language scope Λ : Sm,Λ,ttext=max(0,1−∑ℓ∈Λ∑i∈ℐmℓtextDmℓi∑ℓ∈Λ∑i∈ℐmℓtextLℓi).S_m, ,t_text= \! (0,1- _ ∈ _i ^text_m D_m i _ ∈ _i ^text_m L_ i ). (3) This micro-average gives every target character equal weight; empty extractions incur full deletion cost, while unavailable evaluator records are excluded. If the denominator is zero, the score is NA. Hierarchy-constrained aggregation. We combine these secondary scores with HCQ, which micro-averages valid questions within each secondary capability, then macro-averages capabilities within each reporting group and across the five groups. With gT_g the capabilities in group g, Tg=|g|T_g=|T_g|, and G the number of reporting groups, Sm,ΛHCQ=1G∑g=1G1Tg∑t∈gSm,Λ,t.S_m, ^HCQ= 1G _g=1^G 1T_g _t _gS_m, ,t. (4) Here G=5G=5 and, in the order Non-text Entities, Text Content, Attributes, Relations, and High-Level Semantics, (T1,…,T5)=(5,1,6,4,5)(T_1,…,T_5)=(5,1,6,4,5). Compositional capability analysis. Beyond capability aggregation, we use the recorded DSG structure for three complementary diagnostics. Component exactness tests whether related requirements are jointly satisfied; cross-prompt root matching separates entity-generation difficulty from failures under attached constraints; and marginal-controlled coupling tests whether excess outcome dependence localizes to explicit DSG edges. A prompt may contain several disconnected structures, so we define its DSG atlas as the maximal weakly connected components p=Gp,1,…,Gp,KpA_p=\G_p,1,…,G_p,K_p\. For a component G of a fixed prompt–language pair, let (G)Q(G) be its questions and (G)=⋃q∈(G)t(q)T(G)= _q (G)\t(q)\ its component tag set. We score each maximal component once for its full tag set rather than enumerating its pairs or lower-order subsets. Components define structural context, not an assumption of statistical dependence; that distinction motivates the edge-controlled test below. For model m and component G, Cm(G)C_m(G) is the complete-component exactness indicator. It equals 1 only when every question is valid and succeeds, equals 0 when any valid question fails, and is undefined when the component cannot otherwise be scored. Unlike mean coverage, this criterion retains root failures even when their descendants are masked: Cm(G)=0,∃q∈(G):vmq=1∧ymq=0,1,∀q∈(G):vmq=1∧ymq=1,NA,otherwise.C_m(G)= cases0,&∃ q (G):\ v_mq=1 y_mq=0,\\ 1,&∀ q (G):\ v_mq=1 y_mq=1,\\ NA,&otherwise. cases (5) Applying the same rule separately to roots and descendants gives root survival Rm(G)R_m(G) and descendant completion conditional on root survival. To distinguish a composition-specific root failure from a generally difficult entity, we conservatively match entity roots across prompts without adding cross-prompt dependency edges. For entity identity e in prompt p, the leave-one-prompt-out baseline is R¯mℓe−p=1|ℓe∖p|∑p′∈ℓe∖pRmℓp′e. R^-p_m e= 1|P_ e \p\| _p _ e \p\R_m p e. (6) Here, ℓeP_ e contains prompts in language ℓ with a matched root e, and Rmℓp′e∈0,1R_m p e∈\0,1\ is its prompt-level root-survival outcome for model m. Matches stay within one language and model and require at least three other prompts. Comparing the local outcome RmℓpeR_m pe with R¯mℓe−p R^-p_m e therefore controls for the model’s baseline ability to realize the same entity. Marginal-controlled coupling. Complete-component exactness decreases as more fallible requirements are conjoined even when their outcomes are independent. We therefore test whether joint success follows recorded DSG topology beyond this ordinary multiplication of marginal success rates. For model m, language ℓ , prompt p, matched pair stratum s, and context c, define P^mℓpsc,11 P^c,11_m ps =1|ℳℓpsc|∑(i,j)∈ℳℓpscYmℓpiYmℓpj, = 1|M_ psc| _(i,j) _ pscY_m piY_m pj, (7) Δmℓpsc ^c_m ps =P^mℓpsc,11−π^mℓs,1−p,cπ^mℓs,2−p,c. = P^c,11_m ps- π^-p,c_m s,1 π^-p,c_m s,2. Here the nonempty set ℳℓpscM_ psc contains ordered matched pairs (i,j)(i,j), Ymℓpi,Ymℓpj∈0,1Y_m pi,Y_m pj∈\0,1\ are their native binary outcomes, and positions 1 and 2 denote the two ordered endpoints. The stratum fixes their ordered capability tags, root/descendant roles, and DSG depths; c is either edge, a direct parent–child pair, or disc, a structurally matched pair from disconnected components in the same prompt. The first term is observed joint success, while the product of leave-one-prompt-out endpoint marginals is the success expected in the same model–language stratum. To isolate dependence associated specifically with a recorded DSG edge, we compare the two residuals after matching: Γmℓ=1|mℓ∗|∑p∈mℓ∗1|mℓp∗|∑s∈mℓp∗(Δmℓpsedge−Δmℓpsdisc). _m = 1|P^*_m | _p ^*_m 1|S^*_m p| _s ^*_m p ( ^edge_m ps- ^disc_m ps ). (8) Here mℓ∗P^*_m contains prompts with matched edge and disconnected contexts, and mℓp∗S^*_m p contains their eligible matched strata; the nested means give equal weight to strata within a prompt and then to prompts. We additionally macro-average these estimates across models. Thus Δ=0 =0 is consistent with marginal multiplication, while Δ>0 >0 indicates excess outcome coupling. The contrast Γ asks whether that excess is stronger on explicit DSG edges than on matched disconnected pairs; it is not a quality gain or a causal composition penalty. For this structural test only, Y is recovered before dependency masking, which would otherwise induce edge dependence mechanically. Matching thresholds and full estimators are reported in the supplement. Together, the scoring and structural paths populate the evidence summarized by the bilingual views in Table 2; the following experiments reuse the same records for ranking, diagnosis, and routing. Model English Chinese Total Ent. Text Attr. Rel. Sem. Total Ent. Text Attr. Rel. Sem. FLUX.2-dev 89.48 97.5 78.6 94.7 89.4 87.1 88.60 97.2 74.4 94.6 89.6 87.3 ERNIE-Image 89.43 97.2 79.7 95.0 89.5 85.7 89.58 97.5 79.7 95.4 90.5 84.8 Z-Image Base 88.11 95.5 79.1 93.6 86.9 85.3 89.15 96.7 79.8 94.3 88.5 86.4 Qwen-Image 87.72 95.7 79.4 93.4 87.0 83.1 88.55 96.3 82.4 93.6 87.9 82.6 Lens 87.34 97.3 72.8 94.6 89.1 82.9 87.37 97.5 73.0 94.4 89.6 82.3 FLUX.2-Klein-9B 87.08 95.8 72.3 94.0 87.9 85.3 85.35 96.7 61.0 94.4 89.1 85.5 HiDream-O1 86.80 95.3 73.1 92.7 87.3 85.5 87.00 95.1 76.1 92.2 87.0 84.6 Z-Image Turbo 85.67 93.9 78.5 91.6 84.0 80.4 85.98 95.1 77.5 91.4 84.8 81.1 FLUX.2-Klein-4B 84.33 96.0 61.9 93.5 86.8 83.4 82.29 95.3 52.0 93.6 88.3 82.3 GLM-Image 83.94 92.6 73.4 88.9 82.9 82.0 84.88 93.1 76.2 89.1 83.9 82.2 LongCat-Image 83.91 96.2 63.8 91.6 84.7 83.3 85.97 96.2 70.0 92.8 87.7 83.2 FLUX.1-dev 76.21 89.9 44.1 88.8 80.1 78.2 32.75 23.0 16.5 47.2 41.8 35.3 HiDream-I1 75.66 91.7 35.2 88.9 81.7 80.8 65.02 78.9 20.6 80.4 73.8 71.4 Table 2: Aligned English and Chinese HCQ results (%). Models are ordered by English total; each language is aggregated independently by equally averaging the five reporting views. Ent., Attr., Rel., and Sem. denote Non-text Entities, Attributes, Relations, and High-Level Semantics. Best values within each language are bold; full 21-dimension profiles are in the supplement. Experiments Experimental Setup Models and generation. We evaluate 13 recent open-source T2I systems: ERNIE-Image (Liu and others 2026), Qwen-Image (Wu and others 2025), Z-Image Base and Turbo (Z-Image Team and others 2025), FLUX.1-dev and FLUX.2-dev (Black Forest Labs 2024), FLUX.2-Klein 4B/9B, GLM-Image (Z.ai 2026), HiDream-O1/I1 (Cai and others 2026; Cai and others 2025), LongCat-Image (Meituan LongCat Team and others 2025), and Lens (Chen and others 2026). We use official defaults and disable configurable prompt enhancement so every generator receives the same prompt. The fixed benchmark contains 6,573 conceptual prompts, 94,547 English questions, and 94,555 Chinese questions. Analyses are run separately by language: the main paper presents both leaderboards side by side and reports bilingual aggregation, dependency, and routing summaries, while complete language-specific profiles are in the supplement. Evaluation protocol. Every model generates both language sets. We apply HCQ (Equation 4) independently to each language; the language-specific leaderboards do not pool English and Chinese evidence. Aggregation robustness. Rank recovery alone confounds scoring noise with rule-induced changes in model gaps. Let g(t)g(t) map capability t to its unique reporting group. We hold the HCQ capability weight αt=1/(GTg(t)) _t=1/(GT_g(t)) fixed and compare the noise introduced by two within-capability rules. Let tP_t be the prompts contributing to capability t, Pt=|t|P_t=|P_t|, nptn_pt the valid question count from prompt p, and Nt=∑p∈tnptN_t= _p _tn_pt. Question-micro weighting assigns each judgment αt/Nt _t/N_t; prompt-first weighting assigns it αt/(Ptnpt) _t/(P_tn_pt). Under independent, zero-mean, equal-variance adjudication errors, their noise-variance ratio is VtpromptVtquestion=(1Pt∑p∈tnpt)(1Pt∑p∈t1npt)≥1. V_t^promptV_t^question= ( 1P_t _p _tn_pt ) ( 1P_t _p _t 1n_pt )≥ 1. (9) Here VtpromptV_t^prompt and VtquestionV_t^question are the variances of capability t’s weighted adjudication-error contribution under the two rules. The arithmetic–harmonic mean inequality makes the ratio at least one, with equality only when every prompt contributes the same number of questions. Thus equal atomic weights minimize variance under the fixed capability construct, while prompt-first weighting amplifies errors in relatively sparse prompts. The full proof, unequal-variance extension, and rank-gap analysis are reported in the supplement. Reliable Ranking Leaderboard. Table 2 aligns the five capability views and HCQ total across languages; complete 21-dimensional profiles are in the supplement. FLUX.2-dev leads in English and ERNIE-Image in Chinese, but no model is best across every view, keeping each scalar order traceable to its diagnostic coordinates. Bootstrap ranking reliability. We test whether the leaderboard survives prompt resampling using 2,000 paired bootstrap draws over the full set of retained conceptual-prompt clusters, resampling every attached question together. The 95% paired intervals exclude zero for 68 of the 78 model pairs. FLUX.2-dev has a 55.2% top-1 probability and a 95% rank interval of [1,2][1,2]; ERNIE-Image has the complementary 44.8% top-1 probability and the same interval. Thus the broad ordering is resolved, but the 0.042-point gap between the first two models is not evidence of deterministic separation. Adjudication-noise robustness. Prompt-first normalization increases both adjudication-noise variance and the influence of one error in a short prompt in English and Chinese (Table 3). The smaller full-benchmark effects would hide this difference at the roughly 1,000-prompt scale of recent benchmarks; complete diagnostics are in the supplement. Var. increase One short-prompt error (p) Language Prompt vs. Q Q Prompt Increase English 23.2% 0.0102 0.0151 48.4% Chinese 21.2% 0.0110 0.0156 41.8% Table 3: Adjudication-noise sensitivity at 1,000 prompts. “Question” gives equal weight to valid atomic judgments; “Prompt” normalizes prompts first. A short prompt contains at most three questions. Lower is more robust. Figure 2: Scaling and root-control views of complete DSG components. Left: prompt-macro exact completion versus the number of distinct capability tags in the full graph. Right: local entity-root success (orange) and the same model’s leave-one-prompt-out success on matched roots (blue). Line samples identify the corresponding model curves. Figure 3: Outcome coupling after removing the joint success expected from ordinary marginal error multiplication. Each row compares question pairs in the same DSG component (orange) with matched pairs from disconnected components (blue). Points are model-macro residuals Δ from Equation 7; bars are 95% prompt-bootstrap intervals. Figure 4: High-frequency DSG-component composition fingerprints. Columns are the three most frequent full tag sets at each cardinality from two through five; frequency selects columns but does not weight cells. Abbreviations: H=Humans, M=Man-Made, Sh=Shape, Col=Color, Mat=Material, Lt=Lighting, St=State, Int=Interaction, Sp=Spatial, and Q/C=Quantification/Comparison. Cell text is prompt-macro conditional descendant completion (%) after every root survives. The color bar gives the deviation in percentage points from the same model’s mean at that tag-set size. Fine-Grained Diagnosis Capability boundaries. The five views in Table 2 separate strong entity and appearance generation from weaker text, relation, and semantic capabilities. The complete 21-dimensional matrices further localize persistent deficits in Text Content, Scientific and Cultural Knowledge, Temporal Setting, and Logical Constraints; cross-language shifts are capability- and model-specific rather than uniform. Effect of dependency masking. The official metrics exclude dependency-invalid descendants. Counting them as failures leaves the broad ordering stable (ρ=0.978ρ=0.978) but changes two positions, indicating that masking removes cascading penalties without manufacturing the ranking; full shifts are in the supplement. DSG-component composition diagnosis. The DSG atlas yields 10,584 maximal components from 5,792 conceptual prompts. In Figure 2, mean exact completion falls from 80.7% for two-tag components to 37.2% for components with seven or more tags. This describes joint difficulty, not interaction: exact success becomes stricter as requirements accumulate even under independent errors. The cross-prompt control uses 4,319–4,347 eligible matched contexts per model. Local roots trail their matched baseline by only 0.14–0.88 points, with nine of thirteen intervals excluding zero. Root survival and conditional descendant completion can therefore locate entity versus attached-structure failure, but do not by themselves establish interaction. Topology-localized coupling. After marginal control, the direct-edge contrast is +1.10+1.10 points (95% prompt-bootstrap interval [+0.69,+1.50][+0.69,+1.50]), whereas the non-ancestral contrast is smaller and unresolved at +0.38+0.38 points [−0.17,+1.01][-0.17,+1.01] (Figure 3). Directly dependent requirements thus co-succeed and co-fail beyond their individual rates, and the excess localizes to explicit DSG topology rather than component membership alone. Chinese results reproduce the direct-edge pattern; full estimates are in the supplement. Figure 4 adds structure-specific resolution: Qwen-Image is weak on a five-tag Man-Made–Color–Material–Spatial–Quantification component, whereas FLUX.2-Klein-4B is comparatively strong on the related Shape variant. Such profiles distinguish models with similar scalar scores; all cells and structural frequencies are in the supplement. Scalable Cost-Aware Routing Question histories reveal which generators satisfy which atomic requirements. We reuse them to select one T2I generator for each complete request; routing does not select the VQA evaluator. Routers and evaluation protocol. Our training-free router, Q-Profile, estimates generator quality from the request’s active views and reference-fold outcomes without learned parameters. Its quality-first variant (Q) selects the predicted maximum; its cost-aware variant (C) selects the least expensive generator within a validation-selected ϵε of that maximum. We retain the original source-stratified nested five-fold choices and do not retune settings after the HCQ revision. English and Chinese scores are computed separately and combined with fixed 50/50 weights. Cost is latency times occupied GPUs per output megapixel (GPU-s/MP); paired intervals resample shared conceptual IDs across languages. Full protocol and retrieval ablations are in the supplement. Method EN ZH Bi. GPU-s/MP Save E/F ERNIE-Image 89.43 89.58 89.51 28.15 0.0/+55.6% FLUX.2-dev 89.48 88.60 89.04 63.34 -125.0/0.0% Uniform random 85.08 82.62 83.85 19.20 +31.8/+69.7% Q-Profile-Q 89.72 89.98 89.85 46.46 -65.0/+26.7% Q-Profile-C 89.25 89.78 89.51 22.16 +21.3/+65.0% Table 4: Out-of-fold routing under HCQ. Bi. is the fixed EN/ZH average; Save E/F is relative to fixed ERNIE-Image/FLUX2-dev. Q and C denote quality-first and cost-aware routing. Quality-cost results. Q-Profile-Q gives the highest routed point estimate, but its paired gain over ERNIE is marginal at the lower endpoint and comes at higher cost. Q-Profile-C matches ERNIE’s point estimate while reducing cost by 21.3% relative to ERNIE and 65.0% relative to FLUX2. Its interval against ERNIE crosses the preregistered 0.20-point non-inferiority margin, so the result supports a quality–cost tradeoff rather than lossless routing. Uniform random performs worse, excluding arbitrary model assignment as the explanation. Conclusion QC-T2I-Bench turns question decomposition from an intermediate step toward a prompt score into reusable evidence. Its attributed records combine five capability groups, dependency validity, and DSG context: HCQ yields complexity-aware rankings, while graph controls separate root realization from attached requirements. Across 13 models and two languages, exact completion falls from 80.7% for two-tag components to 37.2% for components with seven or more tags; after marginal control, excess coupling is resolved on direct DSG edges but not matched non-ancestral pairs, localizing rather than causally identifying composition-associated dependence. The same records support a training-free router that matches ERNIE’s point estimate with 21.3% lower GPU-s/MP, although its interval precludes a lossless-routing claim. Thus one auditable evidence base supports ranking, diagnosis, and cost-aware selection without treating any scalar score as universal. Future work should calibrate question construction, DSG parsing, and adjudication across domains. References Ban et al. (2026) Y. Ban, T. Xie, S. An, Y. Hong, E. Frick, I. Hsu, W. Chiang, I. Stoica, and C. Hsieh Arena-t2i hard: benchmarking and improving faithfulness with dependency-aware checklist. External Links: 2606.31711, Link Cited by: Fine-grained evaluation benchmarks.. Black Forest Labs (2024) Black Forest Labs FLUX. Note: https://github.com/black-forest-labs/flux Cited by: Introduction, Models and generation.. Cai et al. (2025) Q. Cai et al. HiDream-I1: a high-efficient image generative foundation model with sparse diffusion transformer. arXiv preprint arXiv:2505.22705. External Links: Link Cited by: Models and generation.. Cai et al. (2026) Q. Cai et al. HiDream-O1-Image: a natively unified image generative foundation model with pixel-level unified transformer. arXiv preprint arXiv:2605.11061. External Links: Link Cited by: Models and generation.. Chang et al. (2026) J. Chang, Y. Fang, P. Xing, S. Wu, W. Cheng, R. Wang, X. Zeng, G. Yu, and H. Chen Oneig-bench: omni-dimensional nuanced evaluation for image generation. Advances in Neural Information Processing Systems 38. Cited by: Capability-specific and closed-set benchmarks.. Chen et al. (2026) D. Chen et al. Lens: rethinking training efficiency for foundational text-to-image models. arXiv preprint arXiv:2605.21573. Cited by: Models and generation.. Cho et al. (2024) J. Cho, Y. Hu, J. Baldridge, R. Garg, P. Anderson, R. Krishna, M. Bansal, J. Pont-Tuset, and S. Wang Davidsonian scene graph: improving reliability in fine-grained evaluation for text-to-image generation. In International conference on learning representations, Vol. 2024, p. 15625–15645. Cited by: Introduction, Introduction, Introduction, Fine-grained evaluation benchmarks., Dependency-aware capability scoring.. Du et al. (2025) N. Du, Z. Chen, Z. Chen, S. Gao, X. Chen, Z. Jiang, J. Yang, and Y. Tai Textcrafter: accurately rendering multiple texts in complex visual scenes. arXiv e-prints, p. arXiv–2503. Cited by: Capability-specific and closed-set benchmarks.. Esser et al. (2024) P. Esser et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, Cited by: Introduction. Fang et al. (2025) R. Fang, A. Yu, C. Duan, L. Huang, S. Bai, Y. Cai, K. Wang, S. Liu, X. Liu, and H. Li FLUX-reason-6m & prism-bench: a million-scale text-to-image reasoning dataset and comprehensive benchmark. arXiv preprint arXiv:2509.09680. Cited by: Capability-specific and closed-set benchmarks.. Geng et al. (2025) Z. Geng et al. X-omni: reinforcement learning makes discrete autoregressive image generative models great again. arXiv preprint arXiv:2507.22058. Cited by: Capability-specific and closed-set benchmarks.. Ghosh et al. (2023) D. Ghosh, H. Hajishirzi, and L. Schmidt Geneval: an object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems 36, p. 52132–52152. Cited by: Fine-grained evaluation benchmarks.. Han et al. (2024) S. Han, H. Fan, J. Fu, L. Li, T. Li, J. Cui, Y. Wang, Y. Tai, J. Sun, C. Guo, and C. Li EvalMuse-40k: a reliable and fine-grained benchmark with comprehensive human annotations for text-to-image generation model evaluation. External Links: 2412.18150, Link Cited by: Fine-grained evaluation benchmarks.. Hessel et al. (2021) J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi Clipscore: a reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 7514–7528. Cited by: Introduction. Hu et al. (2024) X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu ELLA: equip diffusion models with llm for enhanced semantic alignment. External Links: 2403.05135 Cited by: Capability-specific and closed-set benchmarks.. Hu et al. (2023) Y. Hu, B. Liu, J. Kasai, Y. Wang, M. Ostendorf, R. Krishna, and N. A. Smith Tifa: accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 20406–20417. Cited by: Introduction, Introduction, Fine-grained evaluation benchmarks.. Huang et al. (2023) K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu T2i-compbench: a comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems 36, p. 78723–78747. Cited by: Capability-specific and closed-set benchmarks.. Jiang et al. (2026) X. Jiang, B. Chen, G. Li, Y. Duan, R. Wang, and J. Zhang OctoT2I: a self-evolving agentic text-to-image router. arXiv preprint arXiv:2606.01803. External Links: Link Cited by: Evidence reuse and generator routing.. Kamath et al. (2025) A. Kamath, K. Chang, R. Krishna, L. Zettlemoyer, Y. Hu, and M. Ghazvininejad GenEval 2: addressing benchmark drift in text-to-image evaluation. arXiv preprint arXiv:2512.16853. Cited by: Fine-grained evaluation benchmarks.. Li et al. (2024) B. Li, Z. Lin, D. Pathak, J. Li, Y. Fei, K. Wu, T. Ling, X. Xia, P. Zhang, G. Neubig, and D. Ramanan GenAI-bench: evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406.13743. Cited by: Introduction, Fine-grained evaluation benchmarks.. Li et al. (2026) N. Li et al. Qwen-image-bench: from generation to creation in text-to-image evaluation. arXiv preprint arXiv:2605.28091. External Links: 2605.28091, Document, Link Cited by: Introduction, Fine-grained evaluation benchmarks.. Li et al. (2025a) O. Li, Y. Wang, X. Hu, H. Huang, R. Chen, J. Ou, X. Tao, P. Wan, X. Qi, and F. Feng Easier painting than thinking: can text-to-image models set the stage, but not direct the play?. arXiv preprint arXiv:2509.03516. Cited by: Capability-specific and closed-set benchmarks.. Li et al. (2025b) Q. Li, K. Chen, C. Su, W. Jitkrittum, Q. Sun, and P. Sangkloy Cost-aware routing for efficient text-to-image generation. arXiv preprint arXiv:2506.14753. External Links: Link Cited by: Evidence reuse and generator routing.. Lin et al. (2024) Z. Lin, D. Pathak, B. Li, J. Li, X. Xia, G. Neubig, P. Zhang, and D. Ramanan Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.01291. Cited by: Fine-grained evaluation benchmarks.. Liu et al. (2026) J. Liu et al. ERNIE-image technical report. External Links: 2605.25347, Link Cited by: Models and generation.. Meituan LongCat Team et al. (2025) Meituan LongCat Team et al. LongCat-Image technical report. arXiv preprint arXiv:2512.07584. External Links: Link Cited by: Models and generation.. Meng et al. (2024) F. Meng et al. Phybench: a physical commonsense benchmark for evaluating text-to-image models. arXiv preprint arXiv:2406.11802. Cited by: Capability-specific and closed-set benchmarks.. Mohebbi et al. (2025) H. Mohebbi, M. Abdulrahman, Y. Miao, P. Poupart, and S. Kothawade Image-POSER: reflective RL for multi-expert image generation and editing. arXiv preprint arXiv:2511.11780. External Links: Link Cited by: Evidence reuse and generator routing.. Niu et al. (2025) Y. Niu et al. Wise: a world knowledge-informed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.07265. Cited by: Capability-specific and closed-set benchmarks.. Wang et al. (2024) Z. Wang, A. Li, Z. Li, and X. Liu GenArtist: multimodal LLM as an agent for unified image generation and editing. In Advances in Neural Information Processing Systems, Vol. 37, p. 128374–128395. Cited by: Evidence reuse and generator routing.. Wei et al. (2025) X. Wei et al. TIIF-bench: how does your t2i model follow your instructions?. arXiv:2506.02161. Cited by: Introduction, Fine-grained evaluation benchmarks.. Wu et al. (2025) C. Wu et al. Qwen-image technical report. arXiv preprint arXiv:2508.02324. Cited by: Introduction, Models and generation.. Wu et al. (2024) X. Wu, D. Yu, Y. Huang, O. Russakovsky, and S. Arora Conceptmix: a compositional image generation benchmark with controllable difficulty. Advances in Neural Information Processing Systems 37, p. 86004–86047. Cited by: Capability-specific and closed-set benchmarks.. Yang et al. (2025) A. Yang, A. Li, B. Yang, et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. External Links: Link Cited by: Question construction and capability coordinates.. Z-Image Team et al. (2025) Z-Image Team et al. Z-image: an efficient image generation foundation model with single-stream diffusion transformer. arXiv preprint arXiv:2511.22699. External Links: Link Cited by: Models and generation.. Z.ai (2026) Z.ai GLM-Image model card. Note: https://huggingface.co/zai-org/GLM-Image Cited by: Models and generation.. Zhao et al. (2024) L. Zhao, Y. Yang, K. Zhang, W. Shao, Y. Zhang, Y. Qiao, P. Luo, and R. Ji DiffAgent: fast and accurate text-to-image API selection with large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6390–6399. Cited by: Evidence reuse and generator routing..