Paper deep dive
VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision
Vu Dinh Xuan, Duc-Hai Nguyen, Minh-Dung Dao, Vu Quynh Giao, Quang Hong Nguyen, Binh-Son Hua, Barry O'Sullivan, David Murphy, Hoang D. Nguyen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/2/2026, 9:27:23 AM
Summary
The paper introduces VisionQ, a benchmark and dataset for evaluating Vision-Language Models (VLMs) as judges in computer vision qualitative analysis. Unlike existing benchmarks that rely on scalar quality scores, VisionQ grounds judgments in specific visual criteria using a six-axis, 51-leaf taxonomy. The dataset comprises 1,409 CVPR and ICCV papers with over 1,800 validated comparison figures and 3,911 hand-annotated data points. The authors propose a criterion-conditioned evaluation protocol and introduce VisionQ-Judge, a DPO-tuned Gemma-4-E4B model that improves accuracy and reduces bias. Evaluations of 20 VLMs reveal significant variability in reliability across different visual criteria.
Entities (15)
Relation Signals (7)
VisionQ-Judge → finetunedfrom → Gemma 4 E4B
confidence 96% · VisionQ-Judge, a DPO-tuned Gemma-4-E4B judge
VisionQ → usestaxonomy → six-axis, 51-leaf taxonomy
confidence 95% · VisionQ comprises ... a six-axis, 51-leaf taxonomy of the visual criteria behind qualitative judgment
VisionQ-Judge → trainedwithmethod → DPO
confidence 94% · VisionQ-Judge, a DPO-tuned Gemma-4-E4B judge trained on symmetric evidence pairs
VisionQ → derivedfromcorpus → CVPR and ICCV papers
confidence 93% · a corpus of 1,409 CVPR and ICCV papers with 1,800+ validated comparison figures
VisionQ-Bench → usedfor → VLM evaluation
confidence 92% · VisionQ-Bench, used to evaluate VLM judges
VisionQ-MCQ → usedfor → VisionQ-Judge training
confidence 92% · VisionQ-MCQ release ... used to train and test VisionQ-Judge
VisionQ → proposesprotocol → criterion-conditioned evaluation protocol
confidence 90% · a criterion-conditioned evaluation protocol that hides method names, captions, and paper identity
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Qualitative comparison figures are central evidence in computer vision papers, and vision-language models (VLMs) are increasingly used to judge them. Yet existing benchmarks score only scalar quality or overall preference, so a judge can be rewarded for picking the preferred image for the wrong visual reason. We introduce VisionQ, the first benchmark built from peer-reviewed CV comparison figures that grounds every judgment in a named visual criterion: each question states the criterion, and a judge is credited only when it selects the output the authors identify as best on that criterion. We call this task criterion-conditioned visual discrimination. VisionQ comprises (1) a corpus of 1,409 CVPR and ICCV papers with 1,800+ validated comparison figures and 3,911 hand-annotated data points linking method crops to author-stated visual claims; (2) a six-axis, 51-leaf taxonomy of the visual criteria behind qualitative judgment; (3) a criterion-conditioned evaluation protocol that hides method names, captions, and paper identity and reports accuracy per criterion; and (4) VisionQ-Judge, a DPO-tuned Gemma-4-E4B judge trained on symmetric evidence pairs, which reduces last-option predictions by 7.0pp and improves accuracy by 2.5pp on a held-out test set. Evaluating 20 open- and closed-source VLM judges, we find that the strongest reach only 63.1% accuracy (chance 32.2%) and that reliability varies sharply across criteria. Code: this https URL. Data: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.00666v1
- Canonical: https://arxiv.org/abs/2610.00666v1
Trouble viewing inline? Open PDF directly →
Full Text
98,782 characters extracted from source content.
Expand or collapse full text
VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision Vu Dinh Xuan 1∗ Duc-Hai Nguyen 2∗ Minh-Dung Dao 2 Vu Quynh Giao 2 Quang Hong Nguyen 3 Binh-Son Hua 4 Barry O’Sullivan 2 David Murphy 2 Hoang D. Nguyen 2† 1 University of Information Technology, VNU-HCM, Vietnam 2 University College Cork, Ireland 3 Hanoi University of Science and Technology, Vietnam 4 Trinity College Dublin, Ireland ∗ Equal contribution † Corresponding author vudx.17@grad.uit.edu.vn, 125109073@umail.ucc.ie, robert.dao@reliable-ai.org, qvu@ucc.ie, Quang.NH232116M@sis.hust.edu.vn, binhson.hua@tcd.ie, b.osullivan@cs.ucc.ie, d.murphy@cs.ucc.ie, hn@cs.ucc.ie Abstract Qualitative comparison figures are central evidence in computer vision papers, and vision-language models (VLMs) are increasingly used to judge them. Yet existing benchmarks score only scalar quality or overall preference, so a judge can be rewarded for picking the preferred image for the wrong visual reason. We introduce VisionQ, the first benchmark built from peer-reviewed CV comparison figures that grounds every judgment in a named visual criterion: each question states the criterion, and a judge is credited only when it selects the output the au- thors identify as best on that criterion. We call this task criterion-conditioned visual discrimination. VisionQ comprises (1) a corpus of 1,409 CVPR and ICCV papers with 1,800+ validated comparison figures and 3,911 hand-annotated data points linking method crops to author-stated visual claims; (2) a six-axis, 51-leaf taxon- omy of the visual criteria behind qualitative judgment; (3) a criterion-conditioned evaluation protocol that hides method names, captions, and paper identity and re- ports accuracy per criterion; and (4) VisionQ-Judge, a DPO-tuned Gemma-4-E4B judge trained on symmetric evidence pairs, which reduces last-option predictions by 7.0 p and improves accuracy by 2.5 p on a held-out test set. Evaluating 20 open- and closed-source VLM judges, we find that the strongest reach only 63.1% accuracy (chance 32.2%) and that reliability varies sharply across crite- ria. Code is available athttps://github.com/ReML-AI/visionqand data at https://huggingface.co/datasets/visionq-anon-2026/VisionQ-1k. 1 Introduction Vision-language models (VLMs) are increasingly used as automated judges [30] of visual quality, reducing reliance on costly per-sample human annotation in tasks ranging from image quality assessment [22,28] to preference-based evaluation of generative outputs [27,25]. Yet these systems are evaluated almost exclusively on verdict accuracy: whether they select the preferred image. A judge that chooses correctly while citing unsupported visual evidence is treated as equivalent to one that correctly identifies the true visual reason for a preference. This evaluation gap makes it difficult for practitioners to trust judge rationales or diagnose systematic failures. Preprint. arXiv:2610.00666v1 [cs.CV] 30 Sep 2026 The gap is especially consequential in computer-vision research, where qualitative comparison figures are central evidence for method evaluation. Assessing such a figure is an inherently structured task: a reviewer compares candidate outputs against a reference or baseline, identifies the visual criteria that differentiate them, and grounds a preference in specific, figure-verifiable evidence. VLMs exhibit systematic biases on such figures, such as preferring the last-listed option in multiple-choice settings [29], and existing evaluation protocols provide no mechanism to check whether a judge’s stated rationale is grounded in the visual evidence it cites. Yet neither IQA benchmarks [22,28] nor VLM-as-judge validation work [6] requires a judge to name the specific visual criterion behind a preference or verify that criterion against the figure. We extensively discuss the related work in Appendix A. We introduce VisionQ, which moves grounding from the judge’s explanation into the question itself. Each question names the visual criterion on which the candidates are compared, taken from an author-stated claim in a peer-reviewed CV paper, so a judge is credited only when its choice follows that criterion rather than overall appeal. This makes criterion-specific failures measurable without relying on a model’s account of its own reasoning: one of the strongest judges we evaluate reaches 64.1% on Image Appearance but only 45.2% on Relation. Because every question keeps its full author claim, the same data also provides ground truth for scoring free-text rationales. VisionQ is intended for teams that use VLM judges in place of human annotation when evaluating generative and reconstruction models; its purpose is judge validation, not automated reviewing or paper authoring. Repeated VLM judgments act as proxy metrics that decide which model variants a team keeps, so a judge that relies on an incidental visual cue turns a local failure into systematic measurement bias. VisionQ is the primary contribution: the taxonomy defines the visual criteria, the annotated corpus supplies the source material, and VisionQ-Judge demonstrates how the benchmark can be used. Figure 1 illustrates the complete pipeline and our contributions are detailed below. Our contributions are: 1.A corpus of 1,409 peer-reviewed CV papers with 1,800+ validated qualitative-comparison figures and 3,911 hand-annotated data points with method-level and claim-level metadata. 2.A six-axis taxonomy of visual evaluation criteria (image appearance, object form, scene layout, relation, reference fidelity, prompt match) grounded in prior benchmark dimensions and corpus-driven claim analysis. 3.A criterion-conditioned evaluation protocol in which every question names one taxonomy criterion and hides method names, captions, and paper identity, with accuracy reported per criterion. 4.VisionQ-Judge, a DPO-fine-tuned Gemma-4-E4B (4B) VLM trained on symmetric evidence pairs built from the 4,524-question VisionQ-MCQ release spanning 513 papers, derived from VisionQ’s structured metadata with no LLM calls at construction time, reducing last-option predictions by 7.0 p and improving preference accuracy by +2.5 p (5.0% relative). 5. A profiling study of 20 open- and closed-source VLM judges on overall and per-criterion accuracy, revealing criterion-specific failure modes that verdict-only benchmarks cannot detect. 2 Taxonomy Construction We construct VisionQ’s taxonomy through a mixed deductive-inductive procedure [17]: the deductive stage seeds axes from prior visual-evaluation literature; the inductive stage refines them against 9,228 grounded qualitative claims extracted from CV papers. Axes must be reviewer-like (criteria a reader naturally invokes, not model-internal properties), figure-observable (checkable from the static figure alone), and specific enough for reproducible annotation; non-visual claims are explicitly excluded. We validate the codebook through a coverage check on1,486real CV-paper figures (Section 2.3). The taxonomy provides the controlled vocabulary for annotation (Section 3) and benchmark synthesis (Section 2.4). 2 1 2 3,911 annotated data points Figure crops; Claims; Axes; Labels; Rationales; Metadata Evaluation axis taxonomy (6 axes) AppearanceObject FormScene Layout Relation Reference Fidelity Prompt Match 34 Final judge model for evaluation Judge profiling (example) VisionQ-Judge (our fine-tuned model) Corpus Construction 1,400+ CV papers (CVPR/ ICCV/ ECCV/ ...) Extract qualitative- comparison figures 1,800+ validated figures Grounding & Annotation Example qualitative- comparison figures Panel grounding (crop/ highlight) Author-stated visual claims “Method B has sharper edges than A.” “Method C better preserves reflections.” Human annotation & verification Fact check; Claim mapping; Axis labeling; Quality control; Adjudication Benchmark Protocols VisionQ-Judge Training (a) Criterion-conditioned multiple choice (2–4 panels) (b) Criterion named in the question e.g. "Which output shows sharper edges?", .... (c) Blind composite Prompt rewrite Paraphrase/ reorder/ compress the evaluation prompt Method identity perturbation Shuffle/ Rename/ Obfuscate method identities Scorecard (per model x per axis) Preference accuracy Per-criterion accuracy Per-axis reliability Evaluate VLM Judges Evaluate on benchmark tasks Open-source VLMs e.g., Qwen 3.6, Gemma 4, GLM-5V etc. Closed-source VLMs e.g., GPT-5.5, Claude Opus 4.6, Grok-4.3, etc. Structured VisionQ metadata Figures; Claims; Axes; Labels; Rationales; IDs Symmetric evidence pairsDPO fine-tuning Optimize on pairwise preferences with symmetric evidence VisionQ-Judge Increase preference accuracy Figure 1: Overview of VisionQ. Five-phase pipeline: corpus construction and figure selection (Section 3), taxonomy-guided annotation (Section 2), benchmark protocols (Section 4), VLM judge evaluation (Section 4), and VisionQ-Judge DPO training (Section 5, bottom strip). 2.1 Two-Stage Construction Deductive stage.We survey four kinds of prior evaluation work and extract their recurring evaluator concerns. IQA and low-level quality benchmarks [22,23,28] contribute criteria for perceptual quality, sharpness, artifact presence, and texture; text-image alignment benchmarks [8,4] contribute object presence, attribute satisfaction, and prompt satisfaction; visual reasoning and compositional benchmarks [9,5] contribute spatial arrangement and relational structure; reference-based protocols from tasks such as super-resolution, inpainting, and depth estimation contribute output-to-ground- truth correspondence. From these we derive six top-level axes: Image Appearance, Object Form, Scene Layout, Relation, Reference Fidelity, and Prompt Match. Inductive stage. We extract9,228grounded qualitative claims from1,409VisionQ papers using LLM-assisted open coding. For each comparison passage, the model emits a structured(claim, axis-class)pair, whereclaimis the author-stated qualitative judgment andaxis-classis a short open-code label naming the visual criterion being evaluated (e.g., thin-structure preservation, boundary accuracy, identity preservation). Because many papers describe the same visual criterion in different language, multiple claims map to the same or near-equivalentaxis-class. We canonicalize these labels by exact-string deduplication and synonym normalization, yielding447distinct candidate atoms. Each atom is therefore a recurring visual judgment type, while the original9,228claims provide support instances and example evidence. Hierarchy induction (CLIO-inspired). The final taxonomy has six axes and51leaves, where a leaf is the finest-grained visual criterion a claim can be assigned to (e.g., Blur or Boundary); the mid-level groups described next are used only during construction. We organize the447atoms into a draft hierarchy [20]: atom labels are embedded and clustered (K=60leaf candidates), an LLM names each cluster contrastively (what distinguishes it from neighbors), and the procedure recurses to roll leaf candidates into16mid-level groups and six top-level axes. Each leaf is then reviewed against 3 ROOT Qualitative evaluation criteria 1 Image Appearance 9 leaves Blur Noise Exposure Contrast Color Lighting Global Artifact Realism Style 2 Object Form 11 leaves Shape Boundary Texture Surface Material Completeness Pose Face Anatomy Detail Segmentation 3 Scene Layout 8 leaves Composition Depth Scale View Spatial Distribution Occlusion Layout Camera Geometry Physical Plausibility 4 Relation 6 leaves Contact Correspondence Attribute Binding Spatial Relation Interaction Part-Whole Relation 5 Reference Fidelity 9 leaves Identity Preservation Background Preservation Reconstruction Fidelity Edit Preservation Geometry Fidelity Style Fidelity Mask Fidelity Landmark Fidelity Segmentation Fidelity 6 Prompt Match 8 leaves Object Presence Attribute Match Spatial Instruction Semantic Match Text Legibility Count Match Action Match Expression Match Figure 2: VisionQ six-axis taxonomy. Each axis branches into leaf nodes to which author-stated claims are assigned. Claims that don’t fit any leaf are assigned to an out-of-scope category with an explicit reason code. example claims; overlapping leaves are split or merged, sparse leaves removed, and every retained leaf documented with a working definition; inclusion/exclusion criteria are fixed at the axis level. Axis definitions and all51leaf definitions are given in Appendix B. LLM use. Every LLM-assisted step in this procedure, the open coding of claims and the contrastive naming of clusters, was run with three LLMs (GPT-5.5 Thinking, Gemini 3.1 Pro, and Claude Opus 4.7). Each model produced a draft; human reviewers compared the drafts, adjusted them, and produced the final labels, so no LLM output enters the taxonomy without human review. 2.2 Final Codebook The codebook contains six primary axes organized around three levels of visual analysis: image-level (Image Appearance, Object Form), scene-level (Scene Layout, Relation), and cross-image (Reference Fidelity, Prompt Match). Image Appearance covers low-level appearance of a whole panel or crop: blur, noise, exposure, contrast, color, lighting, global artifacts, realism, and style. Object Form covers the internal quality of visible objects and regions: shape, boundary, texture, surface, material, completeness, pose, face, anatomy, fine detail, and segmentation quality. Scene Layout covers scene-level spatial organisation: composition, depth, scale, viewpoint, spatial distribution, occlusion layout, camera geometry, and physical plausibility. Relation covers relationships between visible entities or parts: contact, correspondence, attribute binding, spatial relation, interaction, and part-whole relation. Reference Fidelity covers correspondence between the method output and a visible reference. It splits internally into Source Preservation (Identity, Background, Edit, Style) for cases where the reference is a source image, and Target Fidelity (Reconstruction, Geometry, Mask, Landmark, Segmentation) for cases where the reference is a target ground truth. Prompt Match covers agreement between the method output and the text prompt or caption that conditions generation. Figure 2 shows the full six-axis tree with all 51 leaf nodes. Adjudication. When a claim plausibly fits multiple axes, we assign it to exactly one axis using a fixed priority order: Prompt Match→Reference Fidelity→Relation→Scene Layout→Object Form→Image Appearance→Out-of-Scope. The order prioritises explicit conditioning signals: first a text prompt, then a visible reference; where neither applies, it proceeds from relational and scene-level criteria to object- and image-level ones. For example, a relation required by the prompt is assigned to Prompt Match, while a relation between visible entities without prompt conditioning is assigned to Relation; likewise, a shape judged against a visible target is assigned to Reference Fidelity, while its quality without a reference is assigned to Object Form. This order is a convention: we did not evaluate alternative orderings, and axis assignments are conditional on it. Two figure-observable rules close known boundary cases: a Part-Whole rule (assign to Relation only when two visible parts 4 are judged jointly) and a Reference-Fidelity grounding guard (assign to Reference Fidelity only when the reference is visible and lexically grounded in the figure). Other claim properties (scope, attribute type, question type, presentation mode (how the compared panels are arranged, e.g., a direct pairwise comparison or a comparison against a reference panel), evidence format) are recorded as metadata rather than additional branches. Out-of-scope claims.Claims that are not figure-observable are assigned to an out-of-scope category with one of nine reason codes:temporal(motion or video),nonvisual-method-property(ar- chitecture, training, parameter count),metric-calibration(numeric scores),latent-internal (feature-space diagnostics),multi-sample-required(diversity claims requiring multiple out- puts),dataset-or-training(training-data or memorization claims),protocol-robustness (robustness to perturbations or test-time conditions),requires-external-knowledge, and not-visible-in-figure. Out-of-scope claims are excluded from axis assignment and from the benchmark. 2.3 Validation We validate the codebook two ways. First, we apply it to1,486figures from810VisionQ papers and check how many of their claims it can classify (the coverage check). Second, we run a qualitative stress test: frontier VLMs are shown the codebook and prompted to invent claims they expect the codebook to struggle with, and we manually check whether it classifies them correctly. Appendix B gives the full axis definitions, inclusion/exclusion criteria, and leaf list used to score both checks. Of1,486coverage-check assignments,1,426(96.0%) fall within the main taxonomy and60(4.0%) receive an out-of-scope code; all ambiguous cases (39.2%) each receive exactly one axis (the priority order serves as the tie-breaking guide), with no schema-invalid assignments. The axis distribution is imbalanced: Object Form43.4%, Reference Fidelity17.0%, Prompt Match 13.1%, Image Appearance10.9%, Relation7.9%, Scene Layout3.6%, Out-of-Scope4.0%. The low out-of-scope rate confirms broad coverage; the dominance of object-level criteria over whole-image appearance supports the need for a CV-paper-specific taxonomy rather than generic IQA dimensions. Field-specific evaluation vocabularies.As a second validation, we verify that the taxonomy’s leaf distinctions correspond to genuine inter-field differences in evaluation practice rather than annotation artefacts. Grouping all3,773in-scope data points by task type reveals strikingly field-specific leaf distributions (Figure 15, Appendix E). Reconstruction 3D papers concentrate57.8%of claim mass in three Object Form leaves (Detail, Completeness, Surface); generative 2D papers place30.2%of claim mass on a single Prompt Match leaf (Semantic Match), reflecting text-to-output faithfulness as the field’s defining criterion; segmentation and detection papers are the most distributed, with no leaf exceeding13.9%of claim mass. These distinct signatures confirm that the 51-leaf vocabulary is warranted: a coarser taxonomy would conflate evaluation criteria that practitioners treat as structurally distinct. 2.4 From Taxonomy to Benchmark The codebook supports a benchmark task: blind multiple-choice questions in which the model identifies which of several candidate crops best satisfies a leaf-level evaluator question. We synthesize MCQs from the same VisionQ figures used in the coverage check. For each figure, we identify candidate panels (method outputs, baselines, ground truth) and pair them with a leaf-level evaluator question drawn from the codebook, for example, “Which crop shows the highest reconstruction fidelity?”. The candidate panels are arranged into a single composite image with hard-rendered (A)/(B)/(C)/(D) banners, so that the model sees only the composite and the question stem; no figure caption, paper claim, or method name is shown. We synthesize332such MCQs, which form VisionQ-Bench, and apply a four-flag quality filter that removes records where the candidate set is not a valid pairwise comparison: cross-row mixing, inclusion of a context panel as a candidate, duplicate method labels, and single-method-family ablations. After filtering,309questions remain eligible for evaluation (duplicate method labels remove6questions, context panels offered as candidates9, and single-method-family ablations8; no candidate set mixes rows). The evaluation protocol and per-model results are reported in Section 4. 5 1 Corpus 2 Taxonomy & labels 3 Datasets 4 Uses Papers 1,409 in total CVPR 2023–24 ICCV 2023 Figures 3,651 extracted 1,800+ selected Author claims 9,228 claims 447 atoms Taxonomy 6 axes 51 leaves Annotated pool 3,911 data points 1,486 figures 810 papers In scope 3,773 data points 1,426 figures 790 papers VisionQ-Bench 332 questions 309 eligible 114 papers VisionQ-MCQ 4,524 questions 707 figures 513 papers Coverage analysis field statistics, selective comparison (§3) Judge evaluation 20 VLM judges, per-axis accuracy (§4) VisionQ-Judge 2,867 train 717 test 9,525 DPO pairs (§5) figure extraction Figure 3: How the VisionQ datasets relate: (1) the source corpus, (2) claims, taxonomy, and annotation, (3) the datasets built from the annotated pool, and (4) where each is used. Every population derives from the 1,409-paper source corpus. Claims extracted from the whole corpus define the taxonomy, which labels the annotated pool. The annotated pool supplies the in-scope data used for the coverage analysis and the two question sets: VisionQ-Bench, used to evaluate VLM judges, and VisionQ-MCQ, used to train and test VisionQ-Judge. Table 1 lists the exact sizes. Table 1: VisionQ populations at a glance. A data point is one annotated comparison row of a figure together with its author-stated claim; a question is one multiple-choice item. PopulationPapers Figures Data pointsQuestions Used for Source corpus1,409Taxonomy; 9,228 claims (§2.1) Candidate figures3,651Figure selection (§3.1) Comparison figures1,800+Annotation (§3.2) Box-annotated papers1,403Coverage denominator (§3.4) Annotated pool8101,4863,911Coverage check (§2.3) In scope7901,4263,773Coverage analysis (§3.4–3.5) VisionQ-Bench114132332 / 309 20 judges, 309 eligible (§4) VisionQ-MCQ5137071,7374,524 VisionQ-Judge (§5) Judge split2,867 / 717 Train / test; 9,525 pairs (§5.2) 3 Dataset Using the six-axis taxonomy defined in Section 2, we construct VisionQ’s dataset from a corpus of peer-reviewed CV papers. The dataset serves two roles: it provides the figure pool from which benchmark items are drawn, and it supplies the author-stated claims that fix the criterion and the correct answer of each question (Section 4). Figure 3 shows how every population used in this paper derives from the source corpus and where it is used; Table 1 lists their exact sizes. 3.1 Source Corpus and Figure Selection VisionQ is built from a corpus of1,409peer-reviewed computer-vision papers from CVPR and ICCV. We collected the papers from recent proceedings (CVPR 2023, ICCV 2023, and CVPR 2024; Figure 12), keeping papers whose experiments include qualitative comparison figures, the evidence VisionQ is built around. We extract all figures from each paper along with their captions, section-level context, and paper metadata including title, abstract, and result-section text. Figures are extracted from high-resolution PDF renders, and each figure is linked to its caption and to the body-text passages that reference it by layout proximity followed by a validation step (mean link confidence 0.93 on the released papers). 6 The full corpus serves two distinct roles, operating over two different populations. First, all1,409 papers feed taxonomy construction: the9,228grounded qualitative claims used to induce the 51-leaf codebook (Section 2) are extracted broadly across the entire corpus, independently of whether a paper later contributes figures to the benchmark. Second, the corpus seeds the benchmark figure funnel described next; this funnel narrows substantially, and the subset of papers that survives it should not be confused with the taxonomy-construction population. Automatic figure extraction over the1,409papers yields3,651candidate figures before quality filtering. From this pool, we identify qualitative-comparison figures: figures that place method outputs alongside baselines or reference images in a shared panel for direct visual comparison. A figure is included if it contains at least one method output compared against a baseline or reference in the same panel and if the comparison supports a visual preference judgment. Candidate figures are scored with comparison cues in the caption and referencing text, such as “qualitative comparison”, “compared with”, “our method”, “ground truth”, and comparative verbs such as “outperforms” or “better”. After this automatic filter, the corpus retains1,800+validated qualitative-comparison figures. 3.2 Hand-Curated Subset: Annotation Funnel and Final Release Not every validated figure yields a usable benchmark item. Deep per-figure annotation adds three layers on top of a validated figure: method-level crop annotations identifying which panel belongs to which method, review metadata linking the figure to its paper context, and author-stated claims extracted from the paper’s result section and figure caption. This succeeds only when a figure contains a well-formed comparison: panels must be cleanly croppable, the compared methods identifiable, and the authors’ claimed winner recoverable from the paper text. Figures failing any of these conditions at annotation or review time are dropped, so the counts below reflect annotation yield under quality gating, not a target coverage budget. Annotating the validated pool produces3,911human-annotated data points spanning1,486figures from810papers; excluding138out-of-scope data points leaves3,773in-scope data points (assigned to a taxonomy leaf), with790papers retaining at least one. This annotated pool is the population used for the taxonomy coverage check in Section 2.3 and the coverage analysis in Section 3.4. Each item is labelled independently by two annotators; when they disagree, a third annotator labels the item and the majority label is retained. From the annotated pool we synthesize multiple-choice questions (Section 3.3) and apply a final round of human review. Reviewers inspected every record and rejected760questions that failed quality review; we additionally enforce that each question contains exactly one panel from the paper’s proposed method (deduplicating violating records and salvaging48ablation-style questions by relabeling), and add a200-question top-up batch drawn from previously untapped annotated figures. The final hand-curated release, VisionQ-MCQ, contains4,524questions covering707figures from 513papers. Author-stated claims serve as ground truth in two ways: they fix the criterion and the correct answer of each question, and, because every question retains the full claim text, they allow a judge’s free-text justification to be checked against the criteria the authors claim their method improves. 3.3 Benchmark Item Construction Each benchmark item is constructed from one qualitative-comparison figure and contains four components: (1) the figure image with candidate outputs, baselines, and reference if present; (2) method labels or anonymized identifiers for each panel; (3) the author-stated claim or figure caption describing the expected judgment; (4) the taxonomy axis or axes to which the claim is assigned. Items without a figure-observable axis assignment (i.e., claims assigned out-of-scope codes) are excluded from the evaluation set. These fields are stored with each item; the judge sees only the composite image and the question (Section 4). One data point can yield several questions: a version with and one without the reference panel, and additional two-choice pairs drawn from the same comparison row. The 4,524 released questions therefore derive from 1,737 distinct data points, and all questions from one data point fall on the same side of every train/test split. Every question has exactly one correct answer, the author-claimed winner; comparisons in which no single winner can be identified 7 are discarded, so there are no ties. VisionQ-Bench is used for evaluation only, and VisionQ-MCQ is split for VisionQ-Judge as described in Section 5.2. 3.4 Coverage Analysis and Field Statistics Coverage breadth. VisionQ’s targeted sampler extensions yield broad coverage across the full paper corpus. 790 of the 1,403 papers with an annotated figure (56.3%) carry at least one in-scope data point, comprising3,773in-scope data points across1,426figures (the1,486annotated figures minus60whose claims are all out of scope), 20 task types, and all 51 observed taxonomy leaves. This exceeds the37.6%figure of the initial release—a 50% relative increase. No single task type accounts for more than18.0%of total data points (neural_radiance_field:18.0%, reconstruction_3d: 12.6%), confirming distributed coverage across domains rather than concentration in a single area. Per-task-type counts and full statistics are reported in Appendix E. Coverage depth. Coverage density (mean data points per covered paper) is lowest for motion_generation(3.1), withoptical_flow_stereoandsegmentation_detectionnext (both below 3.6), and highest fornovel_view_synthesis(6.5), withface_centricand generative_3dnext (both above 6). This reflects domain-specific figure practices: novel-view- synthesis and 3D-generation papers typically include dense spatial comparison panels, while seg- mentation and motion-generation papers tend toward sparser per-sample comparisons. The per-field depth breakdown is shown in Figure 14 (Appendix E). 3.5 Selective Comparison Analysis A structural property of qualitative CV evaluation not visible from aggregate coverage statistics is what papers choose not to compare. Author claims identify the dimension on which a method is shown to win; they reveal nothing about the field-standard dimensions that the comparison omits. We term this phenomenon selective comparison: a paper demonstrates improvement on its cited winning leafNwhile leaving untested the top-Kfield-standard leaves (the X-dims) that papers in its task type consistently apply. Setup. For each covered paper, let cited_leaves be the set of primary taxonomy leaves for which the paper contributes at least one in-scope data point—the dimensions it explicitly evaluated. Let top-Kfield leaves be theKmost frequent leaves in its task type, derived from the coverage matrix in Section 3.4. The untested X-dims are defined astop-K leaves\ cited_leaves(K=10throughout). A paper is a selective comparison candidate if it has at least one untested X-dim from its field’s top-K. Prevalence. Applying this analysis to the 790 covered papers (K=10) yields 789 selective com- parison candidates (99.9%). The distribution of untested X-dims reveals the extent of selectivity: 465 papers (58.9%) test exactly one leaf from their field’s top-10 standard criteria, leaving the remaining 9 standard criteria unexamined; 163 papers (20.7%) test exactly two; 95 papers (12.0%) test none—their comparison figures evaluate criteria that fall entirely outside their field’s top-10. The mean untested X-dim count is 8.73 out of 10, meaning the average covered paper leaves 8–9 of its field’s 10 most commonly applied evaluation dimensions uncovered. Figure 4 shows the full distribution. These numbers are not an indictment of paper quality: reporting one or two well-chosen qualitative comparisons per paper is standard practice and sufficient to support a claim. Rather, they quantify a structural feature of the literature—qualitative comparisons are highly selective by construction, and a paper that wins on its cited criterion may not generalise to the broader set of criteria its field applies. VisionQ provides the infrastructure to probe this gap systematically: for each candidate paper, VisionQ-Judge can assess whether the paper’s method also wins on the dimensions its own comparisons omitted, providing an automated sanity check on the scope of qualitative claims. This is a diagnostic of evaluation coverage, not a judgment of any paper’s merit or correctness. Appendix E.2 describes how a 200-paper sample for this assessment is drawn; running the assessment is left to future work. 8 01234 Criteria tested, out of the field's 10 most common 0 100 200 300 400 500 Papers 95 (12%) 465 (59%) 163 (21%) 56 (7%) 10 (1%) Figure 4: How many of their field’s 10 most commonly applied criteria the 789 selective-comparison candidates test in their qualitative comparisons. Most papers (59%) test exactly one. Question shown to the judge: “Which method demonstrates better sharper detailed textures without artifacts?” Criterion: Object Form / TextureCorrect answer: (B) Author claim (hidden from the judge): “Our method renders sharper detailed textures without artifacts.” AnswerModels (B), correctGPT-5.1, GPT-5.3-codex, GPT-5.4, Gemma-4-31B-it, Grok-4.3, Grok-4.20, Mistral Medium 3.5, Qwen 3.6-27B, Qwen 3.5-9B, MiMo v2.5, VisionQ-Judge, Gemma-4-E4B-it (base) (A)GPT-5.5, Haiku 4.5, Gemma-4-26B-A4B-it (C)Opus 4.6, GLM-5V Turbo, Nemotron-Omni (D)Sonnet 4.6, GPT-5.2, Qwen 3.6-35B-A3B Figure 5: An example VisionQ-Bench question, exactly as the judges see it: one composite image with lettered panels and the question text, without method names, caption, or author claim. The author claim identifies (B), the proposed method’s output, as best on the criterion. Twelve of the 21 evaluated models choose (B), including VisionQ-Judge; GPT-5.5, one of the two strongest judges overall, chooses (A), and the other errors are spread over (A), (C), and (D). 4 Benchmark Protocols and Model Evaluation VisionQ evaluates whether a model can select the method crop that best satisfies a taxonomy-grounded qualitative criterion. Each item contains a lettered composite image and a leaf-level evaluator question, but hides paper identity, method names, captions, and author claims. This isolates visual judgment from method-name priors and textual leakage. Figure 5 shows one question exactly as the judges see it. Two question sets. VisionQ provides two question sets built from the same annotated figure pool (Table 1). VisionQ-Bench (309 questions, this section) is a fixed evaluation set, frozen before the release was expanded so that every judge is compared on identical items. VisionQ-MCQ (4,524 questions, Section 3.2) is the expanded release, generated with a later sampler that recovers more valid comparisons per figure and adds reference-context variants; we use it to train and evaluate VisionQ-Judge under a paper-disjoint split (Section 5). Eligible evaluation slice. We generate 332 VisionQ-Bench questions and retain 309 eligible questions after filtering invalid candidate sets. The eligible slice spans six taxonomy axes: Object Form (n = 143), Reference Fidelity (n = 66), Image Appearance (n = 39), Relation (n = 31), Prompt Match (n = 22), and Scene Layout (n = 8). Per-axis accuracy with paper-clustered intervals 9 for every model is in Table 5. Axes withn < 30are marked low-confidence in per-axis profiles. Scene Layout (n = 8) and Prompt Match (n = 22) are too small for reliable per-axis conclusions; their size reflects annotation yield under quality gating (Section 3.2), not a sampling choice, and we report them for completeness. Metrics. We report raw accuracy (as a decimal in the text and as a percentage in figures and tables); a request that does not return a valid option (an unparseable response or an API failure) counts as incorrect. Because the random baseline differs by item choice count, we also report chance-normalized score: norm_score = acc− random 1− random . This score is 0 at chance, 1 at perfect accuracy, and negative below chance. It appears in the released dashboard; all comparisons in this paper use raw accuracy. Model comparison. Across 20 evaluated judges (19 external VLMs and VisionQ-Judge, with its Gemma-4-E4B base shown for reference), the strongest models reach 0.631 accuracy on the eligible slice (chance level: 0.322). OpenAI GPT-5.3-codex and GPT-5.5 tie for the highest point estimate, followed by GPT-5.2 and GPT-5.4. The tuned VisionQ-Judge reaches 0.482 on the same eligible slice, compared with 0.450 for its base model; this is the checkpoint trained before the release was expanded, for which VisionQ-Bench is the held-out test split, so none of its items were seen in training. With 4B effective parameters, VisionQ-Judge is within 1 p of Gemma-4-26B-A4B (0.485), Mistral Medium 3.5 (0.489), and Haiku 4.5 (0.492), and ahead of Qwen 3.5-9B (0.472) and Grok-4.20 (0.398). Figures 10 and 11 in Appendix C rank all 20 models overall and by axis. Full per-axis and per-leaf profiles are given in Appendix C. 5 VisionQ-Judge: Specialising a VLM via DPO Off-the-shelf VLMs exhibit two systematic failure modes on qualitative CV figures: positional bias (a model-specific tendency to over-predict certain letter options regardless of visual content [29]) and a domain gap from never having been trained to compare method outputs in peer-reviewed figures. We address both by fine-tuning a multimodal VLM via DPO on symmetric evidence pairs derived from VisionQ’s structured metadata. 5.1 LLM-Free DPO Dataset Construction An earlier iteration of our pipeline used a language model to generate question-answer pairs from paper captions, introducing the risk of hallucinated evidence. We instead build DPO pairs entirely from structured fields already present in the VisionQ data records: the author-stated claim text, the visual attribute description, winner method identification, and per-method bounding-box crops. No LLM calls are made at dataset construction time. Figure 6 summarises the full construction pipeline. More details on how we construct the DPO pairs can be found in Appendix D.1. 5.2 Symmetric Evidence Pairs Symmetric Construction. Initial experiments used long explanations forchosenresponses vs. bare letters forrejected. This length asymmetry (50:1 ratio) caused the model to optimize for explanation length rather than accuracy, leaving held-out performance stagnant at 37.3%. To eliminate stylistic shortcuts, we makechosenandrejectedstructurally identical. Each pair consists of a letter followed by a one-sentence evidence string (e), with only the letter reference differing. For a winner wand each loserr(one pair per wrong option), we define:y chosen = (w) e w andy rejected = (r) e r (Figure 7). The evidencee r is generated by replacing all instances of “imagew” in the original claim with “imager” via regex. This normalization succeeded in 90.4% of cases; otherwise, we defaulted to bare letters. This approach ensures near-zero initial KL divergence. Because both responses share identical length and style, the DPO objective cannot be improved through response form, only by choosing the correct letter. 10 SourceFilterFilterFilter data points (3,911) Mode & status filter Winner detection Distractor selection Raw JSON pair_compare reference_pair joint_pair SELF_REF regex bbox slug match SKIP_GT regex max 4 methods Question builder VisionQ-MCQ (4,524 questions) Pair constructor 1–N questions per data point: with / without reference panel, plus 2-choice pairs after human review: 760 rejected, 200 added one DPO pair per wrong option winner slug, method slugs input Figure 6: LLM-free DPO dataset construction pipeline. All fields are extracted directly from structured metadata. The structured VisionQ metadata yields 4,524 questions spanning 513 papers and 46 of the 51 taxonomy leaves. claim_text visual_attr normalize_claim() winner name ! “image w” normalised claim chosen: (w) evidence e w rejected: (r) evidence e r names the winner, “image w” letter swap: “image w” ! “image r” one pair per wrong option r identical body ) initial KL ¼ 0 Figure 7: Symmetric evidence pair construction. The same evidence bodyeis used in both chosen and rejected responses; only the letter reference differs. This ensures near-zero initial KL divergence, so the objective can only be improved by choosing the correct letter, not through response length or style. Implementation. Per-method crops are composited into a single labelled tile before tokenisation, avoiding multi-image tensor shape errors in TRL’sDPOTrainer(see Appendix D.2 for tiling details). We fine-tunegoogle/gemma-4-E4B-itwith LoRA. We cap each leaf at 300 questions (4,524→ 3,584) so that frequent leaves do not dominate training, then apply a paper-disjoint, leaf-stratified split, yielding 2,867 training and 717 test questions. Position-swap augmentation adds a reordered copy of each training question, and pairing the correct answer with each wrong option gives 9,525 training pairs; full detail in Appendix D.3. 11 Table 2: Main results on the VisionQ-Judge test set (717 questions). ModelOverall2-choice 3-choice 4-choice (n=717)(n=376)(n=157)(n=184) Base (Gemma-4-E4B-it)0.4990.5900.4010.397 DPO-tuned0.5240.6280.4330.391 ∆ (p)+2.5+3.7+3.2 −0.5 95% CI (p)[−1.9, +7.0] Table 3: Position-bias: predicted letter frequencies vs. gold. Bold values indicate the biased slot;† marks near-gold calibration. SettingSourceABCD 4-choice Gold28.8%23.4%24.5%23.4% Base15.2%18.5%29.9%36.4% Tuned17.9% 22.8% † 31.0%28.3% † 3-choice Gold30.6%34.4%35.0%- Base20.5%23.3%56.2%- Tuned19.2% 32.2% † 48.6%- 2-choice Gold51.9%48.1%-- Base36.4%63.6%-- Tuned42.8%57.2%-- 5.3 Results Main result. Table 2 shows overall accuracy on the held-out test set. DPO fine-tuning with symmetric evidence improves overall accuracy by +2.5 p or 5.0% relative (from 49.9% to 52.4%). The improvement holds for 2-choice (+3.7 p) and 3-choice (+3.2 p) questions; 4-choice accuracy is essentially unchanged (−0.5 p, one question). Uncertainty. Questions from the same source paper are not independent, so we report 95% confidence intervals from a paired bootstrap that resamples source papers (10,000 resamples). The overall gain is modest (+2.5 p, 95% CI−1.9 to +7.0 p). The effect DPO reliably produces is the reduction of positional bias: the share of answers on the last letter falls from 54.1% to 47.1% against a gold share of 38.9% (−7.0 p, 95% CI−10.9 to−3.2), and accuracy on the 438 questions whose correct answer is not the last letter rises from 36.5% to 43.8% (+7.3 p, 95% CI +1.7 to +13.2). Position-bias correction. Table 3 reveals a consistent last-option bias in the base model. Across all question types, the base model disproportionately predicts the final answer letter: option B in 2-choice questions (63.6% vs. 48.1% gold,+15.5 p excess), option C in 3-choice (56.2% vs. 35.0% gold,+21.2 p), and option D in 4-choice (36.4% vs. 23.4% gold,+13.0 p). After DPO fine-tuning with symmetric evidence, this bias is substantially reduced in all settings. The correction is most complete in 4-choice, where D-excess falls to 4.9 p (28.3% vs. 23.4% gold); in 2-choice, B-excess shrinks to 9.1 p (57.2% vs. 48.1% gold). The 3-choice C-bias is partially corrected (48.6% vs. 35.0% gold, 13.6 p excess remaining) and represents the residual calibration challenge. The bias falls steadily during training (4-choice D-share 36.4% for the base model, then 35.3%, 31.5%, and 28.3% after 300, 600, and 900 steps), and the accuracy gain is concentrated where the base model’s bias hurt it: on questions whose correct answer is not the last letter, accuracy rises from 36.5% to 43.8% (+7.3 p), while on questions whose answer is the last letter it moves from 71.0% to 65.9%. Symmetric evidence is designed to support this correction: because chosen and rejected differ only in which letter the evidence references, the DPO objective cannot be improved through length or style cues. Position-swap augmentation was applied in the same run, so we do not attribute the change to either component alone. Figure 8 shows the same distributions graphically. 12 AB 0% 20% 40% 60% % of predictions 2-choice ABC 0% 20% 40% 60% % of predictions 3-choice ABCD 0% 20% 40% 60% % of predictions 4-choice Predicted letter distribution vs gold (position-bias analysis) GoldBaseTuned Figure 8: Predicted letter-frequency distributions for the base and DPO-tuned models vs. gold on the 717-question test set, for 2-, 3-, and 4-choice questions. The base model over-predicts the last option (B, C, and D respectively); after DPO training the predicted frequencies move toward the gold distribution. 5.4 Per-Leaf Analysis Figure 9 shows the per-leaf accuracy delta for leaves withn≥ 6. We discuss the five most improved and five least improved leaves in turn. Top-5 improving leaves.Lighting (+28.6 p,n = 7), Anatomy (+25.0 p,n = 8), Blur (+21.1 p, n = 19), Attribute Binding (+20.0 p,n = 10), and Landmark Fidelity (+18.2 p,n = 11). Lighting, Anatomy, and Attribute Binding each come from a single source paper in the test split, so these gains should be read as paper-level observations. The remaining leaves cover perceptual sharpness, visual attribute grounding, and spatial precision—settings where the winning method’s advantage is visually distinctive and well-supported at 512 px tile resolution. The gains suggest that DPO training most effectively corrects positional bias on leaves where the visual difference is holistic and immediately legible from the composite tile. Bottom-5 leaves.Noise (−36.4p,n = 11), Semantic Match (−27.6p,n = 29), Segmentation (−14.3p,n = 14), Identity Preservation (−14.3p,n = 14), and Texture (−5.3p,n = 38). Two mechanisms plausibly explain these regressions: (1) Ceiling disruption: Noise and Semantic Match were the highest-accuracy leaves for the base model (72.7% and 69.0% respectively), leaving little room for DPO to improve; training instead perturbs well-calibrated, high-confidence predictions. Semantic Match (five papers) is the clearest case; Noise comes from a single test paper. (2) Subtle discriminative signals: identity and texture differences require fine-grained comparison of facial characteristics and surface micro-structure that symmetric evidence alone does not effectively target within a 512 px composite tile. Limitations. Several caveats bound our conclusions. (i) Small test support: many leaves have n < 15; per-leaf confidence intervals are wide, and 19 of the 38 leaves in the test split come from a single paper. Semantic Match (five papers) is the only multi-paper leaf whose change has a paper-clustered 95% interval that excludes zero. (i) Single model: we evaluate one base model (Gemma-4-E4B-it); results may not generalise to larger models or different architectures. (i) Synthetic evidence: the evidence strings are extracted programmatically; they are verifiably grounded in author claims but do not cover the full space of visual reasoning required for each leaf. Future directions.The ceiling-disruption pattern motivates curriculum or difficulty-weighted DPO, where already well-calibrated leaves are down-weighted during training to prevent high-confidence predictions from being perturbed. The residual 3-choice last-option bias (C at 48.6% vs. 35.0% gold) suggests that targeted oversampling of 3-choice items during training may further close this gap. Higher tiling resolution (672–768 px) would better preserve fine-grained signal for Identity 13 402002040 Accuracy delta (p) Texture Identity Preservation Segmentation Semantic Match Noise -5.3 n=38 -14.3 n=14 -14.3 n=14 -27.6 n=29 -36.4 n=11 Bottom-5 leaves 20020 Accuracy delta (p) Lighting Anatomy Blur Attribute Binding Landmark Fidelity +28.6 n=7 +25.0 n=8 +21.1 n=19 +20.0 n=10 +18.2 n=11 Top-5 leaves Per-leaf accuracy delta: DPO-tuned vs base Figure 9: Per-leaf accuracy delta (DPO-tuned vs. base) for taxonomy leaves withn≥ 6test questions. Left panel: five leaves with the largest decline. Right panel: five leaves with the largest improvement. Blur, Anatomy, and Lighting show the largest gains and Noise and Semantic Match the largest declines; Lighting, Anatomy, Attribute Binding, and Noise each come from a single test paper. Preservation and Texture leaves at the cost of increased VRAM. Multi-hop evidence chains (“imagew has smoother surfaces because its boundary is within 2 px of the ground-truth mask”) would provide stronger training signal for Reference Fidelity leaves. 6 Discussion and Limitations VisionQ makes qualitative figure judgment in CV papers a measurable, reproducible task for the first time. Beyond the benchmark itself, the six-axis taxonomy provides a vocabulary for criterion-level evaluation, and for rationale evaluation built on it, that may generalize to other visual comparison settings. We discuss the limitations that bound the current scope. Venue bias. The source corpus draws from CVPR and ICCV, which over-represent certain visual do- mains (e.g., image generation, segmentation, depth) relative to the broader CV literature. Benchmarks from other venues or tasks may exhibit different axis distributions. Dependence on author-stated claims. Ground truth for each question is derived from author claims, which may be aspirational, imprecise, or biased toward favorable comparisons. Claims are not independently verified against human perceptual ground truth. Axis imbalance. Axis sizes follow annotation yield: Object Form dominates and Scene Layout is sparse, so per-axis results for small axes are indicative only. Possible pretraining exposure. The source figures come from public 2023–2024 papers that may appear in the pretraining data of the evaluated VLMs. We reduce leakage by removing paper identity, captions, and method names and by recomposing cropped panels into new layouts with reassigned letters, so memorisation would require recognising the source figure and mapping the preferred panel to a new option. This mitigates but does not rule out contamination, and we do not test for it directly. As an exploratory check, accuracy on 2023 papers is not higher than on 2024 papers (mean difference −0.7p across the 21 models; higher for only 6), so we see no sign that longer public availability helps; because every evaluated model was released after 2024, this is not a direct test. No human baseline. We do not measure human accuracy on VisionQ-Bench, so the best score (0.631) should be read against chance rather than as human-level performance. Single-model demonstration. VisionQ- Judge is trained from one base model (Gemma-4-E4B-it); the DPO results are evidence for this model and setting, not across model families. Static-figure limitation. Temporal claims about video quality or motion consistency cannot be evaluated from static comparison figures and are excluded via out-of-scope codes. Methods evaluated primarily on video tasks are therefore underrepresented in the hand-curated subset. Verdicts vs. rationales. VisionQ scores which panel a judge selects under a stated criterion; it does not yet score the free-text rationales judges produce, nor establish that a stated rationale causally explains the verdict. The released claim-level annotations support 14 rationale scoring as a next step. Attribution methods such as gradient-based saliency [19] or attention rollout [1] would be needed for causal faithfulness, but require white-box access unavailable for the black-box judges we evaluate. Ethical and licensing considerations. Figures come from papers in the CVPR and ICCV proceedings, which are publicly available through CVF Open Access; copyright in the figures remains with their authors and publishers. We release our annotations, taxonomy, and questions under C BY 4.0 and our code under the MIT licence; figure crops are redistributed for non-commercial research use only, each linked to its source paper. VisionQ is intended for diagnostic evaluation of VLM judges; automated reviewing, scoring, or generation of scientific papers is outside its intended use. Authors who do not want their figures included can request removal through the dataset repository, and we remove them in the next release. Despite these limitations, VisionQ establishes the first evaluation infrastructure for reviewer-like qualitative figure judgment, and we release the corpus metadata, taxonomy, questions, model predic- tions, and benchmark and training code to support further development of evaluation methods that go beyond verdict accuracy. Acknowledgments and Disclosure of Funding VisionQ is built entirely from figures, captions, and result-section text in1,409peer-reviewed computer-vision papers (predominantly CVPR and ICCV, 2023–2024; venue and year recovered for 1,399of the1,409); source titles are listed in the accompanyingpaper_list.csvreleased with the code and data. We gratefully acknowledge these authors, whose qualitative comparisons and reported claims are the evidentiary basis for VisionQ’s taxonomy, dataset, and benchmark. This publication has emanated from research conducted with the financial support of Taighde Éireann – Research Ireland under Grant number 23/RC/13506. For the purpose of open access, the author has applied a C BY public copyright licence to any Author Accepted Manuscript version arising from this submission. References [1]S. Abnar and W. Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. [2]Z. Chen, H. Yao, Z. Zhao, and M. Yang. Advancing multimodal judge models through a capability-oriented benchmark and MCTS-driven data generation.arXiv preprint arXiv:2603.00546, 2026. [3]W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning, 2024. [4]J. Cho, A. Zala, and M. Bansal. Davidsonian scene graph: Improving reliability in fine- grained evaluation of text-to-image generation. In International Conference on Learning Representations, 2024. [5] X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-Y. Ma, and R. Kr- ishna. BLINK: Multimodal large language models can see but cannot perceive. In European Conference on Computer Vision, 2024. [6]L. Guerdan, S. Barocas, K. Holstein, H. Wallach, Z. S. Wu, and A. Chouldechova. Validating LLM-as-a-judge systems under rating indeterminacy. In Advances in Neural Information Processing Systems, 2025. [7]J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. [8] Y. Hu, B. Liu, Z. Kasner, H. Peng, A. Korhonen, M. Ostendorf, and R. Krishna. TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In International Conference on Computer Vision, 2023. 15 [9]K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu. T2I-CompBench: A comprehensive benchmark for open-world compositional text-to-image generation. In Advances in Neural Information Processing Systems, 2023. [10]Z. Huang, Y. He, J. Yu, F. Zhang, C. Si, Y. Jiang, Y. Zhang, T. Wu, Q. Jin, N. Chanpaisit, Y. Wang, X. Chen, L. Wang, D. Lin, Y. Qiao, and Z. Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [11] M. Kim, S. Lee, and D. Park. Vlm-subtlebench: How far are vlms from human-level subtle comparative reasoning?, 2026. URL https://arxiv.org/abs/2603.07888. [12] S. Kim, J. Shin, Y. Choi, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne, and M. Seo. Prometheus: Inducing fine-grained evaluation capability in language models. In International Conference on Learning Representations, 2024. [13] Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In Advances in Neural Information Processing Systems, 2023. [14]M. Ku, T. Li, K. Zhang, Y. Han, and W. Chen. VIEScore: Towards explainable metrics for conditional image synthesis evaluation. In Advances in Neural Information Processing Systems, 2024. [15]R. Liu, H. Weingord, S. Mittal, P. Dungarwal, A. Nandula, B. Ni, S. Basu, H. Chen, N. K. Ahmed, L. Li, J. Zhang, K. Goswami, S. Mukherjee, B. Kveton, P. Mathur, F. Dernoncourt, Y. Zhao, Y. Wang, R. A. Rossi, Z. Tu, and H. Du. Human-aligned mllm judges for fine- grained image editing evaluation: A benchmark, framework, and analysis, 2026. URLhttps: //arxiv.org/abs/2602.13028. [16]I. Mehmood, I. A. Shah, M. R. Luo, and B. Deegan. Vision-language models vs human: Perceptual image quality assessment, 2026. URL https://arxiv.org/abs/2603.24578. [17]R. C. Nickerson, U. Varshney, and J. Muntermann. A method for taxonomy development and its application in information systems. European Journal of Information Systems, 22(3):336–359, 2013. [18]A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. [19]R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In International Conference on Computer Vision, 2017. [20] A. Tamkin, M. McCain, K. Handa, E. Durmus, L. Lovitt, A. Rathi, S. Huang, A. Mountfield, J. Hong, S. Ritchie, T. Belonax, K. K. Troy, D. Amodei, J. Kaplan, J. Clark, and D. Ganguli. Clio: Privacy-preserving insights into real-world AI use. arXiv preprint arXiv:2412.13678, 2024. [21]J. Wang, K. C. K. Chan, and C. C. Loy. Exploring CLIP for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023. [22]H. Wu, Z. Zhang, E. Zhang, C. Chen, L. Liao, A. Wang, C. Li, W. Sun, Q. Yan, G. Zhai, and W. Lin. Q-bench: A benchmark for general-purpose foundation models on low-level vision. In International Conference on Learning Representations, 2024. [23]H. Wu, Z. Zhang, W. Zhang, C. Chen, C. Li, X. Hou, A. Wang, W. Sun, Q. Yan, G. Zhai, and W. Lin. Q-Align: Teaching LMMs for visual scoring via discrete text-defined levels. In International Conference on Machine Learning, 2024. [24]T. Wu, J. Zou, J. Liang, L. Zhang, and K. Ma. VisualQuality-R1: Reasoning-induced image quality assessment via reinforcement learning to rank. In Advances in Neural Information Processing Systems, 2025. 16 [25]X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023. [26]T. Xiong, X. Wang, D. Guo, Q. Ye, H. Fan, Q. Gu, H. Huang, and C. Li. LLaVA-Critic: Learning to evaluate multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. [27]J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong. ImageReward: Learning and evaluating human preferences for text-to-image generation. In Advances in Neural Information Processing Systems, 2023. [28] Z. Zhang, H. Wu, C. Li, Y. Zhou, W. Sun, X. Min, Z. Chen, X. Liu, W. Lin, and G. Zhai. A-bench: Are LMMs masters at evaluating AI-generated images? In International Conference on Learning Representations, 2025. [29] C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations, 2024. [30]L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, 2023. 17 A Related Work A.1 Image Quality Assessment and Low-Level Visual Benchmarks Image quality assessment (IQA) benchmarks have established rigorous protocols for measuring VLM performance on low-level visual quality tasks. Q-Bench [22] introduced a suite of low-level visual perception tasks covering quality description, comparison, and rating, revealing strong low-level capabilities in current foundation models. Q-Align [23] extended this by training VLMs on discrete human quality ratings, achieving strong alignment with mean opinion scores across multiple IQA datasets. A-Bench [28] specifically targets AI-generated image evaluation, addressing the distribu- tional gap between natural and synthetic content. CLIP-IQA [21] shows that CLIP representations capture both technical quality and aesthetic properties, extending perceptual assessment beyond distortion-based metrics. More recently, VisualQuality-R1 [24] integrates reinforcement learning to reason about quality in ranked comparisons, producing chain-of-thought rationales alongside quality verdicts. Concurrent work [16] directly compares VLM quality assessments against human perceptual judgments, finding attribute-dependent variability and clear gaps on contrast-related criteria. These benchmarks demonstrate strong VLM capability at scalar perceptual quality assessment. However, they score individual images or unstructured pairs, with no reference to the structured comparison task that qualitative figures in CV papers impose on a reviewer. In contrast, VisionQ asks a judge to compare candidate outputs drawn from paper figures and choose the one that best satisfies a named visual criterion. A.2 VLM-as-Judge and Preference Evaluation The LLM-as-judge paradigm was established by Zheng et al. [30], who used GPT-4 as an automated evaluator for open-ended dialogue quality via pairwise comparison. Chatbot Arena [3] extended this to crowd-sourced human preference elicitation at scale. This framework was extended to vision through preference-based reward models; ImageReward [27] trains such a model from human annotations for text-to-image generation. HPSv2 [25] constructs a large-scale human preference dataset for T2I scoring, and PickScore [13] learns human preference from an open dataset of real user selections. Prometheus [12] and LLaVA-Critic [26] take a different approach, training language and vision- language models specifically to produce fine-grained evaluation with natural-language feedback. VIEScore [14] introduces explainability into T2I evaluation by prompting VLMs to score images on semantic and perceptual dimensions with rationale generation. Concurrent benchmarks have extended judge evaluation further: M-JudgeBench [2] decomposes MLLM judgment into ten fine-grained capability dimensions, and a concurrent benchmark for fine-grained image editing evaluation [15] decomposes judgments into 12 interpretable factors covering edit quality and instruction fidelity. Systematic failure modes of judge systems have also received attention. Guerdan et al. [6] identified conditions under which LLM-as-judge ratings become indeterminate, showing that verdict reliability depends on item-level rating variance. Wang et al. [29] demonstrated that VLMs exhibit position and label biases in multiple-choice settings, which can corrupt pairwise preference outcomes. Despite these advances, existing judge benchmarks measure verdict-level accuracy or calibration. Verdict- only evaluation cannot expose a judge that cites incorrect visual evidence while selecting the right image. VisionQ addresses this by conditioning each verdict on a named visual criterion, so a judge is credited only for choices that follow it, and by keeping the author-stated claims needed to evaluate its justifications. A.3 Visual Reasoning, Text-Image Alignment, and Compositional Benchmarks Text-image alignment evaluation has a history rooted in embedding-space metrics: CLIP [18] learns joint text-image representations that serve as the backbone for many downstream alignment metrics, and CLIPScore [7] adapts CLIP similarity into a reference-free evaluation metric that correlates well with human alignment judgments. More recent benchmarks probe VLMs on structured visual reason- ing and compositional alignment. TIFA [8] evaluates text-to-image faithfulness by checking whether generated images satisfy fine-grained question-answer pairs derived from the prompt. DSG [4] con- structs structured dependency graphs over prompt clauses to enable reliable fine-grained alignment evaluation. T2I-CompBench [9] provides a comprehensive compositional evaluation suite covering attribute binding, spatial relations, and non-spatial properties. BLINK [5] tests VLMs on multi-image perception tasks including spatial reasoning, visual correspondence, and depth estimation, finding 18 that current models perform near chance on many such tasks. VBench [10] decomposes video generation evaluation into multiple quality and semantic dimensions, demonstrating that per-axis analysis exposes failure modes that aggregate metrics miss. VLM-SubtleBench [11] specifically targets subtle visual differences across 13K triplets, showing that proprietary VLMs lag human performance by over 30 percentage points on spatial and temporal discrimination. These benchmarks provide useful evaluation dimensions that partially overlap with our taxonomy axes, such as spatial relations and attribute binding. However, they are designed for text-to-image or video generation evaluation on curated prompts, not for qualitative comparison figures extracted from peer-reviewed CV papers grounded in author-stated claims. VisionQ is the first benchmark constructed from this source and designed for this reviewer-like task. A.4 Qualitative Evaluation in Computer-Vision Papers Qualitative evaluation in computer-vision papers has traditionally relied on human inspection of comparison figures or crowd-sourced preference studies [27,25]. These studies aggregate pairwise preferences across annotators but do not require annotators to identify specific visual criteria or ground their preference in a paper’s stated claims. Automated substitutes for this task, using VLM judges to assess method outputs in peer-reviewed figures, have not been benchmarked. No prior benchmark tests whether a judge can replicate the reviewer’s structured comparison process: comparing candidates against references, locating relevant visual differences, and attributing them to specific criteria. In contrast, VisionQ supplies figure-level ground truth derived from author-stated claims and conditions every judgment on a named visual criterion, enabling criterion-level evaluation and providing claim-level ground truth for rationale grounding. Taken together, these four bodies of work reveal a shared gap: no existing benchmark provides a controlled vocabulary for the visual criteria that drive reviewer-like judgment, nor a protocol that ties each verdict to a named, figure-observable criterion. VisionQ addresses both through a purpose-built taxonomy and evaluation infrastructure, which we describe next. B Full Taxonomy Codebook This appendix reproduces the codebook underlying the six-axis taxonomy of Section 2. For each axis we give its definition, the evaluator question used to phrase leaf-level judgments, the positive scope (what the axis includes), and the negative scope (what it excludes and where those claims are assigned instead); most cross-axis assignment is expressed directly in this exclusion scope, but Image Appearance additionally carries a short list of explicit assignment rules because it is the most common source of confusion with neighboring axes. Table 4 then lists all51leaves grouped by axis with a one-line working definition for each; per-leaf evaluator questions, example claims, and the intermediate mid-level grouping used during construction (Section 2) are maintained in the internal codebook and omitted here for space, except the Reference Fidelity split (Source Preservation vs. Target Fidelity) shown in the table’s Group column. The nine out-of-scope reason codes are defined in Section 2; we omit the corresponding out-of-scope category tree here for space. B.1 Image Appearance Definition. Whole-panel, crop-level, or panel-level low-level visual appearance. Evaluator question. Does the figure, crop, or panel have acceptable low-level visual appearance? Includes. Blur, noise, exposure, contrast, color, lighting, global artifact, realism, and style. Excludes. Object shape errors, spatial relation errors, prompt mismatch, reference mismatch, point distribution, keypoint accuracy, mask accuracy, and latent attention-map behavior. • Feature-map / attention-heatmap / t-SNE / latent-internal visualization -> Out-of-scope / latent-internal. Exception: only when the generated output image itself is a heatmap and that output’s appearance is being judged. • Multi-sample diversity / mode coverage / ‘outputs all look the same’ -> Out-of-scope / multi-sample-required. NOT Image Appearance / Realism. • Scale consistency / view consistency / perspective drift / physical plausibility -> Scene Layout. NOT Image Appearance / Realism, even when the natural-language phrasing is ‘realistic’. 19 •Face deformation / body anatomy / finger count / malformed limbs / warped contours -> Object Form (Face / Anatomy / Shape / Boundary). NOT Image Appearance / Realism or Lighting. B.2 Object Form Definition. Internal quality of visible objects, regions, bodies, faces, surfaces, and masks. Evaluator question. Are visible objects or regions internally well-formed? Includes. Shape, boundary, texture, surface, material, completeness, pose, face, anatomy, detail, and segmentation boundary quality. Excludes. Whole-scene arrangement, pairwise relation, explicit prompt matching, and visible reference or ground-truth matching. B.3 Scene Layout Definition. Scene-level spatial organization and physical arrangement. Evaluator question. Are scene-level spatial properties correct? Includes. Composition, depth, scale, view, spatial distribution, occlusion layout, camera geometry, and physical plausibility. Excludes. Object-internal defects, exact pairwise relation claims, explicit reference matching, and prompt-specific layout instructions. B.4 Relation Definition. Correctness of relationships between visible entities, parts, attributes, or correspondence points. Evaluator question. Are relationships between visible entities correct? Includes. Contact, correspondence, attribute binding, spatial relation, interaction, and part-whole relation. Excludes. Single-entity pose plausibility, ground-truth/keypoint accuracy, bounding-box accuracy, segmentation quality, and protocol robustness. A single malformed or missing part on one entity is assigned to Object Form (Anatomy or Completeness), not Relation. B.5 Reference Fidelity Definition. Whether an output preserves or matches a visible reference—source image, target image, ground truth, annotation, mask, landmark, or before/after state. Pure baseline / method comparison without a visible reference is not Reference Fidelity. Evaluator question. Does the output preserve or match a visible reference (source, target, ground truth, annotation, mask, landmark, or before/after state)? Includes. Identity preservation, background preservation, edit preservation, style fidelity (reference- driven), reconstruction fidelity, geometry fidelity, mask fidelity, landmark fidelity, segmentation fidelity. Excludes. Prompt-only matching (-> Prompt Match), absolute image appearance with no reference (-> Image Appearance), single-object form with no visible reference (-> Object Form), protocol robustness (-> out-of-scope), pure baseline/method comparison without a visible reference (assign to whichever axis the actual visual criterion belongs to). B.6 Prompt Match Definition. Match to a text prompt, caption, instruction, requested content, or requested attribute. Evaluator question. Does the output match the text prompt, caption, or instruction? Includes. Object presence, attribute match, spatial instruction, semantic match, text legibility, count match, action match, and expression match. Excludes. Lighting, expression, physical plausibility, or object quality when not specified by prompt/caption/instruction. 20 Table 4: Full51-leaf codebook, grouped by axis. Group indicates the internal Reference Fidelity split (Source Preservation vs. Target Fidelity); all other axes are ungrouped. AxisGroupLeafDefinition Image Appear- ance Blur Focus loss, smear, or blur visible at the panel or crop level. NoiseRandom speckle, grain, sensor-like corruption, or noisy visual artifacts. ExposureOver-exposure, under-exposure, saturation clip- ping, or visibility loss from brightness. ContrastInsufficient or excessive tonal separation that harms visible interpretation. ColorColor cast, color inconsistency, or visually implau- sible color independent of prompt/reference crite- ria. LightingIllumination, shadow, reflection, or lighting quality visible in the output. Global ArtifactPanel-wide artifact not better explained by object form, layout, relation, prompt, or reference match. RealismOverall perceptual plausibility or photorealism when not tied to a specific object, prompt, or refer- ence. StyleGlobal visual style or aesthetic quality when judged directly rather than as reference or prompt match. Object FormShapeObject geometry, silhouette, deformation, or struc- tural shape correctness. BoundaryObject or region edge quality, contour sharpness, silhouette completeness, or boundary artifacts. TextureLocal texture quality, texture continuity, or texture corruption on visible regions. SurfaceSurface smoothness, topology, wrinkles, holes, or visible surface defects. MaterialMaterial appearance or material property quality of an object or region. Completeness Missing, truncated, duplicated, or incomplete ob- ject parts. Pose Pose plausibility of a single visible object, body, or articulated entity when not judged against ground truth. FaceFace-specific form quality including expression quality when not prompt- or reference-driven. AnatomyBody, limb, hand, joint, or anatomical plausibility of one entity. DetailFine detail, thin structures, small parts, or local feature preservation. SegmentationMask or segmentation visual quality when judged as object-region form rather than ground-truth match. Scene LayoutCompositionOverall arrangement of regions, objects, fore- ground, background, or panel contents. DepthDepth ordering, 3D arrangement, or perceived scene depth. ScaleRelative or absolute size plausibility at scene level. ViewViewpoint, novel-view layout, or multi-view scene- level consistency. Spatial Distribu- tion Distribution, spacing, clumping, or spread of visi- ble scene elements or point samples. Occlusion Lay- out Occlusion ordering or impossible scene-level over- lap patterns. Camera Geome- try Camera perspective, projection geometry, or view- point geometry visible in the figure. 21 AxisGroupLeafDefinition Physical Plausi- bility Scene-level physical plausibility such as floating objects, floor contact at scene scale, or impossible arrangements. RelationContactWhether two visible entities or parts touch, con- nect, intersect, or maintain contact correctly. CorrespondenceVisible correspondence between entities, parts, points, views, or panels. Attribute Bind- ing Correct binding of visible attributes to the right entity or part. Spatial RelationPairwise relation such as left/right, above/below, inside/outside, near/far, or in-front/behind. Interaction Visible interaction between people, objects, agents, tools, or scene elements. Part-Whole Re- lation Relation between two or more visible parts judged jointly (e.g., wheel misaligned with axle). Reference Fi- delity Source Preserva- tion Identity Preser- vation Preservation of subject/person/object identity against a visible source/reference image. Source Preserva- tion Background Preservation Preservation of background content from a visible source/reference image. Source Preserva- tion Edit Preserva- tion Preservation of non-edited content in before/after or edit-reference setups. Source Preserva- tion Style Fidelity Style copied from a visible reference image (DreamBench-style transfer). NOT for prompt- requested style or intrinsic rendering style. Target Fi- delity Reconstruction Fidelity Reconstruction fidelity against a target, source, or ground truth (e.g., PSNR/SSIM/LPIPS-style judg- ments on visible reference). Target Fi- delity Geometry Fidelity Geometric fidelity to a visible source, target, anno- tation, or ground truth. Target Fi- delity Mask FidelityMask fidelity against a visible mask, annotation, or ground truth. Target Fi- delity Landmark Fidelity Keypoint, landmark, or pose-coordinate fidelity against visible ground truth or annotation. Target Fi- delity Segmentation Fidelity Segmentation output fidelity against visible ground-truth segmentation. Prompt MatchObject PresenceRequested object, entity, category, or scene ele- ment is present or absent as specified. Attribute MatchRequested color, material, style, state, or other attribute is correctly assigned. Spatial Instruc- tion Prompt-specified spatial arrangement or relation is satisfied. Semantic Match Overall semantic content matches the caption, prompt, or instruction. Text LegibilityRequested text, glyphs, symbols, or readable writ- ing are rendered correctly. Count MatchRequested number of objects or entities is correct. Action MatchRequested action, activity, or event is depicted cor- rectly. Expression Match Requested facial expression, emotion, or pose state is depicted correctly. C Full Benchmark Profile The full experiment reports 20 models on the eligible slice (n = 309). Figures 10 and 11 show overall and per-axis accuracy with Wilson intervals over questions; the released dashboard adds per-axisnorm_score, parameter-scaling plots for models with disclosed sizes, and per-model strongest/weakest leaves. Table 5 reports every model’s per-axis accuracy with paper-clustered 22 20%30%40%50%60%70%80% Accuracy on eligible split (%) Grok-4.20 Gemma-4-E4B-it (base of VisionQ-Judge) Qwen 3.5-9B VisionQ-Judge (Ours) Gemma-4-26B-A4B-it Mistral Medium 3.5 Anthropic Haiku 4.5 NVIDIA Nemotron-Omni Anthropic Sonnet 4.6 Qwen 3.6-27B Grok-4.3 Gemma-4-31B-it Qwen 3.6-35B-A3B Anthropic Opus 4.6 GLM-5V Turbo OpenAI GPT-5.1 Xiaomi MiMo v2.5 OpenAI GPT-5.4 OpenAI GPT-5.2 OpenAI GPT-5.3-codex OpenAI GPT-5.5 Headline accuracy with Wilson 95% CI random ≈ 32% Figure 10: Headline accuracy of 20 models on VisionQ-Bench (n = 309). VisionQ-Judge is the checkpoint for which VisionQ-Bench is the held-out test split; its base model is shown for reference. Unparsed responses count as incorrect. 2035506580 Acc (%) Grok-4.20 Gemma-4-E4B-it (base of VisionQ-Judge) Qwen 3.5-9B VisionQ-Judge (Ours) Gemma-4-26B-A4B-it Mistral Medium 3.5 Anthropic Haiku 4.5 NVIDIA Nemotron-Omni Anthropic Sonnet 4.6 Qwen 3.6-27B Grok-4.3 Gemma-4-31B-it Qwen 3.6-35B-A3B Anthropic Opus 4.6 GLM-5V Turbo Xiaomi MiMo v2.5 OpenAI GPT-5.1 OpenAI GPT-5.4 OpenAI GPT-5.2 OpenAI GPT-5.5 OpenAI GPT-5.3-codex Object Form (n=143) 2035506580 Acc (%) Reference Fidelity (n=66) 2035506580 Acc (%) Image Appearance (n=39) 2035506580 Acc (%) Relation (n=31) Per-axis accuracy with Wilson 95% CI (axes with n ≥ 30; models sorted by overall accuracy) Figure 11: Per-axis accuracy with Wilson 95% CI (axes with more than 30 samples; models sorted by overall accuracy) 95% intervals, which are wider than the Wilson intervals in the figures because questions from the same paper are not independent. D VisionQ-Judge Training Details D.1 LLM-Free DPO Dataset Construction Source records. Each record in the VisionQ semantic dataset contains: (i) aclaimobject with winner_method,visual_attribute_text, andclaim_text; (i) ataxonomyobject with the taxonomy leaf assignment andresolver.gate_decision; (i)evidence.bboxeslisting per- method image crops with theircrop_pathandmethod_rawslug (a normalised method-name 23 Table 5: Accuracy (%) per axis on VisionQ-Bench with 95% confidence intervals from a bootstrap that resamples source papers. Column headers give questions / source papers. Relation, Prompt Match, and Scene Layout rest on few papers, so their intervals are wide and category-level conclusions for them are indicative only. ModelOverall Object Form Ref. Fidelity Img. Appearance Relation Prompt Match Scene Layout 309 / 109143 / 5066 / 2339 / 1631 / 822 / 98 / 4 OpenAI GPT-5.3-codex63 [57,69]64 [55,73]59 [45,71]69 [53,83]58 [46,78]59 [29,88]75 [40,100] OpenAI GPT-5.563 [57,69]68 [58,77]61 [48,72]64 [48,79]45 [31,60]64 [38,85]62 [29,100] OpenAI GPT-5.261 [55,67]66 [57,75]55 [43,66]62 [46,76]58 [48,75]59 [29,84]50 [20,100] OpenAI GPT-5.461 [55,67]64 [56,73]56 [43,68]67 [49,82]55 [44,65]59 [28,86]38 [0,86] OpenAI GPT-5.159 [53,66]59 [50,69]53 [40,66]69 [52,85]52 [35,68]64 [33,89]75 [54,100] Xiaomi MiMo v2.559 [53,65]64 [55,73]52 [40,64]67 [50,81]45 [30,58]59 [36,81]50 [20,100] GLM-5V Turbo59 [53,65]57 [49,65]53 [41,65]77 [62,89]55 [45,73]68 [42,90]25 [0,86] Anthropic Opus 4.658 [52,65]62 [52,73]61 [49,71]56 [39,72]39 [20,62]59 [29,84]50 [20,100] Qwen 3.6-35B-A3B58 [51,64]62 [54,70]48 [35,62]62 [38,83]45 [32,61]64 [35,89]62 [29,100] Gemma-4-31B-it57 [51,64]64 [54,72]52 [40,64]62 [44,79]42 [20,63]50 [13,83]50 [0,86] Grok-4.356 [49,62]62 [52,72]56 [45,67]46 [29,63]42 [17,59]50 [15,81]75 [40,100] Qwen 3.6-27B55 [49,62]61 [51,70]47 [36,58]51 [31,71]48 [35,62]59 [25,88]62 [29,100] Anthropic Sonnet 4.654 [48,61]59 [49,68]55 [44,66]54 [37,69]39 [19,50]59 [28,88]25 [0,86] NVIDIA Nemotron-Omni 51 [45,57]56 [49,63]47 [32,61]49 [28,69]39 [21,49]55 [26,79]38 [0,86] Anthropic Haiku 4.549 [43,55]50 [39,60]44 [32,56]54 [37,69]55 [45,64]50 [21,79]38 [0,71] Mistral Medium 3.549 [42,56]55 [44,65]32 [23,40]64 [43,82]35 [19,48]45 [16,69]75 [0,100] Gemma-4-26B-A4B-it49 [42,55]54 [45,63]39 [28,52]59 [43,73]32 [17,50]45 [14,76]50 [20,100] VisionQ-Judge (Ours)48 [42,55]52 [41,62]39 [28,51]59 [42,76]39 [33,48]45 [14,76]50 [31,100] Qwen 3.5-9B47 [41,53]53 [44,63]36 [23,48]46 [25,67]45 [22,69]50 [25,71]38 [20,75] Gemma-4-E4B-it (base)45 [39,51]48 [38,57]41 [29,52]56 [41,73]32 [23,48]41 [15,64]38 [14,100] Grok-4.2040 [35,45]41 [33,48]38 [28,49]41 [26,57]52 [35,65]18 [0,43]50 [20,100] string); (iv) aprovenance.has_winner_evidence_in_groupflag. We retain records where validation.overall_status ∈proposed, validated,gate_decision = main_tree, and has_winner_evidence = True, yielding the candidate pool from which MCQ items are drawn. Mode filtering. We further restrict to presentation modes that genuinely compare competing methods:pair_compare,reference_pair, andjoint_pair. Modes such assingle_panel(no comparison partner) are excluded. Winner identification.Thewinner_methodfield contains the paper’s method name, which must be matched against themethod_rawslugs of the figure’s bounding boxes. We apply a two-stage matching procedure. First, we split the winner name on comma/semicolon delimiters and attempt case-insensitive substring matching against method_raw for each part of length≤5 words. Second, if no direct match is found, we apply a self-reference regex to detect “ours / our method / the proposed . . . ” and fall back to the bbox withmethod=ours. Records where neither strategy identifies a winner bbox are discarded. This process, combined with the utilization-based sampler, yields 4,524 usable questions covering 46 of the 51 taxonomy leaves. Distractor selection.For each winner, we select up to three distractor methods from the remaining bboxes. We exclude bboxes whosemethod_rawmatches a ground-truth/reference regex to prevent the judge from learning to reject reference images rather than competing methods. For each slug, we keep only the single best bbox (typemainpreferred, then lowestbbox_id) to avoid repeated crops of the same method. The resulting comparison has 2–4 methods; winner is assigned a random letter position (seeded for reproducibility). D.2 Image Preprocessing and Tiling Each VisionQ record provides individual bounding-box crops for each method. A 4-method question therefore has 4 separate image files. Multimodal DPO trainers (we use TRL’sDPOTrainer) have a known failure mode with multi-image prompts: image-token counts and vision-patch feature counts diverge across chosen and rejected when images differ, causing a tensor shape error during loss computation. We circumvent this by compositing all method crops into a single labelled tile. The tiling procedure: (1) square-pad each crop to its longest side using neutral grey; (2) resize to≤ 512px (bicubic); (3) arrange in a1×Ngrid forN ≤ 2or2×⌈N/2⌉forN ≥ 3; (4) overlay a circled letter badge (A–D) at 24 pt in the top-left corner via PILImageDraw. This matches inference-time behaviour and published VLM-DPO recipes [26, 12]. 24 Figure 12: Source papers by venue and year, over the full1,409-paper corpus underlying VisionQ (1,399 with known venue). 2024 is CVPR-only because ICCV is held only in odd years. D.3 Training Configuration Base model. google/gemma-4-E4B-it (4.5B effective parameters and 8B total parameters with embeddings, bfloat16) on a single 46 GB GPU;∼16 GB resident with per-sample batch size 1 and gradient accumulation 8 (effective batch 8). LoRA adapter. Rankr=16,α=32,dropout 0.05.Target modules: .*language_model\..*\.(q|k|v|o|gate|up|down)_proj$, scoping adaptation to the lan- guage decoder only and skipping Gemma 4’s Gemma4ClippableLinear vision-tower wrapper. DPO hyperparameters.β = 0.3, learning rate5× 10 −6 with linear decay and 5% warmup, max_length = 1024 tokens, max_image_size = 512 px. We train the model for 1 epoch. Train/test split. Each leaf is first capped at 300 questions (4,524→3,584, seed 42). A paper- disjoint, leaf-stratified split with targetf test = 0.15then yields 2,867 training and 717 test questions; the realised test share is 20% because whole papers are assigned to one side. Position-swap augmen- tation gives 5,729 training questions and 9,525 training pairs. E Coverage Analysis Supplement This appendix reports the per-task-type coverage statistics and field evaluation signature visualisations supporting Sections 2.3, 3.4, and 3.5. E.1 Source Corpus Distribution Figure 12 shows the full1,409-paper source corpus underlying VisionQ by venue and year. Venue and year are known for1,399of the1,409papers (99.3%); the remaining10could not be identified and are excluded from the figure. The comparison-group statistics below are computed over the bounding-box-annotated figures, a distinct population from both the3,651candidate figures before quality filtering (Section 3.1) and the1,486-figure coverage-annotated subset used for taxonomy validation (Section 2.3): each annotated figure contains one or more side-by-side image groups (e.g., method vs. baseline vs. ground truth); across9,382such groups (median4images),88.0%contain at least two images,76.3%contain at least three, and64.2%contain at least four – sufficient to seed the2–4-candidate multiple-choice questions of Section 2.4. At least one group with≥ 2images appears in97.4%of papers and95.0%of annotated figures, so benchmark coverage is not driven by a small subset of comparison-rich papers. The full paper list (title, venue, year, task type) is released as paper_list.csv alongside the code and data. Dataset composition. Table 6 summarises the composition of the VisionQ dataset. 25 Table 6: VisionQ dataset composition. ItemCount Notes Papers with≥ 1 annotated figure1,403 Of the 1,409 source papers (Section 3.1) Task types20 Task-type labels assigned to each paper Annotated figures1,486Qualitative-comparison figures with≥1 data point Total data points3,911Sample-level records (figure× group) In-scope data points3,773After excluding 138 out-of-scope records (60 fig- ures) Papers with≥ 1 data point79056.3% of the 1,403 papers with an annotated figure Taxonomy leaves observed51Full 51-leaf codebook covered Per-task-type paper and coverage counts. Figure 13 shows the number of papers per task type. The five largest fields (neural_radiance_field, generative_2d, reconstruction_3d, point_cloud_analysis, segmentation_detection) together account for 784 of the 1,403 papers (55.9%). Figure 14 reports total data-point counts and mean data points per covered paper. novel_view_synthesis has the highest evaluation density (6.5 data points per covered paper), followed by face_centric and generative_3d (both above 6), despite their moderate corpus share. 0255075100125150175200 Number of papers slam localization other face centric tracking image restoration optical flow stereo hand centric gaussian splatting motion generation depth estimation scene understanding human avatar generative 3d human pose mesh novel view synthesis reconstruction 3d segmentation detection point cloud analysis generative 2d neural radiance field 19 20 22 25 27 29 32 38 39 43 50 56 63 75 81 138 141 144 166 195 Papers per task type (N = 1,403 papers with an annotated figure) Figure 13: Number of papers per task type in the VisionQ corpus (N =1,403papers with an annotated figure). The five largest fields together account for over half of them. 0100200300400500600700 Total data points other motion generation optical flow stereo tracking slam localization scene understanding hand centric image restoration face centric gaussian splatting human pose mesh depth estimation human avatar segmentation detection generative 3d novel view synthesis point cloud analysis generative 2d reconstruction 3d neural radiance field Coverage depth: total data points 0123456 Mean data points per covered paper other motion generation optical flow stereo tracking slam localization scene understanding hand centric image restoration face centric gaussian splatting human pose mesh depth estimation human avatar segmentation detection generative 3d novel view synthesis point cloud analysis generative 2d reconstruction 3d neural radiance field Coverage density Figure 14: Coverage depth per task type. Left: total data-point count. Right: mean data points per covered paper (papers with at least one in-scope data point). Novel view synthesis has the highest evaluation density, followed by face-centric and 3D-generation papers. 26 Field evaluation signatures. Figure 15 shows the full task-type×leaf coverage matrix, row- normalised to percentage of each field’s claim mass. Figure 16 shows the corresponding primary- axis distribution. Together these visualisations document the field-specific evaluation vocabularies described in Section 2.3: fields use the taxonomy in structurally distinct ways that cannot be captured by a coarser-grained scheme. Detail Semantic Match Reconstruction Fidelity Completeness Surface Correspondence Boundary Shape Texture Global Artifact Realism Geometry Fidelity Blur Action Match Segmentation Fidelity Spatial Relation Segmentation Landmark Fidelity Identity Preservation Pose Taxonomy leaf neural radiance field generative 2d point cloud analysis segmentation detection reconstruction 3d novel view synthesis human pose mesh generative 3d human avatar scene understanding depth estimation motion generation gaussian splatting hand centric optical flow stereo image restoration tracking face centric other slam localization Task type 19.09.010.47.65.44.72.91.32.99.93.82.85.32.20.91.81.00.01.60.0 9.530.27.32.91.41.84.83.25.92.75.40.00.73.41.11.40.00.02.30.2 20.17.89.912.810.29.15.23.71.80.01.01.01.80.36.01.31.00.50.00.3 0.011.51.96.71.412.013.91.01.91.41.94.81.01.49.12.910.02.90.00.0 25.02.710.919.113.72.14.66.31.10.61.13.61.10.00.80.00.60.80.00.6 17.35.110.52.03.46.26.01.73.74.54.82.36.02.00.04.30.60.64.50.0 6.58.99.74.80.024.20.00.01.60.00.00.00.03.20.04.80.07.30.012.1 10.122.93.57.96.64.00.017.25.71.37.50.00.01.80.00.90.00.00.00.0 23.22.820.411.33.52.14.90.01.40.70.70.00.01.40.00.70.00.02.81.4 0.045.81.40.00.05.60.00.00.00.00.08.30.00.09.70.016.70.00.01.4 23.86.924.66.25.41.55.412.30.01.50.04.60.00.00.00.00.00.00.00.0 0.07.02.30.00.04.70.00.00.016.30.018.60.018.60.09.30.04.70.04.7 22.12.116.88.49.53.20.010.50.03.23.20.05.33.20.00.02.10.00.02.1 12.56.90.05.64.26.94.20.05.60.00.015.30.00.00.00.00.00.00.018.1 23.90.026.10.06.50.015.213.08.72.20.00.00.00.00.04.30.00.00.00.0 12.30.015.10.06.89.61.42.715.16.80.00.01.40.00.00.00.00.04.10.0 4.34.30.08.715.24.30.08.76.50.00.00.00.08.70.02.20.021.70.04.3 24.02.717.313.30.04.05.34.06.70.00.00.00.00.00.00.00.010.70.00.0 0.036.119.40.00.013.90.00.05.60.00.00.00.02.80.00.05.68.30.00.0 9.82.035.35.90.00.00.00.05.90.00.011.80.03.90.00.00.013.70.00.0 Share of each task type's claim mass per taxonomy leaf (20 most frequent leaves shown) 0 10 20 30 40 % of task type's total claim mass Figure 15: Task-type×leaf matrix. Each cell states a leaf’s share of the task type’s total claim mass (row-normalised over all 51 leaves); only the top-20 leaves by overall count are displayed, so rows sum to less than 100%. Deep red cells indicate a field’s concentrated reliance on a narrow leaf; uniformly pale rows (e.g., segmentation_detection) indicate broad evaluation vocabulary. Reconstruction 3D papers concentrate 57.8% of claim mass in three Object Form leaves (Detail, Completeness, Surface); generative 2D papers place 30.2% on Semantic Match alone. This matrix is the source of the per-field top-K leaf rankings used in the selective comparison analysis (Section 3.5). neural radiance field generative 2d point cloud analysis segmentation detection reconstruction 3d novel view synthesis human pose mesh generative 3d human avatar scene understanding depth estimation motion generation gaussian splatting hand centric optical flow stereo image restoration tracking face centric other slam localization 0 20 40 60 80 100 % of data points Primary-axis distribution per task type Object Form Reference Fidelity Prompt Match Image Appearance Relation Scene Layout Figure 16: Primary-axis distribution per task type (row-normalised %). generative_2d and scene_understanding are Prompt Match–dominated; reconstruction_3d and gaussian_splatting are Object Form–dominated; novel_view_synthesis shows the most balanced distribution across the six axes. Field specialisation score.Figure 17 shows the Shannon entropy of each field’s leaf-count distribu- tion. Lower entropy indicates evaluation concentrated on a narrow set of leaves. scene_understanding (11 active leaves) and optical_flow_stereo (8 leaves) are the most specialised; novel_view_synthesis (28 leaves) and neural_radiance_field (33 leaves) are the most diverse. Low-entropy fields have 27 a dominant evaluation criterion, making the omission of secondary criteria in individual papers structurally more consequential for selective comparison analysis. 0.00.51.01.52.02.53.03.54.0 Shannon entropy of leaf distribution (bits) scene understanding other optical flow stereo slam localization depth estimation face centric motion generation human pose mesh tracking reconstruction 3d image restoration hand centric gaussian splatting human avatar generative 3d point cloud analysis segmentation detection generative 2d neural radiance field novel view synthesis 11 leaves 8 leaves 8 leaves 10 leaves 13 leaves 11 leaves 11 leaves 13 leaves 13 leaves 26 leaves 14 leaves 13 leaves 16 leaves 21 leaves 20 leaves 24 leaves 24 leaves 33 leaves 33 leaves 28 leaves Field specialisation: lower entropy = narrower leaf focus Figure 17: Shannon entropy of leaf-count distributions per task type (bits). Lower entropy indicates that a field concentrates evaluations on a narrow leaf set. Bars are annotated with the number of active leaves. Top-5 leaves per task type. Figure 18 shows the top-5 leaves by data-point count for each of the 20 task types. The across-panel diversity visually confirms field-specific evaluation vocabularies: human_pose_mesh papers concentrate on Correspondence and Anatomy; neural_radiance_field papers spread across Detail, Reconstruction Fidelity, and Global Artifact; generative_3d papers favour Semantic Match and Shape. E.2 Selective Comparison: Sampling Details The 200-paper stratified sample for the model-based selective comparison assessment (Section 3.5) is constructed as follows. Among 789 selective-comparison-eligible papers (K=10, at least one untested X-dim, at least two candidate methods available for comparison), 200 slots are allocated proportionally to the eligible-paper count per task type (minimum one per task type). Within each task type, papers are ranked jointly by (i) number of untested X-dims (descending) and (i) total comparison-figure count (descending). Papers with higher X-dim counts and more comparison figures are prioritised to maximise the probability of detecting selective comparison hits within a fixed computational budget. The full distribution of untested X-dim counts in the eligible pool is as follows:n x =10: 95 papers (12.0%);n x =9: 465 (58.9%);n x =8: 163 (20.7%);n x =7: 56 (7.1%);n x =6: 10 (1.3%). The mean is 8.73 untested dims out of 10. The stratified sample preserves this distribution approximately and spans all 20 task types. 28 020406080100120 Completeness Semantic Match Global Artifact Reconstruction Fidelity Detail neural radiance field 020406080100120 Realism Texture Reconstruction Fidelity Detail Semantic Match generative 2d 020406080 Correspondence Reconstruction Fidelity Surface Completeness Detail point cloud analysis 051015202530 Segmentation Fidelity Segmentation Semantic Match Correspondence Boundary segmentation detection 020406080100120 Shape Reconstruction Fidelity Surface Completeness Detail reconstruction 3d 0102030405060 Boundary Blur Correspondence Reconstruction Fidelity Detail novel view synthesis 051015202530 Semantic Match Reconstruction Fidelity Pose Anatomy Correspondence human pose mesh 01020304050 Realism Completeness Detail Shape Semantic Match generative 3d 051015202530 Anatomy Lighting Completeness Reconstruction Fidelity Detail human avatar 051015202530 Physical Plausibility Geometry Fidelity Segmentation Fidelity Segmentation Semantic Match scene understanding 051015202530 Completeness Semantic Match Shape Detail Reconstruction Fidelity depth estimation 02468 Spatial Relation Interaction Global Artifact Geometry Fidelity Action Match motion generation 05101520 Completeness Surface Shape Reconstruction Fidelity Detail gaussian splatting 024681012 Correspondence Attribute Match Detail Geometry Fidelity Pose hand centric 024681012 Texture Shape Boundary Detail Reconstruction Fidelity optical flow stereo 0246810 Correspondence Detail Texture Reconstruction Fidelity Noise image restoration 0246810 Completeness Anatomy Action Match Surface Landmark Fidelity tracking 0.02.55.07.510.012.515.017.5 Attribute Match Landmark Fidelity Completeness Reconstruction Fidelity Detail face centric 024681012 Noise Landmark Fidelity Correspondence Reconstruction Fidelity Semantic Match other 0.02.55.07.510.012.515.017.5 Attribute Match Detail Geometry Fidelity Landmark Fidelity Reconstruction Fidelity slam localization Top-5 taxonomy leaves per task type (number of data points) Figure 18: Top-5 taxonomy leaves per task type by data-point count (one panel per task type). The distinct leaf profiles across panels confirm that each field applies a recognisably different evaluation vocabulary, motivating the field-signature approach to selective comparison detection. 29