Paper deep dive

Program-Verified Self-Evolution for Vision-Language Models

Ahmed Heakl, Sungik Choi, Moontae Lee, Salman Khan

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 83

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Self-evolving vision-language models train on questions they generate from unlabeled images. Since these questions have no gold answers, prior methods label them by majority vote over sampled answers or by a model judge. In a human evaluation, we find that 24\% of majority-vote labels and 18\% of model-judge labels produced during self-evolution are wrong. To address this problem, we present Verifiable QA Generation for Self-Evolving Models (VQS), which changes how the model judges answers. Instead of voting on an answer, the model parses each image into a structured record, such as a scene graph, a chart table, or a diagram graph. Fixed programs then write a question from the record and compute its answer. The model still acts as a visual checker, but it only confirms the individual facts the program reads, one short claim at a time. These claim-level checks select the parser's training targets, so the parser also improves without labels. Human raters find 94\% of VQS answers correct, against 76\% for majority voting. Across ten benchmarks, VQS improves Qwen3-VL by up to 3.18 points at the 2B, 4B, and 8B scales and outperforms the strongest self-evolving baseline at each. Gains keep growing over three training rounds, reaching 3.84 points at 2B. Code is released at this https URL

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

79,886 characters extracted from source content.

Expand or collapse full text

Under Submission PROGRAM-VERIFIED SELF-EVOLUTION FOR VISION- LANGUAGE MODELS Ahmed Heakl 1,2 , Sungik Choi 1 , Moontae Lee 1,4 , Salman Khan 2,3 1 LG AI Research, 2 MBZUAI, 3 Australian National University, 4 University of Illinois at Chicago https://github.com/ahmedheakl/VQS ABSTRACT Self-evolving vision-language models train on questions they generate from un- labeled images. Since these questions have no gold answers, prior methods label them by majority vote over sampled answers or by a model judge. In a human eval- uation, we find that 24% of majority-vote labels and 18% of model-judge labels produced during self-evolution are wrong. To address this problem, we present Verifiable QA Generation for Self-Evolving Models (VQS), which changes how the model judges answers. Instead of voting on an answer, the model parses each image into a structured record, such as a scene graph, a chart table, or a diagram graph. Fixed programs then write a question from the record and compute its an- swer. The model still acts as a visual checker, but it only confirms the individual facts the program reads, one short claim at a time. These claim-level checks select the parser’s training targets, so the parser also improves without labels. Human raters find 94% of VQS answers correct, against 76% for majority voting. Across ten benchmarks, VQS improves Qwen3-VL by up to 3.18 points at the 2B, 4B, and 8B scales and outperforms the strongest self-evolving baseline at each. Gains keep growing over three training rounds, reaching 3.84 points at 2B. 1INTRODUCTION GQA OK-VQA InfoVQAScienceQA MMMU MMBench EmbSpatial LogicVistaMMStar SEEDBench BaseVISEVQS (ours) Figure 1: VQS improves every bench- mark.Scores are for Qwen3-VL-2B. VISE loses on LogicVista and MMStar. A vision-language model can improve without human labels by writing the questions it trains on. Recent work does this by splitting one model into a proposer that asks and a solver that answers (He et al., 2026; Thawakar et al., 2025; Sunil et al., 2026; Huang et al., 2026). In each round, the proposer generates questions from unlabeled images, the solver answers them, and the solver is trained on the pairs that match a pseudo- label. Answer verification is the hardest step, since gold answers do not exist in this loop. Standard reinforcement learning with verifiable re- wards compares each answer with a gold label (Shao et al., 2024; Yu et al., 2026). However, self-generated questions have no gold label, so prior work falls back on proxy labels such as a majority vote over samples (Wei et al., 2026; Zuo et al., 2026; He et al., 2026), agreement among reasoning steps (Sunil et al., 2026), consistency under an image transformation (Venkatra- man et al., 2026), or a model judge (Chen et al., 2025; Wu et al., 2026; Yuan et al., 2024). Prior works use proxies that accept many wrong answers. First, in our human evaluation of 500 gen- erated questions (Table 2), 24% of majority-vote labels and 18% of model-judge labels are wrong. Second, the errors also grow as the loop runs. VisPlay’s pseudo-label accuracy falls from 72% to 61% over three rounds (He et al., 2026), and R-Zero’s falls from 79% to 63% (Huang et al., 2026). Third, prior work reaches the same conclusion independently, showing that majority-vote self-reward eventually favors confident but incorrect answers and collapses (Shafayat et al., 2025). 1 arXiv:2609.33855v1 [cs.CV] 27 Sep 2026 Under Submission (b) VQS (Program-Verified Self-Evolution) (a) Prior Self-Play the reward comes from the solver's own answers Image Questioner ? How many mugs are shown? Solver 5 answers ·2 ·1 ·2 ·2 ·1 Answer distribution 60% 40% Majority 2 used as label True 1 discarded Policy Update FrozenTrainable QA Generation Schema- Constrained Parser 1 Structured parse z Pear Mug left of left of Book Fixed Template Count: answer choose randomly obj answer Is there a in the image? obj Filter: Q: How many of Mugs in the image? A: 1 How many mugs are shown? ? Paraphrase Question Verification and Filtration Visual checker Image + Question 2 Blind Gate 1 0 2 NA 4 answers 0.25 How many mugs are shown? KeepDrop Threshold 1/2 0.25 00.5 1 p(Ans) Extract Claims 1 2 Is there only "1" mug shown? Are there any mugs in the image? Solver Training with GRPO3 Trained Solver VLM π θ Q Solver VLM π θ rollouts 8 answer y · 1 · 2 · 1 · 1 · 2 · 1 · 2 · 1 reward r 1.0 0 1.0 1.0 0 1.0 0 1.0 Accepted examples Image + Question + Answer GRPO Update π θ (1) ↑ π θ (2) ↓ Update Questioner reward: Uncertainty Entropy 0.67 agreescattered ? 1 > Wrong answer "2" is paid more 2 0 1 Solver reward: match majority [ [ Number of Correct answers Too hardKeep Difficulty band 8 rollouts + Question Accepted examples 23 4 56 708 Too easy Figure 2: Majority-vote labels vs. computed answers. (a) In prior self-play, the solver’s majority answer becomes the label. Here three of five answers say 2 mugs, so the wrong answer is rewarded and the correct answer “1” is discarded. (b) VQS computes the answer instead. (1) A schema- constrained parser turns the image into a structured parse. A fixed template reads the parse (filter mugs, then count) to write a question and compute its answer “1” and the parser then paraphrases the question. (2) A visual checker verifies each claim behind the answer, a blind gate drops questions answerable without the image, and a difficulty band keeps those the solver gets right in some but not all of 8 rollouts. (3) GRPO rewards each rollout by exact match against the computed answer, so training reinforces the correct answer “1” and suppresses the wrong one “2”. We present VQS (Verifiable QA Generation for Self-Evolving Models), a framework that computes answers from verified visual facts rather than estimating them from model outputs. As illustrated in Fig. 2, a parser first converts an input image into a navigable structure, such as a scene graph for a photograph, a structured table for a chart, or a set of nodes and directed edges for a diagram. Fixed templates (i.e., deterministic programs) then traverse these structures to generate questions alongside their computed answers. To catch wrong answers, a checker verifies the atomic visual facts behind an answer one at a time. We then train the solver on these verified QA pairs. These claim-level checks select the parser’s training targets by choosing the most precise of K parses, so the parser also improves without labels. A single model is used for the parser, solver, and checker. Our contributions are: (1) VQS converts photographs, charts, diagrams, and infographics into struc- tured records, then generates questions and computes answers through deterministic programs. (2) A label-free algorithm improves the parser by selecting its training set from candidate parses with claim-level visual verification. (3) VQS improves the average over ten benchmarks by up to 3.18 points (Fig. 1) at the 2B, 4B, and 8B scales and beats the strongest self-evolving method at ev- ery scale. (4) VQS keeps improving over three training cycles on the same unlabeled images by generating new QA pairs from templates each cycle, raising the 2B gain from 3.18 to 3.84 points. 2RELATED WORK Self-evolving vision-language models. R-Zero (Huang et al., 2026) uses a questioner and a solver trained against each other. VisPlay (He et al., 2026) trains a questioner with a reward that peaks when the solver is maximally uncertain. EvoLMM (Thawakar et al., 2025) rewards the solver by how often its answer is the majority among five samples. iReasoner (Sunil et al., 2026) adds agreement 2 Under Submission 1 Sample structured parses Check atomic claims Unlabeled image Parser VLM z₁z₂ z₃ z₄ Unverified right of Visual checker Image + one claim Example: z₂ All 4 candidates Attributes Mug: blue Parser Training + decoding Fields Shown z₂ Mug: blue Pear: yellow Book: blue left of Object presence Mug Pear Book Schema-constrained 2 Relations Mug right of book Pear left of book Pear: yellow Book: blue 3Verified-claim fraction Mean 0.75 p(z₁) +0.25 1.00 8/8 p(z₂) 0.75 6/8 p(z₃) 0.75 6/8 p(z₄) 0.50 4/8 Selected z₁ 00.51 Margin min 0.10 min 4 Thresholds Whole z₁ Whole z₁ Parse z₁ Margin over mean left of Mug red Pear yellow Book blue left of left of 8 claims 4 SFT Trained parser πφ , Pear yellow Book left of left of Mug red blue Book What's right to the red object? Solver-data Curation Templates Figure 3: Label-free parser training. (1) The parser samples K structured parses of an unlabeled image under a schema-constrained decoder. (2) A visual checker verifies each parse one claim at a time. In the example, z 2 wrongly calls the mug blue and places it to the right of the book, so it passes 6 of 8 claims. (3) Each parse is scored by its fraction of verified claims. We keep the best parse z 1 (8/8) only because it beats the mean of all four parses (0.75) and contains at least m=4 claims. (4) The kept parses train the parser with SFT. The trained parser then feeds the templates that write questions and compute their answers for the solver. between reasoning steps. Vision-Zero (Wang et al., 2025) turns the loop into a checkable game. VISE (Venkatraman et al., 2026) drops the two-role setup and rewards the model for predicting the same box under a known image transformation. Without gold answers, all of these methods take their reward from the model’s outputs, and this reward can become less accurate as training goes on. For example, VisPlay’s estimated pseudo-label accuracy falls from 72% to 61% over three rounds as its questioner gets stronger, and R-Zero’s falls from 79% to 63%. Programs that write questions. CLEVR (Johnson et al., 2017) attaches a functional program to every question and runs it on the scene graph it rendered from. GQA (Hudson & Manning, 2019) scales this to real photographs using human-annotated graphs. ProVision (Zhang et al., 2024) and SpatialVLM (Chen et al., 2024a) run generators over a model-produced parse at very large scale, but both produce fine-tuning data and neither verifies a generated claim. 3METHOD Below we describe how VQS builds the solver’s training data, trains the solver, and trains the parser. In each training cycle, the parser is trained first. Algorithm 1 gives the full procedure. Setup. The only training data is a set of unlabeled images X . We initialize two models from the same pretrained model: (1) the parser π φ , which maps an image to a structured description that we call a parse, and (2) the solver π θ , which answers questions about the image. The parser also serves as the checker and the text-only model described below (details in Section 4). A third component, the program libraryT , is a fixed set of templates written once by hand and never trained (See A.5). From images to questions.To build the solver’s training data, the parser first maps each image x to a parse z.Each parse must follow a schema S d(x) fixed per domain d(x) ∈ photo, chart, diagram, infographic, and we decode it with a constrained decoder so that it is al- ways well formed. A photo parse holds objects with names, attributes, boxes, and relations. A chart parse holds series of (category, printed value, numeric value) points and the axis ticks. A diagram 3 Under Submission Algorithm 1 VQS (one training cycle) Require: unlabeled imagesX , templatesT , pretrained weights π 0 1: φ,θ ← π 0 ▷ parser and solver start from the same model Train the parser (label-free best-of-K) 2: for x∈X do 3:sample z (1..K) ∼ π φ (·| x) under the schema 4:score each z (k) by its fraction of verified claims P(z (k) )▷ Eq. (5) 5:keep (x,z ⋆ ) if it beats the mean of all K parses by δ and has≥ m claims▷ Eq. (6) 6: end for 7: fine-tune φ on the kept targets with cross-entropy loss Build the solver training data 8: for x∈X do 9: z ← π φ (x), decoded greedily under the schema 10:for each applicable template t: (q,a,h)← t(z), then paraphrase q → ̃q 11:drop the question unless it passes the fact-check, the blind gate, and the difficulty band ▷ Eq. (1) 12: end for Train the solver 13: for each GRPO step do 14:draw n rollouts per question and score each against a▷ Eq. (3) 15:normalize rewards within each group and update θ with GRPO▷ Eq. (4) 16: end for 17: return π θ ▷ π θ can re-parse the images and start the next cycle parse holds nodes and directed arrows. An infographic parse holds entries with their text, number, and unit. Figure 4 shows examples, and Appendix A.4 gives the full schemas for each domain. Each applicable template t∈T then reads the parse z and returns a question q, its answer a, and its hop count h, which is the number of operations the template performs: (q,a,h) = t(z). Templates combine these operations: filter a set, read a field or attribute, count a set, take a maximum or minimum, follow a relation, rank, compare, do arithmetic, read a trend, count a node’s arrows, and turn a box into a coarse position. Finally, the parser rewrites q in natural wording, ̃q ∼ π φ (·| x,z,q), while the answer a stays the same. Which questions enter the training data. We keep a question only if it passes three checks. The fact-check verifies the facts behind the answer. The template asserts one claim per field it reads, giving a set of atomic claims C(t,z), and a binary checker v(c,x) ∈ 0, 1 answers yes or no to each claim. The blind gate removes questions that do not need the image. A text-only model π blind answers the question J times without the image, u j ∼ π blind (· | ̃q), and we drop it if more than a fraction λ of these answers are correct. The difficulty band removes questions that are too easy or too hard. The solver about to be trained answers the question n times, y i ∼ π θ (· | x, ̃q), and we drop it if the solver is always right or always wrong, since all rollouts then get the same correctness score and give almost no learning signal (Eq. (3)). Formally, we keep the question only if Q c∈C(t,z) v(c,x) = 1 | z fact-check , 1 J X j eq(u j ,a)≤ λ |z blind gate , 0 < ˆp < 1, ˆp = 1 n X i eq(y i ,a) |z difficulty band ,(1) where eq(y,a)∈0, 1 is exact match between an answer y and a. Solver Training. The reward is an exact match with the computed answer, plus a small format term, r(y;a) = 0.9· eq(y,a) + 0.1· fmt(y),(2) where fmt(y) ∈ 0, 1 checks that y follows the required answer format. We train the solver with GRPO (Shao et al., 2024) rather than SFT. GRPO learns on-policy, while SFT on template-written text may pull the model away from its distribution. For each question, GRPO normalizes the rewards of the n rollouts within their group, A i = r i − mean(r 1 ...r n ) std(r 1 ...r n ) + ε , r i = r(y i ;a),(3) 4 Under Submission (a) Natural image Parser VLM Object–relation graph holding on on on white bowl oatmeal poached egg toast plate white banana sliced 6 objects · 4 relations (b) Chart 0 20 40 60 80 million € 7.11 2007 12.5 2008 8.26 2009 3.63 2010 11 2011 65.09 2012 1.6 2013 1.44 2014 21.5 2015 10.76 2016 Parser VLM Series → values 2011 "11" 2012 "65.09" 2013 "1.6" total investment million euros 3 of 10 values shown (c) Diagram Parser VLM Directed graph ADULT EGGS LARVA PUPA 4 nodes 4 edges (d) Infographic Parser VLM Entry → field records Humans "160" 160 psi Alligator "2,000" 2000 psi Jaguar "1,500" 1500 psi entries[ ] text / number + unit 3 of 12 entries shown Figure 4: Examples of structured scene parsing. The parser converts natural images, charts, diagrams, and infographics into domain-specific JSON records that preserve spatial relations, chart values and ticks, graph structure, and verbatim strings for deterministic QA construction. and updates the solver with a clipped objective and a KL penalty against a frozen reference π ref , L solver (θ) =−E " min π θ (y i ) π θ old (y i ) A i , clip π θ (y i ) π θ old (y i ) , 1− ε lo , 1 + ε hi A i # + β KL[π θ ∥π ref ], (4) where ε lo and ε hi set the clipping range and β weights the KL penalty. Parser Training. The parser is trained first, before the solver data is built, and without labels (Fig. 3). For each image x, we sample K parses and score each by the fraction of its claimsF(z), one per field, that the checker confirms: z (1..K) ∼ π φ (·| x), P(z) = 1 |F(z)| X c∈F(z) v(c,x), z ⋆ = arg max k P(z (k) ).(5) We keep the best parse z ⋆ only when it clearly beats the average of all K parses, so that each kept target carries a real learning signal. If all K parses score the same, the chosen parse is no better than what the parser already produces. We also require the best parse to have at least m claims, since small parses usually get inflated verification scores: keep (x,z ⋆ ) iff P(z ⋆ )− 1 K X k P(z (k) )≥ δ and |F(z ⋆ )|≥ m.(6) Finally, we fine-tune the parser on the kept targets (x,z ⋆ ) with the standard cross-entropy loss. 4EXPERIMENTAL SETUP Models and data. We train Qwen3-VL-2B, 4B and 8B Instruct (Bai et al., 2025). We take the training images from Venkatraman et al. (2026) and discard their questions, answers, and labels, so no human annotation is used at any stage. Benchmarks. We use lmms-eval (Zhang et al., 2025) to evaluate on 10 benchmarks: GQA (Hudson & Manning, 2019), OK-VQA (Marino et al., 2019), InfoVQA (Mathew et al., 2022), ScienceQA (Lu et al., 2022), MMMU (Yue et al., 2024), MMBench (Liu et al., 2024), EmbSpatial (Du et al., 2024), LogicVista (Xiao et al., 2024), MMStar (Chen et al., 2024b), SEEDBench (Li et al., 2024). We use the default hyperparameters for each benchmark. Implementation Details. We used 8xAMD MI210 cards for training and evaluation. Every rein- forcement learning run uses GRPO via EasyR1 (Zheng et al., 2025) and the parser is fine-tuned with LLaMA-Factory (Zheng et al., 2024). For parsing, we sample each image K=4 parses at tempera- ture 1.0 and top-p 0.95, up to 2048 tokens, under a per-domain JSON schema the decoder enforces, and score each by the fraction of its claims the checker confirms. We keep the best parse only if 5 Under Submission Table 1: Main results across three scales. Every model trains on unlabeled images only, with no human question, answer or gold label at any stage. Subscripts give the change against that backbone’s own base, in green when positive and red when negative. VQS gives the largest average gain at all three scales, +3.18 at 2B, +2.32 at 4B and +2.43 at 8B, against +1.01, +1.04 and +0.66 for the strongest published method. InfoVQA stands for infographicsVQA, SQA for ScienceQA, MMB for MMBench, ESB for EmbSpatial, and LogicV for LogicVista. MethodGQAOK-VQA InfoVQASQAMMMUMMBESBLogicVMMStarSEEDAvg. Qwen3-VL-2B-Instruct Base58.2540.7669.0279.4238.9274.4868.5435.0455.6272.1659.22 VisPlay58.65+0.40 41.15+0.39 69.96+0.94 80.74+1.32 39.27+0.35 74.52+0.04 68.56+0.02 34.93-0.11 55.37-0.25 71.62-0.54 59.48+0.26 Vision-Zero 58.98+0.73 41.39+0.63 70.93+1.91 81.96+2.54 39.58+0.66 75.07+0.59 69.72+1.18 35.28+0.24 55.58-0.04 71.53-0.63 60.00+0.78 EvoLMM59.01+0.76 38.03-2.73 70.69+1.67 83.01+3.59 39.08+0.16 74.62+0.14 69.32+0.78 34.99-0.05 55.50-0.12 71.17-0.99 59.54+0.32 iReasoner59.13+0.88 38.13-2.63 70.82+1.80 83.12+3.70 39.11+0.19 74.75+0.27 69.67+1.13 35.09+0.05 55.59-0.03 71.25-0.91 59.67+0.45 VISE59.41+1.16 41.24+0.48 71.43+2.41 83.61+4.19 40.67+1.75 76.72+2.24 70.14+1.60 31.92-3.12 55.02-0.60 72.18+0.02 60.23+1.01 VQS (ours)59.52+1.2742.60+1.8472.01+2.9986.81+7.3946.78+7.8678.26+3.7871.54+3.0036.83+1.7957.04+1.4272.61+0.4562.40+3.18 Qwen3-VL-4B-Instruct Base60.3247.8677.7387.5145.1783.4276.7949.7862.6775.7166.70 VisPlay60.06-0.26 48.04+0.18 77.96+0.23 87.30-0.21 45.42+0.25 83.47+0.05 76.11-0.68 49.34-0.44 62.13-0.54 74.83-0.88 66.47-0.23 Vision-Zero 61.30+0.98 48.41+0.55 78.28+0.55 88.46+0.95 46.27+1.10 84.83+1.41 77.02+0.23 49.11-0.67 62.45-0.22 75.01-0.70 67.19+0.49 EvoLMM61.20+0.88 46.72-1.14 78.58+0.85 88.46+0.95 46.02+0.85 84.21+0.79 76.94+0.15 49.57-0.21 62.37-0.30 74.77-0.94 66.88+0.19 iReasoner61.32+1.00 46.89-0.97 78.76+1.03 88.62+1.11 46.13+0.96 84.34+0.92 76.98+0.19 49.65-0.13 62.46-0.21 74.84-0.87 67.00+0.30 VISE61.82+1.50 48.81+0.95 81.45+3.72 90.04+2.53 48.89+3.72 84.51+1.09 77.46+0.67 47.66-2.12 61.73-0.94 75.00-0.71 67.74+1.04 VQS (ours)62.86+2.5448.19+0.3380.23+2.5093.11+5.6054.78+9.6183.93+0.5177.80+1.0150.10+0.3263.29+0.6275.82+0.1169.01+2.32 Qwen3-VL-8B-Instruct Base61.5449.8481.2390.8850.1284.7177.6652.2364.9674.9668.81 VisPlay61.66+0.12 49.93+0.09 81.36+0.13 91.02+0.14 52.61+2.49 84.62-0.09 78.24+0.58 51.88-0.35 64.52-0.44 74.34-0.62 69.02+0.21 Vision-Zero 62.12+0.58 50.31+0.47 82.44+1.21 92.35+1.47 52.89+2.77 85.11+0.40 77.96+0.30 52.33+0.10 64.75-0.21 74.33-0.63 69.46+0.65 EvoLMM61.94+0.40 50.18+0.34 82.02+0.79 91.96+1.08 51.56+1.44 85.02+0.31 77.51-0.15 52.01-0.22 64.59-0.37 74.12-0.84 69.09+0.28 iReasoner62.02+0.48 50.24+0.40 82.15+0.92 92.11+1.23 51.83+1.71 85.09+0.38 77.82+0.16 52.09-0.14 64.67-0.29 74.18-0.78 69.22+0.41 VISE62.43+0.89 50.61+0.77 82.83+1.60 92.81+1.93 52.69+2.57 85.46+0.75 78.62+0.96 50.78-1.45 64.17-0.79 74.30-0.66 69.47+0.66 VQS (ours)61.97+0.4353.00+3.1683.13+1.9094.65+3.7757.56+7.4485.82+1.1178.49+0.8354.24+2.0168.14+3.1875.39+0.4371.24+2.43 it beats the mean of all K parses by δ=0.10 and asserts at least m=4 claims, which turned 12,000 images into 4,792 targets. The parser is then fully fine-tuned on those targets with the vision tower frozen, per-device batch 4 with 8 gradient accumulation steps, learning rate 1e−5, 2 epochs, 284 steps. At data-curation time it decodes greedily. The fact-checker v is the same as the parser an- swering yes or no to one claim at a time, greedy, with four new tokens. The blind gate π blind is the same as the parser with no image. It generates J=4 answers and drops a question if more than a fraction λ=0.5 of them are correct. The difficulty band runs the base solver at the training settings, 8 rollouts at temperature 1.0 with a 256-token budget, and keeps a question only when it is answered correctly at least once and not every time. We train the solver with a curriculum that sorts each template family’s questions by hop count h, from fewest to most (Table 6). 5RESULTS We answer five questions: (1) Does VQS beat prior self-evolving methods, and how do its gains change with model size? (2) How many generated questions pass the filters, and are their answers correct? (3) Does label-free training make the parser more accurate? (4) Which filters and questions help the solver most? (5) Does the model keep improving over more training cycles? Main Results. Table 1 compares VQS with five self-evolving methods on ten benchmarks at three Qwen3-VL scales. Existing consistency-based baselines using majority voting or sample agreement show relatively little improvement, as they depend significantly on the base model performance. For example, EvoLMM and iReasoner raise ScienceQA, where the 2B base is already strong, but lower OK-VQA, where the base is weak. VisPlay stays close to the base scores on most benchmarks, because its questioner is rewarded for questions the solver is least sure about, and a majority vote is least reliable on such questions. VQS gives the largest average gain at every scale (+3.18 over base at 2B, against +1.01 for the strongest baseline) and lifts all ten benchmarks at each. Its labels come from programs run on checked parses, not from the solver, so training can move the model toward answers it did not favour and rarely rewards a wrong one (Table 2). This helps most where the base is weak. At 2B, the majority-vote methods gain almost nothing on MMMU, while VQS gains more than twice as much as any baseline (+7.86 vs +1.75). MMMU and ScienceQA have the largest gains as they 6 Under Submission 04896 GRPO Step 0.6 0.7 0.8 Training Reward (b) reward over training 8B4B2B 010305070 Template Families 0 2 4 6 Gain Over Base   (c) more families help reasoning perception all 0 2 4 Gain Over Base 2B4B8B Backbone Size 0% 4% 8% Answers RL Changes (a) RL moves bigger models less changedgain Figure 5: What scale and coverage change. (a) Share of benchmark answers RL changes, mea- sured per sample against each model’s base. (b) Smoothed training reward. (c) 2B gain over base against the number of template families. ask the model to read a chart, table, or figure and combine several facts, exactly what the multi-step templates train (Table 5). VISE, the strongest baseline, lowers the scores on LogicVista and MMStar at every scale, while VQS raises both. Like MMMU and ScienceQA, both benchmarks contain many questions that require combining several facts. See A.3 for results on non-Qwen model families. Effect of model scale. Gains from VQS are largest at the smallest backbone and shrink with size, from +3.18 at 2B to +2.43 at 8B. All three models reach a training reward of about 0.80 (Fig. 5b), so the larger models learn the generated task as well as the 2B model, but less of what they learn transfers to the benchmarks. Since an average gain cannot distinguish a model that barely changed from one that changed many answers in opposite directions, we also compare each run with its base sample by sample and count the benchmark answers that change. This share falls from 7.82% at 2B to 5.58% at 4B and 4.93% at 8B (Fig. 5a), which matches the smaller gains of the larger models. Answer sourceCorrect (%) Kept (%) Templates + checker94.4%76.0 Majority vote76.4%92.0 Model judge82.2%90.8 Table 2: Human evaluation. Correct is accuracy among retained questions. Do the generated answers match the image? Since each answer is computed from the parse, a parsing error can still produce a wrong answer, so human raters check the generated answers against the images. We compare three answer sources on 500 shared questions, with 125 per visual domain and coverage across template families. VQS executes templates on checked parses. Majority vot- ing selects the most frequent of five solver answers and keeps it only if p(y majority ) > 0.3. A model judge se- lects an accepted answer from the same five candidates. Table 2 reports both correctness and reten- tion: VQS reaches 94.4% correctness while keeping 76% of questions, compared with 76.4% for majority voting and 82.2% for the model judge. Does paraphrasing change the question? The parser rewrites each template question in natural wording, but neither the checker nor the blind gate tests whether the rewrite still asks what the program computed. We check this on 437 questions from 70 template families. The final question still matches its program in 99.8% of cases at 2B, 100% at 4B, and 99.1% at 8B. All four 8B errors swap the two years in a yes/no comparison, so the stored answer becomes wrong. Parser target Parse Prec. Answer Acc. Solver Acc. Untrained80.886.858.0 Random82.191.061.5 Whole parse85.094.062.0 Per claim87.696.162.4 Table 3: Parser target selection. Parse precision, cor- rect answer accuracy, and solver accuracy (%). Parser training selection. We test how to pick the parser’s training target from its K samples without labels. We compare three rules against the untrained parser: a ran- dom parse, the parse a checker accepts as a whole with one yes or no, and the parse with the highest fraction of verified claims. We rate parse precision and answer accu- racy by hand on 150 samples. We also train a 2B solver on the questions built from each parser and report its average over the ten benchmarks. Table 3 shows that per-claim checking is best on all three measures. Over whole-parse checking, it raises parse precision from 85.0 to 87.6 and answer accuracy from 94.0 to 96.1. Solver accuracy follows the same order. Training the parser with any rule lifts it from 58.0 to at least 61.5, and the three rules differ by less than one point. Questions from the untrained parser even drop the solver below the 2B base (59.22, Table 1), likely because 13.2% of their answers are wrong. One reason per-claim checking works better is that a single yes 7 Under Submission 248 Count K 0 20 40 60 80 100 Rate (%) 86.5 87.6 88.1 Precision 31.2 39.9 45.8 Retained (a) Candidate parses 00.100.20 Margin δ 84.9 87.6 89.1 Precision 67.5 39.9 19.8 Retained (b) Selection margin 248 Count m 88.4 87.6 86.8 Precision 54.1 39.9 24.3 Retained (c) Minimum claims Figure 7: Parser target selection: quality and coverage. Each panel varies one setting around (K,δ,m) = (4, 0.10, 4). Parse precision is the percentage of correct visual claims after training. Images retained is the percentage of source images yielding a selected training target. or no cannot tell a parse with one error from a parse with five. Hence, when many samples tie, the retained parse is often chosen by chance. In contrast, the fraction of verified claims gives a graded score that separates them. Another reason is that a checker judges one short claim more reliably than a long structure (Min et al., 2023; Jing et al., 2024), in line with findings that checking each fact separately reduces errors (Dhuliawala et al., 2024). GQAChartQAAI2D-RST 0 25 50 75 100 Parsing F1 (%) 79.6 83.3 75.2 84.6 89.9 82.6 UntrainedTrained Figure 6: Parser Structured-F1 Score. Does label-free training improve the parser? Every VQS answer is computed from the parse, so a parsing error becomes a wrong answer. The parser is also trained on the checker’s verdicts, so it could learn to please the checker without becoming more accurate. We therefore evaluate the parser against human annotations. We mea- sure F1 against GQA scene graphs (Hudson & Manning, 2019), ChartQA tables (Masry et al., 2022), and AI2D- RST diagram annotations (Hiippala et al., 2021). Figure 6 compares F1 by domain. Label-free training improves F1 in all three domains, by 5.0 points on GQA, 6.6 on ChartQA, and 7.4 on AI2D-RST. The largest gain comes on diagrams, where the un- trained parser is weakest (75.2), although diagrams remain the hardest domain after training (82.6). Parser hyperparameter selection. We also vary the candidate count K, the selection margin δ, and the minimum claim count m (Fig. 7). Raising K or δ improves precision, but only slightly, so we choose a middle setting that keeps computation affordable. Raising m, however, slightly lowers precision. Parses with more claims can produce more questions, but longer parses are more likely to contain hallucinated content (Li et al., 2025). Base123 Training cycle 0 2 4 6 Gain over base +2.2 +2.6 +3.8 +5.9 Knowledge & Reasoning Perception All benchmarks General Figure 8: Gains across cycles. Average gain over base per benchmark category for Qwen3-VL-2B. Does VQS keep improving over several rounds? All results so far (Table 1) come from a single round, or train- ing cycle, in which the parser and the solver are each trained once. Here we test whether more cycles help the solver. At the start of each new cycle, we merge the LoRA weights from the previous solver cycle, generate new QA pairs from our templates on the same images, and com- bine them with the QA pairs from earlier cycles. We then sample 8 answers to each question, and compute s as the fraction of these answers that are correct. We weight each question by w(s) = exp(−((s− 0.5)/0.15) 2 ), give zero weight to questions that are always or never solved, and resample 8K training items with these weights. As a re- sult, questions solved about half the time appear most of- ten. Finally, we train the model from the previous cycle on these items for 96 GRPO steps, with the same settings as all other runs. The average gain over base grows with every cycle (Fig. 8), from 3.18 points after cycle 1 to 3.41 and 3.84 after cycles 2 and 3. The third cycle adds more than the second (0.43 vs 0.23 points), so the curve has not flattened. As with a single cycle, knowledge and reasoning benchmarks gain the most, 5.94 points after three cycles, led by ScienceQA (+8.53) and MMMU (+8.44). Perception and general benchmarks gain 2.62 and 2.24 points. Nine of the ten benchmarks improve in every cycle, and the only exception, MMStar, peaks after cycle 1 but still ends 0.93 points above base. 8 Under Submission PipelineSolver Acc. Parser Prec. Untrained parser58.080.8 Full VQS62.487.6 ✗ constrained decoding61.985.3 ✗ fact-checker60.987.6 ✗ blind gate61.887.6 ✗ difficulty band62.187.6 Table 4: Component ablations at 2B. Which filters improve solver accuracy? Table 4 re- moves one component at a time at 2B. The filters act after parsing, so removing one changes solver accuracy but not parser precision. Removing parser training causes the largest drop, lowering parser precision from 87.6 to 80.8 and solver accuracy from 62.4 to 58.0. Removing the fact-checker causes the second-largest drop, lower- ing solver accuracy to 60.9 while parser precision stays at 87.6, so unchecked answers alone cost 1.5 points. Each remaining component costs 0.3 to 0.6 points. Rank FamilyGain ▲ 1Chart differences7.4 ▲ 2Diagram connectivity5.8 ▼ 1Object existence-0.2 ▼ 2Attribute lookup0.4 Table 5: Per-family gains. The two largest and two smallest. Which families gain the most? We measure the solver’s gain from each template family, and Table 5 lists the two families with the largest gains and the two with the smallest. The largest gains come from families that combine several reads, such as subtracting two chart values or following diagram arrows. Families that need a single read, such as object existence and attribute lookup, barely change, likely because the base model already answers them reliably. This matches the main results, where reasoning-heavy benchmarks such as MMMU and ScienceQA gain the most. Lastly, Fig. 5c shows that adding template families keeps raising the gain over base. The average gain grows from 1.9 points with 10 families to 3.18 with all 70, and it is still rising. Reasoning benchmarks gain more than perception benchmarks at every family count, reaching 4.7 against 2.4 points. ComparisonSettingSolver acc. Family difficulty Easy60.8 Middle62.4 Hard61.1 GeneratorFixed templates62.4 Generated programs61.4 CurriculumShuffled61.3 Global hops61.7 Within-family hops62.4 Difficulty bandInitial selection62.4 Refreshed selection62.3 Table 6: Question design at 2B. Each block changes one choice at an equal training budget. Solver accuracy averages the 10 benchmarks in Table 1. Bold marks the VQS default. Which questions are most useful for training? Beyond a correct answer, a question helps the solver only if it needs the image and the cur- rent solver answers it correctly sometimes, but not always. We test four design choices at equal training budgets (Table 6).For difficulty, we split template families into easy, middle, and hard thirds by the base solver’s accuracy and train on each third alone. For the generator, we replace fixed templates with model-written questions and programs on the same parses, following PROVE (Prabhu et al., 2025). Both generators use the same checkers, and human raters check whether each question asks what its program computes. For the curriculum, we compare shuffled order with ordering by hop count, across all questions or within each family. For the difficulty band, we compare selecting questions once with the base solver against reselecting them with the current solver during training. Middle families work best (62.4), ahead of hard (61.1) and easy (60.8), which matches prior findings that questions of interme- diate difficulty give the most learning signal (Bae et al., 2026; Gao et al., 2026). Generated programs lose 1.0 point because only 85.3% of their questions match their program. Ordering by hop count helps, and ordering within each family works best, raising accuracy from 61.3 (shuffled) to 62.4. Reselecting during training brings no gain (62.3), so one selection with the base solver is enough. 6CONCLUSION In this work, we show that self-evolving vision-language models can generate hallucinated questions and answers that still pass consistency-based verifiers. In our human evaluation, 24% of majority- vote labels and 18% of model-judge labels are wrong. To enable principled generation and verifica- tion, we propose VQS, a template-based self-evolving framework that computes answers with fixed programs over a structured parse of each image and checks every fact the programs read, lowering this error to 6%. The same checks also train the parser without labels. VQS applies across four image domains, and across ten benchmarks it improves Qwen3-VL by up to 3.18 points, beating the strongest baseline at every scale and reaching 3.84 points at 2B after three cycles. 9 Under Submission REFERENCES Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Vol- ume 1: Long Papers), p. 700–719, 2026. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, p. 14455–14465, 2024a. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056–27087, 2024b. Yixing Chen, Yiding Wang, Siqi Zhu, Haofei Yu, Tao Feng, Muhan Zhang, Mostofa Patwary, and Jiaxuan You. Multi-agent evolve: Llm self-improve through co-evolution. arXiv preprint arXiv:2510.23595, 2025. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models. In Findings of the association for computational linguistics: ACL 2024, p. 3563–3578, 2024. Mengfei Du, Binhao Wu, Zejun Li, Xuan-Jing Huang, and Zhongyu Wei. Embspatial-bench: Bench- marking spatial understanding for embodied tasks with large vision-language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 346–355, 2024. Zhaolin Gao, Joongwon Kim, Wen Sun, Thorsten Joachims, Sid Wang, Richard Yuanzhe Pang, and Liang Tan. Prompt curriculum learning for efficient llm post-training. In International Conference on Learning Representations, volume 2026, p. 58614–58652, 2026. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Yicheng He, Chengsong Huang, Zongxia Li, Jiaxin Huang, and Yonghui Yang. Visplay: Self- evolving vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 26274–26284, 2026. Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Sera- fina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. Ai2d-rst: a multimodal corpus of 1000 primary school science diagrams: T. hiippala et al. Language Resources and Evaluation, 55(3):661–688, 2021. Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Ji- axin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning llm from zero data. In International Conference on Learning Representations, volume 2026, p. 130770–130790, 2026. Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 6693–6702. IEEE, 2019. Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du. Faithscore: Fine-grained evaluations of hallucinations in large vision-language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 5042–5063, 2024. 10 Under Submission Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2901–2910, 2017. Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 13299–13308. IEEE, 2024. Zhuowei Li, Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang, and Dimitris N Metaxas. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. arXiv preprint arXiv:2502.03628, 2025. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, p. 216–233. Springer, 2024. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in neural information processing systems, 35:2507–2521, 2022. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 3190–3199. IEEE, 2019. Ahmed Masry, Jia Qing Tan, Shafiq Joty, Enamul Hoque, et al. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, p. 2263–2279, 2022. Minesh Mathew, Viraj Bagal, Rub ` en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 2582–2591. IEEE, 2022. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual pre- cision in long form text generation. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 12076–12100, 2023. Viraj Prabhu, Senthil Purushwalkam, An Yan, Caiming Xiong, and Ran Xu. Trust but verify: Pro- grammatic vlm evaluation in the wild. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 3258–3267. IEEE, 2025. Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train? arXiv preprint arXiv:2505.21444, 2025. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Meghana Sunil, Manikandarajan Venmathimaran, and Muthu Subash Kavitha.ireasoner: Trajectory-aware intrinsic reasoning supervision for self-evolving large multimodal models. In Findings of the Association for Computational Linguistics: ACL 2026, p. 29361–29372, 2026. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram ́ e, Morgane Rivi ` ere, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. Omkar Thawakar, Shravan Venkatraman, Ritesh Thawkar, Abdelrahman Shaker, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, and Fahad Khan. Evolmm: Self-evolving large multi- modal models with continuous rewards. arXiv preprint arXiv:2511.16672, 2025. 11 Under Submission Shravan Venkatraman, Ritesh Thawkar, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Salman Khan, and Fahad Shahbaz Khan. Paying more attention to visual tokens in self-evolving large multimodal models. In European Conference on Computer Vision, p. 41–59. Springer, 2026. Qinsi Wang, Bo Liu, Tianyi Zhou, Jing Shi, Yueqian Lin, Yiran Chen, Hai Helen Li, Kun Wan, and Wentian Zhao. Vision-zero: Scalable vlm self-improvement via strategic gamified self-play. arXiv preprint arXiv:2509.25541, 2025. Lai Wei, Yuting Li, Chen Wang, Yue Wang, Linghe Kong, Weiran Huang, and Lichao Sun. First sft, second rl, third upt: Continual improving multi-modal llm reasoning via unsupervised post- training. Advances in Neural Information Processing Systems, 38:62293–62318, 2026. Zhengxian Wu, Kai Shi, Chuanrui Zhang, Zirui Liao, Jun Yang, Ni Yang, Qiuying Peng, Luyuan Zhang, Hangrui Xu, Tianhuang Su, et al. When models judge themselves: Unsupervised self- evolution for multimodal reasoning. arXiv preprint arXiv:2603.21289, 2026. Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973, 2024. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 9556–9567, 2024. Jieyu Zhang, Le Xue, Linxin Song, Jun Wang, Weikai Huang, Manli Shu, An Yan, Zixian Ma, Juan Carlos Niebles, Silvio Savarese, et al. Provision: Programmatically scaling vision-centric instruction data for multimodal language models. arXiv preprint arXiv:2412.07012, 2024. Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evalua- tion of large multimodal models. In Findings of the Association for Computational Linguistics: NAACL 2025, p. 881–916, 2025. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations), p. 400–410, 2024. Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, Yuwen Xiong, and Richong Zhang. EasyR1: An efficient, scalable, multi-modality RL training framework. https: //github.com/hiyouga/EasyR1, 2025. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Xinwei Long, Ermo Hua, Biqing Qi, et al. Ttrl: Test-time reinforcement learning. Advances in Neural Information Processing Systems, 38:131459–131483, 2026. 12 Under Submission AAPPENDIX Contents Training and evaluation settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Benchmark categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 More Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Is the checker independent of the parser? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Does VQS transfer to other model families? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Parsing an image into a structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Rewriting the templated question . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 The blind gate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Training-time prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Model-written programs (ablation) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Baseline prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Qualitative examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Traced questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 The program behind a template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 VQS, VISE and the base model on the same benchmark questions . . . . . . . . . . . . . . . . . . . . . . . . 23 A.1TRAINING AND EVALUATION SETTINGS We train the solver with GRPO (Shao et al., 2024) on top of a LoRA adapter of rank 64, applied to all linear layers with the vision modules excluded, so the vision tower stays frozen and no visual parameters are trained. We use AdamW at a constant learning rate of 1e−5, a low-variance KL penalty against a frozen reference with β = 0.01, and clip ratios of 0.2 and 0.3 with dual clipping at 3.0. For each prompt we draw 8 rollouts at temperature 1.0 with a 256-token response budget and cap training images at 1,003,520 pixels. We collect 256 prompts per rollout batch, update on a global batch of 128, and train for 96 steps. The three scales share these settings, except that the per-device micro-batch is 16 at 2B and 4 at 8B. Table 7 lists the full settings. A.2BENCHMARK CATEGORIES We group the ten benchmarks into three categories by the main skill each one tests (Table 8). Per- ception benchmarks ask about what is visible in the image, such as objects, relations, printed text, and spatial layout. Knowledge & Reasoning benchmarks also need something the image does not supply, such as outside facts, subject knowledge, or logical reasoning. General benchmarks mix both kinds of questions, so we keep them as a separate group. A category’s score is the unweighted mean of its benchmarks, and the overall score is the mean over all ten, as in Table 1. A.3MORE RESULTS Is the checker independent of the parser? The parser, checker, and solver share the same Qwen3-VL weights, so the checker could share the parser’s blind spots and confirm its mistakes. To test this, we grade the same claims with five checkers of different families and sizes (Table 9). From 150 parses, we extract 1,417 atomic claims. Of these, 1,170 come from the parser, and 247 are traps in which we change a number or a label, so the correct verdict is always no. All checkers use the same prompt and greedy decoding. The other models only audit our checker and never touch the training data, so training still uses a single model. Our checker rejects 244 of the 247 traps, so it does not simply approve what the parser writes. Agreement between checkers depends more on size than on family. The two 8B checkers from different families agree with each other more (κ=0.877) than our 2B checker agrees with Qwen3-VL-8B (κ=0.837). SmolVLM2-2.2B agrees least with every other checker (κ from 0.62 to 0.66) and accepts the most traps (5.7%), so it is a weaker checker rather than an independent one. An independent check therefore needs a larger model, not just a different family. The risk also depends on the claim type. Among the claims our checker accepts, 13 Under Submission Table 7: Solver training settings. Identical across 2B, 4B, and 8B except for the per-device micro- batch. SettingValue Adapter LoRA rank, alpha64, 128 Target modulesall linear layers, . * visual. * excluded Trainable visual parameters0.00M Gradient checkpointingon Optimiser AlgorithmAdamW, betas (0.9, 0.999), weight decay 0.01 Learning rate1e−5, constant GRPO Advantage estimatorGRPO KL penaltylow-variance estimator, β = 0.01, frozen reference Clip ratio (low, high)0.2, 0.3, dual clipping at 3.0 PPO epochs per step1 Dynamic samplingoff Rollout Rollouts per prompt8 at temperature 1.0, top-p 1.0 Prompt budget2048 tokens Response budget256 tokens Training image cap1,003,520 pixels Batching and schedule Rollout batch256 prompts Global update batch128 Micro-batch per device16 at 2B, 4 at 8B GPU memory utilisation0.60 Steps96 Table 8: Benchmark categories. Each benchmark belongs to one category. CategoryBenchmarkWhat it tests PerceptionGQAobjects, attributes, and relations InfoVQAtext and layout in infographics EmbSpatial (ESB)spatial relations in embodied scenes Knowledge & ReasoningOK-VQAoutside knowledge ScienceQA (SQA)school science MMMUcollege-level subjects LogicVista (LogicV)logical reasoning over images GeneralMMBench (MMB)many perception and reasoning skills MMStarquestions that require the image SEED-Bench (SEED)broad visual understanding at least two other checkers reject 26.7% of diagram arrows, 8.3% of chart values, and none of the object-existence claims. Diagram arrows and chart values are therefore where the checker most needs improvement. Since another checker’s rejection is not ground truth, we treat these rates as warning signs rather than error rates. Does VQS transfer to other model families? We also apply VQS to Gemma3-12B- It (Team et al., 2025), InternVL3-8B-Instruct (Zhu et al., 2025), and Llama-3.2-11B-Vision- Instruct (Grattafiori et al., 2024) (Table 10). VQS improves all ten benchmarks for every family, raising the average by 1.77 points on Gemma, 1.51 on InternVL3, and 1.84 on Llama. Gains are large on MMMU, InfoVQA, and EmbSpatial, about 3 to 4 points per family, and smallest on SEED- Bench and OK-VQA, under 0.7 points. The weakest base, Llama (55.77), gains the most, and the strongest, InternVL3 (64.46), gains the least. All three gains are smaller than on Qwen3-VL (+2.32 to +3.18, Table 1). 14 Under Submission Table 9: Checker independence. All five checkers grade the same 1,417 claims. Accepts is the share of the 1,170 parser claims a checker calls true. Trap YES is the share of the 247 corrupted claims it wrongly accepts. κ is Cohen’s kappa with our checker over all claims. CheckerRelation to the parserAccepts (%) Trap YES (%)κ vs. ours Qwen3-VL-2B (ours)same weights94.41.2– Qwen3-VL-8Bsame family, 4× larger93.31.60.837 InternVL3.5-8Bdifferent family, 8B93.14.00.810 Qwen2.5-VL-7Bprevious Qwen generation89.50.40.807 SmolVLM2-2.2Bdifferent family, same size86.15.70.660 Table 10: Results on other model families. Every model trains on unlabeled images only. Sub- scripts give the change against that backbone’s base. Abbreviations follow Table 1. VQS improves all ten benchmarks for every family, raising the average by +1.77 on Gemma3-12B-It, +1.51 on InternVL3-8B-Instruct and +1.84 on Llama-3.2-11B-Vision-Instruct. MethodGQAOK-VQA InfoVQASQAMMMUMMBESBLogicVMMStarSEEDAvg. Gemma3-12B-It Base61.2062.4052.1094.3046.5076.4024.5041.8058.9074.2059.23 VQS (ours)62.02+0.8262.71+0.3155.29+3.1995.23+0.9350.84+4.3479.06+2.6627.70+3.2042.87+1.0759.80+0.9074.47+0.2761.00+1.77 InternVL3-8B-Instruct Base65.4064.1068.8096.5051.2081.3028.1048.6063.7076.9064.46 VQS (ours)66.01+0.6164.36+0.2671.81+3.0196.71+0.2154.87+3.6783.17+1.8731.99+3.8949.42+0.8264.32+0.6277.08+0.1865.97+1.51 Llama-3.2-11B-Vision-Instruct Base56.0055.4046.9091.8050.7072.8021.2036.5055.1071.3055.77 VQS (ours)57.13+1.1356.05+0.6549.87+2.9793.47+1.6753.64+2.9476.04+3.2424.18+2.9837.79+1.2956.22+1.1271.68+0.3857.61+1.84 A.4PROMPTS Text in braces, such as q, is a Python format field filled at call time. The prompts in Section A.4.6 belong to the two baselines we compare against, which are the only places a model is asked to judge or vote on an answer. A.4.1PARSING AN IMAGE INTO A STRUCTURE One prompt per domain, each paired with a JSON schema enforced by constrained decoding, so valid JSON holds by construction rather than by instruction. Charts Read this chart. List every series and every data point you can see. Copy`printed_value` exactly as printed on the chart (keep %, $, B, commas). Put the plain number in`numeric_value`. Use the real category and axis names. Diagrams Read this diagram. List every labelled part as a node, copying labels verbatim. Set has_arrows to true ONLY if the diagram really draws arrows from one labelled part to another, like a cycle or a food web. Many diagrams only label parts and have no such arrows -- for those set has_arrows to false and leave edges empty. Never invent an arrow. 15 Under Submission Natural images Describe this photo as structure. List the main objects with their visible attributes and boxes in 0-1000 coordinates as [x0,y0,x1,y1]. Then list how the objects relate to each other, using simple predicates like "on", "holding". Infographics Read this infographic. List every labelled figure or statistic as an entry. Copy`printed_value` exactly as printed (keep %, $, commas, K/M/B). Put the plain number in`numeric_value` and the unit in `unit`. The schemas below are the ones the decoder enforces through vLLM’s guided decoding, as JSON Schema with whitespace compacted. Every field is required and no object admits extra fields, so a parse can neither omit a field nor add one. A numeric value may be null: a value that is drawn but not legible is recorded as null rather than guessed, and the templates skip it. Charts: output schema "type": "object", "additionalProperties": false, "required": ["kind", "title", "x_axis_title", "y_axis_title", "series"], "properties": "kind": "const": "chart", "title": "type": "string", "x_axis_title": "type": "string", "y_axis_title": "type": "string", "series": "type": "array", "maxItems": 6, "items": "type": "object", "additionalProperties": false, "required": ["series_name", "points"], "properties": "series_name": "type": "string", "points": "type": "array", "maxItems": 24, "items": "type": "object", "additionalProperties": false, "required": ["category", "printed_value", "numeric_value"], "properties": "category": "type": "string", "printed_value": "type": "string", "numeric_value": "type": ["number", "null"] Diagrams: output schema "type": "object", "additionalProperties": false, "required": ["kind", "title", "has_arrows", "nodes", "edges"], "properties": "kind": "const": "diagram", "title": "type": "string", "has_arrows": "type": "boolean", "nodes": "type": "array", "maxItems": 20, "items": "type": "object", "additionalProperties": false, "required": ["node_id", "node_label"], "properties": 16 Under Submission "node_id": "type": "integer", "node_label": "type": "string", "edges": "type": "array", "maxItems": 30, "items": "type": "object", "additionalProperties": false, "required": ["from_node_id", "to_node_id", "edge_label"], "properties": "from_node_id": "type": "integer", "to_node_id": "type": "integer", "edge_label": "type": "string" Natural images: output schema "type": "object", "additionalProperties": false, "required": ["kind", "objects", "relations"], "properties": "kind": "const": "photo", "objects": "type": "array", "maxItems": 10, "items": "type": "object", "additionalProperties": false, "required": ["object_id", "object_name", "attributes", "box_0_1000"], "properties": "object_id": "type": "integer", "object_name": "type": "string", "attributes": "type": "array", "maxItems": 3, "items": "type": "string", "box_0_1000": "type": "array", "minItems": 4, "maxItems": 4, "items": "type": "integer", "relations": "type": "array", "maxItems": 12, "items": "type": "object", "additionalProperties": false, "required": ["subject_id", "predicate", "object_id"], "properties": "subject_id": "type": "integer", "predicate": "type": "string", "object_id": "type": "integer" Infographics: output schema "type": "object", "additionalProperties": false, "required": ["kind", "title", "entries"], "properties": "kind": "const": "infographic", "title": "type": "string", "entries": "type": "array", "maxItems": 20, "items": "type": "object", "additionalProperties": false, "required": ["entry_label", "printed_value", "numeric_value", "unit"], "properties": "entry_label": "type": "string", "printed_value": "type": "string", "numeric_value": "type": ["number", "null"], "unit": "type": "string" 17 Under Submission A.4.2REWRITING THE TEMPLATED QUESTION The template program emits the question; the model then rewrites it. The instruction forbids adding or removing conditions and restricts the rewrite to words already present, because an unconstrained rewrite silently changes the computation. The gold answer is never shown and never touched. Paraphrase Rewrite this question so it sounds natural, keeping its meaning exactly the same. Do not answer it. Do not add or remove any condition. Use only the words and entities already in the question. Reply with only the rewritten question. Question: q A.4.3THE BLIND GATE The shipped question is answered k=4 times by the same model with the image withheld. Blind gate (no image) Answer this question with your best guess. You cannot see the image, so guess from the wording alone. Reply with the answer only, no explanation. Question: q A.4.4TRAINING-TIME PROMPTS Each training item is the image, the question, and a suffix fixing the answer format. The format string is chosen by the program’s answer type. Answer-format suffix answer type = word Answer the question using a single word or phrase. answer type = number Answer the question using a single number. multiple-choice pool Answer with the option’s letter from the given choices directly. A.4.5MODEL-WRITTEN PROGRAMS (ABLATION) In the ablation where the model writes the question, the answer and the program, the parse is sup- plied as JSON and only the executor decides correctness. Question, answer and program written by the model You are given an image and a JSON description of it that was read off the image. graph Write ONE question about the image, its short answer, and a Python program that COMPUTES that answer from the JSON. Rules for the program: - exactly one function: def solve(g): ... where g is the JSON above, as a dict - return the answer; it will be compared to your "answer" field as a string 18 Under Submission - read the values out of g. Do NOT write the answer as a literal in the program - plain Python only: indexing, loops, if, len, sum, min, max, sorted, round, abs, str, int, float - no imports; use g["key"] and for x in g["list"]; string methods like .lower() and .strip() are fine Rules for the question: - answerable from the image alone by someone who cannot see the JSON - the answer must be a single word, number or short phrase - do not name the answer in the question Return JSON with keys question, answer, program. A.4.6BASELINE PROMPTS These are the prompts of the two baselines we compare our generator against. Both start from the same free-form question prompt. Question generation (both baselines) Write one question about this image that can be answered with a single word, phrase or number by looking at the image. Reply with only the question. Majority vote. The question is answered 5 times at T=1.0 with the standard format suffix, and the plurality answer is kept as the label. There is no prompt beyond the suffix. Judge You are grading one answer to a question about an image. Question: q Proposed answer: a Look at the image and decide whether the proposed answer is correct. Reply with exactly one word: YES if it is correct, NO if it is not. 19 Under Submission A.5QUALITATIVE EXAMPLES This section shows what the generator produces. Everything in it comes from Qwen3-VL-2B pars- ing, checking and gating its own images. Table 11 lists twenty template families with a question each one wrote. Section A.5.1 follows single questions from the parse to the answer key, including two that the filters removed, and Section A.5.2 shows the code that computes one family’s answer. Levels follow the generator’s operand depth: L1 lookup, L2 single hop, L3 two operands, L4 derived operand, L5 comparison of aggregates. Table 11: Template families, with one question each. A slot in⟨angle brackets⟩ is filled from the parse, and a part in [square brackets] appears only when the chart has several series. Each example is a question the family wrote for a real image, shown after paraphrasing where the paraphrase passed the guard, with the key its program computed. We checked every key against its image. FamilyLvTemplate→ example questionAnswer key Photos existenceL2Are there any⟨objects⟩ in the image? → Are there any windows in the image? no relationexistenceL2Are there any⟨objects⟩⟨relation⟩ the⟨anchor⟩? → Are there any chairs to the right of the painting? yes attributefiltered existence L3Is there a⟨attribute⟩⟨object⟩ in the picture? → Is there white grass in the picture? no statefiltered existence L3Do you see a⟨state⟩⟨object⟩ in the picture? → Do you see an open oven door in the picture? yes gated object recognition L3Which kind of⟨category⟩ is⟨attribute⟩? → Which kind of furniture is green? bench relationchainL4Are there any⟨objects⟩⟨relation⟩ the⟨anchor⟩ that is⟨relation⟩ the ⟨target⟩? → Are there any watches to the left of the man that is to the right of the baseball bat? yes Charts retrieveL1What is the value of⟨category⟩[ for⟨series⟩]? → What is the value of Indiana? 0.53 extremumL2Which category has the⟨highest| lowest⟩ value[ for⟨series⟩]? → Which category has the highest value? 2025* compareL2Which has a higher value,⟨a⟩ or⟨b⟩[ for⟨series⟩]? → Which has a higher value, 2018 or 2020? 2018 diffL3How much higher is⟨a⟩ than⟨b⟩[ for⟨series⟩]? → How much higher is 2019 than 2018 for Stayovers? 31984 ratioL3What is the ratio of⟨a⟩ to⟨b⟩[ for⟨series⟩]? → What is the ratio of Somewhat prefer theater to Strongly prefer streaming for June 2020? 0.44 series compareL3In⟨category⟩, which is higher,⟨series⟩ or⟨series⟩? → In 2016, which is higher, China or United States? China ranked arithmeticL5Is the sum of the two highest values greater than the sum of the three lowest?No Infographics igchartmetadataL1What is the title of this chart?THE WORLD’S 10 LARGEST STADIUMS topkL2Which are the⟨k⟩ highest[ in⟨title⟩]? → Which are the 3 highest in HOW AMERICANS RATE BUSINESS FIGURES? Bill Gates, Warren Buffett and Mark Cuban multi itemfilterL2Which have a value greater than⟨threshold⟩[ in⟨title⟩]? → Which have a value greater than 37 in HOW AMERICANS RATE BUSINESS FIGURES? Bill Gates, Warren Buffett, Mark Cuban and Elon Musk igdiffL3How much higher is⟨a⟩ than⟨b⟩? → How much higher is China than the U.S.? 994 compoundlookupL3Which is highest[ in⟨title⟩] and what is its value? → Which is highest in HOW AMERICANS RATE BUSINESS FIGURES and what is its value? Bill Gates, 45 Diagrams diagram identification L1What is shown in this diagram? → What is depicted in this diagram? Types of Volcanoes existenceL2Does every labelled item in this diagram have an arrow attached?yes 20 Under Submission A.5.1TRACED QUESTIONS Each card follows one generated question through the pipeline. Parse lists the fields the template read, Program lists the operations that compute the answer, and Question gives the template question with its paraphrase. Fact check gives the checker’s yes or no verdict on each claim behind the answer, and Blind gate lists four answers from the same model without the image (Eq. (1)). The last two cards show questions that the filters removed. Photo · gatedobjectrecognition · L3KEPT Parsefurniture objects whose color is green:bench Program gatedobjectrecognition(category=furniture, color=green), which must find exactly one kind QuestionWhich kind of furniture is green? Fact check read back the bench’s color: “green”✓ (attribute read-back) Blind gatesofa · couch · “The green furniture is a chair.” · “leafy green furniture” 0/4 match✓ Answer key bench Photo · attributefilteredexistence · L3KEPT Parsegrass objects carrying the attribute white: none Program attributefilteredexistence(object=grass, attribute=white) QuestionIs there a white grass in the picture? → Is there white grass in the picture? (paraphrase accepted) Fact check asked whether there is white grass: “no”✓ Blind gateYes · Yes · Yes · Yes 0/4 match✓ Answer key no Chart · rankedarithmetic · L5KEPT ParseNumber of visitors in millions: 2007 2.7 · 2008 2.75 · 2009 3.53 · 2010 3.13 · 2011 3.61 · 2012 3.8 · 2013 3.75 Programtwo highest 3.8 + 3.75 = 7.55; three lowest 2.7 + 2.75 + 3.13 = 8.58; 7.55 > 8.58 is false QuestionIs the sum of the two highest values greater than the sum of the three lowest? Fact check 7/7 values read back exactly, e.g. 2009→ 3.53✓ Blind gateYes · Yes · Yes · Yes 0/4 match✓ Answer key No Chart · diff · L3KEPT Program | Stayovers(2019)− Stayovers(2018)| on a stacked chart with three series QuestionHow much higher is 2019 than 2018 for Stayovers? Fact check read back 463,685 and 431,701; the difference recomputed from them, 31,984, matches the key✓ Blind gate10 · 1 · 10 · 1 0/4 match✓ Answer key 31984 Chart · ratio · L3KEPT ProgramSomewhat prefer theater÷ Strongly prefer streaming, in the June 2020 series QuestionWhat is the ratio of Somewhat prefer theater to Strongly prefer streaming for June 2020? Fact check read back 16 and 36; the ratio recomputed from them, 16/36 = 0.444, matches the key✓ Blind gate1.2 · 0.22 · 0.67 · 0.50 0/4 match✓ Answer key 0.44 21 Under Submission Infographic · igdiff · L3KEPT Program | China− U.S.| QuestionHow much higher is China than U.S.? → How much higher is China than the U.S.? (paraphrase accepted) Fact check read back 1,334 and 340; the difference recomputed from them, 994, matches the key✓ Blind gate“10000000” · “China is taller than the U.S. by about 3,000 miles.” · two more free-text guesses 0/4 match✓ Answer key 994 Infographic · igchartmetadata · L1 KEPT Parsetitle: THE WORLD’S 10 LARGEST STADIUMS Programread the title field of the parse QuestionWhat is the title of this chart? Fact check read back “The World’s 10 Largest Stadiums”✓ (case is ignored) Blind gate“Line Chart” · “Average household income by age group” · two free-text guesses 0/4 match✓ Answer key THE WORLD’S 10 LARGEST STADIUMS Diagram · diagramidentification · L1KEPT Parsetitle: Types of Volcanoes Programread the title field of the parse QuestionWhat is shown in this diagram? → What is depicted in this diagram? (paraphrase accepted) Fact check read back the title: “Types of Volcanoes”✓ Blind gate“The diagram depicts a chloroplast.” · “A bird in the sky.” · two more free-text guesses 0/4 match✓ Answer key Types of Volcanoes Chart · rankedaggregate · L4DROPPED BY FACT CHECK ParsePercentage of the GDP: ’00 4.1 · ’01 3.9 · . . . · ’20 4.7 · ’21 4.8 · ’22 4.8 · ’23 4.8 (24 values; the parse stops at ’23) Programthree highest 4.8 + 4.8 + 4.8 = 14.4 QuestionWhat is the sum of the three highest values? (paraphrase “three largest” rejected: new content word) Fact check 21 of 24 read-backs pass; ’21 read back as 5.45✗, ’22 as 5.4✗, ’23 as 5.45✗. The chart shows 5.4 to 5.5 for these years and reaches 5.9 by ’30*, so the key 14.4 was wrong. Answer key 14.4, never used 22 Under Submission Chart · seriescompare · L3DROPPED BY BLIND GATE Programcompare the China and United States series at 2016, on a stacked chart with seven series QuestionIn 2016, which is higher, China or United States? Fact check read back 1471.18 for China✓ and 189.3 for United States, where the chart shows 60.12 (189.3 is Thailand’s value). The comparison recomputed from the two read-backs, China, still matches the key✓, so the probe passes despite the misread. Blind gateChina · China · China · China 4/4 match✗. The key is correct, but a model without the image answers it from world knowledge, so the question teaches nothing about the image. Answer key China, never used A.5.2THE PROGRAM BEHIND A TEMPLATE A family is a question template paired with an executor that computes the answer from the parse. Below is the pair for ranked arithmetic, the family of the third card. The executor returns nothing rather than a guess when the parse cannot support a unique answer: too few values, or a tie. The cells it read (allprov) are what the fact check later asks the model to verify. rankedarithmetic: template and executor # question side: the template, filled from the parse q = (f"Is the sum of the two highest subj or 'values'sr greater than the " f"sum of the three lowest?") # answer side: the executor, run on the parsed values of one series if fam == "ranked_arithmetic": k, m = int(prog["k"]), int(prog["m"]) # here k = 2, m = 3 if len(vals) < k + m: return None # too few values: no question vs = sorted(vals, reverse=True) top, bot = sum(vs[:k]), sum(vs[-m:]) if top == bot: return None # a tie has no answer: no question return ("Yes" if top > bot else "No"), "free", allprov A.5.3VQS, VISE AND THE BASE MODEL ON THE SAME BENCHMARK QUESTIONS Each card shows one benchmark item answered by three 2B models: the base model, VISE, and VQS after one training cycle. All responses are taken from our lmms-eval runs, where the three models received the same prompt. Red text explains why the other two answers are wrong. GQAsample 420 QuestionIs the woman near the man standing on the bricks? Goldyes VQS (ours) yes✓ VISEno✗ Baseno✗ Why wrong The woman beside the bearded man in the background is standing on the brick pavement. 23 Under Submission OK-VQAsample 3993 QuestionHow old are the boys? Gold5 VQS (ours) 5✓ VISE7✗ Base7✗ Why wrong None of the ten human annotators answered 7, so it scores 0 under VQA accuracy; the players are small children. InfoVQAsample 150 detail of the “by state” panel QuestionHow many states are having a poor job Outlook? Gold5 VQS (ours) 5✓ VISE4✗ Base4✗ Why wrong Five states sit under the thumbs-down in “Jobs outlook by state”: Rhode Island, New Jersey, Washington, Oregon and Alaska. Four is the size of the thumbs-up group (Georgia, Hawaii, Florida, Utah). ScienceQAsample 934 QuestionContext: Use the data to answer the question below. Is the following statement about our solar system true or false? The volume of Mars is more than three times as large as Mercury’s. A. false B. true GoldA VQS (ours) A. false✓ VISEB. true✗ BaseB. true✗ Why wrong The table gives Mars 160 and Mercury 60 billion km 3 , so Mars is 160/60≈ 2.7 times Mercury’s volume, less than three. MMMUsample 758 QuestionLook at the two pictures below. Which picture appears to have more order? A. a B. b C. same of the two GoldA VQS (ours) A. a✓ VISEB. b✗ BaseB. b✗ Why wrong In (b) the jar lies on its side and the marbles are scattered over the table and the floor; in (a) they are all inside the upright jar, the more ordered state. 24 Under Submission MMBenchsample 98 QuestionSelect the best answer. Which property do these three objects have in common? A. blue B. smooth C. flexible GoldB VQS (ours) B✓ VISEC✗ BaseC✗ Why wrong Only the silk tie is flexible; the glass bottle and the wooden baseball bat are rigid. All three are smooth. EmbSpatialsample 3056 QuestionWhich object, in relation to your current position, holds the farthest placement in the image? A) chair B) Unknown C) box D) shelves GoldD VQS (ours) D✓ VISEC✗ BaseC✗ Why wrong The box is the grey block in the foreground, the object nearest the camera. The shelves on the desk at the back of the room are the farthest. LogicVistasample 255 QuestionAssuming the report has overestimated revenues by 15%, what was the actual revenue generated by Binders in the quarter October to December? Select from A, B, C, and D. (A) 30,435 (B) 25,500 (C) 29,750 (D) 35,000 GoldA VQS (ours) A✓ VISEC✗ BaseC✗ Why wrong A 15% overestimate means the reported 35 is 1.15 times the actual value, so the actual revenue is 35/1.15≈ 30.4 thousand. Both baselines take 85% of the reported value instead, and both say the chart covers three quarters when it shows four. VISE also ends with a malformed <CAnswer> tag. 25 Under Submission LogicVista, sample 255: full responses VQS (ours) ...The chart shows the revenue in thousands for four different companies over four quarters. The blue line represents Binders. In the "Oct - Dec" quarter, the chart indicates a revenue of 35 thousand. Since the report has overestimated revenues by 15%, we need to calculate the actual revenue by dividing the reported value by (1 + 0.15). This gives us 35 / 1.15 ≈ 30.435. Therefore, the actual revenue generated by Binders in the quarter October to December is approximately 30,435. <answer>A</answer> VISE ...The chart shows the revenue for four companies (Moen, Binders, Graff, and Smiths) over three quarters. The y-axis represents revenue in thousands of dollars, and the x-axis represents the time period. First, we need to determine the reported revenue for Binders in the "Oct - Dec" quarter from the chart. The blue line represents Binders. At the "Oct - Dec" point on the x-axis, the reported revenue is 35,000 (thousands). The report has overestimated revenues by 15%. This means that the actual revenue is 100% - 15% = 85% of the reported value. So, the actual revenue is: (85/100) * 35,000 = 0.85 * 35,000 = 29,750 Therefore, the actual revenue generated by Binders in the quarter October to December was 29,750. <CAnswer> C</CAnswer> Base ...The chart shows the revenue for four companies (Moen, Binders, Graff, and Smiths) over three quarters. The y-axis represents revenue in thousands of dollars, and the x-axis represents the time period. First, we need to determine the reported revenue for Binders in the "Oct - Dec" quarter from the chart. The blue line represents Binders. At the "Oct - Dec" mark on the x-axis, the reported revenue is 35 thousand dollars. The report has overestimated revenues by 15%. This means that the actual revenue is 100% - 15% = 85% of the reported value. So, the actual revenue is: (85/100) * 35,000 = 0.85 * 35,000 = 29,750 Therefore, the actual revenue generated by Binders in the quarter October to December was 29,750. <answer>C</answer> MMStarsample 152 QuestionWhich image is the brightest one? Options: A: upper left, B: upper right, C: down left, D: down right GoldC VQS (ours) C✓ VISEB✗ BaseB✗ Why wrong The upper-right panel is fireworks against a night sky and mostly black; the lower-left panel, a clear daytime sky, is bright across the whole frame. SEED-Benchsample 2433 QuestionWhich of the following objects are detected in the image? A. Person B. Yellow and blue sculpture C. Ball D. All of the above GoldB VQS (ours) B✓ VISED✗ BaseD✗ Why wrong There is no person and no ball in the image, so “All of the above” cannot hold; the human figures are illuminated Statue of Liberty sculptures. 26

Instant research discovery