Paper deep dive

It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scaling

Puzhuo Zheng, Hasan Kurban

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 49

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Test-time scaling lifts large language model reasoning by sampling many candidate solutions and selecting among them, yet the same recipe transfers poorly to vision-language models (VLMs): recent work shows that simple majority voting beats selection methods built on the model's own self-verification, apparently because at the selection layer an image-grounded answer and a confident guess from the language prior look the same. A natural fix is to make the selection signal one that cannot be computed without the image. We study Perturbation Grounded Selection (Pgs), a label-free, training-free rule that scores each candidate by whether the model re-derives it under label-preserving perturbations of the input (cropping, background masking, mild photometric or geometric jitter); Pgs recovers majority voting when the perturbation set is empty. The decisive question is not whether Pgs beats chain-of-thought only majority voting, but whether the perturbation term adds anything once decoding format and budget are controlled. We therefore introduce a format-matched control (MatchedCtrl): the same short, no-CoT draws spent on the original image. Across TextVQA, MATH-Vision, MMMU, and ViLP, with a Qwen headline (three-seed means) and LLaVA-OneVision coverage in matched-budget selector tables, Pgs appears to beat plain majority voting by up to +31.8 points on TextVQA (Qwen), but MatchedCtrl tracks or exceeds Pgs within noise on every benchmark, including the vision-required ViLP; no Qwen category shows a significant gain over this control. The stability gap is real and image-dependent (up to +0.48), yet does not predict per-instance wins. The result is negative and diagnostic: perturbation consistency is at best a partial diagnostic of visual dependence and, on its own, not a usable selection signal once format is controlled; gains reported against CoT-only majority voting overstate such methods.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

47,362 characters extracted from source content.

Expand or collapse full text

It’s the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scaling Puzhuo Zheng, Hasan Kurban Abstract Test-time scaling lifts large language model reasoning by sampling many candidate solutions and selecting among them, yet the same recipe transfers poorly to vision-language models (VLMs): recent work shows that simple majority voting beats selection methods built on the model’s own self-verification, apparently because at the selection layer an image-grounded answer and a confident guess from the language prior look the same. A natural fix is to make the selection signal one that cannot be computed without the image. We study Perturbation-Grounded Selection (Pgs), a label-free, training-free rule that scores each candidate by whether the model re-derives it under label-preserving perturbations of the input (cropping, background masking, mild photometric or geometric jitter); Pgs recovers majority voting when the perturbation set is empty. The decisive question is not whether Pgs beats chain-of-thought-only majority voting, but whether the perturbation term adds anything once decoding format and budget are controlled. We therefore introduce a format-matched control (MatchedCtrl): the same short, no-CoT draws spent on the original image. Across TextVQA, MATH-Vision, MMMU, and ViLP, with a Qwen headline (three-seed means) and LLaVA-OneVision coverage in matched-budget selector tables, Pgs appears to beat plain majority voting by up to +31.8+31.8 points on TextVQA (Qwen), but MatchedCtrl tracks or exceeds Pgs within noise on every benchmark, including the vision-required ViLP; no Qwen category shows a significant gain over this control. The preserve/destroy stability gap is real and image-dependent (up to +0.48+0.48), yet does not predict per-instance wins. The result is negative and diagnostic: perturbation consistency is at best a partial diagnostic of visual dependence and, on its own, not a usable selection signal once format is controlled; gains reported against CoT-only majority voting overstate such methods. We release code and audit tooling. ViLP-F A seed grows into a tree. What grows into a tree in the image? Sample N=8N=8reasoning tracesseedseedseedleafleafprior-drivenseed×5groundedleaf×3 Majority vote never re-reads x seedpicks frequent⇒ wrongPgs (ours): re-score under edits of xxcropmask bgjitterleafleafleafleaf“leaf” stable across views ⇒ correct Figure 1: PGS on Real ViLP-F example (Qwen2.5-VL-7B). A VLM samples N=8N=8 traces for an image–question pair whose prompt plants a language prior (“A seed grows into a tree”) while the image-grounded answer is “leaf”. The prior guess “seed” is most frequent (55 vs. 33), so majority voting picks it and is wrong—it never re-reads the image. Pgs (ours) instead re-scores each candidate by adding re-rendering x under label-preserving edits (center crop, background mask, photometric jitter): “leaf” stays modal across views. So Pgs might selects the stable, image-grounded answer. Majority voting is the empty-perturbation special case of Pgs (Section 3). 1 Introduction Sampling many candidate solutions at inference and selecting a good one (test-time scaling) has become one of the most reliable ways to improve reasoning in large language models (LLMs) (Wei et al. 2022; Wang et al. 2023; Snell et al. 2024; Brown et al. 2024). The recipe assumes a usable selection signal: either many samples agree on the right answer (self-consistency / majority voting) or the model can verify its own candidates well enough to pick the best one (best-of-N with self-verification). This recipe transfers poorly to vision-language models, and recent work converges on one cause. For RL-tuned VLMs on visual math, majority voting beats verification-centric selection and the self-correction “aha moment” yields no reliable gain (Wu et al. 2026a). VLMs often attend to the correct evidence yet still answer wrongly (“seeing but not believing” (Liu and others 2025)) and can hold a fixed answer while their internal representation drifts under label-preserving edits (Wani et al. 2026), so grounded and ungrounded answers are indistinguishable at the output level and output stability is not evidence of grounding. Self-consistency assumes correctness correlates with answer frequency, which fails when spurious paths dominate (Chung et al. 2025); and across seven VLMs, single-model majority voting yields only modest, chain-of-thought-dependent gains that vanish once outputs are correlated (Tong et al. 2026). The common thread: the selection layer cannot tell an image-grounded answer from a confident guess, because the usual signals (frequency, verbalized confidence) never test whether the answer depends on the pixels. The selection layer, not the generation layer, is where vision-language test-time scaling leaks. The field’s responses have largely gone in two directions, both of which leave the core problem untouched for a practitioner with modest resources. One trains the deficiency away with reinforcement learning or tool-use curricula so the model learns when to re-examine visual evidence (Wu et al. 2026b; Zhu et al. 2026; Marsili and Gkioxari 2025); this needs training compute and data many groups do not have. The other intervenes inside a single generation, for example by masking deep-layer attention toward evidence regions (Liu and others 2025) or adding an external verifier or process reward model that scores candidates (Marsili and Gkioxari 2025); these need access to model internals, a second trained network, or supervision. Neither gives a label-free, training-free fix at the selection layer that runs on a single consumer GPU, which is precisely the regime in which test-time scaling is most attractive. We turn this diagnosis into a design constraint: if the usual selection signals fail because they never test whether an answer depends on the image, then a corrective signal must be one that cannot be computed without the image. A natural candidate, which we call Perturbation-Grounded Selection (Pgs), scores each of the N sampled answers by re-rendering the visual input under a set of label-preserving perturbations, transformations that should not change the correct answer to a genuinely image-grounded question (cropping toward question-relevant regions, masking plausibly irrelevant background, mild photometric or geometric jitter), and measuring how stable the model’s support for that answer is across these views. The intuition is that an answer re-derived from many perturbed views is grounded in the pixels, whereas an answer that survives only on the original view is the fingerprint of a language-prior guess. Majority voting is the empty-perturbation special case of Pgs, so this is a strict generalization of the standard rule rather than a competing heuristic (Section 3). Pgs inherits none of the costs that make trained fixes inaccessible (no training, labels, reward model, or second network, at a constant multiple of ordinary best-of-N cost), so if a purely inference-time grounding signal can improve selection, Pgs is where it should show. We therefore ask a sharper question than “does Pgs beat majority voting?”: does the perturbation signal add anything once the comparison is fair? The comparison is not fair by default, and this is the crux. Pgs’s perturbation-side draws are short, answer-only samples with no chain of thought, whereas plain majority voting aggregates only the N long CoT samples, so any gain of Pgs over MV conflates the perturbation reweighting we care about with a switch in decoding format (CoT→short) that we do not, and that format is not incidental for visual tasks (Sammani et al. 2026; Kaya et al. 2026). We isolate the two with a format-matched control (MatchedCtrl): reuse the N CoT answers and add the same number of short, no-CoT answers, but drawn from the original image with no perturbation. MatchedCtrl spends Pgs’s exact extra budget in its exact decoding style and changes only whether those short answers pass through perturbed views, so comparing Pgs to MatchedCtrl rather than to CoT-only MV tests grounding rather than format. Our measurements across four automatically scored benchmarks—with Qwen as the headline model and LLaVA-OneVision in the selector tables—return a negative result. Against plain MV, Pgs appears to help substantially (up to +31.8+31.8 points on TextVQA; three-seed mean), but against MatchedCtrl it shows no reliable advantage on any benchmark, including the vision-required ViLP, and no category yields a significant gain. The preserve/destroy stability gap the method is built on is real and image-dependent (up to +0.48+0.48), yet its per-instance value does not predict when Pgs wins. Perturbation consistency is thus at best a partial diagnostic of visual dependence, not a usable label-free selection signal once decoding format is held fixed. Contributions. (1) A decoding-format confound and its control: comparing a perturbation- or consistency-based selection rule against chain-of-thought-only majority voting conflates the selection mechanism with a decoding-format change, and our format-matched control (MatchedCtrl), the same short, no-CoT budget spent on the unperturbed image, isolates it, naming a cause distinct from the diversity (Tong et al. 2026) and internal-structure (Cai et al. 2026) accounts of weak VLM selection. (2) A negative result under that control: across four benchmarks (Qwen headline; LLaVA-OneVision in matched-budget selectors), Pgs, a label-free, training-free rule that strictly generalizes majority voting, gives no advantage over MatchedCtrl on any Qwen headline benchmark, including the vision-required ViLP, and a real, image-dependent stability gap (up to +0.48+0.48) does not predict when it wins. (3) An experiment-facing isolation protocol: we formalize the information and budget difference among MV, Pgs, and MatchedCtrl, and state what would count as evidence that perturbation reweighting supplies visual grounding at the selection layer (Section 4). (4) Released measurement and audit tooling (the Pgs score, stability gap, paired bootstrap CIs), so the format-matched comparison can be reused to audit perturbation- and consistency-based selection claims for VLM test-time scaling. 2 Related Work Test-time scaling and self-verification. LLM test-time scaling selects among many samples via majority vote or best-of-N verification (Wang et al. 2023; Brown et al. 2024; Snell et al. 2024). The same recipe is attractive for VLMs, yet the selection layer is fragile: for RL-tuned visual math models, majority voting beats verification-centric selection and self-correction gains are unreliable (Wu et al. 2026a), while self-refinement often degrades open VLMs (Ahmadpour et al. 2025). Adaptive compute allocation and trained multi-view policies (Jeddi et al. 2026; Avogaro et al. 2026; Yang et al. 2025; Zhu et al. 2026) improve generation—when to look again, which crop to take—rather than how to aggregate already-drawn candidates. Pgs is orthogonal: it leaves the generator unchanged and only reweights answers at selection time, with no training and no second network. Perturbation signals and confounds. Input perturbations already appear as decoding interventions (e.g., contrastive decoding (Leng et al. 2024; Wan et al. 2025)) and as confidence or consistency signals (Kim and Chelikavada 2026; Chou et al. 2026). Closest neighbors differ in what they require (attention access, trained modules) or in what they claim (generation-time grounding vs. selection-time reweighting); a compact design-properties comparison is in the Supplementary Material (Miyazawa and Okuno 2026; Liu and others 2025). Parallel accounts of weak VLM selection emphasize sample diversity (Tong et al. 2026), internal commitment versus correctness (Cai et al. 2026), and the decoding substrate (Baxevanakis and Yang 2026; Sammani et al. 2026; Kaya et al. 2026) (short answers often dominate verbose CoT on visual tasks). We add a format-isolation account: when a perturbation-based selector spends its extra budget as short, no-CoT draws, a gain over CoT-only majority voting can be almost entirely a CoT→short effect that a format-matched control (MatchedCtrl) removes. Under that control, preserve/destroy stability remains useful as an uncertainty diagnostic (Zhang et al. 2024), but not as a selector. 3 Method 3.1 Setup and notation A VLM defines a distribution pθ(a∣x,q)p_θ(a x,q) over answers a given an image x∈x and a question q. Test-time scaling draws N candidates a1,…,aN∼pθ(⋅∣x,q)a_1,…,a_N p_θ(· x,q) (with chain-of-thought, then extracting the final answer) and applies a selection function S(ai,x,q)→a^S(\a_i\,x,q)→ a. Let ans(⋅)ans(·) map a sampled generation to its normalized final answer and let =ans(ai)A=\ans(a_i)\ be the set of distinct candidate answers. Throughout, the image x and question q are fixed for an instance; what changes across selectors is which additional draws are taken and how they enter S. Majority voting. The standard rule is SMV=arg⁡maxc∈∑i[ans(ai)=c]S_MV= _c _i1[ans(a_i)=c]. Each sample aia_i is generated conditioned on the image x, so MV does use vision at generation time; the aggregation step itself only counts extracted answers and does not re-read x. That is a limitation of the selection layer, not a claim that MV ignores the image. Separately, Wu et al. (2026a) show that self-verification-based selection fails to integrate visual evidence effectively and is outperformed by MV—a generation–verification gap that motivates seeking selection signals beyond unverified self-checks, while still leaving open whether a perturbation-based reweighting can beat a format-matched control. 3.2 Label-preserving perturbations Definition 1 (Label-preserving perturbation set). A perturbation set =t1,…,tMT=\t_1,…,t_M\ is a finite collection of maps tm:→t_m:X such that for the true answer a⋆a to (x,q)(x,q), a⋆a remains the correct answer to (tm(x),q)(t_m(x),q) for all m. Examples used here: (i) crops toward question-relevant regions proposed by a cheap saliency heuristic; (i) masking of background regions unlikely to contain the answer; (i) mild photometric jitter (brightness/contrast) and small-angle rotation or rescaling. Perturbations are label-preserving by construction, not by assumption about the model: a crop that still contains the evidence, a mask over background, or a brightness change does not alter the ground-truth answer to a well-posed visual question. We make this concrete and auditable in Section 5 via the label-destroying control, which deliberately removes the evidence region and must reduce Pgs’s support for the correct answer if the signal is genuinely grounded. 3.3 The selection rule For candidate answer c∈c , define its grounded support g(c)= g(c)\;= ∑i[ans(ai)=c]⏟original-view votes _i1[ans(a_i)=c]_original-view votes (1) +λ∑m=1Mwmρ(c∣tm(x),q), \;+\;λ _m=1^Mw_m\,ρ (c t_m(x),q ), where ρ(c∣tm(x),q)ρ(c t_m(x),q) is the model’s re-derivation strength for answer c under perturbed view tm(x)t_m(x), wm≥0w_m≥ 0 weights perturbation m, and λ≥0λ≥ 0 trades off original-view agreement against perturbation stability. We instantiate ρ as the consistency score ρ(c∣tm(x),q)=1K∑k=1K[ans(a(m,k))=c],ρ(c t_m(x),q)= 1K _k=1^K1 [ans(a^(m,k))=c ], (2) with a(m,k)∼pθ(⋅∣tm(x),q)a^(m,k) p_θ(· t_m(x),q): we draw K short samples per perturbed view and count how often they re-derive c. The selected answer is a^=arg⁡maxc∈⁡g(c) a= _c g(c). Algorithm 1 Perturbation-Grounded Selection (Pgs) 1:image x, question q, model pθp_θ, perturbations T, counts N,KN,K, weight λ 2:sample a1,…,aN∼pθ(⋅∣x,q)a_1,…,a_N p_θ(· x,q); ←ans(ai)A←\ans(a_i)\ 3:for each perturbation tm∈t_m do 4: sample a(m,1),…,a(m,K)∼pθ(⋅∣tm(x),q)a^(m,1),…,a^(m,K) p_θ(· t_m(x),q) 5: record ρ(c∣tm(x),q)ρ(c t_m(x),q) for every c∈c 6:end for 7:compute g(c)g(c) by Eq. (1) for every c∈c 8:return a^=arg⁡maxc⁡g(c) a= _cg(c) Proposition 1 (Majority voting is a special case). If =∅T= (equivalently λ=0λ=0), then g(c)g(c) reduces to the original-view vote count and a^=SMV a=S_MV. This is immediate from Eq. (1). Pgs is therefore a strict generalization of majority voting rather than a competing heuristic: the original-view vote count is always present in g(c)g(c), and the perturbation term only re-weights candidates. We do not claim Pgs can never underperform majority voting (a large λ can in principle override a correct original-view majority), so the choice of λ matters. Unless noted otherwise, all primary results use a fixed operating point λ=2λ=2 (chosen once for the protocol, not computed per example). Separately, we also evaluate a label-free selection rule that does not look at ground truth: split the K samples of each perturbed view into two halves, run selection on each half, and among the λ values whose two halves agree on the chosen answer at least 80%80\% of the time, take the largest. Choosing the smallest such λ would trivially return λ=0λ=0, since majority voting always agrees with itself. That rule is reported in SensAblations alongside the full λ sweep and a label-aware oracle; it is not the λ used to produce Table 1. 3.4 Cost The full procedure is Algorithm 1. The generation cost is N+MKN+MK forward samples versus N for plain best-of-N / CoT-only majority voting. Throughout the experiments we hold the total number of generations fixed at a matched budget (default N+MK=32N+MK=32; Section 5), so differences among Pgs, and MatchedCtrl are about how those draws are spent rather than about spending more. Perturbed-view samples are short (answer-only), so the MKMK term stays cheap relative to N long CoT traces; MatchedCtrl spends that same short-answer mass on the original image. 4 What Would Count as a Grounding Gain? The empirical claim is not that Pgs can never help relative to some baseline, but that a widely used comparison is misleading, and that under the comparison that isolates the intended mechanism the gain disappears. This section fixes that comparison before the numbers appear. 4.1 Three selectors, three information budgets Fix an instance (x,q)(x,q) and a generation budget B=N+MKB=N+MK. Write CoTN(x)CoT_N(x) for N chain-of-thought samples on the original image and ShortK(y)Short_K(y) for K short, no-CoT samples on an image y. • MV uses only CoTN(x)CoT_N(x) and returns a majority vote. Cost N. • Pgs uses CoTN(x)CoT_N(x) together with ShortK(tm(x))m=1M\Short_K(t_m(x))\_m=1^M and returns arg⁡maxc⁡g(c) _cg(c) (Eq. (1)). Cost N+MKN+MK. • MatchedCtrl uses CoTN(x)CoT_N(x) together with ShortMK(x)Short_MK(x)—the same short-answer mass, still on the original image—and returns a majority vote on the pooled answers. Cost N+MKN+MK. MV and Pgs differ in two ways at once: decoding format of the extra draws (CoT vs. short) and whether those draws see perturbed pixels. MatchedCtrl matches Pgs on budget and short-answer format and differs only in whether the short draws are routed through T. Therefore: A gain of Pgs over MV is not evidence of perturbation grounding. A gain of Pgs over MatchedCtrl would be. Conversely, if acc(Pgs)≈acc(MatchedCtrl)acc( Pgs) (MatchedCtrl), the perturbation term is not buying selection accuracy beyond spending the same short-answer budget on the original view. That is the decision-relevant null for contribution (2). 4.2 Diagnostic vs. routing; operating point Even under a null against MatchedCtrl, the perturbation channel may still track visual content. We separate two roles: • Diagnostic (StabilityGap/BlankAblation). The preserve/destroy stability gap, and the collapse of Pgs when perturbation inputs are blanked, test whether ρ depends on the image. A large gap can coexist with a selection null. • Routing (RoutingTest). For the same instances, does a larger gap predict a larger per-example gain [Pgs]−[MatchedCtrl]1[ Pgs]-1[MatchedCtrl]? If not, the gap is not a usable switch for when to trust perturbation weighting over MatchedCtrl. A grounding gain at the selection layer would require (i) a reliably positive Δ(Pgs−MatchedCtrl) ( Pgs-MatchedCtrl) overall or in a pre-specified slice, and/or (i) a monotone routing relationship in RoutingTest. Section 5 tests both and finds neither. Primary tables fix λ=2λ=2, pool union, and vote weight 11, holding total generations at N+MK=32N+MK=32 unless noted. Sensitivity to λ, M, K, and perturbation family is reported in SensAblations; those ablations do not overturn the MatchedCtrl null. 5 Experiments We evaluate the isolation protocol of Section 4: whether Pgs improves over the format-matched control MatchedCtrl, and whether the stability gap routes that comparison—not merely whether Pgs beats CoT-only majority voting. The design therefore reports MV (for the familiar but confounded contrast), MatchedCtrl (for the decision-relevant null), and the StabilityGap/BlankAblation/RoutingTest diagnostics that separate “the signal sees the image” from “the signal improves selection.” We evaluate on four benchmarks that vary in how much the answer depends on the image: TextVQA (Singh et al. 2019), MATH-Vision (Wang et al. 2024), MMMU (Yue and others 2024), and ViLP (Luo et al. 2025), the last of which pairs a language-prior-aligned answer with a vision-required answer for each question and is the natural stress test for a grounding-at-selection claim. Coverage uses two open VLMs—Qwen2.5-VL-7B-Instruct (Qwen Team 2025) and LLaVA-OneVision-7B (Li et al. 2024). The headline mechanism table (Table 1) and the StabilityGap/ BlankAblation/ SensAblations/RoutingTest analyses report Qwen three-seed means for a single readable story; matched-budget selector comparisons and reproducibility tables report both models (Table 2, Table 3; and compute tables in the Supplementary Material). Unless stated otherwise, Pgs is recomputed offline with candidate pool union, vote weight vw=1vw=1, and λ=2λ=2 on saved runs. Accuracy is hard-match rate in percent (TextVQA: any-annotator match). Table 1: Main results on Qwen2.5-VL-7B-Instruct. Mean hard-match accuracy over seeds 0,23,42\0,23,42\. Pgs appears to beat plain majority voting (MV) by up to +31.8+31.8 p (column Δ ), yet the format-matched control MatchedCtrl (shaded)—the same short, no-CoT budget on the original image—tracks Pgs within seed noise on every benchmark (TextVQA: MatchedCtrl still ahead). Accuracy is hard-match rate (%; TextVQA any-annotator match); Pgs uses union, vote weight 11, λ=2λ=2. Accuracy (%) Bench. N MV Pgs MatchedCtrla Δ Gapc TextVQA 300300 54.454.4 86.286.2 87.687.6 31.831.8 0.4780.478 MATH-V 299299 25.625.6 26.026.0 25.325.3 0.40.4 0.0250.025 MMMU 900900 49.549.5 50.350.3 50.150.1 0.70.7 0.0710.071 ViLP 600600 53.253.2 52.352.3 53.753.7 −0.9-0.9 0.4450.445 a MatchedCtrl, format-matched control: reuse the N CoT answers ++ M⋅KM·K short no-CoT answers on the original image (N+MK=32N+MK=32). b Δ=acc(Pgs)−acc(MV) =acc( Pgs)-acc(MV), from seed-mean accuracies. c Mean Gap ρpreserve−ρdestroy _preserve- _destroy over seeds; ViLP gaps are computed on non-prior (grounded) slots in the dump. 5.1 Setup and controls Protocols. For each example we draw N original-view answers with chain-of-thought (CoT) and, for Pgs, K short, no-CoT answers on each of M label-preserving perturbations. Pgs scores candidates by votes plus λ times re-derivation support ρ estimated from the perturbation views. We report: • MV: majority vote over the N original-view CoT samples only. • Pgs: ρ-weighted selection as above. • MVLift: paired difference Pgs−MV Pgs-MV (percentage points). • MatchedCtrl: format-matched control that reuses the N CoT answers and adds M⋅KM·K short no-CoT answers on the original image (N+MK=32N+MK=32), matching Pgs’s extra budget and decoding style with no perturbation. • StabilityGap: mean paired stability gap ρpreserve−ρdestroy _preserve- _destroy on the MV-selected candidate under label-destroying crops. Comparing Pgs to MV confounds perturbation weighting with the CoT↔ short format change; comparing Pgs to MatchedCtrl isolates whether routing those short answers through perturbed views and ρ helps beyond keeping them on the original image. Baselines. Our comparison isolates the perturbation mechanism against the format-matched control (MatchedCtrl), the decisive test for our claim; it is not a survey of selectors. Still, under the same matched-budget protocol on each VLM we evaluate standard label-free alternatives—self-certainty (SC), SC with Borda voting (Kang et al. 2026), confidence-weighted self-consistency (Taubenfeld et al. 2025), and mean token-entropy selection—against MV, MatchedCtrl, and Pgs at N+MK∈16,32N+MK∈\16,32\ (Table 2, Table 3). Both LLaVA-OneVision-7B and Qwen appear in these selector tables, so the MatchedCtrl null is not an artifact of a single checkpoint family. Confidence selectors use the matched N+MKN+MK decode pool after teacher-force rescoring; budget 1616 is an offline subsample (N=4N=4, K=2K=2) of the budget-3232 dumps. Consistent with the intro diagnosis, these selectors do not systematically beat MatchedCtrl: SC and entropy often fall below MatchedCtrl, while SC+Borda and CISC typically track MatchedCtrl within noise. Where Pgs also fails to separate from MatchedCtrl, the null remains diversity / format rather than grounding; where absolute numbers differ across models, the gap is still relative to a control that already absorbs the strongest confidence-based aggregators. Table 2: Label-free selectors vs. MV/MatchedCtrl/Pgs at matched budget N+MK=32N+MK=32 (rescored confidence metrics; mean± over seeds). Benchmark Model MV MatchedCtrl SC Entropy SC+Borda CISC PGS MATH-Vision LLaVA-OV-7B 18.8± 1.1 20.7± 0.9 15.8± 1.1 17.8± 0.7 19.2± 2.3 19.5± 0.8 20.3± 0.2 MATH-Vision Qwen2.5-VL-7B 25.6± 1.6 25.3± 1.0 17.9± 0.5 15.4± 0.6 25.1± 1.0 25.4± 1.0 26.0± 0.7 MMMU LLaVA-OV-7B 47.4± 0.5 47.6± 0.6 43.9± 0.3 45.7± 0.3 47.5± 0.7 47.9± 0.7 48.3± 0.7 MMMU Qwen2.5-VL-7B 49.5± 0.3 50.1± 0.4 44.7± 0.5 44.7± 0.7 50.1± 0.5 50.4± 0.4 50.3± 0.7 TextVQA LLaVA-OV-7B 78.8± 0.2 81.3± 0.3 78.8± 0.8 80.4± 0.8 82.2± 0.5 81.6± 0.5 81.7± 0.3 TextVQA Qwen2.5-VL-7B 54.4± 2.0 87.6± 0.4 47.1± 2.1 84.0± 4.1 88.1± 0.5 88.0± 0.3 86.2± 0.7 ViLP LLaVA-OV-7B 49.9± 0.3 49.4± 0.3 50.4± 0.5 49.1± 0.1 50.4± 0.9 50.1± 0.7 49.5± 0.7 ViLP Qwen2.5-VL-7B 53.2± 0.7 53.7± 0.4 50.9± 0.6 48.6± 0.6 54.0± 0.2 54.0± 0.4 52.3± 0.7 Table 3: Label-free selectors vs. MV/MatchedCtrl/Pgs at matched budget N+MK=16N+MK=16 (offline subsample from budget-3232 dumps; mean± over seeds). Benchmark Model MV MatchedCtrl SC Entropy SC+Borda CISC PGS MATH-Vision LLaVA-OV-7B 20.9± 2.2 20.7± 1.2 17.4± 1.4 19.0± 0.7 20.4± 0.6 20.3± 1.2 22.9± 0.8 MATH-Vision Qwen2.5-VL-7B 23.3± 1.6 26.8± 2.6 23.0± 1.9 21.6± 2.2 26.9± 2.3 26.3± 2.6 26.8± 1.1 MMMU LLaVA-OV-7B 48.2± 1.2 48.3± 0.6 44.8± 0.1 47.1± 0.4 47.6± 0.4 48.2± 0.9 48.8± 1.1 MMMU Qwen2.5-VL-7B 49.3± 0.8 51.0± 0.5 45.4± 0.6 45.7± 0.6 51.3± 0.1 51.2± 0.1 50.6± 0.3 TextVQA LLaVA-OV-7B 75.7± 0.3 81.8± 0.2 79.3± 0.7 81.3± 0.3 81.8± 0.4 82.0± 0.7 81.6± 0.2 TextVQA Qwen2.5-VL-7B 57.0± 2.0 87.9± 0.4 49.9± 2.9 84.7± 2.7 88.4± 0.7 88.4± 0.2 88.7± 0.7 ViLP LLaVA-OV-7B 49.8± 0.3 49.5± 0.4 50.5± 0.2 49.4± 0.2 50.1± 0.8 50.1± 0.6 48.9± 0.8 ViLP Qwen2.5-VL-7B 52.9± 0.8 53.3± 0.3 51.8± 1.2 49.7± 0.3 53.7± 0.4 53.6± 0.2 51.2± 0.7 Reproducibility and benchmarks. We evaluate TextVQA, MATH-Vision, MMMU, and ViLP (Score / ViLP-P / ViLP-F splits as noted per table; ViLP StabilityGap gaps use non-prior grounded slots in the dumps, N=600N=600 per seed). Decode settings, exact (N,M,K)(N,M,K), compute footprint for both VLMs, dataset versions/splits/licenses, and checkpoint revisions appear in the Supplementary Material. Default generation budget is N=8N=8, M=6M=6, K=4K=4 (N+MK=32N+MK=32) on a single NVIDIA RTX 4090. 5.2 Main results Table 1 is the headline comparison (three-seed means on Qwen dumps). Against plain MV, Pgs gains substantially on TextVQA (+31.8+31.8 p) and is near-flat on MATH-V, MMMU, and ViLP. That comparison is unfair in exactly the sense of Section 4: MV aggregates only the N CoT samples, whereas both Pgs and MatchedCtrl add M⋅KM·K short, no-CoT answers. The all-CoT MV pool is therefore the weaker aggregator at a smaller effective budget in short-answer mass: accuracy rises at matched N+MKN+MK even when the extra draws never leave the original image. MatchedCtrl spends those same short answers on the original image and, under this format-matched control, tracks Pgs within seed noise on every benchmark (TextVQA 87.687.6 vs. 86.286.2; MATH-V 25.325.3 vs. 26.026.0; MMMU 50.150.1 vs. 50.350.3; ViLP 53.753.7 vs. 52.352.3), with MatchedCtrl still ahead on TextVQA. The MV→ lift is therefore the effect of adding short no-CoT mass, not evidence that re-derivation support ρ supplies visual grounding at the selection layer; isolating the perturbation piece leaves no reliable advantage. The StabilityGap reinforces this reading rather than rescuing it. Large mean preserve/destroy gaps on TextVQA and ViLP (+0.478+0.478 / +0.445+0.445) coexist with no win over MatchedCtrl, while smaller gaps on MATH-V and MMMU (+0.025+0.025 / +0.071+0.071) likewise fail to separate Pgs from MatchedCtrl. Preserve/destroy asymmetry is real and image-dependent, but its magnitude does not identify when perturbation-weighted selection beats format-matched short-answer MV. In other words, the diagnostic channel can fire without a routing gain over MatchedCtrl—the distinction Section 4 insists on. 5.3 Ablations: the signal is visual but does not route selection Table 4: BlankAblation (earlier single-seed Qwen dual-PGS run; not present in the multi-seed dumps used for Table 1). Holding the original-view CoT pool fixed and blanking the perturbation inputs collapses accuracy where the signal is visual (TextVQA, ViLP-F, MMMU) but not on symbolic MATH-V. Absolutes are not aligned with the multi-seed main table; only the within-row comparison is meaningful. Benchmark N MV Pgs BlankAblation TextVQA 302302 55.055.0 87.787.7 7.97.9 MATH-V 304304 22.422.4 26.326.3 23.423.4 MMMU 900900 49.949.9 50.250.2 23.323.3 ViLP-F 266266 48.948.9 46.246.2 0.40.4 Blanking the perturbation inputs (BlankAblation, Table 4) collapses Pgs’s accuracy where the signal is strong (TextVQA 87.7→7.987.7\!→\!7.9, ViLP-F 46.2→0.446.2\!→\!0.4, MMMU 50.2→23.350.2\!→\!23.3) while symbolic MATH-V is unaffected, confirming the score is genuinely image-dependent rather than a format artifact of short decoding alone. Yet image-dependence and the stability gap decouple (MMMU drops sharply while its multi-seed mean gap is only +0.071+0.071), so the gap is only a partial measure of visual dependence: BlankAblation can fail even when StabilityGap looks mild. Ablations over perturbation family, λ, M, and K (SensAblations, Supplementary Material: perturbation-family / λ-sweep / λ λ tables; recomputed offline from the same multi-seed Qwen dumps as Table 1) show Pgs is active almost only on OCR-heavy TextVQA (crop/geometry-sensitive) while other benchmarks are flat, consistent with a format-sensitive rather than a universal grounding effect. Neither ablation overturns MatchedCtrl: where the signal is visually live, matched short answers on the original image still absorb the lift. Figure 2: No category shows a significant gain of Pgs over the format-matched control MatchedCtrl. Per-category difference Δ=acc(Pgs)−acc(MatchedCtrl) =acc( Pgs)-acc(MatchedCtrl) with 95% bootstrap confidence intervals (Supplementary Material), pooled over seeds 0,23,42\0,23,42\ on Qwen dumps. No interval lies entirely in the “favors Pgs” half-plane (shaded); the ViLP interval lies entirely below zero. Points are Δ ; whiskers are the 95% CI. 5.4 RoutingTest: where the gap predicts help (it does not) We break results down by perception-heavy vs. text-heavy categories and correlate per-instance gain ([Pgs]−[MatchedCtrl])(1[ Pgs]-1[MatchedCtrl]) with the StabilityGap; on ViLP we additionally report prior→ flip rates relative to MatchedCtrl. These analyses ask the routing question of Section 4 directly: even if the mean Δ(Pgs−MatchedCtrl) ( Pgs-MatchedCtrl) is near zero, a pre-specified slice or a monotone gap→ relationship could still salvage a conditional use of ρ. The Supplementary Material category table and Figure 2 close the isolation protocol under the decision-relevant baseline: no category yields a significant positive Δ(Pgs−MatchedCtrl) ( Pgs-MatchedCtrl) (every 95% bootstrap CI overlaps zero). The stability gap also fails to route the selector: instance-level correlations between gap and gain are near zero (Supplementary Material), gap quartiles are non-monotone, and Pgs’s recoveries of MatchedCtrl errors on ViLP are rare (Supplementary Material). A large measured re-derivation advantage therefore does not forecast when perturbation-weighted selection beats format-matched short-answer MV. The gap remains a useful diagnostic of preserve/destroy asymmetry; it is not a reliable routing signal for the selector studied here. 6 Discussion The controls converted a plausible mechanism into a measured null. StabilityGap and BlankAblation show that the re-derivation score tracks visual content when the pixels are removed or destroyed; once the same short draws sit on the original image, the perturbation term buys nothing at selection, and the per-instance gap does not route the selector (RoutingTest). Diagnostic image dependence is therefore not a routing gain over a format-matched control (Section 4). Gains against CoT-only majority voting can be almost entirely a decoding-format effect—here as large as +31.8+31.8 p on TextVQA—and should not be read as evidence that perturbation consistency supplies visual grounding. ViLP sharpens the point: even where language priors and image answers are split, MatchedCtrl still meets or beats Pgs, so a large preserve/destroy gap coexists with a selection null rather than rescuing it. This isolates a format piece that coexisting 2026 accounts do not foreground in the same way: diversity-limited self-consistency (Tong et al. 2026) and commitment-versus-correctness (Cai et al. 2026) explain other failure modes of VLM selection, while MatchedCtrl shows that a CoT↔ short confound can inflate apparent perturbation gains even when sample diversity and internal stability are held aside. Holding the number of draws fixed is not enough if the extra draws change answer format; the control must also hold format fixed on the original view. Progress beyond MatchedCtrl will likely need an information source that answer-level consistency alone does not supply—denser visual probes, process-level rewards, or cross-model disagreement. We do not show that a trained verifier, or multi-model ensemble beats MatchedCtrl; whether such a signal recovers the gap remains open, and any such claim must still clear a format-matched control. Practical takeaways. Any selection rule that spends extra draws in a different decoding format than the MV baseline needs a format-matched control; without one, a format effect is easily misread as a grounding gain. Short-answer aggregation on the original image is a strong, cheap baseline under matched budget, and in our grid it is the hurdle that confidence-style and perturbation-weighted selectors alike must clear. Perturbation consistency remains worth computing as a partial diagnostic of visual dependence (StabilityGap/BlankAblation)—useful for auditing whether a score still sees the image—but not as a drop-in selector once format is controlled. Reporting MV lifts without a MatchedCtrl-style arm should be treated as incomplete for grounding-at-selection claims. Limitations Our conclusion is bounded, and stating the bounds is part of reading it honestly. The evidence covers two open VLMs (LLaVA-OneVision-7B and Qwen2.5-VL-7B) on four automatically scored benchmarks. A stronger family or larger decode budget could in principle show a benefit ours do not; we do not claim universality beyond the measured grid. Main results are aggregated over three decoding seeds (0,23,42\0,23,42\) as mean± (Supplementary Material); paired Pgs-vs-MatchedCtrl comparisons carry instance-level bootstrap CIs (RoutingTest), the right uncertainty for the claim that no category shows a significant gain over MatchedCtrl. Three seeds cannot rule out rarer decoding regimes. The perturbation families remain hand-designed heuristics (crop, mask, mild photometric and geometric jitter); a learned set could carry more signal, but would still need a MatchedCtrl-style control before any gain is attributed to grounding, and our SensAblations sweeps already show that family choice does not overturn the null under matched format. We do not include a stronger-signal reference point (trained verifier or multi-model ensemble) that beats MatchedCtrl, so recoverability under training is not demonstrated. And the negative finding is about answer selection: the stability gap remains a partial diagnostic of visual dependence—none of which we test here for abstention or routing. Ethics Statement Pgs is inference-only and uses public benchmarks and open models, raising no annotation- or participant-related concerns. Favoring image-grounded answers may reduce confidently-wrong outputs that ignore the pixels, but a stability signal can be satisfied by inputs that are stable yet wrong, so Pgs is not a correctness or safety guarantee and does not replace verification in safety-critical use; it also inherits the underlying model’s biases and failure modes. Deploying perturbation probes still incurs extra decode cost and can amplify whatever stereotypes or dataset artifacts the base VLM already exhibits under short-answer sampling. Because our headline finding is negative under MatchedCtrl, the main ethical risk of overstating Pgs as a grounding selector is mitigated by the release itself: we release the analysis code, perturbation specifications, and evaluation tooling with paired bootstrap CIs, so the null against format-matched short-answer aggregation is reproducible. We encourage downstream work that claims a grounding gain at selection to publish the same MatchedCtrl-style contrast rather than MV-only lifts. The authors used AI-based tools for assistance with language editing and software debugging during the development of this work. All AI-generated suggestions were reviewed, verified, and modified by the authors. References M. Ahmadpour, A. Meighani, P. Taebi, O. Ghahroodi, A. Izadi, and M. Soleymani Baghshah (2025) Limits and gains of test-time scaling in vision-language reasoning. arXiv preprint arXiv:2512.11109. Cited by: §2. N. Avogaro, N. Debnath, L. Mi, T. Frick, J. Wang, Z. He, H. Hua, K. Schindler, and M. Rigotti (2026) SPARC: separating perception and reasoning circuits for test-time scaling of VLMs. In Proceedings of the 43rd International Conference on Machine Learning (ICML), Cited by: §2. S. Baxevanakis and P. Yang (2026) Test-time scaling for small VLMs on multilingual visual MCQ. arXiv preprint arXiv:2607.09438. Cited by: §2. B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. Ré, and A. Mirhoseini (2024) Large language monkeys: scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787. Cited by: §1, §2. M. Cai, L. Kulik, and F. Choudhury (2026) ARBITER: reasoning trajectory basins and majority vote failures in test-time sampling. arXiv preprint arXiv:2605.26172. Cited by: §1, §2, §6. S. Chou, S. Chandhok, J. J. Little, and L. Sigal (2026) Test-time consistency in vision language models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Cited by: §2. H. Chung, T. Hsiao, H. Huang, C. Cho, J. Lin, Z. Zhang, J. Hsieh, and Y. Chen (2025) Revisiting test-time scaling: a survey and a diversity-aware method for efficient reasoning. arXiv preprint arXiv:2506.04611. Cited by: §1. A. Jeddi, M. N. Le, A. Kazerouni, H. C. Karaimer, H. Nguyen, I. Mohomed, M. Brudno, A. Levinshtein, K. G. Derpanis, B. Taati, and R. Grzeszczuk (2026) AVIS: adaptive test-time scaling for vision–language models. arXiv preprint arXiv:2606.11576. Cited by: §2. Z. Kang, X. Zhao, and D. Song (2026) Scalable best-of-n selection for large language models via self-certainty. Advances in neural information processing systems. Cited by: §5.1. M. O. Kaya, D. Elliott, and D. P. Papadopoulos (2026) Efficient test-time scaling for small vision-language models. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. K. Kim and K. Chelikavada (2026) Zoom consistency: a free confidence signal in multi-step visual grounding pipelines. arXiv preprint arXiv:2604.15376. Cited by: §2. S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing (2024) Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, et al. (2024) Llava-onevision: easy visual task transfer. arXiv preprint arXiv:2408.03326. Cited by: §5. Z. Liu et al. (2025) Seeing but not believing: probing the disconnect between visual attention and answer correctness in VLMs. arXiv preprint arXiv:2510.17771. Cited by: §1, §1, §2. T. Luo, A. Cao, G. Lee, J. Johnson, and H. Lee (2025) Probing visual language priors in VLMs (ViLP). In International Conference on Machine Learning (ICML), Cited by: §5. D. Marsili and G. Gkioxari (2025) No labels, no problem: training visual reasoners with multimodal verifiers. arXiv preprint arXiv:2512.08889. Cited by: §1. T. Miyazawa and H. Okuno (2026) Answer self-consistency with margin-triggered question re-arbitration for the CVPR 2026 VidLLMs challenge. arXiv preprint arXiv:2606.04323. Cited by: §2. Qwen Team (2025) Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923. Cited by: §5. F. Sammani, T. Chamiti, and N. Deligiannis (2026) On test-time scaling for vision-language models. In European Conference on Computer Vision (ECCV), Cited by: §1, §2. A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach (2019) Towards VQA models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Cited by: §5. C. Snell, J. Lee, K. Xu, and A. Kumar (2024) Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314. Cited by: §1, §2. A. Taubenfeld, T. Sheffer, E. Ofek, A. Feder, A. Goldstein, Z. Gekhman, and G. Yona (2025) Confidence improves self-consistency in llms. In Findings of the Association for Computational Linguistics: ACL 2025, Cited by: §5.1. Y. Tong, Y. Hou, S. Cui, A. Bosselut, and M. Sachan (2026) Diversity matters: revisiting test-time compute in vision-language models. In International Conference on Machine Learning (ICML), Cited by: §1, §1, §2, §6. D. Wan, J. Cho, E. Stengel-Eskin, and M. Bansal (2025) Contrastive region guidance: improving grounding in vision-language models without training. In European Conference on Computer Vision (ECCV), Cited by: §2. K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li (2024) Measuring multimodal mathematical reasoning with math-vision dataset. In Advances in Neural Information Processing Systems, Cited by: §5. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou (2023) Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. F. A. Wani, A. Suglia, R. Saxena, A. P. Gema, W. Kwan, F. Barez, M. S. Bucarelli, F. Silvestri, and P. Minervini (2026) Same answer, different representations: hidden instability in VLMs. arXiv preprint arXiv:2602.06652. Cited by: §1. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou (2022) Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1. M. Wu, M. Li, J. Yang, J. Jiang, K. Yan, Z. Li, H. Yu, M. Zhang, and K. Nahrstedt (2026a) Aha moment revisited: are VLMs truly capable of self verification in inference-time scaling?. arXiv preprint arXiv:2506.17417. Cited by: §1, §2, §3.1. M. Wu, J. Yang, J. Jiang, M. Li, et al. (2026b) VTool-r1: vlms learn to think with images via reinforcement learning on multimodal tool use. In International Conference on Learning Representations (ICLR), Cited by: §1. Y. Yang, J. Liu, Z. Zhang, S. Zhou, R. Tan, J. Yang, Y. Du, and C. Gan (2025) MindJourney: test-time scaling with world models for spatial reasoning. In Advances in Neural Information Processing Systems, Cited by: §2. X. Yue et al. (2024) MMMU: a massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §5. R. Zhang, H. Zhang, and Z. Zheng (2024) VL-Uncertainty: detecting hallucination in large vision-language models via uncertainty estimation. arXiv preprint arXiv:2411.11919. Cited by: §2. C. Zhu, J. Lin, Y. Long, P. Cao, T. Wang, J. Pang, and X. Liu (2026) Thinking with imagination: agentic visual spatial reasoning with world simulators. arXiv preprint arXiv:2606.06476. Cited by: §1, §2.

Instant research discovery