Paper deep dive
Unsupervised Multimodal Reward Modeling
Yueying Li, Jingxuan Fan, Kianté Brantley, Sham M. Kakade, Hanlin Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/12/2026, 6:16:47 AM
Summary
This paper introduces an unsupervised multimodal reward modeling method that eliminates the need for human preference labels. It employs a linear probe on an intermediate layer of a frozen Qwen3-VL backbone, trained using in-batch Bradley-Terry loss against self-perturbed captions (same-image perturbations). The approach demonstrates that the choice of probe layer and negative sampler (perturbations) are the critical factors for performance, outperforming GPT-4o on benchmarks like VL-RewardBench with significantly less compute. A verifier-driven bootstrap loop further improves policy performance by reducing hallucination.
Entities (8)
Relation Signals (7)
Unsupervised Multimodal Reward Modeling → evaluatedon → VL-RewardBench
confidence 95% · 6.6 points ahead on VL-RewardBench
Unsupervised Multimodal Reward Modeling → usesbackbone → Qwen3-VL
confidence 95% · We build one with no human preference label: a linear head on an intermediate layer of a frozen Qwen3-VL backbone
Unsupervised Multimodal Reward Modeling → usestrainingobjective → Bradley-Terry
confidence 92% · trained by in-batch Bradley–Terry against same-image perturbations
Unsupervised Multimodal Reward Modeling → outperforms → GPT-4o
confidence 90% · one recipe transferred without re-tuning beats or at least matches GPT-4o on three benchmarks
Unsupervised Multimodal Reward Modeling → reducesmetric → CHAIR
confidence 90% · cuts caption hallucination by 5.1 CHAIR points
Unsupervised Multimodal Reward Modeling → generalizesto → GLM-4.1V-9B
confidence 85% · The same recipe with no re-tuning lifts each of them above its own untrained judge... GLM-4.1V-9B
Unsupervised Multimodal Reward Modeling → outperforms → RLAIF-V
confidence 85% · At an equal pair budget, perturbations teach a frozen head 3 to 5 points more than curated RLAIF-V pairs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy. We build one with no human preference label: a linear head on an intermediate layer of a frozen Qwen3-VL backbone, trained to score a web caption above a perturbed version of the same caption; labeled data enters only to select the layer. The layer is the decisive choice, worth 5.8 points over the last layer, and one recipe transferred without re-tuning beats or at least matches GPT-4o on three benchmarks at 32B, 6.6 points ahead on VL-RewardBench, from a few thousand parameters at about a thousandth of fine-tuning compute. At an equal pair budget, perturbations teach a frozen head 3 to 5 points more than curated RLAIF-V pairs. Because the backbone is frozen, the objective is convex, and the perturbation sampler specifies the reward, which the benchmark does not identify: two heads trained on disjoint halves of the same perturbations tie on the benchmark and differ by 6.4 points in hallucination judging. Retraining the head on the policy's own verifier-labeled candidates cuts caption hallucination by 5.1 CHAIR points and lifts image-generation pass rate by 5.4, while scoring lower on every reward benchmark. Together, an intermediate-layer probe, free same-image perturbations, and a verifier-driven bootstrap turn give a recipe for multimodal reward modeling that needs no annotator, costs a fraction of fine-tuning, and is specified by the comparisons it is trained on rather than by a benchmark rank.
Tags
Links
- Source: https://arxiv.org/abs/2610.12707v1
- Canonical: https://arxiv.org/abs/2610.12707v1
Trouble viewing inline? Open PDF directly →
Full Text
97,996 characters extracted from source content.
Expand or collapse full text
Unsupervised Multimodal Reward Modeling Yueying Li †thanks: Work done outside of NVIDIA, while conducting research at MIT and Cornell. Affiliation: Massachusetts Institute of Technology Affiliation: Cornell University Jingxuan Fan Affiliation: Scale AI Kianté Brantley Affiliation: Harvard University Sham M. Kakade Affiliation: Harvard University Hanlin Zhang Affiliation: October 8, 2026 Affiliation: Harvard University Abstract Vision-language reward models are trained on annotated preference pairs, which are expensive to collect and can disagree with each other, and are ranked by benchmark accuracy. We build one with no human preference label: a linear head on an intermediate layer of a frozen Qwen3-VL backbone, trained to score a web caption above a perturbed version of the same caption; labeled data enters only to select the layer. The layer is the decisive choice, worth 5.8 points over the last layer, and one recipe transferred without re-tuning beats or at least matches GPT-4o on three benchmarks at 32B, 6.6 points ahead on VL-RewardBench, from a few thousand parameters at about a thousandth of fine-tuning compute. At an equal pair budget, perturbations teach a frozen head 3 to 5 points more than curated RLAIF-V pairs. Because the backbone is frozen the objective is convex and the perturbation sampler specifies the reward, which the benchmark does not identify: two heads trained on disjoint halves of the same perturbations tie on the benchmark and differ by 6.4 points in hallucination judging. Retraining the head on the policy’s own verifier-labelled candidates cuts caption hallucination by 5.1 CHAIR points and lifts image-generation pass rate by 5.4, while scoring lower on every reward benchmark. Together, an intermediate-layer probe, free same-image perturbations and a verifier-driven bootstrap turn give a recipe for multimodal reward modeling that needs no annotator, costs a fraction of fine-tuning, and is specified by the comparisons it is trained on rather than by a benchmark rank. Figure 1: Reward modeling with no human preference label. (a) A linear probe on a frozen Qwen3-VL backbone, trained on caption perturbations, beats or at least matches GPT-4o at 32B. (b) Two probes trained on disjoint halves of the same data tie on the benchmark and differ by 6.4 points on hallucination judging: the sampler, not the score, sets the reward. (c) Retraining the head on the policy’s own verifier-labelled candidates cuts caption hallucination 5.1 CHAIR points below the base generators (Section 3.4). 1 Introduction Reinforcement learning from human feedback (Christiano et al., 2017; Ouyang et al., 2022) has made the reward model the channel through which preferences enter post-training. For vision-language models the same design is now standard: a scalar scorer over an image, a prompt and a response, trained on annotated preference pairs and ranked on preference benchmarks (Yasunaga et al., 2025; Li et al., 2025; Hu et al., 2025). The design has two costs: multimodal preference pairs are expensive to annotate and the annotations disagree with each other (Yu et al., 2025). Once the model is built, benchmark accuracy is how it is compared, selected and trusted. VL-RewardBench accuracy correlates with Best-of-N accuracy on MMMU-Pro for generative judges (Pearson’s r>0.9r>0.9; Li et al., 2025); what sets that number for a scalar reward model, and when it stops predicting deployment, is the question this paper takes up. Text reward models show that a reward is a comparison before it is a number: Bradley–Terry training identifies a reward only up to a shift that may differ across prompts (Bradley and Terry, 1952; Rafailov et al., 2023), so a score of .8 has no stable meaning across prompts, candidate pools or checkpoints. The scalar hides the relative preference that was always the operative signal (Sun et al., 2025; Zhang, 2026), so what a reward model learns is set by which comparisons it is trained on. And those comparisons need no annotator: web continuations as chosen against in-batch continuations as rejected train a text reward model with no label at all (Fan et al., 2026). We take both findings to the vision-language setting, make the comparison the object of study, and remove the human labels entirely. The reward model is a linear head on the hidden states of an intermediate layer of a frozen backbone, trained by in-batch Bradley–Terry (Bradley and Terry, 1952) against same-image perturbations of web captions or same-seed re-renders of a perturbed prompt (Fig. 2, Section 2). The claim is about the training data: the probe layer is selected on one labeled benchmark, which we therefore treat as development and never as transfer evidence, and the loop of Section 3.4 uses verifier labels, not annotators. Two things then decide what the head learns: the representation it reads, which sets the signal a linear head can extract, and the comparison distribution it is trained on, which sets the direction of the reward. Every choice in the paper is one of the two, and the verified policy errors of Section 3.4 are the second variable’s final instance. A head of a few thousand parameters built this way beats GPT-4o by 6.6 points on VL-RewardBench at 32B, for about a thousandth of the compute of full backbone fine-tuning, and is level with GPT-4o on Multimodal RewardBench and MMRB2. At an equal pair count, the mined pairs teach a frozen head more than curated RLAIF-V pairs do. Figure 2: (a) The reward from a linear probe. A pair (image, prompt, real caption, same-image perturbation) passes once through the frozen backbone ϕ [rgb]0.0078,0.3125,0.4883φ; the layer-N−k [rgb]0.0078,0.3125,0.4883N-k states are mean-pooled over response tokens into h [rgb]0.0078,0.3125,0.4883h and scored by a bias-free head w [rgb]0.6523,0.4141,0w, trained with in-batch matrix Bradley–Terry; the sampler qneg [rgb]0.0039,0.418,0.3047q_neg is the only place supervision enters. (b) One turn of the bootstrap loop (Section 3.4): cheap verifiers label the policy’s own Best-of-N candidates and the same head is retrained on the minted pairs. Blue frozen, orange trained, green supervision. Because the backbone is frozen, the objective is transparent enough to say what that number is. It is ranking noise-contrastive estimation over fixed features (Gutmann and Hyvärinen, 2010; Ma and Collins, 2018), so with a fixed quadratic penalty it is convex with a unique minimizer, characterized as an average of the negative sampler’s difference vectors weighted by how badly the head still misorders each pair (Section 2.2). AdamW does not optimize that fixed objective, so where the trained heads land is measured rather than derived. In our runs no training-side change (pool size, steps, curriculum, loss form or optimizer setting) moves the benchmark by more than .014, so the representation and the sampler are the two choices that matter. The representation is the backbone’s, and our findings about it hold across backbones. On four Qwen3-VL sizes and on a second family, GLM-4.1V-9B, the best probe layer sits well below the top, twelve layers down for 8B, 32B and GLM-4.1V-9B. The same recipe with no re-tuning lifts each of them above its own untrained judge (Section 3.2). The sampler is the other choice, and it is where the benchmark stops being informative. Which pairs train the head decides which reward it learns. Two probes trained on disjoint subsets of the same perturbations, split by whether CLIP can see the edit, reach similar benchmark accuracy and differ by 6.4 points in hallucination judging (Fig. 1b). Deployed for Best-of-N reranking, benchmark rank orders utility in neither direction, and where the probe does beat majority vote, by 7 to 11 points, a training-free log-likelihood judge matches it. The analysis leaves one sampler to try: the policy’s own errors. We mint those labels with cheap verifiers on Best-of-N candidates, for captions and for generated images; the resulting head cuts caption hallucination by 5.1 CHAIR points below the base Qwen3-VL-2B and 8B generators, while its VL-RewardBench accuracy falls below that of the probe it replaced (Fig. 1c). One DPO (Rafailov et al., 2023) turn on the policy’s own pairs also cuts its hallucination by 3 to 6 CHAIR (Rohrbach et al., 2018) points at no QA cost (Table 8). Our contributions combine probing, negative sampling and self-improvement, each studied on the policy side before (Section 4), into one reward-side recipe that uses no human preference label. (i) Probed, not trained in: a frozen vision-language backbone with a linear head (2,048 parameters at 2B) at a depth set by the backbone, at or above GPT-4o on three benchmarks at 32B (Section 3.2). (i) Supervision as a sampler: same-image perturbations written as a mixture of operators that apply to captions and to rendered images alike, with an argument for which reward direction they select and an ablation of the operators in both modalities (Section 2.2, Table 3). (i) What the score tracks: benchmark accuracy plateaus under every training-side change we tried, the learned direction is set by the sampler, and benchmark rank predicts Best-of-N rank in neither direction (Sections 3.3 and 3.3). (iv) A bootstrap loop that replaces annotators with cheap verifiers on the policy’s own candidates, the one supervision that moved deployment, shown for captions and for image generation (Section 3.4). 2 Methodology 2.1 Algorithm We score each response with a bias-free linear head on mean-pooled response-token states from layer N−k [rgb]0.0078,0.3125,0.4883N-k of a frozen Qwen3-VL backbone (Qwen Team, 2025b). The triple (layer, pooling, head) is the design space the rest of the paper sweeps (Fig. 2a; colours as in its caption). For an image x, a prompt q and a response y, let ht(N−k) [rgb]0.0078,0.3125,0.4883h^(N-k)_t be the hidden state of layer N−k [rgb]0.0078,0.3125,0.4883N-k at response token t; the reward is sw(x,q,y)=w⊤1|y|∑t∈yht(N−k), [rgb]0.6523,0.4141,0s_w(x,q,y)\;=\; [rgb]0.6523,0.4141,0w \, 1|y| _t∈ y [rgb]0.0078,0.3125,0.4883h^(N-k)_t, (1) so the 2B model trains 2,0482,048 parameters and the backbone none. A micro-batch of B preference pairs (xi,qi,yi+,yi−)i=1B\(x_i,q_i,y_i^+,y_i^-)\_i=1^B gives the in-batch matrix Bradley–Terry loss, every chosen response against every rejected one, ℒ=1B2∑i=1B∑j=1B−logσ(β(si+−sj−)),L= 1B^2 _i=1^B _j=1^B- σ\! (β\,( [rgb]0.6523,0.4141,0s^+_i- [rgb]0.6523,0.4141,0s^-_j) ), (2) with β=0.1β=0.1 the inverse temperature and B=8B=8, so each positive meets eight negatives per step rather than one. Note that supervision enters only through the negative sampler, which is what makes the sampler the object Section 2.2 reasons about. For a positive (x,q,y+)(x,q,y^+) the rejected response is drawn from a mixture of perturbation operators applied to the positive itself, y−∼qneg(⋅∣x,q,y+)=∑oπo(y+;x,q),∑oπo=1, [rgb]0.0039,0.418,0.3047y^-\; \; [rgb]0.0039,0.418,0.3047q_neg(· x,q,y^+)\;=\; _o _o\, [rgb]0.0039,0.418,0.3047T_o(y^+;x,q), _o _o=1, (3) and the modality fixes what the operators act on. Unlike the pixel-level perturbations of ImageNet-C (Hendrycks and Dietterich, 2019), which perturb the input, these operators edit the response and change a fact while leaving the input untouched. For text responses nearT_near substitutes k editable spans of y+y^+ (colour, number, size or spatial words), spliceT_splice replaces its tail with another caption’s, shufT_shuf permutes its clauses, and ×T_× returns the caption of another image; the recipe uses π×=0.15 _×=0.15 and k=1k=1 (Figs. 6 and D). For example, a caption reading two brown horses beside a red fence becomes two brown horses beside a blue fence: one colour word moves, the token count is unchanged, and the pair differs only in a fact about the image. For image responses the operator acts on the prompt and a fixed-seed generator G renders both sides, y+=G(q,s)y^+=G(q;s) and y−=G(near(q),s)y^-=G(T_near(q);s). The usual label-free alternative is to have a model write the rejected response, as in constitutional and AI-feedback pipelines (Bai et al., 2022; Lee et al., 2024; Yu et al., 2025). We do not, because such a negative differs from the human positive in author, length and style as well as in fact, and by Proposition 1 the reward learns every one of those differences; length alone is a known shortcut for reward models (Singhal et al., 2024). In our own test a head trained on human captions against the policy’s captions for the same images scores .36 on Multimodal RewardBench, below chance, and picks the more hallucinated caption in every Best-of-N cell (Appendix F). Editing one fact inside the same response holds author, length and style fixed by construction. These negatives cost nothing to label: the caption operators are string edits with no model in the loop, and the 22k-pair bank builds in 10 seconds on one CPU core. Rendering the image arm’s 1,212 pairs takes 5 GPU-hours (Appendix K). 2.2 What the Objective Identifies We read Eq. 2 as ranking noise-contrastive estimation (Gutmann and Hyvärinen, 2010; Ma and Collins, 2018), with the reward a linear energy over frozen features (LeCun et al., 2006) and β the InfoNCE temperature (van den Oord et al., 2018). Write dij=hi+−hj−d_ij=h^+_i-h^-_j, so that Eq. 2 is 1B2∑ijℓ(βw⊤dij) 1B^2 _ij (β\,w d_ij) with ℓ(z)=log(1+e−z) (z)= (1+e^-z). We keep three objects apart: an idealized objective over unrestricted scores (Proposition 1); the linear head under a fixed penalty λ2w⊤Dw λ2w Dw (Corollary 1), a strongly convex problem with one minimizer that the sampler and β define and gradient descent reaches from any start; and the heads AdamW trains, whose stationary points depend on its preconditioner (Xie and Li, 2024), which we measure (Section 3.1). Appendix I reads the two statements and Appendix J proves them. Proposition 1 (the optimum is a density ratio against the sampler). Let P and Q be the joint laws of a positive and of a negative triple (x,q,y)(x,q,y), the latter with the sampler of Eq. 3 averaged over the positive it edits, q¯(y∣x,q) q(y x,q), and assume P and Q are mutually absolutely continuous. Over unrestricted scores, the cross-entry terms of Eq. 2 are minimized exactly by s⋆(x,q,y)=1βlogdPdQ(x,q,y)+C=1βlogp(y∣x,q)q¯(y∣x,q)+C,s (x,q,y)\;=\; 1β dPdQ(x,q,y)\;+\;C\;=\; 1β p(y x,q) q(y x,q)\;+\;C, (4) with C a constant. The B same-entry terms, where y−y^- is an edit of the very y+y^+ it is scored against, are coupled and are not minimized by Eq. 4. Those coupled terms carry weight 1/B=12.5%1/B=12.5\% at B=8B=8 and move even the unrestricted optimum off Eq. 4 (Appendix J). Our sampler also violates the absolute-continuity assumption, since a caption of another image has probability zero for this one, so Eq. 4 describes what the cross-entry terms reward and the linear head is analyzed through Corollary 1. Length drops out of the ratio when the negatives match its law conditionally on content, and a matched marginal is not enough (Appendix I); our bank holds length fixed on two thirds of its pairs and shortens the negative by about 5% on the rest, so nothing makes the target length-free, and Section 3.2 finds the length artifact this allows. Why perturbations provide useful supervision. Independent negatives from the caption marginal, our ×T_×, make s⋆s the image–caption pointwise mutual information up to a constant (Section I.2): a score of topic association, which can rank an accurate and an inaccurate response on the same topic equally. A same-context edit changes one fact and nothing else. In the stylized model of Section I.3 the quality factor cancels from the PMI score, while the edited pairs recover the correct ordering with margin logη1−η η1-η even when a fraction 1−η1-η of the originals is wrong. Perturbations thus supply one quality comparison, an original above its degraded copy, without requesting a judgment. Corollary 1 (the sampler’s pair law sets the direction). With the penalty λ2w⊤Dw λ2w Dw, the population minimizer of Eq. 2 satisfies Dw⋆=βλ[σ(−βw⋆⊤d)d],d=h+−h−,D\,w \;=\; βλ\;E [\,σ(-β\,w d)\;d\, ], d=h^+-h^-, (5) the expectation over the pair law of the matrix loss: Dw⋆Dw is the mean difference vector of the sampler’s pairs, each weighted by the probability the head still misorders it, so two samplers with the same perturbations but different pair laws give different heads in general. The law of the differences, not their span, sets the direction; this is how the two CLIP-split heads of Section 3.3 can reach similar benchmark accuracy while differing by 6.4 points in hallucination judging. Nothing in the objective sets the head’s length coefficient to zero, and Section 3.2 finds it with opposite signs at 2B and 8B under one sampler. Finally, Eq. 4 orders a policy’s own samples by a ratio against the perturbation process, so it ranks them by how unlike a perturbation each one looks and has no term for how wrong it is. The design hypothesis this leaves is that a sampler drawn from the policy’s own errors ranks them better: Section 3.3 measures how far the perturbation sampler falls short and Section 3.4 tests the hypothesis. 2.3 Experimental Setup We build pairs from 70k PixMo web captions (Deitke et al., 2024) and train heads on Qwen3-VL Instruct backbones at 2B, 4B, 8B and 32B, with the same negative bank and a frozen vision tower at every size. The headline recipe is the 2B backbone at probe layer N−4N-4, head only, 200 steps of 256 pairs with AdamW at learning rate 10−310^-3 and weight decay 0.010.01; every later run changes one element of it, and the curated comparison set is RLAIF-V (Yu et al., 2025). Every head trains on the same schedule of 256 pairs per step, so two runs are compared at equal pairs seen along the training timeline (Fig. 3, right). We evaluate on Multimodal RewardBench (Yasunaga et al., 2025), VL-RewardBench (Li et al., 2025) and MMRB2 (Hu et al., 2025), with GPT-4o (OpenAI, 2024) as the anchor on all three and no re-tuning between them. Per-benchmark pair counts, the MMRB2 scoring construction and the confidence bands are in Appendix C. For Best-of-N we sample N=16N=16 candidates from a Qwen3-VL-2B or 8B policy over 2,500 prompts of COCO captioning (Lin et al., 2014), MathVista (Lu et al., 2024), MMMU (Yue et al., 2024) and Claude-written hard questions on COCO images. We measure CHAIRs (Rohrbach et al., 2018) or exact match against six training-free controls, from random to the oracle, with 95% confidence intervals from a paired bootstrap over prompts. 3 Experiments 3.1 Two Choices Set the Accuracy: The Probe Layer and the Sampler Two choices set the accuracy, and neither is an optimizer setting: the probe layer alone adds 5.8 trainable points over the last layer, and mined negatives beat curated RLAIF-V pairs by 3.5 at equal pair count. Table 2 puts those two and every other ablation of the paper on one backbone under one recipe. Every number is a real end-to-end run except the layer sweep that nominated the probe depth: linear heads trained briefly on precomputed hidden states of every layer, a relative instrument that appears only in Fig. 3 and Appendix D. Figure 3: Left: layer-sweep trainable accuracy against layers below the top for 2B / 4B / 8B / 32B (bands: 75% binomial intervals); the optimum is never the last layer, and sweep numbers rank depths within a model only. Right: Multimodal RewardBench pair accuracy over the first 250 steps by negative type (2B full fine-tune, 674-pair eval subset): same-image perturbations transfer, cross-image negatives fall below the untrained base, curated pairs plateau lower. Probe depth. The best probe layer is never the top of the network: it is N−4N-4 for 2B, N−6N-6 for 4B and N−12N-12 for both 8B and 32B. That depth is chosen on Multimodal RewardBench trainable accuracy, so we treat that benchmark as development throughout and VL-RewardBench and MMRB2, which no choice in the recipe has seen, as the transfer evidence. The gap over the last layer is 3 to 7 points in the layer sweep (Fig. 3) and 5.8 trainable points on the real 4B run. The optimum deepens from 2B to 8B and then holds at twelve layers below the top for 8B, 32B and GLM-4.1V-9B (36, 64 and 40 layers), while the 36-layer 4B peaks at N−6N-6, so the depth is set by the backbone and not by its layer count; each peak is a plateau several layers wide (Appendix D). Negative construction. We find that same-image perturbations transfer in both modalities and that cross-image negatives do not (Table 3). For text, ×T_× alone reaches 99% training accuracy and falls below the untrained base on the benchmark, because it amplifies a length and style shortcut the same-image operators mostly remove: that head scores .224 on the length-sensitive safety/bias column against .451. For images the replica is exact, with the deranged negative transferring nothing to MMRB2 text-to-image (52.7 against a label prior of 53.2) while the same-seed planted edit accounts for all of it (56.0). Perturbation density and the operator mix are in Appendix D. Every training-side change leaves the benchmark within .014 of the recipe. Changing the loss form or the negative composition, training on four times the pairs or for twice the steps, or ordering the negatives from easy to hard or from hard to easy moves a real 2B run by at most .014. In the sweep setting, changing the negative pool or the temperature schedule moves it by less than .005. The sharpest row is a 540-configuration grid over head learning rate, β and weight decay: all 540 reach .995 or better in distribution, and selecting among them out of distribution changes the benchmark by .0003. Mined negatives against curated pairs at a matched pair budget. At 15k pairs the mined-negative head is 3 to 4 trainable points ahead of curated RLAIF-V pairs on Multimodal RewardBench and 2 to 5 on VL-RewardBench, including the hallucination split RLAIF-V was built for (Table 5). Unfreezing the backbone on the same 15k curated pairs recovers 11 points overall to .613: a frozen head matches full fine-tuning on mined-negative data where real pairs need the unfrozen model (Appendix D). 3.2 One Untuned Recipe Reaches GPT-4o Across Backbones and Benchmarks Table 1: One recipe on three reward benchmarks. Each probe is one frozen linear head at its probe depth, trained once on caption perturbations and evaluated with no re-tuning. Published judges are quoted from each benchmark’s paper; our Multimodal RewardBench rows († ) are over the 4,711 released pairs, which omit the withheld toxicity split (GPT-4o recomputed on them: .728 overall, Gemini 1.5 Pro .734), and safety/bias is a length artifact for every mined-negative head (Section 3.2). VL-RewardBench macro is the mean of its three splits. The 2B and 8B rows use the 60k-pair bank, the 4B and 32B rows the 15k-pair rebuild (Appendix D). Bold: column maximum among our Qwen3-VL rows. The GLM-4.1V-9B rows are a second backbone family under the identical recipe (15k pairs, 200 steps, N−12N-12 of 40 layers, no re-tuning). Model Overall General Knowl. Reasoning Safety VQA Corr. Pref. Math Coding Bias Claude 3.5 Sonnet .720 .626 .678 .739 .686 .651 .768 .856 Gemini 1.5 Pro .720 .635 .677 .663 .689 .555 .945 .872 GPT-4o .715 .626 .690 .720 .676 .621 .748 .872 Ours: 2B probe, N−4N-4 .708† .669 .610 .587 .654 .490 .998 .850 Ours: 4B probe, N−6N-6 .724† .713 .621 .624 .716 .500 .951 .855 Ours: 8B probe, N−12N-12 .659† .735 .638 .635 .708 .579 .187 .860 Ours: 32B probe, N−12N-12 .733† .726 .636 .643 .739 .567 .742 .911 Ours: GLM-4.1V-9B probe, N−12N-12 .692† .636 .602 .514 .562 .612 .935 .857 base 4B (log-likelihood judge) .571† .449 .558 .581 .582 .364 .191 .892 base GLM-4.1V-9B (log-likelihood judge) .565† .440 .495 .573 .611 .404 .167 .892 VL-RewardBench Overall Macro General Halluc. Reason. Gemini-1.5-Pro (strongest published) .672 – – – – GPT-4o (published) .658 – – – – Ours: 32B probe, N−12N-12 .724 .649 .475 .810 .663 Ours: 8B probe, N−12N-12 .682 .607 .425 .765 .631 Ours: 4B probe, N−6N-6 .680 .600 .420 .774 .606 Ours: 2B probe, N−4N-4 .669 .592 .425 .762 .587 Ours: GLM-4.1V-9B probe, N−12N-12 .602 .531 .348 .678 .568 base 8B (log-likelihood judge) .542 .525 .459 .554 .561 base 4B (log-likelihood judge) .525 .518 .464 .522 .568 base 2B (log-likelihood judge) .569 .550 .525 .598 .527 base GLM-4.1V-9B (log-likelihood judge) .570 .529 .425 .616 .546 The 32B probe is at or above GPT-4o on all three benchmarks and separably above it on one (Table 1). On VL-RewardBench it is the strongest entry on the table, 6.6 points above GPT-4o at 3.6 standard errors of the difference; on Multimodal RewardBench and on MMRB2 reasoning the benchmark cannot resolve the margin, and separating either at two standard errors would take 13 and 3.5 times the released pairs. The stronger reference than GPT-4o is the backbone itself as a generative judge, which on MMRB2, the one benchmark we ran it on, leads the probe by 4.7 points (64.6 against 59.9 on the four-task average, Appendix C). In exchange, the probe decodes nothing: it needs one forward pass per response, where the judge generates a judgment in both presentation orders. Larger backbones improve the intermediate-layer probe on all three benchmarks, while the base log-likelihood judge stays near chance and the last-layer head trails at every size (Fig. 7). The recipe is not specific to Qwen3-VL: on GLM-4.1V-9B, a second family with 40 layers, the same recipe at N−12N-12 beats its last layer and its untrained judge on both benchmarks (Table 1). A head trained only on caption perturbations also transfers a usable signal to response-image tokens once the backbone is large enough (MMRB2 editing, level with GPT-4o), which starts Section 3.4. Our training data shares no prompt with the benchmarks and 0.1% of images with Multimodal RewardBench (Appendix B). What the columns mean. The safety/bias column of Multimodal RewardBench is a length artifact, the one reward models are known to carry (Singhal et al., 2024). A head trained on same-image perturbations, two thirds of which match length exactly, scores .998 on it at 2B and .19 at 8B, where its correlation with response length flips sign. The gap is not seed noise: eight layer-sweep seeds give .967±.007.967±.007 at 2B and .569±.013.569±.013 at 8B, the same ordering (Appendix C). We therefore compare heads on the five trainable categories, where the 8B probe’s .659 overall against .708 at 2B becomes .658 against .602. 3.3 Two Probes, One Score, Two Rewards Same benchmark score, different reward. We train two probes on the two extreme quartiles of the same mined-negative data, ranked by the drop in CLIP (Radford et al., 2021) image–text cosine from the real to the perturbed caption, 15k pairs each at 2B. They reach similar accuracy on Multimodal RewardBench, .597 against .603 trainable, and then separate three ways (Fig. 1b): VL-RewardBench hallucination 72.0 against 78.4, reasoning 63.4 against 56.8, and as Best-of-N scorers the CLIP-visible head beats probe-8B on CHAIRs by 3.8 points while losing 2.9 on QA. The same pattern, a small benchmark gap against a large hallucination gap, holds across sizes, consistent with Corollary 1: the two samplers differ from the positives along different coordinates, and the benchmark does not separate them. Whether a scorer helps Best-of-N depends on the pool, not on its benchmark score. Across eight captioning and QA cells and a hard-QA ladder, a probe’s benchmark accuracy does not predict how much it improves Best-of-N selection, in either direction (Tables 6 and F). On captioning every mined-negative probe stays at greedy level where the oracle reaches .06 to .09 CHAIRs, and on MathVista and MMMU, where candidate agreement tracks correctness, majority vote is 1.4 to 3.8 points ahead of the probe, whose one positive role there is as a vote weighter. Where the 8B generator is confidently wrong instead (majority wrong on 32% of prompts), every external scorer beats majority vote: probe-2B by 10.8 points, probe-8B by 7.2 and the training-free log-likelihood judge by 9.2, all with 95% CIs excluding zero. The trained head does not significantly beat the training-free judge: probe-2B against log-likelihood is +1.6+1.6 to +2.1+2.1 points with CIs including zero, and probe-2B beats probe-8B by 3.6 points, the reverse of their benchmark order. What decides the gain is the pool: whether the generator’s majority can be trusted, and so whether any external scorer should replace the vote. The benchmark measures neither. 3.4 Learning from the Policy’s Own Errors: Better Deployment, Lower Benchmarks Figure 4: One turn of the loop. (a) CHAIRs@16 per captioning cell (hatched: verified probe trained without COCO-2). (b) Judging (VL-RewardBench) against ranking (Δ vs greedy). Figure 2b changes one element of the recipe of Section 2.1, the negative sampler: the policy’s own Best-of-N candidates, labelled by a cheap verifier, are paired correct against wrong for the same prompt. We train the head to separate verified successes from failures among the policy’s own outputs, the comparison that Best-of-N selection makes at deployment. It lowers captioning CHAIRs where the perturbation probes did not: a head trained on 3,462 such pairs cuts CHAIRs by 5.1 points against greedy and by 4.1 points on a task it never trained on (Fig. 1c, Fig. 4a, Table 7). On QA it matches probe-8B as a selector, so the utility claim rests on captioning, the open-ended task where no vote is available. CHAIR is both harvest label and metric, so the gain is in absent COCO-object mentions rather than faithfulness at large (Appendix A). The verified head is the weakest judge of the three heads (Fig. 4b) on VL-RewardBench, .615 against .669 and .682 for the perturbation probes, yet the one that ranks the policy’s captions below greedy’s hallucination rate (the other benchmarks in Table 7). Both follow from the sampler: 70% of the verified pairs are QA correctness pairs, so the categories they do not cover fall while math rises. The two rewards differ because the two comparisons do. Benchmark pairs contrast other models’ responses on annotator-chosen errors, close to what perturbations produce; verified pairs contrast this policy’s own successes and failures. In our runs the head trained on the second kind improves candidate selection and loses VL-RewardBench accuracy. Serving both the benchmark and the deployment is open: train on the union of the two samplers, give the head more than one direction, or draw benchmark pairs from the policy under deployment. The actor side of the turn. One DPO turn on pairs the policy generated itself lowers its COCO hallucination rate from .446 to .390 CHAIRs with probe-picked pairs, .414 with verified and .412 with random pairs, at no cost on MathVista or MMMU (Table 8). The label adds little over random pairs, so the selector’s effect on the actor is still to be shown (Appendix G). Figure 5: The loop transposes to image generation: verifier-minted pairs gain 5.4 points over random selection at 8B, while the 2B head trained on caption perturbations loses 6.5. (a) Best-of-N on 160 held-out hard compositional prompts: verifier pass rate of the image each selector picks from N Qwen-Image samples (bands: 75% binomial intervals; paired differences against random at N=12N=12 in the text). (b) One prompt and each selector’s pick out of twelve, judged by the GenEval-style OWLv2 verifier (Ghosh et al., 2023; Minderer et al., 2023) on object presence, count, colour and relative position. The selectors see only the image. The same turn on image generation. The loop transposes to image generation point for point (Fig. 5): Qwen-Image (Qwen Team, 2025a) produces pools of 12 on GenEval-style prompts (Ghosh et al., 2023) and a label-free OWLv2 verifier (Minderer et al., 2023) plays CHAIR’s role. On 160 held-out hard prompts (random .765, oracle .956), the text-trained 2B head is 6.5 points below random (95% CI −11.6-11.6 to −1.6-1.6), and the mined-negative image heads are +2.5+2.5, on par with CLIP (+3.0+3.0). A head trained on 3,000 verifier-minted pairs is +5.4+5.4 at 8B (95% CI +0.5+0.5 to +10.3+10.3), 2.4 points above CLIP and 2.9 above the mined-negative image heads on the same prompts. The two verified heads cross over as MMRB2 judges, so here too the better ranker is the worse judge. 4 Related Work Multimodal reward models and the benchmarks that rank them. A multimodal reward model is built by annotating preference pairs and fitting a scorer on them, from LLaVA-RLHF and RLHF-V (Sun et al., 2024; Yu et al., 2024) to M-RLHF’s 120k human comparisons (Zhang et al., 2025) and the scoring heads of IXC-2.5-Reward and UnifiedReward (Zang et al., 2025; Wang et al., 2025). Removing the annotator usually means a model in their place (Bai et al., 2022; Lee et al., 2024; Li et al., 2024; Yu et al., 2025); we use a sampler, as hallucination-injection DPO does for the policy (Zhou et al., 2024; Zhao et al., 2023b). Multimodal RewardBench (Yasunaga et al., 2025), VL-RewardBench (Li et al., 2025) and MMRB2 (Hu et al., 2025) rank those models and generative judges (Zheng et al., 2023; Xiong et al., 2025) by pairwise accuracy against human labels (Lambert et al., 2025; Malik et al., 2026; Frick et al., 2025, cf.). We take them as given and ask what the number measures. In text, policies optimized toward reward models of similar accuracy can perform quite differently (Wen et al., 2025), while an outcome reward model’s score tracks downstream accuracy better than validation loss does (Qi et al., 2026); we ask when a reward model’s own benchmark score stops tracking deployment. Probing and probe depth. The scorer we set against those judges is a probe, and linear probes are classically an analysis tool for encoders (Alain and Bengio, 2017; Tenney et al., 2019; Burns et al., 2023); intermediate layers often give better features than the last (Skean et al., 2025). LLaVA-Reward (Zhou et al., 2025) is the closest prior use of a probe as a reward, mixing depths through a cross-attention module trained on annotated preferences; ours is a 2,048-parameter head with no label. Density-ratio estimation and negative sampling. A probe on a frozen backbone approximates a density ratio, which noise-contrastive estimation fits against a noise distribution (Gutmann and Hyvärinen, 2010; van den Oord et al., 2018) that the negative-sampler literature designs to shape the representation (He et al., 2020; Robinson et al., 2021; Kalantidis et al., 2020; Wang and Isola, 2020), and in vision-language models as word-order and attribute swaps of the true caption (Yuksekgonul et al., 2023; Hsieh et al., 2023). With frozen features those choices act only through the objective. Closest to our objective is Fan et al. (2026), a label-free in-batch Bradley–Terry reward on text with web continuations as the negative process; their invariance theorem gives a nuisance’s coefficient in the unrestricted optimum, which Proposition 1 restates for our matrix loss. Best-of-N and learning from self-generated data. A reward built from that ratio is then used, like a verifier (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2024), under optimization pressure, and proxy rewards degrade under it (Gao et al., 2023; Coste et al., 2024; Stiennon et al., 2020). We locate an earlier failure, before any pressure is applied, where the proxy’s benchmark score does not predict its Best-of-N value (Beirami et al., 2025) against self-consistency (Wang et al., 2023). Expert iteration and STaR (Anthony et al., 2017; Zelikman et al., 2022; Singh et al., 2024) improve a policy on its own verified outputs, and GKD (Agarwal et al., 2024) diagnoses the train–inference mismatch on the policy side; ours is the reward-side analogue. 5 Concluding Remarks Frozen vision-language representations support strong reward modeling with only a linear head and caption perturbations, but probe depth and training comparisons determine which signals the head extracts. Similar benchmark scores can conceal different specializations, and higher preference accuracy need not yield better Best-of-N selection. Retraining on verified policy outputs improves caption selection while lowering benchmark accuracy, so the supervision that helps deployment is drawn from the policy’s own errors. Anyone choosing a reward model should test how well it picks the best of their own policy’s samples, and a paper that reports one should state which pairs it was trained on. Acknowledgement S.K. acknowledges the support from the National Science Foundation Grant under award IIS 2229881; H.Z. and S.K. acknowledge the Chan Zuckerberg Initiative Foundation for establishing the Kempner Institute for the Study of Natural and Artificial Intelligence. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In ICLR, Cited by: §4. Alain and Bengio (2017) G. Alain and Y. Bengio Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations (ICLR) Workshop, Cited by: §4. Anthony et al. (2017) T. Anthony, Z. Tian, and D. Barber Thinking fast and slow with deep learning and tree search. In NeurIPS, Cited by: §4. Bai et al. (2022) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional ai: harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. Cited by: §2.1, §4. Beirami et al. (2025) A. Beirami, A. Agarwal, J. Berant, A. D’Amour, J. Eisenstein, C. Nagpal, and A. T. Suresh Theoretical guarantees on the best-of-n alignment policy. In International Conference on Machine Learning (ICML), Cited by: §4. Bradley and Terry (1952) R. A. Bradley and M. E. Terry Rank analysis of incomplete block designs: i. the method of paired comparisons. Biometrika 39 (3/4), p. 324–345. Cited by: §1. Burns et al. (2023) C. Burns, H. Ye, D. Klein, and J. Steinhardt Discovering latent knowledge in language models without supervision. In International Conference on Learning Representations (ICLR), Cited by: §4. Christiano et al. (2017) P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §4. Coste et al. (2024) T. Coste, U. Anwar, R. Kirk, and D. Krueger Reward model ensembles help mitigate overoptimization. In International Conference on Learning Representations (ICLR), Cited by: §4. Deitke et al. (2024) M. Deitke, C. Clark, S. Lee, et al. Molmo and pixmo: open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146. Cited by: Appendix B, §2.3. Fan et al. (2026) J. Fan, Y. Li, Z. Qi, D. Zhang, K. Brantley, S. M. Kakade, and H. Zhang Scaling reward modeling without human supervision. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Appendix G, Appendix I, §1, §4. Frick et al. (2025) E. Frick, T. Li, C. Chen, W. Chiang, A. N. Angelopoulos, J. Jiao, B. Zhu, J. E. Gonzalez, and I. Stoica How to evaluate reward models for RLHF. In International Conference on Learning Representations (ICLR), Cited by: §4. Gao et al. (2023) L. Gao, J. Schulman, and J. Hilton Scaling laws for reward model overoptimization. In ICML, Cited by: §4. Ghosh et al. (2023) D. Ghosh, H. Hajishirzi, and L. Schmidt GenEval: an object-focused framework for evaluating text-to-image alignment. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks, Cited by: Figure 5, §3.4. Gutmann and Hyvärinen (2010) M. Gutmann and A. Hyvärinen Noise-contrastive estimation: a new estimation principle for unnormalized statistical models. In AISTATS, Cited by: §1, §2.2, §4. He et al. (2020) K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick Momentum contrast for unsupervised visual representation learning. In CVPR, Cited by: §4. Hendrycks and Dietterich (2019) D. Hendrycks and T. Dietterich Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), Cited by: Figure 6, §2.1. Hsieh et al. (2023) C. Hsieh, J. Zhang, Z. Ma, A. Kembhavi, and R. Krishna SugarCrepe: fixing hackable benchmarks for vision-language compositionality. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, Cited by: §4. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), Cited by: Appendix G. Hu et al. (2025) Y. Hu, R. Askari-Hemmat, M. Hall, E. Dinan, L. Zettlemoyer, and M. Ghazvininejad Multimodal rewardbench 2: evaluating omni reward models for interleaved text and image. arXiv preprint arXiv:2512.16899. Cited by: Appendix B, §1, §2.3, §4. Kalantidis et al. (2020) Y. Kalantidis, M. B. Sariyildiz, N. Pion, P. Weinzaepfel, and D. Larlus Hard negative mixing for contrastive learning. In NeurIPS, Cited by: §4. Lambert et al. (2025) N. Lambert, V. Pyatkin, J. Morrison, et al. RewardBench: evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: NAACL 2025, p. 1755–1797. Cited by: §4. LeCun et al. (2006) Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. J. Huang A tutorial on energy-based learning. In Predicting Structured Data, Cited by: §2.2. Lee et al. (2024) H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, and S. Prakash RLAIF vs. rlhf: scaling reinforcement learning from human feedback with ai feedback. In International Conference on Machine Learning (ICML), Cited by: §2.1, §4. Li et al. (2025) L. Li, Y. Wei, Z. Xie, et al. VL-rewardbench: a challenging benchmark for vision-language generative reward models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Appendix B, §1, §2.3, §4. Li et al. (2024) L. Li, Z. Xie, M. Li, S. Chen, P. Wang, L. Chen, Y. Yang, B. Wang, L. Kong, and Q. Liu VLFeedback: a large-scale AI feedback dataset for large vision-language models alignment. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §4. Li et al. (2023) Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen Evaluating object hallucination in large vision-language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: Appendix A. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations (ICLR), Cited by: §4. Lin et al. (2014) T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick Microsoft coco: common objects in context. In European Conference on Computer Vision (ECCV), Cited by: §2.3. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, Cited by: Appendix J. Lu et al. (2024) P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K. Chang, M. Galley, and J. Gao MathVista: evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), Cited by: §2.3. Ma and Collins (2018) Z. Ma and M. Collins Noise contrastive estimation and negative sampling for conditional models: consistency and statistical efficiency. In EMNLP, Cited by: §1, §2.2. Malik et al. (2026) S. Malik, V. Pyatkin, S. Land, J. Morrison, N. A. Smith, H. Hajishirzi, and N. Lambert RewardBench 2: advancing reward model evaluation. In International Conference on Learning Representations (ICLR), Cited by: §4. Minderer et al. (2023) M. Minderer, A. Gritsenko, and N. Houlsby Scaling open-vocabulary object detection. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Figure 5, §3.4. OpenAI (2024) OpenAI GPT-4o system card. arXiv preprint arXiv:2410.21276. Cited by: §2.3. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1. Qi et al. (2026) Z. Qi, F. Nie, A. Alahi, J. Zou, H. Lakkaraju, Y. Du, E. Xing, S. Kakade, and H. Zhang EvoLM: in search of lost training dynamics for language model reasoning. Advances in Neural Information Processing Systems 38, p. 82906–82937. Cited by: §4. Qwen Team (2025a) Qwen Team Qwen-image technical report. arXiv preprint arXiv:2508.02324. Cited by: Appendix G, §3.4. Qwen Team (2025b) Qwen Team Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Note: Qwen3-VL-2B/8B-Instruct vision-language models Cited by: §2.1. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), Cited by: §3.3. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Appendix G, §1, §1. Robinson et al. (2021) J. Robinson, C. Chuang, S. Sra, and S. Jegelka Contrastive learning with hard negative samples. In ICLR, Cited by: §4. Rohrbach et al. (2018) A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko Object hallucination in image captioning. In EMNLP, Cited by: §1, §2.3. Singh et al. (2024) A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P. J. Liu, et al. Beyond human data: scaling self-training for problem-solving with language models. Transactions on Machine Learning Research (TMLR). Cited by: §4. Singhal et al. (2024) P. Singhal, T. Goyal, J. Xu, and G. Durrett A long way to go: investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716. Cited by: §2.1, §3.2. Skean et al. (2025) O. Skean, M. R. Arefin, D. Zhao, N. Patel, J. Naghiyev, Y. LeCun, and R. Shwartz-Ziv Layer by layer: uncovering hidden representations in language models. In International Conference on Machine Learning (ICML), Cited by: §4. Stiennon et al. (2020) N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano Learning to summarize from human feedback. In NeurIPS, Cited by: §4. Sun et al. (2025) H. Sun, Y. Shen, and J. Ton Rethinking reward modeling in preference-based large language model alignment. In International Conference on Learning Representations (ICLR), Note: arXiv:2411.04991 Cited by: §1. Sun et al. (2024) Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y. Shen, C. Gan, L. Gui, Y. Wang, Y. Yang, K. Keutzer, and T. Darrell Aligning large multimodal models with factually augmented RLHF. In Findings of the Association for Computational Linguistics: ACL 2024, p. 13088–13110. Cited by: §4. Tenney et al. (2019) I. Tenney, D. Das, and E. Pavlick BERT rediscovers the classical nlp pipeline. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §4. Uesato et al. (2022) J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins Solving math word problems with process- and outcome-based feedback. arXiv preprint arXiv:2211.14275. Cited by: §4. van den Oord et al. (2018) A. van den Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §2.2, §4. Wallace et al. (2024) B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik Diffusion model alignment using direct preference optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Appendix G. Wang and Isola (2020) T. Wang and P. Isola Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, Cited by: §4. Wang et al. (2023) X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. In ICLR, Cited by: §4. Wang et al. (2025) Y. Wang, Y. Zang, H. Li, C. Jin, and J. Wang Unified reward model for multimodal understanding and generation. arXiv preprint arXiv:2503.05236. Cited by: §4. Wen et al. (2025) X. Wen, J. Lou, Y. Lu, H. Lin, X. Yu, X. Lu, B. He, X. Han, D. Zhang, and L. Sun Rethinking reward model evaluation: are we barking up the wrong tree?. In International Conference on Learning Representations (ICLR), Cited by: §4. Xie and Li (2024) S. Xie and Z. Li Implicit bias of AdamW: ℓ∞ _∞-norm constrained optimization. In International Conference on Machine Learning (ICML), Cited by: Appendix J, §2.2. Xiong et al. (2025) T. Xiong, X. Wang, D. Guo, Q. Ye, H. Fan, Q. Gu, H. Huang, and C. Li LLaVA-Critic: learning to evaluate multimodal models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4. Yasunaga et al. (2025) M. Yasunaga, L. Zettlemoyer, and M. Ghazvininejad Multimodal rewardbench: holistic evaluation of reward models for vision language models. arXiv preprint arXiv:2502.14191. Cited by: Appendix B, §1, §2.3, §4. Yu et al. (2024) T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H. Zheng, M. Sun, and T. Chua RLHF-v: towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4. Yu et al. (2025) T. Yu, H. Zhang, Y. Yao, et al. RLAIF-V: open-source AI feedback leads to super GPT-4V trustworthiness. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Appendix B, §1, §2.1, §2.3, §4. Yue et al. (2024) X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. MMMU: a massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.3. Yuksekgonul et al. (2023) M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou When and why vision-language models behave like bags-of-words, and what to do about it?. In International Conference on Learning Representations (ICLR), Cited by: §4. Zang et al. (2025) Y. Zang, X. Dong, P. Zhang, Y. Cao, Z. Liu, S. Ding, S. Wu, Y. Ma, H. Duan, W. Zhang, K. Chen, D. Lin, and J. Wang InternLM-xcomposer2.5-reward: a simple yet effective multi-modal reward model. In Findings of the Association for Computational Linguistics (ACL), Cited by: §4. Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman STaR: bootstrapping reasoning with reasoning. In NeurIPS, Cited by: §4. Zhang (2026) D. Zhang What Is RLCD? The Secret Behind Jev. Note: Blog post, https://di-zhang-llm.github.io/blog/what-is-rlcd-the-secret-behind-jevPosted September 21, 2026; accessed September 25, 2026 Cited by: §1. Zhang et al. (2025) Y. Zhang, T. Yu, H. Tian, C. Fu, P. Li, J. Zeng, W. Xie, Y. Shi, H. Zhang, J. Wu, et al. M-RLHF: the next step forward in multimodal LLM alignment. In International Conference on Machine Learning (ICML), Cited by: §4. Zhao et al. (2023a) Y. Zhao, A. Gu, R. Varma, et al. PyTorch fsdp: experiences on scaling fully sharded data parallel. Proceedings of the VLDB Endowment. Cited by: Appendix D. Zhao et al. (2023b) Z. Zhao, B. Wang, L. Ouyang, X. Dong, J. Wang, and C. He Beyond hallucinations: enhancing LVLMs through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839. Cited by: §4. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, Cited by: §4. Zhou et al. (2025) S. Zhou, R. Zhang, H. Zhu, B. Kveton, Y. Zhou, J. Gu, J. Chen, and C. Chen Multimodal llms as customized reward models for text-to-image generation. In IEEE/CVF International Conference on Computer Vision (ICCV), Cited by: §4. Zhou et al. (2024) Y. Zhou, C. Cui, R. Rafailov, C. Finn, and H. Yao Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411. Cited by: §4. Appendix Contents 1 Introduction 2 Methodology 2.1 Algorithm 2.2 What the Objective Identifies 2.3 Experimental Setup 3 Experiments 3.1 Two Choices Set the Accuracy: The Probe Layer and the Sampler 3.2 One Untuned Recipe Reaches GPT-4o Across Backbones and Benchmarks 3.3 Two Probes, One Score, Two Rewards 3.4 Learning from the Policy’s Own Errors: Better Deployment, Lower Benchmarks 4 Related Work 5 Concluding Remarks References A Limitations and Broader Impacts B Reproducibility C Evaluation Protocol D Negative Mining and Probe Depth E Extended Results for Reward-Model Training F Extended Results for Best-of-N G Actor Training H CLIP Studies I Reading the Two Statements I.1 What Constructed Comparisons Supervise I.2 Independent Negatives: Density Ratios and PMI I.3 When Perturbations Expose an Additional Quality Signal J Proofs K Compute Resources Appendix A Limitations and Broader Impacts Our approach lowers the additional training and preference-annotation requirements for multimodal reward modeling by learning a small scoring head over a frozen backbone. This may make task-specific reward models more accessible to researchers and practitioners with limited annotation budgets, while retaining the inference cost of the underlying backbone. Constructing training comparisons explicitly also makes the intended supervision easier to inspect and revise. The broader implication concerns how reward models are evaluated and selected. Our experiments show that higher preference-benchmark accuracy need not yield better selection of a policy’s outputs. Reporting the training comparison distribution alongside deployment-specific evaluations can therefore support more informed model selection. Conversely, using benchmark performance alone to justify deployment risks overlooking systematic errors and shortcuts, including response-length biases. These benefits have a limited demonstrated scope. Our experiments cover two backbone families, and verifier-based adaptation is evaluated for one round. Captioning improvements are measured with CHAIR, which also supplies the training labels; they establish fewer absent COCO-object mentions rather than general factuality or preserved informativeness, and a CHAIR-independent metric such as POPE (Li et al., 2023) would be needed to settle the latter. Best-of-N cells are single-seed, the hard-QA set was written and verified by one LLM family, and our Multimodal RewardBench overall omits the withheld toxicity split (Section 3.2). The method does not establish safety or fairness capabilities, and broader deployment requires independent evaluation of output quality, coverage, and task-specific failure modes. Appendix B Reproducibility The recipe is fully specified by Eqs. 1, 2 and 3 and the hyper-parameters of Section 2.3 and Appendix D: a frozen Qwen3-VL checkpoint, a probe layer, mean pooling over response tokens, and one bias-free linear head trained with AdamW for 200 steps. Every input is public. Positives are PixMo-Cap captions (Deitke et al., 2024); negatives are string edits of those captions with no model in the loop, so the 22k-pair bank rebuilds in 10 seconds on one CPU core from the operator definitions in Appendix D. The curated comparison set is public RLAIF-V (Yu et al., 2025), and all three evaluation benchmarks (Yasunaga et al., 2025; Li et al., 2025; Hu et al., 2025) are used as released, with the pair counts and the scoring construction stated per benchmark in Section 2.3. Contamination is checked by a 256-bit difference hash of every training image against every evaluation image, counting a pair as a near-duplicate at ≤10≤ 10 differing bits, plus exact prompt matching. Against Multimodal RewardBench the perturbation set shares 5 near-duplicate images of 4,711 (0.1%) and no prompt; against VL-RewardBench, the 1,000 MMRB2 prompt images and the 1,000 COCO Best-of-N images it shares none. The image arm renders 1,212 PixMo-derived prompts with Qwen-Image, so its training images are generated and its prompts are not those of MMRB2. The same audit exposes the curated baseline: RLAIF-V pairs overlap VL-RewardBench on 326 of 1,247 rows and Multimodal RewardBench on 18, which is why every RLAIF-V-trained row also reports the decontaminated subset. Source for pair construction, training, every selector in the Best-of-N tables and the plotting scripts behind every figure will be released with the paper. Appendix C Evaluation Protocol Multimodal RewardBench releases 4,711 of its 5,211 pairs, the toxicity split being withheld, in eight categories; our overall number is therefore comparable to published ones only through the recomputation of Table 1. VL-RewardBench has 1,247 pairs (general 181, hallucination 749, reasoning 317), of which 326 overlap the RLAIF-V training set, so we also score anything trained on RLAIF-V on the 921 clean rows. On the 326 overlapping rows the RLAIF-V fine-tunes score .80 against .60 to .64 for models that never saw the data; on the clean rows their lead disappears. MMRB2 has 1,000 judge-labelled pairs per task (text-to-image, editing, interleaved generation, reasoning; 996 reasoning pairs are scoreable). We score it with the prompt in the user turn and the response, images included, in the assistant turn, the construction the heads were trained on. A scalar reward is order-invariant, so its accuracy equals MMRB2’s dual-order accuracy; exact ties count 0.5. We also report reasoning on the 350-pair text-only-output subset, the convention that benchmark’s paper uses for scalar reward models. On the four-task average under this construction the 32B probe scores 59.9 and the same backbone used as a generative judge 64.6; GPT-4o as a judge scores 59.7. Scoring prompt and response in one user turn instead costs the text heads 2 to 3 points on reasoning and 5 to 9 on its text-only subset. The GPT-4o margins in Table 1 are quoted as unpaired standard errors because the benchmark papers do not release per-item predictions, so a paired interval is not available. Shaded bands in every figure are 75% binomial intervals of the plotted accuracy over its evaluation set; seed-to-seed spread of the heads is narrower than the line width. Eight seeds of the layer-sweep heads put the safety/bias column of Section 3.2 at .967±.007.967±.007 at 2B and .569±.013.569±.013 at 8B, where the correlation between head score and response length is −0.57-0.57 at 2B and +0.44+0.44 at 8B. The Best-of-N intervals of Section 3.3 are over prompts at one selector seed. The Best-of-N prompts are 500 each of COCO captioning (two prompt sets, COCO and COCO-2), MathVista and MMMU, plus 500 Claude-written hard questions on COCO images, of which 241 are counting or spatial; candidates are sampled at temperature 1.0 and top-p 0.95, plus the greedy decode. The mid-training runs of Section 3.4 continue from the mined-negative probe on 3,462 verified pairs harvested from those pools (MathVista 1,352, MMMU 1,072, COCO 511, COCO-2 527), with leave-one-task-out runs that drop every COCO-2 or every MMMU pair, and a non-convex check that unfreezes the backbone on 15k RLAIF-V pairs from either checkpoint. Heads train with AdamW, learning rate 10−310^-3, weight decay 0.010.01, β=0.1β=0.1, micro-batch 8 and effective batch 256 or 1,024 as stated; the backbones have 28, 36, 36 and 64 layers. Appendix D Negative Mining and Probe Depth Two pair banks appear in the benchmark tables: the 2B and 8B rows of Table 1 use the original 60k-pair bank, the 4B and 32B rows and every ablation the rebuilt 15k-pair bank, which costs the 2B probe 2 to 3 points (.708 to .682 on Multimodal RewardBench), so the 32B row leads from the smaller bank. Perturbation density does not move either benchmark: one, three or all editable spans stay within .008 on both benchmarks (Table 4) while the training task gets much easier. The three token-edit heads beat the mixed-operator recipe on VL-RewardBench by 2.8 to 3.4 points at equal Multimodal RewardBench accuracy (Table 3). That comparison also drops the cross-image and the length-changing operators, so it does not isolate the mix. Table 2: Every ablation and analysis of the paper on one backbone (Qwen3-VL-4B). Frozen linear head, mean pooling, probe on layer N−6N-6 unless stated, 200 steps of 256 pairs, single seed, no re-tuning between benchmarks. MMRB1 = Multimodal RewardBench (4,711 pairs; trainable = the five categories without safety/bias and VQA); VL-RB = VL-RewardBench (1,247 pairs, hallucination split 749). The Δ -CLIP heads are 5k-pair quartiles of the same perturbations split by whether CLIP can see the edit; the verified heads train on 2,400 verifier-minted pairs from a fresh head or from the mined-negative head; the base judge scores by response log-likelihood with no training. setting MMRB1 overall MMRB1 trainable VL-RB VL-RB halluc. mined-negative probe, N−6N-6 (recipe) .724 .633 .680 .774 last-layer control .696 .575 .659 .700 frozen head on RLAIF-V 15k .676 .598 .618 .677 Δ -CLIP blind / visible .687 / .707 .615 / .631 .656 / .673 .721 / .765 verified, fresh / mid-training .705 / .723 .615 / .632 .616 / .690 .650 / .762 base log-likelihood judge .571 .507 .525 .522 Figure 6: Negative mining by perturbation, laid out after ImageNet-C (Hendrycks and Dietterich, 2019). (a) A PixMo image and three operators applied to its human caption: attribute swaps every colour, number and spatial word; shuffle reorders and drops sentences; splice continues the true caption with another image’s. Red marks the edit, orange the foreign text. (b) The image-response arm perturbs the prompt instead and renders both prompts with Qwen-Image under the same seed, so the two prompts differ only in the edited span. Table 3: Negative-mining ablation in both modalities. Top: text responses, 2B backbone. The frozen-head rows use 15k pairs at N−4N-4 on the official benchmarks; the two full-fine-tune rows are the June series that first separated the negative types (overall Multimodal RewardBench; their safety/bias column reads .224 and .451, the length shortcut). Bottom: image responses, 2B image-trained head at N−12N-12 on 1,091 same-seed prompt pairs; held-out is accuracy on the head’s own negative type, MMRB2 text-to-image is judge agreement (label prior 53.2). The by kind row splits held-out accuracy of the combined head by the planted edit. text negatives y−y^- head val MMRB1 trainable VL-RB halluc. cross-image only, ×T_× full FT .99 .559 – – – same-image perturbations, mixed full FT – .582 – – – token edit, k=1k=1 span, nearT_near frozen head .820 .673 .574 .709 .781 token edit, k=3k=3 spans frozen head .973 .673 .563 .713 .797 token edit, all spans (13.6) frozen head .996 .681 .576 .707 .797 mixed operators, 15% ×T_× (recipe) frozen head .92 .682 .584 .679 .769 image negatives y−y^- held-out, own type MMRB2 T2I cross-image only, G(q′,s′)G(q ;s ) .93–1.00 52.7 planted edit only, G((q),s)G(T(q);s) .79–.85 56.0 both (recipe) .79–.84 56.2 by kind: object swap / near-miss / multi-edit 2B .944 / .714 / .837 8B .917 / .786 / .884 Table 4: Perturbation density, k=1k=1 / 3 / all editable spans per rejected caption (all == 13.6 edits on average), 21,521 identical images and captions per setting, token substitutions only. Frozen 2B N−4N-4 head, 200 steps of 1,024, single seed. The training task gets much easier (val .82 → .996) and neither benchmark moves. setting val MMRB1 VL-RewardBench overall overall macro general halluc. k=1k=1 (1.0 edit) .820 .673 .709 .665 .630 .781 k=3k=3 (3.0 edits) .973 .673 .713 .660 .613 .797 k=k=all (13.6 edits) .996 .681 .707 .644 .547 .797 ref: mixed-operator probe – .682 .679 – – .769 Table 5: Mined negatives against curated RLAIF-V pairs at a matched pair budget, 2B backbone, 15k pairs per setting. Frozen heads: N−4N-4, 200 steps of 1,024 pairs. Full fine-tune: lr 10−510^-5, 400 steps of 256, vision tower frozen. The tokens column is the response-token count of the 15k pairs (both sides; Qwen3-VL tokenizer): perturbation captions are longer than RLAIF-V responses (204 against 86 tokens per side), so the RLAIF-V heads see fewer tokens at the same pair count. The val column is in-distribution pair accuracy on each setting’s own held-out pairs. 326 of the 1,247 VL-RewardBench rows overlap the RLAIF-V training set, so RLAIF-V models are also read on the 921 clean rows; mined-negative pairs overlap none. Single seed; the comparison is matched in pairs and not in tokens. training pairs (15k) head tokens val MMRB1 trainable VL-RB VL-RB clean perturbations, token-edit frozen head 6.4M .820 .673 .574 .709 .747 perturbations, mixed operators frozen head 6.1M .92 .682 .584 .679 – RLAIF-V frozen head 2.6M .551 .577 .544 .660 .672 RLAIF-V full FT 2.6M .637 .686 .613 .735 .727 base 2B (log-likelihood judge) – – – .563 – .569 – Figure 3 gives the depth sweep per model, Table 3 the operator ablation in both modalities, Table 4 the density variants and Table 5 the matched-budget comparison against RLAIF-V. On that matched budget the mined-negative head reads .574 to .584 trainable on Multimodal RewardBench against .544 for RLAIF-V and .679 to .709 on VL-RewardBench against .660, with hallucination .769 to .781 against .737; its in-distribution validation accuracy is .82 to .92 against .55 for the curated pairs. The comparison is matched in pairs and not in tokens, since perturbation captions run 2.4 times longer, which leaves the RLAIF-V head 2.6M response tokens against 6.1M. Unfreezing the backbone on the same 15k real pairs gains 11 points overall and 7 trainable, to .613, which puts it above the mined-negative probe on the trainable categories and level with it on decontaminated VL-RewardBench (.727 against .747). Perturbation operators act on one caption at a time: a near-miss substitution of a colour, number or spatial word; a splice of another caption’s tail; a clause shuffle; and, for 15% of pairs, the caption of another image. In the 15k-pair bank the operators and their share are near-miss substitution 50%, attribute substitution 15%, splice 17%, shuffle 10% and cross-image 8%. Measured as the ratio of rejected to chosen length in words, the two substitutions leave length unchanged (1.00), splice shortens to 0.82, shuffle to 0.69 (it drops about 30% of sentences) and cross-image captions are 1.10 times as long, so across the bank a negative is about 5% shorter than its positive. On GLM-4.1V-9B the same recipe at the last layer scores .628 on Multimodal RewardBench and .598 on VL-RewardBench against .692 and .602 at N−12N-12 (Table 1); the untrained judge scores .565 and .570. Density variants perturb one, three or all editable spans (13.6 on average) of the same 21,521 captions. Heads train with AdamW, learning rate 10−310^-3, weight decay 0.010.01, β=0.1β=0.1, micro-batch 8, effective batch 256 or 1,024 as stated, 200 steps, on precomputed vision-tower outputs; the 32B confirm run used FSDP (Zhao et al., 2023a) over eight GPUs with 1,024 effective. The layer sweep trains a linear head on the precomputed hidden states of each layer for 8k pairs and five seeds; it ranks depths within a model and nothing else, and it misleads in two documented ways. Last-token pooling ranks first in the sweep at 4B, 8B and 32B and loses 7.5 points to mean pooling in the one real run. Sweep heads trained under a different regime than the real pipeline disagree with it by several points, in either direction. Appendix E Extended Results for Reward-Model Training Figure 7: Three reward benchmarks against backbone size, by component (shaded: confidence bands). The linear probe at N−kN-k improves with size on all three and beats the last-layer probe at every size. The full fine-tune matches it on MMRB1 only from 8B up and trails it on VL-RewardBench at 32B (early-stopped at step 150 of 400). The RLAIF-V probe trails on MMRB1 everywhere and leads on VL-RewardBench only at 32B, where the two share source images. Figure 7 separates the components behind Table 1 at each backbone size. Appendix F Extended Results for Best-of-N Figure 8: Accuracy of the selected candidate against N on 500 hard questions and the 241-question counting/spatial subset, for the 2B and 8B generators, with every training-free control on every panel. Where the generator is confidently wrong (8B) every external scorer beats majority vote; on the 2B pool probe, log-likelihood and vote are within noise. probe-8B omitted (it tracks probe-2B); its numbers are in Appendix F. Table 6: Best-of-16 on hard verifiable QA (500 questions on COCO images, and the 241-question counting/spatial subset), accuracy of the selected candidate for each selector. greedy is N=1N=1; oracle is pass@16. Paired bootstrap over prompts; single seed per selector. Where the 8B generator is confidently wrong (mean top-share .85) every external scorer beats majority vote. On the 2B pool, the probes and log-likelihood are within noise of each other and of the vote. cell greedy random shortest loglik-2B probe-2B probe-8B majority wvote oracle hard QA, gen 2B .684 .672 .580 .714 .742 .762 .736 .752 .940 hard QA, gen 8B .648 .648 .448 .760 .776 .740 .668 .742 .884 counting/spatial, gen 2B .535 .502 .440 .577 .631 .622 .606 .627 .929 counting/spatial, gen 8B .481 .436 .253 .602 .622 .610 .485 .614 .809 Table 7: One turn of the loop, 2B. Top: CHAIRs at N=16N=16 per captioning cell (lower is better); verified is the head trained on 3,462 verifier-minted pairs, held-out the same recipe with every COCO-2 pair excluded, scored on COCO-2. Pooled paired bootstrap against greedy: −5.1-5.1 CHAIR points, 95% CI −7.1-7.1 to −3.1-3.1; held-out −4.1-4.1, 95% CI −6.9-6.9 to −1.1-1.1. Bottom: the same heads as judges on the two unseen reward benchmarks, against the mined-negative probe they replace (the 15k-pair rebuild that scores every Best-of-N cell). The best ranker is the worst recent judge. captioning cell greedy probe-8B verified held-out oracle COCO, gen 2B .346 .376 .312 – .066 COCO, gen 8B .374 .400 .332 – .090 COCO-2, gen 2B .362 .396 .294 .324 .056 COCO-2, gen 8B .402 .414 .344 .358 .094 judge VL-RB halluc. MMRB2 reas. verified, fresh init .615 68.8 .539 verified, perturbation init .615 68.9 .540 mined-negative probe .679 76.9 .533 Figure 8 and Table 6 give the hard-QA ladder with every control on every panel and Table 7 the loop turn per captioning cell. Initializing the verified head from the mined-negative probe matters only when the pair budget is too small to reach the optimum. At 2B a fresh head and a continued one score within 0.3 points of each other everywhere (Table 7); at 4B, where 2,400 pairs do not reach the optimum, the continued head is 7 points ahead (Table 2). A planted-decoy test (3.4 LLM-written single-error decoys and 4 template perturbations per captioning pool) shows that the mined-negative feature fires inside a real pool: every scorer ranks a source above its own one-edit decoy at least 93% of the time. The margin, however, is 0.3 to 1.1 within-pool standard deviations, and CHAIRs of the pick does not move. The probe therefore detects that an edited sentence is worse than its source but cannot tell which of 16 real captions is the most truthful. Truth-preserving edits separate the signals: the probes do not prefer the source over a true edit (probe-2B .512, probe-8B .291) while log-likelihood prefers it 91% of the time. Model-written negatives. Two 2B heads were trained with the policy’s own captions in the pair instead of a string edit, same recipe otherwise. Self-corruption (chosen: the policy’s caption; rejected: its perturbation) ties greedy and random in all four captioning cells and scores .689 on Multimodal RewardBench, the same as the off-policy head. Human-versus-policy (chosen: the PixMo caption; rejected: the policy’s caption for the same image) scores .360 on Multimodal RewardBench, below chance, and picks a more hallucinated caption than random in all four cells (CHAIRs@16 .422 / .432 / .414 / .422 against random .370 / .384 / .406 / .402). This head learned to tell a human author from a model; on a benchmark where both responses are model-written, that direction anti-correlates with the label. Appendix G Actor Training We train the 2B policy on its own selected candidates against a size-matched random-selection control. DPO (Rafailov et al., 2023) uses β=0.1β=0.1, learning rate 10−610^-6, two epochs, and three corpora from the same 500 prompts per task: verifier-minted pairs, probe-8B argmax-versus-argmin pairs, and random pairs as the control. Policies are re-evaluated greedily on all four tasks. The actor half of the turn shows no selector effect, though it has a safe recipe. DPO costs nothing on QA: no change is outside ±1.8± 1.8 points, and the per-object hallucination rate on COCO falls for every corpus (CHAIRi .083 to .073, .079 and .082 for the verified, RM-picked and random pairs), with the label adding little over random pairs (Table 8). The size-matched random control plays the role of the untrained-initialization control of Fan et al. (2026). The RM-picked corpus is the only one that shortens captions (7%), yet its per-object rate also falls, so the change is not purely length. One DPO turn on self-generated pairs is a mild, label-agnostic hallucination reducer, and the actor side of the loop still lacks a demonstrated selector effect. Table 8: Actor, DPO (2B policy). Full-parameter DPO (β=0.1β=0.1, lr 10−610^-6, 2 epochs, vision tower frozen) on three corpora from the same 500 prompts × 4 tasks: verified (3,462 pairs), rm (probe-8B argmax vs argmin, 2,000), random (two random candidates, 1,999; the control). Greedy regeneration at 512 tokens; Δ against the regenerated base, paired bootstrap over prompts (2,000 resamples), every QA CI including zero. CHAIRi is hallucinated object mentions over all object mentions, pooled over the 500 COCO captions, which shorter captions do not lower by themselves. Best value per column in bold. COCO MathVista MMMU setting words CHAIRi ↓ acc (Δ ) acc (Δ ) base (regenerated) 245 .083 .546 .334 verified (3,462 pairs) 242 .073 .542 (−.004-.004) .346 (+.012+.012) rm (2,000) 228 .079 .528 (−.018-.018) .338 (+.004+.004) random (1,999, control) 245 .082 .534 (−.012-.012) .332 (−.002-.002) The DPO base policy was regenerated at 512 new tokens under the identical protocol. Between 19 and 27% of DPO pairs were skipped by the 1,024-token cap, mostly long QA chains of thought, so the QA share of each corpus is below nominal; the update was small (loss .69 to .64 to .68). The image-generation counterpart (LoRA (Hu et al., 2022) on Qwen-Image (Qwen Team, 2025a) for 160 steps on 640 on-policy images, across seven Diffusion-DPO (Wallace et al., 2024) and related fine-tuning variants) has the same shape. No variant differs from the base generator on the hard prompts (all Δ within [−.04,+.07][-.04,+.07]), and every selection variant minus its random control is inside ±2± 2 points. Appendix H CLIP Studies Figure 9: CLIP view of the perturbation pairs (22,000 same-image pairs, CLIP ViT-B/32). (a) Image–caption cosine of the real and the perturbed caption; the perturbation lowers the median from .303 to .294 and adds a low-score tail. (b) The drop Δ per pair, which the dissociation split of Section 3.3 ranks on, on a log count axis; the dashed line is the top-quartile cut (Δ≥.008 ≥.008) that defines the CLIP-visible arm. CLIP accepts 77 tokens and the captions average 150 words, so for 43% of pairs the edit falls beyond its window and Δ=0 =0 exactly; the CLIP-blind quartile is drawn from those pairs and is omitted from the histogram. Figure 9 shows the score distribution the quartile cuts are taken from. Filtering the positives by image–caption CLIP score gives .685 / .698 / .678 on Multimodal RewardBench for the top, random and bottom quartile and .670 / .686 / .682 on VL-RewardBench. By contrast, the Δ split ranks each pair by the drop in CLIP cosine from the real to the perturbed caption. The CLIP-blind quartile has near-zero drop, part of it because CLIP truncates at 77 tokens and the edit falls outside the window. Appendix I Reading the Two Statements The convexity fact of Section 2.2 predicts that, when gradient descent runs on a fixed-penalty objective, initialization, step size and step count cannot move the minimizer. It predicts nothing about AdamW, whose stationary points depend on its preconditioner, and nothing about the negative pool, the hardness schedule or the temperature, each of which changes the population objective rather than the path to its minimizer. The temperature is the least obvious of these: writing u=βwu=β w turns the weight decay into λ2β2∥u∥2 λ2β^2 u ^2, so β acts as an effective regularization strength that rescales w⋆w and tilts its direction. That those interventions nonetheless leave the benchmark where it was (Section 3.1) is an empirical finding, and Eq. 5 explains it. Each intervention re-weights the same difference vectors, and on the benchmark’s pairs the re-weighted heads order alike. The CLIP split of Section 3.3 is the same equation evaluated at a re-weighting large enough to separate on hallucination. The value at the minimizer is the best linear separation of positives from negatives in the layer N−kN-k feature space, which is consistent with depth moving the plateau while optimizer settings do not; convexity fixes the endpoint of the fixed-penalty problem and not the distance to it, which is where initialization re-enters (Section 3.4). Proposition 1 is stated for the decoupled objective because that is the one with a closed-form optimum, and for mutually absolutely continuous P and Q because that is what keeps the log ratio finite. Our sampler does not satisfy the hypothesis: a caption of another image has probability zero as a caption of this one, so on those pairs the unrestricted optimum runs to ±∞±∞ and separates the two laws with unbounded margin. The in-distribution accuracy of the recipe is the finite-sample face of that, with all 540 configurations of Section 3.1 at .995 or better, and it is why we analyze the linear head through the fixed-penalty problem of Corollary 1 rather than through Eq. 4. The diagonal terms alone would leave a context-dependent offset free, making scores comparable within a prompt and not across prompts; the matrix form removes that freedom, and our same-image-only ablation (Section 3.1) confirms that the offset does not change within-prompt accuracy. The matrix form also decides what the target does with length. Write a response as content c and length l. If the negatives match the positives’ length law conditionally on content, p(l∣c)=q¯(l∣c)p(l c)= q(l c), then dP/dQdP/dQ depends on c alone and length drops out of Eq. 4 (cf. Fan et al., 2026). A matched marginal is not enough: with two contents and two lengths, the joint tables P=(.4.1.1.4),Q=(.1.4.4.1)P= pmatrix.4&.1\\ .1&.4 pmatrix, Q= pmatrix.1&.4\\ .4&.1 pmatrix share the length marginal (.5,.5)(.5,.5) and still have ratios 44 and 1/41/4 that depend on length. Our substitution edits hold length fixed pairwise and splice and shuffle shorten the negative, so the bank meets neither condition exactly and nothing makes the target of Eq. 4 length-free. A linear head cannot represent Eq. 4 exactly and is characterized by the error-weighted mean difference of Eq. 5 instead, whose coefficient on length is not the target’s; the sign flip of Section 3.2 is that head-side coefficient changing with the backbone under one sampler. I.1 What Constructed Comparisons Supervise Fix a context x (an image and its prompt) and a finite response set Y, and let Jx(y,y′)J_x(y,y ) be the probability that a training comparison designates y as chosen and y′y as rejected, so that the conditional population loss is ℒx(s)=∑y,y′∈Jx(y,y′)ℓ(s(x,y)−s(x,y′)),ℓ(v)=log(1+e−v).L_x(s)\;=\; _y,y J_x(y,y )\, (s(x,y)-s(x,y ) ), (v)= (1+e^-v). (6) If μ(y∣x)μ(y x) is the distribution of real captions and Kx(y′∣y)K_x(y y) constructs the negative, then Jx(y,y′)=μ(y∣x)Kx(y′∣y)J_x(y,y )=μ(y x)K_x(y y): independent negatives use Kx(y′∣y)=ν(y′)K_x(y y)=ν(y ), our cross-image operator ×T_×, and a deterministic edit uses Kx(y′∣y)=y′=(y)K_x(y y)=1\y =C(y)\. Judged candidates give a JxJ_x in which winner and loser are dependent after labeling, so their marginals alone do not determine the objective; the joint comparison law does. These are population statements about unrestricted scalar scores and carry no finite-sample or downstream guarantee. I.2 Independent Negatives: Density Ratios and PMI With Jx(y,y′)=μ(y∣x)ν(y′)J_x(y,y )=μ(y x)\,ν(y ) and a common support SxS_x, the argument of Proposition 1 with the context held fixed gives the unrestricted minimizers of Eq. 6 on SxS_x, s∗(x,y)=logμ(y∣x)ν(y)+ϕ(x),y∈Sx,s (x,y)\;=\; μ(y x)ν(y)+φ(x), y∈ S_x, (7) with ϕ(x)φ(x) arbitrary. If ν=μYν= _Y, the caption marginal of the corpus law μ(x,y)μ(x,y), then s∗(x,y)=logμ(x,y)μX(x)μY(y)+ϕ(x)=PMIμ(x,y)+ϕ(x),s (x,y)\;=\; μ(x,y) _X(x)\, _Y(y)+φ(x)\;=\;PMI_μ(x,y)+φ(x), (8) the idealized case of using other images’ captions as negatives. PMI measures image–caption association, not correctness or preference agreement; shared support is an assumption a finite corpus does not guarantee; and the result does not apply to edited copies or to judge-ordered candidates. I.3 When Perturbations Expose an Additional Quality Signal The following stylized example separates topic association from response quality. Represent a response as y=(t,c)y=(t,c), where t denotes its topic and c∈0,1c∈\0,1\ indicates an incorrect or correct version. Suppose the corpus has distribution μ((t,c)∣x)=px(t)bη(c),bη(1)=η,bη(0)=1−η,12<η<1.μ ((t,c) x )=p_x(t)\,b_η(c), b_η(1)=η, b_η(0)=1-η, 12<η<1. (9) Thus topics depend on the context, but the fraction of correct versions is the same across contexts and topics. Let p(t)p(t) be the corpus topic marginal and assume px(t)>0p_x(t)>0 on a common finite topic set. Independent corpus negatives have distribution μY(t,c)=p(t)bη(c) _Y(t,c)=p(t)b_η(c), so Eq. 7 gives sind∗(x,t,c)=logpx(t)bη(c)p(t)bη(c)+ϕ(x)=logpx(t)p(t)+ϕ(x).s _ind(x,t,c)\;=\; p_x(t)\,b_η(c)p(t)\,b_η(c)+φ(x)\;=\; p_x(t)p(t)+φ(x). (10) The quality factor cancels: the optimum distinguishes topics by their association with the context but assigns equal scores to correct and incorrect versions within a topic. Now consider an idealized edit (t,c)=(t,1−c)C(t,c)=(t,1-c) that switches the version while preserving the topic, with the original always labeled chosen. This label agrees with correctness with probability η and is reversed with probability 1−η1-η; the corpus need not be perfectly correct. For a fixed topic, write Δt=s(x,t,1)−s(x,t,0) _t=s(x,t,1)-s(x,t,0). Its conditional loss and minimizer are ℒedit,t(Δt)=ηℓ(Δt)+(1−η)ℓ(−Δt),Δt∗=logη1−η>0.L_edit,t( _t)=η\, ( _t)+(1-η)\, (- _t), _t = η1-η>0. (11) Indeed, the derivative is σ(Δt)−ησ( _t)-η, which vanishes uniquely at the stated margin. Perturbation training therefore ranks the correct version above the incorrect one, a distinction absent from the independent-negative optimum in this example. A Bradley–Terry judge with reward r(x,t,c)=γcr(x,t,c)=γ c, γ>0γ>0, gives the same within-topic ordering with margin γ; the edited pairs recover the ordering with a margin set by η instead. Under the stated factorization and edit assumptions, the example shows one additional quality distinction, not a general superiority of perturbations. Changing quality-bearing content inside an otherwise matched response can supply a preference that corpus association alone does not teach; the experiments test whether actual edit rules do so and whether that preference transfers. Appendix J Proofs Convexity of the regularized objective. Write the population objective as L(w)=ℓ(βw⊤d),d=h+−h−,ℓ(z)=log(1+e−z).L(w)\;=\;E\, (β\,w d ), d=h^+-h^-, (z)= (1+e^-z ). The function ℓ is twice differentiable with ℓ′(z)=σ(z)σ(−z)> 0, (z)\;=\;σ(z)\,σ(-z)\;>\;0, so it is strictly convex, and a strictly convex function of an affine map is convex. The Hessian of L is ∇2L(w)=β2[ℓ′(βw⊤d)dd⊤].∇^2L(w)\;=\;β^2\,E [ (β w d )\,d\,d ]. It is positive definite on spandspan\d\, because the weights ℓ′ are strictly positive, and zero on the orthogonal complement, because w⊤dw d does not depend on that component. Hence L is strictly convex on the span and constant on its complement. Adding the penalty turns the objective into Lλ(w)=L(w)+λ2w⊤Dw,λ>0,D≻0,L_λ(w)\;=\;L(w)\;+\; λ2\,w Dw, λ>0,\;D 0, which is λλmin(D)λ\, _ (D)-strongly convex on all of ℝmR^m, with m the feature dimension. It therefore has exactly one minimizer. If ∥d∥2<∞E d ^2<∞, it is also LsL_s-smooth with Ls=β24∥d∥2+λλmax(D)L_s= β^24E d ^2+λ _ (D), since ℓ′≤14 ≤ 14, and gradient descent with a constant step below 2/Ls2/L_s converges to the minimizer from every initialization. Isotropic weight decay is D=ID=I. AdamW (Loshchilov and Hutter, 2019) is not gradient descent on LλL_λ for a fixed D. With a frozen diagonal preconditioner M, its update w←w−α(M∇L(w)+λw)w← w-α(M∇ L(w)+λ w) stops where ∇L(w)+λM−1w=0,∇ L(w)+λ M^-1w=0, which moves with M: for L(w)=log(1+e−w)L(w)= (1+e^-w) and λ=1λ=1 the fixed point solves w=Mσ(−w)w=Mσ(-w), giving 0.400.40 at M=1M=1 and 1.041.04 at M=4M=4. The actual M follows the gradient history, and Xie and Li (2024) characterize AdamW’s limit points as KKT points of an ℓ∞ _∞-constrained problem instead. The statements in this appendix therefore concern the fixed-penalty problem; the AdamW heads of the experiments are measured, not derived (Section 3.1). For gradient descent, initialization, step size, sample order and the number of steps change only the iterate sequence, so at fixed p, qnegq_neg, β, λ and D none of them changes the minimizer. The negative pool, its hardness and its composition change qnegq_neg and therefore the law of d, so they may move the minimizer; Eq. 5 says how, through the error-weighted mean of d. The temperature is not neutral either. Writing u=βwu=β w, Lλ(w)=ℓ(u⊤d)+λ2β2‖u‖2,L_λ(w)\;=\;E\, (u d )\;+\; λ2β^2\,\|u\|^2, so β enters as an effective regularization strength λ/β2λ/β^2 and a change in it rescales w⋆w and, since ridge shrinkage is isotropic while [dd⊤]E[d ] is not, tilts its direction as well. Only the accuracy of a ranking is invariant to the scale of w⋆w , not to its direction. ∎ Proof of Proposition 1. Let P=μ⊗pP=μ p and Q=μ⊗q¯Q=μ q be the joint laws of a positive and of a negative triple a=(x,q,y)a=(x,q,y), with q¯ q the marginal law of a negative as in the statement. The off-diagonal population loss over unrestricted scores s is J(s)=a∼P,b∼Qℓ(β(s(a)−s(b))),J(s)\;=\;E_a P,\;b Q\; (β\,(s(a)-s(b)) ), where a and b are independent because they come from different micro-batch entries. We assume throughout that P and Q are mutually absolutely continuous, so that dP/dQdP/dQ is finite and positive almost everywhere. The hypothesis is what makes the minimizer a real-valued function: on a pair where one of the two densities vanishes, the stationarity condition below puts δ⋆δ at ±∞±∞, the infimum of J over real-valued scores is not attained, and every sequence approaching it separates the two laws with unbounded margin. Our own sampler is of that kind, which is why the recipe’s weight decay and its linear class, and not Eq. 4, are what pin down the head that gets trained (Corollary 1). Group the contributions of each unordered pair a,b\a,b\ of triples. The pair appears once with a as the positive, at density P(a)Q(b)P(a)Q(b), and once with b as the positive, at density P(b)Q(a)P(b)Q(a). For a fixed difference δ=s(a)−s(b)δ=s(a)-s(b) its contribution is P(a)Q(b)ℓ(βδ)+P(b)Q(a)ℓ(−βδ).P(a)Q(b)\, (βδ)\;+\;P(b)Q(a)\, (-βδ). This is strictly convex in δ, and its derivative vanishes where σ(βδ)=P(a)Q(b)P(a)Q(b)+P(b)Q(a),that is, atβδ⋆=logP(a)Q(a)−logP(b)Q(b).σ(βδ)\;=\; P(a)Q(b)P(a)Q(b)+P(b)Q(a), is, at β\,δ \;=\; P(a)Q(a)\;-\; P(b)Q(b). The function s⋆=β−1log(dP/dQ)+Cs =β^-1 (dP/dQ)+C attains δ⋆δ for every pair simultaneously, so it minimizes every summand and therefore J. Any other minimizer attains δ⋆δ on almost every pair as well, because each summand is strictly convex in δ, so it differs from s⋆s by a constant on the support of P and Q. Since the two laws share the context law μ, dPdQ(x,q,y)=μ(x,q)p(y∣x,q)μ(x,q)q¯(y∣x,q)=p(y∣x,q)q¯(y∣x,q), dPdQ(x,q,y)\;=\; μ(x,q)\,p(y x,q)μ(x,q)\, q(y x,q)\;=\; p(y x,q) q(y x,q), which is Eq. 4. The pairs run over triples and not over responses within one context, so no context-dependent offset is left free; only the constant C is. The factor β−1β^-1 is where the temperature enters the target, and a linear head absorbs it into the scale of w. The B diagonal terms are outside this argument. There b is an edit of the same a it is scored against, so the pair law is P(a)qneg(b∣a)P(a)\,q_neg(b a) rather than the product P(a)Q(b)P(a)Q(b), the two orderings of a,b\a,b\ no longer enter at weights that depend only on the marginals, and δ⋆δ above is not the pointwise minimizer. We therefore state the proposition for the decoupled objective, whose terms carry weight 1−1/B=87.5%1-1/B=87.5\% at B=8B=8. The coupling moves the optimum. Take two responses with p=(.8,.2)p=(.8,.2), an edit that always swaps them, so q¯=(.2,.8) q=(.2,.8), and β=1β=1. Up to a constant from equal-response pairs, the full objective at B=8B=8 is 78(.64ℓ(Δ)+.04ℓ(−Δ))+18(.8ℓ(Δ)+.2ℓ(−Δ))= .66ℓ(Δ)+.06ℓ(−Δ), 78 (.64\, ( )+.04\, (- ) )+ 18 (.8\, ( )+.2\, (- ) )\;=\;.66\, ( )+.06\, (- ), minimized at Δ=log11 = 11, while the cross-entry terms alone give log16 16. ∎ Proof of Corollary 1. The population objective with the penalty is Lλ(w)=ℓ(βw⊤d)+λ2w⊤Dw,L_λ(w)\;=\;E\, (β\,w d )\;+\; λ2\,w Dw, with the expectation over the pair law of the matrix loss: each micro-batch contributes B same-entry pairs and B2−B^2-B cross-entry pairs, at total weights 1/B1/B and 1−1/B1-1/B. Since ℓ′(z)=−σ(−z) (z)=-σ(-z), ∇Lλ(w)=−β[σ(−βw⊤d)d]+λDw,∇ L_λ(w)\;=\;-β\,E [σ(-β\,w d)\,d ]\;+\;λ\,Dw, and LλL_λ is strongly convex (previous paragraph), so its unique minimizer is the unique zero of this gradient, which is Eq. 5. The inner product of Dw⋆Dw with a unit vector v gives its coefficient along v. In particular Dw⋆Dw is a positive combination of difference vectors and lies in their span, which is all of ℝmR^m whenever the difference vectors span it, and then constrains nothing; what Eq. 5 fixes is the combination. Now take D=ID=I and suppose the features encode length along a unit direction v with v⊤d=0v d=0 almost surely on the same-entry pairs. This is an assumption on the backbone, which pairwise length preservation (exact for the substitution operators, approximate for the bank) makes plausible and does not imply. Those pairs then contribute nothing to the head’s coefficient v⊤w⋆v w , and the cross-entry pairs, which compare captions of different images, contribute βλ(1−1B)×[σ(−βw⋆⊤d)v⊤d], βλ (1- 1B )\,E_× [σ(-β\,w d)\,v d ], an error-weighted mean of gaps between unrelated captions, which the objective does not set to zero. Pairwise preservation on the diagonal therefore leaves the coefficient along v to the cross-entry pairs rather than fixing it at zero, the same conclusion as for the unrestricted optimum above, where a comparison that holds a nuisance fixed leaves its coefficient undetermined. For D≠ID≠ I the same computation gives v⊤Dw⋆v Dw , and the head’s coefficient v⊤w⋆v w can be nonzero even when every difference is orthogonal to v: with D=diag(1,2)D=diag(1,2) and d=(1,−1)d=(1,-1), w⋆∝(1,−12)w (1,- 12) and v=(1,1)/2v=(1,1)/ 2 gives v⊤d=0v d=0 and v⊤w⋆>0v w >0. For two samplers q¯1,q¯2 q_1, q_2 with difference laws ρ1≠ρ2 _1≠ _2, Eq. 5 gives minimizers w1,w2w_1,w_2 that coincide only if the two error-weighted means do, a condition on the laws and not on their spans. A benchmark pair set ek\e_k\ measures the sign pattern sign(w⊤ek),k=1,…,K,sign (w e_k ), k=1,…,K, which coincides for w1w_1 and w2w_2 whenever the two heads order the benchmark’s pairs alike, however far apart they are on other pairs. ∎ Appendix K Compute Resources Training a head is the cheapest step in the study. Once the frozen backbone’s hidden states are stored, a 540-configuration grid at 2B fits in about nine GPU-minutes, and a single head is 200 steps of a d→1d→ 1 linear map. The 32B confirm run used FSDP over eight GPUs at effective batch 1,024. The compute goes to everything that is not the reward model: one forward pass of the backbone per pair to store its hidden states, Best-of-N sampling at N=16N=16 over 2,500 prompts for two policy sizes, and the image arm’s renders. The image arm needs two same-seed Qwen-Image renders per prompt at about 7 GPU-seconds each, about 5 GPU-hours for its 1,212 training pairs. The caption negative bank itself is free: 10 seconds on one CPU core for 22k pairs, with no model in the loop. The figure the paper quotes against fine-tuning, about a thousandth of the compute, counts head training against the full fine-tune of the same backbone on the same pairs and excludes the backbone forward passes both settings share.