Paper deep dive
VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning
Jiahe Feng
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:GRPO-style reinforcement learning has advanced vision-language reasoning, but its sequence-level rewards provide limited credit assignment for individual reasoning tokens. On-policy distillation (OPD) addresses this issue by providing dense supervision on student-generated trajectories, yet standard OPD assigns similar importance to all teacher-preferred tokens, including template phrases and visually irrelevant continuations. We introduce Visual Counterfactual On-Policy Distillation (VC-OPD), a grounding-aware distillation method that prioritizes tokens whose correctness depends on image-specific evidence. After a teacher-supervised warm start that stabilizes response formats and reduces noisy rollouts, VC-OPD performs on-policy distillation with two teacher evaluations for each student trajectory: one under the original image and one under a counterfactual image with corrupted instance-specific evidence. The original-image teacher distribution determines the distillation target, while the original--counterfactual teacher gap estimates token-level visual dependence and softly reweights the loss. To avoid unstable response-level update scales, VC-OPD further applies mass-preserving normalization to the token weights. This design preserves the dense supervision of OPD while shifting learning capacity toward visually grounded reasoning tokens. Experiments on multimodal reasoning benchmarks show that VC-OPD improves visual reasoning performance and produces more effective token-level distillation.
Tags
Links
- Source: https://arxiv.org/abs/2610.14939v1
- Canonical: https://arxiv.org/abs/2610.14939v1
Trouble viewing inline? Open PDF directly â
Full Text
61,786 characters extracted from source content.
Expand or collapse full text
VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning Jiahe Feng Affiliation: College of Intelligence and Computing Affiliation: Tianjin University Affiliation: Tianjin, China Email: fjh3475227047@tju.edu.cn Abstract GRPO-style reinforcement learning has advanced vision-language reasoning, but its sequence-level rewards provide limited credit assignment for individual reasoning tokens. On-policy distillation (OPD) addresses this issue by providing dense supervision on student-generated trajectories, yet standard OPD assigns similar importance to all teacher-preferred tokens, including template phrases and visually irrelevant continuations. We introduce Visual Counterfactual On-Policy Distillation (VC-OPD), a grounding-aware distillation method that prioritizes tokens whose correctness depends on image-specific evidence. After a teacher-supervised warm start that stabilizes response formats and reduces noisy rollouts, VC-OPD performs on-policy distillation with two teacher evaluations for each student trajectory: one under the original image and one under a counterfactual image with corrupted instance-specific evidence. The original-image teacher distribution determines the distillation target, while the originalâcounterfactual teacher gap estimates token-level visual dependence and softly reweights the loss. To avoid unstable response-level update scales, VC-OPD further applies mass-preserving normalization to the token weights. This design preserves the dense supervision of OPD while shifting learning capacity toward visually grounded reasoning tokens. Experiments on multimodal reasoning benchmarks show that VC-OPD improves visual reasoning performance and produces more effective token-level distillation. 1 Introduction Large vision-language models (VLMs) have achieved strong progress in multimodal reasoning, yet post-training them remains substantially harder than post-training text-only language models. Recent reinforcement and preference-based methods, including PPO, GRPO, multimodal RLHF, reinforcement fine-tuning, and verifiable-reward optimization, have improved mathematical and long-chain reasoning (Schulman et al., 2017; Shao et al., 2024; Sun et al., 2023; Yu et al., 2023; Liu et al., 2025a; Shen et al., 2025; Zhang et al., 2025a; Tan et al., 2025). However, VLM reasoning has an additional prerequisite: the model must first identify and use the relevant visual evidence. This has motivated perception-aware post-training methods, such as corrupted-image objectives, perception-first curricula, self-rewarding decomposition, and grounded visual exploration (Wang et al., 2025c; Chen et al., 2025; Li et al., 2025; Sarch et al., 2025). These studies indicate that effective VLM post-training should not only optimize final answers, but also assign learning signals to tokens that are genuinely grounded in the image. On-policy distillation (OPD) is a natural framework for such fine-grained supervision. Instead of relying only on sequence- or step-level rewards, OPD samples trajectories from the student and asks a teacher to score the same trajectories, providing dense token-level feedback under the studentâs own state distribution (Agarwal et al., 2024). Recent theoretical analyses further interpret OPD as KL-regularized token-level policy optimization, where the teacherâstudent likelihood ratio acts as a dense reward-like signal (Yang et al., 2026; Ko et al., 2026). Nevertheless, token-level density does not imply grounding awareness. Standard OPD treats all teacher-preferred tokens as equally distillable, although many tokens in VLM responses are generic templates, discourse markers, or language-prior continuations rather than image-dependent content. This limitation is illustrated in Figure 1. The studentâteacher gap used by standard OPD highlights both visually meaningful tokens and generic reasoning expressions, revealing where the student differs from the teacher but not whether the discrepancy is grounded in the image. In contrast, a factualâcounterfactual teacher gap, obtained by comparing teacher scores under the original image and a visually weakened counterfactual image, more selectively emphasizes tokens tied to visual evidence, such as angle symbols, parallel-line relations, and answer-relevant quantities. Figure 1: Token-level grounding visualization. Standard studentâteacher gaps highlight both visual and generic tokens, whereas factualâcounterfactual teacher gaps more selectively emphasize image-dependent tokens. Token-level statistics provide the same message at scale. As shown in Figure 2, studentâteacher gaps are heavily concentrated near zero, suggesting that dense OPD supervision often yields weak effective gradients. By contrast, large factualâcounterfactual gaps appear on only a sparse subset of tokens, consistent with the structure of VLM reasoning: only some tokens encode objects, quantities, spatial relations, or diagram-specific information, while many others mainly serve as linguistic scaffolding. The position-wise pattern further shows that visual dependence is stronger in early perception-related positions and weaker in later reasoning or answer-formatting stages. These observations motivate a grounding-aware OPD objective that preserves dense supervision while reallocating update strength toward visually dependent tokens. We propose Visual Counterfactual On-Policy Distillation (VC-OPD), a grounding-aware OPD framework for vision-language reasoning. VC-OPD decouples the direction and magnitude of token-level updates. For each student-generated trajectory, the factual teacher under the original image provides the teacherâstudent discrepancy that determines the update direction. The same teacher is then evaluated under a counterfactual image where instance-specific visual evidence is weakened. The resulting factualâcounterfactual log-probability gap estimates each tokenâs dependence on the original visual input and is converted into a soft grounding weight that modulates the update magnitude. In this way, VC-OPD retains the dense token-level supervision of OPD while concentrating learning on tokens that are more likely to encode visual perception and grounding. VC-OPD uses a simple two-stage pipeline. First, we warm-start the student with teacher-generated responses to stabilize response format and reduce noisy on-policy rollouts. Second, we perform visual counterfactual on-policy distillation: the student rolls out under the real image, the teacher scores the same trajectory under both factual and counterfactual visual conditions, and the resulting visual-dependence score reweights the token-level OPD objective. We further apply mass-preserving normalization to preserve the relative emphasis on visually grounded tokens while keeping the overall token weight stable across responses. Our contributions are threefold: ⢠We identify a grounding-related limitation of standard OPD for VLM reasoning: the teacherâstudent discrepancy captures model disagreement but does not explicitly distinguish image-dependent tokens from generic linguistic continuations. ⢠We introduce the factualâcounterfactual teacher log-probability gap as a token-level proxy for visual dependence, enabling grounding-aware reweighting of student-generated trajectories. ⢠We formulate VC-OPD with a directionâmagnitude separation: the factual teacherâstudent discrepancy determines the update direction, while the counterfactual signal modulates its magnitude through soft weighting and mass-preserving normalization. (a) Studentâteacher gap (b) Factualâcounterfactual gap (c) Position-wise visual gap Figure 2: Token-level signal statistics. Standard OPD gaps are often small, while factualâcounterfactual gaps reveal sparse and position-dependent visual grounding signals. 2 Related Work 2.1 Reinforcement learning for vision-language models Reinforcement learning has become an important post-training paradigm for vision-language models (VLMs), building on the success of policy optimization in language reasoning. Modern VLM RL inherits trust-region and proximal policy methods such as TRPO and PPO, as well as critic-free variants such as GRPO, which has shown strong performance in mathematical reasoning (Schulman et al., 2015; Schulman et al., 2017; Shao et al., 2024). These optimization paradigms have been extended to multimodal alignment and reasoning through RLHF, reward modeling, preference alignment, and reinforcement fine-tuning (Sun et al., 2023; Yu et al., 2023; Zhang et al., 2025c; Ji et al., 2025; Liu et al., 2025a; Shen et al., 2025; Zhang et al., 2025a; Tan et al., 2025; Liu et al., 2025b). A growing line of work further argues that visual perception is a central bottleneck in VLM RL. PAPO introduces perception-aware policy optimization by comparing model behavior under original and corrupted visual inputs; PeBR proposes a two-stage curriculum that improves perception before reasoning; Vision-SR1 decomposes visual reasoning into perception and language reasoning with self-rewarding signals; ViGoRL grounds intermediate reasoning steps to visual coordinates (Wang et al., 2025c; Chen et al., 2025; Li et al., 2025; Sarch et al., 2025). These methods show that successful multimodal reasoning requires models to use instance-specific visual evidence rather than relying on language priors or generic reasoning templates. However, most existing VLM RL methods still optimize sequence-level rewards, step-level rewards, or auxiliary perception losses, and do not provide dense teacher-guided token-level credit assignment. Our method addresses this gap by introducing visual counterfactual signals into on-policy distillation. 2.2 On-policy distillation for reasoning models Knowledge distillation transfers capabilities from large teachers to smaller students, but classical distillation and supervised fine-tuning usually optimize the student on fixed teacher-provided trajectories, which can introduce distribution mismatch (Hinton et al., 2015; Kim and Rush, 2016). On-policy distillation (OPD) reduces this mismatch by sampling trajectories from the student and querying the teacher on those trajectories, thereby aligning supervision with the studentâs own rollout distribution (Agarwal et al., 2024; Yang et al., 2026; Ko et al., 2026). Recent work has provided a stronger theoretical and practical foundation for OPD. GKD formalizes on-policy distillation for autoregressive models (Agarwal et al., 2024). G-OPD interprets OPD as dense KL-constrained policy optimization and generalizes the reference-policy design (Yang et al., 2026). REOPOLD further shows that the teacherâstudent likelihood ratio can be viewed as a token-level reward and improves training efficiency through relaxed token selection and reward clipping (Ko et al., 2026). Other OPD variants study context distillation, reasoning-prefix supervision, and entropy-aware objectives (Ye et al., 2026; Zhang et al., 2026; Jin et al., 2026). Compared with sparse reward RL, OPD provides dense token-level supervision, which is attractive for long-chain reasoning. While effective for text reasoning, existing OPD methods typically treat all generated tokens as distillable. This assumption is suboptimal for VLM reasoning, where many tokens are structural markers, template phrases, or language-prior continuations, and only a subset is truly supported by visual evidence. We extend the generalized OPD perspective to multimodal reasoning by using a counterfactual teacher to estimate token-level visual dependence and reweighting OPD updates toward visually grounded tokens. 3 Method 3.1 Preliminaries: on-policy distillation as token-level RL We consider a vision-language reasoning setting with textual input x, visual input v, and autoregressive output sequence y=(y1,âŚ,yT)y=(y_1,âŚ,y_T). The student policy is denoted by Ďθâ(ytâŁx,v,y<t) _θ(y_t x,v,y_<t), and the rollout policy Îź is typically the current student or a lagged copy. During training, we sample trajectories on-policy under the real visual input: yâźÎź(â âŁx,v).y Îź(¡ x,v). (1) On-policy distillation (OPD) differs from conventional offline distillation in one essential aspect: the teacher does not supervise teacher-forced trajectories, but instead evaluates trajectories that are actually generated by the student itself. As a result, the distillation signal is aligned with the studentâs own state distribution, which reduces the trainâtest mismatch that commonly arises in offline knowledge distillation. This property is particularly important for reasoning tasks, where small deviations in early tokens may lead the student to entirely different downstream trajectories. Recent work has shown that OPD can be interpreted through the lens of token-level policy optimization. Let st=(x,v,y<t)s_t=(x,v,y_<t) denote the autoregressive state and at=yta_t=y_t the sampled token. If we measure the discrepancy between the student and the teacher with the reverse KL divergence, DRKL(Ďθ(â âŁst)âĽĎtea(â âŁst))=atâźĎθ[logĎθ(atâŁst)âlogĎtea(atâŁst)],D_RKL\! ( _θ(¡ s_t)\,\|\, _tea(¡ s_t) )=E_a_t _θ [ _θ(a_t s_t)- _tea(a_t s_t) ], (2) then maximizing the negative reverse KL induces the token-level learning signal rt=logâĄĎteaâ(atâŁst)âlogâĄĎθâ(atâŁst).r_t= _tea(a_t s_t)- _θ(a_t s_t). (3) This quantity can be viewed as a dense per-token reward: if the teacher assigns higher probability to the sampled token than the student, the reward is positive, encouraging the student to increase its probability; otherwise the reward becomes small or negative. In this sense, OPD can be understood as a form of dense RL where the reward is not provided by an external environment, but is instead induced by the discrepancy between the teacher and the student on student-generated trajectories. Under this view, a broad class of OPD-style methods can be written as weighted log-likelihood optimization: ââĄ(θ)=âyâźÎźâ[ât=1TsgâĄ(At)âlogâĄĎθâ(ytâŁx,v,y<t)],L(θ)=-E_y Îź [ _t=1^Tsg(A_t) _θ(y_t x,v,y_<t) ], (4) where AtA_t is an advantage-like token weight and sgâĄ(â )sg(¡) denotes stop-gradient. Therefore, once OPD is viewed as token-level RL, the central modeling question becomes how to define AtA_t. For text-only reasoning, this weight mainly reflects how strongly the teacher prefers a continuation. In multimodal reasoning, however, teacher preference alone is insufficient: a token may be preferred either because it is genuinely grounded in visual evidence or because it is a generic reasoning phrase, a structural template token, or a continuation supported only by language priors. Our goal is therefore to redesign the token-level weight so that student updates are concentrated on tokens that are truly supported by visual evidence. Figure 3: Overview of VC-OPD. The framework first warm-starts the student with filtered teacher-generated reasoning responses. During visual counterfactual on-policy distillation, student rollouts are scored by the teacher under factual and counterfactual visual inputs. The factual teacher provides the update direction, while the factualâcounterfactual gap produces grounding-aware token weights that modulate the final token-level advantage. 3.2 Two-stage training Our training consists of two stages. The first stage provides a stable initialization, and the second stage performs visual counterfactual on-policy distillation. Stage I: warm-start. Before entering on-policy training, we first warm-start the student using teacher-generated reference responses yây . Given (x,v)(x,v), the teacher produces a structured target response, and the student is optimized with standard autoregressive supervision: âwarmâ(θ)=â(x,v,yâ)âźâ[ât=1TâlogâĄĎθâ(ytââŁx,v,y<tâ)].L_warm(θ)=-E_(x,v,y ) [ _t=1^T _θ(y_t x,v,y _<t) ]. (5) The purpose of this stage is not to solve the final problem by imitation alone, but to place the student in a reasonable response regime before on-policy optimization. In practice, this reduces rollout noise, stabilizes response format, and makes the subsequent teacher feedback more informative. Stage I: visual counterfactual on-policy distillation. After warm-start, the student generates its own response trajectory under the real image, yâźÎź(â âŁx,v).y Îź(¡ x,v). (6) We then evaluate the same trajectory with the teacher under two visual conditions: the original image and a counterfactual image in which instance-specific evidence has been weakened. The factual branch determines where the student should move, while the counterfactual branch determines how strongly each token should be updated. 3.3 Counterfactual teacher scoring and grounding-aware weighting For each sampled token yty_t, we define the factual teacher score and the counterfactual teacher score as pT+â(yt)=pTâ(ytâŁx,v,y<t),pTââ(yt)=pTâ(ytâŁx,v~,y<t),p_T^+(y_t)=p_T(y_t x,v,y_<t), p_T^-(y_t)=p_T(y_t x, v,y_<t), (7) where v~=câĄ(v) v=c(v) is a corrupted visual input produced by a visual perturbation operator câĄ(â )c(¡). The role of câĄ(â )c(¡) is not to destroy the entire input, but to weaken instance-level visual evidence while preserving the general input format, so that the teacher remains in a comparable prediction regime. Based on these two teacher evaluations, we compute two quantities. The first is a direction term, Atdir=logâĄpT+â(yt)âlogâĄÎźâĄ(ytâŁx,v,y<t),A_t^dir= p_T^+(y_t)- Îź(y_t x,v,y_<t), (8) which compares the factual teacher and the student on the sampled token. This term determines the basic update direction: when the student underestimates a token relative to the factual teacher, the token receives a positive update signal; when the student already overestimates it, the update is weakened or reversed. The second is a visual dependence score, Îtvis=logâĄpT+â(yt)âlogâĄpTââ(yt), _t^vis= p_T^+(y_t)- p_T^-(y_t), (9) This metric quantifies the shift in teacher token support under visual perturbation. We use this difference as a proxy for token-level visual dependence, not a rigorous causal estimate of image effects. A positive value means the original factual image yields stronger teacher support for the token, while near-zero values show insensitivity to visual perturbations. The score hinges on whether the perturbation removes task-relevant visual evidence while preserving sufficient context. Rather than hard-selecting tokens with a binary rule, we convert the visual dependence score into a continuous soft weight: wt=ĎâĄ(ÎtvisâĎvTv),w_t=Ď\! ( _t^vis- _vT_v ), (10) where ĎâĄ(â )Ď(¡) is the sigmoid function, Ďv _v is a visual threshold, and TvT_v is a temperature parameter. This design preserves continuity during training: highly grounded tokens receive larger weights, while weakly grounded tokens are downweighted but not necessarily discarded. A practical issue is that the number and strength of visually grounded tokens may vary substantially across samples, which can lead to unstable update scales if wtw_t is used directly. To stabilize optimization, we adopt a mass-preserving normalization: wÂŻt=wtâ Nâj=1Twj+Ďľ,N=ât=1Tmt, w_t=w_t¡ N _j=1^Tw_j+Îľ, N= _t=1^Tm_t, (11) where mtm_t is the response mask and N is the number of valid response tokens. This normalization preserves the relative emphasis across tokens while keeping the total effective weight approximately constant for each response, thereby preventing large variations in update magnitude across batches. 3.4 Final VC-OPD objective We combine the direction term and the normalized grounding weight to define the final token-level advantage: AtVC=wÂŻtâ A~tdir,A_t^VC= w_t¡ A_t^dir, (12) where A~tdir A_t^dir denotes a normalized version of AtdirA_t^dir for numerical stability. The key design principle is that the factual teacherâstudent discrepancy determines the direction of the update, while the counterfactual branch modulates only its magnitude according to visual grounding strength. The final stage-I objective is âVCâ-âOPD(θ)=âyâźÎź(â âŁx,v)[ât=1Tsg(AtVC)logĎθ(ytâŁx,v,y<t)].L_VC -OPD(θ)=-E_y Îź(¡ x,v) [ _t=1^Tsg\! (A_t^VC ) _θ(y_t x,v,y_<t) ]. (13) This objective retains the on-policy nature of OPD while making the token-level update explicitly grounding-aware. Tokens that are both under-modeled by the student and strongly supported by real visual evidence receive larger updates; tokens that are mainly driven by language priors are correspondingly suppressed. 4 Experiments 4.1 Experimental setup We use Qwen2.5-VL (Bai et al., 2025b) as the primary backbone for our main experiments. Unless otherwise specified, Qwen2.5-VL-3B-Instruct is the student, while larger Qwen2.5-VL models serve as teachers. All methods start from the same student checkpoint and are trained on ViRL39K (Wang et al., 2025a), which contains approximately 39K verifiable visual-language reasoning examples. VC-OPD uses a two-stage procedure consisting of teacher-supervised warm-start and visual counterfactual on-policy distillation. We further evaluate Qwen3-VL and InternVL3.5 to test cross-backbone generalization, with additional details provided in Appendix E and F. We evaluate seven multimodal reasoning benchmarks covering mathematical, geometry, multi-disciplinary, logical, and counting tasks: Geometry3K (Lu et al., 2021), MathVista (Lu et al., 2024), MathVerse (Zhang et al., 2024), We-Math (Qiao et al., 2024), MMMU-Pro (Yue et al., 2024), LogicVista (Xiao et al., 2024), and Super-CLEVR Counting (Li et al., 2023). All results use exact match after answer extraction, with the average reported as the unweighted mean across benchmarks. Unless otherwise specified, we use the @8 setting with inference temperature 1.01.0. All experiments are conducted on four NVIDIA A800 GPUs. Detailed hyperparameters and implementation details are provided in Appendix B. Table 1: Main results on multimodal reasoning benchmarks. We report exact-match accuracy under the @8 setting with inference temperature 1.01.0. Avg. denotes the unweighted average over all listed benchmarks. Model Geo3K MVer. MVist. WeM. MMMU Logic Count Avg. Qwen2.5-VL-3B-Instruct 24.96 38.08 58.50 53.22 24.50 34.72 51.71 40.81 7B teacher â 3B student + GRPO 30.52 40.60 59.04 59.10 25.86 38.34 57.81 44.47 + PAPO 30.95 41.50 62.38 60.09 27.11 38.67 62.56 46.18 + RKL 32.12 40.75 62.08 60.23 28.40 39.41 63.50 46.64 + VC-OPD 34.35 41.90 64.10 61.46 32.20 42.40 65.68 48.87 ÎRKL _RKL (%) +6.94 +2.82 +3.25 +2.04 +13.38 +7.59 +3.43 +4.78 32B teacher â 3B student + RKL 45.09 43.25 65.50 64.56 31.24 41.52 65.80 50.99 + VC-OPD 57.58 44.40 65.72 65.74 33.50 42.90 67.24 53.87 ÎRKL _RKL (%) +27.70 +2.66 +0.34 +1.83 +7.23 +3.32 +2.19 +5.64 Qwen2.5-VL-7B-Instruct 36.80 43.90 66.40 64.80 33.80 43.50 64.90 50.59 32B teacher â 7B student + RKL 48.70 47.40 69.70 66.30 36.20 46.00 69.00 54.76 + VC-OPD 55.40 48.90 70.90 68.10 38.80 48.60 71.20 57.41 ÎRKL _RKL (%) +13.76 +3.17 +1.72 +2.71 +7.18 +5.65 +3.19 +4.85 4.2 Main results Table 1 reports the main results on multimodal reasoning benchmarks. We adopt Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct as the student models, with Qwen2.5-VL-7B and Qwen2.5-VL-32B serving as teacher models. For the Qwen2.5-VL-3B student, we evaluate both 7B- and 32B-teacher settings, while the Qwen2.5-VL-7B student is distilled from the 32B teacher. In the 7B-teacher setting with the 3B student, we compare VC-OPD with GRPO, PAPO, and reverse-KL on-policy distillation (RKL). For all distillation settings, RKL serves as the direct on-policy distillation baseline, and ÎRKL _RKL denotes the relative improvement of VC-OPD over RKL. VC-OPD consistently improves over RKL across all evaluated studentâteacher settings. With the 7B teacher and the Qwen2.5-VL-3B student, VC-OPD improves the average accuracy from 46.6446.64 to 48.8748.87, yielding a 2.232.23-point absolute gain and a 4.78%4.78\% relative improvement. With the 32B teacher and the same 3B student, VC-OPD further improves the average accuracy from 50.9950.99 to 53.8753.87, corresponding to a 2.872.87-point absolute gain and a 5.64%5.64\% relative improvement. For the Qwen2.5-VL-7B student, the base model achieves an average accuracy of 50.5950.59, while distillation from the 32B teacher improves the RKL performance from 54.7654.76 to 57.4157.41, corresponding to a 4.85%4.85\% relative improvement. The consistent gains across different student and teacher scales demonstrate that VC-OPD remains effective as the student model capacity increases, while preserving the advantages of grounding-aware token-level distillation. 4.3 Ablation study We conduct ablation studies under the 7B-teacher setting to isolate the effect of each component in VC-OPD. Starting from vanilla on-policy distillation, we progressively examine the contribution of warm-start training, counterfactual teacher scoring, soft visual reweighting, and mass-preserving normalization. Table 2: Ablation study under the 7B-teacher setting. Warm, CF, Soft, and Mass denote warm-start, counterfactual teacher scoring, soft visual reweighting, and mass-preserving normalization, respectively. Variant Warm CF Soft Mass Math Avg. Vision Avg. Overall Avg. Vanilla OPD â â â â 48.80 43.77 46.64 w/o warm-start â â â â 49.37 45.55 47.73 w/o counterfactual teacher â â â â 49.42 44.63 47.37 w/ hard mask â â â â 49.86 45.92 48.17 w/o mass norm. â â â â 50.05 46.10 48.36 VC-OPD â â â â 50.45 46.76 48.87 Table 2 shows that each component contributes to the final performance of VC-OPD. Vanilla OPD already provides dense token-level supervision, but its performance remains limited because all generated tokens are treated uniformly. Adding counterfactual visual scoring improves both math-oriented and visually grounded benchmarks, indicating that token-level visual dependence is a useful signal for multimodal distillation. Warm-start training also matters: removing it lowers the overall average from 48.8748.87 to 47.7347.73, suggesting that stable initial rollouts are important before applying on-policy teacher feedback. Replacing soft reweighting with a hard mask further degrades performance, showing that continuous token weighting is preferable to binary token selection. Finally, removing mass-preserving normalization reduces the overall average to 48.3648.36, confirming that stabilizing the effective token weight is beneficial for robust optimization. Overall, the full VC-OPD configuration achieves the best performance, supporting the design choice of combining counterfactual visual dependence with stable soft token-level reweighting. 4.4 Training dynamics We further compare the optimization behavior of GRPO, RKL, and VC-OPD under the same 200-step training budget. As shown in Table 3 and Figure 4, both RKL and VC-OPD improve faster than GRPO, suggesting that dense teacher-guided token supervision is more sample-efficient than outcome-level reward optimization. VC-OPD further achieves a more stable trajectory than RKL and reaches the best final performance on both Geo3K and LogicVista, indicating that visual counterfactual reweighting improves the effectiveness and stability of token-level distillation. Table 3: Step-wise validation accuracy during training. We report exact-match accuracy (%) on Geo3K and LogicVista every 50 optimization steps. Method Geo3K LogicVista 50 100 150 200 50 100 150 200 GRPO 26.10 27.75 28.90 30.52 34.20 36.30 37.45 38.34 RKL 29.20 30.35 31.25 32.12 37.05 38.20 38.85 39.41 VC-OPD 31.80 32.45 33.20 34.35 39.10 40.55 41.45 42.40 (a) Geo3K validation accuracy (b) Training reward Figure 4: Training dynamics over 200 optimization steps. VC-OPD improves faster than GRPO, is more stable than RKL, and achieves the highest validation accuracy and training reward. Compared with GRPO, RKL improves more rapidly because it provides dense token-level teacher feedback instead of relying only on sequence-level rewards. However, RKL still applies supervision uniformly across tokens, which may amplify visually irrelevant or template-like continuations. VC-OPD addresses this issue by reweighting token updates according to visual counterfactual dependence. As a result, it maintains a smoother reward curve and consistently outperforms RKL at later training stages. 4.5 Hyperparameter analysis We analyze the proportion of visually dependent tokens emphasized by the soft visual weighting function. The goal is to verify that VC-OPD benefits from a balanced grounding signal rather than overly dense or overly sparse token reweighting. Table 4: Sensitivity analysis of the visual token ratio. Default setting is marked with â. Visual ratio Math Vision Overall 20% 49.92 46.18 48.32 10%â 50.45 46.76 48.87 5% 49.58 45.90 48.00 Table 4 shows that a moderate visual-token ratio performs best. Since visually grounded tokens form only a sparse subset of the response, using a large ratio such as 20%20\% assigns substantial weight to weakly visual or template-like tokens, reducing the selectivity of VC-OPD. In contrast, using only 5%5\% of tokens makes the supervision too sparse, resulting in insufficient effective updates. The 10%10\% setting provides a favorable trade-off between grounding selectivity and optimization density and is therefore used as the default configuration. 4.6 Alternative Counterfactual Constructions We further investigate whether VC-OPD depends on a particular counterfactual construction. The counterfactual image is intended to weaken instance-specific visual evidence while preserving a meaningful visual context. We compare the default patch dropout with random dropout, Gaussian blur, contiguous region masking, object-level erasing, saliency-guided corruption, and mismatched-image counterfactuals under identical training and evaluation settings. Table 5: Comparison of alternative counterfactual visual constructions under the Qwen2.5-VL-3B student with the Qwen2.5-VL-7B teacher. The RKL result is shown as the reference baseline. Counterfactual Configuration Math Avg. Vision Avg. Overall Avg. No counterfactual RKL 48.80 43.77 46.64 Random dropout â 49.86 46.20 48.29 (+3.54%) Patch dropout 20% 49.72 45.98 48.12 (+3.17%) Patch dropout 60% 50.45 46.76 48.87 (+4.78%) Patch dropout 100% 49.20 45.55 47.64 (+2.14%) Gaussian blur Kernel 31Ă3131Ă 31, Ď=10Ď=10 50.06 46.35 48.47 (+3.92%) Contiguous region masking 60% masked area 50.30 46.55 48.69 (+4.40%) Object-level erasing Randomly erase object instances 49.62 45.70 47.94 (+2.79%) Saliency-guided corruption Mask high-saliency regions 50.40 46.83 48.87 (+4.78%) Mismatched image Random batch permutation 47.22 41.05 44.36 (-4.89%) Table 5 shows that VC-OPD remains effective across a range of counterfactual constructions. All alternatives except mismatched-image counterfactuals outperform RKL, indicating that the proposed objective is not tied to a specific patch-dropout implementation. Patch dropout at 60%60\% provides a favorable balance between weakening instance-specific evidence and preserving a meaningful visual prediction context, and is therefore retained as the default configuration. The mismatched-image condition performs substantially worse, suggesting that replacing the original visual context with unrelated evidence does not provide an appropriate counterfactual for estimating token-level visual dependence. More targeted constructions, such as object-level erasing and saliency-guided corruption, are promising but may require additional models or computations, whereas patch dropout provides a simple and lightweight alternative. 5 Conclusion We introduced VC-OPD, a visual counterfactual extension of on-policy distillation for grounded vision-language reasoning. By comparing teacher scores under factual and counterfactual visual inputs, VC-OPD estimates token-level visual dependence and uses it to reweight dense distillation updates. The factual teacherâstudent discrepancy determines the update direction, while the counterfactual signal adjusts the update magnitude through soft weighting and mass-preserving normalization. Experiments across multiple student scales and the Qwen2.5-VL, Qwen3-VL, and InternVL3.5 families show consistent improvements over reverse-KL on-policy distillation in the evaluated settings. These findings support grounding-aware token-level credit assignment as a useful complement to general multimodal reasoning objectives. 6 Limitations VC-OPD is primarily designed to improve visually grounded token-level credit assignment rather than generic text-only reasoning. Its gains may therefore be more modest on tasks dominated by linguistic logic, symbolic manipulation, or visually invariant deduction. Moreover, visual dependence is not equivalent to reasoning importance: some visually invariant transition or reasoning-control tokens may still play an important role in organizing the reasoning trajectory and can therefore be underweighted. The factualâcounterfactual teacher gap is also based on visual perturbations rather than a perfect causal intervention, and the additional counterfactual teacher evaluation introduces computational overhead. Future work could combine visual-dependence weighting with complementary signals for reasoning importance and develop more precise and efficient counterfactual interventions. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. arXiv preprint arXiv:2306.13649. Cited by: §1, §2.2, §2.2. Bai et al. (2025a) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, R. Fang, C. Gao, C. Ge, et al. Qwen3-vl technical report. ArXiv abs/2511.21631. External Links: Link Cited by: Appendix E. Bai et al. (2025b) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §4.1. Chen et al. (2025) Y. Chen, L. Li, T. Xi, L. Zeng, and J. Wang Perception before reasoning: two-stage reinforcement learning for visual reasoning in vision-language models. arXiv preprint arXiv:2509.13031. Cited by: §1, §2.1. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: §2.2. Ji et al. (2025) J. Ji, X. Chen, R. Pan, C. Zhang, H. Zhu, J. Li, D. Hong, B. Chen, J. Zhou, K. Wang, J. Dai, C. Chan, Y. Tang, S. Han, Y. Guo, and Y. Yang Safe rlhf-v: safe reinforcement learning from multi-modal human feedback. arXiv preprint arXiv:2503.17682. Cited by: §2.1. Jin et al. (2026) W. Jin, T. Min, Y. Yang, S. R. Kadhe, Y. Zhou, D. Wei, N. Baracaldo, and K. Lee Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079. Cited by: §2.2. Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Cited by: §2.2. Ko et al. (2026) J. Ko, S. Abdali, Y. J. Kim, T. Chen, and P. Cameron Scaling reasoning efficiently via relaxed on-policy distillation. arXiv preprint arXiv:2603.11137. Cited by: §1, §2.2, §2.2. Li et al. (2023) Z. Li, X. Wang, E. Stengel-Eskin, A. Kortylewski, W. Ma, B. Van Durme, and A. Yuille Super-clevr: a virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §4.1. Li et al. (2025) Z. Li, W. Yu, C. Huang, R. Liu, Z. Liang, F. Liu, J. Chen, D. Yu, J. Boyd-Graber, H. Mi, and D. Yu Self-rewarding vision-language model via reasoning decomposition. arXiv preprint arXiv:2508.19652. Cited by: §1, §2.1. Liu et al. (2025a) Z. Liu, Z. Sun, Y. Zang, X. Dong, Y. Cao, H. Duan, D. Lin, and J. Wang Visual-rft: visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785. Cited by: §1, §2.1. Liu et al. (2025b) Z. Liu, Y. Zang, Y. Zou, Z. Liang, X. Dong, Y. Cao, H. Duan, D. Lin, and J. Wang Visual agentic reinforcement fine-tuning. arXiv preprint arXiv:2505.14246. Cited by: §2.1. Lu et al. (2024) P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K. Chang, M. Galley, and J. Gao MathVista: evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, Cited by: §4.1. Lu et al. (2021) P. Lu, R. Gong, S. Jiang, L. Qiu, S. Huang, X. Liang, and S. Zhu Inter-gps: interpretable geometry problem solving with formal language and symbolic reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, Cited by: §4.1. Qiao et al. (2024) R. Qiao, Q. Tan, G. Dong, M. Wu, C. Sun, X. Song, Z. GongQue, S. Lei, Z. Wei, M. Zhang, et al. We-math: does your large multimodal model achieve human-like mathematical reasoning?. arXiv preprint arXiv:2407.01284. Cited by: §4.1. Sarch et al. (2025) G. Sarch, S. Saha, N. Khandelwal, A. Jain, M. J. Tarr, A. Kumar, and K. Fragkiadaki Grounded reinforcement learning for visual reasoning. arXiv preprint arXiv:2505.23678. Cited by: §1, §2.1. Schulman et al. (2015) J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel Trust region policy optimization. arXiv preprint arXiv:1502.05477. Cited by: §2.1. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1, §2.1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §2.1. Shen et al. (2025) H. Shen, P. Liu, J. Li, C. Fang, Y. Ma, J. Liao, Q. Shen, Z. Zhang, K. Zhao, Q. Zhang, R. Xu, and T. Zhao VLM-r1: a stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615. Cited by: §1, §2.1. Sun et al. (2023) Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y. Shen, C. Gan, L. Gui, Y. Wang, Y. Yang, K. Keutzer, and T. Darrell Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525. Cited by: §1, §2.1. Tan et al. (2025) H. Tan, Y. Ji, X. Hao, M. Lin, P. Wang, Z. Wang, and S. Zhang Reason-rft: reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.20752. Cited by: §1, §2.1. Wang et al. (2025a) H. Wang, C. Qu, Z. Huang, W. Chu, F. Lin, and W. Chen VL-rethinker: incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837. Cited by: §4.1. Wang et al. (2025b) W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao, Z. Wang, Z. Chen, H. Zhang, et al. InternVL3.5: advancing open-source multimodal models in versatility, reasoning, and efficiency. ArXiv abs/2508.18265. External Links: Link Cited by: Appendix F. Wang et al. (2025c) Z. Wang, X. Guo, S. Stoica, H. Xu, H. Wang, H. Ha, X. Chen, Y. Chen, M. Yan, F. Huang, and H. Ji Perception-aware policy optimization for multimodal reasoning. arXiv preprint arXiv:2507.06448. Cited by: §1, §2.1. Xiao et al. (2024) Y. Xiao, E. Sun, T. Liu, and W. Wang LogicVista: multimodal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973. Cited by: §4.1. Yang et al. (2026) W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin Learning beyond teacher: generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125. Cited by: §1, §2.2, §2.2. Ye et al. (2026) T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei On-policy context distillation for language models. arXiv preprint arXiv:2602.12275. Cited by: §2.2. Yu et al. (2023) T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H. Zheng, M. Sun, and T. Chua RLHF-v: towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:2312.00849. Cited by: §1, §2.1. Yue et al. (2024) X. Yue, T. Zheng, Y. Ni, Y. Wang, K. Zhang, S. Tong, Y. Sun, B. Yu, G. Zhang, H. Sun, Y. Su, W. Chen, and G. Neubig MMMU-pro: a more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813. Cited by: §4.1. Zhang et al. (2026) D. Zhang, Z. Yang, S. Janghorbani, J. Han, A. Ressler, Q. Qian, G. D. Lyng, S. S. Batra, and R. E. Tillman Fast and effective on-policy distillation from reasoning prefixes. arXiv preprint arXiv:2602.15260. Cited by: §2.2. Zhang et al. (2025a) J. Zhang, J. Huang, H. Yao, S. Liu, X. Zhang, S. Lu, and D. Tao R1-vl: learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937. Cited by: §1, §2.1. Zhang et al. (2024) R. Zhang, D. Jiang, Y. Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K. Chang, P. Gao, and H. Li MathVerse: does your multi-modal llm truly see the diagrams in visual math problems?. In European Conference on Computer Vision, Cited by: §4.1. Zhang et al. (2025b) Y. Zhang, X. Lu, S. Yin, C. Fu, W. Chen, X. Hu, B. Wen, K. Jiang, C. Liu, T. Zhang, H. Fan, K. Chen, J. Chen, H. Ding, K. Tang, Z. Zhang, L. Wang, F. Yang, T. Gao, and G. Zhou Thyme: think beyond images. ArXiv abs/2508.11630. External Links: Link Cited by: Appendix H. Zhang et al. (2025c) Y. Zhang, T. Yu, H. Tian, C. Fu, P. Li, J. Zeng, W. Xie, Y. Shi, H. Zhang, J. Wu, X. Wang, Y. Hu, B. Wen, F. Yang, Z. Zhang, T. Gao, D. Zhang, L. Wang, R. Jin, and T. Tan M-rlhf: the next step forward in multimodal llm alignment. arXiv preprint arXiv:2502.10391. Cited by: §2.1. Zheng et al. (2025) Z. Zheng, M. Yang, J. Hong, C. Zhao, G. Xu, L. Yang, C. Shen, and X. Yu DeepEyes: incentivizing "thinking with images" via reinforcement learning. ArXiv abs/2505.14362. External Links: Link Cited by: Appendix H. Appendix A Additional theoretical analysis This appendix provides additional theoretical justification for VC-OPD. The main text already shows that OPD can be interpreted as token-level policy optimization induced by the reverse KL divergence. Here we focus on three properties that are specific to VC-OPD: (i) the factualâcounterfactual teacher gap is a pointwise visual-dependence signal, (i) mass-preserving normalization stabilizes the total update scale while preserving relative token importance, and (i) the counterfactual branch modulates update magnitude without changing the update direction. A.1 Counterfactual teacher gap as visual likelihood ratio Let st=(x,v,y<t)s_t=(x,v,y_<t) and at=yta_t=y_t. For the same sampled token, VC-OPD evaluates the teacher under the factual visual input and the counterfactual visual input: pT+â(at)=pTâ(atâŁx,v,y<t),pTââ(at)=pTâ(atâŁx,v~,y<t).p_T^+(a_t)=p_T(a_t x,v,y_<t), p_T^-(a_t)=p_T(a_t x, v,y_<t). (14) The visual dependence score used in VC-OPD is Îtvis=logâĄpT+â(at)âlogâĄpTââ(at)=logâĄpTâ(atâŁx,v,y<t)pTâ(atâŁx,v~,y<t). _t^vis= p_T^+(a_t)- p_T^-(a_t)= p_T(a_t x,v,y_<t)p_T(a_t x, v,y_<t). (15) Thus, Îtvis _t^vis is a pointwise log-likelihood ratio between the factual and counterfactual teacher distributions. A positive value means that the real image increases the teacherâs support for the token relative to the counterfactual image, while a value close to zero means that the token is largely invariant to the removal of instance-specific visual evidence. This justifies using Îtvis _t^vis as a token-level grounding signal. VC-OPD converts this score into a non-negative soft weight: wt=ĎâĄ(ÎtvisâĎvTv).w_t=Ď\! ( _t^vis- _vT_v ). (16) Since the sigmoid function is monotonic, if Îivis>Îjvis _i^vis> _j^vis, then wi>wjw_i>w_j. Therefore, the soft weighting function preserves the ranking induced by the visual likelihood ratio: tokens more dependent on real visual evidence receive larger update weights. A.2 Mass-preserving normalization Directly using wtw_t may make the total update scale vary across responses, because different examples contain different numbers of visually grounded tokens. VC-OPD therefore uses the normalized weight wÂŻt=mtâwtâ Nâj=1Tmjâwj+Ďľ,N=ât=1Tmt. w_t=m_tw_t¡ N _j=1^Tm_jw_j+Îľ, N= _t=1^Tm_t. (17) where mtm_t is the response mask. Ignoring the small numerical constant Ͼξ, the normalized weights satisfy ât=1TwÂŻt=N. _t=1^T w_t=N. (18) Thus, the total token weight is approximately equal to the number of valid response tokens. At the same time, for any two valid tokens i and j, wÂŻiwÂŻj=wiwj. w_i w_j= w_iw_j. (19) Therefore, mass-preserving normalization keeps the relative visual emphasis unchanged while stabilizing the overall update magnitude. This property explains why VC-OPD can emphasize visually grounded tokens without changing the effective learning scale across batches. A.3 Directionâmagnitude separation The final VC-OPD advantage is AtVC=wÂŻtâ A~tdir,Atdir=logâĄpT+â(at)âlogâĄÎźâĄ(atâŁst),A_t^VC= w_t¡ A_t^dir, A_t^dir= p_T^+(a_t)- Îź(a_t s_t), (20) where A~tdir A_t^dir is a normalized version of the factual teacherâstudent direction term. The resulting objective is âVCâ-âOPDâ(θ)=âyâźÎźâ[ât=1TsgâĄ(AtVC)âlogâĄĎθâ(atâŁst)].L_VC -OPD(θ)=-E_y Îź [ _t=1^Tsg\! (A_t^VC ) _θ(a_t s_t) ]. (21) Taking the gradient gives âθâVCâ-âOPD=âyâźÎźâ[ât=1TsgâĄ(wÂŻtâA~tdir)ââθâlogâĄĎθâ(atâŁst)]. _θL_VC -OPD=-E_y Îź [ _t=1^Tsg\! ( w_t A_t^dir ) _θ _θ(a_t s_t) ]. (22) This is still an advantage-weighted policy-gradient estimator, but the advantage is reweighted by token-level visual dependence. Since wÂŻtâĽ0 w_t⼠0, the counterfactual branch cannot reverse the update direction: signâĄ(wÂŻtâAtdir)=signâĄ(Atdir)whenever âwÂŻt>0.sign\! ( w_tA_t^dir )=sign\! (A_t^dir ) w_t>0. (23) Therefore, the factual teacherâstudent discrepancy determines whether the student should increase or decrease the probability of a sampled token, while the counterfactual teacher only determines how strongly this update should be applied. This separation is important for stability: the corrupted visual input is used to estimate grounding strength, but it does not directly decide the sign of the optimization signal. A.4 View as selective token-level distillation VC-OPD can also be interpreted as performing OPD under a reweighted token distribution. For a fixed response, define qâĄ(tâŁy)=mtâwtâj=1Tmjâwj.q(t y)= m_tw_t _j=1^Tm_jw_j. (24) Then the VC-OPD objective is proportional to âyâźÎź[Nâ tâźq(â âŁy)[sg(A~tdir)logĎθ(atâŁst)]].-E_y Îź [N¡E_t q(¡ y) [sg\! ( A_t^dir ) _θ(a_t s_t) ] ]. (25) Thus, VC-OPD can be viewed as standard token-level OPD applied to a token distribution biased toward visually grounded tokens, while preserving the total update mass. This interpretation formalizes the main design goal of VC-OPD: retain the dense supervision advantage of OPD, but reallocate learning capacity toward tokens that are supported by real visual evidence. Appendix B Training Details B.1 Stage-I warm-start Before visual counterfactual on-policy distillation, we first warm-start the student with teacher-generated responses. For each training example, the teacher is prompted to produce a reasoning response followed by a boxed final answer. The prompt template is: content | trim You FIRST think about the reasoning process as an internal monologue and then provide the final answer. The reasoning process MUST BE enclosed within <think> </think> tags. The final answer MUST BE put in . After generation, we extract the final boxed answer and compare it with the ground-truth answer using the same answer-extraction and exact-match evaluation protocol as in the main experiments. Only examples for which the teacher produces a correct final answer are retained for warm-start training. This filtering removes noisy teacher trajectories and ensures that the student is initialized with reliable response formats and valid reasoning paths before entering on-policy optimization. The warm-start stage uses standard autoregressive supervision on the filtered teacher responses. Table 6 summarizes the warm-start hyperparameters. Table 6: Warm-start training hyperparameters. Hyperparameter Value Student model Qwen2.5-VL-3B-Instruct Teacher model Qwen2.5-VL-32B-Instruct Optimizer AdamW Learning rate 1Ă10â61Ă 10^-6 Batch size 256 Response format <think> reasoning + answer B.2 Stage-I on-policy training After warm-start, we perform on-policy optimization using the same training prompts. GRPO is trained with verifiable outcome-level rewards, while VC-OPD uses the student-generated trajectories for teacher scoring under both factual and counterfactual visual inputs. The stage-I implementation is based on the verl training framework. For VC-OPD, the factual teacher branch provides the token-level distillation direction, while the counterfactual teacher branch estimates visual dependence and reweights the token-level objective. Table 7 reports the main hyperparameters used for GRPO and VC-OPD. Table 7: Stage-I training hyperparameters for GRPO and VC-OPD. Hyperparameter GRPO VC-OPD Optimizer AdamW AdamW Learning rate 1Ă10â61Ă 10^-6 5Ă10â65Ă 10^-6 Batch size (rollouts) 5 1 Max response length 8192 8192 Max prompt length 4096 4096 Generation temperature 1.0 1.0 Top-p 1.0 1.0 Tensor parallel size 2 2 Visual token ratio â 10% Counterfactual corruption â Patch dropout 60% Appendix C Log-probability Diagnostics We further visualize the log-probability traces during VC-OPD training to verify that the factual and counterfactual teacher scoring remains numerically stable. At each optimization step, we record the average response-token log probability for the student model, the factual teacher, and the counterfactual teacher. The light curve denotes the raw step-wise value, and the darker curve denotes the smoothed trend. (a) Student model (b) Factual teacher (c) Counterfactual teacher Figure 5: Log-probability diagnostics during VC-OPD training. We plot the average response-token log probability of the student, factual teacher, and counterfactual teacher over training steps. The curves stabilize after the early optimization phase, indicating that both teacher-scoring branches remain in a comparable and numerically stable prediction regime. As shown in Figure 5, the student log probability increases rapidly in the early stage and then gradually stabilizes, reflecting the effect of warm-starting followed by on-policy adaptation. The factual and counterfactual teacher branches remain stable throughout training, which is important because VC-OPD relies on their log-probability difference to estimate token-level visual dependence. The counterfactual branch does not serve as an independent training target; instead, it provides a relative grounding signal by measuring how teacher support changes when instance-specific visual evidence is weakened. These diagnostics suggest that the counterfactual scoring procedure does not introduce unstable probability estimates and can be used reliably for grounding-aware token reweighting. Appendix D Token-level Grounding Verification The qualitative visualization in Figure 1 suggests that the factualâcounterfactual teacher gap more selectively highlights image-dependent tokens than the standard studentâteacher gap. To quantitatively verify this mechanism, we conduct a token-level grounding verification study on a subset of validation examples. We sample model responses from the validation set and annotate response tokens as visually grounded if their correctness depends on image-specific evidence, such as objects, quantities, spatial relations, diagram symbols, chart elements, or answer-relevant visual attributes. Tokens that only serve as discourse markers, generic reasoning templates, or language-prior continuations are annotated as non-visual. Given these annotations, we compare four token scoring signals: ⢠Random score: a random token ranking baseline. ⢠Token log-probability: the factual teacher log-probability of each token. ⢠Studentâteacher gap: the standard OPD score based on the teacherâstudent log-probability difference. ⢠Factualâcounterfactual gap: the proposed visual-dependence score used by VC-OPD. For each scoring signal, we rank all response tokens within each example and evaluate how well the top-ranked tokens match the human visual-grounding annotations. We report Precision@5%, Precision@10%, Recall@10%, and AUPRC. These metrics measure whether a scoring signal can retrieve visually grounded tokens from the full response. As shown in Table 8, the factualâcounterfactual gap provides the strongest alignment with human-annotated visually grounded tokens. The random baseline roughly reflects the sparsity of visually grounded tokens, while token log-probability performs only slightly better because high-probability tokens often correspond to common function words or response templates. The studentâteacher gap improves over these baselines, but it still ranks many generic reasoning tokens highly because it measures teacherâstudent disagreement rather than visual dependence. In contrast, the factualâcounterfactual gap achieves the highest precision and recall, indicating that it more accurately prioritizes tokens whose correctness depends on image-specific evidence. This result supports the central mechanism of VC-OPD: the counterfactual teacher branch does not merely identify high-discrepancy tokens, but provides a token-level signal that is more closely aligned with visual grounding. Table 8: Token-level grounding verification. We compare different token scoring signals for identifying human-annotated visually grounded tokens. Scoring signal P@5% P@10% Recall@10% AUPRC Random score 11.8 11.6 10.1 12.0 Token log-probability 18.7 17.4 15.2 18.6 Studentâteacher gap 36.5 31.8 27.7 34.2 Factualâcounterfactual gap 58.9 52.1 45.4 57.6 Appendix E Generalization to Qwen3-VL To evaluate whether VC-OPD transfers to newer VLM backbones, we further conduct experiments with Qwen3-VL-2B Bai et al. [2025a] and Qwen3-VL-4B students using Qwen3-VL-8B as the teacher. RKL and VC-OPD use the same student initialization, training data, rollout budget, and evaluation protocol. We additionally report the performance of the original base models to provide a direct reference for the effectiveness of post-training. Table 9: Generalization results on the Qwen3-VL family. We report exact-match accuracy (%) under the same evaluation protocol as the main experiments. Model Geo3K MVer. MVist. WeM. MMMU Logic Count Avg. Qwen3-VL-2B + Base 28.20 39.50 59.80 56.60 27.50 36.80 55.10 43.36 + RKL 39.50 42.80 63.00 60.50 31.40 40.90 61.20 48.47 + VC-OPD 42.40 44.20 64.50 61.90 33.80 42.80 63.80 50.49 ÎRKL _RKL (%) +7.34 +3.27 +2.38 +2.31 +7.64 +4.65 +4.25 +4.16 Qwen3-VL-4B + Base 35.20 42.80 64.00 61.80 32.20 41.50 62.00 48.50 + RKL 43.20 45.50 66.70 64.50 35.60 44.80 67.30 52.51 + VC-OPD 46.70 47.80 68.30 66.40 38.40 47.50 70.40 55.07 ÎRKL _RKL (%) +8.10 +5.05 +2.40 +2.95 +7.87 +6.03 +4.61 +4.87 VC-OPD consistently improves over RKL for both Qwen3-VL student scales. For the 2B and 4B students, VC-OPD achieves average accuracies of 50.4950.49 and 55.0755.07, corresponding to relative improvements of 4.16%4.16\% and 4.87%4.87\% over RKL, respectively. Compared with the corresponding base models, the gains are also substantial, with absolute improvements of 7.137.13 and 6.576.57 points in average accuracy. These results indicate that the proposed visual-dependent token reweighting remains effective when transferred to the newer Qwen3-VL family. Appendix F Generalization to InternVL3.5 To assess cross-family generalization, we further implement VC-OPD on the VERLâInternVL framework and evaluate InternVL3.5-2B Wang et al. [2025b] and InternVL3.5-4B students with an InternVL3.5-8B teacher. RKL and VC-OPD use the same student initialization, training data, rollout budget, and evaluation protocol. We also include the corresponding base-model performance as a reference for evaluating the effect of post-training. Table 10: Generalization results on the InternVL3.5 family. We report exact-match accuracy (%) under the same evaluation protocol as the main experiments. Model Geo3K MVer. MVist. WeM. MMMU Logic Count Avg. InternVL3.5-2B + Base 30.80 39.40 60.50 28.40 31.20 41.00 56.20 41.07 + RKL 37.90 42.10 64.90 35.20 34.00 39.80 61.00 44.99 + VC-OPD 40.80 44.20 66.10 37.40 35.80 42.30 63.10 47.10 ÎRKL _RKL (%) +7.65 +4.99 +1.85 +6.25 +5.29 +6.28 +3.44 +4.70 InternVL3.5-4B + Base 39.20 49.80 71.10 35.80 38.80 53.20 64.80 50.39 + RKL 44.70 53.10 73.50 47.00 42.00 54.10 68.50 54.70 + VC-OPD 47.30 55.00 74.40 49.10 44.30 56.00 71.20 56.76 ÎRKL _RKL (%) +5.82 +3.58 +1.22 +4.47 +5.48 +3.51 +3.94 +3.76 VC-OPD consistently outperforms RKL for both InternVL3.5 student scales, achieving relative improvements of 4.70%4.70\% and 3.76%3.76\% in average accuracy for the 2B and 4B students, respectively. Relative to the corresponding base models, VC-OPD further improves average accuracy by 6.036.03 and 6.376.37 points, respectively. The consistent gains across both student scales and a different VLM family demonstrate that VC-OPD is not restricted to the Qwen architecture used in the main experiments, and supports its cross-family generalization. Appendix G Training Efficiency VC-OPD evaluates the teacher under both factual and counterfactual visual conditions. The two conditions share the same student-generated trajectory and are concatenated along the batch dimension for a single batched teacher invocation. In our implementation, the additional counterfactual branch increases the wall-clock time per optimization step by approximately 20%20\%, rather than doubling the end-to-end training cost. Table 11 compares the wall-clock time required by RKL and VC-OPD to reach the same target Geo3K accuracy. Although VC-OPD introduces additional teacher computation at each optimization step, it consistently reaches the reported accuracy targets earlier than RKL. The difference becomes particularly evident at higher accuracy levels: VC-OPD reaches 32%32\% accuracy in 7.27.2 hours compared with 14.514.5 hours for RKL, and further reaches 33%33\% and 34%34\% in 14.414.4 and 17.417.4 hours, respectively, whereas RKL does not reach these targets within the reported training budget. These results indicate that the extra per-step computation can be offset by faster optimization progress, leading to improved time-to-accuracy. Table 11: Wall-clock time (hours) required to reach target Geo3K accuracies for Qwen2.5-VL-3B. Target Accuracy 29% 30% 31% 32% 33% 34% RKL 2.5 4.0 7.0 14.5 â â VC-OPD 1.5 2.8 5.0 7.2 14.4 17.4 Appendix H Comparison with Recent Visual Reasoning Methods We additionally compare the Qwen2.5-VL-7B student distilled from the Qwen2.5-VL-32B teacher with recent visual reasoning post-training methods on four shared benchmarks. Since DeepEyes Zheng et al. [2025] and Thyme-VL Zhang et al. [2025b] report results obtained under potentially different training data, evaluation splits, and tool-augmented inference procedures, this comparison should be interpreted as an external reference rather than a strictly controlled comparison. Table 12 reports the results on MathVista, MathVerse, We-Math, and LogicVista. VC-OPD achieves an average accuracy of 59.13%59.13\% across the four benchmarks. Compared with the reported results of DeepEyes and Thyme-VL, VC-OPD shows particularly large gains on We-Math, while the differences on the other benchmarks are more modest. Because the compared methods may differ in training resources, inference settings, and evaluation protocols, these results are provided primarily to contextualize the empirical performance of VC-OPD rather than to establish a controlled state-of-the-art comparison. Table 12: Comparison with recent visual reasoning methods on shared benchmarks. Method MathVista MathVerse We-Math LogicVista Avg. DeepEyes 70.10 47.30 38.90 47.70 51.00 Thyme-VL 70.00 39.10 39.30 49.00 49.35 VC-OPD 70.90 48.90 68.10 48.60 59.13