Paper deep dive
Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning
Feiding, Yongkang Zhang, Yuhao Liao, Zijian Zeng, Chunzheng Zhu, Yaozong Zheng, Yafei Liu, Yeling Peng, Youwei Wang, Sibo Wang, Huiming Yang, Linglin Liao, Shunzhi Yang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/31/2026, 2:05:55 AM
Summary
The paper introduces 'Difference Feedback' (DF), a mechanism for generating process-level supervision for Vision-Language Models (VLMs) during reinforcement learning. By training a repair model to generate minimal edits to erroneous trajectories, DF constructs token-level masks that gate gradient updates, effectively localizing and correcting reasoning errors without requiring expensive human step-by-step annotations. The method is compatible with existing frameworks like GRPO and PPO and demonstrates consistent performance improvements on multimodal benchmarks.
Entities (5)
Relation Signals (3)
Difference Feedback → integrateswith → GRPO
confidence 100% · As a plug-in module, Difference Feedback can be integrated into multiple optimization methods such as GRPO
Difference Feedback → evaluatedon → MMMStar
confidence 95% · Experiments on multimodal reasoning benchmarks including MMMStar and MathVista show an average 3% improvement
Difference Feedback → improves → VLM alignment
confidence 95% · Our approach offers an effective, low-cost solution for accurate vision–reasoning process alignment.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision--language models (VLMs) are increasingly aligned via Group Relative Policy Optimization (GRPO)-style training. However, relying solely on terminal outcome rewards yields sparse credit assignment in multi-step reasoning, weakening the linkage between visual evidence and intermediate steps and often causing unstable optimization and visual hallucinations. We propose Differential Feedback, which automatically constructs token/step-level supervision masks by repairing erroneous reasoning trajectories, explicitly marking the key positions that require correction. Without costly large-scale step-by-step human annotations, our method enables process-level visual alignment and can be seamlessly integrated into existing GRPO-like frameworks. Experiments on multimodal reasoning benchmarks including MMMStar and MathVista show an average 3% improvement under matched compute budgets. Our approach offers an effective, low-cost solution for accurate vision--reasoning process alignment.
Tags
Links
- Source: https://arxiv.org/abs/2603.27482v1
- Canonical: https://arxiv.org/abs/2603.27482v1
Trouble viewing inline? Open PDF directly →
Full Text
43,419 characters extracted from source content.
Expand or collapse full text
11institutetext: Shenzhen International Graduate School, Tsinghua University, China Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning Feiding Yongkang Zhang Yuhao Liao Zijian Zeng Chunzheng Zhu Yaozong Zheng Yafei Liu Yeling Peng Youwei Wang Sibo Wang Huiming Yang Linglin Liao Shunzhi Yang Abstract Vision–language models (VLMs) are increasingly aligned via Group Relative Policy Optimization (GRPO)-style training. However, relying solely on terminal outcome rewards yields sparse credit assignment in multi-step reasoning, weakening the linkage between visual evidence and intermediate steps and often causing unstable optimization and visual hallucinations. We propose Differential Feedback, which automatically constructs token/step-level supervision masks by repairing erroneous reasoning trajectories, explicitly marking the key positions that require correction. Without costly large-scale step-by-step human annotations, our method enables process-level visual alignment and can be seamlessly integrated into existing GRPO-like frameworks. Experiments on multimodal reasoning benchmarks including MMMStar and MathVista show an average 3% improvement under matched compute budgets. Our approach offers an effective, low-cost solution for accurate vision–reasoning process alignment. Figure 1: Difference Feedback (DF) provides fine-grained process supervision for VLM alignment. When the policy produces an incorrect trajectory, a small-edit repair is generated; the difference between the two outputs yields a token-level mask that gates gradient updates. 1 Introduction Vision-Language Models (VLMs) have demonstrated impressive performance on visual question answering, cross-modal reasoning, and multi-turn dialogue by integrating visual encoders with large language model generation. To improve instruction following and alignment, [21, 23] introduce reinforcement learning and preference optimization into VLM training, including GRPO [17], PPO [16], GSPO [24], and SAPO [8]. Despite impressive engineering progress, the training signal design universally relies on outcome supervision: models receive a single global reward or preference judgment only after generation completes, without fine-grained feedback on intermediate reasoning steps or critical visual decisions. Outcome supervision creates significant credit assignment difficulties in multimodal, multi-step reasoning. Different tokens, reasoning steps, and visual evidence choices contribute unequally to the final outcome; a single outcome signal amplifies policy gradient variance and makes updates susceptible to sampling noise—manifesting as training instability, slow convergence, and wasteful gradient updates. In challenging samples involving fine-grained recognition, counting, spatial relationships, or cross-modal constraints, errors typically concentrate in a few critical fragments (e.g., a wrong attribute word, a misidentified region, or a logical jump). Outcome signals cannot localize these critical steps, limiting stable improvement on long-tail hard cases. Concrete failure modes include inability to learn from hard examples and responses that ignore visual content in favor of language priors. Process-level supervision is a natural remedy: training a Process Reward Model (PRM) to score intermediate steps, or combining test-time search with step scoring. However, explicit process supervision requires expensive step-level annotation that is difficult to scale; test-time search incurs substantial computational overhead and is highly sensitive to evaluator quality. In multimodal tasks, process annotation further requires reliable visual evidence verification, amplifying cost and noise. Thus, obtaining stable, effective process-level training signals without large-scale annotation or heavy search overhead remains a central challenge in VLM alignment. We propose Difference Feedback (DF) to address this challenge (Fig. 1). Difference Feedback is not a specific optimizer, but rather a general mechanism for generating process-level supervision signals. When the policy model produces an incorrect trajectory, we train a multimodal repair model to generate a corrected answer while making relatively small edits to the original response. We then align the original answer with the repaired answer and automatically derive token-/step-level supervision masks from their differences, explicitly identifying which positions need to be corrected. This supervision signal can be plugged into a variety of alignment objectives, where the difference mask serves as token-level gradient gating to reduce ineffective updates. As a result, Difference Feedback preserves the scalability of training while providing more stable and interpretable local correction signals for long-horizon multimodal reasoning. We evaluate our approach on multiple multimodal reasoning and alignment benchmarks. Experimental results show that Difference Feedback consistently improves performance across different evaluation datasets. Overall, this work opens a new general pathway for VLM alignment: automatically constructing process-level supervision via output differences, leading to stronger multimodal alignment and reasoning capabilities. Contributions. The main contributions of this work are summarized as follows: • We introduce the Difference Feedback (DF) mechanism. Without modifying the neural architecture, the objective function, or requiring large-scale step-by-step human annotations, DF provides a general method for generating process-level supervision. Through a “repair → alignment → difference mask” pipeline, the method automatically constructs token-/step-level correction signals, and uses the difference mask to gate token-level updates, reducing the probability of gradients updating irrelevant tokens. • Compatibility with diverse alignment objectives. As a plug-in module, Difference Feedback can be integrated into multiple optimization methods such as GRPO, DAPO, GSPO, and PPO, reducing ineffective updates and improving the training efficiency of multimodal models. • Consistent improvements on multimodal reasoning and alignment tasks. Across multiple benchmarks, Difference Feedback yields consistent performance gains. 2 Related Work Alignment and Reinforcement Learning(RL) fine-tuning typically rely on sequence-level outcome reward/preference signals, causing credit assignment difficulties and high-variance updates in multi-step generation. Process supervision mitigates this but requires expensive step-level annotation and PRMs [12]. To avoid step-level labeling, one line of work constructs denser token-level signals from outcome feedback: [4] redistributes rewards using internal reward model information; [11] decomposes contributions for reward redistribution; [3] allocates marginal contributions from a Shapley perspective. Another line learns token-wise signals from preference optimization, typified by DPO [14] and token-level variants [25, 26, 5, 20]; [6] uses per-token generation probability as token-level reward. Unlike these methods that answer “how much credit should each token receive,” our work asks “which tokens/steps are the critical bottlenecks causing errors.” We localize erroneous positions by introducing lightly-edited repairs: rather than outputting token-level reward values, we generate token/step masks via edit alignment (Levenshtein/LCS) and inject them into GRPO/PPO/GSPO objectives, focusing negative advantage on segments requiring correction. Related work on minimal contrastive edits [15] demonstrates the value of minor edits for causal span localization. Self-refinement methods [13, 10] focus on test-time iterative revision. Implicit process rewards [22] provide token-level reward allocation but face a language capability–reasoning trade-off that limits scalability. To our knowledge, we are the first to introduce process-level supervision into multimodal RL training, using repair differences to improve multimodal alignment under sparse outcome supervision. 3 Method 3.1 Notation and Problem Setup We are given a multimodal input x=(,)x=(I,q), where I denotes an image (or a sequence of video frames) and q denotes a textual instruction/question. The policy model (VLM) is denoted by πθ _θ, which autoregressively generates an output sequence =(y1,…,yT).y=(y_1,…,y_T). We define the state at step t as st=(x,<t)s_t=(x,y_<t). We consider a family of alignment optimization objectives (θ)J(θ), whose training signals are typically derived from terminal rewards/preferences, e.g., a sequence-level reward R(x,)R(x,y) or a preference pair (+,−)(y^+,y^-). Such terminal supervision is sparse, making credit assignment difficult; in practice, this manifests as high-variance gradients and many ineffective updates. To address this issue, we propose Difference Feedback (DF): a general mechanism for constructing process-level supervision signals, which can be plugged into and enhance various optimization objectives such as GRPO, PPO, and GSPO. 3.2 Difference Feedback Generation (DF) Core idea. When the policy output y is incorrect (or receives a low score), we use a multimodal repair model ρϕ _φ to produce a repaired output ~ y with minimal edits. We then align y and ~ y and derive a token-/step-level supervision mask m from their differences, indicating which positions should be updated. We first train the multimodal repair model (described in Sec. 3.3), and then use it to generate difference feedback (detailed in Sec. 3.4). 3.3 Train the Repair Model: Two-Stage (Supervised Fine-Tuning(SFT) + RL) 3.3.1 Stage 1: SFT Warm Start We manually construct a small repair dataset (1,000 samples) rep=(x,,,∗)D_rep=\(x,y,a,y^*)\, where y denotes an incorrect output, ∗y^* denotes the repaired target output, and a denotes the correct reference answer. The pair (,)(y,a) can be obtained by prompting the policy model to generate candidate outputs followed by correctness filtering, while a and ∗y^* are manually annotated. We initialize the repair model using supervised fine-tuning (SFT) with the objective: ℒSFT(ϕ)=−(x,,,∗)∼rep[∑t=1T∗logρϕ(yt∗∣x,,,<t∗)].L_SFT(φ)=-E_(x,y,a,y^*) _rep [ _t=1^T^* _φ(y^*_t x,y,a,y^*_<t) ]. (1) 3.3.2 Stage 2: Learning to Repair with Reinforcement Learning To prevent the repair model from rewriting the entire response, we further train it with reinforcement learning to encourage correct repairs with relatively small edits. For a repaired output ~ y, we define a task correctness reward rtask(x,~)∈[0,1]r_task(x, y)∈[0,1]. This reward reuses the correctness evaluation used for training the policy model: for mathematical problems we use a rule-based reward that checks only the final answer, while for general tasks we use an LLM-based judge. We also define an edit cost measuring how many tokens in y are modified: Δedit(,~)=EditDist(,~). _edit(y, y)=EditDist(y, y). (2) Reward-hacking auditing (gating). We introduce an auditor based on Qwen2.5-VL-72B-Instruct to inspect repair trajectories. The audit outcome is formalized as a gating factor gaud(x,,~)∈0,1g_aud(x,y, y)∈\0,1\. If the auditor detects potential reward-hacking behaviors (e.g., the answer remains correct after removing the image while the reasoning still claims visual evidence, template-based shortcuts, or spurious difference masks caused by compensatory reasoning or paraphrasing—i.e., mask hallucination), we set gaud=0g_aud=0; otherwise gaud=1g_aud=1. The auditor does not participate in gradient updates and is used only for sample filtering and failure-mode statistics. We report both the audit hit rate and the agreement rate with manual inspection to evaluate its reliability. Image-removal penalty (C1). To discourage shortcut behaviors where the model can answer correctly without using visual information, we construct a counterfactual sample without visual input x∅=(∅,)x_ =(I_ ,q), where ∅I_ is a blank image (all zeros) or a random-noise image. We define a visual-dependence gating factor: gvis(x,~)=[rtask(x∅,~)<1]∈0,1.g_vis(x, y)=I\! [r_task(x_ , y)<1 ]∈\0,1\. (3) If the repaired output is still judged correct after removing the image, we treat it as not relying on genuine visual evidence and set the reward to zero (gvis=0g_vis=0). We further verify with the policy model whether the problem can be solved using text alone; if so, the corresponding training sample is removed. Final repair reward (with auditing and C1). We integrate the auditing and visual-dependence constraints into the repair reward: rrep(x,,~)=gaud(x,,~)⋅gvis(x,~)⋅(rtask(x,~)−λ⋅Δedit(,~)max(1,||)).r_rep(x,y, y)=g_aud(x,y, y)· g_vis(x, y)· (r_task(x, y)-λ· _edit(y, y) (1,|y|) ). (4) In Stage 2, the repair model is optimized using the same reinforcement learning algorithm as the policy (for example, if the policy uses GRPO, the repair model also uses GRPO), with the only difference being the reward function. 3.4 Obtaining Difference Feedback Given incorrect trajectory y, we independently sample n repairs from the repair model: ~(i)∼ρϕ(⋅∣x,,),i=1,…,n, y^(i) _φ(· x,y,a), i=1,…,n, (5) where a is a reference signal (annotated answer, a preferred response in preference data, or a correct trajectory uniformly sampled from corG_cor). The repair model is initialized as ϕ←θφ←θ to keep repairs in-distribution. We define the feasible set: (x,)=~(i)∣rtask(x,~(i))=1,S(x,y)= \ y^(i) r_task(x, y^(i))=1 \, (6) and select the optimal (minor-edit) repair: ~=argmin~∈(x,)Δedit(,~). y= _ y (x,y) _edit(y, y). (7) 3.4.1 Difference Alignment and Supervision Mask. Given original output y and repair ~ y, the alignment operator A yields two-sided token-level masks: (,′)=(,~),(m,m )=A(y, y), (8) where ∈0,1Tm∈\0,1\^T marks tokens in y requiring correction and ′∈0,1T~m ∈\0,1\ T marks the corresponding repair tokens in ~ y. Concretely, A is implemented via the Levenshtein edit path (or LCS): mt=1m_t=1 at substitution/deletion/insertion positions and mt=0m_t=0 otherwise. Soft Weights (Optional). Beyond binary masks, soft weights wt∈[0,1]w_t∈[0,1] with distance-based decay can be used: wt=mt⋅exp(−α(t−τ)).w_t=m_t· \! (-α(t-τ) ). (9) For clarity we use binary masks below; soft weights are a direct drop-in replacement. 3.5 Plug-in Injection of Difference Feedback: A Unified View We formalize the plug-in nature of Difference Feedback (DF) as follows. For any objective based on token-level log-likelihood, DF introduces a gating term gtg_t—either a binary mask or a soft weight—to reweight the contribution of each token to the update: gt∈[0,1],gt=mtorgt=wt.g_t∈[0,1], g_t=m_t\ \ or\ \ g_t=w_t. (10) Intuitively, gtg_t determines whether a token should receive credit assignment from terminal supervision. Tokens identified by DF as responsible for the error receive higher update weights, while irrelevant tokens are suppressed. Due to space limitations, we present below the concrete formulations of DF when integrated with GRPO and PPO. 3.6 DF-GRPO: Repair-Pair-Based Group Relative Policy Optimization Sampling an initial group ′G . For each input x, we sample a group of candidate trajectories from the old policy πθold _ _old: ′(x)=yii=1G′,yi∼πθold(⋅∣x),G (x)=\y_i\_i=1^G , y_i _ _old(· x), (11) where |′(x)|=G′|G (x)|=G . Splitting trajectories by reward threshold τr _r. Using the terminal reward r(x,y)r(x,y) and a threshold τr _r, we partition ′(x)G (x) into correct and erroneous trajectories: cor(x)=y∈′(x)∣r(x,y)≥τr,err(x)=y∈′(x)∣r(x,y)<τr.G_cor(x)=\y (x) r(x,y)≥ _r\, _err(x)=\y (x) r(x,y)< _r\. (12) Sampling a reference a from correct trajectories. If cor(x)≠∅G_cor(x)≠ , we uniformly sample a reference trajectory a∼Unif(cor(x)).a (G_cor(x) ). (13) If no correct trajectory exists, this sample cannot provide a meaningful reward signal and is skipped. Repairing erroneous trajectories to obtain repG_rep. For each erroneous trajectory yi∈err(x)y_i _err(x), we generate a repaired trajectory using the repair model: y~i∼ρϕ(⋅∣x,yi,a),yi∈err(x), y_i _φ(· x,y_i,a), y_i _err(x), (14) and collect them into rep(x)=y~i∣yi∈err(x).G_rep(x)=\ y_i y_i _err(x)\. (15) Constructing the optimization group G (size G). We remove all correct trajectories from ′(x)G (x) and add the repaired versions of erroneous trajectories: (x)=(′(x)∖cor(x))∪rep(x)=err(x)∪rep(x),G(x)= (G (x) _cor(x) )\ ∪\ G_rep(x)=G_err(x)\ ∪\ G_rep(x), (16) and denote |(x)|=G.|G(x)|=G. (17) Group-normalized advantages in GRPO. For each trajectory z∈(x)z (x), we compute a sequence-level advantage A^(z) A(z) obtained by normalizing rewards within the group (x)G(x). The token-level likelihood ratio is defined as wz,t(θ)=πθ(zt∣x,z<t)πθold(zt∣x,z<t).w_z,t(θ)= _θ(z_t x,z_<t) _ _old(z_t x,z_<t). (18) Two-sided difference gating for error–repair pairs. For each error–repair pair (yi,y~i)(y_i, y_i), we apply an alignment operator A (see Sec. 3.4.1) to obtain token-level difference masks on both sides: (mi,1:|yi|,mi,1:|y~i|′)=(yi,y~i), (m_i,1:|y_i|,\ m _i,1:| y_i| )=A(y_i, y_i), (19) where mi,t∈0,1m_i,t∈\0,1\ marks positions in the erroneous trajectory yiy_i that require correction, and mi,t′∈0,1m _i,t∈\0,1\ marks the corresponding repair tokens in y~i y_i. We apply the gating to the two trajectories respectively: gz,t(x)=mi,t,z=yi,yi∈err(x),mi,t′,z=y~i,y~i∈rep(x).g_z,t(x)= casesm_i,t,&z=y_i,\ \ y_i _err(x),\\[3.0pt] m _i,t,&z= y_i,\ \ y_i _rep(x). cases (20) The resulting token-level gated advantage is A^z,tDF=gz,t⋅A^(z). A^DF_z,t=g_z,t· A(z). (21) DF-GRPO objective. DF-GRPO(θ) _DF-GRPO(θ) =x∼,(x)[1G∑z∈(x)1|z|∑t=1|z|ℒz,tclip]−βkl(θ), =E_x ,\,G(x) [ 1G _z (x) 1|z| _t=1^|z|L_z,t^clip ]- _kl\,K(θ), (22) where ℒz,tclip=min(wz,t(θ)A^z,tDF,clip(wz,t(θ),1−ϵ,1+ϵ)A^z,tDF).L_z,t^clip= (w_z,t(θ)\, A^DF_z,t,clip(w_z,t(θ),1-ε,1+ε)\, A^DF_z,t ). (23) 3.7 DF-PPO: Masked PPO-Clip (General Advantage Injection) PPO defines the token-level likelihood ratio: rt(θ)=πθ(yt∣st)πθold(yt∣st).r_t(θ)= _θ(y_t s_t) _ _old(y_t s_t). (24) The standard PPO-Clip surrogate objective (to maximize) is PPO(θ)=[∑t=1Tmin(rt(θ)A^t,clip(rt(θ),1−ϵ,1+ϵ)A^t)].J_PPO(θ)=E [ _t=1^T (r_t(θ) A_t,\ clip(r_t(θ),1-ε,1+ε) A_t ) ]. (25) Key idea: different gating for correct and incorrect trajectories. Difference Feedback is applied only to “incorrect/low-reward” trajectories in order to localize the segments that require correction. For “correct/high-reward” trajectories, we retain the full PPO positive reinforcement signal to avoid discarding useful learning signals. Specifically, let the terminal reward (or correctness signal) be R(x,)R(x,y) and the threshold be τr _r. We define the gating coefficient as gt(x,)=1,R(x,)≥τr,mt(,~),R(x,)<τr,g_t(x,y)= cases1,&R(x,y)≥ _r,\\ m_t(y, y),&R(x,y)< _r, cases (26) where, when R(x,)<τrR(x,y)< _r, we generate a repaired trajectory ~∼ρϕ(⋅∣x,,) y _φ(· x,y,a) using the repair model, and obtain the truncated difference mask mtm_t via alignment (Eq. (8)). Masked advantage. We gate the advantage as A^tDF=gt⋅A^t, A^DF_t=g_t· A_t, (27) which yields the DF-PPO objective (to maximize): DF-PPO(θ)=[∑t=1Tmin(rt(θ)A^tDF,clip(rt(θ),1−ϵ,1+ϵ)A^tDF)]−βkl(θ).J_DF-PPO(θ)=E [ _t=1^T (r_t(θ) A^DF_t,\ clip(r_t(θ),1-ε,1+ε) A^DF_t ) ]- _klK(θ). (28) When the trajectory is correct, gt≡1g_t≡ 1 and DF-PPO reduces to the standard PPO update. When the trajectory is incorrect, gt=mtg_t=m_t restricts negative advantages to only the tokens or steps that require correction, thereby significantly reducing erroneous credit assignment and ineffective updates caused by sparse terminal supervision. 3.8 Why Difference Feedback Reduces Ineffective Updates: A Counterfactual Path Perspective (Brief) The central role of Difference Feedback (DF) is not to claim a strict variance-reduction guarantee for arbitrary tasks, but rather to mitigate misattribution under terminal supervision by leveraging a counterfactual reference trajectory provided by the repair model. In this work, the quality of a trajectory is defined in terms of terminal correctness. Given a reward function rtaskr_task, any generated sequence that is judged correct at the final step can be regarded as an effective trajectory and can therefore serve as a valid training signal. We do not require the intermediate reasoning in the repaired trajectory to match the original trajectory or to follow a unique reasoning form. Instead, the key question is whether there exists a trajectory that can flip the terminal judgment from incorrect to correct, and which local segments must be modified to achieve such a flip. Concretely, when the policy output y is an incorrect or low-scoring trajectory, we construct a repaired output ~ y that satisfies the correctness criterion while applying relatively small edits, as described in Eq. (7). We then apply an alignment operator to derive a difference mask and its truncated variant (Eq. (8)). This construction implies a direct and interpretable property: there exists a counterfactual path that flips the terminal judgment from incorrect to correct, and the required modifications are concentrated in the difference segments. The small-edit constraint further discourages opportunistic success via large-scale rewriting, making the resulting difference mask more likely to highlight the key bottleneck segments responsible for the judgment flip. Consequently, the difference mask can be viewed as a local correction indicator that is sufficient to flip the terminal decision. Under purely terminal reward or preference supervision, sequence-level advantages are typically broadcast across the entire generated output. This causes negative attribution to accumulate on many tokens that are unrelated to the actual error, resulting in substantial ineffective updates and unstable training dynamics. DF addresses this issue by applying token-level gating gtg_t (Eq. (10)) on erroneous trajectories, so that negative advantages and local preference constraints primarily act on the difference segments indicated by the counterfactual path, rather than uniformly affecting the entire sequence. It is important to emphasize that a counterfactual modification being sufficient to flip the judgment does not imply that masked tokens are causally irrelevant to the error. Compensatory effects, solution multiplicity, or semantic rewriting may cause the apparent small edits to deviate from the true causal bottleneck. However, the goal of the gating mechanism is not strict causal attribution. Instead, it aims to reduce the broadcast of negative signals and the update noise on irrelevant tokens under sparse terminal supervision, thereby improving learning efficiency and training stability. 3.9 Computational Complexity and Scalability The additional computational overhead introduced by Difference Feedback mainly arises from a single inference pass of the repair model, which is triggered only for incorrect samples. The expected cost can be approximated as: CostDF≈Costbase+perr⋅Cost(ρϕ decode),Cost_DF _base+p_err·Cost ( _φ decode ), (29) where perrp_err denotes the proportion of samples that trigger the repair process, which typically decreases as training progresses. Unlike test-time search methods, Difference Feedback does not introduce exponential branching during decoding. Consequently, it maintains favorable scalability and can be efficiently applied to large-scale VLM alignment training. 4 Experiments 4.1 Implementation Details We conduct experiments using the base models Qwen2.5-VL-7B and Qwen2.5-VL-32B [1]. The standard baseline is trained with GSPO, while our method uses DF-GSPO. Training is performed on our curated multimodal mathematical reasoning dataset (image–text pairs) using 32 NVIDIA A800 GPUs. Unless otherwise specified, we use the following hyperparameters: a rollout batch size of 128, rollout temperature of 0.7, and a learning rate of 1×10−61× 10^-6. The reward threshold τr _r is set to 1, n is set to 2 and the edit penalty coefficient λ is set to 0.5. For the base model InternVL3.5-8B-MPO [18], we compare GRPO with DF-GRPO. Training is conducted using the MMPR-Tiny dataset [18]. The rollout batch size is 512, rollout temperature is 0.7, and the learning rate is 1×10−61× 10^-6. We use the same reward threshold τr=1 _r=1 , n is set to 2 and edit penalty coefficient λ=0.5λ=0.5. 4.1.1 Compute-Matching Protocol To ensure fair comparison, we use identical model architectures and optimization settings across methods, and match the overall training computation by accounting for the total GPU time consumed. Specifically, for DF-based methods we include both the cost of training the repair model and the cost of policy optimization. For example, the GPU time used by the GRPO baseline is matched to the sum of (i) the GPU time spent training the repair model and (i) the GPU time spent training the policy model with DF-GRPO. This protocol ensures that all compared methods operate under comparable computational budgets. 4.2 Main Results Table 1 presents comprehensive results at 7B and 32B scales across benchmarks covering both specialized and general multimodal reasoning. Table 1: Main experimental results. For selected experiments, we report the mean over five random seeds together with the 95% bootstrap confidence interval (mean ± 95% CI). Improvements over the baseline methods are statistically significant under a paired bootstrap test (p<0.01p<0.01). Model MathVista MMStar MMMU MathVerse MathVision AI2D InternVL3.5-8B-MPO [18] 75.9 – 71.2 54.8 52.6 – InternVL3.5-8B-MPO (PRIME [7]) 78.5 – 73.6 61.9 56.9 – InternVL3.5-8B-MPO (GRPO) 78.4 – 73.4 61.5 56.8 – InternVL3.5-8B-MPO (DF-GRPO) 79.9±1.179.9_± 1.1 – 74.8±0.674.8_± 0.6 63.5±0.963.5_± 0.9 58.2±1.058.2_± 1.0 – Qwen2.5-VL-7B [2] 68.2 63.9 58.6 49.2 25.1 83.9 ThinkLite-7B [19] 74.3 63.7 53.1 52.2 29.9 83.0 R1-ShareVL-7B [21] 75.4 67.0 58.1 52.8 29.5 84.5 Vision-G1 [23] 76.1 66.0 53.4 – – – Vision-R1 [9] 73.5 – – 52.4 – – Qwen2.5-VL-7B (GSPO) 73.6±1.073.6_± 1.0 65.9±0.865.9_± 0.8 56.7±1.556.7_± 1.5 50.8±1.350.8_± 1.3 27.4±1.427.4_± 1.4 84.5±0.884.5_± 0.8 Qwen2.5-VL-7B (DF-GSPO) 76.1±1.876.1_± 1.8 68.3±1.368.3_± 1.3 58.4±0.858.4_± 0.8 53.7±0.953.7_± 0.9 30.4±1.230.4_± 1.2 85.6±0.685.6_± 0.6 Qwen2.5-VL-32B [2] 74.7 69.5 70.0 49.9 38.4 84.6 R1-ShareVL-32B [21] 77.6 70.2 70.1 59.0 40.3 86.2 Qwen2.5-VL-32B (GSPO) 75.5±1.375.5_± 1.3 68.3±1.468.3_± 1.4 66.8±0.966.8_± 0.9 57.2±0.757.2_± 0.7 35.7±1.235.7_± 1.2 85.7±0.585.7_± 0.5 Qwen2.5-VL-32B (DF-GSPO) 78.8±1.278.8_± 1.2 71.5±1.571.5_± 1.5 70.6±0.870.6_± 0.8 61.3±1.861.3_± 1.8 41.6±0.941.6_± 0.9 87.8±0.787.8_± 0.7 Comparison with RL-trained MLLMs. Training Qwen2.5-VL with GSPO and DF-GSPO under matched GPU time, DF-GSPO consistently outperforms GSPO at 7B—with notable gains on MathVista and MathVerse. GSPO exhibits performance drops on MMMU, likely due to catastrophic forgetting from wasteful updates; DF-GSPO substantially mitigates this. At 32B, standard GSPO shows drops on multiple metrics, suggesting larger models are more susceptible to wasteful updates; DF-GSPO recovers and surpasses all baselines. Using InternVL3.5-8B-MPO as a distinct backbone, DF-GRPO outperforms GRPO, confirming the generality of the DF mechanism. Comparison with Other Process Reward Methods. PRIME [7] applies implicit process rewards [22]. DF outperforms PRIME across all evaluated metrics, likely because PRIME suffers from a fundamental objective inconsistency: PRM training progressively degrades the model’s base language capability, limiting scalability. 4.3 Ablation Study Goal and Setup. To quantify the contribution of each component in Difference Feedback (DF), we conduct an ablation study on Qwen2.5-VL-7B using the same training data, hyperparameters, and evaluation protocol as in the main experiments. Unless explicitly removed, all other components remain identical: the same GSPO configuration, rollout temperature and batch size, KL coefficient, and number of training steps. Control Variants. The ablated variants are listed below (the default configuration is denoted as DF-GSPO): • GSPO (w/o DF): No difference-feedback gating is applied, equivalent to standard GSPO (gt≡1g_t≡ 1). • DF w/ LCS Align: The alignment operator A is replaced from the Levenshtein shortest-edit-path alignment to LCS-based alignment. • DF w/ Soft Weight: Replace the binary mask with the soft weighting scheme wtw_t defined in Eq. (9). • DF w/ SFT-only Repair: The repair model is trained only with Stage-1 repair SFT, without the Stage-2 RL stage that encourages correct repairs with relatively small edits. • DF w/o Edit Penalty: Remove the edit penalty term in the repair reward (Eq. (4), setting λ=0λ=0) to examine the effect of excessive rewriting on mask quality. • DF w/o C1 (Image-Removal Penalty): Remove the image-removal penalty (Eq. (4)), i.e., the counterfactual test on x∅x_ is disabled. This variant evaluates the contribution of C1 in suppressing degeneration caused by text-only shortcuts or ignoring visual evidence. Table 2: Ablation study (7B). Impact of each component on multimodal reasoning and alignment performance. Accuracy (%), higher is better. Method MathVista MMStar MMMU MathVerse MathVision AI2D GSPO (w/o DF) 73.6 65.9 56.7 50.8 27.4 84.5 DF w/ LCS Align 75.9 67.5 57.1 52.3 29.7 84.6 DF w/ Soft Weight 75.5 67.6 57.2 53.5 30.3 85.4 DF w/ SFT-only Repair 73.3 65.1 56.5 50.5 26.2 84.3 DF w/o Min-Edit (λ=0λ=0) 75.4 67.0 56.9 52.1 28.1 84.8 DF w/o C1 (Image-Removal) 75.2 68.1 57.0 53.3 30.2 84.9 DF-GSPO (Full) 76.1 68.3 58.4 53.7 30.4 85.6 Result analysis. (1) Removing DF (GSPO w/o DF) leads to a clear overall drop, indicating that difference-based gating effectively alleviates misattribution under terminal supervision. (2) LCS alignment performs worse than the shortest edit path, as it does not model substitution operations well, resulting in inaccurate difference localization and degraded performance. (3) The SFT-only repair model degrades substantially, suggesting that without Stage-2 (SFT+RL) training it cannot reliably produce counterfactual answers that are both correct and involve relatively small edits, which in turn reduces the quality of the difference masks. (4) DF w/o Edit Penalty (λ=0λ=0) consistently performs worse, indicating that excessive rewriting expands the difference mask from critical corrections to many irrelevant edits, thereby increasing gating noise. (5) Removing C1 leads to a mild degradation, suggesting that the counterfactual image-removal test helps suppress shortcut behaviors such as ignoring visual inputs or relying on text-only heuristics. Sensitivity analysis of λ. We analyze the effect of the edit-penalty coefficient λ (Eq. (4)) on the potential mismatch between superficial edits and true causal errors, which may lead to reward hacking. Repair accuracy is evaluated using the auditing model together with manual verification. As shown in Table 4, when λ<0.5λ<0.5, the repair accuracy remains relatively stable. As λ increases further, the repair model becomes increasingly reluctant to modify the original output; consequently, many errors cannot be corrected to the correct answer, and reward-hacking behaviors become more likely. Effect of the number of repair samples n. In Table 4, we study the performance of DF-GSPO under different numbers of repair samples n while keeping the total GPU time fixed. We observe that when n=2n=2, the model trained with DF-GSPO achieves the best performance on MathVista. As the number of samples increases, the additional computational overhead reduces the effective number of optimization steps within the same GPU budget, resulting in smaller performance gains. Table 3: Sensitivity of λ. Acc@1: pass@1; Acc@2: pass@2 (n=2n=2). λ 0.2 0.4 0.5 0.6 0.8 Acc@1 (%) 81.3 81.1 80.9 78.2 74.2 Acc@2 (%) 83.2 82.9 82.1 79.6 75.7 Table 4: Effect of sampling count n (matched GPU time). n 1 2 3 4 Acc@1 (%) 75.52 76.11 76.07 75.95 5 Conclusion We address the widely observed problem of sparse terminal supervision in RL-based alignment for VLMs (e.g., GRPO), which leads to difficult credit assignment, high-variance updates, unstable training, slow learning on hard examples, and insufficient utilization of visual evidence. We propose Difference Feedback (DF), a novel process-supervision paradigm that automatically constructs token-/step-level supervision through a repair → alignment → masking pipeline. Specifically, a multimodal repair model generates a correct version of the erroneous trajectory with relatively small edits; the resulting difference mask is then injected into the alignment objective as a gradient gating signal, focusing negative credit assignment on the critical segments responsible for the error. This mechanism substantially reduces ineffective updates and yields consistent improvements across multiple model scales on multimodal reasoning benchmarks. We anticipate that continued scaling of process-supervised RL training will further drive fundamental advances in multimodal intelligence. 6 Limitations Potential failure modes include misalignment between minimal surface edits and the true causal error spans due to paraphrasing or reasoning style shifts. In addition, the effectiveness of Difference Feedback depends on the repair model producing correct trajectories with relatively small edits; inaccurate or overly large repairs may introduce noisy difference masks. Finally, DF requires reliable correctness signals and incurs additional inference from the repair model, which may introduce modest overhead in large-scale training. Ethics Statement This work involves no significant ethical risks. Reproducibility Statement Code will be anonymously released via a public repository (GitHub) to enable reproduction and independent verification of all results. References [1] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. (2025) Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §4.1. [2] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. (2025) Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: Table 1, Table 1. [3] M. Cao, S. Zhang, X. Chang, and D. Precup (2025) SCAR: shapley credit assignment for more efficient rlhf. External Links: 2505.20417 Cited by: §2. [4] A. J. Chan, H. Sun, S. Holt, and M. van der Schaar (2024) Dense reward for free in reinforcement learning from human feedback. External Links: 2402.00782 Cited by: §2. [5] H. Chen, T. Yang, S. Gao, R. Chen, X. Quan, H. Tian, and T. Yao (2025) Discriminative policy optimization for token-level reward models. External Links: 2505.23363 Cited by: §2. [6] Z. Chen, K. Zhou, W. X. Zhao, J. Wan, F. Zhang, D. Zhang, and J. Wen (2024) Improving large language models via fine-grained reinforcement learning with minimum editing constraint. External Links: 2401.06081 Cited by: §2. [7] G. Cui, L. Yuan, Z. Wang, H. Wang, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, et al. (2025) Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. Cited by: §4.2, Table 1. [8] C. Gao, C. Zheng, X. Chen, K. Dang, S. Liu, B. Yu, A. Yang, S. Bai, J. Zhou, and J. Lin (2025) Soft adaptive policy optimization. arXiv preprint arXiv:2511.20347. Cited by: §1. [9] W. Huang, B. Jia, Z. Zhai, S. Cao, Z. Ye, F. Zhao, Y. Hu, and S. Lin (2025) Vision-r1: incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749. Cited by: Table 1. [10] A. Kumar, V. Zhuang, R. Agarwal, Y. Su, J. D. Co-Reyes, A. Singh, K. Baumli, S. Iqbal, C. Bishop, R. Roelofs, L. M. Zhang, K. McKinney, D. Shrivastava, C. Paduraru, G. Tucker, D. Precup, F. Behbahani, and A. Faust (2024) Training language models to self-correct via reinforcement learning. External Links: 2409.12917 Cited by: §2. [11] J. Li, T. Chang, F. Zhang, K. Kuang, and L. Chen (2024) R3HF: reward redistribution for enhancing reinforcement learning from human feedback. External Links: 2411.08302 Cited by: §2. [12] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023) Let’s verify step by step. External Links: 2305.20050 Cited by: §2. [13] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark (2023) Self-refine: iterative refinement with self-feedback. External Links: 2303.17651 Cited by: §2. [14] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. External Links: 2305.18290 Cited by: §2. [15] A. Ross, A. Marasović, and M. Peters (2021-08) Explaining NLP models via minimal contrastive editing (MiCE). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online, p. 3840–3852. External Links: Document, Link Cited by: §2. [16] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1. [17] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y.K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300 Cited by: §1. [18] W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao, et al. (2025) InternVL3.5: advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265. Cited by: §4.1, Table 1. [19] X. Wang, Z. Yang, C. Feng, H. Lu, L. Li, C. Lin, K. Lin, F. Huang, and L. Wang (2025) SoTA with less: mcts-guided sample selection for data-efficient visual reasoning self-improvement. arXiv preprint arXiv:2504.07934. Cited by: Table 1. [20] H. Xia, S. Gao, Q. Ge, Z. Xi, Q. Zhang, and X. Huang (2024) Inverse-q*: token level reinforcement learning for aligning large language models without preference data. External Links: 2408.14874 Cited by: §2. [21] H. Yao, Q. Yin, J. Zhang, M. Yang, Y. Wang, W. Wu, F. Su, L. Shen, M. Qiu, D. Tao, et al. (2025) R1-sharevl: incentivizing reasoning capability of multimodal large language models via share-grpo. arXiv preprint arXiv:2505.16673. Cited by: §1, Table 1, Table 1. [22] L. Yuan, W. Li, H. Chen, G. Cui, N. Ding, K. Zhang, B. Zhou, Z. Liu, and H. Peng (2024) Free process rewards without process labels. arXiv preprint arXiv:2412.01981. Cited by: §2, §4.2. [23] Y. Zha, K. Zhou, Y. Wu, Y. Wang, J. Feng, Z. Xu, S. Hao, Z. Liu, E. P. Xing, and Z. Hu (2025) Vision-g1: towards general vision language reasoning with multi-domain data curation. arXiv preprint arXiv:2508.12680. Cited by: §1, Table 1. [24] C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. (2025) Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: §1. [25] H. Zhong, Z. Shan, G. Feng, W. Xiong, X. Cheng, L. Zhao, D. He, J. Bian, and L. Wang (2024) DPO meets ppo: reinforced token optimization for rlhf. External Links: 2404.18922 Cited by: §2. [26] W. Zhou, S. Zhang, L. Zhao, and T. Meng (2024) T-reg: preference optimization with token-level reward regularization. External Links: 2412.02685 Cited by: §2.