Paper deep dive
When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning
Xiuyi Lou, Zicheng Xu, Yu-Neng Chuang, Hoang Anh Duy Le, Zhaozhuo Xu, Guanchu Wang, Vladimir Braverman
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/10/2026, 3:06:11 AM
Summary
The paper identifies a critical failure mode in critic-free reinforcement learning for LLMs called Positive-Credit Contamination, where uniform credit assignment reinforces contextually erroneous low-probability tokens. To address this, the authors propose Tail-Aware Credit calibratiOn (TACO), which computes a tail-risk score based on token probability and local entropy to calibrate positive credit. TACO softly suppresses updates for risky tokens while preserving useful rare patterns, consistently outperforming GRPO-style baselines across multiple mathematical and scientific reasoning benchmarks with improved training stability.
Entities (8)
Relation Signals (7)
TACO → outperforms → GRPO
confidence 98% · Experimental results across three LLMs and eight benchmarks show that TACO consistently outperforms GRPO-style baselines.
TACO → appliesto → Large Language Models
confidence 97% · Reinforcement learning (RL) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs).
TACO → mitigates → Positive-Credit Contamination
confidence 97% · TACO calibrates uniform credit assignment to suppress undesirable positive updates, directly addressing the identified failure mode.
GRPO → suffersfrom → Positive-Credit Contamination
confidence 96% · widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences, resulting in indiscriminate reinforcement of flawed reasoning behavior.
TACO → uses → Tail-Risk Score
confidence 95% · TACO first computes a tail-risk score that incorporates the local generation context to assess each token's risk.
TACO → evaluatedon → AIME-2024
confidence 94% · We evaluate the trained models on six mathematical reasoning benchmarks: AIME 2024, AIME 2025, AMC 2023, MATH-500, Minerva Math, and Olympiad Bench.
Tail-Risk Score → computedfrom → Local Entropy
confidence 93% · TACO combines token-level rarity with the uncertainty of the local policy distribution, using local entropy as a reference for expected surprisal.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences. We identify a critical failure mode of this design, which we refer to as Positive-Credit Contamination: low-probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior. To mitigate this issue, we propose Tail-Aware Credit calibratiOn (TACO), a method that calibrates uniform credit assignment to suppress undesirable positive updates. TACO first computes a tail-risk score that incorporates the local generation context to assess each token's risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertainty-driven exploration. TACO then uses this score to tune positive credit for risky tokens without removing their gradients entirely, so that recurring useful rare patterns can accumulate reinforcement while incidental noise is progressively dampened. Experimental results across three LLMs and eight benchmarks show that TACO consistently outperforms GRPO-style baselines. Notably, TACO improves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.07976v1
- Canonical: https://arxiv.org/abs/2607.07976v1
Trouble viewing inline? Open PDF directly →
Full Text
51,615 characters extracted from source content.
Expand or collapse full text
When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning Xiuyi Lou 1∗ , Zicheng Xu 1∗ , Yu-Neng Chuang 2 , Hoang Anh Duy Le 2 , Zhaozhuo Xu 3 , Guanchu Wang 4 , Vladimir Braverman 1† 1 Johns Hopkins University, 2 Rice University, 3 Workato, 4 University of North Carolina at Charlotte Abstract Reinforcement learning (RL) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences. We identify a critical failure mode of this design, which we refer to as Positive-Credit Contamination: low- probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior. To mitigate this issue, we propose Tail-Aware Credit calibratiOn (TACO), a method that calibrates uniform credit assignment to suppress undesirable positive updates.TACOfirst computes a tail-risk score that incorporates the local generation context to assess each token’s risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertainty- driven exploration.TACOthen uses this score to tune positive credit for risky tokens without removing their gradients entirely, so that recurring useful rare patterns can accumulate reinforcement while incidental noise is progressively dampened. Experimental results across three LLMs and eight benchmarks show thatTACO consistently outperforms GRPO-style baselines. Notably,TACOimproves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: https://github.com/xiuyilou/TACO. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has become a common post-training paradigm for reasoning-oriented large language models (LLMs). Recent models such as OpenAI’s o-series and DeepSeek-R1 demonstrate that large-scale RL post-training can substantially improve performance on automatically verifiable tasks, including mathematics and programming [14,2]. Early RLVR methods often build on Proximal Policy Optimization (PPO), which typically relies on a learned critic for advantage estimation [18,15]. In contrast, Group Relative Policy Optimization (GRPO) replaces the learned critic with group-relative advantages, reducing value-modeling overhead while achieving strong empirical performance [19]. The simple yet effective design has made GRPO a widely adopted backbone for reasoning RL, motivating variants such as Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) and Group Sequence Policy Optimization (GSPO) [34, 38]. However, this simplification computes a single completion-level advantage and broadcasts it uniformly to every generated token, even though not every token in a rewarded completion is equally reliable. Specifically, tokens from the low-probability tail of the policy distribution, which are unlikely under the generation context, can be sampled in rewarded completions even when they are semantically ∗ Equal contribution. † Correspondence to Vladimir Braverman. Preprint. arXiv:2607.07976v1 [cs.CL] 8 Jul 2026 AIME24AIME25 AMC23 Olympiad GPQA-D AIME24AIME25 AMC23 Olympiad GPQA-D AIME24AIME25 AMC23 Olympiad GPQA-D Accuracy (%) Qwen3-1.7B-BaseQwen3-4B-BaseQwen2.5-Math-7B 9.5 7.3 46.7 29.0 20.7 14.4 9.1 49.5 31.7 24.4 25.7 20.8 68.1 36.4 26.1 27.1 23.9 71.9 40.1 29.2 29.9 16.8 72.7 46.7 23.6 32.4 19.8 78.4 49.4 24.8 GRPOTACO (Ours) Figure 1: TACO consistently improves over GRPO across representative benchmarks and models. irrelevant or erroneous; we refer to such locally unreliable tokens as implausible tail tokens. Yet, these tokens’ local effects may be bypassed or overlooked by the overall correct reasoning process and final answer [22,10,6]. These tokens then receive the same positive credit as reliable ones, resulting in indiscriminate reinforcement of both well-formed reasoning behavior and locally flawed continuations. Consequently, the accumulation of such updates throughout training progressively biases the policy away from well-calibrated reasoning patterns. We refer to this failure mode as Positive-Credit Contamination. To improve upon the uniform credit assignment in GRPO-style methods, existing work generally seeks to differentiate token-level updates based on their estimated importance to the final outcome. One line of work leverages external signals such as counterfactual analysis, temporal-difference propagation, or execution feedback to evaluate each token’s contribution and scale its credit accord- ingly [9,16,21,7]. Another line incorporates intrinsic signals such as token entropy or distributional divergence as proxies for per-token significance, concentrating updates on tokens deemed critical to the reasoning process and attenuating routine ones [20,12,35,26]. However, existing methods exhibit two fundamental limitations. First, most of these methods depend on additional inference or auxiliary models, introducing non-trivial resource overhead. Second, current methods lack a local semantic perspective on token reliability. Implausible tail tokens may appear in high-contribution or high-entropy positions that existing methods regard as informative, causing unreliable credit to be mis- takenly amplified. Together, these limitations restrict the effectiveness of existing credit-assignment methods in reasoning-oriented RL training. To overcome these limitations, we introduce Tail-Aware Credit calibratiOn (TACO). Unlike existing methods,TACOcalibrates uniform credit assignment from a local semantic lens to suppress undesirable positive updates, with only negligible additional computational cost. Specifically,TACOestimates each token’s risk of being an implausible tail token under its generation context, softly suppressing positive credit for high-risk tokens while preserving full reinforcement for low-risk ones. Therefore,TACO reduces harmful positive reinforcement of unreliable behaviors while preserving useful exploration. Empirically, we evaluateTACOacross three LLMs and six mathematical reasoning benchmarks, as well as two out-of-distribution (OOD) scientific reasoning benchmarks.TACOconsistently im- proves over GRPO-style baselines across all settings, with Figure 1 highlighting the gains on five representative benchmarks. Our contributions can be summarized as follows: • Positive-Credit Contamination. We identify a failure mode of GRPO-style RLVR in which locally implausibletokens receive positive reinforcement. •Tail-Aware Credit Calibration. We introduceTACO, a context-aware method that calibrates positive token-level credit with negligible computational overhead. •Comprehensive Evaluation.TACOconsistently improves over baselines across multiple benchmarks, maintaining stable performance under long-horizon training. 2 Related Work Low-probability tokens in RLVR. Recent research suggests that tokens exhibit heterogeneity, making uniform update rules in RLVR suboptimal, particularly for low-probability tokens [12]. 2 On one hand, rare tokens sustain exploration by preserving diverse reasoning continuations and preventing premature policy collapse [5], motivating protective mechanisms such as clip-higher [34] and low-probability regularization [5]. On the other hand, rare tokens can destabilize optimization: their large gradient magnitudes can overshadow updates for high-probability tokens, and tokens with both low probability and low local entropy have been identified as primary drivers of training instability [33,11]. These findings motivated gradient-aware methods that dampen extreme token updates to enhance training robustness. Together, both directions highlight the importance of low- probability tokens in GRPO-style optimization, motivating further examination of their properties. Token-level credit allocation in RLVR. GRPO-style training assigns the same trajectory-level reward to every token in a completion, which can misallocate credit when tokens contribute unequally to the final outcome. Two main directions address this limitation. The first leverages external signals to redistribute credit: OAR estimates per-token outcome influence [9]; GRPO-λpropagates credit backward via temporal-difference methods [16]; TEMPO and EGCA concentrate updates on pivotal decision points using response structure and intermediate execution feedback [21,7]. The second reshapes token-level updates based on intrinsic signals: GTPO/GRPO-S and HAPO use uncertainty measures such as token entropy as importance indicators, so that high-uncertainty or exploratory tokens receive prioritized treatment over routine ones [20,12]. However, these methods primarily model contribution or importance, leaving the contextual validity of credited tokens underexplored. 3 Preliminary 3.1 Notations Letπ θ be an autoregressive policy over vocabularyV. For a promptqand completiono i = (o i,1 ,...,o i,T i ), we denotec i,t = (q,o i,<t )as the context at stept, andp i,t = π θ (o i,t | c i,t )as the sampled-token probability. The local entropy isH i,t =− P v∈V π θ (v | c i,t ) logπ θ (v | c i,t ) . In this work, we aim to identify unreliable tokens from generation-time statistics underc i,t to calibrate their credit, suppressing undesirable positive updates while preserving sound ones. 3.2 Group Relative Policy Optimization GRPO is a widely adopted RLVR framework for reasoning-oriented LLM post-training. Given a promptq, GRPO samples a group ofGcompletionso i G i=1 using the policyπ θ old . Each completion o i receives an outcome-level verified rewardR i , and its sequence-level advantage is computed as ˆ A i = R i −μ σ , whereμandσare the mean and standard deviation of rewards within the sampled group. GRPO then updates the policy by optimizing the following objective, which aggregates over all generated tokens: J GRPO (θ) = E q,o i ∼π θ old " 1 P i T i G X i=1 T i X t=1 ℓ i,t (θ; ˆ A i,t ) # ,(1) whereℓ i,t (θ; ˆ A i,t )is the PPO-style clipped surrogate term 3 . Since updates are token-level while advantages are completion-level, GRPO broadcasts the same sequence-level advantage to every token position: ˆ A i,t = ˆ A i . This broadcast rule assigns identical credit regardless of token-level differences. In reasoning traces, this can dilute useful learning signals and lead to suboptimal policy updates [32]. We examine this issue in detail below. 3.3 Positive-Credit Contamination Correct reasoning traces can contain locally unreliable tokens even when the final answer is correct. This occurs because erroneous calculations, irrelevant detours, or malformed continuations may be corrected or bypassed before the final response. Prior work on process supervision and step-level verification shows that outcome-based feedback can overlook flawed intermediate reasoning [22, 10]. In addition, models may produce repeated or incoherent text inside generated traces without necessarily disrupting the overall logical flow [4,25]. We observe such locally unreliable tokens in real correct reasoning traces, as illustrated by the case studies in Figure 2. 3 The full GRPO loss function can be referred to Equation (1) in the DeepSeek-R1 technical report [2]. 3 Figure 2: Examples of unreliable tokens from various sources in real correct reasoning traces. 010002000300040005000600070008000 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Average Reward H = 32 ( = 0.19) H = 16 (= 0.16) H = 8 (= 0.12) oracle broadcast (a) Trace length H 010002000300040005000600070008000 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Average Reward n opt = 4 ( = 0.19) n opt = 8 (= 0.16) n opt = 16 (= 0.16) (b) Optimal-action count n opt 010002000300040005000600070008000 Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Average Reward G = 32 (= 0.08) G = 16 (= 0.12) G = 8 ( = 0.12) (c) Group size G Figure 3: Demonstration of the effect of Positive-Credit Contamination on a synthetic sequential MDP. Experiments are conducted to analyze how (a) trace lengthH, (b) optimal-action countn opt , and (c) group sizeGaffect the contamination behavior.∆denotes the final-step reward gap between oracle and broadcast; the largest gap in each panel is bolded. The phenomenon that LLMs can produce locally unreliable tokens has been widely studied in the decoding literature, typically through the low-probability tail of auto-regressive policies. Prior work identifies an unreliable subset of this region, where sampled continuations can become incoherent or off-topic; we refer to tokens from this locally implausible subset as implausible tail tokens [4]. Standard inference-time strategies such as nucleus sampling suppress these tokens by truncating the tail, which has been shown to improve generation quality [4]. In GRPO-style training, however, full-vocabulary rollouts allow implausible tail tokens to enter the trajectories used for policy updates. While implausible tail tokens in failed completions can be penalized by negative advantages, they become problematic when they occur in completions with positive advantages, since the broadcast rule assigns them the same positive credit as contextually sound tokens. We formalize this as a critical failure mode of the broadcast rule, which we refer to as Positive-Credit Contamination. The root cause is that the outcome reward verifies only the final answer, but does not establish the contextual validity of each individual token continuation. As a result, implausible tail tokens can accumulate positive reinforcement across training, progressively biasing the policy toward bad continuations. To demonstrate the effect of Positive-Credit Contamination on training performance, we conduct a controlled experiment in a synthetic sequential Markov Decision Process (MDP) where optimal and tail actions are explicitly defined. Each trajectory consists of multiple independent steps with local step rewards, while the trajectory-level return is the sum of these rewards. This design allows us to isolate credit contamination by comparing two credit assignment rules under the same rollout distribution and reward function: the broadcast rule, which assigns the group-normalized trajectory advantage to every step, and oracle step-level credit assignment, which assigns credit using each step’s own reward. Therefore, any performance gap between the two rules reflects cross-step credit contamination caused by trajectory-level advantage broadcast. We study three axes that mirror LLM reasoning training: trace lengthH, optimal-action sparsity controlled byn opt , and group sizeG. As shown in Figure 3, oracle credit assignment consistently outperforms the broadcast rule across all settings, and the gap widens with longer traces, sparser 4 correct actions, and smaller group sizes. These worst-case conditions directly correspond to core properties of LLM RLVR: long chain-of-thought generation, scarce optimal continuations, and limited rollout budgets in practice, suggesting that Positive-Credit Contamination poses a significant challenge for reasoning-oriented post-training. Details of the synthetic MDP are provided in Appendix B. 4 Methodology In this section, we introduce Tail-Aware Credit calibratiOn (TACO) to calibrate token-level credit assignments and mitigate Positive-Credit Contamination.TACOconsists of two modules: adaptive tail-risk estimation, which scores each token’s likelihood of being a tail token; and tail-aware credit calibration, which softly suppresses positive credit for high-risk tokens while leaving low-risk ones unchanged. Together, these modules allowTACOto suppress undesirable positive updates while preserving useful exploration with negligible computational cost. 4.1 Adaptive Tail-Risk Estimation Mitigating Positive-Credit Contamination requires identifying tail tokens within rewarded comple- tions, so that their positive credit can be selectively suppressed. However, directly recognizing these tokens during training requires token-level oracle labels, incurring significant annotation cost and is computationally infeasible. TACO therefore uses observable generation-time statistics as an efficient proxy for estimating whether a sampled token falls into the implausible tail of the policy distribution. To estimate tail risk,TACOexamines the local next-token distribution at each generation step. A natural signal is the sampled-token probability: a token assigned very low probability under the current context is more likely to fall outside the policy’s reliable generation region. However, probability alone is insufficient since rare tokens are heterogeneous. In high-entropy contexts, a low probability token may still reflect useful exploration among many plausible alternatives. In low-entropy contexts, the same level of rarity is more likely to indicate a locally implausible continuation. Therefore,TACO combines token-level rarity with the uncertainty of the local policy distribution. Formally, given a tokeno i,t sampled at positiontof completioni,TACOuses its sampled-token probabilityp i,t together with the local entropy H i,t . Entropy serves as a context-level reference for how surprising a token is expected to be. The further a token’s surprisal exceeds the level expected by the local entropy, the more likely it is to be in the unreliable tail, as such deviations indicate generation beyond the policy’s well-calibrated region. TACOtherefore defines the tail-risk scorer tail i,t as how far the token’s surprisal exceeds that expected by the local entropy: r tail i,t = − logp i,t |z token surprisal − H i,t |z expected surprisal + logα,(2) A largerαmakes tail-risk identification more aggressive, identifying more low-probability tokens as risky, but risks suppressing useful rare tokens together with implausible ones. Withαfixed, a positive score indicates that the token is considered risky and subject to credit suppression. 4.2 Tail-Aware Credit Calibration The tail-risk score provides a continuous estimate of how likely each sampled token is to be locally implausible. Based on this estimate,TACOassigns each token a risk-dependent weight that softly reduces its positive credit as the risk increases. This smooth down-weighting preserves partial gradients for genuinely useful low-probability patterns, allowing them to accumulate reinforcement across rewarded trajectories, while progressively dampening accidental noise. Formally,TACOdefines the token weight w i,t as: w i,t = ( 1− λ 1− exp −r tail i,t , r tail i,t > 0, 1,r tail i,t ≤ 0, .(3) Tokens with non-positive risk scores retain full weight, while high-risk tokens receive smoothly decaying weights, bounded below by1− λ, whereλ ∈ (0, 1)is a hyperparameter controlling the 5 Algorithm 1 TACO: Tail-Aware Credit Calibration Require: Prompt datasetD; policy π θ ; verifierR; group size G; hyperparameters α,λ 1: for each training iteration do ## Standard GRPO rollout 2: π θ old ← π θ 3:Sample prompt batchB ∼D 4:Generate G completions for each prompt using π θ old 5:Compute rewards and group-normalized advantages ˆ A i ▷ sequence-level credit ## Tail-aware credit calibration 6:Estimate token-level tail risk r tail i,t using local surprisal and entropy by Eq. (2) 7:Convert tail risk into credit-suppression weights w i,t by Eq. (3) 8:Calibrate positive sequence-level credit into token-level advantages ˆ A TACO i,t by Eq. (4) ## Standard GRPO policy update 9:Update π θ with the GRPO clipped surrogate using ˆ A TACO i,t ▷ token-level credit 10: end for maximum suppression strength.TACOthen applies this weight to the broadcast advantage to produce a calibrated token-level advantage: ˆ A TACO i,t = w I[ ˆ A i >0] i,t ˆ A i .(4) Only positive advantages are modulated; negative advantages are left unchanged to preserve the suppression signal from failed trajectories. 4.3 Algorithm of TACO TACO integrates into standard GRPO training loop by replacing the broadcast advantage ˆ A i with calibrated token-level credit ˆ A TACO i,t from Eq.(4). All other components of the GRPO surrogate remain unchanged. SinceTACOrelies solely on the token probabilities and entropy computed during the forward pass, it introduces negligible overhead. Algorithm 1 summarizes the full procedure. 5 Experiments In this section, we conduct experiments to verify the effectiveness of our method, aiming to answer the following research questions: RQ1: DoesTACOimprove performance across diverse reasoning benchmarks? RQ2: DoesTACOsustain performance gains under longer RL training while avoiding late-stage collapse? RQ3: How does TACO affect token-level training behavior? 5.1 Experimental Setup Models. We evaluateTACOon three LLMs from different model families and scales: Qwen3- 1.7B-Base, Qwen3-4B-Base [31], and Qwen2.5-Math-7B [30]. For the main experiments, we use DAPO-Math-17K 4 as the training dataset. Our training codebase is built onverl[23], and we follow its standard GRPO training recipe. All methods share the same configuration, with method-specific hyperparameters set to the values reported in their original papers. Datasets. We evaluate the trained models on six mathematical reasoning benchmarks: AIME 2024 [36], AIME 2025 [37], AMC 2023 [13], MATH-500 [10], Minerva Math [8], and Olympiad- Bench [3]. To assess out-of-distribution generalization beyond mathematics, we further evaluate on two scientific reasoning benchmarks: MMLU-Pro [24] and GPQA-Diamond [17]. We report avg@32 for AIME and AMC, avg@4 for MATH-500, Minerva Math, and OlympiadBench, and avg@16 for MMLU-Pro and GPQA-Diamond. 4 https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k 6 Table 1: Main results on in-domain mathematical reasoning benchmarks and out-of-distribution scientific reasoning benchmarks. Best results are in bold and second-best results are underlined. The ∆ row reports absolute improvement over GRPO. MethodAIME24 AIME25 AMC23 MATH-500 Minerva Olympiad MMLU-Pro GPQA-D Avg. Qwen3-1.7B-Base GRPO9.487.2946.7266.2025.8329.0124.9420.7128.77 GRPO w/ Adv. Reweighting11.258.4447.03 64.7524.6329.1129.8423.3629.80 STAPO12.299.3846.4168.3523.0730.3624.3217.8029.00 TACO (Ours)14.389.06 49.4568.3525.7431.7130.4524.4331.70 ∆ vs. GRPO(+4.90) (+1.77) (+2.73) (+2.15)(-0.09) (+2.70)(+5.51)(+3.72) (+2.93) Qwen3-4B-Base GRPO25.7320.8368.1376.5034.6536.3536.2626.1440.57 GRPO w/ Adv. Reweighting23.5421.5669.3878.85 35.9438.2839.2026.6441.67 STAPO24.6921.9872.8977.0033.6436.8338.8425.8841.47 TACO (Ours)27.0823.8571.8880.0535.6740.0641.9029.1743.71 ∆ vs. GRPO(+1.35) (+3.02) (+3.75) (+3.55)(+1.02) (+3.71)(+5.64)(+3.03) (+3.14) Qwen2.5-Math-7B GRPO29.9016.7772.6683.3051.0146.7430.7523.6444.35 GRPO w/ Adv. Reweighting28.9617.2974.8482.9550.1846.7729.0524.4344.31 STAPO28.7517.0873.2884.3053.5847.8127.6424.4944.62 TACO (Ours)32.4019.7978.4484.6555.5149.4130.5324.8446.95 ∆ vs. GRPO(+2.50) (+3.02) (+5.78) (+1.35)(+4.50) (+2.67)(-0.22)(+1.20) (+2.60) Baselines. We compareTACOagainst three baselines. GRPO is the standard critic-free RLVR baseline, we implement it with clip-higher asymmetric clipping to enable long-term training [19,34]. GRPO w/ Adv. Reweighting reweights low-probability tokens to reduce their over-dominance in policy-gradient updates [33]. STAPO dampens updates from tokens with disproportionately large gradients for more stable optimization [11]. Additional details are provided in Appendix A. 5.2 Performance on Reasoning Tasks (RQ1) Table 1 shows the accuracy (%) of TACO and baseline methods across all benchmarks. Performance Gain.As shown in Table 1,TACOsurpasses all baselines on average and consistently improves performance across different models and benchmarks. These gains suggest thatTACO improves performance by calibrating implausible positive credit among low-probability tokens, going beyond merely stabilizing their updates as in STAPO and GRPO w/ Adv. Reweighting. By down- weighting unreliable positive credit while preserving useful rare-token exploration,TACOprovides a more robust optimization signal. OOD Generalization.TACOdemonstrates strong generalization beyond the mathematical training domain.TACOmaintains stable gains on MMLU-Pro and GPQA-Diamond, which cover scientific fields including physics, chemistry, and biology . By selectively suppressing tail tokens while preserving credit for useful exploratory ones,TACOavoids overly aggressive gradient suppression that could limit cross-domain generalization. This design helps the policy develop general reasoning capabilities rather than overfitting to domain-specific patterns. Model Scale Consistency.TACOachieves consistent improvements across models of different parameter scales. Because the tail-risk score uses each policy’s token probability and local entropy, the calibration naturally adapts to each model’s uncertainty profile, allowingTACOto improve reasoning capability without requiring model-specific tuning. 5.3 Analysis of Training Dynamics To examine howTACOaffects training, we compare its reward, policy entropy, and response length dynamics with baseline methods in Figure 4. 7 050100150200250300 Training Step −0.9 −0.8 −0.7 −0.6 −0.5 Reward Ours STAPO GRPO Adv. Reweighting (a) Training reward 050100150200250300 Training Step 0.2 0.4 0.6 0.8 1.0 1.2 Entropy (b) Policy entropy 050100150200250300 Training Step 1000 1200 1400 1600 1800 2000 2200 Response Length (c) Response length Figure 4: Training dynamics ofTACOand baseline methods on Qwen3-1.7B-Base. Results on other models are provided in Appendix C. 0100200300400500600 Training Step 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Accuracy (%) OursGRPO (a) AIME24 test accuracyprogre 0100200300400500600 Training Step 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Entropy (b) Policy entropy Figure 5: Comparison of TACO and GRPO on Qwen3-1.7B-Base with extended training. Learning Effectiveness. As shown in Figure 4a,TACOachieves consistently higher training accu- racy than baselines. This improvement stems from the selective nature ofTACO’s credit calibration: by suppressing positive credit for locally implausible tokens, each policy update concentrates reinforce- ment on contextually more reliable reasoning steps, leading to more effective policy optimization. Stable Entropy and Sustained Exploration. As shown in Figure 4b and 4c,TACOmaintains relatively lower and more stable token entropy compared to baselines while simultaneously producing longer responses throughout training. The stable entropy indicates thatTACOsuppresses erratic tail-token updates without collapsing the policy’s exploration capacity. Meanwhile, the increased response length suggests that the policy develops more complex and complete reasoning ability under this stable optimization. Together, these dynamics suggest thatTACO’s credit calibration enables the policy to maintain effective exploration and develop more robust reasoning capabilities. 5.4 Long-Horizon Training Stability (RQ2) A common challenge in GRPO-style training is that the policy tends to plateau or degrade as training progresses, limiting the benefit of extended optimization. To evaluateTACO’s robustness in this regime, we extend training of both GRPO andTACOon Qwen3-1.7B-Base from the default 300 steps to 600 steps. As shown in Figure 5, GRPO’s test accuracy on AIME24 plateaus around step 450 and begins to degrade thereafter, whileTACOcontinues to improve throughout the extended phase. The entropy dynamics further show that GRPO exhibits large fluctuations as training proceeds, whereas TACOmaintains a smooth and stable entropy profile, avoiding both entropy collapse and explosion. These results suggest thatTACOenables more stable long-horizon optimization, allowing the policy to continue benefiting from extended training in regimes where standard GRPO stagnates. 5.5 Behavioral Analysis (RQ3) To further illustrate howTACOmitigates Positive-Credit Contamination, we present a rewarded trace sampled during training in Figure 6. Although the completion reaches the correct answer, its early generation contains unreliable tail tokens, including unnecessary table formatting, corrupted math 8 Prompt.Given an isosceles right triangleABCwith ̸ A=90 ◦ ,AD=48 √ 2, andAE=52 √ 2, compute the distanceXYbetween the altitude feet fromDandEtoBC. Completion w/Adv. = 2.6458 Let'srestatetheprobleminatable,thenproceedwiththesolution.problem solution|--Problem:ABCisani.osalesTrianglewith:angleA=90^circ Points:D,EIn:ABand:ACGEFCIn단한는게Daisy이거조르지마You're absolutelyright--thelistofpointsand...RestatedProblem:LetABCbean isoscelestrianglewithangleA=90deg...FinalAnswer:boxed(100). Figure 6: A case from Qwen3-4B-Base.TACOselectively downweights tail tokens (shown in red; darker red indicates stronger suppression), while preserving coherent reasoning steps (shown in blue). Table 2: Sensitivity Analysis ofαandλon Qwen3-1.7B-Base model. Best results are in bold and second-best results are underlined. α λAIME24 AIME25 AMC23 MATH-500 Minerva Olympiad MMLU-Pro GPQA-D Avg. Qwen3-1.7B-Base 0.010.615.109.5847.5068.1524.8228.9730.9523.9931.13 0.010.914.389.0649.4568.3525.7431.7130.4524.43 31.70 0.005 0.612.818.6448.1567.2024.7229.3828.8526.2630.75 0.005 0.913.239.0646.5767.5525.5530.0829.8525.8130.96 terms (i), broken formula (In), mixed-language noise (Daisy), and irrelevant text (–the,list). Under the GRPO broadcast rule, these unreliable behaviors would be mis-amplified by receiving the same positive advantage as useful reasoning tokens. In contrast,TACOassigns suppressed credit to these tokens while preserving full credit for the later valid solution steps, showing thatTACOcan help suppress harmful local behaviors without weakening coherent reasoning. Additional qualitative cases and training diagnostics are provided in Appendix C. 5.6 Hyperparameter Sensitivity Study We evaluate the sensitivity ofTACOto its two key hyperparameters on Qwen3-1.7B-Base model: the tail-risk strictnessαand the suppression strengthλ. We varyαandλaround the default configuration while keeping all other training settings fixed. As shown in Table 2, our default setting, (α,λ) = (0.01, 0.9), achieves the best average performance, whileTACOremains effective across a reasonable range of settings. This suggests that the gains do not rely on a brittle hyperparameter choice. We also observe that settingαtoo large, e.g.,α = 0.1, leads to clear early-stage performance degradation. This suggests that overly aggressive tail-risk identification may suppress useful low- probability behaviors together with unreliable ones. 6 Conclusion In this work, we identify Positive-Credit Contamination as a failure mode of GRPO-style RLVR, where unreliable tail tokens can receive undesired positive updates. To address this issue, we propose Tail-Aware Credit calibratiOn (TACO), which calibrates token-level credit to avoid reinforcing flawed local continuations.TACOestimates each token’s tail risk from its sampled probability and local entropy, then uses this risk to softly reduce positive credit for high-risk tokens while preserving sound reasoning patterns. Experiments across three LLMs and eight reasoning benchmarks show thatTACO consistently improves over GRPO-style baselines and supports more stable long-horizon training. These results demonstrate the effectiveness of TACO for improving reasoning-oriented RLVR. 9 References [1] Zigeng Chen, Xinyin Ma, Gongfan Fang, Ruonan Yu, and Xinchao Wang. Verithinker: Learning to verify makes reasoning model efficient. CoRR, abs/2505.17941, 2025. doi: 10.48550/ARXIV. 2505.17941. URL https://doi.org/10.48550/arXiv.2505.17941. [2]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. [3]Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 3828–3850. Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.ACL-LONG.211. URL https://doi.org/10.18653/v1/2024.acl-long.211. [4] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URLhttps://openreview. net/forum?id=rygGQyrFvH. [5] Guanhua Huang, Tingqiang Xu, Mingze Wang, Qi Yi, Xue Gong, Siheng Li, Ruibin Xiong, Kejiao Li, Yuhao Jiang, and Bo Zhou.Low-probability tokens sustain explo- ration in reinforcement learning with verifiable reward. CoRR, abs/2510.03222, 2025. doi: 10.48550/ARXIV.2510.03222. URL https://doi.org/10.48550/arXiv.2510.03222. [6] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=IkmD3fKBPQ. [7] Abhijit Kumar, Natalya Kumar, and Shikhar Gupta. Execution-grounded credit assignment for grpo in code generation. arXiv preprint arXiv:2603.16158, 2026. [8] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, Novem- ber 28 - December 9, 2022, 2022. URLhttp://papers.nips.c/paper_files/paper/ 2022/hash/18abbeef8cfe9203fdf9053c9c4fe191-Abstract-Conference.html. [9]Ziheng Li, Liu Kang, Feng Xiao, Luxi Xing, Qingyi Si, Zhuoran Li, Weikang Gong, Deqing Yang, Yanghua Xiao, and Hongcheng Guo. Outcome-grounded advantage reshaping for fine- grained credit assignment in mathematical reasoning. CoRR, abs/2601.07408, 2026. doi: 10.48550/ARXIV.2601.07408. URL https://doi.org/10.48550/arXiv.2601.07408. [10]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. URL https://arxiv.org/abs/2305.20050. [11]Shiqi Liu, Zeyu He, Guojian Zhan, Letian Tao, Zhilong Zheng, Jiang Wu, Yinuo Wang, Yang Guan, Kehua Sheng, Bo Zhang, Keqiang Li, Jingliang Duan, and Shengbo Eben Li. STAPO: stabilizing reinforcement learning for llms by silencing rare spurious tokens. CoRR, abs/2602.15620, 2026. doi: 10.48550/ARXIV.2602.15620. URLhttps://doi.org/10. 48550/arXiv.2602.15620. [12]Zheng Liu, Mengjie Liu, Siwei Wen, Mengzhang Cai, Bin Cui, Conghui He, and Wentao Zhang. Heterogeneous adaptive policy optimization: Tailoring optimization to every token’s nature, 2026. URL https://arxiv.org/abs/2509.16591. 10 [13]math-ai Team.AMC23: American mathematics competitions 2023 test set.https: //huggingface.co/datasets/math-ai/amc23, 2024. Hugging Face dataset. Accessed: 2026-05-06. [14]OpenAI. Openai o1 system card. CoRR, abs/2412.16720, 2024. doi: 10.48550/ARXIV.2412. 16720. URL https://doi.org/10.48550/arXiv.2412.16720. [15]Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. CoRR, abs/2203.02155, 2022. doi: 10.48550/ARXIV.2203.02155. URLhttps: //doi.org/10.48550/arXiv.2203.02155. [16]Prasanna Parthasarathi, Mathieu Reymond, Boxing Chen, Yufei Cui, and Sarath Chandar. Grpo- λ: Credit assignment improves llm reasoning, 2025. URLhttps://arxiv.org/abs/2510. 00194. [17]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. [18]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URLhttp://arxiv.org/ abs/1707.06347. [19]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. doi: 10.48550/ARXIV.2402.03300. URL https://doi.org/10.48550/arXiv.2402.03300. [20]Hongze Tan, Zihan Wang, Jianfei Pan, Jinghao Lin, Hao Wang, Yifan Wu, Tao Chen, Zhihang Zheng, Zhihao Tang, and Haihua Yang. Gtpo and grpo-s: Token and sequence-level reward shaping with policy entropy. arXiv preprint arXiv:2508.04349, 2025. [21]Hieu Tran, Zonghai Yao, and Hong Yu. Exploiting tree structure for credit assignment in RL training of llms. CoRR, abs/2509.18314, 2025. doi: 10.48550/ARXIV.2509.18314. URL https://doi.org/10.48550/arXiv.2509.18314. [22] Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. CoRR, abs/2211.14275, 2022. doi: 10.48550/ARXIV.2211.14275. URL https://doi.org/10.48550/arXiv.2211.14275. [23]verl-project. verl: Volcano engine reinforcement learning for llms.https://github.com/ verl-project/verl, 2024. GitHub repository. Accessed: 2026-05-06. [24] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URLhttp://papers.nips.c/paper_files/paper/2024/hash/ ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets_and_Benchmarks_Track. html. [25] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=SJeYe0NtvH. 11 [26]Can Xie, Ruotong Pan, Xiangyu Wu, Yunfei Zhang, Jiayi Fu, Tingting Gao, and Guorui Zhou. Unlocking exploration in RLVR: uncertainty-aware advantage shaping for deeper reasoning. CoRR, abs/2510.10649, 2025. doi: 10.48550/ARXIV.2510.10649. URLhttps: //doi.org/10.48550/arXiv.2510.10649. [27]Zicheng Xu, Guanchu Wang, Yu-Neng Chuang, Guangyao Zheng, Alexander S. Szalay, Zirui Liu, and Vladimir Braverman. DTS: enhancing large reasoning models via decoding tree sketching. CoRR, abs/2511.00640, 2025. doi: 10.48550/ARXIV.2511.00640. URLhttps: //doi.org/10.48550/arXiv.2511.00640. [28]Zicheng Xu, Guanchu Wang, Guangyao Zheng, Yu-Neng Chuang, Alex Szalay, Xia Hu, and Vladimir Braverman. Self-ensemble: Mitigating confidence distortion for large language models. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 16603–16615. Association for Computational Linguistics, 2025. URL https://aclanthology.org/2025.findings-emnlp.902/. [29]Zicheng Xu, Ruixuan Zhang, Yu-Neng Chuang, Xiuyi Lou, Hoang Anh Duy Le, Oren Gal, Alexander S Szalay, Zhaozhuo Xu, Guanchu Wang, and Vladimir Braverman. Learning at the right pace: Adaptive data scheduling improves llm reinforcement learning. arXiv preprint arXiv:2606.22305, 2026. [30]An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. CoRR, abs/2409.12122, 2024. doi: 10.48550/ARXIV.2409. 12122. URL https://doi.org/10.48550/arXiv.2409.12122. [31]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. [32]Matthew Y. R. Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, and Aviral Kumar. Int: Self-proposed interventions enable credit assignment in LLM reasoning. CoRR, abs/2601.14209, 2026. doi: 10.48550/ARXIV.2601.14209. URLhttps://doi.org/10.48550/arXiv.2601. 14209. [33] Zhihe Yang, Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, and Yun- jian Xu. Do not let low-probability tokens over-dominate in rl for llms. arXiv preprint arXiv:2505.12929, 2025. [34]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an open-source LLM reinforcement learning system at scale. CoRR, abs/2503.14476, 2025. doi: 10.48550/ARXIV.2503.14476. URL https://doi.org/10.48550/arXiv.2503.14476. [35]Song Yu and Li Li. ERPO: token-level entropy-regulated policy optimization for large reasoning models. CoRR, abs/2603.28204, 2026. doi: 10.48550/ARXIV.2603.28204. URLhttps: //doi.org/10.48550/arXiv.2603.28204. [36]Yifan Zhang and Math-AI Team. American invitational mathematics examination (AIME) 2024. https://huggingface.co/datasets/math-ai/aime24, 2024. Hugging Face dataset. Ac- cessed: 2026-05-06. 12 [37]Yifan Zhang and Math-AI Team. American invitational mathematics examination (AIME) 2025. https://huggingface.co/datasets/math-ai/aime25, 2025. Hugging Face dataset. Ac- cessed: 2026-05-06. [38] Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. CoRR, abs/2507.18071, 2025. doi: 10.48550/ARXIV.2507.18071. URLhttps: //doi.org/10.48550/arXiv.2507.18071. A Implementation Details Training Setup. We train all methods with vLLM-based rollout generation and no KL penalty. Unless otherwise specified, we use group size8, learning rate1× 10 −6 with 10 warmup steps, maximum prompt length 1024, maximum response length 4096, rollout temperature 1.0, top-p = 1.0, and asymmetric clipping with lower and upper clip ranges(0.2, 0.28). For the Qwen3 series, the rollout batch size is 256 and the PPO mini-batch size is 64; Qwen3-1.7B-Base and Qwen3-4B-Base are trained for 300 and 250 steps, respectively. For Qwen2.5-Math-7B, the rollout batch size is 512, the PPO mini-batch size is 32, and the model is trained for 200 steps. Baselines and Evaluation.For method-specific hyperparameters, TACO uses coefficients0.01and 0.9. For GRPO with Advantage Reweighting, we set the reweighting coefficient to0.1. For STAPO, we set the selected token ratio to20%and the threshold to0.002. During evaluation, we use top- p = 0.7and temperature0.9for all models and benchmarks. We follow existing work [27,1,28,29] to construct evaluation prompts, extract final answers, and build validation sets. We select the checkpoint with the highest validation accuracy for each method and model for final reporting. B Controlled Experiment Task Setting. We instantiate the synthetic sequential MDP described in Section 3.3 as follows. Each trajectory has lengthH. At each step, the agent samples from a large action space|A| = 10 5 . Among them,n opt actions are optimal with reward1and all remaining actions receive reward0. The trajectory return is the sum of step rewards, and we report the normalized return by dividing it byH. The policy is a per-step tabular softmaxθ ∈ R H×|A| . To mimic a pretrained non-uniform prior, logits of optimal actions are initialized fromN (1, 1), while all other logits are initialized fromN (0, 1). Hyper-parameters. The policy is trained with SGD using learning rate0.2for8000steps, with evaluation every100steps. We varyH,n opt , and group sizeGto study the effects of trace length, optimal-action sparsity, and group-level credit noise. All curves are averaged over ten random seeds. C Additional Training Dynamics and Analysis As illustrated in Figure 7 and Figure 8, the training dynamics on both Qwen3-4B-Base and Qwen2.5- Math-7B exhibit patterns consistent with those observed on Qwen3-1.7B-Base. TACO achieves higher training accuracy while maintaining stable token entropy and producing longer responses during training. These dynamics provide empirical evidence that TACO improves optimization efficiency and helps develop more robust reasoning capabilities. Figure 9 shows diagnostics over positive-advantage response tokens. The thresholdτrepresents the mean probability boundary below which tokens are identified as risky. The reliable-token ratio averages0.981with a median of0.998, indicating that only about1.9%of positive-advantage tokens are downweighted on average. The average credit weightwremains close to one, increasing from 0.996to0.9998, whileτincreases mildly from0.0071to0.0095. These results show that the calibration modifies only a small subset of low-confidence tokens, yet has a substantial effect on training dynamics and final performance. This suggests that sparse tail tokens being mis-amplified during training can exert a disproportionate influence on policy optimization, while our calibration effectively suppresses these harmful local updates. 13 050100150200250 Training Step −0.8 −0.7 −0.6 −0.5 −0.4 −0.3 −0.2 −0.1 0.0 Reward Ours STAPO GRPO Adv. Reweighting (a) Training reward 050100150200250 Training Step 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Entropy (b) Policy entropy 050100150200250 Training Step 1000 1200 1400 1600 1800 2000 2200 2400 Response Length (c) Response length Figure 7: Training dynamics of TACO and baseline methods on Qwen3-4B-Base. 0255075100125150175200 Training Step −0.6 −0.4 −0.2 0.0 Reward Ours STAPO GRPO Adv. Reweighting (a) Training reward 0255075100125150175200 Training Step 0.1 0.2 0.3 0.4 0.5 Entropy (b) Policy entropy 0255075100125150175200 Training Step 900 950 1000 1050 1100 Response Length (c) Response length Figure 8: Training dynamics of TACO and baseline methods on Qwen2.5-Math-7B. Figures 10 and 11 show two additional rewarded traces from different models. The Qwen3-1.7B-Base case contains tail tokens related to instruction or search leakage, such as off-context assistant/search text and unrelated help messages. The Qwen2.5-Math-7B case contains malformed template frag- ments and serialized control artifacts, such asist,_that,Pro, andabb. In both cases,TACOassigns lower credit to these unreliable tokens while preserving credit for the later coherent solution steps. D Limitations and Future Works In this work, we proposeTACOto mitigate Positive-Credit Contamination in GRPO-style RLVR by calibrating token-level credit for unreliable tail tokens. While our experiments focus mainly on mathematical reasoning, a natural future direction is to extendTACOto other verifiable domains, such as code generation and tool use, as well as to open-ended tasks such as creative writing, which may require richer reward signals and more careful calibration designs. In these broader settings, the main idea ofTACOremains applicable: credit assignment should distinguish unreliable local continuations from useful behaviors. Future work can also combineTACOwith methods that use model-generated reasoning traces for self-improvement, further enhancing the stability and effectiveness of post-training. E Computational Infrastructure The computational infrastructure information is given in Table 3. 14 050100150200250300 Training Step 0.90 0.92 0.94 0.96 0.98 1.00 In-ratio (a) Reliable-token ratio 050100150200250300 Training Step 0.0070 0.0075 0.0080 0.0085 0.0090 0.0095 Threshold τ (b) Threshold τ . 050100150200250300 Training Step 0.9960 0.9965 0.9970 0.9975 0.9980 0.9985 0.9990 0.9995 1.0000 Weight w (c) Average weight w. Figure 9: Training diagnostics over positive-advantage response tokens. Prompt.Letfbe a non-constant polynomial such that f(x−1) +f(x) +f(x+ 1) = f(x) 2 2013x for all nonzero real numbersx. Find the sum of all possible values off(1). Completion w/Adv. = 2.6458 Notjustahint.Ifyouthinkitmighthelpsooneroratsomepointduringyour search,useassistant,butasmentioned,don'tuseassistantasatitletoany case....Errorscanalsobethereasonwhythesuggestionwon'tappearwhencurrent queriesdon'tseemtobesufficient.Here'ssometheoryaboutsearchqueriesand results.Thetheorypartended.Continueasnormalonthatnote.Assistant:To solvethegivenproblem,weanalyzethefunctionalequation....Now,tofindf(1), wesimplysubstitutex= 1intof(x) = 6039x:f(1) = 6039.Thus,thesumofall possiblevaluesis 6039. Figure 10: A case from Qwen3-1.7B-Base. Prompt.An airline serves a dinner to all the passengers on an airplane. They get their choice of steak or fish. Three steak meals and three fish meals are set aside for the six-member crew. If the meals are distributed to the crew members randomly, what is the probability that both pilots get the fish? The answer is in the form m n , where gcd(m,n) = 1. Please provide the value ofm+n. Completion w/Adv. = 2.6458 WewillalsohandleitusingPython'sfractionclassfromthefractionsmoduleto ensureit'sinthesimplestform.Answerist_that_rightwhenever_the_answer _is(abb|def|Pro|Com)Assistant:Let'scalculatetheprobabilitystepbystep toensurewegetthecorrectanswer....Theprobabilitythatbothpilotsgetfish is 1 5 . Thus,m= 1andn= 5,som+n= 6.Therefore,theansweris 6. Figure 11: A case from Qwen2.5-Math-7B. 15 Table 3: Experiment configuration and computing infrastructure. NameValue Data typetorch.bfloat16 Flash-AttentionTrue Computing InfrastructureGPU GPU ModelNVIDIA-H200 GPU Memory141 GB GPU Number 4 CUDA Version12.9 CPU Memory512GB 16