Paper deep dive
Semifactual Credit-Augmented Policy Optimization
Junshu Pan, Zhizhang Fu, Shulin Huang, Yiran Ding, Zifan Cheng, Wenqi Shao, Qiaosheng Zhang, Yue Zhang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions remain sensitive to task-irrelevant prompt features. We investigate this sensitivity through semifactual prompt interventions that preserve the underlying problem and its answer. Our analysis reveals substantial variation in token-level sensitivity and shows that suppressing high-drift token candidates during decoding improves reasoning accuracy without updating model weights. These findings highlight a limitation of Group Relative Policy Optimization (GRPO), which assigns the same outcome-derived advantage to every response token and may reinforce potential spurious dependence alongside useful reasoning. Motivated by this observation, we introduce Semifactual Credit-Augmented Policy Optimization (SCAPO), a causally inspired variant of GRPO that incorporates semifactual stability into token-level credit assignment. SCAPO measures token probability drift for fixed responses under semifactual interventions and uses normalized stability scores to reduce advantages for relatively unstable tokens during early training, while granting no additional credit for stability alone. On Qwen3-4B-Base and Qwen3-1.7B-Base, SCAPO improves AIME 2024-2026 accuracy over GRPO by 5.63 and 4.17 percentage points, respectively. At both model scales, SCAPO achieves the best results on most evaluated mathematics benchmarks and all evaluated out-of-distribution benchmarks among the compared methods. These results suggest that semifactual stability provides an effective training signal for improving reasoning and generalization through finer-grained credit assignment in RLVR. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.40360v1
- Canonical: https://arxiv.org/abs/2609.40360v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
75,657 characters extracted from source content.
Expand or collapse full text
Semifactual Credit-Augmented Policy Optimization SEMIFACTUAL CREDIT-AUGMENTED POLICY OPTIMIZATION Junshu Pan 1,2,3 Zhizhang Fu 2 Shulin Huang 1,2 Yiran Ding 2 Zifan Cheng 1 Wenqi Shao 3,4 Qiaosheng Zhang 3,4 Yue Zhang 2∗ 1 Zhejiang University 2 Westlake University 3 Shanghai Innovation Institute 4 Shanghai AI Laboratory panjunshu,zhangyue@westlake.edu.cn ABSTRACT Reinforcement learning with verifiable rewards (RLVR) has improved the reason- ing capabilities of large language models (LLMs), yet their predictions remain sensitive to task-irrelevant prompt features. We investigate this sensitivity through semifactual prompt interventions that preserve the underlying problem and its an- swer. Our analysis reveals substantial variation in token-level sensitivity and shows that suppressing high-drift token candidates during decoding improves reasoning accuracy without updating model weights. These findings highlight a limitation of Group Relative Policy Optimization (GRPO), which assigns the same outcome- derived advantage to every response token and may reinforce potential spurious dependence alongside useful reasoning. Motivated by this observation, we intro- duce Semifactual Credit-Augmented Policy Optimization (SCAPO), a causally inspired variant of GRPO that incorporates semifactual stability into token-level credit assignment. SCAPO measures token probability drift for fixed responses under semifactual interventions and uses normalized stability scores to reduce advantages for relatively unstable tokens during early training, while granting no additional credit for stability alone. On Qwen3-4B-Base and Qwen3-1.7B-Base, SCAPO improves AIME 2024–2026 accuracy over GRPO by 5.63 and 4.17 per- centage points, respectively. At both model scales, SCAPO achieves the best results on most evaluated mathematics benchmarks and all evaluated out-of-distribution benchmarks among the compared methods. These results suggest that semifactual stability provides an effective training signal for improving reasoning and general- ization through finer-grained credit assignment in RLVR. The code is available at https://github.com/DtYXs/SCAPO. 0120200300400500600 Policy optimization steps 5 10 15 20 25 30 AIME accuracy (%) SCAPO credit augmentation (a) Qwen3-4B-Base GRPO FIPO SCAPO 25.42 27.71 22.08 02004006008001000 Policy optimization steps 4 6 8 10 12 AIME accuracy (%) SCAPO credit augmentation (b) Qwen3-1.7B-Base GRPO FIPO SCAPO 7.01 11.88 7.71 Figure 1: SCAPO achieves the highest AIME accuracy on the two Qwen3 base models. Aggregate AIME 2024–2026 accuracy for SCAPO, GRPO, and FIPO. SCAPO reaches GRPO’s final accuracy in fewer than half as many policy optimization steps and achieves higher final accuracy than baselines at both model scales. Shaded regions mark SCAPO’s semifactual credit augmentation phase. ∗ Corresponding author. 1 arXiv:2609.40360v1 [cs.LG] 30 Sep 2026 Semifactual Credit-Augmented Policy Optimization 1INTRODUCTION Reinforcement learning with verifiable rewards (RLVR) has advanced the development of large language models (LLMs) with strong reasoning capabilities, such as OpenAI o1 (OpenAI, 2024) and DeepSeek-R1 (Guo et al., 2025). Group Relative Policy Optimization (GRPO) (Shao et al., 2024), a representative RLVR method, optimizes automatically checkable outcomes without process-level supervision. Recent evidence suggests that RLVR can diminish dependence on spurious correlations and improve generalization under distribution shifts (Fu et al., 2026). However, robustness evaluations still reveal sensitivity to irrelevant information and input perturbations (Mirzadeh et al., 2025; Huang et al., 2025). Understanding how training factors affect generalization in RLVR and how to further improve it remains underexplored. We study spurious dependence in RLVR at the token level. Building on prior studies of spurious feature reliance (Wang et al., 2022; Fu et al., 2026), we probe causal invariance (Peters et al., 2016; Arjovsky et al., 2019) in LLM inference through semifactual prompt interventions (Goodman, 1947; Lu et al., 2022). Specifically, we alter task-irrelevant prompt features while preserving the mathematical problem and its answer, using drift in the token probabilities of a fixed response as a proxy for potential spurious dependence. We find that suppressing high-drift token candidates during decoding improves the accuracy of Qwen3-4B-Base (Yang et al., 2025) from 15.8% to 30.0% on a 1,000-question mathematical diagnostic panel without updating model weights (see Section 2). The above results suggest that LLMs are highly sensitive to token-level spurious features. However, GRPO assigns the same outcome-derived advantage to every valid token in a response, without directly accounting for this token-level sensitivity. As a result, positive outcome credit may reinforce potential spurious dependence alongside useful reasoning. To address the above problem, one intuitive way is to change the RL training process, incorporating semifactual sensitivity into token-level credit assignment. In this paper, we introduce Semifactual Credit-Augmented Policy Optimization (SCAPO), a causally inspired variant of GRPO that aims to turn semifactual stability into a token-level credit signal during training. This signal reveals differences in token sensitivity that final-answer correctness alone cannot distinguish. Verifiable rewards provide response-level supervision, while semifactual stability refines token-level credit assignment. In particular, SCAPO uses the rollout policy to teacher-force each sampled response under the original prompt and its semifactual perturbations, measuring the probability drifts of each response token. After aggregating and normalizing these drifts within each prompt group, it adds only the negative part of the resulting stability score to the GRPO advantage as a detached token-level credit augmentation. Consequently, relatively unstable tokens receive lower advantages, while stability alone earns no additional credit, since stability does not imply correctness. We apply this credit augmentation during the early phase of training to shape trajectory selection, then continue optimizing the resulting policy with standard GRPO. SCAPO improves AIME 2024–2026 accuracy (Mathematical Association of America, 2026) over GRPO by +5.63 and +4.17 points on Qwen3-4B-Base and Qwen3-1.7B-Base (Yang et al., 2025), respectively. As shown in Figure 1, SCAPO reaches GRPO’s final AIME accuracy in fewer than half as many policy optimization steps. Moreover, across both model scales, SCAPO outperforms GRPO on all evaluated benchmarks and achieves the best performance on most of the competition-level mathematics benchmarks among all the compared RLVR methods. SCAPO also achieves the highest accuracy on out-of-distribution benchmarks at both model scales. In addition, our ablations further support the value of aligning credit corrections with semifactual sensitivity and selectively reducing credit for relatively unstable tokens. These results suggest that semifactual stability complements outcome rewards with an effective signal for finer-grained credit assignment in RLVR. Our contributions can be summarized as: • We study token-level spurious dependence in LLM inference through semifactual prompt interventions, revealing heterogeneous sensitivity and showing that suppressing high-drift tokens during decoding can improve reasoning accuracy without weight updates. (Sec. 2) • We introduce SCAPO, an approach to token-level credit augmentation in GRPO using detached negative-only corrections derived from fixed-response semifactual stability, without requiring process supervision or an external reward model. (Sec. 3) 2 Semifactual Credit-Augmented Policy Optimization 10 −8 10 −6 10 −4 10 −2 1 Original probability p (0) t 10 −8 10 −6 10 −4 10 −2 1 Perturbed probability p ( k ) t (a) Probability transitions 24.0%p (k) t = p (0) t omitted 33.4% p ( k ) t > p (0) t increase 42.6% p ( k ) t < p (0) t decrease 10 −2 10 −1 10 0 10 1 Within-bin share (%) 10 −6 10 −4 10 −2 10 0 Mean drift d t (log scale) 0 1 2 3 4 Token share per bin (%) d t = 0: 12.82% (b) Token instability 0123 Relative mean drift (×) Reflection markers Discourse connectives Words Punctuation Math symbols Numbers 2.74 2.32 1.53 1.19 0.47 0.37 (c) Instability by token category Figure 2: Semifactual instability varies across tokens. (a) Probability transitions under semifactual prompt interventions. (b) Distribution of mean token drift. (c) Relative category mean drift normalized by the overall mean, with reflection markers and discourse connectives showing higher sensitivity. Results use frozen Qwen3-4B-Base before RL. • We empirically demonstrate SCAPO’s effectiveness on Qwen3-4B-Base and Qwen3-1.7B- Base, achieving the best results on most benchmarks among the compared RLVR methods. These results show that semifactual stability can serve as an effective training-time signal for token-level credit assignment in RLVR. (Sec. 4) 2TOKEN-LEVEL SEMIFACTUAL SENSITIVITY We probe potential token-level spurious dependence with semifactual sensitivity. Using a frozen base model, we characterize how this sensitivity varies across tokens and then examine whether suppressing unstable token candidates improves decoding. Constructing semifactual perturbations.Semifactual perturbations alter incidental features of a prompt while preserving its underlying mathematical problem and final answer (Kenny & Keane, 2021). Following prior work on reasoning robustness under input perturbations (Mirzadeh et al., 2025; Huang et al., 2025), we construct four types of perturbations: paraphrase, minor typo noise, irrelevant scenario wrapping, and appended irrelevant context. We use GPT-5.5 (OpenAI, 2026) to generate these perturbations, instructing it to preserve quantities, mathematical expressions, constraints, and the requested quantity. Example: Paraphrase Original: What is the smallest odd number with four different prime factors? Paraphrase: What is the smallest odd number that has four distinct prime factors? Answer: 1155 All four perturbed prompts for this example and details of perturbation construction are provided in Appendix B.1. Diagnostic setup and drift measurement. To characterize semifactual sensitivity before RL training, we sample one response per original prompt from frozen Qwen3-4B-Base on 1,000 questions selected from DAPO-Math-17K. Sampling details are provided in Appendix B.2. Letp (0) t andp (k) t denote the probabilities assigned to the identical sampled token at positiontunder the original and the k-th perturbed prompts, respectively. The probability drift measured by the bounded-symmetric distance is defined as follows: d (k) t = 2 tanh logp (0) t − logp (k) t /2 = |p (0) t − p (k) t | p (0) t + p (k) t /2 .(1) Normalizing by the local mean probability avoids the scale bias of absolute differences and preserves relative probability changes for low-probability tokens. The distance remains within[0, 2], bound- ing the effect of extreme probability ratios for numerical robustness. We average across the four perturbations to obtain d t = 1 4 P 4 k=1 d (k) t . 3 Semifactual Credit-Augmented Policy Optimization Heterogeneity in token sensitivity. Figure 2(a) shows both increases and decreases in token probability across a wide range of original confidence levels. Figure 2(b) reveals heterogeneity in semifactual sensitivity. Nonzero probability drift magnitudes span several orders of magnitude, while 12.82% of positions show no recorded change. Semifactual sensitivity also varies across token categories, as shown in Figure 2(c). Reflection markers adapted from (Wang et al., 2025a) and discourse connectives drawn from (Das et al., 2018) exhibit mean drifts of 2.74×and 2.32×the overall mean, respectively, whereas mathematical symbols and numbers exhibit 0.47×and 0.37×, respectively. This contrast suggests that tokens lexically associated with organizing and reflection are more sensitive to these interventions than mathematical symbols or numbers. See Appendix B.3 for more details on token category definitions and statistics, and Appendix B.4 for illustrative case studies of token-level semifactual sensitivity. 0102030 Accuracy (%) Base sampling d-filtered 15.8% 30.0% +14.2 p Figure 3: Semifactual filtering improves rollout accuracy. Suppressing high-drift token candidates during rollout generation improves Qwen3-4B-Base’s accuracy by 14.2 percentage points on the same 1,000- question diagnostic panel. Suppressing unstable token candidates improves rollout accuracy. To test whether this semifactual sensitivity signal can help improve reasoning accuracy, we evaluate Qwen3-4B-Base under standard sampling and an onlined-filtered decoding strategy on the same 1,000 selected questions. Specifically, at each decoding step, we compute drift for every candidate token in the vocabulary, rank non-EOS candidates in descending drift order, and mask the longest prefix whose cumula- tive probability mass does not exceed 0.8. We always retain the EOS token, and then renormalize the remain- ing probabilities. With model weights unchanged, ac- curacy rises from 15.8% to 30.0% (+14.2 points), as shown in Figure 3. This result demonstrates that semi- factual sensitivity provides an actionable token-level signal and motivates its use for finer-grained credit assignment during RLVR training. More details and additional results are provided in Appendix C. 3SEMIFACTUAL CREDIT-AUGMENTED POLICY OPTIMIZATION Building on the decoding benefit of suppressing unstable token candidates, we use semifactual stability to probe potential spurious dependence and refine token-level credit assignment during RLVR training. Our guiding principle is to reduce the advantages assigned to relatively unstable tokens without granting additional credit for stability alone, since stability does not imply correctness. In this section, we introduce Semifactual Credit-Augmented Policy Optimization (SCAPO), a GRPO- based algorithm that combines trajectory-level outcome advantages with token-level semifactual stability for finer-grained credit assignment. Figure 4 presents an overview of SCAPO, which consists of semifactual prompt intervention, token probability drift probing, relative stability signal construction, and policy optimization with GRPO advantages augmented by the semifactual credit signal. 3.1PRELIMINARIES: GROUP RELATIVE POLICY OPTIMIZATION Given a problem–answer pair(x,a) ∼ D, the rollout policyπ old samples a group ofGresponses y i G i=1 under the original promptx, wherey i = (y i,1 ,...,y i,T i )denotes thei-th response. Each response receives a verifiable outcome rewardR i =V(y i ,a)∈0, 1. GRPO constructs a trajectory- level advantage from the relative rewards within the group (Shao et al., 2024): A i = R i − μ R σ R + ε A , i = 1,...,G,(2) whereμ R andσ R are the mean and standard deviation of the group rewards, andε A > 0is a numerical stabilizer. The sameA i is applied to every valid token iny i , so it cannot distinguish token-level dependence on task-irrelevant prompt features. 4 Semifactual Credit-Augmented Policy Optimization T012 Original problem (a) Semifactual Prompt Intervention Fixed response group Teacher-force the response under each prompt and compute sampled token probabilities Prompt Token Position in Response Paraphrase ... ... ... ... ... ... ... 0123 ... T (d) Semifactual Credit-Augmented Policy Optimization Outcome reward ... Semifactual advantage Compute bounded symmetric probability drift Normalize over all tokens in within the same prompt group Relative Stability Stability Position GRPO advantage + = Final advantage ... Policy update (b) Token Probability Drift Probing (c) Relative Stability Signal Construction What is the smallest odd number with four different prime factors? Semifactual intervention What is the smallest odd number that has four distinct prime factors? Typo noise What is the smallest odd number with four diferent prime factors? Scenario wrapper During a classroom warm-up, what is the smallest odd number with four different prime factors? Irrelevant context What is the smallest odd number with four different prime factors? [tag: mx93q] Sample responses from Lower (less stable) Higher (morestable) ... ... Verifiable reward GRPO clipped objective Retain only the negative part of the stability scores Stop-gradient Figure 4: Overview of SCAPO. (a) Semifactual prompt interventions are paired with a fixed group of sampled responses. (b) Teacher forcing measures sampled-token probability drift under each intervention. (c) Group-wise normalization and aggregation yield relative stability scores, retaining only their negative part. (d) The scaled stability correction, with gradients stopped, is added to the GRPO advantage to produce token-level advantages for policy optimization. For sampled tokeny i,t , the importance ratio between the current policyπ θ and the rollout policy, together with its clipped counterpart, is r i,t (θ) = π θ (y i,t | x,y i,<t ) π old (y i,t | x,y i,<t ) , ̄r i,t (θ) = clip(r i,t (θ), 1− ε low , 1 + ε high ). (3) Following the token-mean formulation (Yu et al., 2025), we update the policy by maximizing the clipped token-level policy gradient objective: J GRPO (θ) =E " 1 P G i=1 T i G X i=1 T i X t=1 min(r i,t A i , ̄r i,t A i ) # .(4) 3.2FIXED-RESPONSE SEMIFACTUAL PROBING To obtain token-level credit signals, we examine how sampled-token probabilities change under semifactual prompt interventions while holding the response fixed. For each original promptx = x (0) , we constructKperturbed promptsx (1) ,...,x (K) using the semifactual perturbations described in Section 2. As shown in Figure 4(a), theGresponses sampled under the original prompt are held fixed across interventions. We teacher-force the same response under each prompt, keeping both the sampled token y i,t and its response prefix y i,<t unchanged: p (k) i,t = π old y i,t | x (k) ,y i,<t , k = 0,...,K.(5) The rollout policy remains fixed during these probes and is updated as training proceeds. 5 Semifactual Credit-Augmented Policy Optimization Using the bounded-symmetric distance from Section 2, we convert these aligned probabilities into token-level drift, as illustrated in Figure 4(b): d (k) i,t = p (0) i,t − p (k) i,t p (0) i,t + p (k) i,t /2 , k = 1,...,K.(6) A largerd (k) i,t indicates greater sensitivity of the sampled token’s probability to the semifactual prompt intervention. 3.3GROUP-RELATIVE STABILITY ESTIMATION We convert token-level drift into a relative stability signal within each prompt group. Letd (k) collect the drifts at all valid token positions across the group’sGresponses for perturbation typek. For any vector v over the valid token positions, define the group-wise standardization zscore x (v) = v− μ x (v) σ x (v) + ε , where the mean and standard deviation are computed over all valid response tokens in the group. As shown in Figure 4(c), we standardize negative drift separately for each perturbation type, average the resulting scores, and normalize the aggregate: u = zscore x 1 K K X k=1 zscore x −d (k) ! .(7) The resultingu i,t measures within-group relative stability, with negative values identifying relatively unstable tokens. Since stability alone does not imply correctness, we retain only the negative part of the signal: u − i,t = min(u i,t , 0).(8) 3.4POLICY OPTIMIZATION WITH SEMIFACTUAL CREDIT We add the semifactual credit signal to the original GRPO advantage, enabling tokens with the same outcome reward to receive different learning signals. As shown in Figure 4(d), the resulting token-level advantage is e A i,t = A i + λ sg u − i,t ,(9) whereλ ≥ 0controls the augmentation strength andsgdenotes stop-gradient. The semifactual correction is held fixed during each policy update. This construction ensures e A i,t ≤ A i , with equality whenu i,t ≥ 0. WhenA i > 0, the stability correction can weaken the positive reinforcement signal, and whenA i < 0, it strengthens the negative learning signal. The method thus refines outcome credit without treating semifactual stability as a token-level correctness label. Substituting e A i,t into Eq. 4 gives SCAPO’s policy objective: J SCAPO (θ) =E " 1 P G i=1 T i G X i=1 T i X t=1 min r i,t e A i,t , ̄r i,t e A i,t # .(10) SCAPO modifies only the advantage, leaving GRPO’s importance ratio, clipping, and loss reduction unchanged. To guide early trajectory selection, we apply semifactual credit augmentation during an initial phase and then continue with standard GRPO. Specifically, at policy optimization stepn, the coefficient is λ = λ 0 , 1≤ n≤ N 0 , 0, n > N 0 , (11) whereλ 0 andN 0 control the strength and duration of credit augmentation. Appendix D provides an analysis under a stochastic-gradient model. 6 Semifactual Credit-Augmented Policy Optimization 4EXPERIMENTS 4.1SETUP Datasets.We train on DAPO-Math-17K (Yu et al., 2025), a curated dataset of approximately 17,000 competition-level mathematics problems with integer answers. For SCAPO, each training prompt is paired withK = 4precomputed perturbed prompts, corresponding to the four perturbation types introduced in Section 2. Training.We initialize from Qwen3-4B-Base and Qwen3-1.7B-Base (Yang et al., 2025) and train directly with RL for 600 and 1,000 policy optimization steps, respectively. We use a rollout batch size of 128, an update batch size of 64, and 8 rollouts per prompt. The maximum response length is 16,384 tokens. We use the R1-style prompt template, binary rule-based rewards, and a constant learning rate of10 −6 . For SCAPO, we setλ 0 = 0.01and apply credit augmentation for the first N 0 = 120and200policy optimization steps on the 4B and 1.7B models, respectively. More detailed training hyperparameters are provided in Appendix E.1. Evaluation.We evaluate on in-distribution competition-level mathematical reasoning benchmarks, including AIME 2024–2026 (Mathematical Association of America, 2026), AMC 2023–2025 (Mathe- matical Association of America, 2025), HMMT 2025–2026 (Harvard–MIT Mathematics Tournament, 2026), BRUMO 2025 (Brown University Math Olympiad, 2025), SMT 2025 (Stanford Math Tour- nament, 2025), Omni-Math (Gao et al., 2025a), Minerva (Lewkowycz et al., 2022), and Olympiad- Bench (He et al., 2024). To assess out-of-distribution generalization, we evaluate NoOp-style distractor variants of AIME (Mirzadeh et al., 2025) and ThinkBench perturbations of AIME (Huang et al., 2025) for robustness to input perturbations. We further evaluate GPQA-Diamond (Rein et al., 2024), a graduate-level science question-answering benchmark, to examine transfer beyond mathe- matics. All main results use the final checkpoints with identical generation and scoring settings across methods within each benchmark. We sample at temperature 0.7 and top-p = 0.9with a maximum response length of 16,384 tokens, score mathematical responses with Math-Verify (Kydlí ˇ cek, 2025), and report average accuracy across sampled responses. Further evaluation details are provided in Appendix E.2. Baselines.We consider the following representative RLVR methods: (1) GRPO (Shao et al., 2024): computes group-relative advantages from outcome rewards and applies token-level clipped policy updates. (2) GSPO (Zheng et al., 2025): uses sequence-level importance ratios and clipping for policy optimization. (3) SAPO (Gao et al., 2025b): replaces hard clipping with smooth, advantage-dependent gating. (4) CF-GRPO (Khandoga et al., 2026): estimates token-level credit through counterfactual masking of reasoning spans. (5) FIPO (Ma et al., 2026): reweights token advantages using discounted future KL. At each model scale, comparisons match the training data, reward function, prompt template, optimizer, rollout settings, training-step budget, and evaluation protocol. Method-specific hyperparameters are provided in Appendix E.1. 4.2MAIN RESULTS Mathematical reasoning.Table 1 shows that SCAPO improves over GRPO on all eight mathemat- ical reasoning metrics at both model scales. AIME accuracy increases by 5.63 and 4.17 percentage points on the 4B and 1.7B models, respectively, while HMMT improves by 4.46 and 2.98 points. SCAPO achieves the highest accuracy on six of eight metrics at 4B and all eight at 1.7B, with the highest average accuracy at both scales. Additional results show that SCAPO achieves the highest Pass@128 on AIME and AMC at both model scales (see Appendix F.2). Out-of-distribution generalization. SCAPO achieves the highest accuracy on all three out-of- distribution benchmarks at both model scales (Table 1). On GPQA-Diamond, accuracy improves over GRPO from 36.26% to 42.45% at 4B and from 27.42% to 31.25% at 1.7B, gains of 6.19 and 3.83 percentage points, respectively. Together with the improvements on NoOp-AIME and ThinkBench-AIME, these results show that the gains extend to both perturbed mathematical problems and scientific reasoning beyond the training domain. 7 Semifactual Credit-Augmented Policy Optimization Table 1: Overall accuracy on competition-level mathematical reasoning and out-of-distribution benchmarks (%). Base denotes the model before RL. AIME variants aggregate 2024–2026, while HMMT and AMC aggregate 2025–2026 and 2023–2025, respectively. Year-specific results are provided in Appendix F.1. Bold andunderlineindicate the best and second-best results, respectively. BenchmarkBaseGRPOGSPOSAPOCF-GRPOFIPOSCAPO Qwen3-4B-Base In-Distribution AIME 24–267.6422.0823.4724.0322.6425.4227.71 AMC 23–2537.4059.7464.7165.4562.0664.5765.65 HMMT 25–262.0811.7113.6915.1814.0915.2816.17 BRUMO 2518.5428.9633.7533.5432.0835.0037.71 SMT 2512.7427.1226.0627.1226.7728.1830.31 Omni-Math26.4835.4538.7138.1836.4138.6039.42 Minerva31.6240.0739.7141.1841.5444.4943.38 Olympiad38.5854.3055.6454.7554.7557.5756.82 Avg.21.8834.9336.9737.4336.2938.6439.65 Out-of- Distribution NoOp-AIME 24–266.7420.5620.1420.4221.7421.7423.61 ThinkBench-AIME 24–266.5322.6422.0823.4723.5423.8926.11 GPQA-Diamond33.7836.2641.9241.1636.5140.1542.45 Avg.15.6826.4828.0528.3527.2628.59 30.72 Qwen3-1.7B-Base In-Distribution AIME 24–263.617.717.3610.006.747.0111.88 AMC 23–2523.3332.9732.8238.5831.9932.4841.39 HMMT 25–260.691.982.084.662.981.494.96 BRUMO 256.8813.5411.8817.7114.3710.0019.17 SMT 254.836.966.8410.14 8.028.1410.85 Omni-Math16.7320.4521.4822.6220.7721.7724.74 Minerva21.6927.9427.2130.1528.3128.3130.88 Olympiad21.8131.3133.6837.8333.9832.6440.80 Avg.12.4517.8617.9221.46 18.3917.7323.08 Out-of- Distribution NoOp-AIME 24–262.366.115.978.066.606.608.40 ThinkBench-AIME 24–263.197.577.229.586.947.0112.01 GPQA-Diamond22.3727.4229.2130.96 27.6128.2431.25 Avg.9.3113.7014.1316.2013.7213.9517.22 4.3ABLATION STUDIES We ablate the source and sign of token-level credit to examine their roles in SCAPO’s performance gains, and the results are shown in Table 2. Table 2: Ablations on Qwen3-1.7B-Base. SCAPO uses negative-only corrections. Method AIME 24–26 AMC 23–25 HMMT 25–26 BRUMO 25 SMT 25Avg. GRPO7.7132.971.9813.546.96 12.63 SCAPO11.8841.394.9619.17 10.85 17.65 Credit signal source Random shuffle8.3332.782.2815.429.43 13.65 Counterfactual6.3230.511.6912.926.96 11.68 Correction sign All-sign10.76 38.833.7715.219.6715.65 Positive-only9.7934.102.5815.428.37 14.05 Source of the credit signal. The random shuffle ablation permutes semifactual corrections within each response, preserving their values but breaking the original token align- ment. The counterfactual ablation re- places answer-preserving semifactual prompts with answer-changing per- turbations, while keeping the credit construction unchanged. SCAPO per- forms best on the five competition- level benchmarks, followed consis- tently by random shuffle and the coun- terfactual control. On AIME 24–26, SCAPO leads these controls by 3.55 and 5.56 percentage points, respectively. These results support aligning credit with semifactual sensitivity, since drift under answer-changing perturbations may reflect legitimate changes in token predictions rather than spurious dependence. 8 Semifactual Credit-Augmented Policy Optimization Sign of the credit correction. We compare negative-only, all-sign, and positive-only corrections by retaining negative, all, or positive stability scores. SCAPO’s negative-only correction achieves the highest average accuracy across the five competition-level benchmarks. This supports reducing credit for relatively unstable tokens without rewarding stability alone. 4.4COMPUTATIONAL EFFICIENCY Training time Semifactual 1.8% Rollout 52.7% Policy update 18.7% Other 26.7% Figure 5: Training time breakdown of SCAPO on Qwen3-4B-Base. Other in- cludes validation, reward computation, old policy log-probability computation, and miscellaneous overhead. As shown in Figure 5, semifactual probing and credit construction account for only 1.8% of the measured training runtime in the 4B experiment, far less than autoregressive rollout. The probes reuse sampled re- sponses through teacher forcing, avoiding fresh gen- eration under each perturbed prompt. They are also confined to the initial credit augmentation phase and incur no further cost during subsequent optimization. Together, these properties keep probing a small fraction of total runtime despite evaluating multiple prompts. Training efficiency is therefore more strongly affected by major stages such as autoregressive generation, whose cost depends on response length. 5RELATED WORK Reinforcement learning with verifiable rewards. RLVR has advanced mathematical reasoning by optimizing automatically verifiable outcomes, with critic-free group-relative methods enabling direct RL training of base models (Shao et al., 2024; Guo et al., 2025; Zeng et al., 2025). Subsequent work improves normalization and sampling (Liu et al., 2025; Yu et al., 2025), adapts importance weighting and clipping (Zheng et al., 2025; Gao et al., 2025b), and promotes exploration through entropy control or off-policy guidance (Cui et al., 2025; Yan et al., 2025). However, response-level advantages remain too coarse to distinguish individual reasoning tokens. Studies of reasoning-path coverage further distinguish improved sampling efficiency from expansion of a model’s reasoning repertoire (Yue et al., 2025). Like prior RLVR methods, our approach optimizes verifiable outcome rewards, but leverages semifactual stability to refine credit assignment at the token level. Credit assignment for reasoning.Fine-grained feedback can be learned from process annotations or outcome labels (Lightman et al., 2024; Wang et al., 2024; Cui et al., 2026), or estimated through Monte Carlo rollouts (Kazemnejad et al., 2025). Token entropy, confidence, eligibility traces, and future policy divergence also guide token selection and weighting (Wang et al., 2025b; Xie et al., 2026; Mou et al., 2026; Ma et al., 2026). Perturbation- and gradient-based attribution methods estimate how reasoning tokens or spans contribute to final-answer predictions, providing signals for selective supervised fine-tuning and importance-weighted policy updates (Ruan et al., 2025; Khandoga et al., 2026; Li et al., 2026). Like prior work on fine-grained credit assignment, we use token-level signals to guide policy updates. Our contribution is to derive negative-only advantage corrections from the sensitivity of a fixed response to semifactual prompt interventions, using teacher-forced token probabilities rather than output-side attribution through span masking or continuation regeneration. 6CONCLUSION We presented SCAPO, a causally inspired approach for finer-grained token credit assignment in RLVR. By probing potential token-level spurious dependence through fixed-response sensitivity to semifactual prompt interventions, SCAPO applies negative-only corrections to GRPO advantages, selectively reducing credit for relatively unstable tokens. Across Qwen3-4B-Base and Qwen3-1.7B- Base, SCAPO outperforms GRPO on all reported benchmark metrics and achieves the best results on most benchmarks among the compared RLVR methods. These results highlight that semifactual interventions can serve not only as diagnostic probes of brittle reasoning but also as training signals that complement outcome rewards. Future work will explore this credit augmentation in larger models, more diverse architectures, and domains beyond mathematical reasoning. 9 Semifactual Credit-Augmented Policy Optimization AI USE STATEMENT We used generative AI to generate semifactual perturbation data, polish writing, correct grammar, and assist with code debugging and script development. The authors take responsibility for the final content of this work. REPRODUCIBILITY STATEMENT Our work is easy to reproduce. The original datasets and pretrained models used in our experiments are publicly available, and detailed experimental hyperparameters are provided in Appendix E. We provide our code, developed on top of the open-source EasyR1 framework, together with all training and evaluation data in the Github repo. REFERENCES Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019. Brown University Math Olympiad. BrUMO 2025, 2025. URLhttps://w.brumo.org/archive. Accessed: 2026-07-15. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards. Transactions on Machine Learning Research, 2026. Debopam Das, Tatjana Scheffler, Peter Bourgonje, and Manfred Stede. Constructing a lexicon of english discourse connectives. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, p. 360–365, 2018. Zhizhang Fu, Guangsheng Bao, Hongbo Zhang, Chenkai Hu, and Yue Zhang. Correlation or causation: Analyzing the causal structures of llm and lrm reasoning process. IEEE Transactions on Audio, Speech and Language Processing, 34:2986–2999, 2026. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models. In International Conference on Learning Representations, volume 2025, p. 100540–100569, 2025a. Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, and Junyang Lin. Soft adaptive policy optimization. arXiv preprint arXiv:2511.20347, 2025b. Nelson Goodman. The problem of counterfactual conditionals. The journal of philosophy, 44(5): 113–128, 1947. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean 10 Semifactual Credit-Augmented Policy Optimization Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. Harvard–MIT Mathematics Tournament. HMMT: Past tournaments, 2026. URLhttps://beta. hmmt.org/w/archive/problems. Accessed: 2026-07-15. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3828–3850, 2024. Shulin Huang, Linyi Yang, Yan Song, Shawn Chen, Leyang Cui, Ziyu Wan, Qingcheng Zeng, Ying Wen, Kun Shao, Weinan Zhang, Jun Wang, and Yue Zhang. Thinkbench: Dynamic out- of-distribution evaluation for robust llm reasoning. Advances in Neural Information Processing Systems, 38, 2025. Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Refining credit assignment in RL training of LLMs. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 29557–29590. PMLR, 2025. Eoin M Kenny and Mark T Keane. On generating plausible counterfactual and semi-factual ex- planations for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, p. 11575–11585, 2021. Mykola Khandoga, Rui Yuan, and Vinay Kumar Sankarapu. Beyond uniform credit: Causal credit assignment for policy optimization. arXiv preprint arXiv:2602.09331, 2026. Hynek Kydlí ˇ cek. Math-Verify: Math verification library, 2025. URLhttps://github.com/ huggingface/math-verify. Version 0.9.0. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843–3857, 2022. Ziheng Li, Liu Kang, Feng Xiao, Luxi Xing, Qingyi Si, Zhuoran Li, Weikang Gong, Deqing Yang, Yanghua Xiao, and Hongcheng Guo. Outcome-grounded advantage reshaping for fine-grained credit assignment in mathematical reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 24681–24693, 2026. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, p. 39578–39601, 2024. 11 Semifactual Credit-Augmented Policy Optimization Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. In Conference on Language Modeling (COLM), 2025. Jinghui Lu, Linyi Yang, Brian Mac Namee, and Yue Zhang. A rationale-centric framework for human- in-the-loop machine learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6986–6996, 2022. Chiyu Ma, Shuo Yang, Kexin Huang, Jinda Lu, Haoming Meng, Shangshang Wang, Bolin Ding, Soroush Vosoughi, Guoyin Wang, and Jingren Zhou. FIPO: Eliciting deep reasoning with future-KL influenced policy optimization. In Conference on Language Modeling (COLM), 2026. Mathematical Association of America. American Mathematics Competitions (AMC), 2025. URL https://maa.org/student-programs/amc/. Accessed: 2026-07-15. Mathematical Association of America. American Invitational Mathematics Examination (AIME), 2026. URL https://maa.org/maa-invitational-competitions/. Accessed: 2026-07-15. Iman Mirzadeh, Keivan Alizadeh-Vahid, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. In International Conference on Learning Representations, volume 2025, p. 94743–94765, 2025. Chaoli Mou, Zhan Zhuang, Xinning Chen, and Yu Zhang. Beyond uniform credit assignment: Selective eligibility traces for rlvr. arXiv preprint arXiv:2605.05965, 2026. OpenAI. Learning to reason with LLMs, September 2024. URLhttps://openai.com/index/ learning-to-reason-with-llms/. OpenAI.GPT-5.5 system card, April 2026.URLhttps://openai.com/index/ gpt-5-5-system-card/. Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):947–1012, 2016. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In Conference on Language Modeling (COLM), 2024. Zhiwen Ruan, Yixia Li, He Zhu, Yun Chen, Peng Li, Yang Liu, and Guanhua Chen. Enhancing large language model reasoning via selective critical token fine-tuning. arXiv preprint arXiv:2510.10974, 2025. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Stanford Math Tournament. Stanford Math Tournament 2025: Tests and solutions, 2025. URL https://w.stanfordmathtournament.org/past-tests/SMT/2025. Accessed: 2026-07-15. Chenlong Wang, Yuanning Feng, Dongping Chen, Zhaoyang Chu, Ranjay Krishna, and Tianyi Zhou. Wait, we don’t need to" wait"! removing thinking tokens improves reasoning efficiency. In EMNLP (Findings), p. 7459–7482, 2025a. Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9426–9439, 2024. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. Advances in Neural Information Processing Systems, 38:115452–115486, 2025b. 12 Semifactual Credit-Augmented Policy Optimization Tianlu Wang, Rohit Sridhar, Diyi Yang, and Xuezhi Wang. Identifying and mitigating spurious corre- lations for improving robustness in nlp models. In Findings of the association for computational linguistics: NAACL 2022, p. 1719–1729, 2022. Xiaoxuan Wang, Han Zhang, Haixin Wang, Yidan Shi, Ruoyan Li, Kaiqiao Han, Chenyi Tong, Haoran Deng, Alexander K. Taylor, Renliang Sun, Yanqiao Zhu, Jason Cong, Yizhou Sun, and Wei Wang. ARLArena: A unified framework for stable agentic reinforcement learning. In International Conference on Machine Learning, 2026. Can Xie, Ruotong Pan, Xiangyu Wu, Yunfei Zhang, Jiayi Fu, Tingting Gao, and Guorui Zhou. Unlocking exploration in rlvr: Uncertainty-aware advantage shaping for deeper reasoning. In Findings of the Association for Computational Linguistics: ACL 2026, p. 19057–19076, 2026. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. Advances in Neural Information Processing Systems, 38:117157–117186, 2025. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2025. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? Advances in Neural Information Processing Systems, 38:57654–57689, 2025. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. SimpleRL- Zoo: Investigating and taming zero reinforcement learning for open base models in the wild. In Conference on Language Modeling (COLM), 2025. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. ALIMITATIONS Our training experiments are limited to mathematical reasoning with dense Qwen3 models of 1.7B and 4B parameters trained on DAPO-Math-17K. Although GPQA-Diamond provides evidence of transfer to scientific reasoning, the effectiveness of SCAPO at larger model and data scales and with broader training domains remains to be evaluated. Future work should also examine other architectures, including mixture-of-experts (MoE) and hybrid models. BSEMIFACTUAL PERTURBATIONS AND TOKEN-LEVEL ANALYSIS B.1PERTURBATION CONSTRUCTION We request GPT-5.5 (OpenAI, 2026) to generate four perturbed prompts per DAPO-Math-17K problem, using the full system and user prompts in Table 3. 13 Semifactual Credit-Augmented Policy Optimization Table 3: Full prompts used to generate semifactual perturbations. The user template is filled with the original problem and reference answer. System prompt Given a math problem and its original answer, produce exactly 4 prompt perturbations. Core requirements: - Preserve the same mathematical problem. - Preserve all numeric values exactly as written. - Preserve all constraints, operations, requested quantity, and the original answer. - Produce exactly one perturbation for each perturbation_type: paraphrase, typo_noise, scenario_wrap, irrelevant_context. - Do not add any condition that helps solve the problem. - Do not solve the problem. - Do not reveal reasoning. - Return JSON only. Math and LaTeX preservation: - Treat all math expressions as frozen text. This includes text inside $...$, $$...$$, \(...\), \[...\], and standalone symbolic expressions such as variables, equations, inequalities, fractions, powers, percentages, and units. - Do not rewrite, reorder, escape, unescape, or modify any LaTeX/math content. - Preserve every LaTeX command exactly as it appears in the original parsed string: examples include , , , , , . - In JSON, escape backslashes correctly so that the parsed string contains exactly one backslash before each LaTeX command. Never output doubled LaTeX commands like \ , and never output raw control characters such as tab or form feed. - Do not alter variables, point labels, entity labels, numbers, units, comparison signs, operations, or answer format. Type-specific requirements: - paraphrase: Rewrite only the natural-language prose that sits outside math expressions. Keep the same scenario, the same entities, the same event and causal order, and every mathematical fact unchanged. Do not introduce a new setting or new objects. - typo_noise: Introduce exactly one character-level edit (insert, delete, substitute, or swap two adjacent characters) in one ordinary English word. The chosen word must NOT be math, LaTeX, a variable, a number, a unit, a proper name, a label, or a mathematical keyword such as "mean", "median", "integer", "prime", "even", "odd". The perturbed text must differ from the original by at least that one non-whitespace character; whitespace-only differences are invalid. The word must still be recognizable to a human reader. - scenario_wrap: Add a light scenario frame around the original problem. This can be a short prefix, a short suffix, or one brief background phrase inserted into the natural-language part of the problem. This should be a small irrelevant context shift, not a full rewrite into a new domain. Keep the original people, objects, quantities, units, formulas, variables, constraints, operations, event order, and requested quantity unchanged. Do not replace core entities or items. Do not change "clips" into "packages", "wallet" into "sensor kit", "flowers" into "crates", or a person into a machine. Good examples: * "At a school craft booth, [original problem]" * "During a budgeting exercise, [original problem]" * "While tracking a reading plan, [original problem]" * "[Original first sentence] This is part of a classroom activity. [remaining original problem]" * "[Original problem] This was recorded in a simple daily log." * "[Original problem] The setting is an ordinary after-school situation." Bad examples: * Rewriting the full story into a different domain. * Replacing the actors or objects with different actors or objects. * Adding hints, extra facts, new conditions, or solution-relevant context. * Adding numbers, formulas, variables, quantities, or mathematical facts. - irrelevant_context: 14 Semifactual Credit-Augmented Policy Optimization System prompt (continued) Keep the original problem text exactly unchanged. Append exactly one short meaningless distractor at the very end (after the final question mark or full stop). Acceptable distractor shapes (pick one; vary the shape and content across samples): * A vacuous logical tautology, e.g. "and true is true", "note that 1 = 1", "recall that each thing equals itself". * A short random alphanumeric string of length 6 to 12 characters, e.g. "5XeflW1ZJc", "qP7mz39a". * A bracketed decorative tag, e.g. "[tag: mx93q]", "[marker-a7f]". The distractor MUST NOT contain: * Any mathematical fact, quantity, variable, unit, equation, or operator that could interact with the problem. * Any hint about the problem, the subject area, or the solution approach. * Any statement about the characters, objects, or scenario that appear in the problem. * Any framing or meta-language such as "question", "problem", "exercise", "consider", "observe", "for practice", "simple", "word problem". Return this JSON shape: "perturbations": [ "perturbed_question": "...", "perturbation_type": "paraphrase | typo_noise | scenario_wrap | irrelevant_context" ] User prompt Original problem: problem Original answer: answer Example: Four semifactual perturbations Original: What is the smallest odd number with four different prime factors? Paraphrase: What is the smallest odd number that has four distinct prime factors? Typo noise: What is the smallest odd number with four diferent prime factors? Scenario wrapper: During a classroom warm-up, what is the smallest odd number with four different prime factors? Irrelevant context: What is the smallest odd number with four different prime factors? [tag: mx93q] Answer: 1155 B.2DIAGNOSTIC SETUP We randomly select 1,000 DAPO-Math-17K questions and their four perturbed prompts to form a fixed diagnostic subset dataset. Frozen Qwen3-4B-Base generates one response per original prompt, with temperature 1.0, top- p = 1.0, and a limit of 8,192 response tokens. We then teacher-force the same response tokens under the original prompt and all four perturbed prompts. Each response token contributes one mean drift d t across the four perturbations, yielding 870,586 original response token drifts. B.3TOKEN TAXONOMY AND STATISTICS We assign each decoded response token to one of the eight mutually exclusive categories in Ta- ble 4. Reflection markers are identified using a 16-entry lexicon adapted from prior work (Wang et al., 2025a) and our generation traces. Discourse connectives are identified using 80 single-part DiMLex-Eng forms that contain no whitespace (Das et al., 2018). Lexical matching ignores case, sur- rounding whitespace, and edge punctuation, with reflection markers taking precedence over discourse connectives. 15 Semifactual Credit-Augmented Policy Optimization Table 4: Token categories and drift statistics underlying Figure 2(c). Share is the percentage of the 870,586 response token positions assigned to each category. Mean is the category’s average drift d t . Relative mean is this average divided by the overall mean (0.01640).denotes a space and newline. CategoryClassification ruleToken examplesShare (%) Mean d t Relative mean Reflection marker a Matches the reflection lexicon after lexical normalization. any,check, Now, However,verify, again 0.29 0.044962.74 Discourse connective Matches the 80-form DiMLex-Eng lexicon after normalization. Excludes reflection markers. and,for,if, or, Thus, Therefore 2.68 0.038072.32 WordContains a Unicode letter and matches neither lexical category nor the math-symbol rule. the,of,is,to, we,in 40.92 0.025061.53 PunctuationAll non-whitespace characters have Unicode category P*. Excludes math symbols. , . : # ), : 6.75 0.019561.19 Math symbolBegins with contains only characters from the fixed math/L A T E X symbol set. Excludes numbers. \,=,+,-, , 26.62 0.007680.47 NumberMatches a signed integer, decimal, or percentage expression. 0, 1, 2, 3, 4, 515.31 0.006120.37 Whitespace-only Decodes entirely to Unicode whitespace. ,, , 6.71 0.007180.44 OtherControl tokens, undecodable UTF-8 fragments, or remaining forms not covered above. <|endoftext|>, ```,``, $, 0.72 0.029251.78 a The reflection lexicon contains the following 16 words:again,ah,alternative,alternatively,another, any, but, check, hmm, however, maybe, now, oh, other, verify, wait. B.4COMPLETE-RESPONSE CASE STUDIES Figures 6 and 7 illustrate two cases with their original and perturbed prompts. Response tokens are shaded by their mean probability drift across the four prompt perturbations, with darker shading indicating greater sensitivity. These examples show that even when the final answer is correct, the probabilities of individual response tokens can change substantially under answer-preserving prompt perturbations. 16 Semifactual Credit-Augmented Policy Optimization Case 1 | Compound interest Original prompt Kimberly borrows 1000 dollars from Lucy, who charged interest of 5% per month (which compounds monthly). What is the least integer number of months after which Kimberly will owe more than twice as much as she borrowed? Paraphrase Kimberly takes a loan of 1000 dollars from Lucy, who charged interest of 5% per month (which compounds monthly). What is the smallest integer number of months after which Kimberly will owe more than twice as much as she borrowed? Typo noise Kimberly borows 1000 dollars from Lucy, who charged interest of 5% per month (which compounds monthly). What is the least integer number of months after which Kimberly will owe more than twice as much as she borrowed? Scenario wrapper During a budgeting exercise, Kimberly borrows 1000 dollars from Lucy, who charged interest of 5% per month (which compounds monthly). What is the least integer number of months after which Kimberly will owe more than twice as much as she borrowed? Irrelevant context Kimberly borrows 1000 dollars from Lucy, who charged interest of 5% per month (which compounds monthly). What is the least integer number of months after which Kimberly will owe more than twice as much as she borrowed? qPzmXaRt Response Mean drift To 0.053 ▁find 0.163 ▁the 0.031 ▁least 0.518 ▁integer 0.300 ▁number 0.027 ▁of 0.003 ▁months 0.003 ▁$ 0.558 n 0.058 $ 0.004 ▁after 0.130 ▁which 0.001 ▁Kimberly 0.096 ▁will 0.047 ▁owe 0.002 ▁more 0.015 ▁than 0.001 ▁twice 0.017 ▁as 0.114 ▁much 0.002 ▁as 0.009 ▁she 0.004 ▁borrowed 0.004 , 0.014 ▁we 0.010 ▁can 0.147 ▁set 0.087 ▁up 0.001 ▁the 0.019 ▁following 0.064 ▁inequality 0.061 : 0.371 ▁$\ 0.038 boxed 0.264 0.021 P 0.266 ( 0.048 1 0.000 ▁+ 0.186 ▁\ 0.105 frac 0.003 0.004 r 0.005 0.000 1 0.044 0 0.003 0 0.000 ) 0.001 0.000 n 0.000 0.163 ▁> 0.012 ▁ 0.022 2 0.000 P 0.019 $, 0.024 ▁where 0.003 ▁$ 0.007 P 0.002 $ 0.003 ▁is 0.002 ▁the 0.001 ▁principal 0.021 , 0.169 ▁$ 0.036 r 0.001 $ 0.000 ▁is 0.000 ▁the 0.000 ▁interest 0.081 ▁rate 0.000 ▁per 0.040 ▁month 0.043 ▁in 0.117 ▁percent 0.021 , 0.009 ▁and 0.001 ▁$ 0.001 n 0.000 $ 0.000 ▁is 0.000 ▁the 0.000 ▁number 0.002 ▁of 0.000 ▁months 0.002 .↵ 0.066 Sub 0.075 stit 0.008 uting 0.000 ▁in 0.159 ▁the 0.011 ▁values 0.042 ▁we 0.179 ▁know 0.054 , 0.027 ▁we 0.005 ▁get 0.060 ▁$\ 0.093 boxed 0.001 0.008 1 0.002 0 0.001 0 0.000 0 0.000 ( 0.002 1 0.000 ▁+ 0.003 ▁\ 0.006 frac 0.000 0.000 5 0.001 0.000 1 0.000 0 0.000 0 0.000 ) 0.000 0.000 n 0.000 0.001 ▁> 0.000 ▁ 0.001 2 0.000 0 0.015 0 0.000 0 0.000 $ 0.070 .↵ 0.003 We 0.122 ▁can 0.004 ▁simplify 0.037 ▁this 0.009 ▁to 0.039 ▁$( 0.139 1 0.000 . 0.007 0 0.000 5 0.000 ) 0.001 n 0.000 ▁> 0.004 ▁ 0.001 2 0.000 $. 0.099 ▁Taking 0.086 ▁the 0.003 ▁logarith 0.100 m 0.000 ▁of 0.010 ▁both 0.000 ▁sides 0.000 , 0.031 ▁we 0.001 ▁get 0.008 ▁$ 0.014 n 0.000 ▁\ 0.062 log 0.017 ( 0.049 1 0.000 . 0.000 0 0.000 5 0.000 ) 0.000 ▁> 0.000 ▁\ 0.000 log 0.000 ( 0.007 2 0.000 )$ 0.028 .↵ 0.021 Div 0.049 iding 0.000 ▁both 0.006 ▁sides 0.000 ▁by 0.000 ▁$\ 0.000 log 0.000 ( 0.000 1 0.000 . 0.000 0 0.000 5 0.000 )$ 0.000 , 0.007 ▁we 0.000 ▁get 0.008 ▁$ 0.004 n 0.000 ▁> 0.000 ▁\ 0.000 frac 0.001 \ 0.000 log 0.000 ( 0.000 2 0.000 ) 0.000 \ 0.000 log 0.000 ( 0.000 1 0.000 . 0.000 0 0.000 5 0.000 ) 0.000 ▁\ 0.126 approx 0.000 ▁ 0.000 1 0.002 4 0.014 . 0.000 2 0.008 $ 0.132 .↵ 0.001 Since 0.063 ▁$ 0.097 n 0.000 $ 0.000 ▁must 0.002 ▁be 0.000 ▁an 0.003 ▁integer 0.000 , 0.001 ▁the 0.044 ▁least 0.247 ▁integer 0.032 ▁value 0.065 ▁of 0.016 ▁$ 0.000 n 0.000 $ 0.000 ▁is 0.105 ▁$\ 0.010 boxed 0.000 0.001 1 0.000 5 0.000 0.014 $. 0.000 <|endoftext|> 0.075 ▁ = space ↵ = newline 0.00.20.40.6 Figure 6: Token-level semifactual sensitivity in a compound-interest problem. 17 Semifactual Credit-Augmented Policy Optimization Case 2 | Five-digit numeral base Original prompt 100_10 in base b has exactly 5 digits. What is the value of b? Paraphrase In base b, 100_10 has exactly 5 digits. What is the value of b? Typo noise 100_10 in base b has exatly 5 digits. What is the value of b? Scenario wrapper During a classroom warm-up, 100_10 in base b has exactly 5 digits. What is the value of b? Irrelevant context 100_10 in base b has exactly 5 digits. What is the value of b? [marker-a7f] Response Mean drift To 0.190 ▁determine 0.073 ▁the 0.001 ▁base 0.166 ▁\( 0.013 ▁b 0.002 ▁\ 0.000 ) 0.000 ▁in 0.205 ▁which 0.000 ▁the 0.039 ▁decimal 0.430 ▁number 0.000 ▁\( 0.010 ▁ 0.003 1 0.000 0 0.000 0 0.000 _ 0.014 1 0.000 0 0.000 0.000 ▁\ 0.000 ) 0.000 ▁has 0.029 ▁exactly 0.000 ▁ 0.009 5 0.000 ▁digits 0.000 , 0.000 ▁we 0.012 ▁need 0.028 ▁to 0.000 ▁use 0.231 ▁the 0.001 ▁relationship 0.153 ▁between 0.000 ▁the 0.005 ▁number 0.006 ▁of 0.000 ▁digits 0.000 ▁in 0.146 ▁a 0.006 ▁number 0.041 ▁and 0.060 ▁the 0.051 ▁base 0.007 ▁of 0.075 ▁the 0.036 ▁numeral 0.143 ▁system 0.000 . 0.073 ▁Specifically 0.021 , 0.000 ▁a 0.059 ▁number 0.002 ▁\( 0.001 ▁N 0.030 ▁\ 0.006 ) 0.000 ▁has 0.092 ▁exactly 0.050 ▁\( 0.001 ▁d 0.009 ▁\ 0.001 ) 0.000 ▁digits 0.000 ▁in 0.000 ▁base 0.000 ▁\( 0.000 ▁b 0.000 ▁\ 0.000 ) 0.000 ▁if 0.000 ▁and 0.071 ▁only 0.000 ▁if 0.000 :↵ 0.064 \[ 0.022 ▁b 0.000 0.000 d 0.001 - 0.000 1 0.000 0.000 ▁\ 0.000 le 0.000 q 0.001 ▁N 0.000 ▁< 0.000 ▁b 0.000 0.000 d 0.000 ▁\ 0.008 ]↵ 0.000 In 0.105 ▁this 0.007 ▁problem 0.034 , 0.000 ▁\( 0.017 ▁N 0.000 ▁= 0.000 ▁ 0.000 1 0.000 0 0.000 0 0.000 _ 0.058 1 0.000 0 0.000 0.000 ▁\ 0.000 ) 0.000 ▁and 0.001 ▁\( 0.013 ▁d 0.000 ▁= 0.000 ▁ 0.000 5 0.000 ▁\ 0.000 ). 0.002 ▁Therefore 0.035 , 0.000 ▁we 0.001 ▁need 0.007 ▁to 0.001 ▁find 0.010 ▁the 0.275 ▁base 0.006 ▁\( 0.000 ▁b 0.000 ▁\ 0.000 ) 0.000 ▁such 0.001 ▁that 0.000 :↵ 0.000 \[ 0.000 ▁b 0.000 0.019 4 0.000 ▁\ 0.000 le 0.000 q 0.000 ▁ 0.000 1 0.000 0 0.000 0 0.000 ▁< 0.001 ▁b 0.000 0.000 5 0.000 ▁\ 0.000 ]↵ 0.000 First 0.018 , 0.000 ▁we 0.064 ▁solve 0.055 ▁the 0.009 ▁inequality 0.008 ▁\( 0.000 ▁b 0.012 0.000 4 0.000 ▁\ 0.000 le 0.000 q 0.000 ▁ 0.000 1 0.000 0 0.000 0 0.000 ▁\ 0.000 ). 0.147 ▁Taking 0.086 ▁the 0.000 ▁fourth 0.000 ▁root 0.000 ▁of 0.000 ▁both 0.001 ▁sides 0.000 , 0.000 ▁we 0.000 ▁get 0.000 :↵ 0.000 \[ 0.000 ▁b 0.000 ▁\ 0.000 le 0.000 q 0.000 ▁\ 0.002 sqrt 0.000 [ 0.000 4 0.000 ] 0.000 1 0.000 0 0.000 0 0.000 0.000 ▁\ 0.000 ]↵ 0.034 Since 0.158 ▁\( 0.000 ▁\ 0.044 sqrt 0.000 [ 0.000 4 0.000 ] 0.000 1 0.000 0 0.000 0 0.000 0.000 ▁\ 0.014 approx 0.002 ▁ 0.000 3 0.000 . 0.000 1 0.000 6 0.000 2 0.044 ▁\ 0.000 ), 0.000 ▁the 0.049 ▁largest 0.030 ▁integer 0.008 ▁\( 0.005 ▁b 0.000 ▁\ 0.000 ) 0.000 ▁that 0.041 ▁satisfies 0.000 ▁this 0.001 ▁inequality 0.000 ▁is 0.000 ▁\( 0.044 ▁b 0.000 ▁= 0.000 ▁ 0.000 3 0.000 ▁\ 0.000 ).↵ 0.022 Next 0.000 , 0.000 ▁we 0.000 ▁solve 0.039 ▁the 0.000 ▁inequality 0.000 ▁\( 0.000 ▁ 0.000 1 0.000 0 0.000 0 0.000 ▁< 0.000 ▁b 0.000 0.000 5 0.000 ▁\ 0.000 ). 0.000 ▁Taking 0.000 ▁the 0.000 ▁fifth 0.000 ▁root 0.000 ▁of 0.000 ▁both 0.001 ▁sides 0.000 , 0.000 ▁we 0.000 ▁get 0.000 :↵ 0.000 \[ 0.000 ▁ 0.119 1 0.000 0 0.000 0 0.000 ▁< 0.012 ▁b 0.000 0.000 5 0.000 ▁\ 0.000 ]↵ 0.092 Since 0.010 ▁\( 0.000 ▁\ 0.019 sqrt 0.000 [ 0.000 5 0.000 ] 0.000 1 0.000 0 0.000 0 0.000 0.000 ▁\ 0.000 approx 0.000 ▁ 0.000 2 0.000 . 0.000 5 0.000 1 0.001 1 0.016 ▁\ 0.012 ), 0.000 ▁the 0.000 ▁smallest 0.000 ▁integer 0.000 ▁\( 0.000 ▁b 0.000 ▁\ 0.000 ) 0.000 ▁that 0.000 ▁satisfies 0.000 ▁this 0.000 ▁inequality 0.000 ▁is 0.000 ▁\( 0.000 ▁b 0.000 ▁= 0.000 ▁ 0.000 3 0.009 ▁\ 0.000 ).↵ 0.011 Thus 0.161 , 0.000 ▁the 0.002 ▁only 0.130 ▁integer 0.005 ▁\( 0.017 ▁b 0.000 ▁\ 0.000 ) 0.000 ▁that 0.000 ▁satisfies 0.000 ▁both 0.000 ▁inequalities 0.026 ▁is 0.062 ▁\( 0.000 ▁b 0.000 ▁= 0.000 ▁ 0.000 3 0.001 ▁\ 0.000 ).↵ 0.064 To 0.018 ▁confirm 0.055 , 0.000 ▁we 0.008 ▁check 0.091 ▁the 0.021 ▁range 0.492 :↵ 0.110 - 0.003 ▁For 0.101 ▁\( 0.000 ▁b 0.000 ▁= 0.000 ▁ 0.000 3 0.002 ▁\ 0.000 ), 0.017 ▁\( 0.006 ▁ 0.004 3 0.000 0.000 4 0.000 ▁= 0.000 ▁ 0.000 8 0.000 1 0.000 ▁\ 0.000 ) 0.039 ▁and 0.000 ▁\( 0.000 ▁ 0.000 3 0.001 0.000 5 0.000 ▁= 0.000 ▁ 0.000 2 0.000 4 0.000 3 0.000 ▁\ 0.000 ). 0.008 ▁Clearly 0.048 , 0.000 ▁\( 0.000 ▁ 0.000 8 0.000 1 0.000 ▁\ 0.000 le 0.000 q 0.000 ▁ 0.000 1 0.000 0 0.000 0 0.000 ▁< 0.000 ▁ 0.000 2 0.000 4 0.000 3 0.000 ▁\ 0.000 ), 0.009 ▁so 0.002 ▁\( 0.004 ▁b 0.246 ▁= 0.000 ▁ 0.000 3 0.000 ▁\ 0.000 ) 0.000 ▁is 0.006 ▁correct 0.043 .↵ 0.002 Therefore 0.000 , 0.000 ▁the 0.000 ▁value 0.003 ▁of 0.000 ▁\( 0.000 ▁b 0.000 ▁\ 0.000 ) 0.000 ▁is 0.000 ▁\ 0.006 (\ 0.000 boxed 0.000 0.000 3 0.001 \ 0.000 ). 0.011 <|endoftext|> 0.002 ▁ = space ↵ = newline 0.00.20.40.6 Figure 7: Token-level semifactual sensitivity in a numeral-base problem. 18 Semifactual Credit-Augmented Policy Optimization Cd-FILTERED DECODING C.1DECODING SETUP We use frozen Qwen3-4B-Base on the 1,000-question panel in Appendix B.2. Both base sampling and d-filtered decoding use the same prompt template, temperature of 1.0, top-p = 1.0, and a maximum response length of 8,192 tokens. We cap the cumulative clean-policy probability mass of masked tokens at 0.8, which is the sole filtering hyperparameter. For each question, we independently generate one response under each decoding strategy. C.2FILTERING PROCEDURE At each decoding step, we compute the mean bounded-symmetric drift of every vocabulary token across the four perturbed prompts. We sort all token candidates by decreasing drift. We mask token candidates in this order until adding the next candidate would exceed the mass cap. The EOS token is always retained. The cap limits the total probability mass removed, not the fraction of tokens masked. Writingp(v)for the clean-policy probability at the current prefix andMfor the masked set, the filtered sampling distribution is p filter (v) = p(v)1v /∈M 1− P w∈M p(w) .(12) We sample a proposal from the clean policy and accept it if it is unmasked. Otherwise, we reject it and resample from the distribution renormalized over unmasked tokens. Drift and the masked set are recomputed at each step using the current filtered-response prefix. C.3RESULTS As shown in Table 5, filtering yields 202 incorrect-to-correct and 60 correct-to-incorrect transitions, a gain of 14.20 percentage points. On average, 50.58 token proposals are rejected and resampled per filtered response (4.58% of decoding steps), with at least one such rejection in each of the 1,000 filtered responses. Table 5: Decoding results on 1,000 questions with frozen Qwen3-4B-Base. The lower block reports question counts for the four combinations of baseline and filtered answer correctness. MetricBase samplingd-filtered Correct answers158300 Accuracy (%)15.8030.00 Paired correctness outcomes (question counts) Filtered correctFiltered incorrect Baseline correct9860 Baseline incorrect202640 DOPTIMIZATION ANALYSIS We analyze credit augmentation under a stochastic-gradient model with a fixed objective F . Lemma D.1 (Bounded auxiliary gradient). Letbg λ andbg 0 be the batch gradients of Eqs. 10 and 4 on the same complete prompt groups, with advantages and stability scores held fixed. If importance ratios are at most R and token log-probability gradients have norm at most B, then ∥bg λ −bg 0 ∥≤ λH, H := RB/2.(13) Proof. Within each group, write⟨·⟩for the valid-token mean andc = (−u) + . Eq. 7 gives⟨u⟩ = 0 and⟨u 2 ⟩≤ 1, hence ⟨c⟩ = 1 2 ⟨|u|⟩≤ 1 2 p ⟨u 2 ⟩≤ 1 2 . 19 Semifactual Credit-Augmented Policy Optimization At differentiable points, the coefficient of a token’s log-probability gradient in the clipped surrogate is rA,r max(A, 0), orr min(A, 0), eachr-Lipschitz inA. ReplacingAbyA− λctherefore changes the group gradient by at mostλRB⟨c⟩≤ λH. Token-mean aggregation across groups preserves this bound. Stochastic-gradient model. Consider θ n+1 = θ n + ηbv n , bv n = bg n + λ n b h n ,(14) wherebg n is the baseline gradient andλ n b h n is the gradient difference induced by credit augmentation, with∥ b h n ∥ ≤ Hby Lemma D.1. LetE n denote the expectation conditional on the history before update n. Assume F is L-smooth, bounded above by F sup , and ∥E n bg n −∇F (θ n )∥≤ β,E n ∥bv n −E n bv n ∥ 2 ≤ σ 2 .(15) The two gradient components may be correlated. The parameterβallows bias in the baseline direction. Theorem D.2 (Finite-duration augmentation). Under the above assumptions, let0 < η ≤ 1/Land use the schedule in Eq. 11. For anyT ≥ 1, set∆ F = F sup −EF (θ 1 )andm T = min(T,N 0 ). Then 1 T T X n=1 E∥∇F (θ n )∥ 2 ≤ 2∆ F ηT + Lησ 2 + β 2 + 2βHλ 0 + H 2 λ 2 0 m T T .(16) Proof. Writev n = ∇F (θ n )andb n =E n bv n − v n , so∥b n ∥ ≤ β + Hλ n . Smoothness and Eq. 15 give E n F (θ n+1 )≥ F (θ n ) + η⟨v n ,v n + b n ⟩− Lη 2 2 ∥v n + b n ∥ 2 + σ 2 ≥ F (θ n ) + η 2 ∥v n ∥ 2 − η 2 (β + Hλ n ) 2 − Lη 2 2 σ 2 , where the second line usesηL≤ 1and2⟨v,v + b⟩ =∥v∥ 2 +∥v + b∥ 2 −∥b∥ 2 . Taking expectations, summing over n, and usingEF (θ T +1 )≤ F sup yields the claim. Implication for the schedule. For an unbiased baseline (β = 0), the augmentation term is H 2 λ 2 0 min(T,N 0 )/T. With fixedλ 0 andN 0 , this contribution to the average stationarity bound decays asN 0 /Tafter augmentation ends. HereTis an arbitrary observation horizon, not a prescribed training budget. EADDITIONAL EXPERIMENTAL DETAILS E.1TRAINING HYPERPARAMETERS Table 6 lists shared training hyperparameters and method-specific settings. A rule-based verifier assigns reward 1 to a correct boxed answer and 0 otherwise, and no separate format reward is used. All models share the following prompt template for mathematical reasoning during training and inference: Prompt template question Please reason step by step, and put your final answer within . All methods, including GSPO and SAPO, use token-mean loss reduction for better performance on mathematical reasoning tasks (Yu et al., 2025; Wang et al., 2026). 20 Semifactual Credit-Augmented Policy Optimization Table 6: Shared and method-specific training hyperparameters. Batch sizes count prompts unless stated otherwise. Each rollout step contains two policy optimization steps. Clipping offsets are relative to 1. HyperparameterQwen3-4B-BaseQwen3-1.7B-Base Data and rollout settings Training datasetDAPO-Math-17K (Yu et al., 2025) Maximum prompt length1,024 Maximum response length16,384 Rollout batch size (prompts)128 Responses per prompt (G)8 Sampling temperature / top-p1.0 / 1.0 Optimization settings OptimizerAdamW, β 1 = 0.9, β 2 = 0.999 Learning rate10 −6 Learning-rate scheduleConstant Weight decay0.01 Gradient clipping norm1.0 Update mini-batch size (prompts)64 Loss reductionToken-mean KL regularizationDisabled Training settings Rollout steps300500 Policy optimization steps6001,000 Mixed precisionBF16 Method-specific settings SCAPO: Correction strength (λ 0 )0.01 SCAPO: Augmentation duration (N 0 )120200 GRPO / CF-GRPO / FIPO / SCAPO: Policy clipping (lower / upper)0.20 / 0.28 Dual-clip coefficient10.0 GSPO: Policy clipping (lower / upper)3× 10 −4 / 4× 10 −4 SAPO: Gate temperatures (τ pos /τ neg )1.0 / 1.05 CF-GRPO: Maximum probed spans per response10 CF-GRPO: Span length5–50 CF-GRPO: Token-weight bounds[0.5, 4.0] FIPO: Future-KL decay32.0 FIPO: Influence-weight clipping[1.0, 1.2] FIPO: Safety threshold10.0 Each rollout batch contains 128 prompt groups and is optimized once in mini-batches of 64 groups (512 responses), giving two policy optimization steps per rollout step. Thus,N 0 = 120/200policy optimization steps correspond to 60/100 rollout steps for 4B/1.7B. Diagnostic checkpoint indices use rollout steps unless stated otherwise. E.2EVALUATION PROTOCOL AND TRAINING CURVES Table 7 summarizes the evaluation datasets and sample counts. 21 Semifactual Credit-Augmented Policy Optimization Table 7: Evaluation benchmarks and sample counts. Slash-separated problem counts follow the listed years. ∗ For GPQA-Diamond, we enumerate all4! = 24permutations of the multiple-choice options to avoid contamination. BenchmarkProblemsSamples per problem AIME 2024 / 2025 / 202630 / 30 / 3016 AMC 2023 / 2024 / 202540 / 45 / 4216 HMMT Feb. 2025 / 202630 / 3316 BRUMO 20253016 SMT 20255316 Omni-Math2,8211 Minerva2721 OlympiadBench6741 NoOp-AIME 2024 / 2025 / 202630 / 30 / 3016 ThinkBench-AIME 2024 / 2025 / 2026120 / 120 / 1204 GPQA-Diamond19824 ∗ For Omni-Math, we use the 2,821-problem subset designed for rule-based evaluation. We evaluate the final checkpoints using identical prompts and sample counts across methods, with temperature 0.7, top-p = 0.9, and a maximum response length of 16,384 tokens. Responses on the mathematical benchmarks are scored with Math-Verify (Kydlí ˇ cek, 2025). We perform rule-based validation before training and every 40 policy optimization steps. Figure 1 presents the corresponding accuracy curves. FADDITIONAL EXPERIMENTAL RESULTS F.1DETAILED BENCHMARK RESULTS Table 8 shows the yearly results underlying Table 1. Table 8: Yearly accuracy (%) for Table 1. Bold and underlining denote the best and second-best results. In-DistributionOut-of-Distribution MethodAIMEAMCHMMTNoOp-AIMEThinkBench-AIME 2425262324252526242526242526 Qwen3-4B-Base Base8.337.926.67 47.66 32.50 32.891.043.038.337.294.588.963.756.88 GRPO23.54 23.13 19.58 66.41 55.69 57.74 10.83 12.50 21.25 21.25 19.17 25.42 22.9219.58 GSPO27.08 22.92 20.42 73.59 61.39 59.82 10.42 16.67 19.58 21.4619.38 24.58 22.7118.96 SAPO26.46 24.17 21.46 72.50 59.31 65.33 12.29 17.8021.25 21.04 18.96 27.71 22.5020.21 CF-GRPO 24.17 24.17 19.58 68.44 59.03 59.23 13.1314.96 23.75 21.4620.00 26.88 23.3320.42 FIPO27.50 27.0821.6769.06 61.8163.24 12.08 18.18 23.3320.63 21.2526.67 22.2922.71 SCAPO29.17 27.71 26.25 70.31 63.19 63.8413.96 18.18 22.71 25.62 22.50 27.5026.4624.38 Qwen3-1.7B-Base Base3.963.753.13 32.66 17.50 20.680.001.332.711.672.714.583.541.46 GRPO10.636.675.8346.4128.19 25.301.252.658.545.004.799.796.466.46 GSPO9.587.085.42 44.38 27.08 27.981.882.276.466.045.42 10.006.465.21 SAPO13.54 8.547.92 51.09 33.6131.993.545.687.507.718.96 11.678.758.33 CF-GRPO9.175.215.83 46.4123.75 27.082.503.418.756.045.009.586.884.38 FIPO8.757.504.79 46.25 25.28 27.081.041.897.925.426.46 10.215.005.83 SCAPO14.79 12.927.92 51.09 37.08 36.764.795.118.339.177.7116.25 11.468.33 22 Semifactual Credit-Augmented Policy Optimization F.2PASS@k RESULTS We report Pass@kon AIME and AMC, measuring the probability of obtaining at least one correct answer withinkattempts. We estimate the curves from 128 responses per problem using the standard unbiased estimator, with the same sampling and scoring settings as the main evaluation. Figure 8 shows that SCAPO achieves the highest Pass@128 at both model scales. 1248163264128 0 20 40 60 Pass@k (%) Qwen3-4B · AIME 2024–2026 1248163264128 0 20 40 60 80 100 Pass@k (%) Qwen3-4B · AMC 2023–2025 1248163264128 k 0 10 20 30 40 50 Pass@k (%) Qwen3-1.7B · AIME 2024–2026 1248163264128 k 0 20 40 60 80 Pass@k (%) Qwen3-1.7B · AMC 2023–2025 BaseGRPOGSPOSAPOCF-GRPOFIPOSCAPO Figure 8: Pass@kon the AIME 2024–2026 and AMC 2023–2025 benchmarks at both model scales. 23