Paper deep dive
CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning
Yafei Zhang, Songshuo Lu, Sicong Liao, Zhi Chen, Yaohua Tang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/4/2026, 2:19:29 AM
Summary
The paper introduces Cancellation-Aware Response Masking (CARM), a sequence-level masking technique for Large Language Model (LLM) reinforcement learning. CARM addresses the issue of 'cancellation' in traditional geometric-mean masking, where opposing token probability log-ratios cancel each other out, hiding significant policy drift. By taking the absolute value of token log-ratios before averaging, CARM detects bidirectional drift more effectively. Theoretical bounds are provided linking the mask threshold to token-ratio violations. Experiments on mathematical reasoning (AIME, BeyondAIME) and code generation (TACO, LiveCodeBench, HumanEval+, MBPP+) demonstrate that CARM outperforms baselines like GeoMean, TRM, and IcePop, improving mean@16 and pass@1 metrics.
Entities (15)
Relation Signals (16)
CARM → improves → pass@1
confidence 95% · increases average pass@1 across four code benchmarks by 2.88 points over the strongest evaluated baseline
CARM → improves → mean@16
confidence 95% · CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to 3.13 percentage points over geometric-mean masking
CARM → isappliedto → Qwen3.5-9B
confidence 95% · Experiments on mathematical reasoning... Qwen3.5-9B... CARM (ours)
CARM → isappliedto → Qwen3.5-4b
confidence 95% · Experiments on mathematical reasoning... Qwen3.5-4B... CARM (ours)
CARM → isevaluatedon → AIME 2024/2025/2026
confidence 95% · Experiments on mathematical reasoning... averaged over AIME 2024/2025/2026
CARM → outperforms → GeoMean
confidence 95% · CARM improves mean@16... by up to 3.13 percentage points over geometric-mean masking
CARM → prevents → cancellation
confidence 95% · CARM... prevents opposing probability changes from canceling.
GeoMean → suffersfrom → cancellation
confidence 95% · Its signed log-ratios can cancel across positions, concealing substantial bidirectional policy drift.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization. A common masking rule uses the length-normalized geometric mean of sampled token probability ratios. Its signed log-ratios can cancel across positions, concealing substantial bidirectional policy drift. We propose \emph{Cancellation-Aware Response Masking} (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging, preventing opposing probability changes from canceling. We prove that accepted responses satisfy a joint bound on the fraction of sampled-token ratios outside a prescribed band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show that CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to $3.13$ percentage points over geometric-mean masking, and increases average pass@1 across four code benchmarks by $2.88$ points over the strongest evaluated baseline. These findings support CARM as a theoretically grounded and effective method for response-level off-policy control in LLM reinforcement learning.
Tags
Links
- Source: https://arxiv.org/abs/2610.02039v1
- Canonical: https://arxiv.org/abs/2610.02039v1
Trouble viewing inline? Open PDF directly →
Full Text
81,054 characters extracted from source content.
Expand or collapse full text
CARM: CANCELLATION-AWARE RESPONSE MASKING FOR LLM REINFORCEMENT LEARNING Yafei ZhangSongshuo LuSicong LiaoZhi ChenYaohua Tang Moore Threads AI ABSTRACT Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathemati- cal reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization. A common masking rule uses the length-normalized geometric mean of sampled token probability ratios. Its signed log-ratios can cancel across positions, concealing substantial bidirec- tional policy drift. We propose Cancellation-Aware Response Masking (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging, preventing opposing probability changes from canceling. We prove that accepted responses satisfy a joint bound on the fraction of sampled-token ra- tios outside a prescribed band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show that CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to 3.13 percentage points over geometric-mean masking, and increases average pass@1 across four code benchmarks by 2.88 points over the strongest evaluated baseline. These findings support CARM as a theoretically grounded and effective method for response-level off-policy control in LLM reinforcement learning. 1INTRODUCTION Reinforcement learning (RL) is widely used to improve the reasoning ability of large language mod- els (LLMs), building on advances in preference-based alignment and execution-feedback learning (Ouyang et al., 2022; Le et al., 2022). In an ideal on-policy setting, responses are generated by the same policy that is being optimized. In practical systems, however, several effects introduce pol- icy mismatch. Mini-batch optimization makes stored responses stale as the policy changes, while actor–learner latency can delay their use. High-throughput rollout engines may also differ from the training backend in batching, numerical precision, and kernel implementations (Yao et al., 2025; Sheng et al., 2025). Consequently, the policy being updated can assign substantially different prob- abilities to sampled tokens than the rollout policy that generated them. Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) regulate pol- icy updates through clipping in their token-level surrogate losses (Schulman et al., 2017; Shao et al., 2024; Guo et al., 2025). Clipping changes how individual tokens contribute to the loss, but does not keep every current-to-rollout probability ratio within a prescribed interval. It also does not decide whether an entire response should participate in the update. Sequence-level masking complements clipping by making this decision from a score that summarizes mismatch across the response. Trust Region Masking (TRM), for example, motivates such filtering through an analysis of how policy mismatch affects optimization over long autoregressive sequences (Li et al., 2026). A simple response-level criterion is geometric-mean masking: it takes the geometric mean of the token probability ratios and retains responses when this aggregate is close to one. We refer to this baseline as GeoMean. In log space, this aggregation corresponds to averaging signed token log- ratios, as used in DeepSeek-V3.2’s off-policy sequence mask (Liu et al., 2025a). The problem is that positive and negative log-ratios can cancel. For example, token ratios of 10 and 0.1 have a geometric mean of one, although each sampled-token probability has changed tenfold. More generally, a 1 arXiv:2610.02039v1 [cs.LG] 1 Oct 2026 02000400060008000 Token position −0.04 −0.02 0.00 0.02 0.04 Mean log-ratio drift (per 64-token block) Full-response bidirectional drift log r t > 0log r t < 0 02000400060008000 Prefix length 0.00 0.01 0.02 0.03 Pre fi x sequence deviation Prefix-averaged log-ratio deviations CARM 0.0245 GeoMean 0.0025 CARM: s= 1.0248 > 1.02 REJECT GeoMean: s= 1.0025 ≤ 1.005 ACCEPT it 'sa math problem context( likely Chinese Ga ok aoor similar competition style ),"m/n format " withintegers usually accepts negative numerators. Also , usually $ m,n $ arecopr imeand$ n>0 $. Let's checkif$f (x )$ could be complex ?No, log base 2 usuallyimpliesreal domain . Domain$ 0\ le x<7 $.$ 9-x $ispositive . So $f (x )$is real . Real response tokens 6784-6886 Highlighted window: s GeoMean = 1.0021, s CARM = 1.0692 log r t > 0log r t < 0 Figure 1: A real unfiltered rollout that GeoMean accepts and CARM rejects. This 8,619-token response has scores s GeoMean = 1.0025 ≤ 1.005 and s CARM = 1.0248 > 1.02. Top: positive and negative log-ratio mass and prefix deviations. Bottom: a 103-token window; red tokens have logr t > 0, green tokens have logr t < 0, and color intensity encodes magnitude. response can appear well aligned under this score because its probability changes balance out, rather than because those changes are small. We propose Cancellation-Aware Response Masking (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging. This prevents opposing probability changes from canceling while treating reciprocal changes symmetrically. Figure 1 illustrates the distinction on a real 8,619-token response: substantial positive and negative log-ratio contributions largely cancel under GeoMean, whereas CARM retains their magnitudes. CARM changes which responses contribute to optimization while retaining the token-level PPO/GRPO surrogate. Beyond preventing cancellation, CARM gives the response-level threshold a direct token-level inter- pretation. We prove that every accepted response satisfies a joint bound on the fraction of sampled- token ratios outside a prescribed, possibly asymmetric band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show improvements over token- and sequence-level baselines. Threshold analyses further show that similar filtering rates can yield different accuracies, while filtering more responses is not consistently better. Our contributions are: • We formalize cancellation in signed-log sequence masking: the aggregate score can indi- cate perfect apparent alignment even when every sampled token ratio violates a prescribed band. • We propose CARM, a sequence-level mask that prevents cancellation by taking absolute values before averaging token log-ratios. We derive a joint bound on the frequency and severity of ratio-band violations in accepted responses, including asymmetric bands, giving the filtering threshold a token-level interpretation. • Experiments on mathematical reasoning and code generation show improvements over sequence- and token-level baselines. Paired-response diagnostics and threshold analyses 2 characterize the selection behavior: higher accuracy is observed at comparable average filtering rates, and accuracy is not monotone in the masked fraction. 2RELATED WORK Policy optimization for LLM post-training. Trust Region Policy Optimization (TRPO) constrains Kullback–Leibler (KL) divergence, while PPO uses a clipped-ratio surrogate (Schulman et al., 2015; 2017). GRPO replaces a learned critic with within-group relative advantages (Shao et al., 2024). An- swer correctness and program execution provide reward signals for improving mathematical reason- ing and code generation (Le et al., 2022; Dou et al., 2024; Guo et al., 2025; Yang et al., 2025; Zeng et al., 2026), complementing inference-time reasoning and program-aided prompting (Wei et al., 2022; Chen et al., 2023; Gao et al., 2023). Decoupled Clip and Dynamic sAmpling Policy Opti- mization (DAPO) refines clipping and sampling (Yu et al., 2025); related work revisits advantage normalization, sequence-level objectives, and reward–policy learning (Hu et al., 2025; Xie et al., 2025; Lu et al., 2026a). These choices determine how sampled responses contribute to optimization. CARM addresses the complementary question of whether a sampled response should contribute to the current policy update. Off-policy and token-level trust-region control. This selection problem arises even in nominally online training: actor–learner delay, repeated updates, and rollout–training engine differences sepa- rate the policy that generates responses from the policy being optimized (Espeholt et al., 2018; Yao et al., 2025; Sheng et al., 2025). Importance weighting and token-level filtering control individual update contributions (Espeholt et al., 2018; Shen et al., 2025). Adaptive clipping balances advantage contributions and mitigates entropy loss (Xi et al., 2026), while divergence-based controls constrain update size (Qi et al., 2026; Yao et al., 2026). These mechanisms regulate the update within a response. CARM instead decides whether to retain the complete response, separating response se- lection from the surrogate applied to retained tokens. Sequence-level correction and masking. Response-level selection must summarize mismatch across an entire sampled path. Length-normalized geometric aggregation removes the direct length dependence of multiplying token ratios (Liu et al., 2025b); it also underlies sequence-level clipping in Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025a). DeepSeek-V3.2 explicitly uses off-policy sequence masking based on the mean signed log-ratio, equivalently the logarithm of a length-normalized geometric-mean ratio, to stabilize GRPO training (Liu et al., 2025a). Its rule applies a one-sided threshold to negative-advantage responses. Our GeoMean baseline uses the same signed-log aggregation with reciprocal-band acceptance (Eq. 5); we examine advantage con- ditioning separately in Section 5.2. TRM takes a worst-position view, rejecting a response when its maximum token-level divergence violates a long-horizon trust region (Li et al., 2026). CARM tar- gets a different limitation: geometric aggregation averages signed log-ratios, so opposing shifts can cancel. Averaging their absolute values measures bidirectional drift rather than net displacement. The resulting mask changes which responses contribute to the update while retaining the token-level policy surrogate. 3METHOD 3.1PROBLEM SETUP Let x be a prompt and let y i = (y i,1 ,...,y i,T i ) be the i-th sampled response. We use response and sequence interchangeably for these generated tokens. We consider nonempty responses (T i ≥ 1) with finite token log-probabilities, so 0 < r i,t < ∞. The token-level ratio of the current policy to the rollout policy is r i,t (θ) = π θ (y i,t | x,y i,<t ) π rollout (y i,t | x,y i,<t ) .(1) For a sequence-level advantage ˆ A i , the conventional asymmetric PPO/GRPO token surrogate is ℓ i,t (θ) = min r i,t (θ) ˆ A i , clip r i,t (θ), 1− ε low , 1 + ε high ˆ A i .(2) 3 For a binary response mask M i ∈0, 1, averaging the retained token losses gives L masked (θ) =− P i M i P T i t=1 ℓ i,t (θ) P i M i T i .(3) The policy-loss contribution is defined as zero if no response is retained. CARM specifies the detached mask independently of loss normalization; Appendix B states the convention used in the experiments. 3.2GEOMEAN MASKING AND RATIO CANCELLATION Let G(y i ) = exp( 1 T i P t logr i,t ) denote the geometric-mean token ratio. We define its log-space deviation and reciprocal-symmetric score as d GeoMean (y i ) = 1 T i T i X t=1 logr i,t , s GeoMean (y i ) = exp(d GeoMean (y i )).(4) The response is retained when M GeoMean i = 1[s GeoMean (y i )≤ τ GeoMean ].(5) For τ GeoMean ≥ 1, this requires the geometric-mean ratio to lie in the reciprocal interval [1/τ GeoMean ,τ GeoMean ]. Equivalently, d GeoMean = | logG| and s GeoMean = max(G,G −1 ). We use GeoMean as a descriptive shorthand for this two-sided rule. It shares signed-log aggregation with DeepSeek-V3.2, whose mask instead uses a one-sided threshold on negative-advantage re- sponses (Liu et al., 2025a). Proposition 1 (Cancellation under signed aggregation). For any B > 1 and any 1 < c < B, there exists an even-length response for which every token ratio lies outside [1/c,c], while s GeoMean (y) = 1. Proof. Let the ratios alternate between B and 1/B. Every token is outside [1/c,c], but the log-ratios alternate between logB and− logB. Their sum is zero, so d GeoMean (y) = 0 and s GeoMean (y) = 1. Proposition 1 does not claim that the exact trajectory likelihood ratio is incorrect. Indeed, π θ (y | x)/π rollout (y | x) = Q t r t . It shows that thresholding a signed aggregate does not bound the magnitudes of the sampled-token log-ratios when used as a response-level filter. Figure 1 shows this cancellation in an actual unfiltered rollout. 3.3CANCELLATION-AWARE RESPONSE MASKING CARM removes direction before aggregation: d CARM (y i ) = 1 T i T i X t=1 |logr i,t |, s CARM (y i ) = exp(d CARM (y i )).(6) Equivalently, each token contributes the reciprocal-invariant mismatch max(r i,t ,r −1 i,t ) in ratio space, since| log(1/r i,t )| =| logr i,t |. The CARM mask is M CARM i = 1[s CARM (y i )≤ τ CARM ].(7) During optimization, the response mask and token-level surrogate use the same current-policy for- ward pass and the stored rollout log-probabilities, before the optimizer update. The binary gate is treated as a constant during backpropagation; a rejected response contributes no policy-loss gra- dient in that update. Algorithm 1 and Appendix B give the computation schedule and numerical conventions. The two scores measure different properties: GeoMean measures the magnitude of the average signed log-ratio, whereas CARM measures the average log-ratio magnitude. Mean-absolute ag- gregation combines contributions linearly, preventing cancellation without selecting only the most extreme token or squaring each deviation. Figure 2 contrasts this rule with GeoMean. 4 Token ratios Keep Reject example: 1<τ< √ 12 Sequence gate Changes only which sequences are retained Example: [4, 1/4, 3, 1/3] GeoMean d GeoMean = ∣ ∣ ∣ ∣ ∣ 1 T ∑ t logr t ∣ ∣ ∣ ∣ ∣ d GeoMean = 0,signs cancel s GeoMean = 1 M GeoMean = 1 CARM d CARM = 1 T ∑ t |logr t | d CARM = 1 2 log 12,magnitudes add s CARM = √ 12 M CARM = 0 Figure 2: CARM detects bidirectional token shifts that geometric-mean masking can hide. GeoMean averages signed log-ratios before taking an absolute value; CARM averages their magnitudes. The response-level gate complements the token-level PPO/GRPO surrogate. A real disagreement case from an unfiltered rollout. We replay both masks on an unfiltered batch of 1,024 responses generated by Qwen3.5-4B. With τ GeoMean = 1.005 and τ CARM = 1.02, 160 responses are accepted by both rules, 668 are rejected by both, 195 are accepted only by Ge- oMean, and one is accepted only by CARM. Figure 1 visualizes a representative GeoMean-only response whose opposing log-ratio masses cancel; Appendix C provides all 8,619 tokens. Its mean absolute log-ratio is 9.79 times the absolute signed mean, while the largest single-token contribution accounts for only 0.43% of the absolute log-ratio mass. The discrepancy is therefore not driven by one extreme token. These unequal thresholds illustrate different filtering decisions; the shared- threshold relation is examined separately below. Proposition 2 (Acceptance-set inclusion). For every response y, s CARM (y)≥ s GeoMean (y). Thus, for any shared threshold τ ≥ 1, y : s CARM (y)≤ τ⊆y : s GeoMean (y)≤ τ.(8) Proof. The triangle inequality gives 1 T X t | logr t |≥ 1 T X t logr t . Exponentiation preserves the inequality, and the accepted-set inclusion follows immediately. Figure 3 replays both statistics on the same 1,024 responses from an unfiltered rollout batch, before any sequence-level gate can alter the training trajectory. Every response obeys d CARM ≥ d GeoMean , with no numerical violations; the mean difference is 0.0319. At every shared threshold, CARM rejects at least as many responses as GeoMean, in agreement with Proposition 2. 3.4THRESHOLD INTERPRETATION Consider a token-ratio band [ℓ,u] with 0 < ℓ < 1 < u. Define the lower- and upper-side violation sets and their fractions as V − =t : r t < ℓ,p − = |V − | T ,(9) V + =t : r t > u,p + = |V + | T .(10) Let p out = p − + p + and define the log-costs of reaching the boundaries by b − = log(1/ℓ) and b + = logu. For p out > 0, define ̄ δ out (y) = X t∈V − log ℓ r t + X t∈V + log r t u |V − | +|V + | ,(11) 5 0.000.010.020.030.04 0.00 0.02 0.04 0.06 0.08 0.10 0.12 linear fit: y=2.87x+0.011 10 −3 10 −2 10 −1 Threshold excess τ−1 (log scale) 0 20 40 60 80 100 Masked sequences (%) GeoMean CARM d GeoMean = ∣ ∣ ∣ ∣ ∣ 1 T T ∑ t=1 logr t ∣ ∣ ∣ ∣ ∣ d CARM = 1 T T ∑ t =1 | log r t | Figure 3: Paired deviations (left) and rejection rates at shared thresholds (right) on 1,024 unfiltered responses. Every response satisfies d CARM ≥ d GeoMean . Green shading denotes rejection by both rules; orange shading denotes rejection only by CARM. and set ̄ δ out (y) = 0 when p out = 0. We call ̄ δ out the mean violation severity: it is the average log-distance by which an out-of-band token passes its nearest boundary. Proposition 3 (Bounds on ratio-band violations). If s CARM (y)≤ τ with τ ≥ 1, then p − b − + p + b + + p out ̄ δ out (y)≤ logτ.(12) Consequently, with b min = min(b − ,b + ), p out ≤ min 1, logτ b min + ̄ δ out (y) ≤ min 1, logτ b min .(13) Thus each response accepted by CARM has a shared budget for the fraction of violating token ratios and their mean severity. At a fixed threshold, more severe violations imply a smaller admissible fraction; the bound does not require a reciprocal-symmetric band. This is a deterministic bound on the sampled-token ratios, not on the full conditional-policy KL divergence. Proof. The acceptance condition is 1 T P t | logr t | ≤ logτ . For a lower violation, | logr t | = b − + log(ℓ/r t ); for an upper violation, | logr t | = b + + log(r t /u). Sum these identities over violating tokens, drop the non-negative in-band contribution, and divide by T to obtain Equation 12. Replacing both boundary costs by b min gives p out (b min + ̄ δ out ) ≤ logτ . Division by the positive denominator and p out ≤ 1 give the fraction bound. Appendix A provides the full derivation. Corollary 1 (Reciprocal-symmetric band). For [1/c,c] with c > 1, define V c = t : r t < 1/c or r t > c, p c = |V c |/T , and let ̄ δ c be the mean of | logr t |− logc over t ∈ V c (zero when p c = 0). If s CARM (y)≤ τ , then p c logc + ̄ δ c ≤ logτ.(14) In particular, p c ≤ logτ/ logc. Proof. Set ℓ = 1/c and u = c in Proposition 3; then b − = b + = logc and ̄ δ out = ̄ δ c . The experimental PPO/GRPO clip parameters ε low = 0.2 and ε high = 0.28 correspond to the numerical band [0.8, 1.28]. For this band, the lower and upper boundary costs are − log 0.8 ≈ 0.2231 and log 1.28 ≈ 0.2469. Consequently, Equation 13 gives p out ≤ 8.88% at τ = 1.02 and p out ≤ 17.58% at τ = 1.04 for any accepted sequence. The PPO clipping interval serves only as a reference band for interpreting violations; CARM does not impose it as a hard per-token constraint. 6 4EXPERIMENTS 4.1EXPERIMENTAL SETUP Experiments use verl (Sheng et al., 2025) and GRPO. We train Qwen3.5-4B and Qwen3.5-9B (Qwen Team, 2026) on DAPO Math (Yu et al., 2025) and evaluate AIME 2024/2025/2026 (Math- ematical Association of America, 2026) and BeyondAIME (Chen et al., 2025) with 16 responses per problem. For code generation, Qwen3.5-9B is trained on the TACO training split (Li et al., 2023) with executable rewards. Evaluation uses the held-out TACO test split, LiveCodeBench-v6 (Jain et al., 2025), and the EvalPlus suites HumanEval+ and MBPP+ (Liu et al., 2023). Metrics are obtained using three random seeds for both tasks. Math-Verify checks mathematical answers (Ky- dlí ˇ cek, 2025); SandboxFusion executes and judges generated code (Cheng et al., 2025). Baselines are Null, IcePop, TRM, and GeoMean. Null retains the same GRPO procedure, including token- level clipping and advantage normalization, but applies no additional off-policy response mask. Appendix D and Table 3 give the datasets, prompts, and configuration. 4.2MATHEMATICAL REASONING Table 1 compares mathematical reasoning across AIME 2024/2025/2026 and BeyondAIME. CARM achieves the highest four-dataset average for both Qwen3.5-4B and Qwen3.5-9B. Its three-year AIME average also exceeds those of Null, IcePop, TRM, and GeoMean: relative to GeoMean, it rises from 69.17% to 70.42% on Qwen3.5-4B and from 74.03% to 77.50% on Qwen3.5-9B. The latter improvement holds separately on all three AIME years. For Qwen3.5-4B, CARM improves AIME 2024, ties on AIME 2025, and is slightly lower on AIME 2026, while improving the aggre- gate. Table 1: Mathematical-reasoning mean@16 across methods (%, higher is better). Average is the mean of AIME 2024, AIME 2025, AIME 2026, and BeyondAIME. Bold and underline denote the best and second-best values within each model block. ModelMethodAIME 2024 AIME 2025 AIME 2026 BeyondAIME Average Qwen3.5-4B Null49.3848.3354.7922.2543.69 IcePop47.0840.8340.4219.3136.91 TRM66.6755.2165.0032.3154.80 GeoMean72.5064.38 70.6238.8861.59 CARM (ours) 77.08 64.3869.7942.56 63.45 Qwen3.5-9B Null76.6766.8867.7145.2564.13 IcePop70.4259.5866.6734.9457.90 TRM78.3367.0876.4645.8166.92 GeoMean80.4266.6775.0047.0667.29 CARM (ours) 82.08 71.04 79.38 49.19 70.42 On BeyondAIME, CARM raises mean@16 from 38.88% to 42.56% on Qwen3.5-4B and from 47.06% to 49.19% on Qwen3.5-9B relative to GeoMean. The gains therefore extend beyond the annual AIME benchmarks. Across all four datasets, the Average increases from 61.59% to 63.45% on Qwen3.5-4B and from 67.29% to 70.42% on Qwen3.5-9B. For the larger model, the aggregate lead accompanies an improvement on every benchmark; for the smaller model, it reflects an overall gain with a mixed year-by-year AIME pattern. Relative to TRM, CARM improves the three-year AIME average by 8.13 points on Qwen3.5-4B and 3.54 points on Qwen3.5-9B despite TRM’s higher rejection rates (Section 5.1). Relative to IcePop, CARM improves the four-dataset Average by 26.54 points on Qwen3.5-4B and 12.52 points on Qwen3.5-9B. 7 4.3CODE GENERATION Table 2 reports pass@1 for Qwen3.5-9B trained on the TACO training split. Testing on TACO measures performance on held-out problems from the same dataset, while LiveCodeBench-v6, Hu- manEval+, and MBPP+ evaluate the resulting policy on three additional benchmarks. Table 2: Evaluation pass@1 across code benchmarks (%, higher is better). TACO denotes the held- out test split, separate from the training split. Average is the mean of the four datasets. Bold and underline denote the best and second-best values. MethodTACO LiveCodeBench-v6 HumanEval+ MBPP+ Average Null34.5038.4685.5775.2258.44 IcePop45.8743.3692.6879.8165.43 TRM36.6044.9787.8076.7261.52 GeoMean46.03 44.5092.2878.3165.28 CARM (ours) 46.70 51.20 94.11 81.22 68.31 CARM improves average pass@1 by 3.03 points over GeoMean and 2.88 points over IcePop, the strongest baseline on average. The gains are not uniform: relative to GeoMean, the increase is modest on TACO, from 46.03% to 46.70%, and largest on LiveCodeBench-v6, from 44.50% to 51.20%. On HumanEval+ and MBPP+, CARM reaches 94.11% and 81.22%, also exceeding Ice- Pop, the strongest baseline on these suites. Thus, the improvement is not confined to held-out TACO problems. 5ANALYSIS We examine whether the gains are explained by filtering more responses, whether advantage sign should override the gate, and whether improvements persist across different sampling criteria. 5.1UPDATE REDUNDANCY AND EFFECTIVE FILTERING 010203040506070 Sequences masked (%) 65 70 75 80 AIME mean@16 (%) Null TRM GeoMean CARM (ours) Figure 4: Sequence filtering versus AIME ac- curacy. Each point is a masking configuration. AIME mean@16 averages the 2024/2025/2026 benchmarks. We use update redundancy to describe improved validation accuracy despite omitting response- level updates, without implying that rejected re- sponses lack useful information. GeoMean and CARM mask an average of 55.65% and 54.52% of sequences during Qwen3.5-4B training, yet improve three-year AIME mean@16 over Null by 18.33 and 19.58 points. For Qwen3.5-9B, the corresponding frac- tions are 52.75% and 52.32%, with gains of 3.61 and 7.08 points. Filtering persists throughout training (Appendix Figure 9). Filtering already- generated responses complements selective roll- out use and data-efficient reasoning (Yu et al., 2025; Sun et al., 2025; Lu et al., 2026b; Wang et al., 2025; Zheng et al., 2025b). Figure 4 compares AIME accuracy with mean sequence-masked fraction.On Qwen3.5-9B, CARM outperforms GeoMean by 3.47 points while their masked fractions differ by only 0.43 points. Accuracy is not monotone in the masked fraction, and TRM rejects more responses with- out outperforming CARM. 8 Paired-score differences (Figure 3) show that rejection rates alone do not characterize retained tra- jectories. Tightening CARM’s frequency–severity budget (Proposition 3) limits sampled-token mis- match but removes update contributions, consistent with the observed non-monotonicity. 5.2ADVANTAGE-CONDITIONAL MASKING ABLATION DeepSeek-V3.2 masks only negative-advantage responses (Liu et al., 2025a). We test this condi- tioning on Qwen3.5-9B with CARM’s score and threshold fixed within each task. Standard mask- ing reaches 75.63% AIME mean@16 (2024/2025/2026) versus 60.28% for negative-only masking; LiveCodeBench-v6 scores are 50.58% versus 42.22% (Figure 5). Even a correctly rewarded response can be stale if the current policy assigns substantially different probabilities to its tokens. PPO clips positive-advantage terms above the upper ratio bound, but other token updates can remain active (Schulman et al., 2017); it does not enforce a response-level trust region (Li et al., 2026). These results suggest that response-level mismatch checks can remain useful even for positive-advantage responses. 5.3SAMPLING CONSISTENCY We assess sampling consistency using bootstrapped best-of-k and worst-of-k: the former requires at least one correct response among k, while the latter requires all k to be correct. Worst-of-k measures sampling repeatability, not distribution-shift or adversarial robustness. Appendix G defines the estimators and reports results for k ∈1, 2, 4, 8, 16. For Qwen3.5-9B, Figure 6 averages results over AIME 2024/2025/2026.CARM improves mean@16, best@16, and worst@16 over GeoMean by 3.47, 2.74, and 2.78 points, respectively; the corresponding gains over TRM are 3.54, 0.30, and 4.61 points. CARM also achieves the highest best-of-k and worst-of-k values throughout the displayed range. 020406080100 Training step 0 20 40 60 80 Accuracy (%) Advantage conditioning Math: standard Math: negative-only Code: standard Code: negative-only Figure 5:Standard (solid) ver- sus negative-only (dashed) CARM masking. 124816 Samples k 70 75 80 85 90 95 Accuracy (%) Best-of-k Null TRM GeoMean CARM (ours) 124816 Samples k 40 50 60 70 80 Accuracy (%) Worst-of-k Figure 6: Bootstrapped best-of-k (left) and worst-of-k (right) accuracy on AIME. 6CONCLUSION We proposed CARM, a response-level off-policy mask that averages absolute token log-ratios to prevent opposing probability changes from canceling. Our analysis shows that signed aggregation can accept responses even when every sampled-token ratio lies outside a reciprocal-symmetric band. In contrast, CARMbounds the frequency and severity of ratio-band violations through a joint budget for each accepted response, giving the threshold a token-level interpretation. Experiments show gains in mathematical reasoning and code generation over the evaluated baselines, together with improved sampling consistency. These findings support CARM as a principled response-selection criterion that complements reward-directed PPO/GRPO optimization. 9 REFERENCES Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732. Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, et al. Seed1.5- Thinking: Advancing superb reasoning models with reinforcement learning, 2025. URL https: //arxiv.org/abs/2504.13914. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023. URL https://openreview.net/forum?id= YfZ4ZPt8zd. Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, et al. FullStack Bench: Evaluating LLMs as full stack coders, 2025. URL https://arxiv.org/ abs/2412.00535. Shihan Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. StepCoder: Improving code generation with reinforcement learning from com- piler feedback. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 4571–4585. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.251. URL https://aclanthology.org/2024. acl-long.251/. Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IM- PALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, 2018. URL https: //proceedings.mlr.press/v80/espeholt18a.html. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided language models. In Proceedings of the 40th In- ternational Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, p. 10764–10799. PMLR, 2023. URL https://proceedings.mlr.press/ v202/gao23f.html. Daya Guo, Dejian Yang, Haowei Zhang, et al.DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning.Nature, 645(8081):633–638, 2025.doi: 10.1038/ s41586-025-09422-z. URL https://doi.org/10.1038/s41586-025-09422-z. Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. REINFORCE++: Stabilizing critic-free policy optimization with global advantage normalization, 2025. URL https://arxiv.org/abs/ 2501.03262. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. Hynek Kydlí ˇ cek.Math-Verify: Math verification library.Software, 2025.URL https: //github.com/huggingface/Math-Verify. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. CodeRL: Mastering code generation through pretrained models and deep reinforcement learn- ing. In Advances in Neural Information Processing Systems, volume 35, p. 21314–21328, 2022. doi: 10.52202/068431-1549. Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. TACO: Topics in algorithmic code generation dataset, 2023. URL https://arxiv. org/abs/2312.14852. 10 Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, and Baoxiang Wang. Trust region mask- ing for long-horizon LLM reinforcement learning. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id= dPaBOu0Xwk. Aixin Liu, Aoxue Mei, Bangcai Lin, et al. DeepSeek-V3.2: Pushing the frontier of open large language models, 2025a. URL https://arxiv.org/abs/2512.02556. Jiacai Liu, Yingru Li, Yuqian Fu, Jiawei Wang, Qian Liu, and Zhuo Jiang. When speed kills sta- bility: Demystifying RL collapse from the training–inference mismatch, September 2025b. URL https://richardli.xyz/rl-collapse. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, 2023. URL https://openreview. net/forum?id=1qvx610Cu7. Songshuo Lu, Hua Wang, Zhi Chen, and Yaohua Tang. URPO: A unified reward & policy op- timization framework for large language models.Proceedings of the AAAI Conference on Artificial Intelligence, 40(38):32329–32337, 2026a. doi: 10.1609/aaai.v40i38.40507. URL https://ojs.aaai.org/index.php/AAAI/article/view/40507. Xiaodong Lu, Xiaohan Wang, Jiajun Chai, Guojun Yin, Wei Lin, Zhijun Chen, Yu Luo, Fuzhen Zhuang, Yikun Ban, and Deqing Wang. Contextual rollout bandits for reinforcement learning with verifiable rewards, 2026b. URL https://arxiv.org/abs/2602.08499. MathematicalAssociationofAmerica.MAAinvitationalcompetitions:Ameri- caninvitationalmathematicsexamination,2026.URL https://maa.org/ maa-invitational-competitions/. Accessed September 2026. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, et al. Training language models to follow instructions with human feedback. In Advances in Neural In- formation Processing Systems, volume 35, p. 27730–27744, 2022. doi: 10.52202/068431-2011. Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, and Wee Sun Lee. Re- thinking the trust region in LLM reinforcement learning. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id= f4g8zblbwp. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen. ai/blog?id=qwen3.5. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, 2015. URL https://proceedings.mlr.press/v37/schulman15.html. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402. 03300. Anqi Shen, Baihui Li, Bin Hu, et al. Every step evolves: Scaling reinforcement learning for trillion- scale thinking model, 2025. URL https://arxiv.org/abs/2510.18855. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. In Proceed- ings of the Twentieth European Conference on Computer Systems, p. 1279–1297, 2025. doi: 10.1145/3689031.3696075. URL https://doi.org/10.1145/3689031.3696075. 11 Yan Sun, Jia Guo, Stanley Kok, Zihao Wang, Zujie Wen, and Zhiqiang Zhang. Efficient rein- forcement learning for large language models with intrinsic exploration, 2025. URL https: //arxiv.org/abs/2511.00794. Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training example. In Advances in Neural Information Processing Systems, volume 38, p. 122721–122764, 2025. doi: 10.52202/085713-4093. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, p. 24824–24837, 2022. doi: 10.52202/068431-1800. Zhiheng Xi, Xin Guo, Yang Nan, et al. BAPO: Stabilizing off-policy reinforcement learning for LLMs via balanced policy optimization with adaptive clipping. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/pdf? id=jIeJJqG7dz. Zhengpeng Xie, Qiang Zhang, Fan Yang, Marco Hutter, and Renjing Xu. Simple policy op- timization. In Proceedings of the 42nd International Conference on Machine Learning, vol- ume 267 of Proceedings of Machine Learning Research, p. 68813–68824. PMLR, 2025. URL https://proceedings.mlr.press/v267/xie25m.html. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, et al. Qwen3 techni- cal report, 2025. URL https://arxiv.org/abs/2505.09388. Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient RL framework secretly brings you off-policy RL training, August 2025. URL https: //fengyao.notion.site/off-policy-rl. Jiarui Yao, Xiangxin Zhou, Penghui Qi, Wee Sun Lee, Liefeng Bo, and Tianyu Pang. Rethinking the divergence regularization in LLM RL, 2026. URL https://arxiv.org/abs/2606. 09821. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, et al. DAPO: An open-source LLM reinforcement learning system at scale. In Advances in Neural Information Processing Systems, volume 38, p. 113222–113244, 2025. doi: 10.52202/ 085713-3775. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, et al. GLM-5: from vibe coding to agentic engineering, 2026. URL https://arxiv.org/abs/2602.15763. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy op- timization, 2025a. URL https://arxiv.org/abs/2507.18071. Haizhong Zheng, Yang Zhou, Brian Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts. In Advances in Neural Information Processing Systems, volume 38, p. 124321–124346, 2025b. doi: 10.52202/085713-4143. 12 ADETAILED PROOFS A.1PROOF OF PROPOSITION 1 Step 1: construct an even-length response. Fix arbitrary constants B > 1 and 1 < c < B. Choose an arbitrary positive integer m and let the response length be T = 2m. For every pair j ∈1,...,m, define r 2j−1 = B, r 2j = 1 B .(15) Thus the response contains exactly m ratios equal to B and exactly m ratios equal to 1/B. These ratios can arise from valid conditional policies: at each odd position, assign the sampled token probabilities 1/(1 + B) under the rollout policy and B/(1 + B) under the current policy, and swap them at each even position. Assign the remaining probability to a second token. Both policies are normalized, all probabilities are positive, and their sampled-token ratios are exactly those in Equation 15. Step 2: verify that every ratio violates the band. For each odd position, r 2j−1 = B > c, so it lies strictly above the upper endpoint of [1/c,c]. For each even position, 1 < c < B and all quantities are positive. Taking reciprocals reverses the inequalities and gives 1 > 1 c > 1 B .(16) Hence r 2j = 1/B < 1/c, so it lies strictly below the lower endpoint. Therefore every one of the T token ratios is outside [1/c,c]. Step 3: evaluate the GeoMean statistic. The logarithm of a reciprocal satisfies log(1/B) = − logB. Grouping the signed log-ratios into the m adjacent pairs in Equation 15 yields T X t=1 logr t = m X j=1 (logB + log(1/B))(17) = m X j=1 (logB− logB)(18) = m X j=1 0(19) = 0.(20) Substitution into Equation 4 gives d GeoMean (y) = 1 T T X t=1 logr t (21) = 0 T (22) = 0,(23) and therefore s GeoMean (y) = exp(d GeoMean (y)) = exp(0) = 1.(24) This proves the proposition: every token can violate the prescribed reciprocal band even though GeoMean reports its minimum possible score. 13 Step 4: evaluate the CARM statistic for comparison. Because B > 1, logB > 0, and hence | logB| =| log(1/B)| = logB. Consequently, d CARM (y) = 1 T T X t=1 | logr t |(25) = 1 2m m X j=1 (| logB| +| log(1/B)|)(26) = 1 2m m X j=1 2 logB(27) = logB,(28) so s CARM (y) = exp(logB) = B. Thus any GeoMean threshold at least one accepts this construc- tion, whereas CARM rejects it whenever τ CARM < B. A.2PROOF OF PROPOSITION 2 Step 1: move to log-ratio coordinates. For a fixed response of positive length T , set z t = logr t ∈ R. For T = 1, both scores are identical. For T ≥ 2, the triangle inequality applied repeatedly to the finite sum gives T X t=1 z t =|z 1 + (z 2 +· + z T )|(29) ≤|z 1 | +|z 2 +· + z T |(30) ≤·≤ T X t=1 |z t |.(31) Step 2: normalize by response length. Since T > 0, division by T preserves the direction of the inequality: 1 T X t z t ≤ 1 T X t |z t |.(32) Using z t = logr t and Equations 4 and 6, the left-hand side is exactly d GeoMean (y) and the right- hand side is exactly d CARM (y). Therefore d GeoMean (y)≤ d CARM (y).(33) Step 3: return to ratio space. The exponential function is strictly increasing onR. Applying it to both sides gives exp(d GeoMean (y))≤ exp(d CARM (y)),(34) s GeoMean (y)≤ s CARM (y).(35) This proves the pointwise score inequality. Step 4: prove accepted-set inclusion. Take an arbitrary response y ∈ y : s CARM (y) ≤ τ. By membership in this set, s CARM (y) ≤ τ . Combining this fact with the pointwise result from Step 3 gives s GeoMean (y)≤ s CARM (y)≤ τ.(36) Thus y ∈ y : s GeoMean (y) ≤ τ. Since the argument holds for every element of the CARM accepted set, Equation 8 follows. Equality condition. For real scalars, equality in the triangle inequality holds exactly when all nonzero z t have the same sign. Accordingly, s GeoMean = s CARM when all nonzero token log-ratios are nonnegative or all are nonpositive. If at least one z t > 0 and at least one z t ′ < 0, the inequality is strict. 14 A.3PROOF OF PROPOSITION 3 Step 1: convert acceptance into an additive budget. By definition, s CARM (y) = exp 1 T T X t=1 | logr t | ! .(37) The acceptance assumption s CARM (y)≤ τ and monotonicity of log(·) imply logs CARM (y)≤ logτ,(38) 1 T T X t=1 | logr t |≤ logτ.(39) Multiplying by the positive length T gives the total absolute-log budget T X t=1 | logr t |≤ T logτ.(40) Step 2: partition positions and define the boundary costs. LetV 0 = 1,...,T\ (V − ∪V + ). Because ℓ < 1 < u, the setsV − ,V 0 , andV + are disjoint and cover all token positions. Define b − = log 1 ℓ =− logℓ, b + = logu.(41) The assumptions 0 < ℓ < 1 < u imply b − > 0 and b + > 0. Step 3: decompose a lower-side violation exactly. Take t ∈ V − . Then 0 < r t < ℓ < 1, so | logr t | =− logr t . Inserting and subtracting logℓ gives | logr t | =− logr t (42) =− logℓ + logℓ− logr t (43) = b − + log ℓ r t .(44) Since ℓ/r t > 1, the final term is positive. It approaches zero at the lower boundary and grows with the token’s log-distance below that boundary. Step 4: decompose an upper-side violation exactly. Take t ∈ V + . Then r t > u > 1, so | logr t | = logr t . Inserting and subtracting logu gives | logr t | = logr t (45) = logu + logr t − logu(46) = b + + log r t u .(47) Because r t /u > 1, this final term is likewise a positive log-distance past the upper boundary. Step 5: sum the exact decompositions. Partition the total absolute-log drift and substitute Equa- tions 44 and 47 term by term: T X t=1 | logr t | =|V − |b − +|V + |b + (48) + X t∈V − log ℓ r t + X t∈V + log r t u (49) + X t∈V 0 | logr t |.(50) 15 Step 6: identify the conditional mean log-distance. If p out > 0, Equation 11 implies X t∈V − log ℓ r t + X t∈V + log r t u (51) = |V − | +|V + | ̄ δ out (y).(52) Divide by T and use (|V − | +|V + |)/T = p out : 1 T X t∈V − log ℓ r t + X t∈V + log r t u = p out ̄ δ out (y).(53) If p out = 0, both sums are empty; our zero convention makes the same identity valid. Step 7: obtain the master budget. The in-band sum in Equation 50 is nonnegative. Drop it, apply Equation 40, divide by T , and use Equation 53: p − b − + p + b + + p out ̄ δ out (y)≤ logτ.(54) This is Equation 12. Step 8: bound the total violation fraction. Let b min = min(b − ,b + ) > 0. Keeping the severity contribution in the master budget gives p out (b min + ̄ δ out ) = (p − + p + )b min + p out ̄ δ out (55) ≤ p − b − + p + b + + p out ̄ δ out (56) ≤ logτ.(57) Because b min + ̄ δ out > 0, division yields p out ≤ logτ b min + ̄ δ out ≤ logτ b min .(58) Combining each inequality with p out ≤ 1 proves Equation 13, including the case p out = 0 under the stated zero convention. Step 9: recover the frequency-only budget. Every term defining ̄ δ out is nonnegative. Dropping the distance term from the master budget therefore gives p − b − + p + b + ≤ logτ . Let b min = min(b − ,b + ) > 0. Then p out b min = (p − + p + )b min (59) ≤ p − b − + p + b + (60) ≤ logτ.(61) Division by b min gives p out ≤ logτ/b min . Independently, p out ≤ 1 because it is a fraction. Taking the smaller upper bound recovers the severity-independent consequence of Equation 13. A.4PROOF OF COROLLARY 1 Step 1: specialize the two endpoints. Set ℓ = 1/c and u = c. Because c > 1, the assumptions 0 < ℓ < 1 < u hold, and the two boundary costs are identical: b − = log(1/ℓ) = logc,(62) b + = logu = logc.(63) Step 2: combine the violation sets. The lower and upper violation sets areV − = t : r t < 1/c andV + =t : r t > c. They are disjoint because c > 1, and their union is exactlyV c . Hence p − + p + = |V c | T = p c .(64) 16 Step 3: identify the reciprocal-band log-distance. For t∈V − , log ℓ r t = log 1/c r t (65) =| logr t |− logc,(66) where the second equality uses r t < 1. For t∈V + , log r t u = logr t − logc(67) =| logr t |− logc,(68) where r t > 1. Thus the conditional mean in Equation 11 is precisely ̄ δ c . Step 4: simplify the master budget. Substitute p − + p + = p c , b − = b + = logc, and ̄ δ out = ̄ δ c into Equation 12: p − logc + p + logc + p c ̄ δ c ≤ logτ,(69) p c logc + p c ̄ δ c ≤ logτ,(70) p c (logc + ̄ δ c )≤ logτ.(71) This is Equation 14. Finally, ̄ δ c ≥ 0, so dropping p c ̄ δ c gives p c logc ≤ logτ . Since logc > 0, division by logc yields p c ≤ logτ/ logc. BIMPLEMENTATION OF CARM Algorithm 1 gives the response-level masking rule, treating the hard threshold decision as constant during backpropagation. The gate changes which response losses contribute to an update while retaining the token-level PPO/GRPO surrogate. Algorithm 1 CARM response-level off-policy masking Require: Finite current and rollout token log-probabilities logπ θ (y i,t | x i ,y i,<t ) and logπ rollout (y i,t | x i ,y i,<t ); response-token mask a i,t ∈ 0, 1 with P t a i,t ≥ 1 for each response; threshold τ CARM ≥ 1 Ensure: Response-acceptance indicators M CARM i 1: for each response y i do 2: T i ← P t a i,t 3:∆ i,t ← stopgrad(logπ θ (y i,t |·)− logπ rollout (y i,t |·)) 4: d CARM,i ← 1 T i P t a i,t |∆ i,t | 5: s CARM,i ← exp(d CARM,i ) 6: M CARM i ← 1[s CARM,i ≤ τ CARM ] 7: end for 8: Gate each per-response PPO/GRPO loss by M CARM i and apply the training implementation’s loss normalization. Mask computation schedule. We store the sampled-token log-probabilities produced during roll- out. During training, each optimizer mini-batch is divided into micro-batches; each micro-batch for- ward supplies the current log-probabilities used by both the response mask and the token surrogate. The optimizer updates the policy after gradient accumulation over the mini-batch. Threshold com- parisons produce a binary gate with no gradient through the decision, as represented by stopgrad in Algorithm 1. Masking is local to loss evaluation rather than permanent removal from the rollout batch. Our experiments use one optimization epoch per rollout batch; with multiple epochs, the gate would be recomputed whenever a response is evaluated under the updated policy. The PPO/GRPO loss implementation in verl also applies dual clipping to negative advantages. Writing ℓ i,t for Equation 2, its surrogate is ℓ dual i,t = ( maxℓ i,t ,c dual ˆ A i , ˆ A i < 0, ℓ i,t , ˆ A i ≥ 0, c dual = 3.(72) The sequence mask multiplies this surrogate without changing its clipping rule or normalization. 17 Numerical convention. The theoretical scores use exact log-ratios. The implementation uses verl’s numerical guard ez t = clip(logr t ,−20, 20) when computing the mask and token surro- gate. If the guard is inactive, the implemented and theoretical scores coincide. Otherwise, the score inequalities and frequency–severity budget apply to the effective ratios er t = exp(ez t ). For a band strictly inside [e −20 ,e 20 ], including [0.8, 1.28], clipping does not change which tokens violate the band. Thus the frequency-only guarantee also holds for the original ratios, while the severity term measures the guarded log-distances. 18 CCOMPLETE RESPONSE FOR THE DISAGREEMENT CASE Figure 1 in the main paper visualizes the complete signed-drift trajectory and a tokenizer-aligned local window. For auditability, the five pages below show all 8,619 response tokens from the same rollout without elision. Each token is rendered in its tokenizer-aligned box: red denotes logr t > 0, green denotes logr t < 0, and color intensity encodes | logr t |. The first-page header repeats the response-level scores and masking decisions. For this response, d GeoMean = 0.002502 and d CARM = 0.024486: signed aggregation cancels 89.78% of the absolute log-ratio mass. The example demonstrates score cancellation and differing mask decisions. Complete token-level view for the Figure 1 disagreement case | Part 1/5 log r t > 0log r t < 0 Here'sa thinking processtosolvetheproblem: 1.**UnderstandtheGoal:**Theuserwantsto find thevalue of $ f (-100 )$ given a functional equationanda specificdefinitionof $ f (x )$ onaninterval. Finally ,expresstheresultasa fraction $ m/n $ and find $ m+n $. 2.**An alyze theGiven Information :** *Domain: $ f (x )$ is defined on $\ mathbbR $. *FunctionalEquation: $ f (x+3)· f (x-4)=-1 $ for all $ x $. * Definition onaninterval: $ f (x)=\ log_ 2(9-x )$ for $ 0 <7 $. * Target :Calculate $ f (-100) $. *Output format : $ m+n $ wheretheansweris $\ frac mn $ ( pres umably $ m,n ,n>0,\ gcd (m,n)= 1 $, though theprompt justsays format $ m/n $ andasks for $ m+n $ ). 3.**ExploretheFunctionalEquation:** Lettheequationbe $ f (x+3) f (x-4)=-1 $. Wewantto find relationsbetweenvalues of $ f $ at different points. Let'sreplace $ x $ with $ x+4 $ toseeapatterninthe arguments . $ f ((x+4)+3) f ((x+4)-4)=-1 f (x+7) f (x)=-1 $. Thislooks promising becauseitlinks $ f (x+7 )$ and $ f (x) $. Let'scheck if thisrelationholds for all $ x $. Yes,it'sderived from the originalidentity for all $ x $. So, $ f (x+7)=-\ frac 1 f (x) $. Let'slookatthenextstepto find aperiodic ity . Apply therelation again for $ f (x+7 )$ : $ f (x+14)=-\ frac 1 f (x+7)=-\ frac 1(-1 /f (x))= f (x) $. So, $ f (x+14)= f (x )$ for all $ x $. The function $ f (x )$ isperiodicwithperiod $ T=14 $. 4.**Calculate TargetArgument :** Weneedto find $ f (-100) $. Since $ f (x )$ hasperiod14,wecan shift the argumentby multiples of 14to bring itintotheknowninterval $ [0,7) $. Wewantto find an integer $ k $ suchthat $ -100+14k $ falls within $ [0,7) $. Let'sapproximate: $ -100/14≈-7.14 $. So $ k $ couldbe7,8,or8. Let's try $ k=8 $: $ -100+14(8)=-100+112=12 $. Wait,the definition $ f (x)=\ log_ 2(9-x )$ is for $ 0≤ qslantx<7 $. $ x=12 $ isnotin $ [0,7) $. However,weknow $ f (12)= f (12-14)= f (-2) $. Let's try tomap $ -100 $ tothe range $ [0,7) $. $ -100+14k∈[0,7) $. Let'scalculate $ -100 14 $. $ 100=7×14+2 $. So $ 100 2 mod14 $. tokens 1-895 Thus $ -100≡-2 12 14 $. Thismeans $ f (-100)= f (12) $. Also $ f (12)= f (12-14)= f (-2) $. Butweneedthevalueinterms of the definition $ f (x)=\ log_ 2 (9-x )$ whichis defined on $ [0,7) $. Soweneedtoevaluate $ f (-100 )$ byrelating ittoavalue $ x _ 0∈[0,7) $. Let'schecktherelation $ f (x+7)=-1 /f (x )$ again . If wehaveavaluein $ [0,7 )$ , say $ x∈[0,7 )$ , then $ f (x+7)=-1 /f (x) $. Notethat $ x+7 $ wouldbein $ [ 7,14) $. Wehave $ f (-100) $. We found $ -100 12 14 $. So $ f (-100)= f (12) $. Wealsoknow $ f (x+14)= f (x) $. Wait,I'm saying $ f (-100)= f (12) $. But $ 12∉[0,7) $. Canweexpress $ 12 $ using the shift relationbackwards? $ f (12)= f (12-14 )$ ?No,that goes backto negative . Let'slookattherelation $ f (x+7) f (x)=-1 $. Thisimplies $ f (x+7)=-1 /f (x) $. Wewanttousethe definition $ f (x)=\ log_ 2(9-x )$ for $ 0 <7 $. Thevalue12isoutsidethis range . However,weknowthe functional equationholds for allreal $ x $. Sowecanuse $ f (x+7 )f (x)=-1 $ torelateavalueat $ 12 $ toavalueat $ 5 $ (since $ 12-7=5 $ ). Let'scheck: $ 5∈[0,7) $. So, $ f (12) f (5)=-1 f (12)=-1 /f (5 ) $. Since $ f (-100)= f (12 )$ ,then $ f (-100)=-1 /f ( 5) $. Nowweneedtocalculate $ f (5) $. Since $ 0 5<7 $, wecanusethe formula directly . $ f (5)=\ log_ 2(9-5)=\ log_ 2(4)=2 $. Therefore , $ f (-100)=-1/2=-\ frac 12 $. 5.**DoubleCheckthe Logic :** *FunctionalEquation: $ f (x+3 )f (x-4)=-1 $. *Let $ x= y +4 $. Then $ f (y +7 )f (y )=-1 $. *Implies $ f (y )=-1 /f (y +7) $. *Period icity : $ f (y +14)= f (y +7+7)=-1 /f (y +7)=-1/(-1 /f (y ))= f (y ) $. Periodis14. * Target : $ f (-100) $. * $ -100=-7×14-2=-98-2 $. No. $ -100/14=-7.1428... $ $ -7×14=-98 $. So $ -100=-7×14- 2 $. Thus $ -100≡-2 14 $. So $ f (-100)= f (-2) $. tokens 896-1,842 Tokens 1-1,842 | 8,619 response tokens | GeoMean s=1.0025 <= 1.005 (accept) | CARM s=1.0248 > 1.02 (reject) 19 Complete token-level view for the Figure 1 disagreement case | Part 2/5 Tokens 1,843-3,632 | Continuation of the same unabridged tokenizer-aligned response. log r t > 0log r t < 0 *Nowwehave $ f (-2) $. Is $ -2∈[0,7 )$ ?No. *Weneedtousetherelation $ f (x+7 )f (x)=-1 $ tomap $ - 2 $ totheinterval $ [0,7) $. *Let $ x=-2-7=-9 $. Then $ f (-9-7 )f (-9)= f (-16 )f (-9 )$ ?No. *Wewant $ x _ new∈[0,7 )$ suchthat $ x _ new $ is relatedto $ -2 $ by a shiftof 7. * $ -2+7=5 $. *Soconsider $ f (5) $. Wehavetherelation $ f (5 )f (5-7 )=-1 f (5 )f (-2)=-1 $. *So $ f (-2)=-1 /f (5) $. *Check validity : $ 5∈[0,7) $. Formula $ f (x)=\ log_ 2(9-x )$ applies. *Calculate $ f (5)=\ log_ 2(9-5)=\ log_ 2(4)= 2 $. *So $ f (-2)=-1/2 $. *Since $ f $ isperiodicwithperiod14, $ f (-100)= f (-2)=-1/2 $. *Wait,is $ f (-100)= f (-2 )$ consistentwithearliermod calc? $ -100=-8×14+12 $. So $ f (-100)= f ( 12) $. $ 12-7=5 $. $ f (12 )f (5)=-1 f (12)=-1/2 $. $ -100 12 14 $. $ -100+112=12 $. Correct. $ -100=-7×14-2 $. So $ -100≡-2\ pmod14 $. $ 12≡-2 14 $. Correct. *Also,isthere any constraintviolation? The definition is given for $ 0 <7 $. Weused $ f (5 )$ whichisinthat range . Weused $ f (x+7 )f (x)=-1 $. Thisholds for allreal $ x $. So $ f (5 )f (-2)=-1 $. $ f (-2)=-1/2 $. $ f (-100)= f (-2)=-1/2 $. *Resultis $ -1/2 $. *Format $ m/n $. So $ m=-1,n=2 $. * Usually intheseproblems $ m,n $ are integers , $ n>0 $. If reduced, $ gcd (m,n)=1 $. * $ m=-1,n=2 $.$ m+n=1 $. *Or maybethey expect $ m $ tobepositive?"m/n format ". Usually impliesrationalnumber. Negative numeratorisallowed. *Let'scheck if there's ambiguity inthe sign . $ f (x)=\ log _ 2(9-x) $. *Domain $ 0 <7 $. For $ x=5 $,$ 9-5=4 $,$\ log _ 24=2 $. Positive. * $ f (-100)=-1/2 $. * $ m=-1,n=2 $.$ m+n=1 $. *Let'sre-read carefully :"m/n format ". Often implieslowest terms. *What if the fraction is simplified differently ?No, $ -1/2 $ issimplest form . *Are $ m,n $ requiredtobecoprime?" give thevalue of m+n tokens 1,843-2,778 ". If Iused $ -2/4 $, sumis2. If $ -1/2 $, sumis1. Standardmathematicalconventionimpliescoprime integers with $ n>0 $. Iwillassumethisstandardconvention. Let'sre- verify theperiodderivation. $ f (x+3 )f (x-4)=-1 $ Replace $ x $ with $ x+4 $:$ f (x+7 )f (x)=-1 $. Period icity : $ f (x+14)=\ frac -1 f (x+7)=\ frac -1-1 /f (x)= f (x) $. Correct. Moduloarithmetic: $ -100=-7×14-2≡-2 14 $. $ -100=-8×14+12 12 14 $. $ f (-100)= f (-2)= f (12) $. Using $ f (x+7 )f (x)=-1 $ for $ x=-2 $: $ f (-2+7 )f (-2)=-1 f (5 )f (-2)=-1\ implies f (-2)=-1 /f (5) $. Using $ x=5 $: $ f (5)=\ log_ 2(9-5)=\ log_ 2(4)=2 $. $ f (-100)=-1/2 $. Isthere any potentialtrap? " defined onR"."0<=x<7". Isitpossible $ f (x)=0 $ somewhere? $\ log_ 2(9-x )$ isnever0unless $ 9-x=1 =8 $, whichisoutside $ [0,7) $. If $ f (x)=0 $, therelation $ f (x+7 )f (x)=-1 $ would imply $ 0=-1 $, contradiction. So $ f (x) 0 $. Isthedomain of log valid? $ 9-x>0 <9 $. Interval $ [0,7 )$ satisfies this. Sothe function iswell -defined . Isthereanother shift ? $ f (x+3 )f (x-4)=-1 $. Maybe usethis directly ? Let $ x-4=-100 =-96 $. Then $ f (-96+3 )f (-100)=-1 f (-93 )f (-1 00)=-1 $. Need $ f (-93) $. $ -93=-7×14-5≡-5 14 $. $ -93 9 14 $. $ f (-93)= f (9) $. $ 9=2+7 $. $ f (9) f (2)=-1 f (9)=-1 /f (2) $. $ 2∈[0,7) $.$ f (2)=\ log_ 2(9-2)=\ log_ 2(7) $. Thislooks messy . $ f (-93 )$ needstobecalculated. Let'ssticktotheperiodderivation. $ f (x+14)= f (x) $. Weestablished $ f (-100)= f (12) $. Then $ f (12 )f (5)=-1 $ (since $ 12-7=5 $ ). $ f (5 )$ is easily calculated. $ f (-100)=-1 /f (5) $. tokens 2,779-3,632 20 Complete token-level view for the Figure 1 disagreement case | Part 3/5 Tokens 3,633-5,377 | Continuation of the same unabridged tokenizer-aligned response. log r t > 0log r t < 0 $ f (5)=\ log_ 2(4)=2 $. $ f (-100)=-1/2 $. Let'sdoublecheck $ f (x+7 )f (x)=-1 $. Original : $ f (x+3 )f (x-4)=-1 $. Shift $ x +7 $: $ f (x+7+3 )f (x+7-4)= f (x+10 )f (x+3)=-1 $. Thisdoesn'tmatch $ f (x+7 )f (x) $. Wait,derivationwas: Let $ y =x-4 $. Then $ x= y +4 $. $ f (y +4+3 )f (y )=-1 f (y +7 )f (y )=-1 $. Thisderivationiscorrect. Wait,earlierIwrote"Replace $ x $ with $ x+4 $ ". Original equation: for any $ x $,$ f (x+3 )f (x-4)=-1 $. Let'ssubstitute specific $ x $. If wewanttolink $ f (x )$ and $ f (x+7 )$ ,weneedthe arguments of $ f $ intheequationtomatch $ x $ and $ x+7 $. In $ f (x+3 )f (x-4 )$ ,the difference between arguments is $( x +3)-(x-4)=7 $. So yes , $ f (u )f (v )$ with $ u-v=7 $ equals $ -1 $. Let $ u=x+7 $,$ v=x $.$ u-v=7 $. Then $ f (u )f (v)=-1 $, so $ f (x+7 )f (x)=-1 $. Yes,derivationstands. Now,checkthe mapping from $ -100 $ to $ [0,7) $. $ -100 12 14 $. $ 12 5 7 $. Wait. $ f (x+7)=-1 /f (x) $. $ f (x+14)= f (x) $. Sowemap $ -100 $ byadding multiples of 14. $ -100+112=12 $. Wewantaninputtothe functiondefinition $ f (t)=\ log_ 2(9 -t )$ for $ t∈[0,7) $. Wehave $ f (12) $. $ f (12 )f (5)=-1 $ because $ 12=5+7 $. Is $ 5∈[0,7 )$ ?Yes. So $ f (5)=\ log_ 2(9-5)=2 $. $ f (12)=-1/2 $. $ f (-100)= f (12)=-1/2 $. Seemssolid. Isthere anyambiguity about $ m,n $ beingintegers ? Usually ,questionslike" find m+nwherem/nisthevalue" imply integer numeratoranddenominatorinlowestterms. $ m=-1,n=2 $. $ m+n=1 $. Sometimes they wantpositive $ n $. Yes, usually $ n>0 $. Sometimes they define $ m,n $ aspositive integers andadda sign to thevalue?No, $ m/n $ format usually allows negative $ m $. If theanswerwas $ -1/2 $, then $ m=-1,n=2 $. If theproblemimplied $ m,n +$ and $ m $ is thenumerator of the simplifiedfraction ( includingsign ),it might be specified .Butstandardinterpretationis $ m,n , n 0,\ gcd (| m | , | n | )=1,n>0 $. So $ m=-1,n=2 $. tokens 3,633-4,521 Sum=1. Wait,couldthe log be $\ ln $ ?The symbol is $\ log_ 2 $. Couldthebasebe something else? $\ log_ 2 $. $ f (5)=\ log_ 2(4)=2 $. Calculationiscorrect. Letmecheck if Imissed any subtleconstraints. " defined on $\ mathbbR $ ". " for any realnumber $ x $ ". Istheperiodic ity unique? $ f (x+14)= f (x) $. Couldithaveasmallerperiod? $ f (x+7)=-1 /f (x) $. Since $ f (x )$ isnotident ically $ 1 $ or $ -1 $ (valueis2 ),periodcannotbe7(unless $ f (x+7)= f (x) f (x) =-1 /f (x) f (x)^2=-1 $, impossible for realnumbers). Soperiodis exactly 14(oramultiple if $ f $ hasother propertiesnot given ,but14is sufficient tosolve). Actually , $ f (x+7)=-1 /f (x )$ means $ f (x )$ isanti -periodicwithperiod7. So $ f (x+14)= f (x) $. $ -100=-7×14-2 $. So $ f (-100)= f (-2) $. $ f (-2)=-1 /f (5) $. Calculation: $ f (5)=2 $. Result $ -0.5 $. Let's quickly check $ x $ mappingagain . $ f (x )$ is given for $ 0 <7 $. Weneed $ f (-100) $. $ -100 14 $. $ 100=70+30=70+28+2 $.$ 100≡ 2 14 $. $ -100≡-2 12 14 $. So $ f (-100)= f (12) $. $ 12=7+5 $. $ f (12)= f (5+7) $. $ f (x+7)=-1 /f (x) $. $ f (5+7)=-1 /f (5) $. $ 5∈[0,7) $. $ f (5)=\ log_ 2(9-5)=\ log_ 2(4)=2 $. $ f (12)=-1/2 $. Everything isconsistent. What if Ievaluate $ f (-2 )$ directly from therelation $ f (x+3 )f (x-4)=-1 $ ? $ f (-2+3 )f (-2-4)=-1 f (1 )f (-6)=-1 $. $ f (-6 )$ ? $ -6 8 14 $.$ f (-6)= f (8) $. $ 8=1+7 $.$ f (8)=-1 /f (1) $. So $ f (1)(-1 /f (1))=-1 -1=-1 $. Taut ology .Doesn'thelp find value of $ f (1) $. Weneedthevaluein $ [0,7) $. $ f (-2) $. $ f (-2+7 )f (-2)=-1 f (5 )f (-2)=-1 $. $ 5∈[0,7) $.$ f (5)=2 $. tokens 4,522-5,377 21 Complete token-level view for the Figure 1 disagreement case | Part 4/5 Tokens 5,378-6,954 | Continuation of the same unabridged tokenizer-aligned response. log r t > 0log r t < 0 $ 2 f (-2)=-1 f (-2)=-1/2 $. Matches. Arethere any constraintson $ m,n $ ? Usually $ m,n $ arecoprime integers . $ m=-1,n=2 $. $ gcd (1,2)=1 $. $ n>0 $. $ m+n=-1+2=1 $. Wait,isitpossible $ m,n $ arepositive? Thevalueis negative .So $ m $ mustbe negative if wewrite $ m/n $. If thequestionimplies $ | val | =m/n $, itwould say "absolute value"orsimilar. Standardinterpretation for $ m/n $ format with negative numberis just $ numerator/denominator $. Onemorecheck. $ f (x+3 )f (x-4)=-1 $. $ x=0 f (3 )f (-4)=-1 $. $ x=100 f (103 )f (96)=-1 $. Wedon'tneedthis. We just need $ f (-100) $. $ f (-100)=-1 /f (-73 )$ ( using $ x-4=-100 iesx=-96 f (-93 )f (-100)=-1 $.$ -93≡ 3 14 $.$ f (3) $. Wait. $ f (-93 )f (-100)=-1 $. $ -93=-7×13-2=-91-2≡-2\ equiv12 14 $. $ f (-93)= f (12)=-1/2 $. $ (-1/2 )f (-100)=-1 f (-100)=2 $. CONTRADICTION! Let'sre-evaluatethispath. $ f (-93 )f (-100)=-1 $. We found $ f (-100)=-1/2 $. So $ f (-93 )$ shouldbe2. Let'scheck $ f (-93 )$ again . $ -93 14 $. $ 93=14×6+9 $.($ 14×6=84 $ ). So $ 93=6×14+9 93 9\ pmod14 $. So $ -93≡-9 5 14 $. So $ f (-93)= f (5) $. $ f (5)=2 $. So $ f (-93)=2 $. Then $ f (-93 )f (-100)=2× f (-100)=-1\ implies f (-100)=-1/2 $. Okay , my previouscalculation of $ -93 14 $ was wrong inthe"Wait"check. $ 14×7=98 $. $ -93=-98+5 $. So $ -93 5 14 $. My previouschecksaid $ -93 12 $ (basedon wrong arithmetic). tokens 5,378-6,188 $ 12-93=-81 $. Notdivisible by 14. Okay ,the consistency check $ f (-93 )$ works. $ f (-93)= f (5)=2 $. $ f (-100)=-1/2 $. Okay ,theresultisrobust. Let'swritedownthesteps clearly . 1.Deriveperiodic ity $ T=14 $ from $ f (x+3 )f (x-4) =-1 $. Substitute $ x+4 $ for $ x $ to get $ f (x+7 )f (x)=-1 $. Then $ f (x+14)= f (x) $. 2.Reduce $ -100 $ modulo14. $ -100=-8×14+12 12 14 $. So $ f (-100)= f (12) $. 3.Usetherelation $ f (x+7 )f (x)=-1 $ toconnect $ f ( 12 )$ totheinterval $ [0,7) $. Let $ x=5 $. Then $ x+7=12 $. $ f (12 )f (5)=-1 $. $ f (12)=-1 /f (5) $. 4.Calculate $ f (5 )$ using the definition $ f (x)=\ log_ 2(9-x) $. $ f (5)=\ log_ 2(9-5)=\ log_ 2(4)=2 $. 5.Calculateresult. $ f (-100)=-1/2 $. 6. Identify $ m,n $. $ m=-1,n=2 $. 7.Compute $ m+n $. $ m+n=1 $. Isthere any reasontodoubtthe format "m/n"? Sometimes fractions arerequiredtobepositive.Butthevalueis clearlynegative . Unlessthequestionimplies $ f (-100)=-\ frac mn $ ? " 则 $ f (-100 )$ 的 值为 \ frac mn 的 格式 ， 请 给出 m+n 的值 ". This literally means"thevalue off (-100)isinthe format m/n ". So if thevalueis-0.5,itis-1/2. So $ m=-1,n=2 $. If thequestionmeantabsolutevalue,itwould say so. Givenit'samathproblemcontext( likely ChineseGaokaoorsimilar competition style ),"m/n format "with integersusually accepts negative numerators. Also, usually $ m,n $ arecoprimeand $ n>0 $. Let'scheck if $ f (x )$ couldbecomplex?No, log base2 usually impliesrealdomain. Domain $ 0 <7 $.$ 9-x $ ispositive. So $ f (x )$ isreal. Everything holdsin $\ mathbbR $. Finalcheck of thequestionph rasing : " f (x+3)* f (x-4)=-1". "0<=x<7". " f (-100)". "m/n format ". tokens 6,189-6,954 22 Complete token-level view for the Figure 1 disagreement case | Part 5/5 Tokens 6,955-8,619 | Continuation of the same unabridged tokenizer-aligned response. log r t > 0log r t < 0 "m+n". Calculationcheck: $ 100=7×14+2 $. $ 100 2 14 $. $ -100≡-2 12 14 $. $ f (-100)= f (12) $. $ f (12)= f (5+7) $. $ f (x+7 )f (x)=-1 $. $ f (5+7 )f (5)=-1 $. $ f (5)=\ log_ 2(9-5)=\ log_ 2(4)=2 $. $ f (12)=-1/2 $. $ m=-1,n=2 $. $ m+n=1 $. What if $ n $ canbe negative ? $ m=1,n=-2 $.$ m+n=-1 $. Standardconvention forfractions :denominatorpositive. Iwillstickwith $ n>0 $. Structure of theresponse: 1.Deriveperiodic ity . 2.Reduce argument . 3.Userecurrencerelation. 4.Evaluate using formula . 5.Statem,n,andsum. Wait,isthere anypossibility that $ f (x+3 )f (x-4)=-1 $ implies something else? $ f (x+7)=-1 /f (x) $. $ f (x) 0 $. If $ f (5)=2 $, then $ f (12)=-0.5 $. Then $ f (19)=-1/(-0.5)=2 $. Then $ f (19)= f (5+14 )$ ,consistent. $ f (26)= f (5+21)= f (-100+?) $. $ -100=19-119=2+17-... $ $ f (-100)= f (12)=-0.5 $. $ f (12)= f (12-14)= f (-2) $. $ f (-2)=-1 /f (5)=-1/2 $. $ f (12)=-1 /f (5)=-1/2 $. Consistent. Isthere anyambiguity with $ x=5 $ ? $ 0 5<7 $. Yes. Isthere anyboundary issue?No. Isitpossibletouse $ f (x-4 )$ relation directly on $ f (-100 )$ ? $ f (x-4)= f (-100) -4=-100 x=-96 $. $ f (-96+3) f (-100)=-1 f (-93) f (-10 0)=-1 $. $ -93 5 14 $. $ f (-93)= f (5)=2 $. $ 2· f (-100)=-1 f (-100)=-1/2 $. Thisisa faster routeto verify without explicitlystating period14 first , justlinking viamod14or just successive7-sh iffs . Wait,Iusedmod14to get $ f (-93)= f (5) $. $ -93=-14×7-5 $. No. tokens 6,955-7,787 $ 14×(-7)=-98 $.$ -93=-98+5 $. Correct. So $ f (-93)= f (5) $. This confirms theresultwithout explicitlycalculating $ -100 mod14 $ directly ,but calculating $ -93 14 $. $ -93 5 $. $ 5∈[0,7) $. $ f (5)=2 $. $ 2· f (-100)=-1 f (-100)=-0.5 $. Bothmethods align perfectly . Finalcheckon $ m+n $. $ m=-1,n=2 +n=1 $. Thequestionasks for "m+n". Isitpossible they want $ | m | +n $ ?No, just "m+n". Isitpossible $ f (-100 )$ isan integer ?No. Okay , ready towrite. </think> 由 已知 条件 $ f (x+3)· f (x-4)=-1 $ ， 令 $ x $ 替换 为 $ x+4 $ ， 则有 ： $$ f ((x+4)+3)· f ((x+4)-4)=-1 $$ 即 $$ f (x+7)· f (x)=-1 $$ 由此 可得 $$ f (x+7)=-\ frac 1 f (x) $$ 再次 应用 此关系 ， 将 $ x $ 替换 为 $ x+7 $ ： $$ f (x+14)= f ((x+7)+7)=-\ frac 1 f (x+7)= -\ frac 1-1 /f (x)= f (x) $$ 说明 函数 $ f (x )$ 是 周期 函数 ，且 周期 $ T=14 $ 。 我们需要 求 $ f (-100 )$ 的值 。 利用 周期性 ， 我们可以 将 自 变量 $ -100 $ 转化为 一个 在 已知定义 域 $ [0,7 )$ 附近的值 。 计算 $ -100 $ 关于 $ 14 $ 的同 余 关系 ： $$ -100=14×(-8)+12 $$ 即 $ -100 12 14 $ 。 所以 $$ f (-100)= f (12) $$ 接下来 利用 $ f (x+7)· f (x)=-1 $ 将 $ f (12 )$ 转化 到区间 $ [0,7 )$ 。 因为 $ 12=5+7 $ ，且 $ 5∈[0,7 )$ ， 所以 ： $$ f (12)· f (5)=-1 f (12)=-\ frac 1 f (5) $$ 又 因为 当 $ 0 <7 $ 时 ， $ f (x)=\ log_ 2(9-x )$ ， 所以 $$ f (5)=\ log_ 2(9-5)=\ log_ 2(4)=2 $$ 代入 上 式 ， 得 ： $$ f (12)=-\ frac 12 $$ 因此 $$ f (-100)=-\ frac 12 $$ 题目 要求 将 结果表示 为 $\ frac mn $ 的形式 ， 并 求 $ m+n $ 的值 。 这里 $ m=-1 $ ， $ n=2 $ （ 通常 约定 分 母 $ n $ 为正 整数 ）。 则 $ m+n=-1+2=1 $ 。 1 <|im_end|> tokens 7,788-8,619 DEXPERIMENTAL SETTINGS D.1DATA Math training data. The mathematical-reasoning experiments use the competition-style prob- lems in the DAPO Math training corpus (Yu et al., 2025). Math-Verify (Kydlí ˇ cek, 2025) extracts the boxed final answer and checks its equivalence to the reference. Null, IcePop, TRM, GeoMean, and CARM use the same prompt order, rollout group size, rewards, optimizer settings, and token-level PPO/GRPO clipping; only the off-policy correction differs. Math evaluation data. AIME 2024, AIME 2025, and AIME 2026 each contain 30 contest prob- lems and are reported separately (Mathematical Association of America, 2026). BeyondAIME con- tributes 100 additional problems designed to extend evaluation beyond the three annual AIME sets 23 (Chen et al., 2025). We sample 16 responses per problem and compute acc/mean@16 from ex- act final-answer correctness. The Average in Table 1 gives AIME 2024, AIME 2025, AIME 2026, and BeyondAIME equal weight. Evaluation is performed before training and periodically during optimization. Code training data. The general-code experiments use problems from the TACO training split (Li et al., 2023), with natural-language task descriptions, input/output specifications, and executable tests. During RL, the reward checks up to the first ten available tests for each training problem. This fixed cap keeps execution cost and sandbox load comparable across methods; held-out benchmark tests are not used by the training reward. All five methods receive the same prompts, decoding budget, executable reward, and optimizer configuration. Code evaluation data. We evaluate each frozen TACO-trained checkpoint on four complementary suites. TACO evaluation uses 1,000 problems from its held-out test split and up to the first ten tests per problem. LiveCodeBench-v6 contains 1,054 temporally collected coding problems and is judged with its complete released tests. HumanEval+ and MBPP+ contain 164 and 378 problems, respectively, and use the stronger EvalPlus test suites rather than only the original base tests. MBPP+ extends the MBPP benchmark (Austin et al., 2021). Every generated program is rejudged with SandboxFusion (Cheng et al., 2025), with functional correctness reported as pass@1. D.2PROMPT TEMPLATES Figures 7 and 8 reproduce the task-specific chat messages used throughout training and evaluation. Dataset problem statements replace the placeholders without changing the surrounding instructions. System: A conversation between User and Assistant. The user asks a question, and the Assistant solves it. User: Please solve the problem step by step, and put your final answer within . <problem> Figure 7: Prompt template for mathematical-reasoning experiments. The placeholder <problem> is replaced by the problem text. System: You are an expert competitive programming assistant. Solve the user’s programming problem in Python 3. You may reason about the solution, algorithm, and edge cases. At the end, provide the final answer as exactly one fenced Python code block. The final code must be complete and executable. It should read from standard input and write to standard output, unless the problem provides starter code requiring a specific function or class interface. User: Solve the following programming problem. Problem: <problem> <starter code, when provided> Return your final solution in this format: ```python # your complete Python solution ``` Figure 8: Prompt template for code-generation experiments. The placeholder <problem> is re- placed by the problem text; the starter-code field is included only when provided by the dataset. 24 D.3TRAINING AND EVALUATION CONFIGURATION Table 3 consolidates the shared settings, method-specific controls, and threshold grids used to pro- duce the reported comparisons. Table 3: Complete experimental configuration for mathematical reasoning and code generation. FieldValue Base modelsQwen3.5-4B and Qwen3.5-9B (math); Qwen3.5-9B (code) Framework / algorithm verl; GRPO-style optimization Training dataDAPO Math (math); TACO training split with executable rewards (code) Evaluation dataAIME 2024/2025/2026 and BeyondAIME; TACO test split, LiveCodeBench-v6, HumanEval+, and MBPP+ Evaluation sampling16 responses/problem (math); one response/problem per evaluation (code) Prompt batch / rollout group128 prompts / 8 responses per prompt PPO mini-batch size8 Training budgetUp to 120 training iterations Maximum response length16,384 tokens Learning rate3× 10 −6 PPO clipping band[1− 0.2, 1 + 0.28] = [0.8, 1.28] Compared methodsNull; IcePop; TRM; GeoMean; CARM (ours) Baseline settingsIcePop token-ratio interval [0.5, 5.0]; TRM cutoff 0.05 GeoMean thresholds Qwen3.5-4B:1.0025, 1.005, 1.01, 1.02, 1.03 Qwen3.5-9B:1.0025, 1.005, 1.01, 1.02, 1.03 CARM (ours) thresholds Qwen3.5-4B:1.02, 1.025, 1.03, 1.04 Qwen3.5-9B:1.02, 1.025, 1.03, 1.04 Evaluation protocol. Both tasks are evaluated with three random seeds. Mathematical-reasoning evaluation uses 16 responses per problem and reports mean@16; code-generation evaluation uses one response per problem and reports pass@1. Code decoding uses temperature 0.7, top-p = 0.9, top-k = 50, and a maximum response length of 16,384 tokens. Generated programs are judged with the same batched SandboxFusion evaluator across methods. For each method, math results use the training iterate with the highest three-year AIME mean@16; code results use the iterate with the highest LiveCodeBench-v6 validation pass@1. Best- and worst- of-k use the same math iterate without separate optimization of either metric. Qwen3.5-4B Be- yondAIME results for Null, GeoMean, and CARM use separate runs matched in configuration and training iteration. TRM implementation. We follow the sampled-token max-filtering option described in Sec- tion 4.3 and Appendix E of Li et al. (2026). It rejects a response when max t 1 2 (logr t ) 2 > 0.05. This is the paper’s proposed approximation when full logits are unavailable, not a full-vocabulary KL implementation. All TRM results in this paper refer to this sampled-token variant. 25 ETHRESHOLD RESULTS Table 4 reports the GeoMean and CARM threshold sweeps, with AIME mean@16 averaged equally across the three years. Table 4: Threshold sweep (%, higher is better). Each row reports the three-year AIME mean@16 for that threshold. Bold and underline denote the best and second-best values within each model block. Qwen3.5-4BQwen3.5-9B MethodτAvg.MethodτAvg. Null–50.83Null–70.42 GeoMean1.0025 69.03GeoMean1.0025 71.67 GeoMean1.00569.17GeoMean1.00574.03 GeoMean1.0166.74GeoMean1.0169.65 GeoMean1.0252.57GeoMean1.0271.67 GeoMean1.0352.43GeoMean1.0369.86 CARM (ours)1.02 70.42CARM (ours)1.0272.99 CARM (ours) 1.02569.24 CARM (ours) 1.025 77.50 CARM (ours)1.0367.99CARM (ours)1.0373.82 CARM (ours)1.0466.81CARM (ours)1.0474.17 FADDITIONAL TRAINING DIAGNOSTICS Figure 9 gives the time-resolved utilization view used in Section 5.1. It distinguishes sustained filter- ing from a mean dominated by a small number of unusually strict updates, and contrasts Qwen3.5-9B on mathematical reasoning and code generation. 020406080100120 Training step 40 50 60 70 80 90 100 Masked sequences (%) Qwen3.5-9B Math 020406080100120 Training step 50 60 70 80 90 100 Qwen3.5-9B Code GeoMeanTRMCARM (ours) Figure 9: Sequence-masked fraction during training for the selected GeoMean, TRM, and CARM (ours) configurations. The left and right panels show Qwen3.5-9B mathematical-reasoning and code-generation training, respectively. Faint curves are raw measurements and opaque curves apply an exponential moving average with weight 0.8. Figure 10 complements validation accuracy with generation- and optimization-level measurements. It compares the same five methods on Qwen3.5-9B mathematical-reasoning training. Faint solid curves show raw measurements, while opaque solid curves show exponential moving averages with weight 0.8. Gradient norm uses a logarithmic axis. 26 020406080100120 Training step 2000 4000 6000 8000 10000 12000 14000 Response length (tokens) 020406080100120 Training step 0 2 4 6 8 Policy entropy 020406080100120 Training step 10 −2 10 −1 10 0 Gradient norm NullIcePopTRMGeoMeanCARM (ours) Figure 10: Training diagnostics for Qwen3.5-9B mathematical reasoning. Panels show response length, policy entropy, and gradient norm across the five methods. Faint curves show raw mea- surements; opaque curves show exponential moving averages with weight 0.8. Gradient norm is displayed on a logarithmic scale. Figure 11 presents the matched diagnostics for code-generation training. The same metrics and smoothing convention make the domain-level differences directly comparable with Figure 10. 020406080100120 Training step 4000 6000 8000 10000 12000 14000 Response length (tokens) 020406080100120 Training step 0 2 4 6 8 Policy entropy 020406080100120 Training step 10 −2 10 −1 10 0 Gradient norm NullIcePopTRMGeoMeanCARM (ours) Figure 11: Code-generation training diagnostics: response length, policy entropy, and gradient norm for Qwen3.5-9B trained on TACO. Faint and opaque curves show raw and smoothed measurements, respectively, using the same smoothing and axis conventions as the mathematical-reasoning diag- nostics. GBEST- AND WORST-OF-k SAMPLING METRICS Figure 6 evaluates inference-time sample scaling at the same Qwen3.5-9B checkpoints reported in Table 5. For each problem, let X 1 ,...,X 16 ∈ 0, 1 denote the correctness of the 16 indepen- dently sampled responses and let c = P 16 j=1 X j . The per-problem empirical accuracy is c/16, and mean@16 averages this quantity over problems and then equally over AIME 2024, 2025, and 2026. Best-of-k asks whether at least one of k sampled responses is correct. It captures the opportunity provided by additional inference-time sampling when an oracle verifier can select a correct candi- date. Worst-of-k asks whether all k sampled responses are correct. It is therefore a strict measure of repeatability: a model scores highly only when its stochastic samples solve the same problem consistently. Worst-of-k should not be read as robustness to distribution shift or adversarial pertur- bations. At k = 1, both statistics reduce exactly to mean@16, which anchors the two curves at the same observed single-sample accuracy. 27 For k ∈ 2, 4, 8, 16, we resample k responses with replacement from the 16 available responses for each problem and apply the maximum (best-of-k) or minimum (worst-of-k) correctness operator. The plotted values average 1,000 bootstrap replicates, using the same models as Table 1. For a problem with c correct responses among 16, the expected best-of-k value is b B k (c) = 1− 1− c 16 k .(73) Equation 73 is the probability that at least one of k independent draws from the empirical re- sponse distribution is correct. It is a with-replacement best-of-k statistic, not the standard without- replacement pass@k estimator. The corresponding expected worst-of-k value is c W k (c) = c 16 k .(74) Equation 74 is the probability that all k draws are correct. It is non-increasing in k, and strictly decreasing for a problem when 0 < c < 16. As with best-of-k, this is a with-replacement statis- tic rather than the literal maximum or minimum over a fixed subset of the original 16 responses. Together, the two curves characterize solution coverage and repeatability. Worst-of-k emphasizes problems with high per-problem correctness probability; an aggregate increase does not imply im- provement on every problem. Table 5: Exact Qwen3.5-9B values underlying Figure 6. All entries are equal-weighted three-year AIME averages in percent. MetricMethodk = 1 k = 2 k = 4 k = 8 k = 16 Best-of-k Null70.4278.0783.3486.4288.26 TRM73.9681.4786.4389.3990.98 GeoMean74.0380.1984.0586.7388.54 CARM (ours) 77.50 83.94 87.74 90.05 91.28 Worst-of-k Null70.4262.7055.3049.0444.38 TRM73.9666.4959.2752.9948.00 GeoMean74.0367.6661.3255.0249.83 CARM (ours) 77.50 71.04 64.44 58.05 52.61 28