Paper deep dive

To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregation

Jiuheng Lin, Chen Zhang, Yansong Feng

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 82

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%

Last extracted: 7/5/2026, 2:05:12 AM

Summary

The paper introduces HIPPO (Hint-Injected Pairwise Policy Optimization), a novel reinforcement learning framework designed to mitigate 'Pre-RL data overlap.' This phenomenon occurs when RL training datasets overlap with pretraining or SFT corpora, causing LLMs to exploit shortcuts by memorizing answers and generating spurious post-hoc reasoning instead of genuine deduction. HIPPO uses hint-injected aggregation—injecting the ground-truth answer into the prompt—to simulate and identify these shortcut-reliant behaviors. It then employs a pairwise reward model (a Bradley-Terry comparator) to maximize the distinction between genuine reasoning and hint-induced shortcuts. Experimental results in mathematical and medical reasoning domains demonstrate that HIPPO improves reasoning quality, stability, and out-of-distribution (OOD) generalization compared to standard RL and other reward-shaping baselines.

Entities (7)

DeepScaleR · dataset · 100%Hint-injected aggregation · methodology · 100%HIPPO · framework · 100%MedQA · dataset · 100%Pairwise reward model · component · 100%Pre-RL data overlap · problemphenomenon · 100%Qwen2.5-7B-Instruct · model · 100%

Relation Signals (5)

HIPPO → addresses → Pre-RL data overlap

confidence 100% · To address this, we introduce HIPPO, a novel RL framework... designed to mitigate Pre-RL data overlap.

Pre-RL data overlap → causes → shortcut-driven rationalization

confidence 100% · causing models to exploit shortcuts by memorizing correct answers and fabricating post-hoc reasoning.

HIPPO → uses → Hint-injected aggregation

confidence 100% · integrates hint-injected aggregation with a tailored pairwise reward model.

HIPPO → uses → Pairwise reward model

confidence 100% · integrates hint-injected aggregation with a tailored pairwise reward model.

Qwen2.5-7B-Instruct → isbackbonefor → HIPPO

confidence 90% · We adopt Qwen2.5-7B-Instruct and Qwen3-4B as our backbone.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:While reinforcement learning (RL) significantly enhances LLM reasoning, its efficacy is severely undermined by Pre-RL data overlap, where RL datasets overlap with pretraining or SFT corpora, causing models to exploit shortcuts by memorizing correct answers and fabricating post-hoc reasoning. To address this, we introduce HIPPO, a novel RL framework that integrates hint-injected aggregation with a tailored pairwise reward model. By utilizing hint injection to deliberately trigger overlap-induced behaviors, the resulting traces naturally serve as explicit anchors for pairwise comparison. This provides highly discriminable preference signals, enabling a lightweight judge model to reliably distinguish genuine reasoning deduction from shortcut-driven rationalization, while the pairwise formulation ensures stable and robust optimization compared to standard PRMs. Extensive experiments demonstrate that HIPPO yields substantial improvements over standard baselines and generalizes effectively to out-of-distribution general tasks, showing it extracts authentic, transferable reasoning skills rather than superficial shortcut patterns.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

81,456 characters extracted from source content.

Expand or collapse full text

To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregation Jiuheng Lin, Chen Zhang and Yansong Feng Corresponding author. Abstract While reinforcement learning (RL) significantly enhances LLM reasoning, its efficacy is severely undermined by Pre-RL data overlap, where RL datasets overlap with pretraining or SFT corpora, causing models to exploit shortcuts by memorizing correct answers and fabricating post-hoc reasoning. To address this, we introduce HIPPO, a novel RL framework that integrates hint-injected aggregation with a tailored pairwise reward model. By utilizing hint injection to deliberately trigger overlap-induced behaviors, the resulting traces naturally serve as explicit anchors for pairwise comparison. This provides highly discriminable preference signals, enabling a lightweight judge model to reliably distinguish genuine reasoning deduction from shortcut-driven rationalization, while the pairwise formulation ensures stable and robust optimization compared to standard PRMs. Extensive experiments demonstrate that HIPPO yields substantial improvements over standard baselines and generalizes effectively to out-of-distribution general tasks, showing it extracts authentic, transferable reasoning skills rather than superficial shortcut patterns.111Code is open sourced at: https://github.com/Infinite-set/HIPPO 1 Introduction Large Language Models (LLMs) have achieved remarkable success, with their reasoning capabilities recently experiencing massive leaps driven by Reinforcement Learning with Verifiable Rewards (RLVR) (Shao et al. 2024; Guo et al. 2025; Yu et al. 2025). However, many commonly used challenging RL datasets inevitably overlap with the models’ pretraining or supervised fine-tuning (SFT) corpora (Wu et al. 2025; balunović2026matharenaevaluatingllmsuncontaminated) due to reuse of scarce high-quality data pools, as illustrated in Figure 1. We formally define this phenomenon as Pre-RL Data Overlap. Pre-RL data overlap severely undermines the efficacy of reinforcement learning. Under Pre-RL data overlap, the model retains partial memory of the solution without genuinely mastering the complete deductive process. While standard RLVR drives models to tackle complex tasks and autonomously discover reasoning patterns through final-answer correctness, exploiting superficial memory for answer retrieval also yields maximal rewards, acting as a trivial shortcut much easier to learn than executing rigorous logical deduction. Consequently, the training process degenerates into enhancing a retrieval shortcut—retrieving the memorized answer, then generating a post-hoc spurious rationale (Lin et al. 2025; Wu et al. 2025). Consequently, Pre-RL data overlap suppresses exploratory depth, degrades reasoning reliability and causing an illusory inflation of capabilities (Yan et al. 2026). Figure 1: An illustration of Pre-RL data overlap, using DeepScaleR (Luo et al. 2025) as an example. See Appendix E for detailed analysis across more RL datasets. Existing studies focus exclusively on detecting data overlap between training corpora and evaluation benchmarks (Tao et al. 2026; Wang et al. 2026a), overlooking the negative impact of data overlap across training phases. Moreover, driven by the persistent demand to inject high-quality reasoning data into base models to establish strong reasoning capability boundaries and maximize their RL potential (Yue et al. 2025; Zhao et al. 2025; Dong et al. 2026), enforcing strict Pre-RL data isolation is both practically unfeasible and detrimental, as it inevitably discards valuable data and hinders model performance. Therefore, we shift our focus from detection to mitigation and ask: How can we extract genuine reasoning gains from limited, partially contaminated datasets? To answer this, we propose Hint-Injected Pairwise Policy Optimization (HIPPO), a novel RL framework designed to mitigate Pre-RL data overlap. We begin by formally modeling the active training policy and a hypothetical policy that has learned to exploit overlap-induced shortcuts. The natural approach to penalize shortcut behaviors is to maximize the distributional KL divergence between them, but directly optimizing this divergence is mathematically intractable and unbounded. To overcome this, we theoretically derive a bounded surrogate objective by leveraging Pinsker’s inequality and the dual form of TV distance (Nguyen et al. 2010), finally reduce the intractable divergence into the reward expectation of generations produced by both policies. Next, we instantiate this hypothetical policy directly on-policy via answer-level hint injection. Since directly intervening in a model’s internal parametric memory during Pre-RL overlap is difficult, this explicit hint serves as an accessible behavioral anchor. While operating on the input space, it effectively simulates the information-asymmetry of Pre-RL data overlap —knowing the answer before reasoning (Chua and Evans 2025; Chen et al. 2025). By deliberately recreating this shortcut-reliant behavior (Marioriyad et al. 2025; Young 2026), we can algorithmically penalize it without requiring access to latent parametric states. Finally, we implement the reward judge as a pairwise Bradley–Terry comparator, maximizing the expected win rate of standard rollouts against hint-injected ones, thereby sidestepping the intransitivity and miscalibration that plague standard reward models (Swamy et al. 2024; Xu et al. 2025; Sutawika et al. 2026). Our experiments validate the effectiveness of HIPPO across mathematical and medical reasoning. Compared to standard RL, HIPPO yields consistent performance gains in both in-domain and out-of-distribution (OOD) scenarios, demonstrating capabilities to extract genuine reasoning from heavily overlapping datasets while maintaining strong generalizability. Furthermore, we show our hint-injected comparison provides a much clearer and more stable preference signal than direct pairwise comparison, enabling cost-efficient small models to serve as reliable judges, and provide evidences for the mechanism by which Pre-RL data overlap degrades RL training. Our main contributions are summarized as follows: (1) We identify the Pre-RL data overlap problem, theoretically reduce it to a bounded surrogate, and propose a mechanistic explanation regarding its impact on reasoning. (2) We propose HIPPO, a novel RL framework employing a hint-injected pairwise reward aggregation to suppress overlap-induced behaviors. (3) Extensive experiments show that HIPPO enhances reasoning and generalizability, while delivering superior optimization stability and clearer preference signals. 2 Methodology In this section, we introduce HIPPO, with its overall framework illustrated in Figure 2. We begin by formally defining the Pre-RL data overlap mitigation task. Then we theoretically reduce the formulation into a bounded and empirically optimizable surrogate to overcome the intractable nature of directly optimizing divergence. Finally, we detail the practical implementation of our framework. Figure 2: The overall framework of HIPPO. Given a task, HIPPO performs standard RL rollouts paired with hint-injected sampling, followed by a pairwise comparison between the standard and the hint-injected traces. 2.1 Preliminary Reinforcement learning for LLMs. Let (x,y)∼(x,y) , where x denotes a prompt and y its reference answer. A language model πθ _θ generates a reasoning trajectory z∈z before emitting a final answer, with Z the space of token sequences. Sampling proceeds autoregressively: z∼πθ(⋅∣x)=∏t=1|z|πθ(zt∣x,z<t)z _θ(· x)= _t=1^|z| _θ(z_t x,z_<t). Under group-based RL training (e.g., GRPO and DAPO; Shao et al. 2024; Yu et al. 2025), for each prompt x we draw a group of G rollouts z(g)g=1G∼πθ(⋅∣x)\z^(g)\_g=1^G _θ(· x), score them with a scalar reward r(⋅)r(·), then form the standardized advantage and compute the PPO clipped surrogate (Schulman et al. 2017). Pre-RL data overlap. Pre-RL data overlap—where the training data used for RL has already been exposed to the model during pre- or post-training—is prevalent and difficult to avoid in practice. To model and mitigate its effect, we formalize the trajectory distribution of the aforementioned contaminated shortcut-reliant hypothetical policy as πC(z∣x) _C(z x). This distribution characterizes a generation process where the model relies on memorized clues to bypass genuine deduction, thereby executing a reasoning shortcut directly leading to critical intermediate steps or ground-truth answer. 2.2 Direct KL Optimization and Its Limitations The core training objective is to ensure that the current policy πθ(z∣x) _θ(z x) remains distinguishable from πC(z∣x) _C(z x) when trained on already contaminated datasets—meaning the policy actively avoids reasoning shortcuts even when it has implicitly memorized the answers due to Pre-RL data overlap. A natural formalization is to maximize the Kullback–Leibler (KL) divergence between the two policies: DKL(πθ∥πC)=z∼πθ[log⁡πθ(z∣x)πC(z∣x)].D_KL\! ( _θ \| _C )=E_z _θ\! [ _θ(z x) _C(z x) ]. (1) Adopting Eq. (1) as a direct RL objective, however, is problematic in two ways. First, the per-sample divergence log⁡[πθ/πC] [ _θ/ _C] is a sum of token-level log-probability differences whose variance scales with trajectory length, overwhelming the policy gradient signal in long reasoning chains (Yu et al. 2025; Liu et al. 2025). Second, the KL divergence is unbounded in θ: πθ _θ can concentrate probability mass on regions where πc _c has near-zero density to make the divergence go to infinity, leading to reward hacking rather than genuine reasoning (Gao et al. 2023). These issues motivate replacing Eq. (1) with a stable and bounded surrogate. 2.3 From KL Divergence to Pairwise Comparisons Directly optimizing the KL divergence is mathematically intractable. To overcome this, we introduce a reward-form surrogate estimated via pairwise comparisons between generations from active policy and contaminated policy. This mechanism provides a highly discriminable preference signal with superior robustness, tractability, and empirical estimability. We now formally derive how this pairwise approach serves as a provable lower bound surrogate for Eq. (1). Lemma 2.1 (Pinsker’s inequality; Tsybakov 2009; Nguyen et al. 2010). For any probability distributions p,qp,q on Z, DKL(p∥q)≥ 2DTV2(p,q),D_KL(p\,\|\,q)\;≥\;2\,D_TV^2(p,q), (2) where DTV(p,q):=supA⊆|p(A)−q(A)|.D_TV(p,q):= _A |p(A)-q(A) |. Pinsker’s inequality furnishes a bounded surrogate lower bound for the unbounded KL objective, offering a safe guarantee for distancing the active policy from the shortcut-reliant behaviors. Lemma 2.2 (Variational representation of TV). For any probability distributions p,qp,q on Z and any measurable function T:→[0,1]T:Z→[0,1], the TV distance admits the equivalent form DTV(p,q)=supT(z∼p[T(z)]−z∼q[T(z)]),D_TV(p,q)\;=\; _T (E_z p[T(z)]-E_z q[T(z)] ), (3) with the supremum attained at T⋆=A⋆T =1_A , A⋆:=z∈:p(z)≥q(z)A :=\z :p(z)≥ q(z)\. This variational form converts a distance between intractable distributions into the expected score gap of a [0,1][0,1]-valued discriminator T, which is density-free and directly amenable to Monte Carlo estimation. Moreover, the supremum is attained precisely when T correctly separates the two distributions, implying that the closer T is to the optimal discriminator, the tighter the lower bound. Concretely, in our setting, consider p=πθ(⋅∣x)p= _θ(· x) and q=πC(⋅∣x)q= _C(· x), for any capable reward model T:→[0,1]T:Z→[0,1], (z∼πθ[T(z)]−z∼πC[T(z)])2≈DTV2(πθ,πH). (E_z _θ[T(z)]-E_z _C[T(z)] )^2\;≈\;D_TV^2( _θ, _H). (4) The leftmost term is a [0,1][0,1]-bounded expectation difference—density-free and estimable from rollouts—that provably lower-bounds the original KL objective. The next step is specifying T. Instantiating T via Pairwise Comparisons. The bound in Eq. (4) is tight only when T closely approximates the optimal discriminator T⋆=A⋆T =1_A , i.e., when T accurately detects whether a trace is influenced by contamination. While tractable, evaluating a single trajectory via a pointwise reward model T(z)T(z) can be problematic: previous works indicate that pointwise RM can only express transitive, Bradley–Terry-style preference structures and are notoriously poorly calibrated across distributions (Xu et al. 2025; Wang et al. 2026c). Pairwise comparators, by contrast, provably reduce variance and robustly handle non-Markovian or intransitive preference signals while being resilient to the compounding errors (Swamy et al. 2024; Zhang et al. 2026). We therefore instantiate T via a pairwise judge J:×→[0,1]J:Z×Z→[0,1], where J(za,zb)J(z_a,z_b) denotes the judgment that zaz_a is preferred to zbz_b. Concretely, we define TJ(z):=z′∼πC(⋅∣x)[J(z,z′)]∈[0,1],T_J(z)\;:=\;E_z _C(· x) [J(z,z ) ]\;∈\;[0,1], (5) which automatically lies in [0,1][0,1], satisfying the requirement of Lemma 2.3 and is agnostic to the implementation of J. Lemma 2.3. Let J be a pairwise comparator satisfying strict symmetry222In practice, we eliminiate the LLM judge’s position bias by averaging both orderings; see Appendix A for details., i.e., J(z,z′)+J(z′,z)=1J(z,z )+J(z ,z)=1 for all (z,z′)(z,z ). Then for any distribution q, the marginal expectation over independent samples is constant: z,z′∼q[J(z,z′)]=12.E_z,z q[J(z,z )]\;=\; 12. (6) Substituting Eq. (5) into Eq. (4) and applying Lemma 2.3 with q=πCq= _C, the subtrahend collapses to a constant: z∼πC[TJ(z)]=z,z′∼πC[J(z,z′)]=12E_z _C[T_J(z)]=E_z,z _C[J(z,z )]= 12. Final Surrogate Objective. Combining Lemmas 2.3–2.3, our RL objective maximizes the following lower bound on the KL divergence: |z∼πθ,z′∼πC[J(z,z′)]−12|. |E_z _θ,\,z _C [J(z,z ) ]- 12 |. (7) Empirically, we observe that traces from the active policy πθ _θ typically exhibit higher quality than the contaminated baseline (i.e., z∼πθ,z′∼πC[J(z,z′)]≥1/2E_z _θ,\,z _C[J(z,z )]≥ 1/2; as detailed in Section 4.3). Therefore, maximizing the absolute value in Eq. (7) is equivalent to maximizing the expectation alone, and we estimate it via an unbiased Monte Carlo reward: rHIPPO(z;x):=1K∑k=1KJ(z,z(k)′)→K→∞TJ(z),r_HIPPO(z;x):= 1K _k=1^KJ (z,z _(k) ) []K→∞T_J(z), (8) where z(k)′∼πC(⋅∣x)z _(k) _C(· x) are K independent trajectories sampled from the contaminated policy. This formulation completely reduces the pathological KL target into a bounded, variance-reduced stable pairwise signal. See more lemma prove and learning objective analysis in Appendix A. 2.4 Practical Instantiation of HIPPO We detail the practical instantiation of the theoretical learning objective during RL training. We first instantiate the contaminated shortcut-reliant policy πC _C. Inspired by observations that prompt injections can reliably induce specific model behaviors (e.g., unfaithful reasoning or stylistic biases) (Marioriyad et al. 2025; Young 2026), we model the shortcut-reliant policy via an answer-level hint injection, denoted as πH(⋅∣x,h) _H(· x,h). Specifically, we inject the ground-truth answer h into the prompt prior to the reasoning generation. This intervention has been empirically shown to effectively elicit the shortcut-reliant behaviors (Marioriyad et al. 2025; Young 2026). Next, we instantiate the pairwise judge J(za,zb)J(z_a,z_b) by comparing standard rollouts against these hint-injected samples. As illustrated in Figure 2, for each rollout z∼πθ(⋅∣x)z _θ(· x) that yields a correct final answer, we sample K hinted responses zh(k)∼πH(⋅∣x,h)z_h^(k) _H(· x,h) under the same model and conduct pairwise comparisons. Recalling that the supremum in Lemma 2.3 is attained when the reward function evaluates to a 0,1\0,1\ binary score, we strictly binarize the continuous preference signal: win rate exceeding a predefined threshold (0.50.5) are mapped to 11, while all others are strictly assigned 0. Finally, this pairwise reward is integrated with standard answer accuracy and format reward to form a comprehensive signal, penalizing flawed reasoning while suppressing shortcut-driven correct traces, ensuring that the policy is rewarded only for genuine, step-by-step deduction. Further details and discussions about reward configuration are provided in Appendix C and D. 3 Experimental Setup Implementation Details We evaluate HIPPO on two representative domains: mathematical and medical reasoning. Training data is curated from DeepScaleR (Luo et al. 2025) for math and MedQA (Jin et al. 2019) for medicine. We filter out trivial instances that the base model can solve in over 50% of its trials, to ensure that the model has not yet mastered the genuine reasoning paths. This filtering yields about 5,000 instances for each domain. To deliberately simulate Pre-RL data overlap, ensuring the model acquires prior knowledge of the answer without the underlying deduction, we then conduct SFT using solely the final answers as targets, supplemented by a 20% subset of data with rationales to prevent reasoning collapse. Finally we perform domain-specific RL training with the same data, to assess the performance under Pre-RL data overlap. We adopt Qwen2.5-7B-Instruct and Qwen3-4B as our backbone. Evaluation Datasets We evaluate on both in-domain and out-of-domain (OOD) tasks to disentangle shortcut-driven gains from genuine reasoning improvements. For math, we evaluate on DeepScaleR (Luo et al. 2025), MATH-500 (Lightman et al. 2023), MMLU-Pro math subset (Wang et al. 2024), and CARP-EN (Zhang et al. 2023). For medicine, we utilize MedQA (Jin et al. 2019), MedMCQA (Pal et al. 2022), the MMLU-Pro Health and Biology subsets, and the GPQA Genetics and Molecular Biology subsets (Rein et al. 2024). OOD generalization is measured on TheoremQA (Chen et al. 2023) together with the remaining MMLU-Pro subsets. To mitigate randomness, we sample 4 responses per question and report the average accuracy. Baselines To systematically disentangle the sources of performance gains and validate our theoretical reduction, we compare HIPPO against two categories of baselines: (i) Standard Training References (SFT-only, SFT with outcome-based RL). These models bound the baseline contribution of the RL phase itself and quantify the gap between contamination-driven memorization and our shortcut-resistant training. (i) Reward-Shaping Alternatives, including: RL with Pointwise RM employs a standard absolute scalar reward model (standard PRM) to evaluate individual trajectory quality independently; Pref-GRPO (Wang et al. 2026c) incorporates a pairwise preference framework, comparing and optimizing directly between all the rollout sequences; Privileged Pairwise Self-Play (SP3F) (Sutawika et al. 2026) further augments the pairwise judge model with privileged information (i.e., gold rationales). These baselines quantify the specific superiority of pairwise RM and our hint-injection mechanism. 4 Results and Analysis 4.1 Main Results We evaluate HIPPO’s effectiveness in Table 1 and Table 2 against all baselines across benchmarks. Superior and Generalizable Performance. HIPPO achieves consistent improvements over the base model and demonstrates superior performance against standard RL and all strong reward-shaping baselines. Moreover, the reasoning capabilities learned by HIPPO are highly generalizable to broader, general-purpose tasks. Despite training on mathematical questions, our approach shows substantial gains on out-of-distribution general reasoning benchmarks. Enhanced Reasoning Quality. Beyond accuracy, our approach elevates the response quality. As shown in Figure 3, while standard RL degrades reasoning quality under Pre-RL data overlap, HIPPO consistently produces more rigorous and coherent rationales across various backbone LLMs. To fully unpack the mechanisms driving these gains, we structure our subsequent analyses by first validating our hint-injection paradigm as a reliable proxy for Pre-RL data overlap through fine-grained quality analysis, then demonstrating its substantial benefits on optimization stability and scaling, and finally mechanistically interpreting the reasoning degradation induced by Pre-RL data overlap. Method Mathematical Reasoning General Reasoning Avg. DeepScaleR MATH-500 CARP-EN TheoremQA MMLU-Pro Standard Training Qwen2.57B_7B 10.1 12.1 41.9 28.3 33.0 25.1 + SFT 38.8 67.2 55.6 39.6 53.0 50.8 + RL 45.2 68.2 58.7 42.4 55.4 54.0 Reward-Shaping PRM RL 45.5 66.2 56.1 40.1 56.0 52.8 Pref-GRPO 42.2 68.3 58.9 40.7 56.8 53.4 SP3F 46.0 68.0 54.1 38.2 56.1 52.5 !20 Ours HIPPO 46.2 69.3 59.3 44.1 58.8 55.5 Table 1: Performance of the model trained on DeepScaleR across mathematical tasks and out-of-domain general reasoning benchmarks. Best scores are bold, with the second underlined. Results for Qwen34B_4B are detailed in Table 8 in Appendix D. Method MedQA MedMCQA MMLU-Pro GPQA Avg. Health Biology Genetics MolBio Qwen2.57B_7B 49.4 53.8 51.3 65.0 21.9 47.9 48.2 + SFT 49.5 48.4 48.2 67.0 26.6 44.4 47.4 + RL 61.2 55.1 57.5 74.5 25.0 44.8 53.0 PRM RL 57.7 52.0 41.1 65.7 21.8 48.7 47.8 Pref-GRPO 56.6 51.8 52.9 71.7 25.0 41.1 49.9 SP3F 61.1 56.0 55.4 68.0 28.0 45.6 52.4 !20 HIPPO 61.7 57.0 56.8 75.0 32.8 48.8 55.4 Table 2: Performance of the model trained on MedQA. Results for Qwen34B_4B are detailed in Table 9 in Appendix D. 4.2 Reasoning Degradation Analysis Under Pre-RL data overlap, standard RL degrades rationale quality. We first investigate the direct impact of Pre-RL data overlap on reasoning capabilities. We sample correct responses using the MedQA training data and conduct pairwise comparisons against responses from the SFT model to observe the performance shifts introduced by subsequent RL training. As illustrated in Figure 3, applying standard RL notably yields lower rationale quality than the SFT baseline on these overlapping data. This degradation indicates that standard RL fails to enhance genuine reasoning (see more detailed mechanistic discussion on the RL-induced shortcut retrieval behavior in Section 5). In contrast, HIPPO outperforms all baselines, demonstrating its capability to extract genuine reasoning rather than cultivating memorization. The reliability of using hint-injection sampling as a behavior proxy for Pre-RL data overlap. In Figure 3, applying hint-injection substantially degrades the quality of generated responses, consistent with prior findings (Chen et al. 2025; Marioriyad et al. 2025; Wang et al. 2026b). Beyond exhibiting a more severe drop in overall quality than that in standard RL under Pre-RL data overlap, we further delve into fine-grained quality characteristics to establish hint-injection as a more observable and comprehensive proxy. As shown in Table 3, using the correlation between reasoning error dimensions as an analytical lens, we find this correlation between most dimensions—measured by co-occurrence agreement—is significantly higher in our hint-injected paradigm than in standard RL failures. This indicates that errors are more coupled under hint-injection than natural Pre-RL data overlap: rationales tend to be either faultless or simultaneously riddled with multiple flaws. Therefore, hint-injection amplifies the flaw density of natural Pre-RL data overlap and significantly improves observability—detecting solely a fraction of these errors suffices for the judge to identify poor rationales, thus lowering evaluation difficulty and enhancing discriminability, as empirically demonstrated next. Figure 3: Response quality of different methods on MedQA overlapped data, measured via pairwise comparisons against the SFT baseline using a strong judge (Gemini-3.1-Pro). Metric Accuracy Knowledge Logic Assertion Accuracy - 0.67 0.74 0.76 Knowledge 0.55 - 0.70 0.73 Logic 0.70 0.66 - 0.84 Assertion 0.75 0.66 0.84 - Table 3: Average quality scores across dimensions evaluated by Gemini-3.1-Pro. The upper-right and lower-left triangles denote Hint-Injection and Standard RL, respectively (detailed criteria in Appendix C). Judge Model Pointwise Acc AUROC Precision@1 Llama3.18b 52.8 68.0 72.7 Qwen2.57b 53.9 73.2 86.9 Qwen34b 78.2 78.4 84.7 Table 4: Performance comparison of different Judge models. Evaluation metrics include judge accuracy in pointwise assessment, alongside AUROC and Precision@1 in pairwise evaluation (we set N=8N=8). 4.3 Why Hint-Injected Pairwise Rewards Work Efficacy of Small Open-Source LLMs in Pairwise Evaluation. To validate the reliability of small pairwise LLM judges, we sample correct reasoning traces together with their hinted counterparts on the validation set, and use a strong LLM (Gemini-3.1-pro) to establish ground-truth quality annotations. we then evaluate judge of varying size and architecture. As shown in Table 4, pointwise evaluation heavily relies on the capability of the judge model. Without sufficient capacity, small models degrade to near-random guessing, which explicitly explains the poor performance of our pointwise RM baseline. In contrast, pairwise judgment demonstrates superior stability across all evaluated models, consistently achieving high AUC-ROC and Precision@1 scores, which shows its effectiveness in both global preference ranking and precise top-quality trace identification. These properties are valuable for group-based RL optimization, as it helps provide a clear and stable gradient signal for policy alignment. Hint-Injection Enhances Discriminability and Transitivity. By serving as a contaminated reference anchor, the hinted reference simplifies the evaluation landscape, yielding more stable and consistent optimization. First, it sharpens the contrast between high- and low-quality traces. We examine this by comparing the pairwise scores derived from our hint-injected approach against a standard pairwise baseline (i.e., comparing pairs of vanilla rollout generations directly). As shown in Figure 4, the average distance between ground-truth high- and low-quality traces increases substantially under hint-injection, yielding a clearer separable boundary and a more informative reward signal. Second, hint-injected pairwise judging significantly mitigates the issue of intransitive preferences. As noted by previous works (Swamy et al. 2024; Xu et al. 2025), pointwise reward models are notoriously prone to it. While switching to pairwise comparison alleviates this issue (Sutawika et al. 2026; Wang et al. 2026c), our hint-injected approach further maximizes this mitigation. We quantify this issue via the Percentage of Non-Transitive (PNT) metric (Xu et al. 2025)—the fraction of trace triplets that violate transitivity. Figure 5 demonstrates that our hint-injected approach consistently achieves lower PNT than direct pairwise comparisons, thus substantially simplifying the preference learning problem. Moreover, our method exhibits a distinctly positive scaling trend—as model capacity increases, our approach continuously drives the PNT lower without bottlenecking, demonstrating significantly superior effectiveness and scalability. See Appendix B.1 for calculation details. Figure 4: Score comparison between the direct and hint-injected pairwise judges. Hint-injection effectively enlarges the decision boundary between high- and low-quality traces. Figure 5: Comparison of PNT statistics between the direct pairwise judge and our hint-injected pairwise judge. Our proposed method consistently exhibits better evaluation stability and overall performance across all tested models. 5 Exploring the Degradation Mechanism While the preceding analyses establish that Pre-RL data overlap degrades reasoning reliability and generalizability, its driving mechanism remains underexplored. We thus propose a hypothesis: familiar questions trigger memorized answer retrieval rather than genuine deduction. Consequently, RL optimizes for memory exploitation over reasoning enhancement. In this section, we validate this hypothesis by demonstrating that Pre-RL data overlap indeed triggers premature answer retrieval, reducing the subsequent reasoning to mere post-hoc rationalizations. Experimental Setup. We design two pilot experiments. First, we examine the internal representations of the model under Pre-RL data overlap using the Logit Lens (Nostalgebraist 2020; Belrose et al. 2025). By directly projecting intermediate hidden states from all layers into the vocabulary space across the entire generation sequence, we verify whether the final answer token becomes identifiable at shallow layers or during the early beginning of token generation. Either scenario implies that the target answer is accessible prior to substantial reasoning, signaling a shortcut. Second, we employ the Independent Causal Mechanisms (ICM) principle (Janzing et al. 2012; Schoelkopf et al. 2012; Jin et al. 2021) to analyze the generative information flow under overlap. ICM postulates that the true causal direction C→EC→ E minimizes Kolmogorov complexity, satisfying K(PC)+K(PE|C)≤K(PE)+K(PC|E)K(P_C)+K(P_E|C)≤ K(P_E)+K(P_C|E), which means the non-causal direction contains redundant information. Since true Kolmogorov complexity is uncomputable, we follow previous works (Burden et al. 2024) and approximate K(⋅)K(·) using negative token log-likelihoods. We evaluate the complexity difference between the two generative directions—[K(y)+K(z∣y)]−[K(z)+K(y∣z)][K(y)+K(z y)]-[K(z)+K(y z)]—across the base, SFT, and SFT+RL models, to reveal the causal direction shifts the reasoning z and the final answer y. Detailed theoretical justifications and experimental settings are provided in Appendix B.3. Figure 6: Logit Lens visualizations of answer probability across hidden layers and generation steps. Model MATH-500 DeepScaleR A+R|A R+A|R A+R|A R+A|R Qwen2.57B 63.22 49.50 13.72 71.42 56.58 14.84 + SFT 96.80 89.88 6.92 104.86 98.78 6.08 + RL 90.45 86.17 4.28 90.85 89.92 0.93 Table 5: Estimated complexity on overlapping data (DeepScaleR training set) compared to a uncontaminated dataset (MATH-500). Results The Logit Lens visualizations in Figure 6 reveal a behavioral divergence after the model undergoes RL. Specifically, we observe a dramatic increase in the target answer’s probability within exceptionally shallow hidden layers. This phenomenon indicates that the model forms an internal representation of the final answer well before executing any meaningful latent computation. Moreover, this shallow-layer prominence persists throughout the entire generation process, providing clear evidence that the model anchors its output to a pre-retrieved answer rather than deriving it sequentially. This observation challenges the authenticity and functional necessity of the generated reasoning chain. The result of causal information flow, shown in Table 5, further confirms the model’s already known the answer before reasoning. On the non-overlapping MATH-500 dataset, the causal direction remains robustly aligned with z→yz→ y, as evidenced by the large positive Δ values across all training stages, indicating that reasoning naturally precedes the conclusion. However, on the overlapping DeepScaleR data, this complexity gap collapses precipitously after RL training, dropping to almost zero (0.930.93). This structural collapse reflects a shift from a strictly forward reasoning process toward a mixed causal dynamic: while the model still genuinely resolves half of the instances, the other half now clearly exhibits a reversed causal flow: the model anchors on the answer-retrieval shortcut, relegating the subsequent chain-of-thought to a fabricated, post-hoc rationalization. Together, these empirical findings validate our hypothesis, demonstrating that Pre-RL data overlap inherently drives RL to replace sequential logical deduction with shortcut-driven memory retrieval. 6 Related Works RL for LLM Reasoning While RL has driven remarkable leaps in LLM reasoning, it carries inherent risks. Without explicit reasoning monitoring, RL can stagnate performance or even induce detrimental behaviors, such as logical inconsistency (Lin et al. 2025), unfaithfulness (Chen et al. 2025), or collapsing into mere memory retrieval due to reward breakdown (Wu et al. 2025). To monitor the reasoning process, various Process Reward Models have been proposed. Although large-scale commercial models deliver state-of-the-art evaluation, they incur prohibitive costs (Chen et al. 2024; Zhang et al. 2026). Alternatively, optimizing the evaluation paradigm can enable much smaller LLMs to serve as PRMs. Pairwise reward modeling, a prominent advancement in this vein, effectively addresses the instability, intransitivity, and miscalibration that plague standard PRMs (Swamy et al. 2024). Furthermore, it better aligns with human annotation behavior by evaluating relative preferences between ambiguous trajectories (Wang et al. 2026c; Zhang et al. 2026). Building upon this pairwise framework, our method introduces hint-injected generation as an explicit anchor. This design further stabilizes the optimization process and yields clearer, more discriminable reward signals, particularly under the challenge of Pre-RL data overlap. Data Overlap Across Training Stages Previous studies mostly adopt a training–evaluation lens, focusing on whether training corpora leak into test benchmarks (Cheng et al. 2025). Within the context of pre-training and SFT, this perspective has driven the development of output-distribution detectors (Dong et al. 2024), paraphrase-evasive probes (Lan et al. 2026), and competition-math audits (balunović2026matharenaevaluatingllmsuncontaminated), alongside stress tests revealing that such detection mechanisms often falter on modern reasoning models (Wang et al. 2026a). Other works extend this paradigm to RL training, investigating whether exposure to specific RL-stage data spuriously inflates benchmark scores (Tao et al. 2026; Wu et al. 2026; Kocyigit and Yildirim 2026). While existing analyses remain rigidly anchored to evaluation leakage, we identify a critical yet underexplored regime where pre-RL corpora act as the contamination source, with the RL stage itself serving as the vulnerable sink. 7 Conclusion We identify the Pre-RL data overlap problem which severely undermines RL efficacy by driving models to exploit memorization shortcuts. We introduce HIPPO, a novel framework that theoretically reformulates an intractable divergence objective into a stable, pairwise-rewarded surrogate. By utilizing hint-injected aggregation to deliberately trigger shortcut behaviors, the resulting traces naturally serve as explicit anchors for pairwise comparison. HIPPO yields a highly discriminable, stable, and learnable preference signal, ultimately achieving superior reasoning capabilities and strong generalizability. References A. Albalak, D. Phung, N. Lile, R. Rafailov, K. Gandhi, L. Castricato, A. Singh, C. Blagden, V. Xiang, D. Mahan, and N. Haber (2025) Big-math: a large-scale, high-quality math dataset for reinforcement learning in language models. External Links: 2502.17387, Link Cited by: Appendix E. N. Belrose, I. Ostrovsky, L. McKinney, Z. Furman, L. Smith, D. Halawi, S. Biderman, and J. Steinhardt (2025) Eliciting latent predictions from transformers with the tuned lens. External Links: 2303.08112, Link Cited by: §5. J. Burden, M. Cebrian, and J. Hernandez-Orallo (2024) Conversational complexity for assessing risk in large language models. External Links: 2409.01247, Link Cited by: §B.3, §5. J. Chen, Z. Cai, K. Ji, X. Wang, W. Liu, R. Wang, J. Hou, and B. Wang (2024) HuatuoGPT-o1, towards medical complex reasoning with llms. External Links: 2412.18925, Link Cited by: §6. W. Chen, M. Yin, M. Ku, P. Lu, Y. Wan, X. Ma, J. Xu, X. Wang, and T. Xia (2023) TheoremQA: a theorem-driven question answering dataset. External Links: 2305.12524, Link Cited by: §3. Y. Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schulman, A. Somani, P. Hase, M. Wagner, F. Roger, V. Mikulik, S. R. Bowman, J. Leike, J. Kaplan, and E. Perez (2025) Reasoning models don’t always say what they think. External Links: 2505.05410, Link Cited by: §1, §4.2, §6. Y. Cheng, Y. Chang, and Y. Wu (2025) A survey on data contamination for large language models. External Links: 2502.14425, Link Cited by: §6. J. Chua and O. Evans (2025) Are deepseek r1 and other reasoning models more faithful?. External Links: 2501.08156, Link Cited by: §1. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: Appendix E. Y. Dong, X. Jiang, H. Liu, Z. Jin, B. Gu, M. Yang, and G. Li (2024) Generalization or memorization: data contamination and trustworthy evaluation for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 12039–12050. External Links: Link, Document Cited by: §6. Y. Dong, X. Jiang, Y. Tao, H. Liu, K. Zhang, L. Mou, R. Cao, Y. Ma, J. Chen, B. Li, Z. Jin, F. Huang, Y. Li, and G. Li (2026) RL-plus: countering capability boundary collapse of llms in reinforcement learning with hybrid-policy optimization. External Links: 2508.00222, Link Cited by: §1. L. Gao, J. Schulman, and J. Hilton (2023) Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning (ICML), External Links: Link Cited by: §2.2. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1. J. He, J. Liu, C. Y. Liu, R. Yan, C. Wang, P. Cheng, X. Zhang, F. Zhang, J. Xu, W. Shen, S. Li, L. Zeng, T. Wei, C. Cheng, B. An, Y. Liu, and Y. Zhou (2025) Skywork open reasoner series. Note: Notion Blog External Links: Link Cited by: Appendix E. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. NeurIPS. Cited by: Appendix E. D. Janzing, J. Mooij, K. Zhang, J. Lemeire, J. Zscheischler, P. Daniušis, B. Steudel, and B. Schölkopf (2012) Information-geometric approach to inferring causal directions. Artif. Intell. 182–183, p. 1–31. External Links: ISSN 0004-3702, Link, Document Cited by: §B.3, §5. Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu (2019) PubMedQA: a dataset for biomedical research question answering. External Links: 1909.06146, Link Cited by: §3, §3. Z. Jin, J. von Kügelgen, J. Ni, T. Vaidhya, A. Kaushal, M. Sachan, and B. Schölkopf (2021) Causal direction of data collection matters: implications of causal and anticausal learning for NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Online and Punta Cana, Dominican Republic, p. 9499–9513. External Links: Link, Document Cited by: §B.3, §5. M. Y. Kocyigit and C. Yildirim (2026) The impact of post-training on data contamination. External Links: 2601.06103, Link Cited by: §6. A. Kumar, V. Zhuang, R. Agarwal, Y. Su, J. D. Co-Reyes, A. Singh, K. Baumli, S. Iqbal, C. Bishop, R. Roelofs, L. M. Zhang, K. McKinney, D. Shrivastava, C. Paduraru, G. Tucker, D. Precup, F. Behbahani, and A. Faust (2024) Training language models to self-correct via reinforcement learning. External Links: 2409.12917, Link Cited by: Appendix G. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, Cited by: §C.1. Y. Lan, Y. Cao, H. Wang, L. Lin, and J. Chen (2026) The illusion of reasoning: exposing evasive data contamination in llms via zero-cot truncation. External Links: 2605.21856, Link Cited by: §6. J. LI, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. C. Huang, K. Rasul, L. Yu, A. Jiang, Z. Shen, Z. Qin, B. Dong, L. Zhou, Y. Fleureau, G. Lample, and S. Polu (2024) NuminaMath. Numina. External Links: Link Cited by: Appendix E. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023) Let’s verify step by step. arXiv preprint arXiv:2305.20050. Cited by: §3. J. Lin, C. Jiang, Z. Wu, J. Sun, and Y. Feng (2025) CLARity: reasoning consistency alone can teach reinforced experts. External Links: 2510.09278, Link Cited by: §1, §6. Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin (2025) Understanding r1-zero-like training: a critical perspective. External Links: 2503.20783, Link Cited by: §2.2. M. Luo, S. Tan, J. Wong, X. Shi, W. Y. Tang, M. Roongta, C. Cai, J. Luo, T. Zhang, L. E. Li, R. A. Popa, and I. Stoica (2025) DeepScaleR: surpassing o1-preview with a 1.5b model by scaling rl. Note: Notion Blog External Links: Link Cited by: Appendix E, Figure 1, §3, §3. S. Mahdavi, M. Li, K. Liu, C. Thrampoulidis, L. Sigal, and R. Liao (2025) Leveraging online olympiad-level math problems for llms training and contamination-resistant evaluation. External Links: 2501.14275, Link Cited by: Appendix E. A. Marioriyad, S. Adim, N. Alighardashi, M. S. Banghshah, and M. H. Rohban (2025) Unspoken hints: accuracy without acknowledgement in llm reasoning. External Links: 2509.26041, Link Cited by: §1, §2.4, §4.2. X. Nguyen, M. J. Wainwright, and M. I. Jordan (2010) Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory 56 (11), p. 5847–5861. Cited by: §1, §2.3. Nostalgebraist (2020) Interpreting gpt: the logit lens. LessWrong. External Links: Link Cited by: §5. A. Pal, L. K. Umapathi, and M. Sankarasubbu (2022) MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, G. Flores, G. H. Chen, T. Pollard, J. C. Ho, and T. Naumann (Eds.), Proceedings of Machine Learning Research, Vol. 174, p. 248–260. External Links: Link Cited by: §3. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman (2024) GPQA: a graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, External Links: Link Cited by: §3. B. Schoelkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. Mooij (2012) On causal and anticausal learning. External Links: 1206.6471, Link Cited by: §B.3, §5. J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel (2018) High-dimensional continuous control using generalized advantage estimation. External Links: 1506.02438, Link Cited by: §A.4. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. ArXiv abs/1707.06347. External Links: Link Cited by: §2.1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §1, §2.1. L. Sutawika, G. Swamy, Z. S. Wu, and G. Neubig (2026) Gained in translation: privileged pairwise judges enhance multilingual reasoning. External Links: 2601.18722, Link Cited by: §1, §3, §4.3. G. Swamy, C. Dann, R. Kidambi, Z. S. Wu, and A. Agarwal (2024) A minimaximalist approach to reinforcement learning from human feedback. External Links: 2401.04056, Link Cited by: §1, §2.3, §4.3, §6. Y. Tao, T. Wang, Y. Dong, H. Liu, K. Zhang, X. Hu, and G. Li (2026) Detecting data contamination from reinforcement learning post-training for large language models. External Links: 2510.09259, Link Cited by: §1, §6. A. B. Tsybakov (2009) Introduction to nonparametric estimation. Springer Series in Statistics, Springer, New York. External Links: Document Cited by: §2.3. H. Wang, H. Li, B. Ko, and H. Zhang (2026a) On the fragility of benchmark contamination detection in reasoning models. External Links: 2510.02386, Link Cited by: §1, §6. X. Wang, J. Han, Z. Jiang, T. Li, J. Liang, S. Jiang, Z. Dai, S. Ma, F. Yu, and Y. Xiao (2026b) Don’t tell the answer, truly guide the reasoning during rl rollouts. External Links: 2510.09388, Link Cited by: §4.2. Y. Wang, Z. Li, Y. Zang, Y. Zhou, J. Bu, C. Wang, Q. Lu, C. Jin, and J. Wang (2026c) Pref-grpo: pairwise preference reward-based grpo for stable text-to-image reinforcement learning. External Links: 2508.20751, Link Cited by: §2.3, §3, §4.3, §6. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. (2024) Mmlu-pro: a more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574. Cited by: §3. F. Wu, A. Tu, W. Xuan, H. Qi, X. Huang, Q. Zeng, S. Talaei, Y. Xiao, P. Xia, X. Tang, Y. Zhuang, B. Hu, H. Cao, W. Shi, R. Yang, N. Liu, H. Yao, G. Liu, L. E. Li, A. Saberi, N. Yokoya, J. Leskovec, and Y. Choi (2026) Position: the hidden costs and measurement gaps of reinforcement learning with verifiable rewards. External Links: 2509.21882, Link Cited by: §6. M. Wu, Z. Zhang, Q. Dong, Z. Xi, J. Zhao, S. Jin, X. Fan, Y. Zhou, H. Lv, M. Zhang, Y. Fu, Q. Liu, S. Zhang, and Q. Zhang (2025) Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. External Links: 2507.10532, Link Cited by: Appendix E, §1, §1, §6. Y. Xu, L. Ruis, T. Rocktäschel, and R. Kirk (2025) Investigating non-transitivity in llm-as-a-judge. External Links: 2502.14074, Link Cited by: §B.1, §B.1, §1, §2.3, §4.3. L. Yan, R. Li, G. Chen, Q. Li, J. Geng, W. Li, V. Wang, and C. Lee (2026) Spurious rewards paradox: mechanistically understanding how rlvr activates memorization shortcuts in llms. External Links: 2601.11061, Link Cited by: §1. R. J. Young (2026) Lie to me: how faithful is chain-of-thought reasoning in reasoning models?. External Links: 2603.22582, Link Cited by: §1, §2.4. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang (2025) DAPO: an open-source llm reinforcement learning system at scale. External Links: 2503.14476, Link Cited by: §1, §2.1, §2.2. Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, and G. Huang (2025) Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. External Links: 2504.13837, Link Cited by: §1. W. Zeng, Y. Huang, Q. Liu, W. Liu, K. He, Z. Ma, and J. He (2025) SimpleRL-zoo: investigating and taming zero reinforcement learning for open base models in the wild. External Links: 2503.18892, Link Cited by: Appendix E. B. Zhang, K. Zhou, X. Wei, W. X. Zhao, J. Sha, S. Wang, and J. Wen (2023) Evaluating and improving tool-augmented computation-intensive math reasoning. arXiv preprint arXiv:2306.02408. Cited by: §3. Q. Zhang, B. Chen, F. Zhang, R. Ding, S. Wang, Q. Wang, Y. Huang, H. Zhang, R. Zhu, P. Wang, A. Ren, X. Li, P. Xie, J. Liu, N. Guo, J. Zhou, and Z. Zha (2026) ArenaRL: scaling rl for open-ended agents via tournament-based relative ranking. External Links: 2601.06487, Link Cited by: §2.3, §6. R. Zhao, A. Meterez, S. Kakade, C. Pehlevan, S. Jelassi, and E. Malach (2025) Echo chamber: rl post-training amplifies behaviors learned in pretraining. External Links: 2504.07912, Link Cited by: §1. Appendix A Proofs A.1 Proof of Lemma 2.3 (Pinsker’s inequality) We prove DKL(p∥q)≥2DTV2(p,q)D_KL(p\|q)≥ 2\,D_TV^2(p,q) in two steps: (i) reduction to a Bernoulli case via the data-processing inequality, and (i) direct verification for Bernoulli distributions. Step (i): Reduction. Let A⋆=z:p(z)≥q(z)A =\z:p(z)≥ q(z)\ and define f(z):=z∈A⋆f(z):=1\z∈ A \. The pushforward distributions P:=f#pP:=f_\#p and Q:=f#qQ:=f_\#q are Bernoulli on 0,1\0,1\: P=Ber(α)P=Ber(α), Q=Ber(β)Q=Ber(β) with α:=p(A⋆)α:=p(A ), β:=q(A⋆)β:=q(A ). By the data-processing inequality for KL divergence, DKL(p∥q)≥DKL(P∥Q)D_KL(p\|q)≥ D_KL(P\|Q). Meanwhile, DTV(P,Q)=|α−β|=p(A⋆)−q(A⋆)=DTV(p,q)D_TV(P,Q)=|α-β|=p(A )-q(A )=D_TV(p,q), where the last equality follows from Lemma 2.3 with the optimal witness T⋆=A⋆T =1_A . Hence it suffices to prove DKL(P∥Q)≥2DTV2(P,Q)D_KL(P\|Q)≥ 2\,D_TV^2(P,Q) for the Bernoulli case. Step (i): Bernoulli case. We need to show that for all α,β∈(0,1)α,β∈(0,1), g(α,β):=αlog⁡αβ+(1−α)log⁡1−α1−β−2(α−β)2≥ 0.g(α,β)\;:=\;α αβ+(1-α) 1-α1-β-2(α-β)^2\;≥\;0. (9) Fix β and regard g as a function of α. Then: • g(β,β)=0g(β,β)=0. • ∂g∂α|α=β=log⁡β−log⁡1−β1−β−4(β−β)=0 ∂ g∂α |_α=β= β- 1-β1-β-4(β-β)=0. • ∂2g∂α2=1α+11−α−4=1α(1−α)−4≥ 4−4=0 ∂^2g∂α^2= 1α+ 11-α-4= 1α(1-α)-4\;≥\;4-4=0, where the inequality uses α(1−α)≤1/4α(1-α)≤ 1/4. Since g is convex in α with g(β,β)=0g(β,β)=0 and ∂αg|α=β=0 _αg|_α=β=0, we conclude g(α,β)≥0g(α,β)≥ 0 for all α∈(0,1)α∈(0,1). □ A.2 Proof of Lemma 2.3 (Variational representation of TV) We show DTV(p,q)=supT:→[0,1](p[T]−q[T])D_TV(p,q)= _T:Z→[0,1] (E_p[T]-E_q[T] ). Upper bound. For any T:→[0,1]T:Z→[0,1], p[T]−q[T]=∑zT(z)(p(z)−q(z)).E_p[T]-E_q[T]= _zT(z) (p(z)-q(z) ). (10) Partition the sum over A⋆:=z:p(z)≥q(z)A :=\z:p(z)≥ q(z)\ and A⋆c:=z:p(z)<q(z)A c:=\z:p(z)<q(z)\: p[T]−q[T]=∑z∈A⋆T(z)(p(z)−q(z))⏟≥ 0+∑z∈A⋆cT(z)(p(z)−q(z))⏟< 0. splitE_p[T]-E_q[T]= _z∈ A T(z) (p(z)-q(z) )_≥\,0\\ + _z∈ A cT(z) (p(z)-q(z) )_<\,0. split (11) Applying T(z)≤1T(z)≤ 1 on A⋆A and T(z)≥0T(z)≥ 0 on A⋆cA c: p[T]−q[T]≤∑z∈A⋆(p(z)−q(z))=p(A⋆)−q(A⋆)≤DTV(p,q), splitE_p[T]-E_q[T]\;≤\; _z∈ A (p(z)-q(z) )\\ =p(A )-q(A )≤\;D_TV(p,q), split (12) where the last inequality follows from the definition DTV(p,q)=supA|p(A)−q(A)|D_TV(p,q)= _A|p(A)-q(A)|. Achievability. Setting T=A⋆T=1_A yields p[A⋆]−q[A⋆]=p(A⋆)−q(A⋆)E_p[1_A ]-E_q[1_A ]=p(A )-q(A ). By the Hahn decomposition, DTV(p,q)=p(A⋆)−q(A⋆)D_TV(p,q)=p(A )-q(A ), so the supremum is attained. □ A.3 Proof of Lemma 2.3 (Symmetry pins the baseline) Under q⊗q q, the variables z and z′z are exchangeable, so z,z′∼q[J(z,z′)]=z,z′∼q[J(z′,z)]=z,z′∼q[1−J(z,z′)]=1−z,z′∼q[J(z,z′)], splitE_z,z q[J(z,z )]=E_z,z q[J(z ,z)]\\ =E_z,z q[1-J(z,z )]\\ =1-E_z,z q[J(z,z )], split (13) whence z,z′∼q[J(z,z′)]=1/2E_z,z q[J(z,z )]=1/2. □ Practical realization. LLM-as-judge implementations of J exhibit non-negligible position bias and therefore violate J(z,z′)+J(z′,z)=1J(z,z )+J(z ,z)=1 in general. We restore symmetry at the implementation level by averaging the two orderings, J~(a,b):=12(J(a,b)+1−J(b,a)), J(a,b)\;:=\; 12 (J(a,b)+1-J(b,a) ), (14) which satisfies J~(a,b)+J~(b,a)=1 J(a,b)+ J(b,a)=1 by construction. All downstream identities are stated for the symmetrized comparator J~ J; for readability we write J in place of J~ J hereafter. A.4 More Analysis on Learning Objectives We compare the variance of the pairwise reward rHIPPOr_HIPPO (Eq. (8)) with that of the per-sample KL reward r∗(z)=log⁡[πθ(z∣x)/πH(z∣x,h)]r^*(z)= [ _θ(z x)/ _H(z x,h)]. KL reward: variance grows with trajectory length. Under autoregressive decomposition, r∗(z)=∑t=1TΔtr^*(z)= _t=1^T _t where Δt=log⁡πθ(zt∣x,z<t)πH(zt∣x,h,z<t) _t= _θ(z_t x,z_<t) _H(z_t x,h,z_<t) and T=|z|T=|z|. When the token-level log-ratios have a common positive variance lower bound σ2>0σ^2>0 and non-negative correlations—a reasonable assumption when πθ _θ and πH _H diverge consistently across steps—we have Var[r∗]≥Tσ2Var[r^*]≥ Tσ^2, scaling at least linearly with trajectory length (Schulman et al. 2018). Pairwise reward: variance is uniformly bounded. Since J∈[0,1]J∈[0,1], the reward rHIPPO∈[0,1]r_HIPPO∈[0,1] as well. By Popoviciu’s inequality, any random variable supported on [a,b][a,b] satisfies Var[X]≤(b−a)2/4Var[X]≤(b-a)^2/4, so Var[rHIPPO]≤14,Var[r_HIPPO]\;≤\; 14, (15) independent of trajectory length T. More precisely, by the law of total variance, Var[rHIPPO]=z[Varz[rdiv∣z]]⏟≤ 1/(4K)+Varz[TJ(z)]⏟≤ 1/4.Var[r_HIPPO]= E_z\! [Var_\z\[r_div z] ]_≤\;1/(4K)+ Var_z[T_J(z)]_≤\;1/4. (16) The first term is the Monte Carlo estimation noise, which vanishes as K→∞K→∞; the second is the irreducible variance due to the variability of z under TJT_J, bounded by 1/41/4 since TJ∈[0,1]T_J∈[0,1]. Appendix B Evaluation Details B.1 Percentage of Non-Transitive cases (PNT) We follow the same set of non-transitivity criteria as Xu et al. (2025); the complete enumeration of violation patterns is given in their appendix and we do not reproduce it here. For the direct (unhinted) baseline, we apply this protocol verbatim: every triple of rollouts drawn for the same prompt is compared by the judge model, and the fraction of triples that exhibit any of the violation patterns is reported as PNT. In our hint-injected setting, the protocol is adapted on a single axis: the middle element of each triple is a hinted rollout rather than an unhinted one. The two edges involving the hinted reference are still decided by the judge model, but the relative quality between the two outer (unhinted) rollouts is determined by their pairwise scores against the hint set rather than by a separate judge query. All other counting rules follow Xu et al. (2025) unchanged. B.2 Gold Quality Score Throughout the analyses in Section 4.2, Section 4.3 and Appendix B.1, the gold scalar score r(z)r(z) is obtained by prompting a stronger judge model (Gemini-3.1-pro) to assign a [0,1][0,1] score on each of a fixed set of quality dimensions. A trace is labeled good only if it receives a passing score on every dimension; otherwise it is labeled bad. The full prompt template is documented in our released code. B.3 Causal Analysis: Detailed Setup This section provides the formal background and the experimental protocol omitted from Section 5. Independent causal mechanisms. The diagnostic in Section 5 rests on the principle of independent causal mechanisms (ICM) (Janzing et al. 2012; Schoelkopf et al. 2012; Jin et al. 2021), which posits that the causal generative process of a system’s variables decomposes into autonomous modules that carry no shared information. A direct corollary is that, for any pair of variables (C,E)(C,E) standing in a true causal relation C→EC→ E, the Kolmogorov complexity of the joint distribution admits the asymmetric factorization K(PC,E)=K(PC)+K(PE∣C)≤K(PE)+K(PC∣E),K(P_C,E)\;=\;K(P_C)+K(P_E C)\;≤\;K(P_E)+K(P_C E), (17) because the causal factorization is built from two independent modules (PCP_C and the mechanism PE∣CP_E C), whereas the anti-causal factorization mixes them and typically carries redundant information. The sign of the gap Δ:=[K(PE)+K(PC∣E)]−[K(PC)+K(PE∣C)] := [K(P_E)+K(P_C E) ]- [K(P_C)+K(P_E C) ] therefore identifies the causal direction. Token-probability surrogate. Kolmogorov complexity is uncomputable. Following Burden et al. (2024), we approximate it by the negative log-likelihood under the policy: K(t∣context)≈−∑i=1|t|log⁡pθ(ti|t<i,context).K(t )\;≈\;- _i=1^|t| p_θ (t_i\, |\,t_<i,context ). (18) Substituting Eq. (18) into Eq. (17) with C instantiated by the reasoning trace R and E by the answer A, we obtain the empirical estimator Δ^(x)=[K(A∣x)+K(R∣A,x)]−[K(R∣x)+K(A∣R,x)], (x)\;=\; [K(A x)+K(R A,x) ]- [K(R x)+K(A R,x) ], (19) which is positive when the data are best explained by R→AR→ A and negative under A→RA→ R. Experimental protocol. For each policy under inspection (Vanilla, SFT-only, SFT-then-RL) and each evaluation set (contaminated, OOD), we sample n rollouts per prompt, retain only those whose final answer is correct, and compute Δ^(x) (x) on each retained instance. The per-instance gaps are then averaged within the eligible subset, so that the reported Δ in Table 5 represents the mean causal-direction gap conditional on answer correctness. Restricting to correct samples removes the confound that incorrect traces may exhibit artificially low K(A∣R)K(A R) for reasons unrelated to causal structure. Reward Variant MedQA (val) Soft reward (raw win-rate) 58.3 Hard reward (αdiv=0.125 _div=0.125) 62.0 Hard reward (αdiv=0.25 _div=0.25) 61.7 Table 6: Hard binarized reward outperforms the soft (win rate) variant, in line with our theoretical analysis. Appendix C More Implementation Details C.1 Training Configuration We implement HIPPO on top of the VeRL library (Kwon et al. 2023), running on a single node with 8×8× NVIDIA A800 GPUs. We use weights 1.01.0 for the answer-correctness term, 1.01.0 for the format term, and 0.250.25 for the hint-injected pairwise term; additional ablations on the pairwise reward configuration are reported in Appendix D.2. At each training step we draw K=8K=8 hinted rollouts per prompt to estimate rHIPPOr_HIPPO. The remaining training hyperparameters are listed in Table 7. Category Parameter Value General Advantage estimator GRPO Gamma (γ) 1 Lambda (λ) 1 Batch size 128 Max prompt length 1024 Gradient checkpointing Enabled Actor Learning rate 1×10−61× 10^-6 Mini-batch size 1024 Dynamic batch size Enabled KL penalty role Loss KL loss type Low-variance KL KL loss coefficient (β) 0.001 Entropy coefficient 0.001 Clip ratio 0.2 Gradient clipping 1.0 Sequence parallel size Model-specific Rollout Backend vLLM Tensor model parallel size 1 Rollouts per sample 8 Nucleus sampling p 1.0 GPU memory utilization 0.6 Sampling temperature 1.0 Qwen2.5-7B Max response length 2048 Table 7: Training configuration for HIPPO. We use VeRL version 0.5. Model DeepScaleR MATH-500 CARP-EN TheoremQA Avg. Qwen34B 27.2 49.6 46.2 29.1 38.0 + SFT 39.5 60.2 48.1 31.8 44.9 HIPPO 62.1 74.6 57.4 51.2 61.3 Table 8: Results on Qwen3-4B trained on DeepScalR. HIPPO improves both in-domain and out-of-domain accuracy, demonstrating backbone-agnostic gains. Model MedQA MedMCQA MMLU-Pro Avg. Health Bio Qwen34B 57.2 62.2 38.1 50.9 52.1 + SFT 51.6 49.7 46.8 72.6 55.2 HIPPO 58.1 53.3 56.7 78.8 61.7 Table 9: Results on Qwen3-4B trained on MedQA. RL Dataset / Family Direct Upstream Sources SimpleRL-Zoo GSM8K (train) + MATH (train) Big-Math-RL-Verified MATH + GSM8K + AMC/AIME + AoPS DeepScaleR AIME (1984–2023) + AMC (≤2023≤ 2023) + Omni-MATH NuminaMath Series (OpenR1, Skywork-OR1) AoPS + MATH + GSM8K + AMC/AIME Table 10: Summary of Pre-RL data overlap lineages in widely adopted RL reasoning datasets. These datasets directly source from benchmarks that have been empirically proven to be exposed to modern base models prior to RL training. C.2 Prompts We document the prompt templates used in our pipeline: the rubric prompt that produces the dimension-wise quality labels for the gold score r(⋅)r(·) (Table 11), the pairwise-judge template used by both the reward judge J and the PNT analysis (Table 12), and the two rollout templates that materialize the unhinted policy πθ _θ and the hinted reference πH _H (Table 13). 1. Accuracy (Score: 1, 0, -1 or -2) Focus: Does the final answer match the correct answer? Score 1 (Correct): The answer is numerically or semantically equivalent to the correct answer. This includes minor formatting differences, unsimplified fractions, scientific notation, rounding to an acceptable range, or the use of equivalent values (e.g., if the model provides a decimal or fraction and the correct answer is an integer rounded from that decimal, this is acceptable). Score 0 (Incorrect): The answer is wrong, missing, or fundamentally different. Score -1 (Reference Error): If the correct answer is incomplete (e.g., a question asks for both maximum and minimum values, but only one value is provided), assign a −1-1 to indicate that the reference data requires review. 2. Knowledge (Score: 1 or 0) Focus: This dimension checks only for errors in memorized facts (e.g., value of constants, standard textbook formulas). It does not penalize misuse of valid knowledge. Score 1 (Correct): All external knowledge (such as mathematical formulas, physical constants, legal statutes, medical symptoms, etc.) used is theoretically correct (e.g., stating E=mc2E=mc^2 or π≈3.14159π≈ 3.14159). Note 1: If the model uses a correct formula (e.g., V=43πr3V= 43π r^3 for the volume of a sphere), even if the formula is not relevant to the problem, this still counts as 1 for Knowledge. We care only about whether the factual knowledge is correct, not whether it is applied in the right context. Note 2: If the model writes the correct formula but calculates it wrong or substitutes the wrong value into the formula, this is still Score 1 for Knowledge. Note 3: If the model hallucinates a specific equation for the problem (e.g., “We assume x+y=10x+y=10”) that turns out to be invalid for the scenario, this is Score 1 here. It is an assumption failure rather than a Knowledge error unless it is a universal theorem. Score 0 (Error): The model hallucinates a false universal truth, such as a non-existent formula or remembering a constant incorrectly (e.g., stating “The sum of angles in a triangle is 360 degrees”, attributing a historical event to the wrong century). 3. Calculation (Score: 1 or 0) Focus: Is the arithmetic execution correct? Score 1 (Correct): All arithmetic calculations are performed correctly based on the numbers present in the reasoning. Note: A wrong final answer does not necessarily indicate a calculation error; it could be due to using the wrong formula or flawed logic. This category focuses only on the correctness of the calculations themselves. Besides, if there is no need for calculations in the problem, then this should be scored as 1. Score 0 (Error): The model makes calculation errors (e.g., calculating the wrong result number, substituting the wrong value into a correct formula). 4. Logic Properness (Score: 1 or 0) Focus: This evaluates the logical coherence of the reasoning process. Check whether each step logically follows from the previous ones, and whether all necessary conditions are satisfied to draw the conclusion. Score 1 (Valid): The reasoning is coherent, and each step logically follows from its premises. Score 0 (Flawed): When the reasoning involves any of the following: • Insufficient conditions: the evidence is insufficient to draw the conclusion (e.g., concluding a quadrilateral is a square solely because it has 4 right angles, missing the necessary condition of equal side lengths). • Self-contradiction: the model makes logical errors such as contradicting its own earlier reasoning (e.g., first states “x>0x>0” and later assumes “x=−5x=-5”), or draws conclusions that do not logically follow from the premises (e.g., reasoning points to Answer A, but the final conclusion is Answer B). • Other logical mistakes. 5. Assertion Correctness (Score: 1 or 0) Focus: This evaluates the correctness of assertions made during the reasoning process. Specifically, it checks if the model introduces any assumptions or numbers that appear out of nowhere and lack support. Score 1 (Grounded): All numbers, variables, and constraints introduced are traceable to the question or previous steps. Score 0 (Hallucinated/Fabricated): When the reasoning involves any of the following: • Fabricated constraints: the model introduces rules or assumptions that cannot be supported by the question or previous steps (e.g., “Assume x is an integer” or “Assume the triangle is equilateral” when this was not provided). • Magic steps/numbers: new numbers or results appear without any explanation or calculation (e.g., suddenly stating “Let’s consider a possible solution of 12769” without showing the calculation or derivation). • Reverse engineering: abruptly stating the correct answer without a sufficient reasoning path. Note 1 (vs. Knowledge error): if the model hallucinates a specific equation for the problem (e.g., “We assume x+y=10x+y=10”) that turns out to be invalid for the scenario, this is not a Knowledge error but an Assertion error, as the specific equation is not a universal theorem. Note 2 (vs. Calculation error): if an incorrect assertion stems from a visible computation/derivation, and the claim is wrong only because the calculation itself is incorrect (e.g., claiming “25+12^5+1 is divisible by 4” due to an incorrect evaluation of (25+1)mod4(2^5+1) 4), this should be treated as a Calculation error, not an Assertion error. However, if the model does not explain where the number/claim comes from at all, and it simply appears out of nowhere, then it is a magic number / fabricated claim, which is an Assertion error. Table 11: Quality-evaluation rubric used to produce the dimension-wise gold labels referenced in Appendix B.2. A trace is labeled good only if it receives a passing score on every dimension. You are an expert evaluator of reasoning chains in the medical domain. Below are two reasoning traces (Chain-of-Thought) for the same medical/mathmetical problem. Problem: question Trace A: trace_a Trace B: trace_b Evaluation Criteria. Compare the traces based on the following 5 dimensions. A ”better” trace is one that exhibits higher rigor and fewer flaws across these categories: 1. Logical Properness & Coherence: seamless logical flow. Penalize logic jumps or self-contradictions. 2. Assertion Correctness & Groundedness: no “magic numbers” or “reverse engineering” to match a target answer. 3. Clinical Reasoning Rigor & Knowledge Validity: follows established clinical guidelines and does not hallucinate false medical facts. 4. Depth & Clarity: comprehensive, step-by-step derivation over abrupt or shallow explanations. Penalize unnecessary verbosity. Task. Analyze both traces according to the criteria above. Determine which one demonstrates a more authentic, efficient, and rigorous reasoning process overall. Output your final decision in the following format: [[Decision]]: [Trace A is better / Trace B is better / Equal quality] Table 12: Pairwise-judge prompt used by the reward comparator J and the PNT analysis. Unhinted rollout (active policy πθ _θ). system: A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>. user: question assistant: <think> Hinted rollout (contaminated reference πH _H). system: A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant provides the final answer first, then follows up with a comprehensive reasoning process. The answer and reasoning process are enclosed within <answer> </answer> and <think> </think> tags, respectively, i.e., <answer>answer here</answer><think>reasoning process here</think>. user: question assistant: <answer> the correct answer </answer><think> Table 13: The two rollout templates. The unhinted template (top) elicits a reasoning-first generation that defines the active policy πθ _θ. The hinted template (bottom) front-loads the ground-truth answer so the subsequent reasoning becomes a post-hoc rationalization, materializing the contaminated reference πC _C used by the pairwise reward rHIPPOr_HIPPO. C.3 Data Use Our use of the datasets is consistent with their intended purposes and complies with their respective licenses. Appendix D Additional Experimental Results D.1 Generalization Across Backbones: Qwen3-4B To verify that the gains of HIPPO are not tied to a specific backbone, we replicate the main experiment on Qwen3-4B. Table 8 and Table 9 reports in-domain and out-of-domain (TheoremQA) accuracy. HIPPO consistently outperforms both the original instruction-tuned model and the SFT-then-standard-RL baseline, confirming that the contamination-resistant signal transfers across model scales and architectures. D.2 Hard vs. Soft Pairwise Reward The main results use the binarized (hard) variant of rHIPPOr_HIPPO, in which the Monte Carlo win-rate is thresholded at 0.50.5. We compare this against a continuous (soft) variant that uses the raw win-rate as the reward, and a hard variant with different weight. As shown in Table 6, the hard variant outperforms the soft variant at matched reward scale. This is consistent with the theoretical analysis: Lemma 2.3 attains its supremum precisely at the indicator witness T⋆=A⋆T =1_A , so a 0,1\0,1\-valued reward is the one that tightens the lower bound in Eq. (4) on the underlying divergence. Appendix E Evidence of Pre-RL Data Overlap in Common RL Reasoning Datasets A critical premise of our work is that many widely adopted reasoning datasets utilized in recent RL post-training pipelines inherently suffer from Pre-RL data overlap. Specifically, the upstream benchmarks used to construct these RL datasets have already been ingested by modern LLMs during their pre-training or supervised fine-tuning (SFT) phases. In this section, we systematically trace four reliable lineages of data overlap, demonstrating how state-of-the-art RLVR (Reinforcement Learning with Verifiable Rewards) datasets inherit this contamination. A high-level summary is presented in Table 10. The datasets primarily trace back to four reliable lineages of overlap. Chain 1: SimpleRL-Zoo (Zeng et al. 2025). This dataset directly utilizes the official training splits of GSM8K (Cobbe et al. 2021) and MATH (Hendrycks et al. 2021). The contamination of these upstream benchmarks is extensively documented, with models like Qwen2.5-Math-7B exhibiting up to a 54.6% exact-match regurgitation rate on MATH prompts (Wu et al. 2025). Chain 2: Big-Math-RL-Verified (Albalak et al. 2025). Comprising over 250k instances, it explicitly includes subsets of MATH, GSM8K, and AMC/AIME. Chain 3: DeepScaleR (Luo et al. 2025). This dataset relies heavily on historical AIME and AMC problems. Chain 4: The NuminaMath Ecosystem (LI et al. 2024). Datasets such as OpenR1-Math-220k and Skywork-OR1 (He et al. 2025) rely on NuminaMath as their foundational data source, a dataset that is itself pervasively utilized as a pre-RL training corpus (Mahdavi et al. 2025). Appendix F Case Study To complement the aggregate metrics with a qualitative view, we present a representative prompt from the MedQA validation set. On this instance, the SFT-then-standard-RL baseline produces a reasoning trace riddled with severe hallucinations to rationalize a memorized correct answer, while HIPPO produces a reliable, faithful derivation. Prompt. A 4-year-old girl from a recently immigrated family presents to the emergency department with episodes of severe coughing lasting up to several minutes followed by vomiting. She had a low grade fever and runny nose over the last 2 weeks but these coughing episodes just began one day prior to presentation. A complete blood count shows a lymphocytic infiltrate and Gram stain reveals a gram-negative coccobacillus. The emergency department physician explains that this organism causes disease by toxin-mediated inactivation of an inhibitory signaling molecule. Which of the following could be used to culture the most likely cause of this disorder? Options: (A) Charcoal yeast with iron and cysteine (B) Eaton agar (C) Loffler medium (D) Regan-Lowe medium (E) Thayer-Martin agar SFT-then-standard-RL output (abbreviated) <think> The most likely cause of this patient’s severe coughing and vomiting is Pseudomonas aeruginosa, a gram-negative coccobacillus. The pathophysiology of Pseudomonas aeruginosa involves the production of a potent exotoxin, pseudomonas exotoxin A (PE), which is a type I protein-toxin. PE is a preformed, heat-stable protein that is secreted by the bacterium. Once released into the host, PE is cleaved by host trypsin to become active. The active form of PE is a highly toxic, highly immunogenic protein that acts as a cytotoxin. It is best known for its ability to inactivate the host’s inhibitory signaling molecule, the inhibitory G-protein alpha subunit (Galpha-i). By deactivating Galpha-i, the toxin prevents the normal negative feedback of the cyclic AMP (cAMP) signaling pathway, leading to an abnormally high concentration of cAMP in the host cell. This elevated cAMP promotes the expression of virulence genes, such as those encoding for exotoxins and biofilm formation, and enhances the bacterium’s ability to colonize and multiply in the host’s respiratory tract and other mucosal surfaces. Pseudomonas aeruginosa is a highly opportunistic pathogen that commonly causes severe infections in immunocompromised or chronically ill patients, as well as in patients with underlying lung diseases such as cystic fibrosis. It is a non-fermenting, oxidase-positive, and catalase-positive organism. To culture this organism, a selective medium containing specific growth factors is required. Regan-Lowe medium is a specialized culture medium used for the selective isolation of Pseudomonas species. It is a nutrient-rich medium that contains iron, cysteine, and vitamins, which are essential for the growth of Pseudomonas. The addition of iron and cysteine provides the necessary sulfur and nitrogen sources for the synthesis of the bacterium’s characteristic, highly mucoid, blue-green pigmentation. </think> <answer> D </answer> HIPPO output (abbreviated) <think> The clinical presentation of severe, prolonged coughing followed by vomiting, combined with a lymphocytic leukocytosis and a Gram stain revealing a gram-negative coccobacillus, is highly characteristic of Bordetella pertussis, the causative agent of whooping cough (pertussis). Bordetella pertussis is a highly specialized, highly virulent Gram-negative bacterium that induces its pathogenesis through the production of two major exotoxins: pertussis toxin (also known as pertactin) and filamentous hemagglutinin. Pertussis toxin is a highly potent ADP-ribosylating toxin that targets the Gα subunit of Gi-coupled G-protein receptors. By inactivating this inhibitory G-protein, the toxin prevents the normal downregulation of adenylate cyclase. This leads to a massive increase in intracellular cAMP, which subsequently activates protein kinase A (PKA). PKA drives the excessive, paroxysmal coughing characteristic of the disease by stimulating the cough center in the medulla oblongata and inhibiting the normal relaxation of the laryngeal muscles. The most reliable method for culturing B. pertussis is the Regan-Lowe medium. This specialized agar is heavily enriched with vitamins, iron, and cysteine, which are critical for the bacterium’s growth. The medium also contains bile salts and crystal violet, which inhibit the growth of other competing organisms. Because B. pertussis is highly fastidious and grows very slowly (with a doubling time of approximately 16 hours), it requires specific growth conditions. Therefore, Regan-Lowe medium is the preferred culture medium for isolating this organism. </think> <answer> D </answer> Table 14: A model response example from different methods in MedQA validation set. Analysis. As illustrated in Table 14, the baseline RL model falls into a severe hallucination trap driven by shortcut exploitation. Despite successfully arriving at the correct final option (D), its underlying reasoning is entirely unreliable. The model explicitly misidentifies the pathogen as Pseudomonas aeruginosa, yet forcibly aligns it with the clinical vignette by attributing the distinct pathophysiological mechanism of Bordetella pertussis (the inactivation of the Gα protein and subsequent cAMP elevation) to Pseudomonas Exotoxin A. Moreover, it explicitly fabricates the claim that Regan-Lowe medium—the definitive culture for B. pertussis—is used to selectively isolate Pseudomonas. In contrast, HIPPO exhibits faithful, forward-chaining logical deduction: accurately diagnosing B. pertussis, explaining its true toxin mechanism, and logically deriving Regan-Lowe medium as the correct clinical decision. Appendix G Limitations Scope of Hint Injection. Currently, our framework exclusively employs the correct answers for hint injection. However, injecting incorrect answers could further elicit diverse reasoning failures, allowing the pairwise reward model to capture finer-grained preference signals. For instance, it could help the judge distinguish between minor calculation slips and severe logical hallucinations among two incorrect traces, thereby providing a denser and more informative signal to boost RL optimization efficiency. We leave the exploration of incorrect-hint injection as a promising avenue for future work. Utilization of Rollout Traces. In our current implementation, the standard and hint-injected rollouts are generated independently from scratch. A more efficient and synergistic approach would be to enforce prefix-sharing or apply token-level perturbations, ensuring both traces share the identical initial reasoning path before divergence. This alignment would make the paired traces strictly comparable, substantially reducing the capability requirements for the judge model. Furthermore, it could naturally facilitate self-correction mechanisms (Kumar et al. 2024). Optimizing the utilization of rollout trajectories to unlock such self-evolution remains an open challenge. Computational Overhead. Although HIPPO utilizes a relatively small 4B LLM as the pairwise reward model, running an additional model concurrently during RL inevitably introduces computational overhead. A faster alternative could involve replacing the LLM judge with a lightweight discriminator. However, training such a discriminator typically requires additional labeled preference data, which compromises the data-efficiency of our current framework. We therefore leave the development of a lightweight, label-efficient judge to future research.

Instant research discovery