Paper deep dive

SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning

Jialong Liu, Yuling Shi, Ning Yang, Xiaodong Gu, Zuchao Li

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 111

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data efficiency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME'24 using only 8% (0.08x) of the training FLOPs required by scaled supervised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available at this https URL

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

107,542 characters extracted from source content.

Expand or collapse full text

SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning Jialong Liu 1 Yuling Shi 2 Ning Yang 3 Xiaodong Gu 2 Zuchao Li 1 Abstract Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. How- ever, its potential for post-training Large Lan- guage Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this ca- pability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise ”reflection patches,” and use reflection- conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal su- pervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data effi- ciency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME’24 using only 8% (0.08×) of the training FLOPs required by scaled super- vised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available athttps://github.com/Galle ons2029/SRPO 1. Introduction Post-training has emerged as the critical phase for unlock- ing the reasoning and decision-making capabilities of large language models (LLMs) (OpenAI et al., 2024; Grattafiori et al., 2024; Shao et al., 2024). Through reinforcement learn- ing (RL) combined with inference-time techniques such as 1 School of artificial intelligence, Wuhan University, Wuhan, China 2 School of computer science, Shanghai Jiao Tong University, Shanghai, China 3 Institute of automation, Chinese Academy of Sciences, Beijing, China. Correspondence to: Zuchao Li<zcli- charlie@whu.edu.cn>. Proceedings of the43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s). chain-of-thought prompting, modern LLMs achieve impres- sive performance on mathematical reasoning, code gener- ation, and interactive decision-making. Yet a fundamental bottleneck remains: current post-training methods scale poorly to long-horizon tasks—those requiring coherent rea- soning across extended interaction sequences with dozens or even hundreds of intermediate decisions (Gao et al., 2025; Laban et al., 2025). The root cause is the credit assignment problem under sparse supervision. Standard RL approaches such as PPO (Schul- man et al., 2017) and GRPO (Shao et al., 2024) receive only terminal feedback (success or failure) after an entire episode, regardless of trajectory length. This provides merelyO(1) bits of information per episode—an information bottleneck that leads to high-variance gradients and sample-inefficient learning (Lu & Lab, 2025). Compounding this difficulty, recent studies reveal that RL fine-tuning often suffers from entropy collapse, progressively shrinking the policy’s explo- ration space without genuinely expanding reasoning capac- ity (Cui et al., 2025; Yue et al., 2025). Self-reflection offers a compelling alternative by converting sparse outcome signals into rich textual feedback (Shinn et al., 2023; Madaan et al., 2023). Rather than receiv- ing a scalar reward, the model analyzes its own fail- ures, identifies specific errors, and generates corrective guidance—mirroring how humans learn from experience through deliberate reflection. However, existing reflection mechanisms suffer from fundamental limitations that restrict their utility for training. Iteratively appending reflections within an ongoing trajectory induces semantic drift, where accumulating context eventually collapses under its own weight (Laban et al., 2025). Furthermore, reflections con- ditioned on frozen parameters often fail to meaningfully explore the solution space, and injected reflection prompts can disrupt native thinking processes, producing chaotic or excessively verbose outputs. To bridge the gap between reflection-based prompting and reinforcement learning, we propose SRPO (Self-Reflective Policy Optimization), a framework that casts self-reflection as a mechanism for dense reward generation. Our cen- tral insight is that an LLM can serve as its own teacher: hindsight reflections on completed trajectories define a reflection-augmented teacher distribution that, via teacher- 1 arXiv:2608.23493v1 [cs.AI] 24 Aug 2026 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning forced scoring of the student’s on-policy rollouts, yields dense token-level supervision. Crucially, we introduce a reset-with-memory mechanism that prepends compact re- flection patches to the original prompt and regenerates from a clean initial state. This design preserves task specifica- tion fidelity while injecting learned guidance, creating a principled asymmetry—training with reflection, inference without—that drives continuous self-improvement. This reflection-enhanced behavior is transferred into the base policy through on-policy distillation with dense token- level supervision. Where sparse terminal rewards provide O(1)bits per episode, SRPO extractsO(T )bits by com- puting the per-token reverse KL divergence between the student and the reflection-augmented teacher distributions — a signal we refer to as hindsight-guided dense supervision, since reflections are generated after observing the full tra- jectory and densify supervision at every token rather than performing precise per-step causal attribution. This avoids the distribution shift inherent in off-policy methods. In summary, this work presents a scalable and effective framework for advancing long-horizon reasoning. We intro- duce SRPO to convert sparse outcome signals into hindsight- guided dense token-level supervision, effectively mitigating the credit assignment problem where standard RL struggles. Empirically, SRPO achieves state-of-the-art results on math- ematical reasoning (AIME’24: 73.3%) and complex agen- tic tasks (WebShop, ALFWorld, SWE-Bench-Lite), outper- forming reflection-trained baselines (SCoRe (Kumar et al., 2025), R 3 L (Shi et al., 2026), RISE (Qu et al., 2024)) and on- policy distillation from a 72B external teacher, while requir- ing∼3.8×fewer total FLOPs than GRPO (full Stage 1+2 accounting). 2. Related Work Post-training has become the standard paradigm for unlock- ing the reasoning potential of pre-trained LLMs (OpenAI et al., 2024; Grattafiori et al., 2024). The dominant approach involves Reinforcement Learning (RL) over language poli- cies, where Proximal Policy Optimization (PPO) (Schulman et al., 2017) and group-based variants like GRPO (Shao et al., 2024) are widely employed to align models with human preferences or objective correctness. Recently, ex- tensions such as GSPO (Zheng et al., 2025) have further improved stability and sample efficiency. However, apply- ing RL to complex reasoning tasks is non-trivial; recent studies highlight fundamental limitations, including entropy collapse and saturation effects (Cui et al., 2025; Yue et al., 2025), which restrict the model’s ability to explore diverse solution paths effectively. Our work directly addresses these inefficiencies by moving beyond sparse terminal rewards. The challenges of RL are amplified in long-horizon agen- tic tasks, which require planning over extended sequences rather than single-turn QA. Recent benchmarks and surveys emphasize unique difficulties in this domain, such as in- struction drift, context management, and error propagation over time (Li et al., 2025; Laban et al., 2025; Deshpande et al., 2025). While large-scale asynchronous RL (Gao et al., 2025) and system-level optimizations have been proposed to handle these complexities, they often demand massive com- putational resources. Unlike methods that rely on external knowledge injection to supplement domain gaps (Ovadia et al., 2024; Song et al., 2025), our approach focuses on opti- mizing the reasoning process itself within the agent’s policy, enabling it to self-correct dynamically during multi-turn interactions. To mitigate errors in long trajectories, a growing body of work leverages the model’s own reflective capabilities. Inference-time techniques like Self-Refine (Madaan et al., 2023) and Reflexion (Shinn et al., 2023) demonstrate that LLMs can improve by critiquing their own outputs, effec- tively framing verbal feedback as a reward signal. Similarly, CRITIC (Gou et al., 2024) extends this to tool-interactive settings. While promising, most of these methods oper- ate solely at inference time or rely on separate retrospec- tive modules (Yao et al., 2024). A second line of work moves reflection into the training loop: SCoRe (Kumar et al., 2025) trains a two-turn self-correction policy with RL, RISE (Recursive Introspection) (Qu et al., 2024) fine-tunes on iterative self-improvement trajectories, and R 3 L (Shi et al., 2026) performs local repair of pivot tokens identified by reflection. These methods treat correction either as an inference-time procedure that doubles compute (SCoRe’s two-turn generation), as a separate supervised task (RISE), or as a localized suffix repair (R 3 L). SRPO differs from all three by exposing the full reflection-augmented teacher dis- tribution at every student-rollout token (via teacher-forced scoring) and internalizing it through on-policy distillation, so that no reflection is needed at inference time and the entire trajectory — not just a local pivot — can be re-routed. We compare against all three as training-time baselines in Section ??. Our framework is also closely related to self-distillation, where models learn from their own high-quality gener- ations (Pham et al., 2022; Yang et al., 2024). While standard self-distillation typically uses off-policy data, re- cent work emphasizes the importance of on-policy distil- lation—learning from the model’s current distribution—to improve robustness and alignment (Agarwal et al., 2024; Pati ̃ no et al., 2025). SRPO advances this direction by in- troducing a specific form of on-policy distillation: using reflection-conditioned teacher scores on student on-policy rollouts as dynamic targets. This allows the model to learn not just from correct answers, but from the process of cor- recting its own mistakes, thereby converting the sparse sig- 2 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning nals characteristic of long-horizon tasks into dense, token- level supervision. 3. Methodology 3.1. Preliminaries Problem Setup. We study post-training of a language model as sequential decision making over text. Given an input promptx(task instruction and optional observation), a policyπ θ generates a completiony = a 1:T token-by-token, which equivalently defines a trajectoryτ = (s 1:T ,a 1:T ) wheres t is the text history up to stept. An environment or automatic evaluator assigns a scalar rewardR(τ ) ∈ R, which is typically sparse in long-horizon settings (often only available at termination). Our goal is to learn a policy that maximizes expected reward while remaining close to a reference policy π ref to preserve language quality: max θ E x∼D,τ∼π θ (·|x) [R(τ )] − β E x∼D [KL (π θ (·| x)∥π ref (·| x))], (1) whereDdenotes the prompt distribution andβcontrols the KL regularization strength. In interactive agent benchmarks, s t may include the full dialogue history and environment feedback, and an episode terminates upon success, failure, or a length budget. Imitation learning. We consider sequential decision mak- ing over text, where a policyπ θ generates an action se- quence (token sequence)a 1:T conditioned on a state/his- torys t . Given expert demonstrationsD = τ (i) with trajectoriesτ = (s 1:T ,a 1:T ), behavior cloning learnsπ θ by maximum likelihood: L BC (θ) =−E (s t ,a t )∼D [logπ θ (a t | s t )].(2) In interactive settings, pure behavior cloning can suffer from compounding errors due to distribution shift. DAgger miti- gates this by aggregating data collected under the learned policy and querying an expert for corrective labels (Ross et al., 2011). Group Relative Policy Optimization (GRPO). For post- training with reinforcement learning, we optimizeπ θ to maximize an expected rewardR(τ )while constraining devi- ation from a reference policyπ ref . GRPO (Shao et al., 2024) is a PPO-style method that replaces a learned value func- tion with a group-relative baseline computed from multiple sampled responses. Concretely, for each prompt/context xwe sample a group ofGrolloutsy (g) G g=1 ∼ π θ (· | x) and obtain scalar rewardsr (g) . GRPO forms normalized advantages using group statistics, e.g. ˆ A (g) = r (g) − μ r σ r + ε , μ r = 1 G G X g=1 r (g) .(3) SRPO � � Standard training Self- Reflection � � � � ∗ � � � � � � � ∗ ＋ Stage1: RefineStage2: Resample Initial prompt Rollout trajetoryAdvantages Policy Model Policy Model Figure 1. Overview of the SRPO Framework. The process consists of two stages using the same modelπ θ . Stage 1: Given a prompt xand a sparse outcomeofrom an initial attempt, the model gen- erates a concise Reflection Patch (p). This patch is prepended to the prompt (Reset-with-Memory) to guide the model in generating a high-quality “Teacher” distributionπ θ (·| [p;x]). Stage 2: The base model (Student), seeing only the original promptx, gener- ates on-policy rollouts. It is optimized to minimize the Reverse KL divergence between its output distribution and the Teacher’s distribution, effectively converting sparse outcomes into dense, token-level supervision. Then it applies a clipped policy-gradient objective with a KL regularizer toπ ref (analogous to PPO (Schulman et al., 2017)), enabling stable optimization without an explicit critic. Related group-based variants further extend this idea for improved efficiency (Zheng et al., 2025). 3.2. Self-Reflective Policy Optimization We present SRPO, a framework that converts sparse out- come signals into dense, token-level supervision via self- reflection, as illustrated in Figure 1. The core premise is to utilize the model’s own reflection-augmented distribution as a dynamic teacher for its base policy. By distilling this self- induced teacher into the student via teacher-forced scoring of the student’s on-policy rollouts, SRPO effectively ad- dresses the limitations of standard post-training paradigms: it provides the dense supervision lacking in sparse-reward RL, mitigates the exposure bias inherent in off-policy SFT, and establishes a self-contained improvement loop without relying on external teacher models. Figure 1 summarizes the two-stage SRPO pipeline. 3.2.1. STAGE 1: REFLECTION-GUIDED STATE AUGMENTATION The first stage transforms sparse terminal feedback into structured, reusable guidance through self-reflection on ini- tial rollouts. Initial Rollout Collection. For each promptx ∼ D, we sample a completiony ∼ π θ (· | x)from the current stu- dent policy. In agentic settings, we execute the trajectory τ = (s 1:T ,a 1:T )in an interactive environment to obtain a terminal outcome signalo(e.g., success/failure indicator, environment feedback, or sparse scalar reward). Critically, such outcome signals are often the only reliable supervi- 3 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning sion in long-horizon tasks, making credit assignment over intermediate decisions fundamentally difficult. Self-Reflection as Credit Assignment. Rather than di- rectly optimizing from sparse outcomes—which provides O(1)bits of information per episode regardless of trajectory length—we leverage the model’s intrinsic reflection capa- bility to perform explicit credit assignment. Given the tuple (x,τ,o), we query the model with a structured reflection prompt to generate a concise hindsight patch p: p = Reflect π θ (x,τ,o).(4) The reflectionpencapsulates two components: (i) a di- agnostic analysis identifying root causes of failure or key decisions in successful trajectories, and (i) actionable guid- ance specifying constraints to enforce, subgoals to priori- tize, or pitfalls to avoid. We deliberately keeppcompact (typically 2–5 bullet points) to ensure it serves as a stable, non-redundant conditioning signal. Initial-State Reconstruction. A critical design choice distinguishes our approach from prior reflection meth- ods (Shinn et al., 2023; Madaan et al., 2023): instead of iter- atively appending reflections within an already-drifted con- text—where reflections can become repetitive, inconsistent, or semantically entangled with accumulated errors—we per- form a reset with memory. We reconstruct an augmented initial state by prepending the reflection patch to the original prompt: ̃x = Reconstruct(p,x) = [p;x].(5) This formulation ensures that: (1) the environment state re- mains unmodified, maintaining consistency with the original task specification; (2) the augmented distributionπ θ (·| ̃x) stays close to the original distributionπ θ (· | x), sincep is prepended rather than interleaved; and (3) the reflection serves as a “prior” that guides subsequent generation with- out contaminating the action space. Rethinking Rollout (Quality Validation). From the recon- structed state, we optionally sample a rethinking rollout: ̃y ∼ π θ (·| ̃x).(6) Empirically, ̃yexhibits substantially higher quality thany, evidence thatπ θ (·| ̃x)is a stronger policy and therefore a valid teacher. Importantly, ̃yis not a training target: Stage 2 consumes only the teacher’s per-token log-probabilities un- der teacher-forcing on the student’s tokens. Self as Teacher. The central insight of our approach is that the model’s own reflection-conditioned policy can serve as its teacher. We define a reflection-augmented teacher policy: π T (·| x) := π θ (·| ̃x) = π θ (·| [p;x]).(7) With a slight abuse of notation, on the student’s on-policy tra- jectory(s t ,a t )we writeπ T (a t | s t ) := π θ (a t | [p;x],y <t ): the teacher is evaluated by teacher-forcing on the stu- dent’s response prefixy <t under the reflection-augmented prompt[p;x]. Stage 2 thus uses teacher scoring of stu- dent tokens under privileged context—not teacher-generated trajectories—which keeps the distillation strictly on-policy and avoids the off-policy mismatch that would arise if reflection-conditioned rollouts were used as targets. Un- like conventional distillation that requires a larger, more capable teacher model, our teacherπ T is the same model π θ but operating under more favorable conditions (i.e., with access to hindsight information). This creates an asymme- try: at training time, the model can leverage reflection to produce better outputs; at inference time, the learned policy must achieve comparable quality without reflection. Stage 2 bridges this gap through on-policy distillation. 3.2.2. STAGE 2: ON-POLICY SELF-DISTILLATION The second stage transfers the reflection-enhanced behavior into the base policy through on-policy distillation, eliminat- ing the inference-time dependency on explicit reflection. On-Policy Sampling. For each promptx∼D, we sample trajectories from the current student policyπ θ (· | x). Un- like off-policy distillation that trains on teacher-generated trajectories, on-policy sampling ensures the student learns to improve from states it actually visits, avoiding the com- pounding errors that arise from distribution shift (Ross et al., 2011). This is particularly crucial in long-horizon settings where small early deviations can lead to drastically different downstream states. Dense Token-Level Supervision via Reverse KL. The core advantage of distillation over reinforcement learning lies in its reward density. While RL provides onlyO(1)bits of supervision per episode (the terminal reward), distillation providesO(T )bits by grading every token in the trajectory. We adopt a per-token single-sample Monte-Carlo estimator of the reverse-KL functional: for a states t visited by the cached behaviour policyπ θ old and a sampled actiona t ∼ π θ old (·| s t ), we define the cached log-ratio reward r t = sg[ logπ T (a t | s t ) − logπ θ old (a t | s t ) ],(8) wheresg[·]is the stop-gradient operator ands t denotes the context up to positiont. Bothπ T andπ θ old are pre- computed and detached before the actor update, sor t car- ries no gradient; the policy is optimised only through the PPO ratioρ t = π θ /π θ old in Eq. (10). Taking expectation underπ θ old (· | s t )recovers−KL(π θ old ∥π T ) (s t ), sor t is an unbiased single-sample estimator of the population re- verse KL ats t . The negative reverse KL rewards tokens that align with the teacher’s distribution while penalizing deviations. Intuitively, tokens receiving high penalty cor- respond to “forking points” where the student’s choices diverge from the reflection-informed teacher—precisely the 4 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning decision points where credit assignment is most valuable. Reverse KL possesses several desirable properties for our setting: (i) it is mode-seeking, encouraging the student to commit to the teacher’s preferred behavior rather than spreading probability mass across suboptimal alternatives; (i) it provides an unhackable reward signal, as low KL al- ways corresponds to high probability under the teacher; and (i) it naturally synergizes with policy gradient methods that optimize sequence-level reverse KL induced by reward models. Advantage Estimation. To reduce variance while main- taining unbiased gradients, we compute advantages using a trajectory-level baseline: ̄r = 1 |V| X t∈V r t , A t = r t − ̄r,(9) whereVdenotes the set of valid (non-padding) token po- sitions. This group-relative normalization, inspired by GRPO (Shao et al., 2024), eliminates the need for a learned value function while providing stable optimization signals. Clipped Policy Gradient Objective. We optimize the stu- dent policy using a PPO-style clipped objective (Schulman et al., 2017) to prevent destructively large updates: L(θ) =−E t [min (ρ t A t , clip(ρ t , 1− ε, 1 + ε)A t )], (10) whereρ t = π θ (a t | s t )/π θ old (a t | s t )is the importance sampling ratio between the current and cached policy. The clipping thresholdε(typically0.1–0.2) bounds the policy update magnitude, ensuring stable learning even with ag- gressive advantage signals. Algorithm 1 summarizes the complete training procedure. Through this iterative process, the student policyπ θ gradu- ally approximates the behavior of the reflection-augmented teacherπ T , effectively internalizing the reasoning capabili- ties initially derived from the reflection mechanism. SRPO inherits the advantages of both distillation and on- policy RL: it provides dense token-level supervision (O(T ) bits per episode vs.O(1)for sparse RL), eliminates ex- posure bias through on-policy sampling, and requires no external teacher models. The reflection-augmented teacher π T (·| [p;x])effectively has access to hindsight information, creating an asymmetry that enables self-improvement: the model learns to internalize decisions that were originally informed by outcome feedback. Counting Stage 1 and Stage 2, SRPO uses5.4× 10 18 FLOPs versus20.8× 10 18 for GRPO, i.e., approximately3.8×fewer total FLOPs. De- tailed theoretical analysis, including information-theoretic perspectives and formal treatment of the self-distillation mechanism, is provided in Appendix B. Algorithm 1 SRPO: Self-Reflective Policy Optimization Require:Policyπ θ , prompt datasetD, clipping threshold ε 1: for each training iteration do 2:for each prompt x∼D do 3:// Stage 1: Reflection-Guided State Augmentation 4:Sample initial rollout: y ∼ π θ (·| x) 5:Execute trajectory τ and obtain outcome o 6:Generate reflection patch: p← Reflect π θ (x,τ,o) 7:Construct augmented prompt: ̃x← [p;x] 8:// Stage 2: On-Policy Self-Distillation 9:Define teacher: π T (·| x) := π θ (·| ̃x)// Self as teacher 10:Sample on-policy trajectory: y ∼ π θ (·| x) 11:for each token a t in y do 12: Compute cached token reward (stop-grad): r t ← logπ T (a t | s t )− logπ θ old (a t | s t )// teacher-forcing on y <t 13:end for 14:Compute advantages: A t ← r t − 1 T P t ′ r t ′ 15:Update θ via clipped policy gradient: 16: L =−E t [min (ρ t A t , clip(ρ t , 1± ε)A t )] 17:end for 18: end for 4. Experiments We evaluate SRPO on reasoning benchmarks spanning math- ematical problem-solving and long-horizon agentic tasks. We compare against strong post-training and inference-time baselines, and ablate self-reflection and key design choices across model scales. 4.1. Experimental Setup Base Models. We evaluate SRPO on models spanning multiple scales to demonstrate generalizability: Qwen3- 1.5B, Qwen3-8B, and Qwen3-32B (Yang et al., 2025) as base models. For agentic tasks, we additionally evaluate on Llama-3.1-8B-Instruct (Grattafiori et al., 2024) to demon- strate cross-family generalization. Datasets and Benchmarks. We evaluate on the following benchmarks: •Mathematical Reasoning: AIME’24 (30 problems, competition-level), MATH-500 (Lightman et al., 2024) (500 problems across 7 categories), and GSM8K (Cobbe et al., 2021) (1,319 test problems) for grade-school math. •Out-of-DistributionGeneralization:Deep- ScaleR (Luo et al., 2025) comprising 1,200 challenging mathematical reasoning problems from diverse domains. 5 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning SFT-400KPPOGRPOOn-Policy Distill ReflexionSelf-RefineSRPO (Ours) 40 50 60 70 80 90 100 Accuracy (%) Best across all benchmarks Mathematical Reasoning Performance Comparison AIME'24 MATH-500 GSM8K DeepScaleR Figure 2. Performance comparison on mathematical reasoning benchmarks. All methods start from the same SFT-400K check- point. SRPO achieves the best performance across all benchmarks with significantly fewer training FLOPs (0.08×). † Inference-time methods with 3 refinement iterations. Detailed numerical results are in Table 5. •Long-Horizon Agentic Tasks: WebShop (Yao et al., 2023) (12,087 shopping tasks with sparse success sig- nals), ALFWorld (Shridhar et al., 2021) (134 household tasks), and SWE-Bench-Lite (Jimenez et al., 2024) (300 real-world GitHub issues). Training Details. For mathematical reasoning, we initialize from an SFT checkpoint trained on 400K prompts from OpenThoughts-3 (Guha et al., 2025). We use a batch size of 256 rollouts, learning rate of1× 10 −5 with cosine decay, and clip ratioε = 0.2. For agentic tasks, we train with 64 prompts per batch with 4 samples per prompt. All experi- ments use 8×H100 GPUs unless otherwise specified. The reflection prompt is structured to generate 2–5 bullet points of diagnostic analysis and actionable guidance. Baselines. We compare against the following methods: •SFT: Supervised fine-tuning on teacher-generated tra- jectories (off-policy distillation). •GRPO (Shao et al., 2024): Group Relative Policy Optimization with sparse outcome rewards. • PPO (Schulman et al., 2017): Standard PPO with learned value function. •On-Policy Distillation (OPD) (Lu & Lab, 2025): Dis- tillation from a larger teacher model (Qwen3-32B→ Qwen3-8B). •Reflexion (?): Iterative reflection with memory accu- mulation (inference-time only). •Self-Refine (?): Iterative self-refinement without train- ing. 4.2. Main Results: Mathematical Reasoning Figure 2 presents our main results on mathematical rea- soning benchmarks. SRPO consistently outperforms all BaseSFTGRPOPPOReflexionSelf-RefineSRPO (Ours) Method 0 10 20 30 40 50 60 70 80 90 Success Rate (%) (a) Long-Horizon Agentic Task Success Rates WebShop ALFWorld SWE-Lite Base SFT GRPO PPO Reflexion Self-Refine SRPO (Ours) Method 0 2 4 6 8 10 12 14 16 18 Average Episode Steps Most efficient (b) Execution Efficiency Figure 3. Performance on long-horizon agentic benchmarks. (a) Success rate comparison across WebShop, ALFWorld, and SWE- Bench-Lite. (b) Average episode steps showing execution effi- ciency. SRPO achieves the highest success rates while maintaining the shortest episode length. Detailed numerical results are in Ta- ble 6. baselines across all four benchmarks, achieving 73.3% on AIME’24—a 3.3 percentage point improvement over stan- dard on-policy distillation and 5.3 points over GRPO. No- tably, SRPO achieves these gains with only 8% of the train- ing FLOPs required by SFT-2M extrapolation, demonstrat- ing superior sample efficiency. Analysis. The performance gap between SRPO and stan- dard on-policy distillation (OPD) highlights the value of self-generated reflection as a teaching signal. While OPD relies on an external, larger teacher model (Qwen3-32B), SRPO uses the student’s own reflection-conditioned outputs as targets. This creates two advantages: (1) the teacher dis- tribution is naturally aligned with the student’s capability frontier, avoiding the “capability gap” problem where stu- dents struggle to imitate behaviors far beyond their current ability; (2) the reflection mechanism provides task-specific, instance-adaptive guidance rather than generic teacher be- havior. On out-of-distribution generalization (DeepScaleR), SRPO shows a larger relative improvement (+7.8% relative to OPD), suggesting that the reflection mechanism helps iden- tify and correct domain-specific failure modes that may not be addressed by generic distillation. 4.3. Main Results: Long-Horizon Agentic Tasks Figure 3 presents results on long-horizon agentic tasks. SRPO achieves the highest success rates across all three benchmarks: 64.7% on WebShop (+7.9% over SFT), 76.8% on ALFWorld (+5.6% over Reflexion), and 31.2% on SWE- Bench-Lite (+4.4% over Reflexion). Credit Assignment in Long Horizons. The performance gap between SRPO and RL-based methods (GRPO, PPO) is particularly pronounced on these tasks, where episodes can span 10–50 actions. RL methods struggle with credit as- signment: a single terminal success/failure signal provides insufficient information to determine which intermediate decisions were critical. In contrast, SRPO’s reflection mech- 6 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning 012345 Number of Refinement Iterations 55 60 65 70 75 80 85 AIME'24 Accuracy (%) +1.8%/iter +1.2%/iter +0.9%/iter (a) Performance vs. Refinement Iterations SFT GRPO SRPO (Ours) 123456 Inference FLOPs (relative) 55 60 65 70 75 80 85 AIME'24 Accuracy (%) Pareto Frontier (b) Performance vs. Inference Compute SFT GRPO SRPO (Ours) Figure 4. Inference-time scaling on AIME’24. (a) Performance vs. number of refinement iterations for different methods. SRPO- trained models show better scaling with additional compute (+1.8%/iter vs +1.2%/iter for GRPO). (b) Performance vs. to- tal inference FLOPs, demonstrating Pareto efficiency. anism explicitly identifies failure modes (e.g., “navigated to wrong room before finding the target object”) and converts them into actionable guidance. Efficiency Gains. Interestingly, SRPO also achieves the shortest average episode length (10.2 steps), indicating that the learned policy is not only more successful but also more efficient. This suggests that the reflection-distilled policy has internalized effective planning strategies rather than relying on trial-and-error exploration. 4.4. Inference-Time Scaling via Self-Refinement We investigate how SRPO-trained models behave when combined with inference-time refinement strategies. Fig- ure 4 shows the relationship between inference compute and performance. Setup. We compare three configurations: (1) base SFT model with iterative self-refinement; (2) GRPO-trained model with self-refinement; (3) SRPO-trained model with self-refinement. For each configuration, we vary the number of refinement iterations from 0 to 5. Results. SRPO-trained models exhibit superior inference- time scaling: each additional refinement iteration yields +1.8% improvement on average, compared to +1.2% for GRPO and +0.9% for SFT. At 3 iterations, SRPO achieves 78.5% on AIME’24, approaching the performance of mod- els 4× larger. This improved scaling can be attributed to the training ob- jective: SRPO explicitly trains the model to benefit from reflection-augmented contexts. The learned policy has inter- nalized the structure of effective reflections and can leverage them more efficiently at inference time. 4.5. Full Fine-Tuning vs. LoRA Figure 5 compares full fine-tuning and LoRA adaptation for SRPO. We evaluate LoRA with ranks 64 and 128, applied to all attention and MLP layers. Full FTLoRA-64LoRA-128 Fine-tuning Method 65 70 75 80 85 90 AIME'24 Accuracy (%) 97.8% 97.2% (a) Performance: Full FT vs. LoRA Qwen3-8B Qwen3-32B Full FTLoRA-64LoRA-128 Fine-tuning Method 0 50 100 150 200 250 300 GPU Memory (GB) 72% 73% (b) Memory Efficiency Qwen3-8B Qwen3-32B Figure 5. Comparison of full fine-tuning vs. LoRA adaptation for SRPO. (a) Performance on AIME’24 showing LoRA-128 achieves 97.8% of full fine-tuning accuracy. (b) GPU memory requirements demonstrating LoRA’s efficiency with 72–73% reduction. Detailed numerical results are in Table 7. Table 1. Ablation study on AIME’24 and WebShop. Each row removes or modifies one component from the full SRPO system. ConfigurationAIME’24WebShop SRPO (Full)73.364.7 Stage 1: Reflection Design w/o reflection (direct retry)65.854.2 w/ verbose reflection (>10 points)70.060.3 w/ outcome-only feedback67.256.8 w/ external teacher reflection71.562.4 Stage 2: Distillation Strategy Forward KL instead of reverse KL69.458.6 Off-policy (teacher trajectories)68.055.9 No clipping (ε =∞)70.261.3 Single-sample advantage (no group)71.162.0 State Reconstruction Append reflection (not prepend)68.557.4 No state reset (iterative)66.352.8 Findings. LoRA-128 achieves 97.8% of full fine-tuning performance on AIME’24 while using only 1.3% of the trainable parameters and 34% of the GPU memory. This efficiency makes SRPO accessible for practitioners with limited compute resources. Notably, the gap between LoRA and full fine-tuning is smaller for SRPO compared to standard SFT (where LoRA lags by 13% after large-scale training, as noted in prior work (Lu & Lab, 2025)). We hypothesize that on-policy distillation’s dense token-level supervision is more compat- ible with low-rank updates, as it provides richer gradient information per sample. 4.6. Ablation Studies Table 1 presents ablation studies examining key design choices in SRPO. Reflection Quality Matters. Removing reflection entirely (“direct retry”) causes a 7.5-point drop on AIME’24, con- firming that self-reflection provides meaningful guidance beyond random exploration. Interestingly, verbose reflec- 7 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning 1.7B8B32B Model Size 0 20 40 60 80 AIME'24 Accuracy (%) +7.8 +5.3 +3.8 (a) Performance vs. Model Size SFT GRPO SRPO (Ours) 10 1 10 0 Total Training FLOPs (relative to GRPO, log scale) 45 50 55 60 65 70 75 80 AIME'24 Accuracy (%) Matched AIME'24 accuracy ~3.8× fewer total FLOPs SRPO: 0.26× (b) Performance vs. Training Compute SFT GRPO SRPO (Ours) Figure 6. Scaling behavior of SRPO. (a) Performance vs. model size shows consistent improvements across scales, with larger gains for smaller models (+7.8 for 1.5B vs +3.8 for 32B). (b) Performance vs. training compute demonstrates SRPO reaches 70% performance with∼10× fewer FLOPs than GRPO. Table 2. Scaling analysis across model sizes on AIME’24. SRPO shows consistent improvements across all scales, with the relative gain increasing for smaller models. Model SizeSFTGRPOSRPO∆ vs GRPO Qwen3-1.5B38.244.552.3+7.8 Qwen3-8B60.068.073.3+5.3 Qwen3-32B71.478.282.0+3.8 tions with more than 10 points actually hurt performance (-3.3 points), suggesting that overly detailed reflections may introduce noise or conflicting advice. This validates our design choice of compact, actionable reflections. Self-Generated vs. External Reflection. Using reflections generated by a larger external teacher (Qwen3-32B) per- forms worse than self-reflection (-1.8 points on AIME’24). We attribute this to distribution mismatch: external reflec- tions may reference strategies or concepts outside the stu- dent’s capability, leading to ineffective guidance. Reverse KL is Critical. Switching from reverse KL to forward KL causes a 3.9-point drop on AIME’24. Forward KL is “mean-seeking” and encourages the student to cover all modes of the teacher distribution, which dilutes focus on the most effective strategies. Reverse KL’s mode-seeking property better aligns with our goal of learning the teacher’s preferred behavior. State Reset is Essential. The “no state reset” configuration, which appends reflections iteratively without resetting to the initial state (similar to Reflexion), performs 7.0 points worse on AIME’24. This confirms our hypothesis that accu- mulated context can become inconsistent and interfere with effective reasoning. 4.7. Scaling Analysis We examine how SRPO scales with model size (Table 2 and Figure 6). Consistent Gains Across Scales. SRPO provides improve- 12345 Helpfulness Score (1-5) 0 5 10 15 20 25 30 35 40 45 Percentage of Reflections (%) 3% 5% 25% 35% 32% 67% 4 8% 2 (a) Distribution of Reflection Quality 12345 Reflection Helpfulness Score 0 10 20 30 40 50 Improvement in Success Rate (%) r = 0.72 High-quality: +34% avg Low-quality: +5% avg (b) Quality-Improvement Correlation Figure 7. Analysis of reflection quality. (a) Distribution of reflec- tion helpfulness scores rated by GPT-4: 67% of reflections receive scores≥4 (useful), while only 8% receive scores≤2 (unhelpful). (b) Strong correlation (r = 0.72) between reflection quality and improvement in rethinking rollout success rate. ments across all model sizes, with gains of +7.8, +5.3, and +3.8 points over GRPO for 1.5B, 8B, and 32B models re- spectively. The larger relative improvement for smaller mod- els suggests that SRPO’s dense supervision is particularly valuable when model capacity is limited. Compute Efficiency. Figure 6 (right) shows performance as a function of training FLOPs. SRPO achieves superior Pareto efficiency: it reaches 70% AIME’24 performance with approximately 10×fewer FLOPs than GRPO and 30× fewer than continued SFT. This aligns with theoretical ex- pectations: distillation providesO(T )bits of supervision per episode compared to O(1) for RL with sparse rewards. 4.8. Analysis: Quality of Self-Generated Reflections To understand when and why self-reflection helps, we an- alyze 500 randomly sampled reflection instances from the AIME’24 training set. Reflection Helpfulness. We use GPT-4 to rate each reflec- tion on a 1–5 scale for helpfulness (1=irrelevant, 5=directly actionable). Figure 7 (left) shows the distribution: 67% of reflections receive scores≥4, indicating that the model can reliably generate useful self-feedback. Only 8% receive scores≤2, typically for problems where the model lacks fundamental knowledge. Correlation with Improvement. Figure 7 (right) shows a strong correlation (r = 0.72) between reflection helpfulness and improvement in rethinking rollout success. High-quality reflections (score 5) lead to a 34% average improvement in success rate, while low-quality reflections (score 1–2) yield only 5% improvement—still positive but marginal. Failure Mode Analysis. We categorize unsuccessful reflec- tions into three types: (1) Generic advice (42%): reflections that provide correct but non-specific guidance (e.g., “check the arithmetic carefully”); (2) Incorrect diagnosis (35%): reflections that misidentify the root cause of failure; (3) Beyond capability (23%): problems requiring knowledge the model does not possess. These failure modes suggest 8 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning Table 3. Comparison of self-distillation (SRPO) vs. distillation from larger teachers. Self-distillation achieves competitive perfor- mance without requiring access to larger models. StudentTeacherAIME’24Teacher FLOPs Qwen3-8B None (GRPO)68.00 Qwen3-32B70.04.0× Qwen3-72B72.59.0× Self (SRPO)73.31.0× Qwen3-1.5B Qwen3-8B48.75.3× Qwen3-32B51.221.3× Self (SRPO)52.31.0× Table 4. Continual learning evaluation. Models are first trained on math (AIME’24) then adapted to code (SWE-Lite). SRPO better preserves original capabilities while acquiring new skills. MethodMath (Before)Code (After)Math (After)Retention SFT60.028.448.280.3% GRPO68.026.759.387.2% SRPO73.331.269.895.2% future directions: incorporating external verification signals or retrieval-augmented reflection. 4.9. Comparison with Larger Teacher Distillation A key advantage of SRPO is eliminating the need for larger teacher models. Table 3 compares SRPO against distillation from progressively larger teachers. Self-Distillation Matches or Exceeds Larger Teachers. Remarkably, SRPO with self-distillation outperforms distil- lation from Qwen3-72B (+0.8 points) while using 9×fewer teacher FLOPs. This suggests that the reflection mechanism effectively “unlocks” latent capabilities within the model that are difficult to transfer through conventional distillation. Implications for Practitioners. This finding has significant practical implications: practitioners can improve their mod- els without access to expensive API calls to larger models or the computational resources to run them. SRPO democra- tizes access to high-quality post-training by leveraging the model’s own capabilities. 4.10. Continual Learning and Catastrophic Forgetting We evaluate SRPO’s behavior in a continual learning set- ting where models must acquire new capabilities without forgetting previous skills. Setup. We first train Qwen3-8B on mathematical reasoning, then adapt to coding tasks (SWE-Bench-Lite). We measure both the new capability acquisition and retention of original mathematical reasoning ability. Results. Table 13 shows that SRPO achieves 95.2% re- tention of mathematical reasoning performance after code adaptation, compared to 87.2% for GRPO and 80.3% for SFT. This improved retention can be attributed to SRPO’s on-policy learning: by training on the model’s own distri- bution, SRPO naturally maintains behaviors that the model already performs well. Connection to Prior Work. This finding aligns with ob- servations in the on-policy distillation literature (Lu & Lab, 2025): on-policy methods cause less catastrophic forgetting than off-policy approaches because they do not force the model to imitate out-of-distribution behaviors. 5. Conclusion We presented SRPO, a post-training method that con- verts episode-level outcomes into hindsight-guided dense, reflection-augmented supervision, overcoming the ineffi- ciency of sparse-reward RL. The mechanism is best un- derstood not as self-distillation but as on-policy RL with reflection-guided exploration: reflection-conditioned teach- ers steer the student into high-reward regions of its own policy space rather than imitating a distribution beyond its representational capacity. Across 10 benchmarks span- ning math, code, agent, science, and logic, SRPO yields reproducible gains (multi-seed bootstrap CIs,p < 0.005), matching or exceeding larger-teacher distillation while re- quiring∼3.8×fewer total FLOPs than GRPO and naturally mitigating catastrophic forgetting under continual learning. Moving forward, we aim to refine reflection quality via ex- ternal verification or better calibration, and to extend the framework beyond verifiable-outcome tasks, including mul- timodal reasoning and longer-horizon tool-use trajectories. 9 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning Acknowledgements This work was supported by the National Natural Science Foundation of China (No. 62306216), the Technology Inno- vation Program of Hubei Province (No. 2024BAB043) and the Fundamental Research Funds for the Central Universi- ties (No. 2042026kf0055). Impact Statement This paper presents a post-training method intended to im- prove the reliability and sample efficiency of long-horizon reasoning and agentic behavior in LLMs. Potential positive impacts include better performance with lower compute and fewer human labels, which can broaden access to capable models. Potential negative impacts include enabling more effective autonomous agents that could be misused (e.g., for scalable cyber abuse) if deployed without appropriate safeguards. We encourage careful evaluation, monitoring, and staged release practices when applying the method to high-stakes or open-ended agentic settings. References Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Garea, S. R., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview .net/forum?id=3zKtaqxLhW. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021. URLhttps://arxiv. org/abs/2110.14168. Cui, G., Zhang, Y., Chen, J., Yuan, L., Wang, Z., Zuo, Y., Li, H., Fan, Y., Chen, H., Chen, W., Liu, Z., Peng, H., Bai, L., Ouyang, W., Cheng, Y., Zhou, B., and Ding, N. The entropy mechanism of reinforcement learning for reasoning language models, 2025. URLhttps: //arxiv.org/abs/2505.22617. Deshpande, K., Sirdeshmukh, V., Mols, J. B., Jin, L., Hernandez-Cardona, E.-Y., Lee, D., Kritz, J., Primack, W. E., Yue, S., and Xing, C. MultiChallenge: A realistic multi-turn conversation evaluation benchmark challeng- ing to frontier LLMs. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.), Findings of the Asso- ciation for Computational Linguistics: ACL 2025, p. 18632–18702, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256- 5. doi: 10.18653/v1/2025.findings- acl.958. URL https://aclanthology.org/2025.findin gs-acl.958/. Gao, J., Fu, W., Xie, M., Xu, S., He, C., Mei, Z., Zhu, B., and Wu, Y. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl, 2025. URL https://arxiv.org/abs/2508.07976. Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., and Chen, W. CRITIC: Large language models can self- correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum ?id=Sx038qxjek. Grattafiori, A., Dubey, A., et al. The llama 3 herd of models, 2024. URLhttps://arxiv.org/abs/2407.2 1783. Guha, E., Marten, R., Keh, S., et al. Openthoughts: Data recipes for reasoning models, 2025. URLhttps://ar xiv.org/abs/2506.04178. Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. LiveCodeBench: Holistic and contamination-free eval- uation of large language models for code, 2024. URL https://arxiv.org/abs/2403.07974. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770. Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., Zhang, L. M., McKinney, K., Shrivastava, D., Paduraru, C., Tucker, G., Precup, D., Behbahani, F., and Faust, A. Training language models to self-correct via reinforce- ment learning. In The Thirteenth International Confer- ence on Learning Representations, 2025. URLhttps: //openreview.net/forum?id=CjwERcAU7w. Laban, P., Hayashi, H., Zhou, Y., and Neville, J. Llms get lost in multi-turn conversation, 2025. URLhttps: //arxiv.org/abs/2505.06120. Li, Y., Shen, X., Yao, X., Ding, X., Miao, Y., Krishnan, R., and Padman, R. Beyond single-turn: A survey on multi-turn interactions with large language models, 2025. URL https://arxiv.org/abs/2504.04717. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum ?id=v8L0pN6EOi. 10 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning Lu, K. and Lab, T. M. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml .20251026. https://thinkingmachines.ai/blog/on-policy- distillation. Luo, M., Tan, S., Wong, J., Shi, X., Tang, W., Roongta, M., Cai, C., Luo, J., Zhang, T., Li, E., Popa, R. A., and Stoica, I. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl.https://pretty-radio-b75.n otion.site/DeepScaleR-Surpassing-O1-P review-with-a-1-5B-Model-by-Scaling -RL-19681902c1468005bed8ca303013a4e2, 2025. Notion Blog. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023. URLhttps://openreview.net/forum?id= S37hOerQLB. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., et al. Gpt-4 technical report, 2024. URLhttps://ar xiv.org/abs/2303.08774. Ovadia, O., Brief, M., Mishaeli, M., and Elisha, O. Fine- tuning or retrieval? comparing knowledge injection in LLMs. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 237–250, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emn lp-main.15. URLhttps://aclanthology.org /2024.emnlp-main.15/. Pati ̃ no, C. M., Rasul, K., Gallou ́ edec, Q., Burtenshaw, B., Paniego, S., Srivastav, V., Frere, T., Beeching, E., Tun- stall, L., von Werra, L., and Wolf, T. Unlocking on-policy distillation for any model family, 2025. Pham, M., Cho, M., Joshi, A., and Hegde, C. Revisiting self-distillation, 2022. URLhttps://arxiv.org/ abs/2206.08491. Qu, Y., Zhang, T., Garg, N., and Kumar, A. Recursive introspection: Teaching language model agents how to self-improve. In Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024. URLht tps://openreview.net/forum?id=DRC9pZ wBwR. Ross, S., Gordon, G. J., and Bagnell, J. A. A reduction of imitation learning and structured prediction to no-regret online learning, 2011. URLhttps://arxiv.org/ abs/1011.0686. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017. URLhttps://arxiv.org/abs/1707.0 6347. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. Shi, W., Chen, Y., Li, Z., Pan, X., Sun, Y., Xu, J., Zhou, X., and Li, Y. R 3 l: Reflect-then-retry reinforcement learning with language-guided exploration, pivotal credit, and positive amplification, 2026. URLhttps://ar xiv.org/abs/2601.03715. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=vAEl hFcKW6. Shridhar, M., Yuan, X., C ˆ ot ́ e, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. Alfworld: Aligning text and embodied environments for interactive learning, 2021. URL https://arxiv.org/abs/2010.03768. Song, Z., Yan, B., Liu, Y., Fang, M., Li, M., Yan, R., and Chen, X. Injecting domain-specific knowledge into large language models: A comprehensive survey. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2025, p. 25297–25311, Suzhou, China, November 2025. Association for Com- putational Linguistics. ISBN 979-8-89176-335-7. doi: 1 0 . 1 8 6 5 3 / v 1 / 2 0 2 5 . f i n d i n g s- e m n l p . 1 3 7 9.URL https://aclanthology.org/2025.findin gs-emnlp.1379/. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 technical report, 2025. URLhttps: //arxiv.org/abs/2505.09388. Yang, Z., Pang, T., Feng, H., Wang, H., Chen, W., Zhu, M., and Liu, Q. Self-distillation bridges distribution gap in language model fine-tuning. In Ku, L.-W., Martins, A., 11 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), p. 1028–1043, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.58. URL https://aclanthology.org/2024.acl-lon g.58/. Yao, S., Chen, H., Yang, J., and Narasimhan, K. Web- shop: Towards scalable real-world web interaction with grounded language agents, 2023. URLhttps://ar xiv.org/abs/2207.01206. Yao, W., Heinecke, S., Niebles, J. C., Liu, Z., Feng, Y., Xue, L., N, R. R., Chen, Z., Zhang, J., Arpit, D., Xu, R., Mui, P. L., Wang, H., Xiong, C., and Savarese, S. Retroformer: Retrospective large language agents with policy gradient optimization. In The Twelfth International Conference on Learning Representations, 2024. URLhttps://open review.net/forum?id=KOZu91CzbK. Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., and Huang, G. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URLhttps: //openreview.net/forum?id=4OsgYD7em5. Zheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B., Gao, C., Dang, K., Liu, Y., Men, R., Yang, A., Zhou, J., and Lin, J. Group sequence policy optimization, 2025. URL https://arxiv.org/abs/2507.18071. 12 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning A. Detailed Experimental Results This section provides detailed numerical results for the main experiments presented in the paper. A.1. Mathematical Reasoning Results Table 5. Performance comparison on mathematical reasoning benchmarks. All methods start from the same SFT-400K checkpoint and are trained with identical prompt template, sampling temperature, max tokens, batch size, rollouts per prompt, and learning schedule (Appendix B.8); the only variable is the teacher distribution. We report mean over 5 independent seeds; the±values are paired-bootstrap standard deviations over 10 000 resamples. “Train FLOPs” is the full Stage 1+Stage 2 cost (Appendix B.4), normalized to GRPO. † Inference-time methods with 3 refinement iterations. MethodAIME’24MATH-500GSM8KDeepScaleRTrain FLOPs SFT-400K (init)45.0±1.952.4±1.069.2±0.628.3±1.4– PPO55.3±2.165.8±1.080.7±0.541.2±1.31.2× GRPO68.0±1.772.2±0.888.4±0.453.8±1.11.0× OPD (Qwen3-32B teacher)70.0±1.578.6±0.692.1±0.355.4±1.00.11× OPD (Qwen3-72B teacher)72.5±1.480.1±0.593.0±0.355.8±1.00.23× GRPO + prompt reflection69.5±1.673.4±0.889.0±0.454.1±1.11.0× OPD + prompt reflection71.8±1.479.0±0.692.4±0.355.6±1.00.12× SCoRe (Kumar et al., 2025)70.2±1.577.9±0.791.5±0.454.0±1.00.34× RISE (Qu et al., 2024)69.8±1.677.2±0.791.1±0.453.7±1.10.28× R 3 L (Shi et al., 2026)71.5±1.578.4±0.691.8±0.454.5±1.00.21× Reflexion † 66.7±1.776.4±0.791.0±0.452.6±1.1– Self-Refine † 65.0±1.875.2±0.790.5±0.451.8±1.2– SRPO (Ours)73.3±1.481.2±0.593.8±0.359.7±0.90.26× p-values vs. GRPO (paired bootstrap) < 0.005 < 0.001 < 0.001 < 0.001— p-values vs. OPD-32B 0.008< 0.001 < 0.001 < 0.001— A.2. Long-Horizon Agentic and Code Task Results A.3. Full Fine-Tuning vs. LoRA 13 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning Table 6. Performance on long-horizon agentic and code-reasoning benchmarks. Success rate / pass@1 (%) is reported. Live- CodeBench (Jain et al., 2024) is evaluated on a contamination-free Aug 2025 – Jan 2026 slice (240 problems). Mean over 5 seeds;±are paired-bootstrap standard deviations. MethodWebShopALFWorldSWE-LiteLiveCodeBenchAvg. Steps Qwen3-8B (base)42.3±1.451.2±1.718.7±1.322.4±1.212.4 SFT (expert traj.)56.8±1.268.4±1.424.3±1.226.1±1.110.8 GRPO51.2±1.162.7±1.522.1±1.228.4±1.011.6 PPO49.8±1.360.3±1.721.5±1.327.0±1.111.9 OPD (Qwen3-32B teacher)57.3±0.969.2±1.326.4±1.131.2±0.911.0 OPD (Qwen3-72B teacher)61.8±0.972.4±1.228.6±1.032.4±0.910.7 SCoRe (Kumar et al., 2025)55.6±1.063.8±1.423.7±1.229.4±1.013.6 RISE (Qu et al., 2024)56.2±1.065.5±1.324.0±1.229.8±1.012.5 R 3 L (Shi et al., 2026)58.9±1.066.4±1.325.3±1.130.5±1.011.8 Reflexion † 58.4±1.271.2±1.326.8±1.230.1±1.014.2 Self-Refine † 55.6±1.267.8±1.425.1±1.229.2±1.113.8 SRPO (Ours)64.7±0.876.8±1.231.2±1.035.1±0.810.2 Table 7. Comparison of full fine-tuning vs. LoRA adaptation for SRPO. LoRA shows competitive performance with significantly reduced memory footprint, though full fine-tuning maintains an advantage on the most challenging benchmarks. ModelMethodAIME’24MATH-500ParamsMemory Qwen3-8B Full FT73.381.28.0B64GB LoRA-6470.079.452M18GB LoRA-12871.780.1104M22GB Qwen3-32B Full FT82.087.332B256GB LoRA-6478.385.1210M68GB LoRA-12879.786.0420M78GB B. Theoretical Insights and Computational Efficiency This section provides a detailed theoretical treatment of SRPO, including information-theoretic analysis, formal justification of the self-distillation mechanism, and computational cost analysis. B.1. Information-Theoretic Perspective The efficiency advantage of our approach can be understood through an information-theoretic lens. Standard RL with sparse terminal rewards provides at mostO(1)bits of supervision per episode—regardless of trajectory length, the model only learns whether the final outcome was successful. In contrast, on-policy distillation with token-level KL rewards provides O(T )bits per episode, whereTis the sequence length. ThisT-fold increase in supervision density translates directly to improved sample efficiency: empirically, distillation-based methods achieve equivalent performance with 10–100×fewer gradient steps compared to sparse-reward RL. A key theoretical motivation for SRPO stems from the observation that on-policy self-distillation inherently circumvents the fundamental limitations of existing post-training paradigms. Unlike pointwise RL with sparse terminal rewards, our framework provides dense supervision at every token position, dramatically improving sample efficiency. Simultaneously, by training on student-generated trajectories rather than fixed teacher demonstrations, we eliminate the exposure bias inherent in standard SFT and ensure the model learns to recover from its own distributional drift. Moreover, since our teacher is simply the same model operating under reflection-augmented conditions, SRPO establishes a principled self-improvement loop that requires no access to larger or more capable external models. 14 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning B.2. Why Self-Distillation Works A natural question arises: how can the model serve as its own teacher? The answer lies in the asymmetry introduced by reflection. Consider a trajectory where the model makes a suboptimal decision at stept ∗ . Without hindsight, the model at stept ∗ lacks information about downstream consequences. With reflection patchp, which explicitly encodes the outcome and diagnosis, the model at stept ∗ effectively has access to future information—transforming an originally difficult decision into a more informed one. Formally, letI(a t ∗ ;o| s t ∗ )denote the mutual information between the optimal action and the outcome given the current state. The reflection patch makes this information explicit: I(a t ∗ ;o| s t ∗ ,p)≫ I(a t ∗ ;o| s t ∗ ).(11) Self-distillation then transfers this information advantage back into the unconditional policy. Through iterative training, the student policyπ θ (·| x)gradually learns to approximate the behavior ofπ θ (·| [p;x]), effectively internalizing the reasoning patterns that were originally guided by explicit reflection. B.3. Avoiding Degenerate Solutions A potential concern is that the model might learn trivial solutions, such as ignoring the reflection entirely or producing reflections that do not generalize. We address this through several design choices: (i) The reflection is generated before seeing the rethinking rollout, preventing information leakage from future tokens. (i) We train on diverse prompts, encouraging generalizable reflection patterns rather than prompt-specific memorization. (i)The on-policy nature of sampling ensures the model learns from states it actually visits, not idealized teacher trajectories. (iv)The compact form of reflection patches (2–5 bullet points) prevents the model from simply memorizing verbose solutions. Empirical evidence (Section 4.3 in the main paper) demonstrates that reflections generalize across problem types and that the learned policy maintains strong performance even when reflection is removed at inference time. B.4. Complete FLOPs Breakdown Earlier drafts of this paper reported a “∼10×fewer FLOPs than GRPO” figure based on Stage 2 alone. For the camera- ready, we report the full Stage 1 + Stage 2 cost, including all preprocessing (Table 8). FLOPs are computed as6NDper forward/backward pass, whereN = 8.03B is the Qwen3-8B parameter count andDis the number of tokens processed per component (average sequence length×rollouts×prompts). Stage 1 components each process the full training set (16K prompts) once; Stage 2 runs 500 iterations with batch size 256. Three clarifying notes. First, Stage 1 is a one-time, embarrassingly parallel preprocessing step. The reflection patchpand the optional rethinking rollout ̃yfor each prompt are fully independent and can be computed offline. In practice, updating the reflection cache every 2–3 epochs maintains performance while reducing overhead by an additional 30–40%. Second, Stage 2 training does not consume ̃y. The teacherπ T = π θ (·| [p;x])is evaluated via teacher-forcing on the student’s on-policy tokens (Eq. 6, main paper), so the only per-iteration teacher cost is a forward pass to obtainlogπ T (a t | s t ). We retain ̃yin the Stage 1 budget because it is generated alongsidepin our pipeline and underpins the quality-gap analysis that empirically validatesπ T . A leaner variant that skips ̃ygeneration would cut total cost from5.4to4.2× 10 18 FLOPs (SRPO/GRPO ratio∼4.95× instead of∼3.8×). Third, the FLOPs ratio understates the practical advantage. Because Stage 1 parallelizes trivially and Stage 2 requires∼4× fewer iterations than GRPO (500vs.2 000+), end-to-end wall-clock time is∼7.5×faster on8×H100 GPUs ( ̃8GPU-hours for SRPO vs. ̃60+ GPU-hours for GRPO). Per-iteration, the student/teacher share weights, so memory remains within a single-GPU budget. Role of the rethinking rollout. The rethinking rollout ̃yplays three roles in our pipeline, none of which is to serve as a training target. (i) Quality validation: comparing ̃yagainst the initial rolloutyconfirms that the reflection patch produces a 15 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning Table 8. Complete FLOPs breakdown for SRPO (Qwen3-8B, AIME’24 training). Stage 1 is one-time, embarrassingly parallel preprocessing that can be cached across epochs; only Stage 2 is repeated each training run. The rethinking rollout ̃yis reported for full transparency but is not consumed as a training target in Stage 2, which uses only teacher-forced scoring of student tokens (see Eq. 6 in the main paper). Removing this row would cut Stage 1 cost by 22.2% without altering the Stage 2 objective. ComponentFLOPs (×10 18 )% of SRPO total Stage 1 (one-time preprocessing) Initial rollout generation1.222.2% Reflection patch generation0.611.1% Rethinking rollout (quality validation)1.222.2% Stage 2 (500 iterations) Student on-policy rollout1.222.2% Teacher logits + KL optimization1.222.2% SRPO total5.4100% GRPO (2 000 iterations)20.8— SRPO / GRPO∼3.8× fewer FLOPs strictly stronger conditional distribution, justifying our choice of teacher. (i) Diagnostic analysis: ̃ysupports the reflection- quality and ablation studies reported elsewhere in this appendix. (i) Implementation amortisation: sampling ̃yshares activations with the teacher-forcing pass used in Stage 2, so amortising the cost is essentially free in our implementation. The Stage 2 training objective (Eq. 6) readslogπ T (a t | s t )wherea t is the student’s own on-policy token; ̃ynever appears in this expression. B.5. Statistical Methodology All main-paper numbers are reported as the mean over 5 independent seeds. Seeds differ in prompt shuffling, dropout masks, and Monte-Carlo sampling of rollouts; the SFT-400K initialization and all hyperparameters are held fixed. Confidence intervals are obtained by paired bootstrap over 10 000 resamples of the per-instance score vectors;p-values are reported for paired comparisons against GRPO and OPD-32B. On AIME’24 (30 problems) we acknowledge that the small sample size inflates per-seed variance, but SRPO outperforms GRPO in 5/5 seeds with the per-instance gain stochastically dominating the GRPO distribution (p < 0.005). On larger benchmarks (MATH-500, GSM8K, WebShop, ALFWorld, LiveCodeBench) all gains achieve p < 0.001. Per-category MATH-500 results show consistent improvements across all 7 subdomains (algebra +6.4, counting/probability +8.1, geometry +7.8, intermediate algebra +9.2, number theory +9.6, precalculus +8.4, prealgebra +7.0), ruling out cherry-picking of favorable categories. B.6. Data Contamination Audit We performed an 8-gram overlap audit between the OpenThoughts-3 training set and every evaluation benchmark used in this paper. For each benchmark instance, the problem text was tokenized with the Qwen3 tokenizer and any 8-gram appearing in the training set was flagged. Results are in Table 9. Table 9. Data contamination audit. 8-gram overlap between OpenThoughts-3 training data and each evaluation benchmark. Numbers in parentheses for MATH-500 indicate the 3 flagged 8-grams, which on manual inspection were common mathematical phrases (e.g. “find the value of x such that”), not problem-level contamination. BenchmarkInstances8-gram overlapsOverlap rate AIME’243000.0% MATH-50050030.6% (common phrases) GSM8K1 31900.0% DeepScaleR1 20000.0% GPQA Diamond44800.0% LogiQA 2.01 57200.0% LiveCodeBench24000.0% 16 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning Removing the 3 flagged MATH-500 instances yields 81.0 (vs. 81.2 with them included), a negligible 0.2-point shift well within the bootstrap standard deviation. Our strongest OOD result, on DeepScaleR (0% overlap, 1 200 unseen competition problems), shows SRPO reaching 59.7 vs. OPD-32B 55.4 and OPD-72B 55.8 — providing a contamination-free reference point for the entropy-expansion claim in Section ??. B.7. Multi-Verifier and Human Evaluation of Reflection Quality To validate that the reflection-quality measurements used in the main paper are not artifacts of a single LLM evaluator, we performed both multi-verifier cross-validation and human evaluation. Multi-verifier cross-validation. We re-scored all 500 sampled AIME’24 reflections with three independent LLM evaluators: Deepseek-V3.2 (the original judge), GPT-5.2, and Qwen3.5-397B-A17. Each evaluator returned an integer score in 1,..., 5on the same rubric. Pairwise agreement rates (defined as identical scores or scores within±1) are 92.4% (DeepSeek vs. GPT-5.2), 91.8% (DeepSeek vs. Qwen3.5), and 93.2% (GPT-5.2 vs. Qwen3.5). The cross-evaluator agreement well exceeds the 75–80% range typical for LLM-as-judge protocols, supporting the validity of the single-evaluator scoring in the main paper. Human evaluation. Two expert annotators (senior PhD students in NLP/LLMs) blind-rated 100 reflections — 50 from AIME’24 and 50 from WebShop — on a ternary scale of Effective / Redundant / Detrimental, with Effective requiring an actionable, problem-specific diagnosis. Cohen’sκbetween annotators is0.81, indicating substantial agreement. Distribution: 68–74% Effective, 18–22% Redundant, 8–10% Detrimental. On a subset of 50 reflections where the initial trajectory was wrong, annotators measured a 52–58% Wrong→Correct fix rate after reflection. The human-identified detrimental rate (8–10%) is consistent with the automatic evaluator’s low-quality threshold (score≤ 2, also 8%), confirming the automatic protocol is well-calibrated and does not systematically favor plausible-sounding reflections. No quality collapse across training. Tracking reflection helpfulness across 500 training iterations shows it is stable: Iter 100 mean3.72± 0.41, Iter 250 mean3.76± 0.38, Iter 500 mean3.79± 0.34. There is no downward drift, ruling out a hypothesized failure mode in which the self-teacher quality degrades as the student approaches it. B.8. OPD Baseline Fairness Controls To ensure that the SRPO vs. external-teacher OPD comparison reflects the design variable (teacher distribution) and not incidental confounds, every OPD-style baseline in this paper uses the alignment matrix in Table 10. Hyperparameters were taken from the OPD/OPSD reference implementations where available; we made no separate tuning pass for SRPO. Table 10. Experimental controls for the SRPO vs. external-teacher OPD comparison. Every variable other than the teacher distribution is held identical across runs. VariableSRPO & OPDAligned? Prompt templatesystem + task✓ Sampling temperature0.7✓ Top-p0.95✓ Max tokens4 096✓ Rollouts per prompt4 (best-of-4)✓ Batch size256✓ Learning rate5× 10 −6 (SRPO) / 2× 10 −6 (72B-OPD ∗ )note ∗ Training iterations500✓ Regeneration depthFull reset (no suffix-only repair)✓ Design variableπ θ (·| [p;x]) vs. π ext (·| x)— ∗ For OPD with the Qwen3-72B teacher, the standard 5× 10 −6 learning rate caused training instability at the K = 0.67 teacher-student KL gap; gradient clipping at 1.0 and a reduced LR of 2× 10 −6 were necessary. SRPO runs at the standard LR. The∼90-token reflection patch is part of the method, not a confound: the semantic-control experiments in Appendix C show that random patches of the same token length yield the same performance as no reflection (66.5vs.65.8on AIME’24), so the gain cannot be attributed to additional context tokens. 17 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning B.9. Impact of Low-Quality Reflections A natural concern with self-reflection as a training signal is that low-quality reflections might amplify errors. We tested this by filtering out reflections automatically scored≤ 2(the bottom 8%) and retraining SRPO on the remaining 92%. The result on AIME’24 is 73.6 vs. the unfiltered 73.3, a negligible +0.3-point shift well within the bootstrap standard deviation. The training dynamics explain why: uninformative reflection patches produce teacher distributionsπ θ (·| [p;x])very close to the unconditioned student distributionπ θ (·| x), so the per-token KL is near zero, the gradient signal is near zero, and these reflections are effectively down-weighted by the loss itself. SRPO is therefore robust to the noise floor of its own self-teacher; aggressive filtering of low-quality reflections is not required for the method to work. B.10. Solution-Diversity Protocol Table 11. Solution-diversity probe on the 30 AIME’24 problems (10 samples per problem). The reflection-conditioned self-teacher expands the student’s reasoning-path entropy rather than contracting it, in contrast to a larger external teacher. MethodUnique Paths / ProblemNovel Correct (%) OPD (Qwen3-72B teacher)2.38.4 SRPO (self-reflection)4.115.7 The diversity numbers reported in Table 11 are obtained as follows. For each of the 30 AIME’24 problems we drewn = 10 samples per method at temperature 1.0 (matched), then computed two quantities. Unique reasoning paths per problem. Each sample is summarized by the embedding sequence of its intermediate solution steps using a fixed sentence-encoder (text-embedding-3-large). We agglomeratively cluster the 10 step-sequence embeddings per problem with cosine-distance threshold0.25(chosen by visual inspection of within-cluster step-sequence overlap on a held-out set) and report the number of resulting clusters, averaged over the 30 problems. Novel correct solutions. For each problem, we mark a sample as “novel” if its cluster does not overlap with any cluster present in the OPD-72B sample set for the same problem (using the same threshold). The percentage of correct novel samples among the 10 samples is then averaged across problems. The contrast between the two methods — 2.3 vs. 4.1 unique paths, 8.4% vs. 15.7% novel-correct rate — is consistent across re-runs with different random seeds and across embedding models (text-embedding-3-large, BGE-large, E5-mistral), with ≤ 0.3-cluster variation between runs. 18 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning C. Reflection Semantics Control Experiments This section provides comprehensive experimental details and extended analysis for the reflection semantics control experiments. These experiments directly address the concern that SRPO’s performance gains might stem from additional context tokens or teacher forcing artifacts rather than the semantic content of reflections. C.1. Experimental Design We design a series of controlled ablations that systematically disrupt the semantic alignment between reflections and problem instances while preserving other factors (format, token count, training procedure). Mismatched Reflection Protocol. For each training batch containing N instances(x i ,τ i ,o i ) N i=1 : 1. Generate reflection patches normally: p i = Reflect(x i ,τ i ,o i ) for all i. 2. For each instance i, randomly sample j ̸= i from the same batch. 3. Construct mismatched teacher input: ̃x mis i = [p j ;x i ]. 4. Generate teacher distribution: π mis T (·| x i ) := π θ (·| ̃x mis i ). This protocol ensures that: (a) the mismatched reflections are structurally valid (generated by the same reflection process), (b) the token length distribution matches that of correct reflections, and (c) the only difference is the semantic relevance to the current problem. Token Length Matching. To eliminate confounds from varying context lengths, we filter mismatched pairs to ensure: |len(p j )− len(p i )|≤ 0.1· len(p i ).(12) Figure 8(b) confirms that the resulting token length distributions are statistically indistinguishable (Kolmogorov-Smirnov test, p > 0.85). Additional Ablation Conditions. Beyond mismatched reflections, we evaluate: • Shuffled Words: Randomly permute words within each reflection sentence while preserving sentence boundaries. • Shuffled Sentences: Randomly permute the order of bullet points/sentences in the reflection. • Template Only: Use a fixed generic reflection applicable to any problem: “• Carefully verify each computational step. • Check boundary conditions and edge cases. • Ensure the final answer addresses the original question.” • Outcome Only: Replace reflection with a simple outcome indicator: “Previous attempt was [correct/incorrect].” C.2. Quantitative Results Table 12 presents comprehensive results across all conditions. Key observations: Mismatched≈No Reflection. The mismatched reflection condition (66.5% on AIME’24) performs nearly identically to the no-reflection baseline (65.8%), with the 0.7-point difference falling within error margins. This directly refutes the hypothesis that performance gains come from “extra tokens” or format-induced regularization. Content Hierarchy.We observe a clear hierarchy: matched>template>outcome-only≈mismatched≈no-reflection. Template-only reflections provide marginal benefit (+2.0 over baseline), suggesting that even generic meta-cognitive prompting has weak positive effects. However, this effect is substantially smaller than task-specific reflections (+7.5), confirming that specific semantic content is essential. 19 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning SRPO (Matched) Mismatched Reflection Template Only No Reflection 0 10 20 30 40 50 60 70 80 Performance (%) 73.3 66.5 67.8 65.8 64.7 55.8 57.2 54.2 = 6.8 baseline ( = 0.7) (a) Reflection Semantics Ablation AIME'24 WebShop 50100150200250 Reflection Token Length 0 5 10 15 20 25 Frequency Matched: =95, =37 Mismatched: =101, =45 Same length distribution Gap from semantics only (b) Token Length Distribution Matched (SRPO) Mismatched Template (=84) Figure 8. Reflection semantics control experiment. (a) Performance comparison across reflection conditions: mismatched reflections (from unrelated problems) and template-only reflections yield performance comparable to no reflection, while matched SRPO shows significant gains. (b) Token length distributions are matched across conditions, isolating the effect of semantic content. Table 12. Comprehensive reflection semantics ablation results. Performance drops sharply when reflections lack task-specific semantic content, regardless of format or token count preservation. ConditionAIME’24WebShopAvg. Tokens∆ vs Full SRPO (Full, Matched)73.3± 0.864.7± 1.592.4– Semantic Disruption Shuffled Words66.2± 1.255.4± 1.992.4-7.1 / -9.3 Shuffled Sentences66.8± 1.156.1± 1.792.4-6.5 / -8.6 Mismatched Reflection66.5± 1.055.8± 1.891.8-6.8 / -8.9 Content Reduction Template Only67.8± 0.957.2± 1.685.0-5.5 / -7.5 Outcome Only67.2± 1.056.8± 1.712.3-6.1 / -7.9 Baselines No Reflection65.8± 1.154.2± 1.90-7.5 / -10.5 Direct Retry65.8± 1.154.2± 1.90-7.5 / -10.5 Semantic Coherence Matters. Shuffled-words and shuffled-sentences conditions perform comparably to mismatched reflections, indicating that disrupting coherence—even within originally correct reflections—eliminates most of the benefit. This suggests the model does not simply extract keywords but relies on the coherent reasoning structure. C.3. Content Analysis To understand what makes reflections effective, we analyze 200 matched and mismatched reflection pairs using GPT-4 as an evaluator. Task-Specific Information. We classify reflection content into four categories: • Task-specific numbers: References to specific values, coefficients, or quantities from the problem. • Constraint mentions: Explicit constraints, boundary conditions, or requirements. • Error diagnosis: Identification of specific mistakes in the previous attempt. • Generic advice: General problem-solving strategies applicable to any task. 20 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning SRPO (Full) Shuffled Words Shuffled Sentences Mismatched Reflection Template Only Outcome Only No Reflection 60.0 62.5 65.0 67.5 70.0 72.5 75.0 77.5 80.0 AIME'24 Accuracy (%) (a) Comprehensive Reflection Ablation (AIME'24) No Reflection Baseline SRPO (Full) Mismatched Reflection Template Only Outcome Only No Reflection 45 50 55 60 65 70 75 WebShop Success Rate (%) -13.8% -11.6% -12.2% -16.2% (b) Reflection Ablation (WebShop) 0100200300400500 Training Iteration 40 45 50 55 60 65 70 75 80 AIME'24 Accuracy (%) (c) Training Dynamics SRPO (Matched) Mismatched Template Only No Reflection Task-Specific Numbers Constraint Mentions Error Diagnosis Generic Advice 0 20 40 60 80 100 Relevance to Problem (%) Mismatched reflections lack task-relevant information (d) Reflection Content Analysis Matched (Relevant %) Mismatched (Relevant %) Figure 9. Extended reflection semantics analysis. (a) Comprehensive ablation on AIME’24 showing that all semantic disruption conditions converge to baseline performance. (b) WebShop results confirm the pattern generalizes to agentic tasks. (c) Training dynamics reveal that semantic-aligned reflections enable faster convergence. (d) Content analysis shows mismatched reflections lack task-relevant information despite identical format. Figure 9(d) shows that matched reflections contain task-relevant information in 78–91% of cases, while the same content in mismatched reflections is relevant to the actual problem only 8–15% of the time (by chance). This explains the performance gap: mismatched reflections provide irrelevant “noise” that the model must learn to ignore. Qualitative Example. Consider a geometry problem asking for the area of a triangle with vertices at(0, 0),(4, 0), and (2, 3). Matched reflection: ”• The base is along the x-axis with length 4. • Height is the perpendicular distance from (2,3) to the x-axis, which is 3. • Apply formula: Area = 1 2 × 4× 3 = 6.” Mismatched reflection (from a number theory problem): ”• Consider modular arithmetic with respect to 7. • Check divisibility conditions for each candidate. • The answer must satisfy both congruence relations.” The mismatched reflection, while grammatically correct and mathematically valid for its original problem, provides no actionable guidance for the geometry problem. 21 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning C.4. Training Dynamics Figure 9(c) shows training curves for different conditions. Key observations: Faster Convergence with Semantic Alignment. Matched reflections achieve 70% accuracy in approximately 250 iterations, while mismatched and no-reflection conditions require>500 iterations to plateau at a lower level (∼66%). This suggests that semantically relevant reflections provide a stronger and more consistent training signal. No Late-Stage Recovery. The mismatched condition does not “catch up” with extended training, confirming that the performance gap is not due to slower learning but fundamental limitations in the training signal quality. C.5. Implications and Conclusions These experiments provide strong evidence for the following claims: 1.Semantic content is essential: Performance gains from SRPO depend critically on the semantic alignment between reflection content and the specific problem instance. 2.Format and tokens are insufficient: Matching the format, style, and token count of reflections does not transfer benefits; only task-relevant semantic content produces improvements. 3. Reflections enable credit assignment: The effectiveness of semantically aligned reflections suggests they successfully perform “soft credit assignment” by highlighting decision points and error sources that would otherwise be difficult to identify from sparse terminal feedback. 4.No teacher forcing artifacts: The failure of mismatched reflections rules out explanations based on teacher forcing regularization or exposure to “future information”—if such mechanisms were responsible, mismatched reflections should provide similar benefits. These findings directly address the reviewer concern that SRPO’s gains might stem from “just adding more text” and provide experimental grounding for the theoretical claims about information flow in the reflection-distillation pipeline. 22 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning D. Qualitative Case Study We provide a representative example illustrating how a concise reflection patch can correct a systematic mistake (misinter- preting the averaging window). The original benchmark instance contains typographical noise; for readability, we show a cleaned rendering below. D.1. Question. Choose the most realistic answer. Beth places 4 whole ice cubes in a frying pan at the start of the 1st minute, then 5 at the start of the 2nd minute, and some more at the start of the 3rd minute, but none in the 4th minute. If the average number of ice cubes per minute placed in the pan while it was frying was 5, how many whole ice cubes can be found in the pan at the end of the 3rd minute? A. 30 B. 0 C. 20 D. 10 E. 11 D.2. Initial Responses Qwen2.5-7B-instruct: Let’s break down the problem step by step. 1. ** Understanding the average ** : The average number of ice cubes placed per minute is 5. This means that over the 3 minutes, the total number of ice cubes placed is $5 × 3 = 15$. 2. ** Counting the ice cubes placed ** : - At the start of the 1st minute, Beth places 4 ice cubes. - At the start of the 2nd minute, Beth places 5 ice cubes. - At the start of the 3rd minute, let’s say Beth places $x$ ice cubes. - At the start of the 4th minute, Beth places 0 ice cubes. 3. ** Setting up the equation ** : The total number of ice cubes placed is $4 + 5 + x + 0 = 15$. 4. ** Solving for $x$ ** : \[4 + 5 + x = 15\] \[9 + x = 15\] \[x = 15 - 9\] \[x = 6\] So, at the start of the 3rd minute, Beth places 6 ice cubes. 5. ** Calculating the total number of ice cubes at the end of the 3rd minute ** : - Ice cubes at the start of the 1st minute: 4 - Ice cubes added at the start of the 2nd minute: 5 - Ice cubes added at the start of the 3rd minute: 6 Thus, the total number of ice cubes at the end of the 3rd minute is $4 + 5 + 6 = 15$. However, the question asks for the number of ice cubes at the end of the 3rd minute, and since the problem specifies that no ice cubes are added in the 4th minute and no ice cubes melt (or are removed) in the frying pan, the total remains the same. Therefore, the correct answer is: ** D. 10 ** This answer is based on the assumption that the problem might have a typo or simplification, as the correct total based on the given conditions is 15, but the closest realistic option provided is D. 10, which might be a result of a potential simplification 23 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning or misinterpretation in the problem statement. Qwen3-8B: <think> Okay, let’s see. So the problem is about Beth putting ice cubes into a frying pan over four minutes. The question is asking how many ice cubes are in the pan at the end of the 3 rd minute, given that the average number of ice cubes per minute placed in the pan while it was frying was 5. The choices are A. 30, B. 0, C. 20, D. 10, E. 11. First, let me make sure I understand the problem correctly. Beth starts putting ice cubes into the pan at the beginning of each minute. In the first minute, she puts 4, then 5 in the second minute, some more in the third minute, and none in the fourth minute. The average number of ice cubes per minute during the frying time is 5. Wait, but how many minutes was the pan frying? The problem says "while it was frying," but it doesn’t specify the duration of frying. Hmm, maybe the frying happens over the four minutes? Or maybe the pan is frying for some time, and during that time, she’s adding ice cubes? Wait, maybe the key is that the average is calculated over the minutes she was adding ice cubes. But the problem says "while it was frying," so perhaps the frying process is happening over those four minutes, and during that time, she added ice cubes at the start of each minute. So the average number of ice cubes per minute placed in the pan during the frying time is 5. But the average is over how many minutes? The problem says "the average number of ice cubes per minute placed in the pan while it was frying was 5." So if she placed ice cubes for 4 minutes (1st, 2nd, 3rd, 4th), then the average would be total ice cubes divided by 4 minutes equals 5. But she didn’t put any in the 4th minute. Wait, so maybe the frying time is only three minutes? Because she didn’t add any in the fourth minute? Or maybe the frying is happening over four minutes, but she added ice cubes in the first three minutes? Wait, the problem says "the average number of ice cubes per minute placed in the pan while it was frying was 5." So maybe the frying process is ongoing, and she added ice cubes at the start of each minute. But the fourth minute she didn’t add any. However, the average is calculated over the time the pan was frying. Hmm, this is a bit confusing. Let me read again. "Beth places 4 whole ice cubes in a frying pan at the start of the 1st minute, then 5 at the start of the 2nd minute, and some more at the start of the 3rd minute, but none in the 4th minute. If the average number of ice cubes per minute placed in the pan while it was frying was 5, how many whole ice cubes can be found in the pan at the end of the 3rd minute?" So the key here is that she placed ice cubes at the start of each minute. So in minute 1, she adds 4. Then minute 2, adds 5. Minute 3 adds some number, let’s call it x. Minute 4 adds 0. So total ice cubes added over the four minutes would be 4 + 5 + x + 0 = 9 + x. But the average is 5 per minute. So average is total ice cubes divided by the number of minutes. Wait, but the problem says "while it was frying." So maybe the frying is happening over the four minutes, but she only added ice cubes in the first three minutes? Wait, but she added none in the fourth minute. So perhaps the frying process is happening for four minutes, and during that time, she added ice cubes at the start of each minute. So the total number of ice cubes added over the four minutes is 4 + 5 + x + 0. Then the average per minute is (total)/4 = 5. Therefore, total = 5 * 4 = 20. So 9 + x = 20. Therefore, x = 11. So then, the total ice cubes added by the end of the 3rd minute would be 4 + 5 + 11 = 20. But wait, the question is asking how many ice cubes can be found in the pan at the end of the 3rd minute. If she added 11 at the start of the third minute, then by the end of the third minute, those 11 would have melted? Or are they still there? Wait, the problem says "the average number of ice cubes per minute placed in the pan while it was frying was 5." So maybe the average is about the rate of adding ice cubes, not about the number in the pan. But the wording is a bit unclear. Let me think again. Alternatively, maybe the average is over the frying time, which might be different from the number of minutes she added ice cubes. Wait, but the problem mentions "at the start of the 1st minute," "start of the 2nd minute," etc., so it’s possible that the frying process is happening over those four minutes, and during each minute, she adds ice cubes at the start. So the total number of ice cubes added over the four minutes is 4 + 5 + x + 0. The average per minute is 5, so total added is 5 * 4 = 20. Therefore, 4 + 5 + x + 0 = 20. Solving for x gives x = 11. So she added 11 ice cubes at the start of the third minute. 24 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning But the question is asking how many ice cubes are in the pan at the end of the third minute. So if she added ice cubes at the start of each minute, then during the third minute, she added 11. But does the ice cube melt during frying? The problem mentions " while it was frying," so maybe the frying is happening over time, and the ice cubes are melting. But the problem doesn’t mention anything about melting. Wait, that’s a problem. If the ice cubes are melting, then the number in the pan would decrease over time. But since the question is about how many are found in the pan at the end of the third minute, maybe we need to consider that the ice cubes are still there? Or maybe the frying is just the process of putting them in, and the melting isn’t considered here? Wait, the problem is a bit ambiguous. Let me check again. The problem says "the average number of ice cubes per minute placed in the pan while it was frying was 5." So "placed in the pan while it was frying" so maybe the frying is happening over the time when she is placing the ice cubes. But if she placed them at the start of each minute, and the frying is ongoing, perhaps the average is calculated over the total time of frying. But the problem doesn’t specify how long the frying lasted. Wait, but the problem mentions minutes 1 to 4. So maybe the frying process is happening for four minutes, and during each minute , she added ice cubes at the start. So the total number of ice cubes added over four minutes is 4 + 5 + x + 0. The average per minute is 5, so total added is 5 * 4 = 20. Therefore, x = 11. But the question is how many ice cubes are in the pan at the end of the third minute. If she added 11 at the start of the third minute, then those are still in the pan unless they melted. But if the problem is not considering melting, then the answer would be 4 + 5 + 11 = 20. But that’s option C. But wait, the answer choices include E. 11. But maybe the problem is not about the total added, but about the average rate of adding ice cubes? Wait, maybe the average is over the time the pan was frying, but the frying time isn’t necessarily four minutes. Wait, but the problem says she placed ice cubes at the start of the first minute, then second, third, and fourth. So maybe the frying process is happening over four minutes, and she added ice cubes at the start of each minute. So the average number of ice cubes per minute placed is 5. So total ice cubes placed over four minutes is 5 * 4=20. Therefore, 4 + 5 + x + 0 = 20. So x=11. Therefore, by the end of the third minute , she has added 4 + 5 + 11 = 20 ice cubes. But if the pan is frying, maybe the ice cubes are melting. But the problem doesn’t mention melting. So maybe the question is just about the total number added, and the answer is 20? But then, why mention "while it was frying"? Maybe that’s just to indicate that the average is calculated during the frying process, which is over four minutes. Alternatively, maybe the average is not about the total number added, but about the rate. Wait, but the average number of ice cubes per minute placed in the pan while it was frying. So maybe it’s total number of ice cubes divided by the number of minutes. So if she placed ice cubes for four minutes, then total is 4 + 5 + x + 0 = 20. Therefore, x = 11. So total in the pan is 20. But then the answer would be C. 20. But maybe the frying process is only three minutes? Because she didn’t add any in the fourth minute. Wait, but the problem says "while it was frying," which could mean the entire time she was adding ice cubes. Alternatively, maybe the average is calculated over the time during which the pan was frying, which is three minutes because she added ice cubes in the first three minutes and then stopped? But the problem says "none in the 4th minute," so maybe the frying continued into the fourth minute, but she didn’t add any ice cubes then. This is confusing. Let me try to think again. The average number of ice cubes per minute placed in the pan while it was frying was 5. So the average is over the duration of frying. But how many minutes was the pan frying? If she added ice cubes at the start of each minute, maybe the frying process is happening over four minutes, and the average is over those four minutes. Therefore, total ice cubes added is 4 + 5 + x + 0 = 20, so x=11. Therefore, total ice cubes in the pan at the end of the third minute would be 4 + 5 + 11 = 20. So answer C. But wait, the problem says "the average number of ice cubes per minute placed in the pan while it was frying was 5." So maybe "placed in the pan" refers to the number of ice cubes added each minute? But then the average would be the average of the number of ice cubes added each minute. If she added 4, 5, x, and 0 over four minutes, then the average is (4 + 25 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning 5 + x + 0)/4 = 5. Then solving that gives x=11, as before. So total ice cubes added is 20. But the question is about how many are in the pan at the end of the third minute. If she added 11 at the start of the third minute, then those are still there unless they melted. But the problem doesn’t mention melting. It just says "how many whole ice cubes can be found in the pan at the end of the 3rd minute?" So maybe the ice cubes are not melting, and the answer is 20. But the problem says "while it was frying," which might imply that the frying is happening, so maybe the ice cubes are melting during frying. But if that’s the case, how much would have melted? The problem doesn’t give any information about the melting rate. So maybe we are supposed to assume that the ice cubes are not melting? Or that the frying process is just the act of putting them in, and the average is about the number added per minute? Alternatively, maybe the average is calculated over the time during which the pan was frying, which is three minutes. Because she didn’t add any in the fourth minute. Wait, but the problem says "while it was frying," so if the frying is happening for four minutes, but she added ice cubes in the first three minutes and none in the fourth. So maybe the average is over the frying time, which is four minutes, but the ice cubes added in the fourth minute is zero. So then, the average would be total ice cubes added divided by four minutes. So total is 20, so 4 + 5 + x + 0 = 20. So x is 11. Then, at the end of the third minute, she has added 4 + 5 + 11 = 20. But if the pan was frying for four minutes, then perhaps during each minute, the ice cubes are melting. However, without any information about the melting rate, we can’t calculate how many are left. Therefore, maybe the problem is not considering melting, and it’s just about the total number of ice cubes added over the four minutes. Wait, but the answer choices include E. 11. Which would be the number of ice cubes added in the third minute. So maybe the problem is not about total added, but about the average number of ice cubes per minute during frying. Wait, but how is that different? Alternatively, maybe the average is calculated over the time the pan was frying, but the time is not four minutes. For example, if the pan was frying for three minutes, then the average would be (4 + 5 + x)/3 = 5. So total would be 15, so 4 + 5 + x = 15, so x = 6. Then total ice cubes in the pan would be 4 + 5 + 6 = 15. But 15 isn’t one of the options. Alternatively, if the average is over the time when she was actively adding ice cubes. But she added ice cubes in minutes 1, 2, 3, and not in 4. So the average is over three minutes. Then (4 + 5 + x)/3 = 5, so total is 15, so x = 6. But again, 15 isn’t an option. Wait, maybe the average is over the entire frying time, which is four minutes, but the ice cubes added in the fourth minute is zero. So total added is 4 + 5 + x + 0 = 20. Then x is 11. So total ice cubes in the pan at the end of the third minute would be 4 + 5 + 11 = 20. So answer C. But why is the fourth minute mentioned? Maybe the frying process is only three minutes? If she added ice cubes at the start of each minute, but the frying process ended at the end of the third minute. Then the average would be over three minutes. Then (4 + 5 + x)/3 = 5. So 9 + x = 15, so x = 6. But again, 6 isn’t an option. Wait, the answer choices are A. 30, B. 0, C. 20, D. 10, E. 11. So maybe the average is calculated over the time when the pan was frying, which is three minutes, but the total is 4 + 5 + x = 5 * 3 = 15, so x = 6. But 6 isn’t an option. Alternatively, maybe the average is over the total time of frying, which is four minutes, but she added ice cubes in the first three minutes. So total is 4 + 5 + x + 0 = 20. So x = 11. Then total ice cubes in the pan at the end of the third minute is 4 + 5 + 11 = 20. So answer C. But if the pan was frying for four minutes, but she added ice cubes at the start of each minute, then the number of ice cubes in the pan at the end of the third minute would be 4 + 5 + x, because the fourth minute hasn’t happened yet. Wait, but the fourth minute is part of the frying time. If she added ice cubes at the start of the fourth minute, but she didn’t add any. So the total ice cubes added is 4 + 5 + x + 0 = 20. Therefore, x is 11. So at the end of the third minute, she has added those 11, so total is 20. 26 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning Alternatively, maybe the problem is about the average number of ice cubes in the pan per minute, not the number placed. But that would be different. For example, if the pan is frying for some time, and ice cubes are melting. But again, without melting information, it’s impossible to calculate. Wait, maybe the problem is not about melting. Maybe "placed in the pan while it was frying " is just referring to the number of ice cubes added per minute, and the average is 5. So over four minutes, she added 4, 5, x, 0. So average is (4 + 5 + x + 0)/4 = 5. Therefore, 9 + x = 20, so x = 11. Then total added is 20. So the number in the pan at the end of the third minute is 4 + 5 + 11 = 20. So answer C. But then why is the fourth minute mentioned? Because the average is over the entire frying time, which includes the fourth minute where she didn’t add any. Therefore, the answer would be 20. But let me check the answer choices again. Option C is 20. Wait, but let me think again. If the average is 5 per minute, and the frying lasted for four minutes, then total ice cubes added is 20. Therefore, the answer is 20. But the problem says "at the end of the 3rd minute," so the fourth minute hasn’t been completed yet. So the fourth minute is part of the frying time, but she didn’t add any ice cubes. Therefore, the total number of ice cubes added during the frying time is still 4 + 5 + x + 0, which is 20. But the number in the pan at the end of the third minute would be 4 + 5 + x. So if x is 11, that’s 20. But if the average is over the frying time, which is four minutes, then the answer is 20. But then why would the answer be 20? Let me check the options again. The options are A. 30, B. 0, C. 20, </think> To solve this problem, we need to carefully interpret the wording and determine what the average of 5 ice cubes per minute refers to. --- ### Understanding the Problem Beth adds ice cubes at the ** start ** of each minute: - ** Start of 1st minute ** : 4 ice cubes - ** Start of 2nd minute ** : 5 ice cubes - ** Start of 3rd minute ** : some number of ice cubes, let’s call it ** x ** - ** Start of 4th minute ** : 0 ice cubes The key phrase is: * "the average number of ice cubes per minute placed in the pan while it was frying was 5." * This suggests that the ** average ** is calculated over the total number of minutes the pan was ** frying ** . Since she placed ice cubes at the ** start ** of each minute, and the ** 4th minute ** is mentioned, we can infer that the frying process lasted ** four minutes ** . Therefore, the average is over ** four minutes ** . --- ### Calculating Total Ice Cubes Placed Let’s denote the total number of ice cubes placed over the four minutes as: $$ Total = 4 + 5 + x + 0 = 9 + x $$ The average number of ice cubes per minute is given as 5, so: 27 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning $$ 9 + x4 = 5 $$ Multiply both sides by 4: $$ 9 + x = 20 $$ Solve for $ x $: $$ x = 11 $$ So, Beth added ** 11 ice cubes at the start of the 3rd minute ** . --- ### Determining Ice Cubes in the Pan at the End of the 3rd Minute The question asks: * how many whole ice cubes can be found in the pan at the end of the 3rd minute? * Since the ice cubes are added ** at the start ** of each minute, and we’re not told that any of them melted during frying, we can reasonably assume that ** the ice cubes remain in the pan ** until the end of the 3rd minute. That is, the ** 11 ice cubes added at the start of the 3rd minute ** are still in the pan at the end of that minute. Therefore, the total number of ice cubes in the pan at the end of the 3rd minute is: $$ 4 + 5 + 11 = 20 $$ --- ### Final Answer $$ C $$ We then do reflection step: Based on the answers is B, reflect on and check the existing reasoning process, and summarize a short tip to help oneself answer correctly next time. Note that the tips cannot contain cheating content, such as directly exposing the answers in the tips. Using <reflection></reflection> to summarize the tips. <reflection> To ensure accuracy in such problems, it’s crucial to ** clearly distinguish between the timing of actions and the duration of the process ** . In this case, the key was understanding that the average was calculated over the ** entire frying time (4 minutes) ** , not just the minutes when ice cubes were added. This means that even though no cubes were added in the 4th minute, the average still includes that full minute. Therefore, the total number of cubes added is 20, and the ice cubes placed at the start of each minute ( including the 3rd minute) remain in the pan unless otherwise stated (e.g., melting or removal). A useful tip is to always verify the time frame of the average and ensure that all events (like cube additions) are included in the calculation, unless the problem explicitly indicates otherwise. This helps avoid misinterpretations and ensures that the final count reflects what is actually asked. 28 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning </reflection> D.3. Reflection patch (example). <reflection> To ensure accuracy in such problems, it’s crucial to ** clearly distinguish between the timing of actions and the duration of the process ** . In this case, the key was understanding that the average was calculated over the ** entire frying time (4 minutes) ** , not just the minutes when ice cubes were added. This means that even though no cubes were added in the 4th minute, the average still includes that full minute. Therefore, the total number of cubes added is 20, and the ice cubes placed at the start of each minute ( including the 3rd minute) remain in the pan unless otherwise stated (e.g., melting or removal). A useful tip is to always verify the time frame of the average and ensure that all events (like cube additions) are included in the calculation, unless the problem explicitly indicates otherwise. This helps avoid misinterpretations and ensures that the final count reflects what is actually asked. </reflection> 29 SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning E. Continual Learning and Catastrophic Forgetting We evaluate SRPO’s behavior in a continual learning setting where models must acquire new capabilities without forgetting previous skills. Table 13. Continual learning evaluation. Models are first trained on math (AIME’24) then adapted to code (SWE-Lite). SRPO better preserves original capabilities while acquiring new skills. MethodBefore AdaptationAfter Code Adaptation MathCodeMath (Retention) SFT60.028.448.2 (80.3%) GRPO68.026.759.3 (87.2%) SRPO73.331.269.8 (95.2%) Setup. We first train Qwen3-8B on mathematical reasoning, then adapt to coding tasks (SWE-Bench-Lite). We measure both the new capability acquisition and retention of original mathematical reasoning ability. Results. Table 13 shows that SRPO achieves 95.2% retention of mathematical reasoning performance after code adaptation, compared to 87.2% for GRPO and 80.3% for SFT. This improved retention can be attributed to SRPO’s on-policy learning: by training on the model’s own distribution, SRPO naturally maintains behaviors that the model already performs well. Connection to Prior Work. This finding aligns with observations in the on-policy distillation literature (Lu & Lab, 2025): on-policy methods cause less catastrophic forgetting than off-policy approaches because they do not force the model to imitate out-of-distribution behaviors. 30

Instant research discovery