Paper deep dive
Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training
Zhengyao Gu, Jonathan Light, Raul Astudillo, Ziyu Ye, Langzhou He, Henry Peng Zou, Wei Cheng, Santiago Paternain, Philip S. Yu, Yisong Yue
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 2:33:25 PM
Summary
The paper introduces Actor-Curator, a scalable, automated framework for reinforcement learning post-training of large language models. It addresses the challenge of curriculum learning by formulating problem selection as a non-stationary stochastic bandit problem. A neural curator dynamically selects training problems from large datasets to maximize expected policy performance improvement, optimized via online stochastic mirror descent with partial feedback. The method achieves significant performance gains and speedups on reasoning benchmarks compared to uniform sampling and other baselines.
Entities (9)
Relation Signals (7)
Actor-Curator → achievesgainon → AIME2024
confidence 95% · Notably, it achieves relative gains of 28.6% on AIME2024
Actor-Curator → achievesgainon → ARC-1D
confidence 95% · and 30.5% on ARC-1D over the strongest baseline
Actor-Curator → appliedto → Large Language Models
confidence 95% · curriculum learning framework for reinforcement learning post-training of large language models (LLMs).
Actor-Curator → uses → Non-stationary Stochastic Bandit
confidence 95% · We formulate problem selection as a non-stationary stochastic bandit problem
Actor-Curator → employs → Online Stochastic Mirror Descent
confidence 90% · derive a principled loss function based on online stochastic mirror descent
Actor-Curator → optimizes → Policy Improvement
confidence 90% · dynamically selects training problems from large problem banks by directly optimizing for expected policy performance improvement.
Actor-Curator → outperforms → Uniform Sampling
confidence 90% · ACTOR-CURATOR consistently outperforms uniform sampling and strong curriculum baselines
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Post-training large foundation models with reinforcement learning typically relies on massive and heterogeneous datasets, making effective curriculum learning both critical and challenging. In this work, we propose ACTOR-CURATOR, a scalable and fully automated curriculum learning framework for reinforcement learning post-training of large language models (LLMs). ACTOR-CURATOR learns a neural curator that dynamically selects training problems from large problem banks by directly optimizing for expected policy performance improvement. We formulate problem selection as a non-stationary stochastic bandit problem, derive a principled loss function based on online stochastic mirror descent, and establish regret guarantees under partial feedback. Empirically, ACTOR-CURATOR consistently outperforms uniform sampling and strong curriculum baselines across a wide range of challenging reasoning benchmarks, demonstrating improved training stability and efficiency. Notably, it achieves relative gains of 28.6% on AIME2024 and 30.5% on ARC-1D over the strongest baseline and up to 80% speedup. These results suggest that ACTOR-CURATOR is a powerful and practical approach for scalable LLM post-training.
Tags
Links
- Source: https://arxiv.org/abs/2602.20532v1
- Canonical: https://arxiv.org/abs/2602.20532v1
Trouble viewing inline? Open PDF directly →
Full Text
108,257 characters extracted from source content.
Expand or collapse full text
Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for Scalable RL Post-Training Zhengyao Gu∗♠ Jonathan Light∗♡♢🖂^* \ Raul Astudillo♣ Ziyu Ye♠ Langzhou He♠ Henry Peng Zou♠ Wei Cheng◆ Santiago Paternain♢ Philip S. Yu♠ Yisong Yue♡ ♠University of Illinois Chicago ♡Caltech ♢RPI ♣MBZUAI ▲University of Chicago ◆NEC Laboratories America ∗Equal contribution 🖂 Corresponding author: jonathan.li.connect@gmail.com Abstract Post-training large foundation models with reinforcement learning typically involves selecting training problems from massive and heterogeneous datasets, where the choice of data has a critical impact on training stability, sample efficiency, and final performance. In this work, we propose Actor-Curator, a scalable and fully automated framework for reinforcement learning post-training of large language models (LLMs) that learns to adaptively curate training problems. Actor-Curator trains a neural curator that dynamically selects problems from large problem banks by directly optimizing for expected policy performance improvement. We formulate problem selection as a non-stationary stochastic bandit problem, derive a principled loss function based on online stochastic mirror descent, and establish regret guarantees under partial feedback. Empirically, Actor-Curator consistently outperforms uniform sampling and strong learning-based baselines across a wide range of challenging reasoning benchmarks, demonstrating improved training stability and efficiency. Notably, it achieves relative gains of 28.6% on AIME2024 and 30.5% on ARC-1D over the strongest baseline and up to 80% speedup. These results suggest that Actor-Curator provides a practical and principled approach to scalable, adaptive curriculum learning for LLM post-training. 1 Introduction Reinforcement learning (RL) has become a central paradigm for post-training foundation models, enabling improvements in reasoning, alignment, and task-specific performance beyond supervised fine-tuning (Shao et al., 2024). In this setting, the choice, ordering, and frequency of training problems play a critical role in determining convergence speed, training stability, and final generalization performance, motivating the use of curriculum learning to adaptively select training data (Bengio et al., 2009; Tzannetos et al., 2023; Parashar et al., 2025). However, applying curriculum learning to modern foundation model post-training is challenging: post-training datasets are large, diverse, and continuously evolving, while actor updates induce complex, non-stationary training dynamics. Traditional curriculum learning approaches—based on manual difficulty annotations, hand-designed problem buckets, or tabular per-problem statistics (Asada et al., 1996; Wu and Tian, 2017; Yengera et al., 2021)—do not scale to such settings, fail to generalize to unseen problems, and are brittle when problem utility changes as the policy improves. Moreover, effective curricula must balance exploration of under-sampled problems with exploitation of those that most improve the current policy, further complicating scalable curriculum design. In this work, we propose Actor-Curator (AC), a scalable and fully automated problem curation framework for RL post-training that jointly trains an actor and a curator in an online, on-policy manner. At the core of Actor-Curator is a learned curator that adaptively selects training problems at each iteration and function-approximates over large, heterogeneous datasets. The curator is trained to directly maximize a policy improvement objective, assigning higher probability to problems expected to induce the greatest improvement in the actor’s performance. Unlike prior curricula that rely on heuristic signals such as absolute mean advantage or difficulty proxies (Chen et al., 2025a; Gao et al., 2025; Wang et al., 2025), our objective is derived from expected policy improvement, providing a principled and actor-aware learning signal. As a result, Actor-Curator naturally adapts to evolving actor training dynamics and allows it to be seamlessly combined with a wide range of RL algorithms. Figure 1: Co-adaptive online training loop of Actor-Curator. At each RL step, a learned curator adaptively selects problems from a large problem bank instead of uniform sampling. The actor is updated on these problems, after which a bandit-style reward based on post-update policy improvement trains the curator. As the actor improves, the curator adapts to prioritize problems that yield the greatest expected performance gains. To optimize the curator, we formalize problem selection as a non-stationary stochastic bandit problem with partial feedback. The curator is trained online and on-policy alongside the actor using online stochastic mirror descent (OSMD) Lattimore and Szepesvari (2017), which explicitly balances exploration and exploitation under non-stationarity. This differs from prior approaches that primarily rely on regression-style objectives and do not explicitly model the bandit structure or partial observability inherent in adaptive data selection (Tzannetos et al., 2023; Gao et al., 2025). To scale beyond tabular formulations, we derive a function-approximation variant of OSMD that trains the curator as a neural network, enabling generalization across problems and robustness to large, dynamic datasets. Finally, we introduce a PPO-style proximal clipping objective to stabilize curator optimization in practice. Empirically, Actor-Curator enables effective curriculum learning at scale without human annotations, difficulty labels, or manual dataset structuring. Across diverse reasoning benchmarks—Countdown, Zebra, MATH, AIME, and ARC-1D—it consistently outperforms uniform sampling and strong baselines, achieving up to 30% higher peak performance on ARC-1D, 28% on AIME24, and up to 80% faster convergence to comparable performance. In summary, our main contributions are: • Automated problem curation framework for RL post-training. We introduce Actor-Curator, a scalable framework that learns a neural curator to adaptively select training problems in an online, on-policy manner, enabling curriculum learning over large, heterogeneous datasets without human annotations or manual structuring. • A policy-improvement–driven bandit formulation of data curation. We cast problem selection as a non-stationary stochastic bandit problem and derive a principled learning signal grounded in policy improvement theory, optimized via an OSMD-based bandit objective with regret guarantees under partial feedback. 2 Problem formulation We study problem curation for reinforcement learning (RL) post-training of large language models (LLMs), where training is performed over large and heterogeneous collections of problems. Our goal is to design an adaptive data selection strategy that determines which training problems an LLM should train on at each iteration in order to maximize overall post-training performance. 2.1 RL post-training setting Let =(i)i=1||X=\ x^(i)\_i=1^|X| denote a large collection of training problems, and let p_X be a fixed evaluation distribution over X. Let π denote a pretrained autoregressive language model, which induces a conditional distribution ∼π(⋅∣) y π(· x) over solutions for each problem x. A reward model R:×→[0,1]R:Y×X→[0,1] assigns a scalar score to each solution. The post-training objective is to maximize expected reward under the evaluation distribution: J(π)≜∼p,∼π(⋅∣)[R(∣)].J(π)\; \;E_ x p_X,\, y π(· x) [R( y x) ]. (1) In this work, the reinforcement learning algorithm, reward model, and rollout procedure are fixed. Our focus is on how training problems are selected across iterations. 2.2 Training dynamics Training proceeds in iterations. At iteration t, a curator selects a subset of training problems t⊂X^t . Given this selection, the actor πtπ^t is rolled out on each ∈t x ^t to produce solutions and corresponding rewards t≜(j)∼πt(⋅∣)j=1||,ℛt≜R((j)∣)j=1||.Y_ x^t \ y^(j) π^t(· x)\_j=1^|Y_ x|, _ x^t \R( y^(j) x)\_j=1^|Y_ x|. These trajectories form the dataset t≜(,t,ℛt)∣∈t,πt+1←(πt,t).D^t \( x,Y_ x^t,R_ x^t) x ^t \, π^t+1 (π^t,D^t). (2) Here A may correspond to any standard post-training algorithm (e.g., GRPO (Shao et al., 2024) or GSPO (Ahmadian et al., 2024)). We emphasize that the curator influences learning only indirectly through data selection, while the actor is solely responsible for policy optimization. 2.3 Curriculum learning as problem selection The central problem addressed in this work is how to choose the training subsets tX^t across iterations. Different choices of tX^t induce different actor updates and therefore different trajectories of policy improvement. We formalize curriculum learning as a sequential decision-making problem. At each iteration t, a curator selects a subset of training problems t⊂X^t . This selection induces a performance improvement J(πt+1)−J(πt)J(π^t+1)-J(π^t). The curator’s objective is to maximize cumulative performance gains over training: maxtt=1T∑t=1T(J(πt+1)−J(πt)). _\X^t\_t=1^T _t=1^T (J(π^t+1)-J(π^t) ). (3) 2.4 Challenges of adaptive problem selection Effective curation is challenging for several reasons: • Large action space. The problem set X is large and might change across time, making it infeasible to manually define curricula or track per-problem statistics. • Partial feedback. At each iteration, feedback is observed only for the problems selected for training; the utility of unselected problems remains unknown. • Non-stationarity. The usefulness of a problem depends on the current actor πtπ^t and changes as the actor improves. It is also highly dependent on the actor update method. • Exploration–exploitation trade-off. The curator must balance exploring under-sampled problems whose utility is uncertain with exploiting problems that are known to drive policy improvement. These challenges motivate a curriculum learning approach that operates at scale, learns online from partial feedback, and explicitly accounts for the non-stationary relationship between training problems and policy improvement. 3 Method Figure 2: Single training iteration of Actor-Curator. At each iteration, a candidate set of problems is sampled from a fixed proposal distribution. The curator reweights this candidate set to select training problems for the actor. After the actor update, per-problem policy improvement is estimated using pre- and post-update policies. The curator observes bandit feedback only on selected problems and is updated using a PPO-style approximation of online stochastic mirror descent. We now present Actor-Curator, a curriculum learning framework that trains a learned curator to adaptively select training problems for RL post-training of large language models. The key idea is to treat problem selection as a non-stationary bandit problem and to train the curator to directly maximize policy improvement—the expected performance gain induced by each actor update—using online stochastic mirror descent (OSMD) under partial feedback. Algorithm 1 Actor–Curator: Self-driven curriculum learning 1:Input: dataset X, pretrained LLM (actor) π0π^0, reward model R, curator model Cϕ0C_φ^0, proposal distribution q~ q 2:Hyperparameters: number of training steps T, rollouts per problem |||Y_ x|, candidate batch size |~t|| X^t|, training batch size |t||X^t| 3:for t=0t=0 to T−1T-1 do ⊳ Training steps 4: t←∅D^t← 5: Proposal step: sample a candidate batch ~t⊂ X^t according to q~ q 6: Selection step: sample a training batch t⊂~tX^t⊂ X^t using the curator CϕtC_φ^t 7: for each problem ∈t x ^t do 8: Roll out solutions t=(j)∼πt(⋅∣)j=1||Y^t_ x=\ y^(j) π^t(· x)\_j=1^|Y_ x| 9: Compute rewards ℛt=R((j)∣)j=1||R^t_ x=\R( y^(j) x)\_j=1^|Y_ x| 10: Add (,t,ℛt)( x,Y^t_ x,R^t_ x) to tD^t 11: end for 12: Actor update: πt+1←(πt,t)π^t+1 (π^t,D^t) 13: Curator utilities: compute ^t U^t using (10) (and πt+1π^t+1) 14: Curator update: ϕt+1←argminϕℒcur(ϕ)φ^t+1← _φ\ L_cur(φ) using (12) 15:end for 3.1 Overview of the training loop Training proceeds in iterations. At iteration t, the following steps are performed: 1. We sample a training subset t⊂X^t based on probabilities produced by the curator. 2. The actor πtπ^t is rolled out on tX^t to collect trajectories and updated to πt+1π^t+1 using any RL update. 3. Compute a per-problem policy improvement estimate for problems in tX^t using πtπ^t and πt+1π^t+1 4. The curator is updated online using bandit feedback derived from these improvement estimates. The curator is trained jointly with the actor in an on-policy manner, allowing the curriculum to adapt dynamically as the actor improves. Figure 2 illustrates this process. Pseudocode is provided in Algorithm 1. 3.2 Policy improvement as the curator learning signal The curator’s objective is to select problems that maximize improvement in the actor’s performance under the fixed evaluation distribution p_X. Performance improvement identity. Define the performance improvement at iteration t as ut≜J(πt+1)−J(πt).u^t J(π^t+1)-J(π^t). In the single-turn setting, the performance difference identity (Kakade and Langford, 2002) gives ut=∼p∼πt+1(⋅∣)[Aπt(∣)],Aπt(∣)≜R(∣)−′∼πt(⋅∣)[R(′∣)].u^t=E_ x p_XE_ y π^t+1(· x) [A_π^t( y x) ], A_π^t( y x) R( y x)-E_ y π^t(· x) [R( y x) ]. (4) Applying importance sampling yields ut=∼p∼πt(⋅∣)[πt+1(∣)πt(∣)Aπt(∣)].u^t=E_ x p_XE_ y π^t(· x) [ π^t+1( y x)π^t( y x)A_π^t( y x) ]. (5) Per-problem utility. Eq. 5 decomposes additively across problems. For each ∈ x , define the per-problem utility ut≜p()∼πt(⋅∣)[πt+1(∣)πt(∣)Aπt(∣)].u_ x^t p_X( x)E_ y π^t(· x) [ π^t+1( y x)π^t( y x)A_π^t( y x) ]. (6) By construction, ut=∑utu^t= _ xu_ x^t. Although the actor update couples all selected problems, utu_ x^t provides a principled first-order credit assignment signal under small policy updates, grounded in policy improvement theory. We provide further explanation in App. E. 3.3 Tabular OSMD formulation We cast curriculum learning as a non-stationary stochastic bandit problem, where each problem x corresponds to an arm with time-varying utility utu_ x^t. The curator is optimized using online stochastic mirror descent (OSMD) under bandit feedback (Lattimore and Szepesvari, 2017). We start with the tabular formulation first, where the curator maintains a probability mass function t∈Δα() p^t∈ _α(X) over a finite set of problems, where t p^t is clipped to the sampling distribution pt(∣~t)≥α>0p^t( x X^t)≥α>0. In Sec. 3.4 we show how to represent ptp^t using a learned model. Utility estimation and OSMD bandit feedback. For each ∈t x ^t, let tY_ x^t denote rollouts from πt(⋅∣)π^t(· x). We estimate Eq. 6 via U^t≜p()∈tpt()A^t(⋅∣),A^t(⋅∣)≜1|t|∑∈tπt+1(∣)πt(∣)A(∣) U^t_ x p_X( x) I\ x ^t\p^t( x) A^t(· x), A^t(· x) 1|Y^t_ x| _ y ^t_ x π^t+1( y x)π^t( y x)A( y x) (7) where A^t A^t is the importance normalized estimated average advantage. This estimate is agnostic to the specifics of the actor update method as long as an updated actor is produced. Theorem 1 (Unbiasedness) [U^t]=utE[ U_ x^t]=u_ x^t. We prove this in App. B. For decoder language models, πt+1(∣)π^t+1( y x) is obtained via a single forward pass of the updated model on the previous solution. OSMD update Given bandit feedback, the curator is updated with a negative-entropy regularizer: pt+1←argmin∈Δα()−η⟨,^t⟩+KL(∥t).p^t+1← _ p∈ _α(X) \-η p, U^t +KL( p\| p^t) \. (8) This yields the exponentiated-gradient update pt+1()∝pt()exp(ηU^t).p^t+1( x) p^t( x) (η U_ x^t). 3.4 Function approximation for curator training Although the OSMD update in Eq. 8 is defined over a distribution on the entire problem set X, explicitly maintaining and updating tabular probabilities is infeasible when X is large. We therefore parameterize the curator using a neural network that assigns a positive score to each problem and implicitly defines a probability distribution. The curator and induced distribution are defined as Cϕ:↦wϕ(),pϕ()≜wϕ()∑′∈wϕ(′),wϕ()>0.C_φ: x w_φ( x), p_φ( x) w_φ( x) _ x w_φ( x ), w_φ( x)>0. OSMD surrogate objective. To implement the OSMD update Eq. 8 with function approximation, we optimize the following surrogate objective: ℒcur(ϕ)=KL(pϕ∥pt)−η⟨pϕ,^t⟩,L_cur(φ)=KL\! (p_φ\;\|\;p^t )-η p_φ, U^t , (9) where ptp^t denotes the curator distribution from iteration t. 3.5 Two-stage sampling Sampling directly from a curator distribution over the full problem set X is computationally infeasible at scale. We therefore adopt a two-stage sampling scheme that separates coverage from adaptive curation while preserving unbiased utility estimation. At iteration t, we first sample a candidate set ~t⊂ X^t from a fixed proposal distribution q~ q. Let q()≜Pr~∼q~(∈~),pt(∣~t)≜wt()∑′∈~twt(′).q( x) _ X q\! ( x∈ X ), p^t( x X^t) w^t( x) _ x ∈ X^tw^t( x ). Here q()q( x) denotes the induced marginal inclusion probability, and we assume q()≥qmin>0q( x)≥ q_ >0 for all ∈ x . Conditioned on ~t X^t, the curator samples a training set t⊂~tX^t⊂ X^t according to the restricted distribution pt(∣~t)p^t( x X^t), where wt()>0w^t( x)>0 is the curator score at iteration t. This allows the curator to prioritize problems while operating only on a small candidate batch. Utility estimation. The unbiased two-stage estimator corresponding to Eq. 7 is U^two,t≜p()∈tq()pt(∣~t)A^t(⋅∣), U^t_two, x\; \;p_X( x)\, I\ x ^t\q( x)\,p^t( x X^t)\; A^t(· x), (10) which corrects for both proposal and curation sampling probabilities. This estimator satisfies [U^two,t]=utE[ U^t_two, x]=u^t_ x (see App. B). Substituting Eq. 10 into the OSMD update Eq. 8 yields the surrogate objective ℒtwo(ϕ) _two(φ) ≜KL(pϕ∥pt)−η∑∈tpϕ(∣~t)pt(∣~t)p()A^t(⋅∣)q(), \! (p_φ\;\|\;p^t )-η _ x ^t p_φ( x X^t)p^t( x X^t)\; p_X( x)\, A^t(· x)q( x), (11) where pϕ(⋅∣~t)p_φ(· X^t) is the conditional curator distribution. Regret guarantee. We now state a regret bound for curator optimization under two-stage sampling. The bound characterizes the curator’s ability to track the best sequence of problem-selection distributions in hindsight despite non-stationary utilities. A proof is provided in App. C. Theorem 2 Assume the curator is updated using OSMD with a negative-entropy regularizer and receives bandit feedback U^two,t U^t_two, x satisfying [U^two,t]=utE[ U^t_two, x]=u^t_ x. Then the cumulative dynamic regret satisfies RegT≤O(T2/3VT1/3),VT≜∑t=2Tmax∈|ut−ut−1|Reg_T\;≤\;O\! (T^2/3V_T^1/3 ), V_T _t=2^T _ x |u_ x^t-u_ x^t-1| where RegTReg_T is the regret against the best available arm, ignoring uniform exploration which we define in App. D, and VTV_T is a measure of how rapid the utility of a problem changes over time t. While we focus on two-stage sampling for efficiency, Actor-Curator is compatible with other approximate sampling schemes (e.g., Metropolis–Hastings), provided marginal inclusion probabilities are roughly proportional to curator-assigned weights. 3.6 Proximal curator optimization Directly optimizing the KL-regularized objective in (9) can be unstable with neural network parameterization (Schulman et al., 2015). Following proximal policy optimization (PPO) (Schulman et al., 2017), we adopt a clipped surrogate objective. Define the importance ratio and sub-objective as ρϕ()≜pϕ(∣~t)pt(∣~t),gt()≜p()A^t(⋅∣)q() _φ( x) p_φ( x X^t)p^t( x X^t), g^t( x) p_X( x) A^t(· x)q( x) Starting from (11), we replace the linear probability ratio term with a clipped surrogate ℒcurPCO(ϕ)≜−η∑∈~tmin( _cur^PCO(φ) -η _ x∈ X^t \! ( ρϕ()gt(),clip(ρϕ(),ρmin,ρmax)gt()), _φ( x)\,g^t( x),\;clip( _φ( x), _ , _ )\,g^t( x) ), (12) where ρmin,ρmax _ , _ are clipping parameters. This objective constrains successive curator updates while preserving the behavior of online mirror descent in practice. 4 Experimental results Benchmarks. We evaluate Actor-Curator on five reasoning and mathematics benchmarks. Countdown is an arithmetic puzzle requiring the composition of integers and operations to reach a target value (Stojanovski et al., 2025). Zebra is a symbolic logic puzzle that requires finding assignments satisfying a set of constraints (Stojanovski et al., 2025). ARC-1D is the one-dimensional variant of the Abstraction and Reasoning Corpus, designed to test inductive generalization (Chollet, 2019; Xu et al., 2023). MATH500 consists of competition-level mathematics problems (Hendrycks et al., 2021). AIME24 contains problems from the 2024 American Invitational Mathematics Examination. We additionally consider hard subsets (countdown-hard, zebra-hard, arc-hard) for validation. We train on 30K problems for Countdown, Zebra, and ARC-1D, and 12K MATH problems for MATH500 and AIME24. Experimental setup. We implement our post-training pipeline using VERL (Sheng et al., 2024). The curator is initialized from a pretrained Qwen3-0.6B model (Yang et al., 2025). Unless otherwise specified, the actor is trained using GSPO, a stabilized variant of GRPO (Ahmadian et al., 2024), on Qwen2.5-3B. Additional details are provided in App. G, including hyper-parameters. Table 1: Peak validation performance on problems within 100 training steps for different methods with qwen2.5-3b. Actor-Curator outperforms both other learning based methods (PCL) and methods that rely on human heuristics (SEC). We see similar results with llama3.2-3b-instruct, as shown in Tab. 4. |||X| Benchmark Method Improvement _ref Uniform SEC PCL AC (Ours) +Δ+ +Δ%+ \% 30,000 Countdown 0.00 44.74 58.87 57.24 62.12 +3.25 +5.52 Countdown-hard 0.00 41.00 51.50 48.00 58.00 +6.50 +12.62 30,000 Zebra 0.00 35.12 36.00 34.12 37.62 +1.62 +4.50 Zebra-hard 0.00 30.50 27.50 26.00 34.50 +4.00 +13.11 30,000 ARC-1D 0.00 26.74 27.87 26.37 36.37 +8.50 +30.51 ARC-hard 0.00 19.50 18.50 18.50 31.00 +11.50 +58.97 12,000 MATH500 61.80 83.00 81.00 79.79 81.00 -2.00 -2.41 AIME24 3.33 23.33 20.00 23.33 30.00 +6.67 +28.57 4.1 Main results Baselines. We compare against state-of-the-art curriculum learning methods for LLM post-training, using the same backbone model and actor update. πref _ref denotes the pretrained model without post-training. Uniform sampling draws training problems uniformly at random. SEC partitions problems into manually defined buckets and updates bucket probabilities based on the sum of absolute advantages (Chen et al., 2025a). PCL trains a value model to estimate success probabilities and prioritizes problems with predicted success near 50%50\% (Gao et al., 2025). PCL is competitive with recent curriculum-based approaches (Yue et al., 2025; Zhang et al., 2025; Zheng et al., 2025b). Performance. We evaluate performance on held-out test sets, recording metrics every 10 training steps. Following prior work (Gao et al., 2025), we report the peak performance achieved within the first 100 steps. Results are summarized in Tab. 1. Across both backbone models and most benchmarks, Actor-Curator consistently outperforms all baselines, with additional results in App. J. Notably, Actor-Curator achieves substantially larger gains on harder benchmarks such as arc-hard and AIME24, indicating that adaptive curation is particularly beneficial in challenging regimes. Efficiency. As shown in Fig. 6, Actor-Curator reaches comparable or higher performance using significantly fewer training steps than uniform sampling, demonstrating improved sample efficiency. Training dynamics. Across datasets, Actor-Curator exhibits more stable optimization and often continues to improve performance after baselines plateau, as shown in Figs. 3, 9 and 10. (a) Countdown training curves (b) Zebra training curves (c) ARC-1D training curves Figure 3: Training dynamics. Actor-Curator Test performance over training on three datasets, showing faster convergence and higher final accuracy. 4.2 Ablations Core components. We ablate two key design choices: the policy-improvement utility and the OSMD bandit objective. Absolute adv replaces the policy-improvement signal with mean absolute advantage, as in (Chen et al., 2025a). Regression loss trains the curator to predict the target utility value using a squared loss, rather than learning a classifier as in OSMD. The predicted utilities are then converted into a sampling distribution via a Boltzmann transform with temperature η, matching the temperature used in OSMD. As shown in Fig. 7(b), both components are critical for achieving strong performance. We include a more detailed account of the motivation behind Absolute Adv. and regression in App. A, as well as their difference with Actor-Curator. Figure 4: Training speed-up. Actor-Curator attains high test accuracy with significantly fewer steps. Figure 5: Difficulty progression of curated problems. Actor-Curator gradually increases the average difficulty over training. Figure 6: Actor gradient norms over training. Actor-Curator yields larger and more sustained updates. Actor update methods. Actor-Curator generalizes across actor optimization algorithms. In Fig. 7(a), Actor-Curator significantly improves GRPO-based training relative to uniform sampling, demonstrating robustness to the choice of actor update. Additional ablation. We provide additional ablation on curator model size (Fig. 7(c)) and candidate batch size (Fig. 8) in App. I. 4.3 Interpretation and analysis Curriculum progression. As shown in Fig. 6, Actor-Curator initially prioritizes easier problems and gradually shifts toward harder ones over training. Problem difficulty is estimated using heuristic annotations. Impact on actor updates. Actor-Curator induces consistently larger actor gradient norms than uniform sampling (Fig. 6), suggesting that curated problems produce more informative learning signals. This aligns with the curator’s objective of prioritizing problems with higher expected policy improvement. 5 Related work RLVR. Reinforcement learning with verification (RLVR) has emerged as an effective paradigm for improving the capabilities of large language models (LLMs) during post-training (Guo et al., 2025; Setlur et al., 2024; Chen et al., 2025b; Wen et al., 2025). Prior work has largely focused on improving the actor update rules (Yu et al., 2025; Dong et al., 2025) or enhancing trajectory generation, often via search-based methods (Zhang et al., 2024; Light et al., 2025b; c). Our work is complementary: rather than modifying the actor or rollout process, we focus on learning a principled curriculum that selects which problems the actor should train on to maximize policy improvement. Curriculum learning for RL. Classical curriculum learning approaches in reinforcement learning typically select tasks of intermediate difficulty, often defined via the agent’s probability of success (Florensa et al., 2017; 2018; Wöhlke et al., 2020; Liu et al., 2025). For example, ProCuRL selects problems whose difficulty is estimated to be near a decision boundary using a learned value network (Tzannetos et al., 2023). In contrast, our approach (1) trains a large language model as a curator that directly operates over language problems, and (2) uses a theoretically grounded, policy-improvement-based target rather than heuristic difficulty estimates. This design allows our method to generalize across different actor update rules and scale to large, heterogeneous datasets. Curriculum learning for LLMs. Recent work on self-improving and self-evolving LLMs has highlighted the importance of curriculum learning in RL-based post-training (Ye et al., 2024; Light et al., 2025a). Several methods adjust problem sampling using bandit-style objectives such as UCB (Chen et al., 2025a; Wang et al., 2025; Gao et al., 2025). However, most existing approaches rely on manual curriculum design, including human-labeled difficulty levels or pre-defined problem buckets that are sampled adaptively (Graves et al., 2017). In contrast, we propose one of the first fully automated curriculum learning frameworks for LLM post-training that requires no human annotations or manual dataset structuring. (a) Actor update ablation (b) Core component ablation (c) Curator model ablation Figure 7: Ablation results. (a) Actor-Curator is compatible with alternative actor update methods such as GRPO and yields consistent performance gains. (b) The combination of the OSMD curator objective and the policy-improvement target achieves superior performance compared to alternative targets and loss functions. (c) Varying curator model size leads to similar long-term performance, indicating robustness to curator capabilities. 6 Limitations While Actor-Curator is agnostic to the specific actor update rule and can be combined with a wide range of post-training algorithms, its gains ultimately depend on the stability and quality of the underlying actor optimization: unstable updates can still lead to noisy dynamics or training collapse, a common failure mode in reinforcement learning that curriculum learning alone cannot fully address. In addition, Actor-Curator inherits standard assumptions from RL-based LLM post-training, most notably access to a reliable reward signal, making it best suited to domains with objectively verifiable rewards such as mathematics, logic puzzles, and code. Finally, learning an explicit curator introduces additional computational overhead beyond standard RL post-training; while modest relative to actor training and rollout generation, this cost is non-negligible, and in our experiments Actor-Curator increases overall wall-clock training time by approximately 9% (see App. H), though this overhead is small compared to the observed gains in training efficiency. 7 Conclusion Our results demonstrate that combining bandit-style optimization, neural function approximation, and policy-improvement-based feedback provides a powerful and general approach to curriculum learning for RL post-training. This combination enables efficient and stable adaptation of training data selection at scale, leading to faster learning and higher final performance, particularly on difficult reasoning problems. Importantly, these gains persist across diverse benchmarks and settings, suggesting that this design generalizes beyond specific tasks or datasets and offers a scalable foundation for improving reinforcement learning post-training of large language models. Beyond empirical gains, Actor-Curator highlights a broader shift in how post-training systems should be designed. Rather than treating training data as a static resource, our results suggest that data selection itself can be optimized online as part of the learning process, adapting in tandem with the evolving policy. This perspective opens the door to post-training pipelines that are less reliant on meticulous dataset engineering and more resilient to distributional mismatch, noise, and continual data growth. References A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker (2024) Back to basics: revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740. Cited by: §2.2, §4. M. Asada, S. Noda, S. Tawaratsumida, and K. Hosoda (1996) Purposive behavior acquisition for a real robot by vision-based reinforcement learning. Machine learning 23 (2), p. 279–303. Cited by: §1. Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009) Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, p. 41–48. Cited by: §1. O. Besbes, Y. Gur, and A. Zeevi (2014) Stochastic multi-armed-bandit problem with non-stationary rewards. In Advances in Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (Eds.), Vol. 27, p. . External Links: Link Cited by: Appendix D. X. Chen, J. Lu, M. Kim, D. Zhang, J. Tang, A. Piché, N. Gontier, Y. Bengio, and E. Kamalloo (2025a) Self-evolving curriculum for llm reasoning. arXiv preprint arXiv:2505.14970. Cited by: §A.3, §1, §4.1, §4.2, §5. Y. Chen, Z. Yang, Z. Liu, C. Lee, P. Xu, M. Shoeybi, B. Catanzaro, and W. Ping (2025b) Acereason-nemotron: advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505.16400. Cited by: §5. F. Chollet (2019) On the measure of intelligence. arXiv preprint arXiv:1911.01547. Cited by: §4. G. Dong, H. Mao, K. Ma, L. Bao, Y. Chen, Z. Wang, Z. Chen, J. Du, H. Wang, F. Zhang, et al. (2025) Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849. Cited by: §5. C. Florensa, D. Held, X. Geng, and P. Abbeel (2018) Automatic goal generation for reinforcement learning agents. In International conference on machine learning, p. 1515–1528. Cited by: §5. C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel (2017) Reverse curriculum generation for reinforcement learning. In Conference on robot learning, p. 482–495. Cited by: §5. Z. Gao, J. Kim, W. Sun, T. Joachims, S. Wang, R. Y. Pang, and L. Tan (2025) Prompt curriculum learning for efficient llm post-training. arXiv preprint arXiv:2510.01135. Cited by: §A.3, §1, §1, §4.1, §4.1, §5. A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu (2017) Automated curriculum learning for neural networks. In international conference on machine learning, p. 1311–1320. Cited by: §5. D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025) DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), p. 633–638. Cited by: §5. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: §4. S. Kakade and J. Langford (2002) Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning, p. 267–274. Cited by: Appendix E, §3.2. T. Lattimore and C. Szepesvari (2017) Bandit algorithms. External Links: Link Cited by: §1, §3.3. J. Light, M. Cai, W. Chen, G. Wang, X. Chen, W. Cheng, Y. Yue, and Z. Hu (2025a) Strategist: self-improvement of llm decision making via bi-level tree search. In The Thirteenth International Conference on Learning Representations, Cited by: §5. J. Light, W. Cheng, B. Riviere, W. Yue, M. Oyamada, M. Wang, Y. Yue, S. Paternain, and H. Chen (2025b) DISC: dynamic decomposition improves llm inference scaling. In Advances in Neural Information Processing Systems, Cited by: §5. J. Light, Y. Wu, Y. Sun, W. Yu, Y. Liu, X. Zhao, Z. Hu, H. Chen, and W. Cheng (2025c) Sfs: smarter code space search improves llm inference scaling. In The Thirteenth International Conference on Learning Representations, Cited by: §5. Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin (2025) Understanding r1-zero-like training: a critical perspective. arXiv preprint arXiv:2503.20783. Cited by: §5. S. Parashar, S. Gui, X. Li, H. Ling, S. Vemuri, B. Olson, E. Li, Y. Zhang, J. Caverlee, D. Kalathil, and S. Ji (2025) Curriculum reinforcement learning from easy to hard tasks improves llm reasoning. External Links: 2506.06632, Link Cited by: §1. J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz (2015) Trust region policy optimization. In International conference on machine learning, p. 1889–1897. Cited by: §3.6. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §3.6. A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, and A. Kumar (2024) Rewarding progress: scaling automated process verifiers for llm reasoning. arXiv preprint arXiv:2410.08146. Cited by: §5. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §A.1, §1, §2.2. G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2024) HybridFlow: a flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256. Cited by: §4. Z. Stojanovski, O. Stanley, J. Sharratt, R. Jones, A. Adefioye, J. Kaddour, and A. Köpf (2025) REASONING gym: reasoning environments for reinforcement learning with verifiable rewards. arXiv preprint arXiv:2505.24760. Cited by: §4. G. Tzannetos, B. G. Ribeiro, P. Kamalaruban, and A. Singla (2023) Proximal curriculum for reinforcement learning agents. arXiv preprint arXiv:2304.12877. Cited by: §1, §1, §5. Z. Wang, G. Cui, Y. Li, K. Wan, and W. Zhao (2025) Dump: automated distribution-level curriculum learning for rl-based llm post-training. arXiv preprint arXiv:2504.09710. Cited by: §1, §5. L. Wen, Y. Cai, F. Xiao, X. He, Q. An, Z. Duan, Y. Du, J. Liu, T. Tanglifu, X. Lv, et al. (2025) Light-r1: curriculum sft, dpo and rl for long cot from scratch and beyond. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), p. 318–327. Cited by: §5. J. Wöhlke, F. Schmitt, and H. van Hoof (2020) A performance-based start state curriculum framework for reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, p. 1503–1511. Cited by: §5. Y. Wu and Y. Tian (2017) Training agent for first-person shooter game with actor-critic curriculum learning. In International Conference on Learning Representations, Cited by: §1. Y. Xu, W. Li, P. Vaezipoor, S. Sanner, and E. B. Khalil (2023) LLMs and the abstraction and reasoning corpus: successes, failures, and the importance of object-based representations. arXiv preprint arXiv:2305.18354. Cited by: §4. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4. Z. Ye, R. Agarwal, T. Liu, R. Joshi, S. Velury, Q. V. Le, Q. Tan, and Y. Liu (2024) Scalable reinforcement post-training beyond static human prompts: evolving alignment via asymmetric self-play. arXiv preprint arXiv:2411.00062. Cited by: §5. G. Yengera, R. Devidze, P. Kamalaruban, and A. Singla (2021) Curriculum design for teaching via demonstrations: theory and applications. Advances in Neural Information Processing Systems 34, p. 10496–10509. Cited by: §1. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. (2025) Dapo: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: §5. Y. Yue, Y. Yuan, Q. Yu, X. Zuo, R. Zhu, W. Xu, J. Chen, C. Wang, T. Fan, Z. Du, et al. (2025) Vapo: efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118. Cited by: §4.1. D. Zhang, S. Zhoubian, Z. Hu, Y. Yue, Y. Dong, and J. Tang (2024) Rest-mcts*: llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems 37, p. 64735–64772. Cited by: §5. R. Zhang, D. Arora, S. Mei, and A. Zanette (2025) SPEED-rl: faster training of reasoning models via online curriculum learning. arXiv preprint arXiv:2506.09016. Cited by: §4.1. C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. (2025a) Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: §A.1. H. Zheng, Y. Zhou, B. R. Bartoldson, B. Kailkhura, F. Lai, J. Zhao, and B. Chen (2025b) Act only when it pays: efficient reinforcement learning for llm reasoning via selective rollouts. arXiv preprint arXiv:2506.02177. Cited by: §4.1. Appendix A Baselines A.1 Group-Based Policy Optimization Group-based policy optimization methods update the policy by grouping multiple rollouts from the same problem and computing advantages relative to the group baseline. GRPO. Group Relative Policy Optimization (GRPO) (Shao et al., 2024) computes advantages by comparing each solution’s reward to the mean reward of all solutions sampled from the same problem. For a problem x with rollouts x=y(1),…,y(m)Y_x=\y^(1),…,y^(m)\ and corresponding rewards R(y(j)|x)j=1m\R(y^(j)|x)\_j=1^m, the advantage for solution y(i)y^(i) is: AGRPO(y(i)|x)=R(y(i)|x)−1m∑j=1mR(y(j)|x).A^GRPO(y^(i)|x)=R(y^(i)|x)- 1m _j=1^mR(y^(j)|x). (13) The policy is then updated using a PPO-style clipped objective: ℒGRPO(π)=x,y(i)∼x[min(ρiAGRPO(y(i)|x),clip(ρi,1−ϵ,1+ϵ)AGRPO(y(i)|x))],L^GRPO(π)=E_x,y^(i) _x [ ( _iA^GRPO(y^(i)|x),clip( _i,1-ε,1+ε)A^GRPO(y^(i)|x) ) ], (14) where ρi=π(y(i)|x)πold(y(i)|x) _i= π(y^(i)|x) _old(y^(i)|x) is the importance ratio and ϵε is the clipping threshold. GSPO. Group-Sequence Policy Optimization (Zheng et al., 2025a) addresses fundamental stability issues in GRPO by defining importance ratios at the sequence level rather than the token level. Unlike GRPO, which applies token-level importance weights that can introduce high-variance noise, GSPO computes importance ratios based on sequence likelihood, aligning with the principle of importance sampling. GSPO optimizes the following sequence-level objective: GSPO(θ)=x∼,yii=1G∼πθold(⋅|x)[1G∑i=1Gmin(si(θ)A^i,clip(si(θ),1−ϵ,1+ϵ)A^i)],J^GSPO(θ)=E_x ,\y_i\_i=1^G _ _old(·|x) [ 1G _i=1^G (s_i(θ) A_i,clip(s_i(θ),1-ε,1+ε) A_i ) ], (15) where the sequence-level importance ratio is defined as: si(θ)=(πθ(yi|x)πθold(yi|x))1|yi|=exp(1|yi|∑t=1|yi|logπθ(yi,t|x,yi,<t)πθold(yi,t|x,yi,<t)).s_i(θ)= ( _θ(y_i|x) _ _old(y_i|x) ) 1|y_i|= ( 1|y_i| _t=1^|y_i| _θ(y_i,t|x,y_i,<t) _ _old(y_i,t|x,y_i,<t) ). (16) The advantage computation remains the same as in GRPO. A.2 Mean Absolute Advantage as Curriculum Reward An ideal curriculum should prioritize training problems that maximize the model’s immediate learning outcomes. A natural way to quantify learning outcomes is through the magnitude of parameter updates induced by the selected training data, which can be approximated by the absolute advantage. In the common setting of RL with verifiable binary rewards, mean absolute advantage has a attractive interpretation. When using group-based RL methods like GRPO with n rollouts per problem, the advantage for the i-th rollout is computed as: A^t,i=ri−mean(r)std(r)=ri−p(1−p), A_t,i= r_i-mean(r)std(r)= r_i-p p(1-p), (17) where p is the empirical success rate over the group. Since the reward rir_i follows a Bernoulli distribution, the expected absolute advantage is: [|A^t,i|] [| A_t,i|] =p⋅1−p(1−p)+(1−p)⋅p(1−p)=2p(1−p). =p· 1-p p(1-p)+(1-p)· p p(1-p)=2 p(1-p). (18) The function g(p)=2p(1−p)g(p)=2 p(1-p) is symmetric around p=0.5p=0.5, strictly concave on [0,1][0,1], and reaches its maximum at p=0.5p=0.5. Thus, maximizing expected absolute advantage is equivalent to prioritizing problems at a success rate of 50%. A.3 Value-Based Curation Value-based curation methods maintain explicit estimates of the utility or learning value associated with training data, which are then used to guide adaptive curriculum selection during RL post-training. The core principle of value-based curation is to learn a utility function QtQ_t that maps training data to expected learning outcomes at training step t. Let S denote the space over which utilities are estimated (e.g., individual problems, problem categories, or problem features). The utility function Qt:→ℝQ_t:S assigns a scalar value to each element s∈s . During curriculum selection, the curator samples from S according to a policy derived from QtQ_t. A common choice is the Boltzmann (softmax) policy: pt(s)=exp(Qt(s)/η)∑s′∈exp(Qt(s′)/η),p_t(s)= (Q_t(s)/η) _s (Q_t(s )/η), (19) where η>0η>0 is a temperature parameter that controls the exploration-exploitation tradeoff: higher temperatures lead to more uniform sampling (exploration), while lower temperatures concentrate probability mass on high-utility items (exploitation). The utility function is updated over time based on observed learning outcomes. After selecting data according to ptp_t and performing an RL update, the curator observes a reward signal rt(s)r_t(s) that measures the actual learning value obtained. These observations are used to refine Qt+1Q_t+1. SEC. Self-Evolving Curriculum (SEC) (Chen et al., 2025a) instantiates value-based curation by defining the utility space =CS=C as a discrete set of problem categories (e.g., difficulty levels, problem types). The utility function Qt:C→ℝQ_t:C is represented as a lookup table, with one scalar value per category. SEC uses temporal difference (TD) learning to update utilities: Qt+1(c)=αrt(c)+(1−α)Qt(c),Q_t+1(c)=α r_t(c)+(1-α)Q_t(c), (20) where α∈(0,1]α∈(0,1] is a learning rate and rt(c)r_t(c) is the mean absolute advantage aggregated over all problems from category c selected at step t. This exponential moving average naturally adapts to non-stationarity as category utilities change with model improvement. Curriculum selection proceeds in two stages: first, a category is sampled according to the Boltzmann policy pt(c)∝exp(Qt(c)/η)p_t(c) (Q_t(c)/η); then, problems are uniformly sampled from the selected category. Regression. An alternative instantiation defines the utility space =S=X directly at the problem level, estimating utilities for individual training problems. The utility function is parameterized by a neural network Qϕ:→ℝQ_φ:X that takes problem representations (e.g., text embeddings) as input. Rather than incremental TD updates, the curator is trained via supervised regression. Let ℋt=(xj,rj)j=1tH_t=\(x_j,r_j)\_j=1^t denote the history of problem-reward observations. The utility network is optimized to minimize mean squared error: ℒMSE(ϕ)=∑(xj,rj)∈ℋt(Qϕ(xj)−rj)2.L_MSE(φ)= _(x_j,r_j) _t (Q_φ(x_j)-r_j )^2. (21) This can be optimized periodically (every K steps) via batch gradient descent, or online via stochastic gradient descent: ϕt+1=ϕt−β∇ϕ(Qϕ(xt)−rt(xt))2, _t+1= _t-β _φ (Q_φ(x_t)-r_t(x_t) )^2, (22) where β is the learning rate. Problems are sampled directly according to pt(x)∝exp(Qϕ(x)/η)p_t(x) (Q_φ(x)/η). PCL (Gao et al., 2025) exemplifies the online regression approach. PCL updates QϕQ_φ concurrently with policy training using only the current batch of observations. At each step t, PCL samples a candidate pool of kmkm prompts and selects the m prompts whose predicted values are closest to a target threshold τ (typically 0.5): Dm=argminS⊆Dkm,|S|=m∑x∈S|Qϕ(x)−τ|.D_m= _S D_km,|S|=m _x∈ S|Q_φ(x)-τ|. (23) This greedy selection can be viewed as an extreme limit of Boltzmann sampling with a sharply peaked distribution. Consider the modified utility Q~(x)=−|Qϕ(x)−τ| Q(x)=-|Q_φ(x)-τ|, which measures negative distance from the threshold. As the temperature η→0η→ 0, the Boltzmann policy p(x)∝exp(Q~(x)/η)p(x) ( Q(x)/η) concentrates all probability mass on prompts nearest to τ, recovering PCL’s greedy selection. This deterministic selection strategy is computationally efficient and ensures the training batch contains only prompts of target difficulty, maximizing the effective ratio of informative gradient signals. After selecting prompts, PCL generates n responses per prompt and updates both the policy and value model. The value model is trained on the observed rewards from the selected prompts: ℒPCL(ϕ)=∑i=1m(Qϕ(xi)−1n∑j=1nr(xi,yi,j))2.L_PCL(φ)= _i=1^m (Q_φ(x_i)- 1n _j=1^nr(x_i,y_i,j) )^2. (24) Comparison to Actor-Curator. The Actor-Curator framework with OSMD differs fundamentally from value-based curation methods in its optimization objective and data selection mechanism. While value-based methods learn utilities Qt(s)Q_t(s) to predict expected learning outcomes and sample accordingly, OSMD directly optimizes a curriculum distribution t p_t to maximize expected policy improvement. Value-based approaches require estimating problem-level or category-level values and making selection decisions based on these estimates—a two-stage process that introduces approximation error. In contrast, OSMD treats curriculum optimization as a first-order problem: the gradient ∇qJ(πt,) _qJ( _t, p) directly specifies how to adjust the data distribution to improve the policy, without requiring intermediate value estimates. Furthermore, value-based methods typically rely on scalar reward signals rt(s)r_t(s) to update utilities, whereas OSMD leverages the full gradient information ∇θℒ(πt;x) _θL( _t;x) to measure the learning value of each problem. This allows OSMD to capture richer information about how individual problems affect policy optimization, beyond what a single scalar reward can convey. Appendix B Policy improvement estimation This section proves unbiasedness of the per-problem bandit feedback estimators used to train the curator. Recall the per-problem utility at iteration t (Eq. (6)): ut=p()∼πt(⋅∣)[πt+1(∣)πt(∣)Aπt(∣)].u_ x^t=p_X( x)\, *E_ y π^t(· x) [ π^t+1( y x)π^t( y x)A_π^t( y x) ]. We also recall the rollout-based estimator (Eq. (7)): A^t(⋅∣)≜1|t|∑∈tπt+1(∣)πt(∣)Aπt(∣),t=(j)∼πt(⋅∣)j=1||. A^t(· x) 1|Y^t_ x| _ y ^t_ x π^t+1( y x)π^t( y x)A_π^t( y x), ^t_ x=\ y^(j) π^t(· x)\_j=1^|Y_ x|. B.1 Single-stage case Theorem 3 (Unbiasedness (single-stage)) For the estimator in Eq. (7), U^t=p()∈tpt()A^t(⋅∣), U^t_ x=p_X( x) I\ x ^t\p^t( x) A^t(· x), we have [U^t]=ut *E[ U^t_ x]=u^t_ x for every ∈ x . Proof: Fix an iteration t and a problem ∈ x . In the single-stage setting, the training set tX^t is sampled directly from X according to the curator distribution pt(⋅)p^t(·), so that Pr(∈t)=pt(). ( x ^t)=p^t( x). First, A^t(⋅∣) A^t(· x) is an unbiased estimator of the population quantity ∼πt(⋅∣)[πt+1(∣)πt(∣)Aπt(∣)], *E_ y π^t(· x) [ π^t+1( y x)π^t( y x)A_π^t( y x) ], by i.i.d. rollout sampling and linearity of expectation. Next, take expectation of U^t U^t_ x conditioning on A^t(⋅∣) A^t(· x): [U^t∣A^t(⋅∣)]=p()[∈tpt()]A^t(⋅∣)=p()A^t(⋅∣), *E\! [ U^t_ x A^t(· x) ]=p_X( x)\, *E\! [ I\ x ^t\p^t( x) ]\, A^t(· x)=p_X( x)\, A^t(· x), since [∈t]=Pr(∈t)=pt() *E[I\ x ^t\]= ( x ^t)=p^t( x). Finally, taking expectation over rollout randomness yields [U^t]=p()[A^t(⋅∣)]=ut, *E[ U^t_ x]=p_X( x)\, *E\! [ A^t(· x) ]=u^t_ x, which proves the claim. □ B.2 Two-stage case Theorem 4 (Unbiasedness (two-stage)) For the two-stage estimator in Eq. (10), U^two,t=p()∈tq()pt(∣~t)A^t(⋅∣), U^t_two, x=p_X( x)\, I\ x ^t\q( x)\,p^t( x X^t)\; A^t(· x), we have [U^two,t]=ut *E[ U^t_two, x]=u^t_ x for every ∈ x . Proof: Fix an iteration t and a problem ∈ x . By definition of the two-stage procedure (Section 3.5), the candidate set ~t X^t is sampled from q~ q, inducing the marginal inclusion probability q()=Pr(∈~t)q( x)= ( x∈ X^t). Conditioned on ~t X^t, the curator selects the training set t⊂~tX^t⊂ X^t according to pt(⋅∣~t)p^t(· X^t). As in the single-stage case, A^t(⋅∣) A^t(· x) is an unbiased estimator of the corresponding population expectation under ∼πt(⋅∣) y π^t(· x). Now condition on the realized candidate set ~t X^t and on A^t(⋅∣) A^t(· x). If ∉~t x∉ X^t, then ∈t=0I\ x ^t\=0 almost surely. If ∈~t x∈ X^t, then by the selection step, [∈t∣~t]=pt(∣~t), *E\! [I\ x ^t\ X^t ]=p^t( x X^t), and therefore [∈tpt(∣~t)|~t,A^t(⋅∣)]=∈~t. *E\! [ I\ x ^t\p^t( x X^t)\; |\; X^t, A^t(· x) ]=I\ x∈ X^t\. Substituting into the estimator gives [U^two,t|~t,A^t(⋅∣)]=p()∈~tq()A^t(⋅∣). *E\! [ U^t_two, x\; |\; X^t, A^t(· x) ]=p_X( x)\, I\ x∈ X^t\q( x)\; A^t(· x). Taking expectation over the proposal step yields [U^two,t|A^t(⋅∣)]=p()[∈~t]q()A^t(⋅∣)=p()A^t(⋅∣), *E\! [ U^t_two, x\; |\; A^t(· x) ]=p_X( x)\, *E[I\ x∈ X^t\]q( x)\; A^t(· x)=p_X( x)\, A^t(· x), since [∈~t]=Pr(∈~t)=q() *E[I\ x∈ X^t\]= ( x∈ X^t)=q( x). Finally, taking expectation over rollout randomness gives [U^two,t]=p()[A^t(⋅∣)]=ut, *E[ U^t_two, x]=p_X( x)\, *E[ A^t(· x)]=u^t_ x, which proves the claim. □ Appendix C Idealized OSMD algorithm Algorithm 2 Sleeping Online Mirror Descent 1:Number of total arms K, number of available arms k each round, horizon T, step size η>0η>0, exploration parameter α∈(0,1/k)α∈(0,1/k). 2:Initialize 1=(1/K,…,1/K) p_1=(1/K,…,1/K). 3:for t=1,2,…,Tt=1,2,…,T do 4: Sample available subset ~t X_t. Compute t(i∣~t)=t,i∑j∈~t,jif i∈~t0otherwise p^t(i X_t)= cases p_t,i _j∈ X_t p_t,j&if $i∈ X_t$\\ 0 &otherwise cases (25) 5: Sample at,1,at,2,…,at,s∼i.i.d.t(⋅∣~t)a_t,1,a_t,2,…,a_t,s i.i.d. p^t(· X_t) 6: (Loss Estimator) For each arm i∈[K]i∈[K], set L^t,i=1s∑r=1sat,r=ilt,it(i∣~t) L_t,i= 1s _r=1^s1\a_t,r=i\ l_t,i p^t(i X_t) (26) 7: (OSMD Update) Update the next distribution by the mirror step t+1∈argminp∈η⟨p,^t⟩+DF(p,t), p_t+1∈ _p \η\, p, L_t +D_F(p, p_t) \, (27) whereDF(,)=∑iuiloguivi D_F( u, v)= _iu_i u_iv_i 8:end for C.1 Setup We formalize the tabular bandit algorithm in Section 3 as Algorithm 2. Under this idealized algorithm. We assume there is a large set of K arms. At each time step t, k arms are uniformly randomly chosen as candidate arms. The settings allows for the pulling of s arms per round, and after each round the loss lt,il_t,i is revealed for each chosen arm i, where i indicates the arm’s original index in [K][K]. Throughout this section and the next, we denote the vectorized quantities with bold font. For example t=(lt,1,…,lt,K) l_t=(l_t,1,…,l_t,K). Algorithm 2 departs from traditional Online Stochastic Mirror Descent (OSMD) due to the availability constraint. The action distribution is conditioned on a randomly sampled available set at each round, and the loss estimator is modified to remain unbiased under this conditional sampling. Eqn. (25) introduces a two-stage sampling process. First, a random k-subset ~t⊆[K] X_t [K] of available arms is drawn uniformly. The learner then constructs a conditional distribution t(i∣~t)=t,i∑j∈~t,jfor i∈~t, p^t(i X_t)= p_t,i _j∈ X_t p_t,j i∈ X_t, and assigns zero probability to arms outside ~t X_t. We sometimes also use ∣~t(i) p_ X_t(i) to denote t(i∣~t) p^t(i X_t). This renormalization ensures that the learner only samples from arms that are available at round t, while still using t p_t as the global state variable that is updated over time. The conditional sampling in Eqn. (25) invalidates the standard OSMD estimator, since pt,ip_t,i is no longer the actual probability with which arm i is sampled. Eqn. (26) addresses this by defining L^t,i=1s∑r=1sat,r=ilt,it(i∣~t). L_t,i= 1s _r=1^s1\a_t,r=i\ l_t,i p^t(i X_t). This estimator uses the conditional probability t(i∣~t) p^t(i X_t) in the denominator, which is the true sampling probability of arm i given the realized availability set. As a result, conditional on ~t X_t, the estimator is unbiased: [L^t,i∣~t]=lt,ii∈~t.E\! [ L_t,i X_t ]=l_t,i\,I\i∈ X_t\. The use of s independent samples further reduces variance and corresponds to a semi-bandit feedback model, but does not change the role of the estimator in the mirror update. Appendix D Regret Analysis This section presents a proof of the regret bound (Theorem 2) for Algorithm 2. Suppose there are K base arms [K]=1,…,K[K]=\1,…,K\ each corresponding to a problem in the dataset =(1),(2),…,(K)X=\ x^(1), x^(2),…, x^(K)\. At each step, a subset ~t X_t of size k is drawn uniformly randomly from X. For every round t, The curator picks s arms at,1,at,2,…,at,s∈[K]a_t,1,a_t,2,…,a_t,s∈[K] from ~t X_t and the loss for each arm lt,at,il_t,a_t,i is revealed. For convenience, write t=(lt,1,…,lt,K) l_t=(l_t,1,…,l_t,K) to be the losses of each arm as a vector. Assume without loss of generality that lt,i∈[0,1]l_t,i∈[0,1] for all t and i. Define the subset-masked loss vector t~t∈ℝK l_t X_t ^K: t~t(i)=t(i)if i∈~t0otherwise l_t X_t(i)= cases l_t(i)&if i∈ X_t\\ 0&otherwise cases (28) Denote the best available arm at time t as mtm_t mt=argmini∈~tlt,i and lt∗=lt,mt.m_t= _i∈ X_tl_t,i and l^*_t=l_t,m_t. (29) We define the best-arm regret to be RegTbest=[∑t=1T(∑i=1slt,at,s−lt∗)]Reg_T^best= *E [ _t=1^T( _i=1^sl_t,a_t,s-l^*_t) ] (30) Note that the regret can be expressed in the vectorized form. Regnbest=[∑t=1T⟨~t,t⟩−⟨mt,t⟩],Reg_n^best= *E [ _t=1^T p_ X_t, l_t - e_m_t, l_t ], (31) where i e_i is a one-hot vector with 11 on index i. Here, we use ∣~t p_ X_t to represent the vector (∣~t(1)),∣~t((2)),…,∣~t((K)))( p_ X_t x^(1)), p_ X_t( x^(2)),…, p_ X_t( x^(K))). In practice, forcing uniform exploration usually have negligible or even positive effect on the performance. However, it incurs linear regret for theoretical analysis. Thus, to focus on how OSMD manages losses in a non-stationary environment, this regret analysis isolate the loss attributable to factors other than uniform exploration. Towards this end, we set the comparator t q_t to be a mixture of the best available arm and uniform distribution over all arms. t=(1−kα)mt+α~t q_t=(1-kα) e_m_t+α 1_ X_t (32) where α>0α>0 and ~t∈ℝK1_ X_t ^K is a binary vector with ~t(i)=11_ X_t(i)=1 for all i∈~ti∈ X_t and zero everywhere else. The best-available regret is defined as RegTBA=[∑t=1T⟨∣~t,t⟩−⟨t,t⟩]Reg_T^BA= *E [ _t=1^T p_ X_t, l_t - q_t, l_t ] (33) Theorem 5 (Restatement of Theorem 2) Let the drift parameter VT=∑t=2TΔt,Δt=maxi∈[K]|lt,i−lt−1,i|V_T= _t=2^T _t, _t= _i∈[K] |l_t,i-l_t-1,i | (34) be a measure of how rapid the arm values changes over time t. Assume without loss of generality that t,i∈[0,1] l_t,i∈[0,1] for all t and i. Then we have RegnBA≤O(T2/3Vn1/3)Reg^BA_n≤ O (T^2/3V_n^1/3 ) (35) for Algorithm 2. To bound the smoothed regret, the proof follows a block decomposition argument. We partition the horizon into contiguous blocks of length B and restart the algorithm at the beginning of each block. Within a single block, the loss sequence is treated as approximately stationary, allowing us to compare the algorithm against a fixed comparator using standard OSMD analysis. The regret within each block is controlled by a stability–variance tradeoff: the mirror descent inequality yields a term of order O(log(1/α)/η)O( (1/α)/η), while the variance of the importance-weighted estimator contributes a term proportional to O(ηB)O(η B). Across blocks, non-stationarity is captured through a variation budget VnV_n, which upper bounds the cumulative discrepancy between the true per-round losses and the frozen losses used in the blockwise analysis, contributing an additive term of order BVnBV_n. The block length B is then optimized to balance the statistical cost of restarting too frequently against the bias induced by treating losses as stationary within each block. This optimization yields a regret bound scaling as O(Vn1/3)O(V_n^1/3), reflecting the classical tradeoff between adaptivity to non-stationarity and estimation error Besbes et al. (2014). Let the length of each aforementioned block be L. Denote the start time of each block as τl=(l−1)L+1 _l=(l-1)L+1, where l=1,2,…,Bl=1,2,…,B and B=T/LB=T/L111For convenience, we assume T is divisible by B and L at the same time.. Denote the time steps of the l-th block as ℐl=τl,…,τl+L−1I_l=\ _l,…, _l+L-1\. RegnBA=[∑t=1n⟨∣~t,t⟩−⟨t,t⟩]=∑l=1B[∑t∈ℐl⟨∣~t,t⟩−⟨t,t⟩]Reg_n^BA= *E [ _t=1^n p_ X_t, l_t - q_t, l_t ]= _l=1^B *E [ _t _l p_ X_t, y_t - q_t, l_t ] (36) D.1 Time-Frozen Comparator This part of the proof introduces a stable reference for comparison in the presence of non-stationary losses. Rather than comparing the learner to the best action at every round, which may change arbitrarily over time, we define a comparator based on the losses at a fixed reference τ. This “time-frozen” comparator serves as a proxy for the per-round best action. The construction allows us to relate the learner’s loss to this frozen benchmark, while the error incurred by freezing time can be bounded by the amount of variation in the losses. Define mtτ∈argmini∈~tyτ,im_t^τ∈ _i∈ X_ty_τ,i to be the τ-frozen best arm of time t. Denote the value associated with this arm (yt∗)τ=yτ,mtτ(y^*_t)^τ=y_τ,m_t^τ. Thus, we can define the time-frozen smooth comparator tτ=(1−kα)mtτ+α~t q_t^τ=(1-kα) e_m_t^τ+α 1_ X_t (37) Lemma 1 Let ℐ=τ,…,τ+L−1I=\τ,…,τ+L-1\ be an arbitrary block. Suppose α>0α>0. Let VnV_n be as defined in Theorem 5. For any sequence of random variables Xtt∈ℐ\X_t\_t with a common support, we have [∑t∈ℐXt−⟨t,t~tZt⟩]≤[∑t∈ℐXt−⟨tτ,t~tZt⟩]+1−kαkα∑t∈ℐ‖t−τ‖1 *E [ _t X_t- q_t, l_t X_tZ_t ]≤ *E [ _t X_t- q^τ_t, l_t X_tZ_t ]+ 1-kαkα _t \| l_t- l_τ\|_1 (38) Proof: By the definition of tτ q_t^τ, we have ⟨tτ,τ~t⟩ q_t^τ, l_τ X_t ≤(1−kα)(yt∗)τ+α∑j∈~tτ,j ≤(1-kα)(y^*_t)^τ+α _j∈ X_t l_τ,j (39) ≤(1−kα(yt∗)τ+α(k(yt∗)τ+k−1) ≤(1-kα(y^*_t)^τ+α(k(y^*_t)^τ+k-1) =mtτ+α(k−1) =m_t^τ+α(k-1) Rearranging (39) and divide by ZtZ_t, we get mtτZt≥⟨tτ,τ~tZt⟩−α(k−1)Zt≥⟨tτ,τ~tZt⟩−k−1k. m_t^τZ_t≥ q_t^τ, l_τ X_tZ_t - α(k-1)Z_t≥ q_t^τ, l_τ X_tZ_t - k-1k. (40) Using the fact that min is a Lipschitz operation, we have |yt∗−(yt∗)τ|=|mini∈~tt,i−mini∈~tτ,i|≤‖t−τ‖∞≤‖t−τ‖1, |y^*_t-(y^*_t)^τ |= | _i∈ X_t l_t,i- _i∈ X_t l_τ,i |≤\| l_t- l_τ\|_∞≤\| l_t- l_τ\|_1, Applying this identity, we have |⟨t,t~tZt⟩−⟨tτ,t~tZt⟩| | q_t, l_t X_tZ_t - q^τ_t, l_t X_tZ_t | ≤1kα|⟨t,t~t⟩−⟨tτ,t~t⟩| ≤ 1kα | q_t, l_t X_t - q^τ_t, l_t X_t | (41) =1−kαkα|yt∗−(yt∗)τ| = 1-kαkα |y^*_t-(y^*_t)^τ | ≤1−kαkα‖t−τ‖1 ≤ 1-kαkα\|y_t-y_τ\|_1 Rearrange (41) and add XtX_t to both sides while taking expectation, we get [∑t∈ℐXt−⟨t,t~tZt⟩]≤[∑t∈ℐXt−⟨tτ,t~tZt⟩]+1−kαkα∑t∈ℐ‖t−τ‖1 *E [ _t X_t- q_t, l_t X_tZ_t ]≤ *E [ _t X_t- ^τ_t, l_t X_tZ_t ]+ 1-kαkα _t \| l_t- l_τ\|_1 (42) □ D.2 Moment Lemmas To apply the standard OMD bound (Theorem 5), we require unbiased loss estimates with controlled variance. Since the learner only observes losses on the sampled subset (~t X_t), we work with an importance-weighted estimator that accounts for both subset sampling and the normalization induced by ZtZ_t. The following two lemmas establish the properties needed for the regret analysis. The first shows that the estimator is unbiased for the scaled loss, ensuring that the expected update direction matches the true loss. The second provides a bound on the second moment of the estimator, which controls the variance term in the OSMD regret bound. Together, these results justify the use of the estimator in the mirror descent analysis and quantify the cost introduced by partial observation and uniform exploration. Lemma 2 (Unbiasedness) Let L^t,i L_t,i be as defined in Algorithm 2. Denote ^t=(L^t,1,L^t,2,…,L^t,K) L_t=( L_t,1, L_t,2,…, L_t,K). For every i∈[K]i∈[K], [L^t,i∣~t]=t,ii∈~t=t~t(i) *E[ L_t,i X_t]= l_t,iI\i∈ X_t\= l_t X_t(i)\\ (43) Additionally, define the time-frozen estimator L^t,iτ:=1s∑r=1sat,r=ilτ,it(i∣~t), L_t,i^τ:= 1s _r=1^sI\a_t,r=i\ l_τ,i p^t(i X_t), (44) and we have [L^t,iτ∣~t]=lτ,ii∈~t=τ~t(i) *E[ L_t,i^τ X_t]=l_τ,iI\i∈ X_t\= l_τ X_t(i) (45) Proof: If i∈~ti∈ X_t, [L^t,iτ|~t]=1s∑r=1st(i∣~t)lt,ii∈~tt(i∣~t)=lt,ii∈~t *E [ L_t,i^τ | X_t ]= 1s _r=1^s p_t(i X_t) l_t,iI\i∈ X_t\ p^t(i X_t)=l_t,iI\i∈ X_t\ Similarly, [L^t,iτ|~t]=1s∑r=1st(i∣~t)lτ,ii∈~tt(i∣~t)=lτ,ii∈~t *E [ L_t,i^τ | X_t ]= 1s _r=1^s p_t(i X_t) l_τ,iI\i∈ X_t\ p^t(i X_t)=l_τ,iI\i∈ X_t\ □ Lemma 3 (Bounded Second Moment) Let L^t,j L_t,j be as defined in Algorithm 2. Suppose we obtain t p^t through t∈argmin∈η⟨,^t⟩+DF(,t−1), p^t∈ _ p η p, L_t +D_F( p, p^t-1), under the constraint t(i)≥α p^t(i)≥α for all i. We have [∑j=1KjtL^t,j2|~t]≤ks *E [ _j=1^K p^t_j L_t,j^2 | X_t ]≤ ks (46) Proof: For j∉~tj ∈ X_t, L^t,j=0 L_t,j=0, For j∈~tj∈ X_t, let Xr,j:=At,r=jτ,jt(j∣~t),consequently L^t,j=1s∑r=1sXr,jX_r,j:=I \A_t,r=j \ l_τ,j p^t(j X_t), L_t,j= 1s _r=1^sX_r,j By construction, Xr,jX_r,j’s are IID in r, and since τ,j∈[0,1] l_τ,j∈[0,1], [Xr,j2∣~t]=∑j∈~t(j∣~t)τ,j2t(j∣~t)2≤1t(j∣~t) *E [X^2_r,j X_t ]= _j∈ X_t p^t(j X_t) l^2_τ,j p^t(j X_t)^2≤ 1 p^t(j X_t) By Jensen’s inequality, we have [(L^t,jτ)2∣~t]=[(1s∑r=1sXr,j)2|~t]≤1s[Xr,j2|~t]≤1st(j∣~t) *E [( L_t,j^τ)^2 X_t ]= *E [ ( 1s _r=1^sX_r,j )^2 | X_t ]≤ 1s *E [X_r,j^2 | X_t ]≤ 1s p^t(j X_t) Summing with weights t p^t over j∈~tj∈ X_t gives ∑j∈~t(∣~t)j[(L^t,jτ)2|~t]≤1s∑j∈~t(∣~t)jZt(∣~t)j=ksZt≤ks _j∈ X_t ( p^t_ X_t )_j *E [( L_t,j^τ)^2 | X_t ]≤ 1s _j∈ X_t ( p^t_ X_t )_j Z_t ( p^t_ X_t )_j= ksZ_t≤ ks □ D.3 One-Step OMD Bound The purpose of the following lemma is to provide a local inequality that governs how a single mirror descent update behaves under the importance-weighted loss estimator. When summed over time, the KL terms telescope while the second-moment terms accumulate in a controlled manner. This structure allows the proof to separate the effect of the update rule from issues caused by partial observability and non-stationarity, which are handled in later steps. Lemma 4 Suppose t p^t’s are obtained as in Algorithm 2. Then, for any comparator t∈Δ~t v_t∈ _ X_t, ⟨∣~t−t,t⟩≤1η(KL(t∥∣~t)−KL(t∥∣~t+1t+1))+ηs22∑i∈Ctt(i∣~t)L^t,i2 p^t_ X_t- v_t, l_t ≤ 1η (KL( v_t\| p^t_ X_t)-KL( v_t\| p^t+1_ X_t+1) )+ η s^22 _i∈ C_t p^t(i X_t) L_t,i^2 (47) Proof: For the simplicity of the proof, fix t. Write C=~tC= X_t. Let pi=itp_i= p^t_i, Z=ZtZ=Z_t, qi=t(i∣~t)q_i= p^t(i X_t), and L^i=L^t,i L_i= L_t,i. Write the updated weights as pi+=piexp(−ηL^i)p_i^+=p_i (-η L_i) and the updated active sum as Z+=∑i∈Cp+Z^+= _i∈ Cp^+. Let qi+=pi+/Z+q_i^+=p_i^+/Z^+. We denote the vector form of the aforementioned quantities in bold. e.g. (i)=pi p(i)=p_i. Note logqiqi+=logpi/Zpiexp(−ηL^i)/Z+=logZ+Z+ηL^i q_iq_i^+= p_i/Zp_i (-η L_i)/Z^+= Z^+Z+η L_i (48) Now, KL(∥+)−KL(∥) ( v\| q^+)-KL( v\| q) =∑i∈Cvilogviqi+−∑i∈Cvilogviqi = _i∈ Cv_i v_iq^+_i- _i∈ Cv_i v_iq_i (49) =∑i∈Cvilogqiqi+ = _i∈ Cv_i q_iq^+_i =logZ+Z+η⟨,^⟩ = Z^+Z+η v, L Rearranging we have η⟨,^⟩=KL(∥+)−KL(∥)−logZ+Zη v, L =KL( v\| q^+)-KL( v\| q)- Z^+Z (50) Next, apply the fact that e−x≤1−x+x2e^-x≤ 1-x+x^2 for x≥0x≥ 0 and taking logs, we have logZ+Z Z^+Z ≤log(∑i∈Cqi(1−ηL^i+η22L^i2)) ≤ ( _i∈ Cq_i (1-η L_i+ η^22 L_i^2 ) ) (51) =log(1−η⟨,^⟩+η22∑i∈CqiL^i2) = (1-η q, L + η^22 _i∈ Cq_i L_i^2 ) ≤−η⟨,^⟩+η22∑i∈CqiL^i2 ≤-η q, L + η^22 _i∈ Cq_i L_i^2 where the last inequality applies log(1+x)≤x (1+x)≤ x for all x>−1x>-1. Now plug (51) into (50), η⟨,^⟩≥KL(∥+)−KL(∥)+η⟨,^⟩+η22∑i∈CqiL^i2η v, L ( v\| q^+)-KL( v\| q)+η q, y + η^22 _i∈ Cq_i L_i^2 (52) Rearranging the inequality and dividing by η, we get ⟨−,^⟩≤1η(KL(∥)−KL(∥))+η2∑i∈CqiL^i2 q- v, L ≤ 1η(KL( v\| q)-KL( v\| q))+ η2 _i∈ Cq_i L_i^2 (53) Substituting the original notation back, we recover the claim. □ D.4 Reduction to Blockwise Fixed Arm The next two lemmas connect the one-step bound to a blockwise analysis under non-stationarity. Lemma 5 controls the discrepancy between the per-round best available arm and a single arm fixed over a block, showing that this gap is governed by the cumulative variation within the block. Lemma 6 then combines this control with the one-step OMD bound to obtain a regret bound against a fixed comparator over the block. Together, these results allow the per-round inequalities to be aggregated while isolating the effect of non-stationarity. Lemma 5 Fix a block ℬ=t0,t0+1,…,t0+L−1B=\t_0,t_0+1,…,t_0+L-1\. Dentoe the blockwise best arm as mℬ∈argmini∈[K]∑i∈ℬL^t,ii∈~tm_B∈ _i∈[K] _i L_t,iI\i∈ X_t\ (54) Let Δt _t be as defined in Theorem 5. Then, ∑t∈ℬ(L^t,mℬmℬ∈Ct−L^t,mt)≤L∑t∈B∖t0Δt _t ( L_t,m_BI\m_B∈ C_t\- L_t,m_t )≤ L _t∈ B \t_0\ _t (55) Proof: Let bt(C)=mini∈CL^i,tb_t(C)= _i∈ C L_i,t. By the Lipschitz property of min , it satisfies |bt(C)−bt−1(C)|≤maxi∈C|L^t,i−L^t−1,i|≤Δt |b_t(C)-b_t-1(C) |≤ _i∈ C | L_t,i- L_t-1,i |≤ _t (56) By telescoping (56) from t0t_0 to t, bt0(Ct)≤bt(Ct)+∑τ=t0+1tΔτ=mt+∑τ=t0+1tΔτb_t_0(C_t)≤ b_t(C_t)+ _τ=t_0+1^t _τ=m_t+ _τ=t_0+1^t _τ (57) By definition of mℬm_B as the best static arm for the block ∑t∈ℬL^t,mℬbℬ∈~t≤∑t∈ℬL^t,mtt0mtt0∈~t _t L_t,m_BI\b_B∈ X_t\≤ _t L_t,m_t^t_0I\m_t^t_0∈ X_t\ (58) Also note that by definition, L^t,mtt0≤L^t0,mtt0+∑τ=t0+1tΔτ=bt0(~t)+∑τ=t0+1tΔτ L_t,m_t^t_0≤ L_t_0,m_t^t_0+ _τ=t_0+1^t _τ=b_t_0( X_t)+ _τ=t_0+1^t _τ (59) Combine (56) and (59), ∑t∈ℬL^t,mℬmℬ∈~t≤∑t∈ℬ(L^t,mt+∑τ=t0+1tΔτ) _t L_t,m_BI\m_B∈ X_t\≤ _t ( L_t,m_t+ _τ=t_0+1^t _τ ) (60) Rearranging, we get ∑t∈ℬL^t,mℬmℬ∈~t−L^t,mt≤L∑t∈ℬ∖t0Δt _t L_t,m_BI\m_B∈ X_t\- L_t,m_t≤ L _t \t_0\ _t (61) □ Lemma 6 Fix a block ℬ=t0,t0+1,…,t0+L−1B=\t_0,t_0+1,…,t_0+L-1\. Run Algorithm 2 with η>0η>0 starting from time t0t_0. Let t∈Δ~t q_t∈ _ X_t be the played distribution at each round t∈ℬt and let ^t L_t be an unbiased estimator of the masked loss vector [^t∣~t]=t~t *E[ L_t X_t]= L_t X_t and that the second moment is bounded by [∑i∈~t,iL^t,i2]≤ks *E [ _i∈ X_t q_t,i L_t,i^2 ]≤ ks (62) for all t. Then, [∑t∈ℬ⟨t,tCt⟩−t,mℬmℬ∈~t]≤log(1/α)η+skηL2 *E [ _t q_t, l_t^C_t - l_t,m_BI\m_B∈ X_t\ ]≤ (1/α)η+ skη L2 (63) Proof: For each t∈ℬt and any comparators t∈Δ~t v_t∈ _ X_t, apply Lemma 4 ∑t∈ℬ⟨t−t,t⟩≤1η∑t∈ℬ(KL(t∥t)−KL(t∥t+1))+ηs22∑t∈ℬ∑i∈~t,iL^t,i2 _t q_t- v_t, L_t ≤ 1η _t (KL( v_t\| q_t)-KL( v_t\| q_t+1) )+ η s^22 _t _i∈ X_t q_t,i L_t,i^2 (64) By the unbiasedness of ^t y_t, [⟨t−t,t⟩]=[⟨t−t,t~t⟩] *E[ q_t- v_t, l_t ]= *E [ q_t- v_t, l_t X_t ] For the KL terms, we have ∑t∈ℬ(KL(t∥t)−KL(t∥t+1)) _t (KL( v_t\| q_t)-KL( v_t\| q_t+1) ) =KL(t∥t0)−KL(t∥t0+L−1) =KL( v_t\| q_t_0)-KL( v_t\| q_t_0+L-1) (65) ≤log(1/α) ≤ (1/α) where the first equality is a result of telescoping with respect to t and the second due to the fact that t0q_t_0 is restarted from uniform. For the second term on the RHS of (64), we invoke Lemma 3. Taking expectation on both sides, we get (63). □ D.5 Proof of Theorem 5 Note that ∑t∈ℬ⟨t,t~t⟩−t,mt=(∑t∈ℬ⟨t,t~t⟩−t,mℬmℬ∈~t)+(∑t∈ℬt,mℬmℬ∈~t−t,mt) _t q_t, l_t X_t - l_t,m_t= ( _t q_t, l_t X_t - l_t,m_BI\m_B∈ X_t\ )+ ( _t l_t,m_BI\m_B∈ X_t\- l_t,m_t ) (66) We bound the first term using Lemma 6 and the second term using 5. Taking the expectation and summing over blocks, we get RegnBA=[∑i=1B∑t∈ℬi⟨t,t~t⟩−t,mt]≤TLlog(1/α)η+ηsk2T+LVnReg_n^BA= *E [ _i=1^B _t _i q_t, l_t X_t - l_t,m_t ]≤ TL (1/α)η+ η sk2T+LV_n (67) Take L∗ L^* =(TVnsklog(1/α)2)2/3, = ( TV_n sk (1/α)2 )^2/3, (68) η∗ η^* =22/3(log(1/α))1/3(sk)−2/3(VnT)1/3. =2^2/3\,( (1/α))^1/3\,(sk)^-2/3 ( V_nT )^1/3. (69) we get RegnBA≤321/3(T2sklog(1/α)Vn)1/3.Reg_n^BA\;≤\; 32^1/3\, (T^2\,sk\, (1/α)\,V_n )^1/3. (70) Appendix E Performance improvement contribution of each problem We provide further justification for the per-problem policy improvement contribution defined in Eq. 6. Intuitively, this quantity measures the marginal contribution of including a training problem x in an actor update to the overall improvement in policy performance under the evaluation distribution p_X. This interpretation is exact for a broad class of tabular policies, where the policy parameters for different problems (or states) are independent. In such settings, updating the policy using trajectories from a problem x affects only the conditional distribution π(⋅∣)π(· x) and leaves the policy unchanged on all other problems. Example: tabular REINFORCE. Consider tabular REINFORCE, where for each problem x the policy π(⋅∣)π(· x) is parameterized independently. At iteration t, suppose we update the policy using rollouts collected only from a single problem x. By construction, this update modifies π(⋅∣)π(· x) but does not change π(⋅∣′)π(· x ) for any ′≠ x ≠ x. The overall performance objective is J(π)=∑′∈p(′)∼π(⋅∣′)[R(∣′)].J(π)= _ x p_X( x )\,E_ y π(· x ) [R( y x ) ]. Since only the conditional policy at x is changed, the performance difference J(πt+1)−J(πt)J(π^t+1)-J(π^t) depends solely on how the expected reward at x changes. Applying the standard performance difference identity (Kakade and Langford, 2002) to the single-turn setting yields J(πt+1)−J(πt)=p()∼πt(⋅∣)[πt+1(∣)πt(∣)Aπt(∣)],J(π^t+1)-J(π^t)=p_X( x)\,E_ y π^t(· x) [ π^t+1( y x)π^t( y x)A_π^t( y x) ], which is exactly the per-problem utility utu^t_ x defined in Eq. 6. Thus, in tabular REINFORCE, updating the policy on a single problem x produces an expected performance improvement of utu^t_ x. The same reasoning extends directly to batch updates: if the policy is updated using a subset of problems tX^t, the total performance improvement decomposes additively as ∑∈tut _ x ^tu^t_ x, and including a problem x in the update contributes exactly utu^t_ x to the expected performance gain. Other tabular methods. This exact additive interpretation applies equally to other tabular reinforcement learning methods, including tabular policy gradient methods, tabular Q-learning, and SARSA, where updates based on a problem or state affect only the corresponding local policy or value parameters. In all such cases, the per-problem utility utu^t_ x captures the true marginal contribution of training on x. Discussion: function approximation. In the presence of function approximation, updating the policy using trajectories from one problem generally affects the policy on other problems as well, breaking the exact additivity described above. Analyzing such cross-problem interference requires strong assumptions on the structure of the function class and the optimization dynamics, and is beyond the scope of this work. Nevertheless, when policy updates are small—as is typical in modern RL post-training algorithms such as PPO-style methods, GRPO, or GSPO—the per-problem utility utu^t_ x remains a first-order approximation to the marginal contribution of problem x to performance improvement. Our empirical results in Sec. 4.1 indicate that this approximation is sufficiently accurate to drive effective curriculum learning at scale. Appendix F Limitations While Actor-Curator is agnostic to the specific actor update rule and can be combined with a wide range of post-training algorithms, its gains ultimately depend on the stability and quality of the underlying actor optimization: unstable updates can still lead to noisy dynamics or training collapse, a common failure mode in reinforcement learning that curriculum learning alone cannot fully address. In addition, Actor-Curator inherits standard assumptions from RL-based LLM post-training, most notably access to a reliable reward signal, making it best suited to domains with objectively verifiable rewards such as mathematics, logic puzzles, and code. Finally, learning an explicit curator introduces additional computational overhead beyond standard RL post-training; while modest relative to actor training and rollout generation, this cost is non-negligible, and in our experiments Actor-Curator increases overall wall-clock training time by approximately 9% (see App. H), though this overhead is small compared to the observed gains in training efficiency. Appendix G Experimental setup G.1 Hyper-parameters Unless otherwise specified, all experiments use the same hyper-parameter configuration for Actor-Curator. The complete set of hyper-parameters is summarized in Table 2. We briefly explain key parameters below, with a focus on those specific to the actor–curator framework. Parameter Value Model configuration Curator model Qwen3-0.6B Actor KL loss Disabled Max problem length 1024 tokens Max solution length 4096 tokens Sampling and batch sizes Candidate batch size |~t|| X^t| 2048 Training batch size |t||X^t| 256 Rollouts per problem |t||Y^t_ x| 8 Actor optimization Actor temperature 1.0 Actor training top-p 1.0 Actor validation top-p 0.7 Actor learning rate 1×10−61× 10^-6 Actor LR warmup ratio 0.05 Actor weight decay 0.1 Actor gradient clipping 1.0 Actor clip range ρmin,ρmax _ , _ [3×10−4, 4×10−4][3× 10^-4,\,4× 10^-4] Curator optimization Curator dormant steps 20 Curator warm-up steps 5 Curator temperature 1.0 Curator top-p 0.9 Curator learning rate 1×10−61× 10^-6 Curator PPO clip range ρmax−ρmin _ - _ 0.2 Table 2: Hyper-parameters used for Actor-Curator across all experiments unless otherwise specified. Candidate and training batch sizes. At each training iteration, a candidate batch ~t X^t of size 2048 is first sampled from the proposal distribution q. The curator then reweights this candidate set and samples a smaller training batch tX^t of size 256, which is used for actor rollouts and updates. This two-stage sampling scheme follows Section 3.5 and allows scalable curation over large problem banks. Rollouts per problem. For each selected problem x∈tx ^t, we generate |xt|=8|Y^t_x|=8 on-policy rollouts from the current actor. These rollouts are used both for the actor update and for estimating per-problem policy improvement signals used to train the curator. Curator dormant and warm-up steps. During the first curator dormant steps (20 iterations), problems are sampled uniformly from the candidate batch, and curator outputs are ignored. This stabilizes early actor learning before meaningful policy-improvement estimates can be obtained. During the subsequent curator warm-up steps (5 iterations), the curator begins to influence sampling, but its parameters are updated conservatively. After warm-up, the curator is fully active and trained online using bandit feedback. Sampling prior. When use sampling prior is enabled, curator-assigned weights are multiplied by the proposal-induced marginal inclusion probability q(x)q(x) before normalization. This encourages coverage of the full dataset and prevents the curator from collapsing onto a narrow subset of problems early in training, consistent with the two-stage unbiased estimator in Equation (12). Proximal curator clipping. Curator updates use the PPO-style clipped OSMD objective described in Section 3.6. The clipping range ρmin,ρmax _ , _ constrains the importance ratio between consecutive curator policies, stabilizing learning under function approximation. The additional clip range parameter controls the maximum allowed deviation between these bounds. G.2 Hardware All experiments were conducted on NVIDIA A100 and H200 GPUs. Actor rollout generation and optimization dominate overall runtime; curator training introduces approximately 14% additional wall-clock cost relative to uniform sampling, as discussed in Appendix E. Appendix H Additional analysis H.1 Overhead We found that Actor-Curator adds about 9% training wall time overhead. We present the overhead by dataset and model in Tab. 3. However, as shown in Fig. 11, this is relatively the compared to the efficiency gains. Table 3: Average wall-time overhead percentage by model and datasets, averaged over training steps 500 steps. Countdown Zebra ARC-1D MATH Qwen2.5-3B-Base 11.86 16.71 17.12 9.56 Llama3.2-3B-IT 9.34 13.63 19.40 9.82 Appendix I Additional Ablation Candidate batch size |~|| X|. As shown in Fig. 8, candidate sizes of 512512 and 20482048 yield similar final performance. Larger batches (e.g., 81928192) lead to unstable training, likely due to reduced exploration and overfitting. Figure 8: Effect of candidate batch size on test performance. While bsz=2048 and bsz=512 converge to similar performance at step 200, bsz=8192 suffers from instability. Selected batch size is held constant at 256. Appendix J Main results (extended) This appendix provides extended empirical results complementing the main paper. We report additional quantitative comparisons across benchmarks, models, and curriculum learning methods, as well as detailed training dynamics over time. These results further substantiate the robustness, efficiency, and stability of AC across diverse problem domains and model backbones. Extended performance comparison. Tab. 4 reports peak validation performance within the first 100 training steps across all benchmarks and models. Compared to uniform sampling, heuristic curricula (SEC), and learning-based baselines (PCL), AC consistently achieves higher peak performance on most benchmarks. The gains are particularly pronounced on harder subsets (e.g., Countdown-hard, Zebra-hard, ARC-hard, and AIME24), highlighting the effectiveness of directly optimizing for expected policy improvement when problem difficulty and utility are highly non-uniform. While performance on MATH500 is largely saturated for some model configurations, AC remains competitive and avoids degradation relative to strong baselines. Training dynamics and stability. Figs. 9 and 10 visualize test performance as a function of training steps for Qwen2.5-3B-Base and Llama3.2-3B-it, respectively. Across benchmarks, AC not only reaches higher peak performance but also exhibits faster convergence and more stable learning dynamics. In many cases, competing methods plateau early or exhibit higher variance, whereas AC continues to make steady progress, effectively raising the performance ceiling. Model-agnostic behavior. The trends observed in Figs. 9 and 10 are consistent across both base and instruction-tuned models, indicating that the benefits of AC are not tied to a specific initialization or training regime. This supports the claim that learning curricula via policy-improvement-driven signals provides a generally applicable mechanism for improving RL post-training efficiency and robustness. Training efficiency. Fig. 11 compares learning curves of Actor-Curator against uniform sampling on Countdown, Zebra, and ARC. Across all three benchmarks, Actor-Curator reaches the same target accuracy substantially earlier, yielding step-level speedups of 58.2%58.2\% on Countdown, 80.7%80.7\% on Zebra, and 24.3%24.3\% on ARC. This indicates that policy-improvement-driven data selection primarily accelerates optimization by prioritizing high-impact problems. Figure 9: Test set performance across training steps (within 100 training steps) for Qwen2.5-3B-Base across benchmarks and methods. (a) Countdown (b) Countdown-hard (c) Zebra (d) Zebra-hard (e) ARC-1D (f) ARC-hard (g) MATH500 (h) AIME24 Figure 10: Test set performance across training steps (within 100 training steps) for Llama3.2-3B-it across benchmarks and methods. (a) Countdown (b) Countdown-hard (c) Zebra (d) Zebra-hard (e) ARC-1D (f) ARC-hard (g) MATH500 (h) AIME24 Figure 11: Training efficiency: Actor-Curator attains significant efficiency increase with relatively low overhead on Countdown, Zebra, and ARC. (a) Countdown (b) Zebra (c) ARC-1D Table 4: Peak validation performance on problems within 100 training steps across models and methods. Actor-Curator outperforms both other learning based methods (PCL) and methods that rely on human heuristics (SEC). Models: Qwen2.5 refers to Qwen2.5-3B-Base; Llama3.2 refers to Llama3.2-3B-it. Benchmark Model Method Improvement _ref Uniform SEC PCL AC (Ours) +Δ+ +Δ%+ \% Countdown Qwen2.5 0.00 44.74 58.87 57.24 62.12 +3.25 +5.52 Llama3.2 0.00 63.12 62.78 59.62 66.25 +3.13 +4.96 CD-hard Qwen2.5 0.00 41.00 51.50 48.00 58.00 +6.50 +12.62 Llama3.2 0.00 58.50 58.00 53.00 60.50 +2.00 +3.42 Zebra Qwen2.5 0.00 35.12 36.00 34.12 37.62 +1.62 +4.50 Llama3.2 0.00 44.50 46.50 48.25 47.12 -1.13 -2.34 Zebra-hard Qwen2.5 0.00 30.50 27.50 26.00 34.50 +4.00 +13.11 Llama3.2 0.00 37.50 38.00 39.50 40.50 +1.00 +2.53 ARC-1D Qwen2.5 0.00 26.74 27.87 26.37 36.37 +8.50 +30.51 Llama3.2 0.00 27.62 26.75 34.50 35.25 +0.75 +2.17 ARC-hard Qwen2.5 0.00 19.50 18.50 18.50 31.00 +11.50 +58.97 Llama3.2 0.00 23.00 24.50 24.00 31.50 +7.00 +28.57 MATH500 Qwen2.5 61.80 83.00 81.00 79.79 81.00 -2.00 -2.41 Llama3.2 41.00 52.20 52.00 52.40 53.60 +1.20 +2.29 AIME24 Qwen2.5 3.33 23.33 20.00 23.33 30.00 +6.67 +28.57 Llama3.2 0.00 13.33 13.33 13.33 16.67 +3.34 +25.06