Paper deep dive
Frontier Learning: Training LLM Reasoners at the Edge of Capability
Robin Faro, Shyam Sundhar Ramesh, Ilija Bogunovic, Aurelien Lucchi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement Learning-based post-training of Large Language Models (LLM) has been successfully applied to improve their reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator's task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model's evolving reasoning capabilities. Across several reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.
Tags
Links
- Source: https://arxiv.org/abs/2609.35426v1
- Canonical: https://arxiv.org/abs/2609.35426v1
Trouble viewing inline? Open PDF directly →
Full Text
82,139 characters extracted from source content.
Expand or collapse full text
Frontier Learning: Training LLM Reasoners at the Edge of Capability Robin Faro11footnotemark: 1 University of Basel Basel, Switzerland robin.faro@unibas.ch Shyam Sundhar Ramesh University College London London, UK shyam.ramesh.22@ucl.ac.uk †thanks: Equal contribution. Ilija Bogunovic University of Basel Basel, Switzerland †thanks: Equal supervision. Aurelien Lucchi22footnotemark: 2 University of Basel Basel, Switzerland Abstract Reinforcement Learning-based post-training of Large Language Models (LLM) has been successfully applied to improve their reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator’s task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model’s evolving reasoning capabilities. Across several reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability. 1 Introduction Recent advancements in the post-training of Large Language Models (LLM) have led to drastic gains in their reasoning capabilities (Ouyang et al., 2022; Lightman et al., 2023; Rafailov et al., 2023; Lambert et al., 2025; Guo et al., 2025). Among various post-training techniques, Reinforcement learning (RL) has emerged as a powerful paradigm, especially for structured reasoning domains such as math and logical reasoning (Guo et al., 2025; Shao et al., 2024), wherein rewards are assigned in a deterministic manner using rule-based or programmatic reward functions. This has enabled RL-based post-training to mitigate standard pitfalls such as reward hacking and yield improved reasoning performance. Standard RL-based post-training pipelines fine-tune LLMs on a fixed pool of problems often using Group Relative Policy Optimization (GRPO) (Guo et al., 2025; Shao et al., 2024). However, a limitation of the GRPO objective is that it provides no optimization signal for problems whose sampled responses all receive the same reward. To address this, recent methods filter or reweight problems within the fixed pool to concentrate optimization on more informative samples (Foster et al., 2025; Yu et al., 2025; Bae et al., 2025). Such approaches are ultimately limited by the coverage of the fixed problem pool and become ineffective once the policy can reliably solve every problem in the dataset. This raises a fundamental question: How should the training distribution adapt online to the model’s evolving capability? To address this, we propose to construct the training batch in an online manner using problems generated from a procedural data generator (Stojanovski et al., 2025; Chen et al., 2025a; Xu et al., 2026), which are capable of synthesizing reasoning tasks at scale. In particular, procedural data generators expose task-specific attributes defining the problem structure and allow one to produce a virtually unbounded stream of training problems. However, the exposed parameters usually do not provide an explicit or monotonic measure of difficulty: different combinations can interact in complex ways, and the difficulty of a configuration is ultimately relative to the capacities of the policy. Given this, we introduce frontier learning, an open-ended RL post-training approach that continually constructs new training data at the edge of the model’s evolving capability using the procedural data generator of a given reasoning task. Our approach actively identifies difficulty levels, defined through the task-specific attributes, that generate problems which are solvable enough to produce signal but not yet reliably solved by the current policy and hence, enables continual post-training using GRPO on a given reasoning task (see Figure 1). Figure 1: Frontier learning with procedural data generation. Levels vary along task-specific difficulty dimensions, here abstracted as structural complexity and reasoning depth. As the model improves, its frontier of capabilities shifts outward; frontier learning uses procedural generation to continually construct training data near this moving region. Related work: RL-based post-training (Guo et al., 2025; Shao et al., 2024; Hu et al., 2025; Team et al., 2025) has shown that deterministic rule-based rewards can elicit strong reasoning. However, GRPO provides no useful signal when all rollouts for a prompt are correct or incorrect (Zheng et al., 2025; Zhang et al., 2026b; Le et al., 2026). Recent filtering and reweighting methods therefore concentrate training on informative prompts (Foster et al., 2025; Yu et al., 2025; Bae et al., 2025; Li et al., 2025b; Gu et al., 2026; Cai et al., 2026; Li et al., 2025a), but remain limited by the coverage of the available problem pool. Curriculum and learnability-based methods (Bengio et al., 2009; Graves et al., 2017; Matiisen et al., 2017; Oudeyer et al., 2007; Willems et al., 2020) similarly prioritize examples near the learner’s ability, while problem-generation methods (Luo et al., 2025; Dai et al., 2026; Sundaram et al., 2026; Zhao et al., 2025) expand the supply of training problems offline or through self-play. Frontier learning instead uses procedural generators online to explore task-specific difficulty levels, estimates level-wise regret, and mutates, explores new levels to continuously track the model’s moving capability frontier. Prior work in unsupervised environment design (Dennis et al., 2020; Jiang et al., 2021; Parker-Holder et al., 2022; Campero et al., 2021) has explored training in online procedural generation environments, including navigation and control tasks. In contrast, our work focuses on post-training of LLMs on reasoning tasks. We discuss related work in detail in Appendix A. Contributions: Our main contributions are as follows. (i) We formalize frontier learning, an open-ended RL post-training approach that uses a procedural generator online to adapt the training curriculum to the model’s evolving capability. It treats the generator’s task-specific attributes as a multidimensional space of levels, without assuming that they induce a known or monotonic difficulty ordering. (i) We design three online mechanisms that together keep the training distribution at the capability frontier: a dynamic level buffer that maintains a set of frontier levels, a regret-based priority signal that identifies and prioritizes frontier levels using only the GRPO rollouts already collected, and random exploration with state-dependent mutation that continually scouts for new frontier levels and admits them to the buffer. (i) We instantiate frontier learning within GRPO and show that it outperforms fixed-pool, domain-randomization and adaptive-sampling baselines across five puzzle, math and graph reasoning tasks and three model families. The gains are largest over long training horizons, where frontier learning keeps improving after the baselines plateau (71.8 vs 33.3 on Dice), and ablations show that regret-guided mutation drives this expansion. 2 Background We address the challenge of post-training a large language model (LLM), denoted by πθ _θ, on a given reasoning task. This typically entails finetuning the LLM using Reinforcement Learning with Verifiable Rewards (RLVR) based on a fixed offline dataset D, comprising problems specific to the given reasoning task, and a reward function r(⋅,⋅)r(·,·). The problems admit verifiable correct answers, and the reward function r is designed to evaluate the correctness of a response y for any problem x∈x . The objective in RLVR is to optimize the expected reward attained by policy πθ _θ on problems sampled from D. Concretely, we define it as J(θ)=x∼,y∼πθ(⋅|x)[r(x,y)]J(θ)=E_x ,y _θ(·|x)[r(x,y)]. Standard RLVR pipelines optimize J(θ)J(θ) via Group Relative Policy Optimization (GRPO) (Shao et al., 2024; Guo et al., 2025), which constructs advantages from within-group reward comparisons, and avoids training a separate value network to assign the same (Schulman et al., 2017). Specifically, for a given problem x, the policy πθ _θ generates n individual responses y1,…,yn\y_1,…,y_n\. Then, the advantage of response i given the rewards ri=r(x,yi)i=1n\r_i=r(x,y_i)\_i=1^n is calculated as follows: Ai=ri−μσr+ϵ,μ=1n∑iri,σr=MAD(ri)=1n∑i|ri−μ|.A_i= r_i-μ _r+ε, μ= 1n _ir_i, _r=MAD(\r_i\)= 1n _i|r_i-μ|. (1) Here, following Dai et al. (2026), we use the mean absolute deviation (MAD) in the denominator rather than standard deviation. This is to ensure the update magnitude of a single problem, determined by the sum of absolute advantages 1n∑i|Ai| 1n _i|A_i|, remains constant. This avoids disadvantaging problems at the capability frontier (all but one rollout wrong), whose absolute advantages are low when defined using the standard deviation (see Dai et al. (2026, Theorems-1,2)). Given the assigned advantages, GRPO optimizes the following clipped surrogate objective: JGRPO(θ)=x∼,yi∼πθ(⋅|x)[1n∑i=1n1|yi|∑t=1|yi|(min(ri,tA^i,t,clip(ri,t,1−ε,1+ε)A^i,t))].J_GRPO(θ)=E_x ,y_i _θ(·|x) [ 1n _i=1^n 1|y_i| _t=1^|y_i| ( (r_i,t A_i,t,\,clip(r_i,t,1- ,1+ ) A_i,t ) ) ]. (2) Despite the success of standard RLVR pipelines to improve the reasoning capabilities of LLMs, they still suffer from the following limitation when built upon a fixed dataset. Gradient collapse: A well-documented failure mode of GRPO with fixed training data (Foster et al., 2025; Bae et al., 2025; Yu et al., 2025; Ramesh et al., 2025) is the zero gradient phenomenon wherein rollouts to a problem are all correct or all wrong. Proposed mitigations filter or resample within a fixed problem pool to increase the proportion of non-zero gradient problems in the training batch (Li et al., 2025b; Chen et al., 2025b; Wang et al., 2025; Zhou et al., 2025; Panaganti et al., 2026; Gu et al., 2026). Such strategies assume informative problems already exist in the data pool and merely need to be identified from the data. The issues with such an assumption are twofold: (i) The predetermined static data might contain very low or no informative training problems, as the problems might be too easy or too hard for the given policy. (i) Even if the dataset is designed to be informative for the policy at its initial state, as training progresses, the policy’s capability improves. And once its capability reaches a certain threshold, the trained policy drifts beyond the utility of the static dataset for GRPO training. Specifically, it would have already mastered all problems in the dataset, leading to continual zero-gradient exposure and no further learning progress. This exposes a fundamental bottleneck in GRPO post-training and points to a structural gap in standard RLVR pipelines: the training data must adapt online to the model’s evolving capability. 3 Frontier Learning In this section, we propose and formalize frontier learning, an open-ended post-training approach that allows continuous adaptation of the training data by sampling appropriate problems from procedural data generators. We begin by discussing the notion of procedural data generators upon which this approach is built. 3.1 Procedural Data Generation Structured reasoning domains like logic, math, puzzles, etc., consist of tasks such as countdown, ARC, geometry (Stojanovski et al., 2025) that can be generated and verified in a rule-based or programmatic manner. In such tasks, each problem is produced by an underlying procedure specified by explicit rules that samples the terms, constraints, or values defining the problem. Such procedurally generated data has been increasingly incorporated in LLM post-training pipelines, especially to improve their general reasoning ability. Given a procedural data generator, one can specify certain task-specific attributes used by the generator to construct problems of a certain reasoning task for training. In particular, we define a certain attribute combination as a level ℓ∈ℒ , where ℒL is the set of all such attribute combinations, which can be intuitively regarded as a structured difficulty class from which arbitrarily many problems can be drawn. As a concrete example, consider the Countdown arithmetic task. A problem consists of a set of input numbers and a target number and the task is to produce an arithmetic expression that uses the input numbers to obtain the target. Here, the level l will specify the number of inputs, the range of those inputs, the range of the target, and the allowed arithmetic operations. For example, one configuration may generate puzzles with four operands sampled from [1,10][1,10] and targets in [20,100][20,100], while another may generate puzzles with six operands sampled from [1,100][1,100] and targets in [100,1000][100,1000]. Sampling x∼lx l then produces one concrete puzzle from the corresponding level. Appendix B provides an additional visual example for Sokoban. Although prior works incorporate data generated by such procedural generators into their pipelines, they primarily use their generative capacity only offline. They draw batches from a fixed dataset, presampled from a distribution over ℒL specified prior to training. This distribution reflects the practitioner’s assumptions about the model’s capability but still leaves both the levels and the generated problems fixed for the entire run. As noted in Section 2, such a dataset faces the same bottleneck as that of a fixed human-curated dataset for GRPO post-training, as it can only capture the informative levels in ℒL according to the policy’s initial capability and might fail to adapt online to the current policy’s capability. 3.2 Objective for Open-ended Post-training We consider the open-ended post-training framework, in which the objective is to continuously stream informative samples, w.r.t. the current policy, into the training distribution using data generated by a procedural generator. This entails actively navigating and exploring the set of levels ℒL, corresponding to the given procedural generator, to track the policy’s evolving capability frontier. Here, the capability frontier is defined as the band of levels that comprises informative problems and contributes to learning progress. This is especially challenging, as this band of informative levels is a thin, moving slice within ℒL during training. The vast majority of levels are either too easy or too difficult. Hence, sampling uniformly from the set of all levels ℒL and filtering would be highly sub-optimal. Hence, an effective approach for this framework must choose which levels to sample from at each step so as to maximize the capability of the final policy. We measure the performance of an approach tackling this framework by the following objective, evaluated at the end of T training steps: J(θ)=ℓ∼ℒ,x∼ℓ,y∼πθ(⋅∣x)[r(x,y)].J(θ)=E_ ,\;x ,\;y _θ(· x)\! [r(x,y) ]. (3) Here, the objective measures the expected performance of the trained policy on instances drawn uniformly from levels ℓ∈ℒ . In practice, we estimate J(θ)J(θ) using a held-out set of levels unseen during training, thereby also assessing the agent’s ability to generalize. In order to effectively perform in the open-ended post-training framework and maximize Equation 3, we delineate three core requirements any approach should satisfy. First, the approach must identify the capability frontier. Second, it must track this frontier over time, maintaining training pressure on levels that are currently near the edge of this frontier and deprioritizing those that have become consistently solved. Third, it must continuously expand the training distribution, by introducing new levels into the distribution that are not yet present but may lie beyond the current frontier. 3.3 Components of Frontier Learning Next, we detail our proposed frontier learning approach, aimed to tackle the open-ended post-training framework discussed in Section 3.2. Frontier learning addresses the core requirements detailed in Section 3.2 through three tightly coupled online mechanisms operating on a dynamic level buffer ℬB, a fixed-capacity set of levels continuously replenished and pruned throughout training. In particular, it uses a regret-based priority signal to identify levels with remaining learning potential and concentrates sampling on the current frontier. Moreover, it conducts exploration to continuously scout ℒL for informative levels that exist beyond the buffer. Finally, it also performs state-dependent mutation to perturb the attributes of frontier levels and incorporate levels at incrementally harder difficulties into the level buffer ℬB. We summarize our approach in Figure 2. The following subsections define each mechanism in turn. Figure 2: One iteration of frontier learning with GRPO. At each step, the algorithm initially constructs the level batch with a mix of exploratory levels and high-regret frontier levels sampled from the buffer. A level sampled from the buffer is locally mutated with probability pmut(ℓ)p_mut( ), by perturbing its attributes in order to evaluate and potentially extend the policy’s capability frontier. The resulting level batch is then passed through the generator to construct the training batch of problems for the GRPO update. The resulting rewards and regret estimates are then used for future level selection. 3.3.1 Regret and Priority We define regret at two levels of granularity: per problem x and per level ℓ . Problem-wise regret: For a problem x drawn from level ℓ , the policy generates n rollouts with s successes. Two quantities characterise the policy’s behaviour on x: the solvability indicator [s≥1]1[s≥ 1], which is 11 if the policy solves x at all and 00 otherwise, and the empirical success rate s(x)=s/ns(x)=s/n. The problem-level regret is their difference: ρ(x)=[s≥1]−s(x).ρ(x)=1[s≥ 1]-s(x). (4) Regret is zero for impossible problems (s=0s=0, no gradient signal exists) and for fully mastered ones (s(x)=1s(x)=1, solvability and success rate coincide), and is strictly positive only in the productive regime where gradient signal flows. Level-wise regret: The regret of a level ℓ is the expected problem-wise regret over problems drawn from its generator: ℛ(ℓ)=x∼ℓ[ρ(x)]=x∼ℓ[[s≥1]−s(x)].R( )=E_x [ρ(x) ]=E_x \! [1[s≥ 1]-s(x) ]. (5) Crucially, a level accumulates high regret if and only if many of its problems are solvable but not yet reliably solved, i.e., many problems lie in the informative band. A level where most problems are either impossible or already mastered yields low ℛ(ℓ)R( ) regardless of a few outliers, correctly reflecting limited learning potential. In practice, we estimate ℛ(ℓ)R( ) by tracking the moving window of problems lW_l sampled from ℓ : ρ^(ℓ)=1|ℓ|∑x∈ℓρ(x). ρ( )= 1|W_ | _x _ ρ(x). (6) Here, ρ(x)ρ(x) is computed directly from the n rollouts collected on x during the rollout phase, and hence no additional rollouts are required. As the policy improves across problems from ℓ , ρ^(ℓ) ρ( ) falls automatically, and the buffer starts prioritizing harder levels that have newly entered the informative band. Priority score: Each level is assigned a priority score that adds a staleness correction to regret: P(ℓ,t)=ℛ(ℓ)+λs⋅(t−tℓ),P( ,t)=R( )+ _s·(t-t_ ), (7) where tℓt_ is the last step at which level ℓ was sampled. The staleness term ensures all levels in the buffer are periodically revisited and helps detect changes in the policy’s capability on dormant levels. 3.3.2 Regret-Guided Sampling with Exploration At each training step t, a batch of nℓn_ levels is assembled as follows: All nℓn_ slots are initially drawn from the dynamic level buffer ℬB by a Zipfian distribution over priority scores P(ℓ,t)P( ,t), concentrating training on the current capability frontier while preserving diversity across the buffer. Each slot is then independently replaced with probability ϕφ with a random level sampled uniformly from ℒL. This ensures that the expected fraction of exploratory levels in the current training batch is ϕφ. Moreover, this helps the training batch to stay anchored at the productive frontier, while also exploring continuously for more difficult levels beyond the current buffer to avoid stagnation. 3.3.3 State-Dependent Mutation It is important to expand the buffer with frontier levels at incrementally harder difficulties so that it remains continually informative. However, neither regret nor exploration systematically target the boundary of the model’s current capability. Mutation fills this gap by applying a small, undirected perturbation to existing buffer levels to produce new levels in their local neighbourhood. At every step t, each level ℓ that was sampled from the buffer and not replaced with a random level due to exploration is mutated with a probability that depends on its current difficulty class defined w.r.t. the average empirical success rate of its problems s^(ℓ)=1||∑x∈ℓs(x) s( )= 1|W| _x _ s(x) in the last W : pmut(ℓ)=pinfs^(ℓ)∈(0.05,0.95) (informative)peasys^(ℓ)>0.95 (too easy)phards^(ℓ)<0.05 (too difficult)p_mut( )= casesp_inf&$ s( )∈(0.05,0.95)$ (informative)\\ p_easy&$ s( )>0.95$ (too easy)\\ p_hard&$ s( )<0.05$ (too difficult) cases (8) with pinf>peasy≥phardp_inf>p_easy≥ p_hard. This concentrates mutation effort on informative levels, while still allowing easy levels to potentially produce new harder variants and enabling impossible levels to produce easier variants in a limited manner so as not to populate the buffer with a lot of impossible levels. The combined effect of state-dependent mutation probabilities is that the buffer evolves to track the model’s improving capability as informative frontier levels are continuously allowed to seed variants at incrementally higher difficulties. 4 Algorithm In this section, we present an instantiation of our frontier learning approach with GRPO in Algorithm 1. In particular, Algorithm 1 comprises of three tightly coupled operations at each step, namely, assembling a regret-guided training batch using a procedural data generator, updating the policy via GRPO, and refreshing the regret state using the newly observed returns (illustrated in Figure 2). We describe the main components of Algorithm 1 below. Algorithm 1 Frontier Learning with GRPO 1: Level space ℒL, reward r, policy πθ _θ, buffer capacity BmaxB_ , initial regret ℛ0R_0, explore fraction ϕφ, staleness coeff. λs _s, mutation probs. pinf,peasy,phardp_inf,p_easy,p_hard 2: Initialise ℬB by grid seeding ℒL; set ℛ(ℓ)←ℛ0R( )\!←\!R_0 and P(ℓ,0)←ℛ(ℓ)P( ,0)\!←\!R( ) for all ℓ∈ℬ 3: for t=1,…,Tt=1,…,T do 4: Sample nℓn_ levels from ℬB by Zipfian sampling over priority scores P(ℓ,t)P( ,t) 5: ℒnew←∅L_new← 6: for each sampled ℓ do 7: With prob. ϕφ, draw ℓ′∼ℒ ; if ℓ′∉ℬ , admit ℓ′ , add it to ℒnewL_new, replace ℓ←ℓ′ ← ⊳ Explore 8: end for 9: for each sampled ℓ∉ℒnew _new do 10: With prob. pmut(ℓ)p_mut( ), perturb ℓ→ℓ′ → ; if ℓ′∉ℬ , admit ℓ′ , replace ℓ←ℓ′ ← ⊳ Mutate 11: end for 12: for each sampled ℓ do 13: Draw npn_p problems with nrn_r rollouts each: x∼ℓx , yi∼πθ(⋅∣x)i=1nr\y_i _θ(· x)\_i=1^n_r. 14: Compute ri=r(x,yi)r_i=r(x,y_i) 15: end for 16: Update θ with GRPO on (x,yi,ri)\(x,y_i,r_i)\ ⊳ Eq. 1 17: for each sampled ℓ do 18: Append rewards to ℓ ’s rolling window; update ℛ(ℓ)R( ) and P(ℓ,t)P( ,t) ⊳ Eqs. 4–7 19: end for 20: end for 21: return πθ _θ Grid seeding initialisation (line 1): The buffer is populated by grid seeding, wherein ℒL is partitioned along each attribute axis into an equal number of cells and one level is sampled uniformly from each cell. This guarantees broad coverage of the difficulty spectrum from the very first step, preventing the cold-start collapse that occurs when random seeding places all initial levels in a single difficulty band. Batch assembly (lines 3–10): The batch is assembled from nℓn_ replay levels drawn from ℬB by Zipfian sampling over priority scores P(ℓ,t)P( ,t) (see Equation 7). Then, each sampled level is independently considered for exploration: with probability ϕφ, we draw a candidate ℓ′∼Uniform(ℒ) (L), check whether ℓ′∉ℬ , and, if novel, admitted to the buffer ℬB and used in place of the original replay level. Then, each non-replaced replay level is offered to the mutator: with probability pmut(ℓ)p_mut( ) (Equation 8), a single attribute is perturbed to produce a candidate child level l′l . If l′l is novel, it is admitted to the buffer ℬB and used in place of the original replay level. The constructed batch is then passed on to the procedural generator. When |B|=Bmax|B|=B_ , admitting a new level evicts the level with the lowest priority score P(ℓ,t)P( ,t). Rollout and policy update (lines 11–15): For each active level ℓ in the batch, npn_p problems x∼ℓx are generated using the procedural generator and n rollouts are sampled from the current policy for each problem x. The full batch is then used to update θ via GRPO (Equation 1). Buffer update (lines 16–18): After the policy update, returns from every trained level are appended to its rolling window and regret ℛ(ℓ)R( ) and priority P(ℓ,t)P( ,t) are recomputed from Equations 5 and 7. Hence, the buffer’s priority scores constantly reflects the updated policy’s capability frontier, and levels with low regret are immediately deprioritised and the priority shifts towards the new frontier levels with high regret at the next batch assembly. 5 Experiments We evaluate our proposed frontier learning with GRPO (Algorithm 1) on five procedurally generated reasoning tasks drawn from Reasoning Gym (Stojanovski et al., 2025), each parameterised in a structured level space ℒL, spanning three domains. In the puzzle domain, Countdown requires combining numbers via basic arithmetic to reach a target (difficulty grows with operands count, their magnitude, and the target range), and Sokoban requires planning box-pushing moves on a grid (harder problems have larger grids, more boxes, and deeper solution traces). Concerning the math domain, we include Dice, a discrete probability task in which the model must compute the exact probability of a target outcome when rolling a set of multi-sided dice (where difficulty is controlled by the number of dice and their face count), and Decimal Arithmetic, which consists of fixed-precision multi-term decimal calculations, scaling in difficulty with term count, decimal places, and required precision. Finally, in the graph domain, Largest Island requires the model to identify the largest connected component of 1-valued cells in a binary grid and report its area. Difficulty scales with grid dimensions and the number of islands, requiring the model to trace connected components over increasingly large graphs. Models and training protocol: We experiment with two model variants from the Qwen3 family (Yang et al., 2025), chosen to explore the effectiveness of frontier learning across two different starting capability regimes. The pre-trained only Qwen3-4B-Base is used for Countdown, Dice, and Decimal Arithmetic. For Sokoban, we use the instruction-tuned Qwen3-4B with thinking mode enabled, since such puzzles require more sustained multi-step reasoning and therefore benefit from a larger reasoning budget. To assess whether the effectiveness of frontier learning generalizes across model families and model sizes, we additionally conduct experiments on Countdown using Llama-3.2-3B-Instruct (Grattafiori et al., 2024) and on Largest Island using Olmo3-7B-Instruct (Olmo et al., 2025). For Countdown, Sokoban, and Decimal Arithmetic, models are trained for T=200T=200 GRPO steps, while for Dice and Largest Island we extend the training horizon to T=500T=500 GRPO steps, in order to analyze how our method behaves under prolonged training. Full implementation details and hyperparameters are reported in Appendix D, while a detailed accounting of the used compute is provided in Appendix H. Baselines: We compare frontier learning against four representative alternatives for selecting training levels from procedural generators. Three of them rely on a fixed level buffer seeded once before training via grid seeding, introducing no new levels during training, and hence isolating the effect of having a fixed pool of levels in contrast to growing it. In particular, Uniform samples uniformly from the seeded buffer without any priority, providing a clean upper bound on what structured initial coverage alone can achieve. Both SEC (Chen et al., 2025b) and PLR (Jiang et al., 2021) attempt instead to recreate a curriculum within the fixed initial buffer by assigning priority scores to levels and sampling more frequently from those estimated to be most productive. SEC estimates productivity by using the absolute GRPO advantage estimated from previous rollouts, while PLR uses the same level-wise regret signal ℛ(ℓ)R( ) as frontier learning (Eq. 5). On the other hand, Domain Randomization (DR) samples fresh levels at every step from the same coverage-aware seeding grid used to initialize the other baselines, with no memory or replay, thereby testing unguided exploration of the level space. Evaluation: For each task, we pre-define a fixed set of evaluation levels ℓv∼ℒ _v , each associated with a total of 200 unique problems xvalx_val. Each level’s configuration is excluded from grid initialization, uniform exploration, and mutation admission. Thus, neither the evaluation instances nor their exact generator configurations are encountered during training. Levels are ordered by a task-specific difficulty heuristic derived from their configuration attributes, yielding a spectrum that we partition into three bins (Easy, Medium, Hard); full definitions and the bin partition are given in Appendix C. The primary metric is accuracy: the mean greedy-decoding accuracy across all evaluation levels, averaged over four seeds. 5.1 Results Performance under the standard training budget: We first evaluate frontier learning on Countdown, Decimal Arithmetic, and Sokoban using the standard budget of 200 GRPO steps. As shown in Table 1, frontier learning achieves the highest average final performance on all three tasks. Table 1: Accuracy (%) on the fixed anchor evaluation set after 200 training steps. Best result per task in bold; second-best underlined. Puzzle Math Task Countdown Sokoban Dec. Arith. Model Qwen3-4B-Base Qwen3-4B Qwen3-4B-Base Method Domain Rand. 44.7±1.7 37.6±2.6 31.0±2.8 Uniform 44.2±4.0 39.5±1.3 28.0±2.6 SEC 40.1±8.1 34.3±6.6 29.1±4.1 PLR 44.3±2.9 41.1±3.9 27.5±2.9 Frontier Learning (ours) 50.9±0.5 45.2±1.9 34.7±3.6 Relative gain vs 2nd (%) +13.9 +10.0 +11.9 The clearest improvement is observed on Countdown, where frontier learning reaches 50.9±0.550.9± 0.5% accuracy, compared with 44.7±1.744.7± 1.7 for the strongest baseline, while the overall gains are smaller on Sokoban and Decimal Arithmetic. The per-difficulty results, shown in Figure 7, provide a more detailed view of these differences. On Sokoban, frontier learning’s largest gain is concentrated on easier configurations that are not yet fully mastered, while more advanced levels remain largely unproductive. On Decimal Arithmetic, the improvement is concentrated on Easy and Medium levels, whereas all methods remain effectively tied on Hard configurations. These results show that frontier learning successfully identifies where the model has the greatest remaining room for improvement and concentrates training on those configurations. Appendix E reports the full learning curves for all methods on these tasks. Table 2: Accuracy (%) on Dice for the baselines, Frontier Learning and its components ablation. Best result in bold; second-best within each block underlined. Method Accuracy Domain Rand. 23.4±4.7 Uniform 31.9±1.5 SEC 33.3±1.2 PLR 31.2±1.9 ACE-GRPO 33.2±2.1 DAPO 25.4±5.9 Frontier Learning (ours) 71.8±6.2 Component ablations Unif+Explore 30.8±1.3 Unif+FlatMut 47.1±10.8 Unif+Explore+FlatMut 36.9±2.9 PLR+Explore 33.5±1.8 PLR+Mut 62.5±4.5 Relative gain vs best baseline (%) +115.6 Long-horizon frontier expansion on Dice: We next extend training on Dice from 200 to 500 GRPO steps to test whether frontier learning continues to track the model’s improving capabilities over a longer horizon. In addition to the baselines introduced above, we include DAPO-style dynamic-filtering baseline (Yu et al., 2025) and an ACE-GRPO sampling baseline (Cai et al., 2026). DAPO discards rollout groups whose rewards are uniformly correct or uniformly incorrect. Because this filtering can consume multiple rollout groups for each optimizer update, we match DAPO to the total rollout budget of the other methods, corresponding to 256k rollouts, rather than to 500 optimizer updates. ACE-GRPO instead prioritizes replay-buffer levels using its Learnability Potential score. Figure 8 shows that the standard baselines improve during the first approximately 100–200 steps and subsequently plateau. Frontier learning instead continues improving throughout the longer training horizon, reaching 71.8±6.271.8± 6.2% accuracy after 500 steps. The strongest standard baseline, SEC, reaches 33.3±1.233.3± 1.2, while ACE-GRPO reaches 33.2±2.133.2± 2.1 and the rollout-matched DAPO baseline reaches 25.4±5.925.4± 5.9. Additionally, Figure 10 separates performance by difficulty. Frontier learning reaches near-perfect performance on Easy and Medium configurations while making substantial progress on regions that remain almost entirely unsolved by the baselines. In particular, it reaches 72.6%72.6\% on Hard levels and 20.7%20.7\% on Extra-hard levels while all the standard baselines remain under 5.5%5.5\% on such categories. The long-horizon improvement is therefore not limited to further optimizing already-mastered configurations, but reflects continued expansion of the model’s capability frontier. Appendix F provides the per-difficulty learning curves, showing how performance across the individual difficulty bins evolves throughout training. We reuse this same experimental setup to run a sweep across step sizes for two baselines, reported in Appendix G.2. Component ablation: We ablate buffer growth, random exploration, mutation, and regret-based prioritization under the same 500-step Dice protocol. All ablations use a dynamic buffer. PLR+Explore removes mutation, PLR+Mut removes random exploration, the Uniform variants remove regret-based prioritization, and FlatMut replaces state-dependent mutation with a level-independent probability. Table 2 shows that buffer growth alone is insufficient: PLR+Explore reaches only 33.5±1.833.5± 1.8. By contrast, PLR+Mut reaches 62.5±4.562.5± 4.5, showing that mutation-driven expansion is key to moving beyond the seeded difficulty range. Mutation alone is also insufficient: Unif+FlatMut reaches 47.1±10.847.1± 10.8, substantially below PLR+Mut, indicating that regret-guided prioritization makes mutation more effective by focusing replay and subsequent mutations on informative levels. The full method reaches 71.8±6.271.8± 6.2, with random exploration providing a further gain, especially on Hard and Extra-hard levels. Table 3: Accuracy (%) on Countdown and Largest Island. Best result per task in bold; second-best underlined. Task Countdown Largest Island Model Llama-3.2-3B-Instruct Olmo3-7B-Instruct Method Domain Rand. 33.8±1.9 45.2±1.8 Uniform 25.3±4.7 45.3±3.4 SEC 29.5±1.6 48.3±4.7 PLR 29.0±1.8 52.0±5.3 ACE-GRPO 31.4±1.9 45.3±5.5 Frontier learning (ours) 39.2±1.3 73.7±3.3 Relative gain vs 2nd (%) +16.0 +41.7 Generalization across model families and sizes: To assess whether frontier learning remains effective outside the Qwen3 family and the 4B model size, we train Llama-3.2-3B-Instruct on Countdown under the standard 200-step budget and Olmo3-7B-Instruct on Largest Island under the extended 500-step budget. Table 3 shows that frontier learning attains the best accuracy in both settings. The per-difficulty learning curves for Largest Island (Figure 3) mirror the Dice results shown in Figure 9: all methods saturate on Easy levels, but on Medium and Hard levels the baselines plateau after roughly 200 steps while frontier learning continues to improve until the end of training, reaching 84.0%84.0\% and 47.2%47.2\% respectively against at most 54.0%54.0\% and 18.2%18.2\% for any baseline. These results indicate that the gains of frontier learning are not tied to a particular model family, scale, or task domain. We additionally use the Llama Countdown setting for the hyperparameter sensitivity analysis in Appendix G.1. Figure 3: Per-difficulty accuracy (%) on Largest Island with Olmo3-7B-Instruct over 500 GRPO steps. All methods saturate on Easy levels, but only frontier learning keeps improving on Medium and Hard levels throughout training, while the baselines plateau after roughly 200 steps. 6 Conclusions We formalize frontier learning, a novel open-ended RL post-training approach that adapts the training curricula for finetuning LLMs on reasoning tasks using a procedural generator. We design three online mechanisms for our approach, namely, a dynamic level buffer to maintain a set of frontier levels, a regret-based priority signal to identify and prioritize the frontier levels and exploration, mutation to constantly scout and incorporate new frontier levels into the buffer. Together, they ensure the training distribution is constantly adapted online to the model’s evolving capability. We instantiate our approach within the GRPO objective and empirically demonstrate that our approach consistently outperforms baselines on several puzzle, math and graph tasks. Our approach is limited to a procedural generator equipped with continuous or discrete task-specific parameters. Extending frontier learning to tasks where problem instances cannot be procedurally generated remains an interesting avenue for future work. References Bae et al. (2025) S. Bae, J. Hong, M. Y. Lee, H. Kim, J. Nam, and D. Kwak Online difficulty filtering for reasoning oriented reinforcement learning. arXiv preprint arXiv:2504.03380. Cited by: Appendix A, §1, §1, §2. Bao et al. (2026) L. Bao, J. Wang, Y. Zhang, Y. Zheng, and R. Paturi Question begets question: self-evolving curriculum for reinforcement fine-tuning on competition mathematics. External Links: 2608.01522, Link Cited by: Appendix A. Bengio et al. (2009) Y. Bengio, J. Louradour, R. Collobert, and J. Weston Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, p. 41–48. Cited by: §1. Cai et al. (2026) Y. Cai, Z. Liu, X. Zhu, C. Wang, and S. Chen AceGRPO: adaptive curriculum enhanced group relative policy optimization for autonomous machine learning engineering. External Links: 2602.07906, Link Cited by: Appendix A, §1, §5.1. Campero et al. (2021) A. Campero, R. Raileanu, H. Küttler, J. B. Tenenbaum, T. Rocktäschel, and E. Grefenstette Learning with AMIGo: adversarially motivated intrinsic goals. In International Conference on Learning Representations, Cited by: §1. Chen et al. (2025a) J. Chen, Q. He, S. Yuan, A. Chen, Z. Cai, W. Dai, H. Yu, Q. Yu, X. Li, J. Chen, H. Zhou, and M. Wang Enigmata: scaling logical reasoning in large language models with synthetic verifiable puzzles. arXiv preprint arXiv:2505.19914. Cited by: §1. Chen et al. (2025b) X. Chen, J. Lu, M. Kim, D. Zhang, J. Tang, A. Piché, N. Gontier, Y. Bengio, and E. Kamalloo Self-evolving curriculum for llm reasoning. arXiv preprint arXiv:2505.14970. Cited by: Appendix A, Appendix A, §2, §5. Dai et al. (2026) Y. Dai, Y. Ji, X. Zhang, Y. Wang, X. Chu, and Z. Lu Harder is better: boosting mathematical reasoning via difficulty-aware GRPO and multi-aspect question reformulation. arXiv preprint arXiv:2601.20614. Cited by: Appendix A, §1, §2. Dennis et al. (2020) M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine Emergent complexity and zero-shot transfer via unsupervised environment design. In Advances in Neural Information Processing Systems, Vol. 33. Cited by: Appendix A, §1. Foster et al. (2025) T. Foster, A. Sims, J. Forkel, M. Fellows, and J. Foerster Learning to reason at the frontier of learnability. arXiv preprint arXiv:2502.12272. Cited by: Appendix A, Appendix A, §1, §1, §2. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma The llama 3 herd of models. External Links: 2407.21783, Link Cited by: §5. Graves et al. (2017) A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu Automated curriculum learning for neural networks. In Proceedings of the 34th International Conference on Machine Learning, p. 1311–1320. Cited by: Appendix A, §1. Gu et al. (2026) Z. Gu, J. Light, R. Astudillo, Z. Ye, L. He, H. P. Zou, W. Cheng, S. Paternain, P. S. Yu, and Y. Yue Actor-curator: co-adaptive curriculum learning via policy-improvement bandits for RL post-training. arXiv preprint arXiv:2602.20532. Cited by: Appendix A, §1, §2. Guan et al. (2025) X. Guan, L. L. Zhang, Y. Liu, N. Shang, Y. Sun, Y. Zhu, F. Yang, and M. Yang RStar-math: small llms can master math reasoning with self-evolved deep thinking. External Links: 2501.04519, Link Cited by: Appendix A. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Ding, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Chen, J. Yuan, J. Tu, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. You, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Zhou, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), p. 633–638. External Links: ISSN 1476-4687, Link, Document Cited by: §1, §1, §1, §2. Hosseini et al. (2024) A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal V-star: training verifiers for self-taught reasoners. External Links: 2402.06457, Link Cited by: Appendix A. Hu et al. (2025) J. Hu, J. K. Liu, H. Xu, and W. Shen REINFORCE++: stabilizing critic-free policy optimization with global advantage normalization. arXiv preprint arXiv:2501.03262. Cited by: §1. Huang et al. (2026) C. Huang, W. Yu, X. Wang, H. Zhang, Z. Li, R. Li, J. Huang, H. Mi, and D. Yu R-zero: self-evolving reasoning llm from zero data. External Links: 2508.05004, Link Cited by: Appendix A. Jiang et al. (2021) M. Jiang, E. Grefenstette, and T. Rocktäschel Prioritized level replay. In International Conference on Machine Learning, p. 4940–4950. Cited by: Appendix A, §1, §5. Lacombe et al. (2026) V. Lacombe, V. Quesnel, and D. Sileo Reasoning core: a scalable procedural data generation suite for symbolic pre-training and post-training. External Links: 2603.02208, Link Cited by: Appendix A. Lambert et al. (2025) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi Tulu 3: pushing frontiers in open language model post-training. External Links: 2411.15124, Link Cited by: §1. Le et al. (2026) T. V. Le, M. Jeon, K. Vu, V. Lai, and E. Yang No prompt left behind: exploiting zero-variance prompts in LLM reinforcement learning via entropy-guided advantage shaping. In The Fourteenth International Conference on Learning Representations, Cited by: Appendix A, §1. Li et al. (2025a) R. Li, H. Huang, F. Wei, F. Xiong, Y. Wang, and X. Chu AdaCuRL: adaptive curriculum reinforcement learning with invalid sample mitigation and historical revisiting. External Links: 2511.09478, Link Cited by: Appendix A, §1. Li et al. (2025b) Z. Li, C. Chen, T. Yang, T. Ding, R. Sun, G. Zhang, W. Huang, and Z. Luo Knapsack rl: unlocking exploration of llms via optimizing budget allocation. arXiv preprint arXiv:2509.25849. Cited by: Appendix A, §1, §2. Lightman et al. (2023) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In The twelfth international conference on learning representations, Cited by: §1. Liu et al. (2026) B. Liu, S. Yu, Y. Jiang, A. Qu, A. Zhao, Z. Liu, J. Kim, Z. Zhou, S. Kim, T. Ren, M. Liu, H. Yu, Z. Chen, W. Shi, P. P. Liang, L. Zettlemoyer, Y. Choi, and N. Jaques SPADE: self-play in adaptive synthetic executable environments. External Links: 2608.19197, Link Cited by: Appendix A. Liu et al. (2025a) J. Liu, Y. Fan, Z. Jiang, H. Ding, Y. Hu, C. Zhang, Y. Shi, S. Weng, A. Chen, S. Chen, Y. Huang, M. Zhang, P. Zhao, J. Yan, and J. He SynLogic: synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond. External Links: 2505.19641, Link Cited by: Appendix A. Liu et al. (2025b) M. Liu, S. Diao, X. Lu, J. Hu, X. Dong, Y. Choi, J. Kautz, and Y. Dong ProRL: prolonged reinforcement learning expands reasoning boundaries in large language models. External Links: 2505.24864, Link Cited by: Table 5. Luo et al. (2025) H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, Y. Tang, and D. Zhang WizardMath: empowering mathematical reasoning for large language models via reinforced evol-instruct. External Links: 2308.09583, Link Cited by: Appendix A, §1. Matiisen et al. (2017) T. Matiisen, A. Oliver, T. Cohen, and J. Schulman Teacher-student curriculum learning. arXiv preprint arXiv:1707.00183. Cited by: Appendix A, §1. Olmo et al. (2025) T. Olmo, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. J. V. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. L. Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi Olmo 3. External Links: 2512.13961, Link Cited by: §5. Oudeyer et al. (2007) P. Oudeyer, F. Kaplan, and V. V. Hafner Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation 11 (2), p. 265–286. Cited by: Appendix A, §1. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. External Links: 2203.02155, Link Cited by: §1. Panaganti et al. (2026) K. Panaganti, Z. Liang, W. Yu, H. Mi, and D. Yu Group distributionally robust optimization-driven reinforcement learning for LLM reasoning. arXiv preprint arXiv:2601.19280. Cited by: Appendix A, §2. Parker-Holder et al. (2022) J. Parker-Holder, M. Jiang, M. Dennis, M. Samvelyan, J. Foerster, E. Grefenstette, and T. Rocktäschel Evolving curricula with regret-based environment design. In Proceedings of the 39th International Conference on Machine Learning, Cited by: Appendix A, §1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn Direct preference optimization: your language model is secretly a reward model. arXiv preprint arXiv:2305.18290. Cited by: §1. Ramesh et al. (2025) S. S. Ramesh, X. Ji, M. Zimmer, S. Yoon, Z. Wang, H. Bou Ammar, A. Lucchi, and I. Bogunovic Multi-task GRPO: reliable LLM reasoning across tasks. arXiv preprint arXiv:2602.05547. Cited by: Appendix A, §2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §1, §1, §2. Stojanovski et al. (2025) Z. Stojanovski, O. Stanley, J. Sharratt, R. Jones, A. Adefioye, J. Kaddour, and A. Köpf REASONING gym: reasoning environments for reinforcement learning with verifiable rewards. arXiv preprint arXiv:2505.24760. External Links: Link Cited by: Appendix A, §1, §3.1, §5. Sundaram et al. (2026) S. Sundaram, J. Quan, A. Kwiatkowski, K. Ahuja, Y. Ollivier, and J. Kempe Teaching models to teach themselves: reasoning at the edge of learnability. arXiv preprint arXiv:2601.18778. Cited by: Appendix A, §1. Team et al. (2025) K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, C. Tang, C. Wang, D. Zhang, E. Yuan, E. Lu, F. Tang, F. Sung, G. Wei, G. Lai, H. Guo, H. Zhu, H. Ding, H. Hu, H. Yang, H. Zhang, H. Yao, H. Zhao, H. Lu, H. Li, H. Yu, H. Gao, H. Zheng, H. Yuan, J. Chen, J. Guo, J. Su, J. Wang, J. Zhao, J. Zhang, J. Liu, J. Yan, J. Wu, L. Shi, L. Ye, L. Yu, M. Dong, N. Zhang, N. Ma, Q. Pan, Q. Gong, S. Liu, S. Ma, S. Wei, S. Cao, S. Huang, T. Jiang, W. Gao, W. Xiong, W. He, W. Huang, W. Xu, W. Wu, W. He, X. Wei, X. Jia, X. Wu, X. Xu, X. Zu, X. Zhou, X. Pan, Y. Charles, Y. Li, Y. Hu, Y. Liu, Y. Chen, Y. Wang, Y. Liu, Y. Qin, Y. Liu, Y. Yang, Y. Bao, Y. Du, Y. Wu, Y. Wang, Z. Zhou, Z. Wang, Z. Li, Z. Zhu, Z. Zhang, Z. Wang, Z. Yang, Z. Huang, Z. Huang, Z. Xu, Z. Yang, and Z. Lin Kimi k1.5: scaling reinforcement learning with llms. External Links: 2501.12599, Link Cited by: §1. Wang et al. (2025) Z. Wang, G. Cui, Y. Li, K. Wan, and W. Zhao DUMP: automated distribution-level curriculum learning for rl-based llm post-training. External Links: 2504.09710, Link Cited by: Appendix A, §2. Willems et al. (2020) L. Willems, S. Lahlou, and Y. Bengio Mastering rate based curriculum learning. arXiv preprint arXiv:2008.06456. Cited by: §1. Xiong et al. (2025) W. Xiong, C. Ye, B. Liao, H. Dong, X. Xu, C. Monz, J. Bian, N. Jiang, and T. Zhang Reinforce-ada: an adaptive sampling framework under non-linear RL objectives. arXiv preprint arXiv:2510.04996. Cited by: Appendix A. Xu et al. (2026) C. Xu, C. Xiao, Z. Peng, X. Wang, and Y. Cao SCALER: synthetic scalable adaptive learning environment for reasoning. arXiv preprint arXiv:2601.04809. Cited by: Appendix A, §1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Link Cited by: §5. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang DAPO: an open-source llm reinforcement learning system at scale. External Links: 2503.14476, Link Cited by: Appendix A, §1, §1, §2, §5.1. Zhang et al. (2026a) S. Zhang, Z. Xiao, S. Liu, G. Sun, K. Zhang, X. Guo, R. Guo, S. Liu, W. Zhao, and G. Jiang CLPO: curriculum learning meets policy optimization for llm reasoning. External Links: 2509.25004, Link Cited by: Appendix A. Zhang et al. (2026b) Z. Zhang, Z. Han, C. Mavromatis, Q. Zhu, Y. Zhang, S. Guan, D. Wang, X. Zhou, S. Wang, S. Adeshina, V. Ioannidis, and H. Rangwala Train less, learn more: adaptive efficient rollout optimization for group-based reinforcement learning. arXiv preprint arXiv:2602.14338. Cited by: Appendix A, §1. Zhao et al. (2025) A. Zhao, Y. Wu, Y. Yue, T. Wu, Q. Xu, Y. Yue, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang Absolute zero: reinforced self-play reasoning with zero data. External Links: 2505.03335, Link Cited by: Appendix A, §1. Zheng et al. (2025) H. Zheng, Y. Zhou, B. R. Bartoldson, B. Kailkhura, F. Lai, J. Zhao, and B. Chen Act only when it pays: efficient reinforcement learning for LLM reasoning via selective rollouts. arXiv preprint arXiv:2506.02177. Cited by: Appendix A, §1. Zhou et al. (2025) J. Zhou, L. Ma, H. Liang, C. Shen, B. Cui, and W. Zhang DARO: difficulty-aware reweighting policy optimization. arXiv preprint arXiv:2510.09001. Cited by: Appendix A, §2. Appendix A Additional Related Work Adaptive sampling and curricula in RLVR. The zero-gradient failure mode of GRPO has motivated methods that concentrate rollout compute on informative problems (Zheng et al., 2025; Zhang et al., 2026b; Le et al., 2026; Xiong et al., 2025). Sampling for Learnability (Foster et al., 2025) and DAPO (Yu et al., 2025) prioritise prompts with mixed rollout outcomes, with theoretical support for targeting intermediate-difficulty problems provided by Bae et al. (2025). Other approaches adapt rollout allocation, task sampling, or difficulty-group weighting (Li et al., 2025b; Ramesh et al., 2025; Chen et al., 2025b; Wang et al., 2025; Zhou et al., 2025; Panaganti et al., 2026). Working with a fixed problem pool, AdaCuRL schedules difficulty buckets with historical revisitation (Li et al., 2025a), while Actor-Curator trains a neural curator to select problems according to estimated policy improvement (Gu et al., 2026). AceGRPO additionally constructs and prioritises training tasks from intermediate execution states (Cai et al., 2026). Frontier learning instead searches a procedural generator’s configuration space, using configuration-level regret to determine where new training problems should be generated. Adaptive procedural generation. Procedural reasoning frameworks such as Reasoning Gym (Stojanovski et al., 2025), SynLogic (Liu et al., 2025a), and Reasoning Core (Lacombe et al., 2026) provide scalable generators of verifiable reasoning problems with controllable task parameters or difficulty, enabling training distributions beyond fixed datasets. Recent work has begun adapting such distributions to the evolving policy. SCALER (Xu et al., 2026) generates problems at difficulty levels that it raises or lowers according to the model’s current accuracy. This relies on a predefined ordering of difficulty levels. Frontier learning considers a different setting in which the generator exposes a multidimensional configuration space whose parameters need not induce a known or monotonic difficulty ordering. Unsupervised environment design and learnability. Frontier learning is also related to unsupervised environment design (UED), where the training environment evolves with the policy. PAIRED generates environments according to protagonist–antagonist regret (Dennis et al., 2020), while PLR prioritises previously encountered levels according to learning-potential estimates (Jiang et al., 2021). More closely related algorithmically, ACCEL (Parker-Holder et al., 2022) mutates high-regret levels to generate new environments near the agent’s evolving capability frontier. Frontier learning builds on this replay-and-editing principle in RLVR, estimating regret from groups of binary, verifiable rollout outcomes and aggregating it across problems from each procedural configuration. These configuration-level statistics guide replay alongside state-dependent mutation and exploration of unseen configurations. This signal is also related to learnability-based curricula (Oudeyer et al., 2007; Graves et al., 2017; Matiisen et al., 2017; Foster et al., 2025; Chen et al., 2025b); unlike symmetric mixed-outcome criteria such as the binary-reward SEC score, our regret prioritises the outer edge of the productive region and uses it not only for sampling but also to determine where the configuration space should expand. LLM-based problem and environment generation. A complementary line of work uses LLMs themselves to generate or transform training problems (Luo et al., 2025; Dai et al., 2026; Hosseini et al., 2024; Guan et al., 2025; Sundaram et al., 2026). Several recent methods couple generation directly to the learner’s evolving capabilities. CLPO uses the learner itself to simplify hard problems and diversify intermediate-difficulty problems through answer-preserving rewriting (Zhang et al., 2026a). Absolute Zero (Zhao et al., 2025) jointly learns to propose and solve tasks through self-play, while R-Zero (Huang et al., 2026) co-evolves a Challenger and Solver, rewarding the Challenger for proposing tasks near the edge of the Solver’s capability. Question-begets-Question (Bao et al., 2026) instead repeatedly selects problems that the current model can mostly solve and uses a teacher to generate new variants for subsequent RL training. SPADE (Liu et al., 2026) uses an LLM Environment Designer to generate executable environments and optimises generation using the Reasoning Agent’s performance gap with and without privileged hints. These methods adapt task generation itself to the learner, typically through an LLM proposer, teacher, or environment designer. Frontier learning instead searches an existing procedural generator’s explicit configuration space, trading generation flexibility for inexpensive and interpretable exploration and mutation without an LLM-based proposal or rewriting stage. Appendix B Procedural Generator Figure 4 illustrates how the Sokoban procedural generator produces levels of varying difficulty by scaling three configuration attributes: grid size (the side length of the square playable area), number of boxes to push onto goals, and maximum solution depth (the minimum number of moves required). Figure 4: Example Sokoban levels at the two extremes of the anchor ladder. Left: L1 — grid 6×66×6, 1 box, depth ≤12≤ 12. Right: L7 — grid 10×1010×10, 6 boxes, depth ≤80≤ 80. Colours: dark grey = wall, beige = floor, brown = box, yellow = goal, green = box on goal. Appendix C Anchor Levels Definition Table 4 gives the full per-level specification for every task. Countdown uses a monotone ladder co-scaling all attributes. Sokoban levels cover distinct combinations of grid size, box count, and solution depth without enforcing a strict monotone ordering on any single axis (difficulty in Sokoban is non-monotone: a smaller grid with more boxes can be harder than a larger one with fewer). Dice is a Cartesian product over two independent axes. Decimal Arithmetic jointly scales three axes. Table 4: Full anchor-level specifications, grouped into Easy/Medium/Hard (and Extra Hard for Dice) difficulty bins used throughout the paper. Countdown: num = num_numbers, mv = max_value, mt = max_target. Sokoban: grid is square; d = max_depth. Decimal: dp = decimal_places, pr = precision, t = terms. Largest Island: grid = rows×cols; isl = num_islands; sz = island_size. Task Level Bin Config Countdown 1 Easy num=3 mv=20 mt=80 2 Easy 3 40 150 3 Easy 3–4 60 250 4 Medium 4 80 350 5 Medium 4–5 100 500 6 Medium 5 120 800 7 Hard 5–6 150 1200 8 Hard 6 180 2000 9 Hard 6 220 3500 10 Hard 6 300 5000 Sokoban 1 Easy grid=6 boxes=1 d12 2 Easy 7 1 20 3 Easy 8 2 30 4 Medium 9 2 45 5 Medium 8 3 45 6 Hard 10 4 65 7 Hard 10 6 80 Dice 1 Easy num_dice=2 faces=8 2 Easy 2 10 3 Easy 2 16 4 Easy 2 20 5 Medium 3 8 6 Medium 3 10 7 Medium 3 16 8 Medium 3 20 9 Hard 4 8 10 Hard 4 10 11 Hard 4 16 12 Hard 4 20 13 Extra Hard 5 8 14 Extra Hard 5 10 15 Extra Hard 5 16 16 Extra Hard 5 20 Decimal Arith. 1 Easy dp=4 pr=8 t=4 2 Easy 4 12 6 3 Easy 6 12 6 4 Medium 6 18 10 5 Medium 10 18 10 6 Medium 10 24 14 7 Hard 14 18 10 8 Hard 14 24 18 9 Hard 18 24 14 Largest Island 1 Easy grid=8×10 isl=2 sz=3 2 Easy 9×12 2 4 3 Easy 10×14 3 4 4 Medium 11×15 3 5 5 Medium 12×16 4 5 6 Medium 13×17 4 6 7 Hard 14×18 5 6 8 Hard 15×18 5 7 9 Hard 15×18 6 6 10 Hard 15×18 6 7 Appendix D Implementation Details and Hyperparameters Tables 5 and 6 report the full set of hyperparameters used in all experiments. All runs using Qwen3-4B-Base or LLaMA3.2 set a maximum response length of 2048 tokens, while for Sokoban we use Qwen3-4B (thinking mode) with a response length of 4096 tokens to accommodate its extended chain-of-thought reasoning budget. All methods are run with 4 random seeds (42, 123, 314, 999). All the runs are performed on H200 or GH200 GPUs, ensuring that experiments belonging to the same task family are run on the same GPU type. Table 5: GRPO training hyperparameters (shared across all methods and tasks). Hyperparameter Symbol Value Learning rate 10−610^-6 Train batch size 64 Rollouts per problem nrn_r 8 PPO mini-batch size 16 PPO micro-batch size per GPU 8 Max problem length (tokens) 1024 Max response length — base model (tokens) 2048 Max response length — thinking model (tokens) 4096 KL coefficient βKL _KL 0 | 10−410^-4† Evaluation batch size 128 Random seeds 4 Clip low ϵlow _low 0.2 Clip high ϵhigh _high 0.2 | 0.28† †A small KL loss and an asymmetric ("clip-higher") PPO clipping range are applied for long-training stabilization only for the 500 steps runs (across all methods), preventing entropy collapse over extended training horizons Liu et al. (2025b). Table 6: Frontier Learning curriculum hyperparameters. Hyperparameter Symbol Value Buffer and sampling Levels per step nℓn_ 4 Problems per level npn_p 16 Buffer capacity BmaxB_ 100 Seed levels 8 Seed sampling mode grid Sampling strategy Zipf Zipf temperature TzT_z 1.0 Staleness coefficient λs _s 0.05 Informative success-rate interval [0.05-0.95] Regret signal Rolling window size nwn_w 16 Initial Regret R0R_0 0.5 Exploration Explore fraction ϕφ 0.3 State-dependent mutation probabilities Informative pinfp_inf 0.4 Too easy (mastered) peasyp_easy 0.25 Too difficult (impossible) phardp_hard 0.02 Unseen (never sampled) punsp_uns 0.0 SEC (baseline) EMA coefficient α 0.1 Appendix E Learning Curves Figure 5 shows accuracy learning curves for Countdown, Decimal Arithmetic and Sokoban over the standard training budget of 200 GRPO steps. Figure 5: Accuracy (%) on the fixed anchor evaluation sets over 200 GRPO steps for Countdown, Sokoban, and Decimal Arithmetic. Appendix F Per-Difficulty Accuracy Breakdown Figure 6 reports the learning curves across the three tasks over 200 GRPO steps across the difficulties bins, while Figure 7 breaks down the final validation performance by difficulty. On Countdown and Decimal Arithmetic, the gains from Frontier Learning are concentrated primarily on the Easy and Medium bins, whereas performance on the Hard bin remains largely comparable across methods. A similar pattern emerges on Sokoban, where Frontier Learning achieves its largest improvements on the easier configurations, with smaller gains as difficulty increases. This behavior is consistent with the objective of Frontier Learning: prioritizing levels that lie near the model’s current learning frontier, where additional training is most likely to yield improvement, rather than allocating substantial compute to levels that are either already mastered or currently too difficult to solve reliably. Importantly, this frontier is not fixed, but shifts as the model’s capabilities improve. This is illustrated by the extended training results in Figure 9, where Frontier Learning continues to improve throughout the 500-step training horizon, progressively reaching substantially higher validation accuracy on harder tasks. These results suggest that, as training progresses, the set of learnable configurations expands toward harder regions of the level space. Figure 6: Validation accuracy across the three difficulty bins throughout the full training horizon for Countdown, Sokoban, and Decimal Arithmetic. Figure 7: Final validation accuracy per difficulty bin for Countdown, Sokoban, and Decimal Arithmetic. Figure 8: Long-horizon Dice learning curves over 500 GRPO steps. DAPO is omitted from the step-based curves because it is matched by total rollout budget rather than optimizer updates. Figure 9: Validation accuracy across the four difficulty bins throughout the full training horizon for Dice. Figure 10: Final validation accuracy per difficulty bin for Dice Appendix G Hyperparameter sensitivity G.1 Frontier Learning We additionally study the sensitivity of frontier learning to the staleness coefficient λs _s and the informative success-rate interval defined in Equation 8. We conduct this ablation on the same Llama-3.2-3B-Instruct Countdown setting used for the cross-model-family evaluation in the main paper. Table 7 reports the results. Table 7: Hyperparameter sensitivity on Countdown using Llama-3.2-3B-Instruct. Variant Accuracy (%) Default frontier learning 39.2±1.339.2± 1.3 λs=0.02 _s=0.02 40.7±1.140.7± 1.1 Informative interval =[0.10,0.90]=[0.10,0.90] 40.5±0.840.5± 0.8 Informative interval =[0.02,0.98]=[0.02,0.98] 39.8±1.039.8± 1.0 λs=0 _s=0 38.8±1.438.8± 1.4 Performance remains within a narrow range across the tested configurations. In particular, varying either the staleness coefficient or the informative success-rate interval leads to only modest changes relative to the default configuration, suggesting that frontier learning is not highly sensitive to these hyperparameters around the chosen defaults. G.2 Step size sensitivity of the baselines To rule out an under-tuned learning rate as an explanation for frontier learning’s gains, we swept the learning rate on Dice (500 steps, seed 42) for two representative baselines: SEC, the best-performing baseline in Table 2, and Uniform, a non-regret-based method (fixed pool, uniform sampling, no prioritization signal). We swept 5×10−7,9×10−7,1×10−6,2×10−6,5×10−6,5×10−5\5× 10^-7,9× 10^-7,1× 10^-6,2× 10^-6,5× 10^-6,5× 10^-5\, holding every other hyperparameter fixed. As it can be observed in Fig. 11 our paper’s learning rate, 1×10−61× 10^-6, is the best- or joint-best-performing value for both methods. Larger learning rates (≥2×10−6≥ 2× 10^-6) are uniformly worse and increasingly unstable, collapsing to near-zero accuracy by 2×10−62× 10^-6–5×10−55× 10^-5; no value we tested lifts either baseline’s final accuracy above ≈ 35%, well below frontier learning’s performance under the same protocol. Figure 11: Learning-rate sweep on Dice (seed 42) for Uniform and SEC, the non-regret-based baseline and the best-performing baseline in Table 2, respectively. †Denotes the learning rate used for the main paper’s results (1×10−61× 10^-6). No swept value raises either baseline’s final accuracy above ≈ 35%, indicating that an under-tuned learning rate does not explain the gap to frontier learning. Appendix H Compute accounting. 200-step experiments: We first report end-to-end compute for the 200-step experiments on Sokoban, Countdown, and Decimal Arithmetic. All methods use the same number of optimizer updates, prompts, rollouts per prompt, and maximum response length. Across these tasks, frontier learning has comparable wall-clock cost to the baselines and does not systematically generate longer responses. Table 8: End-to-end compute accounting on Sokoban (200 steps). Method Runtime (h) Response length (tokens) Accuracy (%) Domain Rand. 15.4± 2.0 3838± 33 37.6± 2.6 PLR 14.8± 1.8 3698± 103 41.1± 3.9 Uniform 15.1± 1.8 3761± 51 39.5± 1.3 SEC 14.8± 1.8 3709± 81 34.3± 6.6 Frontier Learning (ours) 14.2± 1.7 3529± 49 45.2± 1.9 Table 9: End-to-end compute accounting on Countdown (200 steps). Method Runtime (h) Response length (tokens) Accuracy (%) Domain Rand. 17.4± 0.1 1538± 14 44.7± 1.7 PLR 11.3± 0.3 1521± 44 44.3± 2.9 Uniform 11.8± 0.3 1546± 50 44.2± 4.0 SEC 11.2± 0.4 1446± 65 40.1± 8.1 Frontier Learning (ours) 11.7± 0.5 1269± 28 50.9± 0.5 Table 10: End-to-end compute accounting on Decimal Arithmetic (200 steps). Method Runtime (h) Response length (tokens) Accuracy (%) Domain Rand. 4.3± 1.1 837± 31 31.0± 2.8 PLR 6.0± 1.1 677± 30 27.5± 2.9 Uniform 5.7± 1.0 642± 33 28.0± 2.6 SEC 6.1± 1.2 665± 54 29.1± 4.1 Frontier Learning (ours) 4.2± 1.1 758± 50 34.7± 3.6 Long-horizon compute on Dice: The 500-step Dice experiment reveals a different regime. Although all methods except DAPO use the same number of optimizer updates, prompts, rollouts per prompt, and maximum response length, realized wall-clock cost differs substantially. Frontier learning requires 27.8±0.627.8± 0.6 hours, compared with 18.7±1.118.7± 1.1 hours for PLR, and produces 1383±261383± 26 tokens per response on average, compared with 877±65877± 65 for PLR. This increase in response length is consistent with the outward movement of the learning frontier observed in the per-difficulty analysis. After 500 steps, frontier learning has nearly saturated Easy and Medium configurations and reaches 72.6% on Hard and 20.7% on Extra-hard levels, whereas no standard baseline exceeds 5.5% on these harder categories. As frontier learning acquires the ability to engage with these more difficult configurations, its sampled problems induce longer reasoning traces. The additional wall-clock cost is therefore partly a consequence of the capability expansion produced during training, rather than simply a larger nominal training budget. However, this additional runtime does not explain frontier learning’s performance advantage. As shown in Table 12, looking at the wall-clock time corresponding to PLR’s complete 500-step run (i.e. the fastest baseline) frontier learning already reaches 68.0±5.568.0± 5.5% accuracy. Moreover, PLR+Mut produces even longer responses (1426±141426± 14 tokens) and requires slightly more wall-clock time (28.5±0.328.5± 0.3 hours) than frontier learning, while attaining lower final performance. Thus, neither longer generations nor greater realized wall-clock compute alone account for the gains of frontier learning. Table 11: End-to-end compute accounting on Dice (500 steps). Method Runtime (h) Response length (tokens) Accuracy (%) Baselines Domain Rand. 19.4± 1.9 907± 104 23.4± 4.7 PLR 18.7± 1.1 877± 65 31.2± 1.9 Uniform 20.8± 1.9 994± 108 31.9± 1.5 SEC 18.9± 2.3 866± 129 33.3± 1.2 ACE-GRPO 22.0± 0.8 1063± 45 33.2± 2.1 Component ablations Unif+Explore 20.4± 1.8 974± 105 30.8± 1.3 Unif+FlatMut 22.6± 1.3 1099± 72 47.1± 10.8 Unif+Exp+FlatMut 20.1± 1.8 941± 110 36.9± 2.9 PLR+Explore 24.3± 0.8 1190± 45 33.5± 1.8 PLR+Mut 28.5± 0.3 1426± 14 62.5± 4.5 frontier learning(ours) 27.8± 0.6 1383± 26 71.8± 6.2 Table 12: Dice performance at matched PLR compute time (18.7 hours). Method Accuracy (%) DR 25.4± 3.7 PLR 31.2± 1.9 Uniform 32.2± 1.7 SEC 33.5± 1.6 AceGRPO 31.8± 3.3 Frontier Learning (ours) 68.0± 5.5 Unif+FlatMut 43.7± 9.8 Unif+Explore 31.9± 1.7 Unif+Exp+FlatMut 37.1± 3.4 PLR+Explore 34.8± 1.2 PLR+Mut 60.7± 4.3 Long horizon compute on Largest Island: Unlike on Dice, frontier learning’s runtime and response length on Largest Island are not elevated relative to the baselines: both sit in the middle of the range they span (Table 13), so, also in this case, frontier learning’s accuracy advantage cannot be attributed to spending more wall-clock time or producing longer responses. Table 13: End-to-end compute accounting on Largest Island (500 steps, Olmo-3-7B-Instruct). Method Runtime (h) Response length (tokens) Accuracy (%) Domain Rand. 63.2± 0.3 1578± 29 45.2± 1.8 PLR 44.3± 6.0 1276± 96 52.0± 5.3 Uniform 53.1± 2.5 1132± 125 45.3± 3.4 SEC 44.4± 7.1 1218± 102 48.3± 4.7 ACE-GRPO 43.8± 4.7 1243± 106 45.3± 5.5 frontier learning (ours) 52.4± 0.7 1498± 6 73.7± 3.3