Paper deep dive

Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation

Gijs Kassenaar, Zhao Yang, Vincent François-Lavet

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 57

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 8/24/2026, 4:26:28 AM

Summary

This paper introduces an adaptive reasoning framework for language models that dynamically allocates test-time compute by selecting one of three reasoning modes (NoThink, Short, Long) as the first token of the response. Using Group Relative Policy Optimization (GRPO) without a separate router, the model learns to route problems by difficulty through shaped rewards, hard per-mode token caps, and a balance term to prevent routing collapse. Evaluated on a 1.5B distilled model trained on MATH, the approach reduces mean response length by 41% while maintaining accuracy comparable to the base model, and demonstrates transferability to other benchmarks like GSM8K with significant token savings.

Entities (10)

GSM8K · dataset · 99%MATH · dataset · 99%MATH500 · dataset · 99%GRPO · algorithm · 98%DeepSeek-R1-Distill-Qwen-1.5B · model · 97%Long · reasoning-mode · 95%NoThink · reasoning-mode · 95%Short · reasoning-mode · 95%Balance Term · mechanism · 92%Routing collapse · problem · 90%

Relation Signals (9)

Adaptive Reasoning Framework → appliedto → MATH

confidence 98% · On a 1.5B distilled model trained on MATH

Adaptive Reasoning Framework → evaluatedon → MATH500

confidence 98% · the resulting policy stays close to the base model's accuracy on the held-out MATH500

Adaptive Reasoning Framework → definesmodes → Short

confidence 95% · choosing, as the first token of its response, one of three modes: Short

Adaptive Reasoning Framework → definesmodes → Long

confidence 95% · choosing, as the first token of its response, one of three modes: Long

Adaptive Reasoning Framework → definesmodes → NoThink

confidence 95% · choosing, as the first token of its response, one of three modes: NoThink

Adaptive Reasoning Framework → reducescomputeby → 41%

confidence 95% · cutting the mean response length from 4,796 to 2,811 tokens (a 41% reduction)

Adaptive Reasoning Framework → transfersto → GSM8K

confidence 95% · it also transfers to other benchmarks... with for instance 76% token reduction on GSM8K

Adaptive Reasoning Framework → uses → GRPO

confidence 95% · The choice is learned inside Group Relative Policy Optimization (GRPO)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Reasoning language models trained with reinforcement learning typically operate under a fixed token budget rather than an explicitly adaptive one, which can lead to over-computation on easy problems and insufficient computation on difficult ones. We study whether a model can learn to allocate its own reasoning effort by choosing, as the first token of its response, one of three modes: \textsc{NoThink} (answer as quickly as possible), \textsc{Short} (brief reasoning), or \textsc{Long} (extended reasoning). The choice is learned inside Group Relative Policy Optimization (GRPO) with no separate router, through a shaped reward that makes each mode worthwhile at a different response length, together with hard per-mode token caps that keep the modes distinct. On a 1.5B distilled model trained on MATH, the three modes emerge without collapsing to a single choice, and the brief modes end up more accurate than \textsc{Long}, which shows that the router sorts problems by difficulty rather than at random. Averaged over three seeds, the resulting policy stays close to the base model's accuracy on the held-out MATH500 ($0.782$ vs.\ $0.796$) while cutting the mean response length from $4{,}796$ to $2{,}811$ tokens (a $41\%$ reduction). Interestingly, it also transfers to other benchmarks without retraining, with the largest savings where problems are easier, with for instance 76\% token reduction on GSM8K and at higher accuracy than the baselines at similar response length. In short, we build a reasoning model that adaptively chooses how much to reason for each problem.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

54,182 characters extracted from source content.

Expand or collapse full text

Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation Gijs Kassenaar Zhao Yang Vincent François-Lavet Abstract Reasoning language models trained with reinforcement learning typically operate under a fixed token budget rather than an explicitly adaptive one, which can lead to over-computation on easy problems and insufficient computation on difficult ones. We study whether a model can learn to allocate its own reasoning effort by choosing, as the first token of its response, one of three modes: NoThink (answer as quickly as possible), Short (brief reasoning), or Long (extended reasoning). The choice is learned inside Group Relative Policy Optimization (GRPO) with no separate router, through a shaped reward that makes each mode worthwhile at a different response length, together with hard per-mode token caps that keep the modes distinct. On a 1.5B distilled model trained on MATH, the three modes emerge without collapsing to a single choice, and the brief modes end up more accurate than Long, which shows that the router sorts problems by difficulty rather than at random. Averaged over three seeds, the resulting policy stays close to the base model’s accuracy on the held-out MATH500 (0.7820.782 vs. 0.7960.796) while cutting the mean response length from 4,7964,796 to 2,8112,811 tokens (a 41%41\% reduction). Interestingly, it also transfers to other benchmarks without retraining, with the largest savings where problems are easier, with for instance 76% token reduction on GSM8K and at higher accuracy than the baselines at similar response length. In short, we build a reasoning model that adaptively chooses how much to reason for each problem. Introduction Figure 1: Three-mode self-routing GRPO. The model emits a routing token (NoThink/Short/Long) as the first token of its response and then reasons under that mode’s shaped reward and hard token cap; the routing token is part of the main policy and is trained end-to-end by GRPO, with no separate router. Large language models (LLMs) have achieved remarkable performance on complex reasoning tasks, driven largely by Chain-of-Thought (CoT) prompting (16). By breaking problems into intermediate steps, models such as DeepSeek-R1 (7) reach state-of-the-art results across mathematical and coding benchmarks. This capability, however, comes at a substantial computational cost: extended reasoning chains consume large numbers of tokens, increasing both inference latency and training cost. The burden falls hardest on smaller models, which often exhibit “verbosity compensation,” generating unnecessarily long explanations to compensate for limited capacity (3; 23). Empirically, incorrect answers tend to be markedly longer than correct ones, indicating that current models frequently spend reasoning tokens inefficiently (1). Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for fine-tuning reasoning models (13). Standard algorithms such as PPO (12) and GRPO (13) operate with a single maximum response length applied uniformly across all training samples. This couples two competing inefficiencies: early in training, the model wastes compute generating long, incorrect traces; later, the same fixed ceiling can prevent the extended chains needed to solve genuinely hard problems. Recent analysis further suggests that RLVR functions primarily as an elicitation mechanism, improving pass@kpass@k for small k but remaining bounded by the base model’s latent capability for large k (20). A principled response to the length problem is temporal discounting: scaling reward by a per-token factor penalizes long rollouts directly, and discount-based variants of GRPO have been shown to shorten reasoning while preserving accuracy (11; 2; 22; 18). Discounting, however, is non-adaptive: it imposes a fixed length penalty regardless of a prompt’s difficulty or the response length its solution actually requires. An easy question and a hard question are pressured to be equally brief, even though the optimal reasoning budget differs. The natural next step is to let the model decide, per prompt, how much reasoning a problem warrants, applying brevity pressure only where it is justified. We study this question directly. We ask a 1.5B distilled reasoning model to emit, as the very first token of its response, one of three discrete routing decisions—NoThink, Short, or Long—and then to reason accordingly. The routing token is part of the main policy and receives gradient through the standard GRPO objective, there is no separate router network. The central difficulty is routing collapse: because the routing decision is a single self-reinforcing discrete token, a mode with even a small reward edge is sampled more, reinforced more, and quickly drives the alternatives out of the policy. Our method overcomes collapse with two mechanisms working together. First, hard per-mode token caps make the modes distinct by construction, so the routing label cannot decouple from behavior. Second, a simple balance term added directly to the advantage, similar to the load-balancing penalties used to discourage expert collapse in Mixture-of-Experts routing (6), keeps the policy from settling on a single mode. A forced-rollout warmup stabilizes the early dynamics. Our contributions are twofold. First, we show that a 1.5B reasoning model can be taught to route problems by difficulty, choosing among three reasoning modes as its first generated token, end-to-end inside GRPO, with no separate router and no supervised warm-up. Second, we show that this routing cuts the reasoning-token budget substantially while placing the policy on or above the accuracy–length Pareto frontier traced by the single-mode baselines — both in-distribution on MATH and on held-out benchmarks. As part of the analysis, a negative result on the Countdown task explains when difficulty routing is and is not learnable. Related Work Reinforcement Learning for Reasoning RL fine-tuning of LLMs (18) began with Proximal Policy Optimization (PPO), which uses a clipped surrogate objective for stable updates (12). To remove the memory overhead of PPO’s value network, Group Relative Policy Optimization (GRPO) computes advantages relative to a group of sampled responses (13). While efficient, GRPO carries optimization biases that recent work has sought to correct. 10 identify a “length bias” arising from dividing each response’s loss by its token length, which systematically favors longer incorrect answers and drives “length explosion.” Their Dr. GRPO removes the two terms they hold responsible: the per-response length normalization in the loss and the within-group standard-deviation scaling of the advantage, yielding updates that no longer reward verbosity. DAPO (19) contributes two ingredients we adopt: a token-level loss aggregation that prevents long traces from being under-weighted, and the observation that the KL penalty toward the reference policy is unnecessary—and indeed counterproductive—once the policy is allowed to move far from its initialization during long-horizon reasoning RL. Discounting in GRPO Because GRPO uses no value function, it normally carries no temporal discount, but a discount is a principled way to penalize length and several recent methods add one. They differ in where the discount appears , which changes the resulting group advantages substantially. The discounted-reasoning method (2) multiplies the sequence reward by γKγ^K, with K the reasoning-token count, before group normalization; this couples advantage magnitude to absolute rollout length, so groups of long rollouts and groups of short rollouts are scaled differently. GRPO-LEAD (22) also shapes the reward before normalization, but standardizes length within the group via a z-score and adds difficulty-aware weighting, which makes the relative advantages comparable across groups regardless of nominal length. GRPO-λ (11) instead applies its decay after normalization, tracing credit backward over token positions so that later tokens, which are often most important for answer, receive larger-magnitude updates while the sign is inherited from the group outcome. Our reward surface builds on the pre-normalization γKγ^K formulation, but rather than applying one global discount we attach a separate base reward and discount to each reasoning mode, turning a uniform length penalty into a per-mode incentive. Adaptive Reasoning-Mode Selection A related line of work shortens reasoning by skipping or abbreviating it according to problem difficulty, and the methods differ in how much training machinery they require. AdaptThink (21) and AutoThink (15) stay within pure RL—the former choosing between a full-reasoning and a no-reasoning mode via a constrained objective with importance sampling for cold start, the latter using a multi-stage curriculum—each roughly halving response length on DeepSeek-R1-Distill-Qwen-1.5B while preserving or improving accuracy. ARM (17) and Thinkless (5) instead pre-install the modes with a supervised stage before RL: ARM supervised-fine-tunes on data annotated in four formats (direct answer, short and long chain-of-thought, and code), and Thinkless distills a reasoning expert and an instruction expert into two control tokens, then runs RL with a decoupled objective so the lone control token is not swamped by the response tokens. A difficulty shared by all of these methods is routing collapse: because the mode decision is a single discrete, self-reinforcing choice, a mode with even a small reward advantage is sampled more, reinforced more, and drives the alternatives out of the policy. The all-one-mode states are stable under policy gradient while the mixed split is not, an effect that is stronger on a small distilled model with low output entropy and a strong prior to reason at length. The methods above stabilize the choice with importance sampling (AdaptThink), curriculum staging (AutoThink), or a supervised stage that pre-installs the modes (ARM, Thinkless). Ours is the leanest of these—no supervised or distillation stage, three modes learned end-to-end with RL—relying instead on a forced warmup, hard per-mode caps, and a balance penalty on the advantage to keep all modes alive. Preliminaries The GRPO Objective GRPO optimizes a policy by sampling a group of G outputs o1,…,oG\o_1,…,o_G\ for each prompt q from the current policy πθold _ _old. It replaces the critic with a group-relative advantage AiA_i, obtained by normalizing rewards rir_i within the group: Ai=ri−mean⁡(r1,…,rG)std⁡(r1,…,rG)+ϵ.A_i= r_i-mean(r_1,…,r_G)std(r_1,…,r_G)+ε. (1) The policy maximizes a clipped surrogate objective with a KL penalty, JGRPO(θ)=⁡[1G∑i=1G1Ni∑t=1Ni(mi,t−βDKL)],J_GRPO(θ)=E\! [ 1G _i=1^G 1N_i _t=1^N_i (m_i,t-β\,D_KL ) ]\!, (2) where mi,t=min⁡(ρi,tAi,clip⁡(ρi,t,1−ϵ,1+ϵ)Ai)m_i,t= \! ( _i,tA_i,\ clip( _i,t,1-ε,1+ε)A_i ) is the clipped surrogate term, DKL=DKL(πθ∥πref)D_KL=D_KL( _θ\,\|\, _ref), and ρi,t=πθ(oi,t∣q,oi,<t)/πθold(oi,t∣q,oi,<t) _i,t= _θ(o_i,t q,o_i,<t)/ _ _old(o_i,t q,o_i,<t) is the token-level importance ratio, with ϵε the clipping threshold and β the penalty strength toward a reference policy πref _ref. We set β=0β=0 throughout, removing the KL penalty entirely. The KL term keeps the policy close to the reference model: while this stabilizes ordinary RLHF, it limits the exploration needed to find new reasoning strategies, and in long-horizon reasoning RL it becomes unnecessary and can be harmful (19). This matters more for routing, which requires the policy to explore new behaviors such as answering quickly or stopping its reasoning early, behaviors the distilled base model rarely produces; a term that anchors the policy to that base model would work against this exploration. Loss Aggregation The advantage AiA_i is a sequence-level quantity, but the loss is summed over tokens, and how token losses are reduced to a scalar affects length dynamics. The default DeepSeek formulation (13) uses sequence-mean, token-mean (SMTM) aggregation—averaging tokens within each sequence, then averaging across the group (the nested 1G∑i1Ni∑t 1G _i 1N_i _t in Eq. (2)). The inner 1Ni 1N_i weights every sequence equally regardless of length, but as a side effect each token in a long sequence contributes a smaller gradient than a token in a short one. We instead use token-mean aggregation (19), which normalizes by the total token count ∑iNi _iN_i: LTM=1∑i=1GNi∑i=1G∑t=1Niℓi,t,L_TM= 1 _i=1^GN_i _i=1^G _t=1^N_i _i,t, (3) so every token carries the same weight regardless of its sequence. This matters for routing, where the modes deliberately produce responses of very different lengths: under SMTM the long-mode tokens would receive weaker gradients than the brief-mode tokens, biasing learning away from the mode that handles the hardest problems. The Normalization-Amplification Problem The discounted-reasoning formulation (2) multiplies the sequence reward by a per-token factor before normalization, Ri=γKiri,R_i=γ^K_ir_i, (4) with KiK_i the reasoning-token count and γ∈(0,1)γ∈(0,1), decaying reward as reasoning grows. This interacts badly with GRPO’s standard-deviation normalization. In vanilla GRPO a group with no reward variance e.g. all rollouts correct, produces all-zero advantages and no learning signal. The γKγ^K discount creates tiny reward differences within such a group (longer correct responses score slightly lower), and the division by stdstd in Eq. (1) amplifies these into full-magnitude advantages, comparable to those from a genuinely mixed group—diverting training toward small length optimizations instead of solving harder problems. We avoid this by removing the standard-deviation normalization entirely and keeping only mean-centering, so the advantage is simply A^i=Ri−μg A_i=R_i- _g, proportional to the actual reward differences; the small differences in an all-correct group then stay small. Method Discounting applies the same penalty to every prompt. Our goal is to let the model allocate reasoning effort per prompt: skip reasoning when a problem is trivial, reason briefly when it is easy, and reason at length when it is hard. We realize this by having the model emit a discrete routing token as the first word of its response and conditioning both reward and a hard token budget on that choice. We first explored this idea with a two-mode (Short/Long) router on the Countdown task; it collapsed to a single mode because Countdown’s uniform difficulty gives the router no accuracy gap to learn from (see the supplementary appendix). That negative result motivates the design below: hard per-mode token caps that make the modes behaviorally distinct, and evaluation on MATH, whose wide spread of difficulty gives routing something to learn. Three Modes with Hard Token Caps We define three modes: • NoThink: the model answers as quickly as possible, using little reasoning. It is not forced to skip reasoning and may still think briefly, but it is pushed toward a short answer; intended for the easiest problems. • Short: the model reasons briefly within a hard token cap, intended for easy problems that benefit from a short trace. • Long: the model reasons at length without a cap, intended for hard problems that need extended search. The routing instruction appended to every prompt at runtime is: “Output NoThink to answer directly, Short for brief reasoning, or Long for extended reasoning.” The routing token sits in the response head and receives gradient like any other token; no data preprocessing is required. The per-mode reward and its discount γ, defined in the next subsection, are the primary mechanism that separates the modes: they make each mode the reward-optimal choice over a different range of response lengths. The hard per-mode caps play two supporting roles. First, they keep the policy from collapsing onto a single mode: because a capped brief mode is scored wrong as soon as its answer runs past the cap, it genuinely fails on problems that need more tokens, so Long always retains a reason to exist and the brief modes cannot quietly absorb hard problems. Second, they save compute during training, since rollouts in the capped modes generate far fewer tokens than uncapped reasoning. Concretely, a response whose final answer token falls beyond its mode’s cap is scored incorrect regardless of content; the cap is enforced during generation. Default caps are 1,024 tokens for NoThink and 3,000 for Short; Long is uncapped. Reward Surface Each mode has a base reward b and a per-token discount γ on the number of tokens LiL_i generated after the routing token: ri=bNTγNTLicorrect, NoThinkbSγSLicorrect, ShortbLcorrect, Long0.0incorrect, valid routing−0.5routing token absent.r_i= casesb_NT\, _NT^L_i&correct, NoThink\\ b_S\, _S^L_i&correct, Short\\ b_L&correct, Long\\ 0.0&incorrect, valid routing\\ -0.5&routing token absent. cases (5) The discounts γNT _NT and γS _S are set so that each mode is the reward-optimal choice over the length range it is designed for, with the crossover between adjacent modes falling at or below the cap of the briefer one. With bases bNT=1.3b_NT=1.3, bS=1.2b_S=1.2, bL=1.0b_L=1.0, the NoThink/Short crossover falls at L≈800L≈ 800 and the Short/Long crossover at L≈3000L≈ 3000, the Short cap. Table 3 lists the resulting rewards and Figure 9 plots the curves, the two crossovers marking where the optimal mode changes. Table 1: Correct-answer reward by mode at selected response lengths L, under default bases and caps. Crossovers occur where two modes tie (bold). L NoThink Short Long 0 1.30 1.20 1.00 256 1.25 1.18 1.00 800 1.14 1.14 1.00 3000 0.80 1.00 1.00 8192 0.35 0.73 1.00 Figure 2: Correct-answer reward for each mode as a function of the number of tokens L generated after the routing token, under the bases and caps of Table 4. NoThink (1.3γNTL1.3\, _NT^L) and Short (1.2γSL1.2\, _S^L) decay with length while Long is flat at 1.01.0. The discounts place the crossovers at LNT∗≈800L_NT^*\!≈\!800 (NoThink/Short) and LS∗≈3000L_S^*\!≈\!3000 (Short/Long), at or below the mode caps. Each mode is reward-optimal in its own length band. Advantage Estimation and the Balance Term We use GRPO mean-centering across the group but disable standard-deviation normalization, for the reason given in Section The Normalization-Amplification Problem: the bases in Eq. (7) are intentionally heterogeneous across modes, and normalizing by within-group std would amplify the small reward differences that arise when all rollouts are correct. Mean-centering cancels any signal injected at the reward level when a group is unmixed (all rollouts pick the same mode), which is the regime in which collapse occurs. To keep pressure toward a balanced split, we add a simple correction directly to the advantage, after centering: A^i←Ai+βbal(p⋆−fmode(i)), A_i← A_i+ _bal (p -f_mode(i) ), (6) where fmodef_mode is the fraction of free rollouts choosing that mode, p⋆=1/3p =1/3 is the target share, and βbal _bal is the balance coefficient. A mode that is over-represented (its fraction above p⋆p ) has its advantage reduced, while an under-represented mode has its advantage raised. To keep the balance term from distorting the accuracy signal, we apply it asymmetrically by outcome: a negative correction (for an over-represented mode) is applied only to incorrect rollouts, and a positive correction (for an under-represented mode) only to correct rollouts. Balancing therefore never penalizes a correct answer or rewards an incorrect one; it redistributes routing mass only in the direction that is consistent with the reward. Because this correction is folded into the advantage, it enters the policy-gradient loss directly rather than as a separate auxiliary load-balancing loss (6). Because the balance term is added after mean-centering, it remains effective in zero reward variance or routing variance groups. Warmup The base model never emits a routing token unprompted, because the chat template prefills <think> and the model proceeds directly to reasoning. For the first warmup_steps training steps we therefore force two rollout per group into each of NoThink, Short, and Long, generating the remaining rollouts freely, and we promote the forced routing token into the response head so that gradient flows through the routing position from the first step.Together with the penalty given to all rollouts that have no routing token, causes the model to quickly learn to route. After warmup, all rollouts are generated freely. Algorithm 1 summarizes one training step. Hyper-parameters used can be found in the supplementary appendix. Algorithm 1 Three-Mode Self-Routing GRPO (one step) Input: prompts q\q\, policy πθ _θ, step s, caps CNT,CSC_NT,C_S Parameter: bases b, discounts γ, βbal _bal, p⋆p 1: Append routing instruction to each q 2: if s≤s≤ warmup_steps then 3: Force 1 rollout per mode; remaining free; promote token 4: else 5: Sample G free rollouts per prompt 6: end if 7: Detect mode from first response token; apply caps 8: Compute rewards by Eq. (7) 9: Mean-center advantages (no std-normalization) 10: Add balance term, Eq. (9), over free rollouts 11: Update πθ _θ with the GRPO objective, Eq. (2) Experimental Setup Datasets and base model. We train on the MATH-lighteval split of competition mathematics problems with verifiable final answers (8) and validate on the held-out MATH-500 set (9) at temperature 0.60.6. A response is correct if its final answer matches the reference, giving a binary reward (r=1r=1 correct, r=0r=0 otherwise) on which the routing shaping of Eq. (7) is applied. For out-of-distribution evaluation we use GSM8K (4) (1,319 grade-school word problems, avg@5) and AIME 2024 and 2025 (30 competition problems each, avg@16). The preliminary binary-routing study uses the Countdown task (14); see the supplementary appendix. We use DeepSeek-R1-Distill-Qwen-1.5B (7), a 1.5B-parameter reasoning model distilled from DeepSeek-R1 on roughly 800K high-quality traces. Training. We run GRPO with group size G=8G=8 for 90 steps from a cold start, with a 16,38416,384-token context cap, on 4 NVIDIA H100 GPUs; hyperparameters are in Table 4. We train three seeds of the router and of each single-mode baseline, reporting mean ± std over seeds throughout. Single-mode baselines. To separate adaptive routing from any single mode, we train three ablations in which every rollout is forced to one mode (NoThink-, Short-, or Long-only), with the same data and hyperparameters but no routing token, balance term, or free rollouts; each learns a fixed reasoning budget against which the router is compared on the accuracy versus length plane (Figure 6). Evaluation. We report MATH-500 validation accuracy, the free-rollout mode distribution and routing entropy, per-mode accuracy, and mean response length. Validation generation is uncapped, measuring production behavior under free generation. Experimental Results Through the experiments we would like to answer three questions: (1) Stability: can three reasoning modes be learned end-to-end inside GRPO without collapsing to a single mode? (2) Meaningfulness: does the learned routing sort problems by difficulty, rather than partitioning them arbitrarily? (3) Efficiency: what accuracy and token cost does routing deliver compared to fixed reasoning budgets, and does it transfer beyond the training distribution? The three subsections below address these questions in turn. Three Modes Emerge and Persist Figure 3 traces the free-rollout mode distribution over training. During the forced warmup free rollouts are almost entirely unknown, with the first routing tokens appearing around step 20. Once routing is free the model at first commits heavily to Long, then the brief modes grow as the balance term takes effect, and by step 90 the split settles near 20/32/47%20/32/47\% (NoThink/Short/Long), with routing entropy close to the theoretical maximum (H≈1.04H≈ 1.04 vs. ln⁡3≈1.10 3≈ 1.10). All three modes persist to the end of training across all seeds, with Long remaining the plurality and no collapse into any single mode. Figure 3: Free-rollout mode shares over training, mean over three seeds (the forced warmup runs to step 45). After warmup all three modes persist without collapsing to any single mode, with Long remaining the plurality. Routing Reflects Problem Difficulty Two independent pieces of evidence show the routing is semantically meaningful rather than an arbitrary partition. The first is the inversion of per-mode accuracy over training (Figure 4). Early on, the forced rollouts reproduce the base model’s prior ordering Long>Short>NoThink Long> Short> NoThink, reflecting a model that reasons at length by default. Around step 50 this ordering inverts: NoThink and Short rise above Long and stay there across all seeds, ending well above it at the final checkpoint. Easy problems are being assigned to the brief modes while hard problems concentrate in Long, whose lower accuracy reflects genuine difficulty rather than mode inferiority. Consistently, the Long mean length grows markedly over training while NoThink and Short stay well within their caps. The second is agreement with an external difficulty signal the model never saw in training: the MATH-500 difficulty levels from 1 (easiest) to 5 (hardest). Figure 5 shows the free-validation routing split per level, and the trend is monotone: NoThink carries most of the easiest level but little of the hardest, Long does the reverse, and Short is most used at intermediate difficulty, where a brief trace is most useful. Per-level accuracy falls steadily with level, confirming the labels track real difficulty. The router therefore allocates reasoning budget in proportion to difficulty, using the cheap modes on easy levels for large token savings and reserving Long for the hard ones. Figure 4: Per-mode accuracy over training (mean, with std bands, over three seeds). The forced base-model ordering (Long>Short>NoThink Long> Short> NoThink) inverts around step 50, evidence that the router assigns problems by difficulty: brief modes get easier problems and Long concentrates the hard ones. Figure 5: Free-validation routing split (NoThink / Short / Long) by MATH-500 difficulty level, with per-level accuracy above each bar. Mass shifts from NoThink on easy levels to Long on hard ones, with Short most used at intermediate difficulty. Token Efficiency, In and Out of Distribution We evaluate efficiency in two regimes. In-distribution refers to MATH-500: held out from training but drawn from the same competition-mathematics distribution as the MATH-lighteval training split. Out-of-distribution refers to GSM8K and AIME, datasets the model never trained on that sit at opposite ends of the difficulty range, far easier and far harder than the training data respectively; they test whether the routing itself transfers. Table 2 reports MATH validation accuracy over training. Accuracy does not rise above the base model: it dips while routing is being established and recovers to 0.7830.783 by step 90, close to the untrained model’s 0.8070.807. The gain is in efficiency: on MATH-500 the router reaches this accuracy at 2,8112,811 tokens on average, 41%41\% fewer than the base (Figure 6). The reduction is established early, when the brief modes absorb the easy problems, and is then partly offset as Long responses lengthen toward the context limit. At the final checkpoint the per-mode validation accuracies reproduce the training ordering, so the brief modes are reliably selected for problems the model can solve briefly. Figure 6 places all methods on the accuracy versus length plane. The single-mode baselines trace a Pareto frontier of fixed reasoning budgets, and free routing lies strictly above it: it reaches an accuracy that the frontier only attains with 27%27\% more tokens, while also sitting slightly higher than the frontier at its own length. In other words, the saving holds not only against the base model but against the frontier traced by the three single-mode baselines, consistent with the router allocating length per problem. Notably, Long-only itself dominates the untrained base, so training helps even under full reasoning. Figure 6: Accuracy versus response length on MATH-500 (mean over three seeds; error bars: std on both axes). The dashed line traces the Pareto frontier of the single-mode baselines (shaded: on or below the frontier). Free routing (ours) lands strictly above it. Up and to the left is better. Table 2: MATH validation accuracy over training (free routing), mean ± std over three seeds, at the checkpoints where all three seeds were evaluated. Step 0 is the untrained base under the routing prompt: accuracy dips while routing is being established, then recovers. Step 0 30 60 90 Accuracy 0.8070.807 0.7270.727 0.7640.764 0.7830.783 ±0.003± 0.003 ±0.012± 0.012 ±0.016± 0.016 ±0.015± 0.015 Δ vs. step 0 −0.080-0.080 −0.043-0.043 −0.024-0.024 To test transfer beyond the training distribution, we evaluate all methods on GSM8K (4) (avg@5) and AIME 2024 and 2025 (avg@16, since the small 30-problem sets are too noisy for a single pass), under the same temperature, prompt, and answer extraction (Figure 7). Two findings stand out. First, the token savings track difficulty. On GSM8K, dominated by easy problems, free routing again lies strictly above the frontier: it slightly exceeds the accuracy of the brief fixed modes at comparable length, using 44%44\% fewer tokens than the frontier requires to reach its accuracy and 76%76\% fewer than the base. On AIME, where nearly all problems require extended reasoning, the router correctly declines to cut budget: it matches the frontier’s accuracy at comparable length, still 12%12\% shorter than the base. The router thus saves the most where problems are heterogeneous enough for adaptive allocation to matter, and preserves budget where they are not. Second, full-reasoning training helps even at fixed length: Long-only matches or slightly exceeds the base on all three benchmarks at fewer tokens, and free routing then trades a little of that accuracy for a large reduction in length. Figure 7: Out-of-distribution accuracy versus response length on GSM8K (top) and AIME 2024+2025 (bottom); error bars: std on both axes. Dashed line and shading as in Figure 6. The above-frontier behavior seen in-distribution transfers zero-shot: free routing exceeds the frontier on GSM8K and matches it on AIME, where nearly all problems demand long reasoning. Discussion and Conclusion We presented three-mode self-routing for GRPO, in which a reasoning model chooses whether to answer quickly, reason briefly, or reason at length by emitting a single routing token. On a small distilled model the reward alone cannot enforce this choice: nothing prevents the policy from emitting a brief-mode token and then reasoning at length, so the label decouples from behavior and one mode takes over, exactly the failure of the two-mode Countdown precursor (in the supplementary appendix). Hard per-mode caps lock the label to behavior and create the genuine accuracy gap the router learns from, while the per-mode discounted reward, balance penalty, and forced warmup keep all three modes alive; together they overcome the collapse that defeats reward-shaping-only designs. The resulting policy routes problems by difficulty, holding validation accuracy near the base model while cutting mean response lengthsubstantially on every benchmark. Routing is best understood as a token-efficiency mechanism, not an accuracy mechanism: in the length/accuracy plane it does not only trade accuracy for tokens but pushes the whole trade-off outward. On two of the three datasets — MATH-500 (Figure 6) and GSM8K (Figure 7) — free routing sits clearly above the Pareto front traced by the single-mode baselines, delivering more accuracy per token than any fixed mode. On the third, AIME (Figure 7), whose problems genuinely require extended reasoning, the router sends nearly everything to Long and the policy lands on the Pareto front. Limitations and future work. The results cover three seeds of one 1.5B model on one mathematical training distribution. The mode caps are task-specific and must be matched to the target task’s length distribution, and uncapped validation makes brief-mode accuracy marginally optimistic relative to the trained objective. The most important next step is therefore breadth: establishing the mechanism on a wider range of mathematical benchmarks and then beyond mathematics, wherever problems differ in how much reasoning they require such as code generation, agentic and tool-use settings, etc. References Agentica Team (2025) Agentica Team DeepScaleR: surpassing o1-preview with a 1.5b model by scaling rl. Note: https://agentica.org/deepscalerAccessed: 2026-06-17 Cited by: Introduction. Ayoub et al. (2025) A. Ayoub, K. Asadi, D. Schuurmans, C. Szepesvári, and K. Bouyarmane Learning to reason efficiently with discounted reinforcement learning. External Links: 2510.23486, Link Cited by: Introduction, Discounting in GRPO, The Normalization-Amplification Problem. Chen et al. (2024) X. Chen, J. Xu, T. Liang, Z. He, J. Pang, D. Yu, L. Song, Q. Liu, M. Zhou, Z. Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187. Cited by: Introduction. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: Datasets and base model., Token Efficiency, In and Out of Distribution. Fang et al. (2025) G. Fang, X. Ma, and X. Wang Thinkless: llm learns when to think. arXiv preprint arXiv:2505.13379. Cited by: Adaptive Reasoning-Mode Selection. Fedus et al. (2022) W. Fedus, B. Zoph, and N. Shazeer Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. Cited by: Introduction, Advantage Estimation and the Balance Term. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: Introduction, Datasets and base model.. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Cited by: Datasets and base model.. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations, Vol. 2024, p. 39578–39601. Cited by: Datasets and base model.. Liu et al. (2025) Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin Understanding r1-zero-like training: a critical perspective. External Links: 2503.20783, Link Cited by: Reinforcement Learning for Reasoning. Parthasarathi et al. (2025) P. Parthasarathi, M. Reymond, B. Chen, Y. Cui, and S. Chandar GRPO-λ: credit assignment improves llm reasoning. External Links: 2510.00194, Link Cited by: Introduction, Discounting in GRPO. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: Introduction, Reinforcement Learning for Reasoning. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: Introduction, Reinforcement Learning for Reasoning, Loss Aggregation. Sun et al. (2025) C. Sun, D. Fulop, and X. Li Countdown to brilliance: evolving math reasoning through train and test time compute. Note: Stanford CS224R Final ReportUses the Jiayi-Pan Countdown-Tasks-3to4 dataset External Links: Link Cited by: Datasets and base model.. Tu et al. (2025) S. Tu, J. Lin, Q. Zhang, X. Tian, L. Li, X. Lan, and D. Zhao Learning when to think: shaping adaptive reasoning in r1-style models via multi-stage rl. arXiv preprint arXiv:2505.10832. Cited by: Adaptive Reasoning-Mode Selection. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, p. 24824–24837. Cited by: Introduction. Wu et al. (2025) S. Wu, J. Xie, Y. Zhang, A. Chen, K. Zhang, Y. Su, and Y. Xiao ARM: adaptive reasoning model. External Links: 2505.20258, Link Cited by: Adaptive Reasoning-Mode Selection. Yang et al. (2026) Z. Yang, Y. Jiang, T. Chen, L. Yang, A. Wong, C. Gao, J. E. Kooi, Z. Li, J. Shi, K. Qiu, Q. Huang, X. Zu, S. Yang, H. Zhang, N. Wong, F. Ilievski, S. Yu, A. Plaat, Z. Ren, M. Hoogendoorn, and V. François-Lavet Modularized reinforcement learning on llms: from mdp creation to exploration and learning. External Links: 2606.21943, Document, Link Cited by: Introduction, Reinforcement Learning for Reasoning. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. DAPO: an open-source llm reinforcement learning system at scale. External Links: 2503.14476, Link Cited by: Reinforcement Learning for Reasoning, The GRPO Objective, Loss Aggregation. Yue et al. (2025) Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, and G. Huang Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. External Links: 2504.13837, Link Cited by: Introduction. Zhang et al. (2025) J. Zhang, N. Lin, L. Hou, L. Feng, and J. Li AdaptThink: reasoning models can learn when to think. arXiv preprint arXiv:2505.13417. Cited by: Adaptive Reasoning-Mode Selection. Zhang and Zuo (2025) J. Zhang and C. Zuo GRPO-lead: a difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models. External Links: 2504.09696, Link Cited by: Introduction, Discounting in GRPO. Zhang et al. (2024) Y. Zhang, S. S. S. Das, and R. Zhang Verbosity is not veracity: demystify verbosity compensation behavior of large language models. arXiv preprint arXiv:2411.07858. Cited by: Introduction. Appendix This appendix (i) documents the Countdown negative result that motivated the hard per-mode caps (App. A); (i) derives the reward surface, advantage estimator, balance term, and rollout protocol (App. B); (i) lists the full hyperparameters and training schedule (App. C); and (iv) reports per-benchmark and per-mode results (App. D). All numbers are the mean over three seeds; ± values and error bars are standard deviations across seeds. Appendix A Binary Self-Routing on Countdown (Negative Result) A binary self-routing attempt on Countdown, whose collapse motivated the hard per-mode caps, preceded the MATH results. Countdown asks the model to combine given numbers with the four operators (each used once) to reach a target; instances have two difficulty levels and a binary verifiable reward. We use DeepSeek-R1-Distill-Qwen-1.5B throughout. We first attempted self-routing with two modes, Short and Long: a correct Short response earned a premium bS>bLb_S>b_L eroded by a per-token discount, while a correct Long earned a flat bLb_L, defining a crossover length below which Short pays and above which Long pays (Fig. 8). This failed, for a reason in the task rather than the constants. For routing to be learnable the modes must differ in value on some problems — there must be prompts a longer budget solves and a shorter one does not. On Countdown that set is essentially empty: correct solutions are short at both levels (≈300≈\!300 tokens), and unsolved instances are not rescued by more tokens (the model exhausts useful search, not length). Forcing the modes apart with a hard cap left their per-level accuracy nearly identical. Empirically the binary router collapsed within 30–50 steps in every configuration; the direction was set by the constants (no premium → Long; too large a premium → Short), and neither a larger balance coefficient nor intermediate premiums held the split. The routing label stayed decoupled from behavior: even when Short dominated its mean length sat near 1,0001,000 tokens — the same as Long— so the crossover never took effect. This is exactly why the three-mode method adds hard per-mode caps: a capped brief mode is scored wrong once its answer runs past the cap, so the label cannot decouple from behavior and Long always retains a reason to exist. It also motivated moving to MATH, whose difficulty spread gives routing something to learn. Figure 8: Schematic two-mode Countdown reward surface. The realised Short length (≈1,000≈\!1,000, grey band) sat well past the designed crossover L∗L^*, so Short and Long produced near-identical traces and the crossover never biased routing. Appendix B Method Details Reward surface and crossover placement Let L be the number of tokens generated after the routing token. Each mode m has a base bmb_m and per-token discount γm∈(0,1] _m∈(0,1]; a correct answer scores bmγmLb_m _m^L: ri=bNTγNTLicorrect, NoThink,bSγSLicorrect, Short,bLγLLicorrect, Long(γL=1),0incorrect, valid routing,−0.5routing token absent/invalid.r_i= casesb_NT _NT^L_i&correct, NoThink,\\ b_S _S^L_i&correct, Short,\\ b_L _L^L_i&correct, Long\ ( _L\!=\!1),\\ 0&incorrect, valid routing,\\ -0.5&routing token absent/invalid. cases (7) The convention is γLγ^L, not e−αLe^-α L. To place the crossover between adjacent modes at the cap of the briefer one, solve bSγSCS=bLb_S _S^C_S=b_L and bNTγNTCNT=bSγSCNTb_NT _NT^C_NT=b_S _S^C_NT: γS=(bLbS)1/CS,γNT=γS(bSbNT)1/CNT. _S= ( b_Lb_S )^1/C_S, _NT= _S ( b_Sb_NT )^1/C_NT. (8) With bNT/bS/bL=1.3/1.2/1.0b_NT/b_S/b_L=1.3/1.2/1.0 and caps CNT/CS=1024/3000C_NT/C_S=1024/3000 this gives γNT≈0.99984 _NT\!≈\!0.99984, γS≈0.99994 _S\!≈\!0.99994 and crossovers LNT∗≈800L^*_NT\!≈\!800, LS∗≈3000L^*_S\!≈\!3000 (Fig. 9, Table 3). Each mode is thus reward-optimal over exactly one length band. Table 3: Correct-answer reward by mode at selected lengths L. Crossovers (bold) mark where the optimal mode changes. L NoThink Short Long 0 1.30 1.20 1.00 256 1.25 1.18 1.00 800 1.14 1.14 1.00 3000 0.80 1.00 1.00 8192 0.35 0.73 1.00 Figure 9: Reward surface with the run gammas. Dashed verticals mark the analytic crossovers; dotted verticals the per-mode caps. Advantage estimation We use GRPO mean-centering without std-normalization, A^i=Ri−μg A_i=R_i- _g. The bases in Eq. (7) are heterogeneous across modes; dividing by the within-group std would amplify the tiny reward differences that arise when all rollouts in a group are correct into full-magnitude advantages (a normalization-amplification pathology, since longer correct traces score slightly lower). Invalid-routing rollouts take the constant −0.5-0.5 penalty. Their inclusion in the group mean is a deliberate choice: the downward drag they exert gives every forced routing token a positive advantage during warmup, which is what bootstraps routing-word emission; once routing exists the caps and balance term carry the signal. Balance term Mean-centering cancels any reward-level signal in an unmixed group (all rollouts pick the same mode) — precisely the collapse regime. We therefore add a correction to the advantage, after centering: A^i←A^i+βbal(p⋆−fmode⁡(i)), A_i← A_i+ _bal (p -f_mode(i) ), (9) where fmodef_mode is the fraction of free rollouts choosing that mode and p⋆=1/3p =1/3. Over-represented modes are nudged down, under-represented up. The fractions and nudge cover free rollouts only (forced rollouts are balanced by construction and would dilute the signal during warmup). Two gates keep the term consistent with the reward: an encourage (positive) nudge is applied only to correct rollouts and a discourage (negative) nudge only to incorrect ones, so balancing never rewards a wrong answer nor penalizes a right one; and a wrong Long attempt is exempted, since Long handles the hardest problems and discouraging it there would push hard problems away from it. βbal _bal is annealed to 00 once routing is established, to verify the split is calibrated to difficulty rather than held at 1/31/3 by the term. Rollout protocol and warmup The base model never emits a routing token unprompted (the chat template prefills <think>), so generation runs in phases. During warmup (⌊n/4⌋ n/4 rollouts per mode are forced — the routing question and that mode’s token are appended after the assistant prefix so vLLM generates in-mode — and the rest are free; after warmup all n are free. After a forced generation the routing token sits at the prompt tail; we promote it into the response head so it receives gradient like any generated token (with P the prompt, Q the routing question, t the routing token): [P∣Q∣t]|gen→[P∣Q]|[t∣gen].[\,P Q t\,]\,\|\,gen\ →\ [\,P Q\,]\,\|\,[\,t \,]. Together with the −0.5-0.5 penalty on rollouts that emit no routing token, this teaches the model to emit the routing word itself; the first free routing tokens appear around step 20. Mode detection and per-mode caps The mode is read from the leading tokens of each response, token-id first (exact match, reliable for the promoted forced tokens) with a prefix-regex fallback for free spelling variants; matching is a prefix match with no trailing word boundary, so continuations such as “LongI need…” are not misread. Per-mode caps are applied before reward and training: a response whose answer falls beyond its mode’s cap is masked and scored wrong, creating the capability gap the router learns from. The cap counts tokens after the routing token (matching L), and forced labels are trusted rather than re-detected. Validation is uncapped, measuring production behavior — which makes brief-mode validation accuracy marginally optimistic relative to the capped objective. Appendix C Experimental Details and Hyperparameters Datasets, model, evaluation. We train on the MATH-lighteval split and validate on held-out MATH-500; a response is correct iff its extracted final answer matches the reference (binary reward), on which Eq. (7) is applied. Out-of-distribution evaluation uses GSM8K (1,3191,319 problems, avg@5) and AIME 2024++2025 (3030 each, avg@16), under the same temperature (0.60.6), prompt, and extraction. The base model is DeepSeek-R1-Distill-Qwen-1.5B. We train three seeds of the router and of each single-mode baseline; the latter force every rollout into one mode with the same data and hyperparameters but no routing token, balance term, or free rollouts. Hyperparameters and schedule. Table 4 lists the settings used. Training runs in two configurations: phase 1 (steps →600\!→\!60) uses the shaped bases/gammas with the balance coefficient annealed to 00 over steps →8560\!→\!85; phase 2 (→9060\!→\!90) flattens the reward (all bases 1.01.0, all γ=1.0γ=1.0), so mode separation is then maintained by the hard caps and the balance term alone — a test that the split does not depend on the reward gradient once routing exists. Table 4: Hyperparameters for three-mode self-routing GRPO. Phase 2 flattens the reward; caps and balance keep the modes distinct. Parameter Value Bases bNT/bS/bLb_NT/b_S/b_L (phase 1) 1.3/1.2/1.01.3/1.2/1.0 Discounts γNT/γS/γL _NT/ _S/ _L 0.99984/0.99994/1.00.99984/0.99994/1.0 Caps CNT/CSC_NT/C_S / context 1024/3000/163841024/3000/16384 Wrong / absent-routing penalty 0.00.0 / −0.5-0.5 Std-normalization disabled Balance coef. / target 1.01.0 / 1/31/3 Balance anneal (start/end) 60/85→060/85→ 0 Group size G 88 Warmup steps 4545 Train batch (prompts) 128128 Mini-batch / micro-batch 32/832/8 Loss aggregation token-mean KL penalty βKL _KL 00 Learning rate (Adam) 1×10−61× 10^-6 Max prompt / response len. 2048/163842048/16384 Total steps / validate every 90/1090/10 Rollout backend vLLM Val. temp. / n / batch 0.6/1/2560.6/1/256 GPUs 4×4× H100 Appendix D Supplementary Results Per-benchmark accuracy and length. Table 5 reports final accuracy and mean length for the router, the three single-mode baselines, and the untrained base. The router stays within one to two points of the base and the Long-only ceiling while cutting length sharply, and the saving scales with benchmark easiness (Table 6): 4.2×4.2× on GSM8K, 1.7×1.7× on MATH-500, and 1.1×1.1× on AIME, whose problems genuinely require extended reasoning. The mechanism is visible in the realised length (Fig. 10): the router’s mean length collapses onto the NoThink-only anchor on the easy GSM8K and onto the Long-only anchor on the hard AIME, with MATH-500 in between — i.e. it sends nearly all of GSM8K to NoThink and nearly all of AIME to Long, difficulty-adaptive out of distribution. Accuracy dips while routing is established and recovers by step 90 (→0.7830.807\!→\!0.727\!→\!0.764\!→\!0.783 at steps 0/30/60/900/30/60/90); routing is a token-efficiency mechanism, not an accuracy one. Table 5: Final accuracy and mean response length (tokens), mean ± std over three seeds. MATH-500 in-distribution; GSM8K/AIME out-of-distribution. Base is the untrained model under the routing prompt. Accuracy Mean length (tokens) MATH-500 GSM8K AIME’24 AIME’25 MATH-500 GSM8K AIME (comb.) Base (no routing) 0.796±.0020.796 ±.002 0.831±.0060.831 ±.006 0.265±.0080.265 ±.008 0.231±.0130.231 ±.013 4743±384743 ± 38 1930±181930 ± 18 12266±2612266 ± 26 NoThink-only 0.668±.0130.668 ±.013 0.757±.0010.757 ±.001 0.137±.0100.137 ±.010 0.105±.0180.105 ±.018 1096±1411096 ± 141 336±52336 ± 52 3768±4533768 ± 453 Short-only 0.727±.0070.727 ±.007 0.756±.0040.756 ±.004 0.218±.0130.218 ±.013 0.166±.0280.166 ±.028 2030±1492030 ± 149 440±13440 ± 13 7259±4707259 ± 470 Long-only 0.802±.0040.802 ±.004 0.837±.0070.837 ±.007 0.297±.0070.297 ±.007 0.230±.0110.230 ±.011 4507±884507 ± 88 1665±541665 ± 54 11881±5611881 ± 56 Free routing (ours) 0.782±.0060.782 ±.006 0.781±.0050.781 ±.005 0.276±.0100.276 ±.010 0.222±.0050.222 ±.005 2810±592810 ± 59 459±18459 ± 18 10716±7410716 ± 74 Table 6: Token reduction of free routing vs. the base model. Base Router Reduction GSM8K 19301930 459459 76%76\% (4.2×4.2×) MATH-500 47434743 28102810 41%41\% (1.7×1.7×) AIME 1226612266 1071610716 13%13\% (1.1×1.1×) Figure 10: Router mean response length per benchmark (black diamond, ± ) against the three single-mode anchors (coloured ticks), benchmarks ordered easy→ (log scale). The router hugs the NoThink anchor on GSM8K and the Long anchor on AIME — realised length is a proxy for the routing split, which the released logs do not record per OOD benchmark. Training dynamics. Figure 11 traces the quantities behind the stability and meaningfulness claims. By step 90 the free split settles near 20/32/47%20/32/47\% (NoThink/Short/Long) with routing entropy H≈1.04H\!≈\!1.04 against the maximum ln≈1.10 3\!≈\!1.10; all three modes persist across seeds, Long remaining the plurality and lengthening toward the context limit while the brief modes stay within their caps. Finally, though the model never sees the MATH-500 difficulty levels in training, the free-validation routing split is monotone in them and sharpens over training (Fig. 12) — independent evidence that routing tracks real difficulty. Figure 11: Free-rollout training dynamics (mean over three seeds, std bands). (a) Mode shares emerge after warmup and persist. (b) Routing entropy approaches ln⁡3 3. (c) Long lengthens while the brief modes stay capped. Figure 12: Free-validation routing share by MATH-500 difficulty level at three checkpoints. The easy→ , hard→ gradient emerges over training.

Instant research discovery