Paper deep dive

Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation

Gijs Kassenaar, Zhao Yang, Vincent François-Lavet

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 56

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%

Last extracted: 8/21/2026, 4:29:52 AM

Summary

This paper introduces an adaptive reasoning framework for language models that dynamically allocates test-time compute by selecting one of three reasoning modes (NoThink, Short, Long) as the first token of the response. Using Group Relative Policy Optimization (GRPO) without a separate router, the model learns to route problems by difficulty through shaped rewards and hard token caps. The approach significantly reduces mean response length (41% reduction) while maintaining accuracy on MATH500 and transferring effectively to other benchmarks like GSM8K.

Entities (8)

Long · reasoning-mode · 99%NoThink · reasoning-mode · 99%Short · reasoning-mode · 99%DeepSeek-R1-Distill-Qwen-1.5B · model · 98%GRPO · algorithm · 98%GSM8K · dataset · 97%MATH500 · dataset · 97%Routing collapse · problem · 95%

Relation Signals (8)

Adaptive Reasoning Framework → definesmodes → NoThink

confidence 98% · choosing, as the first token of its response, one of three modes: NoThink

Adaptive Reasoning Framework → definesmodes → Short

confidence 98% · choosing, as the first token of its response, one of three modes: Short

Adaptive Reasoning Framework → definesmodes → Long

confidence 98% · choosing, as the first token of its response, one of three modes: Long

Adaptive Reasoning Framework → evaluateson → MATH500

confidence 95% · stays close to the base model's accuracy on the held-out MATH500

Adaptive Reasoning Framework → evaluateson → GSM8K

confidence 95% · transfers to other benchmarks... 76% token reduction on GSM8K

Adaptive Reasoning Framework → trains → DeepSeek-R1-Distill-Qwen-1.5B

confidence 95% · On a 1.5B distilled model trained on MATH

Adaptive Reasoning Framework → uses → GRPO

confidence 95% · The choice is learned inside Group Relative Policy Optimization (GRPO)

Adaptive Reasoning Framework → mitigates → Routing collapse

confidence 90% · Our method overcomes collapse with two mechanisms working together

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Reasoning language models trained with reinforcement learning typically operate under a fixed token budget rather than an explicitly adaptive one, which can lead to over-computation on easy problems and insufficient computation on difficult ones. We study whether a model can learn to allocate its own reasoning effort by choosing, as the first token of its response, one of three modes: \textsc{NoThink} (answer as quickly as possible), \textsc{Short} (brief reasoning), or \textsc{Long} (extended reasoning). The choice is learned inside Group Relative Policy Optimization (GRPO) with no separate router, through a shaped reward that makes each mode worthwhile at a different response length, together with hard per-mode token caps that keep the modes distinct. On a 1.5B distilled model trained on MATH, the three modes emerge without collapsing to a single choice, and the brief modes end up more accurate than \textsc{Long}, which shows that the router sorts problems by difficulty rather than at random. Averaged over three seeds, the resulting policy stays close to the base model's accuracy on the held-out MATH500 ($0.782$ vs.\ $0.796$) while cutting the mean response length from $4{,}796$ to $2{,}811$ tokens (a $41\%$ reduction). Interestingly, it also transfers to other benchmarks without retraining, with the largest savings where problems are easier, with for instance 76\% token reduction on GSM8K and at higher accuracy than the baselines at similar response length. In short, we build a reasoning model that adaptively chooses how much to reason for each problem.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

52,431 characters extracted from source content.

Expand or collapse full text

Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation Gijs Kassenaar, Zhao Yang ∗ , Vincent François-Lavet Vrije Universiteit Amsterdam g.kassenaar@student.vu.nl, z.yang3@vu.nl, vincent.francoislavet@vu.nl Abstract Reasoning language models trained with reinforcement learn- ing typically operate under a fixed token budget rather than an explicitly adaptive one, which can lead to over-computation on easy problems and insufficient computation on difficult ones. We study whether a model can learn to allocate its own reason- ing effort by choosing, as the first token of its response, one of three modes: NoThink (answer as quickly as possible), Short (brief reasoning), or Long (extended reasoning). The choice is learned inside Group Relative Policy Optimization (GRPO) with no separate router, through a shaped reward that makes each mode worthwhile at a different response length, together with hard per-mode token caps that keep the modes distinct. On a 1.5B distilled model trained on MATH, the three modes emerge without collapsing to a single choice, and the brief modes end up more accurate than Long, which shows that the router sorts problems by difficulty rather than at random. Av- eraged over three seeds, the resulting policy stays close to the base model’s accuracy on the held-out MATH500 (0.782 vs. 0.796) while cutting the mean response length from 4,796 to 2,811 tokens (a 41% reduction). Interestingly, it also transfers to other benchmarks without retraining, with the largest sav- ings where problems are easier, with for instance 76% token reduction on GSM8K and at higher accuracy than the base- lines at similar response length. In short, we build a reasoning model that adaptively chooses how much to reason for each problem. Introduction Large language models (LLMs) have achieved remarkable performance on complex reasoning tasks, driven largely by Chain-of-Thought (CoT) prompting (Wei et al. 2022). By breaking problems into intermediate steps, models such as DeepSeek-R1 (Guo et al. 2025) reach state-of-the-art results across mathematical and coding benchmarks. This capability, however, comes at a substantial computational cost: extended reasoning chains consume large numbers of tokens, increas- ing both inference latency and training cost. The burden falls hardest on smaller models, which often exhibit “ver- bosity compensation,” generating unnecessarily long expla- nations to compensate for limited capacity (Chen et al. 2024; Zhang, Das, and Zhang 2024). Empirically, incorrect an- swers tend to be markedly longer than correct ones, indicat- ing that current models frequently spend reasoning tokens inefficiently (Agentica Team 2025). ∗ Corresponding Author Input Prompt (e.g., MATH question) Appended Instruction: “Output NOTHINK to answer directly, SHORT for brief rea- soning, or LONG for extended reasoning.” Main LLM Policy (No Separate Router) Generates First Token SHORTNOTHINKLONG Easy Medium Hard Behavior Direct Answer Budget Limit 1,024 Tokens Reward Shape High Base, Steep Length Penalty Behavior Brief Reasoning Budget Limit 3,000 Tokens Reward Shape Medium Base, Mild Length Penalty Behavior Extended Search Budget Limit Uncapped Reward Shape Standard Base, No Penalty Unified GRPO Objective (Includes Advantage Load Balanc- ing to maintain mode distribution) End-to-End Gradient Update Figure 1: Three-mode self-routing GRPO. The model emits a routing token (NoThink/Short/Long) as the first token of its response and then reasons under that mode’s shaped reward and hard token cap; the routing token is part of the main policy and is trained end-to-end by GRPO, with no separate router. Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for fine-tuning reason- ing models (Shao et al. 2024). Standard algorithms such as PPO (Schulman et al. 2017) and GRPO (Shao et al. 2024) operate with a single maximum response length applied uni- formly across all training samples. This couples two compet- ing inefficiencies: early in training, the model wastes com- pute generating long, incorrect traces; later, the same fixed ceiling can prevent the extended chains needed to solve gen- uinely hard problems. Recent analysis further suggests that arXiv:2608.20256v1 [cs.AI] 20 Aug 2026 RLVR functions primarily as an elicitation mechanism, im- proving pass@k for small k but remaining bounded by the base model’s latent capability for large k (Yue et al. 2025). A principled response to the length problem is temporal discounting: scaling reward by a per-token factor penalizes long rollouts directly, and discount-based variants of GRPO have been shown to shorten reasoning while preserving ac- curacy (Parthasarathi et al. 2025; Ayoub et al. 2025; Zhang and Zuo 2025; Yang et al. 2026). Discounting, however, is non-adaptive: it imposes a fixed length penalty regardless of a prompt’s difficulty or the response length its solution actually requires. An easy question and a hard question are pressured to be equally brief, even though the optimal reasoning budget differs. The natural next step is to let the model decide, per prompt, how much reasoning a problem warrants, applying brevity pressure only where it is justified. We study this question directly. We ask a 1.5B distilled reasoning model to emit, as the very first token of its response, one of three discrete routing decisions—NoThink, Short, or Long—and then to reason accordingly. The routing token is part of the main policy and receives gradient through the standard GRPO objective, there is no separate router network. The central difficulty is routing collapse: because the routing decision is a single self-reinforcing discrete token, a mode with even a small reward edge is sampled more, reinforced more, and quickly drives the alternatives out of the policy. Our method overcomes collapse with two mechanisms working together. First, hard per-mode token caps make the modes distinct by construction, so the routing label can- not decouple from behavior. Second, a simple balance term added directly to the advantage, similar to the load-balancing penalties used to discourage expert collapse in Mixture-of- Experts routing (Fedus, Zoph, and Shazeer 2022), keeps the policy from settling on a single mode. A forced-rollout warmup stabilizes the early dynamics. Our contributions are twofold. First, we show that a 1.5B reasoning model can be taught to route problems by difficulty, choosing among three reasoning modes as its first generated token, end-to-end inside GRPO, with no separate router and no supervised warm-up. Second, we show that this routing cuts the reasoning-token budget substantially while placing the policy on or above the accuracy–length Pareto frontier traced by the single-mode baselines — both in-distribution on MATH and on held-out benchmarks. As part of the anal- ysis, a negative result on the Countdown task explains when difficulty routing is and is not learnable. Related Work Reinforcement Learning for Reasoning RL fine-tuning of LLMs (Yang et al. 2026) began with Prox- imal Policy Optimization (PPO), which uses a clipped surro- gate objective for stable updates (Schulman et al. 2017). To remove the memory overhead of PPO’s value network, Group Relative Policy Optimization (GRPO) computes advantages relative to a group of sampled responses (Shao et al. 2024). While efficient, GRPO carries optimization biases that recent work has sought to correct. Liu et al. (2025) identify a “length bias” arising from dividing each response’s loss by its token length, which systematically favors longer incorrect answers and drives “length explosion.” Their Dr. GRPO removes the two terms they hold responsible: the per-response length normalization in the loss and the within-group standard- deviation scaling of the advantage, yielding updates that no longer reward verbosity. DAPO (Yu et al. 2025) contributes two ingredients we adopt: a token-level loss aggregation that prevents long traces from being under-weighted, and the ob- servation that the KL penalty toward the reference policy is unnecessary—and indeed counterproductive—once the pol- icy is allowed to move far from its initialization during long- horizon reasoning RL. Discounting in GRPO Because GRPO uses no value function, it normally carries no temporal discount, but a discount is a principled way to penalize length and several recent methods add one. They differ in where the discount appears , which changes the resulting group advantages substantially. The discounted- reasoning method (Ayoub et al. 2025) multiplies the se- quence reward by γ K , with K the reasoning-token count, before group normalization; this couples advantage magni- tude to absolute rollout length, so groups of long rollouts and groups of short rollouts are scaled differently. GRPO- LEAD (Zhang and Zuo 2025) also shapes the reward before normalization, but standardizes length within the group via a z-score and adds difficulty-aware weighting, which makes the relative advantages comparable across groups regard- less of nominal length. GRPO-λ (Parthasarathi et al. 2025) instead applies its decay after normalization, tracing credit backward over token positions so that later tokens, which are often most important for answer, receive larger-magnitude updates while the sign is inherited from the group outcome. Our reward surface builds on the pre-normalization γ K for- mulation, but rather than applying one global discount we attach a separate base reward and discount to each reason- ing mode, turning a uniform length penalty into a per-mode incentive. Adaptive Reasoning-Mode Selection A related line of work shortens reasoning by skipping or ab- breviating it according to problem difficulty, and the methods differ in how much training machinery they require. Adapt- Think (Zhang et al. 2025) and AutoThink (Tu et al. 2025) stay within pure RL—the former choosing between a full- reasoning and a no-reasoning mode via a constrained objec- tive with importance sampling for cold start, the latter using a multi-stage curriculum—each roughly halving response length on DeepSeek-R1-Distill-Qwen-1.5B while preserving or improving accuracy. ARM (Wu et al. 2025) and Think- less (Fang, Ma, and Wang 2025) instead pre-install the modes with a supervised stage before RL: ARM supervised-fine- tunes on data annotated in four formats (direct answer, short and long chain-of-thought, and code), and Thinkless distills a reasoning expert and an instruction expert into two control tokens, then runs RL with a decoupled objective so the lone control token is not swamped by the response tokens. A difficulty shared by all of these methods is routing col- lapse: because the mode decision is a single discrete, self- reinforcing choice, a mode with even a small reward ad- vantage is sampled more, reinforced more, and drives the alternatives out of the policy. The all-one-mode states are stable under policy gradient while the mixed split is not, an effect that is stronger on a small distilled model with low output entropy and a strong prior to reason at length. The methods above stabilize the choice with importance sampling (AdaptThink), curriculum staging (AutoThink), or a super- vised stage that pre-installs the modes (ARM, Thinkless). Ours is the leanest of these—no supervised or distillation stage, three modes learned end-to-end with RL—relying in- stead on a forced warmup, hard per-mode caps, and a balance penalty on the advantage to keep all modes alive. Preliminaries The GRPO Objective GRPO optimizes a policy by sampling a group of G outputs o 1 ,...,o G for each prompt q from the current policy π θ old . It replaces the critic with a group-relative advantage A i , obtained by normalizing rewards r i within the group: A i = r i − mean(r 1 ,...,r G ) std(r 1 ,...,r G ) + ε .(1) The policy maximizes a clipped surrogate objective with a KL penalty, J GRPO (θ) =E " 1 G G X i=1 1 N i N i X t=1 m i,t − β D KL # , (2) where m i,t = min ρ i,t A i , clip(ρ i,t , 1 − ε, 1 + ε)A i is the clipped surrogate term, D KL = D KL (π θ ∥π ref ), and ρ i,t = π θ (o i,t | q,o i,<t )/π θ old (o i,t | q,o i,<t ) is the token- level importance ratio, with ε the clipping threshold and β the penalty strength toward a reference policy π ref . We set β = 0 throughout, removing the KL penalty en- tirely. The KL term keeps the policy close to the reference model: while this stabilizes ordinary RLHF, it limits the ex- ploration needed to find new reasoning strategies, and in long-horizon reasoning RL it becomes unnecessary and can be harmful (Yu et al. 2025). This matters more for routing, which requires the policy to explore new behaviors such as answering quickly or stopping its reasoning early, behaviors the distilled base model rarely produces; a term that anchors the policy to that base model would work against this explo- ration. Loss Aggregation The advantage A i is a sequence-level quantity, but the loss is summed over tokens, and how token losses are reduced to a scalar affects length dynamics. The default DeepSeek formulation (Shao et al. 2024) uses sequence- mean, token-mean (SMTM) aggregation—averaging tokens within each sequence, then averaging across the group (the nested 1 G P i 1 N i P t in Eq. (2)). The inner 1 N i weights every sequence equally regardless of length, but as a side effect each token in a long sequence contributes a smaller gradi- ent than a token in a short one. We instead use token-mean aggregation (Yu et al. 2025), which normalizes by the total token count P i N i : L TM = 1 P G i=1 N i G X i=1 N i X t=1 ℓ i,t ,(3) so every token carries the same weight regardless of its se- quence. This matters for routing, where the modes delib- erately produce responses of very different lengths: under SMTM the long-mode tokens would receive weaker gradi- ents than the brief-mode tokens, biasing learning away from the mode that handles the hardest problems. The Normalization-Amplification Problem The discounted-reasoning formulation (Ayoub et al. 2025) multiplies the sequence reward by a per-token factor before normalization, R i = γ K i r i ,(4) with K i the reasoning-token count and γ ∈ (0, 1), decaying reward as reasoning grows. This interacts badly with GRPO’s standard-deviation normalization. In vanilla GRPO a group with no reward variance e.g. all rollouts correct, produces all-zero advantages and no learning signal. The γ K discount creates tiny reward differences within such a group (longer correct responses score slightly lower), and the division by std in Eq. (1) amplifies these into full-magnitude advan- tages, comparable to those from a genuinely mixed group— diverting training toward small length optimizations instead of solving harder problems. We avoid this by removing the standard-deviation normalization entirely and keeping only mean-centering, so the advantage is simply ˆ A i = R i − μ g , proportional to the actual reward differences; the small dif- ferences in an all-correct group then stay small. Method Discounting applies the same penalty to every prompt. Our goal is to let the model allocate reasoning effort per prompt: skip reasoning when a problem is trivial, reason briefly when it is easy, and reason at length when it is hard. We realize this by having the model emit a discrete routing token as the first word of its response and conditioning both reward and a hard token budget on that choice. We first explored this idea with a two-mode (Short/Long) router on the Countdown task; it collapsed to a single mode because Countdown’s uni- form difficulty gives the router no accuracy gap to learn from (see the supplementary appendix). That negative result moti- vates the design below: hard per-mode token caps that make the modes behaviorally distinct, and evaluation on MATH, whose wide spread of difficulty gives routing something to learn. Three Modes with Hard Token Caps We define three modes: • NoThink: the model answers as quickly as possible, us- ing little reasoning. It is not forced to skip reasoning and may still think briefly, but it is pushed toward a short answer; intended for the easiest problems. • Short: the model reasons briefly within a hard token cap, intended for easy problems that benefit from a short trace. • Long: the model reasons at length without a cap, intended for hard problems that need extended search. The routing instruction appended to every prompt at runtime is: “Output NoThink to answer directly, Short for brief rea- soning, or Long for extended reasoning.” The routing token sits in the response head and receives gradient like any other token; no data preprocessing is required. The per-mode reward and its discount γ, defined in the next subsection, are the primary mechanism that separates the modes: they make each mode the reward-optimal choice over a different range of response lengths. The hard per- mode caps play two supporting roles. First, they keep the policy from collapsing onto a single mode: because a capped brief mode is scored wrong as soon as its answer runs past the cap, it genuinely fails on problems that need more to- kens, so Long always retains a reason to exist and the brief modes cannot quietly absorb hard problems. Second, they save compute during training, since rollouts in the capped modes generate far fewer tokens than uncapped reasoning. Concretely, a response whose final answer token falls be- yond its mode’s cap is scored incorrect regardless of content; the cap is enforced during generation. Default caps are 1,024 tokens for NoThink and 3,000 for Short; Long is uncapped. Reward Surface Each mode has a base reward b and a per-token discount γ on the number of tokens L i generated after the routing token: r i =              b NT γ L i NT correct, NoThink b S γ L i S correct, Short b L correct, Long 0.0incorrect, valid routing −0.5routing token absent. (5) The discounts γ NT and γ S are set so that each mode is the reward-optimal choice over the length range it is designed for, with the crossover between adjacent modes falling at or below the cap of the briefer one. With bases b NT = 1.3, b S = 1.2, b L = 1.0, the NoThink/Short crossover falls at L ≈ 800 and the Short/Long crossover at L≈ 3000, the Short cap. Table 3 lists the resulting rewards and Figure 9 plots the curves, the two crossovers marking where the optimal mode changes. Advantage Estimation and the Balance Term We use GRPO mean-centering across the group but dis- able standard-deviation normalization, for the reason given in Section : the bases in Eq. (7) are intentionally heteroge- neous across modes, and normalizing by within-group std would amplify the small reward differences that arise when all rollouts are correct. Mean-centering cancels any signal injected at the reward level when a group is unmixed (all rollouts pick the same Table 1: Correct-answer reward by mode at selected response lengths L, under default bases and caps. Crossovers occur where two modes tie (bold). LNoThinkShortLong 01.301.20 1.00 2561.251.18 1.00 8001.141.14 1.00 30000.801.00 1.00 81920.350.73 1.00 010002000300040005000600070008000 Tokens generated after routing token, L 0.4 0.6 0.8 1.0 1.2 1.4 Correct-answer reward L * NT 800 L * S 3000 NoThink Short Long Figure 2: Correct-answer reward for each mode as a function of the number of tokens L generated after the routing to- ken, under the bases and caps of Table 4. NoThink (1.3γ L NT ) and Short (1.2γ L S ) decay with length while Long is flat at 1.0. The discounts place the crossovers at L ∗ NT ≈ 800 (NoThink/Short) and L ∗ S ≈ 3000 (Short/Long), at or be- low the mode caps. Each mode is reward-optimal in its own length band. mode), which is the regime in which collapse occurs. To keep pressure toward a balanced split, we add a simple correction directly to the advantage, after centering: ˆ A i ← A i + β bal p ⋆ − f mode(i) ,(6) where f mode is the fraction of free rollouts choosing that mode,p ⋆ = 1/3 is the target share, andβ bal is the balance co- efficient. A mode that is over-represented (its fraction above p ⋆ ) has its advantage reduced, while an under-represented mode has its advantage raised. To keep the balance term from distorting the accuracy signal, we apply it asymmetrically by outcome: a negative correction (for an over-represented mode) is applied only to incorrect rollouts, and a positive correction (for an under-represented mode) only to correct rollouts. Balancing therefore never penalizes a correct an- swer or rewards an incorrect one; it redistributes routing mass only in the direction that is consistent with the reward. Because this correction is folded into the advantage, it enters the policy-gradient loss directly rather than as a separate aux- iliary load-balancing loss (Fedus, Zoph, and Shazeer 2022). Because the balance term is added after mean-centering, it Algorithm 1 Three-Mode Self-Routing GRPO (one step) Input: promptsq, policy π θ , step s, caps C NT ,C S Parameter: basesb, discountsγ,β bal ,p ⋆ 1: Append routing instruction to each q 2: if s≤ warmup_steps then 3: Force 1 rollout per mode; remaining free; promote token 4: else 5: Sample G free rollouts per prompt 6: end if 7: Detect mode from first response token; apply caps 8: Compute rewards by Eq. (7) 9: Mean-center advantages (no std-normalization) 10: Add balance term, Eq. (9), over free rollouts 11: Update π θ with the GRPO objective, Eq. (2) remains effective in zero reward variance or routing variance groups. Warmup The base model never emits a routing token unprompted, because the chat template prefills <think> and the model proceeds directly to reasoning. For the first warmup_steps training steps we therefore force two rollout per group into each of NoThink, Short, and Long, generating the remain- ing rollouts freely, and we promote the forced routing token into the response head so that gradient flows through the routing position from the first step.Together with the penalty given to all rollouts that have no routing token, causes the model to quickly learn to route. After warmup, all rollouts are generated freely. Algorithm 1 summarizes one training step. Hyper-parameters used can be found in the supplementary appendix. Experimental Setup Datasets and base model. We train on the MATH- lighteval split of competition mathematics problems with verifiable final answers (Hendrycks et al. 2021) and validate on the held-out MATH-500 set (Lightman et al. 2024) at temperature 0.6. A response is correct if its final answer matches the reference, giving a binary reward (r = 1 cor- rect, r = 0 otherwise) on which the routing shaping of Eq. (7) is applied. For out-of-distribution evaluation we use GSM8K (Cobbe et al. 2021) (1,319 grade-school word prob- lems, avg@5) and AIME 2024 and 2025 (30 competition problems each, avg@16). The preliminary binary-routing study uses the Countdown task (Sun, Fulop, and Li 2025); see the supplementary appendix. We use DeepSeek-R1-Distill-Qwen-1.5B (Guo et al. 2025), a 1.5B-parameter reasoning model distilled from DeepSeek-R1 on roughly 800K high-quality traces. Training. We run GRPO with group size G = 8 for 90 steps from a cold start, with a 16,384-token context cap, on 4 NVIDIA H100 GPUs; hyperparameters are in Table 4. We train three seeds of the router and of each single-mode baseline, reporting mean± std over seeds throughout. Single-mode baselines. To separate adaptive routing from any single mode, we train three ablations in which every rollout is forced to one mode (NoThink-, Short-, or Long- only), with the same data and hyperparameters but no routing token, balance term, or free rollouts; each learns a fixed reasoning budget against which the router is compared on the accuracy versus length plane (Figure 6). Evaluation. We report MATH-500 validation accuracy, the free-rollout mode distribution and routing entropy, per- mode accuracy, and mean response length. Validation gener- ation is uncapped, measuring production behavior under free generation. Experimental Results Through the experiments we would like to answer three ques- tions: (1) Stability: can three reasoning modes be learned end- to-end inside GRPO without collapsing to a single mode? (2) Meaningfulness: does the learned routing sort problems by difficulty, rather than partitioning them arbitrarily? (3) Ef- ficiency: what accuracy and token cost does routing deliver compared to fixed reasoning budgets, and does it transfer be- yond the training distribution? The three subsections below address these questions in turn. Three Modes Emerge and Persist Figure 3 traces the free-rollout mode distribution over train- ing. During the forced warmup free rollouts are almost entirely unknown, with the first routing tokens appearing around step 20. Once routing is free the model at first com- mits heavily to Long, then the brief modes grow as the balance term takes effect, and by step 90 the split settles near 20/32/47% (NoThink/Short/Long), with routing en- tropy close to the theoretical maximum (H ≈ 1.04 vs. ln 3 ≈ 1.10). All three modes persist to the end of train- ing across all seeds, with Long remaining the plurality and no collapse into any single mode. Routing Reflects Problem Difficulty Two independent pieces of evidence show the routing is semantically meaningful rather than an arbitrary partition. The first is the inversion of per-mode accuracy over train- ing (Figure 4). Early on, the forced rollouts reproduce the base model’s prior ordering Long > Short > NoThink, reflecting a model that reasons at length by default. Around step 50 this ordering inverts: NoThink and Short rise above Long and stay there across all seeds, ending well above it at the final checkpoint. Easy problems are being assigned to the brief modes while hard problems concentrate in Long, whose lower accuracy reflects genuine difficulty rather than mode inferiority. Consistently, the Long mean length grows markedly over training while NoThink and Short stay well within their caps. The second is agreement with an external difficulty signal the model never saw in training: the MATH-500 difficulty levels from 1 (easiest) to 5 (hardest). Figure 5 shows the free- validation routing split per level, and the trend is monotone: NoThink carries most of the easiest level but little of the hardest, Long does the reverse, and Short is most used at Figure 3: Free-rollout mode shares over training, mean over three seeds (the forced warmup runs to step 45). After warmup all three modes persist without collapsing to any single mode, with Long remaining the plurality. intermediate difficulty, where a brief trace is most useful. Per- level accuracy falls steadily with level, confirming the labels track real difficulty. The router therefore allocates reasoning budget in proportion to difficulty, using the cheap modes on easy levels for large token savings and reserving Long for the hard ones. 020406080 Training step 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Per-mode accuracy NoThink Short Long Figure 4: Per-mode accuracy over training (mean, with std bands, over three seeds). The forced base-model ordering (Long > Short > NoThink) inverts around step 50, ev- idence that the router assigns problems by difficulty: brief modes get easier problems and Long concentrates the hard ones. Token Efficiency, In and Out of Distribution We evaluate efficiency in two regimes. In-distribution refers to MATH-500: held out from training but drawn from the same competition-mathematics distribution as the MATH- Figure 5: Free-validation routing split (NoThink / Short / Long) by MATH-500 difficulty level, with per-level accuracy above each bar. Mass shifts from NoThink on easy levels to Long on hard ones, with Short most used at intermediate difficulty. lighteval training split. Out-of-distribution refers to GSM8K and AIME, datasets the model never trained on that sit at opposite ends of the difficulty range, far easier and far harder than the training data respectively; they test whether the rout- ing itself transfers. Table 2 reports MATH validation accuracy over training. Accuracy does not rise above the base model: it dips while routing is being established and recovers to 0.783 by step 90, close to the untrained model’s 0.807. The gain is in efficiency: on MATH-500 the router reaches this accuracy at 2,811 to- kens on average, 41% fewer than the base (Figure 6). The reduction is established early, when the brief modes absorb the easy problems, and is then partly offset as Long responses lengthen toward the context limit. At the final checkpoint the per-mode validation accuracies reproduce the training order- ing, so the brief modes are reliably selected for problems the model can solve briefly. Figure 6 places all methods on the accuracy versus length plane. The single-mode baselines trace a Pareto frontier of fixed reasoning budgets, and free routing lies strictly above it: it reaches an accuracy that the frontier only attains with 27% more tokens, while also sitting slightly higher than the frontier at its own length. In other words, the saving holds not only against the base model but against the frontier traced by the three single-mode baselines, consistent with the router al- locating length per problem. Notably, Long-only itself dom- inates the untrained base, so training helps even under full reasoning. To test transfer beyond the training distribution, we eval- uate all methods on GSM8K (Cobbe et al. 2021) (avg@5) and AIME 2024 and 2025 (avg@16, since the small 30- problem sets are too noisy for a single pass), under the same temperature, prompt, and answer extraction (Figure 7). Two findings stand out. First, the token savings track difficulty. On 10002000300040005000 Mean response length (tokens) 0.60 0.65 0.70 0.75 0.80 0.85 MATH-500 accuracy Free routing (ours) 0.78 @ 2,811 Base 0.79 @ 4,796 Long-only 0.80 @ 4,507 Short-only 0.72 @ 2,030 NoThink-only 0.66 @ 1,096 Figure 6: Accuracy versus response length on MATH-500 (mean over three seeds; error bars: std on both axes). The dashed line traces the Pareto frontier of the single-mode baselines (shaded: on or below the frontier). Free routing (ours) lands strictly above it. Up and to the left is better. Table 2: MATH validation accuracy over training (free rout- ing), mean± std over three seeds, at the checkpoints where all three seeds were evaluated. Step 0 is the untrained base under the routing prompt: accuracy dips while routing is be- ing established, then recovers. Step0306090 Accuracy0.8070.7270.7640.783 ±0.003±0.012±0.016±0.015 ∆ vs. step 0−0.080 −0.043 −0.024 GSM8K, dominated by easy problems, free routing again lies strictly above the frontier: it slightly exceeds the accuracy of the brief fixed modes at comparable length, using 44% fewer tokens than the frontier requires to reach its accuracy and 76% fewer than the base. On AIME, where nearly all prob- lems require extended reasoning, the router correctly declines to cut budget: it matches the frontier’s accuracy at compara- ble length, still 12% shorter than the base. The router thus saves the most where problems are heterogeneous enough for adaptive allocation to matter, and preserves budget where they are not. Second, full-reasoning training helps even at fixed length: Long-only matches or slightly exceeds the base on all three benchmarks at fewer tokens, and free routing then trades a little of that accuracy for a large reduction in length. Discussion and Conclusion We presented three-mode self-routing for GRPO, in which a reasoning model chooses whether to answer quickly, rea- son briefly, or reason at length by emitting a single routing token. On a small distilled model the reward alone cannot enforce this choice: nothing prevents the policy from emit- ting a brief-mode token and then reasoning at length, so the label decouples from behavior and one mode takes over, exactly the failure of the two-mode Countdown precursor (in the supplementary appendix). Hard per-mode caps lock the label to behavior and create the genuine accuracy gap 400800120016002000 0.70 0.75 0.80 0.85 0.90 GSM8K accuracy GSM8K (avg@5) 4000600080001000012000 Mean response length (tokens) 0.12 0.16 0.20 0.24 0.28 AIME accuracy AIME 2024+2025 (avg@16) Free routing (ours) Base Long-only Short-only NoThink-only Figure 7: Out-of-distribution accuracy versus response length on GSM8K (top) and AIME 2024+2025 (bottom); er- ror bars: std on both axes. Dashed line and shading as in Fig- ure 6. The above-frontier behavior seen in-distribution trans- fers zero-shot: free routing exceeds the frontier on GSM8K and matches it on AIME, where nearly all problems demand long reasoning. the router learns from, while the per-mode discounted re- ward, balance penalty, and forced warmup keep all three modes alive; together they overcome the collapse that de- feats reward-shaping-only designs. The resulting policy routes problems by difficulty, holding validation accuracy near the base model while cutting mean response lengthsubstantially on every benchmark. Routing is best understood as a token-efficiency mechanism, not an ac- curacy mechanism: in the length/accuracy plane it does not only trade accuracy for tokens but pushes the whole trade-off outward. On two of the three datasets — MATH-500 (Fig- ure 6) and GSM8K (Figure 7) — free routing sits clearly above the Pareto front traced by the single-mode baselines, delivering more accuracy per token than any fixed mode. On the third, AIME (Figure 7), whose problems genuinely re- quire extended reasoning, the router sends nearly everything to Long and the policy lands on the Pareto front. Limitations and future work. The results cover three seeds of one 1.5B model on one mathematical training distri- bution. The mode caps are task-specific and must be matched to the target task’s length distribution, and uncapped valida- tion makes brief-mode accuracy marginally optimistic rela- tive to the trained objective. The most important next step is therefore breadth: establishing the mechanism on a wider range of mathematical benchmarks and then beyond math- ematics, wherever problems differ in how much reasoning they require such as code generation, agentic and tool-use settings, etc. References Agentica Team. 2025. DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL. https://agentica.org/ deepscaler. Accessed: 2026-06-17. Ayoub, A.; Asadi, K.; Schuurmans, D.; Szepesvári, C.; and Bouyarmane, K. 2025. Learning to Reason Efficiently with Discounted Reinforcement Learning. arXiv:2510.23486. Chen, X.; Xu, J.; Liang, T.; He, Z.; Pang, J.; Yu, D.; Song, L.; Liu, Q.; Zhou, M.; Zhang, Z.; et al. 2024. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168. Fang, G.; Ma, X.; and Wang, X. 2025. Thinkless: LLM Learns When to Think. arXiv preprint arXiv:2505.13379. Fedus, W.; Zoph, B.; and Shazeer, N. 2022. Switch Trans- formers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Re- search, 23(120): 1–39. Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforce- ment Learning. arXiv preprint arXiv:2501.12948. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving with the MATH Dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2024. Let’s verify step by step. In International Con- ference on Learning Representations, volume 2024, 39578– 39601. Liu, Z.; Chen, C.; Li, W.; Qi, P.; Pang, T.; Du, C.; Lee, W. S.; and Lin, M. 2025. Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783. Parthasarathi, P.; Reymond, M.; Chen, B.; Cui, Y.; and Chan- dar, S. 2025. GRPO-λ: Credit Assignment Improves LLM Reasoning. arXiv:2510.00194. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347. Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Rea- soning in Open Language Models. arXiv:2402.03300. Sun, C.; Fulop, D.; and Li, X. 2025. Countdown to Bril- liance: Evolving Math Reasoning through Train and Test Time Compute. Stanford CS224R Final Report. Uses the Jiayi-Pan Countdown-Tasks-3to4 dataset. Tu, S.; Lin, J.; Zhang, Q.; Tian, X.; Li, L.; Lan, X.; and Zhao, D. 2025. Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL. arXiv preprint arXiv:2505.10832. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.; and Zhou, D. 2022. Chain-of- Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Sys- tems, 35: 24824–24837. Wu, S.; Xie, J.; Zhang, Y.; Chen, A.; Zhang, K.; Su, Y.; and Xiao, Y. 2025. ARM: Adaptive Reasoning Model. arXiv:2505.20258. Yang, Z.; Jiang, Y.; Chen, T.-C.; Yang, L.; Wong, A.; Gao, C.; Kooi, J. E.; Li, Z.; Shi, J.; Qiu, K.; Huang, Q.; Zu, X.; Yang, S.; Zhang, H.; Wong, N.; Ilievski, F.; Yu, S.; Plaat, A.; Ren, Z.; Hoogendoorn, M.; and François-Lavet, V. 2026. Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning. arXiv:2606.21943. Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Fan, T.; Liu, G.; Liu, L.; Liu, X.; et al. 2025. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476. Yue, Y.; Chen, Z.; Lu, R.; Zhao, A.; Wang, Z.; Yue, Y.; Song, S.; and Huang, G. 2025. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837. Zhang, J.; Lin, N.; Hou, L.; Feng, L.; and Li, J. 2025. Adapt- Think: Reasoning Models Can Learn When to Think. arXiv preprint arXiv:2505.13417. Zhang, J.; and Zuo, C. 2025. GRPO-LEAD: A Difficulty- Aware Reinforcement Learning Approach for Concise Math- ematical Reasoning in Language Models. arXiv:2504.09696. Zhang, Y.; Das, S. S. S.; and Zhang, R. 2024. Verbosity Is Not Veracity: Demystify Verbosity Compensation Behavior of Large Language Models. arXiv preprint arXiv:2411.07858. Appendix This appendix (i) documents the Countdown negative result that motivated the hard per-mode caps (App. A); (i) de- rives the reward surface, advantage estimator, balance term, and rollout protocol (App. B); (i) lists the full hyperpa- rameters and training schedule (App. C); and (iv) reports per-benchmark and per-mode results (App. D). All numbers are the mean over three seeds; ± values and error bars are standard deviations across seeds. A Binary Self-Routing on Countdown (Negative Result) A binary self-routing attempt on Countdown, whose collapse motivated the hard per-mode caps, preceded the MATH re- sults. Countdown asks the model to combine given numbers with the four operators (each used once) to reach a target; instances have two difficulty levels and a binary verifiable re- ward. We use DeepSeek-R1-Distill-Qwen-1.5B throughout. We first attempted self-routing with two modes, Short and Long: a correct Short response earned a premium b S > b L eroded by a per-token discount, while a correct Long earned a flat b L , defining a crossover length below which Short pays and above which Long pays (Fig. 8). This failed, for a reason in the task rather than the con- stants. For routing to be learnable the modes must differ in value on some problems — there must be prompts a longer budget solves and a shorter one does not. On Countdown that set is essentially empty: correct solutions are short at both levels (≈ 300 tokens), and unsolved instances are not rescued by more tokens (the model exhausts useful search, not length). Forcing the modes apart with a hard cap left their per-level accuracy nearly identical. Empirically the binary router collapsed within 30–50 steps in every configuration; the direction was set by the constants (no premium→ Long; too large a premium→ Short), and neither a larger balance coefficient nor intermediate premi- ums held the split. The routing label stayed decoupled from behavior: even when Short dominated its mean length sat near 1,000 tokens — the same as Long— so the crossover never took effect. This is exactly why the three-mode method adds hard per-mode caps: a capped brief mode is scored wrong once its answer runs past the cap, so the label cannot decouple from behavior and Long always retains a reason to exist. It also motivated moving to MATH, whose difficulty spread gives routing something to learn. B Method Details Reward surface and crossover placement Let L be the number of tokens generated after the routing token. Each mode m has a base b m and per-token discount γ m ∈ (0, 1]; a correct answer scores b m γ L m : r i =              b NT γ L i NT correct, NoThink, b S γ L i S correct, Short, b L γ L i L correct, Long (γ L =1), 0incorrect, valid routing, −0.5routing token absent/invalid. (7) 02004006008001000120014001600 Tokens after routing token, L 0.6 0.8 1.0 1.2 1.4 Correct-answer reward designed crossover L * realised SHORT length (1000) Binary Countdown router: the crossover never bites (schematic) SHORT b S L S LONG b L (flat) Figure 8: Schematic two-mode Countdown reward surface. The realised Short length (≈1,000, grey band) sat well past the designed crossover L ∗ , so Short and Long produced near-identical traces and the crossover never biased routing. Table 3: Correct-answer reward by mode at selected lengths L. Crossovers (bold) mark where the optimal mode changes. LNoThinkShortLong 01.301.201.00 2561.251.181.00 8001.141.141.00 30000.801.001.00 81920.350.731.00 The convention is γ L , not e −αL . To place the crossover be- tween adjacent modes at the cap of the briefer one, solve b S γ C S S = b L and b NT γ C NT NT = b S γ C NT S : γ S = b L b S 1/C S , γ NT = γ S b S b NT 1/C NT . (8) With b NT /b S /b L = 1.3/1.2/1.0 and caps C NT /C S = 1024/3000 this gives γ NT ≈ 0.99984, γ S ≈ 0.99994 and crossovers L ∗ NT ≈ 800, L ∗ S ≈ 3000 (Fig. 9, Table 3). Each mode is thus reward-optimal over exactly one length band. Advantage estimation We use GRPO mean-centering without std-normalization, ˆ A i = R i − μ g . The bases in Eq. (7) are heterogeneous across modes; dividing by the within-group std would am- plify the tiny reward differences that arise when all roll- outs in a group are correct into full-magnitude advantages (a normalization-amplification pathology, since longer correct traces score slightly lower). Invalid-routing rollouts take the constant−0.5 penalty. Their inclusion in the group mean is a deliberate choice: the downward drag they exert gives every forced routing token a positive advantage during warmup, which is what bootstraps routing-word emission; once rout- ing exists the caps and balance term carry the signal. Balance term Mean-centering cancels any reward-level signal in an un- mixed group (all rollouts pick the same mode) — precisely 010002000300040005000600070008000 Tokens after routing token, L 0.4 0.6 0.8 1.0 1.2 1.4 Correct-answer reward L * NT 800L * S 3039 Phase-1 reward surface (run gammas); each mode wins one band NoThink 1.3 L NT Short 1.2 L S Long 1.0 (flat) Figure 9: Reward surface with the run gammas. Dashed ver- ticals mark the analytic crossovers; dotted verticals the per- mode caps. the collapse regime. We therefore add a correction to the advantage, after centering: ˆ A i ← ˆ A i + β bal p ⋆ − f mode(i) ,(9) where f mode is the fraction of free rollouts choosing that mode and p ⋆ = 1/3. Over-represented modes are nudged down, under-represented up. The fractions and nudge cover free rollouts only (forced rollouts are balanced by construc- tion and would dilute the signal during warmup). Two gates keep the term consistent with the reward: an encourage (posi- tive) nudge is applied only to correct rollouts and a discourage (negative) nudge only to incorrect ones, so balancing never rewards a wrong answer nor penalizes a right one; and a wrong Long attempt is exempted, since Long handles the hardest problems and discouraging it there would push hard problems away from it. β bal is annealed to 0 once routing is established, to verify the split is calibrated to difficulty rather than held at 1/3 by the term. Rollout protocol and warmup The base model never emits a routing token unprompted (the chat template prefills <think>), so generation runs in phases. During warmup (⌊n/4⌋ rollouts per mode are forced — the routing question and that mode’s token are appended after the assistant prefix so vLLM generates in-mode — and the rest are free; after warmup all n are free. After a forced generation the routing token sits at the prompt tail; we promote it into the response head so it receives gradient like any generated token (with P the prompt, Q the routing question, t the routing token): [P | Q| t ]∥ gen → [P | Q ]∥ [t| gen ]. Together with the−0.5 penalty on rollouts that emit no rout- ing token, this teaches the model to emit the routing word itself; the first free routing tokens appear around step 20. Mode detection and per-mode caps The mode is read from the leading tokens of each response, token-id first (exact match, reliable for the promoted forced tokens) with a prefix-regex fallback for free spelling variants; matching is a prefix match with no trailing word boundary, so continuations such as “LongI need. . . ” are not misread. Per- mode caps are applied before reward and training: a response whose answer falls beyond its mode’s cap is masked and scored wrong, creating the capability gap the router learns from. The cap counts tokens after the routing token (match- ing L), and forced labels are trusted rather than re-detected. Validation is uncapped, measuring production behavior — which makes brief-mode validation accuracy marginally op- timistic relative to the capped objective. C Experimental Details and Hyperparameters Datasets, model, evaluation. We train on the MATH- lighteval split and validate on held-out MATH-500; a re- sponse is correct iff its extracted final answer matches the reference (binary reward), on which Eq. (7) is applied. Out- of-distribution evaluation uses GSM8K (1,319 problems, avg@5) and AIME 2024+2025 (30 each, avg@16), under the same temperature (0.6), prompt, and extraction. The base model is DeepSeek-R1-Distill-Qwen-1.5B. We train three seeds of the router and of each single-mode baseline; the latter force every rollout into one mode with the same data and hyperparameters but no routing token, balance term, or free rollouts. Hyperparameters and schedule. Table 4 lists the settings used. Training runs in two configurations: phase 1 (steps 0 → 60) uses the shaped bases/gammas with the balance coefficient annealed to 0 over steps 60→85; phase 2 (60→ 90) flattens the reward (all bases 1.0, all γ = 1.0), so mode separation is then maintained by the hard caps and the balance term alone — a test that the split does not depend on the reward gradient once routing exists. D Supplementary Results Per-benchmark accuracy and length. Table 5 reports fi- nal accuracy and mean length for the router, the three single- mode baselines, and the untrained base. The router stays within one to two points of the base and the Long-only ceil- ing while cutting length sharply, and the saving scales with benchmark easiness (Table 6): 4.2× on GSM8K, 1.7× on MATH-500, and 1.1× on AIME, whose problems genuinely require extended reasoning. The mechanism is visible in the realised length (Fig. 10): the router’s mean length collapses onto the NoThink-only anchor on the easy GSM8K and onto the Long-only anchor on the hard AIME, with MATH-500 in between — i.e. it sends nearly all of GSM8K to NoThink and nearly all of AIME to Long, difficulty-adaptive out of distribution. Accuracy dips while routing is established and recovers by step 90 (0.807→0.727→0.764→0.783 at steps 0/30/60/90); routing is a token-efficiency mechanism, not an accuracy one. Training dynamics. Figure 11 traces the quantities behind the stability and meaningfulness claims. By step 90 the free split settles near 20/32/47% (NoThink/Short/Long) with routing entropy H≈1.04 against the maximum ln 3≈1.10; Table 4: Hyperparameters for three-mode self-routing GRPO. Phase 2 flattens the reward; caps and balance keep the modes distinct. ParameterValue Bases b NT /b S /b L (phase 1)1.3/1.2/1.0 Discounts γ NT /γ S /γ L 0.99984/0.99994/1.0 Caps C NT /C S / context1024/3000/16384 Wrong / absent-routing penalty0.0 /−0.5 Std-normalizationdisabled Balance coef. / target1.0 / 1/3 Balance anneal (start/end)60/85→ 0 Group size G8 Warmup steps45 Train batch (prompts)128 Mini-batch / micro-batch32/8 Loss aggregationtoken-mean KL penalty β KL 0 Learning rate (Adam)1× 10 −6 Max prompt / response len.2048/16384 Total steps / validate every90/10 Rollout backendvLLM Val. temp. / n / batch0.6/1/256 GPUs4× H100 all three modes persist across seeds, Long remaining the plu- rality and lengthening toward the context limit while the brief modes stay within their caps. Finally, though the model never sees the MATH-500 difficulty levels in training, the free- validation routing split is monotone in them and sharpens over training (Fig. 12) — independent evidence that routing tracks real difficulty. GSM8KMATH-500AIME '24+'25 Benchmark (easy hard) 10 3 10 4 Mean response length (tokens, log) router NoThink router mixed router Long Router length tracks difficulty out of distribution NoThink-only Short-only Long-only Free routing (ours) Figure 10: Router mean response length per benchmark (black diamond,±std) against the three single-mode anchors (coloured ticks), benchmarks ordered easy→hard (log scale). The router hugs the NoThink anchor on GSM8K and the Long anchor on AIME — realised length is a proxy for the routing split, which the released logs do not record per OOD benchmark. Table 5: Final accuracy and mean response length (tokens), mean ± std over three seeds. MATH-500 in-distribution; GSM8K/AIME out-of-distribution. Base is the untrained model under the routing prompt. AccuracyMean length (tokens) MATH-500 GSM8K AIME’24 AIME’25 MATH-500 GSM8K AIME (comb.) Base (no routing)0.796±.002 0.831±.006 0.265±.008 0.231±.0134743±381930±1812266±26 NoThink-only0.668±.013 0.757±.001 0.137±.010 0.105±.0181096±141336±523768±453 Short-only0.727±.007 0.756±.004 0.218±.013 0.166±.0282030±149440±137259±470 Long-only0.802±.004 0.837±.007 0.297±.007 0.230±.0114507±881665±5411881±56 Free routing (ours) 0.782±.006 0.781±.005 0.276±.010 0.222±.0052810±59459±1810716±74 020406080 Training step 0.0 0.2 0.4 0.6 0.8 1.0 Free-rollout share warmup end (a) Mode shares emerge and persist NoThink Short Long unknown 020406080 Training step 0.0 0.2 0.4 0.6 0.8 1.0 Routing entropy (nats) ln 3 (max) (b) Routing entropy approaches maximum 020406080 Training step 1000 2000 3000 4000 5000 6000 Mean response length (tokens) NoThink cap Short cap (c) Long lengthens; brief modes stay capped NoThink Short Long Figure 11: Free-rollout training dynamics (mean over three seeds, std bands). (a) Mode shares emerge after warmup and persist. (b) Routing entropy approaches ln 3. (c) Long lengthens while the brief modes stay capped. Table 6: Token reduction of free routing vs. the base model. Base Router Reduction GSM8K193045976% (4.2×) MATH-5004743281041% (1.7×) AIME122661071613% (1.1×) L1L2L3L4L5 MATH-500 difficulty level 0% 20% 40% 60% 80% 100% Free-validation routing share step 30 L1L2L3L4L5 MATH-500 difficulty level step 60 L1L2L3L4L5 MATH-500 difficulty level step 90 NoThink Short Long Routing split by problem difficulty over training (easy NoThink, hard Long) Figure 12: Free-validation routing share by MATH-500 difficulty level at three checkpoints. The easy→NoThink, hard→Long gradient emerges over training.

Instant research discovery