Paper deep dive

Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking

Taehyeon Kim, Hyunsoo Lee, Youngsoo Jang, Moontae Lee

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 66

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large reasoning models (LRMs) achieve strong performance by externalizing explicit reasoning traces before producing the answer, yet suffer from overthinking challenge that allocates uniformly heavy computation to queries of varying difficulty. While proprietary models mitigate this via opaque routing, open-source LRMs still lack an efficient mechanism to internalize adaptive reasoning due to both expensive training cost and limited disclosure of training recipes. In response, we introduce RPO (Root-token Policy Optimization), a framework that enables LRMs to self-determine when to reason by training only the initial root token (e.g., whether to invoke the <think> tag) via group relative reward and group-wise advantages. By focusing on this pivotal branching point, RPO drastically reduces training overhead and VRAM usage. Across multiple model families and scales, RPO learns difficulty-aware adaptive thinking at just ~2% of the training compute of prior adaptive-reasoning methods.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

64,080 characters extracted from source content.

Expand or collapse full text

Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking Taehyeon Kim ♠ Hyunsoo Lee ♠ Youngsoo Jang ‡,†, * Moontae Lee ♠,§,† ♠ LG AI Research ‡ UNIST § University of Illinois Chicago Abstract Large reasoning models (LRMs) achieve strong performance by externalizing explicit reason- ing traces before producing the answer, yet suffer from overthinking challenge that allo- cates uniformly heavy computation to queries of varying difficulty. While proprietary models mitigate this via opaque routing, open-source LRMs still lack an efficient mechanism to inter- nalize adaptive reasoning due to both expensive training cost and limited disclosure of training recipes. In response, we introduce RPO (Root- tokenPolicyOptimization), a framework that enables LRMs to self-determine when to rea- son by training only the initial root token (e.g., whether to invoke the<think>tag) via group relative reward and group-wise advantages. By focusing on this pivotal branching point, RPO drastically reduces training overhead and VRAM usage. Across multiple model fami- lies and scales, RPO learns difficulty-aware adaptive thinking at just∼2% of the training compute of prior adaptive-reasoning methods. 1 Introduction The advent of large reasoning models (LRMs) has redefined the benchmark requiring complex reason- ing (Guo et al., 2025; Jaech et al., 2024). The effi- cacy of LRMs has been attributed to the additional computation allocated at test time through explicit reasoning paths, typically encapsulated within de- limiters such as<think>...</think>. By ex- ploring logical traces, reflecting on internal errors, and self-verifying intermediate steps, these mod- els have achieved breakthroughs in mathematical and symbolic reasoning tasks. These days, this think-before-answer paradigm has emerged as the de facto standard. Despite the potential of LRMs, there remains a significant limitation that general LRMs tend to * Work completed while employed at LG AI Research. † Corresponding authors. Queries Reasoning samples Non-Reasoning samples Reasoning Model Adaptive Reasoning Model SFT Self-distillation Root-token policy optimization Adaptive Reasoning Model Root-token Figure 1: Overview of our training pipeline. We gener- ate paired reasoning samples and non-reasoning samples from input queries through the original reasoning model. These samples are self-distilled to manifest adaptive rea- soning. Finally, root-token policy optimization (RPO) is applied to the model, refining the policy by updating only the initial branching token (i.e., root-token) to im- prove difficulty-aware adaptive reasoning. apply this exhaustive explicit reasoning uniformly across all queries, regardless of their intrinsic com- plexity. This results in the overthinking problem (Feng et al., 2025; Sui et al., 2025) where sub- stantial computational resources and latency are wasted on trivial questions. Consequently, a surge of emerging research has been driven into striking a balance between reasoning quality and compu- tational efficiency (Zhang et al., 2025; Fang et al., 2025; Wu et al., 2025; Wang et al., 2025b). While proprietary services such as GPT5 (OpenAI, 2025) and Claude (Anthropic, 2024) rely on black-box routing to mitigate overthinking, a novel model- centric alternative is required to internalize adap- arXiv:2610.17767v1 [cs.CL] 13 Oct 2026 Math query To solve the problem, ... Okay, so I need to figure out the value of this series: 1 - 2 + 3 - 4 + 5 - 6 + ... + 99 - 100. Hmm, let's see. It alternates between adding an odd number and subtracting an even number. Let me write out the first few terms to see if I can spot a pattern. ...Skip... Compute: $1-2+3-4+5- … +99-100$. Non-Thinking Thinking Long path Short path (1) Self-Taught Rollouts (Data curation) Correctness Self-distillation (2) Root-Token Policy Optimization: Efficient GRPO Non-reasoning Root token Reasoning Root token Reward + Advantage Reward Reward 1-token Update Only Figure 2: Detailed overview of our training framework. After supervised finetuning (SFT) with self-generated correct samples, RPO involves updates only the initial decision token with group normalized rewards and advantages. tive reasoning via policy optimization with open dataset for the open-source community. While adaptive reasoning models that learn when to reason have been explored via policy op- timization, these studies have faced the following challenges: 1.Training inefficiency. Optimizing over long con- text trajectories with policy optimization incurs prohibitive VRAM and temporal costs, hinder- ing accessible and iterative development. 2.Data curation. There is a lack of standardized methods for baking difficulty-aware datasets that effectively balance reasoning traces with direct responses across heterogeneous domains. 3.Initialization gap. Prior works train from scratch or adapt base models, often avoiding adapting the original LRM due to its training sensitivity. To alleviate these challenges, we propose RPO (Root-tokenPolicyOptimization), an effi- cient training method that enables adaptive reason- ing by learning only the initial root token (e.g., the decision to open or close the<think>tag). Our method involves two stages (Figure 1): (1) offline synthesis which generates paired reasoning and non-reasoning responses via forced chat templates, and (2) RPO, which restricts gradient updates to the pivotal branching decision. Our design is com- plementary to recent token-selective RL that up- dates only a small subset of decision tokens (e.g., high-entropy forking tokens) rather than the full tra- jectory (Wang et al., 2025a); RPO pushes this idea to the extreme by optimizing only the root routing token (Figure 2). We demonstrate that RPO dras- tically enhances memory and compute efficiency during training while preserving the model’s origi- nal capability. Our contributions are as follows: •We propose RPO, a novel framework, termed as root-token optimization to enable adaptive reasoning by optimizing only the initial root- token whether to reason, significantly reducing training overhead as well as efficient difficulty- aware routing (§3). •Our empirical results demonstrate RPO enables LRMs learning when to reason over diverse LRM families and scales. We also provide an analysis on calibration for an explicit reasoning knob (e.g., target Think% or routing threshold for root-token probability) (§4, §5). • We demonstrate that difficulty-aware policies in mathematical domains elicit adaptive reasoning in broader scientific and code domain (§5). 2 Related work 2.1 Efficient reasoning models LRMs allocate additional test-time computation to explicit reasoning traces often with specific delimiters such as<think>. . .</think>(Guo et al., 2025; Jaech et al., 2024). This think-before- answer format enables iterative exploration and self-verification, but it is frequently applied uni- formly across inputs. Such uniform long-form rea- soning leads to the overthinking challenge with high latency and compute on intrinsically easy queries and sometimes producing redundant rea- soning tokens (Feng et al., 2025; Sui et al., 2025). A broad line of work has pursued efficiency via adaptive computation and early stopping. Classi- cal dynamic-computation methods and early-exit approaches terminate generation once confidence criteria are met, reducing unnecessary computation without changing the core model objective (Schus- ter et al., 2022; Bae et al., 2025). Recent reasoning- focused extensions similarly aim to avoid excessive reasoning tokens by learning or heuristically deter- mining when further context is unlikely to help, and by truncating or suppressing low-information reasoning segments (Sui et al., 2025). Another direction controls reasoning length through prompt- or budget-based strategies. Meth- ods that rely on fixed or predicted token budgets can reduce verbose CoT while maintaining ac- curacy, but often require external budget selec- tion rules or calibrated estimators, which may not transfer cleanly across domains (Han et al., 2024). In parallel, internal-rationale approaches such as Quiet-STaR (Zelikman et al., 2024) encourage mod- els to form latent rationales during generation, while simplified recipes such as s1 (Muennighoff et al., 2025) show that carefully constructed 1k data and prompting can recover much of test-time scaling benefits with minimal training overhead. As an orthogonal axis over these approaches, our goal is not to enforce a budget or modify the archi- tecture, but to internalize the gating decision itself whether to open a long explicit reasoning trace in the first place. RPO optimizes only the root token that triggers the<think>branch, enabling adaptive reasoning with minimal training footprint and without backpropagating over whole reasoning trajectories. 2.2 Difficulty-aware policy optimization The question of learning when to reason has re- cently been reframed as difficulty-aware policy opti- mization: the model should allocate explicit reason- ing only when needed, while responding directly to easy or non-reasoning queries. This direction is es- pecially relevant under the explicit<think>styled format, where the core idea is a discrete branching action that enters a long reasoning mode versus pro- duces a direct response (Zhang et al., 2025; Fang et al., 2025; Wu et al., 2025; Wang et al., 2025b). Representative methods learn such policies through RL or preference learning. AdaptThink (Zhang et al., 2025) trains a policy to select think- ing versus non-thinking modes conditioned on dif- ficulty, while Thinkless (Fang et al., 2025) uses preference/RL signals to reduce redundant think- ing tokens in general conversations. Beyond bi- nary gating, ARM (Wu et al., 2025) expands the action space to multiple reasoning formats (e.g., direct answer, short CoT, long CoT, code), and AdaReasoner (Wang et al., 2025b) treats reason- ing configurations as actions in a model-agnostic RL framework. Together, these works show that adaptive reasoning is learnable and can outperform static long-CoT inference. However, existing policy optimization ap- proaches face practical hurdles that motivate our design. First, many studies are benchmark-centric and largely evaluated on math-heavy suites, leaving open how well their learned routing calibrates (e.g., whether the thinking rate remains aligned with dif- ficulty on open-domain and scientific queries). Sec- ond, optimizing policies over whole long reason- ing trajectories is often training-cost dominated: GRPO-style post-training incurs expensive cost of VRAM, gradient computation, and wall-clock costs (Guo et al., 2025), also detailed in §3.3 and Table 5. Third, effective training frequently depends on cu- rated external prompts, data from larger teacher models, paired supervision, or proprietary query streams for routing signals, and these factors are rarely opened to the community. 3 Method 3.1 Motivation Adaptive reasoning is fundamentally a routing problem. Before generating an answer, the model must decide whether to spend tokens on explicit de- liberation. In open-domain tasks, supervising this routing is difficult because task difficulty is latent: correctness can be ambiguous and the same query may be easy or hard depending on a model’s cali- bration and knowledge. As a result, fixed heuristics (e.g., always long CoT or static budgets) are often miscalibrated. Why math as a proxy We ground our training signal in mathematical reasoning because it ad- mits deterministic verification and a model-relative notion of when reasoning pays off. For each input queryx, we elicit two mode-conditioned behaviors: (i) a direct-answer mode (SHORT) and (i) an explicit-reasoning mode (LONG). Let c S (x),c L (x)∈0, 1denote exact correctness un- der each mode. Rather than relying on external difficulty labels, we can use the difference between mode-conditioned success rates (e.g., Pass@k) as an empirical difficulty proxy: instances where LONG is more reliable than SHORT are treated as harder for the current model, and vice versa. This model-specific signal provides supervision for 0.00.20.40.60.81.0 Threshold τ on p short (x) 0 200 400 # Instances Routed to Short Routed to Long Figure 3: Thresholding on a SHORT-mode confidence signal yields a smooth routing trade-off about prob- ability threshold vs number of routed instances over MATH500 dataset (Hendrycks et al., 2021). learning routing policies without external difficulty labels. Self-taught difficulty signalsAfter a lightweight SFT step that teaches the model to follow our SHORT/LONG templates (§3.2), we observe that routing can be anticipated at the generation root. We obtain a model-specific difficulty signal by con- trasting mode-conditioned accuracy on the same query (e.g., Pass@kunder SHORT vs. LONG). To expose this signal for routing, we parameterize the mode decision at the generation root: under our template, the first newline token(s) after<think> select LONG vs. SHORT ( . ). Letz S de- note the SHORT root token ( ) and define a root-level SHORT-preference proxy as the model’s conditional probability at this decision point: p short (x) = p θ (z S | x, <think>). We route by thresholding: π τ (x) = ( SHORTif p short (x)≥ τ, LONGotherwise. Varyingτyields a simple calibration knob (Fig- ure 3): largerτmakes the router more conservative, assigning fewer queries to SHORT and increasing the fraction routed to LONG. This synthetically produces an interpretable accuracy–routing trade- off, where intermediate thresholds preserve overall accuracy while reducing LONG usage (Table 1). Beyond accuracy, routing is motivated by com- pute. We measure generation lengthL(x)(out- put tokens) and find that LONG exhibits a heavy- tailed length distribution (Figure 4), i.e., a small fraction of queries trigger disproportionately long traces. This makes the routing decision directly Method (τ )SHORTLONGOverall Original0.00 (0)83.6 (500)83.6 (500) Adaptive (0.75)93.6 (62)82.4 (438)83.8 (500) Adaptive (0.70)89.7 (155)78.8 (345)82.2 (500) Table 1: Accuracy (%) and routed instance counts (in parentheses) under threshold routing over MATH500 dataset (Hendrycks et al., 2021) with R1-1.5B model. (a) SHORT(b) LONG Figure 4: Token-length histograms (log-scaled fre- quency) for SHORT and LONG mode. translate into cost control, and motivates learning a lightweight gating policy at the root that reduces unnecessary LONG usage without the loss of the model capability. 3.2 SFT for mode selection Mode definition We define two generation modes using only the reasoning-tag template at the generation root. The token<think>is treated as a special token, and . tokenized into different root tokens. Concretely, we use: •LONG: prefix<think> that the model continues with explicit reasoning content. •SHORT: prefix<think> </think> to suppress explicit reasoning. Self-generated trajectories from OpenR1-Math queries We construct SFT supervision from OpenR1-Math queries by sampling multiple gen- erations from the given LRM under each mode template (SHORT/LONG). We retain only gener- ations that (i) satisfy the mode-specific template and (i) match the exact answer, yielding two pools of correct demonstrations,S S andS L . We then randomly sampleKinstances from each pool for SFT. We do not enforce that a problem must be correct in both modes. SFT We use a SFT loop (1) to manifest mode- conditioned style without performance degrada- tion under the default chat template. LetAcc orig = Eval(π 0 ; DEFAULTCHAT)be the initial accu- racy, and letAcc(π) = Eval(π; DEFAULTCHAT) Algorithm 1 SFT for mode selection Require:base modelπ 0 , promptsD, samples per promptM, tolerance ε, budget K 1: a 0 ← Eval(π 0 ; DEFAULTCHAT) ▷ default template 2: π ← π 0 3: while Eval(π; DEFAULTCHAT)− a 0 ≤ ε do 4: S S ← CollectCorrect(π,D,M; SHORTPREFIX) 5: S L ← CollectCorrect(π,D,M; LONGPREFIX) 6: T ← Sample(S S ,K) ∪ Sample(S L ,K) 7: π ← SFT(π,T ) 8: end while denote the same metric for an SFT candidateπ. We iteratively refresh the correctness-filtered SFT dataset and continue training only while the perfor- mance gap remains within a tolerance ε: Acc(π)− Acc orig ≤ ε. The loop is intentionally general. Empirically, across LRMs we tested, a single iteration typically sufficed (i.e., the constraint was satisfied without repeated resampling). 3.3 Root-token policy optimization (RPO) SFT establishes SHORT/LONG behaviors, but reli- able adaptive selection requires directly optimizing the gating policy. The post-SFT model already knows how to produce either style; what remains is learning when to invoke each one. We therefore view adaptive reasoning as a branching problem: once the gate is chosen, the continuation is simply a mode-conditioned rollout rather than a learning target. Motivated by this hypothesis, we apply RPO only to the first gating token (the root), treat- ing the remainder of each sampled trajectory purely as environment output. Root-token action space We fix a<think>pre- fix and parameterize gating by the first newline tokenization: z ∈ , . Because<think>is a special token and . distinct tokens,zis a well-defined cat- egorical action. Selectingz = LONG: the model continues generating inside the<think> block (explicit reasoning). Choosingz = enters SHORT: the template immediately closes the thinking block, i.e.,<think> </think> . RPO optimizes onlyp θ (z | x); subsequent tokens are masked from the gradient. Balanced rollouts For each queryx, we collect nrollouts with a fixed split:n/2SHORT andn/2 Algorithm 2 Root-token policy optimization Require:policyπ θ (init from SFT), referenceπ ref (frozen), promptsD rl , rollouts per promptn(even), context budget S, temperature τ , KL weight β, mode weights (α S ,α L ) 1: Define gating tokens: z ∈ , 2: Define templates: SHORT(z= ), LONG(z= ) 3: for each training step do 4:Sample a minibatchB ⊂D rl 5:for each prompt x∈B do 6: Balanced rollouts: generaten/2trajectories with SHORT template and n/2 with LONG 7: (z (x) i ,y (x) i ) n i=1 ← Rollout(π θ ,x; S,τ) ▷ z (x) i is the root newline token(s) 8: Compute correctness indicatorsc (x) i ← 1Ans(y (x) i ) is correct 9: LetC S (x) ← P i:z (x) i = c (x) i , C L (x) ← P i:z (x) i = c (x) i 10:for i = 1 to n do 11:r (x) i ← α S · C S (x) ifz (x) i = else r (x) i ← α L · C L (x) 12:end for 13:Store root-token log-probsℓ (x) i ← logπ θ (z (x) i | x) and ℓ (x),ref i ← logπ ref (z (x) i |x) 14:end for 15:Per-mode group-relative advantages : 16: I m ←(x,i) : x∈B, m (x) i = m 17: μ m ← 1 |I m | P (x,i)∈I m r (x) i , σ 2 m ← 1 |I m | P (x,i)∈I m (r (x) i − μ m ) 2 18: A (x) i ← (r (x) i − μ m (x) i )/(σ m (x) i + ε) ∀(x,i) 19:Root-token-only update (mask all tokens except z): 20: θ ←Update θ, ∇ θ − 1 |B| P x∈B 1 n P n i=1 A (x) i ℓ (x) i + β · KL(π θ (z | x)∥ π ref (z|x)) 21: end for LONG. LetC S (x)andC L (x)be the numbers of correct rollouts within the SHORT and LONG groups, respectively. This balanced sampling pre- vents trivial mode collapse and yields comparable advantage statistics across modes. Prefix-level reward For a fixed promptx, we refer to then/2rollouts that share a forced mode as the per-prompt mode group (one for SHORT, one for LONG). Each rollout is rewarded by the success count of its per-prompt mode group: r (x) i = ( α S · C S (x)if rollout i is in SHORT, α L · C L (x) if rollout i is in LONG. We setα L =1and use a mild SHORT preference α S =1 + 1 n to reflect the objective of saving com- pute without sacrificing the correctness. Because r (x) i depends on the per-prompt mode-group out- come, the update favors routing toward the mode that yields higher group-level correctness for that prompt. ModelMethodUpdated tokens TKV / rollout (GiB)Act(bp) / rollout (GiB)Train+Opt (GiB)Peak @ B=8 (GiB) 1.5BOriginal GRPO12K2.3813.1822.35146.86 (1.00×) RPO12.381.10× 10 −3 22.3541.40 (0.28×) RPO with LoRA update12.381.10× 10 −3 0.0719.12 (0.13×) 7BOriginal GRPO12K6.3535.16104.31436.34 (1.00×) RPO16.352.93× 10 −3 104.31155.11 (0.36×) RPO with LoRA update16.352.93× 10 −3 0.2151.01 (0.17×) Table 2: Peak VRAM (GiB) for long-context GRPO with uniform contextS=13K (1K prompt+12K generation) andB=8concurrent rollouts. RPO gains memory efficiency by updating the root token (T=1). LoRA further reduces trainable-state memory with fewer trainable parameters. Per-mode group-relative advantages To opti- mize switching under the enforced split, we stan- dardize advantages within each mode by pooling re- wards. Per-prompt-mode would be degenerate: by construction,r (x) i = α m i ·C m i (x)is constant across then/2rollouts that share a mode within prompt x, so the within-mode-group variance is identi- cally zero. Cross-prompt variation inC S (x)and C L (x)is what supplies the non-degenerate vari- ance needed for advantage estimation. Concretely, for rolloutiof promptxin modem (x) i ∈ S,L, letI m = (x,i) : x ∈ B, m (x) i = mdenote the group-level mode pool. Then, A (x) i = r (x) i − μ m (x) i σ m (x) i + ε , μ m = 1 |I m | X (x,j)∈I m r (x) j , σ 2 m = 1 |I m | X (x,j)∈I m r (x) j − μ m 2 . Pooling is per-mode (separateμ S ,σ S and μ L ,σ L ) so that the standardized advantages remain on a comparable scale even when the two modes have systematically different success rates; pooling is group-level so that within-mode variance is non- zero. The signal eachA (x) i encodes is therefore “how does promptx’s mode-msuccess compare to the group-wide mode-mbaseline,” which the policy converts into an updated routing distribu- tion overz. This per-mode group relative scheme presupposes prompt-difficulty diversity withinB. RPO objective with KL regularizationWe reg- ularize the log-probability of the root actionzwith KL-to-reference loss: max θ E x " 1 n n X i=1 A i logp θ (z i | x) # − β KL(p θ (z| x)∥p θ ref (z| x)).(1) whereθ ref is the post-SFT checkpoint. In imple- mentation, we generate full trajectories to evaluate correctness and compute rewards, but retain only the root log-probabilities. Other non-root tokens are discarded from the training graph, and the up- date is performed solely on p θ (z|x). Balanced difficulty sampling We perform RPO on a compact set of∼1K queries comprised of a part of s1k (Muennighoff et al., 2025) and compression dataset (Arora and Zanette, 2025). With the post-SFT model, We estimate mode- conditioned success rates and exclude items where LONG is rarely correct (acc L <τ min =0.25). We then stratify the remaining queries into easy and hard regimes and sample them in a 1:1 ratio to sta- bilize updates and avoid mode collapse. Concretely, easy includes queries where SHORT is already com- petitive (e.g.,acc S ≥τ easy =0.75), while hard in- cludes queries where LONG is consistently prefer- able (e.g., acc S <acc L ≤τ hard with τ hard =0.5). Why root-only updates are memory-efficient Both vanilla GRPO and RPO must work with KV- cache over the full rollout context length S during sampling. The training-time gap arises from back- propagation. Vanilla GRPO updates all generated tokens, whereas RPO masks the trajectory and updates only the root gating token (T=1). As a result, the dominant activation term scales with the number of updated tokens, M bp ∝ B· T · L· d· b· κ, whereBis the number of rollouts processed con- currently,Landdare model depth and width,b is the activation precision (e.g., bf16), andκag- gregates saved intermediates (e.g., attention/MLP). Thus, settingT=1removes the linear dependence on long CoT length. Table 2 quantifies this effect under long-context rollout scenarios, where activa- tion memory dominates the feasibility boundary. Training configurationUnless noted otherwise, we use three H100 GPUs in a node. Each iteration ModelStageMATH500AIME24AIME25Avg P@1#Tok Think% P@1#Tok Think% P@1#Tok Think%P@1#Tok Think% RAR↓ Qwen3-1.7BBaseline (LONG)91.14,539100.045.615,569100.038.115,830100.058.311,979100.01.72 Baseline (SHORT)73.09780.012.33,8980.011.52,1800.0 32.32,3520.00.00 + SFT85.53,76769.834.412,55872.930.412,64875.650.19,65872.81.45 + RPO79.52,75431.843.813,31593.831.714,23196.351.710,10074.01.43 Qwen3-8BBaseline (LONG)96.64,777100.074.414,184100.064.816,950100.078.611,970100.01.27 Baseline (SHORT)84.11,1180.025.45,6460.020.83,1590.0 43.43,3080.00.00 + SFT90.63,39053.554.012,35059.246.712,66159.263.89,46757.30.90 + RPO90.52,79425.052.912,78456.442.512,02953.162.09,20244.80.72 Exaone4-1.2B Baseline (LONG)91.05,416100.054.018,529100.043.819,124100.062.914,356100.01.59 Baseline (SHORT)75.68540.021.93,0750.015.22,7180.0 37.62,2160.00.00 + SFT84.63,01540.937.39,52439.831.310,56249.451.17,70043.40.85 + RPO83.52,45525.533.86,59221.927.36,66523.348.25,23723.60.49 Table 3: Baseline→SFT mode selection→RPO on math benchmarks. Pass@1 (P@1) is the average accuracy (%) over 16 runs. #Tok is average generated tokens. Think% is the fraction routed to LONG. Avg columns are simple averages over (MATH500, AIME24, AIME25). RAR (↓) is the reasoning-per-accuracy ratio, defined as RAR avg = t avg a avg +ε , where a avg and t avg are Avg Pass@1 and Avg Think% as fractions. Lower is better. 20406080100 Thinking ratio (Long routing) (%) 40 50 60 70 80 90 Pass@1 (%) AIME24 AIME25 MATH500 Figure 5: Think% versus Pass@1 on Qwen3-8B with RPO. Each point indicates an intermediate checkpoint. samples 8 queries and collectsn=16rollouts per query (8 per mode) with temperature 0.6 and top-p 0.95. We allow maximum context length 16,384 for rollouts. 4 Experiment 4.1 Setup Our pipeline follows Baseline→SFT→RPO, and we report (i) Pass@1 accuracy, (i) average gen- erated tokens, and (i) the fraction routed to LONG (Think%) in Table 3. We report results on diverse models with different families and sizes: Qwen3- 1.7B, Qwen3-8B, Qwen3-32B (Yang et al., 2025), Exaone4-1.2B (Research et al., 2025) and R1-1.5B (Guo et al., 2025). All models use a<think> style template and share the same two-mode gat- ing mechanism implemented via the first newline token(s) after <think>. 4.2 Datasets and evaluation Math benchmarks We evaluate on three stan- dard math reasoning sets: MATH500 (Hendrycks et al., 2021), AIME24 (MAA, 2024), and AIME25 (MAA, 2025), scientific reasoning dataset as GPQA (Rein et al., 2024), and coding task with LiveCodeBench 1 . Following prior work, correct- ness is determined by exact match on the final an- swer parsed byboxed. At inference time, we match the original technical reports including tem- perature and sampling settings. SFT data for mode selectionWe use mathemat- ical queries from OpenR1-Math data (Open-R1, 2025) and self-generate candidate trajectories with the given model under each mode template. We filter candidates by correctness and basic template validity, producing two pools of correct demonstra- tions for SHORT and LONG. We then sample 20K instances per mode (40K total) for SFT, without requiring overlap between the two pools. Implementation detailsWe implement SFT and RPO in OpenRLHF (Hu et al., 2024), with a cus- tomized lightweight asynchronous rollout engine that overlaps generation and post-processing. Un- less noted, we use max context 16K, batch size 128, and learning rate10 −6 . For RPO, we sample 16 rollouts per prompt (8/8 SHORT/LONG) and apply gradients only to the root gating token. We run < 60 update steps for all model sizes. 4.3 Main results Table 3 reports math performance across stages (Baseline→SFT→RPO). To summarize reason- ing efficiency in one number, we use RAR (↓), the reasoning-per-accuracy ratio:RAR avg = t avg a avg +ε , 1 LiveCodeBench is sliced from March 25 2025 to April 15 2025 MethodMATH500AIME24Avg P@1Think%P@1Think%P@1RAR↓ Prior hybrid-reasoning baselines (reported in their tables) Original (LONG)83.2100.029.4100.056.31.78 Original (SHORT)67.20.014.00.040.60.00 Rejection Finetuning † (Zhang et al., 2025)72.466.625.279.048.81.49 AdaptThink † (Zhang et al., 2025)82.023.231.059.656.50.74 Thinkless † (Fang et al., 2025)81.951.6 27.3100.054.61.39 Ours (same backbone family; evaluated with our protocol) + SFT75.137.820.868.548.01.11 + RPO77.620.525.077.751.30.96 Table 4: Comparison with accuracy and Think% over R1-1.5B model.†presents the performance from the original paper. wherea avg is Avg Pass@1 andt avg is Avg Think% (both as fractions). RPO improves the accuracy routing trade- off On the larger and mid-scale backbones, RPO reduces Think% with modest accuracy changes, yielding the best (lowest) RAR. For ex- ample, on Qwen3-8B, Avg Think% decreases from 57.3% to 44.8% while Avg Pass@1 stays close (63.8%→62.0%), improving RAR from 0.90 to 0.72; on Exaone4-1.2B, Think% drops from 43.4% to 23.6% and RAR improves from 0.85 to 0.49. On Qwen3-1.7B, the post-SFT operat- ing point is already conservative on the LONG side (Think%=72.8, RAR=1.45); RPO yields a comparable operating point (Think%=74.0, RAR=1.43) that recovers accuracy via routing re- finement rather than via further Think% compres- sion. SFT already enables routing; RPO refines it. SFT alone yields a strong reduction in Think% (and tokens), showing that mode-following is largely learned at the template level. RPO then fine-tunes the root decision to better match query difficulty. The direction of this refinement is configuration- dependent: on Qwen3-1.7B and R1-1.5B (Table 4), RPO recovers accuracy on top of SFT by reallocat- ing LONG toward the harder subset; on Qwen3-8B and Exaone4-1.2B, RPO instead trades a small ac- curacy reduction for further Think% compression. Comparison with other GRPO-based methods Table 4 compares RPO with recent hybrid rea- soning baselines on the same R1 1.5B model. RPO lowers the fraction routed to LONG to 20.5% and reaches a two-task average accuracy of 51.3 with RAR=0.96. AdaptThink achieves a stronger absolute operating point (Avg P@1=56.5, RAR=0.74) but at a far larger training budget (Ta- ble 5); the comparison should therefore be read as two distinct points on a budget-accuracy fron- tier rather than a head-to-head ranking. Addition- ally, Table 5 highlights the compute-efficiency gap. Trajectory level GRPO is compute heavy because gradients traverse the full chain of thought. With matched accounting on an 8B reference model as an example, RPO reduces total training compute to about 0.02× AdaptThink. 5 Analysis Controlling the degree of routingFigure 5 plots P@1 against Think% for intermediate RPO check- points. Points with comparable accuracy but differ- ent thinking ratios indicate multiple feasible operat- ing points. Thus, calibration is practical: choose a checkpoint that satisfies a compute/routing budget while maintaining accuracy. The achievable fron- tier depends on the RL prompt set and easy/hard mix, so checkpoint selection is dataset-conditioned rather than universal. LiveCodeBench and GPQA Table 6 extends evaluation beyond math to coding and scientific QA, where the optimal amount of thinking is more heterogeneous. Across models, SFT already intro- duces a meaningful reduction in explicit reasoning, while RPO further reshapes the accuracy routing profile by updating only the root decision. The effect is model-size dependent: on Qwen3-8B and Exaone4-1.2B, RPO compresses average decoding effort and lowers RAR; on Qwen3-32B, where SFT already produces a near-baseline operating point (Avg Think%=94.6, RAR=1.52), RPO yields a comparable but not strictly better RAR (1.55), re- flecting that the room for further routing-only gains shrinks once the underlying solver is sufficiently strong and the SFT operating point is already con- servative. Practically, this suggests treating the post-RPO checkpoint as a controllable router (with τ-sweeps) rather than as a fixed point that uni- formly dominates SFT. Calibration of root confidence Figure 6 reportsreliabilitydiagramswithQwen3- 1.7B model for the root SHORT confidence p short (x)=p θ (z S |x, <think>). We binp short into 10 uniform intervals and compute (i) Corr, the Pearson correlation between per query confidence and empirical accuracy, and (i) calibration errors ECE= P b n b N |acc b − conf b |and MCE = max b |acc b − conf b |, whereacc∗ bandconf∗ b are the average accuracy and confidence within binb(Guo et al., 2017). Overall,p short (·)tends MethodSFT specGRPO specForward PFLOPsBackward PFLOPsTotal PFLOPsRel. Total (vs. AdaptThink) AdaptThink (Zhang et al., 2025)40K @ 16K40K @ 16K, K=16, T upd ≈16K174,080348,160522,2401.00× ThinkLess (Fang et al., 2025)334K @ 24K1K @ 24K, K=8, T upd ≈24K131,328262,656393,9840.75× RPO (Ours)8K @ 16K1K @ 16K, K=16, T upd =16,1444,09710,2410.02× Table 5: PFLOPs accounting for an 8B dense Transformer (full-length sequences at the stated context). We assume 2PFLOPs/token for forward and4Pfor backward. For AdaptThink, the number of dataset is set to 40K as reported. ModelStageLiveCodeBenchGPQA-DiamondAvg P@1#Tok Think% P@1#Tok Think%P@1#Tok Think% RAR↓ Exaone4-1.2B Baseline42.212,159100.050.87,145100.046.59,652100.02.15 + SFT34.69,28190.146.25,03770.140.47,15980.11.98 + RPO36.76,69261.741.72,88135.039.24,78748.41.26 Qwen3-8BBaseline49.514,878100.058.87,355100.054.2 11,117100.01.85 + SFT30.210,41063.031.15,22750.130.77,81956.61.85 + RPO35.411,53254.755.46,91067.245.49,22161.01.34 Qwen3-32B† Baseline59.113,498100.066.25,108100.062.79,303100.01.60 + SFT58.311,45790.666.34,82298.562.38,14094.61.52 + RPO54.410,62387.265.04,76897.6 59.77,69692.41.55 Table 6: LiveCodeBench/GPQA-Diamond results. Mod- els without†are trained with full-parameter SFT;†in- dicates a LoRA-tuned model. 0.00.20.40.60.81.0 Confidence (P(nothink)) 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy ECE = 0.2717 MCE = 0.5635 Corr = 0.200 math500 - P(nothink) Perfect Calibration (Confidence, Accuracy) Accuracy (a) MATH500. 0.00.20.40.60.81.0 Confidence (P(nothink)) 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy ECE = 0.2234 MCE = 0.5786 Corr = 0.420 aime25 - P(nothink) Perfect Calibration (Confidence, Accuracy) Accuracy (b) AIME25. Figure 6: Reliability diagrams (Guo et al., 2017) for the root confidencep short (x)=p θ (z S |x, <think>). Each bin plots empirical accuracy against predicted confi- dence, with the diagonal indicating perfect calibration. to correlate with correctness, but the signal is weak and dataset dependent. On AIME25, which is substantially harder, mass concentrates at low p short (·), so the router rarely assigns high SHORT confidence, whereas MATH500 shows a broader spread with more high confidence bins. Further discussions are in Appendix. 6 Conclusion We introduce RPO, a root token policy optimiza- tion framework for adaptive thinking that updates only the initial gating token while treating the re- mainder of the rollout as environment output. Pre- cisely, RPO turns static LONG style generation into a lightweight routing problem, reducing sig- nificant training overhead during GRPO. Across multiple model families and scales, RPO improves the accuracy routing trade-off under most scenarios, achieving lower reasoning per accuracy ratios over expert scientific reasoning and coding as well as math reasoning. Some configurations trace a com- parable but not strictly dominating frontier, sup- porting the use of RPO as a controllable routing layer rather than a uniform replacement for SFT. Limitations While our work supports diverse LRM families and scales, several limitations suggest avenues for future exploration. First, broader evaluation is needed under advanced benchmark datasets requir- ing agentic behaviors with tools. Second, while root confidence is informative under our branch- ing problem setting, it can be not only sensitive to bucket composition (i.e., easy and hard queries for RPO), but weakly calibrated towards its query difficulty. Third, we focus on a binary routing decision and do not optimize intra mode behav- iors such as how long to think or when to stop, which could further improve compute allocation. Fourth, RPO’s gain over the SFT operating point is configuration-dependent: on Qwen3-1.7B, the post- SFT routing already saturates the achievable RAR, so RPO yields a comparable rather than strictly better operating point; on Qwen3-32B, where we use LoRA tuning and SFT already produces a near- baseline conservative router, RPO likewise traces a comparable but not strictly dominating frontier. We therefore position RPO as a controllable rout- ing layer to be selected via checkpoint or thresh- old, rather than a uniform replacement for SFT. Finally, due to limited compute, we do not provide a more exhaustive ablation of the RL training dy- namics regarding policy optimization to uncover how RPO enables such gating behaviors in model aspect or a fully controlled comparison against al- ternative GRPO variants based on their publicly released checkpoints and exact training recipes. While mitigating such behaviors would be valu- able, it is beyond the scope of this study. We leave other discussion points including the design of root token choice, and generalizability in Appendix. Acknowledgments We thank Dahyun Lee for providing extensive feed- back on our paper. References Anthropic. 2024. Claude 3.5 Sonnet Model Card Ad- dendum. Technical report. Daman Arora and Andrea Zanette. 2025. Training lan- guage models to reason efficiently. arXiv preprint arXiv:2502.04463. Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Haru- tyunyan, Ziwei Ji, Aaron Courville, and 1 others. 2025. Mixture-of-recursions: Learning dynamic re- cursive depths for adaptive token-level computation. arXiv preprint arXiv:2507.10524. Gongfan Fang, Xinyin Ma, and Xinchao Wang. 2025. Thinkless: Llm learns when to think. arXiv preprint arXiv:2505.13379. Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. 2025. Efficient reasoning models: A survey. arXiv preprint arXiv:2504.10903. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International conference on machine learn- ing, pages 1321–1330. PMLR. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948. Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. 2024. Token-budget-aware llm reasoning. arXiv preprint arXiv:2412.18547. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Ja- cob Steinhardt. 2021. Measuring mathematical prob- lem solving with the math dataset. arXiv preprint arXiv:2103.03874. Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. 2024. Openrlhf: An easy- to-use, scalable and high-performance rlhf frame- work. arXiv preprint arXiv:2405.11143. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720. MAA. 2024.American invitational mathemat- ics examination (aime).https://w.maa.org/ math-competitions/aime. Accessed: 2025-09-18. MAA. 2025.American invitational mathemat- ics examination (aime).https://w.maa.org/ math-competitions/aime. Accessed: 2025-09-18. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xi- ang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393. Open-R1. 2025. OpenR1-Math-220k. Hugging Face Datasets. Accessed: 2026-01-06. OpenAI. 2025. Gpt-5 system card. Technical report, OpenAI. Technical Report. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling. LG Research, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Yemuk Choi, Kyubeen Han, Seokhee Hong, Junwon Hwang, Taewan Hwang, and 1 others. 2025. Exaone 4.0: Unified large language models integrating non-reasoning and reasoning modes. arXiv preprint arXiv:2507.11407. Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. Ad- vances in Neural Information Processing Systems, 35:17456–17472. Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, An- drew Wen, Shaochen Zhong, Hanjie Chen, and 1 others. 2025. Stop overthinking: A survey on ef- ficient reasoning for large language models. arXiv preprint arXiv:2503.16419. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shix- uan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. 2025a. Beyond the 80/20 rule: High-entropy minority tokens drive effective rein- forcement learning for llm reasoning. arXiv preprint arXiv:2506.01939. Xiangqi Wang, Yue Huang, Yanbo Wang, Xiaonan Luo, Kehan Guo, Yujun Zhou, and Xiangliang Zhang. 2025b.Adareasoner: Adaptive reason- ing enables more flexible thinking. arXiv preprint arXiv:2505.17312. Siye Wu, Jian Xie, Yikai Zhang, Aili Chen, Kai Zhang, Yu Su, and Yanghua Xiao. 2025. Arm: Adaptive reasoning model. arXiv preprint arXiv:2505.20258. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025.Qwen3 technical report.arXiv preprint arXiv:2505.09388. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. 2024. Quiet-star:Language models can teach them- selves to think before speaking.arXiv preprint arXiv:2403.09629. Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. 2025.AdaptThink: Reasoning Mod- els Can Learn When to Think.arXiv preprint arXiv:2505.13417. A Discussions A.1 Root-level gating is tokenization agnostic We demonstrate that the design of root token in RPO framework is tokenization agnostic over a broader range of model families. Other than the set- tings in the body, we observe that models without specific routing tokens can achieve robust mode routing by prepending arbitrary prefixes to distin- guish between reasoning and non-reasoning modes. Qwen3 models enable mode selection through dis- tinct single tokens for reasoning and non-reasoning modes. Interestingly, even for models lacking such routing tokens, adding arbitrary prefixes to distin- guish between reasoning and non-reasoning modes enables routing while preserving base reasoning be- havior. In our experiments with R1-Distill-Qwen, a reasoning-only model, we prepend a pair of in- struction prefix sentences before the<think>in the chat template prior to SFT. Then, it is observed that the model keeps the reasoning accuracy when per- forming inference with the reasoning prefix, and routing get possible when no prefix is provided. Several different sentences (about 20∼30 tokens) are tested and it is found that this behavior is consis- tent regardless of the sentence content (i.e., token sequence for prefix) (Table 8). We verify these with RPO training: when the prefixes are prepended during rollout and trained only on the prefix tokens, routing improved similarly as mentioned in the body of the work that utilizes only single routing tokens for training. A.2 Alternative root token markers A potential concern is that se- mantic meaning in natural text and could introduce ambiguity. We emphasize that the routing decision is confined to a single categorical action at the root position immediately after<think>; newline to- kens appearing later in the generation do not affect routing, and RPO masks gradients for all non-root tokens. Nevertheless, to verify that our method is not sensitive to the specific marker choice, we com- pare three marker types on Qwen3-8B (Table 8). A.3 What generalizes and what does not RPO often transfers when the target setting re- wards selective compute, since the update mainly reshapes when the model spends tokens rather than teaching new domain knowledge. In practice, a router learned from verifiable signals (i.e., math problems) can still reduce LONG usage on sci- ModelStageMath500AIME24AIME25LiveCodeBenchGPQA P@1 #Tok Think% P@1#TokThink% P@1#TokThink% P@1#TokThink% P@1 #Tok Think% Qwen3-8B Baseline (think) 96.9 5,547100.075.8 15,602100.068.3 18,419100.049.5 14,879100.058.8 5,838100.0 + SFT90.9 3,25145.850.4 11,19650.044.4 11,59351.334.4 10,51954.456.6 7,09296.8 + RPO88.0 2,41227.845.29,43738.140.69,65039.628.78,36240.157.1 6,90896.2 Qwen3-32B Baseline (think) 97.3 4,806100.079.4 13,235100.069.8 16,431100.059.1 13,498100.066.2 5,108100.0 + SFT92.1 2,60242.759.89,30956.354.2 10,37958.358.3 11,45790.666.3 4,82298.5 + RPO92.1 1,97520.463.1 10,29161.558.1 11,91168.154.4 10,62387.265.0 4,76897.6 Table 7: LoRA-based mode control and RPO on Qwen3. For each model size, we compare the<think>-forced baseline (LONG) with LoRA SFT for mode selection and a subsequent RPO update. We report Pass@1 (%), average generated tokens, and THINK% (fraction routed to LONG). We set batch size 64 and learning rate2× 10 −7 , and use the Qwen default prompt for math tasks. This table reports a LoRA-tuned variant of Qwen3-8B/32B and is therefore not directly comparable to the full-parameter SFT Qwen3-8B in Tab. 6; baseline numerical drift across the two tables reflects independent decoding runs under matched settings. Marker typeMathAvg P@1Think%LCB P@1GPQA P@1 Newline fork (default)62.044.835.455.4 Prefix sentence A61.845.135.155.2 Prefix sentence B61.546.334.854.9 Table 8: Comparison of root token marker types on Qwen3-8B after +RPO. Newline fork uses . . Prefix sentence A/B use distinct instructional prefixes (∼20–30 tokens) prepended before<think>to distin- guish modes. entific and coding benchmarks, but accuracy im- provements are not guaranteed because the opti- mal routing rule is dataset dependent (see the het- erogeneous Think%/P@1 trade-offs across Live- CodeBench and GPQA-Diamond in Table 6). Cal- ibration does not transfer perfectly either. Root confidence is informative but can be weakly cor- related on hard suites wherep short concentrates near low values. While RPO adaptively selects the mode depending on the learned query difficulty, in practice, deploying such way can be brittle across real-world scenarios. In this situation, we may need to treat checkpoint or threshold selection as dataset- conditioned calibration in case-by-case, since sim- ilar accuracy can coincide with materially different thinking ratios. B Implementation details B.1 Hyperparameter optimization for SFT Through fine-tuning R1-Distill, Qwen3, and EXAONE4 models on math benchmarks, we ob- served that preserving the both accuracy and num- ber of tokens of original model requires differ- ent hyperparameter configurations depending on model family and size. For instance, R1-Distill- 1.5B maintained accuracy with learning rate 5e-5, while accuracy degraded at other values; in con- trast, R1-Distill-7B required learning rate 1e-5 to preserve performance. When applying SFT with self-distilled data on Qwen3 models, we observed distinct behaviors across model sizes and modes. In reasoning mode, the 1.7B model exhibited a tendency to generate longer sequences, whereas the 8B model main- tained the original token count. In non-reasoning mode, both models generally produced longer out- puts; however, distilling with presence penalty 1.5 successfully maintained token counts comparable to the original models. For EXAONE4, using a lower temperature of 0.4 in non-reasoning mode preserved both accuracy and token efficiency simi- lar to the base model. B.2 Mode-specific GRPO rollout strategies For Qwen3 models, the non-reasoning mode pre- fix ( ) and reasoning mode prefix ( ) each correspond to a single distinct token. When gen- eratingnrollout samples, we prepended each to- ken ton/2sample prompts before generation. For EXAONE4, the non-reasoning mode prefix corre- sponds to two consecutive reasoning mode tokens. Unlike Qwen3, we prepended the reasoning mode prefix to all rollout samples initially. Forn/2sam- ples, we appended an additional token to enforce non-reasoning generation, and for the remaining samples, we enforced reasoning mode generation by masking the logits of the initial reasoning mode prefix token during sampling. B.3 SFT Dataset and training For SFT training, we used OpenR1-Math prompts to generate SFT dataset. We employed different strategies depending on model capabilities. For Qwen3 and EXAONE4 models, we applied self- distillation to generate both reasoning and non- reasoning trajectories. For Qwen3, we used tem- BackboneCheckpointModeMath AvgNon-math Avg P@1Think%P@1Think% Qwen3-8B Pre forced-think (LONG)78.6100.054.2100.0 forced-no-think (SHORT)43.40.034.90.0 +SFT adaptive63.857.330.756.6 forced-think (LONG)78.1100.052.0100.0 forced-no-think (SHORT)43.40.034.00.0 +RPO adaptive62.044.845.461.0 forced-think (LONG)77.3100.053.8100.0 forced-no-think (SHORT)42.60.035.00.0 Exaone4-1.2B (v2) Pre forced-think (LONG)62.9100.046.5100.0 forced-no-think (SHORT)37.60.031.40.0 +SFT adaptive60.080.140.480.1 forced-think (LONG)62.8100.044.7100.0 forced-no-think (SHORT)43.50.030.90.0 +RPO adaptive58.048.439.248.4 forced-think (LONG)60.8100.045.6100.0 forced-no-think (SHORT)41.50.031.20.0 Table 9: Robustness to forcing thinking vs. non-thinking after post-training. Math Avg is the mean P@1 over MATH500/AIME24/AIME25. Non-math Avg is the mean P@1 over LiveCodeBench and GPQA-Diamond. Think% is the fraction routed to thinking (100% and 0% under forced modes). perature 0.6, top-p 0.95, and top-k 20 for reasoning mode, and temperature 0.7, top-p 0.8, and top-k 20 for non-reasoning mode. For EXAONE4, we used temperature 0.4 and top-p 0.95 for both modes, with top-k 20 applied only in reasoning mode. We generated approximately 20K examples for each mode, resulting in about 40K total training samples per model. For Qwen3 V2 (LoRA), we employed n=4 sampling with temperature 0.6, top-p 0.95, top- k 20, and presence penalty 1.5 for both reasoning and non-reasoning modes. We selected the shortest correct solution from the sampled candidates to construct the SFT dataset. For R1-1.5B and 7B, which runs exclusively in reasoning mode, we utilized external resources. We sampled the original OpenR1-Math solution for reasoning mode trajectories and used Qwen2-Math- 72B-Instruct to distill corresponding non-thinking trajectories with temperature 0.7 and top-p 0.8. Ad- ditionally, since R1-Distilled-Qwen lacks specific control tokens to distinguish between reasoning and non-reasoning modes (unlike Qwen3’s . ), we prepended a short instructional prefixes to each mode. B.4 Dataset sizes for LoRA updates For our framework with LoRA updates, we use substantially fewer SFT queries than full parameter tuning. Mode selection only requires teaching the model to follow the two templates and to expose the root decision token, rather than relearning problem solving. Concretely, we draw math queries from OpenR1-Math (Open-R1, 2025), self generate can- didate trajectories under each mode, and keep only correctness and template valid demonstrations, but sample only 4K per mode (8K total) for LoRA SFT, instead of 20K per mode used in full fine tun- ing. We keep the RPO stage unchanged and run root token GRPO with the same rollout and reward settings as in the main experiments, so the reduced data primarily affects the SFT mode controllability step rather than the routing optimization. B.5 SFT stop criterion and sensitivity As described in 1, the SFT stage is intentionally designed as a controllability enabler rather than a capability-improving stage. We apply an explicit stop criterion: training continues only while the default-template accuracy remains within a toler- anceεof the original model. This prevents over- training, which can degrade both overall accuracy and routing quality (e.g., high Think% but low per- formance due to template drift). To illustrate sensitivity to SFT duration, we report forced-LONG accuracy at different SFT epochs on R1-Distill models in Table 10. Over- Table 10: SFT sensitivity to training epochs on R1- Distill models under forced-LONG evaluation. ModelStageMATH500 P@1 AIME24 P@1 R1-Distill-1.5B Baseline83.229.4 SFT 1 ep75.120.8 SFT 3 ep77.622.3 R1-Distill-7B Baseline93.554.2 SFT 1 ep90.443.5 SFT 2 ep88.044.5 training can degrade the solver: for R1-Distill- Qwen-1.5B, SFT at 1 epoch drops MATH500 from 83.2 to 75.1, while 3 epochs partially recovers to 77.6 but at higher training cost. For R1-Distill- Qwen-7B, 1 epoch (90.4) outperforms 2 epochs (88.0) on MATH500, confirming that the optimal SFT duration is model-dependent and motivating the tolerance-based stop rule. C Additional results C.1 LoRA updates Table 7 shows LoRA-based SFT and subsequent RPO on Qwen3 models. LoRA SFT mainly teaches template-following and root-level mode controllability, so it already yields meaningful re- ductions in Think% and tokens compared to the always-<think>baseline. Building on this control- lable initialization, RPO further shifts the root gat- ing distribution and provides a stronger compute- accuracy tradeoff, often reducing Think% substan- tially while keeping task performance non-trivial across both math and non-math benchmarks. The resulting operating point is model-size dependent, but consistently shows that root-only RL can re- fine routing without requiring full-trajectory policy updates. C.2 Robustness to hard forcing of thinking after post-training Our method learns a root-level routing policy that chooses between a thinking and a non-thinking gen- eration mode. A potential failure mode is entan- glement: post-training might specialize the model such that performance depends on the router, or that one mode becomes unreliable when selected deterministically. We therefore evaluate each checkpoint under two hard forcing interventions. In forced-think (LONG), we always select the reasoning root to- ken, requiring an explicit trace. In forced-no-think (SHORT), we always select the non-reasoning root Table 11: Budget-matched comparison on MATH500 (R1-Distill-1.5B). Under equal PFLOPs (∼10K), full-trajectory GRPO becomes under-trained and achieves only modest routing (Think%=90.2%), while RPO achieves comparable accuracy with substantially lower Think%. Different RPO checkpoints trace a Pareto frontier between accuracy and compute. SettingUpdate scopePFLOPsMATH500 P@1Think% Full GRPO (budget-matched)all tokens∼10K80.890.2 RPO (ckpt A)root-only∼10K77.620.5 RPO (ckpt B)root-only∼10K80.443.4 token, suppressing the trace. We report P@1, aver- age generated tokens, and the fraction of examples routed to thinking (Think%). Table 9 summarizes results on math reason- ing (MATH500/AIME24/AIME25) and non-math benchmarks (LiveCodeBench/GPQA-Diamond). Across backbones and checkpoints, we find that post-trained models remain functional under both forced regimes: forcing thinking preserves a large portion of the always-think baseline accuracy, while forcing non-thinking yields non-trivial per- formance with substantially reduced tokens. Over- all, these results indicate that SFT/RPOprimarily reshapes the routing distribution (i.e., when to al- locate reasoning), without collapsing either mode into a brittle specialization. C.3 Budget-matched comparison: root-only vs. full-trajectory RL A key question is whether the accuracy gap be- tween RPO and full-trajectory methods (e.g., AdaptThink) stems from the root-only formula- tion itself or simply from differences in training budget. To disentangle these effects, we compare RPO with full-trajectory GRPO (i.e., AdaptThink) under the same PFLOPs budget on MATH500 us- ing R1-Distill-Qwen-1.5B (Table 11). Under equal PFLOPs, full-trajectory GRPO is under-trained and barely reduces Think% (90.2%), whereas RPO achieves meaningful routing (20.5– 43.4% Think) with comparable or slightly lower accuracy. This confirms that root-only updates are more sample-efficient for the routing objective un- der low-budget regimes, which is the primary use case RPO targets. C.4 Post-RPO threshold routing To verify that inference-time controllability is pre- served after RPO training, we sweep the routing thresholdτon the final +RPO checkpoint (Ta- ble 12). Higherτroutes more queries to LONG, in- Table 12: Threshold routing (τ) sweep on the final +RPO checkpoint (Qwen3-8B).τcontrols the fraction routed to LONG. τThink% MathAvg P@1 Non-mathAvg P@1 0.9060.063.046.0 0.8052.362.545.8 0.7044.862.045.4 0.60 38.160.844.2 Table 13:Outcome bucket distribution on the RPO training set (Qwen3-8B). DifficultyLONG-onlySHORT-onlyBoth-correctBoth-wrong Easy (acc S ≥ 0.75)3.2%1.8%89.5%5.5% Medium28.4%4.1%42.3%25.2% Hard (acc S < acc L ≤ 0.5) 45.7%2.3%8.6%43.4% creasing accuracy at the cost of more tokens; lower τsaves compute with a controlled accuracy reduc- tion. This demonstrates thatτprovides a smooth, monotonic compute–accuracy trade-off even af- ter RPO training, with the forced-LONG/forced- SHORT results (Table 9) serving as upper/lower capability bounds. C.5 Automatic labeling quality Our automatic labeling assigns each query to one of four outcome buckets based on paired evaluation under both SHORT and LONG modes: (i) LONG- only-correct: only LONG succeeds, (i) SHORT- only-correct: only SHORT succeeds, (i) both- correct: both modes succeed, and (iv) both-wrong: neither succeeds. This paired design directly mit- igates the concern that SHORT might answer cor- rectly by chance: such instances fall into SHORT- only-correct or both-correct and are not used as evidence that LONG is required. Table 13 reports the bucket distribution for Qwen3-8B on the RPO training set. On hard queries (e.g., AIME-level), the LONG-only-correct bucket dominates, indicating a strong and consis- tent mode separation signal. The SHORT-only- correct bucket is small across all difficulty lev- els, suggesting that chance-correct SHORT events rarely contaminate the supervision. To make the bucket assignments concrete, we provide representative training-set queries per bucket together with a mechanistic rationale for why each problem falls into the assigned bucket under Qwen3-8B mode-conditioned generation (Ta- ble 14). The rationales are not post-hoc rationaliza- tions: they identify the specific reasoning step (e.g., modular case enumeration, sign disambiguation) at Table 14: Representative training-set queries per out- come bucket (Qwen3-8B). Queries are abbreviated; ra- tionales identify the reasoning step that drives the bucket assignment. BucketQuery (abbrev.)Mechanistic rationale LONG-only“Find the number of pos- itive integersn ≤ 2024 such thatn 2 +n + 1≡ 0 (mod 7).” Requires enumerating n mod 7(solutions: n ≡ 2, 4), then count- ing via floor arithmetic. SHORT typically guesses a single residue or skips the residue scan and returns an off-by-one count. SHORT-only“Compute cos(π/3).”Closed-formrecall (1/2). SHORT returns the memorized value. LONGoccasionally derives via half-angle identitiesandcon- flatescos(π/6)with cos(π/3),yielding √ 3/2. Both-correct“Solvelog 2 x = 5forx.”Single inverse step:x = 2 5 = 32 . No branch points or memorization gaps; both modes con- verge. Both-wrong“Find the smallest posi- tive integernsuch that the decimal expansion of1/n has period exactly 23.” Requires knowing pe- riod equalsord n (10) whengcd(n, 10)=1and that10 23 − 1admits a non-trivial prime fac- tor.Beyond Qwen3- 8B’s reliable recall in ei- ther mode. which SHORT mode systematically truncates, or at which LONG mode introduces a derivation tangent. C.6 Misrouting analysis We analyze routing errors by counterfactual evalu- ation: for each query where the adaptive model is incorrect, we re-evaluate under the opposite forced mode to determine whether the error is attributable to the gate decision. We categorize errors into: •False-SHORT (harmful misroute): the model chose SHORT but LONG would have suc- ceeded. These typically involve hidden multi- step constraints or subtle arithmetic that re- quire explicit deliberation. • False-LONG (wasteful misroute): the model chose LONG when SHORT would have suf- ficed, or when overthinking amplified distrac- tors or hallucinated intermediate claims. • Both wrong: neither mode succeeds (out-of- distribution or missing knowledge). • Both correct: compute-saving opportunity, not an error. Table 15: Misrouting composition on MATH500 (R1- Distill-1.5B, 500 queries). Loweringτ(0.75→0.70) in- creases the number of false-SHORT errors (+12) while only slightly reducing false-LONG errors (−4), reflect- ing a clear compute-risk trade-off. τFalse-SHORTFalse-LONGBoth-wrong 0.754770 0.70 16730 Table 16:Representative misrouting cases on MATH500 (R1-Distill-1.5B,τ=0.70). Adaptive reports the mode chosen by the learned router and its correct- ness; Counterfactual reports the opposite forced mode’s outcome on the same query. CategoryQuery (abbrev.)AdaptiveCounterfactual False-SHORT“How many distinct ar- rangements are there of the letters of MISSISSIPPI?” SHORT,✗ (returns 11!)LONG,✓ (11!/(4! 4! 2!) = 34,650) False-LONG“Compute P 10 k=1 k.” LONG,✗ (drifts to P k 2 formula)SHORT,✓ (55) Both-wrong“Find the smallestnsuch that the decimal expansion of1/nhas period exactly 23.” LONG,✗SHORT,✗ Both-correct“Solve3x + 7 = 22forx.”SHORT,✓LONG,✓ Table 15 reports the misrouting composition on MATH500 (R1-Distill-1.5B) at two threshold set- tings, extracted from counterfactual re-evaluation. As shown in Table 15, false-LONG (conservative over-routing) is the dominant error type, while false-SHORT errors are relatively rare and increase predictably asτdecreases. This suggests that the learned router errs on the side of caution (spending LONG when unnecessary) rather than harmfully skipping reasoning. A plausible source of this con- servative bias is the pretraining prior that corre- lates more tokens with trying harder. Root-level post-training can reshape the gate, but may not fully overwrite this prior, motivating future work on calibration-aware training. To make the failure modes diagnosable rather than aggregate, we provide representative cases per category in Table 16. Each row shows the routed mode under the adaptive policy together with the counterfactual outcome under the opposite forced mode, isolating whether the error is attributable to the gate decision (false-SHORT/false-LONG) or to a solver limitation (both-wrong). The false-SHORT example illustrates a query whose surface form (a permutation count) admits a one-line answer but requires a divisor correction that SHORT systemat- ically omits; the false-LONG example illustrates a query whose closed-form invites SHORT, while LONG introduces an unnecessary derivation chain that drifts to a sibling identity. Table 17: LiveCodeBench pass@kresults with token cost. Under pass@10 evaluation, the adaptive pol- icy reduces token cost while maintaining competitive pass@10 performance. ModelStagepass@1pass@10Avg Tokens Qwen3-32B Baseline59.175.313,498 + SFT58.377.111,457 + RPO54.472.810,623 Exaone4-1.2B Baseline42.262.912,159 + SFT34.656.09,281 + RPO36.758.26,692 C.7 Pass@k evaluation on coding tasks We additionally report pass@10 alongside token cost on LiveCodeBench (Table 17). Under the pass@klens, the adaptive policy often matches or improves pass@10 relative to the baseline while reducing average token cost, which is the trade-off view most relevant for coding deployment where multiple samples are drawn. D AI Usage We utilize AI tools (OpenAI’s ChatGPT and Google’s Gemini) to support code generation for experiments and help refine the text in a sentence- level. The core idea of our method, design of the framework, and the majority of the manuscript are written by the authors.

Instant research discovery