Paper deep dive

Towards Understanding Pause Token Fine-Tuning Dynamics: A Mode Retention Perspective

Jaehyeon Kim, Suhwan Kim, Nakyung Lee, Yeongoon Kim, Jimin Seo, Giho Lee, Jungwoo Lee

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 102

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 9/7/2026, 5:16:19 AM

Summary

This paper investigates the training dynamics of pause tokens in Large Language Models (LLMs), proposing that they function as a mechanism for mode retention and non-myopic compression rather than just computational expressivity. The authors introduce Masked Boundary Pause (MBP), a fine-tuning strategy where pause tokens are inserted at reasoning-step boundaries with their loss masked. Experiments on Qwen and Llama models (1B-8B) demonstrate that MBP improves math and code reasoning performance while preserving general language capabilities, outperforming standard Supervised Fine-Tuning (SFT) and other pause placement strategies. The method is also shown to extend effectively to Group Relative Policy Optimization (GRPO).

Entities (3)

Masked Boundary Pause · method · 98%Llama · model · 95%Qwen · model · 95%

Relation Signals (11)

Masked Boundary Pause → improves → Reasoning

confidence 95% · MBP consistently improves reasoning, achieving gains of up to 6 points on math and 2.5 points on code

Qwen → isevaluatedwith → Masked Boundary Pause

confidence 95% · Across 1B-8B Qwen and Llama models, MBP consistently improves reasoning

Llama → isevaluatedwith → Masked Boundary Pause

confidence 95% · Across 1B-8B Qwen and Llama models, MBP consistently improves reasoning

Masked Boundary Pause → isbasedon → Mode Retention

confidence 93% · H1, mode retention... We formalize a training rule consistent with both - Masked Boundary Pause (MBP)

Masked Boundary Pause → isbasedon → Non-myopic Compression

confidence 93% · H2, non-myopic compression... We formalize a training rule consistent with both - Masked Boundary Pause (MBP)

Masked Boundary Pause → preserves → general language understanding

confidence 92% · while preserving general language understanding abilities.

Masked Boundary Pause → extendsto → GRPO

confidence 90% · this mode-preserving strategy extend gains to GRPO.

GSM8K → isusedtoevaluate → Masked Boundary Pause

confidence 90% · We evaluate reasoning on math (GSM8K...)

MATH → isusedtoevaluate → Masked Boundary Pause

confidence 90% · We evaluate reasoning on math (...MATH...)

HumanEval → isusedtoevaluate → Masked Boundary Pause

confidence 90% · and code (HumanEval(+), MBPP(+))

MBPP → isusedtoevaluate → Masked Boundary Pause

confidence 90% · and code (HumanEval(+), MBPP(+))

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Pause-token methods improve LLM reasoning by inserting special tokens into sequences. Prior work explains these gains through computational expressivity. However, there is relatively little investigation into the training dynamics of pause tokens. We explore how pause tokens reshape the training dynamics of fine-tuning. Two controlled pilots expose distinct asymmetries. On a synthetic continual-learning task, masked pauses overwrite a previously-learned distribution roughly 4x less at matched final adaptation (H1, mode retention); on a synthetic math-reasoning probe, the boundary-adjacent token comes to encode substantially more downstream-step information (H2, non-myopic compression). We formalize a training rule consistent with both - Masked Boundary Pause (MBP), pause tokens placed at reasoning-step boundaries with their loss masked. Across 1B-8B Qwen and Llama models, MBP consistently improves reasoning, achieving gains of up to 6 points on math and 2.5 points on code, while preserving general language understanding abilities. We further demonstrate that this mode-preserving strategy extend gains to GRPO. These results recast pause tokens as a training-dynamics intervention on the retention-adaptation trade-off, rather than merely an inference-time computation device.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

98,771 characters extracted from source content.

Expand or collapse full text

Towards Understanding Pause Token Fine-Tuning Dynamics: A Mode Retention Perspective Jaehyeon Kim Affiliation: HodooAI Lab, Seoul, South Korea Suhwan Kim Affiliation: Seoul National University, Seoul, South Korea Nakyung Lee Affiliation: Seoul National University, Seoul, South Korea Yeongoon Kim Affiliation: Seoul National University, Seoul, South Korea Jimin Seo Affiliation: Seoul National University, Seoul, South Korea Giho Lee Affiliation: HodooAI Lab, Seoul, South Korea Jungwoo Lee Affiliation: HodooAI Lab, Seoul, South Korea Affiliation: Seoul National University, Seoul, South Korea *Equal contribution. Corresponding author.Correspondence: junglee@snu.ac.kr Abstract Pause-token methods improve LLM reasoning by inserting special tokens into sequences. Prior work explains these gains through computational expressivity. However, there is relatively little investigation into the training dynamics of pause tokens. We explore how pause tokens reshape the training dynamics of fine-tuning. Two controlled pilots expose distinct asymmetries. On a synthetic continual-learning task, masked pauses overwrite a previously-learned distribution roughly 4× less at matched final adaptation (H1, mode retention); on a synthetic math-reasoning probe, the boundary-adjacent token comes to encode substantially more downstream-step information (H2, non-myopic compression). We formalize a training rule consistent with both — Masked Boundary Pause (MBP), pause tokens placed at reasoning-step boundaries with their loss masked. Across 1B-8B Qwen and Llama models, MBP consistently improves reasoning, achieving gains of up to 6 points on math and 2.5 points on code, while preserving general language understanding abilities. We further demonstrate that this mode-preserving strategy extend gains to GRPO. These results recast pause tokens as a training-dynamics intervention on the retention–adaptation trade-off, rather than merely an inference-time computation device. 1 Introduction Figure 1: Comparison between standard fine-tuning and masked boundary pause fine-tuning. Standard SFT applies next-token prediction (NTP) loss at every task-token position, allowing gradients from the target token to flow directly into the preceding context representation. This can strongly adapt the model to the task-data distribution, but may also erode the rich predictive distribution learned during pre-training. Masked boundary pause fine-tuning (MBP) instead inserts pause tokens at selected boundaries and masks their loss, thereby blocking the direct gradient path into the boundary context state. The model receives update signals only indirectly through neighboring token predictions, enabling task adaptation while better preserving the original pre-training distribution. Recent progress in large language models (LLMs) has produced a wide range of training-time and test-time methods for improving reasoning: chain-of-thought prompting, test-time scaling, and reinforcement learning for reasoning achieve strong performance on complex tasks Wei et al. (2022b); Snell et al. (2025); Wang et al. (2023). However, these methods incur substantial costs in generation length, inference latency, sampling budget, or training compute. Pause- and filler-token methods stand out within this landscape Goyal et al. (2024); Burtsev et al. (2021). They improve performance through a small and architecture-preserving change by inserting special tokens into sequences without modifying the model or the optimizer. Pause-token gains have been attributed to forward-pass computation and expressivity Pfau et al. (2024); London and Kanade (2025), focal-loss amplification at hard tokens Kim et al. (2025), and structural anchoring at sentence boundaries Liu et al. (2026). These explanations center on effects at the inserted positions themselves. Yet much less is known about how pause training shapes the pretrained distribution from which fine-tuning starts, even though this distribution is central to modern adaptation pipelines that increasingly warm-start from existing checkpoints Gururangan et al. (2020); Roziere et al. (2023); Azerbayev et al. (2024). We argue that this is precisely where an underexamined role of pause tokens is most consequential. Pause tokens also participate in the forward pass during fine-tuning, where they alter the context under which target tokens are predicted and reshape how gradient flows through the surrounding positions. Wu et al. (2024); Pfau et al. (2024) These training-dynamics effects are present even when pause tokens are absent at inference and never appear as supervised targets, and they directly modulate the retention–adaptation trade-off central to fine-tuning. In a controlled continual-learning setting, we observe that loss-masked pauses reduce overwriting of a previously learned distribution by roughly 4×4× at matched final adaptation, revealing a separation that neither inference-time nor low-confidence framings predict. We investigate this through two controlled pilots that yield distinct empirical phenomena; a retention asymmetry on synthetic continual learning and an information-packing asymmetry on iGSM Ye et al. (2025) probing, which we elevate to working hypotheses H1 (mode retention) and H2 (non-myopic compression). We then formalize a training rule consistent with both hypotheses, Masked Boundary Pause (MBP)11 1 Code is available at https://github.com/jkpizza/emnlp2026-pause-token-dynamics., and test the hypotheses’ scaled predictions across 1B–8B fine-tuning and GRPO. The headline finding is that boundary placement and loss masking are jointly necessary for the gains, with specific results reported in §5–§7.3. In summary, our contributions are: • Through controlled pilots, we identify two hypotheses: (H1) mode retention, and (H2) non-myopic compression. • Motivated by these, we propose Masked Boundary Pause (MBP) fine-tuning, where pause tokens are inserted at sentence or reasoning-step boundaries while masking their cross-entropy loss. • Across 1B–8B models, MBP improves math and code reasoning while preserving general capability; gains extend to GRPO. A four-way placement ablation shows that boundary placement and loss masking are jointly necessary. 2 Preliminaries We formalize masked-pause fine-tuning under a single notation that subsequent sections re-use, and locate prior pause-token methods within the same framework. 2.1 Pause-token training Formulation. Let (x,y)(x,y) be a (prompt, response) pair drawn from a fine-tuning dataset D, with response y=(y1,…,yN)y=(y_1,…,y_N). Standard supervised fine-tuning (SFT) minimizes the per-token conditional cross-entropy of y given x; we recover it as the g⁡(x,y)=∅g(x,y)= case of the formulation below. A pause-token training method is parameterized by a position selector g that returns insertion indices g(x,y)=i1<i2<⋯<iM⊆0,…,Ng(x,y)=\i_1<i_2<·s<i_M\ \0,…,N\. With the conventions i0=0i_0=0 and iM+1=Ni_M+1=N, these indices partition y into M+1M+1 contiguous segments un=(yin−1+1,…,yin),n=1,…,M+1,u_n=(y_i_n-1+1,…,y_i_n), n=1,…,M+1, (1) with y=u1|u2|⋯|uM+1y=u_1 u_2 ·s u_M+1. A distinguished pause token τ is inserted between adjacent segments, yielding the pause-augmented sequence y~=u1|τ|u2|τ|⋯|τ|uM+1 y=u_1 τ u_2 τ ·s τ u_M+1 (2) of length N+MN+M. Let Sτ=k:y~k+1=τS_τ=\k: y_k+1=τ\ be the set of prediction positions whose target is a pause token. Following prior work Goyal et al. (2024); Kim et al. (2025), we exclude SτS_τ from the loss: pause tokens appear in the autoregressive context but never as supervised outputs. The per-example training loss is ℒg(θ)=−1N∑k∉Sτlogpθ(y~k+1∣x,y~1:k),L^g(θ)=- 1N _k∉ S_τ p_θ ( y_k+1 x, y_1:k ), (3) averaged over (x,y)∼(x,y) . Setting g⁡(x,y)=∅g(x,y)= recovers standard SFT. Relation to existing work. Prior work differs in the position selector g: Append Goyal et al. (2024); Pfau et al. (2024), DIT Kim et al. (2025), and Boundary at sentence endings Liu et al. (2026). We additionally test Random placement as an unstructured control. Of these, Goyal et al. (2024) and Kim et al. (2025) exclude pause positions from the loss as in Eq. 3, whereas Liu et al. (2026) trains pauses as predicted targets. Our method, Masked Boundary Pause Training (§5), takes g=Boundaryg= Boundary with SτS_τ masked, and Section 6.1 compares the four placements. 2.2 GRPO Formulation. Group Relative Policy Optimization (GRPO) Shao et al. (2024) optimizes a policy πθ _θ on prompts q∈q by sampling G responses yii=1G∼πθold(⋅∣q)\y_i\_i=1^G _ _old(· q) from the previous policy, scoring each with a reward ri=R⁡(q,yi)r_i=R(q,y_i), and computing the group-relative advantage Ai=(ri−μr)/(σr+ϵ)A_i=(r_i- _r)/( _r+ε) where μr,σr _r, _r are the group’s mean and standard deviation of rewards. With per-token policy ratio ρi,t(θ)=πθ(yi,t∣q,yi,<t)/πθold(yi,t∣q,yi,<t) _i,t(θ)= _θ(y_i,t q,y_i,<t)/ _ _old(y_i,t q,y_i,<t), GRPO maximizes GRPO(θ) _GRPO(θ) =⁡[1G∑i,tci,t(θ)−βDKL], =E\! [ 1G _i,tc_i,t(θ)-β D_KL ], (4) ci,t(θ) c_i,t(θ) =min(ρi,tAi, = \! ( _i,t\,A_i,\, OPENclip⁡(ρi,t, 1−ϵ, 1+ϵ)Ai). ( _i,t,\,1-ε,\,1+ε)\,A_i ). where DKLD_KL is a per-token KL penalty against a reference policy. 3 Pilot Experiments The introduction posed a mechanistic question: what is masked-pause fine-tuning actually doing to a pretrained model’s representations? We approach this in two controlled studies. The first asks whether masked pauses change what gets overwritten during adaptation (§3.1). The second asks what gets encoded at the boundary-adjacent token (§3.2). Each pilot yields a distinct empirical asymmetry that we crystallize into a working hypothesis. 3.1 Modular continual learning: a retention asymmetry Our first pilot asks whether masked pauses change what gets overwritten during adaptation. We use the simplest setting that can support a clean before/after comparison, a synthetic continual-learning task with two disjoint regimes. Detailed experimental settings are provided in Appendix A; Appendix A.2 extends the pilot to a four-regime chain. Setup. Sequences are generated by a Modular Linear Chain (MLC), xt+1=(atxt+bt)modp,x_t+1=(a_t\,x_t+b_t) p, (5) where a regime is defined by a length-L coefficient sequence (at,bt)t=1L\(a_t,b_t)\_t=1^L. We sample two disjoint coefficient sets to define regimes D1D_1 and D2D_2, and train a small GPT-style Transformer sequentially: Phase 1 trains on D1D_1 until accuracy exceeds 0.990.99; Phase 2 forks from a single Phase-1 checkpoint and continues on D2D_2 under two matched conditions sharing architecture, optimizer, batch size, learning rate, and update budget: • No pause: standard next-token prediction on raw D2D_2 sequences. • Pause: one pause token inserted after every content token; pause-target positions are excluded from the loss. Figure 2: Phase-2 dynamics from a D1D_1-pretrained backbone (n=8n=8 seeds, ±1σ± 1σ). Top: D2D_2 adaptation (held-out loss, log scale). Bottom: D1D_1 retention (held-out loss, linear scale). Both conditions reach the same D2D_2 floor; Pause retains D1D_1 ∼4× 4× better than No pause. Results. Figure 2 shows the Phase-2 loss curves on both regimes. Both conditions adapt to D2D_2 to the same ∼10−7 10^-7 floor, with No pause marginally faster in descent but indistinguishable at convergence. The conditions diverge sharply in how much they overwrite D1D_1: No pause plateaus near 66 nats of held-out D1D_1 loss while Pause plateaus near 1.51.5 — a ∼4× 4× retention gap, consistent across seeds. The gap occurs at matched D2D_2 fit and is not explained by tokens-processed per step. Hypothesis. Masked-pause training matches the new regime’s final fit while preserving the previous regime substantially better: H1 (mode retention). Masked-pause training reduces overwriting of previously-learned distributions during adaptation, relative to standard fine-tuning at matched final performance. 3.2 iGSM step probing: an information-packing asymmetry The MLC pilot shows masked pauses change adaptation dynamics, but not what the representation absorbs. To localize the mechanism, we probe what the boundary-adjacent token — the period immediately preceding an inserted [PAUSE] — encodes after training. We use iGSM Ye et al. (2025), a synthetic mathematical reasoning dataset where each reasoning step concludes with a deterministic integer answer, providing the clean per-step probe targets this analysis requires. Setup. We supervise-fine-tune Qwen3-0.6B-Base Yang et al. (2025) on 3K iGSM problems under two conditions — No pause and Pause (one [PAUSE] inserted at each step boundary, loss masked) — and compare against the pretrained Base checkpoint (n=3n=3 seeds each). For each held-out problem we extract the final-layer hidden state at the boundary-adjacent period of every reasoning step and fit two linear probes: one for the integer answer of the current step, one for the next step. Both fine-tuned conditions reach comparable held-out task accuracy, so the probe compares representations at matched task fit. Results. Figure 3 shows masked-pause SFT yielding the highest decodability on both targets: current-step accuracy roughly doubles over no pause (0.2820.282 vs. 0.1330.133, and clearly above the base 0.1820.182), and next-step accuracy is highest as well (0.1040.104 vs. 0.0620.062 for no pause and 0.0750.075 for base). Training the same boundary pauses as prediction targets preserves most of the effect (0.2700.270 and 0.0940.094), indicating that the compression follows primarily from pause placement, with masking a small additional increment. The pattern is consistent across seeds. Figure 3: Linear-probe accuracy on the boundary-adjacent token’s final hidden state (n=3n=3 seeds, ±1σ± 1σ). Left: current step’s integer answer. Right: next step’s integer answer. Pause (unmasked) trains the same boundary pauses as prediction targets. Hypothesis. Under ordinary SFT, the period token must predict the first token of the next reasoning step; inserting a pause replaces that target with [PAUSE], and masking additionally removes its loss. Either way the period’s representation is freed from immediate next-token specialization and can instead absorb gradient from downstream positions: H2 (non-myopic compression). Boundary pauses redirect gradient flow such that boundary-adjacent tokens encode more downstream information than under ordinary SFT. 3.3 From hypotheses to predictions Motivated by H1 and H2, the rest of the paper tests three predictions that follow if they hold at scale on real reasoning tasks: • RQ1 (scaling). Does masked boundary pause fine-tuning at 1B–8B improve math and code reasoning while preserving general capability (MMLU Hendrycks et al. (2021a), GPQA Rein et al. (2024), BBH Suzgun et al. (2023)) better than standard fine-tuning? H1 predicts this trade-off. • RQ2 (placement). Does pause placement affect performance at matched pause-token budget? If so, which strategy is optimal? H1 and H2 both act locally at the inserted positions, predicting that placement matters and that boundary insertion outperforms random, prompt-append Goyal et al. (2024), and low-confidence Kim et al. (2025) alternatives. • RQ3 (RL exploration). Do GRPO rollouts under MBP retain wider behavioral support than standard GRPO — more per-prompt reward variance, slower distribution sharpening, and improved convergence? H1 predicts so. Sections 5–8 formalize MBP and answer these questions via 1B–8B math/code SFT with OOD-retention measurement, and GRPO experiments. 4 A retention certificate for masked-pause SFT H1 admits a first-order retention certificate. The intuition: old D1D_1 predictions on no-pause prefixes survive D2D_2 fine-tuning when their top-token margins outlast the logit movement induced by adaptation. Bounding this movement uniformly over a protected prefix set converts retention into a comparison between the certified drifts of two procedures. Let Φ⁡(r) (r) denote the fraction of protected D1D_1 prefixes whose original margin exceeds r. Theorem 4.1 (Retention certificate, informal). In the local linearized regime around the D1D_1-trained parameters and matched to the same D2D_2 fitting level, let δO _O be the certified protected drift of ordinary SFT on no-pause D1D_1 logits. Masked-pause SFT has certified protected drift at most βctxδO _ctx _O, where βctx _ctx measures how separated the pause-perturbed prefix is from the ordinary prefix in the Jacobian sense. Consequently, Retain⁡(WSFT) (W_SFT) ≥Φ⁡(2δO), ≥ (2 _O), Retain⁡(Wpause) (W_pause) ≥Φ⁡(2βctxδO). ≥ (2 _ctx _O). Since Φ is non-increasing, whenever βctx<1 _ctx<1, masked-pause SFT inherits a stronger retention guarantee than ordinary SFT. The condition βctx<1 _ctx<1 has a geometric reading: masked-pause SFT fits D2D_2 with less drift on D1D_1 no-pause logits whenever the pause-perturbed state is sufficiently separated from the ordinary state. This separation is structural rather than architectural — a property of where pauses are inserted, which the boundary selector of MBP encodes by design. The full setup, lemma statements (margin stability, protected drift bounds), and proofs are in Appendix B, C. 5 Method Motivated by H1 and H2 in §3, we formalize the masked pause training strategy that are predicted to exhibit optimal performance. 5.1 Masked Boundary Pause Training MBP specializes the position selector g from §2 to a boundary selector gBoundaryg_Boundary that produce segments unu_n of Eq. 1 that are semantic units: sentences, reasoning steps, or code lines. Further details can be found in Appendix E.2. Boundary placement is the advantageous choice for both hypotheses: • H1 (mode retention). Step boundaries are by construction where semantic transitions occur, and we expect fine-tuning’s structural learning to concentrate there. A pause embedding at these positions absorbs that learning, preserving the pretrained distribution that would otherwise drift (§3.1). • H2 (non-myopic compression). Inserted pause positions encourage implicit summarization at the preceding token (§3.2), so semantic boundaries are the natural placement: there the compression encapsulates a complete semantic unit, rather than being forced to predict the next token mid-sentence. The MBP training loss is Eq. 3 with g=gBoundaryg=g_Boundary, and all placement baselines in §6.1 use the same pause-token budget M so that comparisons isolate the role of g. 5.2 Extension to GRPO MBP extends to GRPO without changing rollouts or rewards: responses are sampled and scored on the clean policy, while pauses appear only when computing the policy update. Given a sampled response yiy_i, we form y~i y_i (Eq. 2) and apply the GRPO objective at non-pause positions, with policy ratio and KL penalty computed under the pause-perturbed prefix (q,y~i,<j)(q, y_i,<j) rather than the clean prefix (q,yi,<j)(q,y_i,<j). Pauses thus remain outside the action space, perturbing only the policy update’s context — the mechanism through which we test RQ3 (§3.3): whether retention of the pretrained distribution improves GRPO’s exploration. 6 Experiment Setup We test the predictions of §3.3 in two settings: downstream math/code supervised fine-tuning (§6.1, RQ1 and RQ2) and GRPO-based reasoning training (§6.2, RQ3). Full hyperparameters and data details are in Appendix D. 6.1 Downstream Math/Code SFT Setup. We fine-tune Qwen3 (1.7B, 4B, 8B-Base)Yang et al. (2025) and Llama-3.1-8B-Instruct Grattafiori et al. (2024) on math data (83K curated DeepMath-103K He et al. (2025) subset) and code data (100K OpenCodeInstruct subset Ahmad et al. (2025)), inserting pauses only on the response side. We apply LoRA Hu et al. (2022) for 8B models. We evaluate reasoning on math (GSM8K, MATH, AMC23, AIME24, Minerva) Cobbe et al. (2021); Hendrycks et al. (2021b); Lewkowycz et al. (2022) and code (HumanEval(+), MBPP(+)) Chen et al. (2021); Austin et al. (2021); Liu et al. (2023) benchmarks; general-capability retention on MMLU, GPQA, HellaSwag, BBH, PIQA Hendrycks et al. (2021a); Rein et al. (2024); Zellers et al. (2019); Suzgun et al. (2023); Bisk et al. (2020). We report pass@1 for math and code, and pass@4 for general-capability benchmarks. Baselines (matched pause-token budget). SFT (no pauses); Append: pauses at the prompt-response boundary Goyal et al. (2024); Random: pauses at random response positions; DIT: pauses before low-likelihood target tokens Kim et al. (2025); MBP (ours): pauses at reasoning-step or code-line boundaries. All pause variants are loss-masked except MBP w/o masking, which supervises pause tokens as targets (§8.1). Matching the budget isolates placement from token count. 6.2 GRPO Setup We compare Standard GRPO against MBP GRPO on Qwen3-1.7B and 4B, training on 5K-problem subsets of DeepMath-103K He et al. (2025) and SimpleRLZoo Zeng et al. (2025). We report pass@1 on the math benchmarks above, and track two training-dynamics metrics: convergence AUC (area under the training reward curve, measuring how quickly reward accumulates), and mixed-group ratio (proportion of rollout groups containing both correct and incorrect rollouts). Model Method GSM8K MATH500 MATH AMC23 MINERVA Avg. Qwen3-1.7B Base Base 65.13 54.20 54.82 35.00 17.65 45.36 SFT 82.41 63.40 62.98 27.50 20.59 51.38 MBP 83.02 61.00 64.56 37.50 23.16 53.85 Qwen3-4B Base Base 82.56 66.00 67.64 37.50 28.68 56.48 SFT 91.36 69.20 71.24 45.00 29.78 61.32 MBP 90.07 73.40 72.32 50.00 30.51 63.26 Qwen3-8B Base Base 88.40 65.40 65.86 42.50 26.84 57.80 SFT 92.95 71.60 72.54 52.50 32.72 64.46 MBP 92.57 75.00 72.84 55.00 35.29 66.14 Llama3.2-1B Instruct Base 38.97 23.80 25.70 15.00 7.72 22.24 SFT 50.72 28.00 38.48 10.00 8.09 27.06 MBP 51.63 31.80 41.66 27.50 14.34 33.39 Llama3.2-3B Instruct Base 70.43 40.00 43.22 10.00 17.65 36.26 SFT 76.80 46.40 55.02 27.50 13.97 43.94 MBP 77.03 48.20 55.92 30.00 19.85 46.20 Llama3.1-8B Instruct Base 77.48 46.40 48.62 26.50 22.43 44.29 SFT 84.84 53.20 56.28 17.50 21.69 46.70 MBP 86.28 58.00 61.02 27.50 22.43 51.05 Table 1: Math-domain results. Average denotes the macro-average over GSM8K, MATH500, MATH, AMC23, and MINERVA. Model Method MBPP MBPP+ HumanEval HumanEval+ Avg. Qwen3-1.7B Base Base 69.05 59.26 58.54 53.05 59.98 SFT 69.58 62.43 67.07 61.59 65.17 MBP 73.02 62.43 70.12 65.24 67.70 Qwen3-4B Base Base 74.34 64.81 76.83 72.56 72.14 SFT 78.04 67.99 79.27 73.78 74.77 MBP 79.10 65.08 82.93 78.05 76.29 Qwen3-8B Base Base 82.80 72.49 88.41 84.15 81.96 SFT 86.51 75.13 85.98 81.71 82.33 MBP 87.30 75.66 86.59 82.32 82.97 Llama3.1-8B Instruct Base 67.46 59.52 67.68 62.20 64.22 SFT 68.25 60.05 68.29 64.02 65.15 MBP 68.52 60.05 69.51 64.02 65.53 Table 2: Code-domain results. Average denotes the macro-average over MBPP, MBPP+, HumanEval, and HumanEval+. 7 Results We answer RQ1 (§7.1), RQ2 (§7.2), and RQ3 (§7.3) in order. Across these results, the findings show the patterns predicted by H1 and H2. 7.1 Scaling and Retention (RQ1) Reasoning gains scale across model sizes and families. Tables 1 and 2 show that MBP outperforms standard SFT on every model variant we tested. Math gains over SFT range from +1.7+1.7 to +6.3+6.3 points across Qwen3 (1.7B-8B-Base) and Llama (3.2-1B/3B-Instruct, 3.1-8B-Instruct); code gains range from +0.6+0.6 to +2.5+2.5 across Qwen3. The improvement is consistent in sign across every model variant, showing that pause training is not a model-specific phenomenon. General capability is preserved. Table 3 reports MMLU, GPQA, HellaSwag, BBH, and PIQA on Qwen3-1.7B-Base after math and code SFT. Standard SFT is at best neutral on general capability after math SFT and degrades it after code SFT. By contrast, MBP improves the general-benchmark average over base in both settings (+4.2+4.2 after math, +2.5+2.5 after code), beating SFT by +3.1+3.1 and +4.8+4.8, respectively. The MBP-vs-SFT gap on general benchmarks is larger than the gap on the trained-domain benchmarks themselves. This suggests that MBP does not trade reasoning gains for retention loss, but improves both simultaneously. These findings support H1 at scale: MBP preserves the pretrained distribution while still acquiring new-task reasoning. Trained Domain Method MMLU GPQA HellaSwag BBH PIQA Avg. Math Base 62.95 27.46 38.69 38.84 62.13 46.01 SFT 62.32 24.78 40.39 42.06 66.10 47.13 MBP 63.93 27.23 48.32 44.13 67.63 50.25 Code Base 62.95 29.91 38.00 38.84 62.13 46.37 SFT 60.19 23.88 36.44 39.15 60.66 44.06 MBP 63.78 25.67 44.01 42.88 68.01 48.87 Table 3: General-domain results on Qwen3-1.7B-Base. Average denotes the macro-average over MMLU, GPQA, HellaSwag, BBH, and PIQA. 7.2 Placement and Masking (RQ2) H1 and H2 predict that boundary placement activates both mechanisms: placements without alignment to semantic transitions (append, random) or correlated with hard tokens rather than transitions (DIT) should fail to do so. We test this on Qwen3-1.7B-Base at matched pause-token budget. Boundary placement is the only consistent winner. Table 4 reveals a 7.57.5-point spread across the four placements on math (from DIT’s −5.0-5.0 to MBP’s +2.5+2.5 relative to SFT), despite identical pause-token budgets. Of the four, only MBP is reliably positive on both math and code domains. The other placements fluctuate: Append and DIT modestly help code (+1.4+1.4, +1.7+1.7) but hurt math (−0.9-0.9, −5.0-5.0). Random is near-flat on code (+0.2+0.2) and clearly negative on math (−1.7-1.7). The inconsistency suggests that random, append, and low-confidence positions activate H1 and H2’s mechanisms only when an inserted pause happens to coincide with a meaningful boundary, whereas Boundary placement targets these positions by construction and delivers reliable gains across domains. Method Math Avg. GSM8K MATH500 MATH AMC23 MINERVA Base 65.13 54.20 54.82 35.00 17.65 45.36 SFT 82.41 63.40 62.98 27.50 20.59 51.38 Append 79.68 59.20 60.96 30.00 22.79 50.53 Random 81.35 60.80 63.26 20.00 23.16 49.71 DIT 68.01 57.20 56.38 30.00 20.22 46.36 MBP (ours) 83.02 61.00 64.56 37.50 23.16 53.85 MBP w/o masking 80.97 62.20 62.64 27.50 23.16 51.29 Method Code Avg. MBPP MBPP+ HumanEval HumanEval+ Base 69.05 59.26 58.54 53.05 59.98 SFT 69.58 62.43 67.07 61.59 65.17 Append 69.84 60.58 70.73 65.24 66.60 Random 70.37 61.11 67.07 62.80 65.34 DIT 69.84 60.58 71.95 65.24 66.90 MBP (ours) 73.02 62.43 70.12 65.24 67.70 MBP w/o masking 70.63 58.99 67.68 62.20 64.88 Table 4: Method ablation results on Qwen3-1.7B-Base. The upper block reports math benchmarks, and the lower block reports code benchmarks. 7.3 Extension to GRPO exploration (RQ3) If MBP preserves the pretrained distribution (H1), GRPO rollouts should retain wider behavioral support, yielding a richer gradient signal. We test this on Qwen3-1.7B-Base and Qwen3-4B-Base. Faster convergence, higher final accuracy. Table 5 shows MBP-GRPO improves final four-benchmark average accuracy by +1.31+1.31 points (Qwen3-1.7B) and +0.40+0.40 points (Qwen3-4B) over standard GRPO, while normalized convergence AUC increases by +0.048+0.048 and +0.045+0.045, respectively. Early rollout diversity drives the gain. Figure 4 shows that during the earliest steps, MBP-GRPO’s mixed-group ratio spikes substantially above standard GRPO, generating more useful gradient updates when the advantage signal is most valuable. This can be observed in reward and accuracy curves. MBP-GRPO pulls ahead from the first steps and stays ahead throughout training. Later, MBP’s mixed-group ratio falls below standard’s, consistent with earlier convergence on solved prompts. The pattern matches H1 in that a retained pretrained distribution provides more diverse rollouts when they most help, and the early advantage compounds. Figure 4: Training dynamics of MBP-GRPO. Columns correspond to Qwen3-1.7B-Base (left) and Qwen3-4B-Base (right), while rows show GSM8K validation pass@1 (top), training reward (middle), and mixed-group ratio (bottom). Red denotes MBP-GRPO and blue denotes standard GRPO. MBP-GRPO exhibits a higher mixed-group ratio in the early stage of training, providing more useful update signals and leading to faster convergence than standard GRPO. Method GSM8K MATH AMC23 Minerva AUC Qwen3-1.7B-Base Standard 80.52 59.68 37.50 20.96 0.366 MBP 81.35 61.24 40.00 21.32 0.414 Qwen3-4B-Base Standard 92.19 74.30 50.00 30.15 0.505 MBP 91.58 75.26 57.50 23.90 0.550 Table 5: Final GRPO pass@1 accuracy (%) on GSM8K, MATH, AMC23, and Minerva, together with normalized training-reward AUC, for Qwen3-1.7B-Base and Qwen3-4B-Base. Full results are reported in Appendix Table 19. 8 Analysis 8.1 Retention Analyses We examine H1’s retention mechanism through a decomposition of post-training correctness, an ablation on loss-masking, and next-token entropy at step boundaries. MBP preserves base-correct examples while solving new ones. If H1 holds at scale, MBP should preserve pretrained capabilities while fitting to the downstream distribution. We measure both directly: BCP (Base-Correct Preservation) is the fraction of base-correct examples still correct after fine-tuning, and NSR (New-Solve Rate) is the fraction of base-wrong examples that become correct. Table 6 shows MBP improves both BCP and NSR. This shows that MBP achieves sufficient adaptation while better preserving the pretrained distribution. Metric Method GSM8K MATH Overall BCP ↑ Standard 92.08 88.33 89.18 MBP 93.36 89.09 90.00 NSR ↑ Standard 63.04 33.42 38.20 MBP 66.74 36.83 41.11 Acc. ↑ Standard 81.96 63.52 67.14 MBP 84.08 65.48 68.87 Table 6: BCP(Base-Correct Preservation), NSR(New-Solve Rate), and final accuracy of Standard-SFT and MBP on Qwen3-1.7B-Base. Masking is essential at boundary placement. We predict that masking pause tokens would be advantageous in retaining the pretrained distribution. To test this, we perform ablation experiments in math and code training where pauses are predicted as targets Liu et al. (2026). Table 4 (MBP w/o masking) shows that switching from masked to predicted costs −2.6-2.6 points on math and −2.8-2.8 on code; on code, predicted boundary placement even falls below standard SFT, confirming that masking is load-bearing. Entropy is retained at step boundaries. H1 predicts that pause training selectively preserves diversity at boundaries. We therefore measure entropy both at step-boundary positions and across all token positions on out-of-distribution (OOD) general language understanding benchmarks, with the in-distribution GSM8K and MATH benchmarks included for comparison (Table 7). At step boundaries, MBP maintains substantially higher entropy than SFT across both in-domain and OOD tasks, suggesting that MBP selectively preserves the base model’s distributional diversity at these positions. In contrast, the average entropy over all tokens remains almost identical between SFT and MBP. Taken together, these results indicate that MBP selectively preserves distributional diversity at step boundaries while committing to a distribution within steps. OOD In-domain Model MMLU GPQA GSM8K MATH Step boundaries Base 0.707 0.607 0.500 0.579 SFT 0.403 0.380 0.278 0.339 MBP 0.463 0.440 0.365 0.380 All tokens Base 0.587 0.489 0.358 0.258 SFT 0.361 0.321 0.189 0.144 MBP 0.368 0.332 0.192 0.145 Table 7: Next-token vocabulary entropy on Qwen3-1.7B-Base rollouts under the Base, math-SFT, and MBP checkpoints, measured at step boundaries and across all response tokens. MMLU and GPQA are out of domain; GSM8K and MATH are in domain. 8.2 Non-Myopic Compression An intuitive method to empirically quantify the extent to which H2 holds would be through mutual information (MI) measurements between step-boundary tokens and the final answer. We propose an MI proxy, I(X;Y)=(x,y)∼pθ(⋅,⋅∣q)[logpθ(x,y∣q)pθ(x∣q)pθ(y∣q)] splitI(X;Y)=E_(x,y) p_θ(·,· q) [ p_θ(x,y q)p_θ(x q)\,p_θ(y q) ] split where x is the intermediate reasoning steps, y is the final answer, and q is the input prompt. This measure evaluates the predictability of the final answer from the intermediate step boundary, which reflects the extent to which boundary tokens compress final answer-related information (Appendix E.4). Table 8 compares the MI proxy values for MBP and standard SFT across mathematical reasoning benchmarks. Results show that MBP compresses significantly more final answer related information compared to boundaries from standard SFT, an empirical observation that is consistent with H2. Dataset Base SFT MBP GSM8K 0.00 1.43 6.31 MATH 0.00 2.64 4.84 DeepMath Eval 0.00 4.67 8.29 Table 8: Base-referenced normalized MI proxy between context-conditioned step-boundary tokens and final answers. Results use macro aggregation over the full evaluation sets. 9 Related Work Pause and Dummy Tokens. Prior works have utilized pause tokens to enhance LLM reasoning by delaying generation Goyal et al. (2024); Herel and Mikolov (2024), and London and Kanade (2025) prove that pause tokens strictly increase the expressivity of constant-depth transformers. Building on this, recent approaches have explored dynamic token insertion at strategic points, like low-confidence positions or sentence boundaries, to better guide cognitive processing Kim et al. (2025); Liu et al. (2026). Unlike these prior works that primarily view token insertion as a mechanism for increasing computational bandwidth or providing direct generation cues, our work uniquely isolates the role of pause tokens as training-time context perturbations. Post-Training for Reasoning and Alignment. Contemporary post-training paradigms for LLMs aim to align behavior and enhance complex problem-solving, typically progressing from SFT to reinforcement learning frameworks such as GRPO Wei et al. (2022a); Chung et al. (2024); Shao et al. (2024). A critical challenge across these stages is the trade-off between reasoning adaptation and mode retention, as fine-tuning on a new reasoning task often degrades the model’s previously established language fluency and factual knowledge Luo et al. (2025); Zheng et al. (2025); Lin et al. (2024). Our work is complementary to these post-training paradigms. We show that a lightweight token-insertion strategy can improve both SFT and GRPO when applied at semantic boundaries and masked from direct prediction, while mitigating the adaptation-retention trade-off. 10 Conclusions In this work, we extend the study of pause tokens to their effect on fine-tuning dynamics. Through controlled pilot experiments, we identify two hypotheses, mode retention (H1) and non-myopic compression (H2), which naturally lead to Masked Boundary Pause Training. Through this lens, we show improvements in math and code reasoning while preserving general capability on unrelated benchmarks. We further show that boundary placement reliably outperforms SFT across model scales, as our hypotheses predict, and that retention also helps exploration in RL training. We hope this training-dynamics perspective, a shift from computational expressivity to train-time mode retention and non-myopic compression, supports further research across domains and adaptation regimes. Limitations Our current study focuses on full fine-tuning and evaluates retention primarily through per-example correctness, using BCP and NSR to measure whether base-correct examples remain correct and base-wrong examples become solved. Additionally, initial LoRA experiments show more moderate gains, suggesting that MBP’s interaction with parameter-efficient fine-tuning requires further study. Future work should examine retention at finer token- or distribution-level granularity and systematically analyze MBP under parameter-efficient adaptation. Acknowledgments This work is in part supported by the National Research Foundation of Korea (NRF, RS-2024-00451435(15%), RS-2024-00413957(15%)), Institute of Information & Communications Technology Planning & Evaluation (IITP, RS-2025-02305453(10%), RS-2025-02273157(10%), RS-2025-25442149(20%), RS-2021-I211343(10%)) grant funded by the Ministry of Science and ICT (MSIT), Institute of New Media and Communications (INMAC), the BK21 FOUR program of the Education, Artificial Intelligence Graduate School Program (Seoul National University), Research Program for Future ICT Pioneers, Seoul National University in 2026, and the KT (Korea Telecom)–(Seoul National University/Korea University) AICT R&D Center (20%). References Ahmad et al. (2025) W. U. Ahmad, A. Ficek, M. Samadi, J. Huang, V. Noroozi, S. Majumdar, and B. Ginsburg OpenCodeInstruct: a large-scale instruction tuning dataset for code LLMs. External Links: 2504.04030, Link Cited by: §6.1. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §6.1. Azerbayev et al. (2024) Z. Azerbayev, H. Schoelkopf, K. Paster, M. D. Santos, S. M. McAleer, A. Q. Jiang, J. Deng, S. Biderman, and S. Welleck Llemma: an open language model for mathematics. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1. Bisk et al. (2020) Y. Bisk, R. Zellers, R. Le Bras, J. Gao, and Y. Choi PIQA: reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, Cited by: §6.1. Burtsev et al. (2021) M. S. Burtsev, Y. Kuratov, A. Peganov, and G. V. Sapunov Memory transformer. External Links: 2006.11527, Link Cited by: §1. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §6.1. Chung et al. (2024) H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei Scaling instruction-finetuned language models. Journal of Machine Learning Research 25 (70), p. 1–53. External Links: Link Cited by: §9. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §D.3, §6.1. Goyal et al. (2024) S. Goyal, Z. Ji, A. S. Rawat, A. K. Menon, S. Kumar, and V. Nagarajan Think before you speak: training language models with pause tokens. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1, §2.1, §2.1, 2nd item, §6.1, §9. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §6.1. Gururangan et al. (2020) S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith Don’t stop pretraining: adapt language models to domains and tasks. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 8342–8360. Cited by: §1. He et al. (2025) Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, Z. Zhang, R. Wang, Z. Tu, H. Mi, and D. Yu DeepMath-103K: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. External Links: 2504.11456, Link Cited by: §D.3, §6.1, §6.2. Hendrycks et al. (2021a) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In International Conference on Learning Representations, External Links: Link Cited by: 1st item, §6.1. Hendrycks et al. (2021b) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: §D.3, §6.1. Herel and Mikolov (2024) D. Herel and T. Mikolov Thinking tokens for language modeling. External Links: 2405.08644, Link Cited by: §9. Hu et al. (2022) E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §6.1. Hu et al. (2025) J. Hu, X. Wu, W. Shen, J. K. Liu, W. Wang, S. Jiang, H. Wang, H. Chen, B. Chen, W. Fang, Xianyu, Y. Cao, H. Xu, and Y. Liu OpenRLHF: a ray-based easy-to-use, scalable and high-performance RLHF framework. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Suzhou, China, p. 656–666. External Links: Link, Document Cited by: §D.2, Appendix E. Kim et al. (2025) E. Kim, S. Kim, and J. Thorne Learning to insert [pause] tokens for better reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, p. 23760–23777. External Links: Link, Document Cited by: §1, §2.1, §2.1, 2nd item, §6.1, §9. Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §6.1. Lin et al. (2024) Y. Lin, H. Lin, W. Xiong, S. Diao, J. Liu, J. Zhang, R. Pan, H. Wang, W. Hu, H. Zhang, H. Dong, R. Pi, H. Zhao, N. Jiang, H. Ji, Y. Yao, and T. Zhang Mitigating the alignment tax of RLHF. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 580–606. External Links: Link, Document Cited by: §9. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §6.1. Liu et al. (2026) Z. Liu, Y. Li, and Y. Xu Think in sentences: explicit sentence boundaries enhance language model’s capabilities. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 2275–2288. External Links: Link, Document, ISBN 979-8-89176-390-6 Cited by: §1, §2.1, §8.1, §9. London and Kanade (2025) C. London and V. Kanade Pause tokens strictly increase the expressivity of constant-depth transformers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1, §9. Luo et al. (2025) Y. Luo, Z. Yang, F. Meng, Y. Li, J. Zhou, and Y. Zhang An empirical study of catastrophic forgetting in large language models during continual fine-tuning. External Links: 2308.08747, Link Cited by: §9. Pfau et al. (2024) J. Pfau, W. Merrill, and S. R. Bowman Let’s think dot by dot: hidden computation in transformer language models. In First Conference on Language Modeling, External Links: Link Cited by: §1, §1, §2.1. Rein et al. (2024) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, External Links: Link Cited by: 1st item, §6.1. Roziere et al. (2023) B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, et al. Code llama: open foundation models for code. arXiv preprint arXiv:2308.12950. Cited by: §1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §2.2, §9. Snell et al. (2025) C. V. Snell, J. Lee, K. Xu, and A. Kumar Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1. Suzgun et al. (2023) M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou, and J. Wei Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 13003–13051. External Links: Link, Document Cited by: 1st item, §6.1. Wang et al. (2023) X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, External Links: Link Cited by: §1. Wei et al. (2022a) J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le Finetuned language models are zero-shot learners. In International Conference on Learning Representations, External Links: Link Cited by: §9. Wei et al. (2022b) J. Wei, X. Wang, D. Schuurmans, M. Bosma, brian ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: Link Cited by: §1. Wu et al. (2024) W. Wu, J. X. Morris, and L. Levine Do language models plan ahead for future tokens?. In First Conference on Language Modeling, External Links: Link Cited by: §1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §3.2, §6.1. Ye et al. (2025) T. Ye, Z. Xu, Y. Li, and Z. Allen-Zhu Physics of language models: part 2.1, grade-school math and the hidden reasoning process. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §A.3, §1, §3.2. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §6.1. Zeng et al. (2025) W. Zeng, Y. Huang, Q. Liu, W. Liu, K. He, Z. Ma, and J. He SimpleRL-Zoo: investigating and taming zero reinforcement learning for open base models in the wild. External Links: 2503.18892, Link Cited by: §6.2. Zheng et al. (2025) J. Zheng, X. Cai, S. Qiu, and Q. Ma Spurious forgetting in continual learning of language models. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §9. Appendix A Pilot Experiment Details A.1 MLC continual learning pilot Task and data. Sequences are generated by the Modular Linear Chain (MLC) recurrence xt+1=(atxt+bt)modpx_t+1=(a_t\,x_t+b_t) p with prime p=37p=37 and sequence length L=8L=8. A regime is a fixed length-L coefficient sequence (at,bt)t=1L\(a_t,b_t)\_t=1^L with each at,bta_t,b_t drawn uniformly from 0,…,p−1\0,…,p-1\. We sample two disjoint coefficient sets to define regimes D1D_1 and D2D_2. Vocabulary size is p+2p+2 (digits, BOS, and one pause token). Model and optimizer. A 2-layer, 4-head GPT-style Transformer with hidden size d=128d=128 and learned positional embeddings. Trained with AdamW (learning rate 3×10−43× 10^-4, weight decay 0.010.01), batch size 6464. Two-phase training. Phase 1: train on D1D_1 with standard next-token prediction until D1D_1 accuracy exceeds 0.990.99 (approximately 2,0002,000 optimizer steps). Phase 2: fork from a single Phase-1 checkpoint and continue training on D2D_2 under two matched conditions (No pause and Pause), each run for an additional 2,0002,000 steps. The Pause condition inserts one pause token after every content token, doubling sequence length; pause-target positions are excluded from the cross-entropy loss. Both conditions share architecture, optimizer, batch size, learning rate, and update budget. We report n=8n=8 seeds with shaded bands at ±1σ± 1σ. Evaluation. We track held-out loss on D2D_2 (adaptation) and D1D_1 (retention) every 55 Phase-2 steps. D2D_2 loss is computed on pause-augmented sequences for Pause and bare sequences for No pause; D1D_1 loss is bare for both conditions (matching the no-pause inference distribution). A.2 Multi-stage continual learning The main pilot (§3.1) uses a single D1→D2D_1→ D_2 transition. To test whether the retention advantage is specific to one adaptation or persists as adaptations accumulate, we extend the same setup to a four-regime chain. Setup. We sample four disjoint coefficient sets defining regimes D1,…,D4D_1,…,D_4 (p=37p=37, L=8L=8); all task, model, and optimizer settings follow Appendix A.1. After Phase-1 training on D1D_1, each condition adapts sequentially to D2D_2, D3D_3, and D4D_4 for 2,0002,000 steps per stage under matched budget and optimizer settings: No pause runs standard SFT on each bare DkD_k, and Pause runs masked-pause SFT. We report n=8n=8 seeds and track held-out bare-format D1D_1 loss. Results. Table 9 reports held-out D1D_1 loss at the end of each stage. The retention gap is positive in 8/88/8 seeds at every stage and never inverts across the 6,0006,000 steps, holding at roughly 33–44 nats throughout the chain. The absolute advantage therefore neither accumulates nor decays as adaptations pile up. Both conditions reach the same ∼10−7 \!10^-7 adaptation floor on every DkD_k, so the retained D1D_1 behavior is not purchased with adaptation quality. After No pause Pause Gap D2D_2 5.76±0.565.76± 0.56 2.74±0.812.74± 0.81 +3.01±0.75+3.01± 0.75 D3D_3 8.60±1.448.60± 1.44 4.62±0.464.62± 0.46 +3.98±1.59+3.98± 1.59 D4D_4 8.83±0.168.83± 0.16 6.04±0.846.04± 0.84 +2.80±0.97+2.80± 0.97 Table 9: Held-out D1D_1 loss (nats, mean ±1σ± 1σ, n=8n=8 seeds) after each stage of the D1→D2→D3→D4D_1→ D_2→ D_3→ D_4 chain. Lower is better; Gap is No pause −- Pause, so positive values favor masked pauses. A.3 iGSM probing pilot Base model and data. Qwen3-0.6B-Base, fine-tuned on 3,0003,000 problems sampled from the iGSM generator Ye et al. (2025). We use the max_op=15 max\_op=15, max_edge=20 max\_edge=20, perm_level=5 perm\_level=5 difficulty setting. SFT. Three conditions, trained for 22 epochs with learning rate 1×10−51× 10^-5, batch size 3232, AdamW optimizer, and bf16 precision: No pause (standard next-token prediction on bare iGSM solutions) and Pause (one [PAUSE] token inserted at every reasoning-step boundary, pause-target positions excluded from the loss), and Pause (unmasked), identical but with pause positions included in the loss. The Base condition is the pretrained checkpoint with no fine-tuning. We report n=3n=3 seeds. Probe training. For each held-out problem, we extract the final-layer hidden state at the period token immediately preceding each reasoning step boundary (the same character position across all four protocols). We train two separate linear probes on these representations: one predicting the integer answer of the current step, one predicting the integer answer of the next step. Probes are trained with a by-problem 70/3070/30 split, with each problem’s boundary positions assigned to either train or test based on the problem split. Appendix B Theoretical analysis for masked-pause SFT B.1 A first-order view of masked-pause mode retention Old D1D_1 behaviours survive D2D_2 fine-tuning as long as their logits do not move enough to erase their old margins. We make this precise on a protected set of D1D_1 no-pause prefixes and then show that masked-pause SFT yields a smaller certified protected logit drift than ordinary SFT. The result is a first-order sufficient condition within the local linearized model around W1W_1; it certifies preservation of the D1D_1 top token on selected no-pause prefixes, not preservation of the full D1D_1 predictive distribution. Margin stability. Whether an old top token survives depends only on whether its margin outlasts the logit movement at that prefix; bounding the movement uniformly over 1C_1 converts this into a retention guarantee. Lemma B.1 (Margin stability). Let W1W_1 be the shared parameters after D1D_1 training and 1C_1 a set of D1D_1 no-pause prefixes whose top-token behaviour we protect. For c∈1c _1, let a⁡(c)=arg⁡maxvzW1(c)va(c)= _vz_W_1(c)_v be the D1D_1 top token and γ⁡(c)γ(c) its top-vs-runner-up margin at W1W_1 — the amount a competing logit must close to flip the prediction at c. Define the margin survival function, the retention rate, and the protected logit drift Φ⁡(r) (r) =1|1||c∈1:γ⁡(c)>r|, = 1|C_1| |\c _1:γ(c)>r\ |, Retain⁡(W) (W) =1|1||c∈1: = 1|C_1| |\c _1: argmaxvzW(c)v=a(c)|, _vz_W(c)_v=a(c)\ |, Δ1(W) _1(W) =maxc∈1⁡∥zW(c)−zW1(c)∥∞, = _c _1 z_W(c)-z_W_1(c) _∞, with Φ non-increasing in r. Then Δ1(W)≤δ⟹Retain⁡(W)≥Φ⁡(2δ). _1(W)≤δ\; \;Retain(W)≥ (2δ). The factor of two appears because the old top logit may move down by δ while a competitor may move up by δ. Comparing the certified retention of two adaptation procedures therefore reduces to comparing their certified protected logit drifts on 1C_1. What masked-pause SFT changes. Ordinary SFT predicts the next D2D_2 target yi+1y_i+1 from the no-pause prefix bib_i; masked-pause SFT predicts the same target from the pause-perturbed prefix bi∘pb_i p and masks the loss for the pause token p itself. Masking is essential: were p a training target, bib_i would again receive a direct supervised gradient, restoring the interference channel masked-pause SFT removes. Four scalars control the resulting first-order drifts on 1C_1. The interference exposure IOI_O is large when ordinary prefixes bib_i push the parameters in directions that strongly move protected D1D_1 logits. Decomposing the Jacobian of bi∘pb_i p into a part inherited from the ordinary boundary Jacobian and a pause-local residual yields a worst-case carry coefficient λ and a worst-case pause-local overlap ν with 1C_1. Finally, κ=MP/MOκ=M_P/M_O is the ratio of cumulative normalized shared-update masses the two procedures need to reach the same D2D_2 fitting level (matched D2D_2 validation loss in our experiments). Lemma B.2 (Protected drift bounds). In the local linearized regime around W1W_1 with IO>0I_O>0, the protected logit drifts of the ordinary and masked-pause endpoints satisfy Δ1(W^O) _1( W_O) ≤MOIO, ≤ M_O\,I_O, Δ1(W^P) _1( W_P) ≤MP(λIO+ν). ≤ M_P (λ I_O+ν ). Equivalently, defining βctx:=κ⁡(λ+ν/IO) _ctx:=κ (λ+ν/I_O ), the masked-pause bound can be written as Δ1(W^P)≤βctxMOIO. _1( W_P)\;≤\; _ctx\,M_OI_O. Both bounds share the ordinary scale MOIOM_OI_O, and in this certificate βctx _ctx is the factor multiplying the masked-pause drift bound. Feeding these bounds through Lemma B.1 turns the comparison into a retention guarantee. Theorem B.3 (First-order context-perturbation retention certificate). Assume the local small-step regime around W1W_1, with ordinary and masked-pause SFT using the same normal D2D_2 targets, reaching the same D2D_2 fitting level, and IO>0I_O>0. Combining the drift bounds of Lemma B.2 with the margin-stability bound of Lemma B.1, Retain⁡(W^O) ( W_O) ≥Φ⁡(2MOIO), \;≥\; (2\,M_OI_O), Retain⁡(W^P) ( W_P) ≥Φ⁡(2βctxMOIO). \;≥\; (2 _ctx\,M_OI_O). Masked-pause SFT yields the stronger certificate whenever βctx<1 _ctx<1. The condition βctx<1 _ctx<1 admits a geometric reading: masked-pause SFT fits D2D_2 with less first-order drift on D1D_1 no-pause logits whenever the pause state bi∘pb_i p is sufficiently separated from the ordinary boundary bib_i in the Jacobian sense captured by λ and ν. This separation is structural rather than architectural — a property of the pause insertion policy, which our empirical preference for reasoning-step boundaries effectively encodes. The full setup, lemmas, proofs, and the role of masking and pause-insertion position are in Appendix B. Appendix C Theoretical analysis for masked-pause SFT - Proofs This appendix contains the full setup, lemmas, and proofs supporting Theorem B.3. The analysis is a local first-order sufficient-condition statement on a protected no-pause prefix set 1C_1. It does not claim preservation of the full next-token distribution, of D1D_1 multimodality, or of the realized retained fraction; it compares certified lower bounds within the local linearized model around W1W_1. C.1 Setup and notation Let V be the vocabulary, W the ordinary shared model parameters, and epe_p the pause embedding. We write ∘ for token-sequence concatenation, so bi∘pb_i p denotes the prefix obtained by appending a single pause token p to bib_i. For a no-pause prefix c, let zW(c)∈ℝ||z_W(c) ^|V| be the next-token logit vector; for a prefix containing p, write zW,ep(s)z_W,e_p(s) when the dependence on epe_p should be explicit. Let W1W_1 be the shared parameters after D1D_1 training and let 1C_1 be the protected set of D1D_1 no-pause prefixes. For each c∈1c _1, the D1D_1 top token, its margin, and the margin survival function are a⁡(c) a(c) =arg⁡maxv∈zW1(c)v, = _v z_W_1(c)_v, γ⁡(c) γ(c) =zW1(c)a⁡(c)−maxv≠a⁡(c)⁡zW1(c)v, =z_W_1(c)_a(c)- _v≠ a(c)z_W_1(c)_v, Φ⁡(r) (r) =1|1||c∈1:γ⁡(c)>r|. = 1|C_1| |\c _1:γ(c)>r\ |. Old-mode retention and protected no-pause logit drift are Retain⁡(W) (W) =1|1|∑c∈1[ = 1|C_1|\! _c _11\! [ argmaxvzW(c)v=a(c)], _vz_W(c)_v=a(c) ], Δ1(W) _1(W) =maxc∈1⁡∥zW(c)−zW1(c)∥∞. = _c _1 z_W(c)-z_W_1(c) _∞. Because every prefix in 1C_1 is a no-pause prefix, both quantities depend only on the shared parameters W; in particular, Δ1(W^P) _1( W_P) and Retain⁡(W^P)Retain( W_P) do not directly involve the learned pause embedding e^P e_P. For a matrix B, the operator norm induced by ℓ∞ _∞ is ∥B∥∞→∞ B _∞→∞ =supu≠0∥Bu∥∞∥u∥∞ = _u≠ 0 Bu _∞ u _∞ =max⁡∑kj⁡|Bjk|, = _j\! _k|B_jk|, and satisfies ∥XY∥∞→∞≤∥X∥∞→∞∥Y∥∞→∞ XY _∞→∞≤ X _∞→∞ Y _∞→∞ whenever XYXY is defined. C.2 Direct supervised-prefix relocation Let bi=(x,y1,…,yi)b_i=(x,y_1,…,y_i) be a no-pause D2D_2 reasoning boundary and ℓ⁡(z,y) (z,y) the softmax cross-entropy loss for target y. The two methods give the per-step direct losses ℓiO(W) _i^O(W) =ℓ⁡(zW(bi),yi+1), = \! (z_W(b_i),\,y_i+1 ), ℓiP(W,ep) _i^P(W,e_p) =ℓ⁡(zW,ep(bi∘p),yi+1), = \! (z_W,e_p(b_i p),\,y_i+1 ), with the pause-token prediction loss masked out in the second case. Thus the direct supervised prefix for the normal target yi+1y_i+1 moves from bib_i to bi∘pb_i p. Earlier tokens of bib_i can still receive indirect gradient through the pause position; those contributions are folded into the pause-state Jacobian below. We denote the resulting (multi)sets of supervised prefixes by O _O =bi: =\b_i: loss predicts a normal D2D_2 token from bi, b_i\, P _P =bi∘p: =\b_i p: loss predicts the same normal token from bi∘p, b_i p\, read as multisets if a prefix is supervised more than once. C.3 First-order interference and Jacobians For a prefix s, define the start-of-adaptation shared-parameter logit Jacobian J1(s)=∂zW(s)∂W|W1sno-pause,∂zW,ep(0)(s)∂W|W1scontainsp,J_1(s)= cases . ∂ z_W(s)∂ W |_W_1&s\ no-pause,\\ . ∂ z_W,e_p^(0)(s)∂ W |_W_1&s\ contains\ p, cases where ep(0)e_p^(0) is the pause embedding at the start of D2D_2 adaptation; in the pause case, ep(0)e_p^(0) is held fixed and the derivative is taken with respect to W only. For c∈1c _1 and a supervised prefix s, the first-order interference matrix and exposure are K1(c,s) K_1(c,s) =J1(c)J1(s)⊤, =J_1(c)J_1(s) , I1(1,) I_1(C_1,S) =max⁡∑s∈c∈1⁡∥K1(c,s)∥∞→∞. = _c _1 _s K_1(c,s) _∞→∞. The operator-norm choice matches the ℓ∞ _∞ logit-drift quantity Δ1 _1. We write IO=I1(1,O)I_O=I_1(C_1,S_O). Operationally, IOI_O is large when ordinary D2D_2 supervised prefixes share gradient directions with protected D1D_1 prefixes. C.4 Pause-state Jacobian decomposition For each paired boundary, write JiO=J1(bi)J_i^O=J_1(b_i) and JiP=J1(bi∘p)J_i^P=J_1(b_i p). Let PiOP_i^O be the orthogonal projection in parameter space onto the row space of JiOJ_i^O, acting by right-multiplication, so that JiPPiOJ_i^PP_i^O projects each row of JiPJ_i^P onto row⁡(JiO)row(J_i^O). Define Jicarry J_i^carry =JiPPiO, =J_i^PP_i^O, Jiloc J_i^loc =JiP(I−PiO), =J_i^P(I-P_i^O), so JiP=Jicarry+JilocJ_i^P=J_i^carry+J_i^loc. Because every row of JicarryJ_i^carry lies in row⁡(JiO)row(J_i^O), there exists AiA_i with Jicarry=AiJiOJ_i^carry=A_iJ_i^O, and we take the canonical minimum-norm choice Ai=JiP(JiO)+,A_i=J_i^P(J_i^O)^+, where (⋅)+(·)^+ is the Moore–Penrose pseudoinverse. If JiO=0J_i^O=0, set Ai=0A_i=0. The carry and pause-local interference summaries are λ λ =maxi⁡∥Ai⊤∥∞→∞, = _i A_i _∞→∞, ν ν =max⁡∑ic∈1⁡∥J1(c)(Jiloc)⊤∥∞→∞. = _c _1 _i J_1(c)(J_i^loc) _∞→∞. Operationally, λ is small when the pause state does not reuse the same gradient-sensitive directions as the ordinary boundary, and ν is small when the pause-local residual does not align with the gradient-sensitive directions of protected D1D_1 prefixes. Whether λ and ν are small for a given architecture and insertion policy is a separate structural question; see Section C.9. C.5 Training-regime quantities Remark C.1 (Local small-step regime). The D2D_2 adaptation trajectory is analyzed in a neighbourhood of W1W_1 in which the linearization zW(s)≈zW1(s)+J1(s)(W−W1)z_W(s)≈ z_W_1(s)+J_1(s)(W-W_1) is accurate; for the full nonlinear model, the same statements hold up to a standard local linearization error in Δ1 _1. For A∈O,PA∈\O,P\, let wtA(s)w_t^A(s) be the normalized weight of supervised prefix s in the minibatch loss at step t and ηt _t the learning rate. Define mA(s) m_A(s) =∑tηtwtA(s), = _t _tw_t^A(s), MA M_A =maxs⁡mA(s),κ=MPMO. = _sm_A(s), κ= M_PM_O. MAM_A is the largest cumulative normalized update mass assigned to any single supervised prefix along the trajectory. Throughout this appendix, “the same D2D_2 fitting level” refers to matched D2D_2 validation loss (or task accuracy, when that is the empirical comparison metric used in the main paper); κ>1κ>1 then quantifies the additional shared-parameter update mass masked-pause SFT requires, if any, to reach that level. C.6 Additional lemmas (Lemmas B.1 and B.2 are stated in the main text; their proofs are included below.) Lemma C.2 (Direct-loss relocation). Write JW(s)=∂zW(s)/∂WJ_W(s)=∂ z_W(s)/∂ W and JW|ep(s)=∂zW,ep(s)/∂WJ_W e_p(s)=∂ z_W,e_p(s)/∂ W (with epe_p held fixed). Then ∇WℓiO(W) _W _i^O(W) =JW(bi)⊤riO, =J_W(b_i) r_i^O, ∇WℓiP(W,ep) _W _i^P(W,e_p) =JW|ep(bi∘p)⊤riP, =J_W e_p(b_i p) r_i^P, with riO=∇zℓ(zW(bi),yi+1)r_i^O= _z (z_W(b_i),y_i+1) and riP=∇zℓ(zW,ep(bi∘p),yi+1)r_i^P= _z (z_W,e_p(b_i p),y_i+1). There is no direct loss term of the form ℓ⁡(zW(bi),p) (z_W(b_i),p), because the pause-token prediction loss is masked. Under the local-linearized convention, JWJ_W and JW|epJ_W e_p reduce to J1J_1 at the start of D2D_2 adaptation. Lemma C.3 (Interference separation). Under the canonical decomposition of Section C.4, I1(1,P)≤λIO+ν.I_1(C_1,S_P)\;≤\;λ\,I_O+ν. If IO>0I_O>0, equivalently I1(1,P)≤(λ+ν/IO)IO=αctxIOI_1(C_1,S_P)≤(λ+ν/I_O)\,I_O= _ctx\,I_O. Lemma C.4 (Interference controls drift). In the local linearized regime, any D2D_2 adaptation procedure with supervised multiset S and normalized update masses m⁡(s)m(s) satisfies Δ1(W) _1(W) ≤max⁡∑s∈c∈1⁡m⁡(s)∥K1(c,s)∥∞→∞ ≤ _c _1 _s m(s) K_1(c,s) _∞→∞ ≤MI1(1,), ≤ M\,I_1(C_1,S), where M=maxs⁡m⁡(s)M= _sm(s). C.7 Proofs Proof of Lemma B.1 (margin stability). Fix c∈1c _1 with γ⁡(c)>2δγ(c)>2δ and any v≠a⁡(c)v≠ a(c). Since each logit moves by at most δ, zW(c)a⁡(c)−zW(c)v z_W(c)_a(c)-z_W(c)_v ≥(zW1(c)a⁡(c)−zW1(c)v)−2δ ≥ (z_W_1(c)_a(c)-z_W_1(c)_v )-2δ ≥γ⁡(c)−2δ> 0. ≥γ(c)-2δ\;>\;0. Thus a⁡(c)a(c) remains the top token under W, and the fraction of c with γ⁡(c)>2δγ(c)>2δ lower-bounds Retain⁡(W)Retain(W). □ Proof of Lemma C.2. Ordinary SFT predicts yi+1y_i+1 from bib_i, so its CE gradient is attached to the output state at bib_i; the chain rule gives the stated form. Masked-pause SFT inserts p after bib_i, masks the would-be loss for predicting p from bib_i, and predicts yi+1y_i+1 from the pause-state output position bi∘pb_i p; again the chain rule gives the stated form. Indirect gradient on earlier tokens flows through JW|ep(bi∘p)J_W e_p(b_i p) and is handled by the carry/local decomposition. □ Proof of Lemma C.3. Fix c∈1c _1. Using JiP=AiJiO+JilocJ_i^P=A_iJ_i^O+J_i^loc, K1(c,bi∘p) K_1(c,b_i p) =J1(c)(JiP)⊤ =J_1(c)(J_i^P) =K1(c,bi)Ai⊤+J1(c)(Jiloc)⊤. =K_1(c,b_i)\,A_i +J_1(c)(J_i^loc) . By submultiplicativity of ∥⋅∥∞→∞ · _∞→∞, ∥K1(c,bi∘p)∥∞→∞ K_1(c,b_i p) _∞→∞ ≤λ∥K1(c,bi)∥∞→∞ ≤λ K_1(c,b_i) _∞→∞ +∥J1(c)(Jiloc)⊤∥∞→∞. + J_1(c)(J_i^loc) _∞→∞. Summing over i and taking maxc∈1 _c _1 on both sides, with max⁡(A⁡(c)+B⁡(c))≤max⁡A⁡(c)+max⁡B⁡(c) (A(c)+B(c))≤ A(c)+ B(c), gives I1(1,P)≤λIO+νI_1(C_1,S_P)≤λ I_O+ν. □ Proof of Lemma C.4. In the linearized regime, the step is ΔWt=−ηt∑swt(s)J1(s)⊤rt(s) W_t=- _t _sw_t(s)J_1(s) r_t(s), where the softmax CE residual rt(s)=qt(s)−eysr_t(s)=q_t(s)-e_y_s satisfies ∥rt(s)∥∞≤1 r_t(s) _∞≤ 1 uniformly along the trajectory. For c∈1c _1, summing first-order logit changes, ∥Δz(c)∥∞ z(c) _∞ ≤∑tηt∑swt(s) ≤ _t _t _sw_t(s) ∥K1(c,s)∥∞→∞∥rt(s)∥∞ K_1(c,s) _∞→∞ r_t(s) _∞ ≤∑sm⁡(s)∥K1(c,s)∥∞→∞. ≤ _sm(s) K_1(c,s) _∞→∞. Taking maxc∈1 _c _1 and then M=maxs⁡m⁡(s)M= _sm(s) gives both inequalities. □ Proof of Lemma B.2. Applying Lemma C.4 to the ordinary supervised multiset OS_O gives Δ1(W^O)≤MOI1(1,O)=MOIO _1( W_O)≤ M_O\,I_1(C_1,S_O)=M_OI_O. For masked-pause SFT, Lemma C.4 on PS_P followed by Lemma C.3 gives Δ1(W^P) _1( W_P) ≤MPI1(1,P) ≤ M_P\,I_1(C_1,S_P) ≤MP(λIO+ν). ≤ M_P (λ I_O+ν ). When IO>0I_O>0, factoring MOIOM_OI_O out of the last bound gives MP(λIO+ν) M_P (λ I_O+ν ) =MPMO(λ+νIO)MOIO = M_PM_O\! (λ+ νI_O )M_OI_O =βctxMOIO.□ = _ctx\,M_OI_O. Proof of Theorem B.3. By Lemma B.2, Δ1(W^O)≤MOIO _1( W_O)≤ M_OI_O and Δ1(W^P)≤βctxMOIO _1( W_P)≤ _ctx\,M_OI_O. Applying Lemma B.1 with δ=MOIOδ=M_OI_O and δ=βctxMOIOδ= _ctx\,M_OI_O respectively yields Retain⁡(W^O) ( W_O) ≥Φ⁡(2MOIO), ≥ (2\,M_OI_O), Retain⁡(W^P) ( W_P) ≥Φ⁡(2βctxMOIO). ≥ (2 _ctx\,M_OI_O). Since Φ is non-increasing and IO>0I_O>0, the condition βctx<1 _ctx<1 gives βctxMOIO<MOIO _ctx\,M_OI_O<M_OI_O, so the masked-pause certificate dominates the ordinary-SFT one. The dominance is strict whenever a nonzero fraction of 1C_1 has γ⁡(c)∈(2βctxMOIO, 2MOIO]γ(c)∈(2 _ctx\,M_OI_O,\,2\,M_OI_O]. □ C.8 Why masking the pause-token loss matters If the pause-token prediction loss is not masked, each inserted pause introduces an extra direct loss ℓipause(W)=ℓ⁡(zW(bi),p) _i^pause(W)= (z_W(b_i),p) whose gradient is attached to the original no-pause boundary bib_i. This re-introduces a supervised multiset pause=bi:loss predicts p from biS_pause=\b_i:loss predicts $p$ from b_i\ and changes the drift certificate to Δ1(W^Punmask) _1( W_P^unmask) ≤Mnorm(λIO+ν) ≤ M_norm(λ I_O+ν) +MpauseIpause, +M_pause\,I_pause, where Ipause=I1(1,pause)I_pause=I_1(C_1,S_pause). Since pauseS_pause consists of the original no-pause boundaries, this added term can erase the benefit of context perturbation. Masking removes this loss and leaves Theorem B.3 clean. C.9 Pause insertion position An insertion policy A determines PAS_P^A and therefore changes λ⁡(A)λ(A), ν⁡(A)ν(A), and κ⁡(A)=MPA/MOκ(A)=M_P^A/M_O, giving βctx(A)=κ⁡(A)(λ⁡(A)+ν⁡(A)/IO) _ctx(A)=κ(A)(λ(A)+ν(A)/I_O). The theorem applies as soon as βctx(A)<1 _ctx(A)<1. The remark below collects the qualitative considerations; it is not part of the formal theorem. Remark C.5 (Why reasoning-step boundaries are natural insertion points). (i) At a reasoning-step boundary the previous step has ended, so the pause state bi∘pb_i p is less pressured to copy a mid-phrase token representation, which tends to reduce the carry coefficient λ⁡(A)λ(A). (i) The pause-local residual at a complete boundary need not preserve ordinary syntactic-continuation features tied to D1D_1, which tends to reduce ν⁡(A)ν(A). (i) Predicting the next reasoning step from a boundary-plus-pause prefix is still a natural next-token task, so the fitting-budget ratio κ⁡(A)κ(A) stays moderate. Insertion that is too sparse leaves most supervised prefixes close to OS_O; insertion at unnatural dense positions can force the pause state to copy the immediately preceding prefix, increasing λ⁡(A)λ(A), ν⁡(A)ν(A), or κ⁡(A)κ(A). Appendix D Experimental Settings D.1 Hyperparameter settings We report the hyperparameter settings used for each training stage. Across experiments, we use FusedAdam as the optimizer, CosineAnnealingLR as the learning-rate scheduler, a warmup ratio of 0.03, and bf16 precision for training, while adapting the learning rate and batch configuration to each training objective and model family. For SFT on mathematical reasoning data, we train each model for 3 epochs with a batch size of 32. The learning rate is selected depending on the model family: 5×10−65× 10^-6 for Qwen models, 1×10−51× 10^-5 for Llama3.2, and 5×10−55× 10^-5 for Llama3.1. The full set of hyperparameters for math SFT is provided in Table 10. For SFT on code generation data, we also train for 3 epochs with a batch size of 32, using the same optimizer, scheduler, warmup ratio, and training precision as in math SFT. The learning rate is model-specific: 5×10−75× 10^-7 for Qwen3-1.7B-Base, 5×10−65× 10^-6 for Qwen3-4B-Base, 5×10−55× 10^-5 for Qwen3-8B-Base, and 1×10−81× 10^-8 for Llama3.1-8B-Instruct. The complete configuration for code SFT is summarized in Table 11. For GRPO training, we use a learning rate of 5×10−75× 10^-7, train for 1 episode, and use a batch size of 128. For each prompt, we sample 8 rollouts and apply a KL coefficient of 0.001. The optimizer, scheduler, warmup ratio, and training dtype are kept consistent with the SFT settings. The GRPO hyperparameters are listed in Table 12. Hyperparameter SFT on math Learning rate Qwen: 5×10−65× 10^-6; Llama3.2: 1×10−51× 10^-5; Llama3.1: 5×10−55× 10^-5 Epochs 3 Batch size 32 Optimizer FusedAdam Scheduler CosineAnnealingLR Warmup ratio 0.03 Training dtype bf16 Table 10: SFT hyperparameters for mathematical reasoning training. Hyperparameter SFT on code Learning rate Qwen3-1.7B-Base: 5×10−75× 10^-7; Qwen3-4B-Base: 5×10−65× 10^-6; Qwen3-8B-Base: 5×10−55× 10^-5; Llama3.1-8B-Instruct: 1×10−81× 10^-8 Epochs 3 Batch size 32 Optimizer FusedAdam Scheduler CosineAnnealingLR Warmup ratio 0.03 Training dtype bf16 Table 11: SFT hyperparameters for code generation training. Hyperparameter GRPO Learning rate 5×10−75× 10^-7 Episodes 1 Batch size 128 Rollouts per prompt 8 Optimizer FusedAdam Scheduler CosineAnnealingLR Warmup ratio 0.03 Training dtype bf16 KL coefficient 0.001 Table 12: GRPO hyperparameters. D.2 Computation resources All training experiments in this paper were conducted with OpenRLHFHu et al. (2025) on 2× NVIDIA RTX PRO 6000 and 2× NVIDIA RTX A100 GPUs. D.3 Dataset Statistics We used a mixture of data from GSM8K, MATH, and DeepMATH Cobbe et al. (2021); Hendrycks et al. (2021b); He et al. (2025) for math domain. Table 13 summarizes the statistics of the mathematical datasets used for training and evaluation. Statistic Train Eval Total Examples 83,438 4,391 87,829 Full SFT tokens 44.14M 2.36M 46.51M Pause tokens 1.34M 71.72K 1.41M Pause-token ratio 3.02% 3.03% 3.03% Pauses/sample 16.003 16.332 16.020 Table 13: Statistics of the step-aligned pause dataset. D.4 Evaluation setup For mathematical reasoning benchmarks, we report pass@1 accuracy under a deterministic decoding setting with temperature 0.0. For code generation benchmarks, we report pass@1 using the EvalPlus package, and a prediction is counted as correct only when it passes all test cases. For general benchmarks, we report pass@4 accuracy with temperature 0.7 using the LightEval package. Appendix E Implementation Details We implemented most components using OpenRLHFHu et al. (2025), and used the default OpenRLHF settings for hyperparameters not explicitly specified. E.1 Objective function Let x denote the prompt and y=(y1,…,yT)y=(y_1,…,y_T) denote the target response tokens. For a prefix s, write pθ(⋅∣s)p_θ(· s) for the model distribution and ℓθ(s,a)=−log⁡pθ(a∣s) _θ(s,a)=- p_θ(a s) for the next-token cross-entropy loss. Standard SFT optimizes ℒSFT=1T∑t=1Tℓθ(x∘y<t,yt).L_SFT= 1T _t=1^T _θ(x y_<t,y_t). For pause-based training, let u=(u1,…,uL)u=(u_1,…,u_L) be the sequence obtained after inserting pause tokens into y, where each uju_j is either a normal target token or the pause token p. Masked-pause SFT uses the same next-token CE loss, but removes the loss terms whose target is p: mj m_j =[uj≠p], =1[u_j≠ p], ℒmask _mask =1∑jmj∑j=1Lmjℓθ(x∘u<j,uj). = 1 _jm_j _j=1^Lm_j\, _θ(x u_<j,u_j). Thus the model is trained to predict normal answer tokens from prefixes that may contain pauses, while it is not directly trained to emit the pause token itself. The pause-predicted variant keeps the same inserted sequence u, but does not mask pause targets: ℒpred=1L∑j=1Lℓθ(x∘u<j,uj).L_pred= 1L _j=1^L _θ(x u_<j,u_j). The distinction is important: masked-pause SFT changes the context used for predicting ordinary tokens, whereas pause-predicted SFT additionally adds direct supervision for predicting p. E.2 Step-Boundary Extraction We identify reasoning-step boundaries with deterministic preprocessing. For math, line breaks are the primary boundary candidates, while wrapped continuations and line breaks inside inline or display-math expressions are preserved. Displayed equations written with \[...\], $$...$$, or standard equation and alignment environments remain intact and are attached to immediately preceding explanatory text when present. Outside math environments, we split at transitions that form an independent reasoning unit, including bullets, final-answer headings, discourse cues, and completed calculations, and merge short non-semantic fragments with the preceding step. For code, we retain the extracted code region, discard surrounding natural-language text and blank lines, and treat each remaining source line as one step. One pause token is inserted at each resulting boundary, and its target position is masked from the loss. E.3 Training Overhead MBP inserts approximately 16 pause tokens per math example and 27 per code example on average. Pause tokens account for approximately 2.2%2.2\% and 9.8%9.8\% of tokens in the corresponding augmented math and code corpora, respectively. Training is otherwise identical to standard SFT: pause positions are simply excluded from the loss, and insertion is a deterministic rule-based preprocessing step. E.4 MI-Proxy Computation For each prompt qiq_i, we take yiy_i to be the final balanced expression in the generated solution and let xi,kx_i,k denote the cumulative solution prefix ending at the k-th step boundary before the answer. For model m, we compute ri,k(m)=log⁡pm(xi,k,yi∣qi)pm(xi,k∣qi)pm(yi∣qi).r_i,k^(m)= p_m(x_i,k,y_i q_i)p_m(x_i,k q_i)\,p_m(y_i q_i). We weight each observed pair by its joint sequence probability, wi,k(m)=pm(xi,k,yi∣qi)ri,k(m)w_i,k^(m)=p_m(x_i,k,y_i q_i)r_i,k^(m), and macro-average first over the KiK_i boundaries of each example and then over the N examples: I^w(m)=1N∑i=1N1Ki∑k=1Kiwi,k(m). I_w(m)= 1N _i=1^N 1K_i _k=1^K_iw_i,k^(m). We report the signed, Base-referenced normalized score ΔB(m) _B(m) =I^w(m)−I^w(B), = I_w(m)- I_w(B), GB(m) G_B(m) =sign⁡(ΔB(m)) =sign\! ( _B(m) ) ⋅log10⁡(1+|ΔB(m)||I^w(B)|), · _10\! (1+ | _B(m)|| I_w(B)| ), with GB(B)=0G_B(B)=0, where B denotes the Base model. All results use macro aggregation over the full evaluation set (filter = all). E.5 Examples of Methods The following examples illustrate the actual data formats used for different pause-token placement strategies, including MBP (Masked Boundary Pause), Random, Append, and DIT. Examples for math data are shown in Table 14, while examples for code data are provided in Table 15 and Table 16. Method Math example MBP Question. Round each number to the nearest tenth and add the results: 543.55, 25.23, and 299.98. Answer. <|PAUSE|>First, let’s round each number to the nearest tenth: <|PAUSE|>1. 543.55 rounds to 543.6 (since the hundredths place is 5). <|PAUSE|>2. 25.23 rounds to 25.2 (since the hundredths place is less than 5). <|PAUSE|>3. 299.98 rounds to 300.0 (since the hundredths place is 8). <|PAUSE|>Now, let’s add the rounded numbers: <|PAUSE|>543.6 + 25.2 + 300.0 = 868.8. <|PAUSE|>So, the final answer is 868.8 868.8. append Question. Round each number to the nearest tenth and add the results: 543.55, 25.23, and 299.98.<|PAUSE|><|PAUSE|><|PAUSE|> <|PAUSE|><|PAUSE|><|PAUSE|><|PAUSE|> Answer. First, let’s round each number to the nearest tenth: 1. 543.55 rounds to 543.6 (since the hundredths place is 5). 2. 25.23 rounds to 25.2 (since the hundredths place is less than 5). 3. 299.98 rounds to 300.0 (since the hundredths place is 8). Now, let’s add the rounded numbers: 543.6 + 25.2 + 300.0 = 868.8. So, the final answer is 868.8 868.8. random Question. Round each number to the nearest tenth and add the results: 543.55, 25.23, and 299.98. Answer. First, let’s round each number to the nearest tenth: 1. 543.55 rounds to 5<|PAUSE|>4<|PAUSE|>3.6 (since the hundredths place is 5). 2. 25.23 rounds to 25.<|PAUSE|>2 (since the hundred<|PAUSE|>ths place is less than 5). 3. 299.98 rounds to 3<|PAUSE|>00.0 (since the hundredths place is 8). Now, let’s add the rounded numbers: 543.6 + 25.2 + 300.0 = 868.8. So, the final<|PAUSE|> answer is \ <|PAUSE|>868.8. dit Question. Round each number to the nearest tenth and add the results: 543.55, 25.23, and 299.98. Answer. <|PAUSE|>First,<|PAUSE|> let’s round each number to the nearest tenth: <|PAUSE|>1. 543.55 rounds to 543.6 (since the hundredths place is 5<|PAUSE|>). 2. 25.23 rounds to 25.2 (since the hundredths place is<|PAUSE|> less than 5). 3. 299.98 rounds to 300.0 (since the hundredths place is 8). Now, let’s add the rounded numbers: 543.6 + 25.2 + 300.0 = 868.8. So, the<|PAUSE|> final answer is<|PAUSE|> 868.8 868.8. Table 14: Examples of pause-token placement for the math task. Method Code example MBP Question. Ninja Atsushi stands guard atop the Ninja Building, which consists of two adjacent buildings of equal height. Each building has a series of walls of varying heights. Atsushi must jump between buildings to patrol the rooftops. Given the heights of the walls for both buildings, determine the minimum height Atsushi must jump for each wall position. Input: two lists of integers, heights_building_1 and heights_building_2, with the same length. Output: a list of integers where each element is the minimum height Atsushi must jump for each wall position. Constraints: the common list length is between 1 and 10510^5, and each wall height is between 1 and 10910^9. Sample input: heights_building_1 = [3, 6, 2], heights_building_2 = [5, 4, 8]. Sample output: [5, 6, 8]. Answer. <|PAUSE|>def min_jumps(heights_building_1, heights_building_2): <|PAUSE|>Calculate the minimum height Atsushi must jump for each wall position between two buildings. <|PAUSE|>:param heights_building_1: list of integers representing the heights of walls in the first building. <|PAUSE|>:param heights_building_2: list of integers representing the heights of walls in the second building. <|PAUSE|>:return: list of integers representing the minimum jump heights for each wall position. <|PAUSE|>if len(heights_building_1) != len(heights_building_2): <|PAUSE|>raise ValueError("Both buildings must have the same number of walls."). <|PAUSE|>return [ max(heights_building_1[i], heights_building_2[i]) for i in range(len(heights_building_1)) ]. append Question. Ninja Atsushi stands guard atop the Ninja Building, which consists of two adjacent buildings of equal height. Each building has a series of walls of varying heights. Atsushi must jump between buildings to patrol the rooftops. Given the heights of the walls for both buildings, determine the minimum height Atsushi must jump for each wall position. Input: two lists of integers, heights_building_1 and heights_building_2, with the same length. Output: a list of integers where each element is the minimum height Atsushi must jump for each wall position. Constraints: the common list length is between 1 and 10510^5, and each wall height is between 1 and 10910^9. Sample input: heights_building_1 = [3, 6, 2], heights_building_2 = [5, 4, 8]. Sample output: [5, 6, 8].<|PAUSE|><|PAUSE|> <|PAUSE|><|PAUSE|><|PAUSE|><|PAUSE|><|PAUSE|><|PAUSE|> <|PAUSE|><|PAUSE|> Answer. def min_jumps(heights_building_1, heights_building_2): Calculate the minimum height Atsushi must jump for each wall position between two buildings. :param heights_building_1: list of integers representing the heights of walls in the first building. :param heights_building_2: list of integers representing the heights of walls in the second building. :return: list of integers representing the minimum jump heights for each wall position. if len(heights_building_1) != len(heights_building_2): raise ValueError("Both buildings must have the same number of walls."). return [ max(heights_building_1[i], heights_building_2[i]) for i in range(len(heights_building_1)) ]. Table 15: Examples of pause-token placement for the code task. Method Code example random Question. Ninja Atsushi stands guard atop the Ninja Building, which consists of two adjacent buildings of equal height. Each building has a series of walls of varying heights. Atsushi must jump between buildings to patrol the rooftops. Given the heights of the walls for both buildings, determine the minimum height Atsushi must jump for each wall position. Input: two lists of integers, heights_building_1 and heights_building_2, with the same length. Output: a list of integers where each element is the minimum height Atsushi must jump for each wall position. Constraints: the common list length is between 1 and 10510^5, and each wall height is between 1 and 10910^9. Sample input: heights_building_1 = [3, 6, 2], heights_building_2 = [5, 4, 8]. Sample output: [5, 6, 8]. Answer. def min_jumps(heights_building<|PAUSE|>_1, <|PAUSE|> heights_building_2): Calculate the minimum height Atsushi must jump for each wall position between two buildings. :param heights_building_1: list of integers representing the heights of walls in the first building. <|PAUSE|>:param heights_building_2: list of integers representing the heights of walls in the second building. :return: list of integers representing the minimum jump heights for each wall position<|PAUSE|>. if len<|PAUSE|>(height<|PAUSE|>s_building_1) != len(heights_building_2): raise ValueError("Both buildings must have the same number of walls."). return [ max(heights_building_1[i], heights_building_2<|PAUSE|>[i<|PAUSE|>]) for i in range<|PAUSE|>(len(heights_building_1<|PAUSE|>)) ]. dit Question. Ninja Atsushi stands guard atop the Ninja Building, which consists of two adjacent buildings of equal height. Each building has a series of walls of varying heights. Atsushi must jump between buildings to patrol the rooftops. Given the heights of the walls for both buildings, determine the minimum height Atsushi must jump for each wall position. Input: two lists of integers, heights_building_1 and heights_building_2, with the same length. Output: a list of integers where each element is the minimum height Atsushi must jump for each wall position. Constraints: the common list length is between 1 and 10510^5, and each wall height is between 1 and 10910^9. Sample input: heights_building_1 = [3, 6, 2], heights_building_2 = [5, 4, 8]. Sample output: [5, 6, 8]. Answer. <|PAUSE|>def min<|PAUSE|>_jumps( heights_building_1, heights_building_2): Calculate the minimum height Atsushi must jump for each wall position<|PAUSE|> between two buildings. <|PAUSE|>:param heights_building_1: list of integers representing the heights of<|PAUSE|> walls in the first building. :param heights_building_2: list of integers representing the heights of walls in the second building. :return: list of integers representing the minimum<|PAUSE|> jump heights for each wall position. <|PAUSE|>if len(heights_building_1) != len(heights_building_2): raise ValueError("Both buildings must have the same number of walls<|PAUSE|>."). <|PAUSE|>return [ max(heights_building_1[i], heights_building_2[i]) for i in range(len(heights_building_1)) ].<|PAUSE|> Table 16: Examples of pause-token placement for the code task, continued. Appendix F Additional Results This section reports the full evaluation results that are not shown in the main text. We include mathematical reasoning, code generation, and general benchmark results for Qwen3-1.7B-Base, Qwen3-4B-Base, Qwen3-8B-Base, Llama3.2-1B-Instruct, Llama3.2-3B-Instruct, and Llama3.1-8B-Instruct. The tables compare the base models, standard SFT, and the pause-insertion variants Append, Random, DIT, MBP, and MBP w/o masking where available. The full results for math and general language understanding tasks are shown in Table 17, while the full results for code and general language understanding tasks are presented in Table 18. Table 19 reports the full six-benchmark GRPO results. Method GSM8K MATH500 MATH AIME24 AMC23 MINERVA MMLU GPQA HellaSwag BBH PIQA Average Qwen3-1.7B-Base Base 65.13 54.20 54.82 0.00 35.00 17.65 62.95 27.46 38.69 38.84 62.13 41.53 SFT 82.41 63.40 62.98 0.00 27.50 20.59 62.32 24.78 40.39 42.06 66.10 44.78 Append 79.68 59.20 60.96 0.00 30.00 22.79 62.93 25.67 41.53 42.62 66.59 44.72 Random 81.35 60.80 63.26 0.00 20.00 23.16 63.64 23.66 48.73 43.96 69.64 45.29 DIT 68.01 57.20 56.38 6.67 30.00 20.22 62.90 25.67 44.75 42.66 69.31 43.98 MBP 83.02 61.00 64.56 3.33 37.50 23.16 63.93 27.23 48.32 44.13 67.63 47.62 MBP w/o masking 80.97 62.20 62.64 6.67 27.50 23.16 63.53 24.55 45.86 44.17 66.70 46.18 Qwen3-4B-Base Base 82.56 66.00 67.64 6.67 37.50 28.68 73.05 30.80 63.69 56.64 73.29 53.32 SFT 91.36 69.20 71.24 6.67 45.00 29.78 73.33 29.91 75.53 54.64 77.91 56.78 Append 89.92 67.00 68.62 10.00 42.50 28.68 73.35 29.02 76.47 54.94 78.45 56.27 Random 91.43 70.40 71.06 10.00 42.50 29.04 73.57 28.79 76.14 54.89 77.64 56.86 DIT 87.19 63.00 64.00 6.67 40.00 29.04 73.60 30.36 70.35 54.49 79.43 54.38 MBP 90.07 73.40 72.32 10.00 50.00 30.51 73.54 26.79 76.04 54.55 79.65 57.90 MBP w/o masking 90.98 69.20 70.14 6.67 40.00 26.84 73.03 26.12 76.07 54.44 75.08 55.32 Qwen3-8B-Base Base 88.40 65.40 65.86 13.33 42.50 26.84 77.79 29.46 70.82 58.48 78.24 56.10 SFT 92.95 71.60 72.54 6.67 52.50 32.72 78.21 27.90 74.80 58.56 85.09 59.41 Append 93.25 75.40 73.00 10.00 60.00 33.82 78.20 35.04 76.45 58.49 84.17 61.62 Random 93.03 73.80 72.26 6.67 47.50 34.56 78.53 22.32 74.40 58.82 84.98 58.81 DIT 93.93 74.40 73.54 6.67 52.50 35.66 78.00 26.12 73.64 58.14 85.53 59.83 MBP 92.57 75.00 72.84 3.33 55.00 35.29 78.25 23.66 75.52 57.28 85.36 59.46 Llama3.2-1B-Instruct Base 38.97 23.80 25.70 0.00 15.00 7.72 42.34 27.90 30.63 32.43 61.26 27.80 SFT 50.72 28.00 38.48 0.00 10.00 8.09 42.09 28.79 28.54 31.77 58.92 29.58 Append 45.87 28.20 33.62 0.00 15.00 6.62 41.34 25.00 29.03 32.33 58.11 28.65 Random 51.93 28.60 38.94 3.33 10.00 7.72 39.70 25.45 26.91 31.49 58.65 29.34 DIT 43.75 21.80 29.12 0.00 17.50 6.25 39.74 24.33 27.62 33.07 56.86 27.28 MBP 51.63 31.80 41.66 0.00 27.50 14.34 40.81 27.68 28.43 32.16 55.82 31.98 MBP w/o masking 52.16 30.60 39.64 0.00 2.50 6.99 40.90 26.12 27.35 32.38 55.11 28.52 Llama3.2-3B-Instruct Base 70.43 40.00 43.22 0.00 10.00 17.65 59.33 27.90 56.28 43.06 76.77 40.42 SFT 76.80 46.40 55.02 3.33 27.50 13.97 55.99 28.57 49.78 41.09 73.50 42.90 Append 73.77 44.60 52.26 6.67 25.00 15.44 56.13 29.24 50.76 40.78 74.05 42.61 Random 75.97 47.80 54.16 3.33 27.50 13.24 56.62 29.69 51.42 41.15 74.10 43.18 DIT 70.36 38.00 43.00 3.33 17.50 12.13 57.60 28.12 52.39 42.41 74.05 39.90 MBP 77.03 48.20 55.92 3.33 30.00 19.85 56.46 25.89 50.04 41.56 73.72 43.82 MBP w/o masking 76.19 47.40 54.58 3.33 20.00 15.81 55.64 25.22 49.93 40.73 73.78 42.06 Llama3.1-8B-Instruct Base 77.48 46.40 48.62 6.67 26.50 22.43 67.56 28.79 73.30 49.42 82.05 48.11 SFT 84.84 53.20 56.28 3.33 17.50 21.69 59.49 24.33 49.52 46.72 74.65 44.69 Append 84.46 49.80 53.08 3.33 20.00 20.59 60.84 26.56 50.24 48.26 73.67 44.62 Random 85.90 52.80 56.44 3.33 20.00 20.59 59.79 29.91 49.93 46.44 73.72 45.35 DIT 77.41 44.00 47.14 3.33 20.00 22.79 58.51 31.92 52.53 45.89 75.46 43.54 MBP 86.28 58.00 61.02 6.67 27.50 22.43 61.30 29.91 50.78 48.13 72.25 47.66 Table 17: Additional SFT results on mathematical reasoning and general benchmarks. All entries report accuracy. Average denotes the macro-average over all reported benchmark columns. Method MBPP MBPP+ HumanEval HumanEval+ MMLU GPQA HellaSwag BBH PIQA Average Qwen3-1.7B-Base Base 69.05 59.26 58.54 53.05 62.95 27.46 38.69 38.84 62.13 52.22 SFT 69.58 62.43 67.07 61.59 60.19 23.88 36.44 39.15 60.66 53.44 Append 69.84 60.58 70.73 65.24 61.53 27.90 36.58 40.65 64.09 55.24 Random 70.37 61.11 67.07 62.80 64.03 24.78 48.36 42.50 68.77 56.64 DIT 69.84 60.58 71.95 65.24 63.71 25.45 50.40 43.09 67.41 57.52 MBP 73.02 62.43 70.12 65.24 63.78 25.67 44.01 42.88 68.01 57.24 Qwen3-4B-Base Base 74.34 64.81 76.83 72.56 73.05 30.80 63.69 56.64 73.29 65.11 SFT 78.04 67.99 79.27 73.78 74.19 28.12 75.35 56.85 78.45 68.00 Append 80.42 68.25 80.49 74.39 74.43 34.15 77.15 53.38 75.63 68.70 Random 78.84 67.20 82.32 78.66 74.37 28.12 76.08 53.22 78.29 68.57 DIT 75.93 66.93 81.10 76.22 74.36 28.12 76.38 53.43 76.77 67.69 MBP 79.10 65.08 82.93 78.05 74.56 29.24 74.72 54.94 78.89 68.61 Qwen3-8B-Base Base 82.80 72.49 88.41 84.15 77.79 29.46 70.82 58.48 78.24 71.40 SFT 86.51 75.13 85.98 81.71 77.96 29.24 75.33 61.25 83.30 72.93 Append 85.19 72.75 83.54 78.05 77.89 23.88 79.01 61.39 82.97 71.63 Random 86.24 75.13 84.76 81.10 78.09 28.12 79.63 59.85 85.75 73.19 DIT 84.92 74.07 87.20 81.10 77.75 32.14 76.98 60.58 83.62 73.15 MBP 87.30 75.66 86.59 82.32 78.02 29.24 78.16 61.31 83.03 73.51 Llama3.1-8B-Instruct Base 67.46 59.52 67.68 62.20 67.56 28.79 73.30 49.42 82.05 62.00 SFT 68.25 60.05 68.29 64.02 67.47 26.56 73.63 49.55 81.88 62.19 Append 69.58 60.85 67.68 62.20 67.44 25.45 73.87 49.47 82.15 62.08 Random 69.31 60.05 68.90 63.41 67.37 25.45 73.60 49.76 82.10 62.22 DIT 70.11 60.58 68.29 64.02 67.42 29.24 73.75 49.58 82.10 62.79 MBP 68.52 60.05 69.51 64.02 67.41 28.35 73.93 49.36 82.10 62.58 Table 18: Additional SFT results on code generation and general benchmarks. Code benchmark entries report pass@1, while general benchmark entries report accuracy. Average denotes the macro-average over all reported benchmark columns. Method GSM8K M500 MATH AIME24 AMC23 Minerva Avg. AUC Qwen3-1.7B-Base Standard 80.52 61.20 59.68 3.33 37.50 20.96 43.87 0.366 MBP 81.35 61.80 61.24 3.33 40.00 21.32 44.84 0.414 Qwen3-4B-Base Standard 92.19 74.40 74.30 13.33 50.00 30.15 55.73 0.505 MBP 91.58 76.00 75.26 16.67 57.50 23.90 56.82 0.550 Table 19: Full GRPO pass@1 accuracy (%) on six mathematical reasoning benchmarks and normalized training-reward AUC for Qwen3-1.7B-Base and Qwen3-4B-Base. Average denotes the macro-average over the six benchmarks.

Instant research discovery