Paper deep dive
SLPO: Scaling Latent Reasoning via a Surrogate Policy
Runyang You, Zhiyuan Liu, Yongqi Li, Wenjie Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/28/2026, 3:31:06 AM
Summary
The paper introduces Surrogate Latent Policy Optimization (SLPO), a method to apply outcome-reward reinforcement learning to autoregressive latent reasoners. SLPO addresses the lack of tractable per-step likelihoods and adaptive stopping in latent reasoning by using a Gaussian surrogate policy for credit assignment and a correctness-supervised stopping head for variable-horizon computation. Empirical results show SLPO improves Pass@k accuracy across multiple backbones and benchmarks, enabling difficulty-adaptive latent computation.
Entities (11)
Relation Signals (10)
SLPO โ appliesto โ Latent Reasoning
confidence 95% ยท We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL to autoregressive latent reasoners
SLPO โ evaluatedon โ GSM8K
confidence 95% ยท We evaluate SLPO on grade-school mathematical reasoning. We train on GSM8K-Aug
SLPO โ uses โ Gaussian Surrogate
confidence 92% ยท SLPO constructs a tractable Gaussian surrogate for stochastic latent transitions
SLPO โ uses โ Stopping Head
confidence 92% ยท We equip the reasoner with a stopping head and initialize it through a correctness-supervised cold start
SLPO โ enables โ Test-Time Scaling
confidence 90% ยท SLPO produces a consistent latent test-time scaling effect
SLPO โ improves โ pass@k
confidence 90% ยท SLPO improves Pass@8 and Pass@16 in all 12 backboneโdataset settings
SLPO โ compatiblewith โ GRPO
confidence 85% ยท The gains persist across RLOO and GRPO
SLPO โ compatiblewith โ RLOO
confidence 85% ยท The gains persist across RLOO and GRPO
Coconut โ โ
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning with verifiable rewards has become the predominant recipe for eliciting test-time scaling in explicit Chain-of-Thought reasoners. Yet this scaling path remains computationally costly, since every intermediate step must be decoded as a language token. Latent reasoning instead carries intermediate computation as continuous vectors and already matches or surpasses explicit CoT at far shorter horizons. Despite this promise, latent reasoners remain largely imitation-bound, while explicit CoT has already moved past imitation via outcome-reward RL. Latent trajectories lack a tractable per-step likelihood and an adaptive stopping interface under fixed thinking budgets, so outcome rewards cannot elicit latent test-time scaling. We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL to autoregressive latent reasoners: an empirical surrogate policy density over latent transitions for trajectory-level credit assignment, and a correctness-supervised stopping head that outcome-reward optimization refines into a variable-horizon policy. Across continuous and soft thinking settings, SLPO improves Pass@$k$ under parallel sampling and allocates longer latent computation to harder instances with higher deterministic accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2607.19691v2
- Canonical: https://arxiv.org/abs/2607.19691v2
Trouble viewing inline? Open PDF directly โ
Full Text
62,558 characters extracted from source content.
Expand or collapse full text
2026-07-20 SLPO Scaling Latent Reasoning via a Surrogate Policy Runyang You 1* Zhiyuan Liu 2*โก Yongqi Li 1โ Wenjie Li 1 1 The Hong Kong Polytechnic University 2 Sichuan University runyang.y@outlook.com, liyongqi0@gmail.com Abstract Reinforcement learning with verifiable rewards has become the predominant recipe for eliciting test-time scaling in explicit Chain-of-Thought reasoners. Yet this scaling path remains computationally costly, since every intermediate step must be decoded as a language token. Latent reasoning instead carries intermediate computation as continuous vectors and already matches or surpasses explicit CoT at far shorter horizons. Despite this promise, latent reasoners remain largely imitation-bound, while explicit CoT has already moved past imitation via outcome-reward RL. Latent trajectories lack a tractable per-step likelihood and an adaptive stopping interface under fixed thinking budgets, so outcome rewards cannot elicit latent test-time scaling. We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL to autoregressive latent reasoners: a differentiable surrogate policy interface over latent transitions for trajectory-level credit assignment, and a correctness-supervised stopping head that outcome-reward optimization refines into a variable-horizon policy. Across two continuous latent reasoners, two backbones, and three held-out benchmarks, SLPO improves Pass@8 and Pass@16 in all 12 backboneโdataset settings, with gains of up to 12.07 percentage points. SLPO further transfers to soft-token inference and learns difficulty-adaptive computation, allocating longer latent trajectories to harder instances. Keywords: latent reasoning, reinforcement learning, test-time scaling ยง Project Page: https://github.com/ModalityDance/SLPO a Latent Reasoningb SLPOc Latent Test-Time Scaling Figure 1: SLPO brings outcome-reward RL to latent reasoning and enables latent test-time scaling. โ Equal contribution. โ Corresponding author. โก Work done as an intern at PolyU. 1 arXiv:2607.19691v2 [cs.CL] 27 Jul 2026 SLPO: Scaling Latent Reasoning via a Surrogate Policy 1 Introduction Large language models have demonstrated strong performance on complex reasoning tasks with chain-of- thought (CoT) reasoning [1,2]. By eliciting intermediate steps in natural language, CoT decomposes difficult problems and allocates additional test-time computation before generating the final answer [3,4]. Recent reasoning models push this additional test-time computation considerably further, either sequentially, by extending and revising ever longer chains [5]; or in parallel, by searching and exploring among ever more trials at inference [2,4]. These inference-time strategies have become known as test-time scaling (TTS), while reinforcement learning with verifiable rewards (RLVR) has emerged as the predominant training-time paradigm for eliciting and optimizing such capabilities [6, 7]. However, this test-time scaling path is computationally costly. Explicit CoT decodes every intermediate step as a natural-language token, but many of these tokens support linguistic coherence, exposition, or redundancy rather than the underlying problem-solving state [8,9]. Each token still incurs the full cost of autoregressive generation, fundamentally limiting the scalability of explicit reasoning. This motivates continuous CoT, or latent reasoning. As illustrated in Fig. 1a, latent reasoning carries intermediate computation as continuous vectors in hidden space rather than as discrete language tokens at each step [10]. Recent latent-reasoning models already match or surpass explicit CoT on standard benchmarks with far shorter intermediate computation [11, 12, 13, 14, 15, 9]. Despite this promise, latent reasoners remain imitation-bound: they are largely trained to align their intermediate states with manually-selected [10,12], compressed [13,14], or rendered [15,9] explicit CoT representations. Explicit CoT has already moved beyond this imitation stage: RLVR [5,6] is now a mature recipe for scaling past teacher imitation in explicit reasoning. Because each reasoning step is a token sampled from the vocabulary distribution, policy gradients can assign trajectory-level credit through a tractable per-step likelihood. Moreover, rollout length is inherently variable, so the same reward can reshape both the reasoning content and test-time compute allocation. However, latent reasoning has not yet reached an equivalent outcome-optimization stage. As shown in Fig. 1a, intermediate steps propagate as continuous vectors that bypass the vocabulary distribution, latent trajectories thus lack a tractable action likelihood through which outcome rewards can assign transition-level credit [16,17]; existing latent reasoners moreover prescribe a fixed thinking budget throughout training and inference [10,12], freezing the compute horizon that RL would otherwise optimize. In sum, absent both a tractable latent-transition likelihood and an adaptive stopping interface, outcome rewards cannot elicit latent test-time scaling. In this work, we introduce Surrogate Latent Policy Optimization (SLPO), which instantiates both com- ponents to bring outcome-level RL to latent reasoning, as illustrated by Fig. 1b. To score each transition, SLPO defines a surrogate likelihood in hidden space, directly converting rollout advantages into credit over vector-based reasoning. To make the rollout horizon learnable, we equip the reasoner with a stopping head and initialize it through a correctness-supervised cold start, providing an initial stopping policy that outcome-reward optimization subsequently refines for adaptive computation. Together, these components enable outcome-reward optimization over complete trajectories in any autoregressive latent reasoner, using standard algorithms such as RLOO or GRPO. Empirically, SLPO produces a consistent latent test-time scaling effect: it raises both Pass@8 and Pass@16 in every evaluated continuous backboneโdataset setting, with gains of up to 12.07 percentage points. This effect persists across RLOO and GRPO and transfers to soft-token latent inference. The learned stopping policy further converts a fixed thinking budget into difficulty-adaptive computation, allocating longer latent trajectories to harder problems. We summarize our main contributions as follows: โข We expose the missing optimization interface between outcome rewards and latent reasoning: continuous latent transitions lack a tractable policy likelihood, while fixed thinking budgets preclude reward-driven 2 SLPO: Scaling Latent Reasoning via a Surrogate Policy compute allocation. โข We introduce SLPO, a surrogate policy interface that enables trajectory-level credit assignment over continuous latent transitions and jointly learns a variable-horizon stopping policy. โข Across two latent reasoners, two backbones, and three held-out benchmarks, SLPO improves both Pass@8 and Pass@16 in all 12 backboneโdataset settings, with gains of up to 12.07 percentage points. The gains persist across RLOO and GRPO, transfer to soft-token inference, and accompany difficulty-adaptive latent computation (Fig. 1c). 2 Related Work Latent reasoning. Chain-of-Thought exposes intermediate computation as discrete tokens, yet much of that verbalization is linguistically redundant relative to the underlying problem state [8,11]. Latent reasoning instead autoregresses in hidden space, compressing explicit traces into continuous thoughts [10,12,13, 14,15,9,18,19]. COCONUT [10] replaces explicit steps with latent states trained through curriculum compression; CODI [12] self-distills CoT into continuous surrogates; CoLaR [13] dynamically compresses reasoning chains with prompt-controlled thinking speed; ReGuLaR [9] grounds variational latent traces with rendered CoT guidance; DART [18] distills autoregressive CoT into non-autoregressive Silent Thought tokens; and Latent-SFT [19] constrains latent tokens to vocabulary-space superpositions. These methods largely optimize latent states under SFT-style supervision derived from explicit reasoning, and typically prescribe a fixed latent budget at inference. A complementary line keeps model weights frozen and scales compute only at test time: parallel latent test-time scaling induces stochastic rollouts via MC-dropout or additive noise, then aggregates trajectories with a learned latent reward model [20]. That framework improves coverage without updating latent transition parameters. Reinforcement learning. Outcome-reward RL has become central to scaling explicit CoT [7,5,6]. Group- relative objectives such as GRPO [21] and leave-one-out baselines such as RLOO [22] operate on token policies with tractable sequence likelihoods. The obstacle for latent reasoning is that continuous hidden transitions do not expose such action log-probabilities [16,17]. Recent latent-policy methods address this gap through vocabulary-mediated or architecture-specific policy interfaces [23,24,13], but most still route latent reasoning through the vocabulary interface. LEPO [23] injects Gumbel noise into soft-token rollouts and optimizes latent and discrete tokens within one trajectory; Latent-GRPO [24] stabilizes GRPO over vocabulary-space latent tokens with masking and path-selection heuristics; CoLaR [13] couples dynamic compression with RL over a learned Gaussian head in its own latent parameterization. These designs depend on token distributions, embedding mixtures, or architecture-specific latent heads, and therefore do not directly carry over to arbitrary vector-based latent reasoners that propagate plain hidden states. To our knowledge, outcome-reward policy optimization over unconstrained hidden-state recurrence, without vocabulary-level probabilities, remains unexplored. 3 Preliminaries We begin by formalizing RL over explicit CoT and reviewing latent CoT, and outline dropout-induced stochastic latent rollouts that underlie the methodology below. For each training instance(ํฅ ํ , ํ โ ํ ),ํฅ ํ denotes the input problem and ํ โ ํ denotes the reference answer. Explicit CoT and RL. In explicit CoT, a model samples a discrete reasoning trace autoregressively from the next-token distribution [1,2]. At reasoning stepํก,ํฆ ํก โผ ํ ํ (ยท | ํฅ ํ , ํฆ <ํก ); after the reasoning trace is sampled, the model generates the final answer tokens conditioned on the trace. The resulting rollout likelihood is 3 SLPO: Scaling Latent Reasoning via a Surrogate Policy therefore available directly from token probabilities: logํ ํ (ํฆ 1:ํ , หํ ํ | ํฅ ํ )= ํ โ๏ธ ํก=1 logํ ํ (ํฆ ํก | ํฅ ํ , ํฆ <ํก )+ |หํ ํ | โ๏ธ ํ =1 logํ ํ (หํ ํ,ํ | ํฅ ํ , ํฆ 1:ํ , หํ ํ,<ํ ). Given a verifiable rewardํ (หํ ํ , ํ โ ํ ), policy-gradient methods assign credit to sampled rollouts by weighting this log-probability term with an advantage such as b ํด ํ = ํ (หํ ํ , ํ โ ํ )โ ํ ํ . Thus explicit CoT exposes the core policy interface needed by RL: the sampled reasoning trajectory already comes with token-level action probabilities. Latent CoT. Continuous CoT replaces the discrete reasoning trace with an autoregressive trajectory in hidden space [10,11]. We write this trajectory asโ ํ,1:ํ max = (โ ํ,1 , . . . , โ ํ,ํ max ), whereโ ํ,ํก โ โ ํ is the latent state at step ํก. The latent update can be written abstractly as โ ํ,ํก = ํ ํ (ํฅ ํ , โ ํ,<ํก ), with the answer decoder conditioning on the completed latent trajectory. During latent reasoning, the intermediate computation bypasses the vocabulary distribution and language-head sampling, which removes the token-policy likelihood that explicit CoT exposes to RL. Prior work has shown that latent reasoning can nevertheless be made stochastic by inference under dropout [20]. Such stochasticity supplies the exploration substrate for RL, and we denote a sampled trajectory by ฬ โ ํ,1:ํ max . However, how to construct a tractable log-likelihood for the realized latent states is still an open question. 4 Methodology Existing latent reasoners typically prescribe a fixed thinking budget, so the model cannot decide when to terminate latent computation on its own. We therefore proceed in two stages. We first attach a stopping head and, for each sampled trajectory, supervise its distribution over stopping times with final-answer correctness evaluated at every candidate length (Sec. 4.1). This introduces a learnable continue-versus-stop decision before outcome-reward optimization. SLPO then makes outcome rewards usable for eliciting latent reasoning scaling: a tractable surrogate likelihood scores continuous latent transitions, and the stopping likelihood is included in the policy objective, so verifiable rewards jointly shape latent reasoning and compute allocation (Sec. 4.2). We close with the inference procedure (Sec. 4.3). 4.1 Stopping-gate Cold Start We augment the latent reasoner with a stopping headํ ํ over latent states. Given a latent stateโ, the head outputs ํ ํ (โ)= ํ(ํ ํ (โ)), which is the probability of terminating latent computation at that state. For an inputํฅ ํ , the released backbone samples ํ stochastic latent trajectories up to a maximum budget: โ (ํ) ํ,1:ํ max โผ ํ ํ (ยท | ํฅ ํ ), ํ= 1, . . . , ํ. For each sampled trajectory, we enumerate candidate stopping lengthsํก โ [ํ min ,ํ max ]. At each candidate length, we halt latent thinking afterโ (ํ) ํ,ํก and decode final-answers explicitly conditioned onํฅ ํ and the resulting prefix: ํ ํ,ํ,ํก โผ ํ ํ (ยท | ํฅ ํ , โ (ํ) ํ,1:ํก ). Using a verifiable reward, the answer-valid stopping set is V (ํ) ํ = ํก โ [ํ min ,ํ max ] | ํ (ํ ํ,ํ,ํก , ํ โ ํ )= 1 . 4 SLPO: Scaling Latent Reasoning via a Surrogate Policy Figure 2: Overview of SLPO. Verifiable outcomes induce rollout advantages that weight the latent-surrogate, answer, and gate terms. For a sampled trajectory, writeํ ํ,ํ,ํก = ํ ํ (โ (ํ) ํ,ํก ) as the stop probability at stepํก. The probability that stopping occurs at length ํก is then ํ ํ (ํ (ํ) ํ = ํก)= ํ ํ,ํ,ํก ร ํ<ํก (1โ ํ ํ,ํ,ํ ). The gate is trained to place probability mass on stopping times insideV (ํ) ํ , which gives L (ํ) stop =โ 1 ํ ํ โ๏ธ ํ=1 log โ๏ธ ํกโV (ํ) ํ ํ ํ (ํ (ํ) ํ = ํก). 4.2 Surrogate Latent Policy Optimization As illustrated in Fig. 2, SLPO couples latent-transition, answer-token, and stopping-time likelihoods in a single reward-weighted objective. Surrogate transition likelihood. SLPO constructs a tractable Gaussian surrogate for stochastic latent transitions from repeated MC-dropout evaluations. For the prefix(ํฅ ํ , โ ํ,<ํก ), we runํพstochastic forward evaluations with independent dropout masks during training: ํง (ํ) ํ,ํก = ํ ํ (ํฅ ํ , โ ํ,<ํก ; ํ (ํ) ํ,ํก ), ํ= 1, . . . , ํพ. These samples parameterize the Gaussian surrogate through an empirical mean and a floored isotropic variance: ํ ํ,ํก = 1 ํพ ํพ โ๏ธ ํ=1 ํง (ํ) ํ,ํก , ํ 2 ํ,ํก = 1 ํพํ ํพ โ๏ธ ํ=1 ํง (ํ) ํ,ํก โ ํ ํ,ํก 2 2 + ํ. The realized state โ ํ,ํก is then scored under the isotropic Gaussian surrogate: log eํ ํ (โ ํ,ํก | ํฅ ํ , โ ํ,<ํก )=โ ํ 2 log 2ํํ 2 ํ,ํก โ โฅโ ํ,ํก โ ํ ํ,ํก โฅ 2 2 2ํ 2 ํ,ํก . MC-dropout generates the sampled trajectories, and the isotropic Gaussian supplies the per-step surrogate score used for advantage-weighted credit assignment (App. A.5). The sampled rollout states are treated 5 SLPO: Scaling Latent Reasoning via a Surrogate Policy as stop-gradient targets; gradients flow through the recomputed moments(ํ ํ,ํก , ํ 2 ํ,ํก ) . Under fixed variance, a positive advantage raiseseํ ํ (โ ํ,ํก )by movingํ ํ,ํก towardโ ํ,ํก , while a negative advantage pushesํ ํ,ํก away (Prop. 1). Rollout policy objective. For each inputํฅ ํ , we sample one or more latent rollouts, decode final answersํ ํ , and compute a verifiable reward ํ ํ = ํ (ํ ํ , ํ โ ํ ). SLPO targets the expected rollout reward ํฝ(ํ)= ํผ (ํฅ ํ ,ํ โ ํ )โผD ํผ ํ ํ โผํ ํ (ยท|ํฅ ํ ) ํ (ํ ํ , ํ โ ํ ) . Hereํ ํ denotes the behavior trajectory law induced by MC-dropout, answer decoding, and the stopping gate. In practice, MC-dropout supplies behavior trajectories, while the Gaussian surrogate converts their sampled latent states into a differentiable score. The resulting update is the gradient of an explicitly defined empirical surrogate objective (Prop. 2); App. A.5 characterizes when this update follows the expected-reward gradient and gives a sufficient condition for local improvement ofํฝ(ํ). After the stopping-gate cold start, each rollout has an adaptive stopping timeํ ํ โค ํ max . Writeํ ํ = (โ ํ,1:ํ ํ , ํ ํ ). The surrogate rollout score combines the latent-transition surrogate, the answer-token likelihood, and the stopping-time likelihood term from Sec. 4.1: e โ ํ (ํ ํ | ํฅ ํ )= ํ ํ โ๏ธ ํก=1 log eํ ํ (โ ํ,ํก | ํฅ ํ , โ ํ,<ํก )+ |ํ ํ | โ๏ธ ํ =1 logํ ํ (ํ ํ,ํ | ํฅ ํ , โ ํ,1:ํ ํ , ํ ํ,<ํ ) + log ํ ํ (ํ ํ ). The gate term assigns credit to stopping decisions: higher-reward stopping times reinforce continue/stop probabilities throughํ ํ,ํก . With advantageํด ํ = ํ ํ โ ํ ํ , whereํ ํ may be a group-relative or leave-one-out baseline, the surrogate loss is L RL =โํด ํ e โ ํ (ํ ํ | ํฅ ํ ). Differentiating this loss yields the detached empirical update characterized in App. A.5 (Prop. 2); RLOO or GRPO specify the stop-gradient advantageํด ํ . The composite score thereby extends outcome-level credit into the latent recurrence while jointly optimizing answer generation and trajectory length. 4.3 Inference At inference, latent reasoning proceeds sequentially. After each latent state, the stopping head emits a stop probability, and the realized stopping time is the first step where this probability exceeds a threshold. The final answer is decoded only after this stopping event. Thus adaptive computation changes the number of latent steps used per instance without requiring inference-time enumeration of candidate prefixes. We leave the algorithm details to App. A.6. 5 Experiment Setup Our experiments test whether SLPO (i) consistently improves outcome-level performance across latent reasoners, (i) remains compatible with different policy-optimization algorithms, (i) transfers across con- tinuous and soft latent interfaces, and (iv) converts a fixed latent budget into difficulty-adaptive test-time computation. Datasets. We evaluate SLPO on grade-school mathematical reasoning. We train onGSM8K-Aug[10], an augmented variant of GSM8K, and evaluate on three held-out benchmarks: (1)GSM8K-Test, the official GSM8K test split; (2)GSM-Hard[20], a harder variant of GSM8K-Test with larger-magnitude numbers; and (3)MultiArith[20], a multi-step arithmetic set. For the soft-latent transfer experiment in Sec. 7.3, we additionally evaluate onMATH500,AIME 2025, andAMC23. The setup follows prior latent-reasoning bench- marks [10, 12, 13, 20]. 6 SLPO: Scaling Latent Reasoning via a Surrogate Policy Table 1: Comparison between latent-reasoning baselines and our variants. Acc is deterministic accuracy with dropout disabled; Pass@ํ for ํ โ 8, 16 is the mean over three independent MC-dropout evaluation seeds (ํ= 0.1). Method GSM8KGSM-HardMultiArithAverage Acc P@8 P@16 Acc P@8 P@16 Acc P@8 P@16 Acc P@8 P@16 GPT2โ124M COCONUT34.12 45.79 49.13 7.66 10.70 12.06 80.86 88.79 91.38 40.88 48.43 50.86 COCONUT+SLPO35.6349.1351.557.6610.8512.5283.1091.3892.5942.1350.4552.22 CODI42.30 52.08 54.36 9.26 11.84 12.67 90.21 96.90 97.41 47.59 53.61 54.81 CODI+SLPO42.7654.1356.719.7112.0613.0590.5297.2497.7647.6654.4855.84 Llama3.2โ1B COCONUT21.23 24.64 38.13 4.55 6.98 9.94 41.1 45.00 63.10 22.29 25.54 37.06 COCONUT+SLPO22.3730.4841.855.778.1910.9346.9057.0771.3825.0131.9141.38 CODI55.22 63.91 67.48 12.82 15.02 15.63 95.52 98.28 98.79 54.59 59.07 60.63 CODI+SLPO55.2765.1370.2813.2015.8616.7796.3898.7999.4854.9559.9362.18 Models and baselines. We apply SLPO to publicly released checkpoints from COCONUT [10] and CODI [12], using bothGPT-2 (124M)andLlama-3.2-1B-Instructas backbones. Released COCONUT and CODI evaluate under a fixed budget ofํ max =6 latent steps as in the original papers;+SLPOkeeps the same vector recurrence but raises the maximum budget toํ max =12 and lets the stopping gate choose the realized length (Sec. 4.3; App. A.2). We compare our variants against commonly used latent-reasoning baselines: COCONUT [10] and CODI [12], CoLaR [13], ReGuLaR [9], DART [18], and Latent-SFT [19], and the explicit-reasoning methods CoT-SFT and iCoT [10]. CoLaR is run at 2รthinking speed with a maximum latent budget of 64 steps; ReGuLaR uses the releasedLlama-3.2-1B-Instructcheckpoint; DART Acc and #L are taken from Jiang et al. [18]on the same Llama-3.2-1B setting (fixedํถ=20 Silent Thought tokens); Latent-SFT Acc and #L follow theํ=2 Llama-3.2-1B setting in Deng et al.[19]. For transfer experiments using the soft-token interface (Sec. 7.3), we apply SLPO toLlama-3.2-1B-InstructandLlama-3.2-3B-Instructand compare against CoT with soft-token inference [25] and LEPO [23]. Training and Evaluation Protocols. Unless otherwise specified,+SLPOvariants first train the stopping head as described in Sec. 4.1, then optimize SLPO with RLOO and rollout hyperparameters(ํพ,ํบ)=(4,8)with GSM8K-Aug, with further details specified in App. A. At test time,+SLPOmodels decode with the learned stopping gate in Sec. 4.3, with the stop threshold selected by validation sweep (App. A.7). We report Acc from a single deterministic rollout (dropout disabled). Pass@ํforํ โ 8,16drawsํindependent MC-dropout rollouts (ํ=0.1) per problem; we repeat this evaluation with three independent dropout seeds and report the mean Pass@ํ(App. A.7). For Tab. 2, we additionally report average reasoning length (#L): token-chain length for explicit CoT, the fixed latent budget for ungated latent baselines, and the realized stopping time under ํ max = 12 for +SLPO. 6 Main Results 6.1 Consistent Scaling Across Latent Reasoners Tab. 1 shows a consistent latent test-time scaling effect: SLPO raises both Pass@8 and Pass@16 on all 12 backboneโdataset combinations. The gains span curriculum-trained COCONUT and distillation-based CODI, GPT-2 and Llama-3.2-1B, and all three held-out benchmarks without altering the underlying vector recurrence. Deterministic accuracy also improves on average for every backboneโreasoner pair. The improvement reaches 7 SLPO: Scaling Latent Reasoning via a Surrogate Policy Table 2: Broader model results on grade-school mathematical reasoning benchmarks. Acc is deterministic accuracy. #L is average reasoning length: fixed latent budget for ungated COCONUT/CODI (ํ max =6), realized stopping time for +SLPOunderํ max =12, and the corresponding length metric reported by each baseline. DART and Latent-SFT Acc/#L are cited from Jiang et al. [18], Deng et al. [19]. TypeMethod GSM8KGSM-HardMultiArithAverage Acc#LAcc#LAcc#LAcc#L GPT2โ124M Explicit iCoT30.105.7055.5030.40 CoT-SFT44.1 25.1 9.8 28.9 90.7 10.9 48.2 21.6 Latent COCONUT34.167.666 80.86 6 40.87 6 CODI42.30 69.266 90.21 6 47.59 6 COCONUT+SLPO35.63 5.73 7.66 6.08 83.10 6.28 42.13 6.03 CODI+SLPO42.76 11.839.7111.9490.5211.4447.6611.74 Llama3.2โ1B Explicit iCoT19.803.9038.2020.60 CoT-SFT54.1 25.4 15.6 34.2 99.3 13.7 56.3 24.4 Latent COCONUT21.23 64.55641.16 22.29 6 CODI55.22 6 12.82 6 95.52 6 54.59 6 CoLaR26.6 5.56 5.87 7.01 86.4 3.23 39.6 5.27 ReGuLaR34.9 3.69 8.27 4.12 89.2 2.28 44.1 3.36 DART42.6 20 10.9 20 84.8 20 46.1 20 Latent-SFT52.4 12.8 12.6 15.0 96.8 7.19 53.9 11.66 COCONUT+SLPO22.37 7.52 5.77 7.88 46.90 7.49 25.01 7.63 CODI+SLPO55.27 6.30 13.20 6.41 96.38 4.65 54.95 5.79 12.07 percentage points, or 26.8% relative, for Pass@8 on MultiArith with Llama-3.2-1B COCONUT. This substantial expansion in successful parallel rollouts is the clearest signature of the latent policy elicited by outcome-reward optimization. 6.2 Accuracy with Adaptive Latent Computation Tab. 2 widens the comparison to explicit CoT-SFT, iCoT, and latent baselines. On Llama-3.2-1B, CODI+SLPO achieves the strongest average accuracy among latent methods with 5.79 latent steps on average; compared with Latent-SFT, it raises average accuracy by 1.05 percentage points while using 50.3% fewer latent steps. On GSM8K, CODI+SLPO surpasses CoT-SFT by 1.17 percentage points while using 75.2% fewer reasoning steps. The same optimization improves COCONUT, showing that SLPO strengthens both curriculum-trained and distillation-trained latent initializations. On GPT-2, CODI+SLPO likewise achieves the strongest average accuracy among the latent methods. These results position SLPO as an outcome-optimization layer that improves latent reasoning while allowing the learned gate to allocate, rather than uniformly spend, the available compute budget. 7 Analysis 7.1 Rollout Hyperparameter Analysis SLPO couples surrogate transition-likelihood estimation with reward optimization through rollout hyperpa- rametersํพandํบ.ํพcontrols the number of stochastic forward passes used to estimate each surrogate in Sec. 4.2, whereasํบis the group size controlling how many complete trajectories are sampled per problem. 8 SLPO: Scaling Latent Reasoning via a Surrogate Policy a ํพ sweep, GSM-Hardb ํบ sweep, GSM-Hardc ํพ sweep, MultiArithd ํบ sweep, MultiArith Figure 3: Controlled hyperparameter sweeps for SLPO. Each panel varies one factor over2,4,8while holding the other fixed (ํพ sweep at ํบ= 4; ํบ sweep at ํพ= 4). Full values are in App. C. Figure 4: Pass@ํforํ โ 1,8,16under SLPO with RLOO versus GRPO on COCONUT and CODI. Blue curves denote COCONUT; red curves denote CODI. Solid curves use RLOO and dashed curves use GRPO. Full values appear in App. B. We vary each factor over2, 4, 8 while holding the other fixed (Fig. 3); full values appear in App. C. Fig. 3 identifies group sizeํบas the primary rollout-scaling axis: increasingํบconsistently improves Pass@2 on both GSM-Hard and MultiArith. Performance is substantially less sensitive toํพ, indicating that a modest number of stochastic forward passes already provides an effective surrogate estimate. SLPO can therefore direct additional rollout compute toward exploring more complete trajectories, where it yields the clearest return. 7.2 Generalization Across Policy-Optimization Algorithms SLPO defines a surrogate likelihood on hidden transitions that converts verifiable final-answer rewards into vector-space credit, as formalized in Sec. 4.2. Here we analyze whether SLPO adapts across outcome-reward policy-optimization algorithms by comparing RLOO and GRPO under the same surrogate, rollout budget, and latent backbones. Fig. 4 shows closely aligned scaling curves for RLOO and GRPO across COCONUT and CODI on MultiArith, GSM-Hard, and GSM8K. Both algorithms preserve SLPOโs gains over the released backbone initialization across latent reasoners and sampling budgets. The surrogate transition likelihood therefore provides a reusable outcome-reward interface across distinct advantage estimators. 7.3 Generalization to Soft Latent Inference Figure 5: Mean generated sequence length dur- ing soft-token outcome-reward optimization on Llama3.2โ1B. SLPO targets outcome-reward optimization over complete la- tent trajectories in any autoregressive latent reasoner whose intermediate steps are vector states propagated by the back- bone rather than vocabulary-level actions, as formalized in Sec. 4.2. Soft latent inference constitutes another vector- 9 SLPO: Scaling Latent Reasoning via a Surrogate Policy Table 3: Soft latent inference results with Llama3.2โ1B. MethodGSM8KMATH500 Acc. P@32 Len. Acc. P@32 Len. CoT37.30 80.50 200.18 21.1 70.40 599.02 CoT (w/ latent inference) 35.78 79.23 188.27 20.80 69.50 605.27 GRPO45.56 87.57 238.50 25.80 69.40 602.98 GRPO (w/ latent inference) 46.25 80.74 245.49 26.60 69.00 581.73 LEPO40.56 77.56 197.43 24.80 70.20 572.27 SLPO (ours)46.70 82.03 256.65 27.20 71.60 642.32 Table 4: Soft latent inference Pass@ํ results with Llama3.2โ3B. MethodGSM8KMATH500AIME 2025AMC23 P@1 P@32 P@1 P@32 P@1 P@32 P@1 P@32 CoT73.22 96.89 43.75 84.40 0.52 13.33 21.48 75.00 CoT (w/ latent inference)73.32 97.42 43.59 85.00 0.83 16.67 21.64 77.50 GRPO74.69 97.27 43.89 83.20 0.42 13.33 24.14 75.00 GRPO (w/ latent inference) 74.41 96.66 43.86 84.80 0.42 16.67 23.98 72.50 Soft Tokens75.27 93.03 44.08 81.60 0.42 10.00 20.55 72.50 HRPO77.10 90.30 39.60 79.60 0.42 6.67 17.50 67.50 LEPO77.29 96.97 46.51 86.20 0.96 16.67 27.03 77.50 SLPO (ours)77.63 95.30 45.00 82.00 3.33 20.00 32.50 77.50 based latent paradigm [24,23], recurring on a vocabulary- weighted token embedding rather than the hidden state. We thus evaluate SLPO under this setting to test whether the objective generalizes across vector-based latent inference mechanisms. Tabs. 3 and 4 compare SLPO with vocabulary- routed baselines onLlama3.2โ1BandLlama3.2โ3B; Fig. 5 tracks mean rollout length during soft-token training on the 1B setup. Implementation details appear in App. A.9. Tabs. 3 and 4 establish that SLPO transfers beyond hidden-state recurrence. At 1B, SLPO achieves the best deterministic accuracy on both GSM8K and MATH500 and the best MATH500 Pass@32. At 3B, it attains the strongest results on the more selective AIME 2025 and AMC23 evaluations, raising AIME 2025 Pass@1 from 0.96 to 3.33 (3.47ร) and AMC23 Pass@1 from 27.03 to 32.50 (+5.47 percentage points; 20.2% relative). Fig. 5 exposes the corresponding sequential scaling behavior: SLPO steadily expands rollout length under reward optimization, whereas GRPO and LEPO remain near initialization or contract. SLPO thus elicits both stronger outcomes and increased test-time computation through a distinct soft-token latent interface. 7.4 Latent Geometry Analysis Outcome-reward RL reinforces successful latent rollouts, but the underlying latent geometry change remains unclear. We thus track two complementary signals to characterize how SLPO reshapes latent thinking geometry: inter-step separation, the mean cosine distance between consecutive latent statesโ ํ,ํก andโ ํ,ํก+1 along each rollout, and relative change in prefix effective rank. Fig. 6a reports the former and Fig. 6b the latter; see metric definitions, sampling details, and prefix-length rank curves in App. D. For this geometry probe we additionally includeSVAMP[10], a grade-school set with lexical perturbations. Fig. 6a reports the relative change in mean inter-step cosine distance after SLPO across backbones and datasets. Inter-step distance increases for every backboneโdataset pair, with the largest change on GSM-Hard. 10 SLPO: Scaling Latent Reasoning via a Surrogate Policy +13.7% +7.2% +17.1% +17.8% +6.3% +3.9% +7.2% +3.8% GSM8K GSM-Hard MultiArith SVAMP 0 5 10 15 20 25 GPT2LLaMA Change in Inter-step Cosine Distance Dataset Relative change (%) a Inter-step cosine distance โ10โ50โ10โ50 GPT2LLaMA Change in Prefix Effective Rank CODICoconut SVAMP MultiArith GSM8K Relative change (%) b Relative rank change Figure 6: Latent geometry pre- and post-SLPO. Fig. a: relative change in mean cosine distance between consecutive latent states across backbones and datasets. Fig. b: relative change in prefix effective rank across backbones, datasets; metric definitions and prefix-length rank curves appear in App. D. 0.0-0.10.1-0.20.2-0.30.3-0.40.4-0.50.5-0.60.6-0.70.7-0.80.8-0.90.9-1.0 5 5.2 5.4 5.6 5.8 6 6.2 Bin mean ยฑ SEMLinear fit Difficulty (1 โ acc@32) Mean latent length Pearson r = 0.297 a GSM8K validation 0.0-0.10.1-0.20.2-0.30.3-0.40.4-0.50.5-0.60.6-0.70.7-0.80.8-0.90.9-1.0 5 5.2 5.4 5.6 5.8 6 6.2 Bin mean ยฑ SEMLinear fit Difficulty (1 โ acc@32) Mean latent length Pearson r = 0.262 b GSM8K test Figure 7: Relationship between problem difficulty and latent length under the learned stopping gate. Bars show bin means with standard errors, and the dashed line shows the linear fit. Per-problem Pearson correlations areํ=0.30 (validation) and ํ= 0.26 (test), both with two-sided ํ < 0.001. Successive latent states therefore become more differentiated after SLPO, producing a clearer stage-wise progression through the latent thinking process. Inter-step separation captures pairwise progression along each rollout. We next track prefix effective rank, which measures how many independent directions the latent prefix spans, with definition given in App. D.4. Fig. 6b reports the relative change in prefix effective rank after SLPO across backbones, datasets, and latent-training methods. Effective rank decreases in every backboneโdatasetโmethod combination. The latent prefix therefore concentrates into a lower-dimensional subspace after SLPO. Together, the rise in inter-step separation and this rank compression show that SLPO yields stronger differentiation between successive thinking steps while steering rollouts through a sharpened set of task- relevant directions. 7.5 Difficulty-Adaptive Latent Compute Allocation The stopping-gate cold start trains the stop gate from the correctness of answers decoded under varying latent lengths (Sec. 4.1). We test whether the resulting gate allocates latent computation by difficulty by 11 SLPO: Scaling Latent Reasoning via a Surrogate Policy measuring difficulty as the empirical failure rateํ ํ =1โ acc@32 ํ and comparing gate-selected latent length across difficulty bins (Fig. 7). As shown in Fig. 7, harder problems receive longer latent trajectories on both validation and test. Easy bins concentrate on shorter prefixes, while the hardest bins draw the longest hidden computation (Pearson ํ=0.30 on validation andํ=0.26 on test; two-sidedํก-tests, bothํ <0.001). The learned gate therefore provides adaptive test-time scaling in latent space: harder instances draw more computation, while easy instances terminate early rather than following a single global budget. 8 Conclusion We introduce SLPO for outcome-reward optimization in autoregressive latent reasoners. SLPO defines a differentiable surrogate policy interface over MC-dropout latent transitions, extending trajectory-level reward credit directly into vector-space reasoning. A correctness-supervised stopping-gate cold start establishes a prior over stopping times, allowing SLPO to optimize adaptive computation alongside latent reasoning. Across continuous and soft thinking inference, SLPO realizes latent test-time scaling through higher Pass@ํ under parallel sampling and longer latent trajectories on harder instances with improved deterministic accuracy. Future work will extend SLPO to larger backbones, open-ended reasoning, and multimodal latent architectures. References [1] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. NIPS โ22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088. [2]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/forum?id=1PL1NIMMrw. [3] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candรจs, and Tatsunori Hashimoto. s1: Simple test-time scaling. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20275โ20321, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.1025. URL https://aclanthology.org/2025.emnlp-main.1025/. [4] Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=4FWAwZtd2n. [5]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [6]Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245, 2025. [7] Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, et al. A survey of reinforcement learning for large reasoning models. arXiv preprint arXiv:2509.08827, 2025. [8] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models, 2025. URL https://arxiv.org/abs/2503.16419. [9] Fanmeng Wang, Haotian Liu, Guojiang Zhao, Hongteng Xu, and Zhifeng Gao. Regular: Variational latent reasoning guided by rendered chain-of-thought, 2026. URL https://arxiv.org/abs/2601.23184. [10]Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason E Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=Itxz7S4Ip3. 12 SLPO: Scaling Latent Reasoning via a Surrogate Policy [11]Xinghao Chen, Anhao Zhao, Heming Xia, Xuan Lu, Hanlin Wang, Yanjun Chen, Wei Zhang, Jian Wang, Wenjie Li, and Xiaoyu Shen. Reasoning beyond language: A comprehensive survey on latent chain-of-thought reasoning, 2025. URL https://arxiv.org/abs/2505.16782. [12]Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. CODI: Compressing chain-of-thought into continuous space via self-distillation. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 677โ693, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.36. URL https://aclanthology.org/2025.emnlp-main.36/. [13]Wenhui Tan, Jiaze Li, Jianzhong Ju, Zhenbo Luo, Ruihua Song, and Jian Luan. Think silently, think fast: Dynamic latent compression of LLM reasoning chains. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=AQsko3PPUe. [14]Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Jiaqi Wang, Xipeng Qiu, and Dahua Lin. Sim-cot: Supervised implicit chain-of-thought, 2025. URL https://arxiv.org/abs/2509.20317. [15] Yifan Wang, Shiyu Li, Peiming Li, Xiaochen Yang, Yang Tang, and Zheng Wei. Render-of-thought: Rendering textual chain-of-thought as images for visual latent reasoning, 2026. URL https://arxiv.org/abs/2601.14750. [16]Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025. [17]Phuc Minh Nguyen, Chinh D. La, Duy M. H. Nguyen, Nitesh V. Chawla, Binh T. Nguyen, and Khoa D. Doan. The reasoning boundary paradox: How reinforcement learning constrains language models. arXiv preprint arXiv:2510.02230, 2025. [18] Nan Jiang, Ziming Wu, De-Chuan Zhan, Fuming Lai, and Shaobing Lian. DART: Distilling autoregressive reasoning to silent thought. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5100โ5108, Suzhou, China, November 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.emnlp-main.256. URL https://aclanthology.org/2025.emnlp-main.256/. [19]Jingcheng Deng, Liang Pang, Zihao Wei, Shichen Xu, Zenghao Duan, Kun Xu, Yang Song, Huawei Shen, and Xueqi Cheng. LLM latent reasoning as chain of superposition, 2025. URL https://arxiv.org/abs/2510.15522. [20]Runyang You, Yongqi Li, Meng Liu, Wenjie Wang, Liqiang Nie, and Wenjie Li. Parallel test-time scaling for latent reasoning models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, USA, November 2026. Association for Computational Linguistics. [21]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [22]Lianghuan Huang, Shuo Li, Sagnik Anupam, Insup Lee, and Osbert Bastani. Effective reinforcement learning for reasoning in language models. arXiv preprint arXiv:2505.17218, 2025. [23] Yuyan Zhou, Jiarui Yu, Hande Dong, Zhezheng Hao, Hong Wang, Jianqing Zhang, and Qiang Lin. LEPO: Latent reasoning policy optimization for large language models, 2026. URL https://arxiv.org/abs/2604.17892. [24] Jingcheng Deng, Zihao Wei, Liang Pang, Junhong Wu, Shicheng Xu, Zenghao Duan, and Huawei Shen. Latent- GRPO: Group relative policy optimization for latent reasoning, 2026. URL https://arxiv.org/abs/2604.27998. [25]Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, and Xin Eric Wang. Soft thinking: Unlocking the reasoning potential of LLMs in continuous concept space. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2505.15778. A Implementation Details This appendix supplements Sec. 5 with model, training, and evaluation details omitted from the main text for space. A.1 Base Models and Initialization All SLPO experiments start from publicly available latent-reasoning checkpoints, using official baseline releases when available and open third-party reimplementations otherwise. COCONUT and CODI use GPT-2 (124M) or Llama-3.2-1B-Instruct, depending on the row in the main tables; CoLaR, ReGuLaR, DART, and 13 SLPO: Scaling Latent Reasoning via a Surrogate Policy Latent-SFT use Llama-3.2-1B-Instruct. We do not retrain the imitation stage of these latent reasoners: The stopping-gate cold start and SLPO are applied directly on top of these initialized weights. For fair comparison with explicit CoT baselines, CoT-SFT and iCoT numbers follow the training protocol reported by Hao et al. [10], Shen et al. [12], Tan et al. [13]. A.2 Stopping-gate Cold Start The stopping-gate cold start is applied first on the same GSM8K-Aug split. For each training problem, the released backbone samplesํ=4 stochastic latent trajectories with rollout dropout (ํ=0.1) up to ํ max =12, the same maximum budget used at SLPO training and gated inference (raised from the ungated COCONUT/CODI horizon of six). Candidate stopping lengths lie in[ํ min ,ํ max ]withํ min =3. For every trajectory and candidate prefix length, answer decoding runs with dropout disabled to stabilize the online supervision signal. The stopping head is trained with the first-stop objective in Sec. 4.1. We do not add a compute-aware auxiliary term. Optimization. This stage uses Adafactor at learning rate 10 โ4 for 15 epochs. For GPT-2 (COCONUT and CODI), we use a per-device batch size of 32 with 2 gradient-accumulation steps; for Llama-3.2-1B (COCONUT and CODI), we use a per-device batch size of 8 with 8 accumulation steps. The stop gate itself is regularized with dropoutํ=0.1 during this cold start. We evaluate every 500 optimizer steps, restoring the checkpoint with the highest validation accuracy under early stopping with patience 20. A.3 SLPO Training Reward and advantage. Each training instance(ํฅ ํ , ํ โ ํ ) receives a binary outcome rewardํ ํ = ํ[หํ ํ = ํ โ ํ ] after answer parsing. Advantages are estimated with RLOO overํบ=8 sampled latent trajectories per problem. The surrogate transition likelihood at each step uses ํพ= 4 independent MC-dropout forwards. Stochastic rollouts. Latent trajectories are sampled with MC-dropout at rateํ=0.1 during rollout generation. The realized latent states are treated as stop-gradient targets; gradients flow through the recomputed moments(ํ ํ,ํก , ํ 2 ํ,ํก )in Sec. 4.2, not through the rollout sampling path itself. Answer-token likelihoods and the first-stop gate term are included in the rollout objective as in Sec. 4.2. Optimization. We optimize with Adafactor at learning rate 10 โ6 . Training runs for up to 40,000 optimizer steps under a constant schedule with warmup over the first 2,000 steps. Each process uses a per-device batch size of 16; with 4 GPUs, the effective prompt batch is 64. We do not apply PPO-style clipping or a KL penalty: the SLPO RLOO objective uses ํฝ= 0 and omits reference-policy log-probability terms. Hardware. SLPO is launched with Accelerate over 4 processes on 4 GPUs (4รRTX 5880 Ada). A.4 Computational Cost Analysis SLPO introduces training-time computation throughํบsampled trajectories andํพstochastic forwards for estimating each surrogate transition likelihood. We profile this cost on the same RTX 5880 Ada GPU class using two unique prompts, with two warmup steps followed by eight measured optimizer steps. Tab. 5 reports mean wall-clock time per step and its rollout, surrogate-construction, and backward components. The profile isolates the SLPO training stage; the one-time stopping-gate cold start is described separately in App. A.2. Under the paper configuration(ํพ,ํบ)=(4,8), an optimizer step takes 1.130 seconds; surrogate construction accounts for 0.274 seconds, or 24% of the total, while rollout generation and the backward pass account for 0.294 and 0.545 seconds. The profile also exposes a direct compute control: reducing the group size fromํบ=8 toํบ=4 atํพ=4 lowers step time to 0.676 seconds (0.60ร), whereas increasingํพfrom 4 to 8 at ํบ=8 raises it to 2.021 seconds (1.79ร). Together with the hyperparameter analysis in Fig. 3, which finds larger returns from rollout diversity than from additional surrogate samples, these measurements support allocating training compute to ํบ while keeping ํพ moderate. 14 SLPO: Scaling Latent Reasoning via a Surrogate Policy Table 5: Per-step computational profile on RTX 5880 Ada. All configurations use MC-dropout. Relative time is normalized by the paper configuration(ํพ,ํบ)=(4,8). Sur./step is the fraction of total step time spent constructing the surrogate likelihood. ํพ ํบ Step (s) Rel. Rollout (s) Surrogate (s) Backward (s) Sur./step 1 8 0.624 0.55ร0.3210.1080.17717% 481.1301.00ร0.2940.2740.54524% 8 8 2.021 1.79ร0.3190.5851.10129% 4 4 0.676 0.60ร0.2430.1460.27222% The surrogate estimator and theํพstochastic forwards are used only during training. At inference, SLPO performs the original latent recurrence together with the learned stopping head; its compute is therefore governed by the realized adaptive trajectory length rather than by ํพ or ํบ. A.5 Surrogate Optimization Guarantees We establish five properties of the surrogate in Sec. 4.2: its advantage-weighted update, the empirical objective implemented by SLPO, its relation to the expected-reward gradient, a sufficient condition for expected-reward ascent, and the regularity induced by its variance floor. Throughout, writeโfor a stop- gradient latent state sampled by MC-dropout,(ํ ํ , ํ 2 ํ )for the isotropic moments recomputed fromํพdropout forwards, and log eํ ํ (โ)=โ ํ 2 log(2ํํ 2 ํ )โ โฅโโํ ํ โฅ 2 2 2ํ 2 ํ . Answer-token and stopping terms are unchanged from Sec. 4.2 and are omitted from the latent-only statements below. Proposition 1 (Mean-gradient component under fixed variance). Fixํ 2 ํ >0 (or treatํ 2 ํ as stop-gradient), and hold (โ, ํด) fixed. Then โํด log eํ ํ (โ)= ํด 2ํ 2 ํ โฅโโ ํ ํ โฅ 2 2 + ํถ(ํด, ํ 2 ํ ), so โํดโ ํ log eํ ํ (โ)= ํด ํ 2 ํ (ํ ํ โ โ) โค โ ํ ํ ํ whenever gradients are taken only throughํ ํ . A positive advantage therefore pullsํ ํ towardโ, and a negative advantage pushes it away. Whenํ 2 ํ is also trainable, this identity describes only the mean-gradient component of the latent update; the variance path is additional. Proof sketch. The isotropic surrogate log-density expands aslog eํ ํ (โ)=โโฅโโํ ํ โฅ 2 2 /(2ํ 2 ํ )+ํ(ํ 2 ํ ) . Multiplying byโํดgives the stated loss identity. Differentiating inํ ํ and applying the chain rule yields the gradient form.โก Proposition 2 (Detached empirical surrogate loss). Draw a frozen rollout batchํ ํ ํบ ํ=1 from MC-dropout under parameters ฬ ํ, compute stop-gradient advantagesํด ํ = sg(ํ ํ โ ํ ํ ), and define the empirical surrogate loss b L surr (ํ; ฬ ํ)=โ 1 ํบ ํบ โ๏ธ ํ=1 ํด ํ e โ ํ (ํ ํ | ํฅ). Then the implemented gradient of the latent, answer, and gate terms in Sec. 4.2 coincides withโ ํ b L surr (ํ; ฬ ํ) evaluated atํ= ฬ ํ, with the sampled latent states in e โ ํ treated as stop-gradient targets. Thus the implementation 15 SLPO: Scaling Latent Reasoning via a Surrogate Policy optimizes an explicit reward-weighted surrogate score on a detached batch, establishing the empirical optimization target of SLPO. Proof sketch. After sampling,(ํ ํ , ํด ํ )no longer depend on the differentiation variableํ. Differentiating b L surr therefore passes only through the scored factors e โ ํ (ํ ํ | ํฅ) , which is precisely the implemented backward pass.โก Proposition 3 (Gradient approximation under score alignment). Letํ ํ (ํ | ํฅ)denote a behavior trajectory law that admits a differentiable score, and suppose the expected-reward gradient admits the score representation ํ ํฝ = ํผ ํโผํ ํ [ ํด(ํ)โ ํ logํ ํ (ํ | ํฅ) ] for a square-integrable advantage ํด(ํ). Define the corresponding population surrogate direction ํ surr = ํผ ํโผํ ํ h ํด(ํ)โ ํ e โ ํ (ํ | ํฅ) i . If ํผ ํโผํ ํ โ ํ e โ ํ (ํ | ํฅ)โโ ํ logํ ํ (ํ | ํฅ) 2 2 โค ํ 2 score , then โฅ ํ surr โ ํ ํฝ โฅ 2 โค โ๏ธ ํผ ํ ํ [ํด(ํ) 2 ] ํ score . Proof sketch. Write the gradient difference as the expectation of the advantage multiplied by the score mismatch. CauchyโSchwarz gives the stated bound.โก Proposition 4 (Local expected-reward ascent). Supposeํฝhas anํฟ-Lipschitz gradient on a neighborhood ofํ, and let ํ surr be an ascent direction obtained from the population surrogate objective. If โฅ ํ surr โโ ํ ํฝ(ํ) โฅ 2 โค ํฟ, Then for any ํ > 0 such that the segmentํ+ ํ ํํ surr : ํ โ [0, 1] lies in this neighborhood, ํฝ(ํ+ ํํ surr ) โฅ ํฝ(ํ)+ ํ โฅํ surr โฅ 2 2 โ ํฟโฅํ surr โฅ 2 โ ํฟํ 2 2 โฅํ surr โฅ 2 2 . Consequently, if โฅํ surr โฅ 2 > ํฟ and 0 < ํ < 2(โฅํ surr โฅ 2 โ ํฟ) ํฟโฅํ surr โฅ 2 , the surrogate step strictly increases the expected reward ํฝ. Proof sketch. ํฟ-smoothness gives ํฝ(ํ+ ํํ surr ) โฅ ํฝ(ํ)+ ํโจโ ํ ํฝ(ํ), ํ surr โฉโ ํฟํ 2 2 โฅํ surr โฅ 2 2 . Writingโ ํ ํฝ(ํ)= ํ surr + ํ with โฅํโฅ 2 โค ํฟ and applying CauchyโSchwarz yields the result.โก 16 SLPO: Scaling Latent Reasoning via a Surrogate Policy Role of MC-dropout. MC-dropout supplies the exploratory behavior trajectories, while a finite-ํพmoment- matched Gaussian score turns those trajectories into a differentiable optimization interface. Largerํพ reduces Monte Carlo error in the estimated moments, and Prop. 3 shows how score alignment controls the population-gradient approximation. Prop. 4 then converts this approximation into an explicit condition for expected-reward improvement. Proposition 5 (Variance-floor regularity). For fixed โ and a variance floor ํ > 0, sup ํ, ํ 2 โฅํ log eํ(โ; ํ, ํ 2 )=โ ํ 2 log(2ํํ), attained atํ= โandํ 2 = ํ. For every fixedํ 2 โฅ ํ,ํ= โis the unique maximizing mean. The floor is essential for the finite optimum: over ํ 2 > 0, sup ํ,ํ 2 >0 log eํ(โ; ํ, ํ 2 )=+โ, because the supremum diverges as ํ 2 โ 0 + along ํ= โ. Proof sketch. Completing the square in the isotropic Gaussian density shows that, for fixedํ 2 , the maximum overํis attained uniquely atํ= โand equalsโ ํ 2 log(2ํํ 2 ). The variance floor maximizes this expression atํ 2 = ํ, giving the finite joint optimum above. Removing the floor and sendingํ 2 โ0 + alongํ= โyields the stated divergence.โก Implications. Prop. 1 identifies the mean-matching force in the latent update when variance is held fixed. Prop. 2 characterizes the implemented step as gradient descent on a detached, reward-weighted surrogate score. Prop. 3 links score alignment to population-gradient approximation, and Prop. 4 gives a sufficient local condition for expected-reward improvement. RLOO and GRPO instantiate the stop-gradient advantage ํด ํ within this shared interface. Prop. 5 establishes the finite optimum induced by the variance floor ํ. A.6 Stopping-gate Inference Alg. 1 gives the inference procedure after the stopping-gate cold start and SLPO. Latent reasoning and answer decoding both run with dropout disabled, and the stopping gate uses deterministic thresholding. Unless otherwise noted, ํ max = 12 matches the cold-start budget in App. A.2. Algorithm 1 Adaptive latent inference with stopping gate Require: Input ํฅ; latent reasoner ํ ํ with stopping head ํ ํ ; maximum budget ํ max ; stop threshold ํ 1: Disable dropout for latent updates and answer decoding 2: Initialize latent prefix ฬ โ <1 โโ 3: for ํก= 1 to ํ max do 4: ฬ โ ํก โ ํ ํ (ํฅ, ฬ โ <ํก ) 5: ํ ํก โ ํ(ํ ํ ( ฬ โ ํก )) 6:if ํ ํก โฅ ํ then 7:break 8:end if 9: end for 10: Let ํ โ โ ํก be the realized stopping step 11: Decode final answer หํโผ ํ ํ (ยท | ํฅ, ฬ โ 1:ํ โ ) 12: return หํ We sweep candidate thresholds0.5,0.6,0.7,0.8,0.9on the validation split and select the threshold with the highest validation accuracy. When a single default threshold is required, we use 0.5. 17 SLPO: Scaling Latent Reasoning via a Surrogate Policy A.7 Inference and Evaluation Accuracy and Pass@ํestimation. Acc uses a single deterministic forward pass per problem: latent reasoning and answer decoding both run with dropout disabled, following the stopping-gate inference path in Sec. 4.3. Forํ >1, one Pass@ํevaluation drawsํindependent stochastic latent rollouts per problem with MC-dropout (ํ=0.1); a problem counts as solved if at least one trajectory yields the correct parsed answer. Because dropout seeds affect the sampled trajectories, we repeat this evaluationํ =3 times with independent seeds and report the mean Pass@ํ over the three runs in Tab. 1. A.8 Baseline Configurations We use the default inference configuration of each latent baseline whenever possible. Ungated COCONUT and CODI use a fixed budget of six latent thoughts, matching the original papers;+SLPOuses the same backbones with maximum budgetํ max =12 and reports the realized stopping time as #L (App. A.2โA.6). CoLaR uses thinking speed 2รwith a maximum latent budget of 64 steps. ReGuLaR follows the released evaluation protocol in Wang et al.[9]. DART Acc and #L are taken from Jiang et al.[18], with #L equal to their fixed Silent Thought countํถ=20 (non-autoregressive; all ST tokens are processed in one forward pass). Latent-SFT Acc and #L are taken from theํ=2 Llama-3.2-1B setting in Deng et al.[19](GSM8K/GSM-Hard/MultiArith; averages exclude SVAMP). For CoT-SFT and iCoT, #L is measured on the generated reasoning chain before answer prediction. For latent methods without the stopping gate, #L is the fixed latent budget used at inference. A.9 Soft Latent Inference Transfer This appendix supplements Sec. 7.3 with the setup behind Tabs. 3 and 4. We keep the soft-token inference mechanism unchanged [25], apply SLPO on Llama3.2โ1B and Llama3.2โ3B, and compare against CoT, vocabulary-routed GRPO, and LEPO [23] under the same stack. This transfer setting uses SLPO without the stopping-gate cold start. Soft-token interface. Each latent step is a probability-weighted embedding rather than a backbone hidden state. The model maps logits โ ํก to ํ ํก = softmax(โ ํก /ํ) and forms ํง ํก = ํ โค ํก ํธ. Answer generation after the latent block still uses the ordinary token likelihood. SLPO objective. We apply the surrogate in Sec. 4.2 by identifying ฬ โ ํ,ํก โก ํง ํ,ํก and scoring transitions in embedding space. The isotropic Gaussian moments and likelihood follow Sec. 4.2, with variance floor ํ=10 โ6 . On the optimized latent prefix, the surrogate termโ b ํด ํ log eํ ํ (ํง ํ,ํก | ํฅ ํ , ํง ํ,<ํก ) replaces the soft-token cross-entropy; subsequent answer tokens retain the standard token log-probability objective. Training and evaluation. Rollouts use a latent block of length 32. We score only an early prefix with the surrogateโ8 steps by default, with a 16-step variant also evaluatedโusingํพ=2 MC-dropout samples per step. Stochastic forwards follow App. A.3 (ํ=0.1). Outcome rewards and group-relative advantages follow the GRPO-style soft-token baseline. We evaluate GSM8K and MATH500 at 1B scale, and AIME 2025 and AMC23 at 3B scale (Tabs. 3 and 4); the 3B runs keep the same interface and surrogate settings, changing only the backbone scale. Other optimization details follow the soft-token baseline recipe shared with LEPO and vocabulary-routed GRPO. B Policy-Optimization Algorithm Comparison This appendix reports the full Pass@ํvalues underlying Fig. 4. We keep the SLPO surrogate, rollout budget, and latent backbones fixed, and swap only the outcome-reward policy-optimization algorithm between RLOO and GRPO. 18 SLPO: Scaling Latent Reasoning via a Surrogate Policy Table 6: Pass@ํ under SLPO with RLOO versus GRPO on COCONUT and CODI. COCONUTCODI DatasetAlgorithm Pass@1 Pass@8 Pass@16 Pass@1 Pass@8 Pass@16 MultiArith RLOO83.10 91.38 92.59 90.52 97.24 97.76 GRPO82.41 90.34 92.24 90.69 96.55 97.59 GSM-Hard RLOO7.66 10.85 12.529.71 12.06 13.05 GRPO7.89 10.47 12.149.86 12.59 12.90 GSM8K RLOO35.63 49.13 51.55 42.76 54.13 56.71 GRPO35.25 47.61 50.72 44.35 54.06 56.18 C Rollout Hyperparameter Sweeps This appendix reports the full Pass@2 values underlying Fig. 3. We vary one rollout hyperparameter over 2, 4, 8 while holding the other fixed at ํพ= 4 or ํบ= 4. a GSM-Hardb MultiArith Figure 8: Full Pass@2 sweeps forํพ(withํบ=4 fixed) andํบ(withํพ=4 fixed). The larger marker at 4 marks the value used in that one-factor sweep. D Latent Geometry Analysis This appendix supplements Sec. 7.4 with the metric definitions, aggregation rules, and figure construction used in the latent-geometry analysis. D.1 Sampling Protocol We compare pre- and post-SLPO checkpoints on the same evaluation instances. For each problemํฅ ํ , we drawํstochastic latent trajectories with MC-dropout at rateํ=0.1, using the same rollout mechanism as SLPO training (Sec. 4.2). Each trajectory is โ (ํ) ํ,1:ํ = โ (ํ) ํ,1 , . . . , โ (ํ) ํ,ํ , ํ= 1, . . . , ํ, with fixed latent budgetํ=6. Unless stated otherwise, we evaluate the first 32 problems per dataset, use ํ=16 dropout samples per problem, and report metrics on COCONUT and CODI with GPT-2 or Llama backbones. 19 SLPO: Scaling Latent Reasoning via a Surrogate Policy D.2 Step Representations Dropout sampling induces local stochasticity around each latent step. To obtain a representative latent vector for step ํก, we aggregate sampled states across trajectories: ํ ํ,ํก = 1 ํ ํ โ๏ธ ํ=1 โ (ํ) ํ,ํก .(1) The sequence (ํ ํ,1 , . . . , ํ ํ,ํ ) is the step-wise representation used by both metrics below. D.3 Inter-Step Separation Inter-step separation measures how far successive step representations move apart along this trajectory. We use cosine distance dcos(ํข, ํฃ)= 1โ ํข โค ํฃ โฅํขโฅ 2 โฅํฃโฅ 2 ,(2) and define the per-problem inter-step score as the mean over consecutive step representations: ํท inter ํ = 1 ํโ 1 ํโ1 โ๏ธ ํก=1 dcos(ํ ํ,ํก , ํ ํ,ํก+1 ).(3) Largerํท inter ํ means successive latent steps occupy more separated positions in hidden space. For a datasetS, we average over problems, ฬ ํท inter = 1 |S| โ๏ธ ํโS ํท inter ํ , and report the relative change from pre- to post-SLPO, ฮ inter = 100ยท ฬ ํท inter post โ ฬ ํท inter pre ฬ ํท inter pre .(4) Fig. a plotsฮ inter for COCONUT across GSM8K, GSM-Hard, MultiArith, and SVAMP. D.4 Prefix Effective Rank To characterize how the latent prefix distributes across the hidden subspace asํกgrows, we track how many independent directions the latent prefixํ 1 , . . . , ํ ํก spans asํกgrows. For prefix lengthํก โฅ2, stack the firstํก step representations into a matrix ํถ ํ,ํก = ๏ฃฎ ๏ฃฏ ๏ฃฏ ๏ฃฏ ๏ฃฏ ๏ฃฏ ๏ฃฐ ํ โค ํ,1 . . . ํ โค ํ,ํก ๏ฃน ๏ฃบ ๏ฃบ ๏ฃบ ๏ฃบ ๏ฃบ ๏ฃป โ โ ํกรํ , row-center across latent steps, ฬ ํถ ํ,ํก = ํถ ํ,ํก โ 1 ฬํ โค ํ,ํก , ฬํ ํ,ํก = 1 ํก ํก โ๏ธ ํ=1 ํ ํ,ํ , and compute the effective rank ํ eff (ํ, ํก)= โฅ ฬ ํถ ํ,ํก โฅ 2 ํน โฅ ฬ ํถ ํ,ํก โฅ 2 2 = ร min(ํก,ํ) ํ=1 ํ ํ ฬ ํถ ํ,ํก 2 ํ 1 ฬ ํถ ํ,ํก 2 ,(5) 20 SLPO: Scaling Latent Reasoning via a Surrogate Policy 23456 1 1.5 2 2.5 3 3.5 23456 1 1.5 2 2.5 3 3.5 4 23456 1 1.5 2 2.5 3 3.5 4 23456 0.95 1 1.05 1.1 1.15 1.2 1.25 1.3 1.35 23456 0.95 1 1.05 1.1 1.15 1.2 1.25 1.3 1.35 1.4 23456 0.95 1 1.05 1.1 1.15 1.2 1.25 1.3 1.35 1.4 BeforeAfter Latent stepLatent stepLatent step Effective rank Effective rank GPT2 ยท GSM8KGPT2 ยท MultiArithGPT2 ยท SVAMP LLaMA ยท GSM8KLLaMA ยท MultiArithLLaMA ยท SVAMP Figure 9: Full prefix effective rank curves pre- and post-SLPO for COCONUT. The plot shows equation 5 at prefix length ํก, averaged over the first 32 problems in each dataset. Curves are grouped by backbone and dataset. whereํ ํ (ยท)denotes theํ-th singular value. This is the stable rank of the centered step-representation matrix: it equals 1 when the step representations lie on one line, and grows as the prefix spans more independent directions. At eachํก, we averageํ eff (ํ, ํก)over problems and plot the resulting before/after curves. Theํฅ-axis therefore marks prefix length rather than a single latent state; because adding step representations also raises the maximum attainable rank, the upward slope alongํกpartly reflects accumulation. The comparison of interest is pre- versus post-SLPO at the same ํก. D.5 Figure Construction Fig. 6 in the main text has two panels. Fig. a reportsฮ inter for COCONUT (equation 3โequation 4). Fig. b reportsํ eff (ํ, ํก)from equation 5 on two representative CODI settings: GPT-2 on SVAMP and Llama on MultiArith. Figs. 9 and 10 extend the rank analysis to all datasetโbackbone combinations for COCONUT and CODI. The main-text conclusion follows the same pre- versus post-SLPO comparison at fixedํก: the post-SLPO curve lies lower while the progression shape across latent steps is preserved. 21 SLPO: Scaling Latent Reasoning via a Surrogate Policy 23456 1 1.2 1.4 1.6 1.8 2 23456 0.9 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 23456 0.95 1 1.05 1.1 1.15 1.2 1.25 1.3 1.35 1.4 23456 1 1.5 2 2.5 3 3.5 23456 1 1.5 2 2.5 3 23456 1 1.2 1.4 1.6 1.8 2 2.2 2.4 BeforeAfter Latent stepLatent stepLatent step Effective rank Effective rank GPT2 ยท GSM8KGPT2 ยท MultiArithGPT2 ยท SVAMP LLaMA ยท GSM8KLLaMA ยท MultiArithLLaMA ยท SVAMP Figure 10: Full prefix effective rank curves pre- and post-SLPO for CODI. The plot shows equation 5 at prefix lengthํก, averaged over the first 32 problems in each dataset. Curves are grouped by backbone and dataset; the main-text rank panels (Fig. 6) show two representative cases from this figure. 22