Paper deep dive
Token-Disentangled Latent Test-Time Scaling for Vision-Language Reasoning
Hao-Xuan Ma, Yihao Liu, Yutao Sun, Yanting Miao, Mengyu Zhou, YiCheng Xiao, Long Chen, Zhenguo Li, Han-Jia Ye, Xiaoxi Jiang, Guanjun Jiang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Latent test-time scaling improves reasoning by refining hidden states during inference, but existing methods typically apply a single scalar reward to all editable latent tokens. For multimodal large language models, this global update ignores that generated tokens play different roles: some are sensitive to visual evidence, while others correspond to uncertain reasoning decisions. We present Token-Disentangled Latent Test-Time Scaling, an inference-time framework that makes latent refinement token-role-aware. Starting from an initial generated trajectory, we optimize a short hidden-state prefix while routing perception-side visual feedback to image-sensitive tokens and reasoning feedback to high-entropy tokens. Tokens selected by neither route are constrained by an anchor regularizer. Across both perception and reasoning benchmarks on Qwen2.5-VL-7B and InternVL3.5-8B, our method lifts macro accuracy over CoT by +2.57 and +1.51 respectively, and outperforms strong output-space test-time scaling baselines under matched decoded-candidate budgets. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.35228v1
- Canonical: https://arxiv.org/abs/2609.35228v1
Trouble viewing inline? Open PDF directly →
Full Text
67,931 characters extracted from source content.
Expand or collapse full text
00footnotetext: ‡ contributors. † authors. Token-Disentangled Latent Test-Time Scaling for Vision-Language Reasoning Hao-Xuan Ma‡ Affiliation: Qwen Business Unit of Alibaba Affiliation: School of Artificial Intelligence, Nanjing University Affiliation: National Key Laboratory for Novel Software Technology, Nanjing University Yihao Liu‡ Affiliation: Qwen Business Unit of Alibaba Yutao Sun‡ Affiliation: Qwen Business Unit of Alibaba Affiliation: Zhejiang University Yanting Miao‡ Affiliation: Qwen Business Unit of Alibaba Affiliation: University of Waterloo Mengyu Zhou† Affiliation: Qwen Business Unit of Alibaba YiCheng Xiao Affiliation: Chinese Academy of Sciences Long Chen Affiliation: The Hong Kong University of Science and Technology Zhenguo Li Affiliation: The Hong Kong University of Science and Technology Affiliation: Frontier Robotics Han-Jia Ye† Affiliation: School of Artificial Intelligence, Nanjing University Affiliation: National Key Laboratory for Novel Software Technology, Nanjing University Xiaoxi Jiang Affiliation: Qwen Business Unit of Alibaba Guanjun Jiang Corresponding author: yehj@lamda.nju.edu.cn; zhoumengyu.zmy@alibaba-inc.com Affiliation: Qwen Business Unit of Alibaba Abstract Latent test-time scaling improves reasoning by refining hidden states during inference, but existing methods typically apply a single scalar reward to all editable latent tokens. For multimodal large language models, this global update ignores that generated tokens play different roles: some are sensitive to visual evidence, while others correspond to uncertain reasoning decisions. We present Token-Disentangled Latent Test-Time Scaling, an inference-time framework that makes latent refinement token-role-aware. Starting from an initial generated trajectory, we optimize a short hidden-state prefix while routing perception-side visual feedback to image-sensitive tokens and reasoning feedback to high-entropy tokens. Tokens selected by neither route are constrained by an anchor regularizer. Across both perception and reasoning benchmarks on Qwen2.5-VL-7B and InternVL3.5-8B, our method lifts macro accuracy over CoT by +2.57+2.57 and +1.51+1.51 respectively, and outperforms strong output-space test-time scaling baselines under matched decoded-candidate budgets. Code is available at https://github.com/Qwen-Applications/TD-LTTS. Figure 1: From global latent refinement to token-disentangled latent refinement. Our method augments latent test-time scaling with token-role-aware routing, separate reasoning and visual rewards, and anchor regularization, improving visual engagement, answer discrimination, and consistency across multimodal reasoning benchmarks. 1 Introduction Multimodal large language models (MLLMs) have recently achieved impressive progress in integrating visual and textual information for complex reasoning tasks (Shao et al., 2024; Deng et al., 2025; Wen et al., 2025). Despite this progress, further improving these capabilities through additional training remains costly, motivating test-time scaling methods that keep the model frozen while allocating extra computation to the same input at inference time (Brown et al., 2024; Geiping et al., 2026). In multimodal reasoning, however, extra inference-time computation is useful only when it targets the source of the model’s error (Liu et al., 2025a). Such errors can arise from distinct bottlenecks: a model may generate a coherent answer while relying on insufficient visual evidence, or it may capture the relevant visual cues but still fail to discriminate among competing answers. Recent latent test-time scaling methods move inference-time computation into hidden states, refining internal representations rather than updating model parameters (Li et al., 2025a; Zhang et al., 2025a). This offers a natural way to reuse existing MLLMs, since the search operates over instance-specific hidden states while leaving the model frozen. However, existing latent refinement objectives typically optimize a short sequence of editable latent tokens with a single global scalar reward, as illustrated in Figure 1. This global feedback is too coarse for multimodal reasoning, where different tokens can play different roles: some are closely tied to visual evidence, while others support reasoning and answer discrimination (Lu et al., 2026; Huang et al., 2025). As a result, tokens with different roles may receive the same credit signal even when they contribute to the error in different ways. More concretely, a candidate-level reward cannot determine whether a failed continuation stems from insufficient visual grounding or flawed answer discrimination, which can assign refinement pressure to latent positions unrelated to the actual failure mode. These observations suggest that effective latent test-time scaling for MLLMs should refine both visual-evidence use and reasoning states. To this end, we propose Token-Disentangled Latent Test-Time Scaling, an inference-time framework that makes latent refinement token-role-aware. Our method retains the standard latent test-time optimization: an MLLM first generates an initial response, a short slice of generated hidden states is selected as editable latent variables, and this slice is refined by decoding and scoring candidate continuations. Instead of broadcasting one global reward to all editable tokens, we decompose the optimization signal into visual-evidence and reasoning components and route them to different latent positions. Visual tokens are selected by their sensitivity to image perturbations, while reasoning tokens are selected by output uncertainty. This enables visual and reasoning feedback to update different parts of the latent sequence. We evaluate our method on six multimodal reasoning benchmarks spanning both perception-heavy and reasoning-heavy settings, using Qwen2.5-VL-7B and InternVL3.5-8B as frozen backbones. Across these benchmarks, our method consistently improves the initial CoT rollout, yielding macro-average gains of +2.57 on Qwen2.5-VL-7B and +1.51 on InternVL3.5-8B. Under the same decoded-candidate budget, our method also outperforms strong output-space test-time scaling baselines on both backbones, suggesting that targeted latent refinement can improve the model’s internal computation rather than merely selecting among sampled answers. Further analyses show that both visual-token routing and reasoning-token routing contribute to the final improvement. Our contributions are threefold: • We identify a multimodal limitation of existing latent test-time scaling: candidate-level scalar rewards cannot distinguish between visual-evidence and reasoning roles within the editable latent sequence. • We introduce token-disentangled latent refinement, which routes visual engagement feedback to image-sensitive tokens and reasoning feedback to uncertain reasoning tokens. • We demonstrate consistent gains over CoT across six multimodal reasoning benchmarks and two frozen MLLM backbones, showing the effectiveness of role-aware latent refinement. 2 Related Work 2.1 Latent Test-Time Scaling Test-time scaling improves reasoning by spending extra computation during inference rather than increasing model size. Early work mainly scaled in output space through repeated sampling, search, or self-correction (Wang et al., 2022; Brown et al., 2024; Welleck et al., 2022), with later studies analyzing compute-optimal allocation (Snell et al., 2024). A parallel line moves this computation into continuous latent states (Geiping et al., 2026; You et al., 2025; Muennighoff et al., 2025): Coconut (Hao et al., 2024) uses continuous thoughts as reusable reasoning states, while SoftCoT and SoftCoT++ introduce soft latent reasoning and test-time latent exploration (Xu et al., 2025b; Xu et al., 2025a). More recent methods refine latent states through search or policy optimization, including LatentSeek (Li et al., 2025a); multimodal variants such as DMLR (Liu et al., 2025a) and VaLR (Jeon et al., 2026) further emphasize preserving visual information during latent computation (Ahmadpour et al., 2025; Zhang et al., 2025b; Kaya et al., 2025). These studies show the promise of latent refinement, but most apply sequence-level, step-level, or globally coupled objectives. Our work instead routes perception and reasoning feedback to different latent tokens. 2.2 Token-Level Credit Assignment in Multimodal Reasoning Reliable multimodal reasoning requires more than producing a plausible final answer: intermediate decisions must remain grounded in the relevant visual evidence. Prior studies show that stronger language reasoning does not eliminate perception bottlenecks, and that only a subset of visual tokens is often decisive for downstream prediction (Xiong et al., 2025; Tong et al., 2024; Jiang et al., 2025). This issue becomes more pronounced in generated reasoning trajectories, where longer chains can dilute visual attention and amplify hallucination (Liu et al., 2025b). These findings motivate token-level credit assignment for multimodal reasoning. Perception-aware optimization and token analyses identify sparse but pivotal visual dependencies (Huang et al., 2025), while ToR (Lu et al., 2026) and PRCO (Miao et al., 2026) explicitly separate perception- and reasoning-related roles during training. Our work brings this role distinction into latent test-time scaling: instead of assigning one candidate-level reward to all editable hidden states, we route perception-side and reasoning-side feedback to different latent tokens of a frozen MLLM during inference. 3 Preliminaries Problem Setup. We first formulate latent test-time scaling in the conventional text-only setting. Given a textual context c, a language model with fixed parameters θ generates a response y=(y1,…,yT)y=(y_1,…,y_T) with hidden states h1,…,hTh_1,…,h_T. At each position, the LM head maps the hidden state to the next-token distribution pθ(yt∣c,y<t)p_θ(y_t c,y_<t). Latent test-time scaling treats a short prefix of these pre-LM-head hidden states as editable latent variables, z(0)=[h1,…,hL],z(k)=zi(k)i=1L,z^(0)=[h_1,…,h_L], z^(k)=\z_i^(k)\_i=1^L, (1) where L is the number of editable latent tokens and k indexes refinement steps. The model parameters remain frozen; only the instance-specific latent prefix is updated. Given a current latent prefix z(k)z^(k), each editable vector induces a local LM-head distribution πi(v∣zi(k))=softmax(Wlmzi(k))v. _i(v z_i^(k))=softmax(W_lmz_i^(k))_v. (2) Decoding from these distributions yields a latent-induced prefix y^1:L y_1:L, after which the frozen model continues generation to produce a candidate response x. We write the induced candidate distribution as pθ(x∣z(k),c)p_θ(x z^(k),c) and the candidate set at step k as (k)=x1(k),…,xM(k).X^(k)=\x^(k)_1,…,x^(k)_M\. (3) A verifier or rule-based reward then assigns a scalar score R(x,c)R(x,c) to each decoded candidate. Prior latent search methods use this scalar to update the entire latent prefix with an objective of the form ℒ=−R(x,c)∑i=1Llogπi(y^i∣zi).L=-R(x,c) _i=1^L _i( y_i z_i). (4) This formulation is natural for unimodal reasoning: the reward is defined at the candidate level, and all editable latent positions are optimized toward the same decoded trajectory-level signal. Modality Gap in Vision-Language Models. For vision-language reasoning, the context becomes ξ=(I,q)ξ=(I,q) with image I and question q. The generated trajectory is now supported by mixed evidence: some latent tokens are tied to visual input—their likelihoods shift under image corruption, indicating that they encode objects, attributes, counts, or spatial relations—while others correspond to uncertain answer comparisons or reasoning transitions, identifiable by high entropy in their LM-head distributions. A single scalar reward can score answer quality or visual engagement, but it cannot tell which latent positions should absorb perception feedback and which should absorb reasoning feedback; broadcasting one signal to both groups risks mixing the two updates and worsening the modality gap. We therefore treat the latent prefix as a mixture of roles and perform token-level credit assignment. We use two reward signals—a reasoning reward RreaR_rea for answer quality and a visual reward RvisR_vis for perception-side engagement—and route them to two disjoint token groups: high-entropy tokens receive reasoning feedback, while image-sensitive tokens receive visual feedback. This sets up the token-disentangled latent update in Section 4. 4 Method We present Token-Disentangled Latent Test-Time Scaling, an inference-time adaptation method for vision-language reasoning. Building on the latent test-time scaling formulation in Section 3, our method changes how reward feedback is assigned to editable latent tokens. Instead of applying one scalar reward to the entire latent prefix, we separate visual feedback from reasoning feedback and route each signal to the tokens most suited for it. Figure 2: Framework of Token-Disentangled Latent Test-Time Scaling. A frozen VLM first produces an initial chain-of-thought response and its hidden-state trajectory. We refine an early latent prefix from this trajectory at test time. Visual tokens (identified by high image sensitivity) receive visual rewards, while reasoning tokens (identified by high entropy) receive reasoning rewards. Token routing prevents visual and reasoning feedback from collapsing into a single global latent update. 4.1 Token-Routed Latent Refinement Objective The standard latent-search objective is to find an editable prefix z=(z1,…,zL)z=(z_1,…,z_L) of length L that improves the expected quality of decoded continuations: z⋆=argmaxzx∼pθ(x∣z,ξ)[R(x,ξ)].z = _zE_x p_θ(x z,ξ)[R(x,ξ)]. (5) Applying a single reward R to every latent token, however, assumes that all tokens in the slice play the same role. This assumption is especially weak for multimodal reasoning: some latent tokens primarily carry visual evidence, while others encode uncertain answer decisions or reasoning transitions. We therefore decompose the context-conditioned feedback into a visual reward and a reasoning reward: Rvis(x,ξ,y),Rrea(x,ξ),R_vis(x;ξ,y), R_rea(x;ξ), (6) where ξ=(I,q)ξ=(I,q) is the multimodal context and y is the initial rollout used as the per-instance visual baseline. We write these rewards as Rvis(x)R_vis(x) and Rrea(x)R_rea(x) when the context is clear, and compute two disjoint token masks: wivis,wirea∈0,1,wiviswirea=0.w_i^vis,w_i^rea∈\0,1\, w_i^visw_i^rea=0. (7) At update k, the feedback candidate is selected from a decoded candidate set (k)X^(k) by the reasoning reward: xk⋆=argmaxx∈(k)Rrea(x).x_k = _x ^(k)R_rea(x). (8) For k≥1k≥ 1, (k)X^(k) is decoded from the current prefix z(k)z^(k) after the k-th latent update; for k=0k=0 we set (0)=yX^(0)=\y\, so the first update is driven by the rollout’s own reward without any extra decoding at z(0)z^(0). The rollout y is also retained throughout as the per-instance reference for relative prediction margins. The selected xk⋆x_k is then used for both reward channels: RreaR_rea keeps answer quality as the selection criterion, while RvisR_vis measures whether that candidate improves visual engagement relative to the initial rollout y. During decoding, we also retain the editable-token prefix that produced each candidate. Let y^k,i⋆ y_k,i denote the token at editable position i for the selected candidate xk⋆x_k , with its log probability evaluated under the current latent distribution πi(⋅∣zi(k)) _i(· z_i^(k)). We first define the routed log-probability sums Svis(k) S_vis^(k) =∑iwivislogπi(y^k,i⋆∣zi(k)), = _iw_i^vis _i( y_k,i z_i^(k)), (9) Srea(k) S_rea^(k) =∑iwirealogπi(y^k,i⋆∣zi(k)). = _iw_i^rea _i( y_k,i z_i^(k)). The token-routed policy loss is ℒpolicy= _policy= −λpgvisRvis(xk⋆)Svis(k) - _pg^visR_vis(x_k )S_vis^(k) (10) −λpgreaRrea(xk⋆)Srea(k). - _pg^reaR_rea(x_k )S_rea^(k). Visual rewards therefore update image-sensitive tokens, while reasoning rewards update high-entropy tokens. We additionally regularize the update with a reasoning-weighted entropy term and an anchor penalty: ℒent=λent∑iwireaHi∑iwirea+ϵ,L_ent= _ent _iw_i^reaH_i _iw_i^rea+ε, (11) ℒanchor=λanchor‖z(k)−z(0)‖22,L_anchor= _anchor\|z^(k)-z^(0)\|_2^2, (12) where Hi=−∑vπi(v∣zi(k))logπi(v∣zi(k)).H_i=- _v _i(v z_i^(k)) _i(v z_i^(k)). (13) The final objective minimized at each refinement step is ℒ=ℒpolicy+ℒent+ℒanchor.L=L_policy+L_ent+L_anchor. (14) 4.2 Visual and Reasoning Rewards Visual reward. The visual reward uses image-token attention as a lightweight visual-engagement proxy. Let IP_I be the image-token positions. For a generated trajectory x=(u1,…,uN)x=(u_1,…,u_N), define ej(x)=1M∑m=1M∑p∈IAm,j,p(ℓ)(I,x),e_j(x)= 1M _m=1^M _p _IA^( )_m,j,p(I,x), (15) where A(ℓ)A^( ) is the layer-ℓ attention tensor and M is the number of heads. With N′=min(N,Nmax)N = (N,N_ ), the trajectory-level engagement score is E(x)=1N′∑j=1N′ej(x).E(x)= 1N _j=1^N e_j(x). (16) Using the initial rollout y as the per-instance baseline, we define Rvis(x)=tanh(E(x)−E(y)τvis).R_vis(x)= ( E(x)-E(y) _vis ). (17) This relative reward also provides visual feedback for the initial candidate set (0)X^(0); baseline details are in Appendix A. The bounded tanh caps the update magnitude so a single instance with degenerate E(y)E(y) cannot dominate the latent step. Reasoning reward. For a decoded candidate x, let c(x)c(x) be its normalized short prediction. We score candidate quality by subtracting a text-only prior from the image-conditioned likelihood: S(c)=logpθ(c∣I,q)−λplogpθ(c∣q).S(c)= p_θ(c I,q)- _p p_θ(c q). (18) We combine a centered bounded score with a relative margin over competing predictions: Rrea(x)=λsr¯(x)+λmarrmar(c(x)).R_rea(x)= _s r(x)+ _marr_mar(c(x)). (19) Here r¯(x) r(x) is derived from the score S(c(x))S(c(x)), while rmarr_mar compares c(x)c(x) against alternatives from the same refinement step. We give the full bounded-score and margin definitions in Appendix B.1. 4.3 Token Routing Visual tokens. We identify visual tokens by measuring how sensitive their likelihoods are to image corruption. Under the original image, the teacher-forced log probability of the generated token corresponding to editable position i is ℓi=logpθ(yi∣I,q,y<i). _i= p_θ(y_i I,q,y_<i). (20) After corrupting the image to obtain I~ I, we recompute ℓ~i=logpθ(yi∣I~,q,y<i). _i= p_θ(y_i I,q,y_<i). (21) The image-sensitivity score is vi=ϕ(ℓ~i−ℓi),ϕ(x)=exp(x)−x−1.v_i=φ( _i- _i), φ(x)= (x)-x-1. (22) Tokens with the highest image-sensitivity scores are routed to the visual branch: wivis=[i∈TopKρv(vjj=1L)].w_i^vis=1 [i _ _v (\v_j\_j=1^L ) ]. (23) Reasoning tokens. Reasoning tokens are identified by uncertainty under the current latent LM-head distribution. We standardize the entropy values within the editable slice, H¯i=Hi−μHσH+ϵ, H_i= H_i- _H _H+ε, (24) where μH _H and σH _H are the mean and standard deviation of Hjj=1L\H_j\_j=1^L. Let visN_vis denote the non-visual tokens, and let H¯vis H_N_vis be the corresponding standardized entropy scores. We then select reasoning tokens by wirea=[i∈TopKρr(H¯vis)].w_i^rea=1\! [i _ _r( H_N_vis) ]. (25) This strict routing keeps the visual and reasoning branches disjoint. Unselected tokens remain anchored to the original trajectory and receive no direct policy reward. 4.4 Inference Procedure Algorithm 1 summarizes our inference procedure. Starting from the model’s own chain-of-thought rollout, we refine an early latent prefix and use decoded candidates as reward feedback for subsequent updates. Candidate sets combine one greedy continuation with stochastic continuations from lightly perturbed latent prefixes, diversifying search while keeping the VLM frozen. Algorithm 1 Token-Disentangled Latent Test-Time Scaling 0: Multimodal context ξ=(I,q)ξ=(I,q), frozen VLM pθp_θ, update steps K 1: Generate initial rollout y and hidden states h1:Th_1:T 2: Initialize editable prefix z(0)=[h1,…,hL]z^(0)=[h_1,…,h_L] 3: Compute image-sensitivity scores vii=1L\v_i\_i=1^L 4: Set (0)=yX^(0)\!=\!\y\, x0⋆=yx_0 \!=\!y; compute Rrea(y)R_rea(y) (with Rvis(y)=0R_vis(y)\!=\!0 by construction) 5: for k=0,…,K−1k=0,…,K-1 do 6: Compute entropy scores Hii=1L\H_i\_i=1^L from z(k)z^(k) 7: Select wvisw^vis by image-sensitivity TopKρvTopK_ _v 8: Select wreaw^rea by standardized entropy TopKρrTopK_ _r among non-visual tokens 9: Update z(k+1)z^(k+1) using Rrea(xk⋆)R_rea(x_k ) and Rvis(xk⋆)R_vis(x_k ) 10: Decode (k+1)X^(k+1) from z(k+1)z^(k+1) and noisy variants of z(k+1)z^(k+1) 11: Select xk+1⋆=argmaxx∈(k+1)Rrea(x)x_k+1 = _x ^(k+1)R_rea(x) 12: Compute Rrea(xk+1⋆)R_rea(x_k+1 ) and Rvis(xk+1⋆)R_vis(x_k+1 ) for the next update 13: end for 14: return final answer aggregated from y∪(1)∪⋯∪(K)\y\ ^(1)∪·s ^(K) 5 Experiment Table 1: Main results on multimodal reasoning benchmarks. Best results in each model block are highlighted in bold, and second-best results are underlined. Method Perception Reasoning MMStar RWQA Hallusion Avg. ScienceQA MathVista LogicVista Avg. Qwen2.5-VL-7B CoT 62.00 63.66 70.56 65.41 89.74 67.80 44.52 67.35 Self-consistency 63.47 65.49 68.77 65.91 89.60 70.90 42.51 67.67 Best-of-N 63.93 66.37 70.56 66.95 90.63 71.60 42.73 68.32 Reward-only 64.67 67.06 70.45 67.39 90.12 70.90 42.95 67.99 LatentSeek(reasoning) 61.60 63.79 69.09 64.83 88.29 66.50 44.07 66.29 LatentSeek(perception) 62.47 62.35 68.24 64.35 90.53 70.20 44.30 68.34 DMLR 60.10 64.17 70.80 65.02 89.14 69.10 43.15 67.13 Ours 64.73 67.45 71.71 67.96 90.73 72.10 46.98 69.94 InternVL3.5-8B CoT 67.13 64.36 66.04 65.84 95.04 69.80 45.19 70.01 Self-consistency 67.13 64.71 66.25 66.03 95.64 70.10 45.90 70.55 Best-of-N 67.80 64.71 66.56 66.36 94.75 72.10 46.98 71.28 Reward-only 68.07 65.10 66.61 66.59 96.98 71.30 46.53 71.60 LatentSeek(reasoning) 67.07 61.31 63.51 63.96 94.65 70.60 45.64 70.30 LatentSeek(perception) 66.20 62.61 64.04 64.28 91.08 72.00 44.97 69.35 DMLR 67.47 63.19 65.65 65.44 95.44 71.80 45.96 71.07 Ours 68.80 65.23 66.46 66.83 97.08 72.30 46.76 72.05 5.1 Experiment Setting Backbones. We evaluate our method on two main open-source MLLM backbones, Qwen2.5-VL-7B-Instruct Bai et al. (2025) and InternVL3.5-8B-Instruct Wang et al. (2025), and further test generality across six additional backbones from different families and post-training recipes in Table 2. In all settings the backbone parameters are frozen, and our method only optimizes instance-specific latent states at test time. Benchmarks. We evaluate on benchmarks covering two capability groups. For perception and visually grounded understanding, we use MMStar Chen et al. (2024), RealWorldQA, and HallusionBench Guan et al. (2024). For reasoning-centric evaluation, we use ScienceQA-IMG Lu et al. (2022), MathVista Lu et al. (2023), and LogicVista Xiao et al. (2024). All methods use the same answer extraction and evaluation protocol. Baselines. We compare against test-time baselines using the same frozen backbone. CoT Wei et al. (2022) uses chain-of-thought prompting. Self-consistency Wang et al. (2022) samples multiple responses and aggregates final answers, while Best-of-N Snell et al. (2024) selects the highest-scoring candidate. Reward-only applies our reasoning reward only for output-space reranking, without latent updates or token routing. LatentSeek Li et al. (2025a) is a prior latent-adaptation baseline; we use its reasoning- and perception-reward variants, denoted LatentSeek(reasoning) and LatentSeek(perception). DMLR Liu et al. (2025a) is a multimodal latent-refinement method that emphasizes preserving visual information during latent computation. Together, these baselines separate our method from output-space scaling and prior latent-space refinement. Implementation Details. Unless otherwise noted, we use K=4K=4 latent refinement steps as the default. To equalize decoding cost, all sampling-based baselines are allocated N=16N=16 candidate continuations, matching the total number of decoded candidates in our method. Our method additionally incurs reward-scoring and image-sensitivity forward passes; full hyperparameters, decoding settings, and per-call budget accounting are deferred to Appendices B, B.2, and C.5. 5.2 Main Results Table 1 reports the main comparison on Qwen2.5-VL-7B and InternVL3.5-8B. Our method achieves the best average on both perception and reasoning groups across both backbones, lifting the Qwen2.5-VL-7B averages by +2.55+2.55/+2.59+2.59 and the InternVL3.5-8B counterparts by +0.99+0.99/+2.04+2.04 over CoT. Because the backbone is frozen, these gains are obtained purely through test-time computation on the latent states and are consistent across both architectures and capability groups, suggesting that the improvement comes from instance-level trajectory refinement rather than any backbone-specific bias. Against output-space baselines under matched decoded-candidate budgets, our method yields stronger group averages on both backbones: on Qwen2.5-VL-7B the reasoning margins over Best-of-N and Reward-only are +1.62+1.62 and +1.95+1.95, and on InternVL3.5-8B they are +0.77+0.77 and +0.45+0.45, with consistent positive gains on perception. Editing the latent that drives subsequent decoding uses the candidate budget more efficiently than sampling-and-reranking, as the refined latent commits the improvement to every later token rather than only the selected candidate. The LatentSeek comparison is sharper: on Qwen2.5-VL-7B both variants fall below CoT on perception (64.8364.83/64.3564.35 vs. 65.4165.41), and the reasoning-reward variant additionally drops reasoning (66.2966.29 vs. 67.3567.35), showing that a uniform rule-based reward can degrade a competent rollout because a reasoning-only signal erodes perception-sensitive tokens and vice versa. Token-routed refinement avoids this regression by dispatching different reward signals to different token roles, retaining the strongest performance on both groups simultaneously. 5.3 Detailed Analysis Table 2: Performance comparison on perception and reasoning benchmarks. ↑ and ↓ indicate performance changes compared with the base model. Base Model Perception Reasoning MMStar RWQA Hallusion Avg. ScienceQA MathVista LogicVista Avg. Qwen2.5-VL-3B (Bai et al., 2025) 59.50 54.07 63.30 58.96 80.76 63.14 40.93 61.61 Ours 62.20 ↑ 2.70 56.13 ↑ 2.06 65.30 ↑ 2.00 61.21 ↑ 2.25 80.91 ↑ 0.15 64.05 ↑ 0.91 41.88 ↑ 0.95 62.28 ↑ 0.67 InternVL3.5-4B (Wang et al., 2025) 69.10 64.87 63.62 65.86 93.75 54.77 41.36 63.29 Ours 70.70 ↑ 1.60 65.13 ↑ 0.26 64.04 ↑ 0.42 66.62 ↑ 0.76 94.86 ↑ 1.11 63.53 ↑ 8.76 43.04 ↑ 1.68 67.14 ↑ 3.85 LLaVA-OV-1.5-8B (An et al., 2025) 62.67 66.80 61.72 63.73 90.68 68.30 43.85 67.61 Ours 65.80 ↑ 3.13 66.67 ↓ 0.13 64.88 ↑ 3.15 65.78 ↑ 2.05 96.43 ↑ 5.75 70.40 ↑ 2.10 46.53 ↑ 2.68 71.12 ↑ 3.51 MiMO-VL-RL-8B (Li et al., 2025b) 48.40 45.36 49.11 47.62 86.71 63.10 37.14 62.32 Ours 51.60 ↑ 3.20 57.39 ↑ 12.03 58.36 ↑ 9.25 55.78 ↑ 8.16 93.85 ↑ 7.14 66.20 ↑ 3.10 48.55 ↑ 11.41 69.53 ↑ 7.22 Qwen3-VL-8B 71.33 69.28 67.19 69.27 93.11 77.10 48.55 72.92 Ours 71.73 ↑ 0.40 68.89 ↓ 0.39 68.45 ↑ 1.26 69.69 ↑ 0.42 94.89 ↑ 1.78 76.90 ↓ 0.20 52.35 ↑ 3.80 74.71 ↑ 1.79 Qwen2.5-VL-32B (Bai et al., 2025) 67.9 67.8 65.3 67.0 93.2 77.4 50.3 73.6 Ours 68.0 ↑ 0.1 69.7 ↑ 1.9 69.7 ↑ 4.4 69.1 ↑ 2.1 93.6 ↑ 0.4 77.8 ↑ 0.4 51.7 ↑ 1.4 74.4 ↑ 0.7 Figure 3: Component ablation. Perception is the average of MMStar and RealWorldQA; reasoning is the average of MathVista and LogicVista (two representative tasks per group, used as a cost-controlled subset of the Table 1 groups). [Ana.1] Component Ablation. We ablate the two reward branches and the token-routing module with other settings fixed; Figure 3 reports two-task representative averages (perception over MMStar+RWQA, reasoning over MathVista+LogicVista) as a cost-controlled subset of the Table 1 groups. Removing the visual reward (w/oRvisw/o \ R_vis) drops both averages from 66.0966.09/59.5459.54 to 64.8864.88/58.1858.18, while removing the reasoning reward (w/oRreaw/o \ R_rea) causes the largest drop (64.1964.19/57.3157.31), identifying bounded-likelihood scoring as the dominant signal and image-token engagement as the necessary corrector for perception-sensitive tokens. Replacing token-aware routing with random routing also degrades both groups (64.7264.72/58.8758.87), confirming that multiple rewards alone are insufficient without role-aware token assignment. Extended sweeps over reward formulations and routing signals appear in Appendices C.3 and C.2; the routing budget ρ sweep and strict-vs-overlap mask comparison show that hard top-K selection is robust within the explored range. [Ana.2] Generality Across Backbones. Table 2 extends the evaluation to six additional backbones covering compact (Qwen2.5-VL-3B, InternVL3.5-4B), 8B (LLaVA-OneVision-1.5-8B, MiMO-VL-RL-8B, Qwen3-VL-8B), and large (Qwen2.5-VL-32B) models from different families and post-training recipes. Our method improves both group averages on all six backbones. The largest improvements appear on the RL-post-trained MiMO-VL-RL-8B, while stronger backbones such as Qwen3-VL-8B show smaller gains as the initial rollout already approaches benchmark headroom; group averages remain positive even when isolated cells drop within the noise band (at most −0.39-0.39). The consistency across model family, scale (3B–32B), and post-training recipe (SFT vs. RL) suggests that the gains arise from instance-level latent refinement rather than any architectural assumption. Figure 4: Effect of the maximum number of latent refinement steps K on the three-benchmark perception and reasoning averages from Table 1. [Ana.3] Test-Time Scaling and Token Budget. We study how performance scales with the maximum number of latent refinement steps K. Figure 4 reports the perception/reasoning averages for K∈1,2,4,8,16K∈\1,2,4,8,16\. Both groups improve monotonically from K=1K=1 to K=4K=4 (perception →67.9665.96\!→\!67.36\!→\!67.96; reasoning →69.9468.34\!→\!69.13\!→\!69.94), and then saturate at K≥8K\!≥\!8 within the noise band (perception 67.8667.86/67.7467.74; reasoning 69.8869.88/69.7869.78, with overlapping confidence intervals across K∈4,8,16K\!∈\!\4,8,16\); the same K=4K=4 optimum appears in the per-benchmark sweep in Appendix C.4. This indicates that a small number of latent updates suffices to correct the initial trajectory and that further refinement yields no additional measurable gain, so we adopt K=4K=4 as the compute-optimal default in the main results. 6 Conclusion We introduced Token-Disentangled Latent Test-Time Scaling, a role-aware latent refinement framework for frozen multimodal large language models. By routing visual feedback to image-sensitive tokens and reasoning feedback to uncertain decision tokens, our method avoids applying a single global reward to all editable latent states. Experiments show consistent improvements over CoT and over test-time scaling baselines, while ablations confirm the importance of both reward decomposition and token-aware routing. Future work can further reduce reward-estimation cost and extend this framework to longer multimodal interactions. Limitations Our method improves frozen MLLMs at test time, but it also introduces additional inference cost because each example requires latent refinement and multiple candidate continuations. The reward signals are proxy objectives rather than guaranteed correctness verifiers, so visual engagement and likelihood-based reasoning scores can be noisy for ambiguous images, underspecified questions, or answers requiring external knowledge; the bounded tanh in RvisR_vis limits but does not eliminate this. Token routing uses a hard top-K selection for branch disjointness; soft or learned routing is left to future work. In addition, our method requires open-weight backbones whose hidden states can be edited and differentiated through, which excludes API-only closed models by construction. Extending the framework to longer multi-turn settings and broader real-world domains remains future work. Acknowledgments This work was supported by the National Key R&D Program of China under Grant No. 2024YFE0202800 and the National Natural Science Foundation of China under Grant Nos. 62522605 and 62376118. Long Chen was additionally supported by the National Natural Science Foundation of China under Grant Nos. 62522216 and 62402408, and the Hong Kong SAR Research Grants Council under Grant Nos. 26208924 and 16219025. References Ahmadpour et al. (2025) Mohammadjavad Ahmadpour, Amirmahdi Meighani, Payam Taebi, Omid Ghahroodi, Amirmohammad Izadi, and Mahdieh Soleymani Baghshah. Limits and gains of test-time scaling in vision-language reasoning. arXiv preprint arXiv:2512.11109, 2025. An et al. (2025) Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, et al. Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661, 2025. Bai et al. (2025) Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. Brown et al. (2024) Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024. Chen et al. (2024) Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? NeurIPS, 37:27056–27087, 2024. Deng et al. (2025) Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement. arXiv preprint arXiv:2503.17352, 2025. Geiping et al. (2026) Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. NeurIPS, 38:41340–41391, 2026. Guan et al. (2024) Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In CVPR, p. 14375–14385, 2024. Hao et al. (2024) Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024. Huang et al. (2025) Siyuan Huang, Xiaoye Qu, Yafu Li, Yun Luo, Zefeng He, Daizong Liu, and Yu Cheng. Spotlight on token perception for multimodal reinforcement learning. arXiv preprint arXiv:2510.09285, 2025. Jeon et al. (2026) Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee, Minsu Cho, and Jinwoo Shin. Vision-aligned latent reasoning for multi-modal large language model. arXiv preprint arXiv:2602.04476, 2026. Jiang et al. (2025) Yutao Jiang, Qiong Wu, Wenhao Lin, Wei Yu, and Yiyi Zhou. What kind of visual tokens do we need? training-free visual token pruning for multi-modal large language models from the perspective of graph. In AAAI, number 4, p. 4075–4083, 2025. Kaya et al. (2025) Mehmet Onurcan Kaya, Desmond Elliott, and Dim P Papadopoulos. Efficient test-time scaling for small vision-language models. arXiv preprint arXiv:2510.03574, 2025. Li et al. (2025a) Hengli Li, Chenxi Li, Tong Wu, Xuekai Zhu, Yuxuan Wang, Zhaoxin Yu, Eric Hanchen Jiang, Song-Chun Zhu, Zixia Jia, Ying Nian Wu, et al. Seek in the dark: Reasoning via test-time instance-level policy gradient in latent space. arXiv preprint arXiv:2505.13308, 2025a. Li et al. (2025b) Jiaze Li, Jingyang Chen, Yuxun Qu, Shijie Xu, Zhenru Lin, Junyou Zhu, Boshen Xu, Wenhui Tan, Pei Fu, Jianzhong Ju, et al. Xiaomi mimo-vl-miloco technical report. arXiv preprint arXiv:2512.17436, 2025b. Liu et al. (2025a) Chengzhi Liu, Yuzhe Yang, Yue Fan, Qingyue Wei, Sheng Liu, and Xin Eric Wang. Reasoning within the mind: Dynamic multimodal interleaving in latent space. arXiv preprint arXiv:2512.12623, 2025a. Liu et al. (2025b) Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025b. Lu et al. (2026) Jinda Lu, Junkang Wu, Jinghan Li, Kexin Huang, Shuo Yang, Guoyin Wang, Jiancan Wu, Xiang Wang, and Xiangnan He. Bridging perception and reasoning: Token reweighting for rlvr in multimodal llms. arXiv preprint arXiv:2603.25077, 2026. Lu et al. (2022) Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 35:2507–2521, 2022. Lu et al. (2023) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. Miao et al. (2026) Ziqi Miao, Haonan Jia, Lijun Li, Chen Qian, Yuan Xiong, Wenting Yan, and Jing Shao. Seeing with you: Perception-reasoning coevolution for multimodal reasoning. arXiv preprint arXiv:2603.28618, 2026. Muennighoff et al. (2025) Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori B Hashimoto. s1: Simple test-time scaling. In EMNLP, p. 20286–20332, 2025. Shao et al. (2024) Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. NeurIPS, 37:8612–8642, 2024. Snell et al. (2024) Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024. Tong et al. (2024) Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, p. 9568–9578, 2024. Wang et al. (2025) Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 35:24824–24837, 2022. Welleck et al. (2022) Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053, 2022. Wen et al. (2025) Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245, 2025. Xiao et al. (2024) Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973, 2024. Xiong et al. (2025) Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. In CVPR, p. 13618–13628, 2025. Xu et al. (2025a) Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. Softcot: Soft chain-of-thought for efficient reasoning with llms. In ACL, p. 23336–23351, 2025a. Xu et al. (2025b) Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. Softcot++: Test-time scaling with soft chain-of-thought reasoning. arXiv preprint arXiv:2505.11484, 2025b. You et al. (2025) Runyang You, Yongqi Li, Meng Liu, Wenjie Wang, Liqiang Nie, and Wenjie Li. Parallel test-time scaling for latent reasoning models. arXiv preprint arXiv:2510.07745, 2025. Zhang et al. (2025a) Guibin Zhang, Fanci Meng, Guancheng Wan, Zherui Li, Kun Wang, Zhenfei Yin, Lei Bai, and Shuicheng Yan. Latentevolve: Self-evolving test-time scaling in latent space. arXiv preprint arXiv:2509.24771, 2025a. Zhang et al. (2025b) Junyu Zhang, Runpei Dong, Han Wang, Xuying Ning, Haoran Geng, Peihao Li, Xialin He, Yutong Bai, Jitendra Malik, Saurabh Gupta, et al. Alphaone: Reasoning models thinking slow and fast at test time. In EMNLP, p. 11340–11365, 2025b. Appendix Overview This appendix is organized as follows: A. We define the initial-rollout visual baseline used by the visual reward. B. We describe implementation details, including the reasoning reward, default configuration, latent editing, and candidate decoding. C. We report supplementary ablation studies covering routing budgets, routing signals, reward components, latent optimization hyperparameters, and matched-budget comparisons. D. We provide qualitative case studies illustrating how visual and reasoning rewards are routed to different latent-token groups, along with a representative failure case. Appendix A Initial-Rollout Visual Baseline The visual reward compares the image-token attention of a decoded candidate against the initial rollout from the same example. Let y be the initial chain-of-thought trajectory and E(⋅)E(·) the trajectory-level visual engagement score defined in Section 4. For a later candidate x, the reward is Rvis(x)=tanh(E(x)−E(y)τvis).R_vis(x)= \! ( E(x)-E(y) _vis ). (26) The initial rollout is therefore the per-instance baseline, so Rvis(y)=0R_vis(y)=0. Concretely, we set (0)=yX^(0)=\y\ so the first latent update is driven by the reasoning reward of y alone, without any extra decoding at z(0)z^(0). From the first update onward, each decoded candidate set (k)X^(k) (k≥1k≥ 1) comes from a perturbed latent prefix, so its candidates have engagement scores that differ from E(y)E(y), and the visual branch receives nonzero relative feedback whenever the selected candidate attends to image tokens more or less strongly than the initial rollout. This signal is used only for image-sensitive latent tokens and is not treated as an answer-correctness verifier. Appendix B Implementation Details B.1 Reasoning Reward Details For a normalized short prediction c, the text-prior-corrected score is S(c)=logpθ(c∣I,q)−λplogpθ(c∣q).S(c)= p_θ(c I,q)- _p p_θ(c q). (27) We map this score to a bounded scalar and center it with a per-step baseline: r(c)=tanh(S(c)/τs),r¯(x)=r(c(x))−bs.r(c)= \! (S(c)/ _s ), r(x)=r(c(x))-b_s. (28) For the relative margin term, let ℬ(c)B(c) denote the competing predictions from the same refinement step, including the initial rollout when available. We compute Δmar(c) _mar(c) =S(c)−maxc′∈ℬ(c)S(c′), =S(c)- _c (c)S(c ), (29) rmar(c) r_mar(c) =tanh(Δmar(c)/τmar). = \! ( _mar(c)/ _mar ). B.2 Default Configuration Unless otherwise specified, all reported results use the configuration below. • Refinement schedule. K=4K=4 refinement steps; editable prefix length L=min(⌊ρT⌋,300)L= ( ρ T ,300) with start position s=0s=0 and ρ=0.5ρ=0.5; Adam optimizer with learning rate 0.050.05. • Token routing. Visual budget ρv=0.4 _v=0.4 on editable tokens by image sensitivity; reasoning budget ρr=0.4 _r=0.4 on non-visual tokens by standardized LM-head entropy; strictly disjoint masks. • Visual reward. τvis=0.2 _vis=0.2, final attention layer, engagement scored over at most 128128 generated tokens. Image sensitivity uses raw-image patch blackening with patch size 1414 and drop probability 0.50.5. • Reasoning reward. Bounded-score weight λs=1.0 _s=1.0, margin weight λmar=0.5 _mar=0.5, text-prior weight λp=1.0 _p=1.0; visual and reasoning policy weights λpgvis=λpgrea=1.0 _pg^vis= _pg^rea=1.0. • Regularization. Anchor weight λanchor=0.05 _anchor=0.05; logit-entropy weight λent=0.01 _ent=0.01. • LatentSeek baseline. A separate configuration closer to its original setting: 1010 latent update steps, ρ=0.2ρ=0.2, learning rate 0.050.05. B.3 Latent Editing and Candidate Decoding The editable states are the last decoder-layer hidden states of the generated response, immediately before the frozen LM head. They are collected from the model’s native generation API with return_dict_in_generate=True and output_hidden_states=True. We do not edit visual-encoder states, model weights, or intermediate decoder layers. The optimized continuous states are not directly inserted into the transformer’s KV cache. At each refinement step, each edited state is first projected through the frozen LM head to obtain a latent-induced prefix distribution. We decode one greedy prefix by taking the argmax token at each editable position, and decode three noisy-prefix variants by adding Gaussian noise with base standard deviation 0.30.3 to the edited states before top-k sampling from the LM-head logits. Each resulting discrete prefix is concatenated with the original multimodal prompt and unedited response prefix before the model continues autoregressive generation with its native generate() function. Every candidate continuation is generated from a fresh generation call. We reset the model’s multimodal generation state before each call and do not reuse the KV cache from the initial rollout or from previous candidates. Multimodal generation inputs such as image tensors, image-grid metadata, image flags, and model-specific token-type ids are copied from the original prompt inputs and extended when needed to match the latent-induced prefix length. Across the default K=4K=4 refinement steps the method decodes 44 candidates per step (one greedy and three noisy); the original rollout is also kept as a reference candidate for scoring and final selection. Short answers are normalized by parsing structured outputs when available (JSON fields such as final answer, final_answer, or answer) and by applying the benchmark adapter’s answer parser for multiple-choice and pairwise-evaluation tasks. The default final aggregation groups candidates by normalized short answer, selects the group with the largest count, and breaks ties by the highest reward within the group; equivalently, the implementation uses the composite score 10⋅count+maxR10·count+ R per normalized-answer group. A reward-weighted log-sum-exp aggregation is also implemented but is not the default. For model-specific hidden-state handling, Qwen-style models use the HuggingFace generation output directly. InternVL models are wrapped with a lightweight generation adapter so that their generate() output exposes the same prompt-plus-generation sequence convention expected by the latent-prefix code. This keeps hidden-state collection, prefix construction, and candidate decoding identical across the supported backbones. Appendix C Additional Ablation Studies All ablations use Qwen2.5-VL-7B as the frozen backbone and the default configuration unless otherwise specified. Results are reported as accuracy (%). We use LogicVista and RealWorldQA as representative reasoning-heavy and perception-heavy benchmarks; the matched-budget comparison (Section C.5) additionally covers MathVista. The baseline rollout values are 44.5244.52 on LogicVista and 63.6663.66 on RealWorldQA, matching the CoT entries in Table 1. Where shown, parenthetical values report the absolute improvement in percentage points over this baseline, computed from unrounded scores prior to rounding, and Macro denotes the two-benchmark average. C.1 Token Routing Budget We sweep the visual and reasoning routing budgets ρv _v and ρr _r and test disjoint vs. overlapping masks and swapped reward-token assignment. Table 3: Diagonal top-k ratio sweep with ρv=ρr _v= _r. ρv=ρr _v= _r LogicVista RealWorldQA 0.100.10 45.10 (+0.58) 66.40 (+2.74) 0.200.20 45.55 (+1.03) 66.80 (+3.14) 0.300.30 46.31 (+1.79) 67.19 (+3.53) 0.400.40 46.98 (+2.46) 67.45 (+3.79) 0.500.50 46.53 (+2.01) 67.06 (+3.40) 0.600.60 45.86 (+1.34) 66.80 (+3.14) 0.800.80 45.41 (+0.89) 66.40 (+2.74) The routing budget is not monotonic: both benchmarks peak at ρ=0.4ρ=0.4. Larger budgets such as 0.60.6 and 0.80.8 as well as very small budgets such as 0.10.1 both reduce the gains, but all sweep settings remain above the baseline, indicating that the method is robust to threshold choice within the explored range. Table 4: Strict disjoint masks vs. overlapping masks at ρ=0.4ρ=0.4. Setting LogicVista RealWorldQA strict disjoint (default) 46.98 (+2.46) 67.45 (+3.79) overlap allowed 46.31 (+1.79) 67.06 (+3.40) Strict disjoint routing is stronger than overlap on both benchmarks, supporting our default of enforcing disjoint visual and reasoning masks. Table 5: Decoupled visual and reasoning routing budgets. ρv _v ρr _r LogicVista RealWorldQA 0.200.20 0.200.20 45.10 (+0.58) 66.27 (+2.61) 0.200.20 0.400.40 45.41 (+0.89) 66.40 (+2.74) 0.200.20 0.600.60 45.19 (+0.67) 66.53 (+2.87) 0.200.20 0.800.80 44.97 (+0.45) 66.27 (+2.61) 0.400.40 0.200.20 46.31 (+1.79) 66.93 (+3.27) 0.400.40 0.400.40 46.98 (+2.46) 67.45 (+3.79) 0.400.40 0.600.60 46.76 (+2.24) 67.19 (+3.53) 0.400.40 0.800.80 46.31 (+1.79) 67.06 (+3.40) 0.600.60 0.200.20 45.86 (+1.34) 66.80 (+3.14) 0.600.60 0.400.40 45.86 (+1.34) 66.67 (+3.01) 0.600.60 0.600.60 45.41 (+0.89) 66.40 (+2.74) 0.600.60 0.800.80 45.19 (+0.67) 66.40 (+2.74) 0.800.80 0.200.20 45.10 (+0.58) 66.27 (+2.61) 0.800.80 0.400.40 45.10 (+0.58) 66.27 (+2.61) 0.800.80 0.600.60 45.19 (+0.67) 66.40 (+2.74) 0.800.80 0.800.80 45.19 (+0.67) 66.40 (+2.74) The decoupled sweep shows that the symmetric setting ρv=ρr=0.4 _v= _r=0.4 is best on both benchmarks. Nearby asymmetric settings such as ρv=0.4,ρr=0.6 _v=0.4, _r=0.6 remain competitive, indicating that the optimum is reasonably flat around the default. Table 6: Swapped-routing sanity check at ρ=0.4ρ=0.4. Setting LogicVista RealWorldQA correctly routed (default) 46.98 (+2.46) 67.45 (+3.79) swapped routing 45.41 (+0.89) 66.27 (+2.61) Swapped routing is clearly weaker than the correctly routed strict setting on both benchmarks, supporting the claim that the direction of reward-token assignment matters rather than gains coming from updating an arbitrary subset of tokens. C.2 Routing Signal We compare the default image-sensitivity routing signal against alternative visual routing signals, and vary the reasoning-side signal while holding the visual channel fixed. We additionally vary the patch size and drop probability used for raw-image patch blackening. Table 7: Visual routing signal ablation. Setting LogicVista RealWorldQA Macro Δ image-sensitivity top-k (default) 46.98 67.45 57.22 +3.13 visual-focus only 46.31 67.19 56.75 +2.66 hard top-k visual focus 46.53 67.06 56.80 +2.71 cosine-prototype score 45.41 66.93 56.17 +2.08 Image-sensitivity top-k outperforms prototype-similarity and visual-focus alternatives, supporting it as the default visual routing signal. Table 8: Reasoning routing signal ablation. Setting LogicVista RealWorldQA Macro Δ neg. log-prob (default) 46.98 67.45 57.22 +3.13 token loss 46.76 67.19 56.97 +2.88 random non-visual 45.41 66.40 55.90 +1.81 Both likelihood-based routing signals clearly outperform random non-visual routing, indicating that reasoning tokens cannot be selected arbitrarily. Negative log-probability is slightly stronger than token loss and is used as the default reasoning-side routing signal. Table 9: Patch-blackening robustness for image-sensitivity routing. Values are two-benchmark Macro averages. Patch / Drop 0.25 0.50 0.75 patch=77 56.50 56.75 56.95 patch=1414 56.40 57.22 56.80 patch=2828 56.60 56.50 56.70 patch=5656 56.60 56.85 56.20 All patch/drop settings remain above Macro 56.2056.20 (vs. base Macro 54.0954.09), so image-sensitivity routing is not overly fragile. The default (patch size 1414, drop 0.500.50) reaches Macro 57.2257.22. Large patches with high drop probability are the weakest combination, suggesting that overly strong image corruption makes the sensitivity signal noisy. Table 10: Readable span-level visualization of token routing for the MMStar dog-counting case. Color legend follows the surrounding paragraph. Field Routed rollout excerpt Question How many dogs can be seen in the image? Options: A: 33, B: 22, C: 11, D: 44. Initial rollout I need to identify all the dogs in the image. There is one dog visible on the left side of the image, lying down. No other dogs are clearly visible in the rest of the room. Final answer: C. Selected refined candidate I need to carefully identify the dogs in the image. There is a dog visible on the left side of the image, lying down. Another dog is partially visible behind the couch near the center of the room. No other dogs are clearly visible. Final answer: B. Routing signal Image-sensitivity top-k/mean =1.79/0.32=1.79/0.32; entropy top-k/mean =1.83/0.54=1.83/0.54. C.3 Reward Components We ablate the visual reward formulation, the reasoning-reward components, the visual-reward temperature τvis _vis, and the attention layer used for engagement scoring. Table 11: Visual reward ablation. Setting LogicVista RealWorldQA Macro visual engagement (default) 46.98 (+2.46) 67.45 (+3.79) 57.22 (+3.13) ECL clue + crop 44.97 (+0.45) 63.40 (−0.26-0.26) 54.19 (+0.09) image-likelihood delta 46.09 (+1.57) 65.10 (+1.44) 55.60 (+1.50) Table 12: Reasoning reward ablation. Setting LogicVista RealWorldQA Macro full (default) 46.98 (+2.46) 67.45 (+3.79) 57.22 (+3.13) bounded only 46.76 (+2.24) 66.93 (+3.27) 56.85 (+2.76) margin only 45.41 (+0.89) 66.27 (+2.61) 55.84 (+1.75) no text prior 45.86 (+1.34) 66.80 (+3.14) 56.33 (+2.24) The default visual engagement reward dominates both alternative formulations. On the reasoning side, removing either the bounded score or the margin term hurts both benchmarks, and removing the text-only prior also degrades performance; all three components contribute to the full reward. Table 13: Sweep over the visual-reward temperature τvis _vis. τvis _vis LogicVista RealWorldQA Macro 0.050.05 46.53 (+2.01) 67.06 (+3.40) 56.80 (+2.71) 0.100.10 46.76 (+2.24) 67.19 (+3.53) 56.97 (+2.88) 0.200.20 (default) 46.98 (+2.46) 67.45 (+3.79) 57.22 (+3.13) 0.500.50 46.76 (+2.24) 67.06 (+3.40) 56.91 (+2.82) 1.001.00 46.53 (+2.01) 66.93 (+3.27) 56.73 (+2.64) Table 14: Attention-layer ablation for visual engagement scoring. Setting LogicVista RealWorldQA Macro last layer (default) 46.98 (+2.46) 67.45 (+3.79) 57.22 (+3.13) middle layer 46.09 (+1.57) 67.19 (+3.53) 56.64 (+2.55) The visual reward is stable across τvis∈[0.05,1.0] _vis∈[0.05,1.0] with the default τvis=0.2 _vis=0.2 as the optimum. Using the final attention layer for engagement scoring outperforms a middle-layer choice, consistent with the late layer carrying more task-aligned image attention. C.4 Latent Optimization Hyperparameters We evaluate ten representative latent-optimization settings that vary the number of refinement steps K, the update-length ratio ρ, and the anchor/logit regularization configuration. Table 15: Latent optimization hyperparameter ablation. Setting Parameter LogicVista Δ RealWorldQA Δ d1_steps1 K=1K=1 45.86 +1.34 66.40 +2.74 d1_steps2 K=2K=2 46.53 +2.01 66.93 +3.27 d1_steps4 (default) K=4K=4 46.98 +2.46 67.45 +3.79 d1_steps8 K=8K=8 46.76 +2.24 67.19 +3.53 d2_rho0p1 ρ=0.1ρ=0.1 45.10 +0.58 66.27 +2.61 d2_rho0p25 ρ=0.25ρ=0.25 45.86 +1.34 66.67 +3.01 d2_rho0p75 ρ=0.75ρ=0.75 46.53 +2.01 67.06 +3.40 d2_rho1p0 ρ=1.0ρ=1.0 44.97 +0.45 65.62 +1.96 d3_anchor0_logitlr0p01 anchor=00, logit_lr=0.010.01 46.53 +2.01 66.93 +3.27 d3_anchor0p05_logitlr0 anchor=0.050.05, logit_lr=00 46.31 +1.79 67.06 +3.40 The default setting (K=4K=4, ρ=0.5ρ=0.5, anchor=0.050.05, logit_lr=0.010.01) achieves the best accuracy on both benchmarks. K=4K=4 outperforms both K=2K=2 and K=8K=8, indicating that a moderate update budget is preferable to under- or over-shooting. The update-length ratio ρ=0.5ρ=0.5 outperforms shorter ratios (ρ≤0.25ρ≤ 0.25) and the very aggressive ρ=1.0ρ=1.0, confirming that updating the full latent span is harmful. The anchor-only and logit-only variants remain effective but neither exceeds the default, indicating that the two regularizers complement each other. C.5 Matched-Budget Comparison To verify that gains are not purely a function of additional decoding compute, we compare our method against Best-of-N and Self-consistency baselines at a matched number of decoded candidate continuations. Since our method additionally performs latent optimization, image corruption, attention extraction, and reward scoring, we report generation calls, scoring calls, average decoded tokens, and GPU time per example. All methods use the same answer extraction and evaluation protocol. Output-space baselines use N=16N=16 decoded candidates, matching the 44 refinement steps with 44 candidate continuations per step in our method. LatentSeek is reported under its 1010-step configuration. One generation call denotes one autoregressive rollout; score calls count reward-scoring forward passes (for our method this includes reasoning and text-prior scoring, visual-engagement attention scoring, and image-sensitivity scoring). Runtime values are estimated from Qwen2.5-VL-7B sharded runs and call-count-matched baseline budgets. Table 16: Matched-budget comparison against output-space scaling baselines. Method Gen. calls Score calls Dec. toks/sample GPU sec/sample MathVista LogicVista CoT 11 00 ≈130≈ 130 ≈3≈ 3 67.8067.80 44.5244.52 Self-consistency 1717 00 ≈2.2k≈ 2.2k ≈39≈ 39 70.9070.90 42.5142.51 Best-of-N 1717 ≈32≈ 32 ≈2.2k≈ 2.2k ≈43≈ 43 71.6071.60 42.7342.73 Reward-only 1717 ≈34≈ 34 ≈2.2k≈ 2.2k ≈45≈ 45 70.9070.90 42.9542.95 LatentSeek(per.) 1111 ≥11≥ 11 ≈1.4k≈ 1.4k ≈35≈ 35 70.2070.20 44.3044.30 Ours 1717 ≈61≈ 61 ≈2.2k≈ 2.2k ≈44≈ 44 72.1072.10 46.9846.98 Our method is roughly on par with Best-of-N and Reward-only in generation calls and GPU seconds per sample, but uses noticeably more scoring calls because the visual-engagement and image-sensitivity passes sit outside the candidate decoding pipeline. Under this accounting, our method still produces the strongest scores on both MathVista and LogicVista, indicating that the gains are not explained by extra decoded tokens alone. Appendix D Qualitative Case Studies We provide qualitative examples from the Qwen2.5-VL-7B runs under the default configuration of Appendix B.2 (τvis=0.2 _vis=0.2, ρv=ρr=0.4 _v= _r=0.4). The examples are selected only for analysis after the benchmark runs complete; they are not used for tuning. To keep the appendix focused on the mechanism rather than on verbose generated chains, we report the benchmark question, the ground-truth short answer, the initial and final short predictions, and the routed reward diagnostics. Table 17: Successful case studies. Each row shows an example where the initial rollout is incorrect and the token-disentangled update recovers the correct short answer. Case Query summary Ground truth Initial → Final Qualitative change MMStar, object counting How many dogs can be seen in the image? B: 22 C → B The initial rollout counts one visible dog. The refined candidate adds a second, partially visible dog behind the couch. RealWorldQA, scene geometry What level is the ground at? Options: flat, incline, decline. B: incline A → B The initial rollout treats the street as flat. The refined answer uses the slope toward the horizon and predicts incline. MathVista, chart reading How many bars have values larger than 100100? 11 2→12→ 1 The initial rollout counts both bars. The refined answer keeps only the bar above the threshold and rejects the bar below 10210^2. LogicVista, mechanical reasoning If the weight is lifted by 1010 m, which pulley rope must be pulled further? C B → C The initial rollout selects the simpler two-pulley system. The refined answer identifies the system requiring the longer rope displacement. HallusionBench, temporal order The plug is removed from the power outlet. Are the images in the correct positive order? No Yes → No The initial rollout assumes an insertion sequence. The refined answer rejects the sequence as inconsistent with the stated removal event. Table 18: Routed reward diagnostics for the cases in Table 17. E(y)E(y) is the visual engagement of the initial rollout and E(x⋆)E(x ) is the engagement of the selected refinement candidate xk⋆x_k that produced the final short answer in column 4 of Table 17. The image-sensitivity and entropy columns report selected-token top-k means versus slice means. Case E(y)→E(x⋆)E(y)→ E(x ) RvisR_vis RreaR_rea: initial → selected Visual / reasoning tokens Image sensitivity top-k / mean Entropy top-k / mean MMStar counting 0.0867→0.11330.0867→ 0.1133 0.1330.133 −0.997→0.858-0.997→ 0.858 12/3012/30 ; 8/308/30 1.79/0.321.79/0.32 1.83/0.541.83/0.54 RealWorldQA incline 0.1305→0.14010.1305→ 0.1401 0.0480.048 −0.935→0.999-0.935→ 0.999 11/2711/27 ; 7/277/27 1.13/0.241.13/0.24 1.37/0.511.37/0.51 MathVista chart 0.0762→0.08260.0762→ 0.0826 0.0320.032 −1.000→0.981-1.000→ 0.981 20/4920/49 ; 12/4912/49 2.65/0.342.65/0.34 1.15/0.231.15/0.23 LogicVista pulley 0.0924→0.10630.0924→ 0.1063 0.0690.069 −0.725→0.919-0.725→ 0.919 28/6828/68 ; 16/6816/68 1.22/0.121.22/0.12 2.31/0.752.31/0.75 HallusionBench sequence 0.1106→0.13090.1106→ 0.1309 0.1010.101 −0.726→−0.398-0.726→-0.398 11/2711/27 ; 7/277/27 1.44/0.341.44/0.34 1.67/0.701.67/0.70 Example token routing visualization. Table 10 visualizes the routed token selection for the MMStar dog-counting example. The visual / reasoning masks are tied to editable latent positions 1..L1..L, so the same set of positions is highlighted on both the initial rollout and the refined candidate; the displayed text differs because the decoded tokens at those positions change after refinement. The actual masks are computed over tokenizer subwords in the editable hidden-state prefix; for readability, adjacent subwords with the same routed role are merged into representative text spans, so span length is illustrative rather than a literal subword count. Pink spans receive the visual reward, blue spans receive the reasoning reward, and gray spans remain anchored. In this example, 12/3012/30 editable tokens are routed to the visual branch and 8/308/30 to the reasoning branch. The visual tokens concentrate on image-dependent phrases such as the object being counted and its location, while the reasoning tokens concentrate on the count decision. These examples show two recurring patterns. First, successful visual corrections are accompanied by higher engagement with image tokens, but the visual reward is not broadcast globally: it is assigned only to the image-sensitive subset, whose top-k scores are substantially larger than the slice mean. Second, reasoning-heavy examples such as the chart and pulley cases rely on high-entropy latent positions. The reasoning branch receives the answer-quality reward, while the visual branch keeps the refinement tied to image evidence. This separation is especially visible in the LogicVista example: the visual reward is modest, but the selected entropy tokens have much larger uncertainty than the slice average, and the reasoning reward changes the final answer from B to C. Note that the selection rule is an argmax over candidates, so even when the absolute reasoning reward stays negative (HallusionBench: −0.726→−0.398-0.726→-0.398), the relative gain over alternatives is sufficient to flip the answer from Yes to No.