Paper deep dive
Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach
Tianbao Jiang, Weicong Ni, Gerard de Melo, Linlin Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Post-training reinforcement learning (RL) algorithms are commonly used to align large vision-language models (LVLMs) with human intent and the requirements of visual reasoning tasks. However, existing RL-based alignment methods are often resource-intensive and encounter mismatches between training objectives and inference-time distributions. To bridge this gap, we propose a novel test-time alignment approach that leverages trajectory-guided structured sampling for dynamic inference-time refinement, achieving better alignment with visual grounding and ensuring logical consistency. Our approach begins with curating a reasoning memory bank via a trajectory learning algorithm, which decomposes complex question solving into ordered sequences of predefined reasoning patterns. It subsequently accomplishes inference-time alignment by first collecting trajectories from reasoning memory bank to establish a global structural reasoning prior, and then using an iterative Markov Chain Monte Carlo (MCMC) algorithm for localized multi-objective refinement of the reasoning trace. Experiments across multiple multimodal reasoning datasets demonstrate that our approach significantly improves accuracy without incurring prohibitive inference overhead. These results establish trajectory-guided test-time sampling as a scalable and effective alternative to traditional post-training alignment, particularly for complex visual reasoning tasks.
Tags
Links
- Source: https://arxiv.org/abs/2608.03204v1
- Canonical: https://arxiv.org/abs/2608.03204v1
Trouble viewing inline? Open PDF directly →
Full Text
68,102 characters extracted from source content.
Expand or collapse full text
by Aligning Large Vision–Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach Tianbao Jiang 0009-0002-5534-4436 East China Normal UniversityShanghaiChina tbjiang@stu.ecnu.edu.cn , Weicong Ni 0009-0009-3399-8827 East China Normal UniversityShanghaiChina 3025394091@q.com , Gerard de Melo 0000-0002-2930-2059 Hasso Plattner InstitutePotsdamGermany University of PotsdamPotsdamGermany gdm@demelo.org and Linlin Wang 0000-0003-0304-7560 East China Normal UniversityShanghaiChina City University of Hong KongHong KongChina llwang@cs.ecnu.edu.cn (2026) Abstract. Post-training reinforcement learning (RL) algorithms are commonly used to align large vision-language models (LVLMs) with human intent and the requirements of visual reasoning tasks. However, existing RL-based alignment methods are often resource-intensive and encounter mismatches between training objectives and inference-time distributions. To bridge this gap, we propose a novel test-time alignment approach that leverages trajectory-guided structured sampling for dynamic inference-time refinement, achieving better alignment with visual grounding and ensuring logical consistency. Our approach begins with curating a reasoning memory bank via a trajectory learning algorithm, which decomposes complex question solving into ordered sequences of predefined reasoning patterns. It subsequently accomplishes inference-time alignment by first collecting trajectories from reasoning memory bank to establish a global structural reasoning prior, and then using an iterative Markov Chain Monte Carlo (MCMC) algorithm for localized multi-objective refinement of the reasoning trace. Experiments across multiple multimodal reasoning datasets demonstrate that our approach significantly improves accuracy without incurring prohibitive inference overhead. These results establish trajectory-guided test-time sampling as a scalable and effective alternative to traditional post-training alignment, particularly for complex visual reasoning tasks. Test-Time Alignment, Large Vision-Language Model †copyright: c†journalyear: 2026†doi: 10.1145/3767308.3835984†conference: 34th ACM International Conference on Multimedia; November 10–14, 2026; Rio de Janeiro, Brazil†isbn: 979-8-4007-2213-4/2026/11†ccs: Computing methodologies Knowledge representation and reasoning†ccs: Computing methodologies Natural language generation 1. Introduction Despite their impressive performance across many tasks (Bai et al., 2025b; Zhu et al., 2025), large vision–language models (LVLMs) remain unreliable on complex visual reasoning tasks that require both strong visual grounding and multi-step inference (Zhang et al., 2025; Yao et al., 2025). In such settings, even after post-training, they often fail to sustain visually grounded and logically coherent intermediate reasoning at inference time, drifting from visual evidence and producing unstable reasoning trajectories. Existing alignment approaches, such as reinforcement learning from human feedback (RLHF) (Yu et al., 2024), direct preference optimization (DPO) (Liu et al., 2025), and reinforcement learning with verifiable rewards (RLVR), primarily address this problem through offline parameter updates. However, such training-based methods are computationally expensive and offer limited control over inference-time reasoning, especially under new or unforeseen input distributions. Figure 1. Standard autoregressive decoding makes locally optimal choices that may lead to errors. In contrast, power-scaled sampling reweights full sequences via a power-transformed probability pαp^α (α>1α>1). Aligning LVLMs at inference time without modifying their underlying weights emerges as a promising direction for addressing the limitations of training-based alignment methods (Lin et al., 2025; Xu et al., 2025a), such as ensemble-based hypothesis reweighting (Lee et al., 2025), and predictive planning-based alignment (Wang et al., 2025c). Within this paradigm, sampling-based test-time alignment offers an attractive trade-off by using additional decoding-time compute to search for outputs that better satisfy task objectives under distribution shift. In practice, this is often achieved by reshaping the model’s output distribution, for example through reward model guided reweighting (Lin et al., 2025), importance-weighted perturbations (Kanai et al., 2025), or iterative refinement. More recently, Karan and Du (2025) demonstrate that sampling from the power-scaled distribution pαp^α via Metropolis–Hastings (MH) yields training-free reasoning improvements that are comparable to those obtained with RLVR. As shown in Figure 1, sequence-level reweighting in power-scaled sampling avoids misleading local token choices and better preserves visual evidence than autoregressive decoding. Despite its promise, applying MH-based power sampling to LVLMs for visual reasoning remains challenging. First, power scaling may amplify language priors (Wang et al., 2025a) and visually unsupported details when visual evidence is weak, exacerbating hallucinations. Second, complex visual reasoning requires long-horizon, multi-step deduction with multiple plausible solution paths. For example, solving the question in Figure 1 requires the model to identify the derivatives at both x=2x=2 and x=5x=5 before comparing them. Sampling can easily drift into high-entropy, unstable reasoning branches. Finally, for long-horizon generation, the induced Markov chain often mixes slowly in a high-dimensional token space (Brown and Rosenthal, 2025), resulting in low acceptance and many expensive proposal rollouts, which substantially increases inference latency. To address these limitations, we propose a training-free test-time alignment framework that allocates computation to targeted, structured refinement during decoding. We first build a reasoning memory bank through our trajectory learning algorithm, where each trajectory encodes the solution process for a complex question as an ordered sequence of high-level reasoning patterns. Given a query, we retrieve the top-k most similar examples from the bank and aggregate their trajectories via majority voting to obtain a single guidance trajectory. This trajectory provides a global prior over the reasoning structure and step order, while the LVLM is responsible for instantiating each pattern into task-specific reasoning content. Based on this decomposition, our structured sampling algorithm initializes one reasoning pattern per iteration and performs MCMC refinement within a sliding window of neighboring reasoning patterns. This localized refinement yields low-variance proposals and mitigates slow mixing, making MCMC practical for long-form LVLM generation. On top of this sampler, we define three complementary objectives to shape the power-scaled target distribution: vision-aware distribution sharpening to facilitate visual grounding, an entropy regularizer to favor reliable trajectories, and a linguistic control term to suppress degenerative generation. In summary, the integration of trajectory-guided structured proposals, iterative Metropolis–Hastings inference, and our synergistic objectives enables effective test-time alignment for complex visual reasoning. Our main contributions are three-folds: • We propose an automatic trajectory learning algorithm rooted in an agentic framework, designed to construct and store candidate trajectories, thereby building a continuous memory of reasoning processes. • We develop a trajectory-guided structured sampling method that retrieves trajectories from memory as a structural reasoning prior, followed by iterative MCMC trajectory refinement that promotes visual grounding and stable generation through vision-aware distribution sharpening, entropy regularization, and linguistic degeneration control, thereby ensuring alignment during inference. • Extensive experiments across multiple reasoning benchmarks demonstrate consistent gains in both accuracy and sampling efficiency, establishing our training-free test-time sampling approach as a highly effective alternative to RLVR. 2. Preliminaries Power-Scaled Sampling. Let V and X denote the visual and textual inputs, respectively, and let =⟨y0,…,yT⟩Y= y_0,…,y_T be a finite output token sequence with yt∈y_t , where V is the model vocabulary. An LVLM induces a normalized autoregressive distribution over Y: (1) P(∣V,X)=∏t=0TP(yt∣V,X,y<t),P(Y V,X)= _t=0^TP(y_t V,X,y_<t), where Y denotes the set of all finite sequences. Recent work shows that strong reasoning gains can be obtained by sampling from a power-transformed sequence distribution (Karan and Du, 2025). Given an exponent α>1α>1, the corresponding unnormalized target is (2) pα(∣V,X)∝P(∣V,X)α,p_α(Y V,X)\ \ P(Y V,X)^α, which amplifies differences in sequence likelihood and concentrates probability mass on high-likelihood trajectories under the base model. Power sampling operates at the sequence level by reweighting complete continuations Y, rather than making greedy or temperature-scaled decisions at each step, as illustrated in Figure 1. Sampling from the power distribution implicitly favors longer-horizon planning by prioritizing globally coherent, high-likelihood trajectories and mitigating failures induced by pivotal tokens (Abdin et al., 2024) that steer generation toward low-likelihood continuations. Figure 2. Overview of our approach. We begin by constructing a reasoning memory bank based on a trajectory learning algorithm. At inference time, retrieved trajectories provide a global structural reasoning prior, while MCMC performs localized updates within selected reasoning segments to iteratively steer generation toward the alignment target. 3. Methodology In this section, we present our test-time alignment approach for visual reasoning with LVLMs (Figure 2). We first introduce the automatic trajectory memory curation pipeline (Section 3.1), which uses an agentic framework to search for effective reasoning paths under predefined reasoning patterns. We then describe the structured sampling algorithm (Section 3.2), where trajectory guidance provides a structural prior for LVLM reasoning and Metropolis–Hastings is used to perform iterative refinement within a local window of adjacent reasoning patterns. Finally, we introduce our refinement target (Section 3.3), which combines vision-aware distribution sharpening, entropy regularization, and linguistic degeneration control. Figure 3. Automatic trajectory memory curation. After defining reasoning patterns and preparing the seed dataset, the reasoning memory bank is built through a two-stage reasoning memory curation process. 3.1. Automatic Trajectory Memory Curation Complex multimodal questions often require high-level reasoning behaviors, such as decomposition (Li et al., 2025c), knowledge grounding, and self-verification (Wang et al., 2025b). However, standard LVLM decoding may fail to activate these behaviors reliably, leading to brittle trajectories on challenging instances. We therefore introduce an automated trajectory discovery pipeline that decomposes the reasoning process into discrete patterns, curating a reusable memory bank of structured reasoning traces to facilitate test-time guidance (Figure 3). Based on insights from prior works (Wu et al., 2026; Li et al., 2025b), we define seven reasoning patterns, as shown in Figure 3. Rather than relying on heuristic prompts or taxonomies of reasoning actions (Li et al., 2025a), we describe each pattern in a pseudo-code style , making offline reasoning experience explicit and reusable during inference. We construct the seed set using 34k de-duplicated question–image pairs from ThinkLite-70k (Wang et al., 2025d), covering mathematical reasoning, natural image understanding, and chart understanding. Trajectory Learning Algorithm. Motivated by Lu et al. (2025), we develop an agentic framework that automatically learns trajectories through iterative collaboration among a planner, an executor, and a manager, as shown in Figure 3. A standard chain-of-thought (CoT) can be represented as =r0,r1,…,rTY=\r_0,r_1,…,r_T\, where rtr_t denotes the t-th rationale and rTr_T contains the final answer. To make a reasoning trajectory executable and editable at the step level, we represent each intermediate state using (i) a task list T, which specifies the remaining tasks, and (i) an information list I, which stores the intermediate results and supporting evidence collected so far. This state representation also provides explicit inputs and outputs for each reasoning pattern. Accordingly, the CoT is reformulated as: (3) =(0,0),(1,1),…,(T,T),Y=\(T_0,I_0),(T_1,I_1),…,(T_T,I_T)\, where TI_T contains the final answer. Given an image V and a question X, the planner LVLM MθM_θ selects the next reasoning pattern ata_t at step t. Its decision is conditioned on the first pending task t(0)T_t^(0) and the reasoning history HtH_t, which records the previously selected patterns and their associated thinking contexts: (4) at=Mθ(V,X,t(0),Ht),t≥0.a_t=M_θ(V,X,T_t^(0),H_t), t≥ 0. To facilitate exploration, we rank patterns by their first-token probabilities and sample from the top candidates. The executor LVLM then applies the selected pattern to generate an intermediate thinking context ctc_t and the corresponding pattern-specific outputs oto_t: (5) ct,ot=Mθ(V,X,a≤t,t,t).c_t,o_t=M_θ(V,X,a_≤ t,T_t,I_t). Finally, the manager LVLM uses ctc_t and oto_t to transform the current state (t,t)(T_t,I_t) into the next state (t+1,t+1)(T_t+1,I_t+1). Specifically, it removes completed tasks from tT_t and updates tI_t by adding newly obtained information or revising existing entries. The updated lists then provide a clear starting state for the next reasoning step. Reasoning Memory Curation. We use depth-first search to collect candidate trajectories. At each step, we retain the highest-ranked candidate patterns until their cumulative probability reaches a threshold τ∈(0,1)τ∈(0,1), and backtrack when a terminal pattern is reached. To estimate reliability, we roll out each candidate trajectory k times and calculate its answer correctness rate, retaining the most reliable traces. For downstream retrieval, we index the question embeddings in a vector database and store the best-performing trajectory for each question as retrievable metadata. 3.2. Trajectory-Guided Structured Sampling At inference, we retrieve the top-k questions most similar to input (V,X)(V,X). Candidates are ranked by an entropy-weighted score combining image and text embedding similarities with n-gram overlap, where lower-entropy views receive larger weights. The associated trajectories form: (V,X)=(j)j=1k,T(V,X)=\ τ^(j)\_j=1^k, where each trajectory is an ordered sequence of predefined reasoning patterns. Given the retrieved set, we construct a guidance trajectory by majority voting at each aligned position: (6) ∗=⟨a1∗,…,aL∗⟩. τ^*= a^*_1,…,a^*_L . Starting from all retrieved trajectories, at each step t, we select the most frequent pattern as at∗a_t^* and retain only trajectories matching the selected prefix ⟨a1∗,…,at∗⟩ a_1^*,…,a_t^* for the next step. Metropolis–Hastings (Metropolis et al., 1953) is a classic Markov Chain Monte Carlo (MCMC) algorithm that approximately samples from an unnormalized target distribution via iterative proposals and a stochastic accept–reject rule. Let 0:T0Y^0_0:T denote the current sequence being refined, and let ppropp_prop be the proposal LVLM. At iteration i, we sample a boundary index t∼0,…,Tt \0,…,T\ and propose ′Y by resampling the suffix t:TiY^i_t:T with ppropp_prop while keeping the prefix 0:t−1iY^i_0:t-1 fixed. The proposal is accepted with probability (7) A(i,′)=min1,pt(′)q(i∣′)pt(i)q(′∣i),A(Y^i,Y )= \1, p_t(Y )\,q(Y^i )p_t(Y^i)\,q(Y ^i) \, where ptp_t is the unnormalized target density over sequences and q is the proposal transition. Repeating for N iterations yields a chain 0,…,N\Y^0,…,Y^N\ that approximately samples from the target. Given the trajectory guidance, we first use a rule-based estimator that takes the aggregated trajectory as input to route trivial questions to full-sequence MCMC refinement in order to reduce overthinking and improve efficiency. For non-trivial questions, we prompt the LVLM to reason by following the guided trajectory and do iterative sampling as shown in Algorithm 1. We highlight three design choices that are central to our approach. Algorithm 1 Trajectory-Guided Structured Sampling 1:proposal LVLM ppropp_prop, inputs (V,X)(V,X), max length T, initial temperature τ0 _0, initial sharpening exponent α0 _0, decay threshold κ, decay factor γ, MCMC steps NMCMCN_MCMC 2:ℬ←[]B←[\,] ⊳ Initialize block boundaries; ℬB stores start indices of reasoning blocks 3:τ←τ0τ← _0, α←α0α← _0 4:while ||<T|Y|<T do 5: Annealed Sampling Schedule: 6: τ←max(0.1,τ⋅γ[|ℬ|>κ]τ (0.1,\,τ·γ^I[|B|>κ]), α←τ−1α←τ^-1 7: Pattern-wise Initialization: 8: U←|ℬ|+1U←|B|+1 ⊳ the max numbers of pattern to generate 9: ←Generation(pprop,,U,τ,α)Y← Generation(p_prop,\,Y,\,U,τ,\,α) 10: Sliding-Window Sampling: perform MCMC-based refinement within the newest two reasoning patterns. 11: ←MCMC(pprop,,NMCMC,τα)Y← MCMC(p_prop,Y,N_MCMC,\,τ\,α) 12: Update ℬB according to updated sequence Y 13: if HasTerminal(Y) then break 14:end while 15:return Y (i) Pattern-wise Initialization. Each iteration adds at most one new reasoning pattern, thus limiting proposal drift and improving acceptance in high-dimensional sequence spaces (Brown and Rosenthal, 2025). This pattern-level granularity also makes the refinement process interpretable. (i) Sliding-Window Sampling. Full-sequence resampling is costly for long outputs. Instead, we update only a local suffix over the most recent two reasoning patterns, reducing inference cost while retaining the benefits of sequence-level refinement. (i) Annealed Sampling Schedule. Since windowed MH refines recent patterns on top of a mostly fixed prefix, excessive temperature can destabilize the existing context. We therefore decrease the sampling temperature τ and increase the sharpening exponent α only when the number of reasoning patterns |||P| exceeds a threshold κ, yielding a smooth transition from exploration to exploitation. 3.3. Multi-Objective Trajectory Refinement Trajectory guidance provides a global prior over desirable reasoning patterns, but local realizations can still drift from visual evidence or fall into decoding pathologies. In this section, we define our test-time alignment target that augments power-based sharpening with step-level alignment objectives, so that alignment can be improved via MCMC refinement without updating model parameters. 3.3.1. Vision-Aware Distribution Sharpening Naively sharpening the sequence distribution can overemphasize language priors under weak visual evidence, leading to increased hallucinations. Inspired by contrastive visual decoding (Leng et al., 2024; Yin et al., 2025), we introduce a vision-aware calibration mechanism that reweights next-token probabilities against a ungrounded baseline. Specifically, the ungrounded branch is constructed by masking visual tokens during prefill, while subsequent decoding steps are synchronized with the visual-grounded branch through teacher-forcing. Let yty_t be the token chosen by the grounded branch at step t, we thereby define the alignment ratio rtr_t as: (8) rt=Pvisual-grounded(yt)+δPungrounded(yt)+δ,r_t= P_visual-grounded(y_t)+δP_ungrounded(y_t)+δ, where δ is a small smoothing constant for numerical stability. Intuitively, a larger rtr_t suggests that the current token is more visually grounded. We then convert rtr_t into a bounded calibration weight s(rt)s(r_t), following adaptive calibration (Huo et al., 2025; Hu et al., 2025): (9) s(rt)=exp(−β[softplus(−lnrt−ϵ)−softplus(−ϵ)]),rt<1,1,rt≥1,s(r_t)= cases \! (-β [softplus(- r_t-ε)-softplus(-ε) ] ),&r_t<1,\\ 1,&r_t≥ 1, cases where β controls suppression strength, ϵ=10−3ε=10^-3 is a small margin, and softplus(x)=ln(1+ex)softplus(x)= (1+e^x). The subtraction term makes the penalty zero at the boundary, ensuring continuity as rt→1−r_t→ 1^-. Next-token probabilities are calibrated with s(rt)s(r_t) as (10) P~(yt∣V,X,y<t)=P(yt∣V,X,y<t)⋅s(rt), P(y_t V,X,y_<t)\;=\;P(y_t V,X,y_<t)· s(r_t), resulting in a vision-aware sharpened sequence likelihood: (11) P~(∣V,X)α=(∏t=0TP~(yt∣V,X,y<t))α. P(Y V,X)^α\;=\;( _t=0^T P(y_t V,X,y_<t))^α. 3.3.2. Entropy Regularization Recent evidence suggests that RL-style alignment is often accompanied by reduced predictive entropy (Cui et al., 2025). Consistently, our preliminary experiment shows that correct generations tend to exhibit lower predictive entropy under the base model. We therefore exponentially downweight trajectories according to their cumulative token-level predictive entropy: (12) RH()=exp(−λH∑t=0TH[P(⋅∣V,X,y<t)]),R_H(Y)= \! (- _H _t=0^TH\! [P(· V,X,y_<t) ] ), where λH _H controls the strength of the entropy penalty. 3.3.3. Linguistic Degeneration Control To reduce premature stopping while discouraging repetitive loops, we introduce a linguistic control term that couples effective length with an explicit repetition signal. Let ℓ() (Y) be the sequence length and let ρ()∈[0,1]ρ(Y)∈[0,1] denote an n-gram repetition ratio. We define the effective length score as: (13) SL()=ℓ()1+ln(ℓ())ρ(),S_L(Y)\;=\; (Y)1+ ( (Y))\,ρ(Y), which grows with length but is increasingly discounted as repetition increases, especially for long outputs through the ln(ℓ()) ( (Y)) factor. We incorporate a linguistic degeneration control factor by exponentiating a shaped effective-length reward: (14) RL()=exp(L⋅(1−exp(−a⋅SL()b))),R_L(Y)\;=\; \! (L· (1- (-a· S_L(Y)^\,b ) ) ), where L sets the saturation level, a controls the sensitivity to changes in SL()S_L(Y), and b shapes the degree of diminishing returns. 3.3.4. Synergistic Joint Alignment Target By integrating these objectives, we define an unnormalized alignment target over complete sequences, with α>1α>1 controls the sharpening strength: (15) pt(∣V,X)∝P~(∣V,X)α⋅RH()⋅RL().p_t(Y V,X)\ \ P(Y V,X)^α· R_H(Y)· R_L(Y). This synergistic formulation yields a balanced energy landscape. The vision-aware term first filters out visually unsupported reasoning branches, after which entropy regularization biases MCMC toward reliable, confident trajectories. Crucially, while entropy regularization alone may favor confidently wrong outputs, its coupling with visual calibration ensures that only visually consistent paths are sharpened. Finally, the linguistic control term prevents collapse into repetitive loops or premature termination. Together, these objectives turn test-time sampling into a robust optimization process that yields concise and logically coherent reasoning trajectories. 4. Experiments Table 1. Performance comparison on five datasets. Method MathVista MathVision MathVerse MMMU MMStar Avg. General Vision-Language Models InternVL2.5-8B (Chen et al., 2025) 64.4 22.0 39.5 54.9 62.8 48.7 Qwen2.5-VL-7B (Bai et al., 2025b) 68.2 - 49.2 58.6 63.9 - InternVL3-8B (Zhu et al., 2025) 71.6 29.3 46.3 62.7 68.2 55.6 Qwen3-VL-8B (Bai et al., 2025a) 77.2 - 62.1 69.6 70.9 - RLVR-Trained Vision-Language Models OpenVLThinker-7B (Deng et al., 2025) 70.2 29.6 50.3 51.9 63.2 53.0 VL-Rethinker-7B (Wang et al., 2025b) 74.9 32.3 54.2 56.7 - - Vision-R1-7B (Huang et al., 2025) 73.5 30.7 52.4 50.5 60.2 53.4 M-EUREKA-7B (Meng et al., 2025) 73.0 31.9 50.3 52.3 64.1 54.3 ThinkLite-VL-7B (Wang et al., 2025d) 75.1 32.9 52.1 55.5 65.0 56.1 Sampling-based Test Time Alingment (Qwen2.5-VL-7B) Qwen2.5-VL-7B (Reproduced) 70.0 27.3 47.1 54.0 61.3 52.0 + Multi-Objective Sampling 73.2 ↑ 3.2 30.9 ↑ 3.6 50.0 ↑ 2.9 54.9 ↑ 0.9 64.0 ↑ 2.7 54.6 ↑ 2.6 + TG Structured Sampling 72.7 ↑ 2.7 30.3 ↑ 3.0 49.5 ↑ 2.4 54.7 ↑ 0.7 64.5 ↑ 3.2 54.3 ↑ 2.3 4.1. Experiment Settings 4.1.1. Datasets We evaluate our framework on diverse multimodal benchmarks requiring multi-hop reasoning, allowing us to assess test-time alignment under long-horizon inference. We cover multimodal math reasoning through MathVista (Lu et al., 2024), MathVerse (Zhang et al., 2024), and MathVision (Wang et al., 2024), while targeting college-level multi-disciplinary tasks in MMMU (Yue et al., 2024) and general LVLM robustness in MMStar (Chen et al., 2024). 4.1.2. Baselines We compare against two categories of models: • Strong general-purpose LVLMs. We include widely used open-source backbones with strong general multimodal performance across model generations: InternVL2.5-8B (Chen et al., 2025), Qwen2.5-VL-7B (Bai et al., 2025b), InternVL3-8B (Zhu et al., 2025), and Qwen3-VL-8B (Bai et al., 2025a). • RLVR-trained LVLMs We include representative reasoning-oriented LVLMs that improve multi-step inference through RL with verifiable rewards (RLVR),such as OpenVLThinker-7B (Deng et al., 2025), VL-Rethinker-7B (Wang et al., 2025b), Vision-R1-7B (Huang et al., 2025), M-EUREKA-7B (Meng et al., 2025), ThinkLite-VL-7B (Wang et al., 2025d). All the baselines share the same Qwen2.5-VL-7B backbone for fair comparison. 4.1.3. Implementation details For trajectory discovery pipeline, we adopt Qwen3-VL-8B as the planner, executor, and manager. After verification, we retain 24k question–trajectory pairs and index the questions in a vector database. In our main experiments, we report results for two variants: (i) Multi-Objective Sampling without trajectory guidance, where we prompt the model to produce intermediate reasoning before answering and perform full-sequence MCMC refinement, as in Karan and Du (2025). and (i) Trajectory-Guided (TG) Structured Sampling, which further incorporates trajectory guidance and block-wise refinement for non-trivial questions. We set the sharpening exponent α=4α=4 and the generation temperature τ=0.25τ=0.25. For the alignment objectives, we use visual calibration weight β=2.0β=2.0, entropy weight λH=1.0 _H=1.0. For linguistic control term, we set a=0.05a=0.05, b=0.8b=0.8 and linguistic control saturation level L=128L=128. For TG structured sampling, we set the decay threshold κ=4κ=4 and decay factor γ=0.5γ=0.5. We evaluate our methods on Qwen2.5-VL-7B, a widely used LVLM that has shown strong potential in prior RL-based studies. Table 2. Ablation on trajectory-guided structured sampling. Method MathVista MMStar Mathvision TG Structured Sampling 72.7 64.5 30.3 w/o Sampling Schedule 71.9↓ 0.8 63.6↓ 0.9 28.6↓ 1.7 w/o Trajectory Guidance 71.8↓ 0.9 63.8↓ 0.7 28.0↓ 2.3 Table 3. Ablation study on alignment target. We sample directly from Qwen2.5-VL-7B under different MCMC target distributions. Method MathVista MathVision MathVerse MMMU MMStar Average Qwen2.5-VL-7B (Reproduced) 70.0 27.3 47.1 54.0 61.3 52.0 + Power Sampling 72.0 29.9 49.3 52.7 63.1 53.4 + Power Sampling + Visual 70.8 30.9 49.5 53.6 64.1 53.8 + Power Sampling + Entropy 70.2 25.0 46.6 52.1 62.7 51.3 + Power Sampling + Entropy + Visual 72.2 31.3 49.8 54.7 64.4 54.5 + Power Sampling + Entropy + Visual + Linguistic 73.2 30.9 50.0 54.9 64.0 54.6 Table 4. Performance on MMStar and MathVision with InternVL2.5-8B and Qwen3-VL-8B. Method MMStar MathVision InternVL2.5-8B (Chen et al., 2025) 61.2 21.4 + Multi-Objective Sampling 62.8 ↑ 1.6 25.0 ↑ 3.6 Qwen3-VL-8B (Bai et al., 2025a) 65.3 47.7 + Multi-Objective Sampling 73.2 ↑ 7.9 51.0 ↑ 3.3 4.2. Main Results Table 1 reports results on five challenging multimodal datasets. Overall, our methods consistently improves the reproduced Qwen2.5-VL-7B baseline across all datasets. Multi-Objective Sampling yields the best average accuracy (54.654.6), while TG Structured Sampling remains highly competitive (54.354.3). Three takeaways stand out. Training-free test-time alignment is competitive with RLVR-trained reasoning models. Without any post-training, our sampling-based refinement closes much of the gap to RLVR-aligned LVLMs and outperforms several RLVR-trained baselines on the reported suite, suggesting that reallocating compute to inference-time refinement is an effective way to improve alignment. Trajectory-guided structured sampling preserves accuracy and improves efficiency. Compared to full-sequence refinement, TG Structured Sampling achieves similar accuracy while using step-aligned, block-wise proposals guided by a voted trajectory prior, reducing unnecessary resampling and making long-horizon refinement more practical (efficiency results in Section 4.7). General-purpose trajectory priors help most on general dataset. Since the trajectory memory is built from a general-purpose seed set (ThinkLite-70k), trajectory guidance is particularly effective on the general dataset MMStar. On more reasoning-intensive math datasets, gains remain consistent but can be slightly smaller, suggesting room for more domain-specialized trajectory libraries. 4.3. Ablation Study on Structured Sampling Table 2 ablates two key components of trajectory-guided structured sampling. Removing annealed sampling schedule consistently degrades performance across all evaluated benchmarks, indicating that a gradual shift from exploration to exploitation is important for stable long-horizon refinement. Disabling trajectory guidance leads to a larger drop, especially on MathVision, suggesting that the retrieved-and-voted trajectory prior provides useful global structure that improves the quality of block-level proposals and subsequent MCMC refinement. Notably, this sensitivity to temperature is amplified by our relatively strong alignment target: overly aggressive exploration can cause block-wise updates to overfit local edits that score well under the target while drifting from a globally coherent solution, trapping the chain in suboptimal regions and reducing final accuracy. Overall, both components contribute to the final accuracy, with trajectory guidance being the primary driver and annealed sampling schedule offering complementary stabilization. 4.4. Ablation Study on Alignment Target Table 3 ablates our test-time alignment target (Eq. 15). Power sampling improves upon the reproduced Qwen2.5-VL-7B baseline across several datasets, indicating that sharpening the sequence distribution is a strong training-free lever. Incorporating the vision-aware term yields further gains over power sampling on every dataset except MathVista, aligning with its role in suppressing language-prior amplification under weak or ambiguous visual evidence. Adding entropy regularization alone consistently underperforms power sampling, suggesting a failure mode in which low-entropy trajectories are favored even when they encode confident mistakes, thereby limiting error correction. Coupling it with the vision-aware term alleviates this issue by filtering unsupported tokens and stabilizing visually grounded refinement. Among all ablations, the joint objective closely matches the best setting. Adding the linguistic control term yields further gains, most notably on MathVista. Results on MMStar (typically short) and MathVision (often long) suggest that a fixed length prior is not universally optimal, and the parameters L should instead be calibrated to the length regime of the target dataset. 4.5. Generalization of Alignment Target To further assess the generalizability of our refinement target, we evaluate it on two additional LVLMs, InternVL2.5-8B and Qwen3-VL-8B, with results summarized in Table 4. We use the same core experimental settings as for Qwen2.5-VL-7B to test whether the refinement target transfers across model families. In particular, both the visual calibration weight (β=2.0β=2.0) and the entropy weight (λH=1.0 _H=1.0) remain effective without retuning. The only parameter that requires mild model-specific adjustment is the linguistic control saturation level L, likely due to differences in linguistic priors and response verbosity. Even so, our method produces consistent improvements across base architectures. Overall, these findings indicate that the proposed synergistic alignment target generalizes well across models and serves as a practical plug-and-play component for improving LVLM reasoning. Figure 4. Likelihood and entropy of Qwen2.5-VL-7B responses on MathVista, averaged by output length. Table 5. Response length and token consumption on three datasets. Multipliers (shown after ×) report token consumption relative to greedy decoding (×1.0× 1.0). Method MathVista MMStar MathVision Response Length Greedy Decoding 242 184 566 Multi-Objective Sampling 216 164 401 TG Structured Sampling 235 187 528 Token Consumption Greedy Decoding 0,24201.0 × 0,18401.0 × 00,566 01.0 × Multi-Objective Sampling 5,66523.4 × 4,28223.3 × 12,840 22.3 × TG Structured Sampling 3,71015.3 × 2,96416.2 × 09,904 17.5 × 4.6. Reasoning Trace Likelihood and Entropy Figure 4 shows the kernel density estimates of average token entropy and log-likelihood for Qwen2.5-VL-7B responses on MathVista under power-scaled sampling (Power), our full refinement target (Ours), our target without the linguistic term (Ours w/o L), and vanilla decoding (Base). By design, both power-scaled sampling and our method shift samples toward higher-likelihood, lower-entropy regions of the base model, favoring more confident continuations. “Ours (w/o L)” achieves the lowest entropy with likelihood comparable to power sampling, highlighting the effect of visual sharpening and entropy regularization. Compared with pure power sampling, “Ours” retains slightly higher entropy and slightly lower sequence likelihood, reflecting the intended multi-objective trade-off. Figure 5. Illustrative case study. 4.7. Output Length and Inference Cost Response length. We do not observe a tendency toward longer responses under multi-objective sampling, as suggested by Karan and Du (2025). Instead, we observe a negative correlation where response length decreases as the alignment objective becomes stronger. This trend indicates that enhanced alignment effectively suppresses linguistic redundancy and filler tokens inherent in vanilla decoding. Rather than relying on verbosity, our sampling objective directs the model toward the most discriminative reasoning steps, thereby increasing information density while preserving performance. By comparison, trajectory-guided structure sampling tends to produce longer responses than multi-objective sampling. Token consumption. We define token consumption as the average number of output tokens generated per query. With visual calibration enabled, each output token incurs two forward passes, and we therefore count each token twice when computing token consumption. As shown in Table 5, sampling from the unnormalized target distribution incurs substantial inference-time compute: multi-objective sampling increases token consumption by about 2222–24×24× over greedy decoding. In contrast, trajectory-guided structured sampling reduces token consumption to 1515–18×18× and achieves a 1.31.3–1.5×1.5× reduction relative to multi- objective sampling, while exhibiting longer response length. 4.8. Case Study Figure 5 visualizes the initial reasoning state at each refinement iteration. Since iteration i inherits the accepted refinement from the previous cycle, consecutive panels reveal how the trajectory evolves through local updates. Iteration 1 establishes the task, while Iteration 2 extends the trajectory with Visual Detection but omits the target brown cylinder, leaving the object inventory incomplete. The initial state of Iteration 3 restores the missing object while preserving the earlier task context. However, its Logical Reasoning remains incomplete because the subtraction does not account for the brown cylinder. Iteration 4 retains the corrected detection and refines the reasoning into explicit steps, 7−1=67-1=6 and 6−1=56-1=5, ultimately yielding the correct answer. This illustrates the capacity of our MCMC-based refinement to incrementally correct local inconsistencies while preserving the overall reasoning context. 5. Related Work 5.1. Test-Time Alignment Methods Test-time alignment adapts a frozen model to new objectives, distribution shifts, or evolving user preferences at inference time rather than updating parameters, encompassing a broad range of methods unified by post-hoc output steering. First, hypothesis reweighting (Lee et al., 2025) leverages a pre-trained diverse ensemble of model heads and dynamically fits mixture weights at inference time using a small target-domain adaptation set. Second, reward-guided decoding (Xu et al., 2025a) achieves efficient test-time alignment by reshaping token-level probabilities with an autoregressive reward model. Recent advances (Lin et al., 2025) further extend this paradigm by employing a single preference-conditioned model, which overcomes the inefficiency of model ensembling and allows different alignment criteria to be dynamically balanced on the fly. Third, control and planning approaches cast decoding as an inference-time optimization problem, using sampling-based control in pre-logit space (Kanai et al., 2025) or subgoal-guided predictive planning to maintain long-horizon consistency (Wang et al., 2025c). Motivated by this view, we propose a sampling-based test-time alignment method that refines LVLM generations via trajectory-guided structured MCMC sampling. 5.2. MCMC-based Autoregressive Sampling Several works have revisited classical Monte Carlo inference as a test-time alternative to training-time alignment, combining MCMC-style updates with autoregressive generation. For instance, Zhao et al. (2024) cast controlled generation as probabilistic inference and introduce twisted Sequential Monte Carlo, learning prefix-level twists to allocate particles toward high-reward continuations. Alternatively, Faria et al. (2024) employ Metropolis–Hastings decoding to target a metric-induced Gibbs distribution via iterative accept-reject refinement with autoregressive proposals. More recently, Karan and Du (2025) propose sampling from a power-scaled distribution defined directly by the base model itself. Our work is the first to investigate these principles for LVLMs. We introduce an efficient block-wise Metropolis–Hastings refinement steered by retrieved trajectories, significantly reducing the overhead of long-horizon generation. To ensure generation quality, we further enhance the sampling target with a synergistic objective. 6. Conclusion This paper presents a sampling-based test-time alignment framework for LVLMs. Our approach is based on an automated, agent-driven trajectory learning algorithm to build a reasoning memory bank, where each trajectory captures high-level reasoning patterns for solving complex multimodal problems. To address the computational challenges of long-horizon generation, we propose trajectory-guided structured sampling, which enables localized refinement instead of costly full-sequence resampling. The retrieved guidance trajectory provides a global prior over reasoning structure and step order, making local refinement practical while preserving overall coherence. On top of this sampler, our synergistic alignment objectives steer generation toward visually grounded, reliable, and non-degenerate outputs. Experiments on five challenging datasets demonstrate that the framework achieves consistent performance gains without parameter updates, while substantially reducing token consumption relative to full-sequence resampling. Overall, our results position trajectory-guided test-time sampling as a practical and efficient paradigm for alignment in complex multimodal reasoning. Acknowledgements.This work was supported by the New Generation Artificial Intelligence National Science and Technology Major Project (Grant No. 2025ZD0123402), the Computational Biology Program (Grant No. 25JS2830402) of Science and Technology Commission of Shanghai Municipality (STCSM), and the Shanghai Municipal Science and Technology Major Project (Grant No. 2025SHZDZX025G06). References M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, J. R. Lee, Y. T. Lee, Y. Li, W. Liu, C. C. T. Mendes, A. Nguyen, E. Price, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, X. Wang, R. Ward, Y. Wu, D. Yu, C. Zhang, and Y. Zhang (2024) Phi-4 technical report. External Links: 2412.08905, Link Cited by: §2. S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu (2025a) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: 1st item, Table 1, Table 4. S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025b) Qwen2.5-vl technical report. External Links: 2502.13923, Link Cited by: §1, 1st item, Table 1. A. Brown and J. S. Rosenthal (2025) Upper and lower bounds on the subgeometric convergence of adaptive markov chain monte carlo. External Links: 2411.17084, Link Cited by: §1, §3.2. L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, and F. Zhao (2024) Are we on the right way for evaluating large vision-language models?. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §4.1.1. Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, L. Gu, X. Wang, Q. Li, Y. Ren, Z. Chen, J. Luo, J. Wang, T. Jiang, B. Wang, C. He, B. Shi, X. Zhang, H. Lv, Y. Wang, W. Shao, P. Chu, Z. Tu, T. He, Z. Wu, H. Deng, J. Ge, K. Chen, K. Zhang, L. Wang, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, and W. Wang (2025) Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. External Links: 2412.05271, Link Cited by: 1st item, Table 1, Table 4. G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, et al. (2025) The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. Cited by: §3.3.2. Y. Deng, H. Bansal, F. Yin, N. Peng, W. Wang, and K. Chang (2025) OpenVLThinker: complex vision-language reasoning via iterative sft-rl cycles. External Links: 2503.17352, Link Cited by: 2nd item, Table 1. G. Faria, S. Agrawal, A. Farinhas, R. Rei, J. G. C. de Souza, and A. Martins (2024) QUEST: quality-aware metropolis-hastings sampling for machine translation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §5.2. N. Hu, X. Duan, J. Zhang, and G. Kang (2025) Enhancing visual reliance in text generation: a bayesian perspective on mitigating hallucination in large vision-language models. In Proceedings of the 33rd ACM International Conference on Multimedia, M ’25, New York, NY, USA, p. 4778–4787. External Links: ISBN 9798400720352, Link, Document Cited by: §3.3.1. W. Huang, B. Jia, Z. Zhai, S. Cao, Z. Ye, F. Zhao, Z. Xu, Y. Hu, and S. Lin (2025) Vision-r1: incentivizing reasoning capability in multimodal large language models. External Links: 2503.06749, Link Cited by: 2nd item, Table 1. F. Huo, W. Xu, Z. Zhang, H. Wang, Z. Chen, and P. Zhao (2025) Self-introspective decoding: alleviating hallucinations for large vision-language models. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §3.3.1. S. Kanai, T. Yoshida, H. Takahashi, H. Kuroki, and K. Hashimoto (2025) Test-time alignment of llms via sampling-based optimal control in pre-logit space. External Links: 2510.26219, Link Cited by: §1, §5.1. A. Karan and Y. Du (2025) Reasoning with sampling: your base model is smarter than you think. External Links: 2510.14901, Link Cited by: §A.4, §B.2, §1, §2, §4.1.3, §4.7, §5.2. Y. Lee, J. Williams, H. Marklund, A. Sharma, E. Mitchell, A. Singh, and C. Finn (2025) Inference-time alignment via hypothesis reweighting. In Second Workshop on Test-Time Adaptation: Putting Updates to the Test! at ICML 2025, External Links: Link Cited by: §1, §5.1. S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing (2024) Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding . In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , Los Alamitos, CA, USA, p. 13872–13882. External Links: ISSN , Document, Link Cited by: §3.3.1. M. Li, N. Zhang, C. Fan, H. Jiao, Y. Fu, S. Peters, Q. Xu, R. Lissitz, and T. Zhou (2025a) Understanding the thinking process of reasoning models: a perspective from schoenfeld’s episode theory. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 18278–18299. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: §3.1. S. Li, Z. Lin, S. Yang, J. Zhao, and W. Chen (2025b) ThinkPilot: steering reasoning models via automated think-prefixes optimization. External Links: 2510.12063, Link Cited by: §3.1. Z. Li, W. Yu, C. Huang, R. Liu, Z. Liang, F. Liu, J. Che, D. Yu, J. Boyd-Graber, H. Mi, and D. Yu (2025c) Self-rewarding vision-language model via reasoning decomposition. External Links: 2508.19652, Link Cited by: §3.1. B. Lin, W. Jiang, Y. Xu, H. Chen, and Y. Chen (2025) PARM: multi-objective test-time alignment via preference-aware autoregressive reward model. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §1, §5.1. Z. Liu, Y. Zang, X. Dong, P. Zhang, Y. Cao, H. Duan, C. He, Y. Xiong, D. Lin, and J. Wang (2025) MIA-DPO: multi-image augmented direct preference optimization for large vision-language models. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1. P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K. Chang, M. Galley, and J. Gao (2024) MathVista: evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §4.1.1. P. Lu, B. Chen, S. Liu, R. Thapa, J. Boen, and J. Zou (2025) OctoTools: an agentic framework with extensible tools for complex reasoning. External Links: 2502.11271, Link Cited by: §3.1. F. Meng, L. Du, Z. Liu, Z. Zhou, Q. Lu, D. Fu, T. Han, B. Shi, W. Wang, J. He, K. Zhang, P. Luo, Y. Qiao, Q. Zhang, and W. Shao (2025) M-eureka: exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. External Links: 2503.07365, Link Cited by: 2nd item, Table 1. N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller (1953) Equation of state calculations by fast computing machines. Journal of Chemical Physics 21 (6), p. 1087–1092. Cited by: §3.2. X. Tian, S. Zou, Z. Yang, M. He, F. Waschkowski, L. Wesemann, P. Tu, and J. Zhang (2025) More thought, less accuracy? on the dual nature of reasoning in vision-language models. External Links: 2509.25848, Link Cited by: §B.2. C. Wang, X. Chen, N. Zhang, B. Tian, H. Xu, S. Deng, and H. Chen (2025a) MLLM can see? dynamic correction decoding for hallucination mitigation. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1. H. Wang, C. Qu, Z. Huang, W. Chu, F. Lin, and W. Chen (2025b) VL-rethinker: incentivizing self-reflection of vision-language models with reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §3.1, 2nd item, Table 1. K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li (2024) Measuring multimodal mathematical reasoning with MATH-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, External Links: Link Cited by: §4.1.1. K. Wang, T. Chen, Y. H. Hung, G. Ko, S. Ding, Y. Wu, Y. F. Wang, C. H. Yang, W. Peng, and P. Hsieh (2025c) Test-time alignment for large language models via textual model predictive control. External Links: 2502.20795, Link Cited by: §1, §5.1. X. Wang, Z. Yang, C. Feng, H. Lu, L. Li, C. Lin, K. Lin, F. Huang, and L. Wang (2025d) SoTA with less: mcts-guided sample selection for data-efficient visual reasoning self-improvement. External Links: 2504.07934, Link Cited by: §3.1, 2nd item, Table 1. J. Wu, M. Feng, G. Zhai, S. Zhang, Z. Lian, F. Lv, P. Shao, R. Jin, Z. Wen, and J. Tao (2026) AStar: boosting multimodal reasoning with automated structured thinking. External Links: 2502.02339, Link Cited by: §3.1. Y. Xu, U. M. Sehwag, A. Koppel, S. Zhu, B. An, F. Huang, and S. Ganesh (2025a) GenARM: reward guided generation with autoregressive reward model for test-time alignment. External Links: 2410.08193, Link Cited by: §1, §5.1. Z. Xu, C. Liu, Q. Wei, J. Wu, J. Zou, X. E. Wang, Y. Zhou, and S. Liu (2025b) More thinking, less seeing? assessing amplified hallucination in multimodal reasoning models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §B.2. H. Yao, J. Huang, W. Wu, J. Zhang, Y. Wang, S. Liu, Y. Wang, Y. Song, H. Feng, L. Shen, and D. Tao (2025) Mulberry: empowering MLLM with o1-like reasoning and reflection via collective monte carlo tree search. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1. H. Yin, G. Si, and Z. Wang (2025) ClearSight: visual signal enhancement for object hallucination mitigation in multimodal large language models. p. 14625–14634. External Links: Document Cited by: §3.3.1. T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H. Zheng, and M. Sun (2024) RLHF-v: towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 13807–13816. External Links: Document Cited by: §1. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen (2024) MMMU: a massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, Cited by: §4.1.1. R. Zhang, D. Jiang, Y. Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K. Chang, P. Gao, et al. (2024) MathVerse: does your multi-modal llm truly see the diagrams in visual math problems?. arXiv preprint arXiv:2403.14624. Cited by: §4.1.1. R. Zhang, B. Zhang, Y. Li, H. Zhang, Z. Sun, Z. Gan, Y. Yang, R. Pang, and Y. Yang (2025) Improve vision language model chain-of-thought reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 1631–1662. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1. S. Zhao, R. Brekelmans, A. Makhzani, and R. Grosse (2024) Probabilistic inference in language models via twisted sequential monte carlo. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: §5.2. J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, Z. Gao, E. Cui, X. Wang, Y. Cao, Y. Liu, X. Wei, H. Zhang, H. Wang, W. Xu, H. Li, J. Wang, N. Deng, S. Li, Y. He, T. Jiang, J. Luo, Y. Wang, C. He, B. Shi, X. Zhang, W. Shao, J. He, Y. Xiong, W. Qu, P. Sun, P. Jiao, H. Lv, L. Wu, K. Zhang, H. Deng, J. Ge, K. Chen, L. Wang, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, and W. Wang (2025) InternVL3: exploring advanced training and test-time recipes for open-source multimodal models. External Links: 2504.10479, Link Cited by: §1, 1st item, Table 1. Appendix A Implementation Details A.1. Definition of Reasoning Patterns Table 6. Reasoning patterns with explicit interfaces. Reasoning pattern Inputs Outputs Task Initialization Question X A single task Decompose Problem A single task A list of subtasks Visual Detection A single task A list of information (derive from image) Knowledge Retrieval A single task A list of information (derive from parametric knowledge) Logical Reasoning A single task A list of information (derive from deduction) Self-Verification A list of information A list of verified information Final Conclusion A list of information Final answer Table 6 defines seven reasoning patterns as step functions with explicit input–output interfaces. Each step consumes either the question X, a single task, or an accumulated information list, and produces a task (or subtask list), an information list, or the final answer. Notably, while the table specifies the formal I/O signatures, the actual execution of each pattern also involves internal reasoning processes that transform inputs into outputs. This design turns free-form chain-of-thought into composable, well-structured trajectories, making complex behaviors such as decomposition and grounding easier to elicit and reuse. Such pseudo-code-style definitions also support our downstream pipeline. They make trajectories comparable at the step level for retrieval and voting, enable sliding-window sampling by regenerating selected patterns while keeping others fixed, and simplify library curation through lightweight checks on structured outputs. Moreover, when scaling to more patterns, these interfaces make it straightforward to introduce rule-based behavior filters that accept, reject, or reroute pattern executions based on their inputs/outputs, steering reasoning toward desired behaviors. Overall, the pattern interfaces yield interpretable trajectories that are easier to construct and are more amenable to step-aligned refinement. A.2. Entropy-Weighted Multi-View Retrieval For a query instance (V,X)(V,X), we initially retrieve a candidate pool of size M=100M=100 via question-only embedding similarity. To account for the heterogeneous information across modalities, we propose an adaptive re-ranking mechanism based on entropy-weighted multi-view fusion. We consider three complementary perspectives: textual semantic similarity stexts_text, visual consistency simgs_img, and lexical n-gram overlap sngs_ng. To dynamically modulate the importance of each view, we utilize an information-theoretic approach to quantify their respective utilities. Specifically, for each view i, we define the probability distribution pip_i over the candidate pool as: (16) pi(j)=si(j)+ϵ∑j′=1M(si(j′)+ϵ).p_i^(j)= s_i^(j)+ε _j =1^M (s_i^(j )+ε ). The reliability of each modality is then inversely proportional to its normalized entropy HiH_i: (17) Hi=−1logM∑j=1Mpi(j)logpi(j).H_i=- 1 M _j=1^Mp_i^(j) p_i^(j). Intuitively, a view that yields a highly skewed distribution (low entropy) is more informative for ranking than one with a uniform distribution (high entropy). The final retrieval score S(j)S^(j) is computed as a weighted sum ∑iwisi(j) _iw_is_i^(j), where wi∝(1−Hi)w_i (1-H_i). This ensures that the top-k results are selected based on the most reliable and discriminative signals for the given query. We use Qwen3-Embedding-0.6B as our text embedding model and clip-vit-base-patch32 as our image embedding model. A.3. Design of Difficulty Estimator After obtaining the guidance trajectory, we apply a simple rule-based estimator to route instances by difficulty. We first remove behaviors that do not directly advance the reasoning process, including Task Initialization, Self-Verification, and Final Conclusion. If the remaining trajectory contains more than one action, we treat the instance as non-trivial, suggesting that it likely requires multi-hop reasoning; we then perform block-wise sampling under trajectory guidance. Otherwise, we treat it as trivial and apply full-sequence MCMC without trajectory guidance, which avoids overthinking while improving inference efficiency. A.4. MCMC for Full-Sequence Sampling For full-sequence sampling, we employ the autoregressive MH sampler proposed by Karan and Du (2025). To ensure sampling efficiency within high-dimensional spaces, a sequence of maximum length T is partitioned into K blocks, each consisting of T/KT/K tokens. The refinement process follows K sequential cycles, during which the number of active blocks eligible for resampling and MH updates increases incrementally. The procedure terminates early if an end-of-sequence (EOS) token is generated. In our implementation, we set the maximum sequence length T=3072T=3072 and the number of blocks K=16K=16. Appendix B More Experiments B.1. Generation Entropy and Answer Accuracy Figure 6. Average smoothed entropy over timestep. In Figure 6, we plot the average smoothed token-level output entropy H over decoding trajectories on the ThinkLite-70k dataset. A consistent gap is observed throughout generation: trajectories that lead to correct answers exhibit markedly lower entropy than incorrect ones. While this trend suggests that entropy can serve as a proximal reliability signal for test-time refinement, it is not an infallible metric. The model may occasionally succumb to being confidently wrong, yielding low-entropy but factually incorrect outputs that stifle beneficial exploration. To mitigate this, our framework employs entropy as a regularization term coupled with a vision-aware calibration mechanism. This dual approach effectively filters out low-entropy hallucinations, leveraging entropy to stabilize the refinement process rather than as a binary criterion for correctness. B.2. Analysis on Response Length Table 7. Response length of different sampling objectives on MathVista and MMStar with Qwen2.5-VL-7B. Method MathVista MMStar Greedy Decoding 242 184 Power Sampling 229 171 Power Sampling + Visual + Entropy 218 164 Figure 7. Effect of MCMC refinement steps on MathVista and MMStar with Qwen2.5-VL-7B. Figure 8. Case study of different sampling variants. Table 7 reports the average response length across three sampling configurations. We observe a consistent trend where the introduction of additional alignment objectives leads to a reduction in average response length. First, naive power sampling leads to shorter responses, which contrasts with the findings of Karan and Du (2025). We attribute this to the tendency of LVLMs to lose attention to visual inputs as generation length increases (Xu et al., 2025b; Tian et al., 2025), leading to performance degradation. We leave a detailed study for future work. Vanilla decoding is frequently bottlenecked by linguistic redundancy, where high-probability but uninformative sequences dominate the output. By coupling visual evidence with entropy-based weighting, our method actively penalizes such filler tokens, redirecting the generation toward critical, discriminative reasoning steps. As a result, the refined trajectories deliver higher accuracy with remarkable conciseness, fundamentally enhancing the information density of the underlying reasoning. A qualitative case study is provided in Figure 8. Specifically, greedy decoding and power-scaled sampling tend to generate either irrelevant contextual fillers or hallucinatory descriptions that contradict the visual evidence, ultimately culminating in incorrect conclusions. In contrast, by filtering out these linguistically dominant but visually inconsistent paths, our method directs the model to converge on the correct answer with fewer tokens. This case illustrates that our approach effectively prunes redundant or divergent reasoning branches, ensuring that the generated trajectory remains both concise and strictly grounded in the provided image. B.3. Scaling Law on MCMC Steps NMCMCN_MCMC controls test-time scaling in our framework by determining the number of Metropolis–Hastings refinement steps performed under the alignment target. Each step corresponds to one propose–accept update, so increasing NMCMCN_MCMC allocates more decoding-time compute to sequence-level correction, while smaller NMCMCN_MCMC favors lower latency. Figure 7 shows that increasing NMCMCN_MCMC generally improves performance on both MMStar and MathVista, although the gains are not strictly monotonic at intermediate steps. On MMStar, accuracy rises from 61.8 at NMCMC=0N_MCMC=0 to 64.2 at NMCMC=8N_MCMC=8, with a slight drop to 64.0 at NMCMC=10N_MCMC=10, suggesting that most of the benefit is already realized within a moderate refinement budget. On MathVista, the trend is more variable at small step counts but becomes clearly positive at larger budgets, improving from 70.5 at NMCMC=0N_MCMC=0 to 73.2 at NMCMC=10N_MCMC=10. Overall, these results suggest a favorable test-time scaling trend with respect to NMCMCN_MCMC: allocating more MCMC refinement steps generally improves reasoning accuracy, although the returns become less smooth and dataset-dependent at intermediate budgets. B.4. Evaluation on Hallucination Benchmarks Table 8. Evaluation of hallucination on Qwen2.5-VL-7B. We report the F1F_1 score for POPE and CHAIR scores (lower scores indicate fewer hallucinations. ). Method POPE F1↑F_1 CHAIR↓S_S CHAIR↓I_I Greedy Decoding 85.9 38.8 9.2 Power Sampling 85.8 39.2 9.3 Power Sampling + Visual 86.1 38.6 9.2 To evaluate the effectiveness of vision-aware distribution sharpening in mitigating hallucinations, we perform a quantitative analysis across two standard benchmarks, as summarized in Table 8. We evaluate object hallucinations using POPE (li-etal-2023-evaluating), a VQA-based probing protocol, and CHAIR (rohrbach-etal-2018-object), which quantifies hallucinations in image captioning by cross-referencing generated tokens with ground-truth objects. As formulated in Equation (18) and (19), CHAIR measures hallucination at both the instance (CHAIRICHAIR_I) and sentence (CHAIRSCHAIR_S) levels: (18) CHAIRI=|hallucinated objects|all mentioned objects,CHAIR_I= |\hallucinated objects\|all mentioned objects, (19) CHAIRS=|captions with hallucinated objects|all captions.CHAIR_S= |\captions with hallucinated objects\|all captions. As reported in Table 8, power sampling with visual calibration yields marginal yet consistent improvements across the evaluated benchmarks. Specifically, our method achieves a POPE F1F_1 score of 86.1, slightly surpassing the greedy decoding baseline. Notably, while vanilla power sampling exhibits a slight increase in hallucination rates (e.g., 39.2 CHAIRSCHAIR_S), the integration of visual priors successfully mitigates this regression, reaching the lowest CHAIRSCHAIR_S of 38.6. These results suggest that our approach helps maintain factual alignment with the visual input. B.5. Hyperparameter sensitivity Figure 9. Hyperparameter sensitivity analysis on the MMStar dataset with Qwen2.5-VL-7B and InternVL2.5-8B. Figure 9 examines the sensitivity of Qwen2.5-VL-7B and InternVL2.5-8B to the calibration weight β and entropy weight λH _H on MMStar. Both models attain their highest accuracy at approximately β=2.0β=2.0 and λH=1.0 _H=1.0. Performance remains relatively stable under moderate deviations from these values, indicating that the method is not overly sensitive to either hyperparameter, although more extreme settings cause noticeable degradation. Importantly, all evaluated configurations consistently outperform their corresponding greedy baselines, demonstrating the robustness of the proposed method across a broad range of hyperparameter choices. Appendix C Visualization Figure 10. Visualization of vision-aware calibration weight s(rt)s(r_t) and length reward RL()R_L(Y) Figure 10 visualizes the two shaping functions used in our alignment target. The top panel shows the effective-length reward RL()R_L(Y) as a function of SL()S_L(Y). In all cases, the reward increases monotonically with effective length and gradually saturates at the target level L, encouraging sufficiently long reasoning traces while avoiding unbounded growth. The parameters a and b control the growth rate and saturation behavior: larger values lead to faster early growth and earlier saturation, while smaller values produce a smoother increase. In our experiments, we set a=0.05a=0.05, b=0.8b=0.8 for all settings. The bottom panel shows the vision-aware calibration weight s(rt)s(r_t) as a function of the alignment ratio rtr_t. The weight is bounded in [0,1][0,1], increases monotonically with rtr_t, and reaches 11 when rt≥1r_t≥ 1, so tokens that are better supported by visual evidence receive full sharpening. The parameter β controls the selectivity of this calibration: smaller β yields a smoother transition, whereas larger β suppresses weakly aligned tokens more aggressively. We use β=2.0β=2.0 in all experimental settings.