Paper deep dive
Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models
Siyao Chen, Jiakang Yuan, Jiaxin Wang, Tao Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 99%
Last extracted: 7/5/2026, 3:00:35 AM
Summary
The paper introduces T²VLA (Test-time VLA), an architecture-agnostic reinforcement learning framework designed for Vision-Language-Action (VLA) models. Unlike traditional RL methods that rely on external environmental feedback (e.g., success signals from simulators), T²VLA leverages the intrinsic correlation between a model's generation confidence and its physical execution success. The framework utilizes a 'Confidence-Driven Dual Expert Bootstrapping' mechanism, which employs a task-conditioned Local Pseudo-Expert for immediate exploration and a Global Expert Pool for long-term stability. Rewards are computed using Dynamic Time Warping (DTW) to measure trajectory similarity, enabling self-bootstrapping policy improvement without external supervision. Experiments on LIBERO and RoboTwin benchmarks demonstrate that T²VLA achieves significant performance gains and approaches oracle RL performance.
Entities (10)
Relation Signals (7)
T²VLA → compatiblewith → OpenVLA-OFT
confidence 100% · T²VLA can adapt to distinct VLA paradigms, including both OpenVLA-OFT and the π series.
T²VLA → employs → Dynamic Time Warping
confidence 100% · We employ Dynamic Time Warping (DTW) to evaluate sequence similarity
T²VLA → evaluatedon → LIBERO
confidence 100% · Extensive experiments on the LIBERO and RoboTwin benchmarks
T²VLA → evaluatedon → RoboTwin
confidence 100% · Extensive experiments on the LIBERO and RoboTwin benchmarks
T²VLA → updatesvia → Group Relative Policy Optimization
confidence 100% · the VLA policy is updated via Group Relative Policy Optimization (GRPO)
T²VLA → uses → Global Expert Pool
confidence 100% · and a Global Expert Pool for training stability.
T²VLA → uses → Local Pseudo-Expert
confidence 100% · leveraging a Local Pseudo-Expert for aggressive exploration
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) has become indispensable for pushing Vision-Language-Action Models (VLAs) beyond static imitation learning. However, existing RL methods typically require external environmental feedback, relying on predefined success signals to guide policy updates. In this work, we show that VLA models possess useful internal evaluative capabilities: in discrete-action VLAs, trajectories with higher generation confidence are significantly more likely to succeed. Based on this observation, we introduce T^2VLA (Test-time VLA), an architecture-agnostic test-time RL framework that enables VLA models to achieve self-bootstrapping policy improvement. Instead of relying on external rewards, T^2VLA leverages trajectory-level similarity to high-confidence expert demonstrations as an intrinsic reward signal. In addition, we propose a Confidence-Driven Dual Expert Bootstrapping mechanism, which dynamically balances a Local Pseudo-Expert for exploration and a Global Expert Pool for training stability. Extensive experiments on the LIBERO and RoboTwin benchmarks show that T^2VLA consistently outperforms supervised baselines and approaches oracle RL performance with ground-truth rewards, achieving effective improvement without external reward feedback. Furthermore, T^2VLA adapts to distinct VLA paradigms, including both OpenVLA-OFT and the pi series.
Tags
Links
- Source: https://arxiv.org/abs/2606.29892v1
- Canonical: https://arxiv.org/abs/2606.29892v1
Trouble viewing inline? Open PDF directly →
Full Text
82,241 characters extracted from source content.
Expand or collapse full text
Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models Siyao Chen 1 , Jiakang Yuan 1 , Jiaxin Wang 2 , and Tao Chen 1,2,3⋆ 1 College of Future Information Technology, Fudan University, Shanghai, China siyaochen25@m.fudan.edu.cn, eetchen@fudan.edu.cn 2 Shanghai Innovation Institute, Shanghai, China 3 MoShen Intelligence, Shanghai, China Abstract. Reinforcement learning (RL) has become indispensable for pushing Vision-Language-Action Models (VLA) beyond static imitation learning. However, existing RL methods typically necessitate external environmental feedback, relying on predefined success signals to guide policy updates. In this work, we demonstrate that VLA models possess strong internal evaluative capabilities: in discrete-action VLAs, trajecto- ries with higher generation confidence are significantly more likely to suc- ceed. Based on the observation, we introduce T 2 VLA (Test-time VLA), an architecture-agnostic test-time RL framework that enables VLA mod- els to achieve self-bootstrapping policy improvement. Instead of relying on external rewards, T 2 VLA leverages the trajectory-level similarity to high-confidence expert demonstrations as an intrinsic reward signal. In addition, we propose a Confidence-Driven Dual Expert Bootstrapping mechanism. By dynamically balancing a Local Pseudo-Expert for aggres- sive exploration and a Global Expert Pool for training stability, T 2 VLA prevents policy collapse while discovering further breakthroughs. Exten- sive experiments on the LIBERO and RoboTwin benchmarks show that T 2 VLA consistently outperforms supervised baselines and approaches oracle RL performance with ground-truth rewards, achieving effective improvement without external reward feedback. Furthermore, T 2 VLA can adapt to distinct VLA paradigms, including both OpenVLA-OFT and the π series. Keywords: Vision-Language-Action Model· Test-Time Reinforcement Learning· Intrinsic Reward 1 Introduction Driven by the rapid evolution of Vision-Language Models [11,18,25,34] (VLMs), Vision-Language-Action (VLA) models [3, 20, 50] have emerged as a transfor- mative paradigm in embodied AI. Recently, Reinforcement Learning [14,21,28] (RL) has become a research hotspot for VLA, as it effectively addresses the prohibitive data collection costs and inherent generalization bottlenecks associ- ated with traditional Supervised Fine-Tuning [4,29,41,48] (SFT). Despite their ⋆ Corresponding author. arXiv:2606.29892v1 [cs.RO] 29 Jun 2026 2Chen et al. （b）Our T²VLA Self-Evaluator Self-Generated Autonomous Bootstrapping Intrinsic Supervision VLA Trajectories Rollout External Reward External Evaluator External LLM Pre-defined Simulator Human Annotator ... （a）Conventional VLA Reinforcement Learning VLA Intrinsic Reward Explicitly Supervised RL External Reward Signals Trajectories Self-Bootstrapping RL Intrinsic Reward Only Env Env Rollout Fig. 1: Comparison of VLA Reinforcement Learning Paradigms. (a) Conven- tional VLA RL relies on explicitly supervised RL, where policy updates are driven by external reward signals from simulators, human annotators, or external LLMs. (b) T 2 VLA (Ours) introduces a self-bootstrapping RL approach. By leveraging a model- internal signal, T 2 VLA derives rewards to drive continuous policy adaptation without external reward supervision. remarkable success in various embodied tasks, existing RL approaches heavily rely on external supervision signals (e.g., environment-provided success flags), confining their efficacy to predefined scenarios. To break free from these predefined scenarios and unlock true autonomous self-improvement, it is crucial to empower VLA models to learn directly from their own unannotated trajectories. This transition towards the “Era of Expe- rience” [37] has recently been successfully validated in VLMs through test-time Reinforcement Learning [51] in mathematics and coding tasks, which enables models to self-optimize using purely unlabeled trajectories through majority voting. However, translating this success to VLA models exposes a critical gap: unlike math problems that yield a singular, discrete answer for easy verification, robotic manipulation requires generating continuous trajectories where multi- ple divergent action sequences can lead to the same goal, thereby eliminating a straightforward ground truth for self-reflection. To bridge this gap, inspired by entropy-based confidence estimation in Large Language Models [12,33,38], we carefully investigate whether VLA models pos- sess internal evaluative capabilities that can substitute for external verification. Our empirical analysis on discrete-action VLAs (Figure 3) reveals a strong pos- itive correlation between internal generation confidence and task success rates (i.e., trajectories with higher confidence are more likely to succeed). This obser- vation indicates that, for token-based VLA policies, the length-normalized mean log-probability of generated trajectories can serve as a useful indicator of phys- ical success, thereby enabling policy optimization without relying on external environmental rewards. Building upon this empirical insight, we introduce T 2 VLA (Test-time VLA), an architecture-agnostic framework for self-bootstrapping policy optimization, as illustrated in Figure 1. To eliminate reliance on extrinsic environmental rewards, we first propose a Confidence-Driven Dual Expert Bootstrapping mech- Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models3 anism, enabling the autonomous extraction of high-quality demonstrations di- rectly from the model’s own rollouts. Leveraging generative confidence, we elect a task-conditioned Local Pseudo-Expert to serve as an immediate optimization anchor, steering the policy toward the most promising behaviors discovered in the current iteration. Concurrently, we maintain a priority-based Global Expert Pool containing the top-K historical trajectories, which provides a stable be- havioral baseline to prevent policy regression. Subsequently, we formulate a self- bootstrapping proxy reward based on behavioral alignment with these discovered experts. We employ Dynamic Time Warping (DTW) [35] to evaluate sequence similarity across continuous robotic actions with varying execution horizons. The final trajectory reward is computed via an adaptive weighting strategy that dy- namically interpolates between local and global DTW similarities based on their relative confidence scores. Finally, the VLA policy is updated via Group Rela- tive Policy Optimization (GRPO) [36], leveraging the group-normalized proxy rewards to drive continuous, self-guided learning. Extensive experiments on multiple benchmarks including LIBERO [24] and RoboTwin 2.0 [9] demonstrate that T 2 VLA can serve as an architecture-agnostic framework that yields substantial gains on various embodied tasks without re- lying on external rewards. Besides, T 2 VLA is compatible with various VLA models such as OpenVLA-OFT [19,21] and π 0 [3]. Our main contributions can be summarized as follows: 1. We propose T 2 VLA, a self-bootstrapping test-time reinforcement learning framework that eliminates the requirement for external supervision signals. By leveraging the intrinsic correlation between VLA generation confidence and physical execution success, our approach enables self-bootstrapping pol- icy optimization without environmental reward signals. 2. We introduce a confidence-driven dual-expert mechanism that synergizes a Local Pseudo-Expert for aggressive exploration with a Global Expert Pool for long-term stability. Coupled with a DTW-based adaptive reward, this provides a reliable soft anchor that prevents error compounding during un- supervised policy updates. 3. We demonstrate the architecture-agnostic efficacy of T 2 VLA across diverse benchmarks. Relying solely on internal signals, our method achieves over a 20% absolute gain on continuous-action and bimanual tasks, and elevates the average success rate of discrete-action models to 97.2%. 2 Related Work 2.1 Vision-Language-Action Models Vision-Language-Action (VLA) models unify perception, reasoning, and con- trol by directly mapping multimodal visual observations and natural language instructions to physical actions [4,50]. Current VLAs broadly follow two archi- tectural paradigms based on their action representations. The first paradigm, 4Chen et al. Discrete-Action VLAs (e.g., RT-1 [4], RT-2 [50], and OpenVLA [20]), lever- ages pre-trained vision-language models (VLMs) to formulate robotic control as an autoregressive token generation problem. The second paradigm, Continuous- Action VLAs (e.g., Octo [41], RDT [27], π 0 [3], π 0.5 [17], and GR00T [2]), syn- thesizes continuous control trajectories conditioned on multimodal embeddings via diffusion models or flow matching techniques. The standard training recipe for both classes of models is grounded in large-scale Supervised Fine-Tuning (SFT) or behavioral cloning [29]. 2.2 Reinforcement Learning for Robotic Manipulation Reinforcement Learning (RL) has been widely adopted to further optimize VLA policies beyond standard SFT. Offline RL-VLA approaches (e.g., GeRM [39], Q-Transformer [6], ReinboT [46], CO-RFT [16]) extract policies strictly from static datasets using pre-annotated rewards. To enable active environmental in- teraction, Online RL-VLA frameworks optimize policies via closed-loop feed- back, extensively exploring policy gradient algorithms across diverse architec- tures (e.g., ThinkAct [15], RIPT-VLA [40], GRAPE [47], πRL [8], SimpleVLA- RL [21]). In these online methods, reward acquisition is primarily based on explicitly designed external supervision, such as sparse binary feedback from predefined simulators [8, 21] or dense proxy rewards engineered via auxiliary models [45]. Instead of requiring external reward signals, our proposed T 2 VLA framework achieves self-bootstrapped RL by relying entirely on intrinsic rewards. 2.3 Test-Time Learning and Adaptation Test-time optimization strategies for robotic manipulation broadly fall into two paradigms: learning and adaptation. Test-Time Learning [7, 42, 44] dynami- cally updates model representations or weights during inference. For example, VLS [26] and ADPro [22] optimize inference-time sampling via VLM-derived gradients and geometric constraints, while EVOLVE-VLA [1] continuously up- dates policy weights using learned progress estimators. Conversely, Test-Time Adaptation employs training-free mechanisms to guide pre-trained policies, typically through action re-ranking via external verifiers (e.g., V-GPS [30], VLA- Pilot [23]), MCTS-based planning (e.g., VLA-Reasoner [13], VLAPS [32]), or heuristic action filtering (e.g., TACO [43]). While these pipelines predominantly depend on auxiliary external evaluators, our T 2 VLA framework performs test- time RL using intrinsic rewards derived from model-internal signals. 3 Method In this section, we present T 2 VLA, a self-bootstrapping test-time RL frame- work for Vision-Language-Action (VLA) models (Figure 2). Following the prob- lem formulation and our empirical motivation (Section 3.1), we introduce a Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models5 A1 A2 VLA Policy input Rollout Trajectories 휏 1 ... Local Pseudo-Expert Global Experts Pool 휏 푙표푐푎푙 ∗ most confident trajectory is expert Expert Pool Maintenance Contains Top-K experts Per task Advantages A3 A4 A5 ... Task1: [Pick] ... Instruction Observation ... Pseudo-Expert Election Dual Experts Bootstrapping Hybrid Reward Calculation DTW Alignment & Similarity Dynamic Hybrid Reward Min-Max scaling similarity reward Hybrid Similarity Reward Fusion Trajectory Confidence w 1-w 푟 푖 푠푖푚 =w∗푆푖푚 푙표푐푎푙 +1−푤∗max(푆푖푚 푔푙표푏푎푙 ) 푆푖푚 푔푙표푏푎푙 푆푖푚 푙표푐푎푙 task1 task2 ... 휏 p 1 1 휏 p 1 2 휏 p 1 퐾 ... ... 휏 p 2 1 휏 p 2 2 휏 p 2 퐾 ... replace 퐿(휃) A1 A2 휏 2 휏 1 휏 1 휏 3 휏 4 휏 5 Fig. 2: Overview of the T 2 VLA Framework. Our pipeline autonomously mines behavioral anchors by identifying a Local Pseudo-Expert from exploratory roll- outs and maintaining a Global Expert Pool. These references are integrated via a DTW-based Hybrid Similarity Reward to compute advantages, enabling contin- uous policy optimization without external reward signals. confidence-driven dual-expert mechanism to autonomously mine behavioral an- chors directly from exploratory rollouts (Section 3.2). These anchors are sub- sequently translated into intrinsic signals via a DTW-based hybrid similarity reward (Section 3.3) to drive continuous policy optimization (Section 3.4). 3.1 Preliminaries Problem Formulation. Given an embodied task, the robotic manipulation pro- cess can be formulated as a Markov Decision Process (MDP). At each timestep t, the VLA model observes a state s t = (o t ,l), comprising a visual observation o t and a language instruction l, and generates an action a t according to a parame- terized policy π θ (a t |s t ). For discrete-action VLAs, the model outputs logits over discretized action tokens that are decoded into executable actions; for flow-based VLAs, actions are generated through iterative continuous refinement. In standard reinforcement learning for VLAs, the model interacts with the environment to collect a rollout trajectory τ = (s 1 ,a 1 ,...,s T ,a T ) of length T. Policy optimization proceeds by sampling a batch of rollouts τ i N i=1 , assigning each trajectory a scalar reward r i , estimating advantages, and updating θ via a policy-gradient objective. The standard formulation maximizes the expected external return: max θ E τ∼π θ [R env (τ )], (1) where R env (τ ) denotes the external environment reward. Consequently, exist- ing RL-based VLA optimization methods inherently rely on explicit external verification signals to guide policy updates. Empirical Motivation and Reformulation. To eliminate the reliance on external environmental reward R env (τ ), we investigate whether VLA models 6Chen et al. Success Rate Confidence Score (Mean Log-Prob) 0 0.2 0.4 0.6 0.8 1 -60-50-40-30-20-15 LIBERO-Spatial (Trend) LIBERO-Object (Trend) Fig. 3: Correlation between VLA generation confidence and task success rate. Analysis of 2, 000 OpenVLA-OFT rollouts on LIBERO suites reveals that higher mean log-probabilities consistently correlate with execution success, validating internal confidence as a reliable foundation for autonomous self-bootstrapping. can inherently evaluate their physical execution. As shown in Figure 3, empir- ical analysis of discrete-action VLAs reveals a strong positive correlation be- tween trajectory-level generation confidence (defined as the mean action log- probability) and execution success. Trajectories assigned higher likelihood by the model consistently achieve higher task success rates. This observation in- dicates that the model’s intrinsic confidence provides a robust signal regarding behavioral quality, suggesting its potential as a foundation for self-bootstrapping. In light of this observation, we propose T 2 VLA, which optimizes the VLA policy as a self-bootstrapping reinforcement learning process during test time. Instead of relying on external environment rewards, we replace the return with an intrinsic reward R self (τ ) computed solely from signals produced by the model itself. Consequently, the optimization objective transitions to: max θ E τ∼π θ [R self (τ )]. (2) This paradigm shift introduces a primary challenge: how to formulate an effective trajectory-level intrinsic reward R self (τ ) without external verification signals. While empirical evidence links confidence with task success, adopting the raw generation confidence directly as a standalone reward scalar proves in- adequate for stable policy optimization. This prompts a critical question: could these high-confidence trajectories themselves serve as reliable anchors to provide effective guidance signals? To answer this, we decouple the overall challenge into two fundamental sub-problems: (i) How to reliably extract and maintain high- quality reference trajectories from exploratory rollouts. (i) How to formulate a robust trajectory reward design based on these established references. Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models7 3.2 Confidence-Driven Dual Expert Bootstrapping To extract high-quality reference trajectories without external verification, we propose a Confidence-Driven Dual Expert Bootstrapping mechanism. This module is designed to autonomously mine and maintain behavioral an- chors from the model’s exploratory rollouts by leveraging internal confidence signals. Specifically, it consists of two primary components: the election of a task-conditioned Local Pseudo-Expert from the current batch, and the mainte- nance of a dynamic Global Expert Pool to preserve historical best practices. Length-Normalized Confidence Estimation. During exploration, the active policy π θ generates a batch of N trajectories D = τ 1 ,...,τ N . For discrete-action VLAs, directly summing action log-probabilities inherently penal- izes long-horizon trajectories and may favor degenerate, short-horizon behaviors. We therefore compute a length-normalized confidence score. For a trajectory τ i with an effective execution horizon T i , determined by environment termination and strictly excluding padding tokens, its confidence score is defined as the mean action log-probability: c disc i = 1 T i T i X t=1 logπ θ (a i,t | s i,t ).(3) This normalization enables confidence scores to be compared across trajectories with different execution horizons. For continuous flow-based VLAs, confidence is estimated from the Gaussian likelihood of transitions along the denoising process. Let L i denote the effective rollout length, measured by the number of valid action-chunk predictions before environment termination. At each prediction step j, we compute ℓ flow i,j = X h,d logN x k+1 i,j,h,d ;μ θ (x k i,j ,k,s i,j ) h,d ,σ θ (x k i,j ,k,s i,j ) h,d ,(4) where x k i,j is the intermediate denoising state at the selected denoising step k, while h and d index the action-chunk horizon and action dimension, respectively. The trajectory-level confidence is then computed as c flow i = 1 L i L i X j=1 ℓ flow i,j .(5) This likelihood characterizes denoising consistency and provides an effective ranking signal for pseudo-expert election. The elected trajectories subsequently serve as behavioral references for computing intrinsic rewards. For simplicity, we use c i to denote the corresponding trajectory-level confi- dence score, either c disc i or c flow i , in the following expert election process. Task-Conditioned Local Expert Election. Since generative confidence strongly correlates with execution success (Section 3.1), the confidence score c i provides a reliable basis for identifying trajectories that are more likely to achieve 8Chen et al. the task. By leveraging c i to anchor the most confident executions from the exploratory rollouts, we can extract behavioral references to drive iterative policy improvement. To steer suboptimal exploratory behaviors toward these successful modes, we formally designate the most confident trajectory as a pseudo-expert. However, during the RL training loop, a sampled batch typically encompasses trajectories across diverse environments and manipulation tasks. To ensure a fair and task-specific evaluation, we first group the trajectories based on their corresponding language instruction l. Within each task-specific subset D l ⊆ D, the Local Pseudo-Expert is explicitly elected as: τ ∗ local,l = arg max τ i ∈D l c i (6) Thus, this local expert encapsulates the most reliable behavioral mode discovered for the specific task during the current iteration, serving as a dynamic, on-policy target for immediate alignment. Dynamic Global Expert Pool. Relying exclusively on the local batch expert can occasionally introduce optimization instability, particularly if an ex- ploratory batch happens to yield overall suboptimal rollouts. To maintain train- ing stability and preserve previously discovered high-quality behaviors, we con- currently maintain a task-conditioned dynamic Global Expert Pool, denoted as P l . To ensure the high quality of this historical knowledge base, we restrict the pool’s entry strictly to the elected local batch experts τ ∗ local,l , filtering out the remaining suboptimal exploratory trajectories. At each training iteration, the newly elected local expert τ ∗ local,l is integrated into P l . The pool acts as a priority-based memory buffer, retaining only the top-K historical experts (with capacity K = 5 in our implementation) sorted by their confidence scores c: P l ← Top-K P l ∪τ ∗ local,l based on c(7) By bounding the capacity, this mechanism evicts stale behaviors from outdated policy, ensuring the pool remains aligned with the evolving policy distribution. 3.3 DTW-Based Hybrid Similarity Reward To address the second sub-problem of rewarding exploratory rollouts, we design a DTW-based Hybrid Similarity Reward. This mechanism is designed to translate expert guidance into informative signals by aligning trajectories and balancing expertise sources. Specifically, it is composed of a DTW-based trajec- tory alignment to handle temporal variations, and an adaptive weighting scheme to balance current batch consensus with historical best practices. Trajectory Alignment via DTW. The stochastic generative nature of VLAs frequently yields action sequences with varying temporal horizons. Stan- dard Euclidean distance enforces strict one-to-one temporal synchronization, causing massive distance penalties under minor temporal shifts even when spa- tial paths are identical. As visualized in Figure 5, such rigid point-wise matching Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models9 fails to capture true structural similarity. To decouple spatial geometry from temporal progression, we employ Dynamic Time Warping (DTW) [35]. By non- linearly warping the time axis to enable many-to-one state mappings, DTW aligns trajectories based on their spatial morphology rather than strict temporal indices, providing a robust similarity measure for heterogeneous rollouts. Let τ A = (a A,1 ,...,a A,M ) and τ B = (a B,1 ,...,a B,T ) denote two action se- quences of continuous end-effector poses a ∈ R d . To prevent dimensions with large physical magnitudes from dominating the metric, all action dimensions are independently min-max normalized to [0, 1]. We construct a cost matrix D ∈ R M×T via dynamic programming: d i,j =∥a A,i − a B,j ∥ 2 + min(d i−1,j ,d i,j−1 ,d i−1,j−1 )(8) The cumulative distance D DTW (τ A ,τ B ) = d M,T is then normalized by the maxi- mum temporal length to ensure length-invariance and yield a bounded similarity score in (0, 1]: Sim DTW (τ A ,τ B ) = 1 1 + D DTW (τ A ,τ B ) max(M,T ) (9) Dynamic Hybrid Reward Design. To adaptively determine the trust placed in the local batch expert, we compute the alignment of each exploratory trajectory τ i (corresponding to task l) against both its task-specific local expert τ ∗ local,l and the global pool P l . The dynamic hybrid similarity reward r sim i in- corporates the maximum similarity score among all historical experts τ p ∈ P l : r sim i = w· Sim DTW (τ i ,τ ∗ local,l ) + (1− w)· max τ p ∈P l Sim DTW (τ i ,τ p )(10) The interpolation weight w ∈ [0, 1] balances current batch consensus and his- torical best practices by normalizing the local expert’s confidence score c ∗ local,l : w = clip c ∗ local,l − c min,l c max,l − c min,l + ε , 0, 1 (11) where c max,l and c min,l represent the maximum and minimum confidence scores currently maintained within the task-specific pool P l , and ε ensures numerical stability. This relative measure ensures that highly confident local experts (w → 1) prioritize immediate on-policy discoveries, while uncertain batches (w → 0) smoothly fall back to historical experts to prevent policy degradation. A detailed analysis validating the necessity of this continuous Min-Max scaling over rigid or hard-gating alternatives is provided in Sec. 4.4. KL Penalty. To prevent over-optimization and maintain the valid behavioral distribution learned during pretraining, we incorporate a Kullback-Leibler (KL) divergence penalty. The final proxy reward r i for trajectory τ i is: r i = r sim i − β T i X t=1 D KL (π θ (·|s i,t )∥π ref (·|s i,t )) (12) where the initial SFT model π ref acts as a behavioral anchor, and β controls the penalty strength. 10Chen et al. 3.4 Policy Optimization via GRPO Given the proxy rewards r 1 ,...,r N for the N sampled trajectories, GRPO computes a baseline-free advantage A i via group normalization: A i = r i − μ(r) σ(r) + ε (13) where μ(r) and σ(r) denote the batch mean and standard deviation. The policy is then updated by maximizing the clipped surrogate objective: L(θ) = 1 N N X i=1 T i X t=1 min (ρ i,t (θ)A i , clip(ρ i,t (θ), 1− ε clip , 1 + ε clip )A i ) (14) where ρ i,t (θ) = π θ (a i,t |s i,t ) π old (a i,t |s i,t ) and ε clip bounds the policy update. By iteratively maximizingL(θ), the policy π θ is continuously aligned with the extracted expert guidance, completing the self-bootstrapping loop. 4 Experiments 4.1 Experimental Setup Benchmarks. We evaluate our framework on two robotic manipulation bench- marks: (1) LIBERO [24]: We evaluate the policy’s robustness and long-horizon planning capacity across four diverse task suites (Spatial, Object, Goal, and Long). (2) RoboTwin 2.0 [9]: We assess bimanual coordination across five tasks with varying execution horizons (100-650 steps). Baseline Models. To demonstrate the architecture-agnostic applicability of T 2 VLA, we instantiate our framework across two generative paradigms: (1) Discrete-Action VLAs: We adopt OpenVLA-OFT [19]. We use the modified architecture and SFT weights provided by SimpleVLA-RL [21], which optimizes the decoding head for RL compatibility. (2) Continuous-Action VLAs: We evaluate flow-matching policies π 0 [3] and π 0.5 [17] by adopting π RL ’s [8] SFT weights and MDP to compute confidence from denoising log-likelihoods. Implementation Details. We optimize T 2 VLA using GRPO [36] with AdamW (peak LR 5× 10 −6 ; cosine schedule for OpenVLA-OFT, constant for π 0 /π 0.5 ). We sample N = 8 trajectories per instruction at temperature 1.6 (OpenVLA-OFT) or 1.0 (π 0 /π 0.5 ). Policy updates apply a universal initial KL penalty β = 0.02. The PPO clip ratio is [0.2, 0.28] for OpenVLA-OFT, and 0.2 for π 0 /π 0.5 (with gradient clipping at 2.0 and 1.0, respectively). Execution hori- zons are capped at 500 steps (OpenVLA-OFT) and 480 steps (π 0 /π 0.5 , max 1024 tokens), with π 0.5 using 4 denoising steps. 4.2 Main Results In this section, we present the main evaluation results on the LIBERO and RoboTwin benchmarks. To verify the effectiveness of our method, we compare Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models11 Table 1: Main Results on the LIBERO Benchmark. Success rates (%) are reported. ∆ indicates the absolute improvement over the baseline. MethodRewardSpatial Object Goal Long Avg. Reference: Prior SFT, RL, & Test-Time Training Baselines Octo [41]None (SFT)78.9 84.6 85.7 51.1 75.1 OpenVLA [20]None (SFT)84.7 79.2 88.4 53.7 76.5 UniVLA [5]None (SFT)96.5 96.8 95.6 92.0 95.2 VLA-RL [28]Env. Success 90.2 94.3 91.8 82.2 89.6 SimpleVLA-RL [21]Env. Success 99.4 99.1 99.2 98.5 99.1 EVOLVE-VLA [1]Learned Critic 95.4 97.4 95.8 94.4 95.8 Our Framework (Self-Rewarding) OpenVLA-OFT [19,21] None (SFT)91.6 95.3 90.6 86.5 91.0 Ours (OpenVLA-OFT) Self-Reward 97.7 99.6 96.1 95.3 97.2 Improvement (∆)+6.1+4.3+5.5+8.8+6.2 π 0 (Base) [3,8]None (SFT)65.3 64.4 49.8 51.2 57.7 Ours (π 0 )Self-Reward 86.3 91.0 82.0 68.0 81.9 Improvement (∆)+21.0+26.6+32.2+16.8+24.2 π 0.5 (Base) [8,17]None (SFT)84.6 95.4 84.6 43.9 77.1 Ours (π 0.5 )Self-Reward 94.9 98.4 91.8 55.1 85.1 Improvement (∆)+10.3+3.0+7.2+11.2+8.0 T 2 VLA against a diverse spectrum of baselines, including prior SFT methods (i.e., UniVLA [5]), explicitly supervised RL methods (i.e., VLA-RL [28] and SimpleVLA-RL [21]), and test-time training (T) approaches (i.e., EVOLVE- VLA [1]). Further, to demonstrate the advantage of our self-bootstrapping mech- anism, we directly compare the performance of our method against its respective base policies under standard data regimes: OpenVLA-OFT is fine-tuned with full demonstrations (Traj-all), while both π 0 and π 0.5 utilize few-shot initializa- tion. Table 1 details our absolute performance gains on LIBERO, categorized by reliance on external rewards. Additionally, Table 2 reports the framework’s effectiveness in bimanual control scenarios across varying execution horizons. Results on the LIBERO Benchmark. Table 1 demonstrates that T 2 VLA (based on OpenVLA-OFT) delivers consistent gains across all four task suites, improving upon the OpenVLA-OFT baseline by +6.2% on average and outper- forming the state-of-the-art SFT model, UniVLA [5] (97.2% vs. 95.2%). Notably, despite operating entirely without external reward models or environmental su- pervision, T 2 VLA surpasses the concurrent EVOLVE-VLA [1] (95.8%), which relies on a separate foundation critic. Moreover, it substantially reduces the per- formance gap to the oracle-guided SimpleVLA-RL [21] (99.1%), which has access to true environmental feedback. These results further demonstrate that intrinsic 12Chen et al. Table 2: Results on RoboTwin 2.0. Success rates (%) using OpenVLA-OFT. Tasks are categorized by execution horizon length. Task Category & NameSFT BaselineOursImprovement (∆) Short Horizon (100-130 Steps) Lift Pot10.139.8+29.7 Beat Hammer28.168.0+39.9 Medium Horizon (150-230 Steps) Place Empty Cup77.384.8+7.5 Long & Extra Long Horizon (280-650 Steps) Handover Block33.144.1+11.0 Stack Bowls40.659.0+18.4 Average37.859.1+21.3 generative log-probabilities can provide a reliable bootstrapping signal that can boost the performance on different embodied tasks. In addition, to verify the broad applicability of T 2 VLA, we extend our eval- uation to continuous flow-matching models. As detailed in Table 1, our approach yields a substantial 24.2% absolute improvement in the average success rate of π 0 [3,8]. Additionally, it boosts the performance of π 0.5 [8,17] on the Goal suite from 84.6% to 91.8%. These results confirm that T 2 VLA is an architecture- agnostic approach that can be flexibly combined with various VLA models. Results on the Bimanual Scenario (RoboTwin 2.0). To validate the versatility of T 2 VLA in adapting to different tasks, we further conduct exper- iments on the RoboTwin 2.0 benchmark. As shown in Table 2, T 2 VLA consis- tently outperforms the SFT baseline across diverse execution horizons. Specifi- cally, it achieves a remarkable performance leap on short-horizon tasks like Beat Hammer (i.e., from 28.1% to 68.0%). For complex long-horizon tasks such as Stack Bowls, it can also achieve an 18.4% absolute improvement. This demon- strates that the DTW-based self-bootstrapping mechanism scales effectively to high-dimensional control scenarios without environment-provided rewards. 4.3 Ablation Studies To investigate the impact of our core designs, we conduct ablation studies on the LIBERO-Long benchmark using OpenVLA-OFT [19,21]. Effectiveness of Dual Expert Bootstrapping. We first investigate the role of the proposed dual-expert bootstrapping mechanism. As detailed in Ta- ble 3, relying exclusively on the local expert captures recent high-confidence behaviors (94.5%) but remains vulnerable to occasional suboptimal generations within a single batch. Conversely, utilizing only the global pool provides sta- ble historical references (93.0%) but slows down continuous adaptation, as the optimization tends to rely heavily on older trajectories. Integrating both mod- ules effectively balances these aspects, achieving the highest success rate (95.3%). Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models13 Table 3: Dual Expert Ablation. Suc- cess rates (%) on LIBERO-Long. The dual-expert design achieves the best per- formance. MethodSR (%) Local Expert Only94.5 Global Expert Only93.0 Dual Expert (Ours) 95.3 Fig. 4: Impact of Expert Pool Capac- ity (K). Evaluated on LIBERO-Long, where K = 5 yields the best performance. Overall, these results indicate that the local and global experts are highly comple- mentary: the former steers the policy toward newly discovered successful modes, while the latter serves as a stabilizing anchor to prevent policy degradation when current rollouts are poor. Their synergy yields a balanced, progressive reward signal that drives consistent performance improvements. Impact of Expert Pool Capacity. We further analyze the sensitivity to the global expert pool capacity (K). As illustrated in Figure 4, all evaluated pool capacities consistently exceed the performance of the Base model (86.5%), val- idating the general effectiveness of our self-bootstrapping framework. Notably, performance peaks at a moderate capacity (K = 5, 95.3%), suggesting a trade- off between reference diversity and quality control. A restricted pool (K = 3, 93.4%) offers insufficient coverage of diverse successful behavioral modes, po- tentially limiting the richness of the supervision signal. On the other hand, an overly expansive pool (K = 10, 91.4%) retains stale trajectories from earlier, less-proficient training stages. The persistence of these outdated references can introduce suboptimal anchors into the reward calculation, which dilute the qual- ity of the guidance signal and impede learning progress. Thus, K = 5 provides an effective balance, maintaining sufficient historical diversity while ensuring the pool remains aligned with high-quality behavioral standards. 4.4 Insight Analysis Why Dynamic Time Warping (DTW) for Trajectory Similarity? We empirically justify selecting DTW over Euclidean distance for trajectory align- ment. In continuous control, successful rollouts frequently exhibit temporal het- erogeneities, such as pacing variations or phase shifts, compared to the reference expert. As visualized in Figure 5, projecting in-situ 3D trajectories onto their principal action axis reveals that Euclidean distance enforces rigid point-wise alignments. This causes residual errors to explode under minor temporal shifts, yielding an artificially depressed similarity score (0.6914). Conversely, DTW computes an optimal non-linear sequence alignment. By mapping corresponding geometric configurations across flexible time intervals, DTW neutralizes tempo- ral misalignments and recovers a robust spatial congruence score (0.9460). This 14Chen et al. Table 4: Comparison of Advantage Shaping Strategies. Evaluated on LIBERO- Long. † indicates severe policy collapse at early stages (< 100 epochs). Expert Fusion Strategy SR (%)Expert Fusion StrategySR (%) Static Weighting (w = 0.5)88.3 † Asymmetric Z-Score Gate91.4 Max Routing87.5 † Sigmoid Gate92.6 Adaptive Margin Fallback (AMF) 90.0 Min-Max Normalization (Ours) 95.3 Action Magnitude Time Step 0102030405060 Batch Expert Rollout Euclidean Alignment Similarity: 0.6914 Action Magnitude Time Step 0102030405060 Batch Expert Rollout DTW Alignment Similarity: 0.9460 (a)(b) Fig. 5: Trajectory alignment comparison. In-situ training trajectories projected onto the principal action axis. (a) Rigid Euclidean matching yields high residual errors under temporal shifts. (b) DTW dynamically warps the time axis to map structurally similar states, recovering a robust spatial similarity measure. geometric-aware matching prevents high-quality exploratory rollouts from being assigned spuriously low rewards. The Necessity of Smooth Expert Fusion. Dynamically balancing the fusion weight (w ∈ [0, 1]) between the local batch and the global pool is essential for training stability. Table 4 demonstrates that rigid assignments (w = 0.5) or hard switching mechanisms (Max Routing) cause early policy collapse (< 100 epochs) due to abrupt gradient shifts. While hard-gating approaches like Adap- tive Margin Fallback (AMF) prevent this collapse, they lack fine-grained adapta- tion. Soft-gating strategies, such as the Temperature-scaled Sigmoid gate, offer smoother adjustments and successfully stabilize learning (92.6%). Ultimately, our proposed Min-Max Confidence Scaling yields the best performance (95.3%) by mapping non-stationary log-probabilities into a strictly bounded, continuous scale which ensures an adaptive and smooth integration of both experts. Analysis of the Bootstrapping Threshold (1-Shot Data Scarcity). To investigate the boundary conditions of the self-bootstrapping framework and stress-test its capability to bootstrap from minimal supervision, we evaluate OpenVLA-OFT under an extreme 1-shot data scarcity setting across all LIBERO suites. As shown in Table 5, our method effectively improves the 1-shot SFT base- line on the Spatial, Object, and Goal suites, yielding approximately 20% absolute gains. However, performance degrades on the Long-horizon suite (from 17.3% to 11.0%). This indicates a critical “prior threshold” for autonomous bootstrap- Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models15 Table 5: Applicability under Challenging Settings. Success rates (%) under limited demonstrations and world-model-generated interactions. The degradation on the Long suite highlights the minimum prior required for successful self-bootstrapping. TaskBaseOurs∆ 1-Shot SFT LIBERO-Spatial63.684.0+20.4 LIBERO-Object54.974.6+19.7 LIBERO-Goal59.683.6+24.0 LIBERO-Long17.311.0-6.3 World-Model Rollouts OpenSora-Spatial61.263.3+2.1 ping: for complex tasks, an excessively weak initialization causes exploratory rollouts to deviate irrecoverably from the expert demonstration. Consequently, the similarity reward degenerates into uninformative noise, and without mean- ingful gradient guidance, the policy optimization collapses into random drift. Applicability to World-Model-Generated Interactions. To evaluate the applicability of T 2 VLA beyond direct interaction with standard simula- tors, we conduct policy optimization using action-conditioned observations syn- thesized by an OpenSora world model [49]. The resulting OpenVLA-OFT policy is evaluated in the original LIBERO-Spatial simulator. As shown in Table 5, our method improves the success rate from 61.2% to 63.3%. This demonstrates that T 2 VLA remains applicable when its exploratory interactions are generated by a learned world model. 5 Conclusion In this work, we explore the self-bootstrapping optimization of Vision-Language- Action (VLA) models without relying on external rewards and propose T 2 VLA, an architecture-agnostic framework for autonomous policy evolution. By au- tonomously mining high-quality demonstrations through the proposed confidence- driven dual expert bootstrapping mechanism and constructing a dynamic DTW- based hybrid similarity reward, T 2 VLA enables VLA models to continuously refine their execution using intrinsic signals. Our T 2 VLA establishes a scalable, intrinsic-reward-driven paradigm, demonstrating that a model’s internal signals can effectively bootstrap continuous and autonomous policy improvement. Acknowledgements This work is supported by National Key R&D Program of China (No. 2026YFE 0101200). It is also supported by Shanghai Natural Science Foundation (No. 23ZR 1402900). The computations in this research were performed using the CFFF platform of Fudan University. 16Chen et al. References 1. Bai, Z., Gao, C., Shou, M.Z.: Evolve-vla: Test-time training from environment feed- back for vision-language-action models. arXiv preprint arXiv:2512.14666 (2025) 2. Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L., Fang, Y., Fox, D., Hu, F., Huang, S., et al.: Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734 (2025) 3. Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al.: π 0 : A Vision-Language-Action Flow Model for General Robot Control. arXiv preprint arXiv:2410.24164 (2024) 4. Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakr- ishnan, K., Hausman, K., Herzog, A., Hsu, J., et al.: Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 (2022) 5. Bu, Q., Yang, Y., Cai, J., Gao, S., Ren, G., Yao, M., Luo, P., Li, H.: Uni- vla: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111 (2025) 6. Chebotar, Y., Vuong, Q., Hausman, K., Xia, F., Lu, Y., Irpan, A., Kumar, A., Yu, T., Herzog, A., Pertsch, K., et al.: Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In: Conference on Robot Learning. p. 3909–3928. PMLR (2023) 7. Chen, D., Wang, D., Darrell, T., Ebrahimi, S.: Contrastive test-time adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. p. 295–305 (2022) 8. Chen, K., Liu, Z., Zhang, T., Guo, Z., Xu, S., Lin, H., Zang, H., Li, X., Zhang, Q., Yu, Z., et al.: π RL : Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv preprint arXiv:2510.25889 (2025) 9. Chen, T., Chen, Z., Chen, B., Cai, Z., Liu, Y., Li, Z., Liang, Q., Lin, X., Ge, Y., Gu, Z., et al.: Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088 (2025) 10. Community, S.: Starvla: A lego-like codebase for vision-language-action model de- veloping. arXiv preprint arXiv:2604.05014 (2026) 11. Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.: Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 (2023) 12. Geng, J., Cai, F., Wang, Y., Koeppl, H., Nakov, P., Gurevych, I.: A survey of confidence estimation and calibration in large language models. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume 1: Long Papers). p. 6577–6595 (2024) 13. Guo, W., Lu, G., Deng, H., Wu, Z., Tang, Y., Wang, Z.: Vla-reasoner: Empowering vision-language-action models with reasoning via online monte carlo tree search. arXiv preprint arXiv:2509.22643 (2025) 14. Guo, Y., Zhang, J., Chen, X., Ji, X., Wang, Y.J., Hu, Y., Chen, J.: Improving vision-language-action model with online reinforcement learning. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). p. 15665–15672. IEEE (2025) 15. Huang, C.P., Wu, Y.H., Chen, M.H., Wang, Y.C.F., Yang, F.E.: Thinkact: Vision- language-action reasoning via reinforced visual latent planning. arXiv preprint arXiv:2507.16815 (2025) Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models17 16. Huang, D., Fang, Z., Zhang, T., Li, Y., Zhao, L., Xia, C.: Co-rft: Efficient fine- tuning of vision-language-action models through chunked offline reinforcement learning. arXiv preprint arXiv:2508.02219 (2025) 17. Intelligence, P., Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., et al.: π 0.5 : a Vision-Language-Action Model with Open-World Generalization. arXiv preprint arXiv:2504.16054 (2025) 18. Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., Sadigh, D.: Pris- matic vlms: Investigating the design space of visually-conditioned language models. In: Forty-first International Conference on Machine Learning (2024) 19. Kim, M.J., Finn, C., Liang, P.: Fine-tuning vision-language-action models: Opti- mizing speed and success. arXiv preprint arXiv:2502.19645 (2025) 20. Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al.: Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246 (2024) 21. Li, H., Zuo, Y., Yu, J., Zhang, Y., Yang, Z., Zhang, K., Zhu, X., Zhang, Y., Chen, T., Cui, G., et al.: Simplevla-rl: Scaling vla training via reinforcement learning. arXiv preprint arXiv:2509.09674 (2025) 22. Li, Z., Yang, R., Chen, R., Luo, Z., Chen, L.: Adpro: a test-time adaptive diffusion policy via manifold-constrained denoising and task-aware initialization for robotic manipulation. arXiv preprint arXiv:2508.06266 (2025) 23. Li, Z., Liu, J., Dong, Z., Teng, T., Rouxel, Q., Caldwell, D., Chen, F.: Towards deploying vla without fine-tuning: Plug-and-play inference-time vla policy steering via embodied evolutionary diffusion. arXiv preprint arXiv:2511.14178 (2025) 24. Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., Stone, P.: Libero: Benchmark- ing knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems 36, 44776–44791 (2023) 25. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023) 26. Liu, S., Singh, I.S., Xu, Y., Duan, J., Krishna, R.: Vls: Steering pretrained robot policies via vision-language models. arXiv preprint arXiv:2602.03973 (2026) 27. Liu, S., Wu, L., Li, B., Tan, H., Chen, H., Wang, Z., Xu, K., Su, H., Zhu, J.: Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864 (2024) 28. Lu, G., Guo, W., Zhang, C., Zhou, Y., Jiang, H., Gao, Z., Tang, Y., Wang, Z.: Vla- rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505.18719 (2025) 29. Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., Martín-Martín, R.: What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298 (2021) 30. Nakamoto, M., Mees, O., Kumar, A., Levine, S.: Steering your generalists: Improv- ing robotic foundation models via value guidance. arXiv preprint arXiv:2410.13816 (2024) 31. Nasiriany, S., Maddukuri, A., Zhang, L., Parikh, A., Lo, A., Joshi, A., Mandlekar, A., Zhu, Y.: Robocasa: Large-scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.02523 (2024) 32. Neary, C., Younis, O.G., Kuramshin, A., Aslan, O., Berseth, G.: Improving pre- trained vision-language-action policies with model-based search. arXiv preprint arXiv:2508.12211 (2025) 18Chen et al. 33. Nguyen, D., Payani, A., Mirzasoleiman, B.: Beyond semantic entropy: Boosting llm uncertainty quantification with pairwise semantic similarity. In: Findings of the Association for Computational Linguistics: ACL 2025. p. 4530–4540 (2025) 34. Peng, T., Li, M., Yuan, J., Zhou, H., Xia, R., Zhang, R., Bai, L., Mao, S., Wang, B., Zhou, A., et al.: Chimera: Improving generalist model with domain-specific experts. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 3011–3022 (2025) 35. Rakthanmanon, T., Campana, B., Mueen, A., Batista, G., Westover, B., Zhu, Q., Zakaria, J., Keogh, E.: Searching and mining trillions of time series subsequences under dynamic time warping. In: Proceedings of the 18th ACM SIGKDD interna- tional conference on Knowledge discovery and data mining. p. 262–270 (2012) 36. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024) 37. Silver, D., Sutton, R.S.: Welcome to the era of experience. Google AI 1, 11 (2025) 38. Song, H., Ji, R., Shi, N., Lai, F., Kontar, R.A.: Inv-entropy: A fully probabilis- tic framework for uncertainty quantification in language models. arXiv preprint arXiv:2506.09684 (2025) 39. Song, W., Zhao, H., Ding, P., Cui, C., Lyu, S., Fan, Y., Wang, D.: Germ: A generalist robotic model with mixture-of-experts for quadruped robot. In: 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). p. 11879–11886. IEEE (2024) 40. Tan, S., Dou, K., Zhao, Y., Krähenbühl, P.: Interactive post-training for vision- language-action models. arXiv preprint arXiv:2505.17016 (2025) 41. Team, O.M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al.: Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213 (2024) 42. Wang, D., Shelhamer, E., Liu, S., Olshausen, B., Darrell, T.: Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726 (2020) 43. Yang, S., Zhang, Y., He, H., Pan, L., Li, X., Bai, C., Li, X.: Steering vision- language-action models as anti-exploration: A test-time scaling approach. arXiv preprint arXiv:2512.02834 (2025) 44. Yuan, J., Zhang, B., Gong, K., Yue, X., Shi, B., Qiao, Y., Chen, T.: Reg-tta3d: Better regression makes better test-time adaptive 3d object detection. In: European conference on computer vision. p. 197–213. Springer (2024) 45. Zang, H., Wei, M., Xu, S., Wu, Y., Guo, Z., Wang, Y., Lin, H., Shi, L., Xie, Y., Xu, Z., et al.: Rlinf-vla: A unified and efficient framework for vla+ rl training. arXiv preprint arXiv:2510.06710 (2025) 46. Zhang, H., Zhuang, Z., Zhao, H., Ding, P., Lu, H., Wang, D.: Reinbot: Amplifying robot visual-language manipulation with reinforcement learning. arXiv preprint arXiv:2505.07395 (2025) 47. Zhang, Z., Zheng, K., Chen, Z., Jang, J., Li, Y., Han, S., Wang, C., Ding, M., Fox, D., Yao, H.: Grape: Generalizing robot policy via preference alignment. arXiv preprint arXiv:2411.19309 (2024) 48. Zhao, T.Z., Kumar, V., Levine, S., Finn, C.: Learning fine-grained bimanual ma- nipulation with low-cost hardware. arXiv preprint arXiv:2304.13705 (2023) 49. Zheng, Z., Peng, X., Yang, T., Shen, C., Li, S., Liu, H., Zhou, Y., Li, T., You, Y.: Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404 (2024) Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models19 50. Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al.: Rt-2: Vision-language-action models transfer web knowledge to robotic control. In: Conference on Robot Learning. p. 2165–2183. PMLR (2023) 51. Zuo, Y., Zhang, K., Sheng, L., Qu, S., Cui, G., Zhu, X., Li, H., Zhang, Y., Long, X., Hua, E., et al.: Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084 (2025) 20Chen et al. Appendix A Overview This appendix provides additional technical details and experimental results for T 2 VLA. The content is organized as follows: – Section B: Details of Evaluation Models and Benchmarks. – Section C: Further Analysis of Confidence-Based Expert Election. – Section D: More Experimental Results. – Section E: Mathematical Formulations of Advantage Shaping Strategies. – Section F: Algorithmic Implementation. – Section G: Analysis of Raw Confidence-based Optimization. – Section H: Analysis of Learning Dynamics. – Section I: Details of Ablation Studies on the Dual Expert Mechanism. – Section J: Training Hyperparameters. – Section K: Case Study: Visualizing the Bootstrapping Threshold. – Section L: Limitations. B Details of Evaluation Models and Benchmarks B.1 VLA Architectures In addition to the OpenVLA-OFT and π 0 /π 0.5 policies evaluated in the main manuscript, we consider two additional VLA architectures: – StarVLA [10]: A discrete-action VLA that autoregressively generates action tokens. – GR00T [2]: A continuous-action VLA that generates action trajectories through a denoising process. B.2 Evaluation Environments We evaluate T 2 VLA on three simulation benchmarks with distinct manipulation requirements: – LIBERO [24]: This benchmark evaluates compositional generalization and long-horizon planning in tabletop manipulation. We report results on its four task suites: • LIBERO-Spatial: Evaluates policy robustness against variations in initial object arrangements and spatial layouts. • LIBERO-Object: Examines the generalization of manipulation skills to different object instances within the same semantic category. • LIBERO-Goal: Assesses the capacity to achieve diverse task end-states using a consistent set of objects. • LIBERO-Long: Tests long-horizon stability in multi-step sequential tasks that are prone to compounding execution errors. Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models21 – RoboTwin 2.0 [9]: This benchmark focuses on bimanual coordination and execution consistency in high-dimensional action spaces under domain ran- domization. Following the classification in SimpleVLA-RL [21], tasks are cat- egorized into four horizon levels based on the required planning steps: • Short Horizon (112–130 steps): Lift Pot and Beat Hammer. These tasks represent scenarios with limited planning horizons, averaging 121 steps. • Medium Horizon (151–223 steps): Place Empty Cup. This category covers tasks with intermediate planning horizons, averaging 176 steps. • Long & Extra Long Horizon (283–637 steps): Handover Block and Stack Bowls. These scenarios involve extended execution cycles, with step counts ranging from 283 to 637. – RoboCasa [31]: This benchmark provides large-scale household manipulation tasks in diverse kitchen scenes. We use the Close Drawer task to evaluate whether T 2 VLA transfers to a household simulation environment beyond LIBERO and RoboTwin 2.0. Group-Level Synchronization for Domain Randomization. Notably, to ensure fair GRPO advantage computation in RoboTwin 2.0, we design a spe- cific group-level synchronization mechanism to handle the randomized initial states (e.g., varying object positions and textures). Specifically, given N paral- lel environments and a required group size of G, we sample N/G independent random seeds. We then duplicate each seed G times via sequence interleaving. This implementation guarantees two critical properties: (1) Intra-group con- sistency: all G rollouts within a single group are initialized with the exact same physical configuration, providing a perfectly aligned baseline for unbiased rela- tive advantage estimation; (2) Inter-group diversity: different groups receive distinct seeds, preserving the robustness and generalization benefits inherent to domain randomization. B.3 World-Model-Generated Interactions To examine whether T 2 VLA can operate beyond direct interaction with stan- dard simulators, we additionally use OpenSora [49] as a learned world model. During policy optimization, OpenSora synthesizes action-conditioned observa- tions for exploratory interactions on LIBERO-Spatial. The resulting OpenVLA- OFT policy is then evaluated in the original LIBERO-Spatial simulator. Thus, OpenSora serves as the source of model-generated interaction observations rather than as an independent evaluation benchmark. C Further Analysis of Confidence-Based Expert Election C.1 Generalization across Models and Interaction Environments In our main manuscript, we observe that trajectories with higher generation confidence are more likely to succeed for discrete-action VLAs. Here, we further 22Chen et al. 5040302010 Generation Confidence 0% 20% 40% 60% 80% 100% Success Rate (a) LIBERO-Spatial (SR=93%) LIBERO-Object (SR=77%) LIBERO-10 (SR=17%) 1.00.80.60.40.2 Generation Confidence 0% 20% 40% 60% 80% 100% (b) OpenSora-Spatial (SR=80%) 0.120.100.080.060.04 Generation Confidence 0% 20% 40% 60% 80% 100% (c) StarVLA (SR=94%) Fig. 6: Generalization of confidence–success relationships. (a) OpenVLA-OFT on LIBERO-Spatial, LIBERO-Object, and the low-success-rate LIBERO-10 setting. (b) OpenVLA-OFT interacting with observations synthesized by an OpenSora world model. (c) StarVLA on LIBERO-Spatial. Across these discrete-action VLA settings, trajectories with higher generation confidence generally achieve higher task success rates. investigate whether this empirical relationship persists across different initial pol- icy strengths, interaction environments, and VLA architectures. Figure 6(a) com- pares OpenVLA-OFT across LIBERO-Spatial, LIBERO-Object, and LIBERO- 10. In particular, the relationship remains visible on LIBERO-10 even when the initial success rate is only approximately 17%, indicating that the confidence ordering is not limited to already strong policies. We additionally evaluate OpenVLA-OFT using action-conditioned observa- tions synthesized by an OpenSora world model [49]. As shown in Figure 6(b), higher-confidence trajectories remain more likely to succeed when evaluated un- der model-generated observations. Figure 6(c) presents the same analysis for StarVLA [10] on LIBERO-Spatial and exhibits a similar trend. These results show that the observed confidence–success relationship persists across the eval- uated discrete-action VLA architectures and interaction settings. C.2 Confidence-Based Expert Election for Continuous VLAs For continuous-action VLAs, the likelihood formulation differs from the token log-probability used by autoregressive policies. GR00T [2] estimates confidence using per-step Gaussian likelihoods along its denoising process, which char- acterize denoising consistency. Figure 7(a) presents the relationship between trajectory-level denoising confidence and task success rate. T 2 VLA does not directly optimize the policy using the numerical confidence score. Confidence is used to rank exploratory rollouts and elect high-confidence trajectories as pseudo-experts. These elected trajectories subsequently serve as behavioral anchors, and the intrinsic reward for each rollout is computed from its DTW-based trajectory similarity to the selected experts. Among 2,000 GR00T rollouts on LIBERO-Object, the high-confidence tra- jectories elected as experts achieve a 96% success rate, compared with an overall Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models23 0.1250.1000.0750.0500.0250.0000.0250.0500.075 Denoising Confidence 0% 20% 40% 60% 80% 100% Success Rate (a) GR00T (SR=64%) Linear Trend Overall SR (64%) 0.600.650.700.750.80 Trajectory Similarity 0% 20% 40% 60% 80% 100% (b) GR00T Base SR (64%) Expert Pool (96%) Fig. 7: Expert election with the continuous-action GR00T policy on LIBERO-Object. (a) Relationship between denoising confidence and task success. (b) Relationship between DTW similarity to the elected experts and task success. The elected high-confidence experts achieve a 96% success rate, and similarity to these ex- perts provides an informative reward signal. rollout success rate of approximately 64%. This result shows that confidence- based election yields a reliable expert set in the evaluated setting. To further assess whether the elected experts provide useful behavioral an- chors, Figure 7(b) groups the remaining rollouts according to their DTW similar- ity to these experts. The success rate generally increases with expert similarity, indicating that trajectories more closely aligned with the elected experts are also more likely to complete the task. Together with the high success rate of the elected experts, this result supports the two-stage design of T 2 VLA: confidence identifies high-quality reference trajectories, and similarity to these references provides an informative intrinsic signal for policy optimization. One limitation is that confidence-based election may omit successful but low-confidence trajec- tories, motivating future work on likelihood-aware expert selection. C.3 Confidence Ordering during Policy Optimization T 2 VLA only requires confidence to preserve an ordinal relationship suitable for relative expert ranking, rather than to represent a calibrated probability of phys- ical success. To examine whether this ordering deteriorates during optimization, we evaluate the initial OpenVLA-OFT policy and the checkpoints at steps 100 and 200. As shown in Figure 8, the confidence–success correlation remains posi- tive throughout training and progressively strengthens from r = 0.58 to r = 0.80 and r = 0.86. We therefore do not observe degradation of the confidence ordering or emerging overconfidence within the evaluated training horizon. D More Experimental Results We further evaluate T 2 VLA with additional VLA architectures and manipula- tion environments. For continuous-action VLA evaluation, we apply our frame- work to GR00T [2] on the LIBERO-Spatial and LIBERO-Object suites. We 24Chen et al. 160140120100806040200 Generation Confidence 0% 20% 40% 60% 80% 100% Success Rate Base Model (SR=93%, r=0.58) Step 100 (SR=96%, r=0.80) Step 200 (SR=95%, r=0.86) Fig. 8: Confidence–success relationship during policy optimization. OpenVLA-OFT checkpoints at the initial policy, step 100, and step 200 are evaluated on LIBERO-Spatial. The correlation remains positive and strengthens from r = 0.58 to r = 0.80 and r = 0.86, respectively. also evaluate the flow-based π 0 policy on the RoboCasa [31] Close Drawer task, which introduces a different household simulation environment from the LIBERO benchmarks used in the main experiments. In all settings, success rate is measured by executing the optimized policy in the corresponding evaluation environment. Table 6: Results across additional models and environments. Success rates (%). ModelSettingBaseOurs∆ GR00TLIBERO-Spatial41.460.9+19.5 GR00TLIBERO-Object58.698.8+40.2 π 0 RoboCasa Close Drawer75.087.5+12.5 As shown in Table 6, our method improves GR00T by 19.5 points on LIBERO- Spatial and 40.2 points on LIBERO-Object. On RoboCasa Close Drawer, the success rate of π 0 improves from 75.0% to 87.5%. These results provide additional evidence that the proposed expert-election and trajectory-alignment formulation applies across VLA architectures and manipulation environments. Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models25 E Mathematical Formulations of Advantage Shaping Strategies In this section, we provide the detailed mathematical definitions for the ex- pert fusion and advantage shaping strategies compared in Table 5 of the main manuscript (Section 4.4). Let c ∗ local,l denote the confidence score of the current task-conditioned local expert, and P l represent the historical expert pool with mean confidence ̄c pool , maximum c max,l , and minimum c min,l . The fusion weight w ∈ [0, 1] determines the contribution of each expert source to the hybrid simi- larity reward. – Static Weighting: A naive baseline where the current batch and historical pool are weighted equally regardless of their relative quality: w = 0.5(15) – Max Routing: A hard-selection strategy that assigns full weight to the more confident source: w = ( 1 if c ∗ local ≥ ̄c pool 0 otherwise (16) – Adaptive Margin Fallback (AMF): A gating mechanism designed to favor on-policy exploration unless the current batch quality drops significantly below the historical average by a fixed margin m = 2.0: w = I(c ∗ local ≥ ̄c pool − m)(17) – Temperature-scaled Sigmoid Gate: A soft-gating mechanism that pro- vides continuous weight transitions based on the confidence gap, utilizing a temperature parameter T = 2.0: w = 1 1 + exp − c ∗ local − ̄c pool T (18) – Asymmetric Z-Score Gate: A distribution-aware scaling method that uses the running mean μ pool and standard deviation σ pool of the expert pool. It applies higher sensitivity to high-confidence discovery: w = σ(k· Z), Z = c ∗ local − μ pool σ pool + ε (19) where k = 2.5 for Z ≥ 0 and k = 5.0 for Z < 0 to aggressively discount suboptimal batches. – Min-Max Confidence Scaling (Ours): Our proposed strategy maps the non-stationary log-probabilities into a continuous, strictly bounded scale [0, 1] using the extrema of the expert pool: w = clip c ∗ local,l − c min,l c max,l − c min,l + ε , 0, 1 (20) 26Chen et al. As empirically observed in the main manuscript, strategies lacking strictly bounded normalization (e.g., Static Weighting and Max Routing) often lead to abrupt gradient shifts and early policy collapse. Our Min-Max scaling ensures an adaptive and smooth integration of expertise, which is essential for stable self-bootstrapping during test-time reinforcement learning. F Algorithmic Implementation The optimization of T 2 VLA is implemented as a self-bootstrapping reinforce- ment learning process. Instead of relying on external environment rewards, refer- ence behaviors are extracted from exploratory rollouts using intrinsic confidence, which then guide policy updates via alignment-based rewards. The complete pro- cedure is summarized in Algorithm 1, operating in four sequential phases: – Phase 1: Exploration & Confidence Estimation. The policy π θ samples a batch of trajectoriesD l . Each trajectory is assigned a length-normalized con- fidence score c i , defined as the mean log-probability of all generated actions, to ensure a length-agnostic evaluation. – Phase 2: Dual Expert Bootstrapping. The trajectory with the highest c i in the current batch is designated as the Local Pseudo-Expert. It is then used to update a Global Expert Pool P l , a priority buffer that retains the top-K historical trajectories to capture high-quality reference behaviors. – Phase 3: Hybrid Reward Synthesis. Dynamic Time Warping (DTW) is utilized to compute spatial alignment between rollouts and experts, effectively decoupling spatial geometry from temporal variance. The final reward is a dynamically weighted fusion of local and global similarities, regularized by a KL-divergence penalty against π ref . – Phase 4: Policy Optimization. Policy parameters θ are updated via GRPO. By estimating advantages through group-level reward normalization instead of a critic network, the framework directly optimizes the policy to maximize the self-bootstrapped rewards. G Analysis of Raw Confidence-based Optimization Following the empirical observation in our main manuscript (Section 3.1) that generation confidence correlates with task success, we investigate the viability of using raw log-probabilities directly as the reward signal for policy optimization. We conduct an exploratory experiment on the LIBERO-Long benchmark, where the policy is optimized to maximize trajectory-level mean log-probabilities. As illustrated in Figure 9, the confidence-driven objective yields an initial performance improvement. Around epoch 60, the evaluation success rate (blue curve) reaches a peak of approximately 90%, surpassing the 86.5% baseline. Throughout the optimization process, the training rollout success rate (orange curve) maintains a consistently high level, fluctuating between 85% and 95%. Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models27 Algorithm 1 Self-Rewarding VLA Adaptation via Confidence-Driven Dual Ex- perts Require: Pretrained VLA π θ , reference VLA π ref , instruction l, batch size N, pool capacity K Initialize: Global Expert Pool P l ←∅ === Phase 1: Trajectory Rollout & Confidence Estimation === 1: Sample a batch of N trajectories D l = τ 1 ,...,τ N from current policy π θ condi- tioned on instruction l 2: for i = 1 to N do 3: Compute length-normalized confidence: c i ← 1 T i P T i t=1 logπ θ (a i,t |s i,t ,l) 4: end for === Phase 2: Dual Expert Bootstrapping === 5: Identify the task-conditioned local expert: τ ∗ local,l ← arg max τ i ∈D l c i 6: Let c ∗ local,l be the confidence score of τ ∗ local,l 7: Update global pool: P l ←P l ∪(τ ∗ local,l ,c ∗ local,l ) 8: Retain only the top-K trajectories in P l sorted by confidence scores === Phase 3: DTW-based Hybrid Similarity Reward === 9: Get max (c max,l ) and min (c min,l ) confidence scores in the current pool P l 10: Compute dynamic interpolation weight: w ← clip c ∗ local,l −c min,l c max,l −c min,l +ε , 0, 1 11: for i = 1 to N do 12: Local alignment: s local ← Sim DTW (τ i ,τ ∗ local,l ) 13: Global alignment: s global ← max τ p ∈P l Sim DTW (τ i ,τ p ) 14: Assign self-supervised reward: r i ← w · s local + (1 − w) · s global − β P T i t=1 D KL (π θ (·|s i,t ,l)∥π ref (·|s i,t ,l)) 15: end for === Phase 4: Policy Optimization via GRPO === 16: Calculate batch reward mean μ(r) and standard deviation σ(r) 17: for i = 1 to N do 18: Compute Group-normalized Advantage: A i ← r i −μ(r) σ(r)+ε 19: end for 20: Update policy parameters θ by maximizing the GRPO surrogate objective using A i However, following the initial peak, the evaluation performance plateaus and eventually degrades to roughly 84%–85%. Consequently, the evaluation curve remains consistently below the training curve for the remainder of the training process. These empirical results suggest that while the scalar confidence metric effec- tively reflects the overall certainty of the generated sequence, using it as a direct reward introduces optimization instabilities. Specifically, relying purely on raw log-probabilities can lead to the misclassification of valid physical executions. During exploration, the policy may generate multiple trajectories that all suc- cessfully complete the task but yield varying confidence scores due to internal model uncertainties. Consequently, the optimization process might penalize per- fectly valid, successful trajectories simply because their relative log-probabilities 28Chen et al. Success Rate Epochs 0.6 0.7 0.8 0.9 1 01224364860728496108120132144156168180192204216228240252 Train RolloutEval (libero_long) Initial: 86.5% Fig. 9: Empirical analysis of pure confidence-based rewards on the LIBERO-Long bench- mark. The curves demonstrate that confidence-driven optimization initially boosts the evaluation performance (blue) to around 90%. However, it eventually plateaus and de- grades below the 86.5% baseline, remaining consistently below the training success rate (orange). This dynamic suggests that a single scalar metric lacks sufficient signal for sustained optimization, highlighting the need for trajectory-level physical rewards. are lower than others within the same batch. This inconsistent reward assign- ment confuses the policy update process, ultimately causing the evaluation per- formance to stagnate without further improvement. This motivates the reward design in T 2 VLA, which explicitly combines the scalar confidence metric with physical execution trajectories. To prevent valid executions from being erroneously penalized, we restrict the scalar metric to iden- tifying high-quality reference experts. Subsequently, we use DTW to measure the spatial alignment between exploratory rollouts and these selected physical ref- erences. This combined formulation provides a trajectory-level reward, ensuring consistent optimization signals for stable policy learning. H Analysis of Learning Dynamics To illustrate the optimization stability of T 2 VLA, Figure 10 presents the learn- ing curves on the LIBERO-Goal benchmark under the 1-shot (traj1) setting, where the baseline policy is fine-tuned on a single demonstration trajectory. This specific model is selected as a representative case study. Its moderate initial success rate (59.6%) exceeds the minimum bootstrapping threshold, preventing the optimization collapse observed in low-competence regimes. Furthermore, it leaves ample margin for continuous policy refinement, clearly demonstrating the progressive learning dynamics. The optimization process is driven exclusively by the proposed self-bootstrapped intrinsic rewards. While ground-truth environ- ment success rates are recorded concurrently to monitor actual task progression, these external signals are strictly excluded from gradient computation and policy updates. As depicted in Figure 10, the baseline policy initiates with an evaluation success rate of 59.6% at epoch 0. Over the course of 140 training epochs, both Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models29 Success Rate Epochs 0.4 0.5 0.6 0.7 0.8 0.9 1 0714212835424956637077849198105112119126133140 Train RolloutEval (libero_goal) Start 0.596 Ours 0.83 Fig. 10: Learning dynamics on the LIBERO-Goal (traj1) benchmark. The plot tracks the success rates of exploratory training rollouts (orange) and periodic evaluations (blue). T 2 VLA steadily improves the policy from an initial 59.6% to a peak of 83.0% using exclusively intrinsic rewards, with environment signals reserved strictly for mon- itoring. the training rollout performance and the periodic evaluation success rate demon- strate a consistent upward trend. Facilitated by the dual-expert reward formu- lation and dynamic confidence scaling, the framework systematically refines its execution capabilities based on self-generated experiences. The evaluation trajectory reveals a rapid initial ascent, reaching approxi- mately 79.0% around epoch 63. Following a brief consolidation phase, the suc- cess rate resumes its growth, achieving a peak of 83.0% near epoch 119. This detailed dynamic confirms that the framework effectively extracts viable op- timization signals from intrinsic model confidence, securing substantial policy improvement entirely independent of external environmental feedback. I Details of Ablation Studies on the Dual Expert Mechanism To provide a detailed analysis of the dual-expert mechanism discussed in our main manuscript (Section 4.3), we align our evaluation with the representative LIBERO-Goal (traj1) setting established in Section H. Figure 11 visualizes the episodic learning dynamics of the isolated and combined expert configurations, starting from the initial 59.6% baseline. Local Expert Configuration. The policy optimized exclusively via the lo- cal expert (orange curve) facilitates active exploration and rapidly captures the latest on-policy feedback. However, because its guidance is strictly bounded by the current sampled batch, the optimization signal remains vulnerable to the variance of immediate exploratory rollouts. This batch-level dependency occa- sionally introduces suboptimal anchors, causing the evaluation success rate to fluctuate during the training process. 30Chen et al. Success Rate Epochs 0.4 0.5 0.6 0.7 0.8 0.9 1 06121824303642485460667278849096102108114120 Ours (Dual Expert) EvalLocal Expert Only (Train) Local Expert Only (Eval) Global Expert Only (Train) Global Expert Only (Eval) Initial (59.6%) Fig. 11: Ablation of the Dual Expert mechanism on the LIBERO-Goal (traj1) bench- mark. The plot contrasts the evaluation success rates of the Local Expert Only (orange) and Global Expert Only (green) configurations against our synergistic Dual Expert ap- proach (blue). The initial SFT baseline is denoted by the dashed gray line. The dynamic weighting effectively combines the responsiveness of the local expert with the stability of the global expert to achieve better performance. Global Expert Configuration. The global expert pool (green curve) pro- vides a highly stable and consistent optimization signal. Crucially, because this historical pool is constructed entirely from the highest-confidence local experts collected over past iterations, its steady upward trajectory inherently validates the fundamental reliability of the local experts. By maintaining a Top-K mem- ory buffer, this configuration effectively filters out isolated batch-level variance. It ensures that the policy systematically accumulates and retains high-quality behaviors, establishing a robust performance foundation throughout the training process. Dual Expert Synergy. Our T 2 VLA framework (blue curve) integrates both components through dynamic confidence scaling. Building upon the solid foundation provided by the global pool, it incorporates the local expert to ac- tively capture highly confident novel behaviors as they emerge. By adaptively balancing the immediate on-policy discoveries with the stable historical refer- ences, this synergy maximizes the advantages of both mechanisms, ultimately achieving the highest peak success rate. J Training Hyperparameters To facilitate reproducibility, we detail the complete set of hyperparameters used for training our T 2 VLA framework on the LIBERO benchmark in Table 7. In this configuration, we build our framework utilizing the OpenVLA-OFT archi- tecture. Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models31 Table 7: Hyperparameters for RL with OpenVLA-OFT. HyperparameterValueHyperparameterValue Training & EnvironmentOptimization & GRPO Total training epochs300Advantage estimatorGRPO Train batch size16GRPO group size (samples per task) 8 Validation batch size128PPO update epochs1 Max environment steps500 Actor learning rate5× 10 −6 (Cosine) Max prompt length256Critic learning rate1× 10 −5 (Constant) Max response length128Clip bounds (ε low , ε high )(0.2, 0.28) Action Generation Initial KL coefficient0.02 Action chunk length8Target KL coefficient0.04 Action token length7Entropy coefficient0.005 Rollout temperature1.6Reward discount rate (γ)1.0 Top-p sampling1.0 K Case Study: Visualizing the Bootstrapping Threshold K.1 Stable Bootstrapping under Strong Initial Priors To visually understand the exploratory dynamics and the mechanism behind the bootstrapping threshold, we implemented an in-situ tracker during training to capture and log the exact micro-batches of exploratory rollouts and their corresponding rewards. We first examine the stable bootstrapping behavior of a highly competent initial policy. Figure 12 displays a batch of 8 trajectories generated by a model fine-tuned on full demonstration data, achieving an 86.5% base success rate on the LIBERO-10 suite. As demonstrated by the 2D projection of cumulative delta-actions, all 8 roll- outs successfully cover the full spatial horizon of the task. The initial policy maintains consistent directional progression, avoiding premature halting. While the trajectories exhibit local variance due to exploration, they universally adhere to the global task geometry. Consequently, the framework evaluates a pool of structurally complete ex- ecutions. The selected expert (marked with a red star) represents a physically valid, full-length trajectory rather than a degenerate path. This visualization confirms that initial policy competence determines the quality of self-generated trajectories: when the base policy possesses sufficient capability, its exploratory rollouts consistently span the required task horizon. The structural completeness of these trajectories provides the valid anchors necessary for grounded reward formulation, ensuring the stable optimization signals required for effective and continuous policy refinement. 32Chen et al. Fig. 12: Exploratory rollouts in a high-competence regime (base success rate 86.5%). The initial policy ensures all 8 rollouts cover the full task horizon without early ter- mination. The framework selects a structurally complete expert (red line), providing valid anchors for stable policy refinement. Fig. 13: A failure case visualizing the bootstrapping threshold on the LIBERO-Long suite (1-shot setting). The framework evaluates 8 exploratory rollouts generated by a weak initial model and assigns the highest confidence (Reward = 1.000) to a spatially truncated trajectory (Expert, red line). Trajectories attempting to explore further (e.g., blue, purple) receive lower scores. Bootstrapping from such incomplete anchors illus- trates the optimization challenges in excessively low-competence regimes. Trust Your Instincts: Confidence-Driven Test-Time RL for VLA Models33 K.2 Optimization Collapse under Weak Initial Priors To contrast with the stable bootstrapping observed under strong priors, we in- vestigate the optimization collapse under weak priors. As discussed in the main manuscript (Section 4.4), our self-bootstrapping framework exhibits a perfor- mance degradation on the LIBERO-Long suite under the extreme 1-shot setting (dropping from 17.3% to 11.0%). Figure 13 visualizes a representative batch of 8 trajectories collected at Epoch 0 under a single task instruction. For visualization clarity, the cumulative delta-action sequences of the end- effector are projected onto a 2D plane. Given the low initial success rate of the baseline model (17.3%), it typically struggles to handle complex long-horizon scenarios. Consequently, a significant portion of the generated rollouts naturally fails to cover the full task horizon. Relying on the internal log-probabilities, the framework must autonomously nominate an expert from a pool that predomi- nantly consists of incomplete executions. As shown in the figure, the trajectory selected as the Expert (marked with a red star, normalized reward = 1.000) is notably brief, terminating early in the spatial execution. Conversely, rollouts that attempt to progress further spatially (e.g., Rollout 7 and Rollout 8) are assigned lower scores. This visualization clarifies the optimization challenges in low-competence regimes. A severely under-trained initial policy exhibits high epistemic uncer- tainty regarding the later stages of a task. Consequently, the model’s internal confidence metric naturally favors shorter paths that remain within familiar, early-stage state distributions, assigning lower log-probabilities to longer trajec- tories that venture into unfamiliar regions to make spatial progress. Bootstrapping from these spatially truncated anchors misaligns the reward signal with actual task completion. The optimization process inadvertently rein- forces early-termination behaviors, leading to the observed performance degra- dation. This case study empirically supports the existence of a minimum initial- ization threshold: a foundational policy must possess sufficient competence to generate rollouts that adequately span the task horizon, thereby providing valid structural anchors for autonomous self-improvement. L Limitations Omission of Low-Confidence Experts. Continuous-action VLAs formulate trajectory likelihood using Gaussian likelihoods along the denoising process. Suc- cessful trajectories assigned relatively low likelihoods may be omitted during expert election. Initialization Threshold. With weak initial task competence, particularly on long-horizon tasks, successful or reasonably complete trajectories occur less frequently, reducing the number of high-quality behavioral references. Physical Deployment. Our experiments are primarily conducted in simu- lation and with world-model-generated interactions. Testing and adaptation on physical robots remain future work.