Paper deep dive
Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models
Haoran Hao, Shahram Najam Syed, Jeff Schneider, Jeffrey Ichnowski
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/26/2026, 5:25:53 AM
Summary
The paper proposes Hierarchical Skill Retrieval (HSR), a framework for data-efficient adaptation of Vision-Language-Action (VLA) models. HSR decomposes long-horizon tasks into skill sequences using an LLM, evaluates plans based on semantic plausibility and skill reliability (estimated via behavior cloning loss), and performs hybrid retrieval combining language similarity with behavior-feature reranking. The policy is adapted via a two-stage process: pretraining on retrieved skill data and finetuning on target demonstrations. HSR improves success rates by 10.3% on the LIBERO benchmark and 21.3% on real-world tasks compared to strong baselines.
Entities (8)
Relation Signals (7)
HSR → improves → LIBERO
confidence 95% · HSR improves the average success rate by 10.3% ... on the LIBERO benchmark
HSR → employs → Hybrid Retrieval
confidence 93% · The selected decomposition is then used for hybrid retrieval. This combines subtask-level language retrieval with behavior-feature reranking
HSR → employs → two-stage adaptation
confidence 93% · we adapt the policy through a two-stage pretraining and finetuning pipeline
HSR → uses → SmolVLA-450M
confidence 92% · We use the pretrained SmolVLA-450M [8] model as the base policy model.
HSR → uses → Qwen3-VL-4B
confidence 92% · we employ Qwen3-VL-4B [46] as the LLM planner for task decomposition.
HSR → outperforms → BR
confidence 90% · HSR improves the average success rate by 10.3% ... over the strongest baseline
HSR → outperforms → STRAP
confidence 90% · HSR improves the average success rate by 10.3% ... over the strongest baseline
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While Vision-Language-Action (VLA) models pretrained on large-scale robot datasets provide a strong foundation for robot manipulation, their performance can degrade when adapted to new tasks with limited task-specific demonstrations. Retrieval offers a practical way to reuse existing demonstrations for data-efficient adaptation, but existing methods often rely on visual similarity, state-action representations, or task-level language matching. These approaches may overlook the hierarchical structure of long-horizon manipulation tasks, where complete task matches are rare but reusable skills are often abundant. To address this challenge, we propose Hierarchical Skill Retrieval (HSR), a retrieval framework for data-efficient VLA adaptation. Specifically, HSR first decomposes a target task into candidate skill sequences. It evaluates each plan based on both semantic plausibility and skill reliability estimated from the prior dataset. The selected decomposition is then used for hybrid retrieval. This combines subtask-level language retrieval with behavior-feature reranking to identify demonstrations that are both semantically relevant and compatible with the target task. Finally, we adapt the policy through a two-stage pretraining and finetuning pipeline, which separates general skill acquisition from task-specific adaptation. Experiments on the LIBERO benchmark and several real-world robot manipulation tasks show that HSR improves the average success rate by 10.3% and 21.3% over the strongest baseline, respectively. These results demonstrate the effectiveness of structured skill-level retrieval for data-efficient VLA adaptation. Videos and code are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.24042v1
- Canonical: https://arxiv.org/abs/2608.24042v1
Trouble viewing inline? Open PDF directly →
Full Text
50,222 characters extracted from source content.
Expand or collapse full text
Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models Haoran Hao Affiliation: Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213. haoranha, jeff4, jichnows@andrew.cmu.edu Shahram Najam Syed Affiliation: Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213. haoranha, jeff4, jichnows@andrew.cmu.edu Jeff Schneider Affiliation: Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213. haoranha, jeff4, jichnows@andrew.cmu.edu Jeffrey Ichnowski Affiliation: Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213. haoranha, jeff4, jichnows@andrew.cmu.edu Abstract While Vision-Language-Action (VLA) models pretrained on large-scale robot datasets provide a strong foundation for robot manipulation, their performance can degrade when adapted to new tasks with limited task-specific demonstrations. Retrieval offers a practical way to reuse existing demonstrations for data-efficient adaptation, but existing methods often rely on visual similarity, state-action representations, or task-level language matching. These approaches may overlook the hierarchical structure of long-horizon manipulation tasks, where complete task matches are rare but reusable skills are often abundant. To address this challenge, we propose Hierarchical Skill Retrieval (HSR), a retrieval framework for data-efficient VLA adaptation. Specifically, HSR first decomposes a target task into candidate skill sequences. It evaluates each plan based on both semantic plausibility and skill reliability estimated from the prior dataset. The selected decomposition is then used for hybrid retrieval. This combines skill-level language retrieval with behavior-feature reranking to identify demonstrations that are both semantically relevant and compatible with the target task. Finally, we adapt the policy through a two-stage pretraining and finetuning pipeline, which separates general skill acquisition from task-specific adaptation. Experiments on the LIBERO benchmark and several real-world robot manipulation tasks show that HSR improves the average success rate by 10.3% and 21.3% over the strongest baseline, respectively. These results demonstrate the effectiveness of structured skill-level retrieval for data-efficient VLA adaptation. Videos and code are available at hoar012.github.io/HSR-Project. I Introduction Foundation models pretrained on large-scale datasets have shown strong generalization ability and broad transferable knowledge [1, 2]. Motivated by recent advances in large multimodal models, vision-language-action (VLA) models [3, 4, 5, 6, 7, 8] extend pretrained vision-language representations to robot control, aiming to transfer their general visual and linguistic understanding to downstream manipulation tasks. A common way to adapt VLA models to downstream tasks is to finetune them on target-task demonstrations [3, 9]. While effective in many settings, this approach is fundamentally limited by data scarcity: collecting high-quality robot demonstrations is expensive and time-consuming, and target datasets are often too small to fully adapt large policy models for reliable real-world deployment. A complementary line of work improves adaptation by reusing large prior datasets as additional training data [10, 11, 12, 13, 14]. Among these methods, retrieval-based approaches are particularly effective because they can identify useful demonstrations from large prior datasets such as Open X-Embodiment [6]. Existing retrieval methods typically rely on similarity metrics between target data and prior data, including distances in latent state-action spaces [10, 11] and subtrajectory-level similarity measures [13, 14]. By augmenting the target dataset with retrieved demonstrations, these methods expand the effective training set and can improve downstream task performance. Fig. 1: Hierarchical Skill Retrieval (HSR): LLM-based task decomposition enables structured demonstration retrieval for data-efficient VLA adaptation. Although retrieval-based methods have shown promising results in several settings, most existing approaches are designed for vision-based policies and primarily rely on visual or motion similarity. As a result, they often overlook higher-level task semantics and relationships between tasks, and thus do not fully exploit language instructions. VLA models, on the other hand, have demonstrated strong capabilities in following language instructions [15, 16] and transferring knowledge across related tasks [4, 17, 18], such as manipulating different objects using similar actions. This suggests that language can provide an important semantic signal for improving task generalization. However, language-based retrieval is also limited when it relies only on coarse instruction similarity. For complex long-horizon tasks, such as “make a cup of tea”, directly retrieving data using the full task instruction is often difficult, since complete task matches rarely exist in prior datasets. In contrast, many of the underlying skills required to complete such tasks, such as “grasp a teapot” or “pick up a tea bag”, are common and well represented. Effectively leveraging the hierarchical structure of complex tasks and retrieving data at the level of reusable skills therefore remains an open challenge. To address these issues, we propose Hierarchical Skill Retrieval (HSR), a framework for data-efficient VLA adaptation based on task decomposition and skill-level retrieval. Specifically, (1) we first cluster the prior dataset into reusable skills, such as Pick, Open, and Close. Given a target task, we use a Large Language Model (LLM)-based planner to decompose the task into subtasks grounded in the available skills. (2) We then score candidate plans, select the most suitable decomposition, and use the resulting subtasks as queries for retrieval. (3) Finally, we adapt the VLA model in two stages: first by pretraining on language-retrieved skill data, and then by finetuning on the target demonstrations together with feature-reranked retrieved samples. The HSR framework leverages both the structure of the target task and the prior dataset, enabling data-efficient adaptation of pretrained VLA models under limited target data. We evaluate the proposed approach in the LIBERO [19] simulation environment and on several real-world long-horizon robot manipulation tasks. Experimental results show that HSR achieves the best average performance across a wide range of tasks and provides particularly large gains on long-horizon composite manipulation tasks. Notably, HSR improves the success rate by 10.3% on LIBERO and by 21.3% on real-world tasks relative to the strongest baseline. Our contributions are summarized as follows: • We propose a hierarchical retrieval framework for long-horizon manipulation, which decomposes tasks into skill sequences and retrieves demonstrations at the skill level based on task structure and semantics. • We introduce a two-stage adaptation strategy. It first uses retrieved demonstrations to acquire transferable skills, and then finetunes on target data together with feature-reranked retrieved samples. • We validate the proposed framework on both the LIBERO benchmark and real-world long-horizon manipulation tasks, showing improved success rates and data efficiency over strong retrieval-based baselines. I Related Work I-A Vision-Language Conditioned Manipulation Earlier work on robot manipulation using deep neural networks typically restricted the input modality to either visual observations or robot states [20]. In recent years, vision-language models (VLMs) [1, 2, 21] pretrained on large-scale image-text pairs have demonstrated strong capabilities in multimodal perception and understanding. Based on these multimodal foundation models, Vision-Language-Action (VLA) policy models [3, 5, 6, 7, 8, 22, 23, 24] aim to transfer the general knowledge learned by VLMs to robot control. By training on multi-task datasets and conditioning on both language instructions and visual observations, VLA models show improved generalization in vision-language manipulation tasks. Pretrained VLA models can leverage large-scale datasets such as Open X-Embodiment [6], enabling cross-task generalization and improved instruction understanding in robot manipulation. However, despite containing rich general knowledge, VLA models often perform suboptimally on specific downstream tasks. In many practical scenarios, downstream tasks lack sufficient data for training or finetuning. As a result, how to effectively adapt VLA models to data-scarce downstream tasks remains an open and underexplored problem. I-B Retrieval for Model Adaptation Retrieval-based methods aim to select relevant data from existing datasets, such as DROID [25] and Open X-Embodiment [6], to improve adaptation to specific target tasks [26]. Most prior work learns latent representations of trajectories and performs similarity-based retrieval. For example, some methods encode state-action pairs [10], optical flow [27], or sub-trajectories [13, 14], and retrieve data based on visual or motion similarity. Other approaches extend this idea by incorporating multiple modalities [12] or applying importance weighting between prior and target datasets [11]. There are also methods that retrieve relevant past experiences and directly learn from them [28]. However, these approaches mainly rely on visual or motion-level similarity, and often do not explicitly consider the semantic and hierarchical structure of manipulation tasks, which is crucial for modern VLA models. Trajectories with similar motion patterns may correspond to different skills, such as pushing versus picking, leading to ambiguous or less relevant retrieval results. Recent work has started to explore semantic-level reasoning in robot tasks. DROC [29] shows that LLMs can generate language-based corrections, which are then used as queries to retrieve relevant knowledge from a knowledge base. MT3 [30] decomposes tasks into alignment and interaction policies, enabling task composition and improved generalization through test-time retrieval. Motivated by these insights, we propose a framework that decomposes long-horizon manipulation tasks into subtasks and retrieves semantically related experiences from a prior dataset. I-C Robot Task Planning Task planning aims to decompose and schedule complex tasks by combining symbolic reasoning, motion planning, and learning-based methods [31]. Effective planning enables zero-shot generalization to novel tasks by reusing existing skills [32, 33], reducing the need for additional data to learn composite tasks. Some approaches rely on human-defined transition models or learn such models from data to generate composed plans [34, 35, 36, 37]. However, building such models often requires substantial human effort, large training datasets, and additional components, such as object detectors [36]. More recent work explores the reasoning capabilities of LLMs for task planning [38, 39, 40, 41, 42, 43]. The resulting subtasks are typically executed by separate policy modules [41, 44]. Most existing approaches focus on task decomposition at test time, while placing less emphasis on how the required skills are trained or adapted. In this work, we use task decomposition as a training-time retrieval interface. Given a target task, we identify a sequence of semantically relevant skills and retrieve training demonstrations that support each skill, thereby improving data-efficient adaptation of the policy. Fig. 2: Overview of the Proposed HSR Framework. HSR identifies existing skills in the prior dataset based on task-instruction clustering. Given a long-horizon target task, 1. HSR first decomposes the task into several subtasks, evaluates and selects the plan with the highest score; 2. for each subtask, HSR retrieves data based on the subtask instruction, and further reranks using representative demonstrations from the target task to filter out low-consistency data; 3. HSR first pretrains the policy on retrieved data to learn general skills, and then finetunes it on task-related data to adapt to the target task. Fig. 3: Illustration of Plan Evaluation. We first cluster priorD_prior into reusable skills. Each clustered skill is then evaluated by finetuning a pretrained VLA model on the training split and measuring the BC loss on its validation set. The resulting loss serves as a lightweight proxy for skill reliability, enabling efficient plan evaluation without costly environment rollouts. I Problem Statement We study few-shot adaptation of a pretrained vision-language-conditioned policy using a small set of target demonstrations together with a large prior dataset. Given a pretrained VLA policy πθ _θ, the policy maps a visual observation oto_t and a language instruction T to an action ata_t: at=πθ(ot,T)a_t= _θ(o_t,T). For each target task, we are given a target dataset t=τii=1ND_t=\ _i\_i=1^N, where each trajectory τi=(ot,at)t=1H _i=\(o_t,a_t)\_t=1^H is an expert demonstration. We assume access to a large prior dataset priorD_prior, which contains demonstrations from diverse tasks and domains. Our objective is to identify a useful subset r⊂priorD_r _prior such that adaptation on t∪rD_t _r yields a policy that better matches expert behavior and performs well on the target task. IV Method IV-A Hierarchical Task Decomposition Language-based retrieval is often insufficient for robot manipulation tasks, where high-level instructions are ambiguous and rarely match complete demonstrations in the prior dataset. This limitation is particularly severe for long-horizon and compositional tasks, such as “make a cup of tea”, which are seldom observed as monolithic behaviors in practice. However, these tasks are typically composed of reusable skills that recur across different tasks, such as “pick up a cup” and “turn on a machine”. Motivated by this observation, we decompose a target task into a sequence of semantic subtasks and use each subtask as a query to retrieve relevant demonstrations from priorD_prior. This skill-level retrieval enables more precise reuse of prior experiences and improves adaptation to long-horizon manipulation. The overall HSR framework is illustrated in Fig. 2. Although LLMs have shown strong performance in task planning, prior work such as SayCan [42] points out an important limitation: LLMs are not grounded in the robot’s actual capabilities and may therefore generate plans containing subtasks that are difficult or impossible to execute. As a result, purely language-based planning may produce decompositions that are poorly matched to the policy’s learned skills. To mitigate this issue, we leverage priorD_prior to estimate the reliability of different skills and use this information to guide task decomposition for retrieval. Our key idea is to prefer decompositions composed of skills that are both semantically relevant to the target task and well supported by the available data. Specifically, we first extract text embeddings for all task instructions in priorD_prior and apply K-means clustering to group them into a set S=s1,…,sKS=\s_1,…,s_K\, where each cluster sis_i corresponds to a semantically coherent skill, such as Find, Pick up, or Pour. For each sis_i, we estimate its reliability using the pretrained VLA model. Rollout-based evaluation provides the most direct measure of policy performance, but is computationally expensive and often impractical, especially in real-world settings. Instead, following DataMIL [45], we use held-out behavior cloning (BC) loss as a lightweight proxy for skill reliability. As illustrated in Fig. 3, we split priorD_prior into training and validation sets and finetune the pretrained VLA model on the training split to obtain πθ′ _θ . We then evaluate πθ′ _θ on the validation trajectories associated with each skill cluster. For a validation trajectory τ=(ot,at)t=1Hτ=\(o_t,a_t)\_t=1^H_τ, let ℓBC(πθ′(ot,T),at) _BC( _θ (o_t,T),a_t) denote the per-step BC loss. We define the average trajectory-level BC loss as ℒtraj(τ)=1Hτ∑t=1HτℓBC(πθ′(ot,T),at).L_traj(τ)= 1H_τ _t=1^H_τ _BC( _θ (o_t,T),a_t). (1) For a skill cluster sis_i, let iV_i denote its validation trajectories. The validation loss of sis_i is computed as ℒval(si)=1|i|∑τ∈iℒtraj(τ),L_val(s_i)= 1|V_i| _τ _iL_traj(τ), (2) and H(si)H(s_i) denotes the average trajectory length in iV_i. To ensure comparability across skill clusters, we normalize both quantities using min-max normalization. Since long-horizon skills are more susceptible to compounding errors, we define the Skill Confidence Score as C(si)=exp(−αℒ~val(si)−βH~(si)),C(s_i)= (-α L_val(s_i)-β H(s_i) ), (3) where ℒ~val(si) L_val(s_i) and H~(si) H(s_i) denote the normalized validation loss and average trajectory length, respectively. We use fixed weights α=0.6α=0.6 and β=0.8β=0.8 across all experiments. Rather than estimating the true success probability, C(si)C(s_i) serves as a rollout-free proxy for skill reliability, favoring skills that are easier to imitate and less susceptible to error accumulation during long-horizon execution. We then query the LLM with the target task description and the set of available skills to generate candidate decomposed plans. Each plan is represented as an ordered sequence of subtasks p=(u1,u2,…,un)p=(u_1,u_2,…,u_n). For each subtask uju_j, we assign its nearest skill cluster based on text-embedding similarity, denoted as s(uj)s(u_j). In addition, we prompt the LLM to provide a semantic score for each plan, reflecting its task coverage, logical consistency, and semantic plausibility with respect to the target instruction. The score is normalized to Ssem(p)∈(0,1]S_sem(p)∈(0,1] before plan ranking. We then evaluate each candidate plan by combining the semantic score with the confidence scores of its constituent skills: Score(p)=Ssem(p)⋅∏j=1nC(s(uj)).Score(p)=S_sem(p)· _j=1^nC (s(u_j) ). (4) We select the plan with the highest score as the final decomposition, and use it for subsequent retrieval from priorD_prior. IV-B Hybrid Skill Retrieval Language-based retrieval is effective at identifying demonstrations that are semantically relevant to a target task. However, VLA policies are also sensitive to low-level factors such as environment layouts, lighting conditions, and camera viewpoints. As a result, demonstrations that are similar in language instruction may still differ substantially from the current execution context, which can degrade policy adaptation and execution. To address this issue, we propose a hybrid skill retrieval strategy that combines language-based retrieval with behavior-feature reranking. The key idea is to first use language similarity to ensure semantic relevance, and then use behavior similarity to select demonstrations that are more compatible with the target task. Specifically, given a subtask, we first retrieve the top M%M\% of episodes from priorD_prior according to language similarity. We then flatten these episodes into candidate frames and rerank them using behavior features. Following Behavior Retrieval [10], we train a VAE encoder on priorD_prior to extract their behavior features. Let zfz_f denote the latent feature of a candidate frame f, and let zjtj=1Nt\z_j^t\_j=1^N_t denote the latent features extracted from tD_t, where NtN_t is the number of target frames. The behavior similarity score of f is defined as s(f)=max1≤j≤Nt(−‖zf−zjt‖2),s(f)= _1≤ j≤ N_t (-\|z_f-z_j^t\|_2 ), (5) which corresponds to the negative distance to the nearest target example in latent space. We retain the top F%F\% of candidate frames with the highest similarity scores as the final retrieved data for policy adaptation. By combining semantic alignment at the episode level with behavior compatibility at the frame level, this hybrid retrieval strategy improves the relevance and effectiveness of the retrieved skills under varying environmental conditions. IV-C Policy Adaptation To better exploit the retrieved data, we adopt a two-stage policy adaptation scheme. Let langD_lang denote the training samples from the language-retrieved episodes and let rerankD_rerank denote the reranked frames constructed from langD_lang. In the first stage, we pretrain the policy on langD_lang to acquire reusable behaviors. In the second stage, we finetune the policy on t∪rerankD_t _rerank for task-specific adaptation. This design differs from prior single-stage co-training methods [10, 11], which jointly optimize the policy on the retrieved data rD_r and the target data tD_t in one step. In contrast, our two-stage strategy explicitly separates general skill acquisition from task-specific adaptation and avoids treating all retrieved data as equally informative. TABLE I: Comparison on LIBERO Benchmark. We compare our method with behavior cloning (BC), training with all data, language-based retrieval, and four retrieval baselines: FR [27], STRAP [14], BR [10], and IWR [11]. We report the success rate (in %) and standard deviation across 3 seeds. The best results are in bold and the second-best results are underlined. Category Pick-Place Spatial Understanding Composite Manipulation Task Soup-S Cheese-B Book Soup-C Moka-M Mug-Mug Mug-P Stove-M Bowl-C Mug-M Average BC 2.0± 2.0 18.0± 3.5 20.7± 4.2 3.3± 1.2 0.0± 0.0 0.0± 0.0 12.7± 3.1 68.0± 5.3 77.3± 4.2 24.7± 5.8 22.7± 1.1 Random 10.0± 5.3 18.0± 3.5 72.0± 9.2 6.7± 2.3 0.7± 1.2 11.3± 2.3 8.0± 5.3 58.7± 4.2 44.0± 2.0 30.7± 9.2 26.0± 3.7 All Data 16.7± 1.2 19.3± 3.1 81.3± 6.1 11.3± 2.3 3.3± 4.2 12.7± 3.1 6.7± 2.3 62.7± 6.4 44.7± 3.1 31.3± 11.7 29.0± 2.4 Language 16.7± 8.1 20.0± 5.3 88.7± 5.0 14.7± 8.1 2.7± 3.1 16.0± 3.5 10.7± 2.3 68.0± 5.3 45.3± 9.0 32.7± 6.1 31.5± 1.1 FR 16.0± 5.3 20.0± 0.0 76.0± 7.2 9.3± 5.0 0.7± 1.2 14.0± 0.0 11.3± 5.8 63.3± 6.1 46.7± 7.6 25.3± 10.3 28.3± 1.6 STRAP 11.3± 1.2 32.0± 4.0 68.0± 7.2 12.0± 3.5 6.7± 1.2 17.3± 1.2 16.7± 1.2 57.3± 4.6 58.0± 5.3 40.0± 11.1 31.9± 2.1 BR 17.3± 4.2 36.7± 4.6 95.3± 2.3 9.3± 2.3 3.3± 1.2 17.3± 2.3 8.7± 2.3 64.0± 3.5 50.7± 9.2 29.3± 2.3 33.2± 1.5 IWR 14.7± 3.1 29.3± 6.4 88.7± 3.1 11.3± 3.1 2.7± 3.1 14.7± 4.2 11.3± 1.2 70.7± 3.1 54.7± 11.7 27.3± 1.2 32.5± 1.0 HSR 18.0± 5.3 32.0± 9.2 90.7± 1.2 12.7± 5.0 15.3± 3.1 18.0± 3.5 26.0± 2.0 73.3± 4.2 81.3± 11.0 67.3± 18.1 43.5± 3.3 V Experiments Models and Training. We use the pretrained SmolVLA-450M [8] model as the base policy model. During adaptation, we update only the parameters of the action expert, resulting in approximately 100M trainable parameters. In the HSR framework, we employ Qwen3-VL-4B [46] as the LLM planner for task decomposition. We use K=10K=10 clusters for both simulation and real-world experiments. For each target task, the planner generates 10 candidate plans, and we select the plan with the highest score. We use a pretrained BERT model [47] to extract text embeddings for retrieval. Environments. We evaluate the proposed framework in both simulation and real-world settings. In simulation, we use the LIBERO benchmark [19]. For real-world evaluation, we design long-horizon manipulation tasks executed on a 7-DoF xArm robot. All experiments are conducted using a single Nvidia RTX 5090 GPU. Simulation. We evaluate on the LIBERO long-horizon subset, which consists of 10 composite manipulation tasks requiring diverse skills such as pick-and-place and drawer closing. For each task, we construct a target dataset tD_t containing 5 demonstrations. Following [11, 14], we use LIBERO-90 as priorD_prior, which includes 90 tasks with 50 demonstrations per task. We retrieve the top 10% of episodes from priorD_prior in the first stage and retain the top 30% of these candidates after reranking in the second stage, corresponding approximately to 3% of frames from priorD_prior. We evaluate the learned policy over 50 episodes using 3 random seeds and report the average success rate with standard deviation. Real-World. We design several real-world manipulation tasks with varying levels of complexity, including Drawer, Trashcan, Cup-Drawer, and Cup-Tea (Fig. 4). For each task, we collect 20 demonstrations as tD_t. We randomly sample 10k trajectories from the DROID dataset [25] as priorD_prior. In the first stage, we use the reranked top 3% of frames from priorD_prior for skill pretraining. In the second stage, we finetune the policy only on the target dataset tD_t, as the cross-embodiment gap between the source data and our xArm platform makes direct co-training less effective. We evaluate the learned policy over 20 trials per task and report the success rate. Each trial uses a different initial state, while the set of states is identical across models for fair comparison. (a) Drawer (b) Trashcan (c) Cup-Drawer (d) Cup-Tea Fig. 4: Experimental Setup of Real-robot Tasks. We evaluate using a 7-DoF xArm on several manipulation tasks: (a) Drawer: Close the drawer, (b) Trashcan: Pick up the crumpled paper and throw it in the trash can, (c) Cup-Drawer: Put the cup into the drawer and close it, (d) Cup-Tea: Put the teabag next to the cup and pour water into the cup. Baselines. We compare HSR with baselines covering target-only finetuning, naive data mixing, language-based retrieval, optical-flow retrieval, subtrajectory retrieval, and state-action representation-based retrieval. Behavior Cloning (BC) finetunes the VLA model using only the target dataset tD_t. Random randomly samples 10% of the data from the prior dataset priorD_prior and co-trains the VLA model with the target dataset tD_t. All Data co-trains the VLA model on the entire prior dataset priorD_prior together with the target dataset tD_t. Language performs simple language-similarity-based retrieval and co-trains the VLA model on the retrieved dataset rD_r and the target dataset tD_t. Flow Retrieval (FR) [27] retrieves data based on similarity computed from samples’ optical flows. STRAP [14] employs vision foundation models and dynamic time warping to retrieve sub-trajectories from priorD_prior. Behavior Retrieval (BR) [10] trains a VAE to encode state-action pairs and retrieves data from priorD_prior based on similarity in the learned latent space. Importance Weighted Retrieval (IWR) [11] uses the same VAE encoder but retrieves data from priorD_prior based on estimated importance weights. V-A Simulation Experiments Table I summarizes the results on the LIBERO benchmark. Finetuning on the target dataset alone (BC) achieves limited performance due to the small number of demonstrations. Random and All Data achieve only modest gains despite using additional prior data, suggesting that simply increasing data volume is insufficient when the prior dataset contains redundant or task-irrelevant demonstrations. Language-based retrieval improves over random sampling, but remains limited on tasks with more complex instructions, suggesting that language similarity alone is insufficient to capture fine-grained semantics. State-action-based retrieval methods, including BR and IWR, further improve performance by leveraging trajectory-level similarity. However, their gains are smaller on composite manipulation tasks, where higher-level task structure matters more. STRAP is the closest baseline to HSR, as it also performs retrieval at the subtrajectory level. Nevertheless, its vision-based similarity criterion is less effective at capturing the semantic structure of long-horizon tasks, which limits its performance on tasks that require reusable skills. Overall, HSR retrieves more useful demonstrations and achieves the best average performance. On tasks that mainly require a single pick-and-place skill, HSR performs comparably to BR, indicating that directly retrieving highly similar demonstrations can already be sufficient in simpler settings. In contrast, the advantage of HSR becomes more evident on complex composite tasks. For example, on Mug-M, HSR outperforms the strongest competing method on this task, STRAP, by 27.3%. On the challenging Bowl-C task, it is the only method that improves over pure behavior cloning. These results demonstrate the effectiveness of HSR in selecting useful prior data and enabling effective policy adaptation. V-B Real-World Experiments For real-robot tasks, data scarcity poses a significant challenge. As shown in Table I, BC trained only on target demonstrations achieves a success rate below 40 %. BR and IWR provide only limited improvement, suggesting that directly transferring prior data remains challenging under the substantial cross-embodiment gap between the source data and the target tasks. In contrast, HSR improves performance by retrieving relevant demonstrations in a structured manner. As a result, HSR achieves higher success rates across real-robot tasks of varying horizons. Notably, the advantage of HSR becomes more pronounced on long-horizon tasks. For the “make a cup of tea” task, which requires several hundred low-level actions to complete, BC trained only on the target demonstrations succeeds in 3 out of 20 trials, whereas HSR achieves 9 out of 20 successful executions, demonstrating a substantial improvement on complex multi-step manipulation. TABLE I: Experimental Results on Real-robot Manipulation. For each task, we report the success rate over 20 trials. We also report the overall success rate across all tasks. The best results are in bold. Task Drawer Trashcan Cup-Drawer Cup-Tea Overall #Subtask 1 2 3 4 BC 12/2012/20 7/207/20 5/205/20 3/203/20 27/8027/80 BR 15/2015/20 3/203/20 4/204/20 5/205/20 27/8027/80 IWR 16/2016/20 4/204/20 7/207/20 4/204/20 31/8031/80 HSR /18/20 /8/20 /13/20 /9/20 /48/80 V-C Ablation Studies Task Decomposition. We remove the task decomposition module and directly retrieve data using the original task instruction, while keeping the rest of the framework unchanged. As shown in Fig. 5, performance drops noticeably, especially on tasks with more complex instructions such as Mug-P and Moka-M. This suggests that using the full task description alone is often insufficient to retrieve demonstrations required by long-horizon manipulation. In contrast, task decomposition breaks a complex instruction into semantically meaningful subtasks, enabling more precise retrieval and improving overall performance. We further compare different decomposition strategies in Table I, which reports the average success rate across all LIBERO-10 tasks. Raw LLM-generated plans already achieve competitive performance, indicating that LLMs can produce reasonable high-level decompositions. Incorporating the available skill set into the prompt further improves performance, suggesting that grounding the planner with executable skills leads to better task decomposition. Adding the plan evaluation module of HSR yields a further improvement, demonstrating the value of selecting decompositions that are both semantically relevant and well supported by the prior data. Manually selected plans achieve performance comparable to that of HSR, indicating that the proposed plan evaluation strategy can approach the quality of manually designed decompositions. Fig. 5: Results of Ablation Studies. We conduct ablation studies on four LIBERO tasks to evaluate the effectiveness of key components of HSR, including task decomposition, feature reranking, and the pretraining-finetuning strategy. The results show that each component improves performance, with the full HSR framework achieving the highest success rates across all tasks. figurePerformance under Varying Reranking Retention Ratios. TABLE I: Ablation on Task Decomposition. Method Avg. Succ. (%) LLM Only 37.1± 2.5 LLM+Skill Set 37.7± 1.2 HSR 43.5± 3.3 Human 44.9± 1.0 Fig. 6: Comparison of Retrieved Data. Distribution of retrieved data for the task “put the yellow and white mug in the microwave and close it”. HSR effectively retrieves demonstrations across subtasks covering both object-placing and microwave-closing skills. Feature Reranking. As shown in Fig. 5, removing the feature reranking module and using only language-retrieved data leads to a clear performance drop, especially on tasks that require precise alignment with the target execution context, such as Soup-S. This suggests that language similarity alone is insufficient to capture low-level compatibility between the retrieved demonstrations and the target task. In contrast, the feature reranking module filters out demonstrations that are less compatible in behavior space and retains those that better match the target task, thereby improving policy adaptation. We also ablate the retention ratio after feature reranking, with results shown in Fig. I. The results indicate that retaining a moderate fraction of the reranked data yields the best performance on most tasks. Retaining too much data introduces additional noise, while retaining too little data may discard useful demonstrations. Training method. Another way to leverage retrieved data is to co-train the policy on both the target and retrieved datasets in a single stage. We compare this co-training baseline with our proposed two-stage pretraining-finetuning strategy. As shown in Fig. 5, both co-training variants, with and without feature reranking, achieve lower success rates on most tasks. One possible reason is that retrieved demonstrations are typically larger in scale, noisier, and less well aligned with the target task than the target demonstrations. Co-training may therefore bias optimization toward the retrieved data, which can be suboptimal when the gap between the retrieved and target datasets is large. In contrast, our strategy first uses retrieved data to learn general skills and then finetunes on the most relevant demonstrations, which helps mitigate this issue and provides greater robustness to diverse data sources and embodiments. We further evaluate two-stage adaptation by replacing our retrieval method with BR. This combined approach improves over the original BR baseline, but still underperforms HSR, further demonstrating the benefit of our retrieval strategy. V-D Retrieved Data Study To illustrate the effect of different retrieval methods, we consider a representative task from the LIBERO benchmark, “put the yellow and white mug in the microwave and close it.” The retrieved demonstrations are visualized in Fig. 6. The language-based retrieval baseline relies only on the original task description and may miss task-specific nuances. FR, STRAP, BR, and IWR incorporate additional motion- or behavior-level information, but may still retrieve demonstrations from tasks with similar motions and different goals. Such mismatched data can mislead policy learning. In contrast, HSR decomposes the long-horizon task into subtasks and retrieves demonstrations for each subtask independently. This allows the method to retrieve demonstrations that are more closely aligned with the specific requirements of each step, leading to more task-relevant retrieved data and improved performance. VI Conclusion We present Hierarchical Skill Retrieval (HSR), a framework for data-efficient adaptation of VLA models to downstream manipulation tasks. By exploiting the hierarchical structure of long-horizon tasks, HSR combines high-level semantic decomposition with low-level behavior-aware retrieval to identify useful prior demonstrations. This allows the policy to reuse transferable skills from prior datasets, rather than relying on exact task-level matches. Experiments on the LIBERO benchmark and real-world long-horizon manipulation tasks show that HSR consistently improves adaptation performance under limited target data. Discussion and Limitations. Despite these promising results, several challenges remain. First, the current framework is evaluated only on manipulation tasks and a single VLA backbone. Evaluating HSR across different policy architectures, robot embodiments, and data modalities, such as videos or images collected in diverse domains, remains an important direction for improving robustness and data efficiency. Second, although we use an LLM for task decomposition, tighter integration of language models into closed-loop control, such as online replanning and failure recovery, remains an important direction for future work. In addition, even VLAs trained with additional data may still fail during deployment. Developing mechanisms such as test-time adaptation [48] to handle such failures is also a promising direction for future work. References [1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, p. 8748–8763. Cited by: §I, §I-A. [2] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid loss for language image pre-training. In International Conference on Computer Vision, p. 11975–11986. Cited by: §I, §I-A. [3] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. (2024) OpenVLA: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246. Cited by: §I, §I-A. [4] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. (2025) π0.5 _0.5: A vision-language-action model with open-world generalization. In 9th Annual Conference on Robot Learning, Cited by: §I, §I. [5] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. (2024) Octo: an open-source generalist robot policy. arXiv preprint arXiv:2405.12213. Cited by: §I, §I-A. [6] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. (2024) Open x-embodiment: robotic learning datasets and rt-x models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 6892–6903. Cited by: §I, §I, §I-A, §I-B. [7] S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh (2024) RT-h: action hierarchies using language. arXiv preprint arXiv:2403.01823. Cited by: §I, §I-A. [8] M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zouitine, M. Aractingi, C. Pascal, M. Russi, A. Marafioti, S. Alibert, M. Cord, T. Wolf, and R. Cadene (2025) SmolVLA: a vision-language-action model for affordable and efficient robotics. arXiv preprint arXiv:2506.01844. Cited by: §I, §I-A, §V. [9] P. Li, Y. Wu, Z. Xi, W. Li, Y. Huang, Z. Zhang, Y. Chen, J. Wang, S. Zhu, T. Liu, and S. Huang (2025) ControlVLA: few-shot object-centric adaptation for pre-trained vision-language-action models. In 9th Annual Conference on Robot Learning, Cited by: §I. [10] M. Du, S. Nair, D. Sadigh, and C. Finn (2023) Behavior retrieval: few-shot imitation learning by querying unlabeled datasets. In Robotics: Science and Systems, Cited by: §I, §I-B, §IV-B, §IV-C, TABLE I, item Behavior Retrieval (BR) [], item Behavior Retrieval (BR) []. [11] A. Xie, R. Chand, D. Sadigh, and J. Hejna (2025) Data retrieval with importance weights for few-shot imitation learning. In Proceedings of The 9th Conference on Robot Learning, J. Lim, S. Song, and H. Park (Eds.), Proceedings of Machine Learning Research, Vol. 305, p. 1–16. Cited by: §I, §I-B, §IV-C, TABLE I, item Importance Weighted Retrieval (IWR) [], item Importance Weighted Retrieval (IWR) [], §V. [12] S. Kumar, S. Dass, G. Pavlakos, and R. Martín-Martín (2025) COLLAGE: adaptive fusion-based retrieval for augmented policy learning. In Proceedings of The 9th Conference on Robot Learning, J. Lim, S. Song, and H. Park (Eds.), Proceedings of Machine Learning Research, Vol. 305, p. 4607–4624. Cited by: §I, §I-B. [13] S. Nasiriany, T. Gao, A. Mandlekar, and Y. Zhu (2022) Learning and retrieval from prior data for skill-based imitation learning. In Conference on Robot Learning (CoRL), Cited by: §I, §I-B. [14] M. Memmel, J. Berg, B. Chen, A. Gupta, and J. Francis (2025) STRAP: robot sub-trajectory retrieval for augmented policy learning. In The Thirteenth International Conference on Learning Representations, Cited by: §I, §I-B, TABLE I, item STRAP [], item STRAP [], §V. [15] L. X. Shi, B. Ichter, M. R. Equi, L. Ke, K. Pertsch, Q. Vuong, J. Tanner, A. Walling, H. Wang, N. Fusai, A. Li-Bell, D. Driess, L. Groom, S. Levine, and C. Finn (2025) Hi robot: open-ended instruction following with hierarchical vision-language-action models. In Forty-second International Conference on Machine Learning, Cited by: §I. [16] S. Yang, H. Li, Y. Chen, B. Wang, Y. Tian, T. Wang, H. Wang, F. Zhao, Y. Liao, and J. Pang (2025) Instructvla: vision-language-action instruction tuning from understanding to manipulation. arXiv preprint arXiv:2507.17520. Cited by: §I. [17] J. Barreiros, A. Beaulieu, A. Bhat, R. Cory, E. Cousineau, H. Dai, C. Fang, K. Hashimoto, M. Z. Irshad, M. Itkina, et al. (2025) A careful examination of large behavior models for multitask dexterous manipulation. arXiv preprint arXiv:2507.05331. Cited by: §I. [18] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2024) π0 _0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. Cited by: §I. [19] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) LIBERO: benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 44776–44791. Cited by: §I, §V. [20] A. Billard and D. Kragic (2019) Trends and challenges in robot manipulation. Science 364 (6446), p. eaat8414. Cited by: §I-A. [21] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 34892–34916. Cited by: §I-A. [22] D. Qu, H. Song, Q. Chen, Y. Yao, X. Ye, Y. Ding, Z. Wang, J. Gu, B. Zhao, D. Wang, et al. (2025) SpatialVLA: exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830. Cited by: §I-A. [23] T. Zhang, H. Duan, H. Hao, Y. Qiao, J. Dai, and Z. Hou (2025) Grounding actions in camera space: observation-centric vision-language-action policy. arXiv preprint arXiv:2508.13103. Cited by: §I-A. [24] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. (2022) Rt-1: robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817. Cited by: §I-A. [25] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. (2024) Droid: a large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945. Cited by: §I-B, §V. [26] H. Hao, J. Han, C. Li, Y. Li, and X. Yue (2025) RAP: retrieval-augmented personalization for multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), p. 14538–14548. Cited by: §I-B. [27] L. Lin, Y. Cui, A. Xie, T. Hua, and D. Sadigh FlowRetrieval: flow-guided data retrieval for few-shot imitation learning. In 8th Annual Conference on Robot Learning, Cited by: §I-B, TABLE I, item Flow Retrieval (FR) [], item Flow Retrieval (FR) []. [28] S. N. Syed, Y. Ahuja, A. Jakobsson, and J. Ichnowski (2025) ExpReS-vla: specializing vision-language-action models through experience replay and retrieval. arXiv preprint arXiv:2511.06202. Cited by: §I-B. [29] L. Zha, Y. Cui, L. Lin, M. Kwon, M. G. Arenas, A. Zeng, F. Xia, and D. Sadigh (2024) Distilling and retrieving generalizable knowledge for robot manipulation via language corrections. In 2024 IEEE International Conference on Robotics and Automation, Cited by: §I-B. [30] K. Dreczkowski, P. Vitiello, V. Vosylius, and E. Johns (2025) Learning a thousand tasks in a day. Science Robotics 10 (108), p. eadv7594. External Links: Document, https://w.science.org/doi/pdf/10.1126/scirobotics.adv7594 Cited by: §I-B. [31] L. P. Kaelbling and T. Lozano-Pérez (2011) Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation, Vol. , p. 1470–1477. External Links: Document Cited by: §I-C. [32] H. Sahni, S. Kumar, F. Tejani, and C. Isbell (2017) Learning to compose skills. arXiv preprint arXiv:1711.11289. Cited by: §I-C. [33] M. Dalal, M. Liu, W. Talbott, C. Chen, D. Pathak, J. Zhang, and R. Salakhutdinov (2025) Local policies enable zero-shot long-horizon manipulation. International Conference on Robotics and Automation. Cited by: §I-C. [34] L. P. Kaelbling and T. Lozano-Pérez (2017) Learning composable models of parameterized skills. In 2017 IEEE International Conference on Robotics and Automation (ICRA), Vol. , p. 886–893. External Links: Document Cited by: §I-C. [35] W. Liu, N. Nie, R. Zhang, J. Mao, and J. Wu Learning compositional behaviors from demonstration and language. In 8th Annual Conference on Robot Learning, Cited by: §I-C. [36] J. Mao, T. Lozano-Pérez, J. Tenenbaum, and L. Kaelbling (2022) Pdsketch: integrated domain programming, learning, and planning. Advances in Neural Information Processing Systems 35, p. 36972–36984. Cited by: §I-C. [37] J. Shao, H. Hao, X. Yang, and Y. Li (2025) Abductive learning for neuro-symbolic grounded imitation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, p. 1221–1232. Cited by: §I-C. [38] G. R. Team, S. Abeyruwan, J. Ainslie, J. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijl, et al. (2025) Gemini robotics: bringing ai into the physical world. arXiv preprint arXiv:2503.20020. Cited by: §I-C. [39] Y. Jin, D. Li, J. Shi, P. Hao, F. Sun, J. Zhang, B. Fang, et al. (2024) Robotgpt: robot manipulation learning from chatgpt. IEEE Robotics and Automation Letters 9 (3), p. 2543–2550. Cited by: §I-C. [40] M. Li, S. Zhao, Q. Wang, K. Wang, Y. Zhou, S. Srivastava, C. Gokmen, T. Lee, E. L. Li, R. Zhang, et al. (2024) Embodied agent interface: benchmarking llms for embodied decision making. Advances in Neural Information Processing Systems 37, p. 100428–100534. Cited by: §I-C. [41] X. Li, M. Zhang, Y. Geng, H. Geng, Y. Long, Y. Shen, R. Zhang, J. Liu, and H. Dong (2024) Manipllm: embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 18061–18070. Cited by: §I-C. [42] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. (2022) Do as i can, not as i say: grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, Cited by: §I-C, §IV-A. [43] Y. Meng, X. Yao, H. Ye, Y. Zhou, S. Zhang, Z. Bing, and A. Knoll (2025) Data-agnostic robotic long-horizon manipulation with vision-language-guided closed-loop feedback. Cited by: §I-C. [44] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg (2023) Progprompt: generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), p. 11523–11530. Cited by: §I-C. [45] S. Dass, A. Khaddaj, L. Engstrom, A. Madry, A. Ilyas, and R. Martín-Martín (2026) DataMIL: selecting data for robot imitation learning with datamodels. In The Fourteenth International Conference on Learning Representations, Cited by: §IV-A. [46] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: §V. [47] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186. Cited by: §V. [48] H. Hao, S. N. Syed, J. Ichnowski, and J. Schneider (2026) FAR: failure-aware retry for test-time recovery and continual policy improvement. External Links: 2607.01111, Link Cited by: §VI.