Paper deep dive
A Vision-Language-Action Model for Adaptive Ultrasound-Guided Needle Insertion and Needle Tracking
Yuelin Zhang, Qingpeng Ding, Longxiang Tang, Chengyu Fang, Shing Shin Cheng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 4/26/2026, 5:38:04 PM
Summary
The paper proposes a Vision-Language-Action (VLA) model designed for adaptive and automated ultrasound-guided needle insertion and tracking on a robotic ultrasound (RUS) system. The framework utilizes a Cross-Depth Fusion (CDF) tracking head to integrate shallow positional and deep semantic features for real-time tracking, and a Tracking-Conditioning (TraCon) register for parameter-efficient adaptation of a pretrained vision backbone. To ensure real-time performance and safety, the system employs an asynchronous VLA pipeline that decouples high-frequency tracking from lower-frequency action generation, alongside an uncertainty-aware control policy that adjusts insertion speed based on needle tip visibility. The model is built upon the Qwen2.5-VL-3B architecture and demonstrates superior accuracy and efficiency compared to traditional hand-crafted modular pipelines.
Entities (7)
Relation Signals (5)
Vision-Language-Action (VLA) Model → controls → Robotic Ultrasound (RUS) System
confidence 100% · A Vision-Language-Action (VLA) model is proposed for adaptive and automated US-guided needle insertion and tracking on a robotic ultrasound (RUS) system.
Vision-Language-Action (VLA) Model → incorporates → Cross-Depth Fusion (CDF) Tracking Head
confidence 100% · The proposed VLA framework integrates the CDF tracking head and the TraCon register with a large-scale vision encoder.
Vision-Language-Action (VLA) Model → isbuilton → Qwen2.5-VL-3B
confidence 100% · The proposed VLA is built based on a Qwen2.5-VL-3B model [3].
Cross-Depth Fusion (CDF) Tracking Head → uses → Tracking-Conditioning (TraCon) Register
confidence 90% · the proposed CDF tracking head... integrates... the proposed learnable Tracking-Conditioning (TraCon) Register
Asynchronous VLA Pipeline → decouples → Uncertainty-Aware Control Policy
confidence 80% · An asynchronous VLA pipeline is proposed to decouple visual analysis from action generation... After needle tracking, an uncertainty-aware control policy and an asynchronous VLA pipeline are presented.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Ultrasound (US)-guided needle insertion is a critical yet challenging procedure due to dynamic imaging conditions and difficulties in needle visualization. Many methods have been proposed for automated needle insertion, but they often rely on hand-crafted pipelines with modular controllers, whose performance degrades in challenging cases. In this paper, a Vision-Language-Action (VLA) model is proposed for adaptive and automated US-guided needle insertion and tracking on a robotic ultrasound (RUS) system. This framework provides a unified approach to needle tracking and needle insertion control, enabling real-time, dynamically adaptive adjustment of insertion based on the obtained needle position and environment awareness. To achieve real-time and end-to-end tracking, a Cross-Depth Fusion (CDF) tracking head is proposed, integrating shallow positional and deep semantic features from the large-scale vision backbone. To adapt the pretrained vision backbone for tracking tasks, a Tracking-Conditioning (TraCon) register is introduced for parameter-efficient feature conditioning. After needle tracking, an uncertainty-aware control policy and an asynchronous VLA pipeline are presented for adaptive needle insertion control, ensuring timely decision-making for improved safety and outcomes. Extensive experiments on both needle tracking and insertion show that our method consistently outperforms state-of-the-art trackers and manual operation, achieving higher tracking accuracy, improved insertion success rates, and reduced procedure time, highlighting promising directions for RUS-based intelligent intervention.
Tags
Links
- Source: https://arxiv.org/abs/2604.20347v1
- Canonical: https://arxiv.org/abs/2604.20347v1
Trouble viewing inline? Open PDF directly →
Full Text
46,819 characters extracted from source content.
Expand or collapse full text
A Vision-Language-Action Model for Adaptive Ultrasound-Guided Needle Insertion and Needle Tracking †thanks: Accepted by ICRA 2026. Yuelin Zhang, Qingpeng Ding, Longxiang Tang, Chengyu Fang, Shing Shin Cheng∗ Research reported in this work was supported in part by Research Grants Council (RGC) of Hong Kong (CUHK 14217822, CUHK 14207823, CUHK 14211425, T45-401/22-N, and AoE/E-407/24-N) and in part by Innovation and Technology Commission of Hong Kong (MHP/096/22, ITS/235/22, ITS/224/23, ITS/225/23, and Multi-scale Medical Robotics Center (InnoHK initiative)).Yuelin Zhang and Qingpeng Ding are with the Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong.Longxiang Tang is with the Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong. Chengyu Fang is with the Shenzhen International Graduate School, Tsinghua University, China.Shing Shin Cheng is with the Department of Mechanical and Automation Engineering, T Stone Robotics Institute, Shun Hing Institute of Advanced Engineering, Multi-Scale Medical Robotics Center, and Institute of Medical Intelligence and XR, The Chinese University of Hong Kong, Hong Kong.∗sscheng@cuhk.edu.hk Abstract Ultrasound (US)-guided needle insertion is a critical yet challenging procedure due to dynamic imaging conditions and difficulties in needle visualization. Many methods have been proposed for automated needle insertion, but they often rely on hand-crafted pipelines with modular controllers, whose performance degrades in challenging cases. In this paper, a Vision-Language-Action (VLA) model is proposed for adaptive and automated US-guided needle insertion and tracking on a robotic ultrasound (RUS) system. This framework provides a unified approach to needle tracking and needle insertion control, enabling real-time, dynamically adaptive adjustment of insertion based on the obtained needle position and environment awareness. To achieve real-time and end-to-end tracking, a Cross-Depth Fusion (CDF) tracking head is proposed, integrating shallow positional and deep semantic features from the large-scale vision backbone. To adapt the pretrained vision backbone for tracking tasks, a Tracking-Conditioning (TraCon) register is introduced for parameter-efficient feature conditioning. After needle tracking, an uncertainty-aware control policy and an asynchronous VLA pipeline are presented for adaptive needle insertion control, ensuring timely decision-making for improved safety and outcomes. Extensive experiments on both needle tracking and insertion show that our method consistently outperforms state-of-the-art trackers and manual operation, achieving higher tracking accuracy, improved insertion success rates, and reduced procedure time, highlighting promising directions for RUS-based intelligent intervention. I INTRODUCTION Ultrasound (US)-guided needle insertion is frequently used in a variety of percutaneous procedures, including minimally invasive interventions like tissue biopsy, tumor ablation, and regional anesthesia [20]. As a non-invasive, portable, safe, and cost-effective imaging technique [27], US offers real-time intraoperative visualization of both the needle and surrounding tissue, thus helping to reduce the risk of inadvertent injuries. Some US needle trackers have been proposed to provide real-time needle position feedback. However, even with accurate needle position feedback, precise needle insertion still requires skillful operators, which can be challenging in resource-limited settings. Robotic ultrasound (RUS) has therefore emerged as a transformative technology for automated needle insertion, offering enhanced precision and consistency. Recent advancements have enabled RUS systems to perform complex tasks such as probe manipulation, optimal image acquisition, and needle steering, thereby reducing operator dependency and improving procedure outcomes [5]. There are many challenges during needle insertion, where procedure outcomes can be affected by occlusion, imaging artifacts, and intermittent needle invisibility [14]. Traditional hand-crafted automated needle-insertion pipelines [15, 5, 13] remain fragile when facing these challenges, which require not only accurate needle tracking for closed-loop control but also generalizable context awareness to proactively adapt to dynamic environments. This highlights the need for better generalizability and high-level reasoning beyond conventional feature engineering methods. Building upon the developments of Large-Language Models (LLMs) [1] and Vision-Language Models (VLMs) [3], Vision-Language-Action (VLA) [21] models have gained increasing attention in both open-world tasks and medical fields, as they can both understand multimodal information and generate actionable outputs. Although VLA models demonstrate robust logical reasoning abilities and strong generalization to dynamic environments, research on VLA-based automated needle insertion and tracking remains limited due to their high computational and data requirements. Achieving satisfactory accuracy while maintaining efficiency in VLA models remains challenging. In this work, a novel VLA framework for automated and adaptive US-guided needle insertion on a RUS system is proposed for the first time. Without reliance on external sensors or pre-registration, the proposed framework holds promise for safer, more standardized, and operator-independent clinical workflows than conventional visual servoing needle steering pipelines. To enable adaptive needle insertion, accurate real-time needle position feedback is indispensable. Instead of incorporating a separate tracker like traditional methods, a Cross-Depth Fusion (CDF) Tracking Head is introduced as an end-to-end tracking head to incorporate cross-layer features from the pretrained vision backbone. Compared to hand-crafted non-end-to-end pipelines with independent needle trackers based on particle filter [5] or one-stream pipelines with computationally expensive VLM-based object grounding [24], the proposed framework not only ensures end-to-end training but also maintains efficiency. By further integrating the proposed learnable Tracking-Conditioning (TraCon) Register, the pretrained vision backbone is parameter-efficiently conditioned for robust tracking. Based on the needle position feedback, the VLA then controls the needle to reach the target with the proposed uncertainty-aware control policy, which ensures procedural success and safety when facing uncertainty due to occlusion, imaging artifacts, or intermittent tip invisibility. Leveraging large pretrained models, it provides better generalizability to dynamic US environments than traditional methods. An asynchronous VLA pipeline is proposed to decouple visual analysis from action generation, which operates at different latencies, thereby achieving precise action generation while ensuring real-time tracking. Extensive experiments demonstrate the effectiveness of the proposed VLA-based tracking-control pipeline. The contributions of our work are fourfold: • To provide accurate real-time needle position feedback for insertion control, a Cross-Depth Fusion (CDF) Tracking Head is developed to integrate shallow positional and deep semantic features, better adapting deep vision backbone for tracking tasks. • A Tracking-Conditioning (TraCon) Register is introduced as a lightweight learnable token for parameter-efficient transferable vision backbone adaptation, facilitating tracking-oriented conditioning. • To achieve adaptive needle insertion, an uncertainty-aware control policy is implemented, ensuring procedural safety and success under imaging artifacts and procedural uncertainty. • To accommodate different execution speeds of tracking head and LLM, an asynchronous VLA pipeline is proposed, simultaneously ensuring real-time needle tracking and precise insertion control. I RELATED WORK I-A Ultrasound Needle Tracking and Robotic Ultrasound Convolutional neural networks (CNNs) and transformer-based deep learning approaches [29, 34] have become widely adopted for tracking applications [16, 36], including ultrasound (US) needle tracking [23, 35, 37]. Given that the needle tip can be obscured by noise and artifacts, subsequent works employ segmentation of the needle shaft as a precursor to tip localization [31]. Another strategy is to incorporate temporal needle motion information into tracking. In [33], a motion prediction mechanism is combined with a visual tracker to enable motion cues beyond vision information. However, after acquiring needle position, the existing needle trackers failed to be integrated with control models to achieve intelligent intervention with needle position feedback. To improve patient outcomes, the robotic ultrasound (RUS) systems are increasingly being integrated with artificial intelligence to enable autonomous procedures. Khadem et al. [13] introduced an autonomous US-guided intervention system that leverages nonlinear model predictive control (MPC) for accurate and adaptive needle targeting in soft tissue. By integrating particle filter-based needle tracking with closed-loop visual servoing, a real-time 3D US-guided robotic needle steering system is proposed in [5] for autonomous flexible needle insertion. With further integration with segmentation-based state estimation, [15] proposed a US needle insertion system addressing noisy tissue features. However, these modular feature-engineering pipelines with stacked separate modules remain brittle in dynamic environments, as they typically rely heavily on the accuracy of the physics-based model and require numerous hyperparameters for complex optimization, not to mention the impractical assumptions of persistent visibility and quasi-static status. It thus motivates a VLA-based controller that unifies perception and action for context-aware needle insertion control. I-B VLA Models in Medical Applications Based on LLMs and VLMs, Vision-Language-Action (VLA) models have rapidly evolved, enabling systems to perceive, interpret, and interact with complex environments through multimodal understanding and reasoning [21, 30]. In the medical domain, VLA models have shown great promises. CapsDT [11] is proposed as a VLA model for endoscopy capsule robot manipulation with diffusion transformer [25], demonstrating state-of-the-art performance in complex gastrointestinal tasks. More recently, EndoVLA [24] introduces a dual-phase VLA framework designed for robotic endoscopy, enabling robust autonomous tracking of abnormal regions through end-to-end instruction imitating and reinforced with task-specific rewards. Although it achieves end-to-end object tracking and action generation within VLA, its use of a coupled structure for both tasks significantly compromises efficiency. Due to the scarcity of RUS action datasets and the inherently high dynamics, noise, and indistinct target features of US images compared to other modalities, such VLA models for US procedures have yet to be proposed, despite their potential effectiveness. Figure 1: The proposed robotic ultrasound (RUS) platform. Figure 2: Structure overview. The proposed VLA framework integrates the CDF tracking head and the TraCon register with a large-scale vision encoder, and decouples tracking from the proposed uncertainty-aware control through an asynchronous pipeline. This design enables accurate real-time needle tracking and precise, context-aware insertion control. I System Design I-A Robotic Ultrasound Platform A robotic ultrasound (RUS) platform was developed for automated needle insertion, which includes a needle insertion robot and a UR5e robotic arm, as shown in Fig. 1. The needle insertion robot consists of a linear manipulator for controlling the needle’s axial displacement xnx_n (defined in the needle coordinate frame) and a servo for adjusting the insertion angle θn _n. The UR5e robotic arm is for manipulating the US probe in three translational degrees of freedom xp=[xp,yp,zp]x_p=[x_p,y_p,z_p], expressed in the world coordinate frame. I-B Vision-Language-Action Model Overview The proposed VLA is built based on a Qwen2.5-VL-3B model [3]. As shown in Fig. 2, the proposed VLA model adopts a pretrained vision encoder ϵV _V based on the Vision Transformer (ViT) [8], a pretrained LLM ϕL _L for interpreting input and generating actions, and a separate Cross-Depth Fusion (CDF) tracking head ϕT _T specifically designed for real-time needle tracking. In addition, the proposed Tracking-Conditioning (TraCon) register is appended to input image embeddings as a learnable, parameter-efficient, and task-oriented token, which is then applied in tracking head and LLM for task adaptation. In object tracking [16], the network typically performs prediction based on the search map x and the template map z, where z encodes the target object’s appearance from the initial frame, while x represents the region in subsequent frames where the target is likely to be located. The model identifies the object in x that most closely matches z and predicts a bounding box representing the target’s location. The framework receives ultrasound imaging observation Ot=[xt,zt]O_t=[x_t,z_t] and the language instruction I, predicts needle position P=ϕT(ϵV(Ot))P= _T( _V(O_t)), and then predicts action A via an asynchronous inference pipeline to achieve automated adaptive needle insertion. I-C Tracking-Conditioning Register The proposed Tracking-Conditioning (TraCon) register is a lightweight learnable token for task-oriented parameter-efficient fine-tuning (PEFT). Most existing VLA models perform PEFT on the pretrained vision encoder by LoRA [12] to ensure that the pretrained model can be adapted to downstream tasks. However, such PEFT methods place high demands on the quantity and quality of labeled training data, while acquiring labeled datasets in medical scenarios is often difficult and expensive. Besides, in the proposed model, the output of the vision encoder is utilized by both the tracking head and the LLM. Directly fine-tuning the vision encoder would inevitably introduce performance degradation due to inconsistent training objectives. Instead of applying PEFT to the whole vision encoder, the proposed TraCon register R∈ℝB×Lr×CR ^B× L_r× C (Lr=4L_r=4 in this paper) is appended to the vision observation OtO_t as a lightweight trainable token (0.5 M parameters), externally conditioning the frozen vision backbone for object tracking and aligning it with both the tracking head and the LLM. After being encoded by the vision encoder ϵV _V, the encoded TraCon register ϵV(R) _V(R) is subsequently fed into both the CDF tracking head ϕT _T and LLM ϕL _L. In CDF head, ϵV(R) _V(R) is fused with vision embeddings via the proposed Conditional Feature Gating. In LLM, ϵV(R) _V(R) is inserted into the input sequence to facilitate cross-modal knowledge transfer. Figure 3: Asynchronous VLA pipeline (left) and its sequence diagram (right). The runtime of the tracking cycle and control cycle is measured while they are running simultaneously on a single NVIDIA A800 GPU. The lengths in the sequence diagram are for illustration only. I-D Cross-Depth Fusion Tracking Head The Cross-Depth Fusion (CDF) tracking head ϕT _T is proposed as a dedicated module for needle tracking, which is shown in Fig. 2. In CDF head, multi-depth features from the ViT vision encoder ϵV _V are simultaneously incorporated to learn shallow positional information and deep semantic information, since the shallow layer outputs in ViT predominantly encode positional information due to their proximity to the input embeddings and limited receptive field, whereas deeper layer outputs capture more semantic information as a result of accumulated global context and hierarchical feature abstraction [8]. Specifically, shallow features and deep features (together with ϵV(R) _V(R)) will first go through a MLP dimension reductor (to reduce backbone dimension 2048 to internal dimension C=512C=512) and a layer normalization [2] to get xSx_S, zSz_S, xDx_D, zDz_D, and r, respectively, where xSx_S and zSz_S are shallow outputs of search and template maps, xDx_D and zDz_D are deep outputs. To incorporate deep semantic features into tracking prediction without compromising the shallow positional features, a cross-depth semantic fusion is first performed separately to (xS,xD)(x_S,x_D) and (zS,zD)(z_S,z_D) to get xfx_f and zfz_f. The core of semantic fusion can be formulated as: SemFus=(softmax(Qch(Kch)TL)Vch)T∈ℝB×L×C,SemFus=(softmax( Q_ch(K_ch)^T L)V_ch)^T ^B× L× C, (1) where Qch=Linear(q)TQ_ch=Linear(q)^T, Kch=Linear(k)TK_ch=Linear(k)^T, and Vch=Linear(v)TV_ch=Linear(v)^T are channel sequences in the dimension of B×C×LB× C× L. The full semantic fusion block employs a standard transformer-style architecture [29], which includes a residual shortcut, layer normalization [2], and a MLP applied after the attention module. With the cross-depth semantically enhanced maps xfx_f and zfz_f, the next stage is the positional correlation. The positional correlation correlates xfx_f with zfz_f by modeling positional attention to precisely localize target zfz_f on search map xfx_f: PosCor=softmax(Q(K)TC)V∈ℝB×L×C,PosCor=softmax( Q(K)^T C)V ^B× L× C, (2) where Q=Linear(q)Q=Linear(q), K=Linear(k)K=Linear(k), and V=Linear(v)V=Linear(v) are all in B×L×CB× L× C. Similar to semantic fusion, the remaining parts of positional correlation also follow the standard transformer block design [29]. Finally, before predicting the bounding box by a convolution layer (stride=1stride=1, kernel=1×1kernel=1× 1), the fused map f is integrated with register r using the proposed Conditional Feature Gating. This operation conditions the primary vision feature map f on auxiliary register embeddings that capture task-specific priors, enabling dynamic modulation without altering the spatial structure of f. It is formulated by: f′=f⊙(α⋅γ(r))+α⋅β(r),f =f (α·γ(r))+α·β(r), (3) where γ,β∈ℝB×Cγ,β ^B× C are linear transformations, and α is a learnable scalar gate. Compared with the existing FiLM (Feature-wise Linear Modulation) [26], a learnable scalar α is added to adaptively control the modulation strengths. This mechanism enhances feature adaptability by conditioning on register-derived priors, promoting task-specific refinements in visual tracking. With this regressive CDF tracking head, the proposed VLA model possesses several intrinsic advantages on tracking. Existing VLA models with object tracking like EndoVLA [24] typically couples action generation and tracking together to the LLM, where tracking is accomplished in an auto-regressive manner with LLM-based object grounding. This results in inefficiency, with tracking and control sharing a low-speed pipeline running at 2 FPS, which falls significantly short of real-time requirements. In contrast, by leveraging the dedicated CDF tracking head with decoupled action and tracking pipelines, the proposed VLA achieves significantly higher tracking efficiency (∼25 25 FPS) with fewer parameters. Furthermore, compared to hand-crafted models that combine separate needle trackers and insertion controllers [13, 5, 15], the CDF head and action generator (LLM) share the same pretrained vision backbone, enabling higher consistency between tracking and control. I-E Asynchronous VLA Pipeline After obtaining the needle position, the next stage is to generate action with LLM for insertion control. If tracking waits in a blocking manner for the LLM to complete before processing the next frame, like the existing one-stream methods [24], real-time tracking then cannot be guaranteed. To address this issue, an asynchronous VLA pipeline is proposed, as shown in Fig. 3. The proposed pipeline includes a real-time branch for needle tracking (∼25 25 FPS) and a deep branch for action generation (∼10 10 FPS). After the vision encoder ϵV _V finishes encoding one frame, the tracking will then start. Once the needle tracking prediction P is obtained by ϕT _T, it will be sent to both user interface (UI) for display and LLM ϕL _L for action generation. While the LLM generates actions, the vision encoding and tracking prediction for the next frame start immediately. Once the LLM completes action prediction, it will receive the latest available vision embedding without waiting to initiate the next round of action generation. I-F Uncertainty-Aware Control Policy In US-guided needle insertion, the tip visibility is dynamic and fragile. When the tip fades, before the change is detected and reacted upon, the needle will lose tracking and a steady insertion speed can translate into millimeters of unseen advancement, leading to discontinuities in tip visualization and clinical danger near critical structures. Instead of relying on the operator to subjectively determine when to pause the needle advancement, a VLA-based uncertainty-aware control policy is proposed, following the control law by skilled operators: see what you’re doing, and act more slowly when uncertainty increases (uncertainty is defined as the inverse of tip visibility) [28, 4]. After the CDF head ϕT _T predicts the needle tip position P, LLM ϕL _L generates action =ϕL(ϵV(Ot),I,P)A= _L( _V(O_t),I,P), where ϵV(Ot) _V(O_t) is the encoded vision embeddings. P is given by the bounding box [x1,y1,x2,y2][x1,y1,x2,y2] of the needle tip, represented by its top-left (x1,y1)(x1,y1) and bottom-right (x2,y2)(x2,y2) coordinates in pixels. The language instruction I depicts the basic information and provides high-level command, given by “You are an ultrasound expert. The target position is [TARGET]. The insertion technique is [TECH]. Control the insertion based on visibility feedback: when visibility decreases, insert with a slower speed. Stop insertion upon reaching the target.” [TARGET] is the position of insertion target given by its center pixel coordinates. [TECH] is the insertion technique (in-plane-static or in-plane-moving [33]). The predicted action A is given by =[θn,vn,vp]A=[ _n,v_n,v_p], where θn _n is the needle insertion angle, vnv_n is the insertion speed, vp=[vp,x,vp,y,vp,z]v_p=[v_p,x,v_p,y,v_p,z] is the probe moving speed. The needle and probe positions are then given by xn=vnΔt+xn,0x_n=v_n t+x_n,0 and xp=vp⋅Δt+xp,0x_p=v_p· t+x_p,0, where xn,0x_n,0 and xp,0x_p,0 are their initial positions. When the needle reaches the target, the model generates a [STOP] token to terminate the procedure. Leveraging the pretrained LLM, the proposed VLA-based control policy provides high-level reasoning based on global contextual information, in contrast to the ungeneralizable segmentation priors [15] and explicit needle deflection predictors [13] used in traditional needle steering controllers. This language conditioning keeps the LLM architecture unchanged while encoding semantically rich control context. Moreover, unlike traditional methods that require hyperparameter-controlled decision making, the proposed VLA model directly learns the control policy introduced by the expert from the training dataset. The insertion can then be adaptively controlled by visual reasoning, thus avoiding lost tip tracking during low-visibility period and ensuring a lower safe velocity near complex tissue structures. I-G 2-Stage Training and Dataset Collection The proposed VLA model is trained by 2 stages. Stage 1 is for pretraining on the tracking task, where only the CDF tracking head and TraCon register are trained and the remaining modules adopted from Qwen2.5-VL-3B [3] are frozen. A US needle tracking dataset 1=(ok,pk)k=1ND_1=\(o_k,p_k)\^N_k=1 was collected for stage 1, where oko_k is the image, pkp_k is the needle position ground truth measured by an optical tracker ClaroNav MicronTracker 3 (with an acceptable RMSE of 0.189 m in our setup). It was acquired using a Wisonic Clover 60 US machine and a Wisonic C5-1 convex transducer with an 18 gauge needle. 1D_1 contains 41,075 frames from 239 videos (1920×10801920× 1080) by 105 in-plane-static (IPS) trials and 134 in-plane-moving (IPM) trials [14, 33] at a velocity of 20 m/s and 3 different insertion angles (30∘30 , 45∘45 , 60∘60 ). The needle and transducer (i.e. US probe) were manipulated by the aforementioned RUS system. Several different materials were used for needle insertion, including phantoms made from fresh pork and solidified agar, as well as simulators composed of silicone and artificial tumors. 1D_1 is divided into training, validation, and testing sets in the ratio 7:1:2. Stage 2 is for fine-tuning the VLA model for adaptive needle insertion. Only the LLM and MLP aligner are fine-tuned with LoRA [12] while the remaining modules including CDF head and TraCon register are frozen. The pretrained vision encoder is frozen during both stages. A US needle insertion dataset 2=(ok,pk,ik,ak)k=1ND_2=\(o_k,p_k,i_k,a_k)\^N_k=1 was collected using the same devices and setup as 1D_1. In addition to image oko_k and needle positions pkp_k, 2D_2 also includes language instructions iki_k and action ground truth aka_k. aka_k was collected through expert demonstrations, in which an experienced operator manually manipulated the needle to reach specified targets, following principles similar to the proposed uncertainty-based control policy. During these demonstrations, the needle velocity vnv_n, insertion angle θn _n, and probe velocity vpv_p were recorded in real time. The operator recorded a [STOP] sign upon reaching the target. 2D_2 contains 3,852 frames from 18 videos by 9 IPS trials and 9 IPM trials. It is only used for training, not for testing. IV Experiments and Results IV-A Implementation Details The evaluation was carried out on needle tracking and needle insertion respectively 111See examples of needle tracking and insertion in supplementary video.. All experiments are implemented using PyTorch on a server with two NVIDIA A800 GPUs (although often considered high-end, A800 is based on an old NVIDIA Ampere architecture that is slower than current flagships). For needle tracking evaluation, several state-of-the-art trackers are involved for comparison, including classic Siamese trackers [16, 10, 6], transformer-based trackers [18, 7], CNN-based trackers [9], and large-scale trackers [19]. All these models were trained on the same dataset (training set of 1D_1) by the same strategy (350 epochs, batch size 48, AdamW optimizer) as ours. Scaling, blur, and shifting were adopted for augmentation. The learning rate was set to 1e-4 and dropped by 10 after 100 epochs. For needle insertion evaluation, the model pretrained on 1D_1 was further fine-tuned on dataset 2D_2 for a single epoch with a batch size of 16 and an AdamW optimizer. The learning rate was initialized to 1e-4 and modulated by a cosine annealing scheduler. IV-B Needle Tracking Evaluation and Ablation Studies Models were trained and evaluated on the training set and testing set of 1D_1. The result is reported as area under curve (AUC) [32] and precision (P) [22] in percentage, as well as average error (Err) and standard deviation (SD) in m. As shown in Tab. I, the proposed tracking pipeline achieves the best performance in almost all comparisons against the other SOTA trackers. Our tracker achieves 10.7% and 16.0% improvement on Err and SD over the second best method, showing advancement on tracking accuracy and robustness. Furthermore, our framework is the only approach that achieves a SD of less than 2 on both the IPS and IPM tasks. The tracking demonstration in Fig. 4 further illustrates that the proposed method attains the most robust and accurate tracking under challenging conditions. This accurate tracking serves as the basis of successful automated insertion. Ablation studies were performed on four variants, as shown in Tab. I. T,1V_T,1 and T,2V_T,2 investigate the TraCon register and the impact of its length LrL_r. The results in T,1V_T,1 indicate that using a longer register does not consistently improve performance and may even lead to undesired outcomes. In contrast, removing the register results in performance degradation across almost all metrics in T,2V_T,2. These findings demonstrate that the proposed TraCon register can effectively enhance tracking through model conditioning. T,3V_T,3 and T,4V_T,4 investigate the significance of fusing cross-depth features. They represent variants where cross-depth semantic fusion is removed and only shallow feature or deep feature is kept respectively. The results show that receiving only shallow or deep features leads to degraded performance. This indicates that taking into account both shallow positional features and deep semantic features is critical for tracking. Figure 4: Needle tracking demonstration in a tissue phantom. The proposed method outperforms two state-of-the-art trackers under dynamic environment and degraded visibility. TABLE I: Evaluation and ablation study results of needle tracking in AUC (%), P (%), Err (m), and SD (m). The methods with the best and the second best performance are in red and cyan. Method In-plane-static (IPS) In-plane-moving (IPM) Mean AUC↑ P↑ Err± ↓ AUC↑ P↑ Err± ↓ AUC↑ P↑ Err± ↓ SiamRPN++ [16] 45.0 60.1 5.55± 3.19 60.2 76.3 4.10± 3.64 51.4 66.9 4.90± 3.41 SiamCAR [10] 53.1 71.0 4.87± 3.28 60.4 83.3 3.38± 3.09 56.2 76.2 4.27± 3.21 SiamBAN [6] 53.0 72.7 4.75± 3.30 65.4 92.0 4.13± 2.99 58.2 80.8 4.50± 3.19 SwinTrack [18] 49.6 70.6 4.99± 3.37 65.8 94.0 3.41± 2.93 56.4 80.4 4.37± 3.09 STMTrack [9] 52.1 74.1 4.96± 3.20 64.3 92.5 3.17± 3.03 57.2 81.9 4.16± 2.98 MixFormer [7] 57.9 77.0 4.32± 2.77 68.0 93.9 3.01± 2.73 62.1 84.1 3.79± 2.77 LoRAT [19] 59.7 79.0 3.82± 2.55 65.1 93.7 2.73± 2.68 62.0 85.2 3.37± 2.62 Ours 60.3 84.0 3.45± 1.92 67.9 95.5 2.29± 1.98 63.5 88.9 3.01± 2.20 Ablations Mean AUC↑ P↑ Err± ↓ Baseline: default 63.5 88.9 3.01± 2.20 T,1V_T,1: Lr=16L_r=16 63.7(+0.2) 88.5(-0.4) 3.10(+0.09)± 2.19(-0.01) T,2V_T,2: Lr=0L_r=0 (w/o TraCon R) 62.8(-0.7) 86.1(-2.8) 3.18(+0.17)± 2.10(-0.10) T,3V_T,3: w/o fusion, only shallow 62.1(-1.4) 86.4(-2.5) 3.49(+0.48)± 2.62(+0.42) T,4V_T,4: w/o fusion, only deep 61.7(-1.8) 87.0(-1.9) 3.32(+0.31)± 2.95(+0.75) IV-C Needle Insertion Evaluation and Ablation Studies The model trained on 1D_1 was further fine-tuned on dataset 2D_2, where only LLM and MLP aligner were trained with LoRA. The comparison is performed between the proposed VLA-based RUS insertion and manual insertion regarding success rate (SUC) in percentage and procedure time (T) in seconds. For SUC, a successful attempt is defined as the needle tip reaching within 5 m of the target point. Any deviation of the needle from the target or complete loss of needle visualization during the process is considered a failure. T denotes the average completion time across all successful trials. For VLA-based RUS insertion, a total of 40 attempts were performed (20 IPS and 20 IPM). For manual insertion, five experienced users each performed 4 IPS and 4 IPM insertions, also resulting in a total of 40 attempts (20 IPS and 20 IPM). Ablation studies on RUS insertion were conducted using the same protocol. For each target position, both RUS and manual insertion were conducted. This study does not compare with conventional hand-crafted pipelines or existing VLA models for US needle insertion, as such methods are scarce and lack open-source implementations. Instead, we focus on comparisons with manual operation, which remains the mainstream approach in clinical practice and directly reflects the practical benefits and translational potential for real-world adoption. As shown in Tab. I, the proposed VLA-based tracking-insertion pipeline achieves a significant 33.3% SUC improvement with less time consumption than manual insertion. For IPM insertion, the SUC improvement even reached 63.6% with an average time reduction of 7.1 s. Furthermore, with the proposed asynchronous pipeline, the average frame rates for tracking and action generation reached 25.1 FPS and 10.4 FPS, respectively. It can not only provide operators with real-time needle position feedback, but also ensure a safe and acceptable action generation frequency, which surpasses that of most existing VLA models [24, 17]. Two examples are shown in Fig. 5, where the insertion speed is dynamically adjusted by the proposed VLA framework to ensure consistent needle visualization and improved outcomes. In the IPM case, needle insertion is slowed during the initial stage due to increased uncertainty. The speed gradually recovers in the middle stage, and as the needle approaches the target, insertion slows again to account for ambiguity caused by tissue occlusion. This dynamic adaptation allows the needle to respond appropriately at any position, thereby improving the success rate. Ablation studies were performed on four variants. I,1V_I,1 adopts the same structure as T,1V_T,1 in Tab. I. The performance degradation demonstrates that poor tracking leads to a higher rate of insertion failures. Without the TraCon register, I,2V_I,2 results in more insertion failures and increased time consumption. It further demonstrates the significance of PEFT enabled by the TraCon register, highlighting its substantial improvement of PEFT results even in scenarios with extremely limited training data (only 18 videos in 2D_2). I,3V_I,3 shows a significant SUC decrease when the asynchronous pipeline is removed, due to the inability to respond promptly to the latest frames. Since the entire pipeline operates synchronously, tracking and action generation must share an inference speed of 13.1 FPS, which falls short of real-time requirements and results in an inevitable discrepancy between the estimated and actual needle positions. I,4V_I,4 removes the uncertainty-aware control policy, inserting the needle at a constant velocity of 10m/s10~m/s and stopping upon reaching the target. The observed decrease in SUC indicates that, when uncertainty increases, continuing insertion at a fixed speed without intervention can lead to tracking loss and insertion failure. This highlights the importance of controlling insertion with environment awareness. Figure 5: Two examples of adaptive needle insertion with uncertainty-aware control. The plot of vnv_n and t is given. The target position is specified by the operator. The proposed framework proactively responds to environmental uncertainty by adjusting the speed when the needle is occluded by tissue to ensure tracking and safety. TABLE I: Evaluation and ablation study results of needle insertion in SUC (%), T (s). Method In-plane-static (IPS) In-plane-moving (IPM) Mean SUC↑ (%) T (s) SUC↑ (%) T (s) SUC↑ (%) T (s) Ours 70.0 12.1 90.0 23.9 80.0 17.3 Manual 65.0 17.1 55.0 31.0 60.0 23.2 Ablations Mean SUC↑ (%) T (s) Baseline: default 80.0 17.3 I,1V_I,1: Lr=16L_r=16 75.0(-5.0) 16.1(-1.2) I,2V_I,2: Lr=0L_r=0 (w/o TraCon R) 72.5(-7.5) 24.5(+7.2) I,3V_I,3: w/o async pipeline 67.5(-12.5) 15.0(-2.3) I,4V_I,4: w/o uncertainty control 70.0(-10.0) 8.7(-8.6) IV-D Discussion The improvement can be attributed to two factors: first, the efficient adaptation of the large-scale pretrained vision backbone; second, the LLM-enabled high-level reasoning. In contrast to existing large-scale vision trackers like LoRAT [19] that directly use the deepest layer outputs of large backbones, the proposed CDF tracking head in this work efficiently fuses and utilizes multi-level features from different layers of the deep vision encoder, avoiding positional information loss like LoRAT. It can be observed in Tab. I that our method outperforms LoRAT on most metrics, benefiting from the CDF head’s cross-depth fusion. Besides, unlike LoRAT, this work does not fine-tune the vision encoder for tracking, as fine-tuning the vision encoder shared by both the tracking head and the LLM could introduce feature imbalance between them, which would negatively affect LLM-based action generation. Furthermore, ablation studies on the TraCon register show that the vision encoder can be externally conditioned instead of internally fine-tuned, even when utilizing a minimal number of trainable parameters. Such a lightweight PEFT approach is crucial for preventing model overfitting in US tasks, given that US datasets usually consist of relatively few samples and insufficient diversity. These improvements ensure accurate needle position tracking even in dynamic environments. Based on accurate needle position feedback, adaptive needle insertion control can be achieved. The results in Tab. I demonstrate the effectiveness of the proposed framework. The LLM is trained on large-scale open-world data, inherently capturing both semantic information and physical commonsense. Compared to the subjective and user-dependent nature of manual insertion, the proposed method leverages the LLM’s generalizable environment-aware capability to enable adaptive control. When imaging is affected by occlusion, artifacts, or intermittent needle visibility, the proposed control policy can make context-aware decisions to ensure continuous visualization of the needle. Last but not least, the asynchronous VLA pipeline enables non-interfering synergy between tracking and control, ensuring real-time tracking and action generation. This contributes to a higher insertion success rate, as shown in Tab. I (I,3V_I,3). As such, the asynchronous pipeline is an indispensable component of the proposed VLA insertion-tracking model. V CONCLUSIONS In this paper, a VLA framework is proposed for adaptive US-guided needle insertion and tracking. Extensive experiments demonstrate that our framework achieves state-of-the-art tracking accuracy and significantly improves insertion success rates compared to manual operation. These results highlight the potential of VLA models for enhancing safety and efficiency in RUS needle insertion. One limitation is that the tracking speed only barely meets real-time requirements, indicating room for further efficiency improvements. In future work, multi-degree-of-freedom probe manipulation will be developed to proactively enhance needle visibility. A dataset with a larger cohort will be collected in the future. References [1] J. Achiam et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §I. [2] J. L. Ba, J. R. Kiros, and G. E. Hinton (2016) Layer normalization. arXiv preprint arXiv:1607.06450. Cited by: §I-D, §I-D. [3] S. Bai et al. (2025) Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §I, §I-B, §I-G. [4] F. Casanova, P. R. Carney, and M. Sarntinoranont (2014) Effect of needle insertion speed on tissue injury, stress, and backflow distribution for convection-enhanced delivery in the rat brain. PLoS One 9 (4), p. e94919. Cited by: §I-F. [5] P. Chatelain et al. (2015) 3D ultrasound-guided robotic steering of a flexible needle via visual servoing. In 2015 IEEE international conference on robotics and automation (ICRA), p. 2250–2255. Cited by: §I, §I, §I, §I-A, §I-D. [6] Z. Chen et al. (2022) SiamBAN: target-aware tracking with siamese box adaptive network. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (4), p. 5158–5173. Cited by: §IV-A, TABLE I. [7] Y. Cui et al. (2022) Mixformer: end-to-end tracking with iterative mixed attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 13608–13618. Cited by: §IV-A, TABLE I. [8] A. Dosovitskiy et al. (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Cited by: §I-B, §I-D. [9] Z. Fu et al. (2021) Stmtrack: template-free visual tracking with space-time memory networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 13774–13783. Cited by: §IV-A, TABLE I. [10] D. Guo et al. (2020) SiamCAR: siamese fully convolutional classification and regression for visual tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6269–6277. Cited by: §IV-A, TABLE I. [11] X. He et al. (2025) CapsDT: diffusion-transformer for capsule robot manipulation. arXiv preprint arXiv:2506.16263. Cited by: §I-B. [12] E. J. Hu et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), p. 3. Cited by: §I-C, §I-G. [13] M. Khadem et al. (2016) Ultrasound-guided model predictive control of needle steering in biological tissue. Journal of Medical Robotics Research 1 (01), p. 1640007. Cited by: §I, §I-A, §I-D, §I-F. [14] A. Kimbowa et al. (2024) Advancements in needle visualization enhancement and localization methods in ultrasound: a literature review. Artificial Intelligence Surgery 4 (3), p. 149–169. Cited by: §I, §I-G. [15] G. Lapouge et al. (2020) Towards 3d ultrasound guided needle steering robust to uncertainties, noise, and tissue heterogeneity. IEEE Transactions on Biomedical Engineering 68 (4), p. 1166–1177. Cited by: §I, §I-A, §I-D, §I-F. [16] B. Li et al. (2019) Siamrpn++: evolution of siamese visual tracking with very deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 4282–4291. Cited by: §I-A, §I-B, §IV-A, TABLE I. [17] S. Li et al. (2024) Robonurse-vla: robotic scrub nurse system based on vision-language-action model. arXiv preprint arXiv:2409.19590. Cited by: §IV-C. [18] L. Lin et al. (2022) Swintrack: a simple and strong baseline for transformer tracking. Advances in Neural Information Processing Systems 35, p. 16743–16754. Cited by: §IV-A, TABLE I. [19] L. Lin et al. (2024) Tracking meets lora: faster training, larger model, stronger performance. In European Conference on Computer Vision, p. 300–318. Cited by: §IV-A, §IV-D, TABLE I. [20] S. Liu et al. (2019) Deep learning in medical ultrasound analysis: a review. Engineering 5 (2), p. 261–275. Cited by: §I. [21] Y. Ma et al. (2024) A survey on vision-language-action models for embodied ai. arXiv preprint arXiv:2405.14093. Cited by: §I, §I-B. [22] M. Muller et al. (2018) Trackingnet: a large-scale dataset and benchmark for object tracking in the wild. In Proceedings of the European conference on computer vision (ECCV), p. 300–317. Cited by: §IV-B. [23] C. Mwikirize, J. L. Nosher, and I. Hacihaliloglu (2019) Learning needle tip localization from digital subtraction in 2d ultrasound. International journal of computer assisted radiology and surgery 14, p. 1017–1026. Cited by: §I-A. [24] C. K. Ng et al. (2025) EndoVLA: dual-phase vision-language-action model for autonomous tracking in endoscopy. arXiv preprint arXiv:2505.15206. Cited by: §I, §I-B, §I-D, §I-E, §IV-C. [25] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4195–4205. Cited by: §I-B. [26] E. Perez et al. (2018) Film: visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: §I-D. [27] R. R. Richardson and R. R. Richardson (2017) Imaging modalities: advantages and disadvantages. Atlas of Acquired Cardiovascular Disease Imaging in Children, p. 1–4. Cited by: §I. [28] Y. Sato, K. Matsueda, and Y. Inaba (2024) Basic techniques and technical tips for ultrasound-guided needle puncture. Interventional Radiology 9 (3), p. 80–85. Cited by: §I-F. [29] A. Vaswani (2017) Attention is all you need. Advances in Neural Information Processing Systems. Cited by: §I-A, §I-D, §I-D. [30] S. Wang et al. (2025) Trackvla: embodied visual tracking in the wild. arXiv preprint arXiv:2505.23189. Cited by: §I-B. [31] A. M. Wijata, B. Pyciński, and J. Nalepa (2024) A needle in a (medical) haystack: detecting a biopsy needle in ultrasound images using vision transformers. In 2024 IEEE International Conference on Image Processing (ICIP), p. 3017–3023. Cited by: §I-A. [32] Y. Wu, J. Lim, and M. Yang (2013) Online object tracking: a benchmark. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2411–2418. Cited by: §IV-B. [33] W. Yan et al. (2023) Learning-based needle tip tracking in 2d ultrasound by fusing visual tracking and motion prediction. Medical Image Analysis 88, p. 102847. Cited by: §I-A, §I-F, §I-G. [34] Y. Zhang et al. (2024) A unified framework for microscopy defocus deblur with multi-pyramid transformer and contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11125–11136. Cited by: §I-A. [35] Y. Zhang et al. (2025) MambaXCTrack: mamba-based tracker with ssm cross-correlation and motion prompt for ultrasound needle tracking. IEEE Robotics and Automation Letters 10 (5), p. 5130–5137. External Links: Document Cited by: §I-A. [36] Y. Zhang et al. (2025) Motion-guided dual-camera tracker for endoscope tracking and motion analysis in a mechanical gastric simulator. In 2025 IEEE International Conference on Robotics and Automation (ICRA), p. 01–07. Cited by: §I-A. [37] Y. Zhang et al. (2025) Mrtrack: register mamba for needle tracking with rapid reciprocating motion during ultrasound-guided aspiration biopsy. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 407–417. Cited by: §I-A.