Paper deep dive

AffordTissue: Dense Affordance Prediction for Tool-Action Specific Tissue Interaction

Aiza Maksutova, Lalithkumar Seenivasan, Hao Ding, Jiru Xu, Chenhao Yu, Chenyan Jing, Yiqing Shen, Mathias Unberath

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 30

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 4/2/2026, 11:50:12 PM

Summary

AffordTissue is a multimodal framework designed for surgical automation that predicts tool-action specific tissue affordance regions as dense heatmaps. By utilizing a temporal vision encoder, language conditioning, and a DiT-style AdaLN decoder, the model provides explicit spatial reasoning for safe interaction during cholecystectomy, outperforming large-scale foundation models like Molmo-VLM and SAM3.

Entities (6)

AffordTissue · framework · 100%Cholecystectomy · surgical-procedure · 100%AdaLN Decoder · decoder-architecture · 95%Molmo-VLM · baseline-model · 95%SigLIP 2 · language-encoder · 95%Video Swin Transformer · vision-encoder · 95%

Relation Signals (4)

AffordTissue → predicts → Tissue Affordance

confidence 100% · we introduce AffordTissue, a multimodal framework for predicting tool-action specific tissue affordance regions

AffordTissue → outperforms → Molmo-VLM

confidence 95% · Experiments demonstrate substantial improvement over vision-language model baselines (20.6 px ASSD vs. 60.2 px for Molmo-VLM)

AffordTissue → utilizes → SigLIP 2

confidence 95% · The architecture includes (a) a language encoder that embeds the text prompt... (a) SigLIP 2

AffordTissue → utilizes → Video Swin Transformer

confidence 95% · The architecture includes... (b) a temporal video transformer that encodes spatiotemporal visual information

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Surgical action automation has progressed rapidly toward achieving surgeon-like dexterous control, driven primarily by advances in learning from demonstration and vision-language-action models. While these have demonstrated success in table-top experiments, translating them to clinical deployment remains challenging: current methods offer limited predictability on where instruments will interact on tissue surfaces and lack explicit conditioning inputs to enforce tool-action-specific safe interaction regions. Addressing this gap, we introduce AffordTissue, a multimodal framework for predicting tool-action specific tissue affordance regions as dense heatmaps during cholecystectomy. Our approach combines a temporal vision encoder capturing tool motion and tissue dynamics across multiple viewpoints, language conditioning enabling generalization across diverse instrument-action pairs, and a DiT-style decoder for dense affordance prediction. We establish the first tissue affordance benchmark by curating and annotating 15,638 video clips across 103 cholecystectomy procedures, covering six unique tool-action pairs involving four instruments (hook, grasper, scissors, clipper) and their associated tasks: dissection, grasping, clipping, and cutting. Experiments demonstrate substantial improvement over vision-language model baselines (20.6 px ASSD vs. 60.2 px for Molmo-VLM), showing that our task-specific architecture outperforms large-scale foundation models for dense surgical affordance prediction. By predicting tool-action specific tissue affordance regions, AffordTissue provides explicit spatial reasoning for safe surgical automation, potentially unlocking explicit policy guidance toward appropriate tissue regions and early safe stop when instruments deviate outside predicted safe zones.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

29,271 characters extracted from source content.

Expand or collapse full text

AffordTissue: Dense Affordance Prediction for Tool-Action Specific Tissue Interaction Aiza Maksutova 1, ⋆ , Lalithkumar Seenivasan 1,⋆ , Hao Ding 1,⋆ , Jiru Xu 1 , Chenhao Yu 1 , Chenyan Jing 1 , Yiqing Shen 1 , Mathias Unberath 1 Johns Hopkins University, Baltimore MD, USA unberath@jhu.edu Abstract. Surgical action automation has progressed rapidly toward achieving surgeon-like dexterous control, driven primarily by advances in learning from demonstration and vision-language-action models. While these have demonstrated success in table-top experiments, translating them to clinical deployment remains challenging: current methods offer limited predictability on where instruments will interact on tissue sur- faces and lack explicit conditioning inputs to enforce tool-action-specific safe interaction regions. Addressing this gap, we introduce AffordTis- sue, a multimodal framework for predicting tool-action specific tissue affordance regions as dense heatmaps during cholecystectomy. Our ap- proach combines a temporal vision encoder capturing tool motion and tissue dynamics across multiple viewpoints, language conditioning en- abling generalization across diverse instrument-action pairs, and a DiT- style decoder for dense affordance prediction. We establish the first tis- sue affordance benchmark by curating and annotating 15,638 video clips across 103 cholecystectomy procedures, covering six unique tool-action pairs involving four instruments (hook, grasper, scissors, clipper) and their associated tasks: dissection, grasping, clipping, and cutting. Exper- iments demonstrate substantial improvement over vision-language model baselines (20.6 px ASSD vs. 60.2 px for Molmo-VLM), showing that our task-specific architecture outperforms large-scale foundation models for dense surgical affordance prediction. By predicting tool-action specific tissue affordance regions, AffordTissue provides explicit spatial reason- ing for safe surgical automation, potentially unlocking explicit policy guidance toward appropriate tissue regions and early safe stop when in- struments deviate outside predicted safe zones. Keywords: Affordance Prediction· Tissue Affordance· Surgical Automation· Surgical Scene Understanding· Multimodal Models 1 Introduction Recent advances in learning from demonstration [6], imitation policies [31, 22], and vision-language-action models (VLAs) [12, 5] have significantly accelerated ⋆ Equal contribution. arXiv:2604.01371v1 [cs.CV] 1 Apr 2026 2Maksutova et al. surgical task automation, with pioneering works demonstrating surgeon-like dex- terous control in controlled table-top experiments [11, 21]. However, with safety taking precedence over efficiency in clinical practice, truly translating these ad- vances to clinical deployment hinges on ensuring safe automation [4, 9]. A funda- mental limitation of current learning-based approaches lies in their “black box” nature: while these models learn complex dexterous manipulation from expert demonstrations, they offer limited predictability regarding where and how the robot will interact with tissue, and whether the action will succeed [4, 9]. Further- more, they provide no explicit mechanism to condition or verify safe interaction. This poses significant safety concerns – there is limited controllability in en- forcing action-specific tissue interaction zones or intervening before potentially harmful contact occurs. Current surgical scene understanding approaches rely predominantly on se- mantic segmentation [16, 2], which can identify anatomical structures but lacks interaction-aware perception-reasoning about where within those structures a specific tool can safely engage for a given action. Affordance prediction has emerged as a powerful paradigm for interaction-aware perception in robotic ma- nipulation, enabling systems to identify and enforce where and how objects can be acted upon [27, 18]. However, these methods predominantly target rigid-object interaction, with limited exploration of deformable tissue dynamics, context- dependent surgical constraints, and safety-critical medical requirements. To this end, we introduce AffordTissue, a multimodal framework for pre- dicting tissue affordance regions as dense heatmaps conditioned on tool-action specifications. Similar to costmaps in robot navigation [15], our affordance repre- sentation encodes interaction suitability with maximum affordance at the region center that decreases toward boundaries. This potentially unlocks two comple- mentary modes for safe automation: (i) a conditioning signal, guiding learned policies toward appropriate tissue regions, and (i) a safety layer, triggering au- tomated early robot stopping when instrument trajectories deviate toward tissue outside the predicted affordance region – before potentially harmful contact oc- curs. Our key contributions are: (i) we introduce dense tissue affordance pre- diction as a novel task that provides explicit spatial reasoning about tool-action specific affordance regions on tissue surfaces; (i) we propose a multimodal ar- chitecture producing tool-action conditioned affordance heatmaps toward action guidance and safety verification; and (i) we curate and annotate 103 cholecys- tectomy videos, establishing the first tissue affordance benchmark. 2 AffordTissue Our proposed AffordTissue framework for predicting tool-action specific affor- dance reformulates image diffusion into a novel dense heatmap prediction task (Fig. 1). It takes in two inputs: (i) a language prompt specifying the tool–action pair, the surgical context, and the prediction objective, and (i) a video sequence consisting of the target (t 0 ) and past frames (t −256 → t −1 ). The architecture includes (a) a language encoder that embeds the text prompt, (b) a tempo- Title Suppressed Due to Excessive Length3 ral video transformer that encodes spatiotemporal visual information, and (c) a multimodal diffusion decoder that fuses the both to produce a task-aware dense tool-tissue interaction heatmap for the target frame. We begin by detailing the pre-trained backbones and architectural components integrated into our pipeline. Afterwards, we formalize the input space and the end-to-end workflow. 2.1 Preliminaries (a) SigLIP 2 [23]: SigLIP 2 is built on SigLIP [29], a CLIP-style model that employs a pairwise sigmoid loss instead of a conventional softmax loss used in CLIP, reducing the model’s dependency on batch normalization and improving training efficiency. SigLIP 2 further incorporates captioning-based pretraining and self-supervised losses, which significantly improve dense feature representa- tion for segmentation and localization. (b) Video Swin Transformer [13]: Video Swin Transformer is a hierarchi- cal backbone that extends the 2D Swin Transformer architecture to the spa- tiotemporal domain. The model processes an input video by partitioning it into 3D tokens, treating each 3D patch as a unique token. It employs a 3D Shifted Window mechanism to partition the tokens into non-overlapping spatiotemporal windows, where self-attention is computed locally. To ensure cross-window and cross-frame information exchange, the window partitions are shifted along the T, H, and W axes in successive layers, allowing the model to attend to temporal dependencies between several frames. (c) Adaptive Layer Normalization (AdaLN) Decoder: Introduced in DiT [20], the decoder is designed to inject conditional information directly into the feature normalization layers of a transformer decoder. Unlike standard cross- attention, which computes pairwise similarities between all tokens, AdaLN per- forms global conditioning by regressing the scale γ and shift β parameters of the Layer Normalization from the input condition. By shifting and scaling activa- tions based on the conditioning task, AdaLN enables efficient embedding fusion that emphasizes spatial regions relevant to the prediction objective, achieving computational efficiency while maintaining representational power. 2.2 Pipeline (a) Text input: Incorporating text in the pipeline is a crucial step for model scalability, as it allows it to differentiate between different tools and actions during the same surgery. We found that simple prompts work best for such tasks. Our chosen prompt consists of a surgical triplet: surgery type, tool type, action type. From VLM best practices, we also define the model’s final objective inside the prompt. (b) Visual input: We input a temporal window of N = 256 frames (stride of 8) to capture tool motion and tissue dynamics from multiple viewpoints, enabling implicit modeling of deformation patterns and surgical intent beyond single-frame approaches. 4Maksutova et al. Language input: Action and tool specific prompt: “You are given a surgical video frame from a cholecystectomy surgery. Predict the tissue affordance region for the given action and tool. Surgery: cholecystectomy. Action: clip. Tool: clipper.” Temporal Visual Encoder (SWIN) Language Encoder (SigLip) Projection Layer Decoder Vision input Past framesTarget frames Generated affordance heatmap SiLU Linear Video latents LayerNorm Modulate Text condition Multi - Head Self - Attention LayerNorm Modulate MLP Predicted frame Logits δ Pixel-level logits (Softmax) Distribution (a) (b) AdaLN Decoder Decoder Fig. 1. Architecture of AffordTissue: a) High-level overview: The pipeline processes tool-action prompts and temporal vision context via frozen SigLIP and Swin encoders, with a trained AdaLN-based decoder fusing these embeddings to output a heatmap. b) Decoder details: The decoder utilizes text-conditioned adaptive layer normalization (AdaLN) to refine temporal vision latents for a specific tool-action pair. This results in pixel-level logits that estimate the probability of each pixel belonging to the safe tissue affordance region. (c) Decoder adaptation: We propose a task-specific adaptation of the AdaLN decoder for dense heatmap prediction, detailed in Fig. 1(b). While DiT employs AdaLN to predict diffusion noise from noised latent representations conditioned on class labels, our architecture replaces both the input space and the objective. Our decoder directly processes temporal vision embeddings with language em- beddings as condition to predict per-pixel logits, estimating the probability that each pixel belongs to the tissue affordance region. This shift demonstrates that AdaLN-based conditioning, originally designed for diffusion modeling, can be effectively repurposed for spatially grounded dense feature prediction in vision- language tasks. (d) Workflow: Our end-to-end pipeline is shown in Fig. 1(a). Given a text prompt, the SigLIP 2 encoder produces a (B, 1152) embedding, which is af- terwards projected to a shared embedding space using MLP. Concurrently, the Video Swin Transformer processes input frames of shape (B,C,T,H,W) to ex- tract spatiotemporal features of shape (B,C,H,W). The AdaLN decoder then fuses these representations to predict per-pixel logits, producing a probabilistic affordance heatmap. 3 Experiment (i) Dataset: We curate a custom dataset of 15638 video clips from 103 videos: Youtube (21 videos), Cholec-80 (34 videos) [24], HeiChole (11 videos) [26], Com- Title Suppressed Due to Excessive Length5 Table 1. Distribution of video clips per tool–action pair across datasets. Each entry describes the total number of clips, with (train/val/test) split shown in parentheses. Dataset type Tool-action pairYoutubeCholec80HeiCholeCHECSurgVU (21 videos)(34 videos)(11 videos)(8 videos)(29 videos) Dissect - Hook1280 (1066/41/173) 4726 (3454/741/531) 1106 (875/97/134) 628 (476/0/152) 2685 (2120/270/295) Dissect - Grasper493 (462/0/31)43 (36/0/7)101 (92/5/4)0295 (234/61/1) Dissect - Scissors209 (209/0/0)12 (12/0/0)100 (97/3/0)0350 (350/0/0) Grasp - Grasper584 (538/3/43)931 (677/189/65)321 (286/9/26)136 (97/0/39)799 (689/66/44) Clip - Clipper125 (111/4/10)185 (143/26/16)87 (73/6/8)099 (76/14/9) Cut - Scissors69 (63/0/6)145 (106/20/19)79 (70/4/5)049 (46/3/0) prehensive Robotic Cholecystectomy Dataset (8 videos) [19], and SurgVU (29 videos) [30]. Table 1 details the distribution of video clips across datasets and train/validation/test splits. Each video clip is annotated for tool-action pairs (language) and tissue affordance. To define target affordance regions, each case was manually annotated with four keypoints outlining the safe tool–tissue inter- action zone. These keypoints form a polygon, from which a target heatmap is generated by centering a Gaussian distribution at the polygon’s centroid. Since the objective is to predict tissue affordance prior to instrument interaction, only frames occurring before the onset of the surgical action are used. The dataset split is performed at the case level, ensuring no data leakage. (i) Training and inference: In our model, the language and the vision encoder are frozen during training and used solely to provide embeddings, and the de- coder parameters are optimized. The model is trained on a single NVIDIA A100 GPU for 100 epochs, using a combination of binary cross-entropy loss and soft intersection-over-union (IoU) loss – a differentiable approximation of the stan- dard IoU metric. Optimization is performed using AdamW [14] with an initial learning rate of 1× 10 −4 and a cosine learning rate scheduler. During training, we select random target frames within the pre-action range. Along with each target frame, we input 256 previous frames with a stride of 8. This corresponds to approximately 10.6 seconds of historical context, which we found sufficient for capturing relevant temporal dynamics. (i) Evaluation Metrics: We divide our evaluation metrics into two groups: logits-conditioned and boundary-conditioned metrics. Logits-conditionedmetrics assess probabilistic overlap of target and predicted heatmaps and include "soft" Dice score [17] computed directly on logits, corresponding to heatmap intensities for each pixel. Boundary-conditionedmetrics evaluate spatial alignment, and include Precision at K (PCK) [3], Hausdorff Distance (HD) in pixels, and Average Symmetric Surface Distance (ASSD) in pixels [8]. For each case, we evaluate eight pre-action frames, chosen with a preference to earlier timestamps. This reduces the likelihood that the model grounds its prediction on the instrument’s position, which becomes closer to target tissue affordance area as the action approaches. 6Maksutova et al. 4 Results and Ablation Study (i) Baseline Comparison: We quantitatively evaluate our pipeline’s perfor- mance in detecting safe tool-tissue interaction against the following baseline models: Molmo-VLM [7], QWEN-8B VLM [28] and SAM3 [1]. Due to the nov- elty of the proposed task, there are no publicly available baselines specifically trained for tissue affordance prediction. To provide meaningful comparison, we evaluate against closely related tasks, such as pointing and segmentation, which approximate different aspects of spatial affordance estimation. For the pointing baseline, we fine-tune Molmo-VLM and QWEN-8B VLM. We augment the ar- chitecture with additional regression heads to predict the four polygon vertices that define the tissue affordance region. For the segmentation baseline, we fine- tune SAM3. None of the above baselines produce a heatmap output directly. To enable a fair comparison, we convert their predictions into heatmap rep- resentations by generating a Gaussian distribution centered at the centroid of the predicted polygon or segmentation mask. From Table 2 we observe that the model achieves an ASSD of 20.557 px and PCK@0.05 and PCK@0.1 of 0.517 and 0.667, respectively. This strong spatial alignment between predicted and ground-truth heatmaps observed quantitatively is supported by a qualitative analysis in Fig. 2. In contrast, the Hausforff Distance (HD) performance is rela- tively high. Qualitative analysis showed that it is primarily caused by occasional secondary, low-confidence heatmap activations that appear on the tool surface. We suggest that in future studies we add a carefully designed maximum-distance penalty to solve this problem. The DICE score is comparatively low across cases. As illustrated in Fig. 2, this is largely due to differences in intensity distributions rather than boundary misalignment. Given that the affordance regions are anno- tated as uniformly safe for interaction, the intensity discrepancy does not pose a problem as of now. The standard IoU metric for localization is not added to our evaluation experiments, as it is included in our training loss. Our model substan- tially outperforms all evaluated baselines. The strongest competitor, fine-tuned Molmo-VLM, shows a degradation of 192.76% in ASSD and 62.34% in HD relative to our approach. We also observe that SAM3 achieves a higher Dice score than our model. However, qualitative inspection of cases where SAM3 outperforms our approach reveals that it often predicts nearly the entire tissue region in the frame as safe for interaction, leading to higher DICE score but lower overall performance. These results indicate that even large-scale founda- Table 2. Comparison of our model against baselines: Molmo-VLM [7], SAM3 [1], and Qwen-VLM [28]. MetricsDICE↑ PCK@0.05↑ PCK@0.1↑ HD(px)↓ ASSD(px)↓ Ours0.1240.5170.667 79.763 20.557 SAM3 [1]0.180.1280.221150.32081.138 Molmo-VLM [7]0.0260.0950.320129.49460.184 Qwen-VLM (8B) [28] 0.0140.0310.022203.214111.271 Title Suppressed Due to Excessive Length7 Fig. 2. Qualitative comparison between ground truth and predicted heatmaps for three tool–action pairs: (i) hook - dissection, (i) clipper - clipping, and (i) scissors -clipping, across six representative timestamps. tion models do not match the performance of our task-specific architecture for dense heatmap prediction in laparoscopic data. (i) Ablation study: We perform extensive ablation study to validate choice of every architecture and input component in the suggested workflow. Table 3 demonstrates the impact of the SigLIP language encoder, the Swin vision en- coder, and the AdaLN decoder. Among given modules, the decoder and tempo- Table 3. Ablation study of our model with a different (i) language encoder, (i) vision encoder, and (i) decoder. MetricsDICE↑ PCK@0.05↑ PCK@0.1↑ HD(px)↓ ASSD(px)↓ Ours0.124 0.5170.667 79.763 20.557 Ours with Bert-based language encoder 0.1060.5020.65184.40323.891 Ours with vision encoder0.0860.4220.609113.61324.329 Ours with cross-attention decoder0.0850.4430.571108.03928.736 Table 4. Ablation study of our model without (i) L (SigLip language encoder) and (i) A (image augmentations). MetricsDICE↑ PCK@0.05↑ PCK@0.1↑ HD(px)↓ ASSD(px)↓ Ours0.1240.5170.667 79.76320.557 Ablation (w/o A) 0.0940.4910.625 127.10129.181 Ablation (w/o L) 0.0680.2050.348 170.48243.135 8Maksutova et al. Table 5. Ablation study of our model without (i) previous frames, (i) action, and (i) tool specification in input. MetricsDICE↑ PCK@0.05↑ PCK@0.1↑ HD(px)↓ ASSD(px)↓ Ours0.1240.5170.667 79.76320.557 Ablation (w/o action)0.1120.3500.542 104.73322.087 Ablation (w/o previous frames) 0.1030.4900.635 85.93224.973 Ablation (w/o tool)0.1010.3200.495 93.70227.302 ral vision encoder choices have the most crucial effect on model’s performance. Replacing AdaLN decoder with cross-attention decoder [25] not only reduces memory efficiency, but also leads to a 35.45% increase in HD and 39.78% in- crease in ASSD. This highlights the importance of adaptive feature modulation in our setting, suggesting that global conditional normalization is better suited for structured heatmap prediction than token-level cross-attention fusion. Simi- larly, changing the Swin Transformer with a 3D ResNet-18 [10] leads to a signif- icant performance degradation - (HD +42.43%, ASSD +18.34%), justifying our choice of vision backbone. Table 4 presents the importance of language encoder and dataset augmentations. The importance of language encoding is evident as the model’s performance drops drastically when it is removed: ASSD increases by 109.8% and Hausdorff Distance by 213.7%. Given that the training set contains different tool–action pairs, the model must be given a mechanism to differentiate between them, making the language encoder a crucial part of the pipeline. In contrast, image augmentations provide a modest but consistent im- provement, showing that while they contribute to robustness, they are secondary to model performance. Table 5 provides an ablation study for different input structures that we use in our pipeline: temporal vision context and action and tool specification in the prompt. Removing the tool specification leads to a substantial performance drop, with ASSD increasing by 32.81%. This result is expected, since many samples in our dataset include frames with multiple instruments, making it confusing for the model to understand for which tool the affordance region should be predicted. In comparison, removing temporal context is shown to be not as important to model’s performance, leading to a more moderate decline (ASSD +21.48%, HD +7.73%). While less critical than text conditioning, temporal information still improves spatial consistency by giving the model multi-view context that refines tissue affordance prediction. 5 Discussion and Conclusion We present AffordTissue, a multimodal framework combining a temporal vi- sion encoder, language conditioning, and a DiT-style decoder for predicting tool-action specific tissue affordance regions as dense heatmaps for six unique tool-action pairs critical to during the cholecystectomy procedure. By explicitly conditioning on tool-action specifications and temporal context, our approach achieves more precise affordance localization than large foundational models such Title Suppressed Due to Excessive Length9 as SAM3 and Molmo-VLM. The predicted affordance maps potentially unlock two complementary modes for safe surgical automation: a conditioning signal guiding learned policies toward appropriate tissue regions, and a safety layer enabling early automated stopping when instruments deviate outside predicted affordance regions. Future directions include extending affordance prediction to ground on the surgical phase for stage-specific reasoning, expanding to model diverse tool-tissue actions, and integration with VLAs for closed-loop surgical automation. References 1. AI, M., et al.: Sam 3: Segment anything with concepts. In: In- ternational Conference on Learning Representations (ICLR) (2026), https://openreview.net/forum?id=r35clVtGzw 2. Allan, M., Kondo, S., Bodenstedt, S., Leger, S., Kadkhodamohammadi, R., Luengo, I., Fuentes, F., Flouty, E., Mohammed, A., Pedersen, M., et al.: 2018 robotic scene segmentation challenge. arXiv preprint arXiv:2001.11190 (2020) 3. Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: New benchmark and state of the art analysis. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014) 4. Attanasio, A., Scaglioni, B., De Momi, E., Fiorini, P., Valdastri, P.: Autonomy in surgical robotics. Annual Review of Control, Robotics, and Autonomous Systems 4(1), 651–679 (2021) 5. Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al.: pi_0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164 (2024) 6. Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., Song, S.: Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research 44(10-11), 1684–1704 (2025) 7. Deitke, M., Schwenk, D., Salvador, J., VanderBilt, L., Aniol, K., et al.: Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146 (2024) 8. Gerig, G., Jomier, J., Chakos, M.: Valmet: A new software tool for assessing and visualizing segmentation accuracy. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer (2001) 9. Haidegger, T.: Autonomy for surgical robots: Concepts and paradigms. IEEE Transactions on Medical Robotics and Bionics 1(2), 65–76 (2019) 10. Hara, K., Kataoka, H., Satoh, Y.: Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 6546–6555 (2018) 11. Kim, J.W., Chen, J.T., Hansen, P., Shi, L.X., Goldenberg, A., Schmidgall, S., Scheikl, P.M., Deguet, A., White, B.M., Tsai, D.R., et al.: Srt-h: A hierarchical framework for autonomous surgery via language-conditioned imitation learning. Science robotics 10(104), eadt5254 (2025) 12. Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al.: Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246 (2024) 10Maksutova et al. 13. Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin trans- former. In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition. p. 3202–3211 (2022) 14. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019) 15. Lu, D.V., Hershberger, D., Smart, W.D.: Layered costmaps for context-sensitive navigation. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. p. 709–715. IEEE (2014) 16. Maier-Hein, L., Eisenmann, M., Sarikaya, D., März, K., Collins, T., Malpani, A., Fallert, J., Feussner, H., Giannarou, S., Mascagni, P., et al.: Surgical data science– from concepts toward clinical translation. Medical image analysis 76, 102306 (2022) 17. Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international confer- ence on 3D vision (3DV) (2016) 18. Nagarajan, T., Feichtenhofer, C., Grauman, K.: Grounded human-object interac- tion hotspots from video. In: Proceedings of the IEEE/CVF International Confer- ence on Computer Vision. p. 8688–8697 (2019) 19. Oh, K.H., Borgioli, L., Mangano, A., Valle, V., Di Pangrazio, M., Toti, F., Pozza, G., Ambrosini, L., Ducas, A., Žefran, M., et al.: Expanded comprehensive robotic cholecystectomy dataset (crcd). arXiv preprint arXiv:2412.12238 (2024) 20. Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 4195–4205 (2023) 21. Scheikl, P.M., Tagliabue, E., Gyenes, B., Wagner, M., Dall’Alba, D., Fiorini, P., Mathis-Ullrich, F.: Sim-to-real transfer for visual reinforcement learning of de- formable object manipulation for robot-assisted surgery. IEEE Robotics and Au- tomation Letters 8(2), 560–567 (2022) 22. Team, O.M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al.: Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213 (2024) 23. Tschannen, M., Gritsenko, A., Wang, X., Naeem, M.F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al.: Siglip 2: Multilingual vision-language encoders with improved semantic understanding, lo- calization, and dense features. arXiv preprint arXiv:2502.14786 (2025) 24. Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N.: Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging 36(1), 86–97 (2016) 25. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems (2017) 26. Wagner, M., Müller-Stich, B.P., Kisilenko, A., Tran, D., Heger, P., Mündermann, L., Lubotsky, D.M., Müller, B., Davitashvili, T., Capek, M., et al.: Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the heichole benchmark. Medical image analysis 86, 102770 (2023) 27. Xu, R., Zhang, J., Guo, M., Wen, Y., Yang, H., Lin, M., Huang, J., Li, Z., Zhang, K., Wang, L., et al.: A0: An affordance-aware hierarchical model for general robotic manipulation. In: Proceedings of the IEEE/CVF International Conference on Com- puter Vision. p. 13491–13501 (2025) 28. Yang, A., Yang, B., Zhang, B., Hui, B., et al.: Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 (2024) Title Suppressed Due to Excessive Length11 29. Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 11975–11986 (2023) 30. Zia, A., Berniker, M., Nespolo, R., Perreault, C., Wang, Z., Mueller, B., Schmidt, R., Bhattacharyya, K., Liu, X., Jarc, A.: Surgical visual understanding (surgvu) dataset. arXiv preprint arXiv:2501.09209 (2025) 31. Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al.: Rt-2: Vision-language-action models transfer web knowledge to robotic control. In: Conference on Robot Learning. p. 2165–2183. PMLR (2023)

Instant research discovery