Paper deep dive
Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Robotics
Omar Mohamed, Edoardo Fazzari, Ayah Al-Naji, Hamdan Alhadhrami, Khalfan Hableel, Saif Alkindi, Ivan Laptev, Cesare Stefanini
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 6:43:19 AM
Summary
The paper introduces TASOT (Text-Augmented Action Segmentation Optimal Transport), an annotation-free framework for surgical temporal segmentation. It extends the Action Segmentation Optimal Transport (ASOT) method by integrating visual features from DINOv3 and textual features from CLIP-encoded captions generated by a vision-language model (Gemini 2.0 Flash). These modalities are fused within a unified unbalanced Gromov-Wasserstein optimal transport objective to achieve temporally coherent segmentation without manual annotations or surgical-domain pretraining. TASOT demonstrates significant performance improvements over zero-shot baselines on datasets including Cholec80, AutoLaparo, and MultiBypass140.
Entities (10)
Relation Signals (9)
TASOT → evaluatedon → Cholec80
confidence 95% · We evaluate TASOT on three public surgical datasets... Cholec80
TASOT → evaluatedon → MultiBypass140
confidence 95% · We evaluate TASOT on... MultiBypass140
TASOT → evaluatedon → AutoLaparo
confidence 95% · We evaluate TASOT on... AutoLaparo
TASOT → extends → ASOT
confidence 95% · TASOT extends the Action Segmentation Optimal Transport (ASOT) formulation by incorporating temporally aligned textual descriptions
TASOT → uses → CLIP
confidence 92% · temporal captions... are encoded via CLIP
TASOT → uses → DINOv3
confidence 92% · Visual representations are extracted using DINOv3
TASOT → uses → Gemini 2.0 Flash
confidence 90% · Each window clip is uploaded to Gemini 2.0 Flash... to generate a structured temporal description
TASOT → usesoptimization → Gromov-Wasserstein
confidence 90% · fusing visual and semantic cues within a unified unbalanced Gromov-Wasserstein optimal transport objective
TASOT → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Automated recognition of surgical phases and steps is a fundamental capability for intraoperative decision support, workflow automation, and skill assessment in robotic-assisted surgery. Existing approaches either depend on large-scale annotated surgical datasets or require expensive domain-specific pretraining on thousands of labeled videos, limiting their practical deployability across diverse robotic platforms and clinical environments. In this work, we propose TASOT (Text-Augmented Action Segmentation Optimal Transport), an annotation-free framework for surgical temporal segmentation that requires no task-specific annotations or surgical-domain pretraining. TASOT extends the Action Segmentation Optimal Transport (ASOT) formulation by incorporating temporally aligned textual descriptions generated directly from the input video, fusing visual and semantic cues within a unified unbalanced Gromov-Wasserstein optimal transport objective. Visual representations are extracted using DINOv3, while temporal captions produced by a vision-language model are encoded via CLIP and temporally aligned to individual frames, providing complementary semantic structure to the transport cost. We evaluate TASOT on three public surgical datasets and four benchmark settings spanning laparoscopic and robotic procedures, showing substantial improvements over the strongest zero-shot baselines: +18.9 F1 on Cholec80, +33.7 on AutoLaparo, +23.7 on StrasByPass70, and +4.5 on BernByPass70. These results suggest that fine-grained surgical workflow understanding in robotic settings can be achieved without manual training annotations or surgical-specific pretraining pipelines, offering a promising alternative for real-world robotic surgical systems.
Tags
Links
- Source: https://arxiv.org/abs/2602.24138v2
- Canonical: https://arxiv.org/abs/2602.24138v2
Trouble viewing inline? Open PDF directly →
Full Text
38,983 characters extracted from source content.
Expand or collapse full text
Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Robotics Omar Mohamed1, Edoardo Fazzari1∗, Ayah Al-Naji1, Hamdan Alhadhrami1, Khalfan Hableel1, Saif Alkindi1, Ivan Laptev1, Cesare Stefanini1 1Authos are with the Dept. of Robotics, Mohamed bin Zayed University of AI, Abu Dhabi, UAE. omar.mohamed, edoardo.fazzari, ayah.al-naji, hamdan.alhadhrami, khalfan.hableel, saif.alkindi, ivan.laptev, cesare.stefanini@mbzuai.ac.ae Abstract Automated recognition of surgical phases and steps is a fundamental capability for intraoperative decision support, workflow automation, and skill assessment in robotic-assisted surgery. Existing approaches either depend on large-scale annotated surgical datasets or require expensive domain-specific pretraining on thousands of labeled videos, limiting their practical deployability across diverse robotic platforms and clinical environments. In this work, we propose TASOT (Text-Augmented Action Segmentation Optimal Transport), an annotation-free framework for surgical temporal segmentation that requires no task-specific annotations or surgical-domain pretraining. TASOT extends the Action Segmentation Optimal Transport (ASOT) formulation by incorporating temporally aligned textual descriptions generated directly from the input video, fusing visual and semantic cues within a unified unbalanced Gromov-Wasserstein optimal transport objective. Visual representations are extracted using DINOv3, while temporal captions produced by a vision-language model are encoded via CLIP and temporally aligned to individual frames, providing complementary semantic structure to the transport cost. We evaluate TASOT on three public surgical datasets and four benchmark settings spanning laparoscopic and robotic procedures, showing substantial improvements over the strongest zero-shot baselines: +18.9 F1 on Cholec80, +33.7 on AutoLaparo, +23.7 on StrasByPass70, and +4.5 on BernByPass70. These results suggest that fine-grained surgical workflow understanding in robotic settings can be achieved without manual training annotations or surgical-specific pretraining pipelines, offering a promising alternative for real-world robotic surgical systems. I Introduction Understanding surgical workflow at the temporal level is a fundamental capability for intelligent robotic-assisted surgery. Automated recognition of surgical phases and steps enables intraoperative guidance, real-time decision support, automated skill assessment, and the foundation for higher-level surgical autonomy [1, 2]. As robotic surgical platforms generate increasingly large volumes of procedural video, the ability to parse and understand these workflows without manual intervention has become a critical bottleneck in the development of clinically deployable surgical AI. Surgical video, however, presents unique perceptual challenges for automated systems. The visual scene is highly complex and dynamic, with frequent occlusions, camera motion, instrument motion, tissue deformation, and ambiguous anatomical structures [3]. These factors make it difficult to distinguish actions based solely on visual appearance [4, 5]. Moreover, approaches that rely on fully supervised segmentation typically require dense annotations for surgical videos, which are extremely expensive, as each frame must often be labeled by a medical expert. Collecting large annotated surgical datasets is therefore laborious and time-consuming [6]. To reduce the annotation burden, most recent efforts have explored “zero-shot” or transfer-based approaches in the surgical domain [7]. Although these approaches achieve strong performance without requiring dense annotations, they typically still rely on large pretrained networks and employ general, complex architectures that are not specifically designed to enforce temporal segment structure [8]. In short, existing surgical video segmentation methods either demand heavy annotation effort or depend on complex pretrained models, and they do not explicitly exploit the structure of OT-based temporal segmentation. Given these limitations, we ask the following question: Is large-scale surgical pretraining truly necessary for effective temporal segmentation, or can an annotation-free approach achieve competitive performance? To address this question, we propose TASOT (Text-Augmented Action Segmentation Optimal Transport), an annotation-free method for surgical phase and step recognition that extends Action Segmentation Optimal Transport (ASOT) [9] by incorporating textual information generated directly from videos. Crucially, TASOT requires no surgical-domain annotations, no manual task-specific labels, and no narrated footage. Instead, we use off-the-shelf encoders for video frames and associated textual data, and let an optimal transport (OT) formulation fuse them. In other words, TASOT aligns video and text sequences within a unified framework, where a temporally consistent unbalanced Gromov–Wasserstein OT [10] formulation ensures coherent segment boundaries across both modalities. We evaluate TASOT on three publicly available datasets, i.e., Cholec80 [3], AutoLaparo [11], and MultiBypass140 [12] from both centers. Across all datasets, TASOT consistently outperforms existing zero-shot models in terms of F1 score, demonstrating that surgical temporal understanding can be achieved without large-scale surgical pretraining. In summary, our contributions are: ∙ We propose the first multimodal OT-based framework for annotation-free temporal segmentation in the surgical domain, introducing a formulation that integrates visual and textual cues within a unified optimal transport objective, regularized by temporally consistent Gromov–Wasserstein constraints. ∙ We demonstrate strong performance in annotation-free surgical temporal segmentation, consistently outperforming existing zero-shot methods on F1 scores across multiple benchmark datasets. I Related Work Surgical workflow recognition has progressed from fully supervised CNN-based architectures [3] through recurrent and temporal convolutional networks [13, 14] to transformer-based models capturing long-range temporal dependencies [15]. More recently, self-supervised and adaptive transformer-based approaches have also been explored for robotic surgical step recognition, improving robustness across surgeons, centers, and imaging conditions [16]. While these approaches achieve strong performance, they require dense frame-level annotations from clinical experts for downstream workflow recognition—a prohibitive bottleneck for deployment across diverse robotic platforms and clinical environments [6, 12]. To reduce annotation burden, video-language pretraining methods have emerged for zero-shot surgical recognition. Building on contrastive frameworks such as CLIP [8] and MIL-NCE [5], domain-specific adaptations including SurgVLP [7], HecVL [17], and PeskaVLP [18] align surgical video with narrated transcripts via contrastive and hierarchical objectives, achieving competitive zero-shot performance. However, since paired surgical video-text corpora are scarce, these methods often rely on multi-stage text acquisition pipelines: SurgVLP and HecVL use Automatic Speech Recognition (ASR) systems to transcribe narrated surgical lecture videos, while PeskaVLP further processes these transcripts using GPT-4 [19] to produce structured procedural summaries. These pipelines are therefore most naturally applicable to procedures where large-scale narrated surgical footage is available—a condition less common in standard robotic operating room settings where surgeries are performed without narration. Furthermore, the resulting models depend on pretrained multimodal backbones learned from curated video-text pairs. In contrast, TASOT generates semantic descriptions directly from silent input video at inference time using a vision-language model, requiring neither narrated surgical footage, ASR transcription, nor offline corpus construction. In parallel, unsupervised temporal action segmentation aims to partition untrimmed videos into coherent segments without frame-level annotations [20]. Early approaches learn frame embeddings and cluster them under temporal regularization [21, 22], later refined through reconstruction [23] or discriminative embedding objectives [24]. More recently, Optimal Transport (OT) has provided a principled framework for joint representation learning and pseudo-label assignment. In particular, ASOT [9] formulates segmentation as an unbalanced Gromov–Wasserstein OT problem without predefined action ordering, achieving strong general-domain performance. HVQ [2] further improves short-action detection via hierarchical vector quantization, but its codebook-based formulation does not explicitly couple representation learning with clustering as in OT-based alignment. Nevertheless, existing OT-based approaches operate purely on visual features. To date, multimodal textual cues have not yet been integrated into an OT framework for surgical temporal segmentation. I Method We introduce TASOT (Text-Augmented Action Segmentation Optimal Transport), an annotation-free multimodal framework for surgical temporal segmentation that extends ASOT [9] by incorporating temporally aligned textual descriptions generated directly from the input video. As illustrated in Fig. 1, TASOT operates in three stages: (i) a captioning pipeline that processes raw surgical video through a vision-language model to produce temporally grounded natural language descriptions; (i) a feature extraction stage that encodes video frames using DINOv3 [25] and caption segments using CLIP [8], producing temporally aligned visual and textual feature streams; and (i) a multimodal optimal transport stage that integrates both streams within a unified unbalanced Gromov-Wasserstein objective to produce temporally coherent segment assignments. Crucially, TASOT requires no surgical-domain annotations, no manual task-specific labels, and no narrated footage. Instead, semantic information is obtained by generating temporally grounded descriptions directly from the input video at inference time using a pretrained vision-language model. I-A Captioning Pipeline Given a surgical video v of duration T seconds, we first divide it into non-overlapping temporal windows of fixed length W (default W=300W=300 s): wm=v[sm,em),sm=mW,em=min((m+1)W,T),w_m=v[s_m,e_m), s_m=mW, e_m= ((m+1)W,\,T ), (1) where sms_m and eme_m denote the start and end times of the m-th window. Each window clip is uploaded to Gemini 2.0 Flash [26], which processes the raw video frames directly and generates a structured temporal description of the surgical activity within that clip. Prompt design. Gemini is prompted with the following template, which was designed to produce clinically grounded, reproducible descriptions without requiring procedure-specific instructions beyond the procedure name: You are watching a PROCEDURE_NAME surgery video. The clip starts at time 0 seconds and lasts CLIP_LEN seconds. First, understand what is happening in the clip as a surgeon would. Then divide the clip into consecutive time segments and describe what happens in each segment. Return ONLY valid JSON with the following fields: start_sec, end_sec, description. Guidelines: use integer seconds within [0, CLIP_LEN]; write clear descriptions of surgical actions mentioning tools and anatomy when visible; produce 10--20 segments per 5-minute clip with preferred durations of 10--25 seconds; avoid vague phrases such as ‘view change’; use generic tool names only. The model returns a JSON array of consecutive temporal segments mS_m: m=(τj(m),τj+1(m),textj(m))j=1Jm,0=τ1(m)<⋯<τJm+1(m)=|wm| gatheredS_m= \ ( _j^(m),\, _j+1^(m),\,text_j^(m) ) \_j=1^J_m,\\ 0= _1^(m)<·s< _J_m+1^(m)=|w_m| gathered (2) where each entry specifies a start time, end time, and natural language description of the surgical action occurring in that interval. Window-local timestamps are converted to global video time as: tj,start(m)=sm+τj(m),tj,end(m)=sm+τj+1(m).t_j,start^(m)=s_m+ _j^(m), t_j,end^(m)=s_m+ _j+1^(m). (3) Window-level segments are then merged and temporally ordered to form the complete video-level caption set. Robustness and failure handling. The pipeline includes dedicated handling for two failure modes. First, if Gemini returns output wrapped in Markdown code fences (e.g., ‘json), these are stripped before JSON parsing. Second, if JSON parsing fails, for instance due to truncated output or timestamp violations, the raw response is logged to disk and the window is skipped, allowing the pipeline to resume from the last successful window on re-execution. Remote files are deleted from the Gemini API after each window to avoid storage accumulation. In practice, parse failures occurred in fewer than 11% of windows across our evaluation datasets, and re-running the pipeline on failed windows resolved all cases. Figure 1: Overview of TASOT. Surgical videos are divided into temporal windows and processed by a vision-language model to generate structured temporal captions. Visual features extracted with DINOv3 and temporally aligned textual features encoded with CLIP are integrated within TASOT through a weighted multimodal transport cost, enabling annotation-free surgical temporal segmentation. I-B TASOT Framework Multimodal Feature Extraction. Given a surgical video sampled at 1 fps, consistent with standard surgical workflow annotation protocols [3], each frame ItI_t is encoded by a frozen DINOv3 [25] backbone to produce visual frame embeddings xtimg∈ℝDx^img_t ^D. DINOv3 is selected for its strong spatial representations learned through self-supervised pretraining, which transfer effectively to the surgical domain without requiring any domain-specific fine-tuning. In parallel, the temporally grounded captions produced by the pipeline previously described are encoded using CLIP [8], producing segment-level text embeddings ej∈ℝDe_j ^D. Each frame at time t is assigned the embedding of the caption segment whose temporal interval [tj,start(m),tj,end(m))[t^(m)_j,start,\,t^(m)_j,end) contains t, yielding a temporally aligned textual feature stream xttext∈ℝDx^text_t ^D. This alignment is a key design choice: rather than treating text as a global video descriptor, TASOT assigns fine-grained semantic context to individual frames, enabling the transport objective to exploit local semantic transitions as cues for segment boundaries. Modality-Specific Prototype Learning. TASOT learns K normalized action prototypes =akk=1K⊂ℝDA=\a_k\_k=1^K ^D that serve as soft cluster centroids within the optimal transport objective. Compared with ASOT [9], which operates on a single visual feature stream through a shared projection head, TASOT introduces modality-specific projection heads for the visual and textual streams independently. Concretely, visual features xtimgx^img_t and textual features xttextx^text_t are projected into a shared latent space through separate learnable linear projections followed by ℓ2 _2 normalization, yielding ztimgz^img_t and zttextz^text_t respectively: ztimg=Wimgxtimg‖Wimgxtimg‖2,zttext=Wtextxttext‖Wtextxttext‖2.z^img_t= W_img\,x^img_t\|W_img\,x^img_t\|_2, z^text_t= W_text\,x^text_t\|W_text\,x^text_t\|_2. (4) This design allows each modality to learn its own alignment to the shared prototype space, preventing the dominant visual modality from suppressing complementary semantic structure in the text stream, a failure mode observed when using feature concatenation, as confirmed in our ablation (Table I). Multimodal Transport Cost. Using the projected embeddings and shared prototypes, TASOT defines modality-specific cosine-distance cost matrices: Ci,kimg=1−⟨ziimg,ak⟩,Ci,ktext=1−⟨zitext,ak⟩.C^img_i,k=1- z^img_i,\,a_k , C^text_i,k=1- z^text_i,\,a_k . (5) These are fused at the cost level into a single multimodal transport cost: Ci,k=βCi,kimg+(1−β)Ci,ktext,C_i,k=β\,C^img_i,k+(1-β)\,C^text_i,k, (6) where β∈[0,1]β∈[0,1] controls the visual-text trade-off. Cost-level fusion is a deliberate design choice with an important geometric justification. TASOT builds on the Fused Unbalanced Gromov-Wasserstein (FUGW) optimal transport formulation [27], which combines a Kantorovich matching cost, encoding visual similarity between frames and prototypes, with a Gromov-Wasserstein structural prior that enforces temporal consistency over the transport plan ⋆T , and an unbalanced penalty that relaxes the requirement for all action classes to be equally represented within a video. The FUGW solver operates directly on the cost matrix C to compute ⋆T . By fusing modalities at the cost level, rather than in the feature space, TASOT ensures that both visual appearance and semantic textual cues jointly shape the matching cost fed into the FUGW objective, allowing the Gromov-Wasserstein temporal consistency prior to simultaneously regularize the alignment of both modalities. Feature concatenation, by contrast, conflates the two modalities before the transport geometry is defined, losing this structural advantage. Temporally Regularized Optimal Transport. A temporal prior R is incorporated into the cost matrix to encourage monotonic alignment between frame indices and action prototypes: C^i,k=Ci,k+ρRi,k,Ri,k=|iN−kK|, C_i,k=C_i,k+ρ\,R_i,k, R_i,k= | iN- kK |, (7) where ρ≥0ρ≥ 0 controls the strength of the temporal regularization. The resulting multimodal cost C is passed to the FUGW OT solver, which minimizes the fused, unbalanced Gromov-Wasserstein objective: ⋆=argmin∈ = T _p αℱGW(v,a,) α\,F_GW(C^v,C^a,T) (8) +(1−α)⟨^,⟩ +(1-α)\, C,T +λDKL(⊤N∥) +λ\,D_KL(T 1_N\|q) where vC^v and aC^a encode temporal structure over frames and prototypes respectively, α balances the GW temporal consistency prior against the multimodal matching cost, and λ controls the degree of unbalanced assignment, allowing a subset of action classes to be absent from a given video, which is essential for the long-tailed class distributions characteristic of surgical workflows. The resulting soft transport plan ⋆∈ℝN×KT ^N× K assigns each frame i to action prototype k with probability Tik⋆T _ik, and serves as pseudo-labels for self-training. Frame-level cluster probabilities are computed from the fused latent embeddings and optimized via cross-entropy loss against the OT assignments, jointly updating both the projection heads and the action prototypes A. IV Experimental Results and Discussion IV-A Experimental Setup We evaluate TASOT on three public surgical datasets: Cholec80 [3] (80 videos, 7 phases), AutoLaparo [11] (21 videos, 7 phases), and MultiBypass140 [12] (140 videos from Strasbourg and Bern, annotated with 12 phases and 46 steps). For MultiBypass140, results are reported separately for each center following the official zero-shot evaluation protocol. Predicted clusters are aligned with ground-truth labels using Hungarian matching [9]. We report segmental F1 score [20] as the primary metric, consistent with prior work. In the primary experiments, the number of clusters K is set equal to the total number of annotated classes per dataset, consistent with standard practice in OT-based unsupervised segmentation [9]. IV-B State-of-the-Art Comparisons TABLE I: Phase and step recognition results on Cholec80 [3], AutoLaparo [11], and MultiBypass140 [12]. TASOT is compared with zero-shot surgical video-language models and the annotation-free ASOT baseline. Best results are shown in bold. Step-level F1 is not reported for zero-shot methods because their pretraining protocols do not use step-level supervision. Method Cholec80 AutoLaparo BernByPass70 StrasByPass70 Phase F1 Phase F1 Phase F1 Step F1 Phase F1 Step F1 !12 Zero-shot Methods MIL-NCE [5] 7.3 7.9 2.1 — 3.1 — CLIP-SVL [8] 19.6 16.2 7.1 — 8.6 — SurgVLP [7] 24.4 16.6 7.2 — 6.9 — HecVL [17] 26.3 18.9 13.6 — 18.3 — PeskaVLP [18] 34.2 23.6 22.6 — 28.6 — !12 Annotation-free Methods ASOT [9] 44.94 42.63 26.98 13.45 56.11 25.37 TASOT (Ours) 53.11 57.30 27.1 23.0 52.3 30.7 Table IV-B compares TASOT with recent zero-shot surgical video-language models and the annotation-free ASOT baseline. TASOT outperforms the strongest zero-shot baseline, PeskaVLP, across all phase recognition benchmarks, with gains of +18.9 F1 on Cholec80, +33.7 on AutoLaparo, +4.5 on BernByPass70, and +23.7 on StrasByPass70, demonstrating robustness across diverse surgical environments and centers. Relative to ASOT, TASOT improves phase recognition on Cholec80, AutoLaparo, and BernByPass70, and delivers clear gains in step recognition on MultiBypass140. In particular, TASOT improves step F1 from 13.45 to 23.0 on BernByPass70 and from 25.37 to 30.7 on StrasByPass70. Although ASOT achieves slightly higher phase F1 on StrasByPass70, the overall results show that TASOT’s multimodal formulation, which integrates textual cues into the OT framework, is particularly beneficial for more challenging settings and fine-grained step segmentation. IV-C Ablation Study TABLE I: Component ablation of TASOT on MultiBypass140. Video-only uses DINOv3 features within ASOT, Text-only uses aligned caption features only, and Feature concatenation replaces cost-level fusion with feature-space fusion. TASOT (multimodal cost) denotes the full model. Best results are shown in bold. Model BernByPass70 StrasByPass70 Phase F1 Step F1 Phase F1 Step F1 Video-only (DINOv3 + ASOT) 27.9 20.3 51.3 27.9 Text-only 18.2 7.98 26.3 9.39 Feature concatenation 27.4 14.3 50.4 21.2 !12 TASOT (multimodal cost) 27.1 23.0 52.3 30.7 TABLE I: Sensitivity to the multimodal weight β on MultiBypass140. β=1.0β=1.0 is visual-only, β=0.0β=0.0 is text-only, and β=0.8β=0.8 is used as the default setting in the remaining experiments. Best results are shown in bold. β BernByPass70 StrasByPass70 Phase F1 Step F1 Phase F1 Step F1 1.0 (visual-only) 27.9 20.3 51.3 27.9 !12 0.8 27.1 23.0 52.3 30.7 0.5 25.6 12.93 42.0 16.17 0.0 (text-only) 18.2 7.98 26.3 9.39 We conduct ablations on MultiBypass140 to: (i) isolate the contribution of each component of TASOT, (i) analyze sensitivity to the multimodal weight β, (i) evaluate the choice of text encoder, and (iv) analyze the sensitivity of TASOT to the number of clusters K. Component isolation. Table I, together with the ASOT baseline in Table IV-B, provides a structured decomposition of TASOT’s performance gains. Specifically, ASOT in Table IV-B represents the original OT framework with its original visual backbone, while Video-only (DINOv3 + ASOT) in Table I isolates the effect of replacing that backbone with DINOv3 while still using no textual input. Comparing these two baselines shows that a substantial portion of the gain, particularly for step recognition, is attributable to the stronger visual encoder. Comparing Video-only (DINOv3 + ASOT) with the full TASOT model then isolates the contribution of the multimodal cost and text branch. Comparing the latter two rows shows that the multimodal cost provides clear gains for step recognition, improving Step F1 from 20.3 to 23.0 on BernByPass70 and from 27.9 to 30.7 on StrasByPass70. Feature concatenation consistently underperforms the multimodal cost formulation, particularly for step recognition, validating the benefit of cost-level fusion. Text-only performance is substantially lower across all settings, confirming that visual features remain the primary signal and that the text branch acts as a structured semantic complement rather than a standalone modality. Sensitivity to multimodal weighting. Table I analyzes the effect of the multimodal weight β in Eq. 6. The results show that the best performance is obtained with an intermediate but vision-dominant setting (β=0.8β=0.8), which consistently improves step recognition over the visual-only baseline while maintaining strong phase recognition. In contrast, equal weighting (β=0.5β=0.5) leads to a clear degradation, and the text-only setting (β=0.0β=0.0) performs substantially worse across all metrics. These results support the interpretation that textual cues act as a complementary signal that improves temporal segmentation when integrated with strong visual features, rather than as a standalone modality. Accordingly, we use β=0.8β=0.8 as the default setting in all remaining experiments. Text encoder choice. Table IV compares CLIP against Gemma [28] as the text encoder, following recent contrastive learning practice [29]. DINOv3 combined with CLIP consistently achieves the highest performance across all metrics. We attribute this to CLIP’s contrastive pretraining on image-text pairs, which produces embeddings that are inherently aligned with visual features, a property that directly benefits the joint OT cost formulation in Eq. 6. Combining Gemma and CLIP via concatenation does not improve over CLIP alone, further supporting the use of cost-level multimodal fusion over feature-space combination. TABLE IV: Text encoder analysis on MultiBypass140 using DINOv3 visual features within TASOT. We compare CLIP, Gemma, and their feature-space concatenation for the textual branch. ∥\,\|\, denotes feature concatenation. Best results are shown in bold. Visual Encoder Text Encoder BernByPass70 StrasByPass70 Phase F1 Step F1 Phase F1 Step F1 !12 DINOv3 CLIP 27.1 23.0 52.3 30.7 DINOv3 Gemma 26.7 22.4 49.2 29.9 DINOv3 Gemma ∥\,\|\, CLIP 26.6 22.1 50.3 30.6 TABLE V: Effect of video-specific cluster counts on MultiBypass140. The k-specific setting is an oracle analysis that uses the number of active classes per video and is not available in practical deployment. Best annotation-free results are shown in bold. Method BernByPass70 StrasByPass70 Phase F1 Step F1 Phase F1 Step F1 !12 Annotation-free Methods TASOT 27.1 23.0 52.3 30.7 TASOT (k-specific, oracle) 49.7 48.8 56.9 52.4 !12 Supervised Methods TeCNO [12] 59.2 47.5 80.7 58.1 MTMS-TCN [12] 62.4 47.9 79.8 57.3 Figure 2: Qualitative results. Ground-truth (GT) and TASOT predictions on MultiByPass140. Phase-level segmentations are temporally coherent with well-aligned boundaries. Step-level segmentation is more challenging due to finer temporal granularity and a fixed cluster count K that may exceed the number of active steps in a given video. Sensitivity to cluster count K. In our primary experiments, K is fixed to the total number of annotated classes per dataset. However, not all classes are necessarily present in every individual video, meaning the model may tend to predict more segments than actually occur. Table IV-C reports performance when K is adapted to the actual number of active classes per video (k-specific). This is an oracle analysis: it requires privileged knowledge of per-video class counts unavailable at inference time and does not represent a practical deployment scenario. It is reported solely to quantify the performance ceiling achievable with ideal cluster count estimation. Under this oracle setting, TASOT achieves substantial gains, with Step F1 increasing from 23.0 to 48.8 on BernByPass70 and from 30.7 to 52.4 on StrasByPass70, the latter nearly matching the supervised TeCNO baseline on steps. These results highlight adaptive cluster count estimation as a promising direction for future work, and confirm that the primary practical limitation of TASOT lies in K selection rather than in the multimodal OT formulation itself. Discussion and limitations. While TASOT shows strong gains over recent zero-shot surgical video-language baselines and the ASOT baseline, these comparisons should be interpreted with appropriate context. In contrast to zero-shot video-language models, TASOT leverages video-conditioned textual descriptions generated at inference time, providing an additional source of semantic information that is specific to each input video. We therefore view the comparison as one between annotation-free multimodal temporal segmentation strategies rather than a strictly like-for-like architectural comparison. In addition, the gains from the text branch are most pronounced for fine-grained step recognition, while visual-only OT baselines remain competitive for some phase-level settings, such as StrasByPass70. Finally, TASOT still depends on the quality of generated captions and on the choice of a fixed cluster count K, both of which remain important directions for future improvement. V Conclusion We presented TASOT, an annotation-free multimodal optimal transport framework for temporal segmentation in robotic surgical systems. By extending the ASOT formulation to incorporate temporally aligned textual descriptions generated directly from silent surgical video, TASOT introduces a principled cost-level fusion of visual and semantic cues within a unified Fused Unbalanced Gromov-Wasserstein objective—requiring no surgical-domain annotations, no manual task-specific labels, and no narrated footage. TASOT outperforms zero-shot surgical video-language baselines across four benchmark settings spanning laparoscopic and robotic procedures, demonstrating that fine-grained workflow understanding in diverse robotic surgical environments can be achieved without large-scale surgical-specific pretraining pipelines. Ablation studies confirm three key findings. First, the multimodal text branch provides complementary gains beyond the stronger DINOv3 visual encoder, particularly for fine-grained step recognition, validating the role of textual cues in the multimodal cost formulation. Second, cost-level fusion within the OT objective outperforms feature concatenation on the more challenging step recognition tasks, supporting the geometric argument that fusing modalities before the transport geometry is defined can lose structural information important for temporally consistent segmentation. Third, the primary practical limitation of TASOT lies in fixed cluster count K: our oracle analysis shows that adapting K to the actual number of active classes per video yields substantial gains, nearly matching supervised baselines on step recognition, identifying adaptive cluster count estimation as the most impactful direction for future work. Beyond surgical robotics, TASOT’s annotation-free multimodal OT formulation is directly applicable to any long, untrimmed procedural video domain where vision-language models can generate temporally grounded descriptions—including industrial assembly, skills training, and neurosurgical workflows. As vision-language models continue to improve in temporal grounding accuracy, the quality of the text branch is expected to improve correspondingly, further strengthening the practical case for annotation-free surgical AI in real robotic operating room deployments. References [1] E. Fazzari and C. Stefanini, “Deep reinforcement learning for surgical robotics with state and image information: A survey,” Research Square, 2026. [2] F. Spurio, E. Bahrami, G. Francesca, and J. Gall, “Hierarchical vector quantization for unsupervised action segmentation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, 2025, p. 6996–7005. [3] A. P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. de Mathelin, and N. Padoy, “Endonet: A deep architecture for recognition tasks on laparoscopic videos,” IEEE Transactions on Medical Imaging, vol. 36, no. 1, p. 86–97, 2017. [4] E. Fazzari, D. Romano, F. Falchi, and C. Stefanini, “Artemis: animal recognition through enhanced multimodal integration system,” International Journal of Machine Learning and Cybernetics, vol. 16, no. 9, p. 5877–5892, 2025. [5] A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman, “End-to-end learning of visual representations from uncurated instructional videos,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, p. 9879–9889. [6] N. Hashemi, M. B. S. Svendsen, F. Bjerrum, S. Rasmussen, M. G. Tolsgaard, and M. L. Friis, “Acquisition and usage of robotic surgical data for machine learning analysis,” Surgical Endoscopy, vol. 37, no. 8, p. 6588–6601, 2023. [7] K. Yuan, V. Srivastav, T. Yu, J. L. Lavanchy, J. Marescaux, P. Mascagni, N. Navab, and N. Padoy, “Learning multi-modal representations by watching hundreds of surgical video lectures,” Medical Image Analysis, p. 103644, 2025. [8] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PmLR, 2021, p. 8748–8763. [9] M. Xu and S. Gould, “Temporally consistent unbalanced optimal transport for unsupervised action segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, p. 14 618–14 627. [10] V. Titouan, R. Flamary, N. Courty, R. Tavenard, and L. Chapel, “Sliced gromov-wasserstein,” Advances in Neural Information Processing Systems, vol. 32, 2019. [11] Z. Wang, B. Lu, Y. Long, F. Zhong, T.-H. Cheung, Q. Dou, and Y. Liu, “Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2022, p. 486–496. [12] J. L. Lavanchy, S. Ramesh, D. Dall’Alba, C. Gonzalez, P. Fiorini, B. P. Müller-Stich, P. C. Nett, J. Marescaux, D. Mutter, and N. Padoy, “Challenges in multi-centric generalization: phase and step recognition in roux-en-y gastric bypass surgery,” International journal of computer assisted radiology and surgery, vol. 19, no. 11, p. 2249–2257, 2024. [13] T. Czempiel, M. Paschali, M. Keicher, W. Simson, H. Feussner, S. T. Kim, and N. Navab, “Tecno: Surgical phase recognition with multi-stage temporal convolutional networks,” in International conference on medical image computing and computer-assisted intervention. Springer, 2020, p. 343–352. [14] S. Ramesh, D. Dall’Alba, C. Gonzalez, T. Yu, P. Mascagni, D. Mutter, J. Marescaux, P. Fiorini, and N. Padoy, “Multi-task temporal convolutional networks for joint recognition of surgical phases and steps in gastric bypass procedures,” International journal of computer assisted radiology and surgery, vol. 16, no. 7, p. 1111–1119, 2021. [15] Y. Liu, M. Boels, L. C. Garcia-Peraza-Herrera, T. Vercauteren, P. Dasgupta, A. Granados, and S. Ourselin, “Lovit: Long video transformer for surgical phase recognition,” Medical Image Analysis, vol. 99, p. 103366, 2025. [16] Y. Ye, W. Wang, Y. Long, C.-F. Ng, Q. Zhou, D. Yang, Q. Dou, M. Xu, and Z. Pan, “Self-supervised adaptive transformer for surgical step recognition in robotic-assisted radical prostatectomy,” IEEE Robotics and Automation Letters, vol. 11, no. 6, p. 6568–6575, 2026. [17] K. Yuan, V. Srivastav, N. Navab, and N. Padoy, “Hecvl: Hierarchical video-language pretraining for zero-shot surgical phase recognition,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2024, p. 306–316. [18] K. Yuan, N. Navab, N. Padoy et al., “Procedure-aware surgical video-language pretraining with hierarchical knowledge augmentation,” Advances in Neural Information Processing Systems, vol. 37, p. 122 952–122 983, 2024. [19] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. [20] G. Ding, F. Sener, and A. Yao, “Temporal action segmentation: An analysis of modern techniques,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 2, p. 1011–1030, 2024. [21] S. Sarfraz, N. Murray, V. Sharma, A. Diba, L. Van Gool, and R. Stiefelhagen, “Temporally-weighted hierarchical clustering for unsupervised action segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, p. 11 225–11 234. [22] F. Sener and A. Yao, “Unsupervised learning and segmentation of complex activities from video,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, p. 8368–8376. [23] R. G. VidalMata, W. J. Scheirer, A. Kukleva, D. Cox, and H. Kuehne, “Joint visual-temporal embedding for unsupervised learning of actions in untrimmed sequences,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, p. 1238–1247. [24] S. Swetha, H. Kuehne, Y. S. Rawat, and M. Shah, “Unsupervised discriminative embedding for sub-action learning in complex activities,” in 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021, p. 2588–2592. [25] O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa et al., “Dinov3,” arXiv preprint arXiv:2508.10104, 2025. [26] Google DeepMind, “Gemini 2.0 flash model card,” Google, Tech. Rep., April 2025, available at https://modelcards.withgoogle.com/assets/documents/gemini-2-flash.pdf. [27] A. Thual, Q. H. Tran, T. Zemskova, N. Courty, R. Flamary, S. Dehaene, and B. Thirion, “Aligning individual brains with fused unbalanced gromov wasserstein,” Advances in neural information processing systems, vol. 35, p. 21 792–21 804, 2022. [28] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love et al., “Gemma: Open models based on gemini research and technology,” arXiv preprint arXiv:2403.08295, 2024. [29] D. Chen, M. Shukor, T. Moutakanni, W. Chung, J. Yu, T. Kasarla, A. Bolourchi, Y. LeCun, and P. Fung, “Vl-jepa: Joint embedding predictive architecture for vision-language,” arXiv preprint arXiv:2512.10942, 2025.