Paper deep dive
A Surgical Foundation Model Reveals Task-Dependent Label Efficiency
Florian Philipp Stilz, Lorenzo Arboit, Vinkle Srivastav, CAMMA International Surgical Partners, Jacques Marescaux, Sergio Alfieri, Pietro Mascagni, Nassir Navab, Nicolas Padoy
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/29/2026, 11:58:32 AM
Summary
The paper introduces SURGE, a surgical foundation model trained on the SurgSpectrum-30M+ dataset (over 30 million frames) to address label efficiency challenges in surgical AI. The authors systematically evaluate label efficiency across 5 task categories and 15 benchmarks, revealing that while perception tasks (segmentation, phase/step recognition) saturate with minimal supervision, fine-grained reasoning tasks (instrument-anatomy interactions, safety maneuvers) require substantially larger annotation budgets. SURGE outperforms prior state-of-the-art models, including general-domain foundation models and task-specific baselines, demonstrating superior label efficiency and scaling behavior.
Entities (21)
Relation Signals (16)
SURGE → trainedon → SurgSpectrum-30M+
confidence 98% · Here, we introduce SURGE, a surgical foundation model trained on SurgSpectrum-30M+, the largest pretraining dataset comprising over 30 million frames
Action Triplet → evaluatedon → CholecT50
confidence 95% · instrument-anatomy interaction tasks (action triplet recognition) on the CholecT50 and ProstaTD benchmarks
Critical View of Safety → evaluatedon → Endoscapes
confidence 95% · clinical quality assessment (Critical View of Safety (CVS)), depicted in Fig. 4c, on the Endoscapes dataset
Action Triplet → evaluatedon → ProstaTD
confidence 95% · instrument-anatomy interaction tasks (action triplet recognition) on the CholecT50 and ProstaTD benchmarks
Semantic Segmentation → evaluatedon → GraSP
confidence 95% · semantic segmentation on prostatectomy (GraSP dataset)
Phase Recognition → evaluatedon → StrasBypass70
confidence 95% · phase recognition, SURGE-H reaches 74.04± 2.09% F1 on gastric bypass (StrasBypass70)
Phase Recognition → evaluatedon → Cholec80
confidence 95% · comparable behavior on cholecystectomy (Cholec80)
SURGE-B → hasparametercount → 85.67M
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Developing label-efficient models is a central challenge in surgical AI due to the high cost and scarcity of expert annotation. While self-supervised foundation models adapt well to new tasks with minimal data, how label efficiency varies across different surgical tasks remains largely unexplored. Here, we introduce SURGE, a surgical foundation model trained on SurgSpectrum-30M+, the largest pretraining dataset comprising over 30 million frames, with checkpoints released to enable further research. We systematically evaluate label efficiency across 5 task categories and 15 benchmarks. These range from temporal and spatial scene understanding to fine-grained reasoning tied to instrument-anatomy interactions and safety-critical maneuvers. SURGE outperforms prior state-of-the-art on all benchmarks, even surpassing task-specific models on complex reasoning tasks. Crucially, we reveal a task-dependent scaling behavior: while scene understanding tasks saturate with minimal supervision, fine-grained reasoning tasks continue improving with substantially larger annotation budgets, providing a blueprint for allocating expert effort in complex domains. Code: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2609.31821v1
- Canonical: https://arxiv.org/abs/2609.31821v1
Trouble viewing inline? Open PDF directly →
Full Text
68,599 characters extracted from source content.
Expand or collapse full text
A Surgical Foundation Model Reveals Task-Dependent Label Efficiency Florian Philipp Stilz a,b,c,∗ , Lorenzo Arboit a,b , Vinkle Srivastav a,b,d , CAMMA International Surgical Partners, Jacques Marescaux e , Sergio Alfieri f , Pietro Mascagni b,f , Nassir Navab c , Nicolas Padoy a,b a University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France b IHU Strasbourg, Strasbourg, France c Technical University of Munich, Computer Aided Medical Procedures, Munich, Germany d Indian Institute of Technology (IIT) Madras, Department of Data Science and AI, Wadhwani School of Data Science and AI (WSAI), Chennai, India e IRCAD, Research Institute Against Digestive Cancer, Strasbourg, France f Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy Developing label-efficient models is a central challenge in surgical AI due to the high cost and scarcity of expert annotation. While self-supervised foundation models adapt well to new tasks with minimal data, how label efficiency varies across different surgical tasks remains largely unexplored. Here, we introduce SURGE, a surgical foundation model trained on SurgSpectrum-30M+, the largest pretraining dataset comprising over 30 million frames, with checkpoints released to enable further research. We systematically evaluate label efficiency across 5 task categories and 15 benchmarks. These range from temporal and spatial scene understanding to fine- grained reasoning tied to instrument–anatomy interactions and safety-critical maneuvers. SURGE outperforms prior state-of-the-art on all benchmarks, even surpassing task-specific models on complex reasoning tasks. Crucially, we reveal a task-dependent scaling behavior: while scene understanding tasks saturate with minimal supervision, fine-grained reasoning tasks continue improving with substantially larger annotation budgets, providing a blueprint for allocating expert effort in complex domains. Code: https://github.com/CAMMA-public/SURGE. 1. Introduction Artificial intelligence is increasingly deployed to support expert decision-making in complex, high-stakes domains, ranging from healthcare and science to engineering and pub- lic safety [1]. In these settings, models are typically trained on annotated datasets that require substantial input from do- main experts. However, generating such annotations is both time-consuming and costly, as expertise is scarce and tasks often demand deep contextual understanding. As a result, the development of new AI systems is frequently constrained not only by model capacity, but increasingly by the availability of high-quality labeled data [1]. Improving the efficiency with which new tasks can be learned is therefore critical. This can be approached from two complementary directions: develop- ing models that achieve strong performance with fewer anno- tations and understanding how annotation requirements vary across tasks to allocate expert effort more effectively. Here, we address both challenges by developing a self-supervised foun- dation model for surgery and systematically characterizing how its performance scales with the amount of task-specific labeled data. We use this scaling behavior to quantify label efficiency across surgical perception and reasoning tasks, pro- viding a principled basis for understanding and reducing the annotation requirements of AI systems in surgery. Surgical procedures are complex, high-risk interventions that require continuous perception and timely reasoning to ∗ Corresponding author: florian.stilz@tum.de support safe and effective decision-making. Automated anal- ysis of endoscopic video acquired during minimally inva- sive surgery has the potential to improve intraoperative guid- ance, surgical training and postoperative assessment. Sur- gical AI [2, 3, 4, 5] is therefore a particularly high-stakes application, as errors and preventable complications during surgery contribute substantially to global morbidity and mor- tality [6, 7], highlighting the need for reliable intraoperative decision support. Importantly, surgical video analysis encom- passes multiple levels of understanding, ranging from tem- poral perceptual tasks, such as recognizing surgical phases and steps [8, 9, 10, 11, 12] and spatial tasks such as seg- menting instruments and anatomy [13, 14, 9, 11, 15, 16, 17], to higher-level reasoning about instrument–anatomy interac- tions [18, 19] and clinically grounded decision-making [13]. These diverse tasks vary substantially in complexity, raising the question of whether they also differ in the amount of su- pervision required to learn them effectively. Recent advances in self-supervised and foundation model pretraining [20, 21, 22, 23] have enabled strong general- purpose visual representations by learning from large amounts of unlabeled data. When applied to surgical video, these approaches often yield competitive performance, particularly when sufficient labeled data is available [24, 25, 26]. Domain- adapted pretraining [24, 25, 27, 26, 28, 29] and multimodal approaches [30, 31] have further improved data efficiency. However, existing work has primarily focused on developing stronger representations, while the extent to which different categories of surgical tasks exhibit distinct label efficiency re- arXiv:2609.31821v1 [cs.CV] 25 Sep 2026 2 Fig. 1: a: Label efficiency across surgical tasks, shown as normalized area under the curve (nAUC) in low label regimes versus relative peak performance. Results are averaged across datasets per task. We observe a consistent ordering of tasks by data efficiency: perception tasks (segmentation (Seg), Phase, Steps) achieve strong performance in low-label regimes, while fine-grained and reasoning tasks (Action Triplet, CVS) require substantially more supervision. Notably, certain tasks exhibit low-data regimes with limited learning signal, indicating threshold effects that are not overcome by pretraining. b: Full-data performance comparison of our models with general-domain and surgical foundation baselines across 15 benchmarks. c: Five tasks are illustrated with representative examples and grouped into perception and clinical reasoning categories. Icons indicating anatomical structures denote the surgical procedures associated with each task (cholecystectomy, hysterectomy, liver resection, gastric bypass, prostatectomy). The nAUC metric is illustrated schematically. 3 mains largely unexplored. In this work, we address this gap through a large-scale, systematic analysis of label efficiency in surgical AI. To this end, we introduce SURGE (Surgical Understanding through Representation and Generalization at Scale), a sur- gical foundation model trained via self-supervised learning on SurgSpectrum-30M+, a dataset of over 30 million sur- gical video frames. Using this unified representation, we evaluate performance across a spectrum of tasks, which we group into perceptual tasks (e.g., spatial and temporal under- standing) and clinically grounded reasoning tasks (e.g., re- lational instrument–anatomy interactions and safety-critical maneuver recognition), which require higher-level judgment built upon perceptual understanding. Beyond overall perfor- mance, we study how these tasks scale with increasing su- pervision. While large-scale domain-adapted pretraining sub- stantially reduces labeled training data requirements, it does not homogenize performance. Instead, tasks exhibit distinct label efficiencies that persist even under stronger supervi- sion and higher-parameter adaptation, suggesting that the effi- ciency with which different aspects of surgical understanding can be learned depends not only on representation quality, but also on the structure of the task itself. 2. Results Understanding label efficiency in surgical computer vision requires disentangling the interplay between representation quality, task structure, and supervision. To this end we build on large-scale domain-adapted self- supervised pretraining, training ViT-Base and ViT-Huge+ on SurgSpectrum-30M+ (over 30 million surgical video frames), resulting in SURGE-B and SURGE-H, providing a strong and unified visual representation for surgical video analysis. As a result, SURGE provides a well-suited foundation for studying label efficiency, as it reduces performance bottlenecks associ- ated with weaker or less robust representations. This unified representation enables us to study supervision scaling while reducing confounding effects from representation quality. We evaluate these representations across a diverse set of surgical tasks, spanning two main categories: surgical percep- tion and more fine-grained and clinically grounded reasoning tasks. Surgical perception tasks are separated into temporal workflow understanding at different granularities (phase and step recognition) and spatial perception (semantic segmenta- tion), while reasoning tasks focus on analyzing surgical exe- cution. These reasoning tasks include clinically grounded as- sessments (Critical View of Safety recognition) [32] as well as relational instrument-anatomy understanding (action triplet recognition), which has shown correlations with patient out- comes [33]. Action triplet recognition refers to predicting sur- gical actions as ⟨instrument, verb, target⟩ tuples of tool-tissue interactions. We systematically vary the amount of supervision, from a single training video to the full training set for each task, and evaluate all models using a unified linear-probing protocol in which only a linear classifier is trained on frozen representa- tions. This protocol isolates the contribution of representation quality to downstream label efficiency. We compare SURGE with general-domain foundation models [23] and prior surgi- cal self-supervised approaches [24, 28, 26]. Across tasks, per- formance improves consistently with increasing supervision, while stronger representations provide substantial gains, par- ticularly in low-label regimes. We quantify label efficiency using normalized area under the curve (nAUC; see Methods and see Fig. 1c). We compute nAUC in two settings: over the full training budget and in a low-label regime consisting of 1 and 3 training videos. 2.1. SURGE outperforms other foundation models across sur- gical AI tasks SURGE achieves the best performance across all 15 bench- marks spanning diverse surgical tasks and supervision regimes (Fig. 1, Figs. 3, 4). For the subsequent label-efficiency analy- sis, we consider 14 benchmarks for which supervision can be systematically varied at the video level. AutoLaparo-Seg [9] is evaluated using its official train/validation/test splits only, as its data cannot be partitioned into individual videos for con- trolled supervision scaling. We illustrate this overall advan- tage with representative results from temporal workflow un- derstanding and fine-grained relational reasoning. For temporal workflow understanding, SURGE-B surpasses the strongest baseline on every dataset, achieving an aver- age F1 score of 69.40 (95% CI: 63.30-75.85), compared with 63.63 (95% CI: 57.33-70.08) for the strongest competing foundation model. Scaling the representation from SURGE-B to SURGE-H further improves performance to 72.23 (95% CI: 66.18-78.51), demonstrating consistent gains with increased model capacity. The advantage extends beyond temporal perception to more fine-grained relational reasoning. On instrument-anatomy in- teraction recognition, SURGE-B achieves 38.72 mAP (95% CI: 34.3-43.13), compared with 32.87 (95% CI: 28.6-37.13) for the strongest per-dataset baselines. SURGE-H further increases performance to 41.40 mAP (95% CI: 38.6-44.2). Thus, domain-adapted pretraining consistently improves sur- gical video representations across both temporal perception and fine-grained relational understanding, with further gains from scaling model capacity. 2.2. Consistently superior label efficiency across all annota- tion regimes We next examine whether the performance gains of SURGE translate into greater label efficiency across the full range of annotation budgets (Fig. 2). Averaged across datasets, SURGE-H achieves an nAUC of 0.914 (95% CI: 0.884–0.942), substantially outperforming the strongest base- lines on each dataset with 0.769, (95% CI: 0.724–0.809). This advantage persists in low-label settings (Fig. 2), where SURGE-H attains a low-label nAUC of 0.757 (95% CI: 0.683–0.835) versus 0.624 (95% CI: 0.549–0.692). We further analyze the effect of model scale. SURGE-B and SURGE-H contain 85.67M and 840.63M parameters, respec- tively, and larger models consistently improve performance, with SURGE-H outperforming SURGE-B in 14 out of 15 4 DINOv3-B DINOv3-H SelfSupSurg SurgNetXL LemonFM SURGE-B SURGE-H Cholec80 AutoLaparo StrasBypass70 BernBypass70 GraSP PmLR50 StrasBypass70 BernBypass70 GraSP Endoscapes-Seg50 GraSP CholecT50 ProstaTD Endoscapes Dataset / Task 0.540.600.670.700.640.790.86 0.670.690.520.700.780.830.88 0.530.540.490.770.650.840.91 0.380.400.310.560.430.550.65 0.650.690.530.730.730.800.94 0.620.610.610.700.700.730.73 0.350.380.280.570.470.680.75 0.290.320.210.450.370.520.61 0.640.680.470.650.680.770.88 0.630.660.450.360.480.750.69 0.730.830.280.450.380.930.96 0.360.400.400.440.440.580.63 0.340.390.280.380.400.510.58 0.450.420.470.450.460.460.52 Phase Steps Segmentation Triplet CVS Low-Label (13 Videos) nAUC DINOv3-B DINOv3-H SelfSupSurg SurgNetXL LemonFM SURGE-B SURGE-H 0.740.790.800.820.800.930.97 0.720.750.600.740.830.900.95 0.730.760.650.880.770.940.98 0.580.650.480.720.660.770.86 0.740.770.620.820.800.870.98 0.770.780.710.810.840.910.91 0.610.660.510.760.700.870.94 0.540.610.400.690.640.790.88 0.750.790.560.740.760.860.95 0.730.720.540.490.610.910.87 0.790.880.290.480.410.950.98 0.580.700.600.630.630.860.89 0.460.550.400.510.570.690.78 0.710.710.730.720.710.810.88 Overall nAUC 0.20.30.40.50.60.70.80.91.0 Normalized AUC Score Model Low-Label & Overall Label Efficiency Fig. 2: Overall and low-data regime label efficiency comparison: measuring normalized AUC (nAUC) across tasks/datasets, highlighting most efficient model both in heatmap color coding and box highlighting. benchmarks, as depicted in Fig. 1b. This improvement is es- pecially pronounced in low-label settings; while task variance results in wide confidence intervals, paired analysis reveals a highly significant and substantial performance gain for the larger model (SURGE-H: 0.757 (95% CI: 0.683–0.835) and SURGE-B: 0.696 (95% CI: 0.619-0.769)); Wilcoxon signed- rank p< 0.01, Cohen’s d = 1.25). Overall, SURGE substantially improves label efficiency in surgical AI, enabling strong performance with reduced anno- tation effort. These gains are consistent across five surgical procedures (Fig. 2), although the magnitude of improvement varies across tasks, reflecting differences in how performance scales with supervision. 2.3. Domain adaptation bridges general and surgical repre- sentation strengths We next examine why SURGE performs consistently across diverse surgical tasks.Prior surgical self-supervised ap- proaches are particularly effective on workflow and clinically grounded tasks (Critical View of Safety), especially in low- data regimes, but are less effective for dense spatial percep- tion and fine-grained instrument-anatomy interaction seman- tics. In contrast, general-domain self-supervised models pro- vide stronger representations for tasks such as semantic seg- mentation and instrument-anatomy interaction (Fig. 2). SURGE combines these complementary strengths. It re- tains the strong performance of surgical self-supervised ap- proaches on workflow and clinically grounded tasks while exceeding general-domain models on dense spatial and interaction-level tasks.For example, SURGE-H achieves Fig. 3: Coarse temporal perception (phase recognition) performance curves: All subset runs are averaged over 3 runs with error bands highlighting the standard deviation. 5 Fig. 4: Additional perception & reasoning task performance curves: a shows spatial understanding (semantic segmentation), b: fine temporal un- derstanding (step recognition), c: safety maneuver analysis (CVS), and in d instrument-anatomy interaction (action triplet). All subset runs are averaged over 3 runs with error bands highlighting the standard deviation. 82.95 mIoU on GraSP, compared with 76.09 for DINOv3- H, and 44.20 mAP on CholecT50, compared with 37.13 for DINOv3-H. These results suggest that large-scale domain adaptation enables representations that capture both general visual structure and fine-grained surgical semantics, provid- ing a potential basis for SURGE’s broad transferability and label efficiency. 2.4. Perception tasks exhibit high label efficiency with task structure dependent variations Our analysis reveals that surgical perception tasks exhibit high label efficiency, with performance saturating under min- imal supervision. Tasks that rely primarily on visual cues or coarse temporal structure achieve strong performance with very limited annotated data, with diminishing gains as more data is added. For instance, in semantic segmentation on prostatectomy (GraSP dataset), SURGE-H attains an mIoU of 76.68± 2.53% using only a single training video, with perfor- mance remaining nearly unchanged as the training set scales to eight videos as shown in Fig. 4a. Similarly, in phase recog- nition, SURGE-H reaches 74.04± 2.09% F1 on gastric bypass (StrasBypass70) with one video and quickly plateaus with ad- ditional data, while achieving comparable behavior on chole- cystectomy (Cholec80) depicted in Fig. 3. In these settings, a single labeled video often suffices to outperform compet- ing pretrained surgical models trained on substantially larger datasets. While this regime of high label efficiency is consistent across perception tasks, we find that structural properties of the task modulate the rate at which saturation is reached. In- creasing the granularity of the prediction objective introduces a more pronounced dependence on labeled data. On Stras- Bypass70, while coarse phase recognition improves modestly with additional data (8.5% F1 gain from 1 to 40 videos), the more fine-grained step recognition task exhibits a substantially steeper scaling curve, improving by 17.3% F1 over the same range seen in Fig. 4b. Thus, increasing temporal granularity substantially increases the amount of supervision required to reach comparable performance. Similarly, the complexity of the annotations influences data requirements even within this otherwise efficient regime. Comparing semantic segmentation across two datasets (GraSP and Endoscapes-Seg50) in Fig. 4a illustrates this effect. GraSP reaches strong performance quickly with limited data, whereas Endoscapes-Seg50 improves more gradually, starting from a lower baseline and continuing to benefit from addi- tional annotations. This difference reflects the nature of the labels: GraSP focuses on a smaller set of well-defined struc- tures, primarily surgical instruments, whereas Endoscapes- Seg50 includes a larger and more fine-grained label space, including small anatomical structures alongside instruments, resulting in a more complex prediction problem. As a result, the model must learn a more complex mapping from visual features to labels, which increases the amount of data required to achieve comparable performance. Taken together, these results show that surgical perception tasks form a regime of high label efficiency characterized by rapid saturation with minimal supervision. Although fac- tors such as task granularity, annotation density, and semantic complexity modulate the precise label efficiency, they do not alter the overall trend of efficient learning in these tasks. In the next section, we show that this regime breaks down for reason- ing tasks, which exhibit distinctly different label efficiency and remain far from saturation even under substantially increased supervision. 2.5. Reasoning tasks exhibit persistent data requirements In contrast to the rapid saturation observed in perception tasks, reasoning tasks exhibit fundamentally different label ef- ficiency, remaining far from saturation even under substan- tially increased supervision. In the instrument-anatomy interaction tasks (action triplet recognition) on the CholecT50 and ProstaTD benchmarks shown in Fig. 4d, SURGE-H maintains a consistent lead over all baselines, yet displays a remarkably persistent log-linear scaled curve that still allows for sustained performance gains under constant increase in labeled data. For instance, on ProstaTD, SURGE-H begins at 20.30 ± 2.53 mAP and re- quires 14 training videos to reach 38.36 mAP, a trajectory that lacks the ’jump-start’ characteristic, where 1-video per- formance is already a significant fraction of the ceiling, ob- served in temporal (phase) or spatial (semantic segmentation) perception tasks. This is also supported by the nAUC met- ric, where perception tasks achieve average scores of 0.925, 6 Fig. 5: Label instances and label efficiency comparison across tasks & higher parameter adaptation results: a shows task comparisons for prostatectomy and b for cholecystectomy. Highlighting severe annotation density differences across tasks. c&d: We select the best model for this analysis (SURGE-H) and compare linear probing (SURGE-H LP) and partial tuning of the backbone (SURGE-H PT). All subset runs are averaged over 3 runs with error bands highlighting the standard deviation. SOTA on Endoscapes is LG-CVS [34] and on CholecT50 is TripletGCN [35]. 0.940, and 0.919 for semantic segmentation, phase recogni- tion, and step recognition, respectively, compared to 0.832 for action triplet recognition. This difference is also present in the low-label regime, where semantic segmentation, phase recognition, and step recognition achieve average low-label nAUC values of 0.825, 0.828, and 0.750, respectively, versus 0.607 for instrument-anatomy interaction (action triplet recog- nition). This structural bottleneck is most pronounced in the clinical quality assessment (Critical View of Safety (CVS)), depicted in Fig. 4c, on the Endoscapes dataset. Despite leveraging the same large-scale pretrained representation, performance for SURGE-H starts at 27.92 ± 1.17 mAP and requires sub- stantially more labeled data, reaching 53.26 mAP only when scaling to 120 training videos. In contrast to other tasks, we observe an apparent threshold-like behavior, where all models exhibit limited learning signal when trained with 10 or fewer videos, and only begin to improve meaningfully beyond this regime. This pattern is further reflected in the label-efficiency metric (nAUC), which reaches only 0.882 when evaluated across all training budgets and drops to 0.520 in the low-label regime. Notably, the low-label nAUC is substantially lower than that observed for any other task, indicating that this task benefits less from limited supervision. This suggests that cer- tain reasoning tasks require a minimum amount of supervision before meaningful generalization can occur. 2.6. Label instances as an indicator of supervision require- ments To further ground these differences in data requirements, we next quantify the underlying supervision effort associated with each task, providing a practical proxy for label efficiency. While true annotation cost is a complex composite of me- chanical time (e.g., pixel-wise segmentation) and cognitive load (e.g., clinical consensus for safety criteria), direct mea- surement of human hours across retrospective, multi-center datasets is very difficult and annotator specific. To overcome this limitation and provide a standardized baseline, we intro- duce label instances, the absolute count of all explicitly anno- tated instances across a dataset, as an objective, macro-level proxy for total annotation effort. By systematically quanti- fying the volume of human-generated labels required per task category, we offer a first-of-its-kind approximation of compar- ative supervision demands. As shown in Fig. 5a-b, this result- ing distribution closely mirrors the previously observed scal- ing patterns, demonstrating that total label instances serves as a powerful indicator of the intrinsic human effort required to solve different surgical tasks. Notably, more clinically relevant tasks requiring clinical reasoning and relational instrument-anatomy understanding (e.g., CVS and action triplet recognition) exhibit substantially higher label counts, reflecting significantly greater annotation effort. Despite this increased supervision, their performance continues to improve with additional annotations without clear signs of saturation. This indicates that these tasks remain data- hungry even under substantially higher annotation volumes. In contrast, temporal and spatial perception tasks rely on comparatively fewer annotations and exhibit more favorable learning efficiency.Among these, semantic segmentation stands out with the highest number of labels per video due to its dense annotation structure, yet still benefits from com- paratively efficient learning dynamics. Overall, these findings reveal a pronounced imbalance in supervision demands across tasks and indicate that label effi- ciency is strongly task-dependent. 7 2.7. Higher parameter adaptation surpasses supervised SOTA without altering label efficiency Given the strong task-dependent differences in label effi- ciency observed above, we next examine whether these dy- namics can be altered through higher parameter adaptation, in which more parameters of the pretrained backbone are up- dated using task-specific supervision rather than training only a linear classifier. While higher parameter adaptation substan- tially improves absolute performance, enabling data-intensive tasks such as relational instrument-anatomy recognition and clinical reasoning to surpass task-specific supervised state-of- the-art baselines, the fundamental label efficiency with respect to supervision remains unchanged. As shown in Fig. 5c-d, partial adaptation consistently shifts performance upwards, yet preserves the relative gains obtained from additional supervision.Notably, this up- ward shift yields absolute state-of-the-art gains of 6.4% mAP on Endoscapes and 15.5% mAP on CholecT50 action triplet recognition. These improvements are achieved using our generalized approach, without relying on the complex, task-specific modifications used by previous baselines, such as segmentation-based supervision [34] or ensemble strate- gies [35]. 3. Discussion Our results provide a systematic view of how surgical AI learns from expert supervision across tasks with fundamen- tally different demands. Large-scale domain-adapted self- supervised pretraining produces a broadly transferable repre- sentation that consistently outperforms both general-domain and prior surgical foundation models across 15 benchmarks, while providing substantial gains in low-label regimes. How- ever, our results also reveal that representation quality alone does not determine label efficiency. Surgical perception tasks typically reach high performance with minimal supervision and rapidly saturate, whereas clinically grounded and rela- tional reasoning tasks continue to benefit from substantially more supervision. Increasing the number of adapted param- eters raises absolute performance and enables state-of-the-art results on these more demanding tasks, but does not funda- mentally alter their scaling with supervision. Together, these findings suggest that representation quality determines how effectively models can exploit available supervision, whereas the structure of the downstream task determines how much supervision is ultimately required. 3.1. The saturation of perception tasks The rapid saturation observed in spatial understanding tasks (semantic segmentation) and temporal understanding (phase recognition) suggests that the field may be approaching a mile- stone in perceptual learning. Our results show that with suf- ficiently large domain-adapted pretraining, the model’s inter- nal representation of surgical instruments, tissue textures, and broad procedural states becomes “task-ready" with very lim- ited label supervision. In these regimes, a single training video acts not as a source of feature learning, but as a “Rosetta Stone" that simply maps existing high-quality representations to class names. This finding suggests an immediate pivot in how we al- locate annotation resources.For standard procedures and instrument-centric tasks, the pursuit of larger datasets may ex- hibit diminishing returns. Instead, these “solved" perceptual components should be leveraged as automated foundations to assist in the more difficult task of mapping complex medi- cal/surgical logic. 3.2. The persistent reasoning gap and the need for hybrid modeling In contrast to the saturation of perception, fine-grained and reasoning tasks such as action triplet recognition and Criti- cal View of Safety (CVS) assessment exhibit a “Reasoning Gap” that is not fully addressed by scale alone. These tasks require the model to move beyond identification toward com- positional logic and clinical judgment. The persistent growth of the scaling curves observed in these domains indicates that the model is actively learning the underlying rules of surgery frame-by-frame, rather than solely relying on pretrained visual archetypes. This resistance to pretraining gains does not suggest a lim- itation of the underlying representations, adaptation strategy, or model capacity, but rather may reflect the current frontier of deep learning. To cross this Reasoning Gap, we appear to require substantially more annotated data efforts compared to prior perception tasks or a transition toward hybrid method- ologies. Such approaches might include a combination of largely-pretrained foundation models with task-specific archi- tectural modeling, the integration of multi-modal clinical in- puts, or the development of pretraining objectives that explic- itly target causal and relational structures to learn those com- plicated problems under limited supervision efficiently. 3.3. A strategic roadmap to jumpstart Surgical AI By characterizing how data requirements vary across surgi- cal tasks, our work provides a practical framework for under- standing their label efficiency and guiding annotation strate- gies. This perspective enables a more principled allocation of labeling effort toward clinically impactful objectives: • Low-shot transfer for perception: Tasks such as in- strument segmentation and coarse video understanding (phase recognition) exhibit high label efficiency and ben- efit strongly from domain-adapted pretraining. For these tasks, annotation requirements can be substantially re- duced, opening the door to few-shot or weakly super- vised approaches. • Targeted annotation for reasoning: In contrast, tasks involving relational instrument-anatomy understanding and clinical assessment remain significantly more data- intensive. Expert annotation effort, being scarce and costly, should therefore be prioritized toward these set- tings. 8 • Toward hybrid surgical AI: Bridging the gap between perception and higher-level understanding is likely to re- quire approaches that go beyond representation learn- ing, combining strong visual foundations with task-aware modeling, temporal reasoning, and integration of clini- cally relevant context. Such approaches may also be im- portant for achieving the robustness required for deploy- ment in real-world clinical settings. Ultimately, our findings suggest that while foundation mod- els have substantially advanced the visual perception capabil- ities of AI, translating these representations into data-efficient reasoning remains a key challenge. By grounding this per- spective in task-dependent data requirements, we outline a path toward more efficient, scalable, and clinically meaningful AI systems. 3.4. Limitations & Future Work While our study provides a controlled analysis of represen- tation quality through linear probing and partial fine-tuning, this protocol inherently underestimates the peak performance achievable through task-specific fine-tuning. To realize its full potential, we release this foundation model as a shared tool for the research community. Future optimization, incorporating this model into specialized pipelines via architectural adapters or proxy losses, will enable practitioners to deeply adapt these baseline representations for their own specialized clinical ap- plications. A more fundamental limitation lies in the “Reasoning Gap" identified in our scaling analysis. The persistent log-linear data requirements for CVS and action triplet recognition sug- gest that distillation-based pretext tasks may not fully cap- ture the causal and logical dependencies of surgery. To re- solve this, future pretraining should move toward embedding a deeper clinical reasoning behind observed videos and mul- timodal foundations, specifically targeting vision-text align- ment (e.g., surgical reports). Such models could enable zero- shot transfer and may facilitate semi-automated annotation in label-sparse environments, effectively “jump-starting" the de- velopment of AI for rare or specialized medical procedures. 4. Methods 4.1. SurgSpectrum-30M+: pretraining dataset generation In order to unlock a truly label-efficient and multi-procedure foundation model, we construct SurgSpectrum-30M+, the largest surgical pretraining corpus to date. The dataset is assembled from a combination of public and proprietary sources across multiple centers and is designed to capture broad variability across surgical practice, spanning 132 pro- cedures and 47 anatomical targets; more details can be seen in Fig.6. We curated this dataset from multiple public sources [8, 36, 11, 19, 37, 38, 39, 40, 41, 42, 43, 10, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 30] as well as in-house surgi- cal recordings and sampled frames from each video at 1 FPS which ensures a certain degree of visual diversity. To avoid any data leakage for potential downstream tasks we ensured to remove all videos/frames associated to validation or test sets of the public sources. As a critical preprocessing step, we train an out-of-body detector to remove non-informative con- tent, including preparation phases and external camera views, thereby retaining only intraoperative endoscopic footage. This filtering step reduces visual noise and enforces domain speci- ficity, resulting in a dataset that emphasizes clinically mean- ingful patterns while preserving diversity. The final corpus comprises 30,381,943 frames from 38,753 videos. 4.2. Model architecture The network employed in this work is based on the trans- former architecture [54], which leverages self-attention to model long-range dependencies and capture global contextual interactions. Originally developed for natural language pro- cessing, transformer-based models have since demonstrated strong performance in visual representation learning by op- erating on images as sequences of patches [55]. This formu- lation enables the encoding of complex spatial and semantic relationships within visual data. Specifically, input images are partitioned into non- overlapping patches of size 16× 16, which are flattened and projected into a learnable embedding space. Positional in- formation is incorporated using rotary positional embeddings (RoPE) [56], allowing the model to capture relative spatial re- lationships between patches. The resulting sequence of patch embeddings is processed by a stack of transformer layers, yielding both global image-level representations and localized patch features. We consider two model scales based on the Vision Trans- former (ViT) design: ViT-Base and ViT-Huge+. Both models are initialized from pretrained DINOv3 checkpoints and sub- sequently adapted to our surgical pretraining dataset. The base model comprises 12 transformer layers with a hidden dimen- sion of 768 and 12 attention heads, whereas the huge+ variant consists of 32 transformer layers with a hidden dimension of 1280 and 20 attention heads, and uses SwiGLU as its feed- forward network. 4.3. Pretraining objectives We adopt a discriminative self-supervised objective com- bining image-level and patch-level supervision. Specifically, the overall loss consists of a DINO loss [57] applied at the image level and an iBOT loss [22] applied at the patch level, complemented by additional regularization terms. At the image level, the DINO objective aligns the outputs of a student and a teacher network operating on different views of the same image. Let p s and p t denote the softmax-normalized outputs of the student and teacher for a given view, respec- tively. G and L correspond to the sets of global and local views of an input image, respectively. The loss is defined as shown in Eq. 1: L DINO =− X g∈G X l∈L p t (g) log p s (l).(1) At the patch level, the iBOT objective, shown in Eq. 2, ex- tends this formulation to masked image modeling. A subset of patches is masked in the student input, while the teacher 9 Fig. 6: SurgSpectrum-30M+. Overview of the proposed large-scale surgical pretraining dataset. Dataset composition is visualized on an absolute scale for total size, while source distributions are reported as percentages. Compared to existing resources, SurgSpectrum-30M+ increases data scale by more than 6× relative to the next largest dataset, comprising over 30 million frames from approximately 39k videos. The dataset spans more than 130 surgical procedures, substantially increasing procedural diversity and enabling large-scale representation learning in the surgical domain. 10 processes the full image. The student predictions for masked tokens are trained to match the corresponding teacher outputs, whereM denotes the set of masked patches: L iBOT =− X g∈G X l∈L X i∈M p (i) t (g) log p (i) s (l).(2) The final objective combines both terms, encouraging con- sistency at both global and local levels. Additional regulariza- tion such as feature decorrelation [58] is also applied denoted asL reg , depicted in Eq. 3: L =L DINO +L iBOT + 0.1·L reg .(3) 4.4. Post-Training refinement FollowingtheestablishedDINOv3pretraining paradigm [23], we apply a post-training refinement stage con- sisting of resolution scaling and Gram anchoring. Prolonged self-supervised training can sometimes degrade patch-level features, reducing performance on dense prediction tasks. As demonstrated by DINOv3, this degradation is mitigated by a high-resolution adaptation step, in which the model is exposed to inputs at varying and higher resolutions. This enables the model to better capture fine-grained spatial details and improves robustness to resolution changes at inference time [23]. In conjunction, we apply Gram anchoring, a regularization objective that enforces consistency between feature correla- tions of the student and a teacher model. Concretely, it aligns the Gram matrices of patch features, thereby preserving spa- tial structure and preventing the collapse of dense representa- tions. This is particularly important at high resolutions, where maintaining coherent local features becomes more challeng- ing [23]. Together, these refinements restore patch-level consistency and significantly enhance performance on dense tasks, while preserving strong global representations. 4.5. Downstream task evaluation To comprehensively evaluate the learned representations, we consider five distinct surgical task categories across a to- tal of 15 benchmarks. These tasks span complementary as- pects of surgical understanding, including temporal workflow analysis, spatial perception, relational reasoning, and clini- cally grounded assessment. Importantly, they cover multi- ple procedures with diverse visual characteristics and work- flow structures, enabling a systematic study of generalization across both tasks and domains. Evaluating across multiple tasks and procedures is critical for two reasons. First, it allows us to assess whether learned representations capture transferable surgical knowledge be- yond a single task or dataset. Second, it provides a controlled setting to analyze how label efficiency varies as a function of task structure, rather than being confounded by dataset- specific properties. To measure label efficiency in training- data subset experiments, we use three independent random video splits and report the mean ± standard deviation across splits. Surgical Workflow Recognition. We evaluate temporal under- standing at two levels of granularity: coarse-grained surgical phases and fine-grained procedural steps. Phase recognition captures high-level stages of a procedure (e.g., preparation, dissection, closure), whereas step recognition focuses on more detailed sub-actions within each phase, requiring finer tempo- ral discrimination. For phase recognition, we consider Cholec80 [8] (cholecys- tectomy), GraSP [11] (prostatectomy), MultiBypass140 [10], comprising StrasBypass70 (gastric bypass,Strasbourg, France) and BernBypass70 (gastric bypass, Bern, Switzer- land), AutoLaparo [9] (hysterectomy), and PmLR50 [59] (liver resection). For step recognition, we evaluate on the re- spective steps annotations of StrasBypass70, BernBypass70, and GraSP. This distinction between phases and steps enables us to study how label efficiency varies with temporal granularity, as step recognition typically requires more precise temporal localization and finer semantic understanding. Surgical Semantic Segmentation. To assess spatial percep- tion, we evaluate semantic segmentation on Endoscapes- Seg50 [13], consisting of 50 videos with annotations for six semantic classes, and GraSP [11], which provides pixel-wise annotations for seven surgical tools. These tasks require dense spatial predictions and provide strong supervision signals at the pixel level, enabling analysis of label efficiency under highly localized supervision. Triplet Recognition. We evaluate relational reasoning via triplet recognition, capturing interactions between surgical instruments, actions, and anatomical structures.For this we consider CholecT50 [18], which defines 100 unique in- strument–verb–tissue combinations for cholecystectomy, and ProstaTD [19] for prostatectomy with 89 interaction classes. These tasks require joint reasoning over multiple entities and represent a higher level of semantic abstraction compared to purely spatial or temporal tasks. Critical View of Safety (CVS). We evaluate clinically grounded understanding using the Critical View of Safety (CVS) task on the Endoscapes dataset [13]. CVS assessment is based on three criteria defined by Strasberg [32], which must be satisfied to ensure safe dissection during cholecys- tectomy. Each frame is annotated independently by three ex- perts, and final labels are obtained via majority voting. This task requires high-level clinical reasoning and is particularly challenging due to its abstract and safety-critical nature. Evaluation metrics. We use task-appropriate evaluation met- rics for each category. For phase and step recognition, we report the F1 score, which captures the balance between pre- cision and recall in temporally structured predictions. For se- mantic segmentation, we use mean Intersection over Union (mIoU), a standard metric for evaluating pixel-wise overlap between predicted and ground-truth masks. For triplet recog- nition and CVS assessment, we report mean Average Preci- sion (mAP), which reflects performance in multi-label and 11 detection-style settings where multiple interactions or criteria may be present simultaneously and are threshold independent. Normalized area under the curve (nAUC). To compactly quantify label efficiency across tasks and datasets, we use the normalized area under the curve (nAUC). The metric summa- rizes model performance as a function of annotation budget. For each task, performance is normalized by the maximum value achieved across all models. We then compute the area under the resulting performance curve using trapezoidal in- tegration and normalize it by the range of the input domain, yielding a value in [0, 1]. Higher nAUC indicates higher per- formance relative to the task-specific best across the annota- tion budget, reflecting greater overall label efficiency. Let (x i , y i ) n i=1 denote performance y i measured at training set sizes x i , sorted such that x 1 < ·< x n . We normalize performance as shown in Eq. 4: ̃y i = y i y max ,where y max = max 1≤ j≤n y j .(4) The normalized area under the curve (nAUC) is computed using trapezoidal integration displayed in Eq. 5: nAUC = 1 x n − x 1 n−1 X i=1 ̃y i + ̃y i+1 2 ( x i+1 − x i ) .(5) To emphasize low-label regime learning behaviour, we ad- ditionally compute a low-label regime nAUC over the first two points ̃y 1 and ̃y 2 representing the performance at 1 and 3 training videos respectively, shown in Eq. 6. Higher low- label regime nAUC indicates stronger performance when only a small number of labeled videos are available: nAUC low-label = 1 x 2 − x 1 · ̃y 1 + ̃y 2 2 ( x 2 − x 1 ) = ̃y 1 + ̃y 2 2 .(6) 4.6. Implementation details Pretraining. We perform large-scale self-supervised pretrain- ing for both model variants, ViT-Base (SURGE-B) and ViT-Huge+ (SURGE-H), on the SurgSpectrum-30M+. Pre- training is conducted for 180k iterations with a batch size of 2048, corresponding to approximately 12 epochs over the dataset. Following this stage, we apply an additional post- refinement phase for 30k iterations with a reduced batch size of 64. This stage incorporates multi-resolution adaptation and Gram anchoring to further improve representation quality. During multi-resolution training, we employ two global crops at resolutions of 512 × 512 and 768 × 768, and three local crops for the student network at 112× 112, 168× 168, and 336× 336. Optimization is performed using a base learn- ing rate of 4× 10 −4 and a weight decay of 0.04. Linear probing evaluation. To evaluate the quality of the learned representations, we adopt a linear probing protocol across all downstream tasks. For image-level prediction tasks (i.e., all except semantic segmentation), we follow the proto- col used in [23] by constructing a global representation via concatenating the [CLS] token with the average pooled patch tokens, which is then passed to a single-layer MLP for task- specific prediction. For semantic segmentation, we directly use the patch-level features and apply a linear classification head to obtain dense predictions, which are subsequently upsampled via bilinear in- terpolation to the original image resolution. For convolutional architectures, we apply an equivalent strategy by operating on the final feature maps. During linear probing, the pretrained backbone remains frozen and only the task-specific head is optimized. To en- sure fair comparison, we perform a systematic hyperparame- ter search over learning rates for each task and dataset over the respective validation sets, and apply the same protocol consistently across all evaluated methods, including both our models and the considered baselines. Learning rates were se- lected from the following set based on validation performance: [ 10 −5 , 2×10 −5 , 5×10 −5 , 10 −4 , 2×10 −4 , 5×10 −4 , 10 −3 , 2×10 −3 , 5× 10 −3 , 10 −2 , 2× 10 −2 , 5× 10 −2 , 10 −1 ]. This protocol isolates the quality of the learned representa- tions and enables a controlled comparison of label efficiency across tasks without confounding effects from task-specific fine-tuning. Higher parameter evaluation. In addition, we consider a higher-capacity adaptation setting in which a subset of the pre- trained backbone layers is fine-tuned together with the task- specific head. We perform this experiment only for SURGE- H, which achieved the best performance in the linear probing experiments, and conduct a hyperparameter search over the number of final layers to fine-tune (k ∈ 1, 3, 6, 9, 12, 15) and the learning rate separately for each of the two datasets de- scribed in section 2.7. The remaining backbone layers are kept frozen, and the best configuration on the respective validation set is used for the final evaluation on the corresponding test set. Data availability The pretraining corpus used in this study, SurgSpectrum- 30M+, is a large-scale aggregation of both public and pri- vate surgical video sources. Publicly available components, including datasets such as Cholec80 [8], ProstaTD [19], SurgVu24 [49], OCHRA rectal cancer surgery [45], and many more (see section 4.1) can be accessed through their respec- tive official repositories. The private portions of the cor- pus consist of institutional data that cannot be publicly re- leased due to patient privacy regulations and institutional data- sharing agreements. For downstream evaluation, all datasets (Cholec80, CholecT50, GraSP, Endoscapes, etc.) are publicly available through their original providers. Code availability To support the reproducibility of our findings and facili- tate further research in surgical foundation models, we provide the pretrained model weights for SURGE-B and SURGE-H, 12 along with the necessary loading-, inference-,and evaluation scripts on all downstream tasks. The implementation is based on PyTorch and is available at https://github.com/CAMMA- public/SURGE. Acknowledgements This work was partially supported by French state funds managed by the ANR under Grants ANR-22-FAI1- 0001 (project DAIOR), ANR-10-IAHU-02 (IHU Strasbourg), ANR-23-IACL-0004 (AI Cluster Grand Est ENACT) and by the Interdisciplinary Thematic Institute HealthTech (ITI 2021- 2028 program of the University of Strasbourg, CNRS and In- serm) via the IdEx Unistra (ANR-10-IDEX-0002) and SFRI (STRATUS project, ANR-20-SFRI-0012).This work has also received funding from the European Union (ERC, Comp- SURG, 101088553). Views and opinions expressed are how- ever those of the authors only and do not necessarily reflect those of the European Union or the European Research Coun- cil. Neither the European Union nor the granting authority can be held responsible for them. Author contributions F.S., N.N. and N.P. conceived and designed the work. F.S. contributed to the technical implementation and conducted ex- periments. L.A. and P.M. provided clinical expertise and col- lected the data for the self-supervised pretraining. The mem- bers of the CAMMA International Surgical Partners provided clinical expertise and access to proprietary datasets. F.S., L.A., V.S., P.M., and N.P. contributed to the drafting and revising of the manuscript. All authors have received and reviewed the manuscript. N.N. and N.P. supervised the research. Competing interests P.M. and N.P. are co-founders and shareholders of Scialyt- ics. The remaining authors declare no competing interests. CAMMA International Surgical Partners Members Luigi Boni 1 , Marta Goglia 2 , Gianfranco Silecchia 2 , Andrea Balla 3 , Salvador Morales-Conde 3 , Giovanni Guglielmo Laracca 4 ,Claudio Fiorillo 5 ,Giuseppe Quero 5 ,Vin- cenzo Tondolo 5 , Ludovica Baldari 1 , Elisa Cassinotti 1 , Ludovica Guerriero 6 , Isacco Montroni 7 , Anna Fagotti 8 , Alice Zampolini 8 , Riccardo Oliva 8 , Xin Wang 9 , Matteo Pavone 8 , Nicolò Bizzarri 8 , Lise Lecointre 10 Affiliations 1. Department of General & Minimally Invasive Surgery Fondazione IRCCS - Ca’ Granda - Ospedale Maggiore Policlinico di Milano, Milan, Italy 2.Department of Medical-Surgical Sciences and Trans- lational Medicine, Faculty of Medicine and Psychology, Sapienza University of Rome, Italy 3.University Hospital Virgen Macarena, University of Sevilla, Seville, Spain 4. Sant’Andrea Hospital, Sapienza University, Rome, Italy 5. Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy 6. Monaldi Hospital, AORN dei Colli, Naples, Italy 7. IRCCS Fondazione Istituto Nazionale dei Tumori, Division of Colon and Rectal Cancer Surgery, Milan, Italy 8.Gynecologic Oncology Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Università Cattolica del Sacro Cuore, Rome, Italy 9. West China Hospital of Sichuan University, Chengdu, China 10. Department of Gynecologic Surgery, University Hospitals of Strasbourg, Strasbourg, France; ICube, Laboratory of Engineering, Computer Science and Imaging, Department of Robotics, Imaging, Teledetection and Healthcare Tech- nologies, University of Strasbourg, CNRS, UMR 7357, Strasbourg, France References [1] Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai. In proceed- ings of the 2021 CHI Conference on Human Factors in Computing Sys- tems, pages 1–15, 2021. [2] Lena Maier-Hein, Matthias Eisenmann, Duygu Sarikaya, Keno März, Toby Collins, Anand Malpani, Johannes Fallert, Hubertus Feussner, Stamatia Giannarou, Pietro Mascagni, et al. Surgical data science– from concepts toward clinical translation.Medical image analysis, 76:102306, 2022. [3] Lena Maier-Hein, Swaroop S Vedula, Stefanie Speidel, Nassir Navab, Ron Kikinis, Adrian Park, Matthias Eisenmann, Hubertus Feussner, Germain Forestier, Stamatia Giannarou, et al. Surgical data science for next-generation interventions.Nature Biomedical Engineering, 1(9):691–696, 2017. [4] S Swaroop Vedula and Gregory D Hager. Surgical data science: the new knowledge domain. Innovative surgical sciences, 2(3):109–121, 2017. [5] Pietro Mascagni, Deepak Alapatt, Luca Sestini, Maria S Altieri, Amin Madani, Yusuke Watanabe, Adnan Alseidi, Jay A Redan, Sergio Alfieri, Guido Costamagna, et al. Computer vision in surgery: from potential to clinical value. npj Digital Medicine, 5(1):163, 2022. [6] Dmitri Nepogodiev, Janet Martin, Bruce Biccard, Alex Makupe, Aneel Bhangu, D Nepogodiev, et al. Global burden of postoperative death. Lancet, 393(10170):401, 2019. [7] Marieke Zegers, Martine C de Bruijne, Bertus de Keizer, Hanneke Merten, Peter P Groenewegen, Gerrit van der Wal, and Cordula Wagner. The incidence, root-causes, and outcomes of adverse events in surgical units: implication for potential prevention strategies. Patient safety in surgery, 5(1):13, 2011. [8] Andru P Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. Endonet: a deep architec- ture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging, 36(1):86–97, 2016. [9] Ziyi Wang, Bo Lu, Yonghao Long, Fangxun Zhong, Tak-Hong Che- ung, Qi Dou, and Yunhui Liu. Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hys- terectomy. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 486–496. Springer, 2022. [10] Joël L Lavanchy, Sanat Ramesh, Diego Dall’Alba, Cristians Gonzalez, Paolo Fiorini, Beat P Müller-Stich, Philipp C Nett, Jacques Marescaux, Didier Mutter, and Nicolas Padoy. Challenges in multi-centric general- ization: phase and step recognition in roux-en-y gastric bypass surgery. International journal of computer assisted radiology and surgery, pages 1–9, 2024. 13 [11] Nicolás Ayobi, Santiago Rodríguez, Alejandra Pérez, Isabela Hernán- dez, Nicolás Aparicio, Eugénie Dessevres, Sebastián Peña, Jessica San- tander, Juan Ignacio Caicedo, Nicolás Fernández, et al. Pixel-wise recognition for holistic surgical scene understanding. Medical Image Analysis, page 103726, 2025. [12] Yiru Ye, Wenlong Wang, Yonghao Long, Chi-Fai Ng, QingYin Zhou, Dongren Yang, Qi Dou, Mengya Xu, and Zhifang Pan.Self- supervised adaptive transformer for surgical step recognition in robotic- assisted radical prostatectomy. IEEE Robotics and Automation Letters, 11(6):6568–6575, 2026. [13] Aditya Murali, Deepak Alapatt, Pietro Mascagni, Armine Vardazaryan, Alain Garcia, Nariaki Okamoto, Guido Costamagna, Didier Mutter, Jacques Marescaux, Bernard Dallemagne, et al. The endoscapes dataset for surgical scene segmentation, object detection, and critical view of safety assessment: Official splits and benchmark. arXiv preprint arXiv:2312.12429, 2023. [14] Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Dug- gal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, et al. 2017 robotic instrument segmentation chal- lenge. arXiv preprint arXiv:1902.06426, 2019. [15] Devanish N Kamtam, Joseph B Shrager, Satya Deepya Malla, Xiaohan Wang, Nicole Lin, Juan J Cardona, Serena Yeung-Levy, and Clarence Hu. A fine-tuned foundational model surgisam2 for surgical video anatomy segmentation and detection. Scientific Reports, 15(1):35961, 2025. [16] Devanish N Kamtam, Joseph B Shrager, Satya Deepya Malla, Xiaohan Wang, Nicole Lin, Juan J Cardona, Serena Yeung-Levy, and Clarence Hu. Surgisam2: fine-tuning a foundational model for surgical video anatomy segmentation and detection. arXiv preprint arXiv:2503.03942, 2025. [17] Ronald LPD de Jong, Tim JM Jaspers, Raf AH Vervoort, Aron FHA Bakker, Yiping Li, Jip L Tolenaar, Jelle P Ruurda, Willem M Brinkman, Josien PW Pluim, Marcel Breeuwer, et al. Surgical anatomy recognition with context learning using foundation representations. arXiv preprint arXiv:2606.22124, 2026. [18] Chinedu Innocent Nwoye, Tong Yu, Cristians Gonzalez, Barbara Seel- iger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nico- las Padoy. Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Medical Image Analysis, 78:102433, 2022. [19] Yiliang Chen, Zhixi Li, Cheng Xu, Alex Qinyang Liu, Xuemiao Xu, Jeremy Yuen-Chun Teoh, Shengfeng He, and Jing Qin. Prostatd: A large-scale multi-source dataset for structured surgical triplet detection. arXiv preprint arXiv:2506.01130, 2025. [20] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hin- ton. A simple framework for contrastive learning of visual representa- tions. In International Conference on Machine Learning (ICML), 2020. [21] Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi-granular self-supervised learning frame- work. arXiv preprint arXiv:2203.14415, 2022. [22] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tok- enizer. arXiv preprint arXiv:2111.07832, 2021. [23] Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldas- sarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Se- ungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. [24] Sanat Ramesh, Vinkle Srivastav, Deepak Alapatt, Tong Yu, Aditya Mu- rali, Luca Sestini, Chinedu Innocent Nwoye, Idris Hamoud, Saurav Sharma, Antoine Fleurentin, et al. Dissecting self-supervised learn- ing methods for surgical computer vision. Medical Image Analysis, 88:102844, 2023. [25] Dominik Bati ́ c, Felix Holm, Ege Özsoy, Tobias Czempiel, and Nassir Navab. Endovit: pretraining vision transformers on a large collection of endoscopic images. International Journal of Computer Assisted Radi- ology and Surgery, 19(6):1085–1091, 2024. [26] Ronald de Jong, HJ Carolus, HA Franciscus, Romy C van Jaarsveld, Richard van Hillegersberg, PW Josien, Peter HN de With, Yasmina al Khalil, Fons van Der Sommen, et al. Scaling up self-supervised learn- ing for improved surgical foundation models. Medical Image Analysis, page 103873, 2025. [27] Martijn R Jong, Tim GW Boers, Kiki N Fockens, Jelmer B Jukema, Car- olus HJ Kusters, Tim JM Jaspers, RAH van Eijck van Heslinga, Floor C Slooter, Maarten R Struyvenberg, Raf Bisschops, et al. Gastronet-5m: A multicenter dataset for developing foundation models in gastrointesti- nal endoscopy. Gastroenterology, 2025. [28] Chengan Che, Chao Wang, Tom Vercauteren, Sophia Tsoka, and Luis C Garcia-Peraza-Herrera. Lemon: A large endoscopic monocular dataset and foundation model for perception in surgical settings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 42659–42669, 2026. [29] Ronald LPD de Jong, Yiping Li, Tim JM Jaspers, Romy C van Jaarsveld, Gino M Kuiper, Franco Badaloni, Richard van Hillegersberg, Jelle P Ruurda, Fons van der Sommen, Josien PW Pluim, et al. Towards effective surgical representation learning with dino models. In Medical Imaging with Deep Learning, 2026. [30] Kun Yuan, Vinkle Srivastav, Tong Yu, Joel L Lavanchy, Jacques Marescaux, Pietro Mascagni, Nassir Navab, and Nicolas Padoy. Learn- ing multi-modal representations by watching hundreds of surgical video lectures. Medical Image Analysis, 105:103644, 2025. [31] Florian Stilz, Vinkle Srivastav, Nassir Navab, and Nicolas Padoy. Clipper: Contextual video-language pretraining on long-form intra- operative surgical procedures for event recognition.arXiv preprint arXiv:2603.24539, 2026. [32] Steven M Strasberg and Michael L Brunt. Rationale and use of the critical view of safety in laparoscopic cholecystectomy. Journal of the American College of Surgeons, 211(1):132–138, 2010. [33] John R Heard, Atharva Deo, Umar Ghaffar, Runzhuo Ma, Cherine Yang, Ujjwal Pasupulety, Jasmine Lin, Carter M Prentice, Hyung L Kim, Melissa Assel, et al. Ai model predicts patient outcomes from surgical gestures and provides insights into explainability. npj Digital Surgery, 1(1):4, 2026. [34] Aditya Murali, Deepak Alapatt, Pietro Mascagni, Armine Vardazaryan, Alain Garcia, Nariaki Okamoto, Didier Mutter, and Nicolas Padoy. La- tent graph representations for critical view of safety assessment. IEEE Transactions on Medical Imaging, 43(3):1247–1258, 2023. [35] Xiaoyang Zou, Derong Yu, and Guoyan Zheng. Capturing action triplet correlations for accurate surgical activity recognition. Computerized Medical Imaging and Graphics, page 102604, 2025. [36] Samuel Schmidgall, Ji Woong Kim, Jeffrey Jopling, and Axel Krieger. General surgery vision transformer: A video pre-trained foundation model for general surgery. arXiv preprint arXiv:2403.05949, 2024. [37] Pablo Azagra, Carlos Sostres, Ángel Ferrández, Luis Riazuelo, Clara Tomasini, O León Barbed, Javier Morlana, David Recasens, Victor M Batlle, Juan J Gómez-Rodríguez, et al. Endomapper dataset of complete calibrated endoscopy procedures. Scientific Data, 10(1):671, 2023. [38] Matthias Carstens, Franziska M Rinner, Sebastian Bodenstedt, Alexan- der C Jenke, Jürgen Weitz, Marius Distler, Stefanie Speidel, and Fiona R Kolbinger. The dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science. Scientific Data, 10(1):3, 2023. [39] Andreas Leibetseder, Sabrina Kletz, Klaus Schoeffmann, Simon Keck- stein, and Jörg Keckstein. GLENDA: gynecologic laparoscopy en- dometriosis dataset. In MultiMedia Modeling - 26th International Con- ference, M 2020, Daejeon, South Korea, January 5-8, 2020, Pro- ceedings, Part I, volume 11962 of Lecture Notes in Computer Science, pages 439–450. Springer, 2020. [40] Martin Wagner, Beat-Peter Müller-Stich, Anna Kisilenko, Duc Tran, Patrick Heger, Lars Mündermann, David M Lubotsky, Benjamin Müller, Tornike Davitashvili, Manuela Capek, et al. Comparative validation of machine learning algorithms for surgical workflow and skill analy- sis with the heichole benchmark. Medical image analysis, 86:102770, 2023. [41] Hanna Borgli, Vajira Thambawita, Pia H Smedsrud, Steven Hicks, Debesh Jha, Sigrun L Eskeland, Kristin Ranheim Randel, Konstantin Pogorelov, Mathias Lux, Duc Tien Dang Nguyen, et al. Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific data, 7(1):283, 2020. [42] Pia H Smedsrud, Vajira Thambawita, Steven A Hicks, Henrik Gjes- tang, Oda Olsen Nedrejord, Espen Næs, Hanna Borgli, Debesh Jha, Tor Jan Derek Berstad, Sigrun L Eskeland, et al. Kvasir-capsule, a video capsule endoscopy dataset. Scientific Data, 8(1):142, 2021. [43] Andreas Leibetseder, Stefan Petscharnig, Manfred Jürgen Primus, Sab- rina Kletz, Bernd Münzer, Klaus Schoeffmann, and Jörg Keckstein. Lapgyn4: a dataset for 4 automatic content analysis problems in the do- 14 main of laparoscopic gynecology. In Proceedings of the 9th ACM Multi- media Systems Conference, MMSys 2018, Amsterdam, The Netherlands, June 12-15, 2018, pages 357–362. ACM, 2018. [44] Zhao Wang, Chang Liu, Shaoting Zhang, and Qi Dou. Foundation model for endoscopy video analysis via large-scale self-supervised pre- train. In International conference on medical image computing and computer-assisted intervention, pages 101–111. Springer, 2023. [45] Walaa Ghamrawi, Nathan Curtis, Jialang Xu, Matt Boal, Evangelos Ma- zomenos, Eddie Edwards, Danail Stoyanov, and Nader Francis. A rectal cancer surgery dataset: Use of artificial intelligence to aid automation of error identification. Scientific Data, 12(1):180, 2025. [46] Tobias Ross, Annika Reinke, Peter M Full, Martin Wagner, Hannes Kenngott, Martin Apitz, Hellena Hempe, Diana Mindroc Filimon, Patrick Scholz, Thuy Nuong Tran, et al. Robust medical instrument segmentation challenge 2019. arXiv preprint arXiv:2003.10299, 2020. [47] Chengan Che, Chao Wang, Tom Vercauteren, Sophia Tsoka, and Luis C Garcia-Peraza-Herrera. Surg-3m: A dataset and foundation model for perception in surgical settings. arXiv e-prints, pages arXiv–2503, 2025. [48] Dimitrios Psychogyios, Emanuele Colleoni, Beatrice Van Amsterdam, Chih-Yang Li, Shu-Yu Huang, Yuchong Li, Fucang Jia, Baosheng Zou, Guotai Wang, Yang Liu, et al. Sar-rarp50: Segmentation of surgical instrumentation and action recognition on robot-assisted radical prosta- tectomy challenge. arXiv preprint arXiv:2401.00496, 2023. [49] Aneeq Zia, Max Berniker, Rogerio Nespolo, Conor Perreault, Ziheng Wang, Benjamin Mueller, Ryan Schmidt, Kiran Bhattacharyya, Xi Liu, and Anthony Jarc. Surgical visual understanding (surgvu) dataset. arXiv preprint arXiv:2501.09209, 2025. [50] Yiting Ma, Xuejin Chen, Kai Cheng, Yang Li, and Bin Sun.Ld- polypvideo benchmark: a large-scale colonoscopy video dataset of di- verse polyps. In International conference on medical image computing and computer-assisted intervention, pages 387–396. Springer, 2021. [51] Vivek Singh Bawa, Gurkirt Singh, Francis KapingA, Inna Skarga- Bandurova, Elettra Oleari, Alice Leporini, Carmela Landolfo, Pengfei Zhao, Xi Xiang, Gongning Luo, et al. The saras endoscopic surgeon action detection (esad) dataset: Challenges and methods. arXiv preprint arXiv:2104.03178, 2021. [52] Deepak Alapatt, Jennifer Eckhoff, Zhiliang Lyu, Yutong Ban, Jean- Paul Mazellier, Sarah Choksi, Kunyi Yang, Po-Hsing Chiang, Noemi Zorzetti, Samuele Cannas, et al. The sages critical view of safety chal- lenge: A global benchmark for ai-assisted surgical quality assessment. arXiv preprint arXiv:2509.17100, 2025. [53] Sahar Nasirihaghighi, Negin Ghamsarian, Leonie Peschek, Matteo Mu- nari, Heinrich Husslein, Raphael Sznitman, and Klaus Schoeffmann. Gynsurg: A comprehensive gynecology laparoscopic surgery dataset. In Proceedings of the 33rd ACM International Conference on Multime- dia, pages 13141–13147, 2025. [54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [55] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. [56] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yun- feng Liu. Roformer: Enhanced transformer with rotary position embed- ding. arXiv preprint arXiv:2104.09864, 2021. [57] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021. [58] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou.Spreading vectors for similarity search.arXiv preprint arXiv:1806.03198, 2018. [59] Diandian Guo, Weixin Si, Zhixi Li, Jialun Pei, and Pheng-Ann Heng. Surgical workflow recognition and blocking effectiveness detection in laparoscopic liver resection with pringle maneuver. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 3220– 3228, 2025.