Paper deep dive
A Hierarchy-Aware Video-Language Model Evaluation and Hyperbolic Baseline for Surgery
Ana Manzano Rodríguez, Pascal Mettes, Marlies P. Schijven, Cees G. M. Snoek
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/24/2026, 4:45:08 AM
Summary
This paper introduces SurgHiBench, a hierarchy-aware evaluation suite for surgical video-language models, and HyperSurg, a hyperbolic baseline model. The authors argue that current models use flat metrics that ignore the hierarchical structure of surgical procedures (phases and steps). SurgHiBench evaluates models on recognition, cross-level consistency, and error severity across four datasets. HyperSurg utilizes hyperbolic geometry (Lorentz hyperboloid) to enforce phase-step containment via entailment cones, demonstrating that hierarchical geometry reduces error severity and improves step recognition compared to Euclidean baselines like CLIP and SurgCLIP.
Entities (12)
Relation Signals (11)
SurgHiBench → evaluates → CLIP
confidence 95% · We evaluate a general-purpose CLIP model
SurgHiBench → evaluates → SurgCLIP
confidence 95% · We evaluate... a Euclidean surgical model
SurgHiBench → evaluates → HyperSurg
confidence 95% · We evaluate a general-purpose CLIP model, a Euclidean surgical model, and... HyperSurg... across four (existing) datasets
HyperSurg → usesgeometry → Hyperbolic Geometry
confidence 95% · HyperSurg, a new hyperbolic model that enforces phase-step containment via entailment cones
HyperSurg → improves → Step Recognition
confidence 90% · Step recognition benefits most from the hierarchical inductive bias (+9.3 pp on average)
HyperSurg → operateson → Lorentz Hyperboloid
confidence 90% · HYPERSURG is our hierarchy-aware model operating on a Lorentz hyperboloid
HyperSurg → reduces → Cross-Level Contradictions
confidence 90% · cross-level contradictions decrease by up to 13 pp
SurgHiBench → usesdatasets → StrasbyPass70
confidence 90% · evaluated across four (existing) video datasets... StrasbyPass70
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Surgical procedures follow a phase-to-step hierarchy, yet the video-language models used to recognize them are evaluated with flat per-level metrics that ignore cross-level coherence and error structure. In this paper we make two contributions to address this problem, (i) we introduce SurgHiBench, the first hierarchy-aware evaluation suite for surgical video understanding, with three tasks measuring recognition, consistency, and severity across granularity levels. We evaluate a general-purpose CLIP model, a Euclidean surgical model, and, as second contribution: (ii) HyperSurg, a new hyperbolic model that enforces phase-step containment via entailment cones, across four (existing) datasets spanning three procedure types. The suite reveals that two models with the same accuracy can produce predictions of very different error severity, ranging from sibling confusions within the correct phase to unrelated cross-phase predictions. Hyperbolic geometry shifts predictions toward the correct procedural neighborhood, and these gains scale with the tree-likeness of each dataset's annotation hierarchy, providing a principled indicator when hierarchy-aware geometry helps.
Tags
Links
- Source: https://arxiv.org/abs/2609.27139v1
- Canonical: https://arxiv.org/abs/2609.27139v1
Trouble viewing inline? Open PDF directly →
Full Text
63,112 characters extracted from source content.
Expand or collapse full text
A Hierarchy-Aware Video-Language Model Evaluation and Hyperbolic Baseline for Surgery Ana Manzano Rodriguez 1,2,3,4 Pascal Mettes 1,2 Marlies P. Schijven 1,3,4,5 Cees G. M. Snoek 1,2 1 Data Science Center HAVA-Lab, University of Amsterdam 2 Video & Image Sense Lab, Informatics Institute, University of Amsterdam 3 Amsterdam UMC Location University of Amsterdam, Surgery 4 Amsterdam Public Health, Digital Health 5 Amsterdam Gastroenterology and Metabolism Abstract Surgical procedures follow a phase-to-step hierarchy, yet the video-language models used to recognize them are eval- uated with flat per-level metrics that ignore cross-level co- herence and error structure. In this paper we make two contributions to address this problem, (i) we introduce SURGHIBENCH, the first hierarchy-aware evaluation suite for surgical video understanding, with three tasks measur- ing recognition, consistency, and severity across granular- ity levels. We evaluate a general-purpose CLIP model, a Euclidean surgical model, and, as second contribution: (i) HYPERSURG, a new hyperbolic model that enforces phase- step containment via entailment cones, across four (exist- ing) datasets spanning three procedure types. The suite re- veals that two models with the same accuracy can produce predictions of very different error severity, ranging from sibling confusions within the correct phase to unrelated cross-phase predictions. Hyperbolic geometry shifts pre- dictions toward the correct procedural neighborhood, and these gains scale with the tree-likeness of each dataset’s an- notation hierarchy, providing a principled indicator when hierarchy-aware geometry helps. 1. Introduction Developing assistance systems for the operating room re- quires understanding what is happening during a proce- dure [13]. Surgical workflow analysis provides this under- standing by recognizing surgical activities at different lev- els of temporal granularity, which is inherently hierarchical: a gallbladder removal (cholecystectomy) comprises phases (e.g. Calot triangle dissection), each containing steps (e.g. dissecting the cystic duct), which in turn consist of fine- grained actions (e.g. grasping, cutting) (Figure 1). Because steps belong to broader procedural contexts, not all recog- nition errors are equal [3, 24]. Confusing two steps within the same phase means the prediction is at least in the right Gastric Bypass 010203040506070809101112 04 Gastrojejunal anastomosis07 Petersen space closure Jejunum opening Gastrojejunal stapling Gastrojejunal defect closure Petersen space exposure Petersen space closure 1:06:441:10:001:19:342:35:012:39:24 00:002:57:24 PROCEDURE PHASES STEPS VIDEO Figure 1. Surgical procedures are hierarchically structured. A gastric bypass decomposes into 12 phases containing 46 steps. For surgical AI, current video-language models embed all levels in a flat space and evaluate each independently, ignoring underlying structure. To address this problem, we introduce (i) a hierarchy- aware evaluation suite and (i) a hyperbolic baseline model. part of the procedure, whereas predicting a step from an entirely different phase places it in an unrelated procedural context. Effective surgical support therefore requires rec- ognizing both the broad phase and the specific step, and ensuring that these predictions are consistent. Despite this clinical need, current surgical video- language models do not explicitly capture hierarchical re- lationships. State-of-the-art models such as SurgVLP [33] learn from narrated surgical video without hierarchical structure, and while the SurgLaVi model [18] learns from hierarchically annotated data, all levels still coexist in the model’s flat embedding space without explicit hierarchical distinction. Evaluation has not addressed this gap either, as all current models are measured with accuracy or F1 at each level independently, without assessing whether predic- tions across levels are consistent or whether errors respect the surgical procedure structure. In practice, two models with the same score may produce errors of very different 1 arXiv:2609.27139v1 [cs.CV] 22 Sep 2026 severity, with standard metrics not being able to distinguish between them. This disconnect between what we evaluate and what clinical deployment demands has been identified as a key barrier for AI adoption in surgery [23]. To address the lack of hierarchy-awareness in surgical AI, this paper makes two core contributions: 1. Contribution I: SURGHIBENCH (Sec. 3), the first hierarchy-aware evaluation suite for surgical video- language models, with three tasks measuring per-level and joint recognition, cross-level consistency, and error severity, evaluated across four (existing) video datasets spanning three procedure types. SURGHIBENCH comes with three baseline models: CLIP, SurgCLIP ∗ (β) , and 2. Contribution I: HYPERSURG (Sec. 4), a new hierarchy-aware surgical video-language baseline model that enforces entailment cone constraints over a hyper- bolic embedding space, where the exponentially grow- ing volume naturally fits the tree-like structure of sur- gical taxonomies, paired with a hierarchy-aware triplet batch sampler. SURGHIBENCH reveals (Sec. 5) that models with compara- ble accuracy differ substantially in how coherent their pre- dictions are. With HYPERSURG, predictions shift from un- related phases toward the correct procedural neighborhood, and cross-level contradictions decrease by up to 13 p. Step recognition benefits most from the hierarchical inductive bias (+9.3 p on average), as the exponentially growing space provides room for fine-grained distinctions that flat geometry collapses. Before detailing our contributions we first discuss related work. 2. Related Work Surgical Vision-Language Models.While vision- language models such as CLIP [20] have transformed gen- eral visual recognition for quite a while, their adaptation to surgery is still in its early stages. SurgVLP [33] was the first to apply contrastive pretraining to narrated sur- gical video, using a ResNet-50 vision encoder aligned with a BioClinicalBERT text encoder. SurgLaVi [18] con- tributed a large-scale dataset of hierarchically annotated sur- gical clips spanning actions, steps, and phases, together with SurgCLIP, a dual-encoder model trained on this data. Subsequent work has aimed to make these representations hierarchy-aware: HecVL [31] constructs three separate em- bedding levels (clip, phase, video) via progressive aggrega- tion of fine-grained representations into coarser ones, while PeskaVLP [32] enriches textual descriptions with LLM- generated hierarchical knowledge and procedural align- ment. These methods rely on increasingly complex archi- tectures (separate embedding spaces, progressive aggrega- tion, LLM-augmented knowledge) yet all remain in Eu- clidean space. HYPERSURG takes an alternative approach: rather than adding architectural complexity, we change the underlying geometry, unifying all levels in a single hyper- bolic space where the structure itself encodes hierarchical granularity. Hierarchical Evaluation. Not all misclassifications are equally costly: confusing adjacent categories in a seman- tic hierarchy is less severe than confusing distant ones, and evaluation metrics should reflect this. Bertinetto et al. [3] showed that classifiers trained to minimize tree distance to the ground truth sacrifice minimal flat accuracy while pro- ducing substantially milder errors. This principle has found clinical uptake. In dermatology, hierarchy-aware prototypes penalize confusing melanoma with a benign nevus (a distant branch) more heavily than confusing it with basal cell carci- noma (a sibling malignancy), reducing dangerous misdiag- noses [30]. In radiology, Srivastava and Mishra [24] showed that models with comparable recognition ability on chest X-rays can differ substantially in error severity. Moreover, existing hierarchy-aware training losses and post-inference reranking methods, designed for taxonomies where only leaf nodes are labels, actually increase severity on medical hierarchies where every node is a valid diagnosis. This sug- gests that reducing error severity in multi-level clinical hier- archies requires representations that intrinsically respect the hierarchy. For action recognition, Long et al. [12] intro- duced sibling-mAP and cousin-mAP, which relax correct- ness criteria along a category tree. However, all of these frameworks target natural-image classification or general action recognition; none has been applied to surgical video, where the procedural hierarchy provides an equally natu- ral tree structure for grading error severity. Within surgi- cal workflow analysis, temporal hierarchy has been studied through phase recognition [4], step recognition [11], and instrument-action detection [22], yet these tasks are invari- ably evaluated independently with flat metrics. Our cross- level consistency evaluation is, to our knowledge, the first to explicitly measure whether a model’s predictions across granularity levels are mutually consistent. Hyperbolic Representations. Hyperbolic spaces are non- Euclidean geometries of constant negative curvature whose volume grows exponentially with radius, making them a natural fit for encoding hierarchies [14]. Poincar ́ e embed- dings [15] showed that hyperbolic spaces can embed tax- onomies with orders-of-magnitude fewer dimensions than Euclidean ones, and Ganea et al. [7] introduced a geometric notion of containment in hyperbolic space, where broader concepts enclose more specific ones within cone-shaped re- gions. Building on these ideas, MERU [5] was the first to apply hyperbolic geometry to vision-language pretraining, learning a two-level cross-modal hierarchy (text ⊃ image) on natural images. HyCoCLIP [16] extended entailment to compositional sub-parts, and PHyCLIP [29] proposed prod- ucts of hyperbolic spaces for joint hierarchy and compo- sitionality. In medical imaging, HYDEN [19] applied hy- 2 perbolic representations to radiology reports. We propose HYPERSURG, a hyperbolic surgical video-language model that enforces the procedural phase-step hierarchy through entailment cones. Concurrent to our work, HyperVLP by Hu et al. [9] also represents surgical video and language in a Lorentz manifold and use hyperbolic entailment to encode hierarchical relations. However, both HyperVLP and prior work evaluate with standard flat metrics, leaving open the hierarchy-aware questions our benchmark addresses. 3. Hierarchy-Aware Evaluation for Surgery We introduce SURGHIBENCH, the first hierarchy-aware evaluation suite for surgical video-language models. SURGHIBENCH combines three complementary tasks that together provide a complete picture of hierarchical aware- ness: whether predictions at each granularity level are cor- rect, whether they are mutually consistent along the levels, and whether predictions land on related or unrelated surgi- cal concepts (Sec. 3.1). These tasks are evaluated across four (existing) surgical video datasets that span a spectrum of tree-likeness in their annotation structure (Sec. 3.2) and three models ranging from general-purpose to hierarchy- aware (Sec. 3.3). SURGHIBENCH is model-agnostic and can evaluate any surgical video-language model that pro- duces phase and step predictions. Each task is defined be- low together with its metrics, all computed per-video and averaged. All data, training details and source code will be made available on our benchmark website. 3.1. Hierarchy-Aware Tasks Task 1: Single- and Multi-Level Recognition. How well does a model recognize phases and steps, individually and jointly? We first evaluate how well a model identifies phases and steps independently, and report video-averaged accu- racy and macro-averaged F1 at each level. While accuracy measures overall correctness, macro-F1 better captures per- formance on rare but clinically important classes by weight- ing all classes equally, which is important given the severe class imbalance typical of surgical datasets. A model may achieve high phase accuracy while performing poorly on steps, or vice versa; to detect such imbalances we also re- port a multi-level score, defined as the geometric mean of phase and step accuracy: Multi-level = p Acc phase × Acc step ,(1) which remains low unless both levels are strong (e.g. 30% phase with 10% step yields 17.3%, while 20% on both yields 20%). These per-level metrics do not show, how- ever, whether predictions are coherent across levels, which the remaining tasks address. Task 2: Cross-Level Consistency. Are the model’s phase and step predictions mutually consistent? A system that predicts the Preparation phase alongside a Clipping step produces contradictory outputs that could mislead clinical decision support [28]. We measure joint correctness across both levels with three metrics. The Both-wrong and Both- right rates measure how often both levels are simultane- ously wrong or right, revealing whether errors are isolated to one level or propagate across both. Step|Phase is step ac- curacy conditioned on the phase being correct, which iso- lates step discrimination from phase errors. These metrics are inspired by hierarchical consistency measures in image classification [10, 17], adapted to surgical video, where only joint phase-step correctness has been reported [21] without analyzing error patterns across levels. Task 3: Hierarchy-Aware Error Severity. Does the model confuse related or unrelated surgical concepts? We capture this with two complementary measures. Sibling step accu- racy [12] measures how often the predicted step falls within the same phase as the ground-truth step, even if the exact step is wrong. Cousin step accuracy measures the opposite: how often the prediction lands in an entirely different phase. A model with high sibling and low cousin accuracy pre- dicts within the correct procedural context, indicating that it captures the coarse structure of the procedure. We also introduce semantic step accuracy, which measures whether the model recognizes the type of surgical action being per- formed regardless of anatomical site (e.g. Jejunojejunal De- fect Closure vs. Gastrojejunal Defect Closure), identifying what is being done even when it confuses where. We man- ually group steps that share the same surgical action across different anatomical sites; full groupings are in the supple- mental (Sec. A.1). 3.2. Surgical Video Datasets Evaluation Datasets. Our evaluation focuses on phase and step recognition, the two granularity levels that enable the cross-level analysis required by our three tasks. We evaluate on four video datasets (Tab. 1): MIPO [8] (50 radius frac- ture fixation videos), StrasbyPass70 and BernBypass70 [11] (70 gastric bypass videos each, recorded at two institutions), and GraSP [1] (13 robot-assisted prostatectomy videos). All four provide phase and step annotations, but differ in how steps relate to phases. In some datasets, each step belongs to exactly one phase, forming a clean tree (MIPO, 100%). In others, steps recur across multiple phases, forming a di- rected acyclic graph (DAG) where the parent–child rela- tionship is ambiguous (GraSP, 33%). This difference stems from how each dataset defines its steps. GraSP’s step vocab- ulary mixes procedural substages (Seminal vesicle dissec- tion, Prevessical dissection) with reusable instrument ac- tions (Suction, Cut suture or tissue, Tie suture). The latter are phase-agnostic—Suction occurs whenever there is blood or fluid (10 of 11 phases), Cut suture or tissue whenever tis- sue must be separated (10 of 11). By contrast, the bypass 3 Table 1. SURGHIBENCH evaluates video-language models on hierarchy-awareness, covering three tasks and four surgical video datasets. Datasets are ordered by tree-likeness, defined as the percentage of steps that belong to exactly one phase (100% = pure tree, lower = more DAG-like). DatasetSourceProcedureVideos Phases † Steps † PromptsTree-likeness MIPOGraeff et al. [8]Radius fracture5058Ours100% StrasbyPass70Lavanchy et al. [11]Gastric bypass701146SurgLaVi [18]74% BernBypass70Lavanchy et al. [11]Gastric bypass701146SurgLaVi [18]54% GraSPAyobi et al. [1]Prostatectomy131121SurgLaVi [18]33% † Classes used for evaluation; see supplementary Sec. A.2 for details. datasets (54–74%) mostly define steps as procedural sub- stages (Gastrojejunal Defect Closure, Retrogastric Dissec- tion, Vertical Stapling) that describe the surgical goal being accomplished, and are therefore largely phase-exclusive by construction. MIPO follows the same goal-level convention (Distal fixation, Proximal fixation). This spectrum allows our evaluation to also test whether the benefits of hierarchy- aware representations are tied to the structural properties of each dataset’s annotation hierarchy. For zero-shot evaluation, we use the text descriptions provided by Perez et al. [18] for GraSP and the bypass datasets; for MIPO, we wrote new prompts following the same narration style (see supplemental Sec. A.2). Pretraining Dataset. The surgical video-language mod- els evaluated in this work are pretrained on a subset of the SurgLaVi dataset [18]. The full dataset contains 5,317 narrated surgical videos sourced from diverse surgical do- mains, across specialties such as general surgery, colorec- tal, gynecology, thoracic, and urology. Of these, 2,464 are part of the open-source subset, called SurgLaVi-β. We use SurgLaVi-β without modification. Each video is divided into temporal segments at three hierarchy levels, with captions generated from the narrator’s descriptions, yielding ∼113K caption–clip pairs. These levels differ in temporal granularity and scope: Actions (Level 1) capture brief, action-focused surgical moments (e.g. “the surgeon uses a stapler to make the first transverse staple line”). Steps (Level 2) aggregate multiple actions into intermediate procedural substages with more context (e.g. “the surgeon dissects into the lesser sac; the first trans- verse stapler firing is performed”). Phases (Level 3) span longer temporal segments with rich procedural detail (e.g. “the surgeon begins the gastric bypass by placing trocars at the angle of His, dissecting along the lesser curvature with a perigastric technique”). This hierarchical structure mirrors the procedural hierarchy of surgery itself, making hierarchy-aware training possible and allowing the model to learn representations that respect the natural granularity of surgical procedures. 3.3. Baseline Models Our evaluation suite comes with three models. Two ex- isting models and one model that we propose as part of this paper. (i) CLIP ViT-B/16 [20], the original OpenAI CLIP model with no surgical pretraining.Since CLIP has no temporal modeling, each frame is encoded inde- pendently and similarities are averaged over the tempo- ral window. It serves as a lower bound on what general- purpose vision-language representations achieve on surgi- cal tasks. (i) SurgCLIP ∗ (β) is our reimplementation of SurgCLIP (β) by Perez et al. [18] with identical architecture (TimeSformer [2] + BERT-base [6]), training procedure, and Euclidean cosine similarity. Since SurgLaVi released model and evaluation code but not the training pipeline, we reimplemented training from the details in their pa- per; the reproduction closely matches their reported results across all benchmarks (see Tab. 8 in supplemental), con- firming a faithful reimplementation. We follow their zero- shot evaluation protocol and test splits, adapting the scripts for Lorentz distance in HYPERSURG. (i) HYPERSURG is our hierarchy-aware model operating on a Lorentz hyper- boloid, described next, using the same architecture, training data, and number of parameters as SurgCLIP ∗ (β) , ensuring a controlled experiment where any difference with our model is attributable to the embedding geometry alone. 4. Hyperbolic Baseline Model for Surgery Having defined how to evaluate hierarchy-awareness, we now propose a baseline model that encodes surgical hier- archy by operating in hyperbolic space, where the tempo- ral granularity of procedures is naturally represented. We cover the geometric background (Sec. 4.1), the architecture (Sec. 4.2), and the training objectives (Sec. 4.3). 4.1. Background: Lorentz Hyperbolic Space Standard contrastive models, like CLIP and SurgCLIP, measure similarity via cosine distance, which depends only on the angle between embeddings and discards their norms. This leaves a single degree of freedom—angular direction—to simultaneously separate semantically distinct videos and distinguish hierarchy levels within the same 4 (a) Hyperbolic space (HyperSurg) (b) Euclidean space (SurgCLIP * (β) ) 0.500.590.690.790.8810.4010.8811.3511.8312.30 norm by level —Lorentznorm by level —Euclidean ‖z‖ ActionStepPhase Figure 2. Hyperbolic geometry organizes the surgical hier- archy by embedding norm; Euclidean cosine space collapses it. (a) In HYPERSURG (green), embedding norms increase from fine-grained actions to coarse phases, reflecting the hierarchical structure. Gray wedges are entailment cones (Equation (7)). (b) In SurgCLIP ∗ (β) (red), L2 normalization for cosine similarity discards any norm signal, preventing cosine-based models from exploiting the hierarchy. Bottom: per-level norm distributions. Note distin- guishability of hierarchical levels in hyperbolic space. video, competing demands that one signal cannot satisfy. The Lorentz hyperboloid decouples these two roles: the dis- tance from the origin encodes generality while the angular direction independently encodes semantic content, provid- ing the missing degree of freedom. Formally, it is defined as the upper part of a two-sheeted hyperboloid in (n+1)- dimensional Minkowski space with curvature−1/c: L n c =x∈R n+1 :⟨x,x⟩ L =−1/c, x 0 > 0,(2) where⟨x,y⟩ L = −x 0 y 0 + P n i=1 x i y i is the Lorentz inner product. The geodesic distance between two points is: d L (x,y) = 1 √ c acosh −c⟨x,y⟩ L .(3) A key property is that the volume of a ball of radius r grows with ratio e (n−1)r/ √ c , whereas in Euclidean space the same volume grows only polynomially (∝ r n ). This ex- ponential capacity makes the radial axis a natural encoding of hierarchy, separating granularity levels that flat geome- try collapses. Figure 2 illustrates this property: in HYPER- SURG, embedding norms increase monotonically from fine- grained actions to coarse phases, whereas L2 normalization in cosine-based models collapses this separation entirely. 4.2. Architecture HYPERSURG follows a dual-encoder paradigm where a video encoder and a text encoder are jointly trained to align surgical clips with hierarchical captions in a shared hy- perbolic embedding space (Figure 3). To isolate the ef- fect of geometry, HYPERSURG uses the same encoders as SurgCLIP ∗ (β) : a TimeSformer [2] (ViT-B/16, divided space- time attention, 12 layers) encodes T=16 uniformly sam- pled frames into a 768-dimensional video representation via mean-pooled spatial patch tokens, and a BERT-base [6] text encoder maps captions to 768-dimensional text representa- tions via the [CLS] token. The two architectures diverge after the encoders. Both project the 768-dimensional representations toR 256 via learned linear layers, but SurgCLIP ∗ (β) then applies L2- normalization to measure cosine similarity, while HYPER- SURG scales them by learnable per-modality log-space fac- tors α v ,α t and maps them onto the Lorentz hyperboloid L 256 c via the exponential map at the origin o = (1/ √ c, 0): exp c o (v) = cosh √ c∥v∥ L o + sinh √ c∥v∥ L √ c∥v∥ L v, (4) where v ∈ T o L n c is the tangent vector at o. Only the space components are stored; the time component is recov- ered via the hyperboloid constraint x 0 = p 1/c +∥x space ∥ 2 when needed for distance computations. The curvature c is a learnable parameter stored in log-space and clamped to [0.1, 10]. 4.3. Training Objectives Lorentz Contrastive Loss. We measure similarity via neg- ative Lorentz distance scaled by a learnable temperature τ : s(v i ,t j ) =−d L (v i ,t j )/τ.(5) The loss follows symmetric InfoNCE over all GPU- gathered pairs: L contr = 1 2 L v2t +L t2v , L v2t =− 1 B X i log e s(v i ,t i ) P j e s(v i ,t j ) .(6) Hierarchy Entailment Loss. The three-level hierarchy described in Sec. 3.2 forms a tree. Following Ganea et al. [7], we enforce this containment via entailment cones. For a parent text embedding p, its entailment cone has half- aperture: ψ(p) = asin 2r min ∥p∥ R √ c ,(7) where r min controls the minimum cone width. A child q (e.g. a step or phase caption from the same video) must lie within the parent’s cone. Let φ(p,q) denote the exterior angle at p in the hyperbolic triangle O–p–q (where O is the origin). The entailment loss penalizes violations with a slack factor η: L entail = 1 |P| X (p,q)∈P max 0, φ(p,q)− η ψ(p) ,(8) 5 “Laparoscopic gastric bypass procedure.” STEP CAPTION “The surgeon begins by mobilizing the splenic flexure of the colon, preparing for a total pancreatectomy in a patient positioned in full left lateral decubitus.” Text Encoder BERT-base Textual Projection Exponential. Mapping Exponential Mapping Visual Projection Vision Encoder TimeSformer SURGICAL VIDEO CLIP (T = 16) Top-down view Action caption Step caption Phase caption Video clip Contrastive loss Entailment loss Losses contrastive entailment PHASE CAPTION “The surgeon mobilizes the splenic flexure of the colon, creating a small window posterior to the pancreas and splenic vessels, revealing the posterior wall of the stomach. Using a Warshaw-type technique, the splenic vessels are stapled while preserving the short gastric vessels. The pancreas is mobilized from lateral to medial, the splenic artery is isolated, clipped, and divided.” ACTION CAPTION “The surgeon mobilizes the splenic flexure of the colon, preparing for further steps in the pancreatectomy.” STEP CLIP PHASE CLIP ACTION CLIP Figure 3. Overview of the proposed hierarchy-aware HYPERSURG baseline model. Surgical video clips at three hierarchy levels (action, step, phase) are encoded by a TimeSformer (ViT-B/16), while the corresponding narration-derived captions are encoded by BERT- base. Both branches project into a shared 256-D Lorentz hyperboloidL 256 c via learned projections and the exponential map. The contrastive lossL contr aligns video–text pairs via Lorentz distance (Equation (5)); the entailment lossL entail enforces parent–child containment across hierarchy levels within the same cone (Equation (8)). where P is the set of parent–child pairs within each batch and η=1.2 provides geometric slack. Hierarchical Triplet Sampling.During training, each batch consists of video-caption pairs sampled from the dataset. Standard random sampling rarely produces within- video hierarchical pairs: with captions spread across thou- sands of videos, a random batch of a few hundred samples is unlikely to contain multiple hierarchy levels from the same video, leaving the entailment loss effectively without train- ing signal. We introduce a novel hybrid sampler that com- poses each batch of B samples from B−3k random sam- ples and k=3 forced triplets, each containing one caption per hierarchy level from the same video. This guarantees ∼9 entailment pairs per batch while maintaining contrastive diversity. Total Loss. L =L contr + λL entail ,(9) with λ=0.1, selected from a range of values (Tab. 10). An ablation study confirms that both the entailment cones and the hierarchy-aware triplet batch sampler contribute beyond Lorentz distance alone (Sec. B). Training details. We train for 50 epochs on 8× NVIDIA RTX A5000 GPUs with an effective batch size of 312, us- ing AdamW (lr=10 −4 , weight decay = 0.02) with OneCy- cleLR (5-epoch warmup, cosine annealing). Video frames are augmented with random resized crop (scale 0.5–1.0) and horizontal flip.Since no training split is provided with the dataset, we hold out 15% of the videos (369 out of 2,464) for monitoring training. Both HYPERSURG and SurgCLIP ∗ (β) are trained with the same procedure, data, and hyperparameters, including masking same-video captions from the contrastive denominator to prevent conflicting gra- dients from clips that share a source video. Full details are provided in supplemental Sec. A.4. 5. Results 5.1. Task 1: Single- and Multi-Level Recognition We present results for Task 1 in Table 2, where we com- pare recognition performance across granularity levels. The CLIP baseline, which has no surgical pretraining, provides a cautionary note about interpreting accuracy in isolation. On MIPO, CLIP achieves higher accuracy than the Euclidean surgical model at both levels (16.2% vs 13.6% phase, 17.0% vs 8.3% step), yet its F1 is lower in both cases (6.6 vs 7.9, 4.9 vs 5.6), indicating that CLIP collapses predictions into the most frequent classes, inflating accuracy while failing to distinguish rarer categories. The pattern holds at the ag- gregate level, where CLIP’s overall F1 remains near chance (4.1 phase, 1.8 step). Surgical pretraining in Euclidean space substantially im- proves phase recognition (23.0% accuracy, F1 16.4), but step accuracy lags behind (14.2%). These results indicate that coarse-grained surgical categories are already separa- 6 Table 2. Task 1: Single- and Multi-Level Recognition. Zero- shot video-averaged accuracy (%), bold = best per dataset. Over- all phase and step recognition both improve with hierarchy- awareness, though per-dataset performance varies. The multi-level score, which requires both levels to be strong, tends to benefit from hierarchy-awareness, with gains scaling with tree-likeness. Phase (↑)Step (↑) Multi-level (↑) AccuracyF1 AccuracyF1 MIPO CLIP16.26.617.04.916.6 SurgCLIP ∗ (β) 13.67.98.35.6 10.6 HYPERSURG20.9 16.217.6 8.9 19.2 StrasbyPass70 CLIP18.54.23.30.77.8 SurgCLIP ∗ (β) 29.5 22.717.27.4 22.5 HYPERSURG30.2 25.232.0 12.0 31.1 BernBypass70 CLIP20.94.21.80.3 6.1 SurgCLIP ∗ (β) 23.9 17.413.55.6 18.0 HYPERSURG17.4 12.529.7 8.5 22.7 GraSP CLIP4.31.56.41.25.2 SurgCLIP ∗ (β) 25.1 17.717.9 9.0 21.2 HYPERSURG26.6 17.014.77.819.8 Overall CLIP15.04.17.11.88.9 SurgCLIP ∗ (β) 23.0 16.414.26.9 18.1 HYPERSURG23.8 17.723.5 9.323.2 ble in flat Euclidean space, while steps require finer distinc- tions among classes that share a common parent phase, and this is where hierarchical structure in the embedding geom- etry makes the difference. With HYPERSURG, step accu- racy rises to 23.5% while phase accuracy remains compara- ble (23.8%), narrowing the gap between the two levels. The multi-level score, which requires both levels to be strong, confirms that the benefit comes from making phase and step recognition work together rather than improving either in isolation (overall 23.2% vs 18.1%), though per-dataset gains depend on the degree of tree-structure in each anno- tation hierarchy. The gains track this spectrum closely: the bypass datasets (54–74% tree-likeness) show strong multi- level improvements, and MIPO (100% tree-likeness) sees the largest. The exception is GraSP, where SurgCLIP ∗ (β) leads on step accuracy: its mixed-granularity step vocabu- lary means that 67% of steps recur across multiple phases (Sec. 3.2), breaking the parent–child containment that en- tailment cones enforce. The more tree-like the hierarchy, the larger the gains, exactly the expected behaviour for a geometric inductive bias designed to exploit tree structure. Worth noting, MIPO is an orthopaedic procedure outside the surgical domains covered by the pretraining data [18], Table 3. Task 2: Cross-Level Consistency (%). Bold = best per dataset. As models become more hierarchy-aware, catastrophic failures where both levels are wrong decrease, and a correct phase prediction more often leads to a correct step, showing that the two granularity levels become coupled rather than independent. Both Wrong (↓) Both Right (↑) Step| Phase (↑) MIPO CLIP82.215.494.3 SurgCLIP ∗ (β) 82.14.027.2 HYPERSURG70.99.444.3 StrasbyPass70 CLIP80.42.210.9 SurgCLIP ∗ (β) 57.94.515.2 HYPERSURG44.87.022.8 BernBypass70 CLIP78.10.83.5 SurgCLIP ∗ (β) 66.13.514.5 HYPERSURG57.24.324.2 GraSP CLIP89.90.720.9 SurgCLIP ∗ (β) 67.310.235.3 HYPERSURG67.89.130.9 Overall CLIP82.64.832.4 SurgCLIP ∗ (β) 68.35.623.0 HYPERSURG60.27.430.6 yet HYPERSURG achieves the best accuracy and F1 at both levels, suggesting that the hierarchical inductive bias gen- eralises to out-of-distribution procedures where domain- specific pretraining alone falls short. 5.2. Task 2: Cross-Level Consistency We present results for Task 2 in Table 3, where we eval- uate whether phase and step predictions are mutually co- herent. The Both Wrong rates are high across the board, which shows how challenging zero-shot recognition at two granularity levels is. CLIP fails on the vast majority of frames (82.6% Both Wrong), and appears to lead on Both Right and Step|Phase on MIPO, but this is the same class- collapse artefact from Task 1. With a phase F1 of only 6.6, Step|Phase is computed on very few frames and the high score is misleading. These metrics should therefore be read jointly with Task 1, as they can overstate coherence when overall accuracy is low. The consistent reduction in Both Wrong from CLIP (82.6%) to SurgCLIP ∗ (β) (68.3%) to HYPERSURG (60.2%) shows that surgical pretraining helps and encoding hier- archical structure aids further, with the largest drop on StrasbyPass70 (from 57.9% for SurgCLIP ∗ (β) to 44.8% for HYPERSURG). The exception is again GraSP, where SurgCLIP ∗ (β) leads for the same structural reasons discussed 7 Table 4. Task 3: Error severity (%). Bold = best per dataset. The hierarchy-aware model’s predictions more often fall within the correct phase and preserve the surgical action, indicating that its step predictions are more context-aware. Sibling (↑) Cousin (↓) Semantic Step (↑) MIPO CLIP20.679.420.4 SurgCLIP ∗ (β) 21.978.19.1 HYPERSURG27.972.124.2 StrasbyPass70 CLIP20.279.88.8 SurgCLIP ∗ (β) 43.456.630.9 HYPERSURG51.348.744.6 BernBypass70 CLIP21.378.79.9 SurgCLIP ∗ (β) 36.863.220.6 HYPERSURG46.353.740.9 GraSP CLIP14.086.010.7 SurgCLIP ∗ (β) 31.668.429.1 HYPERSURG29.270.824.4 Overall CLIP19.081.012.4 SurgCLIP ∗ (β) 33.466.622.4 HYPERSURG38.761.333.5 in Task 1. Step|Phase shows that when the phase is cor- rect, the hierarchy-aware model picks the right step more often (30.6% vs 23.0% overall), suggesting that hierarchy- awareness couples the two levels through the embedding geometry. 5.3. Task 3: Hierarchy-Aware Error Severity The previous tasks measured whether the model is correct; this task examines where predictions fall in the procedu- ral hierarchy and whether they preserve the surgical ac- tion. We present results in Table 4. CLIP shows the low- est sibling accuracy across all datasets (19.0% overall), in- dicating that its predictions fall in unrelated phases almost uniformly. Surgical pretraining improves this substantially (33.4% sibling accuracy), meaning a larger share of step predictions fall within the correct phase, and hierarchy- awareness pushes it further (38.7%). Semantic step accu- racy follows the same trend, rising from 12.4% (CLIP) to 22.4% (SurgCLIP ∗ (β) ) to 33.5% (HYPERSURG), meaning the model more often recognises the surgical action being performed even when it confuses the anatomical site. The top confusions illustrate what these numbers mean in prac- tice. On BernBypass70, HYPERSURG’s dominant confu- sion for Gastrojejunal Anastomosis Reinforcement is Gas- trojejunal Defect Closure, a sibling step within the same phase that involves suturing the same anatomical structure. BernBypass70 BBP02 · frame 4122 Omentum Division correct phase Exposure of the Omentum GT Omental Transection HyperSurg Disassembling different phase Drainage Insertion SurgCLIP*(β) sibling cousin GraSP CASE050 · frame 12815 Bladder Neck Reconstruction correct phase Pull Suture GT Pass Suture to the bladder neck HyperSurg Severing of the prostate from the urethra different phase Cut the tissue between the prostate and the urethra SurgCLIP*(β) sibling cousin Figure 4. Per-level metrics cannot distinguish between errors. On both frames, each model predicts the correct phase but the wrong step, so accuracy and F1 record the same outcome. The errors differ in where they land in the hierarchy: the orange pre- diction is a sibling of the ground truth, a different step within the correct phase, while the red one is a cousin, from an unrelated phase. SURGHIBENCH measures this distinction. By contrast, SurgCLIP ∗ (β) confuses it with Horizontal Sta- pling, a step from a different phase (Gastric Pouch Creation) at a different anatomical site with a different surgical goal. Figure 4 shows the same pattern on GraSP and BernBy- pass70: the hierarchy-aware prediction remains within the correct phase, the Euclidean one does not. Flat accuracy treats both confusions identically, but these results provide insights that only hierarchy-aware evaluation can report and that directly address the needs of the surgical domain, where models with similar accuracy can produce errors of very dif- ferent severity that should not be overlooked. We provide qualitative examples of successful and failed predictions for both SurgCLIP ∗ (β) and HYPERSURG, across all four datasets, in supplementary Figure 5. 6. Conclusion In this work, we posit that phases and steps in surgical video should not be treated as independent recognition tasks. We introduce a hierarchy-aware evaluation suite that measures cross-level consistency and error severity alongside per- level accuracy, showing across four datasets and three pro- cedure types that video-language models with comparable accuracy can result in very different error patterns. We propose HYPERSURG, a hierarchy-aware baseline that pro- duces more accurate and consistent cross-level predictions by aligning the embedding geometry with the hierarchical structure of procedures. The resulting gains scale with the tree-likeness of each dataset’s annotation hierarchy, provid- ing a principled indicator of when hierarchy-aware repre- sentations will help. As richer annotations become avail- 8 able, we expect both the evaluation framework and the ge- ometric approach to extend naturally to deeper hierarchies, providing the community with tools to build surgical work- flow systems that are not only accurate, but evaluated in ways that reflect what matters in the operating room. Acknowledgements This work was supported by the University of Amsterdam’s Data Science Centre, as part of the HAVA Lab. References [1] Nicolas Ayobi, Santiago Rodriguez, Alejandra Perez, Isabela Hernandez, Nicolas Aparicio, Eugenie Dessevres, Sebas- tian Pe ̃ na, Jessica Santander, Juan Ignacio Caicedo, Nicolas Fernandez, and Pablo Arbelaez. Pixel-wise recognition for holistic surgical scene understanding. Medical Image Anal- ysis, page 103726, 2025. 3, 4, 11 [2] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning (ICML), 2021. 4, 5 [3] Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A. Lord. Making better mis- takes: Leveraging class hierarchies with deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 12503–12512, 2020. 1, 2 [4] Tobias Czempiel, Magdalini Paschali, Matthias Keicher, Walter Simson, Hubertus Feussner, Seong Tae Kim, and Nassir Navab.TeCNO: Surgical phase recognition with multi-stage temporal convolutional networks.In Inter- national Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 343–352, 2020. 2 [5] Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Ramakrishna Vedantam. Hyperbolic image- text representations. In International Conference on Ma- chine Learning (ICML), pages 7694–7731, 2023. 2 [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional trans- formers for language understanding.In North American chapter of the association for computational linguistics: human language technologies (NAACL-HLT), pages 4171– 4186, 2019. 4, 5 [7] Octavian-Eugen Ganea, Gary B ́ ecigneul, and Thomas Hof- mann. Hyperbolic entailment cones for learning hierarchi- cal embeddings. In International Conference on Machine Learning (ICML), pages 1646–1655, 2018. 2, 5 [8] Camille Gra ̈ eff, Nicolas Padoy, Philippe Liverneaux, and Thomas Lampert. Introducing surgical workflow recognition in orthopaedic surgery with timestamp supervision. Comput- ers in Biology and Medicine, 197:110995, 2025. 3, 4 [9] Yaojun Hu, Kun Yuan, Nassir Navab, Haochao Ying, Jian Wu, and Nicolas Padoy. HyperVLP: Enhancing hierarchi- cal surgical video-language pre-training in hyperbolic space. arXiv preprint arXiv:2606.31245, 2026. 3 [10] Ruobing Jiang, Mengzhe Liu, Haobing Liu, and Yanwei Yu. Hierarchy-consistent learning and adaptive loss balancing for hierarchical multi-label classification. In ACM Interna- tional Conference on Information and Knowledge Manage- ment (CIKM), 2025. 3 [11] Joel L. Lavanchy, Sanat Ramesh, Diego Dall’Alba, Cristians Gonzalez, Paolo Fiorini, Beat P. M ̈ uller-Stich, Philipp C. Nett, Jacques Marescaux, Didier Mutter, and Nicolas Padoy. Challenges in multi-centric generalization: Phase and step recognition in Roux-en-Y gastric bypass surgery. Interna- tional Journal of Computer Assisted Radiology and Surgery, 19:2249–2257, 2024. 2, 3, 4, 11 [12] Teng Long, Pascal Mettes, Heng Tao Shen, and Cees G. M. Snoek. Searching for actions on the hyperbole. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1138–1147, 2020. 2, 3 [13] Lena Maier-Hein, Swaroop S. Vedula, Stefanie Speidel, Nassir Navab, Ron Kikinis, Adrian Park, Matthias Eisen- mann, Hubertus Feussner, Germain Forestier, Stamatia Gian- narou, Makoto Hashizume, Darko Katic, Hannes Kenngott, Michael Kranzfelder, Anand Malpani, Keno M ̈ arz, Thomas Neumuth, Nicolas Padoy, Carla Pugh, Nicolai Schoch, Danail Stoyanov, Russell Taylor, Martin Wagner, Gregory D. Hager, and Pierre Jannin. Surgical data science for next- generation interventions. Nature Biomedical Engineering, 1 (9):691–696, 2017. 1 [14] Pascal Mettes, Mina Ghadimi Atigh, Martin Keller-Ressel, Jeffrey Gu, and Serena Yeung. Hyperbolic deep learning in computer vision: A survey. International Journal of Com- puter Vision, 132(9):3484–3508, 2024. 2 [15] Maximilian Nickel and Douwe Kiela.Poincar ́ e embed- dings for learning hierarchical representations. In Advances in Neural Information Processing Systems (NeurIPS), pages 6338–6347, 2017. 2 [16] Avik Pal, Max van Spengler, Guido Maria D’Amely di Me- lendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes. Compositional entailment learning for hyperbolic vision-language models.In International Conference on Learning Representations (ICLR), 2025. 2 [17] Seulki Park, Youren Zhang, Stella X. Yu, Sara Beery, and Jonathan Huang. Visually consistent hierarchical image clas- sification. In International Conference on Learning Repre- sentations (ICLR), 2025. 3 [18] Alejandra Perez, Chinedu Nwoye, Ramtin Raji Kermani, Omid Mohareri, and Muhammad Abdullah Jamal.Sur- gLaVi: Large-scale hierarchical dataset for surgical vision- language representation learning. Medical Image Analysis, 110:103982, 2026. 1, 2, 4, 7, 11, 13 [19] Zhi Qiao, Linbin Han, Xiantong Zhen, Jiahong Gao, and Zhen Qian. HYDEN: Hyperbolic density representations for medical images and reports. In International Conference on Computational Linguistics (COLING), 2025. 2 [20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.Learning transferable vi- sual models from natural language supervision. In Inter- 9 national Conference on Machine Learning (ICML), pages 8748–8763, 2021. 2, 4 [21] Sanat Ramesh, Diego Dall’Alba, Cristians Gonzalez, Tong Yu, Pietro Mascagni, Didier Mutter, Jacques Marescaux, Paolo Fiorini, and Nicolas Padoy. Multi-task temporal con- volutional networks for joint recognition of surgical phases and steps in gastric bypass procedures. International Jour- nal of Computer Assisted Radiology and Surgery, 16:1111– 1119, 2021. 3 [22] Sanat Ramesh, Vinkle Srivastav, Deepak Alapatt, Tong Yu, Aditya Murali, Luca Sestini, Chinedu Innocent Nwoye, Idris Hamoud, Saurav Sharma, Antoine Fleurentin, Georgios Exarchakis, Alexandros Karargyris, and Nicolas Padoy. Dis- secting self-supervised learning methods for surgical com- puter vision. Medical Image Analysis, 88:102844, 2023. 2 [23] Ana Manzano Rodriguez, Cees G. M. Snoek, and Marlies P. Schijven. Bridging the gap: Exposing the hidden challenges towards adoption of artificial intelligence in surgery. British Journal of Surgery, 112(11), 2025. 2 [24] Satwik Srivastava and Deepak Mishra. Severity of error in hierarchical datasets. Scientific Reports, 13:21903, 2023. 1, 2 [25] Andru P. Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. En- doNet: A deep architecture for recognition tasks on laparo- scopic videos. IEEE Transactions on Medical Imaging, 36 (1):86–97, 2017. 11 [26] Martin Wagner, Beat-Peter M ̈ uller-Stich, Anna Kisilenko, Duc Tran, Patrick Heger, Lars M ̈ undermann, David M. Lubotsky, et al. Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark. Medical Image Analysis, 86:102770, 2023. 11 [27] Ziyi Wang, Bo Lu, Yonghao Long, Fangxun Zhong, Tak- Hong Cheung, Qi Dou, and Yunhui Liu. AutoLaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy. In International Conference on Medical Image Computing and Computer- Assisted Intervention (MICCAI), pages 486–496, 2022. 11 [28] Lawrence W. Way, Lygia Stewart, Walter Gantert, Kingsway Liu, Crystine M. Lee, Karen Whang, and John G. Hunter. Causes and prevention of laparoscopic bile duct injuries: Analysis of 252 cases from a human factors and cognitive psychology perspective. Annals of Surgery, 237(4):460–469, 2003. 3 [29] Daiki Yoshikawa and Takashi Matsubara. PHyCLIP: ℓ 1 - product of hyperbolic factors unifies hierarchy and compo- sitionality in vision-language representation learning. In In- ternational Conference on Learning Representations (ICLR), 2026. 2 [30] Zhen Yu, Toan D. Nguyen, Lie Ju, Yaniv Gal, Maithili Sashindranath, Paul Bonnington, Lei Zhang, Victoria Mar, and Zongyuan Ge. Hierarchical skin lesion image classifica- tion with prototypical decision tree. npj Digital Medicine, 8: 26, 2025. 2 [31] Kun Yuan, Vinkle Srivastav, Nassir Navab, and Nicolas Padoy.HecVL: Hierarchical video-language pretraining for zero-shot surgical phase recognition. In International Conference on Medical Image Computing and Computer- Assisted Intervention (MICCAI), 2024. 2 [32] Kun Yuan, Vinkle Srivastav, Nassir Navab, and Nicolas Padoy. Procedure-aware surgical video-language pretrain- ing with hierarchical knowledge augmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2, 11 [33] Kun Yuan, Vinkle Srivastav, Tong Yu, Joel L. Lavanchy, Jacques Marescaux, Pietro Mascagni, Nassir Navab, and Nicolas Padoy.Learning multi-modal representations by watching hundreds of surgical video lectures. Medical Im- age Analysis, 105:103644, 2025. 1, 2 10 A Hierarchy-Aware Video-Language Model Evaluation and Hyperbolic Baseline for Surgery Supplementary Material A. SURGHIBENCH Evaluation Suite A.1. Semantic Step Groupings These groupings are used to compute the semantic step ac- curacy reported in Task 3 (Sec. 5.3). Two steps are “se- mantically related” if they perform the same type of action at different anatomical sites or are closely related surgical variants. The groupings were defined manually by inspect- ing each dataset’s step vocabulary and identifying steps that share the same surgical action. Table 5 lists all groupings per dataset. Table 5. Semantic step groupings used for Task 3. Steps within each group share the same surgical action at different anatomical sites. GroupSteps MIPO (8 steps) FixationDistal fixation, Proximal fixation Plate handlingIntroduction of the plate, Modification of plate positioning StrasbyPass70 / BernBypass70 (46 steps) Defect closureGastrojejunal Defect Closure, Jejunoje- junal Defect Closure ReinforcementGastrojejunal Anast. Reinforcement, Je- junojejunal Anast. Reinforcement StaplingHorizontal, Vertical, Gastrojejunal, Je- junojejunal Stapling Limb measurementBiliary Limb Measurement, Alimentary Limb Measurement Limb openingJejunum Opening, Biliary Limb Open- ing, Alimentary Limb Opening DissectionFat Pad, Lesser Curvature, Retrogastric Dissection Space closure/exposure PetersenSpaceExposure/Closure, Mesenteric Defect Exposure/Closure GraSP (21 steps) Lymph node dissection Dissection Illiac Lymph Nodes, Dissec- tion Obturator Lymph Nodes Anatomical dissectionPrevessical Dissection, Prostate Dissec- tion, Seminal Vessicle Dissection, Denon Dissection PackingPack Lymph Nodes, Pack Prostate CuttingCut Prostate, Cut, Cut Bladder Pass suturePass Suture Urethra, Pass Suture Neck A.2. Zero-Shot Evaluation and MIPO Prompts All zero-shot evaluations use text descriptions as prompts, with no prompt engineering, ensembling, or templates (e.g. “a photo of ”). The model computes cosine similar- ity (or Lorentz distance for HYPERSURG) between the video embedding and each text prompt to produce per-class scores. For GraSP, BernBypass70, and StrasbyPass70, we use the text descriptions provided by SurgLaVi [18], not from the original dataset annotation files. BernBypass70 and StrasbyPass70 share the same annotation vocabulary and prompts; the dataset defines 12 phases [11], but fol- lowing SurgLaVi [18] and existing zero-shot studies [32], we exclude the “other” class for fair comparison. MIPO is not part of SurgLaVi and its annotations do not include text descriptions, so we wrote prompts following the same nar- ration style, describing the surgical actions visible in each phase and step. Tables 6 and 7 list the MIPO prompts. A.3. Replication of SurgCLIP (β) To validate our reimplementation of SurgCLIP ∗ (β) , we reproduce the zero-shot evaluation protocol of Perez et al. [18] (their Table 2) on the six benchmarks they use for phase and step recognition: Cholec80 [25] and HeiC- hole [26] (cholecystectomy), AutoLaparo [27] (hysterec- tomy), StrasbyPass70 and BernBypass70 [11] (gastric by- pass), and GraSP [1] (prostatectomy). Cholec80, Auto- Laparo, and HeiChole provide only phase annotations and therefore cannot be evaluated on the cross-level tasks of SURGHIBENCH, which is why they are reported in this sup- plementary rather than in the main paper. Step recognition is reported only for GraSP, the only dataset for which Sur- gLaVi reports step-level results. Table 8 includes the orig- inal SurgCLIP (β) results reported by Perez et al. alongside our reimplementation SurgCLIP ∗ (β) , allowing readers to as- sess replication fidelity. A.4. Additional Training Details Per-level training dynamics. The contrastive loss con- verges at different rates per hierarchy level: phase loss (0.032) < step loss (0.050) < action loss (0.109) at the end of training. Phases (Level 3), whose rich captions map to few unique labels, are easiest to discriminate contrastively. Actions (Level 1), being brief action-level descriptions, are hardest. HYPERSURG preserves this natural difficulty gra- dient across hierarchy levels rather than collapsing it. Optimizer. AdamW with lr=10 −4 , weight decay 0.02, and gradient clipping at max norm 1.0. LayerNorm parameters, 11 Table 6. Zero-shot prompts: MIPO phases (5 classes). The MIPO repository documentation reports 4 phases and 6 steps; the released annotation files contain 5 and 8 distinct labels, respectively. We use the annotation file counts throughout. PhasePrompt ApproachI incise the skin over the flexor carpi radialis tendon and elevate the pronator quadratus to expose the distal radius. ClosureI close the incision with intradermal sutures and apply a sterile compress bandage over the wound. FixationI place the volar locking plate and secure the distal radius fracture with K-wires, distal locking screws and proximal screws. InstallationI prepare the volar locking plate with the aiming guide, apply a Velpeau bandage and inflate the pneumatic tourniquet. Verification I perform fluoroscopic controls with anteroposterior and lateral views and check flexor tendon freedom and wrist mobility. Table 7. Zero-shot prompts: MIPO steps (8 classes; see Tab. 6 caption). StepPrompt ApproachI incise the skin over the flexor carpi radialis tendon and elevate the pronator quadratus muscle to expose the fracture. ClosureI close the wound with intradermal absorbable sutures and apply a sterile compress dressing. Distal FixationI place K-wires and insert distal locking screws near the wrist joint using a drill guide. InstallationI prepare the volar locking plate with the aiming guide and inflate the pneumatic tourniquet. Introduction of the PlateI introduce the volar locking plate along the distal radius volar cortex without screws. Modification of Plate Posi- tioning I remove and reposition the volar plate to optimize alignment with the volar cortex. Proximal FixationI insert cortical screws through the proximal plate holes into the intact diaphyseal shaft. VerificationI perform fluoroscopic control with anteroposterior and lateral views to confirm reduction and screw placement. bias terms, positional embeddings, and learnable scalar pa- rameters (temperature, curvature, projection scales) are ex- cluded from weight decay. Scheduler. OneCycleLR with 5-epoch linear warmup (ini- tial lr = 10 −6 ), cosine annealing, and a final learning rate of 10 −6 . Batch Composition. Each GPU processes 39 samples: 30 randomly sampled and 9 from 3 forced triplets (one caption per hierarchy level from the same video). Total effective batch size across 8 GPUs: 312. Numerical Stability. All Lorentz distance computations (including acosh) are performed in float32, even when the rest of the forward pass uses bfloat16 mixed precision. The argument to acosh is clamped to [1 +ε,∞) to prevent NaN gradients. Learnable Parameters. Temperature τ and curvature c are stored in log-space and clamped to [0.01, 1.0] and [0.1, 10.0] respectively. Projection scaling factors α v ,α t are stored in log-space, initialized to log(1/ √ 256), and clamped to (−∞, 0] so that the effective scales e α v ,e α t ∈ (0, 1]. At convergence: τ=0.035, c=0.815, e α v =0.054, e α t =0.060. B. HYPERSURG Ablation We ablate the three components of HYPERSURG — Lorentz distance, hierarchy-aware triplet batch sampler, and entailment cones — by training variants with differ- ent combinations. Table 9 compares four configurations: (i) Lorentz distance only (no batch sampler, no cones), (i) Lorentz distance with entailment cones but random batches, (i) Lorentz distance with the triplet batch sampler but no cones, and (iv) the full HYPERSURG with all three components. Both components improve over the Lorentz-only base- line, but they play different roles. The triplet batch sampler is more reliable, improving both phase and step on every multi-level dataset. Where the full model pulls ahead is on the Bypass datasets, where cones and the sampler reinforce each other and produce gains of up to +6.7,p on phase and +8.6,p on step over the sampler alone. The excep- tions follow the expected pattern. On GraSP, whose DAG- like hierarchy violates the tree assumption behind entail- ment cones, adding cones provides no further benefit at the phase level. On MIPO, the sampler alone edges out the full model on step accuracy, though cones still improve phase accuracy (+4.1,p). Overall, the full model improves over the Lorentz-only baseline by +6.0,p on phase and +6.9,p on step, raising the multi-level score from 16.9% to 23.2%. 12 Table 8. Reimplementation validation.Zero-shot down- stream performance (video-averaged, %) comparing the original SurgCLIP (β) results from Perez et al. [18] (their Table 2) with our reimplementation SurgCLIP ∗ (β) , trained on the same data with the same procedure. Phase (↑)Step (↑) AccuracyF1 AccuracyF1 Cholec80 SurgCLIP (β) 58.0 39.4— SurgCLIP ∗ (β) 57.8 34.5— AutoLaparo SurgCLIP (β) 55.7 46.0— SurgCLIP ∗ (β) 58.0 44.3— HeiChole SurgCLIP (β) 57.0 44.0— SurgCLIP ∗ (β) 63.6 35.5— BernBypass70 SurgCLIP (β) 18.3 15.1— SurgCLIP ∗ (β) 23.9 17.4— StrasbyPass70 SurgCLIP (β) 31.2 26.1— SurgCLIP ∗ (β) 29.5 22.7— GraSP SurgCLIP (β) 34.8 28.014.2 11.1 SurgCLIP ∗ (β) 25.1 17.717.99.0 Table 9. Ablation of HYPERSURG components (video-averaged accuracy, %). All variants use Lorentz distance. The full model achieves the best overall performance, with both components con- tributing complementary gains. Bold = best per dataset. Phase (↑) Step (↑) Multi-level (↑) MIPO Lorentz only13.811.912.8 + Entailment cones9.320.613.8 + Triplet sampler16.822.419.4 + Both (HYPERSURG)20.917.619.2 StrasbyPass70 Lorentz only20.424.122.2 + Entailment cones24.320.222.2 + Triplet sampler23.525.424.4 + Both (HYPERSURG)30.232.031.1 BernBypass70 Lorentz only12.617.714.9 + Entailment cones13.817.415.5 + Triplet sampler13.021.116.6 + Both (HYPERSURG)17.429.722.7 GraSP Lorentz only24.212.917.7 + Entailment cones26.722.024.2 + Triplet sampler28.113.819.7 + Both (HYPERSURG)26.614.719.8 Overall Lorentz only17.816.716.9 + Entailment cones18.520.118.9 + Triplet sampler20.420.720.0 + Both (HYPERSURG)23.823.523.2 13 Table 10. Sensitivity of HYPERSURG to the entailment loss weight λ (video-averaged accuracy, %). All variants use the full configuration (Lorentz + cones + triplet sampler), only λ varies. We selectλ=0.1 as it yields the most balanced performance across both levels. Bold = best per dataset. Phase (↑) Step (↑) Multi-level (↑) MIPO λ=0.0118.617.418.0 λ=0.0522.820.021.4 λ=0.120.917.619.2 λ=0.217.523.620.3 λ=0.313.719.916.5 StrasbyPass70 λ=0.0122.126.524.2 λ=0.0527.231.129.1 λ=0.130.232.031.1 λ=0.226.125.425.7 λ=0.328.832.330.5 BernBypass70 λ=0.0110.222.815.2 λ=0.0518.226.321.9 λ=0.117.429.722.7 λ=0.217.723.420.4 λ=0.317.628.622.4 GraSP λ=0.0131.118.323.9 λ=0.0533.810.018.4 λ=0.126.614.719.8 λ=0.235.717.725.1 λ=0.328.616.121.5 Overall λ=0.0120.521.320.3 λ=0.0525.521.922.7 λ=0.123.823.523.2 λ=0.224.322.522.9 λ=0.322.724.222.2 Sensitivity to λ. The entailment loss weight λ (Equa- tion (9)) controls how strongly the model enforces parent– child containment via entailment cones. Table 10 reports phase and step accuracy of HYPERSURG across the four multi-level datasets for five values of λ, with all other hy- perparameters held fixed. No single value of λ dominates across all datasets. While a lower value (λ=0.05) favor phase accuracy, a higher value (λ=0.3) favor step accuracy, λ=0.1 strikes the best balance between both levels, achieving the highest overall multi- level score (23.2%) and the best or near-best results on three of four datasets. C. Qualitative Results To complement the quantitative results, Figure 5 presents qualitative examples of both successful and failed predic- tions from SurgCLIP ∗ (β) and HYPERSURG across all four hierarchical datasets. 14 FrameGround Truth SurgCLIP * (β) HyperSurg BernBypass (BBP02, frame 910) PHASE Gastric Pouch Creation STEP Lesser Curvature Dissection 1. Drainage Insertion +0.696 2. Gastric Opening +0.663 3. Fat Pad Dissection +0.645 4. Retrogastric Dissection +0.645 5. NullStep +0.639 1. Lesser Curvature Dissection -0.685 2. Retrogastric Dissection -0.687 3. Gastric Opening -0.740 4. Omental Transection -0.761 5. Vertical Stapling -0.778 MIPO (video24, frame 21) PHASE Installation STEP Installation 1. Introduction of the Plate +0.580 2. Distal Fixation +0.545 3. Approach +0.531 4. Installation +0.494 5. Proximal Fixation +0.425 1. Installation -0.765 2. Closure -0.785 3. Introduction of the Plate -0.795 4. Distal Fixation -0.804 5. Approach -0.812 StrasbyPass (SBP02, frame 90) PHASE Gastric Pouch Creation STEP Retrogastric Dissection 1. Retrogastric Dissection +0.683 2. Omental Transection +0.670 3. Drainage Insertion +0.669 4. Fat Pad Dissection +0.666 5. Lesser Curvature Dissection +0.666 1. Gastric Opening -0.633 2. Retrogastric Dissection -0.637 3. Gastric Pouch Reinforcement -0.655 4. Vertical Stapling -0.656 5. Gastric Remnant Reinforcement -0.660 GraSP (CASE041, frame 60) PHASE Right pelvic isolated lymphadenectomy STEP Identification and dissection of the iliac vein and artery 1. Cutting and dissection of the external iliac vein's lymph node +0.599 2. Identification and dissection of the iliac vein and artery +0.578 3. Idle +0.509 4. Cut suture or tissue +0.501 5. Prevesical dissection +0.500 1. Idle -0.842 2. Cutting and dissection of the external iliac vein's lymph node -0.863 3. Prevesical dissection -0.889 4. Identification and dissection of the iliac vein and artery -0.892 5. Cut between the prostate and bladder neck -0.904 BernBypass (BBP29, frame 720) PHASE Gastrojejunal Anastomosis STEP Gastrojejunal Stapling 1. Gastrojejunal Stapling +0.332 2. Jejunojejunal Stapling +0.326 3. Suture of Small Bowel Lesion +0.325 4. Horizontal Stapling +0.323 5. Fat Pad Dissection +0.323 1. Jejunojejunal Stapling -0.585 2. Jejunal Transection -0.606 3. Biliary Limb Opening -0.656 4. Alimentary Limb Opening -0.668 5. Biliary Limb Measurement -0.696 Correct (rank #1)Ground truth (not #1)Wrong prediction (#1) PHASE ✓STEP cousin GT @ #8 PHASE ✓STEP exact PHASE ✗STEP cousin GT @ #4 PHASE ✗STEP exact PHASE ✓STEP exact PHASE ✓STEP sibling GT @ #2 PHASE ✗STEP sibling GT @ #2 PHASE ✗STEP cousin GT @ #4 PHASE ✗STEP exact PHASE ✗STEP cousin GT @ #8 PHASE ✓ / ✗ — phase verdictSTEP exact / sibling / cousin — step severityGT @ #n — rank of ground truth Figure 5. Where does each model get it right, and when wrong, how bad is the error? Each row shows a test frame with ground-truth labels and top-5 step predictions. Top two rows: HYPERSURG predicts the exact step while SurgCLIP ∗ (β) predicts a cousin from a different phase. Row 3: HYPERSURG predicts the wrong step but stays within the correct phase (sibling, 0.004 margin), while SurgCLIP ∗ (β) gets the exact step. Flat metrics treat both errors identically. Row 4: HYPERSURG predicts Idle, a cross-phase error on GraSP, consistent with its DAG-like hierarchy. Row 5 (subtree lock-in): HYPERSURG commits to the wrong phase subtree, with all five predictions from Jejunojejunal Anastomosis instead of the ground-truth Gastrojejunal Anastomosis, while SurgCLIP ∗ (β) ranks the correct step first. Both the strengths and this failure mode are consistent with the same geometric property: hierarchical containment helps when the correct subtree is identified, but can trap the model when sibling phases are visually similar. 15