Paper deep dive
SurgAtt-Tracker: Online Surgical Attention Tracking via Temporal Proposal Reranking and Motion-Aware Refinement
Rulin Zhou, Guankun Wang, An Wang, Yujie Ma, Lixin Ouyang, Bolin Cui, Junyan Li, Chaowei Zhu, Mingyang Li, Ming Chen, Xiaopin Zhong, Peng Lu, Jiankun Wang, Xianming Liu, Hongliang Ren
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 2:21:56 PM
Summary
The paper introduces SurgAtt-Tracker, a framework for tracking surgical attention in minimally invasive surgery by modeling focus as a dense attention heatmap. It utilizes temporal proposal reranking and motion-aware refinement to handle occlusion and multi-instrument interference. The authors also present SurgAtt-1.16M, a large-scale benchmark dataset for training and evaluating surgical attention tracking across diverse procedures.
Entities (10)
Relation Signals (8)
SurgAtt-1.16M → includes → SurgAtt-SZPH
confidence 95% · We integrate SurgAtt-SZPH with two public datasets... to form the full SurgAtt-1.16M benchmark
SurgAtt-Tracker → outputs → Attention Heatmap
confidence 95% · model surgeon focus as a dense attention heatmap... SurgAtt-Tracker... yields the attention heatmap
SurgAtt-Tracker → uses → Motion-Aware Adaptive Refine Module
confidence 95% · SurgAtt-Tracker... Motion-Aware Adaptive Refine module fuses visual cues with geometric motion history
SurgAtt-Tracker → uses → Attention Score Rerank Module
confidence 95% · SurgAtt-Tracker decouples localization into three cohesive stages... the Attention Score Rerank module... discards noisy detection confidence
SurgAtt-Tracker → uses → Multi-Scale ROI Decoder
confidence 92% · SurgAtt-Tracker... Multi-Scale ROI Decoder acts as a feature transducer
SurgAtt-1.16M → includes → AutoLaparo
confidence 90% · We integrate SurgAtt-SZPH with two public datasets, AutoLaparo... to form the full SurgAtt-1.16M benchmark
SurgAtt-1.16M → includes → Hamlyn
confidence 90% · We integrate SurgAtt-SZPH with two public datasets... Hamlyn... to form the full SurgAtt-1.16M benchmark
SurgAtt-Tracker → supports →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Accurate and stable field-of-view (FoV) guidance is critical for safe and efficient minimally invasive surgery, yet existing approaches often conflate visual attention estimation with downstream camera control or rely on direct object-centric assumptions. In this work, we formulate surgical attention tracking as a spatio-temporal learning problem and model surgeon focus as a dense attention heatmap, enabling continuous and interpretable frame-wise FoV guidance. We propose SurgAtt-Tracker, a holistic framework that robustly tracks surgical attention by exploiting temporal coherence through proposal-level reranking and motion-aware refinement, rather than direct regression. To support systematic training and evaluation, we introduce SurgAtt-1.16M, a large-scale benchmark with a clinically grounded annotation protocol that enables comprehensive heatmap-based attention analysis across procedures and institutions. Extensive experiments on multiple surgical datasets demonstrate that SurgAtt-Tracker consistently achieves state-of-the-art performance and strong robustness under occlusion, multi-instrument interference, and cross-domain settings. Beyond attention tracking, our approach provides a frame-wise FoV guidance signal that can directly support downstream robotic FoV planning and automatic camera control.
Tags
Links
- Source: https://arxiv.org/abs/2602.20636v1
- Canonical: https://arxiv.org/abs/2602.20636v1
Trouble viewing inline? Open PDF directly →
Full Text
76,225 characters extracted from source content.
Expand or collapse full text
SurgAtt-Tracker: Online Surgical Attention Tracking via Temporal Proposal Reranking and Motion-Aware Refinement Rulin Zhou Guankun Wang An Wang Yujie Ma Lixin Ouyang Bolin Cui Junyan Li Chaowei Zhu Mingyang Li Ming Chen Xiaopin Zhong Peng Lu Jiankun Wang Xianming Liu Hongliang Ren Abstract Accurate and stable field-of-view (FoV) guidance is critical for safe and efficient minimally invasive surgery, yet existing approaches often conflate visual attention estimation with downstream camera control or rely on direct object-centric assumptions. In this work, we formulate surgical attention tracking as a spatio-temporal learning problem and model surgeon focus as a dense attention heatmap, enabling continuous and interpretable frame-wise FoV guidance. We propose SurgAtt-Tracker, a holistic framework that robustly tracks surgical attention by exploiting temporal coherence through proposal-level reranking and motion-aware refinement, rather than direct regression. To support systematic training and evaluation, we introduce SurgAtt-1.16M, a large-scale benchmark with a clinically grounded annotation protocol that enables comprehensive heatmap-based attention analysis across procedures and institutions. Extensive experiments on multiple surgical datasets demonstrate that SurgAtt-Tracker consistently achieves state-of-the-art performance and strong robustness under occlusion, multi-instrument interference, and cross-domain settings. Beyond attention tracking, our approach provides a frame-wise FoV guidance signal that can directly support downstream robotic FoV planning and automatic camera control. 1 Introduction Figure 1: SurgAtt-Tracker enables AI-guided endoscope control by predicting an attention heatmap from raw endoscopic video. Minimally Invasive Surgery (MIS) has revolutionized modern clinical practice across diverse specialties, ranging from general surgery to gynecology and urology (Dupont et al., 2021). By performing sophisticated procedures through small incisions guided by endoscopic imaging, laparoscopic surgery offers profound clinical benefits, including minimized intraoperative blood loss, reduced post-operative pain, and accelerated patient recovery compared to traditional open surgery (Taylor et al., 2016; Dupont et al., 2022). Centrally, the laparoscope provides the primary perceptual interface between the surgeon and the operative field, making Field-of-View (FoV) quality critical to surgical safety and precision. However, precise visual guidance remains challenging, as traditional manual control relies on coordinated interaction between the surgeon and a dedicated assistant (Merola et al., 2002). Prolonged procedures often lead to physical fatigue for the assistant, which manifests as image instability, unintended drifting of the FoV, or loss of target tracking. Furthermore, communication delays between the surgeon and assistant often lead to misplaced views that do not meet the surgeon’s immediate needs (Fang et al., 2024; Gao et al., 2025). These issues disrupt the surgical flow and increase operative risks, ultimately undermining the benefits of the minimally invasive approach. To address the limitations of manual control, extensive studies have investigated automated or semi-automated FoV control strategies. Approaches based on explicit external inputs utilize cues such as gaze (Fujii et al., 2018) and voice commands (Sandoval et al., 2021) to directly control camera motion. While intuitive, these interaction-heavy paradigms inevitably increase the surgeon’s cognitive load and may interrupt the surgical workflow. In contrast, some researchers explore internal surgical cues, most notably instrument motion, as implicit indicators of surgeon intent, enabling more natural FoV adjustment without additional user interaction (Gruijthuijsen et al., 2022). Prior approaches proposed a cognitive robotic endoscope that integrates a learned camera quality classifier with predefined policies to select optimal viewpoints (Bihlmaier and Woern, 2014). Subsequent strategies include centering the FoV on a single instrument (Yang et al., 2019), enclosing multiple tools within a predefined region (Huang et al., 2022), or adjusting viewpoints via model-based visual servoing (Zhang et al., 2023). To improve adaptability across surgical phases, probabilistic models have been introduced to encode surgeon preferences from historical data (Li et al., 2021). Notably, Li et al. (Li et al., 2024) proposed a heuristic decision framework that extracts domain knowledge from clinical videos and formulates FoV control as a constrained optimization problem. Despite their robustness and explainability, these methods typically assume that surgeon attention can be approximated by instrument distributions, which may fail in scenarios involving multiple objects and rapid focus shifts. More recently, data-driven and learning-based approaches have been explored to enhance the intelligence of FoV control. These include imitation learning (Rivas-Blanco et al., 2019; Li et al., 2022) and attention-aware models that associate surgical actions with camera motion (Gao et al., 2022). Parallel to this trend, human–AI collaborative frameworks have been proposed to balance autonomy and surgeon control, combining lightweight commands with scene understanding to infer intent-aligned viewpoints (Gao et al., 2025). Although previous works have explored various cues for guiding automated FoV adjustment, none explicitly formulated the surgeon’s visual attention as a dense, spatially distributed heatmap (as shown in Fig. 1), which can be used as a continuous and spatially expressive signal for FoV guidance beyond discrete directional cues. Existing video-level heatmap visualizations (Droste et al., 2020; Xu et al., 2025) are usually used for post-hoc analysis or monitoring rather than real-time attention modeling rooted in the surgeon’s latent perceptual priorities. Rather than directly predicting camera motion or prescribing FoV control policies, we argue that a prerequisite for intelligent endoscopic assistance is the ability to reliably track where the surgeon is visually attending over time. In practice, surgical attention is not a static object nor a single instrument, but a context-dependent perceptual target that shifts across tissues, tools, and procedural stages (Wang et al., 2026). It may focus on an active instrument during dissection, transition to exposed tissue during inspection, or rapidly relocate in response to bleeding or unexpected events. This observation motivates us to decouple attention modeling from downstream camera control and to treat surgical attention tracking as a learning problem. We represent attention as a dense spatial heatmap that captures graded focus intensity, rather than a single point or bounding box, enabling nuanced modeling of distributed and smoothly evolving visual priorities. However, learning to track surgical attention in real time presents several unique challenges. First, attention is a latent cognitive state that lacks explicit visual markers, making direct supervision infeasible. Second, frame-wise detectors often rank attention-relevant regions inconsistently due to occlusion, motion blur, and low inter-class separability, leading to unstable Top-1 predictions. Third, attention exhibits strong temporal coherence yet can undergo abrupt shifts, requiring models to balance short-term consistency with responsiveness. To address these challenges, we introduce SurgAtt-Tracker, a framework that explicitly tracks surgical attention by exploiting temporal coherence among ambiguous detector proposals. SurgAtt-Tracker formulates attention tracking as a proposal reranking and refinement problem conditioned on cross-frame consistency, enabling stable dense attention heatmap prediction without relying on direct regression. To support this formulation, we propose SurgAtt-1.16M, a large-scale benchmark for surgical attention tracking spanning diverse laparoscopic procedures. Built upon a clinically grounded annotation protocol that bridges discrete expert attention regions and continuous heatmap representations, SurgAtt-1.16M enables systematic learning and evaluation of temporally coherent surgical attention. In summary, our contributions are fourfold: (i) We formulate surgical attention tracking as a spatio-temporal learning problem and propose dense heatmap as its core modeling primitive; (i) We propose SurgAtt-Tracker, a holistic framework that robustly tracks attention by exploiting proposal-level reranking and motion-aware refinement; (i) We introduce SurgAtt-1.16M, a large-scale benchmark that enables systematic heatmap attention evaluation and generalization analysis across procedures and institutions. (iv) Extensive experiments across multiple datasets demonstrate state-of-the-art performance and strong robustness of our SurgAtt-Tracker under occlusion, multi-instrument interference, and cross-domain settings. Figure 2: Overview of the SurgAtt-1.16M dataset, illustrating its anatomical coverage, data sources, and unified organization across organs, procedures, and annotation types. 2 Surgical Attention Tracking: Task and Dataset To advance surgical FoV control from heuristic instrument tracking to intent alignment, we formalize the task of Surgical Attention Tracking (SAT) and introduce SurgAtt-1.16M, a large-scale benchmark for systematic training and evaluating SAT under a unified protocol. 2.1 Task Formulation We define SAT as a spatio-temporal density estimation problem. Let =I1,I2,…,ITV=\I_1,I_2,...,I_T\ be a surgical video sequence. At each frame ItI_t, the primary goal is to predict the primary focus of the surgeon’s operation. Unlike standard object tracking, surgical attention is not a rigid object but a latent cognitive state inferred from visual cues. To bridge the gap between discrete surgical interactions and continuous visual focus, we represent attention using two complementary modalities: First, we define the Latent Attention Center (BtB_t) as a bounding box localizing the primary interaction (e.g., tissue-tool contact), serving as a discrete anchor for training stability and camera centering. Second, we model Attention Density (tH_t) as a dense heatmap to capture the spatial spread and uncertainty of focus; this continuous representation offers a more nuanced tracking target than binary boxes and serves as the primary output for evaluation. 2.2 Data Construction: The SurgAtt-SZPH Subset Existing public datasets often lack the complexity required to model expert attention shifts. A core contribution of this work is the curation of SurgAtt-SZPH, a high-quality clinical dataset derived from 141 hours of laparoscopic footage across 25 patients (Rectum/Stomach resections). Crucially, valid attention modeling requires high operational density. Randomly sampled surgical frames are often static. We therefore employed a two-stage filtering pipeline: (1) Motion filtering using RAFT to discard static intervals (<<20 hours retained), followed by (2) Expert clinical review where surgeons screened segments based on four criteria: (i) operational complexity, (i) coverage of common surgical instruments, (i) inclusion of representative procedures, and (iv) correctness of surgical operations. This yielded 125 interaction-rich clips (approx. 1.12M frames) lasting 4–8 minutes each. Details are in Appendix A.1. 2.3 Annotation Protocol: From Discrete Intent to Continuous Attention Defining ground truth (GT) for latent attention remains challenging: eye-tracking signals are often noisy and reactive, while raw instrument detections fail to capture surgical intent. Thus, we propose a hierarchical annotation strategy that derives continuous attention from discrete expert rules. Step 1: Hierarchical Intent Labeling (Bt)(B_t). Three surgical assistants annotated the primary target box BtB_t for every frame following a strict clinical hierarchy: (i) Tissue-Tool Interaction: The geometric center of the contact area where the instrument actively engages with tissue (e.g., dissection, suturing) takes highest priority; (i) Active Effector Dominance: In multi-instrument scenes, the tool performing the primary maneuver (e.g., electrocautery) takes precedence over passive retractors or static grippers; (i) Navigational Centering: During camera movement or global inspection (no active tool action), the attention target defaults to the anatomical center of the FoV. Step 2: Modeling Visual Persistence (t)(H_t). Human attention exhibits temporal persistence rather than instantaneous disappearance. To model this, we avoid directly rasterizing BtB_t and instead model attention as a temporally decaying Gaussian process. Each discrete box BtB_t is mapped to an anisotropic Gaussian kernel and accumulate them over time using exponential decay. This transforms the jittery discrete sequence into a smooth, probability-like density map tH_t. The mathematical derivation and parameters for this conversion are detailed in Appendix A.2. 2.4 Benchmark Composition We integrate SurgAtt-SZPH with two public datasets, AutoLaparo (Gynecology) (Wang et al., 2022) and Hamlyn (Urology) (Giannarou et al., ), adapting their labels to our unified heatmap protocol to form the full SurgAtt-1.16M benchmark. As detailed in Fig. 2, the benchmark comprises 1.16 million frames, offering the largest and most diverse testbed for surgical attention tracking to date, spanning multiple organs and procedures. 3 Method 3.1 Overview We conceptualize Surgical Attention Tracking as a temporally conditioned proposal evolution problem instead of isolated frame-wise detection. Standard detectors often fail to maintain tracking stability because surgical attention lacks rigid visual features, leading to jittery confidence scores and frequent ID switches. However, a key observation is that while the Top-1 detection prediction is unstable, the true attention target is consistently present within the Top-K proposal set (high recall). To exploit this, SurgAtt-Tracker decouples localization into three cohesive stages: hypothesis generation, temporal reranking, and geometric refinement. As illustrated in Figure 3-A, the framework operates on a target frame ItI_t conditioned on a reference frame It−nI_t-n that represents the previous attention state. Initially, a pre-trained, frozen detector acts as a generic region proposer (Sec. 3.2.1), projecting the image into a high-recall discrete search space (Top-K boxes) and and extract multi-scale features with the Multi-Scale ROI Decoder. Subsequently, the Attention Score Rerank module (Sec. 3.2.3) discards noisy detection confidence in favor of temporal consistency through the cross-attention mechanism. Finally, to overcome the discretization error of fixed proposals, the Motion-Aware Adaptive Refine module (Sec. 3.2.4) fuses visual cues with geometric motion history to predict continuous corrections, yielding the precise final box BtrB_t^r. 3.2 Architecture Details Figure 3: Overview of SurgAtt-Tracker. A frozen detector produces Top-K proposals and multi-scale pyramid features, which are converted into box-aligned embeddings by the Multi-Scale ROI Decoder (B); the AS-Rerank module performs temporal proposal reranking to select the Top-1 attention region (C), and MAA-Refine further refines it using motion-aware geometry (D) and visual evidence to yield the attention heatmap HtH_t. 3.2.1 Frozen Detector & Proposal Generation Our framework begins by transforming the global localization problem into a tractable discrete selection task. We employ a detector D(⋅)D(·) pre-trained on SurgAtt-SZPH as a high-recall hypothesis generator. Crucially, we maintain D(⋅)D(·) in a frozen state throughout tracking training. This decouples feature extraction from tracking dynamics, preventing the model from overfitting to specific detector failure modes while ensuring stable feature distributions for cross-frame matching. Given an input frame ItI_t, this module outputs two distinct primitives. First, it defines a Discrete Search Space (ℬtB_t), extracting the Top-K proposals ℬt=Btkk=1KB_t=\B_t^k\_k=1^K based on raw detection confidence. While the Top-1 box is often unstable due to visual ambiguity, empirical analysis confirms that the true attention target is consistently contained within this expanded candidate set (high recall). Second, to handle the varying scale of surgical interactions, we extract Multi-Scale Semantics (ℱtyF_t^y) in the form of feature pyramids ℱty=Ft(s)s∈3,4,5F_t^y=\F_t^(s)\_s∈\3,4,5\ from the detector neck. These embeddings provide the robust visual descriptors required for the subsequent attention-based retrieval. 3.2.2 Multi-Scale ROI Decoder Effective temporal reranking demands a robust proposal-level representation that is discriminative and spatially aligned. In surgical scenes, the scale of attention targets varies drastically, ranging from minute instrument tips during dissection to large anatomical regions during inspection, which renders single-scale feature extraction insufficient. To address this, the Multi-Scale ROI Decoder (MSR) acts as a feature transducer, mapping the discrete bounding box candidates back into the continuous semantic space of the frozen detector’s feature pyramid ℱtyF_t^y. As shown in Figure 3-B, for each proposal B, we extract semantics across spatial resolutions to simultaneously capture fine-grained texture and global context. Specifically, we employ ROIAlign to harvest fixed-size features from each pyramid level proportional to the box coordinates. These multi-resolution descriptors are projected via a scale-specific layer ϕs(⋅) _s(·) with positional encoding and fused via summation to yield a unified, box-aligned embedding ft(B)f_t(B): ft(B)=∑s∈3,4,5Align(ϕs(Ft(s)),πs(B)),f_t(B)= _s∈\3,4,5\Align\! ( _s\! (F_t^(s) ),\, _s(B) ), (1) where πs(⋅) _s(·) maps B from image coordinates to the coordinate system of the s-th feature map, and Align(⋅)Align(·) denotes the interpolation-based ROI feature extraction operator. This process ensures that subsequent tracking modules operate on consistent, scale-invariant visual descriptors regardless of the target’s changing size. 3.2.3 Attention Score Rerank Module While the frozen detector provides discriminative feature pyramids, its Top-K proposal ranking relies solely on static, per-frame confidence scores, which are often unstable under surgical conditions such as occlusion, motion blur, and rapid viewpoint changes. To address this, we introduce the Attention Score Rerank (AS-Rerank) module, which shifts the selection criterion from static detection confidence to dynamic temporal consistency. Given that surgical attention exhibits pronounced trajectory coherence, this module identifies the optimal candidate at time t by measuring its compatibility with a trusted reference state. As illustrated in Figure 3-C, AS-Rerank operates as a cross-frame retrieval mechanism. It takes the trusted reference ROI embedding frf_r (from frame IrI_r) and the set of current proposal embeddings ftkk=1K\f_t^k\_k=1^K (from frame ItI_t) as input. We employ a multi-head cross-attention mechanism to compute a scalar reranking logit stks_t^k for each proposal: stk=Att(fr,ftk),s_t^k=Att\! (f_r,\,f_t^k ), (2) where Att(⋅)Att(·) is a learnable matching function that weighs feature similarity against the historical context. Finally, we select the index k^t=argmaxkstk k_t= _ks_t^k that maximizes this temporal affinity, outputting the Top-1 hypothesis proposal B^t=Btk^t B_t=B_t k_t and its corresponding feature f^t f_t for subsequent geometric refinement. 3.2.4 Motion-Aware Adaptive Refine Module Although the AS-Rerank module successfully identifies the most plausible candidate from the proposal pool, the selected box B^t B_t remains quantized by the detector’s anchor grid, often leading to residual misalignment in center positioning and scale. To transcend this discretization limit, the Motion-Aware Adaptive Refine (MAA-Refine) module performs continuous geometric correction (Figure 3-D). Crucially, visual-only refinement is often ambiguous in low-texture surgical scenes, so we condition the correction on explicit motion prior encoded from the reference state. The module fuses two inputs: the visual semantic embedding f^t f_t of the selected Top-1 proposal, and a geometric motion descriptor gt=Geo(B^t,Brs)g_t=Geo( B_t,B_r^s). Here, BrsB_r^s represents the trusted attention state from the reference frame It−nI_t-n (using the previous prediction B^t−1 B_t-1 during inference). The descriptor gtg_t explicitly encodes the spatial evolution of the target, capturing normalized coordinates, relative center displacement, and log-scale aspect variations. By concatenating f^t f_t and gtg_t into a lightweight MLP, the network learns to adjust the box based on both local tissue texture and the momentum of the surgeon’s attention. Unlike standard offset regression, which predicts unconstrained Cartesian shifts, we adopt a polar-based formulation that decouples directionality from magnitude. The network predicts a correction tuple consisting of a displacement direction θt _t, a scalar magnitude dtd_t, and channel-wise scale factors t=(stw,sth)s_t=(s_t^w,s_t^h). Let ψ(B^t)=(cx,cy,w,h)ψ( B_t)=(c_x,c_y,w,h) denote the parameterization of the coarse proposal. We update the center via a directed step dtd_t along the unit vector (θt)=(cosθt,sinθt)u( _t)=( _t, _t), and modulate the dimensions via multiplicative scaling: (cxr,cyr) (c_x^r,c_y^r) =(cx,cy)+dt(θt), =(c_x,c_y)+d_t\,u( _t), (3) (wr,hr) (w^r,h^r) =(w,h)⊙(stw,sth), =(w,h) (s_t^w,s_t^h), where ⊙ denotes element-wise multiplication. The final refined box BtrB_t^r is reconstructed from these parameters and serves as the precise anchor for generating the continuous attention heatmap tH_t. 3.3 Training Objectives To jointly optimize discrete proposal selection and continuous geometric refinement, our training objective is composed of two hierarchical components: a reranking loss that enforces correct ordering within the hypothesis pool, and a refinement loss that regularizes the precise regressors. The total objective is defined as: ℒtotal=ℒrerank(t)+ℒrefine(t).L_total=L_rerank^(t)+L_refine^(t). (4) We compute these losses on the target frame ItI_t, isolating valid proposals (those with non-zero overlap) to prevent noise from background outliers. 3.3.1 Reranking losses The goal of reranking is to assign high scores to proposals that are spatially closest to the GT BtgtB_t^gt. Let ℬt=Btkk=1KB_t=\B_t^k\_k=1^K be the set of valid proposals and st=stkk=1Ks_t=\s_t^k\_k=1^K be the predicted logits. We identify the optimal proposal index kt⋆k_t as the one minimizing the center distance to the GT: kt⋆=argmink‖c(Btk)−c(Btgt)‖2.k_t = _k \|c(B_t^k)-c(B_t^gt) \|_2. (5) To ensure robust ranking, we employ a multi-task objective combining hard classification, soft geometric regularization, and listwise ranking: 1. Hard Top-1 Classification (ℒceL_ce). We maximize the probability of selecting the single best candidate kt⋆k_t using standard cross-entropy: ℒce=CE(softmax(st),kt⋆).L_ce=CE\! (softmax(s_t),\,k_t ). (6) 2. Soft Geometric Regularization (ℒgeoL_geo). Hard classification ignores the spatial distribution of the prediction. To inject geometric awareness, we compute a soft-aggregated box B¯t B_t by weighing all proposals using their predicted probabilities πtk=exp(stk/τ)∑jexp(stj/τ) _t^k= (s_t^k/τ) _j (s_t^j/τ) (with temperature τ=0.15τ=0.15). We then minimize the Huber loss between the center of this aggregated box and the GT: ℒgeo=ℒHuber(‖c(B¯t)−c(Btgt)‖2),L_geo=L_Huber\! (\|c( B_t)-c(B_t^gt)\|_2 ), (7) which encourages the probability mass to concentrate around the spatial location of the target, smoothing the decision surface. 3. Top-M Listwise Ranking (ℒrankL_rank). Standard Top-1 supervision induces a sharp decision boundary that suppresses all non-target proposals equally, regardless of their geometric utility. However, surgical attention is often spatially smooth; proposals slightly offset from the exact center still capture valid context. To exploit this, we enforce a local ranking constraint: the model’s confidence distribution should mirror the geometric error distribution within the elite candidate pool. We select the Top-M (M=5M=5) proposals closest to the GT and construct a “geometric teacher” distribution qtq_t that decays with spatial error ϵt _t: qtk=exp(−ϵtk/σ)∑jexp(−ϵtj/σ),ptk=exp(stk)∑jexp(stj).q_t^k= (- _t^k/σ) _j (- _t^j/σ), p_t^k= (s_t^k) _j (s_t^j). (8) By minimizing the cross-entropy between the predicted distribution ptp_t and this geometric prior qtq_t (with smoothing factor σ=15σ=15px), i.e., ℒrank(t)=−∑kqtklogptkL_rank^(t)=- _kq_t^k p_t^k, we ensure that tracking confidence degrades smoothly rather than abruptly. This provides informative gradients for determining relative quality among high-overlap candidates, preventing the tracker from discarding useful near-miss hypotheses. Overall, the final reranking loss is the weighted sum: ℒrerank(t)=ℒce+λgeoℒgeo+λrankℒrank,L_rerank^(t)=L_ce+ _geoL_geo+ _rankL_rank, (9) where we empirically set λgeo=0.1 _geo=0.1 and λrank=0.5 _rank=0.5 to balance the relative gradient magnitudes across terms. 3.3.2 Refinement loss Once the optimal proposal is selected, our goal is to fine-tune its geometry to precisely match the GT BtgtB_t^gt. We supervise the final refined box BtrB_t^r by decoupling the regression task into two orthogonal components: translation alignment and scale calibration. 1. Center Alignment Loss (ℒdistL_dist). To ensure precise localization, we penalize the spatial displacement between the predicted center c(Btr)c(B_t^r) and the GT center c(Btgt)c(B_t^gt). We employ a normalized Huber loss to be robust against outliers while maintaining sensitivity to small errors: ℒdist(t)=ℒHuber(‖c(Btr)−c(Btgt)‖2W2+H2),L_dist^(t)=L_Huber\! ( \|c(B_t^r)-c(B_t^gt)\|_2 W^2+H^2 ), (10) where the diagonal normalization factor W2+H2 W^2+H^2 ensures the loss magnitude is invariant to image resolution. 2. Log-Space Scale Loss (ℒscaleL_scale). Direct regression of width and height is sensitive to object scale variance; for instance, a 10-pixel error is negligible for large organs yet catastrophic for small instrument tips. To address this, we regress dimensions in logarithmic space, thereby equalizing relative errors across scales. Let (B)=(w,h)s(B)=(w,h) be the size vector of box B. We define: ℒscale(t)=ℒHuber(log(Btr)−log(Btgt)).L_scale^(t)=L_Huber\! ( (B_t^r)- (B_t^gt) ). (11) The total refinement loss is a weighted combination: ℒrefine(t)=λdistℒdist(t)+ℒscale(t),L_refine^(t)= _distL_dist^(t)+L_scale^(t), (12) where we set λdist=0.1 _dist=0.1 to balance the gradients between translation and scaling tasks. 4 Experiments and Result Analysis Table 1: Quantitative comparison of attention heatmap prediction on SurgAtt-SZPH dataset using NSS, C, SIM, MSE, and MAE. Model NSS↑ C↑ SIM↑ MSE↓ MAE↓ U-Net–based Unet (Ronneberger et al., 2015) 1.051 0.393 0.490 0.079 0.191 TransUNet (Chen et al., 2021) 1.947 0.697 0.651 0.033 0.101 UNeXt (Valanarasu and Patel, 2022) 1.964 0.712 0.646 0.029 0.109 EMCAD (Rahman et al., 2024) 2.203 0.776 0.711 0.0261 0.089 Regression-based SASNet (Song et al., 2021) 1.690 0.669 0.608 0.032 0.138 RCMNet (Xu et al., 2025) 1.763 0.684 0.618 0.036 0.115 SalFoM (Moradi et al., 2024) 2.030 0.702 0.647 0.028 0.094 Object Tracking–based AQATrack (Xie et al., 2024) 1.093 0.397 0.559 0.070 0.199 ODTrack (Zheng et al., 2024) 1.343 0.212 0.572 0.068 0.193 SPMTrack-B (Cai et al., 2025) 1.720 0.571 0.643 0.060 0.149 LoRAT-B (Lin et al., 2024) 1.837 0.597 0.669 0.053 0.131 LoRATv2-B (Lin et al., ) 1.853 0.649 0.676 0.047 0.126 MCITrack (Kang et al., 2025) 1.979 0.691 0.690 0.041 0.107 Detection-based YOLOv11 (Khanam and Hussain, 2024) 2.195 0.796 0.721 0.030 0.099 YOLOv12 (Tian et al., 2025) 2.285 0.793 0.723 0.028 0.090 RT-DETR (Zhao et al., 2023) 2.359 0.820 0.737 0.025 0.087 RT-DETRv2 (Lv et al., 2024) 2.370 0.822 0.739 0.024 0.086 SurgAtt-Tracker(Ours) 2.580 0.871 0.829 0.015 0.051 4.1 Implementation Details We group baselines into four categories: (i) U-Net-based, (i) regression-based, (i) object-tracking-based, and (iv) detection-based methods. To enable fair comparison, we convert all model outputs into a normalized attention heatmap under a unified evaluation protocol; model-specific adaptations are deferred to the supplementary material. Following prior works (Droste et al., 2020; Xu et al., 2025), we report five metrics: NSS, C, SIM, MSE and MAE, where the former three assess distributional alignment and the latter two measure pixel errors. All frames are resized to 960×540960×540. We use AdamW (lr 1×10−41×10^-4) and run all experiments on a single NVIDIA A100. Additional implementation details, including metrics, model-specific adaptations, training schedules, are provided in the appendix A.3 Table 2: Zero-shot and fine-tuning performance on the SurgAtt-AutoLaparo and SurgAtt-Hamlyn datasets. Model NSS↑ C↑ SIM↑ MSE↓ MAE↓ SurgAtt-AutoLaparo Zero-shot EMCAD 1.378 0.593 0.648 0.077 0.214 SalFoM 1.408 0.531 0.594 0.078 0.206 MCITrack 1.440 0.578 0.503 0.074 0.196 YOLOv12 1.551 0.658 0.520 0.059 0.146 RT-DETRv2 1.864 0.708 0.556 0.053 0.156 SurgAtt-Tracker (Ours) 1.949 0.721 0.575 0.035 0.088 Fine-tuning YOLOv12 2.516 0.756 0.671 0.022 0.065 RT-DETRv2 2.715 0.773 0.708 0.019 0.058 SurgAtt-Tracker (Ours) 2.741 0.837 0.769 0.014 0.048 SurgAtt-Hamlyn Zero-shot EMCAD 1.098 0.552 0.603 0.211 0.359 SalFoM 1.133 0.497 0.574 0.164 0.235 MCITrack 1.035 0.398 0.533 0.275 0.447 YOLOv12 1.171 0.575 0.640 0.081 0.198 RT-DETRv2 1.269 0.651 0.594 0.178 0.180 SurgAtt-Tracker (Ours) 1.693 0.686 0.711 0.046 0.128 Fine-tuning YOLOv12 1.476 0.685 0.694 0.057 0.162 RT-DETRv2 1.744 0.764 0.740 0.037 0.126 SurgAtt-Tracker (Ours) 2.195 0.890 0.841 0.016 0.071 Figure 4: Qualitative comparison of attention heatmaps across diverse surgical scenarios: (A) single-instrument case; (B) multi-instrument without tissue interaction; (C) multi-instrument with tissue interaction; (D) multi-instrument with smoke interference. 4.2 Main Results Evaluation and Visualization on SurgAtt-SZPH. We evaluate all baselines and SurgAtt-Tracker on SurgAtt-SZPH, training each model for 4 epochs. SurgAtt-Tracker first pre-trains the YOLOv12 detector for two epochs, then freezes it and optimizes the remaining modules (MSR, AS-Rerank, MAA-Refine) for another 2 epochs under a unified training/inference setting. Table 1 compares attention heatmap prediction on SurgAtt-SZPH across four types of baselines. U-Net/regression baselines often produce diffuse attention with unstable peaks and limited temporal coherence, which caps both correlation (NSS/C/SIM) and pixel-wise errors (MSE/MAE). This behavior is also evident in Figure 4: EMCAD/SalFoM frequently spreads activation over a broad region and shifts the peak away from the GT focus, especially under cluttered tool appearances. Tracking-based methods introduce temporal cues but are primarily optimized for appearance/boundary consistency and can fail under target disappearance or multi-instrument interference (initialized with GT in our evaluation), leading to accumulated drift. As shown in Figure 4(B–C), MCITrack is more prone to mis-association when multiple instruments are present, yielding off-target or over-smoothed heatmaps. Detection-based baselines are strongest overall; RT-DETRv2 achieves the best prior performance (NSS=2.370, C=0.822, SIM=0.739), yet YOLOv12 confidence-driven per-frame ranking remains unreliable under occlusion, blur, and viewpoint changes. Qualitatively, Figure 4(D) further highlights that smoke interference can disrupt per-frame selection, resulting in peak instability or spatial bias even for strong detectors. In contrast, SurgAtt-Tracker consistently outperforms all competitors, achieving (NSS=2.580, C=0.871, SIM=0.829) with substantially lower errors (MSE=0.015, MAE=0.051). Compared with RT-DETRv2, our method improves NSS/C/SIM by 8.9%/6.0%/12.2% and reduces MSE/MAE by 37.5%/40.7%. Consistent with these gains, Figure 4 shows that our predictions remain sharper and better aligned with GT across diverse scenarios (A–D), including multi-instrument clutter and smoke. Importantly, SurgAtt-Tracker runs at 12.5 FPS in an online setting, approaching real-time capability for closed-loop endoscopic control. We further report per-scene performance on five SurgAtt-SZPH subsets in Appendix A.5.2. Overall, these results validate our formulation of heatmap prediction as temporally consistent retrieval over a high-recall proposal set (AS-Rerank), followed by motion-conditioned continuous refinement (MAA-Refine). Evaluation on SurgAtt-AutoLaparo and Hamlyn. Table 2 evaluates robustness under domain and resolution shifts on two surgical scenarios: uterine tumor resection (AutoLaparo, 1920×10801920×1080) and nephrectomy (Hamlyn, 384×192384×192). In the zero-shot setting, SurgAtt-Tracker achieves strong transfer on both datasets (AutoLaparo: NSS=1.949, MAE=0.088; Hamlyn: NSS=1.693, MAE =0.128). After fine-tuning, our performance further improves substantially over our zero-shot model, with NSS increasing by +40.6%/+29.7% and MAE reduced by 45.5%/44.5% on SurgAtt-AutoLaparo/Hamlyn, respectively, confirming the robustness and transferability of our framework across different pixel videos and scene variations (For more visual results, see the appendix Section A.5.2). 4.3 Ablation Studies Table 3: Reports detector-only best-case performance under three rules (Conf/MinErr/MaxIoU), serving as oracle baselines; SurgAtt-Tracker further incorporates AS-Rerank and MAA-Refine. Method Rerank Refine Err↓ IoU↑ NSS↑ C↑ MAE↓ Detector-only baselines YOLOv12 (Conf) – – 52.192 0.589 2.285 0.793 0.090 YOLOv12 (MinErr) – – 30.408 0.649 2.525 0.857 0.063 YOLOv12 (MaxIoU) – – 32.744 0.665 2.437 0.866 0.064 SurgAtt-Tracker Ours (w/o Refine) ✓ × 36.975 0.627 2.313 0.823 0.079 Ours (full) ✓ ✓ 27.525 0.690 2.580 0.871 0.051 Table 4: Ablation on the rerank loss weight (Top-M). Top-M NSS↑ C↑ SIM↑ MSE↓ MAE↓ 0 1.973 0.729 0.621 0.047 0.144 3 2.379 0.859 0.792 0.024 0.086 5 2.580 0.871 0.829 0.015 0.051 7 2.195 0.812 0.753 0.033 0.107 Table 5: Impact of temporal gap sampling during training. Sample Gap NSS↑ C↑ SIM↑ MSE↓ MAE↓ Avg Gap 2.592 0.863 0.788 0.045 0.090 Random Gap 2.339 0.862 0.798 0.034 0.065 Archer Gap 2.580 0.871 0.829 0.015 0.051 Ablation on Reranking and Refinement. Table 3 diagnoses why detector confidence is not sufficient and verifies the complementary roles of our reranking and refinement. Although YOLOv12 is capable of producing high-quality candidates, its confidence-based Top-1 selection is often suboptimal: In contrast, an oracle choice within YOLOv12 proposals (MinErr / MaxIoU) reveals that better boxes already exist in the candidate pool (Err = 30.41; IoU = 0.665). Building on this candidate pool, our AS-Rerank module effectively recovers better hypotheses by reordering proposals according to temporal consistency, reducing localization error from 52.192 to 36.975 and improving NSS from 2.285 to 2.313 (see Appendix A.6 for detailed analysis of AS-Rerank). Adding MAA-Refine module reduces Err from 36.98 to 27.53, while consistently boosting heatmap metrics. Ablation on Top-M Ranking. We ablate the Top-M setting in the ℒrankL_rank. As shown in Table 4, performance improves from M=0M=0 to M=5M=5 but degrades when M is further increased. Top-5 supervision strikes a good balance between learning meaningful relative orderings and suppressing noisy candidates, yielding the best overall results. Ablation on Temporal Sample Gap Training solely on adjacent pairs (n=1n=1) may lead to an identity shortcut, encouraging the model to copy the previous prior under slow motion and reducing robustness to rapid tool/camera motion. We therefore sample multi-scale temporal gaps during training: for each target frame t, we draw n∼Cat(,)n\! \!Cat(N,P) with =1,2,4,8,16,32N=\1,2,4,8,16,32\ and =0.4,0.2,0.1,0.1,0.1,0.1P=\0.4,0.2,0.1,0.1,0.1,0.1\, and use r=t−nr=t-n as the reference. Results in Table 5 confirm consistent gains in heatmap quality and localization stability over uniform and random sampling. 5 Conclusion We introduce SurgAtt-1.16M, a large-scale benchmark unified from multiple sources for studying surgical attention tracking across diverse organs and conditions, and propose SurgAtt-Tracker, an holistic framework that tracks attention via proposal reranking and motion-aware refinement, achieving state-of-the-art results. Limitations include reliance on detector proposal coverage, potential drift under prolonged occlusions or abrupt camera motion, and a current focus on attention estimation rather than camera control. Future work will integrate SurgAtt-Tracker into robotic systems for safe, stable, and intent-aware endoscope control. Impact Statement This research bridges a critical gap in medical robotics by providing the first large-scale benchmark to enable attention-aware endoscopic control. By open-sourcing SurgAtt-1.16M, we aim to democratize access to high-quality clinical data, accelerating the development of intelligent surgical assistants that can reduce procedural variability and cognitive fatigue. While our framework advances surgical autonomy, we emphasize that attention tracking is probabilistic; clinical deployment requires rigorous “human-in-the-loop” validation to prevent potential misguidance during rare adverse events. All patient data in this study was strictly de-identified to safeguard privacy. References A. Bihlmaier and H. Woern (2014) Automated endoscopic camera guidance: a knowledge-based system towards robot assisted surgery. In ISR/Robotik 2014; 41st International Symposium on Robotics, p. 1–6. Cited by: §1. W. Cai, Q. Liu, and Y. Wang (2025) SPMTrack: spatio-temporal parameter-efficient fine-tuning with mixture of experts for scalable visual tracking. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 16871–16881. Cited by: Table 1. J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and Y. Zhou (2021) Transunet: transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306. Cited by: Table 1. R. Droste, J. Jiao, and J. A. Noble (2020) Unified image and video saliency modeling. In European conference on computer vision, p. 419–435. Cited by: §1, §4.1. P. E. Dupont, B. J. Nelson, M. Goldfarb, B. Hannaford, A. Menciassi, M. K. O’Malley, N. Simaan, P. Valdastri, and G. Yang (2021) A decade retrospective of medical robotics research from 2010 to 2020. Science robotics 6 (60), p. eabi8017. Cited by: §1. P. E. Dupont, N. Simaan, H. Choset, and C. Rucker (2022) Continuum robots for medical interventions. Proceedings of the IEEE 110 (7), p. 847–870. Cited by: §1. J. Fang, L. Li, X. Li, H. Mo, P. Guo, X. Xiao, and Y. Qu (2024) A force-driven and vision-driven hybrid control method of autonomous laparoscope-holding robot. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 5857–5863. Cited by: §1. K. Fujii, G. Gras, A. Salerno, and G. Yang (2018) Gaze gesture based human robot interaction for laparoscopic surgery. Medical image analysis 44, p. 196–214. Cited by: §1. H. Gao, W. Fan, L. Qiu, X. Yang, Z. Li, X. Zuo, Y. Li, M. Q. Meng, and H. Ren (2022) SAVAnet: surgical action-driven visual attention network for autonomous endoscope control. IEEE Transactions on Automation Science and Engineering 20 (4), p. 2655–2667. Cited by: §1. Y. Gao, Z. Li, J. Zhao, J. Li, and J. Li (2025) A human–ai collaborative framework for surgical field-of-view adjustment: design and experimental validation. Journal of Robotic Surgery 19 (1), p. 259. Cited by: §1, §1. [11] S. Giannarou, D. Stoyanov, D. Noonan, G. Mylonas, J. Clark, M. Visentini-Scarzanella, P. Mountney, and G. Yang Hamlyn centre laparoscopic / endoscopic video datasets. Hamlyn Centre, Imperial College London. External Links: Link Cited by: §2.4. C. Gruijthuijsen, L. C. Garcia-Peraza-Herrera, G. Borghesan, D. Reynaerts, J. Deprest, S. Ourselin, T. Vercauteren, and E. Vander Poorten (2022) Robotic endoscope control via autonomous instrument tracking. Frontiers in Robotics and AI 9, p. 832208. Cited by: §1. Y. Huang, J. Li, X. Zhang, K. Xie, J. Li, Y. Liu, C. S. H. Ng, P. W. Y. Chiu, and Z. Li (2022) A surgeon preference-guided autonomous instrument tracking method with a robotic flexible endoscope based on dvrk platform. IEEE Robotics and Automation Letters 7 (2), p. 2250–2257. Cited by: §1. B. Kang, X. Chen, S. Lai, Y. Liu, Y. Liu, and D. Wang (2025) Exploring enhanced contextual information for video-level object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 4194–4202. Cited by: Table 1. R. Khanam and M. Hussain (2024) Yolov11: an overview of the key architectural enhancements. arXiv preprint arXiv:2410.17725. Cited by: Table 1. P. Langley (2000) Crafting papers on machine learning. In Proceedings of the 17th International Conference on Machine Learning (ICML 2000), P. Langley (Ed.), Stanford, CA, p. 1207–1216. Cited by: §A.6. B. Li, B. Lu, Y. Lu, Q. Dou, and Y. Liu (2021) Data-driven holistic framework for automated laparoscope optimal view control with learning-based depth perception. In 2021 IEEE International Conference on Robotics and Automation (ICRA), p. 12366–12372. Cited by: §1. B. Li, Y. Lu, W. Chen, B. Lu, F. Zhong, Q. Dou, and Y. Liu (2024) Gmm-based heuristic decision framework for safe automated laparoscope control. IEEE Robotics and Automation Letters 9 (2), p. 1969–1976. Cited by: §1. B. Li, R. Wei, J. Xu, B. Lu, C. H. Yee, C. F. Ng, P. Heng, Q. Dou, and Y. Liu (2022) 3D perception based imitation learning under limited demonstration for laparoscope control in robotic surgery. In 2022 International Conference on Robotics and Automation (ICRA), p. 7664–7670. Cited by: §1. [20] L. Lin, H. Fan, Z. Zhang, Y. Huang, Y. Wang, Y. Xu, and H. Ling LoRATv2: enabling low-cost temporal modeling in one-stream trackers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: Table 1. L. Lin, H. Fan, Z. Zhang, Y. Wang, Y. Xu, and H. Ling (2024) Tracking meets lora: faster training, larger model, stronger performance. In ECCV, Cited by: Table 1. W. Lv, Y. Zhao, Q. Chang, K. Huang, G. Wang, and Y. Liu (2024) RT-detrv2: improved baseline with bag-of-freebies for real-time detection transformer. External Links: 2407.17140, Link Cited by: Table 1. S. Merola, P. Weber, A. Wasielewski, and G. H. Ballantyne (2002) Comparison of laparoscopic colectomy with and without the aid of a robotic camera holder. Surgical Laparoscopy Endoscopy & Percutaneous Techniques 12 (1), p. 46–51. Cited by: §1. M. Moradi, M. Moradi, F. Rundo, C. Spampinato, A. Borji, and S. Palazzo (2024) Salfom: dynamic saliency prediction with video foundation models. In International Conference on Pattern Recognition, p. 33–48. Cited by: Table 1. M. M. Rahman, M. Munir, and R. Marculescu (2024) Emcad: efficient multi-scale convolutional attention decoding for medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11769–11779. Cited by: Table 1. I. Rivas-Blanco, C. J. Perez-del-Pulgar, C. López-Casado, E. Bauzano, and V. F. Muñoz (2019) Transferring know-how for an autonomous camera robotic assistant. Electronics 8 (2), p. 224. Cited by: §1. O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, p. 234–241. Cited by: Table 1. J. Sandoval, M. A. Laribi, J. Faure, C. Brèque, J. Richer, and S. Zeghloul (2021) Towards an autonomous robot-assistant for laparoscopy using exteroceptive sensors: feasibility study and implementation. IEEE Robotics and Automation Letters 6 (4), p. 6473–6480. Cited by: §1. Q. Song, C. Wang, Y. Wang, Y. Tai, C. Wang, J. Li, J. Wu, and J. Ma (2021) To choose or to fuse? scale selection for crowd counting. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35, p. 2576–2583. Cited by: Table 1. R. H. Taylor, A. Menciassi, G. Fichtinger, P. Fiorini, and P. Dario (2016) Medical robotics and computer-integrated surgery. In Springer handbook of robotics, p. 1657–1684. Cited by: §1. Y. Tian, Q. Ye, and D. Doermann (2025) Yolov12: attention-centric real-time object detectors. arXiv preprint arXiv:2502.12524. Cited by: Table 1. J. M. J. Valanarasu and V. M. Patel (2022) Unext: mlp-based rapid medical image segmentation network. In International conference on medical image computing and computer-assisted intervention, p. 23–33. Cited by: Table 1. B. Wang, J. Zhang, Z. Pan, S. Zhao, M. Li, and H. Liu (2026) Automatic field-of-view adjustment for multi-surgical tools of a novel continuum laparoscopic robot. Measurement 257, p. 118544. Cited by: §1. Z. Wang, B. Lu, Y. Long, F. Zhong, T. Cheung, Q. Dou, and Y. Liu (2022) AutoLaparo: a new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 486–496. Cited by: §2.4. J. Xie, B. Zhong, Z. Mo, S. Zhang, L. Shi, S. Song, and R. Ji (2024) Autoregressive queries for adaptive tracking with spatio-temporal transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19300–19309. Cited by: Table 1. M. Xu, W. Mo, G. Wang, H. Gao, A. Wang, L. Bai, C. Lyu, X. Yang, Z. Li, and H. Ren (2025) Etsm: automating dissection trajectory suggestion and confidence map-based safety margin prediction for robot-assisted endoscopic submucosal dissection. In 2025 IEEE International Conference on Robotics and Automation (ICRA), p. 4513–4519. Cited by: §1, §4.1, Table 1. B. Yang, W. Chen, Z. Wang, Y. Lu, J. Mao, H. Wang, and Y. Liu (2019) Adaptive fov control of laparoscopes with programmable composed constraints. IEEE Transactions on Medical Robotics and Bionics 1 (4), p. 206–217. Cited by: §1. C. Zhang, W. Zhu, J. Peng, Y. Han, and W. Liu (2023) Visual servo control of endoscope-holding robot based on multi-objective optimization: system modeling and instrument tracking. Measurement 211, p. 112658. Cited by: §1. Y. Zhao, W. Lv, S. Xu, J. Wei, G. Wang, Q. Dang, Y. Liu, and J. Chen (2023) DETRs beat yolos on real-time object detection. External Links: 2304.08069 Cited by: Table 1. Y. Zheng, B. Zhong, Q. Liang, Z. Mo, S. Zhang, and X. Li (2024) Odtrack: online dense temporal token learning for visual tracking. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38, p. 7588–7596. Cited by: Table 1. Appendix A Appendix A.1 Dataset Processing and Detailed Supplementation Figure 5: Construction pipeline of the SurgAtt-SZPH dataset. Raw laparoscopic videos are curated into high-quality surgical clips via optical-flow–based operation analysis and expert screening. Videos are sampled at 25 fps and grouped into five representative surgical scenes. During annotation, surgeons mark attention regions with bounding boxes, which are converted into continuous attention heatmaps for supervision. The resulting dataset provides dense, high-fidelity attention annotations across diverse surgical scenarios. A.1.1 Dataset Annotation Pipline The curation process is illustrated in Figure 5 and consists of two stages to efficiently obtain informative and clinically meaningful attention supervision. Stage 1 (a1): motion-based mining Starting from 142 hours of raw endoscopic videos, we run RAFT optical flow to estimate frame-to-frame motion and discard temporally static or low-motion segments that typically contain limited viewpoint changes and weak attention transitions. This step narrows the search space to segments with moderate activity, improving annotation efficiency while preserving diverse camera motions and instrument–tissue interactions. Stage 2 (a2): clinician screening Candidate segments are reviewed by expert surgeons to ensure clinical relevance and data quality, using four criteria: (1) operational complexity, (2) coverage of common surgical instruments, (3) inclusion of representative procedures, and (4) correctness of surgical operations. We then sample 30 minutes per patient and split them into clips of 4–8 minutes to balance temporal continuity with manageable clip length. The resulting SurgAtt-SZPH subset covers five surgical sites (High/Middle/Low Rectum, Distal/Proximal Stomach) and contains 125 clips with about 1M frames at 25 fps. Annotation motivation Rather than exhaustively segmenting objects, surgeons label a single box that captures the clinically attended region (e.g., the active dissection area or target anatomy) per frame, which is then converted into a normalized attention heatmap. This design (i) aligns supervision with surgeon intent and viewpoint control, (i) yields dense, temporally coherent targets suitable for tracking and heatmap evaluation, and (i) substantially reduces annotation cost compared with pixel-level labeling while remaining robust to appearance ambiguity in endoscopic scenes. Table 6: SurgAtt-1.16M forms a complete benchmark by aggregating three subsets under a unified annotation protocol, and summarizes the data source, organ, procedure, image resolution, and the number of frames in the train/test splits. Dataset Name Data Source Surgical Organ Procedure Image Size Train Test Total SurgAtt-SZPH Anonymous Hospital Rectum, Stomach Rectal and gastric cancer resections 1920x1080 800,000 200,000 1,000,000 SurgAtt-AutoLaparo AutoLaparo Uterus Hysterectomy 1920x1080 60,000 15,000 75,000 SurgAtt-Hamlyn Hamlyn Centre Kidney Nephrectomy 384x192 68,478 14,380 82,858 SurgAtt-1.16M Aggregated Multiple Multiple laparoscopic procedures Multiple 928,478 229,380 1,157,858 Table 7: Two-stage curation and annotation pipeline for the SurgAtt-SZPH (SurgAtt-1M) subset. We first mine informative surgical segments via optical-flow–based motion filtering (A), then clinicians select high-quality clips under four clinical criteria (B), and finally surgeons annotate attention boxes that are converted into attention heatmaps at 25 fps across five surgical sites (C). Category Distal Gastric Proximal Gastric High Rectum Middle Rectum Low Rectum Total Split Train Test Train Test Train Test Train Test Train Test Patients 4 1 4 1 4 1 4 1 4 1 25 Videos 20 5 20 5 20 5 20 5 20 5 125 Frames 180K 45K 180K 45K 180K 45K 180K 45K 180K 45K 1.12M A.1.2 SurgAtt-1.16M Dataset Details Supplement. Dataset statistics and composition. Table 7 reports detailed statistics of the SurgAtt-SZPH subset, stratified by five surgical sites (Distal/Proximal Gastric and High/Middle/Low Rectum). We adopt a patient-level split for each site to avoid identity leakage: each site contains 5 patients, with 4 patients for training and 1 patient for testing. In total, SurgAtt-SZPH includes 25 patients and 125 surgical clips. The clips are obtained by sampling comparable-duration segments per patient and segmenting them into multiple shorter videos, yielding a balanced number of clips per site (Train/Test = 20/5) and a near-uniform frame budget per site (Train/Test = 180K/45K), for an overall size of 1.12M frames. These balanced per-site statistics enable controlled evaluation of attention tracking under different anatomical regions and surgical contexts. Full benchmark aggregation. Table 6 summarizes the full SurgAtt-1.16M benchmark, which aggregates SurgAtt-SZPH with re-labeled versions of AutoLaparo and Hamlyn. Specifically, AutoLaparo contributes uterus/hysterectomy cases, and Hamlyn provides kidney/nephrectomy cases with different visual characteristics and resolutions. We re-annotate these external sources under the same attention definition and heatmap generation protocol as SurgAtt-SZPH, so that all subsets can be evaluated with a unified metric suite and a consistent preprocessing pipeline. As a result, SurgAtt-1.16M expands organ/procedure coverage while preserving annotation consistency, supporting both in-domain studies on SurgAtt-SZPH and cross-domain generalization evaluation on heterogeneous surgical settings. A.2 Heatmap Generation Details Given a discrete sequence of per-frame attention boxes, we convert box annotations into dense ground-truth attention maps by (i) mapping each box to a box-aligned anisotropic Gaussian kernel, (i) temporally accumulating kernels with exponential decay to simulate visual persistence, and (i) applying smoothing and robust normalization to obtain a bounded heatmap in [0,1][0,1], as shown in Figure 6. Box-to-kernel mapping. For a frame t, each annotated box Bti=(cx,cy,w,h)B_t^i=(c_x,c_y,w,h) (center, width, height; in pixels after resizing to the evaluation resolution) is mapped to an anisotropic Gaussian kernel G(;ti,ti)G(x; μ_t^i, _t^i), with mean ti=(cx,cy) μ_t^i=(c_x,c_y) and diagonal covariance ti=diag(σx2,σy2) _t^i=diag( _x^2, _y^2). Following our implementation, we set σx=max(1,sw) _x= (1,\,s\,w) and σy=max(1,sh) _y= (1,\,s\,h) with scale factor s (default s=0.45s=0.45), and evaluate the kernel within a truncated window of ±3σ± 3σ along each axis for efficiency. When multiple boxes are present in the same frame, we aggregate them by summation: G~t()=∑ia(wi,hi)G(;ti,ti), G_t(x)\;=\; _i\,a(w_i,h_i)\;G(x; μ_t^i, _t^i), (13) where a(w,h)a(w,h) is an optional area compensation term to reduce scale bias. By default, we use a(w,h)=1/max(wh,1)a(w,h)=1/ (wh,1) (“sqrt”), which prevents large boxes from dominating the heatmap solely due to their area. Temporal accumulation (visual persistence). We simulate human visual persistence by exponentially decaying the previous density map and adding the current-frame kernel: t()=(1−α)t−1()+G~t(),0=,M_t(x)\;=\;(1-α)\,M_t-1(x)\;+\; G_t(x), _0=0, (14) where α∈(0,1)α∈(0,1) controls memory length (default α=0.22α=0.22). If frame t has no valid annotation, we simply apply the decay term, i.e., G~t≡0 G_t≡ 0. Post-processing and robust normalization. To reduce high-frequency artifacts and improve spatial coherence, we apply Gaussian smoothing with kernel size k (default k=9k=9): t=(t;k).S_t\;=\;G(M_t;k). (15) We then normalize by a robust upper bound using the p-th percentile Qp(t)Q_p(S_t) (default p=99.5p=99.5) to suppress outliers and stabilize dynamic range: t=clip(tmax(Qp(t),ϵ), 0, 1),H_t\;=\;clip\! ( S_t (Q_p(S_t),ε),\,0,\,1 ), (16) where ϵε is a small constant. The resulting t∈[0,1]H_t∈[0,1] is stored as a grayscale map and optionally visualized by overlaying a colorized heatmap on the resized frame. Implementation notes. All boxes are parsed from YOLO-format labels (normalized (cx,cy,w,h)(c_x,c_y,w,h)) and converted to pixel coordinates under a fixed output resolution (default 960×540960× 540). The kernel is peak-normalized and truncated to ±3σ± 3σ, and we optionally inflate the box scale prior to kernel generation (default inflation 0). Unless stated otherwise, we use α=0.22α=0.22, s=0.45s=0.45, k=9k=9, and p=99.5p=99.5 throughout. Figure 6: Visualization of heatmap generation. A.3 Evaluation Metric Definitions Let P∈[0,1]H×WP∈[0,1]^H× W be the predicted attention heatmap and G∈[0,1]H×WG∈[0,1]^H× W the ground-truth heatmap for a frame, both normalized to [0,1][0,1]. Let N=HWN=HW and denote by PiP_i and GiG_i the i-th pixel after vectorization. MAE. MAE(P,G)=1N∑i=1N|Pi−Gi|.MAE(P,G)= 1N _i=1^N |P_i-G_i |. (17) MSE. MSE(P,G)=1N∑i=1N(Pi−Gi)2.MSE(P,G)= 1N _i=1^N (P_i-G_i )^2. (18) C (Pearson correlation coefficient). Let μP=1N∑i=1NPi _P= 1N _i=1^NP_i and μG=1N∑i=1NGi _G= 1N _i=1^NG_i. Then C(P,G)=∑i=1N(Pi−μP)(Gi−μG)∑i=1N(Pi−μP)2∑i=1N(Gi−μG)2+ϵ,C(P,G)= _i=1^N(P_i- _P)(G_i- _G) _i=1^N(P_i- _P)^2\ _i=1^N(G_i- _G)^2+ε, (19) where ϵε is a small constant for numerical stability. SIM (histogram intersection). We first form ℓ1 _1-normalized non-negative maps: P^i=max(Pi,0)∑j=1Nmax(Pj,0)+ϵ,G^i=max(Gi,0)∑j=1Nmax(Gj,0)+ϵ. P_i= (P_i,0) _j=1^N (P_j,0)+ε, G_i= (G_i,0) _j=1^N (G_j,0)+ε. (20) If the denominator is zero (i.e., the map sums to zero), we use a uniform distribution P^i=1N P_i= 1N (and similarly for G G). SIM is then defined as SIM(P,G)=∑i=1Nmin(P^i,G^i).SIM(P,G)= _i=1^N ( P_i, G_i). (21) NSS (Normalized Scanpath Saliency). For NSS, we follow a standard saliency protocol by z-normalizing the prediction and averaging scores on ground-truth salient locations, where salient locations are defined as the top 5%5\% pixels of the ground-truth heatmap. We z-normalize the prediction map: Piz=Pi−μPσP+ϵ,σP=1N∑i=1N(Pi−μP)2.P_i^z= P_i- _P _P+ε, _P= 1N _i=1^N(P_i- _P)^2. (22) Since our supervision provides dense heatmaps, we construct a binary salient-location mask F∈0,1H×WF∈\0,1\^H× W from G by thresholding at the per-image top-quantile: Fi=[Gi≥Qq(G)],F_i=I\! [G_i≥ Q_q(G) ], (23) where Qq(G)Q_q(G) denotes the q-quantile of pixel values in G and we use q=0.95q=0.95 (top 5%5\% pixels) in all experiments. Finally, NSS(P,G)=1∑i=1NFi∑i=1NPizFi,NSS(P,G)= 1 _i=1^NF_i _i=1^NP_i^z\,F_i, (24) with the denominator clamped to at least 1 for numerical stability. Algorithm 1 Unified Training and Inference of SurgAtt-Tracker 1: Input: frames Itt=1T\I_t\_t=1^T; training only: GT boxes Btgt\B_t^gt\ 2: Mode: ∈train,infer mode∈\ train, infer\ 3: Train gaps: =1,2,4,8,16,32N=\1,2,4,8,16,32\, =0.4,0.2,0.1,0.1,0.1,0.1P=\0.4,0.2,0.1,0.1,0.1,0.1\ 4: Initialize B^1 B_1 (e.g., detector Top-1 on I1I_1) 5: for t=2t=2 to T do 6: if =train mode= train then 7: n←SampleGap(,)n (N,P), r←t−nr← t-n 8: else 9: n←1n← 1, r←t−1r← t-1 10: end if 11: (ℬr,ℱry)←D(Ir)(B_r,F_r^y)← D(I_r), (ℬt,ℱty)←D(It)(B_t,F_t^y)← D(I_t) 12: if =train mode= train then 13: Brs←BoxSelection(ℬr,Brgt)B_r^s (B_r,B_r^gt) 14: else 15: Brs←B^t−1B_r^s← B_t-1 16: end if 17: fr←MSR(ℱry,Brs)f_r (F_r^y,B_r^s) 18: ftkk=1K←MSR(ℱty,ℬt)\f_t^k\_k=1^K (F_t^y,B_t) 19: stkk=1K←AS-Rerank(fr,ftkk=1K)\s_t^k\_k=1^K -Rerank(f_r,\f_t^k\_k=1^K) 20: k^t←argmaxkstk k_t← _ks_t^k 21: B^t←Btk^t B_t← B_t k_t, f^t←ftk^t f_t← f_t k_t 22: Btr←MAA-Refine(f^t,Geo(B^t,Brs))B_t^r -Refine( f_t,Geo( B_t,B_r^s)) 23: Ht←Heat(ℱty,Btr)H_t (F_t^y,B_t^r) 24: end for A.4 Unified Training and Inference Pipeline Alg. 1 summarizes the end-to-end pipeline used by SurgAtt-Tracker for both training and online inference. Given a video clip Itt=1T\I_t\_t=1^T, a frozen detector D(⋅)D(·) is applied to each queried frame to produce (i) a Top-K proposal set ℬt=Btkk=1KB_t=\B_t^k\_k=1^K and (i) multi-scale neck features ℱtyF_t^y. The tracker then uses a reference frame r and a reference box BrsB_r^s to compute a reference ROI embedding frf_r, which conditions proposal reranking on the target frame t. Training gap sampling. Using only adjacent pairs (r=t−1r=t-1) during training may over-emphasize near-static transitions and encourage a shortcut where the refiner behaves almost as identity, i.e., simply propagating the previous spatial prior while still achieving low loss. To expose the model to diverse displacements and fast tool motion, we sample the temporal gap n from a categorical distribution n∼Cat(,)n (N,P) with =1,2,4,8,16,32N=\1,2,4,8,16,32\ and =0.4,0.2,0.1,0.1,0.1,0.1P=\0.4,0.2,0.1,0.1,0.1,0.1\, and set r=t−nr=t-n in training. Training mode. For each target frame t≥2t≥ 2, we sample a gap n and set r=t−nr=t-n (clipped to the valid range if necessary). We run the detector on both frames IrI_r and ItI_t to obtain (ℬr,ℱry)(B_r,F_r^y) and (ℬt,ℱty)(B_t,F_t^y). To avoid teacher-forcing mismatch while keeping the optimization stable, we construct the reference box BrsB_r^s via an oracle selection on frame r: among valid detector proposals, choose the one with minimum center error to the ground-truth box BrgtB_r^gt. Then we extract the reference ROI embedding fr=MSR(ℱry,Brs)f_r=MSR(F_r^y,B_r^s) and the target proposal embeddings ftkk=1K=MSR(ℱty,ℬt)\f_t^k\_k=1^K=MSR(F_t^y,B_t) using the same multi-scale ROI extractor MSR(⋅)MSR(·). The reranker AS-RerankAS -Rerank outputs scores stkk=1K\s_t^k\_k=1^K conditioned on frf_r, from which we select k^t=argmaxkstk k_t= _ks_t^k and obtain the Top-1 box B^t B_t and its embedding f^t f_t. Finally, the refiner MAA-RefineMAA -Refine predicts the refined box BtrB_t^r using the selected embedding and a motion-geometry descriptor Geo(B^t,Brs)Geo( B_t,B_r^s). All losses are computed on frame t only (reranking supervision and refinement regression), while r serves as a reference for conditioning. Inference mode. At test time, the same modules are executed online with a one-step reference (n=1n=1, r=t−1r=t-1), and the reference box is set to the previous prediction, Brs←B^t−1B_r^s← B_t-1. This matches the realistic tracking condition where only past predictions are available. We then rerank the Top-K proposals on ItI_t, select B^t B_t, refine it to BtrB_t^r, and optionally render the attention heatmap Ht=Heat(ℱty,Btr)H_t=Heat(F_t^y,B_t^r). A.5 Additional Experimental Setting and Results A.5.1 Experimental Setting Baseline implementation details. To ensure a fair and consistent comparison, we follow a unified evaluation protocol and adopt minimal, task-aligned adaptations for different categories of baselines. U-Net–based methods. For U-Net–based attention models, we strictly preserve the original network architectures and do not introduce any structural modifications. The only change is to replace the original loss with a binary cross-entropy (BCE) loss computed against the ground-truth attention heatmaps. All models are trained to directly predict a single-channel, normalized heatmap in [0,1][0,1], which is evaluated under the same metric suite as our method. Regression-based methods. For regression-based baselines, we do not alter either the network architecture or the original loss formulation. These models are trained and evaluated directly on our dataset following their default settings, with their predicted outputs converted into normalized attention heatmaps when necessary for metric computation. Object-tracking–based methods. For object-tracking approaches, we adopt a YOLO-style annotation format to ensure training consistency across datasets. During inference, we provide the ground-truth bounding box of the first frame as initialization, following standard tracking evaluation protocols. The tracker then propagates bounding boxes over time, and the resulting box sequences are converted into attention heatmaps using the same box-to-heatmap generation procedure as described in Section A.2. These heatmaps are evaluated using identical attention metrics. Detection-based methods. Detection-based baselines are trained using the YOLO format without architectural modification. At inference time, detected bounding boxes are transformed into attention heatmaps using the same generation strategy as above, enabling direct comparison with tracking- and heatmap-based approaches under a unified evaluation framework. Figure 7: Radar plots on five subsets (Distal/Proximal Gastric Cancer, Low/Middle/High Rectum) comparing representative SOTA methods over NSS/C/SIM (higher better) and MSE/MAE (lower better). Figure 8: Qualitative comparison of attention heatmap predictions on SurgAtt-SZPH across five surgical scenarios. SurgAtt-Tracker produces sharper and more stable attention aligned with clinically relevant regions compared with representative SOTA baselines. Figure 9: Qualitative comparison on SurgAtt-Hamlyn under zero-shot and fine-tuning settings. SurgAtt-Tracker shows more stable and better-localized attention than RT-DETRv2 and YOLOv12 across both regimes. Figure 10: Qualitative comparison on SurgAtt-AutoLaparo under zero-shot and fine-tuning settings. Our method maintains compact and consistent attention maps under domain shift and further improves after fine-tuning. A.5.2 Main Experiment Visualization and Analysis Per-scene analysis on SurgAtt-SZPH. Figure 7 visualizes a per-scene comparison across five SurgAtt-SZPH subsets, covering two gastric cancer scenes (distal/proximal) and three rectum scenes (low/middle/high). Each radar plot summarizes five complementary metrics: NSS, C, and SIM evaluate alignment between predicted and ground-truth attention distributions (higher is better), while MSE and MAE measure pixel-wise heatmap errors (lower is better). Across all scenes, SurgAtt-Tracker consistently forms the outer envelope on NSS/C/SIM while remaining closest to the optimal low-error region on MSE/MAE, indicating both stronger correlation and sharper localization. Notably, the improvements persist under diverse scene characteristics (e.g., varying anatomy, viewpoint changes, and instrument interactions), suggesting that our temporal-consistent reranking and motion-aware refinement generalize reliably beyond any single surgical condition. Qualitative analysis on SurgAtt-SZPH. Figure 4 presents qualitative comparisons of attention heatmap predictions on the SurgAtt-SZPH dataset across five representative surgical scenarios: Low Rectum, Middle Rectum, High Rectum, Distal Gastric Cancer, and Proximal Gastric Cancer. Each row shows the ground-truth attention heatmap and predictions from representative detection-, tracking-, saliency-, and segmentation-based baselines, together with our method. Across all scenarios, SurgAtt-Tracker produces attention maps that are both spatially concentrated and well aligned with clinically relevant regions. In rectal surgery scenes (A–C), where attention often shifts subtly along tissue boundaries or instrument–tissue interaction zones, our predictions closely match the ground truth in both location and extent. In contrast, detection-based methods (e.g., YOLOv12, RT-DETRv2) tend to generate coarse or overly centralized responses that fail to adapt to fine-grained attention shifts, while object-tracking methods (e.g., MCITrack) may drift toward instruments or background regions when visual cues become ambiguous. In gastric cancer scenarios (D–E), which typically involve larger anatomical structures and more pronounced camera motion, saliency- and segmentation-based methods (e.g., SalFoM, EMCAD) often produce diffuse or over-smoothed heatmaps, spreading attention across broad regions and reducing localization precision. Our method maintains compact, single-mode attention distributions that remain centered on the surgical target despite specular highlights, smoke, or partial occlusions. This robustness is attributed to the combination of proposal-level temporal reranking, which stabilizes attention selection over time, and motion-aware refinement, which constrains updates under rapid viewpoint changes. Overall, these qualitative results are consistent with the quantitative improvements reported in the main paper and appendix. They demonstrate that modeling surgical attention as temporally consistent retrieval over a high-recall proposal set, followed by motion-conditioned continuous refinement, yields attention heatmaps that are sharper, more stable, and more clinically interpretable across diverse surgical scenes. Zero-shot and fine-tuning generalization analysis. Figures 9 and 10 present qualitative comparisons on the SurgAtt-Hamlyn and SurgAtt-AutoLaparo datasets, evaluating three representative SOTA methods (RT-DETRv2, YOLOv12, and our approach) under both zero-shot and fine-tuning settings. These datasets differ substantially from SurgAtt-SZPH in organ type, surgical procedure, imaging resolution, and visual appearance, providing a challenging testbed for cross-domain generalization. In the zero-shot setting, SurgAtt-Tracker demonstrates strong robustness to domain shifts. On SurgAtt-Hamlyn, which involves kidney surgery with smaller target regions and frequent instrument occlusions, our method consistently produces compact and well-localized attention heatmaps that remain aligned with the clinically relevant regions. In contrast, detection-based baselines often exhibit either spatial drift or over-smoothed responses, particularly when specular highlights or motion blur dominate the scene. Similar trends are observed on SurgAtt-AutoLaparo, where changes in organ texture and instrument appearance cause competing methods to spread attention across large regions, while our predictions remain stable. Under the fine-tuning setting, all methods benefit from domain-specific supervision; however, notable qualitative differences persist. While RT-DETRv2 and YOLOv12 improve localization accuracy, their predictions often remain sensitive to background clutter or instrument dominance. SurgAtt-Tracker further sharpens attention boundaries and maintains temporal consistency, producing single-mode heatmaps that better match the ground truth across diverse frames. This suggests that the gains of our approach stem not only from fine-tuning but also from its design principles—namely, proposal-level temporal reranking and motion-aware refinement, which explicitly model attention evolution over time. These qualitative results indicate that SurgAtt-Tracker generalizes more effectively across datasets in the zero-shot regime and continues to provide stable, well-aligned attention estimates after fine-tuning. The observed behavior is consistent with our quantitative results and supports the effectiveness of decoupling attention selection from motion-conditioned refinement when transferring across heterogeneous surgical domains. A.6 Additional Ablation Study Figure 11: Top-K analysis of proposal ranking quality. AS-Rerank reorders the same YOLOv12 candidate pool using temporal consistency, increasing the likelihood that oracle-best proposals (Min-Err / Max-IoU) appear in the first few ranks. AS-Rerank contributes. We adopt a strong detector (YOLOv12) as a high-recall proposal generator rather than a final decision maker. In surgical scenes, detector confidence is often poorly calibrated to clinical attention quality due to specular highlights, smoke, blood occlusion, and frequent appearance changes of instruments and tissues. As a result, selecting the Top-1 box purely by confidence can miss the most attention-aligned region, even when it is already present in the Top-K candidate set. Figure 11 provides evidence for this design choice. The plot reports, for each Top-K, the fraction of frames where the best proposal (defined by oracle criteria such as minimum center error or maximum IoU w.r.t. the annotated box) appears within the top ranks. We observe that YOLOv12 indeed produces high-quality hypotheses early in the list, but these hypotheses are not consistently ranked as Top-1 under confidence. After applying AS-Rerank, the same candidate pool is reordered by temporal consistency, which increases the likelihood that the oracle-best proposal is promoted into the first few ranks (especially Top-1/Top-2). This directly explains why the reranking-only variant improves both localization quality and downstream heatmap metrics without changing the detector backbone: AS-Rerank does not create new boxes; it recovers better hypotheses already present in the candidate set by leveraging temporal coherence. Finally, building on the reranked Top-1 hypothesis, MAA-Refine further reduces residual localization error by conditioning the correction on motion cues from the reference state, yielding additional gains beyond reranking alone. Together, these results justify our two-stage design: a high-recall detector to provide diverse candidates, temporal reranking to select the most consistent attention hypothesis, and motion-aware refinement to obtain a precise and stable final prediction.