Paper deep dive
RefineRank: Joint Box Refinement and Ranking for Surgical Spatio-Temporal Grounding
Linzhe Jiang, Jiayuan Huang, Changhao Zhang, Chunyang Jiang, Zhehua Mao, Mobarak I. Hoque
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/26/2026, 5:05:52 AM
Summary
The paper introduces RefineRank, a method for Surgical Spatio-Temporal Grounding (STG) that combines a frozen medical vision language model (MedVLM) and a frozen open-set detector (GroundingDINO) using a compact trainable module called RefineNet. RefineNet predicts coordinate corrections and quality scores for candidate boxes, enabling precise localization and ranking without retraining the backbones. It achieves state-of-the-art results on MedVidBench.
Entities (10)
Relation Signals (7)
RefineRank → uses → RefineNet
confidence 100% · RefineRank realizes this idea... The single trainable component, RefineNet...
RefineRank → achievesscoreon → MedVidBench
confidence 95% · On the MedVidBench Official Rankings (Verified), RefineRank records 0.421 STG mIoU
RefineNet → combines → GroundingDINO
confidence 95% · RefineNet, combines ... with the proposals of a frozen open-set detector
RefineNet → combines → MedVLM
confidence 95% · RefineNet, combines the language and regional features of a frozen medical vision language model
MedVidBench → contains → CholecTrack20
confidence 90% · The controlled study uses a fixed split of the STG portion of the MedVidU training data, spanning CholecTrack20
MedVidBench → contains → CoPESD
confidence 90% · The controlled study uses a fixed split of the STG portion of the MedVidU training data, spanning ... CoPESD
MedVidBench → contains → EgoSurgery
confidence 90% · The controlled study uses a fixed split of the STG portion of the MedVidU training data, spanning ... EgoSurgery
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Surgical spatio-temporal grounding (STG) requires locating, at each video time specified by a procedural question, the object that the question asks about. Existing approaches face a trade-off: vision language models understand the question context but produce imprecise coordinates, whereas open-set detectors provide localized candidate boxes whose confidence does not reflect which box answers the question. We introduce RefineRank, which closes this gap at the candidate-box level. A compact trainable module, RefineNet, combines the language and regional features of a frozen medical vision language model with the proposals of a frozen open-set detector: it predicts a bounded coordinate correction and a quality score for every candidate box, and a fixed decoding rule returns the original or refined box with the highest score. On the MedVidBench Official Rankings (Verified), RefineRank records 0.421 STG mIoU, the highest displayed STG score, while its global multi-metric rank is 11. In a controlled evaluation on separate training and evaluation videos, coordinate correction raises the candidate oracle upper bound from 0.6772 to 0.7302, and ranking the joint pool of original and refined candidates by their RefineNet scores improves STG mIoU from 0.2719 to 0.4534, whereas separately trained selectors over the same pool reach at most 0.4186. These results show that a small box-level module can reconcile question understanding with precise localization without retraining either backbone. Code is available at [this https URL](this https URL).
Tags
Links
- Source: https://arxiv.org/abs/2608.23928v1
- Canonical: https://arxiv.org/abs/2608.23928v1
Trouble viewing inline? Open PDF directly →
Full Text
42,949 characters extracted from source content.
Expand or collapse full text
RefineRank: Joint Box Refinement and Ranking for Surgical Spatio-Temporal Grounding Linzhe Jiang †thanks: Corresponding author. Affiliation: UCL Hawkes Institute, University College London, London, UK E-mail linzhe.jiang.23@ucl.ac.uk Jiayuan Huang Affiliation: Visual Understanding Research Group, Department of Informatics, King’s College London, London, UK Changhao Zhang Affiliation: UCL Hawkes Institute, University College London, London, UK E-mail linzhe.jiang.23@ucl.ac.uk Chunyang Jiang Affiliation: School of Medicine, Nankai University, Tianjin, China Zhehua Mao Affiliation: UCL Hawkes Institute, University College London, London, UK E-mail linzhe.jiang.23@ucl.ac.uk Mobarak I. Hoque Affiliation: Division of Informatics, Imaging and Data Sciences, University of Manchester, Manchester, UK Abstract Surgical spatio-temporal grounding (STG) requires locating, at each video time specified by a procedural question, the object that the question asks about. Existing approaches face a trade-off: vision language models understand the question context but produce imprecise coordinates, whereas open-set detectors provide localized candidate boxes whose confidence does not reflect which box answers the question. We introduce RefineRank, which closes this gap at the candidate-box level. A compact trainable module, RefineNet, combines the language and regional features of a frozen medical vision language model with the proposals of a frozen open-set detector: it predicts a bounded coordinate correction and a quality score for every candidate box, and a fixed decoding rule returns the original or refined box with the highest score. On the MedVidBench Official Rankings (Verified), RefineRank records 0.421 STG mIoU, the highest displayed STG score, while its global multi-metric rank is 11. In a controlled evaluation on separate training and evaluation videos, coordinate correction raises the candidate oracle upper bound from 0.6772 to 0.7302, and ranking the joint pool of original and refined candidates by their RefineNet scores improves STG mIoU from 0.2719 to 0.4534, whereas separately trained selectors over the same pool reach at most 0.4186. These results show that a small box-level module can reconcile question understanding with precise localization without retraining either backbone. Code is available at https://github.com/linzhe001/RefineRank. Keywords: Surgical video Spatio-temporal grounding Box refinement Candidate ranking 1 Introduction Surgical spatio-temporal grounding (STG) answers procedural questions such as “which instrument is the surgeon holding at 01:35?” by locating the queried object in the video. The question names a target, a temporal interval, and a sampling step, which together determine an ordered set of requested video times; the system must return one object box at each requested time. The target may be a small instrument, an anatomical region, or a surgeon’s hand, and it can be occluded or visually similar to nearby objects. The task therefore requires the spatial, temporal, and language interactions emphasized by prior STG models [19, 7, 10], together with the domain language and fine visual distinctions of surgical video [16]. Two model families address complementary halves of this problem. Medical vision language models (MedVLMs), multimodal large language models adapted to medical images and video, can interpret complex clinical questions, but semantic understanding alone does not ensure accurate coordinates. Grounded multimodal models such as Kosmos-2, Shikra, and Ferret add location tokens, coordinate interfaces, regional representations, or dedicated grounding data to obtain spatial outputs [14, 2, 20]. Open-set detectors offer the complementary strength: given a text query, a detector such as GroundingDINO [11] returns a set of candidate boxes, each with coordinates and a confidence score. That confidence measures how well a box matches the detector query rather than how well it answers the complete timestamped surgical question, so the box that best answers the question is often not the highest-scoring one. Combining the two families is nontrivial. Learning a dense alignment between their feature spaces would require correspondences across different tokenizations, dimensions, spatial grids, and pretraining objectives, a problem that joint multimodal detectors address through grounded pretraining and internal cross-modal fusion [8, 11]. Learning a comparable alignment between two existing frozen backbones from surgical STG data is harder still. Candidate boxes offer a compact alternative: each proposed box already has coordinates, a detector score, and a matching region in the MedVLM visual features, so the two models can be connected through the boxes themselves rather than through their internal features. RefineRank realizes this idea. It keeps a frozen MedVLM and a frozen GroundingDINO detector and connects them only through the detector’s candidate boxes. The single trainable component, RefineNet, reads the MedVLM’s language and regional features at each candidate box and jointly learns two outputs: a bounded correction that improves the box coordinates, and a quality score that reflects how well the box answers the question. A fixed decoding rule with no learned parameters returns the highest-scoring candidate from the joint pool of original and refined boxes, so no separate selector module is needed. In this way, question understanding improves detector grounding without learning a dense alignment between the two feature spaces. Our evaluation separates the two questions this design raises: whether refinement creates better candidate boxes, and whether the learned scores select better boxes. RefineNet’s corrections raise the localization upper bound of the candidate pool, its scores improve final selection over detector confidence and over separately trained selectors, and a gap to the candidate oracle remains. We make three contributions. First, RefineRank, a complete grounding pipeline whose compact trainable RefineNet module jointly learns scores and corrections for detector candidates from MedVLM language and regional features while both backbones remain frozen. Second, a controlled evaluation that separates selection quality from localization potential by comparing the final prediction with the best available box. Third, a selector analysis showing that selectors trained only after RefineNet has modified the boxes do not improve on the built-in decoding rule of RefineRank. 2 Related Work Spatio-temporal video grounding. TubeDETR directly predicts temporally localized tubes with transformer queries [19]. STCAT uses a single stage to maintain spatial and temporal consistency [7]. CG-STVG coordinates static and dynamic vision language streams [10]. These systems learn dense video and language interactions. RefineRank instead uses RefineNet to improve and rank detector boxes before applying a fixed rule that selects one box at each requested time. The requested timestamps are known, so the model does not predict an entire tube directly from video tokens. The datasets and evaluators differ, so their published scores are not compared numerically here. Grounded vision language models. MDETR learns early multimodal fusion for detection conditioned on text [8]. GroundingDINO combines grounded pretraining with tight language and vision fusion for open-set detection [11]. Multimodal language models approach the same spatial problem through explicit grounding interfaces. Kosmos-2 represents regions with location tokens [14]. Shikra generates and accepts coordinates in natural language [2]. Ferret joins coordinates with continuous region features [20]. These works train grounding inside a single model. RefineRank instead forms a pipeline from two frozen models and the compact RefineNet module after GroundingDINO has proposed boxes that both models can refer to. Medical video grounding. MedGRPO introduces MedVidBench and releases the uAI-NEXUS-MedVLM-1.0a-7B-RL checkpoint [16]. RefineRank uses this checkpoint as its frozen MedVLM, which supplies the detector query and multimodal features, together with a separate frozen GroundingDINO detector. The MedVLM + GroundingDINO baseline uses that query and ranks boxes by detector confidence; it is not a trained fusion architecture. RefineNet is the only trainable component that combines their outputs. Localization quality and box refinement. Cascade R-CNN progressively raises proposal quality [1]. IoU-Net predicts localization confidence [6]. GFL and VarifocalNet align dense detector ranking with localization quality [9, 21]. The same distinction between localization and ranking is relevant to surgical STG. Unlike a detector head trained with its feature pyramid, RefineNet operates on precomputed detector boxes and frozen multimodal features. Its two quality outputs assign scores to the original and refined versions of each proposal. 3 Method The complete pipeline of RefineRank is shown in Figure 1: the frozen MedVLM, frozen GroundingDINO, the trainable RefineNet module, and a fixed decoding rule that returns the candidate with the highest score. The MedVLM supplies question and regional features, GroundingDINO supplies box locations, and only RefineNet is optimized. A GroundingDINO detection before correction is called an original detector box, or simply an original box. Its corrected version is called a refined box. Figure 1: The complete RefineRank pipeline. The frozen MedVLM reads the complete timestamped question and sampled frames. It supplies the detector query, qlastq_ last, and two visual grids. GroundingDINO applies the query to the requested frames and provides original boxes and their metadata fdino,if_ dino,i. ROI pooling produces riinterr_i inter and rifinalr_i final for the trainable RefineNet module. Its ranking head hrankh_ rank outputs logits sis_i and s^i s_i, while the box head hboxh_ box outputs the correction i δ_i. The final pool contains original boxes scored by sis_i and refined boxes scored by s^i s_i. At each requested time, the fixed decoding rule returns the candidate with the highest corresponding score; it has no learned parameters and no separate selector module is used. 3.1 Problem Setting The input question specifies a target, a temporal interval, and a sampling step. These values determine an ordered set of requested times. The system must return one axis-aligned box at each time, and STG mIoU averages box IoU over the requested times. During training, g denotes the target box. Each frame contains several original boxes, and detector confidence does not necessarily identify the box that answers the complete question. RefineNet is the only trained component: GroundingDINO box generation, MedVLM feature extraction, and final box selection remain fixed. This design also bounds what the pipeline can recover. The complete system can rank and locally correct an available original box, but it cannot locate a target if GroundingDINO does not propose a box that covers it. 3.2 Frozen Models and Original Detector Boxes The two frozen backbones provide complementary inputs: question understanding and visual features from the MedVLM, and candidate boxes from GroundingDINO. The frozen MedVLM receives the complete timestamped question and sampled video frames. As illustrated in Figure 1, it analyzes the question and extracts target keywords as the detector query, exposes its final language state qlastq_ last (the last-layer hidden state at the final prompt token), and supplies intermediate and final visual grids. Frozen GroundingDINO [11] applies the detector query to each requested RGB frame. Low detection thresholds favor recall. At most 12 raw detections are retained. When a detection covers a broad region, fixed smaller boxes derived from it are also added. Duplicate or invalid boxes are removed before ranking. The original box set is capped at 128 boxes per requested frame and is constructed without target boxes. Each original box bi=(x1i,y1i,x2i,y2i)b_i=(x_1i,y_1i,x_2i,y_2i) is described by a metadata vector fdino,i∈ℝ24f_ dino,i ^24 comprising 12 values and their 12 missing-value indicators. The values are the detector confidence; the upstream temporal selection score; tube coverage, defined as the fraction of frames with a detection; tube smoothness, defined as one minus the mean consecutive-box center displacement normalized by the frame diagonal; the normalized time offset |ti−t|/T|t_i-t|/T and exact-match indicator [|ti−t|≤10−6]1[|t_i-t|≤ 10^-6]; the four normalized box coordinates; and the log area and log aspect ratio. Here, tit_i and t denote the candidate and requested times, and T is the span of the sampled frames. Because GroundingDINO is applied independently to each requested frame, every tube contains one requested time: its temporal selection score and time offset are zero, whereas its coverage, smoothness, and exact-match values are one. These tube-level fields therefore preserve compatibility with multi-frame tubes but do not vary across candidates in this pipeline. The values are standardized with training-set statistics; an unavailable value is set to zero after standardization and its indicator is set to one. Thus, fdino,if_ dino,i describes detector output and box geometry rather than an internal GroundingDINO feature. The MedVLM is the uAI-NEXUS-MedVLM-1.0a-7B-RL checkpoint released by MedGRPO [16]. For each requested frame, the same frozen model processes that frame with the complete timestamped question. Its final language state is qlast∈ℝ3584q_ last ^3584. The intermediate visual grid at block 23 and the final grid after visual token merging are retained. ROI pooling weighted by area over the grid cells intersecting bib_i gives riinter∈ℝ1280r_i inter ^1280 from block 23 and rifinal∈ℝ3584r_i final ^3584 from the final grid. Both features are pooled at the original box coordinates. They are not recomputed after box refinement. Weighting cells by their exact overlap with the box avoids rounding a small box to a single nearest patch. 3.3 Joint Ranking and Refinement RefineNet turns each candidate box into a joint score and correction from the four inputs shown in Figure 1: qlastq_ last, riinterr_i inter, rifinalr_i final, and fdino,if_ dino,i. The query and regional features are ℓ2 _2-normalized, and separate linear maps project all four inputs to 128 dimensions. The projected regional and metadata features are added. An MLP with two layers combines this box representation with the projected query and their elementwise product. The ranking head hrankh_ rank outputs logits sis_i and s^i s_i for the original and refined boxes. The box head hboxh_ box outputs i=(δx,i,δy,i,δw,i,δh,i) δ_i=( _x,i, _y,i, _w,i, _h,i). The scores are logits. A sigmoid is applied only when a probability is needed for score calibration or candidate filtering. Since the sigmoid is monotonic, it does not change their ranking. A componentwise tanh bounds the center offsets δx,i _x,i and δy,i _y,i to [−0.5,0.5][-0.5,0.5] and the logarithmic scale offsets δw,i _w,i and δh,i _h,i to [−log2,log2][- 2, 2]. Let the original box bib_i have center (cx,i,cy,i)(c_x,i,c_y,i) and size (wi,hi)(w_i,h_i). RefineNet decodes the corrected center and size as cx,i′ c _x,i =cx,i+δx,iwi, =c_x,i+ _x,iw_i, cy,i′ c _y,i =cy,i+δy,ihi, =c_y,i+ _y,ih_i, (1) wi′ w _i =wiexp(δw,i), =w_i ( _w,i), hi′ h _i =hiexp(δh,i). =h_i ( _h,i). The result is converted to corner coordinates to obtain the refined box bi′b _i. Coordinates are clipped to the normalized image range. Boxes with non-finite coordinates or non-positive area are discarded. The same bounds are used when the target g is encoded relative to bib_i for regression. Original and refined IoUs, yi=IoU(bi,g)y_i=IoU(b_i,g) and y^i=IoU(bi′,g) y_i=IoU(b _i,g), supervise the two logits. Each score is trained with the same RankingLoss, ℒrank(s,y)=−∑i∈pilogqi+λcalℓSL1(σ(s),y),L_ rank(s,y)=- _i p_i q_i+ _ cal\, _ SL1 (σ(s),y ), (2) where V is the set of valid (non-padded) boxes of one example, p=softmax(y/τ)p=softmax(y/τ) with τ=0.1τ=0.1 and q=softmax(s)q=softmax(s) are distributions over V, σ is the sigmoid, and ℓSL1 _ SL1 is the Smooth L1 loss [5] between the sigmoid scores and the IoU targets, averaged over V and weighted by λcal=0.25 _ cal=0.25. The listwise term is averaged over examples with at least one positive target IoU; examples with maxiyi=0 _iy_i=0 contribute only the calibration term. The refined IoU target is detached from box decoding. Box supervision uses ℒbox=ℓSL1(ℐ,ℐ∗)+12ℓGIoU(bℐ′,g),L_ box= _ SL1 ( δ_I, δ_I^* )+ 12\, _ GIoU (b _I,g ), (3) where ℐI contains the eight valid original boxes with the highest yiy_i, the subscript ℐI denotes averaging over these boxes, i∗ δ_i^* encodes g relative to bib_i under the bounds of Eq. 1, and ℓGIoU=1−GIoU _ GIoU=1-GIoU uses the generalized IoU [15]. The total objective is the sum of the two ranking losses and the box loss. Batches are padded to at most 128 boxes. A Boolean mask removes padded boxes before softmax and from every loss. Algorithm 1 Training RefineNet 1: Frozen features qlastq_ last, rinterr inter, rfinalr final, fdinof_ dino, original boxes b, targets g, and masks for valid boxes 2: RefineNet parameters θ and optimizer O 3: Trained parameters θ 4: for each minibatch do 5: (s,s^,)←RefineNetθ(qlast,rinter,rfinal,fdino)(s, s, δ) _θ(q_ last,r inter,r final,f_ dino) 6: b′←Decode(b,)b ← Decode(b, δ) 7: y←IoU(b,g)y (b,g) 8: y^←StopGrad(IoU(b′,g)) y← StopGrad(IoU(b ,g)) 9: ℐ←TopKValid(y,8)I← TopKValid(y,8) 10: ℒrank←RankingLoss(s,y)+RankingLoss(s^,y^)L_ rank← RankingLoss(s,y)+ RankingLoss( s, y) 11: ℒbox←BoxLoss(b,b′,,g,ℐ)L_ box← BoxLoss(b,b , δ,g,I) 12: ℒ←ℒrank+ℒboxL _ rank+L_ box 13: ZeroGrad() ZeroGrad(O) 14: Backward(ℒ) Backward(L) 15: ClipGradNorm(θ,1) ClipGradNorm(θ,1) 16: Step() Step(O) 17: end for Algorithm 1 applies Eq. 1 in Decode. RankingLoss is the objective of Eq. 2. TopKValid excludes padding and returns at most eight boxes per example. BoxLoss is the objective of Eq. 3 over these boxes. Only θ enters the optimizer. The cached MedVLM and GroundingDINO outputs receive no gradient. 3.4 Candidate Pool and Final Selection At inference, RefineRank keeps both versions of every candidate so that an accurate original box is never forced to move: the correction head produces (bi′,s^i)(b _i, s_i) alongside each original candidate and its score (bi,si)(b_i,s_i). A fixed filter first keeps 16 original boxes, then greedily fills the remaining positions, up to 48 candidates, from the union of the original and refined boxes. Writing s(c)s(c) and b(c)b(c) for the RefineNet score and the box of candidate c, each step adds the candidate that maximizes U(c)=λqσ(s(c))+λdd(c∣),d(c∣)=1−maxc′∈IoU(b(c),b(c′)),U(c)= _ q\,σ (s(c) )+ _ d\,d(c ), d(c )=1- _c IoU (b(c),b(c ) ), (4) where S is the set of already selected candidates, σ is the sigmoid, and the quality and diversity weights are λq=0.7 _ q=0.7 and λd=0.3 _ d=0.3. The diversity term d(c∣)d(c ) is one minus the largest IoU between b(c)b(c) and any selected box, so it favors candidates that do not duplicate an already selected location. Final selection requires no learned module. Let tC_t be the retained candidates at requested time t. For c∈tc _t, its RefineNet score is sis_i when c=bic=b_i and s^i s_i when c=bi′c=b _i, and the decoding rule simply returns argmaxc∈ts(c) _c _ts(c). This rule over the joint pool of original and refined boxes defines the primary result in Sec. 5; no additional selector is part of the final RefineRank prediction. The selector ablation replaces only these scores with separately trained scoring rules while keeping the same candidate pool and argmax decoder. 4 Experimental Protocol 4.1 Training and Evaluation Data Table 1: Video-separated split of the controlled study. Each example is one requested timestamp with its sampled frame, question, and target box. The closed MedVidBench leaderboard test† split is not included. Videos Examples Dataset Train Eval Train Eval CholecTrack20 7 2 807 212 CoPESD 5 2 134 118 EgoSurgery 11 3 1405 624 Total 23 7 2,346 954 The daggered test label denotes the closed MedVidBench leaderboard test split: its ground-truth annotations are hidden and scores are returned only through the benchmark server. The controlled study uses a fixed split of the STG portion of the MedVidU training data, spanning CholecTrack20 [13], CoPESD [18], and EgoSurgery [3]. Each training example contains one requested timestamp, its sampled frame, the complete question, and one target box. The split covers 30 videos and 3,300 examples in total, divided into 23 training videos with 2,346 examples and 7 evaluation videos with 954 examples (Table 1). Videos are assigned to either training or evaluation, never both. This assignment is shared by every controlled comparison. Results are reported per dataset and by their simple mean. The leaderboard test† split is held out from this controlled split and is not used for training or offline evaluation. 4.2 Box and Feature Preparation Detector queries, original boxes, and MedVLM features are computed before training RefineNet. The same frozen MedVLM supplies the detector query and the features illustrated in Figure 1. The same precomputed detector boxes are used throughout the comparison. RefineNet adds one refined box for every original box before applying the predefined filtering step. Padding is excluded from evaluation. Each MedVLM forward receives the requested RGB frame and the complete timestamped question. The stored outputs are the final language state, the visual grid after token merging, and visual blocks 7, 15, 23, and 31. Only block 23 and the final output are consumed by the main model. Blocks 7, 15, and 31 are retained for the controlled layer comparison. Region features are pooled at the coordinates of the original box. The refined box is not sent through the backbone again. GroundingDINO and MedVLM remain in evaluation mode throughout feature extraction. 4.3 Baselines and Reporting The direct MedVLM baseline parses coordinates generated by the frozen checkpoint. Invalid coordinate generations yield no box. The MedVLM + GroundingDINO baseline applies the MedVLM-derived query and selects from the original boxes using detector confidence. RefineRank constructs the joint pool of original and refined boxes described in Sec. 3, ranks every retained candidate by its RefineNet score, and returns the box with the highest score. No additional selector is trained for this prediction. The selector ablation uses the same pool and argmax decoder but replaces the RefineNet scores with scores from separately trained selectors that are not components of RefineRank. STG mIoU is computed at the annotated requested times for every method. The candidate oracle is also reported for diagnosis. It uses the target annotation to choose the box with the greatest IoU at each time and is not a usable prediction method. It measures the best result possible with the available boxes. For the MedVLM + GroundingDINO baseline, the candidate oracle considers only original boxes. For RefineRank, it considers the union of original and refined boxes and therefore measures the localization potential added by the correction head. The public comparison uses the MedVidBench Official Rankings (Verified) snapshot accessed on 15 August 2026.11 1 https://huggingface.co/spaces/UII-AI/MedVidBench-Leaderboard The official leaderboard provides a global position obtained by averaging per-metric ranks over ten metrics. Because this work concerns STG, Table 2 instead orders verified entries by STG mIoU and reports the top five. RefineRank appears under the exact submission name uAI-NEXUS-MedVLM-1.0a-7B-RL-STG_final. 4.4 Training and Controls RefineNet has 1,251,334 trainable parameters and is trained for 40 epochs with AdamW [12]. The learning rate is 3×10−43× 10^-4, weight decay is 10−210^-2, and the gradient norm is clipped to 1. Batches are balanced across the three datasets. All query and ROI tensors are precomputed and receive no gradients, and the optimizer contains no GroundingDINO or MedVLM parameter. The feature study uses a separate MLP that is trained for 10 epochs. It operates on the joint pool of original and refined candidates produced by the trained and frozen RefineNet: RefineNet supplies the refined boxes, but the MLP re-scores every candidate independently and neither its input features nor the selection rule use the RefineNet scores. Unlike the 40-epoch RefineNet, it does not adjust boxes. Its purpose is to compare input features, so it is not a variant of RefineNet. The training and evaluation sets, the candidate pool, and final selection rule are held fixed while the MLP input changes from metadata alone to combinations with qlastq_ last and regional features from visual blocks 7, 15, 23, and 31. For the selector ablation, RefineNet is first trained and then frozen. Its box head generates the refined boxes, which are combined with the original boxes to form the same fixed candidate pool used by RefineRank. ExtraTrees [4], an MLP that scores each box independently from the metadata, query, and block-23 regional features, and a standard Transformer encoder [17] are then trained separately on this fixed output. No selector receives the RefineNet scores as an input feature. They do not update RefineNet and are not components of RefineRank. Table 4 includes RefineRank as the reference row: no selector is trained, and its decoding rule ranks the candidates directly by RefineNet’s own scores. Each ablation selector instead supplies replacement scores to the same argmax decoder. 5 Results 5.1 MedVidBench Official Benchmark Table 2: MedVidBench Official Rankings (Verified), accessed 15 August 2026. Entries are ordered by STG mIoU, and the top five are shown. MedVidBench Leaderboard STG mIoU uAI-NEXUS-MedVLM-1.0a-7B-RL-STG_final (ours) 0.421 uAI-NEXUS-MedVLM-1.0a-7B-RL [16] 0.202 uAI-NEXUS-MedVLM-1.0c-4B-SFT 0.190 uAI-NEXUS-MedVLM-1.0a-7B-SFT 0.177 uAI-NEXUS-MedVLM-1.0b-4B-RL 0.176 Table 2 lists the five highest STG mIoU results in the official snapshot. RefineRank ranks first on this metric with 0.421, while its global leaderboard rank is 11 because the global ordering aggregates ten metrics. 5.2 Does Refinement Create Better Candidate Boxes? Table 3: Controlled STG mIoU on separate training and evaluation videos. Dataset averages weight the three datasets equally. Method Dataset STG mIoU Selected avg. Oracle avg. Cholec CoPESD Ego MedVLM [16] 0.0929 0.0170 0.0190 0.0429 0.0429 MedVLM + GroundingDINO [16, 11] 0.1350 0.3371 0.3435 0.2719 0.6772 RefineRank (ours) 0.3960 0.5972 0.3671 0.4534 0.7302 Refinement expands what the candidate pool can contain. With the original GroundingDINO boxes alone, the candidate oracle in Table 3, the best available box chosen with annotations at each requested time—reaches 0.6772. Adding the refined boxes produced by RefineNet’s correction head raises this upper bound to 0.7302. The correction head therefore adds genuine localization potential beyond what the frozen detector proposes on its own. 5.3 Does the Learned Scoring Function Select Better Boxes? A well-localized box helps only if it is also selected. MedVLM-guided GroundingDINO often proposes a useful box but does not assign it the highest detector confidence: selecting by detector confidence gives an average of 0.2719, far below the 0.6772 candidate oracle over the same original boxes (Table 3). Good candidates are thus often present but poorly ranked for the complete surgical question. RefineNet’s learned scores close part of this gap. Ranking the joint pool of original and refined boxes by their RefineNet scores reaches 0.4534, a gain of 0.1815 over the MedVLM + GroundingDINO baseline, with the largest improvements on CholecTrack20 and CoPESD. RefineRank therefore improves both the candidate boxes and their ranking, although a substantial gap to the 0.7302 candidate oracle remains. 5.4 Do Separately Trained Selectors Improve RefineRank? Table 4: Selector study on the same joint pool of original and refined candidates generated after RefineNet training. RefineNet is frozen for every row. RefineRank trains no additional selector: its decoding rule, which has no learned parameters, uses RefineNet’s own scores. ExtraTrees, MLP, and the Transformer encoder are separately trained replacement scoring rules and are not RefineRank modules. None of them receives the RefineNet scores as an input feature. The MLP combines the metadata, query, and block-23 regional features; it is the same configuration as the block-23 row of Table 5 and therefore reports the same values. Candidate scoring rule Dataset STG mIoU Dataset avg. Cholec CoPESD Ego ExtraTrees [4] 0.2842 0.4243 0.2781 0.3289 MLP 0.3373 0.5767 0.3417 0.4186 Transformer encoder [17] 0.3447 0.5415 0.3261 0.4041 Native RefineNet scores (RefineRank) 0.3960 0.5972 0.3671 0.4534 Better refined boxes do not make selection automatic. RefineRank’s decoding rule uses RefineNet’s own scores directly and reaches 0.4534 on the joint pool. After RefineNet training is complete, the model is frozen and the strongest separately trained selector ablation is the MLP at 0.4186; ExtraTrees and the Transformer encoder also remain below RefineRank. Because all four rows use the same RefineNet-generated candidate pool and argmax decoder, the comparison isolates whether replacing RefineNet’s own scores with an additional learned scoring rule improves selection. In this study, none does. The result indicates that RefineNet already learns the strongest evaluated box-quality signal, while the remaining candidate oracle gap shows that its scores still do not always promote the best available box. 6 Ablations and Qualitative Analysis 6.1 Input Feature Study with a Separate Box Scorer Table 5: Input feature study with a separately trained box scoring MLP. The candidate pool is the joint pool of original and refined boxes generated by the trained and frozen RefineNet, and every row re-scores all of its candidates after 10 epochs of MLP training. The RefineNet scores are used neither as input features nor at selection time. Candidate MLP input Dataset STG mIoU Dataset avg. Cholec CoPESD Ego mean fdinof_dino 0.1512 0.3578 0.3213 0.2767 fdinof_dino + qlastq_last 0.2528 0.6069 0.3536 0.4044 fdinof_dino + qlastq_last + visual block 7 0.3779 0.5488 0.3260 0.4176 fdinof_dino + qlastq_last + visual block 15 0.3537 0.5411 0.3558 0.4169 fdinof_dino + qlastq_last + visual block 23 0.3373 0.5767 0.3417 0.4186 fdinof_dino + qlastq_last + visual block 31 0.2622 0.6040 0.3213 0.3959 A separate 10-epoch MLP provides the input study reported in Table 5. It is not an ablation of hrankh_ rank in the trained RefineNet module. RefineNet is trained and frozen first, and its correction head generates the refined half of the candidate pool; the MLP then re-scores every candidate of this joint pool and changes one feature input at a time. Ranking uses only the MLP’s own scores: the RefineNet scores sis_i and s^i s_i are neither MLP inputs nor selection signals. Its values need not match the RefineRank results in Table 3. The block-23 row is the same configuration as the MLP row of Table 4, so the two tables report identical values for it. The fdinof_ dino row uses the 24 dimensional metadata vector and is not the raw GroundingDINO confidence baseline. Within this diagnostic model, adding qlastq_ last raises the average from 0.2767 to 0.4044. Intermediate visual features provide a smaller additional gain. Block 23 has the highest equally weighted average, but blocks 7, 15, and 23 remain closely grouped and their order varies by dataset. Block 23 is used as the aggregate choice, without treating one depth as universally superior. 6.2 Spatial Response Across Depth The response maps in Figure 2 show localized evidence around the displayed tools in intermediate blocks, whereas block 31 is less spatially differentiated in these examples. This visual pattern is consistent with the aggregate ablation but does not explain it causally. The target is used only to construct this response map, never as a RefineNet input, and each map is normalized independently. Neither color intensity nor spatial spread should be interpreted as calibrated confidence or attention. Figure 2: Spatial responses across frozen MedVLM visual blocks 7, 15, 23, and 31. For this visualization only, target region features are pooled after the multimodal forward and compared with every spatial token. Each map is normalized independently and is not an attention map or a causal explanation. 6.3 Examples from the Same Frame Figure 3: Six representative evaluation examples. Columns show direct MedVLM, MedVLM + GroundingDINO, RefineRank, and target boxes. The bottom two rows show shared failures. A displayed RefineRank prediction may therefore be either an original box or its refined counterpart, whichever received the higher score. The examples in Figure 3 illustrate how selecting an original box and refining its position are complementary. A refined box improves one hand localization, while several examples are solved by assigning a higher score to a better original detector box without moving it. In the two shared failures, the available proposals do not support a correct selection. These examples illustrate the correction and scoring roles but do not estimate how frequently either behavior occurs. 6.4 Limitations The controlled analysis uses one fixed video-separated split so that all methods share the same data assignment. Evaluation on additional splits would further establish stability and is left to future work. The leaderboard result is taken from the MedVidBench Official Rankings (Verified) snapshot accessed on 15 August 2026. No learned fusion baseline between MedVLM and GroundingDINO was trained or evaluated. Consequently, the controlled comparison shows an improvement over direct MedVLM coordinates and the MedVLM + GroundingDINO baseline, but it does not establish that fusion based on boxes is superior to other learned fusion designs. Regional features are pooled only at the original detector coordinates. A refined box is not encoded again, so its score cannot use visual content newly included or excluded by the correction. RefineRank also cannot recover a target when GroundingDINO does not propose a box that covers it. Candidate oracle values use target annotations and only show an upper bound, while the spatial response maps are descriptive rather than explanations of individual scores. 7 Conclusion RefineRank is a complete pipeline that combines the question understanding of a frozen MedVLM, the localized candidate boxes of a frozen GroundingDINO detector, and the compact trainable RefineNet module. A fixed decoding rule then ranks the joint pool of original and refined candidates by their RefineNet scores and returns the box with the highest score, with no additional selector module. This built-in rule is the strongest evaluated selection policy, and selectors trained separately on the fixed RefineNet outputs do not improve it. The gap to the candidate oracle therefore reflects both localization and ranking errors. Further evaluation should cover other video sets and learned fusion baselines. References [1] Z. Cai and N. Vasconcelos (2018) Cascade R-CNN: delving into high quality object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 6154–6162. External Links: Document, 1712.00726 Cited by: §2. [2] K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao (2023) Shikra: unleashing multimodal LLM’s referential dialogue magic. arXiv preprint arXiv:2306.15195. External Links: 2306.15195, Link Cited by: §1, §2. [3] R. Fujii, H. Saito, and H. Kajita (2024) EgoSurgery-Tool: a dataset of surgical tool and hand detection from egocentric open surgery videos. arXiv preprint arXiv:2406.03095. External Links: 2406.03095 Cited by: §4.1. [4] P. Geurts, D. Ernst, and L. Wehenkel (2006) Extremely randomized trees. Machine Learning 63 (1), p. 3–42. External Links: Document Cited by: §4.4, Table 4. [5] R. Girshick (2015) Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), p. 1440–1448. External Links: Document, 1504.08083 Cited by: §3.3. [6] B. Jiang, R. Luo, J. Mao, T. Xiao, and Y. Jiang (2018) Acquisition of localization confidence for accurate object detection. In Computer Vision – ECCV 2018, p. 816–832. External Links: Document, 1807.11590 Cited by: §2. [7] Y. Jin, Y. Li, Z. Yuan, and Y. Mu (2022) Embracing consistency: a one-stage approach for spatio-temporal video grounding. In Advances in Neural Information Processing Systems, Vol. 35, p. 29192–29204. External Links: Document, 2209.13306 Cited by: §1, §2. [8] A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion (2021) MDETR: modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 1780–1790. External Links: Document, 2104.12763 Cited by: §1, §2. [9] X. Li, W. Wang, L. Wu, S. Chen, X. Hu, J. Li, J. Tang, and J. Yang (2020) Generalized focal loss: learning qualified and distributed bounding boxes for dense object detection. In Advances in Neural Information Processing Systems, Vol. 33, p. 21002–21012. External Links: 2006.04388 Cited by: §2. [10] Z. Lin, C. Tan, J. Hu, Z. Jin, T. Ye, and W. Zheng (2023) Collaborative static and dynamic vision-language streams for spatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 23100–23109. External Links: Document Cited by: §1, §2. [11] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang (2024) Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. In Computer Vision – ECCV 2024, p. 38–55. External Links: Document, 2303.05499 Cited by: §1, §1, §2, §3.2, Table 3. [12] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: 1711.05101 Cited by: §4.4. [13] C. I. Nwoye, K. Elgohary, A. Srinivas, F. Zaid, J. L. Lavanchy, and N. Padoy (2025) CholecTrack20: a multi-perspective tracking dataset for surgical tools. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: 2312.07352 Cited by: §4.1. [14] Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and F. Wei (2023) Kosmos-2: grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824. External Links: 2306.14824, Link Cited by: §1, §2. [15] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese (2019) Generalized intersection over union: a metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 658–666. External Links: Document, 1902.09630 Cited by: §3.3. [16] Y. Su, A. Choudhuri, Z. Gao, B. Planche, V. N. Nguyen, M. Zheng, Y. Shen, A. Innanje, T. Chen, E. Elhamifar, and Z. Wu (2026) MedGRPO: multi-task reinforcement learning for heterogeneous medical video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Note: Accepted at CVPR 2026 External Links: 2512.06581, Link Cited by: §1, §2, §3.2, Table 2, Table 3, Table 3. [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30, p. 5998–6008. External Links: 1706.03762 Cited by: §4.4, Table 4. [18] G. Wang, H. Xiao, H. Gao, R. Zhang, L. Bai, X. Yang, Z. Li, H. Li, and H. Ren (2024) CoPESD: a multi-level surgical motion dataset for training large vision-language models to co-pilot endoscopic submucosal dissection. arXiv preprint arXiv:2410.07540. External Links: 2410.07540 Cited by: §4.1. [19] A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid (2022) TubeDETR: spatio-temporal video grounding with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 16442–16453. External Links: Document, 2203.16434 Cited by: §1, §2. [20] H. You, H. Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao, S. Chang, and Y. Yang (2024) Ferret: refer and ground anything anywhere at any granularity. In International Conference on Learning Representations, External Links: 2310.07704, Link Cited by: §1, §2. [21] H. Zhang, Y. Wang, F. Dayoub, and N. Sunderhauf (2021) VarifocalNet: an IoU-aware dense object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8514–8523. External Links: Document, 2008.13367 Cited by: §2.