Paper deep dive

MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning

Yingying Fan, Penghui Du, Leyan Zhu, Runze He, Zimeng Wu, Yuxuan Zhang, Liang Chen, Jiahao Xie, Jiangtang Wang, Shuai Shao, Anchao Yang, Yutong Bai, Yan Wang

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 67

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/17/2026, 4:48:50 AM

Summary

The paper introduces MedClaw, a training-free agent harness for long-horizon temporal reasoning in surgical videos. It separates reasoning (text orchestrator) from perception (frozen vision-language sub-agents) and uses a Heuristic Skill Distillation (HSD) loop to evolve context via reusable skills rather than optimizing model weights. The authors also introduce MedClawBench, a benchmark of 1,123 doctor-grounded questions for evaluating long-horizon reasoning on neurosurgery and lecture videos.

Entities (8)

MedClaw · method · 98%MedClawBench · dataset · 97%Heuristic Skill Distillation · method · 95%Yingying Fan · person · 95%Beijing Jiaotong University · organization · 90%directed re-look · skill · 90%SVU-31K · dataset · 90%Opus 4.8 · model · 85%

Relation Signals (6)

MedClaw → evaluatedon → MedClawBench

confidence 97% · To evaluate this agent, we introduce MedClawBench... Across both datasets... our agent consistently outperforms

MedClaw → uses → Heuristic Skill Distillation

confidence 95% · We further propose a gradient-free, reward-gated Heuristic Skill Distillation loop... MedClaw keeps every model frozen and instead distills reusable skills through Heuristic Skill Distillation

Yingying Fan → affiliatedwith → Beijing Jiaotong University

confidence 90% · Yingying Fan is with the School of Automation and Intelligence, Beijing Jiaotong University

MedClawBench → containsdatafrom → SVU-31K

confidence 90% · 864 from the held-out test split of the public SVU-31K corpus

Heuristic Skill Distillation → produces → directed re-look

confidence 90% · yielding reusable retrieval skills, notably directed re-look

MedClaw → usesmodel → Opus 4.8

confidence 85% · The orchestrator is a text-only policy π θ (Opus-4.8)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Understanding tens-of-minutes surgical videos requires long-horizon temporal reasoning, answering what happens before, after, or across stages of a procedure by grounding the question in visual evidence spread across time. Existing approaches handle this poorly: a one-shot vision-language model (VLM) compresses the whole procedure to fit its context window and loses the detail a "before" or "after" question depends on, while video agents that train the model where to look are data-hungry and transfer poorly to out-of-domain surgery. We build an agent harness that separates reasoning from perception and improves by evolving context rather than optimizing weights. A text-only orchestrator plans which evidence to gather and issues an auditable sequence of tool calls, while frozen vision-language sub-agents execute each call over the pixels, viewing, cropping, inspecting frames, and retrieving external knowledge. We further propose a gradient-free, reward-gated Heuristic Skill Distillation loop that mines the agent's own low-scoring traces and keeps a candidate skill only when it raises a validation reward, yielding reusable retrieval skills, notably directed re-look. Growing an external skill library rather than tuning weights, the loop adapts from only about 100 labeled examples, far fewer than supervised or reinforcement fine-tuning requires. To evaluate this agent, we introduce MedClawBench, a de-leaked, doctor-grounded benchmark of 1,123 questions over self-built long neurosurgery recordings and a held-out public lecture-video test split. Across both datasets and all four evaluation dimensions, our agent consistently outperforms one-shot VLMs and general video-agent frameworks, with the largest gains on the long, out-of-domain neurosurgery videos. Project page: this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

63,969 characters extracted from source content.

Expand or collapse full text

IEEE TRANSACTIONS AND JOURNALS TEMPLATE1 MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning Yingying Fan, Penghui Du, Leyan Zhu, Runze He, Zimeng Wu, Yuxuan Zhang, Liang Chen, Jiahao Xie, Jiangtang Wang, Shuai Shao, Anchao Yang, Yutong Bai, Yan Wang, IEEE Member Abstract — Understandingtens-of-minutessurgical videosrequires long-horizon temporal reasoning, answering what happens before, after, or across stages of a procedure by grounding the question in visual evidence spread across time. Existing approaches handle this poorly: a one-shot vision–language model (VLM) compresses the whole procedure to fit its context window and loses the detail a “before” or “after” question depends on, while video agents that train the model where to look are data-hungry and transfer poorly to out-of-domain surgery. We build an agent harness that separates reasoning from perception and improves by evolving context rather than optimizing weights. A text-only orchestrator plans which evidence to gather and issues an auditable sequence of tool calls, while frozen vision–language sub-agents execute each call over the pixels, viewing, cropping, inspecting frames, and retrieving external knowledge. We further propose a gradient-free, reward-gated Heuristic Skill Distillation loop that mines the agent’s own low-scoring traces and keeps a candidate skill only when it raises a validation reward, yielding reusable retrieval skills, notably directed re-look. Growing an external skill library rather than tuning weights, the loop adapts from only about 100 labeled examples, far fewer than supervised or reinforcement fine-tuning requires. To evaluate this agent, we introduce MedClawBench, a de-leaked, doctor-grounded benchmark of 1,123 questions over self-built long neurosurgery recordings and a held-out public lecture-video test split. Across both datasets and all four evaluation dimensions, our agent consistently outperforms one-shot VLMs and general video-agent frameworks, with the largest gains on the long, out-of-domain neurosurgery videos. Project page: https://fyycs.github.io/medclaw/. Index Terms— Long-horizon surgical video reasoning, Agent harness, Heuristic Skill Distillation. This work was supported in part by the National Natural Science Foundation of China under Grant 82402252, in part by the Noncom- municable Chronic Diseases-National Science and Technology Major Project under Grant no.2023ZD0515200. Yingying Fan is with the School of Automation and Intelligence, Beijing Jiaotong University, Beijing 100044, China, and also with the Depart- ment of Electrical and Computer Engineering, University of Maryland, College Park, MD 20742, USA (e-mail: kristewtoday@gmail.com). Penghui Du, Leyan Zhu, Runze He, Zimeng Wu, Liang Chen, Jiahao Xie are with UniPat.ai. Shuai Shao and Jiangtao Wang are with Suzhou Institute for Advanced Research, University of Science and Technology of China. Yuxuan Zhang is with University of British Columbia. Anchao Yang and Yutong Bai are with the Department of Neuro- surgery, Beijing Tiantan Hospital, Capital Medical University (Email: yang.anchao@163.com; baiyutong@mail.ccmu.edu.cn). Yan Wang is with School of Automation and Intelligence, Beijing Jiaotong University, No.3 Shangyuancun, Haidian District, Beijing, 100044, China. (Email: wangyan9509@gmail.com). I. INTRODUCTION S URGICAL videos are a primary medium for teaching, credentialing, and retrospective review [1], [2], yet they are among the hardest inputs for medical video understand- ing [3]–[5]. A procedure runs for tens of minutes, its clinically meaningful information is spread across time, and the answer to “what is done before or after a given step” depends on how the operation evolves rather than on any single frame [5], [6]. We call this ability, grounding a question about the temporal evolution of a whole procedure in the visual evidence itself, long-horizon temporal reasoning. It is precisely the ability a trainee must acquire from watching operations, and precisely the ability that current surgical video systems fail to deliver. Two families of methods dominate, and neither fits the task. A one-shot vision–language model (VLM) reads the whole procedure at once [7], but a tens-of-minutes video far exceeds any context window [8], so it is compressed to fit and loses the very detail a “before” or “after” question needs (Fig. 1). It can only answer from the coarse, compressed view, and it can never go back to re-examine which few seconds actually matter [9]. The task is instead inherently iterative: decide where to look, look, then re-decide, since the answer usually lies outside the window that depicts the named event. This calls for an agent, and a second family builds one, but trains the model where to look through supervised fine-tuning (SFT) or reinforcement learning (RL) [10], [11]. That demands large amounts of task-specific trajectory data, overfits the training distribution, and transfers poorly to an out-of-domain, long-horizon setting such as long neurosurgery, precisely the regime where the ability is most needed and labeled data is scarcest [12], [13]. We therefore ask whether the ability to decide where to look can be obtained in a training-free manner, and our central idea is to improve by evolving context rather than optimizing weights. We separate the two abilities the task needs, deciding where to look and actually looking, and hand each to a frozen model coordinated only in context. Our agent harness pairs a text-only orchestrator, which reasons about the question and issues an auditable sequence of tool calls, with frozen vision–language sub-agents that execute those calls on the pixels: viewing the full video, cropping a candidate time window (crop video), inspecting individual frames (selectframe, cropframe), and querying a surgical knowledge base. Localization follows a coarse-to-fine strategy, arXiv:2608.14015v1 [cs.CV] 14 Aug 2026 2IEEE TRANSACTIONS AND JOURNALS TEMPLATE Fig. 1.Long-horizon temporal reasoning on a long surgery video, where an “after” question is answered by evidence that lies outside the window depicting the named event. (a) A one-shot VLM reads the whole procedure in a single pass, losing the detail the question needs. (b) SFT- based agents learn where to look through supervised or reinforcement fine-tuning, which needs massive data and transfers poorly out of domain. (c) MedClaw keeps every model frozen and instead distills reusable skills through Heuristic Skill Distillation, collecting failure traces, diagnosing the gap, and retaining a skill only when it passes a reward gate, adapting from only about one hundred examples while keeping every decision explicit and auditable. moving from the whole procedure down to the exact moment. No weight is ever updated: the orchestrator supplies strategy, the sub-agents supply perception, and everything happens in context, which turns the otherwise latent “which segment to look at” into a inspectable, re-examinable action trace. Which retrieval strategies matter, and how are they reused without any parameter update? Open-ended surgical QA has no automatic success signal, so self-evolving loops that keep whatever the model proposes accumulate plausible-but-useless rules that can even lower answer quality. Our Heuristic Skill Distillation (HSD) loop is instead gradient-free and reward- gated: it reads the agent’s own low-scoring traces, diagnoses the missing temporal evidence, proposes candidate strategies, and retains one only when it lifts a validation reward scored on four dimensions (correctness, detail, context, temporal understanding), so a strategy is kept because it works, not because it looks plausible (Fig. 1). Each retained strategy, notably directed re-look (revisiting the window adjacent to a localized event), is stored as a skill that names when it applies and how to call the existing tools, and the orchestrator holds it in a skill library and consults it when a question matches its trigger. Since the loop edits context, it is strikingly data- efficient: it mines its skills from about one hundred preference examples, where the SFT and RL agents we compare against fit millions of parameters to far larger labeled sets and transfer poorly beyond them. Evaluating this agent requires a benchmark that truly iso- lates long-horizon temporal reasoning. However, existing re- sources do not: they suffer from four shortcuts, questions with no long-horizon structure (short clips, or long videos that still ask only single-moment questions), a narrator who speaks the answer aloud, a “standard next step” that follows from textbook priors, and questions answerable from a single frame. We therefore introduce MedClawBench, a de-leaked, doctor-grounded benchmark of 1,123 questions: 259 over long neurosurgery recordings (microvascular decompression, tumour resection, epileptogenic-lesion resection) and 864 from the held-out test split of the public SVU-31K corpus [5]. We validate the automatic judge against expert ratings on both splits (κ) so reported scores reflect clinical judgment rather than a self-referential metric. Our contributions are as follows: • A reproducible agent harness that factorizes long-video reasoning into a text orchestrator and frozen vision– language sub-agents. The orchestrator gathers evidence through a small set of tools, viewing the full video, crop- ping a candidate time window (crop video), inspecting individual frames (selectframe, cropframe), and querying a surgical knowledge base, so evidence seeking becomes an auditable trace of tool calls while every model weight remains frozen. • A Heuristic Skill Distillation loop that improves the agent by evolving context rather than optimizing weights: from roughly one hundred preference examples it dis- covers reusable temporal evidence-retrieval skills, no- tably directed re-look, under objective reward gating, and packages each as a skill in a separate library that the orchestrator consults when a question matches the skill’s declared trigger, improving the agent with no parameter update and keeping every retrieval decision auditable. • MedClawBench, a de-leaked, doctor-grounded bench- mark (259 self-built neurosurgery questions and 864 held- out SVU questions, 1,123 in total, blind-filtered and κ- validated) that isolates long-horizon temporal reasoning on surgical videos. • State-of-the-art results. Our agent consistently outper- forms one-shot VLMs and general video-agent frame- works across both datasets and all four evaluation di- mensions, with the largest margins on the long, out-of- domain neurosurgery videos where long-horizon temporal reasoning matters most. I. RELATED WORK A. Agent Harnesses for Video Reasoning A growing line of work casts video understanding as tool- augmented reasoning, where a language model reasons and acts in an interleaved loop [14] and calls external tools [15]– [17]. For long videos, these systems adaptively retrieve ev- idence, selecting or sampling informative frames [18]–[21] and localizing temporal windows via multi-turn spotlighting or agentic synthesis [9], [22], [23], often using reinforcement learning to shape retrieval or broader spatiotemporal reasoning AUTHOR et al.: MEDCLAW: HEURISTIC AGENT HARNESS FOR LONG-HORIZON SURGICAL VIDEO REASONING3 policies [11], [24]–[28]. A convenient way to organize these behaviors is an agent harness that separates a planning text orchestrator from frozen vision–language sub-agents, with all coordination carried in context [29], and such harnesses also serve as auditable evaluators or task solvers on realistic benchmarks [30], [31]. In the clinical domain, MedScope [6] interleaves reasoning with coarse-to-fine tool calls to “think with videos,” and a broader line of medical agents trains tool-augmented reasoning with reinforcement learning [32], [33]. These methods, however, train the backbone itself to decide where to look, which demands large task-specific trajectory data and transfers poorly to an out-of-domain, long- horizon setting such as long neurosurgery. In contrast, we keep both the orchestrator and the sub-agents frozen and draw retrieval strategies from an external skill library, so the gain is attributable to knowing how to look rather than to a fine-tuned backbone. To our knowledge, this is the first training-free agent harness for long-horizon temporal reasoning on surgical video. B. Heuristic Learning beyond Gradients A complementary paradigm improves a frozen model by searching over its context rather than its weights. Reasoning- bootstrapping and self-feedback methods let a model rewrite its own reasoning or outputs from verbal or reward sig- nals [34], [35], experiential agents distill successful trajectories into reusable skills or memories recalled later [36]–[39], and context- and prompt-optimization methods edit instructions or context to raise an objective without back-propagation [40]– [42], with reward-gated variants keeping only edits that pass an explicit check [29]. Many of these presuppose an environment that scores success automatically. Without one, as in open- ended surgical question answering, plausible-but-useless skills can accumulate on the model’s own judgment. We instantiate this heuristic-learning idea as Heuristic Skill Distillation: the agent diagnoses its own blind failures against the gold, distills only operational lessons into candidate skills, and keeps a candidate only when it raises a validation four-dimensional reward. While such context optimization has been explored mainly for text and general multimodal tasks, we bring it to surgical video, distilling temporal evidence-retrieval skills for a frozen clinical-video agent. C. Long Surgical and Clinical Video Understanding Surgical procedures are long, so reasoning over them is fundamentally a long-video problem. Early work focused on localized, clip-level recognition of phases, tools, and ac- tions [3], [43], and early surgical VQA answered questions about a single endoscopic scene [4], later grounding the answer in a domain-specialized language model [44]. To move beyond short clips, recent efforts scale to full-length procedures: SurgVidLM contributes the large-scale SVU-31K corpus of surgical video–instruction pairs [5], from whose held-out test split we draw the lecture-video portion of our benchmark. OphClip pretrains an ophthalmic surgical video– language model [45], MedGen scales granularly annotated medical video [46], and corpora such as SurgPub-Video [47], SurgBench [48], SurgViVQA [49], and EyePCR [50] broaden surgical VQA coverage. General long-form video under- standing has advanced through memory-augmented and long- context models [7], [8], [51]–[54] and strong open founda- tion VLMs [55]–[57] surveyed in [58], [59], but these are rarely evaluated on long surgical procedures. Medical vision– language agents likewise equip a model with clinical tools for diagnosis and interpretation [60]–[63]. Despite this progress, most medical video QA resources use short, pre-trimmed clips with temporally local content, and several carry spoken narration that states the answer aloud. Fine-grained, evidence- annotated clinical data [6] begins to encourage grounding, yet the long-horizon setting, aggregating evidence for what happens before or after a step across a full-length procedure, remains largely unaddressed. Our benchmark isolates exactly this ability, pairing long neurosurgery recordings with a held- out lecture-video split under a blind filter that discards ques- tions answerable without watching the video. I. METHOD A. Task Formulation Given a surgical video V and a natural-language question q, the task is to produce a free-form answer a grounded in what the video actually shows. We deliberately keep answers open- ended rather than multiple-choice: a distractor-based format lets a model exploit textual priors [64], whereas a free- form answer must name the specific instruments, structures, and actions observed in this procedure. To probe how an agent handles both the visual-detail and the temporal sides of long-horizon surgical video understanding, we follow SVU- 31K [5] and group every question into two reasoning types. Visual Perception Reasoning (VPR) questions, once the right moment is localized, turn on a visual detail at that moment, such as which instrument is in use, an anatomical structure, a count, or a finding. Visual Temporal Reasoning (VTR) questions instead turn on the process itself, how the procedure unfolds: what happens after a given step, what was done before or in preparation for an event, or how a state changes over a long horizon. For VTR the evidence often lies outside the window that depicts the named event, so it can be recovered only by relating moments across the procedure, which makes VTR the more temporally demanding of the two. B. Overview Answering a long-horizon temporal question in one forward pass is hard for a fixed model since the evidence sits in a few seconds out of tens of minutes, requiring the system to decide where to look, inspect that evidence, and revise its decision. We build an agent harness with three parts that each do only what they are good at. A text orchestrator π θ plans which evidence to gather and when to stop (Section I-C); a tool set T exposes the video at three granularities and adds a retrieval tool over an external knowledge base (Section I-D); and frozen vision sub-agents ψ answer the concrete visual query each tool is handed (Section I-E). No weights are updated: neither π θ nor ψ is fine-tuned, and competence comes from the loop and its skill library rather than from updated parameters. On top of this frozen harness, a Heuristic Skill 4IEEE TRANSACTIONS AND JOURNALS TEMPLATE Question: What was done right before the coagulated vessel was cut with curved scissors? <skill>LOAD surgical-video-relook skill. </skill> <tool_call>CALL view_full_video</tool_call> <think>I'l start by loading the surgical-video-relook skill since this is a temporal (before) question.</think> >> After locating the event window [a, b] (from view_full_video), the answer for >> these questions is NOT inside [a, b] —it is in the ADJACENT window. Crop it. [00:0039:15]Sampled Frames >> [00:00 - 01:30]: Opening of the dura mater and initial exposure of the cerebellar surface. >> ... (Summarizationof the full video) >> RELEVANT_WINDOW: 23:50-24:50 <think>Let me crop the window just before the cut to see what preceded it.</think> <tool_call>CALL crop_video(from=22:50, to=23:50)</tool_call> >> 22:20 Dissection of arachnoid membrane using a micro-probe. >> 22:34 Surgical field is cleared, showing the brain surface. >> ... (Timeline of cropped video) <tool_call>CALL select_video_frame(at=23:42)</tool_call> >> Anatomy/tissue: - The central/lower portion of the field contains ... >> Retractors/field exposure: - Several pale/white retractor elements ... >> Instrument(s): - A thin, dark, rod-like instrument or tip enters from ... >> Vessel/tissue of interest: - The suspected small vessel or vascular strand ... <think>The crop timeline confirmed the sequence clearly: coagulation of the vessel with bipolar forceps .... This is consistent with standard microvascular decompression technique where a vessel is coagulated before being divided.</think> <answer>The vessel was coagulated (sealed) using bipolar forceps. This bipolar coagulation cauterized the vessel so it could be safely divided without bleeding.</answer> 22:50 23:50 Load Knowledge base: ·Surgicalpriors ·Anatomy ·Procedures Skills: · Derivedfrom failures ·Reusable (a) Collect Failure Traces (b) Diagnose the Gap (c) Reward-gate Retention Mining SetFrozenAgent HarnessAnswer !푎&Trajectory 휏 Answer !푎& Trajectory 휏 TypeA: Operational Gap Different Looking behavior could fix it TypeB: Unreachable Gap Gold is not visible in the video Type C: Already Correct No Change Discard Propose candidate skill 푠 Candidate Skills 푠 ! ,...,푠 " Reward 푅 # =피푅푎 # R & >푅 ' +훿? Discard NO YES 푆=푠|R & >푅 ' +훿 Retain Tuning HeuristicSkillDistillation ExternalKnowledge & Tunable Skills 푠 ! 푠 " 푠 # 푠 ... 푠 % Baseline Reward 푅 ' =피푅푎 ∅ Agent w/o Skills Install Each Alone Fig. 2. Overview of MedClaw. Left: a real trace on a temporal “before” question, where the orchestrator loads the relevant skill and issues an auditable sequence of tool calls that frozen vision sub-agents execute on the pixels, viewing the full video to localize a window, cropping it, and inspecting a frame, before grounding its answer. Top right: the orchestrator is backed by an external knowledge base queried via RAG and by a library of tunable skills derived from past failures. Bottom right: these skills come from our Heuristic Skill Distillation loop, which collects low- scoring traces on a mining set, diagnoses each gap as operational, unreachable, or already correct, and retains a candidate only if it raises the four-dimensional reward on a validation set. Distillation loop (Section I-F) mines reusable retrieval skills from the harness’s own low-scoring trajectories and stores them in a tunable skill library. Fig. 2 provides an overview of the framework: the left panel illustrates an end to end inference trajectory, where the orchestrator loads a relevant skill and issues tool calls executed by the vision sub-agents, while the panel right shows the external knowledge base, the skill library, and the distillation loop that fills it. C. Orchestration Let V be a surgical video and q a question. The orchestrator is a text-only policy π θ (Opus-4.8) with frozen parameters θ that never sees V directly, and it acts through a fixed tool set T = T 1 ,...,T 5 (Section I-D), each tool being a call T i (V,·) answered by the frozen vision sub-agents ψ (Section I-E). Following a reason-act loop [14], at step t the orchestrator conditions on the running context c t to emit an action a t , and the selected tool then returns a textual observation o t : a t = (T i t ,u t )∼ π θ (·| c t ), o t = T i t (V,u t ),(1) where u t are the corresponding tool arguments (e.g. a time range). The context is then extended as c t+1 = c t ⊕ (a t ,o t ), and the loop repeats until the policy emits a STOP action and produces the final answer a = π θ (c T ). All state lives in c t , so no gradient step or external memory is involved, and the entire trajectory ρ = (a 1 ,o 1 ),..., (a T ,o T ) is recorded. Because the orchestrator has no direct access to V , every visual need must be externalized as an explicit tool call, which makes ρ fully auditable and allows gains to be attributed to specific retrieval behaviors. D. Tool Suite The tool set T forms a coarse-to-fine progression from the whole clip to a segment to a single frame, so the orchestrator localizes first and then verifies the exact instrument, structure, or action at the moment that matters. Each tool takes the video and typed arguments and returns a textual observation. • view fullvideo(V,q) localizes where to look. It runs a coarse-to-fine pass: a Stage-1 low-resolution reading of the full procedure emits a dense timeline, one short description per time interval [a k ,b k ] of what is visible in that interval, together with a candidate time window for the queried event. A Stage-2 re-watch of that widened window at higher resolution then returns the tightest relevant sub-window [s ⋆ ,e ⋆ ] that actually contains the queried event. This puts temporal localization on the model that can see the pixels rather than on the text orchestrator. • cropvideo(V,s,e,q) inspects what happens in a chosen range [s,e]. It re-reads that segment at crop resolution and returns a dense description, and is the tool the orchestrator uses to read the localized window or an adjacent one. • select videoframe(V,t,q) returns a description of the single frame at timestamp t from an independent frame model, used as a cross-model check of the specific instrument or action at that instant. • crop videoframe(V,t,x 1 ,y 1 ,x 2 ,y 2 ,q)extracts the frame at t, crops the normalized spatial region (x 1 ,y 1 ,x 2 ,y 2 ), and describes that region, for fine spatial disambiguation of small or overlapping structures. • search surgicalkb(q, type,k)isaretrieval- augmented-generation(RAG)toolthatreturnsthe AUTHOR et al.: MEDCLAW: HEURISTIC AGENT HARNESS FOR LONG-HORIZON SURGICAL VIDEO REASONING5 top-k passages from a surgical knowledge base when a term needs grounding. The knowledge base holds general surgical background only, with no benchmark answers: PubMed abstracts retrieved for surgical keywords, and literature-grounded summaries of surgical techniques and endoscopic visual cues generated from those keywords, embedded with a sentence encoder and indexed in FAISS under cosine similarity. E. Vision Sub-agents Every tool call is executed by frozen vision sub-agents ψ that perform actual perception and return text. They employ no planning or task-specific weights, and comprise two experts that map directly onto the two granularities of the tool set. • A video expert (Gemini-3.1-flash-lite/Gemini-3.5-flash) serves view fullvideo and cropvideo: it ingests a clip or a temporal segment and returns timelines, per- interval descriptions, and localized windows, handling ev- erything that requires reasoning over time. • Aframeverifier(GPT-5.5)serves select videoframeand cropvideoframe: it ingests a single frame or a spatial crop of one frame and returns a description of that instant, giving an independent, higher-resolution check on a specific structure. Keeping perception behind this fixed interface makes ψ inter- changeable, so the same orchestrator π θ runs over a cheaper or stronger backbone with no other change, which we exploit to separate the contribution of the harness from that of the backbone. F. Heuristic Skill Distillation The harness above is fixed, but which retrieval behaviours it should prefer is not obvious a priori. We therefore add a heuristic skill distillation loop (Fig. 2, bottom right) that discovers reusable retrieval strategies from the harness’s own mistakes and stores them in an external skill library S the orchestrator consults at inference. Learning here evolves this library rather than any model weight, and it is strikingly data- efficient: it uses only 100 labeled questions from the SVU training split, kept disjoint from every evaluation set and split once into a 60-question mining set on which skills are distilled and a 40-question validation set on which they are gated, whereas the trained agents we compare against fit millions of parameters to far larger corpora. Mining candidate skills from failed traces. We walk the min- ing set one question at a time. On each question, the inference- time orchestrator first answers blind: without seeing the gold, it uses the same tools as at test time (a view fullvideo pass, then cropvideo and selectvideoframe as needed) and produces an answer ˆa and its trajectory ρ. It is then shown the gold, asked to diagnose the gap between ˆa and the gold, and to classify it into one of three cases. A type A gap is operational: the gold’s content was present in the video and a different way of looking would have caught it, by cropping a tighter or adjacent window, escalating the crop resolution, selecting a different frame, or disambiguating two look-alike instruments or structures. A type B gap is unreachable: the gold refers to something no amount of looking can recover, such as off-screen narration, the surgeon’s stated intent, or a pathology result, or the backbone simply cannot resolve the pixels. Type C means the blind answer was already correct. Only type A carries an actionable lesson. When the orchestrator finds one, it writes the lesson itself through the native skill manage tool: it first inspects the current library and, if a skill already states this same rule, updates that entry (sharpening it or adding the new example) rather than creating a near-duplicate, otherwise it adds a new skill. Walking the mining set this way accumulates a small set of candidate skills, capped at five, each mined directly from a trajectory that failed rather than supplied by us. Reward-gated retention. Mining only proposes candidates, and whether keeping each one is decided by the validation re- ward. Let R(a)∈ [1, 5] be the mean of four judge dimensions per answer. Before mining, we measure the empty-library reward R 0 =E[R(a ∅ )], the no-skill baseline, on the validation set. We then test mined candidates one at a time. To test a candidate s we install it alone, re-answer the entire validation set, re-judge, and take R s =E[R(a s )]. The candidate is kept only if it clears both gates: it must raise the mean four- dimensional reward by a margin, R s > R 0 +δ with δ = 0.05 on the 1–5 scale, and it must not drop any individual validation question by two points or more. A rejected skill is removed and leaves no trace. This is what separates the loop from reward- based post-training: the reward gates an auditable, perception- level skill in an external library, and never edits a weight or a fixed prompt. The learned skill library. Each retained skill names the condition under which it fires and how to call the tools with concrete arguments, so its trigger is precise and every retrieval it prompts stays visible in the trajectory. In practice, the loop converges to a compact library dominated by one temporal skill, directed re-look. The rule it mines is that for a question about what precedes or follows a located event, the answer lies in the window adjacent to that event rather than inside it. Once the event window is localized, the skill therefore crops and re-reads the following window for an “after” question, or the preceding window for a “before” or preparation question, and grounds the answer there rather than in the event window itself. This is exactly the failure mode that our VTR partition is built to stress, and the harness recovers it not by design but by mining and reward-gating its own temporal mistakes. IV. MEDCLAWBENCH We introduce MedClawBench, a benchmark for open- ended question answering over long surgical videos (an 18.4- minute median, with 41% running beyond twenty minutes; Fig. 3(b)), built to measure long-horizon temporal reasoning in the medical domain. By construction, every question is free of the four shortcuts that let existing resources be answered without long-horizon reasoning: weak temporal structure, nar- ration leakage, textbook-prior answerability, and single-frame answerability. Table I positions it against existing surgical video QA datasets: frame-level resources such as EndoVis18- VQA [65] and PitVQA [44], and the few-frame clip-level 6IEEE TRANSACTIONS AND JOURNALS TEMPLATE TABLE I COMPARISON WITH SURGICAL-VIDEO QA DATASETS. VIDEO LEN.: MEDIAN SOURCE-VIDEO DURATION.≥ 20 MIN: SHARE OF VIDEOS AT LEAST TWENTY MINUTES LONG. TEMP.: LONG-HORIZON TEMPORAL REASONING. DE-LEAK: NOT ANSWERABLE WITHOUT WATCHING THE VIDEO (BY EXPLICIT FILTERING OR BY THE NATURE OF THE SOURCE). NEURO.: INCLUDES NEUROSURGERY VIDEOS. DURATIONS AND PERCENTAGES ARE ROUNDED. DatasetVideo len.≥ 20 min Temp. De-leak Neuro. EndoVis18-VQA [65]per-frame–✗✓✗ PitVQA [44]per-frame–✗✓ SurgViVQA [49]8 frames–short✓✗ SVU-31K (test) [5]13.9 min28%partial✗ MedClawBench (ours) 18.4 min41%✓ neurosurgery split28.5 min71%✓ SurgViVQA [49] carry no long-horizon structure, though as pure operative footage their questions cannot be answered without the video. The only full-video corpus that also asks temporal questions, SVU-31K (test) [5], is built on much shorter videos (32% under ten minutes) and, being narrated lecture material, leaks answers through speech yet applies no de-leaking. MedClawBench is the only entry that combines long surgical videos, explicit de-leaking, and neurosurgery content in a single benchmark. A. Data Construction MedClawBench contains 1,123 questions drawn from two complementary sources, so that it spans both a long, self- recorded surgical domain and a large, externally sourced lecture domain. Fig. 3 summarizes its composition along three axes: question type (Fig. 3(a)), source-video duration (Fig. 3(b)), and surgical type (Fig. 3(c)). Neurosurgery. The first half comprises 259 questions over 14 long neurosurgery recordings from one centre, spanning three procedure types: microvascular decompression (MVD, ten videos, 186 questions), tumour resection (three videos, 49) and epileptogenic-lesion resection (one video, 24). These procedures are long (a 28.5-minute median, with 71% running beyond twenty minutes) and visually homogeneous, making neurosurgery the harder, long-horizon half of the benchmark. General-purpose vision models lack the clinical knowledge to caption these procedures reliably, and automatically generated golds were frequently “correct textbook knowledge unrelated to the video,” as flagged by our neurosurgeon collaborator. We therefore ground the labels in expert neurosurgeon annotation rather than model captions: a neurosurgeon seg- ments each procedure into time intervals and annotates the clinical event in each, and, conditioned on these events, GPT- 5.5 drafts diversified, event-anchored questions whose gold is tied to a specific annotated interval, with embedding-based de- duplication. Each candidate then passes the grounding filters and the faithfulness check below, and is labelled with one of ten fine-grained types (Fig. 3(a)) that partition into VTR (109) and VPR (150), following Section I-A. The temporal (VTR) types ask what step immediately follows (temporal-after) or precedes (temporal-before) a named event, how distant phases relate (long-range-temporal), or what unfolds within a stated interval (segment-content). The perceptual (VPR) types instead ask about a single moment: which instrument performs a step, what action occurs, a counting of items, a finding (an observed anatomical or pathological detail), a comparison (relating two structures or states visible at that moment), or a case-specific detail unique to the individual procedure. SVU (lecture). We use only the question–answer pairs from the test split of SVU-31K, a lecture-video corpus released by SurgVidLM [5], and apply the filtering cascade below to retain 864 questions, reusing their original Visual Temporal Reason- ing (163) and Visual Perception Reasoning (701) labels. B. Filtering Every candidate must survive a cascade of filters that certify it can be answered only by watching the moment it targets. All candidates must pass the shared duration, blind, and local filters; SVU questions additionally pass a narration filter, and neurosurgery questions a faithfulness filter. (A) Duration: we keep only questions whose source video is at least ten minutes long, so that the benchmark stresses long-horizon reasoning over an extended procedure rather than perception on a short clip. For the answerability filters that follow we let a restricted answerer respond and use an automatic judge (GPT-5.5) to score its answer’s overall correctness against the gold on a 1–5 scale, against a fixed threshold τ =3. (B) Blind: a text- only model (DeepSeek) answers with no video access, yielding s B , and we require s B < τ to drop questions solvable from priors alone. (C) Local: a model answers using only the dense observations extracted from the targeted window, yielding s C , and we require s C ≥ τ so that a kept question is answerable from its own interval of visible evidence rather than a coarse whole-procedure overview. (D) Narration (SVU): we re- answer each question from the audio track alone, with no frames, and discard it if this scores≥ τ , since its answer would then be recoverable by listening rather than by watching. (E) Faithfulness (neurosurgery): since the neurosurgery question and gold are drafted by GPT-5.5, a fact-checker checks them against the neurosurgeon’s interval annotation and keeps a candidate only if it aligns with that annotation, discarding any content GPT-5.5 hallucinated beyond it. V. EXPERIMENTS A. Setup We evaluate on MedClawBench, comprising 259 long neurosurgery questions and 864 held-out SVU lecture-video questions, for 1,123 questions in total. Every answer is scored on four dimensions, including correctness of information (CI), detail orientation (DO), contextual understanding (CU), and temporal understanding (TU) [5], each on a scale 1–5 by an automatic judge, and we report the average (Avg) of the four. Higher is better on all metrics. Our agent is implemented on top of the open-source Hermes agent framework. 1 1 https://github.com/nousresearch/hermes-agent AUTHOR et al.: MEDCLAW: HEURISTIC AGENT HARNESS FOR LONG-HORIZON SURGICAL VIDEO REASONING7 (a) Neurosurgery question types (n = 259) 050100150200 Count Lower GI MVD Upper GI Urology & Pelvic HPB* Gynecology Hernia Tumour resection Thoracic Other Endoscopic Epileptogenic lesion 200 186 186 124 106 76 68 49 40 34 30 24 29 10 25 25 12 9 17 3 7 12 5 1 (c) Surgical type distribution Questions Videos VPR 150 VTR 109 finding (29) counting (29) instrument (28) case-specific (28) action (18) comparison (18) temporal-after (29) long-range (28) temporal- before (27) segment- content (25) Video duration (min) 0 5 10 15 20 25 30 35 40 Percentage of videos (%) 32 28 24 10 6 (b) Video-duration distribution SVU-31KOurs Fig. 3. MedClawBench statistics. (a) Neurosurgery question types (259): the inner ring is the Visual Temporal (VTR, 109) vs. Visual Perception (VPR, 150) split and the outer ring the ten fine-grained types. (b) Source-video duration for MedClawBench (Ours) against SVU-31K, as the percentage of videos per bin; our filter drops all sub-ten-minute videos and shifts mass toward the long (≥ 20 min, shaded) tail. (c) Surgical-type distribution over both splits (# questions, # videos), with the neurosurgery half split into MVD, tumour resection and epileptogenic-lesion resection (HPB* stands for Hepatobiliary & Pancreas). B. Baselines We compare against three families of models. Gen- eral video-language models: Qwen2.5-VL-7B-Instruct [55], InternVL3-8B [56], and VideoLLaMA3-7B [57]. Video- reasoning models: Video-R1-7B [24], VideoChat-R1-7B [25], and Video-RFT [28]. Long-video understanding agents: LongVT-7B-RFT [11] and ReWatch-R1-7B [10]. We further include the surgical video-language model SurgVidLM [5]. Fi- nally, as the one-shot vision backends inside our own harness, we report Gemini-3.1-flash-lite, Gemini-3.5-flash, GPT-5.5, and Opus-4.8 answering each question in a single pass. Med- Claw is our frozen-weight agent, which issues an auditable sequence of tool calls (viewing, cropping, frame inspection, knowledge retrieval), including a coarse-to-fine pass inside the full-video view and the discovered directed re-look skill. C. Main Results Table I reports results separately for Visual Temporal Reasoning (VTR) and Visual Perception Reasoning (VPR), which together make up each split. On the long neurosurgery split, MedClaw uses the stronger flash visual backbone and reaches a four-dimensional average of 2.90, ahead of the strongest open-source video model we test (VideoChat-R1- 7B at 2.14) by +0.76. On SVU, where the clips are shorter and the two visual backbones are comparable, MedClaw uses the cheaper lite visual backbone. The table therefore reports a single MedClaw row with the best deployed configuration for each split: flash for neurosurgery and lite for SVU. The ∆ row compares MedClaw against the corresponding one- shot backbone in each split (Gemini-3.5-flash for neurosurgery and Gemini-3.1-flash-lite for SVU): using the same visual backbone, MedClaw improves every dimension by +0.17 to +0.56, which shows that our agent harness is what drives the gain. Every open-source baseline, including the surgical video- language model SurgVidLM (1.94) and the re-watching model ReWatch-R1-7B (1.54), trails all of the one-shot backends on neurosurgery, underscoring how far long-horizon neurosurgery falls outside their training distribution. D. Ablation Studies We ablate using the long neurosurgery split (259 questions). The harness lifts diverse backbones. We compare two settings for each backbone (Table I): (i) a one-shot pass, where the model answers the question directly from the video, against (i) wrapping the same model inside our harness as the visual sub-agent that sees the video, in place of Gemini. Across every backbone we test, wrapping it in our harness improves its four-dimensional average, by +0.54 for Qwen2.5-VL, +0.55 for Video-R1, and up to +1.02 for ReWatch-R1-7B, indicating that our generic harness delivers consistent gains when applied to a broad range of video-understanding models. Removing each component. Starting from the full model, we remove or alter one learned component at a time (Ta- ble IV): the discovered directed re-look skill, coarse-to-fine localization, and Heuristic Skill Distillation itself (replac- ing the distilled skills with a hand-written general looking- discipline skill). Directed re-look matters most: removing it drops the average by −0.38. This skill targets before and after temporal questions, where after localizing the named event it re-examines the window adjacent to that event, since the answer lies just outside the moment the event occupies rather than in the coarse first pass. Coarse-to-fine localization (−0.16) shows that narrowing from the whole procedure down to the exact moment, rather than reading one compressed pass, is needed to find the evidence a question targets. Heuristic Skill Distillation (−0.13) shows that skills mined and reward- gated from the agent’s own traces outperform a hand-written looking-discipline skill, so the distilled skills contribute be- yond generic looking advice. Every component thus makes a measurable contribution, and directed re-look is the largest. Swapping the orchestrator. The harness should not depend on one particular reasoner. We keep the perception side fixed and swap only the text-only orchestrator (Table V). Replacing Opus-4.8 with either open-weight DeepSeek reasoner costs about a third of a point (DeepSeek-V3 −0.33, DeepSeek- R1 −0.34). A weaker orchestrator thus degrades the harness gracefully rather than breaking it, while Opus-4.8 remains the strongest choice for driving the tool-call loop. 8IEEE TRANSACTIONS AND JOURNALS TEMPLATE TABLE I MAIN RESULTS ON MEDCLAWBENCH, BROKEN DOWN BY REASONING TYPE (VTR, VPR) WITH THE FOUR SCORING DIMENSIONS EACH 1–5: CI (CORRECTNESS), DO (DETAIL), CU (CONTEXT), TU (TEMPORAL). BEST IN BOLD, SECOND BEST UNDERLINED . SHADED CELLS MARK THE ONE-SHOT BACKBONE USED FOR THE ∆ ROW. MEDCLAW USES THE FLASH VISUAL BACKBONE ON NEUROSURGERY AND THE CHEAPER LITE BACKBONE ON SVU, AND THE ∆ ROW COMPARES AGAINST THE CORRESPONDING ONE-SHOT BASELINE IN EACH SPLIT. Cat.Method Neurosurgery (259)SVU (864) VTR (109)VPR (150)VTR (163)VPR (701) CIDOCUTUCIDOCUTUCIDOCUTUCIDOCUTU VLM Qwen2.5-VL-7B1.621.312.282.011.801.612.532.901.641.542.312.012.251.852.842.73 InternVL3-8B1.351.051.781.651.651.432.292.601.511.181.981.751.811.162.112.09 VideoLLaMA3-7B1.491.091.961.741.731.392.282.501.601.292.171.791.871.342.262.28 Reasoning Video-R1-7B1.611.172.242.281.771.552.553.031.671.312.332.022.101.502.683.04 VideoChat-R1-7B1.751.292.332.032.041.652.673.021.871.612.462.122.301.742.772.74 Video-RFT1.641.102.061.942.031.512.453.011.851.422.372.062.221.642.642.76 Agent LongVT-7B-RFT1.751.262.181.931.801.402.232.291.531.242.041.651.981.432.472.45 ReWatch-R1-7B1.461.101.791.531.571.161.791.851.701.232.181.801.911.302.352.29 SurgicalSurgVidLM1.531.282.211.831.661.552.532.631.821.672.611.972.241.902.872.69 1-shotGemini-3.1-flash-lite 2.101.802.832.592.272.123.073.352.602.533.282.773.403.143.883.73 Gemini-3.5-flash1.961.772.712.282.232.132.963.362.792.773.442.983.623.394.053.94 Opus-4.82.152.012.932.492.292.403.193.432.362.343.132.522.572.423.152.96 Ours MedClaw2.402.283.192.802.642.693.383.533.033.093.693.183.673.654.173.95 ∆ (1-shot baseline) +0.44 +0.51 +0.48 +0.52 +0.41 +0.56 +0.42 +0.17+0.43 +0.56 +0.41 +0.41 +0.27 +0.51 +0.29 +0.22 TABLE I BACKBONE-LIFT ABLATION ON THE NEUROSURGERY SPLIT (259 QUESTIONS): ONE-SHOT VS. THE SAME BACKBONE WRAPPED AS THE SUB-AGENT IN OUR FROZEN-WEIGHT HARNESS. SCORES ARE THE AVERAGE OVER THE FOUR DIMENSIONS. BEST PER COLUMN IN BOLD. SettingVideo-R1-7BQwen2.5-VL-7BReWatch-R1-7B one-shot2.062.041.54 + harness2.602.582.56 Gain+0.55+0.54+1.02 TABLE IV COMPONENT ABLATION ON THE NEUROSURGERY SPLIT (259 QUESTIONS): FROM THE FULL MEDCLAW HARNESS, WE REMOVE OR ALTER ONE LEARNED COMPONENT AT A TIME. SCORES ARE THE FOUR DIMENSIONS AND THEIR AVERAGE; ∆ IS THE CHANGE IN AVERAGE FROM THE FULL MODEL. ConfigurationCIDOCUTUAvg∆ MedClaw (full)2.542.523.303.222.90— w/o coarse-to-fine2.362.363.193.032.74−0.16 w/o directed re-look2.192.172.932.762.51−0.38 w/o HSD2.372.393.173.142.77−0.13 Judge reliability. To confirm that the automatic four- dimensional scores reflect clinical judgment rather than a self- referential metric, we validate our GPT-5.5 judge against neu- rosurgeon ratings, reporting the quadratic weighted Cohen’s κ [66] between judge and expert per dimension on a held-out subset (Table VI). On the neurosurgery subset (30 items scored by a neurosurgeon), the judge reaches substantial agreement on every dimension (overall κ = 0.72), and its mean scores TABLE V ORCHESTRATOR ABLATION ON THE NEUROSURGERY SPLIT (259 QUESTIONS): THE VISUAL SUB-AGENT AND ALL TOOLS ARE HELD FIXED; ONLY THE TEXT-ONLY ORCHESTRATOR CHANGES. SCORES ARE THE FOUR DIMENSIONS AND THEIR AVERAGE; ∆ IS THE CHANGE FROM THE OPUS-4.8 ORCHESTRATOR. OrchestratorCIDOCUTUAvg∆ Opus-4.82.542.523.303.222.90— DeepSeek-V32.072.362.942.892.57−0.33 DeepSeek-R12.222.213.042.752.56−0.34 TABLE VI JUDGE RELIABILITY: QUADRATIC WEIGHTED COHEN’Sκ BETWEEN THE GPT-5.5 JUDGE AND EXPERT RATINGS, PER DIMENSION, ON EACH SPLIT. Dimension NeurosurgerySVU κ (GPT-5.5) κ (GPT-5.5) CI (correctness)0.710.87 DO (detail)0.660.86 CU (context)0.720.85 TU (temporal)0.740.86 Overall0.720.86 track the expert’s almost exactly (2.95 vs. 2.94 overall). On the SVU subset (70 items scored by surgical experts), agreement is almost perfect on every dimension (overall κ = 0.86, mean 3.64 vs. 3.65); the higher figure is expected, since the lecture domain is less specialized than neurosurgery and thus easier to rate consistently. Across both splits the automatic scores align closely with expert judgment. AUTHOR et al.: MEDCLAW: HEURISTIC AGENT HARNESS FOR LONG-HORIZON SURGICAL VIDEO REASONING9 VI. LIMITATIONS AND FUTURE WORK Our benchmark only contains a limited number of questions, and the neurosurgery portion, in particular, is drawn from only a small set of recordings. In future work, we plan to annotate a larger collection of neurosurgery videos, focusing on long recordings beyond twenty minutes, and to generate more temporal-reasoning questions over them, so as to build a substantially larger benchmark on which to further test the performance and generality of our method. VII. CONCLUSION We addressed long-horizon temporal reasoning on long surgical videos without training any model to decide where to look. Separating reasoning from perception, we built a training-free agent harness in which a text orchestra- tor plans and unmodified vision–language sub-agents look, and a gradient-free, reward-gated Heuristic Skill Distillation loop mines reusable retrieval skills, notably directed re-look, from about one hundred preference examples rather than by fine-tuning. On MedClawBench, our de-leaked and doctor- grounded benchmark, the harness surpasses one-shot VLMs and general video-agent frameworks across all four dimen- sions, with the largest margins on the long, out-of-domain neurosurgery videos where long-horizon temporal reasoning is most stressed. Our ablations show that directed re-look contributes the most, that the harness delivers consistent gains across diverse frozen backbones, and that it survives swapping the orchestrator. Two properties of this training-free formula- tion are of particular relevance to clinical practice: auditability and data-efficient adaptation. Because every retrieval step is realized as an explicit tool call over unchanged weights, the evidence underlying each answer is exposed as an inspectable trace, allowing a clinician to review where the agent attended and how its conclusion was derived. Adaptation, in turn, is confined to an external skill library rather than to the model parameters, enabling extension to unseen procedures or specialties from a small number of examples, without retraining or large-scale annotation. REFERENCES [1] J. L. Green, V. Suresh, P. Bittar, L. Ledbetter, S. K. Mithani, and A. Allori, “The utilization of video technology in surgical education: a systematic review,” journal of surgical research, vol. 235, p. 171– 180, 2019. [2] J. D. Birkmeyer, J. F. Finks, A. O’reilly, M. Oerline, A. M. Carlin, A. R. Nunn, J. Dimick, M. Banerjee, and N. J. Birkmeyer, “Surgical skill and complication rates after bariatric surgery,” New England Journal of Medicine, vol. 369, no. 15, p. 1434–1442, 2013. [3] D. Kiyasseh, R. Ma, T. F. Haque, B. J. Miles, C. Wagner, D. A. Donoho, A. Anandkumar, and A. J. Hung, “A vision transformer for decoding surgeon activity from surgical videos,” Nature biomedical engineering, vol. 7, no. 6, p. 780–796, 2023. [4] L. Seenivasan, M. Islam, A. K. Krishna, and H. Ren, “Surgical-vqa: Visual question answering in surgical scenes using transformer,” in International Conference on Medical Image Computing and Computer- Assisted Intervention. Springer, 2022, p. 33–43. [5] G. Wang, J. Wang, W. Mo, L. Bai, K. Yuan, M. Hu, J. Wu, J. He, Y. Huang, N. Padoy et al., “Surgvidlm: Towards multi-grained sur- gical video understanding with large language model,” arXiv preprint arXiv:2506.17873, 2025. [6] W. Li, Y. Zhang, H. Sun, X. He, H. Gao, C. Ma, M. Hu, G. Wang, S. Yao, R. Yang et al., “Medscope: Incentivizing” think with videos” for clinical reasoning via coarse-to-fine tool calling,” in Forty-third International Conference on Machine Learning, 2026. [7] Y. Chen, F. Xue, D. Li, Q. Hu, L. Zhu, X. Li, Y. Fang, H. Tang, S. Yang, Z. Liu et al., “Longvila: Scaling long-context visual language models for long videos,” in International Conference on Learning Representations, vol. 2025, 2025, p. 18 227–18 246. [8] R. Qian, X. Dong, P. Zhang, Y. Zang, S. Ding, D. Lin, and J. Wang, “Streaming long video understanding with large language models,” Ad- vances in Neural Information Processing Systems, vol. 37, p. 119 336– 119 360, 2024. [9] Z. He, X. Qu, Y. Li, S. Huang, D. Liu, and Y. Cheng, “Framethinker: Learning to think with long videos via multi-turn frame spotlighting,” arXiv preprint arXiv:2509.24304, 2025. [10] C. Zhang, Z. Wang, Y. Ma, J. Peng, Y. Wang, Q. Zhou, J. Song, and B. Zheng, “Rewatch-r1: Boosting complex video reasoning in large vision-language models through agentic data synthesis,” arXiv preprint arXiv:2509.23652, 2025. [11] Z. Yang, S. Wang, K. Zhang, K. Wu, S. Leng, Y. Zhang, B. Li, C. Qin, S. Lu, X. Li et al., “Longvt: Incentivizing” thinking with long videos” via native tool calling,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026, p. 33 816–33 826. [12] T. M. Ward, D. M. Fer, Y. Ban, G. Rosman, O. R. Meireles, and D. A. Hashimoto, “Challenges in surgical video annotation,” Computer Assisted Surgery, vol. 26, no. 1, p. 58–68, 2021. [13] K. C. Demir, H. Schieber, T. Weise, D. Roth, M. May, A. Maier, and S. H. Yang, “Deep learning in surgical workflow analysis: a review of phase and step recognition,” IEEE Journal of Biomedical and Health Informatics, vol. 27, no. 11, p. 5405–5417, 2023. [14] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022. [15] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian et al., “Toolllm: Facilitating large language models to master 16000+ real-world apis,” in International Conference on Learning Representations, vol. 2024, 2024, p. 9695–9717. [16] S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive apis,” Advances in Neural Information Processing Systems, vol. 37, p. 126 544–126 565, 2024. [17] S. Hao, T. Liu, Z. Wang, and Z. Hu, “Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings,” Advances in neural information processing systems, vol. 36, p. 45 870–45 894, 2023. [18] D. Gao, L. Zhou, L. Ji, L. Zhu, Y. Yang, and M. Z. Shou, “Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, p. 14 773–14 783. [19] Y. Yao, Y. Yun, J. Wang, H. Zhang, D. Zhao, K. Tian, Z. Wang, M. Qiu, and T. Wang, “K-frames: Scene-driven any-k keyframe selection for long video understanding,” arXiv preprint arXiv:2510.13891, 2025. [20] L. Yao, H. Wu, K. Ouyang, Y. Zhang, C. Xiong, B. Chen, X. Sun, and J. Li, “Generative frame sampler for long video understanding,” in Findings of the Association for Computational Linguistics: ACL 2025, 2025, p. 17 900–17 917. [21] S. Jeoung, G. Huybrechts, B. Ganesh, A. Galstyan, and S. Bodapati, “Adaptive video understanding agent: Enhancing efficiency with dy- namic frame sampling and feedback-driven reasoning,” arXiv preprint arXiv:2410.20252, 2024. [22] H. Ge, Y. Wang, K.-W. Chang, H. Wu, and Y. Cai, “Framemind: Frame- interleaved video reasoning via reinforcement learning,” arXiv preprint arXiv:2509.24008, 2025. [23] R. Liu, Z. Liu, J. Tang, Y. Ma, R. Pi, J. Zhang, and Q. Chen, “Longvideoagent: Multi-agent reasoning with long videos,” in Proceed- ings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026, p. 40 404–40 416. [24] K. Feng, K. Gong, B. Li, Z. Guo, Y. Wang, T. Peng, J. Wu, X. Zhang, B. Wang, and X. Yue, “Video-r1: Reinforcing video reasoning in mllms,” Advances in Neural Information Processing Systems, vol. 38, p. 99 114–99 137, 2026. [25] X. Li, Z. Yan, D. Meng, L. Dong, X. Zeng, Y. He, Y. Wang, Y. Qiao, Y. Wang, and L. Wang, “Videochat-r1: Enhancing spatio-temporal per- ception via reinforcement fine-tuning,” arXiv preprint arXiv:2504.06958, 2025. [26] Z. Liu, Z. Sun, Y. Zang, X. Dong, Y. Cao, H. Duan, D. Lin, and J. Wang, “Visual-rft: Visual reinforcement fine-tuning,” in Proceedings 10IEEE TRANSACTIONS AND JOURNALS TEMPLATE of the IEEE/CVF International Conference on Computer Vision, 2025, p. 2034–2044. [27] Y. Chen, W. Huang, B. Shi, Q. Hu, H. Ye, L. Zhu, Z. Liu, P. Molchanov, J. Kautz, X. Qi et al., “Scaling rl to long videos,” Advances in Neural Information Processing Systems, vol. 38, p. 172 842–172 870, 2026. [28] Q. Wang, Y. Yu, Y. Yuan, R. Mao, and T. Zhou, “Videorft: Incen- tivizing video reasoning capability in mllms via reinforced fine-tuning,” Advances in neural information processing systems, vol. 38, p. 4350– 4376, 2026. [29] Y. Zhang, P. Du, B. Li, C. Wei, J. Miao, H. Zhang, S. Cai, Y. Wang, D. Jiang, Y. Zhang et al., “Rewardharness: Self-evolving agentic post- training,” arXiv preprint arXiv:2605.08703, 2026. [30] Y. Zhang, Y. Wang, Y. Zhu, P. Du, J. Miao, X. Lu, W. Xu, Y. Hao, S. Cai, X. Wang et al., “Clawbench: Can ai agents complete everyday online tasks?” arXiv preprint arXiv:2604.08523, 2026. [31] M. Zhuge, C. Zhao, D. Ashley, W. Wang, D. Khizbullin, Y. Xiong, Z. Liu, E. Chang, R. Krishnamoorthi, Y. Tian et al., “Agent-as-a-judge: Evaluate agents with agents,” arXiv preprint arXiv:2410.10934, 2024. [32] Y. Jiang, Y. Zhang, P. Zhang, Y. Li, J. Chen, X. Shi, and S. Zhen, “Incentivizing tool-augmented thinking with images for medical image analysis,” arXiv preprint arXiv:2512.14157, 2025. [33] Y. Jiang, Q. Li, B. Xu, H. Sun, C. Ding, J. Dong, Y. Cai, X. Zhang, and J. Yin, “Ibisagent: Reinforcing pixel-level visual reasoning in mllms for universal biomedical object referring and segmentation,” arXiv preprint arXiv:2601.03054, 2026. [34] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang et al., “Self-refine: Iter- ative refinement with self-feedback,” Advances in neural information processing systems, vol. 36, p. 46 534–46 594, 2023. [35] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Re- flexion: Language agents with verbal reinforcement learning,” Advances in neural information processing systems, vol. 36, p. 8634–8652, 2023. [36] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, “Voyager: An open-ended embodied agent with large language models,” arXiv preprint arXiv:2305.16291, 2023. [37] A. Zhao, D. Huang, Q. Xu, M. Lin, Y.-J. Liu, and G. Huang, “Expel: Llm agents are experiential learners,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 17, 2024, p. 19 632–19 642. [38] J. Liu, Y. Su, P. Xia, S. Han, Z. Zheng, C. Xie, M. Ding, and H. Yao, “Simplemem: Efficient lifelong memory for llm agents,” arXiv preprint arXiv:2601.02553, 2026. [39] P. Xia, K. Zeng, J. Liu, C. Qin, F. Wu, Y. Zhou, C. Xiong, and H. Yao, “Agent0: Unleashing self-evolving agents from zero data via tool-integrated reasoning,” arXiv preprint arXiv:2511.16043, 2025. [40] C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen, “Large language models as optimizers,” in International Conference on Learning Representations, vol. 2024, 2024, p. 12 028–12 068. [41] O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam et al., “Dspy: Compiling declarative language model calls into self-improving pipelines,” arXiv preprint arXiv:2310.03714, 2023. [42] Q. Zhang, C. Hu, S. Upasani, B. Ma, F. Hong, V. Kamanuru, J. Rainton, C. Wu, M. Ji, H. Li et al., “Agentic context engineering: Evolving contexts for self-improving language models,” arXiv preprint arXiv:2510.04618, 2025. [43] A. Zia, M. Berniker, R. Garcia Nespolo, C. Perreault, K. Bhattacharyya, X. Liu, Z. Wang, S. Kondo, S. Kasai, K. Hirasawa et al., “Intuitive surgical surgtoolloc challenge results: 2022-2023,” arXiv e-prints, p. arXiv–2305, 2023. [44] R. He, M. Xu, A. Das, D. Z. Khan, S. Bano, H. J. Marcus, D. Stoyanov, M. J. Clarkson, and M. Islam, “Pitvqa: Image-grounded text embedding llm for visual question answering in pituitary surgery,” in International Conference on Medical Image Computing and Computer-Assisted Inter- vention. Springer, 2024, p. 488–498. [45] M. Hu, K. Yuan, Y. Shen, F. Tang, X. Xu, L. Zhou, W. Li, Y. Chen, Z. Xu, Z. Peng et al., “Ophclip: Hierarchical retrieval-augmented learn- ing for ophthalmic surgical video-language pretraining,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025, p. 19 838–19 849. [46] R. Wang, J. Chen, K. Ji, Z. Cai, S. Chen, Y. Yang, and B. Wang, “Medgen: Unlocking medical video generation by scaling granularly- annotated medical videos,” arXiv preprint arXiv:2507.05675, 2025. [47] Y. Li, X. Yang, D. Xu, Y. Yu, L. Zhao, X. Hu, J. Li, and P.-A. Heng, “Surgpub-video: A comprehensive surgical video dataset for enhanced surgical intelligence in vision-language model,” arXiv preprint arXiv:2508.10054, 2025. [48] J. Wei, Z. Xiao, D. Sun, L. Gong, Z. Yang, Z. Liu, and J. Wu, “Surg- bench: A unified large-scale benchmark for surgical video analysis,” arXiv preprint arXiv:2506.07603, 2025. [49] M. O. Drago, L. Carlini, P. C. Balyemez, D. Pierantozzi, C. Lena, C. Hassan, D. Stoyanov, E. D. Momi, S. Bano, and M. I. Hoque, “Surg- vivqa: temporally grounded video question answering for surgical scene understanding,” International Journal of Computer Assisted Radiology and Surgery, p. 1–9, 2026. [50] G. Wang, Y. Wennuo, X. Ma, Z. Zhong, Z. Wu, E. Wu, R. Qu, W. P. Cheah, J. Ren, and L. Shen, “Eyepcr: A comprehensive benchmark for fine-grained perception, knowledge comprehension and clinical reason- ing in ophthalmic surgery,” arXiv preprint arXiv:2509.15596, 2025. [51] B. He, H. Li, Y. K. Jang, M. Jia, X. Cao, A. Shah, A. Shrivastava, and S.-N. Lim, “Ma-lmm: Memory-augmented large multimodal model for long-term video understanding,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, p. 13 504–13 514. [52] Y. Li, C. Wang, and J. Jia, “Llama-vid: An image is worth 2 tokens in large language models,” in European Conference on Computer Vision. Springer, 2024, p. 323–340. [53] Z. Ma, C. Gou, H. Shi, B. Sun, S. Li, H. Rezatofighi, and J. Cai, “Drvideo: Document retrieval based long video understanding,” in Pro- ceedings of the Computer Vision and Pattern Recognition Conference, 2025, p. 18 936–18 946. [54] H. Kwan, H. Li, R. Zhang, M. Zhang, X. Kong, A. Rao, J. Xie, and S. Liu, “Video-mme-logical: A controlled diagnostic benchmark for video temporal-logical reasoning,” arXiv preprint arXiv:2606.27828, 2026. [55] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2502.13923 [56] J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao et al., “Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,” arXiv preprint arXiv:2504.10479, 2025. [57] B. Zhang, K. Li, Z. Cheng, Z. Hu, Y. Yuan, G. Chen, S. Leng, Y. Jiang, H. Zhang, X. Li et al., “Videollama 3: Frontier multimodal foundation models for image and video understanding,” arXiv preprint arXiv:2501.13106, 2025. [58] C. X. Liang, P. Tian, C. H. Yin, Y. Yua, A.-H. Wei, M. Li, X. Song, T. Wang, Z. Bi, M. Liu et al., “A comprehensive survey and guide to multimodal large language models in vision–language tasks,” Computa- tion, vol. 14, no. 6, p. 125, 2026. [59] Y. Tang, J. Bi, S. Xu, L. Song, S. Liang, T. Wang, D. Zhang, J. An, J. Lin, R. Zhu et al., “Video understanding with large language models: A survey,” IEEE Transactions on Circuits and Systems for Video Technology, 2025. [60] B. Li, T. Yan, Y. Pan, J. Luo, R. Ji, J. Ding, Z. Xu, S. Liu, H. Dong, Z. Lin et al., “Mmedagent: Learning to use medical tools with multi- modal agent,” in Findings of the Association for Computational Linguis- tics: EMNLP 2024, 2024, p. 8745–8760. [61] N. Fathi, A. Kumar, and T. Arbel, “Aura: A multi-modal medical agent for understanding, reasoning and annotation,” in International Workshop on Agentic AI for Medicine. Springer, 2025, p. 105–114. [62] Z. Wang, J. Wu, L. Cai, C. H. Low, X. Yang, Q. Li, and Y. Jin, “Medagent-pro: Towards evidence-based multi-modal medical diagnosis via reasoning agentic workflow,” arXiv preprint arXiv:2503.18968, 2025. [63] D. Wang, T. Cheng, S. Wang, Y. F. Chen, and Y. Yin, “Smr-agents: Syn- ergistic medical reasoning agents for zero-shot medical visual question answering with mllms,” Information Processing & Management, vol. 63, no. 1, p. 104297, 2026. [64] H. Yao, J. Huang, Y. Qiu, M. K. Chen, W. Liu, W. Zhang, W. Zeng, X. Zhang, J. Zhang, Y. Song et al., “Mmreason: An open-ended multi-modal multi-step reasoning benchmark for mllms toward agi,” in 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025, p. 273–283. [65] M. Allan, S. Kondo, S. Bodenstedt, S. Leger, R. Kadkhodamoham- madi, I. Luengo, F. Fuentes, E. Flouty, A. Mohammed, M. Pedersen et al., “2018 robotic scene segmentation challenge,” arXiv preprint arXiv:2001.11190, 2020. [66] J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement, vol. 20, no. 1, p. 37–46, 1960.

Instant research discovery