Paper deep dive
LazySloth: Bounded LLM-based Lazy Tree Search for Fast Long Video Comprehension
Arka Mukherjee, Kaleen Shrestha, Larissa Zhu, Maja Matarić
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Modern vision-language models (VLMs) have shown promising results in long-video understanding due to the rich semantic information they can capture. However, most methods focus on coarse captioning of extracted image frames that are computationally inefficient and require models with large context windows. While past work has explored efficient methods through multimodal retrieval-augmented generation (RAG), they rely on lossy embeddings that lose temporal context and fine-grained detail. Few works to date have investigated how VLM-based query-relevant information retrieval can be optimized. We introduce LazySloth, an efficient tree-based search method that speeds up video comprehension and retrieval tasks 2.9-8.3x (compared to existing agentic methods) through bounded captioning of portions of the video considered irrelevant by a VLM of the video. Compared to contemporary specialized video-understanding VLMs and RAG-based methods, LazySloth achieved similar or better final task accuracy across two recent open-source base VLMs--Gemma 4 31B and Qwen3.6 27B--across four benchmarks. LazySloth reduced the gap between the base open-source model and a closed-source model, GPT-4o. Ablations showed that replacing VLM scene understanding with CLIP-based retrieval cost 8.8-19.9% in accuracy, while lazy tree construction matches eager construction at a fraction of the captioning cost. With LazySloth, we demonstrate the possibility of faster long-video comprehension without substantial loss in performance.
Tags
Links
- Source: https://arxiv.org/abs/2609.37426v1
- Canonical: https://arxiv.org/abs/2609.37426v1
Trouble viewing inline? Open PDF directly →
Full Text
64,002 characters extracted from source content.
Expand or collapse full text
LazySloth: Bounded LLM-based Lazy Tree Search for Fast Long Video Comprehension Arka Mukherjee Kaleen Shrestha Larissa Zhu Maja Matarić Abstract Modern vision-language models (VLMs) have shown promising results in long-video understanding due to the rich semantic information they can capture. However, most methods focus on coarse captioning of extracted image frames that are computationally inefficient and require models with large context windows. While past work has explored efficient methods through multimodal retrieval-augmented generation (RAG), they rely on lossy embeddings that lose temporal context and fine-grained detail. Few works to date have investigated how VLM-based query-relevant information retrieval can be optimized. We introduce LazySloth, an efficient tree-based search method that speeds up video comprehension and retrieval tasks 2.9-8.3x (compared to existing agentic methods) through bounded captioning of portions of the video considered irrelevant by a VLM of the video. Compared to contemporary specialized video-understanding VLMs and RAG-based methods, LazySloth achieved similar or better final task accuracy across two recent open-source base VLMs–Gemma 4 31B and Qwen3.6 27B–across four benchmarks. LazySloth reduced the gap between the base open-source model and a closed-source model, GPT-4o. Ablations showed that replacing VLM scene understanding with CLIP-based retrieval cost 8.8–19.9% in accuracy, while lazy tree construction matches eager construction at a fraction of the captioning cost. With LazySloth, we demonstrate the possibility of faster long-video comprehension without substantial loss in performance. 1School of Computer Engineering, Kalinga Institute of Industrial Technology (KIIT) Bhubaneswar 2Computer Science Department, Viterbi School of Engineering, University of Southern California 1 Introduction The frontier for video understanding has moved from short, curated clips (Mangalam, Akshkulakov, and Malik 2023) to long-form content such as feature-length films, hour-long lectures, and egocentric recordings 11 1 https://ego4d-data.org/ (Hong et al. 2025; Shu et al. 2025), to weeks-long videos spanning multiple topics (Rege et al. 2026). Existing work on answering a question from such content focused on explicit localization (finding the handful of seconds that matter) (Zuo et al. 2025) and comprehension (reasoning over what happens in the localized frames) (Yeo et al. 2026). Modern VLMs are exceptionally good at processing long-context inputs across modalities (Wang et al. 2025c), improving over contrastive methods that showed little success (Bagad, Tapaswi, and Snoek 2023). Figure 1: Overview of how LazySloth bounds captioning. Each level captures more fine-grained information, with the parent root representing all of the video. Grayed-out nodes are never visited, which reduces the total time to navigate through the hierarchical tree. However, the semantic expertise of VLMs comes at a tradeoff, and two main paradigms of VLM-based video understanding have emerged. The first places sampled frames directly into the model’s context and asks for an answer in one shot (Shen et al. 2025; Shu et al. 2025). This is bounded by the context window and the quadratic cost of attention: an hour of video sampled at even one frame per second (FPS) yields thousands of frames and hundreds of thousands of visual tokens, forcing frame budgets so aggressive that the evidence needed to answer is often not sampled. To combat this, a second paradigm has emerged that converts the video into text: a VLM captions every frame or every fixed-length segment, and an LLM then reasons over the resulting corpus (Zuo et al. 2025; Yeo et al. 2026; Wang et al. 2025b). This circumvents the context limit and yields an easily searchable artifact. However, this requires one VLM call per unit of video, so the cost scales linearly with duration regardless of the question asked. A question about a five-second event still requires captioning the other fifty-nine minutes. Since the VLM call is much more expensive than any other component of the system, this wasted captioning is the bottleneck. Figure 2: Overview of the LazySloth pipeline. (A) Corpus creation, which produced an ordered list of frames after applying optional uniform frame sampling to the input video. (B) Hierarchical frame bunching, which initialized the first few levels of the search tree. (C) Lazy generation of tree nodes by the captioner VLM C. (D) Agentic Direct Corpus Interaction (DCI), where the search VLM S explored the tree with backtracking to traverse alternative branches. (E) Final answer prediction by the reasoner VLM (R), conditioned on the textual evidence and representative frames collected during the search. Multimodal retrieval-augmented generation (RAG) addresses precisely this issue by retrieving candidate segments with contrastive image-text encoders before invoking a VLM (Radford et al. 2021; Most et al. 2025). Retrieval is cheap, but the representation is lossy (Yu et al. 2025; Lumer et al. 2026) as the contrastive encoder compresses a frame or even a pool of frames into a single vector. This discards temporal, motion, and fine-grained detail. Efficient methods retrieve with lossy embeddings while accurate methods caption extensively with VLMs. Few works have asked how VLM-based retrieval itself should be optimized. To address this, we introduce LazySloth (Figure 2), a tree-based search method that generates captions lazily. LazySloth summarizes the video as a top-down tree that is never fully instantiated. We show a LLM a root summary of the whole clip and a handful of coarse branch summaries, from which it descends into the branch it determines is most relevant to the question. Only that branch’s children are captioned, and we repeat this descent until an interval is small enough to expand into per-frame detail. Finally, a multimodal reasoner answers from the evidence surfaced along the path together with the key frames the search resolved to. This bounds the captioning cost to the depth of the descent rather than the length of the video, and grows logarithmically where prior methods grow linearly. Across four long-video benchmarks and two recent open-source VLMs, LazySloth was 2.9–8.3× faster than existing agentic methods while matching or exceeding both specialized long-video VLMs and RAG-based pipelines in accuracy. It also narrowed the accuracy gap between the tested open-source models and a closed-source model, GPT-4o. Our contributions are: 1. LazySloth, a lazy hierarchical search method that bounds VLM captioning to the branch the search actually explores, making captioning cost logarithmic rather than linear in video length. 2. Evidence that VLM-based retrieval on frame bunches (that captures temporal change unlike individual frames) can be made efficient without falling back on lossy embeddings. 3. An evaluation across four long-video benchmarks and two open base VLMs against specialized video VLMs, RAG pipelines, and agentic baselines, showing 2.9–8.3× speedups at comparable or better accuracy. 4. Ablations that separately isolate the contribution of the hierarchy, of its laziness, and of VLM-based versus embedding-based scene understanding. 2 Related Work 2.1 Vision–Language Models (VLMs) General Purpose Video VLMs Open VLM series such as VideoLLaMA 22 2 https://github.com/damo-nlp-sg/videollama3, Qwen-VL 33 3 https://github.com/qwenlm/qwen3-vl, InternVL 44 4 https://github.com/OpenGVLab/InternVL, LLaVA-Video/OneVision 55 5 https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-2, Gemma 66 6 https://deepmind.google/models/gemma/gemma-4/, and GLM 77 7 https://z.ai/blog/glm-5 are extensively used for building agentic video-understanding systems. These VLMs, however, ingest video data as sequences of uniformly sampled frames within their context length budgets, and rely on aggressive truncation or down-sampling for longer and high-resolution videos. Although this allows for graceful scaling, smaller open models degrade after roughly 30 frames (Wu et al. 2024), consistent with the lost-in-the-middle effect (Liu et al. 2024). Using context-as-memory did not provide usable memory (see experiments in Appendix B), which motivated our search for efficient retrieval systems that support hundreds of frames. Specialized VLMs for Long Videos Specialized VLMs have also emerged to address the unique problems of long video understanding. VideoLLaMA 3 (Zhang et al. 2025), a well-known model in this space, utilizes high-quality vision-centric image-text datasets in its training corpus and achieved 2-3% gains on benchmarks like Video Multi-Modal Evaluation (Video-MME) (Fu et al. 2025) and Multi-task Long Video Understanding (MLVU) (Zhou et al. 2025). Other notable models include VideoChat-Flash (Li et al. 2025a) and video-SALMONN 2+ (Tang et al. 2026), which employs model architecture and training data innovations to improve video understanding. More recently, VideoChat-R1 (Li et al. 2025b) integrates spatio-temporal specific rewards with reinforcement learning to improve temporal grounding of VLMs. The core limitations of frame sampling and utilizing memory as context, however, remain (Section 4). 2.2 Long Video-Understanding Datasets and Benchmarks Several high-quality long video-understanding datasets and benchmarks have been released to evaluate foundation model capabilities and train small-scale specialized VLMs. LongVideoBench (Wu et al. 2024) and Video-MME (Fu et al. 2025) are widely used for model evaluation in a multiple-choice question (MCQ) format, with videos sourced from web platforms such as YouTube88 8 https://w.youtube.com. More recently, evaluation has focused on extremely long (weeks-long, rather than hours) video understanding in VLMs (Wang et al. 2025a). The Ego4D and EgoLife 99 9 https://ego4d-data.org/, https://egolife-ai.github.io/ corpora have also been utilized for testing long-format egocentric content, with datasets such as EgoSchema (Mangalam, Akshkulakov, and Malik 2023) and EgoMemReason (Wang et al. 2026) testing short and week-long comprehension, respectively, in MCQ format. Benchmarks studied in this paper are summarized in Table 1. 2.3 Methods for Long Video-Understanding RAG-Based Methods Multimodal retrieval-augmented generation (RAG) has been explored for long-video understanding. Yeo et al. (2026) studied RAG-based methods such as the text-based LightRAG (Guo et al. 2025) and HippoRAG (Gutierrez et al. 2024), and the multimodal Video-RAG (Luo et al. 2025). Our method resolves the concern of relying on lossy multimodal embeddings by utilizing semantic LLM-based scene understanding (Section 4.3). Agentic methods Several agentic methods have been proposed in the literature to improve VLM-based long video comprehension techniques. Some notable recent work includes VideoLucy (Zuo et al. 2025), which evaluated the effectiveness of coarse-to-fine-grained memory-based retrieval. Other related work has evaluated the effect of storing specialized memory banks for episodic, semantic, and visual data separately (Yeo et al. 2026), while Wang et al. (2025b) uses self-reflection-based iterative searching with VLMs. However, these works did not evaluate how VLM-based long video search can be optimized without significant performance reductions. 3 LazySloth Our proposed method, LazySloth (Figure 2), answers a query Q over a long video without captioning every frame. It utilizes three VLMs: a captioner model, C, that builds a top-down caption tree lazily, a searcher VLM S, and a reasoner VLM R. S repeatedly navigates into a branch that it determines to be most relevant. R finally answers the query given the selected caption corpus (temporal context) plus the key frames of the region the search narrowed to (visual evidence). After decoding the video clip for frame extraction, we get an ordered frame list F=⟨f1…fN⟩F= f_1… f_N (timestamps are provided in context). The top-down lazy tree is then initiated, spanning all N frames, which we caption with C on ≤mb≤ m_b uniformly sampled representative frames. At every step, the root is split into k (branch factor) roughly equal children, each of which represents a contiguous spans of the video. Each span is captioned the same way with C. We initialize the evidence corpus as the root caption plus the child captions, and the searcher S’s selected key-frame range currPos is set to the whole clip [0,N)[0,N) at timestamp t. Each node is assigned an id. Algorithm 1 Hierarchical lazy tree generation and retrieval 1: Video V, query Q, labels L; captioner C, searcher S, reasoner R; branch factor k, leaf size ℓ , key-frame budget mrm_r 2: answer y y 3: F←SampleFrames(V)F← SampleFrames(V); N←|F|N←|F| 4: C captions root [0,N)[0,N) and its k children 5: ℱ←childrenF 6: ℱprev←∅F_prev← 7: ℰ←root, child captionsE←\root, child captions\ 8: [a,b)←[0,N)[a,b)←[0,N) 9: while round budget remains do ⊳ text-only tree search, budget based on backtracking limit 10: S reads Q and the frontier captions caption C:c∈ℱ\caption $C$\!:\!c \ and emits ONE action: 11: if Drill C with c∈ℱ∪ℱprevc _prev and bc−ac>ℓb_c-a_c> then ⊳ c∈ℱprev⇒c _prev backtrack one level 12: split C into k children; C captions only them ⊳ unchosen siblings never captioned 13: ℱprev←F_prev← frontier of C 14: ℱ←childrenF 15: [a,b)←[ac,bc)[a,b)←[a_c,b_c) 16: ℰ+=their captionsE +=their captions 17: else if Expand C with c∈ℱ∪ℱprevc _prev and bc−ac≤ℓb_c-a_c≤ then 18: C captions each frame of C 19: [a,b)←[ac,bc)[a,b)←[a_c,b_c) 20: ℰ+=per-frame captionsE +=per-frame captions 21: else if Answer ySy_S then ⊳ ySy_S: searcher’s provisional answer 22: break 23: else⊳ if S outputs an invalid id, Drills on a leaf node, or Expands on a non-leaf 24: prompt S to retry 25: K←K← evenly sample ≤mr≤ m_r frames of F[a:b)F[a\!:\!b) in time order ⊳ key frames of the searcher-narrowed region 26: yR←R(Q,L,ℰ,K)y_R← R(Q,L,E,K) ⊳ multimodal: ℰE = temporal context, K = visual evidence 27: return y^←yR y← y_R if yR≠Insufficienty_R≠ Insufficient else ySy_S A text-only top-down search then follows, during which S sees the question, an action menu, and a transcript showing the root and current-level captions. Here, searcher S must output exactly one action from the following list: 1. DRILL <id>: This action is used to descend into a frontier branch. A node is split into ≤k≤ k children, which are then captioned lazily by C. For the sibling nodes not chosen by S, LazySloth never runs captioning, which bounds the cost. The newly generated children are now the frontier of the tree, currPos is updated, and the new captions are appended to the evidence corpus. We do not discard the previous frontier; instead, we retain it in the evidence corpus so S can DRILL a sibling id to backtrack one level. 2. EXPAND <id>: If a branch already has ≤ℓ≤ frames (i.e, it is at leaf size), the VLM can choose to EXPAND instead. This step captions every frame in that leaf range episodically, which is then appended to the evidence corpus. Finally, currPos is set to that leaf. 3. PROVISIONAL ANSWER <Q>: The searcher S’s answer to Q, based on the collected evidence, which ends the search loop. Finally, both the textual evidence collected from the captioned corpus by the searcher S and ≤mr≤ m_r representative frames in the leaf nodes of the hierarchical tree that S navigated down to are passed to the multimodal reasoner R, which then reasons on a per-frame basis and emits an output label or open-ended text based on the query. The impact of a post-hoc multimodal reasoner is ablated in Table 4. Let d=⌈logk(N/ℓ)⌉d= _k(N/ ) denote the number of levels navigated until a leaf of at most ℓ frames is reached. Since each navigation captions at most k new nodes and the final expansion captions at most ℓ frames, the total number of captioning operations is bounded by ≤dk+ℓ=k⌈logkNℓ⌉+ℓ=O(klogkN+ℓ).C≤ dk+ =k _k N + =O\! (k _kN+ ). (1) For a fixed branching factor k and leaf size ℓ , this simplifies to O(klogkN)O(k _kN), compared to the O(N)O(N) frame-captioning cost incurred by dense captioning. We summarized LazySloth’s optimized video search in Algorithm 1. 3.1 Evaluation Benchmarks Video- MME LongVideo Bench LV Bench Ego Schema Source YouTube Web YouTube Ego4D Domain 6 domains Multi-topic Longform Egocentric # Videos 900 753 103 500 # Questions 2,700 1,337 1,549 500 # Options 4 4–5 4 5 Avg. length (min) 17.0 7.9 67.3 3.0 Range (min) 0.2–59.7 0.1–59.6 30.1–139.9 3.0 fixed Total duration (h) 255.3 99.8 115.5 25.0 Table 1: Long-video comprehension benchmarks used in our evaluation: Video-MME (Fu et al. 2025), LongVideoBench (Wu et al. 2024), LVBench (Wang et al. 2025a), and EgoSchema (Mangalam, Akshkulakov, and Malik 2023). All four are multiple-choice and scored by exact match. The suite spans web, film/television, and egocentric content, and in total, we evaluate almost 500 hours of video. We evaluated the proposed framework on four established MCQ video question-answering benchmarks with well-maintained leader boards, spanning movie clips, general web videos, long-form videos, and egocentric videos, following the discussion in Section 2.2: Video-MME (Fu et al. 2025), LongVideoBench (Wu et al. 2024), LVBench (Wang et al. 2025a), and EgoSchema (Mangalam, Akshkulakov, and Malik 2023). Table 1 summarizes each dataset’s statistics. For all of these benchmarks, we conducted a full run and reported task accuracy as an exact match of the LLM’s output MCQ label to the ground truth. 3.2 Baselines For each dataset, we first quantified random baseline performance as 1/N1/N, where N is the number of MCQ options for Video-MME, LongVideoBench, LVBench, and EgoSchema. We compared LazySloth to multiple classes of methods starting with general video-understanding models, specifically Qwen3.6 27B1010 10 https://huggingface.co/Qwen/Qwen3.6-27B and Gemma 4 31B 1111 11 https://huggingface.co/google/gemma-4-31B-it, using at most 64 frames sampled from each video to stay within their context windows (Yin et al. 2024; Fu et al. 2025; Doorenbos, Spurio, and Gall 2025). Next, specialized video-understanding models such as VideoChat-R1 (Li et al. 2025b) and VideoLLaMA 3 (Zhang et al. 2025) were tested. To understand how our method compares to RAG-based methods, we compared it against the text-based LightRAG (Guo et al. 2025) followed by the multimodal Video-RAG (Luo et al. 2025). Finally, among modern agentic baselines, we included VideoLucy (Zuo et al. 2025) and WorldMM (Yeo et al. 2026). 3.3 Ablations We designed two ablations to investigate (a) the behavior of lazy tree generation, (b) the impact of the tree data structure, and (c) the effectiveness of using VLMs instead of multimodal embeddings. We replaced the searcher VLM S with embedding-based retrieval by scoring the tree nodes using Contrastive Language–Image Pretraining Score (CLIPScore) (Hessel et al. 2021). This enabled greedy best-first search (Greedy BFS) over the tree. The searcher S first navigated that tree lazily with DRILL and EXPAND, and generated its answer using the collected evidence. This allowed us to illustrate the impact of VLM-based scene understanding. Next, we ablated the effect of lazy tree generation by eagerly constructing the full hierarchy with CLIPScore-based nodes. The searcher S then received the full tree as context while collecting evidence. We kept all other components, including the backtracking from the main method, intact, ablating the lazy, on-demand (top-down, implicitly pruned) tree construction of LazySloth. These comparisons allowed us to illustrate the efficiency gains of Algorithm 1 over naive brute-forced methods. 3.4 Experimental Setup & Models Tested All experiments on existing agentic methods and LazySloth were repeated on two base searcher S and reasoner R VLMs (Qwen3.6 27B and Gemma 4 31B). We ran Qwen 3.6 and Gemma 4’s compiled GGML Universal File (GGUF) locally through LM Studio1212 12 https://lmstudio.ai/. The captioner model C was set to Nvidia’s Nemotron 3 Nano Omni 30B for an equitable comparison of each method’s merit (which was chosen based on caption quality experiments outlined in Appendix C). We loaded Nemotron 3’s compiled GGUF locally through LM Studio. All experiments were run on four Nvidia H100 80 GB, one RTX Pro 6000 96 GB, and two RTX 3090 24 GB GPUs. All videos were sampled at 1 FPS and models initialized with a context window of 79,104 tokens to fit on our available hardware. Accuracy (%) Method Video-MME (w/o subs) LongVideo Bench LV Bench Ego Schema Random baseline 25.0 21.3 25.0 20.0 Direct inference (64 frames) GPT-4o‡ 71.9 66.7 48.9 72.2 Qwen 3.6 27B 45.7 31.8 27.7 38.4 Gemma 4 31B 65.8 52.8 40.7 65.4 Specialized video models VideoChat-R1.5 52.5 49.8 33.4 41.6 VideoChat-Flash† 55.6 — 33.2 44.1 VideoLLaMA 3 56.2 51.0 35.4 60.6 Time-R1† — — 37.6 31.1 RAG-based LightRAG† 46.6 — 30.4 — Video-RAG 7B† 55.4 — 33.1 — Agentic VideoLucy (Qwen) 34.3 43.9 35.4 51.1 VideoLucy (Gemma) 71.3 65.7 53.8 70.9 WorldMM (Qwen) — — 29.5 — LazySloth (Qwen) 61.4 53.2 39.1 73.9 LazySloth (Gemma) 66.5 59.5 45.9 69.1 Table 2: Accuracy results on four long video-understanding benchmarks. Bold marks the best open-source result per benchmark; underline marks the second-best open-source result. † Numbers sourced from Yeo et al. (2026). ‡ GPT-4o numbers are taken from the public leaderboards of the respective benchmarks; we do not report a newer GPT release, as none could be located on the maintained leaderboards. 4 Results Table 2 shows the accuracy of LazySloth compared to existing models and methods. Our method outperformed specialized models and retrieval-based methods. On Qwen 3.6 27B, LazySloth exceeded the strongest specialized video-understanding model on every benchmark: by 5.2 points over VideoLLaMA 3 on Video-MME, 2.2 on LongVideoBench, 1.5 over Time-R1 on LVBench, and 13.3 over VideoLLaMA 3 on EgoSchema. Importantly, these gains were realized with a general-purpose base model (i.e., backbone), and no video-specific training. LazySloth also outperformed the best RAG-based baseline (Video-RAG 7B) by at least 6.0 points on the two benchmarks where their numbers were available, which was consistent with our claim that embedding retrieval discards fine-grained detail required for sophisticated video understanding. Among other agentic methods such as VideoLucy, using Qwen3.6 27B as the backbone, we saw that LazySloth improved over VideoLucy by 27.1, 9.3, 3.7, and 22.8 points, and over WorldMM by 9.6 points on LVBench. The margin over direct inference was 11.4–35.5 points. We note that VideoLucy underperformed compared to direct inference on Video-MME with the same backbone (34.3% vs. 45.7%), whereas LazySloth improved on it by 15.7%. Improvements were markedly smaller on Gemma 4 31B, where LazySloth trailed VideoLucy on all four benchmarks by 1.8–7.9 points, which demonstrated that the improvement margin was largely dependent on the specific model: on average, Qwen3.6 27B gained 21.0% in accuracy with our pipeline compared to 4.1% for Gemma 4 31B over direct inference. Further experiments with more backbone VLMs would confirm this pattern as a claim. Despite lower performance gains with Gemma 4, LazySloth achieved comparable accuracies at 2.9–8.3× lower median wall-clock time per video than competing methods, as we discuss next. 4.1 Efficiency Analysis Figure 3: Efficiency comparison of LazySloth, a brute-forced eager tree, VideoLucy, and WorldMM by median seconds per video (log y, broken above the working band) on LVBench, which was chosen as it has the longest videos in our test corpus. Bootstrapped 95% confidence intervals are plotted at each video length value. All methods shared the Qwen3.6-27B reasoner and Nemotron captioner. To better understand the gains realized with LazySloth, we first compared the median wall clock time across our method and the two agentic baselines, VideoLucy and WorldMM. As a control, we included the brute-forced eager bottom-up tree (Section 3.3). For these tests, we focused only on the LVBench dataset as it contained the longest videos in our test set (Table 1). Figure 3 shows that our method was the fastest, with median time to answer a single LVBench query at 194 seconds (secs) compared to 611 secs on VideoLucy, and 1,139 secs on WorldMM. On the longest videos (140 minutes long), our method was 3× faster than VideoLucy and 7.7× faster than WorldMM. The gap with the brute-forced eager bottom-up tree remained disproportionately high at 287× slower than LazySloth. Direct inference, which feeds 64 frames into the VLM’s memory, stayed constant at 57 secs, and resulted in notably lower task accuracies. Additional discussion about efficiency can be found in Appendix D. Next, we decomposed each method into three categories: time spent captioning, time spent reasoning, and overhead (e.g., video decode, I/O, etc). Figure 4 shows the results on LVBench. We observed that reasoning dominated WorldMM’s median wall clock time (811s), as it used triple-extraction with VLMs across episodic, semantic, and visual memories. VideoLucy, on the other hand, was dominated by its multi-frame coarse-window captions (67%) as compared to reasoning. Our method optimized both frontiers, and neither captioning nor reasoning dominated, ensuring 2.9–8.3× faster median wall clock time than the baselines. Figure 4: WorldMM vs VideoLucy vs LazySloth median wall clock time per video on LVBench decomposition by captioning, reasoning, and other (I/O, etc.). VideoLucy spends a lot of time captioning; WorldMM spends a lot on reasoning. Our method optimizes both frontiers for the lowest total time. 4.2 Behavioral Analysis Figure 5: Comparing the accuracy difference between LazySloth and direct inference across benchmarks and models with respect to search tree depth. Figure 6: Accuracy with respect to the caption budget used by the tree search with Qwen3.6 27B, where budget=1budget=1 root ++ 44 seed branches ++ 44 per successful DRILL ++ 44 per EXPAND. Beyond a single descent into the tree, accuracy declined across all benchmarks. Figure 7: Analysis of temporal localization on correctly answered LongVideoBench questions that were annotated with specific timestamps. Blue: the portion where LazySloth’s final window contained the referenced span. Gray: the rate expected from placing a window of the same width at random, averaged per question. We next characterized how LazySloth’s search behaves and what its behavior predicts about the final accuracy. Figure 5 plots the accuracy difference between LazySloth and direct inference against the depth the search reached. On Qwen3.6 27B, we noticed positive differences across all tested benchmarks, while on Gemma 4 31B, we noticed the differences were zero throughout. This was consistent with the model-based performance differences in Section 4. Multi-level tree search did not impact Gemma 4 31B’s final accuracy, and likely negatively hurt its long video capabilities. Next, Figure 6 analyzes the performance of LazySloth with respect to the caption budget during search on Qwen 3.6 27B. The first descent into the tree (moving from five caption calls to nine) clearly helped the VLM, as accuracy rose by 25%. Beyond that, accuracy fell on Video-MME, LongVideoBench, and LVBench. On EgoSchema, accuracy declined monotonically, likely as videos in the dataset were extremely short (three minutes each). This decline was largely correlated with harder questions, where the search failed to resolve the query cheaply in higher levels of the tree. We additionally investigated LazySloth’s ability to find the exact span in the video that contains the answer to the query. Compared to randomly choosing similar-sized windows, Figure 7 shows each VLM significantly improved (1.8−2.1×1.8-2.1×) at identifying the correct window (Qwen3.6 27B n=163n=163 and Gemma 4 31B n=171n=171). However, while the choice was not arbitrary, the final window did not contain the actual span for 86−91%86-91\% of the questions, where the model still answered the question correctly. 4.3 Ablation Results We replaced VLM captioning with a frozen CLIP encoder, and found that accuracy fell by 8.88.8–19.919.9% with CLIPScored nodes compared to a Qwen3.6 27B base model (Table 3). VLM-based scene understanding was a major contributor to LazySloth’s performance. Holding the scorer fixed at CLIPScore, the lazy top-down and eager bottom-up construction differed by at most 3.13.1% with no consistent direction (Video-MME 41.541.5 vs. 43.443.4, LongVideoBench 40.540.5 vs. 42.142.1, EgoSchema 55.655.6 vs. 56.456.4). Lazy expansion largely did not affect accuracy, explaining how LazySloth speeds up video retrieval (Figure 3) without a massive performance loss. Finally, we explored the effect of having a separate reasoner VLM in LazySloth, and found that the search trajectory largely determined the final answer (Table 4). The reasoner improved micro-averaged accuracy by 0.30.3%, with no clear directionality. The searcher’s provisional answer was thus very nearly as accurate as the final one, and removing the reasoner’s VLM call could improve efficiency further. Accuracy (%) Configuration Video-MME LongVideoBench LVBench EgoSchema LazySloth 61.4 53.2 39.1 73.9 Scene understanding (CLIPScore, greedy best-first) lazy top-down tree 41.5 40.5 27.2 55.6 eager bottom-up tree 43.4 42.1 30.3 56.4 Table 3: Ablation study of LazySloth with Qwen3.6 27B. Group (a) varies the memory structure while keeping VLM-based scene understanding; group (b) replaces VLM captioning and search with a frozen CLIP encoder that scores each tree node by CLIPScore and navigates by greedy BFS. Dataset Searcher (%) Reasoner (%) Δ (Reasoner−-Searcher) EgoSchema 76.6 76.9 +0.3 LongVideoBench 58.6 60.8 +2.2 LVBench 42.5 41.4 −1.1-1.1 Video-MME 62.3 62.6 +0.2 MCQ Micro-Avg 58.0 58.3 +0.3 Table 4: Agreement between the searcher’s selected answer and the reasoner’s final answer with Qwen 3.6 27B. We observe that using a separate reasoner results in a 0.3% accuracy gain over always trusting the search model’s answer. 5 Conclusion We introduce LazySloth, a tree-based search method for long-video understanding that bounds captioning to regions a VLM deems relevant rather than exhaustive captioning. Across four benchmarks and two open-source VLMs, LazySloth was 2.92.9 – 8.3×8.3× faster than existing methods while matching or exceeding specialized video-understanding VLMs and RAG-based methods in accuracy. Detailed failure analysis showed gaps in VLM behavior where it navigated to the wrong window or degraded in performance with extra caption calls. Further, our ablations show LazySloth remained competitive due to VLM-based scene understanding, and lazy construction was close in accuracy with eager construction, thereby resulting in a method that speeds up long-video retrieval without a substantial cost in task performance. 6 Limitations One of the empirical shortcomings of LazySloth is that we observed our method assumed that long-video comprehension can be reliably localized to subtrees, as we observed poor backtracking and navigation to other branches. Our method also relied on the underlying caption quality generated across all resolutions, which we quantify in Appendix C. Additionally, our tests remain limited to two base models, without uncertainty quantification with multiple runs. References Bagad, Tapaswi, and Snoek (2023) Bagad, P.; Tapaswi, M.; and Snoek, C. G. 2023. Test of Time: Instilling Video-Language Models with a Sense of Time . In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2503–2516. Los Alamitos, CA, USA: IEEE Computer Society. Doorenbos, Spurio, and Gall (2025) Doorenbos, L.; Spurio, F.; and Gall, J. 2025. Video Panels for Long Video Understanding. El Haddad, Bohy, and Dutoit (2025) El Haddad, K.; Bohy, H.; and Dutoit, T. 2025. The Interaction Behavior Dataset: A Dataset of Smiles and Laughs in Dyadic Interaction. In 2025 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR), 399–403. Fu et al. (2025) Fu, C.; Dai, Y.; Luo, Y.; Li, L.; Ren, S.; Zhang, R.; Wang, Z.; Zhou, C.; Shen, Y.; Zhang, M.; Chen, P.; Li, Y.; Lin, S.; Zhao, S.; Li, K.; Xu, T.; Zheng, X.; Chen, E.; Shan, C.; He, R.; and Sun, X. 2025. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 24108–24118. Guo et al. (2025) Guo, Z.; Xia, L.; Yu, Y.; Ao, T.; and Huang, C. 2025. LightRAG: Simple and Fast Retrieval-Augmented Generation. In Christodoulopoulos, C.; Chakraborty, T.; Rose, C.; and Peng, V., eds., Findings of the Association for Computational Linguistics: EMNLP 2025, 10746–10761. Suzhou, China: Association for Computational Linguistics. ISBN 979-8-89176-335-7. Gutierrez et al. (2024) Gutierrez, B. J.; Shu, Y.; Gu, Y.; Yasunaga, M.; and Su, Y. 2024. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. Hessel et al. (2021) Hessel, J.; Holtzman, A.; Forbes, M.; Le Bras, R.; and Choi, Y. 2021. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 7514–7528. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics. Hong et al. (2025) Hong, W.; Cheng, Y.; Yang, Z.; Wang, W.; Wang, L.; Gu, X.; Huang, S.; Dong, Y.; and Tang, J. 2025. MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8450–8460. Jiang et al. (2020) Jiang, X.; Zong, Y.; Zheng, W.; Tang, C.; Xia, W.; Lu, C.; and Liu, J. 2020. DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild. In Proceedings of the 28th ACM International Conference on Multimedia, 2881–2889. Li et al. (2025a) Li, X.; Wang, Y.; Yu, J.; Zeng, X.; Zhu, Y.; Huang, H.; Gao, J.; Li, K.; He, Y.; Wang, C.; Qiao, Y.; Wang, Y.; and Wang, L. 2025a. VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling. arXiv:2501.00574. Li et al. (2025b) Li, X.; Yan, Z.; Meng, D.; Dong, L.; Zeng, X.; He, Y.; Wang, Y.; Qiao, Y.; Wang, Y.; and Wang, L. 2025b. VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning. arXiv:2504.06958. Liu et al. (2024) Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12: 157–173. Lumer et al. (2026) Lumer, E.; Cardenas, A.; Melich, M.; Mason, M.; Dieter, S.; Subbiah, V. K.; Basavaraju, P. H.; and Hernandez, R. 2026. Comparison of Text-Based and Image-Based Retrieval in Modern Multimodal Retrieval Augmented Generation Large Language Model Systems. In Proceedings of the 18th International Conference on Agents and Artificial Intelligence - Volume 5: ICAART, 4619–4626. INSTICC, SciTePress. ISBN 978-989-758-796-2. Luo et al. (2025) Luo, Y.; Zheng, X.; Li, G.; Yin, S.; Lin, H.; Fu, C.; Huang, J.; Ji, J.; Chao, F.; Luo, J.; and Ji, R. 2025. Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. Mangalam, Akshkulakov, and Malik (2023) Mangalam, K.; Akshkulakov, R.; and Malik, J. 2023. EgoSchema: a diagnostic benchmark for very long-form video language understanding. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23. Red Hook, NY, USA: Curran Associates Inc. Most et al. (2025) Most, A.; Winjum, J.; Bhattarai, M.; Jones, S.; Ranasinghe, N. R.; Biswas, A.; and O’Malley, D. 2025. Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval. In Proceedings of the 2025 ACM Symposium on Document Engineering, DocEng ’25. New York, NY, USA: Association for Computing Machinery. ISBN 9798400713514. Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 8748–8763. PMLR. Rege et al. (2026) Rege, A.; Sadhu, A.; Li, Y.; Li, K.; Vinayak, R. K.; Chai, Y.; Lee, Y. J.; and Kim, H. J. 2026. Agentic Very Long Video Understanding. In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds., Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 46575–46602. San Diego, California, United States: Association for Computational Linguistics. ISBN 979-8-89176-390-6. Reverdy et al. (2022) Reverdy, J.; O’Connor Russell, S.; Duquenne, L.; Garaialde, D.; Cowan, B. R.; and Harte, N. 2022. RoomReader: A Multimodal Corpus of Online Multiparty Conversational Interactions. In Calzolari, N.; Béchet, F.; Blache, P.; Choukri, K.; Cieri, C.; Declerck, T.; Goggi, S.; Isahara, H.; Maegaard, B.; Mariani, J.; Mazo, H.; Odijk, J.; and Piperidis, S., eds., Proceedings of the Thirteenth Language Resources and Evaluation Conference, 2517–2527. Marseille, France: European Language Resources Association. Shen et al. (2025) Shen, X.; Xiong, Y.; Zhao, C.; Wu, L.; Chen, J.; Zhu, C.; Liu, Z.; Xiao, F.; Varadarajan, B.; Bordes, F.; Liu, Z.; Xu, H.; Kim, H. J.; Soran, B.; Krishnamoorthi, R.; Elhoseiny, M.; and Chandra, V. 2025. LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding. In Forty-second International Conference on Machine Learning. Shu et al. (2025) Shu, Y.; Liu, Z.; Zhang, P.; Qin, M.; Zhou, J.; Liang, Z.; Huang, T.; and Zhao, B. 2025. Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 26160–26169. Siniukov et al. (2024) Siniukov, M.; Yin, Y.; Fast, E.; Qi, Y.; Monga, A.; Kim, A.; and Soleymani, M. 2024. SEMPI: A Database for Understanding Social Engagement in Video-Mediated Multiparty Interaction. In Proceedings of the 26th International Conference on Multimodal Interaction, ICMI ’24, 546–555. New York, NY, USA: Association for Computing Machinery. ISBN 9798400704628. Tang et al. (2026) Tang, C.; Li, Y.; Yang, Y.; Zhuang, J.; Sun, G.; Li, W.; MA, Z.; and Zhang, C. 2026. video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models. Vuillecard et al. (2024) Vuillecard, P.; Farkhondeh, A.; Villamizar, M.; and Odobez, J.-M. 2024. CCDb-HG: Novel Annotations and Gaze-Aware Representations for Head Gesture Recognition. In 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), 1–9. Wang et al. (2025a) Wang, W.; He, Z.; Hong, W.; Cheng, Y.; Zhang, X.; Qi, J.; Ding, M.; Gu, X.; Huang, S.; Xu, B.; Dong, Y.; and Tang, J. 2025a. LVBench: An Extreme Long Video Understanding Benchmark. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), 22958–22967. Wang et al. (2025b) Wang, X.; Zhang, Y.; Zohar, O.; and Yeung-Levy, S. 2025b. VideoAgent: Long-Form Video Understanding with Large Language Model as Agent. In Leonardis, A.; Ricci, E.; Roth, S.; Russakovsky, O.; Sattler, T.; and Varol, G., eds., Computer Vision – ECCV 2024, 58–76. Cham: Springer Nature Switzerland. ISBN 978-3-031-72989-8. Wang et al. (2025c) Wang, Z.; Yoon, J.; Yu, S.; Islam, M. M.; Bertasius, G.; and Bansal, M. 2025c. Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoning. In Christodoulopoulos, C.; Chakraborty, T.; Rose, C.; and Peng, V., eds., Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 28126–28140. Suzhou, China: Association for Computational Linguistics. ISBN 979-8-89176-332-6. Wang et al. (2026) Wang, Z.; Zhang, Y.; Yu, S.; Zhang, C.; Zhao, Z.; Yoon, J.; Lee, H.; Bertasius, G.; and Bansal, M. 2026. EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding. arXiv:2605.09874. Wu et al. (2024) Wu, H.; Li, D.; Chen, B.; and Li, J. 2024. LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding. In Globerson, A.; Mackey, L.; Belgrave, D.; Fan, A.; Paquet, U.; Tomczak, J.; and Zhang, C., eds., Advances in Neural Information Processing Systems, volume 37, 28828–28857. Curran Associates, Inc. Yeo et al. (2026) Yeo, W.; Kim, K.; Yoon, J.; and Hwang, S. J. 2026. WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 25599–25609. Yin et al. (2024) Yin, S.; Fu, C.; Zhao, S.; Li, K.; Sun, X.; Xu, T.; and Chen, E. 2024. A survey on multimodal large language models. National Science Review, 11(12): nwae403. Yu et al. (2025) Yu, S.; Tang, C.; Xu, B.; Cui, J.; Ran, J.; Yan, Y.; Liu, Z.; Wang, S.; Han, X.; Liu, Z.; and Sun, M. 2025. VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents. In The Thirteenth International Conference on Learning Representations. Zhang et al. (2025) Zhang, B.; Li, K.; Cheng, Z.; Hu, Z.; Yuan, Y.; Chen, G.; Leng, S.; Jiang, Y.; Zhang, H.; Li, X.; Jin, P.; Zhang, W.; Wang, F.; Bing, L.; and Zhao, D. 2025. VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding. arXiv:2501.13106. Zhou et al. (2025) Zhou, J.; Shu, Y.; Zhao, B.; Wu, B.; Liang, Z.; Xiao, S.; Qin, M.; Yang, X.; Xiong, Y.; Zhang, B.; Huang, T.; and Liu, Z. 2025. MLVU: Benchmarking Multi-task Long Video Understanding. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 13691–13701. Zuo et al. (2025) Zuo, J.; Deng, Y.; Kong, L.; Yang, J.; Jin, R.; Zhang, Y.; Sang, N.; Pan, L.; Liu, Z.; and Gao, C. 2025. VideoLucy: Deep Memory Backtracking for Long Video Understanding. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. Appendices Appendix A Symbol Glossary Table 11 presents a glossary of all formalization symbols used in Section 3 and Algorithm 1. Appendix B FPS Sweep Results Table 5 showcases uniform FPS sampling results on a few short video datasets: DFEW (Jiang et al. 2020), CCDb-IB (El Haddad, Bohy, and Dutoit 2025), CCDb-HG (Vuillecard et al. 2024), CCDb+, SEMPI (Siniukov et al. 2024), and RoomReader (Reverdy et al. 2022). We specifically selected them as they are a few seconds long and therefore fit entirely into VLM context windows at high FPS sampling rates. The sweep showed that adding frames did not add usable information. Neither model improved monotonically as sampling rose from 1 FPS to every frame. Qwen 2.5 VL 7B1313 13 https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct on RoomReader fell from 47.0% at 1 FPS to 11.5% when given the full frame set, and on SEMPI from 54.0% to 32.7%. Gemma 4 E4B1414 14 https://huggingface.co/google/gemma-4-E4B, on the other hand, stayed essentially flat across all four rates. Since these clips are short enough to fit entirely in context, the ceiling is not the context window but the model’s ability to use what is already in it. This experiment further motivated a retrieval-based design. Tier Dataset “All” (31 FPS) 24 FPS 12 FPS 1 FPS Qwen 2.5 VL 7B (Unsloth bf16 GGUF) DFEW C: 26.7± 22.726.7 \,±\,22.7 R: 12.3± 6.712.3 \,±\,6.7 C: 27.4± 22.427.4 \,±\,22.4 R: 14.7± 7.414.7 \,±\,7.4 C: 32.2± 24.032.2 \,±\,24.0 R: 26.2± 13.726.2 \,±\,13.7 C: 32.5± 22.732.5 \,±\,22.7 R: 30.9± 21.830.9 \,±\,21.8 14.05± 2.5114.05 \,±\,2.51 CCDb-IB C: 23.5± 10.123.5 \,±\,10.1 R: 18.0± 8.418.0 \,±\,8.4 C: 28.2± 8.428.2 \,±\,8.4 R: 21.7± 7.921.7 \,±\,7.9 C: 22.6± 8.222.6 \,±\,8.2 R: 22.2± 7.022.2 \,±\,7.0 C: 27.2± 8.027.2 \,±\,8.0 R: 14.8± 6.714.8 \,±\,6.7 12.63± 2.3912.63 \,±\,2.39 CCDb-HG C: 16.7± 22.516.7 \,±\,22.5 R: 7.4± 10.07.4 \,±\,10.0 C: 16.2± 21.516.2 \,±\,21.5 R: 10.1± 13.510.1 \,±\,13.5 C: 16.8± 20.416.8 \,±\,20.4 R: 13.9± 12.913.9 \,±\,12.9 C: 20.1± 23.820.1 \,±\,23.8 R: 17.8± 13.717.8 \,±\,13.7 16.73± 2.7416.73 \,±\,2.74 CCDb+ C: 54.4± 20.754.4 \,±\,20.7 R: 20.1± 12.120.1 \,±\,12.1 C: 59.8± 25.559.8 \,±\,25.5 R: 22.7± 12.122.7 \,±\,12.1 C: 60.7± 25.160.7 \,±\,25.1 R: 30.2± 12.230.2 \,±\,12.2 C: 56.7± 19.956.7 \,±\,19.9 R: 57.5± 10.557.5 \,±\,10.5 49.87± 3.3549.87 \,±\,3.35 SEMPI C: 32.7± 15.432.7 \,±\,15.4 R: 1.2± 2.61.2 \,±\,2.6 C: 44.8± 18.344.8 \,±\,18.3 R: 3.4± 3.93.4 \,±\,3.9 C: 43.2± 17.743.2 \,±\,17.7 R: 35.4± 11.335.4 \,±\,11.3 C: 54.0± 19.554.0 \,±\,19.5 R: 56.3± 15.456.3 \,±\,15.4 50.07± 3.6250.07 \,±\,3.62 RoomReader C: 11.5± 7.011.5 \,±\,7.0 R: 4.0± 5.04.0 \,±\,5.0 C: 32.8± 17.132.8 \,±\,17.1 R: 21.9± 10.921.9 \,±\,10.9 C: 34.1± 21.234.1 \,±\,21.2 R: 29.3± 18.029.3 \,±\,18.0 C: 47.0± 38.547.0 \,±\,38.5 R: 56.0± 10.056.0 \,±\,10.0 49.86± 3.4549.86 \,±\,3.45 Gemma 4 E4B (Unsloth bf16 GGUF) DFEW C: 30.8± 26.730.8 \,±\,26.7 R: 36.2± 23.136.2 \,±\,23.1 C: 31.3± 25.931.3 \,±\,25.9 R: 35.6± 22.935.6 \,±\,22.9 C: 30.9± 25.330.9 \,±\,25.3 R: 38.3± 25.438.3 \,±\,25.4 C: 32.8± 25.032.8 \,±\,25.0 R: 38.1± 20.338.1 \,±\,20.3 14.05± 2.5114.05 \,±\,2.51 CCDb-IB C: 25.3± 11.425.3 \,±\,11.4 R: 26.4± 9.626.4 \,±\,9.6 C: 26.4± 12.326.4 \,±\,12.3 R: 26.6± 9.226.6 \,±\,9.2 C: 26.1± 10.526.1 \,±\,10.5 R: 27.5± 9.927.5 \,±\,9.9 C: 24.2± 9.624.2 \,±\,9.6 R: 26.8± 8.526.8 \,±\,8.5 12.63± 2.3912.63 \,±\,2.39 CCDb-HG C: 16.3± 23.516.3 \,±\,23.5 R: 21.1± 22.721.1 \,±\,22.7 C: 18.6± 24.218.6 \,±\,24.2 R: 22.8± 24.222.8 \,±\,24.2 C: 18.0± 23.518.0 \,±\,23.5 R: 20.9± 23.020.9 \,±\,23.0 C: 18.4± 23.718.4 \,±\,23.7 R: 16.2± 14.216.2 \,±\,14.2 16.73± 2.7416.73 \,±\,2.74 CCDb+ C: 63.0± 25.763.0 \,±\,25.7 R: 8.1± 5.68.1 \,±\,5.6 C: 65.8± 26.465.8 \,±\,26.4 R: 9.7± 7.69.7 \,±\,7.6 C: 63.8± 23.963.8 \,±\,23.9 R: 10.1± 7.110.1 \,±\,7.1 C: 63.5± 27.063.5 \,±\,27.0 R: 7.4± 5.57.4 \,±\,5.5 49.87± 3.3549.87 \,±\,3.35 SEMPI C: 65.0± 12.065.0 \,±\,12.0 R: 61.0± 12.061.0 \,±\,12.0 C: 68.1± 9.868.1 \,±\,9.8 R: 67.5± 12.967.5 \,±\,12.9 C: 68.5± 13.268.5 \,±\,13.2 R: 71.3± 12.071.3 \,±\,12.0 C: 61.2± 17.561.2 \,±\,17.5 R: 64.5± 12.564.5 \,±\,12.5 50.07± 3.6250.07 \,±\,3.62 RoomReader C: 36.9± 23.836.9 \,±\,23.8 R: 39.6± 27.639.6 \,±\,27.6 C: 49.0± 32.649.0 \,±\,32.6 R: 48.8± 32.448.8 \,±\,32.4 C: 47.1± 37.647.1 \,±\,37.6 R: 47.5± 33.147.5 \,±\,33.1 C: 48.9± 43.548.9 \,±\,43.5 R: 41.6± 31.341.6 \,±\,31.3 49.86± 3.4549.86 \,±\,3.45 Table 5: Per quantization × sampling-strategy estimates of accuracy (%) for Qwen 2.5 VL 7B and Gemma 4 E4B. Each video cell reports Constrained (C) and Reasoning (R) settings as mean ± std; image-based datasets are frame-independent and span all sampling columns. Appendix C Captioning Quality Tests Because the captioner C is held frozen across every method, we tested their sensitivity before committing to Nvidia’s Nemotron 3 Nano Omni. Table 6 shows that across six viable candidates, CLIPScore on frames randomly sampled from the six tested long-video datasets spanned only 0.2331 to 0.2447 CLIPScore, a range of roughly 5%. Importantly, only Gemma 4 E4B scored particularly worse at 0.2153. Since modern VLMs are capable of captioning images, this result is not quite surprising. This brings us to the proposition of efficiency. To make LazySloth fast, we need C to balance captioning quality with latency. Across the six candidates, we note that the per-caption latency spanned 0.59 to 2.70 seconds, demonstrating a difference of 4.6× from the fastest to the slowest model. Since C is invoked once per tree node and therefore sits in the inner loop of every method we compare, latency is the dimension that actually separates the candidates. Nemotron 3 Nano Omni was the fastest captioner that was not a quality outlier. Therefore, we chose this model for LazySloth and across all methods so that no baseline is advantaged or penalized. This allowed us to cleanly compare the merit of each retrieval method. Moreover, looking at Table 7, we want to discuss Nemotron’s high CLIPScore despite the fact that it wrote the second-shortest captions. As seen in Table 6, CLIPScore is directly correlated to average caption length (a known property of the metric). This makes Nemotron’s per-word caption quality competitive. Second, the per-dataset spread within any model is comparable to the spread between models, which means the differences in Table 7 do not account for a strict ranking. Model CLIP Score ↑ SigLIP Score ↑ Avg. length (words) Latency (s) ↓ google/gemma-4-12b 0.2447 0.1367 32.6 1.27 qwen/qwen3.6-27b 0.2435 0.1403 31.4 2.29 google/gemma-4-31b 0.2417 0.1348 29.1 2.70 qwen/qwen3.5-9b 0.2417 0.1402 31.5 0.92 qwen2.5-vl-7b-instruct 0.2341 0.1292 27.7 0.98 nvidia/nemotron-3-nano-omni 0.2331 0.1340 25.1 0.59 google/gemma-4-e4b-it 0.2153 0.1175 21.8 0.77 Table 6: Reference-free caption quality of the candidate frozen captioners C, pooled over all evaluated datasets. Captions were generated from the same sampled frames with the identical pipeline prompt, then scored by CLIPScore (Hessel et al. 2021) and a SigLIP2 cross-check. Latency is wall-clock seconds per caption. The shaded row is the captioner used in all experiments reported in this paper. Model Ego Schema LongVideo Bench MMBench Video Movie Chat Video MME google/gemma-4-31b 0.236 0.239 0.246 0.248 0.238 qwen/qwen3.6-27b 0.236 0.238 0.245 0.262 0.237 google/gemma-4-12b 0.227 0.239 0.248 0.264 0.245 qwen/qwen3.5-9b 0.236 0.229 0.244 0.259 0.240 qwen2.5-vl-7b-instruct 0.229 0.230 0.230 0.256 0.226 nvidia/nemotron-3-nano-omni 0.234 0.220 0.230 0.252 0.230 google/gemma-4-e4b-it 0.181 0.219 0.220 0.229 0.229 Table 7: Per-dataset CLIPScore for each candidate captioner. Rows are ordered by the pooled CLIPScore of Table 6. Appendix D Additional Efficiency Analysis of LazySloth For a fine-grained understanding of the efficiency analysis in Section 4.1, we broke down median inference time under each method and direct inference in Table 8 for the benchmark with the longest videos, LVBench. We found that inference time under each retrieval-based method grew linearly with video length, but the informative quantity is the slope of growth. While LazySloth paid 2.43 additional seconds per extra minute of video (s/min), VideoLucy paid 5.53s, and WorldMM paid 19.86s/min. We note that these differences were significant, as WorldMM paid 17.43 s/min more than LazySloth (t=13.2t=13.2, p<10−39p<10^-39) and VideoLucy 3.10 s/min more (t=4.9t=4.9, p<10−5p<10^-5). LazySloth therefore addressed the scaling problem that these methods struggled with. This difference, however, stemmed from both the number of caption calls and how they were structured. As shown in Table 9, LazySloth’s caption count was effectively compared to WorldMM and the brute-forced baseline, growing only from 35 on 40-minute-long videos to 45 under 140-minute-long videos due to the O(klogkN)O(k _kN) bound of Equation (1). WorldMM grew from 97 to 330, and the brute-forced eager trees from 3,600 to 12,603, a factor of 280 at the longest videos. VideoLucy, however, is the interesting case as it issued 27 to 54 calls, statistically indistinguishable from ours, yet still ran roughly three× slower. This is because VideoLucy re-captions dense fixed-length overlapping windows, so each call carries more frames and more visual tokens than a LazySloth bunch caption. Figure 4 shows that captioning consumed 67% of its total time. Our bounded search cost precisely addressed this issue. Length LazySloth VideoLucy WorldMM Brute-forced Eager DI 40 min 149 s 509 s (3.4×) 735 s (4.9×) 7.0 h (169×) 57 s 60 min 176 s 553 s (3.1×) 1109 s (6.3×) 10.5 h (214×) 57 s 90 min 251 s 727 s (2.9×) 1560 s (6.2×) 15.8 h (226×) 57 s 120 min 287 s 1000 s (3.5×) 2393 s (8.3×) 21.0 h (263×) 57 s 140 min 307 s 936 s (3.0×) 2371 s (7.7×) 24.5 h (287×) 57 s Table 8: Median inference time comparison across different video lengths on LVBench. Speedup is shown relative to LazySloth. Length LazySloth VideoLucy WorldMM Brute-forced Eager 40 min 35 27 (0.8×) 97 (2.8×) 3,600 (103×) 60 min 35 35 (1.0×) 149 (4.3×) 5,401 (156×) 90 min 38 46 (1.2×) 220 (5.8×) 8,102 (215×) 120 min 34 54 (1.6×) 291 (8.6×) 10,801 (320×) 140 min 45 42 (0.9×) 330 (7.3×) 12,603 (280×) Table 9: Median caption calls across different video lengths on LVBench. Speedup is shown relative to LazySloth with Qwen3.6 27B. Figure 8: WorldMM vs VideoLucy vs LazySloth median captions generated per-video on LVBench. While VideoLucy doesn’t caption significantly more than LazySloth, the difference in wall clock time arises from how the two methods utilize caching and optimize multi-frame inputs to VLMs. Appendix E Additional Benchmark-wise Behavioral Analysis To better understand the impact of tree depth searched by the searcher S on final accuracy, we plotted the aggregate depth trend of Figure 5 by benchmark and backbone in Figure 10. Direct inference has no search and therefore no depth of its own; it is scored on exactly the questions that fall in each of LazySloth’s strata. The decline we plotted in Figure 10 against Direct Inference indicated whether the deeper-searched strata are indeed the harder questions. This helps us eliminate an important confound that could misguide this discussion. The takeaway, therefore, is the performance difference between direct inference and LazySloth, which we plotted with 95% confidence interval bars. We note that for Qwen3.6 27B, LazySloth consistently improved performance from direct inference, with the differences reaching statistical significance at higher tree depths across the four benchmarks, most sharply on EgoSchema and LongVideoBench. On Gemma 4 31B, however, the two curves were essentially coincident on EgoSchema and LVBench, and LazySloth directionally improved performance on LongVideoBench, although not statistically significant. Since both the blue and orange lines in the plot refer to identical questions within each stratum, the difference cannot be attributed to question difficulty, and instead reflects how much a given base model gains from our method. Gemma 4 31B consistently struggled to gain any performance across the board, supporting our claim that gains are model-dependent, and Gemma 4 31B lacks any headroom to improve performance with retrieval-based or agentic frameworks. Further, Figure 9 examines which branch the search descends into first against randomly picking any branch from the generated children. We note that models preferred the first frontier split consistently more than the other three branches, which have a roughly equal chance of being picked. The first descent landed in the opening quarter of the video on 49% of Video-MME and LongVideoBench questions, 54% of LVBench, and 64% of EgoSchema, which was roughly twice the random rate. Finally, Table 10 matched LazySloth against direct inference on the same questions and split the result by the question type. We note that gains in performance were not uniform. Question types, mostly those requiring a moment to be located and then reasoned about, produced the largest improvements (LongVideoBench’s T2O gained 44.7%, SOS 38.3%, and TOS 37.0%, and Video-MME’s Information Synopsis and Temporal Reasoning gained 25.5% and 23.2% respectively). This is an architectural improvement LazySloth brings to the table. Types answerable from any representative frame, as expected, gained almost nothing. Our method helps the most in cases where finding the right moment is the task, which explains the performance differences reported in Table 2. Figure 9: Share of questions for which LazySloth (aggregated over Qwen3.6 27B and Gemma 4 31B) chose the first, second, third, or fourth quarter of the video at the first level of the search tree. LazySloth’s first descent was strongly biased toward the start of the video (roughly twice the rate of random branch selection). The dashed reference shows an empirically measured random baseline. Figure 10: Tree depth searched by the two tested base VLMs, Gemma 4 31B and Qwen3.6 27B, across the four datasets with LazySloth and Direct Inference. Since Direct Inference does not create trees to search in, it is scored on exactly the questions that fall in each of LazySloth’s strata. Benchmark Question type N LazySloth Direct inf. Δ McNemar χ2χ^2 LongVideoBench S 97 38.1 12.4 +25.8 18.6∗ LongVideoBench E3E 94 52.1 28.7 +23.4 13.8∗ LongVideoBench S2E 93 57.0 43.0 +14.0 4.4∗ LongVideoBench S2A 88 64.8 62.5 +2.3 0.0 LongVideoBench O2E 87 50.6 42.5 +8.0 1.9 LongVideoBench TAA 82 46.3 26.8 +19.5 7.0∗ LongVideoBench SOS 81 61.7 23.5 +38.3 23.1∗ LongVideoBench T2A 79 54.4 51.9 +2.5 0.0 LongVideoBench T2O 76 65.8 21.1 +44.7 28.7∗ LongVideoBench T3O 74 50.0 14.9 +35.1 22.3∗ LongVideoBench TOS 73 47.9 11.0 +37.0 20.5∗ LongVideoBench T3E 73 43.8 24.7 +19.2 7.0∗ LongVideoBench SAA 72 58.3 27.8 +30.6 17.0∗ LongVideoBench S2O 72 61.1 40.3 +20.8 7.3∗ LongVideoBench O3O 66 54.5 28.8 +25.8 8.8∗ LongVideoBench E2O 65 47.7 43.1 +4.6 0.2 LongVideoBench T2E 65 52.3 27.7 +24.6 9.4∗ Video-MME Object Reasoning 453 57.8 36.0 +21.9 55.5∗ Video-MME Object Recognition 354 65.3 60.5 +4.8 2.5 Video-MME Information Synopsis 322 72.4 46.9 +25.5 54.7∗ Video-MME Action Recognition 313 55.0 52.7 +2.2 0.4 Video-MME Action Reasoning 285 54.7 33.3 +21.4 33.0∗ Video-MME Counting Problem 267 37.1 32.2 +4.9 1.6 Video-MME Attribute Perception 222 67.6 63.5 +4.1 1.0 Video-MME Temporal Reasoning 177 42.4 19.2 +23.2 20.8∗ Video-MME OCR Problems 139 56.8 61.2 -4.3 0.6 Video-MME Spatial Reasoning 56 76.8 58.9 +17.9 4.0∗ Video-MME Temporal Perception 55 70.9 61.8 +9.1 0.8 Video-MME Spatial Perception 54 70.4 61.1 +9.3 1.1 Table 10: Search gain by question type using Qwen 3.6 27B as the base VLM. LVBench and EgoSchema are omitted as they do not contain segregated question types. ∗ marks χ2>3.84χ^2>3.84 (p<0.05p<0.05). Symbol Meaning V Input video clip. Q Query/task (MCQ question + options, open-ended question, or classification prompt). L Label set — MCQ letters / class labels; ∅ for open-ended QA (free-text answer). C Frozen captioner VLM; only ever produces captions (kept fixed so an experiment varies only the search VLM). S Search VLM + reasoner VLM — one model in two roles: text-only navigation, then multimodal final answer. ϕφ Frame sampling rate, measured in FPS, used to decode the clip. F=⟨f1,…,fN⟩F= f_1,…,f_N Ordered extracted frames; each fi=(idxi,ti,pathi)f_i=(idx_i,t_i,path_i) = native index, timestamp, JPEG path. N Number of extracted frames, =|F|= F . k Branch factor — children per Drill, and the root’s fan-out. ℓ Leaf size — frame-range threshold; once a node spans ≤ℓ≤ frames, Drill stops and Expand (per-frame captions) is allowed. mbm_b Max representative frames shown to C when captioning one node/bunch (evenly subsampled); “b” = bunch. mrm_r Max key frames shown to the reasoner S (evenly subsampled, time order); “r” = reasoner. R Max VLM search rounds (round budget). smaxs_ Consecutive no-progress rounds before an answer is forced (=3=3). ℱF (frontier) The currently-shown sibling nodes — the candidates for Drill/Expand. prev / ℱprevF_prev The previous frontier, retained so S can Drill a sibling to backtrack one level. ℰE (evidence) The caption corpus surfaced along the path (root → branch → per-frame captions). [a,b)[a,b) / currPos The position range into F the traversal narrowed to; the reasoner’s key-frame window (starts [0,N)[0,N)). node.pos =[ac,bc)=[a_c,b_c) A node’s half-open position range into F; its span bc−acb_c-a_c is compared against ℓ . level / id Node depth (root =0=0) / node id (its start frame index — stable, unique within a level). cap[⋅]cap[\,·\,] On-disk caption cache, keyed by the frozen captioner C (semantic tree for nodes, episodic for frames); reused across queries. yag,yre,y^y_ag,y_re, y The VLM’s answer, the reasoner’s answer, and the final prediction (y^=yre y=y_re, or yagy_ag if the reasoner returns Insufficient). Table 11: Notation reference for formalization of the LazySloth framework in Section 3 and Algorithm 1.