Paper deep dive
Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos
Penglei Sun, Yehua Huang, Zhuoli Tao, Xiang Li, Runwei Guan, Yaoxian Song, Kaiyong Zhao, Henghui Ding, Bo Han, Yang Yang, Xiaowen Chu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/23/2026, 2:44:23 AM
Summary
This paper introduces SkyAnchor, a memory-augmented Multimodal Large Language Model (MLLM) designed for small object understanding in streaming aerial videos. It addresses challenges of tiny targets and continuous context by proposing a Semantics-Aware Token Router and a Hierarchical Memory Bank. The authors also present DroneEyes, a new pixel-level, open-vocabulary referring-segmentation dataset for aerial targets, demonstrating state-of-the-art performance on both the new dataset and the SkyFind benchmark.
Entities (10)
Relation Signals (9)
SkyAnchor → evaluatedon → DroneEyes
confidence 95% · Experiments on DroneEyes show that SkyAnchor outperforms existing MLLMs
SkyAnchor → uses → Hierarchical Memory Bank
confidence 95% · SkyAnchor... design a Hierarchical Memory Bank that decouples temporal information
SkyAnchor → uses → Semantics-Aware Token Router
confidence 95% · SkyAnchor... introduce a Semantics-Aware Token Router that preserves target-relevant information
Hierarchical Memory Bank → contains → Tracking Memory
confidence 92% · The Semantic Memory layer... while the Tracking Memory layer caches recent spatial states
Hierarchical Memory Bank → contains → Semantic memory
confidence 92% · The Semantic Memory layer stores object-level appearance features... while the Tracking Memory layer caches recent spatial states
DroneEyes → createdby → SkyAnchor Authors
confidence 90% · we present DroneEyes... From the data perspective, we construct DroneEyes
SkyAnchor → evaluatedon → SkyFind
confidence 90% · On the unseen SkyFind benchmark, SkyAnchor improves over the previous domain-specific SOTA
SkyAnchor → builtontopof → Qwen2.5-VL
confidence 88% · We build upon Qwen2.5-VL.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Language-guided aerial perception aims to understand user-specified tiny targets in complex unmanned aerial vehicle (UAV) scenes. In real UAV deployment, the UAV must respond while it flies, so such perception runs in an online streaming manner, where frames arrive sequentially and the model responds to each one without access to future frames. However, applying current Multimodal Large Language Models (MLLMs) to this setting raises two challenges. First, targets viewed from the air are often tiny, yet the visual compression in existing MLLMs treats all regions equally and discards their fine-grained details. Second, understanding a continuous stream requires past-frame context, yet retaining the entire history is infeasible on resource-constrained onboard hardware, whereas discarding it causes the target to drift or disappear. We address the tiny object and streaming challenges from both data and method perspectives. From the data perspective, we present \textbf{DroneEyes}, the \textbf{first} pixel-level and open-vocabulary referring-segmentation dataset for tiny aerial targets, comprising $2,140$ high-definition videos and $176,623$ pairs across Object Description and Referring Expression tasks, with dense per-frame masks. From the method perspective, we propose \textbf{SkyAnchor}, an MLLM with two designs to the above challenges: a Semantics-Aware Token Router that preserves small-target under a reduced visual-token budget, and a Hierarchical Memory Bank that keeps the target consistently understood on streams.
Tags
Links
- Source: https://arxiv.org/abs/2607.19857v1
- Canonical: https://arxiv.org/abs/2607.19857v1
Trouble viewing inline? Open PDF directly →
Full Text
71,958 characters extracted from source content.
Expand or collapse full text
Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos Penglei Sun1, Yehua Huang1, Zhuoli Tao2, Xiang Li1, Runwei Guan1, Yaoxian Song3, Kaiyong Zhao4, Henghui Ding5, Bo Han6 , Yang Yang1 , Xiaowen Chu1✉ ✉ Corresponding author.1The authors are with Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China, Emails: psun012,yhuang704,xli906@connect.hkust-gz.edu. cn, runwayrwguan,yyiot,xwchu@hkust-gz.edu.cn. 2The author is with University of Freiburg, Freiburg, German, Emails: zhuoli.tao@students.uni-freiburg.de. 3The author is with Hangzhou City University, Hangzhou, China, Emails: songyaoxian@westlake.edu.cn. 4The author is with XGRIDS, Shenzhen, China, Emails: kyzhao@xgrids.com. 5The author is with Fudan University, Shanghai, China, Emails: hhding@fudan.edu.cn. 6The author is with Department of Computer Science, Hong Kong Baptist University, Hong Kong, China, Emails: bhanml@comp.hkbu.edu.hk. Abstract Language-guided aerial perception aims to understand user-specified tiny targets in complex unmanned aerial vehicle (UAV) scenes. In real UAV deployment, the UAV must respond while it flies, so such perception runs in an online streaming manner, where frames arrive sequentially and the model responds to each one without access to future frames. However, applying current Multimodal Large Language Models (MLLMs) to this setting raises two challenges. First, targets viewed from the air are often tiny, yet the visual compression in existing MLLMs treats all regions equally and discards their fine-grained details. Second, understanding a continuous stream requires past-frame context, yet retaining the entire history is infeasible on resource-constrained onboard hardware, whereas discarding it causes the target to drift or disappear. We address the tiny object and streaming challenges from both data and method perspectives. From the data perspective, we present DroneEyes, the first pixel-level and open-vocabulary referring-segmentation dataset for tiny aerial targets, comprising 2,1402,140 high-definition videos and 176,623176,623 pairs across Object Description and Referring Expression tasks, with dense per-frame masks. From the method perspective, we propose SkyAnchor, an MLLM with two designs to the above challenges: a Semantics-Aware Token Router that preserves small-target under a reduced visual-token budget, and a Hierarchical Memory Bank that keeps the target consistently understood on streams. Experiments on DroneEyes show that SkyAnchor outperforms existing MLLMs on both object description (by 2×2× in GPT score) and referring expression (by 15.3%15.3\% in region-and-contour accuracy). On the unseen SkyFind benchmark, SkyAnchor improves over the previous domain-specific SOTA by 25.9%25.9\% in IoU@0.5 and 95.5%95.5\% in IoU@mean on the unseen maritime test split, demonstrating strong cross-domain generalization. Finally, we deploy SkyAnchor on a real UAV platform, where our optimization pipeline achieves a 3.05×3.05× speedup over vanilla PyTorch, demonstrating its practical value for UAV surveillance and operator assistance. Our project can be seen in https://sites.google.com/view/skyanchor/main-page. I Introduction Figure 1: Comparison of DroneEyes with existing daily-life and aerial offline tasks. (a) Daily-life datasets (e.g., MeViS) feature large objects irrelevant to aerial perception. (b) Existing aerial datasets (e.g., SkyFind) target small objects using bound box assume offline access to complete clips or static images. (c) DroneEyes try to understand small objects using segmentation in streaming aerial video. The model must respond at each step without access to future frames, while accurately understanding small objects. Unmanned aerial vehicles (UAVs) are increasingly deployed in surveillance, search-and-rescue, and infrastructure inspection [10, 11, 22, 30]. To assist operators in these applications, UAVs are expected to follow natural-language instructions and ground the referred targets in streaming aerial video, e.g., “Find the white truck.” [36]. Since the UAV must respond while it flies, this perception cannot be performed offline over a complete clip, but must run in an online streaming manner. This setting poses two intrinsic challenges. First, targets viewed from the air are often tiny, occupying only a small portion of high-resolution frames [28], which demands pixel-accurate perception to separate them from cluttered background. Second, frames are captured and consumed sequentially, so the model must respond to each frame the moment it arrives without accessing future frames [24], while the resource-constrained onboard device cannot cache the entire visual history of an ever-growing stream. Existing datasets do not provide the annotations required for this task, as illustrated in Fig. 1. First, daily-life datasets such as MeViS and LaMOT [6, 14] feature large objects that are far from the tiny targets seen from aerial viewpoints, and they draw their referring expressions from a closed set of common categories. Second, existing aerial datasets such as SkyFind and AerialMind [33, 3] capture the bird’s-eye view but offer coarse supervision, typically bounding boxes or frame-level captions, without pixel-accurate masks that delineate small targets from cluttered background. To fill this gap, we construct DroneEyes, the first pixel-level and open-vocabulary referring-segmentation dataset for tiny aerial targets. It comprises 2,1402,140 high-definition drone videos and 176,623176,623 pairs across Object Description and Referring Expression tasks. Unlike existing aerial datasets, DroneEyes preserves all videos at their native high resolution and annotates dense per-frame segmentation masks over an open vocabulary along continuous flight videos, so that the fine-grained shape of each tiny target is delineated from the background throughout the stream. The dataset is organized into two task levels from perception to end-to-end grounding. Level 1 (L1) provides a target region and requires the model to describe what is in that region, fostering fine-grained perception of small objects. Level 2 (L2) provides a natural-language referring expression and requires the model to segment the specified target, probing end-to-end referring segmentation. Beyond data, although recent multimodal large language models (MLLMs) have shown strong video understanding capabilities, the two challenges above also carry over to the model side when they are applied to streaming aerial video [18]. First, some targets occupy only small portions of high-resolution frames [28], leaving most visual tokens on background regions [31]. Full-token processing is accurate but costly on edge devices, while existing compression in general MLLMs is efficient but degrades tiny-object information. For instance, Qwen series [25] applies a fixed-ratio token merging that reduces all spatial regions equally. It will treat a small vehicle the same as a large building, so that small targets are merged into surrounding background tokens. Second, understanding a continuous video stream requires past-frame context, yet retaining the entire history is infeasible for streams, whereas discarding it causes the target to drift or disappear. For instance, SAM 3 [2] accumulates per-frame memory features whose storage scales linearly with sequence length. This strategy cannot sustain long-term identity. To address the above challenges of MLLMs, we propose SkyAnchor, an MLLM designed for small-object understanding in streaming aerial video. For the first challenge, we introduce a Semantics-Aware Token Router that preserves target-relevant information through semantics-aware weighted aggregation under a fixed token budget. For the second challenge, we design a Hierarchical Memory Bank that decouples temporal information into two complementary layers. The Semantic Memory layer stores object-level appearance features to preserve target identity over the streaming term, while the Tracking Memory layer caches recent spatial states to support short-term mask propagation. The two layers work jointly to keep a stable understanding of the target on streams. Experiments on DroneEyes validate the effectiveness of SkyAnchor against both general-purpose MLLMs and specialist segmentation models. On the L1 object description task, SkyAnchor achieves 2×2× performance over the strongest general-purpose baselines across metrics and demonstrating clear advantages in small-object understanding. On the L2 referring segmentation task, SkyAnchor-3B reaches best region-and-contour accuracy (&ℱJ\&F), surpassing the existing models by 15.3%15.3\% despite using far fewer parameters. To assess cross-domain generalization, we further evaluate SkyAnchor on the SkyFind benchmark without any fine-tuning. On the unseen maritime test split, SkyAnchor surpasses the previous domain-specific SOTA by 25.9%25.9\% in IoU@0.5 and 95.5%95.5\% in IoU@mean, demonstrating strong robustness under aerial domain shifts. Finally, we deploy SkyAnchor on a real UAV platform with NVIDIA Jetson AGX Orin 64GB. With our optimization pipeline, the system achieves a 3.05×3.05× speedup over vanilla PyTorch. It demonstrates cloud-free online monitoring on the target platform [8, 40]. In conclusion, this paper makes the following contributions: • From the data perspective, we construct DroneEyes, a large-scale aerial streaming video dataset comprising 2,1402,140 high-definition drone videos and 176,623176,623 pairs with dense per-frame segmentation masks. To our knowledge, DroneEyes is the first aerial streaming understanding dataset with the pixel-level annotation. • From the method perspective, we propose SkyAnchor, an MLLM that introduces a Semantics-Aware Token Router for preserving fine-grained details of tiny targets under the same budget and a Hierarchical Memory Bank for consistent understanding on streams, enabling efficient small-object understanding in streaming aerial video. • On DroneEyes, SkyAnchor with only 3B parameters achieves state-of-the-art results on both L1 object description and L2 referring segmentation, outperforming all MLLMs. On the SkyFind benchmark without any fine-tuning, SkyAnchor achieves improvements over the previous domain-specific SOTA on the unseen maritime test split, with relative gains of 25.9%25.9\% in IoU@0.5 and 95.5%95.5\% in IoU@mean, demonstrating strong cross-domain generalization. • From the deployment perspective, we develop a optimization pipeline and validate SkyAnchor on real-world surveillance videos captured by the UAV. Our optimization pipeline achieves 3.05×3.05× speedup over vanilla PyTorch, confirming practical applicability for onboard inference without cloud offloading. The rest of this paper is structured as follows. We begin with a review of related work in Section I. Section I introduces the DroneEyes dataset with the construction engine. Section IV formulates the problem and introduces the proposed framework SkyAnchor. Section V presents the experiments and analyses. Finally, Section VI concludes the paper and outlines future directions. TABLE I: Comparative Dataset for Video Segmentation. We compare the existing video segmentation dataset from venues. Seg. means segmentation task. Granularity indicates the spatial precision of the annotations. Pixel-level denotes dense per-frame segmentation masks and Bbox-level denotes bounding-box annotations. Expr. denotes instance level referring expressions for visual grounding. Bboxes are detection annotations without language. QA pairs serve video question answering. Avg O/I Ratio is the arithmetic mean of (areaobject/areaimage)×100%(area_object/area_image)× 100\% across all annotations in the dataset. Dataset Venue Aerial Perspective Granularity Modality Open Vocab. Tiny Target Avg O/I Ratio Tasks Videos Expression Resolution Seg. QA Ref-Youtube-VOS [29] ECCV 2020 Pixel-Level Video ✓ 18.31 % ✓ 3,978 15K Expr. ≤ 2.7K ShareGPT4Video [4] NIPS 2024 Frame-Level Video ✓ - ✓ 40K 40K Captions ≤ 720p MeViS [7] TPAMI 2025 Pixel-Level Video ✓ 1.4% ✓ 2,006 33K Expr. ≤ 720p JTD-UAV [34] CVPR 2025 ✓ BBox-Level Video ✓ 0.21% ✓ 1,328 3K QA pairs 1080p LaMOT [14] ICRA 2025 BBox-Level Video ✓ ✓ 0.63% ✓ 62 145 Expr. ≤ 720p AerialMind [3] AAAI 2026 ✓ BBox-Level Video ✓ ✓ 0.25% 93 24.6K Expr. ≤ 720p SkyFind [33] TPAMI 2026 ✓ BBox-Level Image ✓ ✓ 0.76% 35K Images 1M Expr. ≤ 4K DroneEyes (ours) - ✓ Pixel-Level Video ✓ ✓ 0.24% ✓ ✓ 2,140 176K pairs ≤ 4K I Related Work I-A Referring Video Object Segmentation Referring video object segmentation (RVOS) segments target objects in videos given natural language expressions. Existing methods follow two main lines. The first builds upon foundation segmentation models. VISA [38], Sa2VA [44] and UniPixel [19] uses a language model to reason on a single frame and hands the mask to SAM 2 [27] for propagation. The second pursues end-to-end architectures that jointly handle language grounding and mask prediction. VideoGLaMM [23] produces spatially grounded masks alongside language outputs. GLUS [15] and VRS-HQ [9] integrate reasoning within a single large language model. InstructSeg [35] and further unify diverse referring and segmentation tasks at the pixel level. However, these methods are designed for daily life level videos with relatively large objects and do not address the challenges specific to aerial scenarios, including small targets, dense distributions, large viewpoint changes, and the requirement for continuous streaming processing. I-B MLLMs for Aerial Understanding MLLMs have recently been adapted to remote sensing and aerial domains. GeoChat [12] SkySenseGPT [21] and EarthGPT [45] improve perception by enriching training data or broadening input modalities to SAR and infrared. SkyFind [45] introduces a million-scale UAV referring expression benchmark and reveals that current MLLMs struggle with small-target localization under heavy background clutter, while JTD-UAV[34] combines MLLMs with tracking for joint localization and intent description in thermal sequences. Both remain limited to box-level output or static grounding without pixel-level video segmentation. Our work addresses this gap by performing referring segmentation on aerial video streams with hierarchical memory and token compression tailored for small objects at high resolution. I DroneEyes Dataset Figure 2: The data label pipeline for DroneEyes. (1) Object Extraction. The candidate objects are detected with human verification, followed by mask propagation. (2) Semantic Enrichment. We use VLM to enrich attribute descriptions via context-aware visual prompting. (3) QA Generation. Two-level QA pairs are generated with LLM-based linguistic refinement. I-A Data Collection and Diversity We curate a dataset of 2,1402,140 high-definition drone videos sourced from the open-access platform Pexels111https://w.pexels.com/. Unlike ground-level datasets, aerial visual understanding depends on fine-grained spatial details. Therefore, we preserve all videos at their native resolutions, typically 1080p or 4K, to retain the visual cues of small objects. To ensure data diversity, the collected videos cover various real-world topologies including urban cityscapes, highway traffic, agricultural fields, and natural terrain under different illumination conditions, altitudes, and camera trajectories such as top-down, oblique, and orbital. This collection provides a diverse distribution of common aerial targets, including vehicles, pedestrians, and infrastructure, supporting the training of grounding-capable vision-language models. I-B Data Engine Annotating dense spatial grounding and complex understanding pairs for drone videos is labor-intensive due to the massive number of small targets. To scale up dataset construction, we develop an automated data engine that couples foundation segmentation models with large language models. The pipeline consists of three main stages: extracting consistent spatial-temporal object trajectories, and enriching these bounding boxes with semantic descriptions to construct difficulty-progressive tasks, as shown in Fig 2. I-B1 Object Extraction To establish consistent object trajectories, we implement a hybrid pipeline combining automated detection and human verification. We first utilize a YOLO-based detector to generate initial bounding box proposals on video keyframes. Human annotators then verify and filter these proposals to ensure high-precision initialization. Using these verified boxes as spatial prompts, we employ Segment Anything Model 3 as our foundation tracker. The tracker propagates segmentation masks bidirectionally through the video to maintain consistent object identities. To construct the final ground truth, we enforce a strict quality control mechanism by retaining only the tracking outputs with high confidence scores. A final manual sampling inspection guarantees the overall accuracy and reliability of the generated annotations. I-B2 Semantic Enrichment Raw spatial trajectories lack semantic information. We leverage a Vision-Language Model to generate descriptive attributes for each tracked object. To overcome the recognition challenge of small aerial targets, we apply context-aware visual prompting. Specifically, we extract object regions with expanded contextual boundaries and apply explicit visual markers. This spatial guidance allows the MLLM to accurately produce category labels and fine-grained visual details. The resulting data bridges the gap between geometric tracking and semantic understanding. I-B3 Data Generation Aerial visual reasoning requires understanding objects at multiple scales, from identifying the attributes of a single tiny vehicle to inferring the functional layout of an entire intersection. To systematically evaluate and train these capabilities, we build upon the semantically enriched annotations to design a two level generation strategy. This strategy requires complex visual understanding: • Level 1 (L1): Object Description. The model identifies an object specified by a bounding box region and describes its visual attributes. • Level 2 (L2): Referring Expression. The model locates a target object based on a natural language description and provides its exact spatial coordinates. To further improve linguistic diversity, we apply an LLM refinement stage. A primary challenge during this text polishing process is preventing the LLM from hallucinating or modifying the original spatial coordinates. We address this issue by decoupling the coordinates from the text using an abstract placeholder mechanism during the LLM prompt stage. The exact bounding box values are restored only in the final post-processing step. I-C Data Statistics As summarized in Table I, existing video benchmarks each cover only part of what real aerial perception requires, whether limited to a single task or relying on heavily compressed footage. Prior aerial video datasets in particular target geometric perception such as box-level detection, where annotations are closed-set category labels with bounding boxes. DroneEyes instead provides open-vocabulary, language-driven supervision. Rather than coarse bounding boxes, it offers pixel-level segmentation masks, and it is the first to pair referring segmentation with question answering on the same continuous aerial stream, with masks and answers aligned to individual frames. In Table I, Our DroneEyes dataset has an average object-to-image ratio of only 0.24%0.24\%, lower than LaMOT (0.63%0.63\%) and SkyFind (0.76%0.76\%), making it one of the more demanding benchmarks for small object drone visual understanding. Such pixel-level annotation is far more informative than bounding boxes for tiny targets, and together with native uncompressed streams ranging from 1080p to 4K, it is what preserves their structural details and enables accurate pixel-level mask generation in complex aerial sensing environments. DroneEyes dataset comprises 176,623 pairs generated from 2,140 valid drone videos (videos containing no detectable objects were excluded). Fig. 3 summarizes the more data statistics for DroneEyes. Figure 3: The statistic of DroneEyes. (a) Distribution of video frame counts across the dataset. (b) Distribution of object visibility ratios across annotation samples. (c) Hierarchical taxonomy of object categories in the dataset. (d) Distribution of object area ratios relative to the image area. (e) Split between Level 1 and Level 2 annotations in the dataset. (f) Word cloud of query text, with word size reflecting term frequency. IV SkyAnchor Method SkyAnchor answers the language query and segments the target across a drone video stream with a vision-language backbone and a mask decoder (Sec. IV-A). Building on this, we design two components: a hierarchical memory bank (Sec. IV-B) and a semantics-aware token router (Sec. IV-C). We further deploy it with an edge inference schedule (Sec. IV-D). We summarize the overall procedure in Alg. 1. Figure 4: Overview of SkyAnchor. (A) Model Overview: Q, optional visual prompt, and I1,…,It\I_1,…,I_t\ are encoded into qZ_q, rZ_r, vZ_v; together with mZ_m, the VLM produces H, from which the Language Head outputs A and the Seg Head outputs sege_seg to drive SAM2 for tM_t. (B) Hierarchical Memory Bank: the Semantic Memory Encoder registers persistent anchors <tsemS^sem_<t from anchor frames (red dashed) and exports mZ_m to the VLM via the Memory Head. The Tracking Memory Encoder caches recent frames (blue dashed) in a FIFO window <ttrkS^trk_<t. (C) Semantics-Aware Token Router: the Visual Router compresses dense ViT tokens into compact vZ_v by merging background tokens while retaining small-target details. IV-A Model Overview IV-A1 Task Statement We address the task of online referring video object segmentation in drone-captured aerial footage. Given a continuous video stream Itt=1∞\I_t\_t=1^∞ and a natural language query Q, the system must produce a segmentation mask tM_t and the answer A for the query for each frame upon arrival: t,=ℱ(It,,<t),M_t,A=F(I_t,Q,S_<t), (1) where <tS_<t denotes the accumulated memory state. This online, causal formulation is essential for real-world drone applications requiring low-latency streaming results. Our model realizes ℱF through a unified architecture comprising two principal components: (i) a vision-language model (VLM) backbone ℱvlmF_vlm that reasons over visual and textual inputs, and (i) a mask prediction decoder ℱdecF_dec that produces per-frame segmentation masks. To handle arbitrarily videos, the memory is explicitly decoupled into permanent semantic anchors semS^sem and sliding tracking states trkS^trk (detailed in Sec. IV-B). IV-A2 Vision-Language Backbone We build upon Qwen2.5-VL. At each anchor frame tat_a (triggered every N frames), the visual encoder extracts dense spatial features from the image. To prevent these from overwhelming the LLM’s attention computation, our Token Router (Sec. IV-C) dynamically compresses them into a compact visual token sequence vZ_v. The query Q is tokenized into text embeddings qZ_q. Additionally, a reference encoder provides spatial prompt embeddings rZ_r, and the semantic memory provides appearance tokens mZ_m from past anchors. The backbone jointly processes these inputs: =ℱvlm([v;q;r;m]).H=F_vlm\! ([Z_v;\,Z_q;\,Z_r;\,Z_m] ). (2) Note that ℱvlmF_vlm runs only at anchor frames to update the semantic understanding; between anchors, mask prediction relies efficiently on the decoder’s tracking memory. Figure 5: Edge inference schedule of SkyAnchor. The top row is the backbone, which runs only on anchor frames, re-anchoring every N−thN-th frame. Dashed boxes mark frames where it is skipped and the cached token is reused. The bottom row is the SAM 2 decoder (orange), which runs on every frame and propagates from the memory bank between anchors. The red dashed box is the tracking memory, a fixed-size FIFO window of size W that slides forward as each new frame enters and the oldest leaves. IV-A3 Mask Prediction Decoder We adopt SAM2 as the decoder ℱdecF_dec. Internally, a segmentation head transforms the backbone’s hidden states H at <seg> positions into dense prompt embeddings sege_seg. The decoder then consumes the memory from Hierarchical Memory Bank based on the sege_seg. The Hierarchical Memory Bank produces combined spatial memory and the SAM2 decoder predict the binary mask: t=ℱdec(<tsem∪<ttrk).M_t=F_dec(S^sem_<t ^trk_<t). (3) By querying both the permanent semantic anchors (semS^sem) and the recent tracking states (trkS^trk), this formulation seamlessly unifies long-term identity preservation and short-term multi-frame propagation. IV-B Hierarchical Memory Bank A fundamental challenge in online video object segmentation is maintaining object identity across temporal spans. We observe that “what an object is” changes slowly, while “where it is” changes rapidly. This motivates a two-level hierarchical memory bank equipped with a Semantic Memory Encoder and a Tracking Memory Encoder. IV-B1 Semantic Memory Layer This layer provides persistent identity information to both the VLM branch and the decoder, preventing long-term drift. Let ta(i)i=1Kt\t_a^(i)\_i=1^K_t denote all previous anchor frames before the current frame t, where ta(i)<t_a^(i)<t. For each anchor frame ta(i)t_a^(i), the Semantic Memory Encoder registers a semantic anchor and materialized differently in the two branches: • VLM-side: When reference masks are available, object-specific appearance features are pooled and projected into a token tasemz^sem_t_a. These tokens form the memory embeddings m=tasem∣ta<tZ_m=\z^sem_t_a t_a<t\ injected into Eq. (2). • Decoder-side: The hidden state at tat_a acts as a sparse conditioning token. The resulting anchor prediction is encoded into SAM2’s native memory format as a 2D spatial feature tasemm^sem_t_a. The spatial semantic memory is defined as: <tsem=ta(i)sem∣ta(i)<ti=1K.S^sem_<t= \m^sem_t_a^(i) t_a^(i)<t \_i=1^K. (4) These continuous representations are stored persistently as conditioning memories. Unlike discretized binary masks, they preserve rich categorical details (e.g., “silver sedan”), which is critical for resolving multi-object ambiguities during decoder-side memory attention. IV-B2 Tracking Memory Layer This layer captures fine-grained spatial continuity. Between consecutive anchor frames, SAM2 propagates masks, and the Tracking Memory Encoder writes each tracked frame τ into a 2D tracking state τtrkm^trk_τ. The tracking memory is defined as: <ttrk=τtrk|τ∈[t−W,t).S^trk_<t= \m^trk_τ\; |\;τ∈[t-W,\,t) \. (5) Unlike persistent semantic anchors, these are transient, non-conditioning memories consumed by SAM2’s memory attention before mask decoding. To bound memory consumption, they are maintained in a FIFO sliding window of size W. IV-C Semantics-Aware Token Router Processing high-resolution drone imagery requires a balance between visual detail and computational cost. We resolve this with a semantics-aware Token Router that dynamically compresses the dense visual tokens into the compact vZ_v consumed by ℱvlmF_vlm. IV-C1 Token Merging The Token Router partitions the dense spatial token grid into non-overlapping k×k× k blocks. Within each block, a lightweight scoring network evaluates the semantic salience of each raw token iv_i to produce an importance score sis_i. The block is then merged into a single output token via softmax-weighted aggregation: ^=Proj(∑i∈ℬsoftmax(si)⋅i), z=Proj\! ( _i softmax(s_i)·v_i ), (6) where Proj(⋅)Proj(·) is a learnable projection. The collection of all z across blocks constitutes vZ_v. The Router uses the same compression ratio for all spatial blocks, but adjusts the aggregation weights according to their visual content. IV-C2 Routing Loss To guide the importance scores toward target-related regions, we guide it with an auxiliary loss supervised by the ground-truth segmentation mask: ℒtotal=ℒvlm+λ1⋅ℒseg+λ2⋅ℒroute,L_total=L_vlm+ _1·L_seg+ _2·L_route, (7) where ℒroute=BCE(,¯gt)L_route=BCE(s,\, M_gt), with s being the predicted importance scores and ¯gt M_gt the ground-truth mask downsampled to the token grid resolution. IV-D Edge Deployment We run SkyAnchor on an edge platform as shown in Fig 5. The multimodal inference runs sparsely in time, while the tracking that emits a mask on every frame stays dense and cheap, so most frames skip the heavy backbone entirely. The backbone ℱvlmF_vlm is invoked only on anchor frames, spaced N frames apart, where it produces the conditioning token sege_seg that drives the decoder. This call is asynchronous, so the per-frame path keeps running on the cached token until the next one is ready and the backbone never stalls the stream. Every frame then passes through the SAM 2 path and memory attention propagates the target over the tracking memory <ttrkS^trk_<t, and the decoder ℱdecF_dec combines the hierarchical memory with the cached token to emit the mask tM_t, so a coherent mask is available even while the backbone is silent. The tracking memory is a fixed-size FIFO window of capacity W, where each frame appends a new state and evicts the oldest once full. On top of this, the backbone, the SAM 2 image encoder and the mask decoder are compiled into static-shape BF16 TensorRT engines, while the per-frame memory attention is captured into a CUDA Graph that replays its many small kernels as one operation. Algorithm 1 SkyAnchor Online Inference 1:Video stream It\I_t\, query Q, anchor interval N, FIFO size W, block size k 2:Mask stream t\M_t\ and answer A 3:Init: sem←∅S^sem\!←\! , trk←∅S^trk\!←\! (capacity W), seg←∅e_seg\!←\! 4:for each frame ItI_t do 5: if tmodN==1t N==1 then 6: ^←Proj(∑i∈ℬsoftmax(si)i) z \! ( _i softmax(s_i)\,v_i ) 7: v←^Z_v←\ z\ 8: ←ℱvlm([v;q;r;m])H _vlm\! ([Z_v;Z_q;Z_r;Z_m] ) 9: seg←SegHead()e_seg (H) 10: sem.append(tsem)S^sem.append(m^sem_t) 11: end if 12: t←ℱdec(<tsem∪<ttrk,seg)M_t _dec\! (S^sem_<t ^trk_<t,\ e_seg ) 13: trk.append(ttrk)S^trk.append(m^trk_t) 14: if |trk|>W|S^trk|>W then 15: pop oldest from trkS^trk 16: end if 17: yield t,M_t,\ A 18:end for V Experiment TABLE I: Quantitative comparison of object description and referring small object segmentation performance on the DroneEyes. Existing MLLM baselines are evaluated using their native offline inference mode, whereas SkyAnchor performs online streaming inference. Seg. LLM denotes the MLLM with segmentation cabilities. FT denotes the model finetuned on the DroneEyes train set. Model Type Source Size L1 - Object Description L2 - Referring Expression ROUGE-L METEOR GPT Scores J ℱ F &ℱ J\&F Gemini-3.0 General MLLM Closed - 18.40 19.10 1.21 1.51 4.01 2.76 GPT-5 General MLLM Closed - 17.44 16.10 1.61 0.87 2.62 1.75 GPT-4o General MLLM Closed - 18.91 21.11 1.42 0.75 2.04 1.39 Claude-Sonnet-4.5 General MLLM Closed - 16.36 20.50 1.47 0.54 1.37 0.95 Kimi-K2.5 [32] General MLLM Closed - 18.13 22.40 1.43 4.14 7.85 6.00 Qwen-3.6-A3B [26] General MLLM Open 35B 17.61 21.80 1.46 13.12 24.63 18.88 Gemma-4 General MLLM Open 31B 19.06 18.85 1.30 16.21 30.10 23.16 SAM3 [2] Seg. MLLM Open 1B - - - 12.67 14.04 13.35 VISA [38] Seg. MLLM Open 7B - - - 10.05 10.43 10.24 VideoGlaMM [23] Seg. MLLM Open 7B - - - 11.34 15.54 13.44 InstructSeg [35] Seg. MLLM Open 3B - - - 27.16 36.59 31.87 GLUS [15] Seg. MLLM Open 7B - - - 20.50 25.55 23.03 VRS-HQ [9] Seg. MLLM Open 7B - - - 27.19 36.26 31.73 Sa2VA [44] Seg. MLLM Open 4B - - - 21.81 29.10 25.45 Sa2VA [44] Seg. MLLM Open 8B - - - 21.36 30.20 25.78 Sa2VA [44] Seg. MLLM Open 26B - - - 27.61 38.49 33.05 UniPixel [19] Seg. MLLM Open 3B - - - 23.69 31.83 27.76 UniPixel [19] Seg. MLLM Open 7B - - - 27.96 36.57 32.26 Sa2VA-FT [44] Seg. MLLM Open 4B - - - 24.21 31.41 27.81 UniPixel-FT [19] Seg. MLLM Open 3B - - - 25.16 33.51 29.34 InstructSeg-FT [35] Seg. MLLM Open 3B - - - 28.61 37.76 33.19 SkyAnchor (ours) Unified Open 3B 45.40 46.31 3.22 30.96 45.23 38.10 V-A Experimental Setup V-A1 Datasets We evaluate our proposed SkyAnchor comprehensively across two distinct domains. • For the primary dynamic video scenarios, we utilize our proposed DroneEyes dataset, which provides unconstrained drone trajectories and pixel-level mask annotations. The dataset is split into training, validation, and testing sets with a ratio of 8:1:1. • To assess cross-domain generalization on static aerial imagery, we further benchmark our model on the recently released SkyFind [33] dataset, following its official validation and test splits. Its validation set (5,0005,000 samples) shares the same distribution as the training set (331,364331,364 samples), whereas its test set (16,54616,546 samples) consists of unseen maritime scenarios and thus introduces a clear domain shift. We perform no training on SkyFind and evaluate our model directly on both the validation and test split. V-A2 Evaluation Metrics For the DroneEyes-L1 subset, we evaluate the generated descriptions using standard natural language generation metrics, including METEOR, ROUGE-L and GPT scores. For GPT scores, we employ a GPT-based score (0–5 scale) using GPT-5.2 as the judge, which evaluates category accuracy, attribute correctness, spatial reasoning, and descriptive fluency given the original video frames as visual context as the existing setting [16]. For DroneEyes-L2 subset, we adopt the widely acknowledged standard metrics: the region similarity (J), the contour accuracy (ℱF), and their average (&ℱJ\&F). For referring image segmentation on SkyFind, we report the standard Intersection over Union at a threshold of 0.5 (IoU@0.50.5) and the mean IoU (IoU@mean), following the SkyFind [33] setting. To better characterize performance under different aggregation protocols, we report both macro-average and micro-average results. The macro-average treats the validation and test splits equally, while the micro-average weights each sample according to the split size. V-A3 Implementation Details Our architecture is built upon the pre-trained Qwen2.5-VL (3B parameters) as the vision-language backbone and SAM 2 as the mask prediction decoder. The model is optimized using AdamW with an initial learning rate of 2×10−52× 10^-5 and a cosine annealing scheduler. The LLM backbone is fine-tuned via LoRA (r=128r=128, α=256α=256). The training is conducted on 8 H800 GPUs with per-device batch size 4. For the Token Router, the merge size is set to 2, the maximum patch size to 960, and the auxiliary routing loss weight λ1=λ2=1 _1= _2=1. The router is trained with a separate learning rate of 10−410^-4, while the SAM 2 decoder uses 5×10−65× 10^-6. In the benchmark experiment, the models are inferenced on the H800 GPU. V-A4 Comparative Methods To evaluate our models, we benchmark it against diverse state-of-the-art methods aligned with different tasks. • For the Object Description task (DroneEyes Level 1 Task), we compare SkyAnchor against general MLLMs. For all baseline models, we sample frames from the input video and provide the target bounding box as normalized text coordinates [xmin,ymin,xmax,ymax][x_ ,y_ ,x_ ,y_ ] in the prompt, asking the model to describe the specified object. • For the Referring Expression task (DroneEyes Level 2 Task), we evaluate spatial localization acuity against existing general and segmentation MLLMs. For general MLLMs, we provide the query and frames and ask them to output the target bounding box as normalized text coordinates [xmin,ymin,xmax,ymax][x_ ,y_ ,x_ ,y_ ]. Segmentation MLLMs are designed for offline video understanding and do not provide a continuous streaming inference interface. We therefore follow their offline evaluation setting. Besides, we finetune SA2VA-4B, UniPixel-3B and InstructSeg-3B on the train set of DroneEyes to compare with SkyAnchor-3B (denoted as the FT series model). • Finally, to demonstrate our model’s generality, we benchmark on the external SkyFind dataset against visual grounding models reported following the original benchmark paper [33]. These visual grounding model are training on the training set of SkyFind. The general MLLMs in our study include closed-source (Gemini-3.0, GPT-5, GPT-4o, Claude-Sonnet-4.5, Kimi-K2.5) and open-source (Qwen-3.6, Gemma-4) models. For segmentation MLLMs, we evaluate the multimodal segmentation model SAM3, as well as MLLMs with inherent segmentation abilities (VISA, VideoGlaMM, InstructSeg, GLUS, VRS-HQ, SA2VA, and Unipixel). V-A5 Training Strategy We adopt a progressive multi-stage training strategy. In Stages 1, we align the base vision language model backbone (Qwen2.5-VL-3B) and the segmentation projection head, and the SAM 2 decoder on a mixture of general-purpose referring segmentation datasets, establishing the model’s core ability to perform language-guided segmentation on individual frames. In Stage 2, we introduce the Token Router and train it jointly on our DroneEyes dataset. All modules are jointly optimized with learning rates. V-A6 Real-world Deployment We deploy SkyAnchor on an NVIDIA Jetson AGX Orin 64GB for real-world edge inference. Aerial videos are collected on the HKUST(GZ) campus in Guangzhou, China, using a DJI Matrice 4E UAV equipped with a 20-MP 4/3 CMOS wide-angle camera. The drone is flying at altitudes of 4545 m and 7777 m. TABLE I: Ablation study on components on SkyAnchor. Components L1 - Object Description L2 - Referring Expression Token Router Memory Bank ROUGE-L METEOR GPT J ℱF &ℱJ\&F 34.26 35.18 2.31 22.34 32.18 27.26 ✓ 37.52 38.64 2.56 24.17 35.60 29.89 ✓ 40.13 41.27 2.82 28.43 41.56 35.00 ✓ ✓ 45.40 46.31 3.22 30.96 45.23 38.10 TABLE IV: Ablation study on Multi-layer Memory on SkyAnchor. Memory Bank L1 - Object Description L2 - Referring Expression Semantic Memory Tracking Memory ROUGE-L METEOR GPT J ℱF &ℱJ\&F 37.52 38.64 2.56 24.17 35.60 29.89 ✓ 43.18 44.52 3.05 26.13 38.85 32.49 ✓ 40.26 41.73 2.81 28.84 42.67 35.76 ✓ ✓ 45.40 46.31 3.22 30.96 45.23 38.10 TABLE V: Comparison with other components using in existing MLLMs. Method L1 - Object Description L2 - Referring Expression ROUGE-L METEOR GPT J ℱF &ℱJ\&F Token Compression Average Pooling 37.92 38.84 2.58 25.43 37.28 31.36 Pixel-Unshuffle [1] 40.16 41.05 2.79 26.88 39.42 33.15 Memory Strategy SAM2 Native Memory [27] 42.74 43.68 3.02 27.06 39.74 33.40 StreamingVLM Memory [37] 43.55 44.47 3.10 28.12 41.18 34.65 SkyAnchor (Ours) 45.40 46.31 3.22 30.96 45.23 38.10 Figure 6: Per-size &ℱJ\&F comparison. All three models are of comparable size (3B–4B) and fine-tuned on the same DroneEyes dataset. SkyAnchor outperforms three finetuned models across all size buckets, with a more margin on tiny objects. Numbers above bars denote the improvement of SkyAnchor over the best baseline. V-B Results on DroneEyes Video Dataset Table I jointly reports the Level-1 object description and Level-2 referring segmentation results on DroneEyes. V-B1 L1 Task Table I reports the Level-1 object description results. SkyAnchor achieves 45.4045.40 ROUGE-L, 46.3146.31 METEOR, and 3.223.22 GPT Score with only 3B parameters, consistently outperforming all compared models. General MLLMs struggle on this task. This shows that textual coordinate prompting is insufficient for precise spatial grounding in aerial imagery, where target objects are typically small and visually similar to surrounding context. We attribute our advantage to the semantics-aware Token Router, which preserves fine-grained visual details of small objects during token compression. By dynamically assigning higher importance scores to target-relevant tokens, the Token Router retains discriminative features of small objects that would otherwise be smoothed out by uniform downsampling. This enables SkyAnchor to generate accurate attribute descriptions even for objects occupying a small portion of the aerial frame. V-B2 L2 Task As shown in Table I, SkyAnchor (3B) achieves the best performance across all metrics, obtaining J, ℱF, and &ℱJ\&F scores of 30.96%30.96\%, 45.23%45.23\%, and 38.10%38.10\%, respectively, outperforming the second-best method SA2VA (26B, &ℱJ\&F = 33.05%33.05\%). General-purpose MLLMs including Gemini-3.0, GPT-5 and Claude-4.5 struggle to produce valid segmentation outputs on this task, indicating limitations of generic multimodal models in fine-grained segmentation of small objects in aerial scenes. Among open-source segmentation-specific models, the Sa2VA series shows gradual improvement as model size increases (4B→ 8B→ 26B, &ℱJ\&F from 25.45%25.45\% to 33.05%33.05\%), yet even the 26B variant falls far short of the 3B SkyAnchor, demonstrating that simply scaling up parameters cannot effectively address domain adaptation challenges in aerial imagery. Overall, SkyAnchor achieves state-of-the-art performance with the smallest parameter count, fully validating the effectiveness of its targeted design for small object referring segmentation in low-altitude aerial scenarios. TABLE VI: Quantitative comparison with existing SOTA models in the SkyFind dataset to evaluate SkyAnchor generation. We report both macro-average and micro-average results. Macro-average treats validation and test splits equally, while micro-average weights each sample according to the split size. Method IoU@0.5 IoU@mean Val Test Macro Avg. Micro Avg. Val Test Macro Avg. Micro Avg. FAOA [42] 23.10 13.08 18.09 15.41 10.21 6.91 8.56 7.68 RSC [41] 29.61 17.59 23.60 20.38 16.38 9.01 12.70 10.72 RefTR [13] 31.22 22.68 26.95 24.66 15.80 13.46 14.63 14.00 TransVG [5] 35.49 22.00 28.75 25.13 20.56 11.90 16.23 13.91 VLVTG [39] 30.29 23.52 26.91 25.09 14.40 13.32 13.86 13.57 SeqTR [46] 37.49 25.74 31.62 28.47 24.45 12.57 18.51 15.33 QRNet [43] 33.90 26.21 30.05 27.99 22.10 11.22 16.66 13.74 SimREC [20] 31.17 21.50 26.34 23.74 22.09 12.15 17.12 14.46 PolyFormer [17] 42.84 31.01 36.93 33.76 25.50 16.44 20.97 18.54 AerialREC [33] 45.21 38.13 41.67 39.77 29.78 20.38 25.08 22.56 SkyAnchor(ours) 32.66 48.02 40.34 44.46 30.50 39.85 35.18 37.68 V-C Results on the Tiny Objects Figure 6 compares SkyAnchor with UniPixel-3B-FT and Sa2VA-4B-FT on the DroneEyes dataset, reporting &ℱJ\&F across buckets defined by the target’s relative area. The three models have comparable parameter counts (3B–4B) and were all fine-tuned (FT) on the same dataset. SkyAnchor consistently outperforms both baselines across all size buckets, and its advantage grows as the target size decreases: in the two smallest-target intervals, ultra-tiny (relative area <10−4%<10^-4\%) and very-tiny (10−4%∼3×10−4%10^-4\% 3× 10^-4\%), SkyAnchor leads the strongest baseline by 9.59.5 and 10.810.8 percentage points, respectively. This suggests that SkyAnchor’s design offers an advantage in small-target segmentation from a drone’s perspective, rather than the improvement stemming merely from model scale or differences in fine-tuning data. V-D Ablation Studies V-D1 Effectiveness of Components We examine the contribution of each proposed component by adding the Token Router and Memory Bank. As reported in Table I, the base model without either component yields a &ℱJ\&F of 27.26%27.26\%, indicating that the plain VLM with SAM2 struggles with small object segmentation in aerial videos. Adding the Token Router alone improves &ℱJ\&F to 29.89%29.89\%, as it reduces background token interference and allows the model to attend more to foreground regions. Introducing the Memory Bank alone brings a gain (35.00%35.00\%), since temporal context across frames provides useful cues for locating and tracking small objects. When both components are combined, the model reaches 38.10%38.10\% in L2 &ℱJ\&F metrics, with consistent improvements on both L1 description metrics. This indicates that the two modules address complementary aspects of the problem. The Token Router improves spatial focus, while the Memory Bank supplies temporal information, and their combination yields the best overall performance. V-D2 Comparison with Other Components Table V presents a replacement-based comparison of the main components in SkyAnchor. In this table, we replace our Token Router with Average Pooling, a simple generic compression baseline, and Pixel-Unshuffle, the fixed-ratio token compression strategy used in Qwen2-VL and Qwen2.5-VL [1]. We also replace our Hierarchical Memory Bank with the native memory used in SAM2 and the streaming memory mechanism used in StreamingVLM [37]. The results show that these alternative designs all lead to lower performance than SkyAnchor. This suggests that our Token Router is more suitable for preserving small-object information during token compression, while our Hierarchical Memory Bank is more effective at maintaining temporal context across frames. V-D3 Ablation on Hierarchical Memory We further investigate the design of the Memory Bank by ablating its two layers: Semantic Memory and Tracking Memory. In this experiment, the Token Router is kept active in all configurations. As shown in Table IV, without any memory layer the model achieves 29.89%29.89\% &ℱJ\&F, relying solely on per-frame processing. Adding Semantic Memory improves the L1 description quality noticeably (GPT score from 2.562.56 to 3.053.05), as it preserves object-level semantic features that help generate more accurate descriptions. Adding Tracking Memory instead leads to a gain on L2 segmentation (&ℱJ\&F from 29.89%29.89\% to 35.76%35.76\%), since it maintains spatial location history that directly benefits mask prediction. The full two-layer memory achieves the best results across all metrics, confirming that Semantic Memory and Tracking Memory serve different purposes and are both needed for the complete pipeline. V-D4 Memory Capacity As shown in Fig.7(a), &ℱJ\&F improves sharply from 20.72%20.72\% (22 frames) and peaks around 1616 frames, after which adding more frames brings no further gain. Fig.7(b) shows the same peak holds regardless of video length, but long videos (>> 500500 frames) degrade slightly beyond 16 frames due to attention dilution from stale context. This indicates that more memory is not always better: 1616 frames already capture the effective temporal context. Figure 7: Ablation study on memory capacity. (a) Segmentation quality (&ℱJ\&F) and inference time as a function of the number of memory frames. (b) QA quality (GPT Score) and inference time as a function of the number of memory frames. V-D5 Token Routing Figure 8: Ablation study on Token Router. (a) Segmentation quality (&ℱJ\&F) and inference time under different setting. (b) QA quality (GPT Score) and inference time under different setting. We conduct an ablation study on the token routing strategy to investigate the trade-off between model effectiveness and inference efficiency. As shown in Fig. 8, we compare four configurations: no routing (retaining 100%100\% of visual tokens), and routing with 2×22× 2, 3×33× 3, and 4×44× 4 spatial merging grids that reduce the token count to 25%25\%, 11.1%11.1\%, and 6.25%6.25\%, respectively. Without token routing, the model achieves a &ℱJ\&F of 38.12%38.12\% but requires 13.3813.38 seconds per query averagely. Introducing the 2×22× 2 token router reduces inference time by 24.6%24.6\% (from 13.3813.38s to 10.0910.09s) with a performance drop of 0.020.02 points in &ℱJ\&F. The additional reduction in inference time is relatively small compared with the decrease in performance. We therefore use 2×22× 2 merging as the default setting. Figure 9: Ablation study of the anchor interval N on SkyAnchor. V-D6 Ablation on Anchor Interval Figure 9 evaluates the effect of the anchor interval N, which determines how often the VLM branch updates the semantic anchor. The segmentation results vary within a limited range for N∈4,8,16,32N∈\4,8,16,32\, with the highest score obtained at N=16N=16. We use N=16N=16 as the default setting because it gives the highest measured score while avoiding frequent VLM inference. V-E Cross-Domain Generalization to Unseen Scene We benchmark SkyAnchor-3B against SOTA methods on SkyFind to evaluate cross-domain generalization, as shown in Table VI. Unlike existing methods optimized on the SkyFind training split, SkyAnchor is evaluated without any additional training on this dataset. On the unseen maritime scenes in the test set, SkyAnchor outperforms the SOTA AerialREC on both metrics, improving IoU@0.5 from 38.1338.13 to 48.0248.02 and IoU@mean from 20.3820.38 to 39.8539.85. Under the micro-average evaluation, SkyAnchor achieves the best overall performance, obtaining 44.4644.46 IoU@0.5 and 37.6837.68 IoU@mean, compared with 39.7739.77 and 22.5622.56 for AerialREC, respectively. Under the macro-average evaluation, which assigns equal importance to the validation and test splits, SkyAnchor achieves the best IoU@mean performance (35.1835.18), while AerialREC obtains a slightly higher IoU@0.5 score (41.6741.67 vs. 40.3440.34). This difference reflects the complementary strengths of the two models: AerialREC performs strongly on the in-domain validation split, whereas SkyAnchor demonstrates superior robustness under the unseen domain shift. These results indicate that models specifically optimized for the source domain may achieve stronger in-domain accuracy, while SkyAnchor provides more stable generalization when the distribution shifts. Figure 10: Qualitative results of SkyAnchor on four challenging video sequences. The input queries are shown in the green boxes. For each case, we display the result from SA2VA-26B (the best baseline model in Table I) and ours with zoomed-in insets for better visualization of details. In case (a), (c) and (d), SA2VA-26B cannot detect the target in the video. In case (b), SA2VA-26B loses the target in some frames. Figure 11: Qualitative results of SkyAnchor for the real world deployment. The six queries are absent in the DroneEyes dataset. For privacy protection purposes, facial features in the video footage have been anonymized. V-F Device Deployment Analysis Table VII reports the deployment efficiency of SkyAnchor on an NVIDIA Jetson AGX Orin 64GB device paired with a DJI Matrice 4E UAV. The vanilla PyTorch implementation achieves only 1.91.9 FPS while consuming approximately 12.812.8 GB of GPU memory. TensorRT compilation increases the throughput to 3.23.2 FPS (1.68×1.68×) and reduces memory usage to approximately 9.09.0 GB through FP16 kernel fusion and reduced Python-side overhead. CUDA Graph capture of the SAM2 memory-attention path further improves the throughput to 3.73.7 FPS (1.95×1.95×), while asynchronous scheduling of the VLM and tracking branches delivers the largest gain, reaching 5.85.8 FPS (3.05×3.05×). The optimized system produces a complete output every approximately 172172 ms, supporting cloud-free online monitoring for latency-tolerant applications such as aerial surveillance, infrastructure inspection, and operator assistance. Although it is not intended for high-speed closed-loop control or native-camera-rate processing, this throughput is consistent with aerial tracking and edge video analytics systems operating at low frame rates or invoking expensive perception modules only every few frames [40, 8]. An SAM2-only configuration reaches 6.96.9 FPS, or approximately 145145 ms per output, indicating that the remaining VLM-related overhead is only about 2727 ms, corresponding to 16%16\% of the complete pipeline latency. TABLE VII: Ablation Study on Progressive Optimization for SkyAnchor Deployment. Configuration FPS ↑ GPU Mem ↓ Speedup Vanilla PyTorch (Base) 1.9 ∼ 12.8G 1.0× TensorRT Backbone Optimization TensorRT 3.2 ∼ 9.0G 1.68× + CUDA Graph Cache 3.7 1.95× + Async VLM 5.8 3.05× SAM2-Only (Upper Bound) 6.9 3.63× V-G Qualitative Results on DroneEyes Figure 10 presents qualitative comparisons between SkyAnchor-3B and SA2VA-26B across four scenarios. SA2VA achieves the second best results in the Table I. In scenario (a), following the query “Please find the red and yellow offshore supply vessel,” SkyAnchor-3B maintains tracking and segmentation as the vessel approaches from a distance, with relatively accurate boundaries under scale changes and textureless water surface. SA2VA-26B loses the target. For scenario (b), with the query “Help me to segment the red SUV,” SkyAnchor-3B segments the target vehicle in a residential street scene, distinguishing it from other parked cars without being affected by roadside trees and flags. SA2VA-26B detects the car in some frames but loses it in others. We attribute it to the memory usage in our SkyAnchor-3B. In scenario (c), under the aerial-view query “Where is the black car?”, SkyAnchor-3B localizes the small target spanning only a few pixels, without drifting to dark distractors such as shadows or roof textures. SA2VA-26B misses the vehicle in this bird’s-eye view. In scenario (d), answering “Can you find a silver car in the video?”, SkyAnchor-3B separates the distant silver car from the visually similar gravel road in an open landscape, showing some ability to recognize color and texture differences. SA2VA-26B fails to identify the target in this low-contrast environment. Across these examples, SkyAnchor shows three clear advantages. First, it maintains more stable tracking under large scale changes and long-range motion, as shown in the ship and red SUV cases. Second, it is more reliable for small targets, such as the black car that occupies only a few pixels in the aerial view. Third, it is less likely to drift to visually similar distractors, including shadows, roof textures, and low-contrast background regions. V-H Qualitative Results on Realworld Deployment Figure 11 shows the online segmentation results across different campus environments, guided by natural language queries222More video visualization demos can be seen in our website: https://sites.google.com/view/skyanchor/main-page. The test cases cover six queries absent in the DroneEyes. First, UAV maneuvers introduce continuous viewpoint changes: in cases (a) and (c), the circular skylight and sculpture are observed from varying oblique angles throughout the flight, yet the hierarchical memory bank preserves their structural semantics and keeps the masks stable without drifting to surrounding regions. Second, target scale varies dramatically across scenes. Case (d) involves large architectural structures such as building-side escalators, while cases (e) and (f) push to the opposite extreme where koi fish and street parking signs occupy only a handful of pixels. The semantics-aware Token Router selectively retains fine-grained cues for small targets while compressing uninformative background regions. Third, complex urban backgrounds introduce severe clutter and distractors. In case (b), pedestrians must be separated from standing plants using only the motion attribute in the query, and in case (c), the red sculpture must be grounded among other red-colored elements via a compositional spatial phrase. The MLLM leverages the full language query, including attribute, category, and spatial modifiers, to suppress distractors and maintain tracking continuity across frames. Across all six scenarios, SkyAnchor produces temporally coherent masks without any per-clip fine-tuning, demonstrating strong cross-domain robustness in real deployment. These deployment results indicate that the model generalizes to different object categories and flight conditions. VI Conclusion This paper present DroneEyes, the first large-scale aerial video referring segmentation dataset, and SkyAnchor, a memory-augmented MLLM framework for online referring segmentation in UAV video streams. DroneEyes addresses the absence of high-resolution aerial video benchmarks with dense pixel-level annotations by providing 2,1402,140 drone videos and 176,623176,623 grounding QA pairs organized into perception-description and referring-segmentation task levels. SkyAnchor tackles two core challenges of this domain: the Semantics-aware Token Router mitigates the extreme foreground-background imbalance by compressing irrelevant background tokens while preserving fine-grained target representations, and the Hierarchical Memory Bank decouples long-term semantic identity from short-term spatial tracking states. Extensive experiments show that SkyAnchor achieves state-of-the-art performance on both DroneEyes and the cross-domain SkyFind benchmark without fine-tuning. Furthermore, we deploy SkyAnchor on the edge platform, achieving streaming inference without cloud offloading. Real-world experiments on campus surveillance videos validate the practical viability of the entire pipeline for online deployment. Future work will pursue further model compression and hardware-aware optimization, enabling the pipeline to better match the native capture rate under fast target or platform motion. Beyond efficiency, we will explore integrating active camera control and autonomous navigation with the segmentation pipeline, where the referring-segmentation output directly drives UAV attitude adjustment and path planning to adaptively keep the target within the field of view under an optimal scale and viewpoint. We further plan to extend the framework toward multi-target referring as well as compositional queries involving relational and long-range temporal reasoning. References [1] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-vl technical report. External Links: 2502.13923, Link Cited by: §V-D2, TABLE V. [2] N. Carion, L. Gustafson, Y. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, et al. (2025) Sam 3: segment anything with concepts. arXiv preprint arXiv:2511.16719. Cited by: §I, TABLE I. [3] C. Chen, S. Liang, R. Guan, X. Sun, H. Zhao, H. Jiang, T. Huang, H. Ding, and Q. Han (2026) Aerialmind: towards referring multi-object tracking in uav scenarios. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 2805–2813. Cited by: TABLE I, §I. [4] L. Chen, X. Wei, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, B. Lin, Z. Tang, et al. (2024) Sharegpt4video: improving video understanding and generation with better captions. Advances in Neural Information Processing Systems 37, p. 19472–19495. Cited by: TABLE I. [5] J. Deng, Z. Yang, T. Chen, W. Zhou, and H. Li (2021) Transvg: end-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 1769–1779. Cited by: TABLE VI. [6] H. Ding, C. Liu, S. He, X. Jiang, and C. C. Loy (2023) Mevis: a large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF international conference on computer vision, p. 2694–2703. Cited by: §I. [7] H. Ding, C. Liu, S. He, K. Ying, X. Jiang, C. C. Loy, and Y. Jiang (2025) MeViS: a multi-modal dataset for referring motion expression video segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: TABLE I. [8] A. Ghosh, S. Iyengar, S. Lee, A. Rathore, and V. N. Padmanabhan (2023) React: streaming video analytics on the edge with asynchronous cloud support. In Proceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation, p. 222–235. Cited by: §I, §V-F. [9] S. Gong, Y. Zhuge, L. Zhang, Z. Yang, P. Zhang, and H. Lu (2025) The devil is in temporal token: high quality video reasoning segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 29183–29192. Cited by: §I-A, TABLE I. [10] X. Huang (2025) The small-drone revolution is coming—scientists need to ensure it will be safe. Nature 637 (8044), p. 29–30. Cited by: §I. [11] E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza (2023) Champion-level drone racing using deep reinforcement learning. Nature 620 (7976), p. 982–987. Cited by: §I. [12] K. Kuckreja, M. S. Danish, M. Naseer, A. Das, S. Khan, and F. S. Khan (2024) Geochat: grounded large vision-language model for remote sensing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 27831–27840. Cited by: §I-B. [13] M. Li and L. Sigal (2021) Referring transformer: a one-step approach to multi-task visual grounding. Advances in neural information processing systems 34, p. 19652–19664. Cited by: TABLE VI. [14] Y. Li, X. Liu, L. Liu, H. Fan, and L. Zhang (2025) Lamot: language-guided multi-object tracking. In 2025 IEEE International Conference on Robotics and Automation (ICRA), p. 6816–6822. Cited by: TABLE I, §I. [15] L. Lin, X. Yu, Z. Pang, and Y. Wang (2025) Glus: global-local reasoning unified into a single large language model for video segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 8658–8667. Cited by: §I-A, TABLE I. [16] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. Advances in neural information processing systems 36, p. 34892–34916. Cited by: §V-A2. [17] J. Liu, H. Ding, Z. Cai, Y. Zhang, R. K. Satzoda, V. Mahadevan, and R. Manmatha (2023) Polyformer: referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 18653–18663. Cited by: TABLE VI. [18] J. Liu, H. Ding, S. He, and Y. Jiang (2026) Event-aware instructed assistant for referring video segmentation. IEEE Transactions on Image Processing. Cited by: §I. [19] Y. Liu, Z. Ma, J. Pu, Z. Qi, Y. Wu, Y. Shan, and C. Chen (2026) Unipixel: unified object referring and segmentation for pixel-level visual reasoning. Advances in Neural Information Processing Systems 38, p. 126078–126108. Cited by: §I-A, TABLE I, TABLE I, TABLE I. [20] G. Luo, Y. Zhou, J. Sun, X. Sun, and R. Ji (2023) A survivor in the era of large-scale pretraining: an empirical study of one-stage referring expression comprehension. IEEE Transactions on Multimedia 26, p. 3689–3700. Cited by: TABLE VI. [21] J. Luo, Z. Pang, Y. Zhang, T. Wang, L. Wang, B. Dang, J. Lao, J. Wang, J. Chen, Y. Tan, et al. (2024) Skysensegpt: a fine-grained instruction tuning dataset and model for remote sensing vision-language understanding. arXiv preprint arXiv:2406.10100. Cited by: §I-B. [22] K. Maninis, S. Caelles, Y. Chen, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool (2018) Video object segmentation without temporal information. IEEE transactions on pattern analysis and machine intelligence 41 (6), p. 1515–1530. Cited by: §I. [23] S. Munasinghe, H. Gani, W. Zhu, J. Cao, E. Xing, F. S. Khan, and S. Khan (2025) Videoglamm: a large multimodal model for pixel-level visual grounding in videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 19036–19046. Cited by: §I-A, TABLE I. [24] B. A. Plummer, K. J. Shih, Y. Li, K. Xu, S. Lazebnik, S. Sclaroff, and K. Saenko (2020) Revisiting image-language networks for open-ended phrase detection. IEEE transactions on pattern analysis and machine intelligence 44 (4), p. 2155–2167. Cited by: §I. [25] Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu (2025) Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §I. [26] Qwen Team (2026-04) Qwen3.6-Plus: towards real world agents. External Links: Link Cited by: TABLE I. [27] N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. (2024) Sam 2: segment anything in images and videos. arXiv preprint arXiv:2408.00714. Cited by: §I-A, TABLE V. [28] A. Rozantsev, V. Lepetit, and P. Fua (2016) Detecting flying objects using a single moving camera. IEEE transactions on pattern analysis and machine intelligence 39 (5), p. 879–892. Cited by: §I, §I. [29] S. Seo, J. Lee, and B. Han (2020) Urvos: unified referring video object segmentation network with a large-scale benchmark. In European conference on computer vision, p. 208–223. Cited by: TABLE I. [30] P. Sun, Y. Song, X. Liu, X. Yang, Q. Wang, T. Li, Y. Yang, and X. Chu (2024) 3d question answering for city scene understanding. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 2156–2165. Cited by: §I. [31] P. Sun, Y. Song, X. Zhu, X. Liu, Q. Wang, Y. Liu, C. Xia, T. Li, Y. Yang, and X. Chu (2025) City-vlm: towards multidomain perception scene understanding via multimodal incomplete learning. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 3448–3457. Cited by: §I. [32] K. Team, T. Bai, Y. Bai, Y. Bao, S. Cai, Y. Cao, Y. Charles, H. Che, C. Chen, G. Chen, et al. (2026) Kimi k2. 5: visual agentic intelligence. arXiv preprint arXiv:2602.02276. Cited by: TABLE I. [33] K. Wang, G. Wu, X. Fu, X. Wang, K. Liu, X. Lu, C. Ge, W. Zhai, and Z. Zha (2026) SkyFind: a large-scale benchmark unveiling referring expression comprehension for uav.. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: TABLE I, §I, 2nd item, 3rd item, §V-A2, TABLE VI. [34] Y. Wang, J. Zhao, Z. Fan, X. Zhang, X. Wu, Y. Zhang, L. Jin, X. Li, G. Wang, M. Jia, et al. (2025) Jtd-uav: mllm-enhanced joint tracking and description framework for anti-uav systems. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 1633–1644. Cited by: TABLE I, §I-B. [35] C. Wei, Y. Zhong, H. Tan, Y. Zeng, Y. Liu, H. Wang, and Y. Yang (2025) Instructseg: unifying instructed visual segmentation with multi-modal large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 20193–20203. Cited by: §I-A, TABLE I, TABLE I. [36] J. Weyler, F. Magistri, E. Marks, Y. L. Chong, M. Sodano, G. Roggiolani, N. Chebrolu, C. Stachniss, and J. Behley (2024) Phenobench: a large dataset and benchmarks for semantic image interpretation in the agricultural domain. IEEE transactions on pattern analysis and machine intelligence 46 (12), p. 9583–9594. Cited by: §I. [37] R. Xu, G. Xiao, Y. Chen, L. He, K. Peng, Y. Lu, and S. Han (2025) Streamingvlm: real-time understanding for infinite video streams. arXiv preprint arXiv:2510.09608. Cited by: §V-D2, TABLE V. [38] C. Yan, H. Wang, S. Yan, X. Jiang, Y. Hu, G. Kang, W. Xie, and E. Gavves (2024) Visa: reasoning video object segmentation via large language models. In European Conference on Computer Vision, p. 98–115. Cited by: §I-A, TABLE I. [39] L. Yang, Y. Xu, C. Yuan, W. Liu, B. Li, and W. Hu (2022) Improving visual grounding with visual-linguistic verification and iterative reasoning. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 9499–9508. Cited by: TABLE VI. [40] T. Yang, D. Li, Y. Bai, F. Zhang, S. Li, M. Wang, Z. Zhang, and J. Li (2019) Multiple-object-tracking algorithm based on dense trajectory voting in aerial videos. Remote Sensing 11 (19), p. 2278. Cited by: §I, §V-F. [41] Z. Yang, T. Chen, L. Wang, and J. Luo (2020) Improving one-stage visual grounding by recursive sub-query construction. In European conference on computer vision, p. 387–404. Cited by: TABLE VI. [42] Z. Yang, B. Gong, L. Wang, W. Huang, D. Yu, and J. Luo (2019) A fast and accurate one-stage approach to visual grounding. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4683–4693. Cited by: TABLE VI. [43] J. Ye, J. Tian, M. Yan, X. Yang, X. Wang, J. Zhang, L. He, and X. Lin (2022) Shifting more attention to visual backbone: query-modulated refinement networks for end-to-end visual grounding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15502–15512. Cited by: TABLE VI. [44] H. Yuan, X. Li, T. Zhang, Y. Sun, Z. Huang, S. Xu, S. Ji, Y. Tong, L. Qi, J. Feng, et al. (2025) Sa2va: marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001. Cited by: §I-A, TABLE I, TABLE I, TABLE I, TABLE I. [45] W. Zhang, M. Cai, T. Zhang, Y. Zhuang, and X. Mao (2024) EarthGPT: a universal multimodal large language model for multisensor image comprehension in remote sensing domain. IEEE Transactions on Geoscience and Remote Sensing 62, p. 1–20. Cited by: §I-B. [46] C. Zhu, Y. Zhou, Y. Shen, G. Luo, X. Pan, M. Lin, C. Chen, L. Cao, X. Sun, and R. Ji (2022) Seqtr: a simple yet universal network for visual grounding. In European Conference on Computer Vision, p. 598–615. Cited by: TABLE VI.