Paper deep dive
Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding
Alberto Presta, Michal Byra, Grzegorz Stefański, Karol Szurkowski, Eryk Kołodziejczyk, Krzysztof Arendt
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/28/2026, 4:17:09 AM
Summary
The paper introduces Pocket-STVG (P-STVG), a lightweight architecture for Spatio-Temporal Video Grounding (STVG) that balances performance and efficiency. It utilizes a temporal-aware video encoder based on MobileViCLIP, a spatial encoder-decoder derived from MDETR, and a shared text encoder. The system operates in two phases: indexing (precomputing video embeddings) and inference (query-based decoding). It supports both weakly supervised (using a 1D U-Net decoder) and zero-shot (using thresholding) temporal grounding, achieving competitive results with fewer than 90M parameters.
Entities (7)
Relation Signals (7)
Pocket-STVG → solvestask → Spatio-Temporal Video Grounding
confidence 98% · We present Pocket-STVG (P-STVG), a lightweight cascade architecture that addresses STVG
Pocket-STVG → usescomponent → MDETR
confidence 95% · P-STVG integrates ... a spatial encoder-decoder derived from MDETR
Pocket-STVG → usescomponent → MobileViCLIP
confidence 95% · P-STVG integrates a temporal-aware video encoder based on MobileViCLIP
Pocket-STVG → hasparametercount → 90M
confidence 92% · Despite requiring fewer than 90M parameters, P-STVG performs on par with weakly supervised methods
Pocket-STVG → supportssetting → zero-shot
confidence 90% · enabling the same framework to operate in both weakly supervised and zero-shot settings.
Pocket-STVG → supportssetting → weakly supervised
confidence 90% · enabling the same framework to operate in both weakly supervised and zero-shot settings.
MDETR → usescomponent → RoBERTa
confidence 85% · However, MDETR exploits RoBERTa [25] text encoder
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Spatio-Temporal Video Grounding (STVG) aims to localize the spatio-temporal tube in a video corresponding to a natural language query. While recent methods achieve strong performance in fully supervised, weakly supervised, and zero-shot settings, they typically rely on computationally expensive architectures, complex training pipelines, or multimodal large language models. We present Pocket-STVG (P-STVG), a lightweight cascade architecture that addresses STVG by combining efficient pre-trained components instead of large end-to-end models. P-STVG integrates a temporal-aware video encoder based on MobileViCLIP, a spatial encoder-decoder derived from MDETR, and a shared aligned text encoder. Temporal localization is performed through either a lightweight 1D U-Net or a simple thresholding strategy, enabling the same framework to operate in both weakly supervised and zero-shot settings. Furthermore, video representations are precomputed independently of the query, yielding an indexing-friendly pipeline for efficient inference and large-scale video collections. Despite requiring fewer than 90M parameters, P-STVG performs on par with weakly supervised methods and improves on earlier zero-shot approaches at a fraction of their memory and computational cost, establishing a favorable performance-efficiency trade-off for STVG.
Tags
Links
- Source: https://arxiv.org/abs/2609.31135v1
- Canonical: https://arxiv.org/abs/2609.31135v1
Trouble viewing inline? Open PDF directly →
Full Text
57,130 characters extracted from source content.
Expand or collapse full text
Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding Alberto Presta Michal Byra Affiliation: Samsung AI Center, Warsaw, Poland Affiliation: IFTR, Polish Academy of Sciences, Warsaw, Poland*corresponding author: a.presta@samsung.com Grzegorz Stefański Affiliation: Samsung AI Center, Warsaw, Poland Karol Szurkowski Affiliation: Samsung AI Center, Warsaw, Poland Eryk Kołodziejczyk Affiliation: Samsung AI Center, Warsaw, Poland Krzysztof Arendt Affiliation: Samsung AI Center, Warsaw, Poland Abstract Spatio-Temporal Video Grounding (STVG) aims to localize the spatio-temporal tube in a video corresponding to a natural language query. While recent methods achieve strong performance in fully supervised, weakly supervised, and zero-shot settings, they typically rely on computationally expensive architectures, complex training pipelines, or multimodal large language models. We present Pocket-STVG (P-STVG), a lightweight cascade architecture that addresses STVG by combining efficient pre-trained components instead of large end-to-end models. P-STVG integrates a temporal-aware video encoder based on MobileViCLIP, a spatial encoder-decoder derived from MDETR, and a shared aligned text encoder. Temporal localization is performed through either a lightweight 1D U-Net or a simple thresholding strategy, enabling the same framework to operate in both weakly supervised and zero-shot settings. Furthermore, video representations are precomputed independently of the query, yielding an indexing-friendly pipeline for efficient inference and large-scale video collections. Despite requiring fewer than 90M parameters, P-STVG performs on par with weakly supervised methods and improves on earlier zero-shot approaches at a fraction of their memory and computational cost, establishing a favorable performance-efficiency trade-off for STVG. 1 Introduction Figure 1: Performance–efficiency comparison on HC-STVG v2. P-STVG occupies a compact operating point, retaining useful grounding accuracy with substantially fewer parameters than recent weakly supervised and zero-shot methods. Marker shapes indicate the supervision setting. Spatio-Temporal Video Grounding (STVG) focuses on localizing, both spatially and temporally, the target action specified by a textual query within a given video sequence. The rapid advancement of multimodal architectures and transformer-based models [3, 40], along with its importance in key applications such as anomaly detection and robotics [51, 43, 6], has propelled STVG into the forefront of research within the deep learning community. However, it is also challenging task as it requires merging complex information originating from different modalities, discarding noise and focusing on specific important targets. While first approaches divided STVG into cascades of sub-stask [52, 35, 46], more recent methods aim to develop compact end-to-end transformer based architectures that output the spatio-temporal tube directly. These models can be categorized into three distinct approaches: fully supervised methods [10, 9, 47], which rely on costly frame-level annotations; weakly-supervised methods [23, 7, 20], which utilize only video-text pairs or partially labeled data; and zero-shot methods [2, 50], which leverage powerful pre-trained vision-language models [29, 24, 30] to eliminate the dependency on annotated data. Fully supervised methods have achieved the best results in terms of both temporal and spatial detection, but they require dense data annotations and often overfit to training dataset, lacking generalization capability. On the other hand, both weakly and zero-shot approaches rely on pre-existing knowledge coming from huge pre-trained models, achieving remarkable results, even if inferior with respect to fully-supervised methods. However, these approaches often employ highly complex iterative optimization strategies [2] or depend on large-scale models during inference [50], which poses challenges for their practical deployment in offline scenarios on resource-constrained edge devices. In general, existing approaches focus on optimization of the spatio-temporal performance rather than models memory footprint, jeopardizing practical scenarios of using the system on mobile devices. On the contrary, recently it has been shown that a large image-text foundation models such as CLIP [29] can be distilled into smaller architectures for efficient learning [39], with straightforward extension to videos [49]. Inspired by recent efficiency-oriented architectures, we propose Pocket-STVG (P-STVG), a lightweight architecture for Spatio-Temporal Video Grounding. The pipeline consists of two stages: indexing and inference. During indexing, spatial and temporal video embeddings are precomputed independently of the query, enabling efficient processing of long videos and large-scale collections. During inference, the query is encoded into text embeddings, after which a lightweight temporal decoder first performs temporal grounding and a spatial decoder subsequently predicts the target bounding boxes. To maximize parameter efficiency, we adopt cross-architecture synthesis by aligning the video-text encoder with the spatial decoder for character-level span prediction, while temporal localization is performed by a lightweight 1D U-Net that effectively models long-range temporal dependencies. Despite requiring fewer than 90M parameters, P-STVG achieves competitive performance with both weakly supervised and zero-shot methods while substantially reducing memory footprint and computational cost. As illustrated in Fig. 1, P-STVG occupies a favorable performance–efficiency operating point, combining compact model size with a simple, non-iterative inference pipeline designed for resource-constrained video grounding scenarios. The contributions of this work are as follows: • We propose P-STVG, a lightweight and indexing-friendly architecture for STVG with fewer than 90M parameters, designed to balance grounding accuracy and computational efficiency in resource-constrained settings. • We introduce a query-independent video indexing stage that precomputes spatial and temporal representations, enabling efficient repeated inference over long videos and large-scale video collections. • We achieve competitive spatio-temporal grounding performance with weakly supervised and zero-shot methods while substantially reducing model size, memory footprint, and computational complexity. • We introduce a lightweight temporal decoder based on a 1D U-Net, together with a parameter-free thresholding alternative, enabling the same framework to operate under both weakly supervised and zero-shot temporal grounding settings. Figure 2: Blueprint of our proposed P-STVG architecture. The horizontal line separates indexing and inference phase, while red boxes represents modules we fine-tuned or trained, while gray boxes denote pre-trained modules that were kept frozen. 2 Related works 2.1 Fully supervised STVG In this setting, end-to-end neural networks are trained on fully annotated datasets with both spatial and temporal labels to predict spatio-temporal tubes. Early STVG methods generated candidate tubes before selecting the correct one [52, 35, 46], whereas recent transformer-based approaches directly predict the target tube in a single-stage pipeline [32, 47, 13, 44, 22, 10]. Among them, [47] first introduced a transformer architecture with a dual-stream encoder and a space-time decoder. This was extended by [9], which incorporated object visual cues for context guidance and a 3D encoder [27] to better capture temporal information. More recently, [10] introduced target-aware modules that generate object queries from video-text cues, improving performance but requiring prior target knowledge, typically provided as an additional word in the query, limiting practical applicability, while [8] exploited an autoregressive transformer for grounding a text-described person or event in long videos. [36] first identified when a queried event occurs, then used a separate query-guided spatial decoder to localize the referred object in each relevant frame. Its main contribution is a semantic bridge between temporal reasoning and precise bounding-box prediction in the spatial decoder. Overall, these methods primarily optimize benchmark performance, often at the expense of model size and generalization. 2.2 Weakly supervised STVG Weakly supervised STVG aims to localize spatio-temporal tubes using only video-level descriptions, avoiding costly frame-level bounding box annotations. Although weak supervision has been widely studied for image grounding [42, 26, 11] and video temporal grounding [54, 5], extending it to spatio-temporal grounding remains relatively unexplored. To mitigate the spurious correlations observed in STVG models, [20] proposed a language decomposition tree with structural attention to align linguistic components and visual regions at multiple levels. [12] instead formulates STVG as region-phrase and frame-sentence alignment, models ground-truth alignments as latent variables, and optimizes the joint distribution using variational EM. Addressing the limited temporal consistency of foundation models such as Grounding DINO [24], [17] employs GPT-3 to decompose complex queries, links textual queries into tubelets, and adopts a curriculum learning strategy from simple to complex videos. Similarly, [7] bridges image foundation models and video grounding through a tubelet referral grounding module that connects frame-level boxes into tubelets, combined with curriculum learning from sparse actions to dense scenes, while [19] generates text-conditioned object tubes and refines their spatial and temporal boundaries by reconstructing masked query phrases. 2.3 Zero-shot STVG Zero-shot STVG seeks to localize spatio-temporal tubes without task-specific training or annotated bounding boxes and temporal intervals. Early approaches adapted pretrained vision-language models not designed for STVG: [31] guides CLIP attention by marking target objects with red circles, while [33] combines cropped and blurred object candidates with a symbolic spatial resolver to improve relational reasoning. The first training-free zero-shot STVG method, [2], leverages pretrained models including CLIP, DINO, Grounding DINO, and SAM [24, 30]. It formulates temporal and spatial localization as latents, using an E-step to suppress background noise and an M-step with Kalman filtering and visual prototypes for spatial grounding. [50] exploits grounding tokens from multimodal large language models (MLLMs) [18, 4, 41]. More recently, agentic based frameworks have been proposed for STVG; [53] exploits Spatial Reasoning Agent and Temporal Reasoning Agent to collaboratively propose, track, verify, and temporally trim an object tube matching a text query. 3 Method 3.1 Preliminaries Given a video-query pair (V,Q), the goal is to localize the spatio-temporal tube BB=bbtt=tsteBB=\b_t\_t=t_s^t_e where bbtbb_t is the bounding box for time t and (tst_s,tet_e) indicate the start and the end of the detected time interval, respectively. In our pipeline, we divide the STVG task into two cascaded sub-tasks: first, temporal grounding, where we identify the correct time interval, followed by spatial grounding, where we extract the bounding boxes. As shown in Fig. 2, our solution is structured into two distinct phases: indexing and inference. The former encompasses all offline steps that do not depend on a specific text query, including the extraction of video embeddings for both spatial and temporal detection, while the latter involves the decoding process, where text embeddings are compared and processed to generate the final tube. 3.2 General architecture During indexing, a video V is divided into K non-overlapping chunks of equal size, resulting in a partition V=sii=1KV=\s_i\_i=1^K, where sis_i stands for the i-th video chunk. Each sis_i is then fed into the temporal video encoder (TVE) obtaining K time-aware semantic video embeddings Ev=eivi=1K=TVE(si)i=1KE^v=\e_i^v\_i=1^K=\TVE(s_i)\_i=1^K. In parallel, for each sis_i we sample NF frames fi1,…,fiNF\f_i^1,...,f_i^NF\, which are fed into the Spatial Image Encoder (SIE) that processes them individually to determine spatial-aware image embeddings Eiimg=eijimgj=1NF=SIE(fij)j=1NFE_i^img=\e_ij^img\_j=1^NF=\SIE(f_i^j)\_j=1^NF. During the inference, we start with an input text query Q, which is fed into the aligned text encoder (ATE) to obtain the corresponding text embedding sq=ATE(Q)s_q=ATE(Q). Next, the embedding is fed into the temporal video decoding (TVD) block along with the EvE^v in order to predict the temporal interval TI=(ts,te)TI=(t_s,t_e). Once we have TITI, we can retrieve E(ts,te)imgE_(t_s,t_e)^img relative to the detected interval and fed them along with sqs_q to the Spatial Image Decoder (SID) to obtain the final bounding boxes BB=bbtt=tste=SID(sq,E(ts,te)img)B=\b_t\_t=t_s^t_e=SID(s_q;E_(t_s,t_e)^img). Before TVDTVD, sqs_q is passed through a projection layer that helps alignment with the SID, which is fine-tuned as explained in Section 3.4. Importantly, in our setting we treat the temporal and spatial grounding as distinct tasks, allowing users to customize their focus and objectives, for example by prioritizing temporal grounding, spatial grounding, or both. 3.3 Temporal video grounding As part of the indexing stage, the video is divided into K non-overlapping equal-size chunks, obtaining V=sii=1KV=\s_i\_i=1^K, possibly achieving a reasonable trade-off between granularity and coherence of video content related information. We designed two distinct TVD techniques, to be used at inference stage. 3.3.1 Temporal Video Encoder (TVE) To extract temporal-aware embeddings, we exploited pre-trained Tiny MobileViCLIP [49]. This architecture is meticulously designed to ensure both reliability and efficiency, enabling easy deployment on edge devices such as smartphones. Specifically, building upon MobileCLIP [39], it employs an enhanced hybrid vision transformer, MCi0, as its image encoder [38]. For the text encoder, it utilizes an efficient hybrid model based on RepMixer [37]. With Tiny MobileViCLIP we obtained video embeddings Ev=eivi=1K=TVE(si)i=1KE^v=\e_i^v\_i=1^K=\TVE(s_i)\_i=1^K, which are used with the text embedding sqs_q to obtain temporal interval of interest at the inference stage. Algorithm 1 Shot-level Temporal Grounding via Thresholding Ev=eivi=1KE^v=\e^v_i\_i=1^K, sqs_q, τ Phase 1: Similarity computation SEvvq←CS(eiv,sq)i=1K∈ℝKS^vq_E^v←\CS(e^v_i,s_q)\_i=1^K ^K S^Evvq←MinMax(SEvvq) S^vq_E^v← MinMax(S^vq_E^v) Phase 2: Thresholding th←mean(S^Evvq)−τ⋅std(S^Evvq)th← mean( S^vq_E^v)-τ· std( S^vq_E^v) ⊳ Define Thresholding ¯←zeros(K) v← zeros(K) for i=1i=1 to K do if S^Evvq[i]≥th S^vq_E^v[i]≥ th then ¯[i]←1 v[i]← 1 else ¯[i]←0 v[i]← 0 end if end for Phase 3: hole filling and time interval extraction ¯←gaps_fill(¯) v← gaps\_fill( v) ts←find_first_moment(¯)t_s← find\_first\_moment( v) te←find_last_moment(¯)t_e← find\_last\_moment( v) return TI=(ts,te)TI=(t_s,t_e) 3.3.2 Temporal Video Decoder with thresholding In this scenario, we extract the temporal interval with an adaptive zero-shot thresholding technique. We first compute the similarity embedding S^Ev S_E^v as follows: SEvvq ^vq_E^v =CS(eiv,sq)i=1K∈ℝK, =\CS(e_i^v,s_q)\_i=1^K ^K, (1) S^Evvq S^vq_E^v =MinMax(SEvvq), =MinMax(S^vq_E^v), where CSCS and MinMaxMinMax represent the cosine similarity function and Min-Max normalization, respectively. Once we have the S^Ev S_E^v, which has values from 0 to 1, we first compute the adaptive threshold in the following way: th=mean(S^Evvq)−τ⋅std(S^Evvq),th=mean( S^vq_E^v)-τ· std( S^vq_E^v), (2) where τ is fixed to 0.5 a priori, without tuning on the evaluation sets, to preserve the fully zero-shot nature of this variant; the supplementary material reports a sensitivity analysis showing that performance is stable over a broad range of τ . For each chunk, we determine its inclusion within a temporal interval based on whether its corresponding similarity score exceeds the predefined threshold, thth, defined in eq. 2, obtaining the binary vector ¯ v. Subsequently, we apply an automatic post-processing step to eliminate noisy gaps in the resulting binary sequence; specifically, we perform hole-filling on a 1D sequence and extract the largest connected component, discarding all other smaller segments. This ensures that only the primary signal interval is preserved while bridging minor internal gaps. Ultimately, the predicted temporal intervals TI=(ts,te)TI=(t_s,t_e) are derived by identifying the first and the last selected shots within the longest continuous interval. This procedure requires no human intervention and is described by Algorithm 1. This approach offers two key benefits: (i) it operates in a zero-shot manner, as no part of the pipeline has been exposed to relevant datasets, and (i) it avoids adding extra parameters, leveraging existing pretrained components’ knowledge to address the task. Figure 3: Diagram of the T-Unet. Values inside boxes are input dimension, output dimension, kernel size, and padding. On square brackets there is embedding dimensions at the specific stage. Algorithm 2 Shot-level Temporal Grounding via T-Unet Ev=eivi=1KE^v=\e^v_i\_i=1^K, sqs_q Phase 1: Input computation i∗←argmaxi=1,…,KCS(eiv,sq)i^*← *arg\,max_i=1,…,KCS(e^v_i,s_q) eimaxv←ei∗ve^v_i_max← e^v_i^* SEvvq←CS(eiv,sq)i=1K∈ℝKS^vq_E^v←\CS(e^v_i,s_q)\_i=1^K ^K SEvvv←CS(eiv,eimaxv)i=1K∈ℝKS^v_E^v←\CS(e^v_i,e^v_i_max)\_i=1^K ^K S~Ev←[SEvvv:SEvvq] S_E^v←[\,S^v_E^v:S^vq_E^v\,] ⊳ Concatenation SEvhad←Hadamard(eiv,sq)i=1K∈ℝK×DS^had_E^v←\Hadamard(e^v_i,s_q)\_i=1^K ^K× D Phase 2: Thresholding with T-Unet ¯←T-Unet(SEvhad,S~Ev) v -Unet(S^had_E^v, S_E^v) Phase 3: hole filling and time interval extraction ¯←gaps_fill(¯) v← gaps\_fill( v) ts←find_first_moment(¯)t_s← find\_first\_moment( v) te←find_last_moment(¯)t_e← find\_last\_moment( v) return TI=(ts,te)TI=(t_s,t_e) 3.3.3 Temporal video decoder with Temporal Unet In this scenario, we introduce a lightweight module termed T-Unet, which employs a UNet-like architecture with one-dimensional convolutions, as illustrated in Fig. 3. The goal of this module is to combine information gathered by EvE^v, acting as an extractor of long-range temporal dependencies, enabling information aggregation between different video chunks. we first extract the shot similarities SEvvqS^vq_E^v between EvE^v and sqs_q, then we define eimaxve^v_i_max as follows: i∗=argmaxi=1,…,KCS(eiv,sq),eimaxv=ei∗v,i^*= *arg\,max_i=1,…,KCS(e^v_i,s_q), e^v_i_max=e^v_i^*, (3) which is the embedding of the video chunk with the highest similarity with respect to the query, used as the reference for the other embeddings. We then concatenate the two similarity vectors obtaining S~Ev=[SEvvv:SEvvq] S_E^v=[S^v_E^v:S^vq_E^v], which constitutes the first input of our decoder. Inspired by SimBase [1], the second input utilizes a straightforward Hadamard product between chunks and query embeddings, SEvhad=Hadamard(eiv,sq)i=1K∈ℝK×D,S^had_E^v=\Hadamard(e_i^v,s_q)\_i=1^K ^K× D, (4) where D is the embedding dimension. This way we avoid complex language-video interactions and significantly simplify the architecture. First, SEvhadS^had_E^v is projected into a latent space of dimension C (compression channel) via an intermediate representation of dimension MC (middle channel). It is then concatenated with SEvvvS^v_E^v and projected into a latent space of dimension MCMC. Finally, resulting embedding is inputted to the T-Unet with M encoder-decoder layers, generating the final binary response for each chunk, denoted as ¯∈ℝK v ^K. The vector ¯ v is passed through a sigmoid layer, constraining its values to the range [0, 1]. Following a procedure analogous to that described in 3.3.2, we apply a threshold operation, resulting in a binary vector where a value of 1 indicates detected shots and 0 signifies excluded ones; the same type of hole-filling post-processing is applied as in TVD with thresholding. The entire process is described in Algorithm 2. 3.4 Spatial video grounding Once the temporal video grounding is addressed, i.e. we have TI=(ts,te)TI=(t_s,t_e), we utilize image embeddings E(ts,te)imgE_(t_s,t_e)^img computed at the indexing stage and extract final bounding boxes by means of the SID BB=bbtt=tste=SID(sq,E(ts,te)img)B=\b_t\_t=t_s^t_e=SID(s_q;E_(t_s,t_e)^img). In order to accomplish this task, we employ MDETR [14], which is a well-known end-to-end integrated multimodal object detection model that identifies objects within images based on natural language queries. However, MDETR exploits RoBERTa [25] text encoder, since it excels at understanding the context of words in a sentence, and it allows Soft Token Prediction that can associate span of text tokens with a detected box. To save parameters and computational resources, it would be optimal to use a single text encoder for both temporal and spatial grounding. This requires fine-tuning the MDETR architecture to align with the text encoder from MobileVICLIP, which is designed for cross-modal alignment rather than natural language understanding. To perform cross-architecture synthesis, we adapt the MobileViCLIP text encoder to support character-level span prediction rather than global sequence representations. In particular, we implemented an extension of the standard BPE tokenizer that generates a deterministic offset mapping =(si,ei)i=1NO=\(s_i,e_i)\_i=1^N, where each pair is the start and end character indices for the i-th subword token in the original query. Given a query description and its associated character span [cstart,cend][c_start,c_end], the model extracts the relevant token T in the following way: =ti∣[si,ei]∩[cstart,cend]≠∅T=\t_i [s_i,e_i]∩[c_start,c_end]≠ \ (5) As performed by MobileViCLIP, we generate a sequence of hidden states H∈ℝL×DH ^L× D where L is the context length and D is the feature dimension. By exploiting character-level offsets, we are able to perform Soft Token Prediction, aligning the visual features extracted from the video backbone with specific sub-sequence of H. Additionally, an attention mask M∈0,1LM∈\0,1\^L is generated to differentiate valid semantic content from padding tokens. This mask is incorporated into the cross-modal transformer layers, ensuring visual features do not attend to null text embeddings, stabilizing the training of the multimodal reasoning heads. Table 1: Quantitative comparison on HCSTVG-v1/v2, between our methods and SOTA algorithms. Here, m_t and v_t stand for m_tIoU and m_vIoU, respectively. Light red indicates our method. STVG-supervised parameters count only modules exposed to STVG annotations Sup. Method STVG-sup. Pars Total Pars. HCSTVG-v1 HCSTVG-v2 (M) (M) m_t m_v v@.3 v@.5 m_t m_v v@.3 v@.5 Full Bridge-STG [36] ∼ 250–400 ∼ 8.98k - - - - 64.1 41.5 67.5 38.6 TA-STVG [10] 206 234 53 39.1 63.1 36.8 60.4 40.2 65.8 36.7 ART-STVG [8] 207 235 - - - - 59.2 39.2 64.4 33.2 TubeDETR [47] 185 185 43.7 32.4 49.8 23.5 53.9 36.4 58.8 30.6 CG-STVG [9] 203 231 52.8 38.4 61.5 36.3 60 39.5 64.5 36.3 Weak WINNER [20] ∼ 22-30 ∼ 162-275 - 14.2 17.2 6.1 - - - - CoSPal [17] ∼ 20-32 ∼ 160-262 - 22.2 31.4 18.9 - 16.0 20.1 13.1 STPro [7] ∼ 22-40 ∼ 162-285 - 17.6 27.0 12.9 - 20.0 31.1 14.6 TubeRMC [19] ∼ 40-100 ∼ 230-300 - 19.38 23.88 6.75 - - - - Zero-shot ASTG [53] 0 ∼ 30-100k - 32.3 54.4 28.2 - 34.8 55.5 31.4 RealVG [45] 0 ∼ 7k - 29.5 40.0 25.8 - 33.1 42.3 27 Unleash [50] 0 ∼ 7k - 24.8 41.5 16.3 - 27.7 44.7 19.5 E3M [2] 0 ∼ 7.5k - 19.1 29.4 10.6 - - - - RedCircle [31] 0 ∼ 149-423 - 9.2 7.8 1.6 - - - - ReClip [33] 0 ∼ 149-423 - 14.4 18.3 4.9 - - - - Zero-shot P-STVG Threshold 0 82.28 35.28 16.86 18.06 3.34 45.19 22.31 27.15 6.82 Spatial zero-shot P-STVG T-Unet 0.795 83.07 40.95 18.92 22.65 3.12 53.11 22.84 31.65 9.15 3.5 Training strategy 3.5.1 MDETR tuning with text encoder alignment We fine-tune the MDETR architecture [14] to align with the MobileViCLIP text encoder, starting from an EfficientNet-based pretrained model [34] and following the original training strategy (soft token prediction and contrastive alignment) on Flickr30k [28], MS COCO [21], and Visual Genome [16]. Inspired by [48], we add a distillation term encouraging our embeddings to match those of a frozen, RoBERTa-based MDETR teacher. Let ims,imtw_im^s,w_im^t and qs,qtw_q^s,w_q^t denote the student/teacher outputs of the image and text encoders, respectively: ℒdistill=λ2⋅(mse(ims,imt)+mse(qs,qt)).L_distill= λ2· (mse(w_im^s,w_im^t)+mse(w_q^s,w_q^t) ). (6) This term is added to the loss of [14] with λ=1λ=1; alignment of the text embedding is enabled by the trainable projection layer (Fig. 2). 3.5.2 T-Unet training The second step is to train T-Unet in order to be able to extrapolate the right time interval, as described in Section 3.3.3. In order to accomplish this task, we adopted the Dice loss function: ℒDice=1−2∑i=1Kp^iyi+ϵ∑i=1Kp^i+∑i=1Kyi+ϵ,L_Dice=1- 2 _i=1^K p_iy_i+ε _i=1^K p_i+ _i=1^Ky_i+ε, (7) where K is the number of shots, p^i∈[0,1] p_i∈[0,1] is the prediction for the i-th shot, yi∈0,1y_i∈\0,1\ is the ground truth, and ϵε is a small constant to ensure stability. We trained this architecture for 50 epochs, using 1×10−41×10^-4 as learning rate with Adam [15], reducing it after 3535 epochs to 5×10−55×10^-5. 4 Experiments 4.1 Datasets and Evaluation We evaluate on HCSTVG-v1/v2 [35] and VidSTG [52]. HCSTVG-v1 contains 5,660 video-text pairs; HCSTVG-v2 extends it to 10,131 training videos and 2,000 test samples. VidSTG comprises over 90k video-text pairs with declarative and interrogative sentences, of which we use only the declarative validation set. Following [17, 47], we report m_tIoU, m_vIoU, and vIoU@R. m_tIoU is the mean temporal IoU between predicted and ground-truth intervals, measuring temporal localisation alone. For a single sample, denoting by SiS_i and SuS_u the intersection and union of the predicted and ground-truth intervals, vIoU=1Su∑t∈SiIoU(b^t,bt)vIoU= 1S_u _t∈ S_iIoU( b_t,b_t), where b^t b_t and btb_t are the predicted and ground-truth boxes at frame t; m_vIoU is its average over the dataset. Finally, vIoU@R is the fraction of samples with vIoU>RvIoU>R, with R∈0.3,0.5R∈\0.3,0.5\. 4.2 Configuration Videos are divided into 20 non-overlapping chunks for HCSTVG-v1/v2 and 32 for VidSTG (roughly one embedding per second), balancing temporal decoder granularity, sufficient frame-level variability for MobileViCLIP, and a manageable number of chunks at indexing time. For T-Unet we fix L=4L=4, MC=128MC=128, and CC=8C=8, giving a ∼ 0.795M-parameter network trained only on temporal labels, ignoring bounding boxes and leaving the rest of the pipeline unchanged. Our method is spatially zero-shot with T-Unet and fully zero-shot with thresholding: no component is ever trained on STVG data. The spatial branch is fine-tuned only on image-grounding datasets and the T-Unet only on temporal labels, so the parameters we report as STVG-supervised in Tab.1 count only the latter. We report total parameters alongside, so that the two accounting conventions can be compared directly. 4.3 Comparison with state-of-the-art methods 4.3.1 Quantitative results Table 1 compares our method against fifteen state-of-the-art (SOTA) methods: five fully supervised [10, 47, 9, 36, 8], four weakly supervised [20, 7, 17, 19], and six zero-shot [50, 2, 31, 33, 45, 53]. We also report trainable and total parameters, estimated when not stated in the original paper (details in the supplementary material). Against weakly supervised methods the picture depends on the dataset. On HCSTVG-v2, both variants exceed all such methods in terms of m_vIoU (22.31 and 22.84 against 20.0 for STPro and 16.0 for CoSPal), despite using no Spatial annotation. On HCSTVG-v1 the margin narrows: P-STVGUnetP-STVG_Unet surpasses WINNER and STPro (18.92 against 14.2 and 17.6) but trails TubeRMC (19.38) and CoSPal (22.2), while the thresholding variant (16.86) exceeds only WINNER. On both datasets our v@.5 remains below these methods, indicating that the gap is concentrated in strict spatial overlap rather than in the coarser localization captured by m_vIoU. These results are obtained with a smaller parameter budget and a single forward pass, split into indexing and inference, rather than the multi-stage training and inference schemes these methods usually employ. To locate the source of this spatial gap, we evaluate the spatial branch with ground-truth temporal intervals. Under oracle temporal localization, P-STVG reaches 39.73 m_vIoU, 62.45 v@.3, and 32.85 v@.5 on HCSTVG-v2, comparable to fully supervised methods such as TA-STVG (40.2 m_vIoU, 36.7 v@.5) and ART-STVG (39.2, 33.2) despite using MDETR as off-the-shelf grounding model. Relative to our end-to-end results (22.84 m_vIoU, 9.15 v@.5), this shows that the spatial decoder is accurate and that most of the loss comes from imprecise temporal intervals. Among zero-shot methods, P-STVG clearly outperforms RedCircle and ReCLIP, while agentic and MLLM-based approaches (ASTG, RealVG, Unleash) achieve higher accuracy using models of several billion parameters and considerably more complex inference. On VidSTG (Tab. 2) the same pattern holds. In the spatial zero-shot setting, P-STVGUnetP-STVG_Unet attains the best temporal accuracy among non-fully-supervised methods (m_tIoU 46.57 against 41.10 for CoSPal) and an m_vIoU comparable to weakly supervised approaches (15.70 against 15.5–16.0), while remaining below them on v@.5. The fully zero-shot variant is weaker (m_vIoU 14.25), exceeding only RedCircle. Throughout, our full architecture stays below 90M parameters, nearly two orders of magnitude smaller than the MLLM-based zero-shot methods, placing P-STVG at a distinct operating point defined by deployment feasibility rather than maximum accuracy. Table 2: Spatio-temporal performance on VidSTG declarative. Method m_tIoU m_vIoU v@.3 v@.5 Bridge-STG 52.6 37.2 52.4 37.4 TA-STVG 51.7 34.4 48.2 33.5 STPro - 15.5 19.4 12.7 COSPAL 41.10 16 20.1 13.1 Tube-RMC - 15.93 25.16 9.09 RedCircle - 8.6 7.6 0.9 E3M - 16.2 20.5 11.9 Unleash - 18 29.8 12.2 RealVG - 29 35.0 25.5 P-STVG Threshold 33.47 14.25 16 3.4 P-STVG T-Unet 46.57 15.70 19 9.16 Table 3: Efficiency analysis on HC-STVG. We report peak VRAM, query time, and TFLOPs under the same inference protocol on an RTX 4060 GPU with 8 GB of VRAM. Blue indicates methods that could not be executed within this hardware budget and were measured on an H100 instead. Method Peak VRAM (GB) Query Time (s) TFLOPs TA-STVG 4.36 0.90 2.87 UNLEASH 14.71 3.35 8.09 P-STVG T-Unet 3.94 0.71 1.11 Table 4: Results in terms of m_tIoU,tIoU@3, and tIoU@5 on HCSTVG-v2 considering different T-Unet configurations. In light red the one chosen for the main architecture architecture configuration training details performance ID MC C L Total pars LR BS m_tIoU tIoU@3 tIoU@5 TVD1TVD_1 128 8 1 351625 1e-4 16 50.11 78.45 54.95 TVD2TVD_2 512 8 1 4749833 1e-4 16 51.66 80.75 58.10 TVD3TVD_3 512 8 2 5536265 5e-5 16 52.14 81.75 59.95 TVD4TVD_4 512 8 3 7896585 5e-5 16 52.47 81.75 61.60 TVD5TVD_5 512 8 4 10256905 1e-4 16 53.12 82.65 61.15 ∗TVD^* 128 8 4 794961 1e-4 16 53.11 81.90 61.75 4.3.2 Efficiency analysis Tab. 3 reports peak VRAM, query time, and TFLOPs at inference, measured for a single text query on a 10-video subset of the HC-STVG validation set using an RTX 4060 with 8 GB of VRAM; the offline indexing stage is reported separately for our method (tab. 5). The subset is representative, as the videos are of comparable complexity. We report the T-Unet variant only, since the difference with respect to thresholding is negligible. We compare against TA-STVG and UNLEASH, representing strong supervised and zero-shot baselines, respectively. Because UNLEASH could not be executed within the 8 GB VRAM constraint and required an H100 GPU instead, its results are shown for reference but should be interpreted under a different hardware budget. Under this protocol, P-STVG achieves lower peak memory usage, shorter query time, and fewer TFLOPs, highlighting the efficiency advantage of the proposed pipeline. 4.4 Ablation study and Internal analysis 4.4.1 Different T-Unet configuration The T-Unet configuration governs the trade-off between m_tIoU and memory footprint. Tab. 4 reports the most representative configurations of compression channels (C), middle channels (MC), and number of layers (L), together with learning rate (LR) and batch size (BS), out of the full grid search summarised in Fig. 4 (blued dots). Performance tracks architecture rather than parameter count: TVD∗TVD^* outperforms TVD3TVD_3 and TVD4TVD_4 despite having roughly an order of magnitude fewer parameters, and matches TVD5TVD_5 with one thirteenth of its budget. Over the full sweep (Fig. 4), the Pearson correlation between m_tIoU and L is 0.62, against 0.11 and 0.20 for MC and C, indicating that depth contributes considerably more than channel width. Figure 4: m_tIoU with respect to MC, C, and L. Each blue dot is one trained configuration from the full grid search. The red line is the linear trend line that best fits them, the light red zone represents the 95% confidence interval around the regression line, and r is the Pearson correlation. Table 5: Evaluation of our pipeline in terms of m_tIoU considering different number of chunks (NC), on hcstvg-v2. We also report memory footprint for a single video (MF) and time for indexing (IT). Light red indicates main architecture configuration. NC m_tIoU T-Unet m_tIoU Thresholding MF (Kb) IT (s) 40 51.87 43.71 82 3.29 24 52.40 44.31 49.3 1.95 16 48.56 44.01 32.9 1.29 12 45.67 43.83 24.7 0.99 20 53.11 45.19 44 1.61 4.4.2 Different number of chunks Tab.5 reports effect of varying number of chunks when dividing the video. Increasing the number of chunks enlarges the video embedding (e.g., 49.3 KB for 24 chunks vs. 44 KB for 20) and increases indexing time due to the extraction of EvE^v by TVE. However, performance does not improve monotonically, likely because of dataset characteristics, increased noise with excessive chunks, and MobileViCLIP requiring a sufficient temporal span to produce meaningful embeddings. Conversely, too few chunks (e.g., 12) reduce performance because of coarser temporal predictions. 5 Conclusion We presented Pocket-STVG, a lightweight framework for spatio-temporal video grounding that assembles efficient pre-trained components into a compact cascade split into an indexing and an inference stage. It comprises a temporal video encoder and aligned text encoder based on MobileViCLIP, a spatial encoder-decoder derived from MDETR, and a temporal video decoder implemented either as simple thresholding or as a lightweight U-Net. Video representations are computed independently of the query, so inference reduces to a single forward pass. With fewer than 90M parameters, P-STVG matches weakly supervised methods in terms of m_vIoU on HCSTVG-v2 and improves on earlier zero-shot approaches, at a fraction of their memory and computational cost. Some limitations should be noted. Under strict spatial overlap our vIoU@0.5 remains below weakly supervised methods on both benchmarks, so the compact pipeline comes at a measurable accuracy cost when precise boxes are required. Furthermore, efficiency is measured on a resource-constrained desktop GPU rather than on mobile hardware Improving temporal precision is therefore the most promising next step, through finer or overlapping chunking and by propagating boxes across frames to stabilize predictions near interval boundaries. The residual spatial margin could be narrowed by replacing per-frame detection with a video-based decoder that exploits temporal continuity, or by adapting the current one through parameter-efficient tuning such as LoRA. Validation on mobile hardware and on longer videos, where query-independent indexing should pay off most, is a natural further direction. References [1] P. Bao and A. C. Kot (2024) Simbase: a simple baseline for temporal video grounding. arXiv preprint arXiv:2411.07945, p. . Cited by: §3.3.3. [2] P. Bao, Z. Shao, W. Yang, B. P. Ng, and A. C. Kot (2024) E3m: zero-shot spatio-temporal video grounding with expectation-maximization multimodal modulation. In European Conference on Computer Vision, p. 227–243. Cited by: Table 8, §1, §2.3, Table 1, §4.3.1. [3] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020) End-to-end object detection with transformers. In European conference on computer vision, p. 213–229. Cited by: §1. [4] L. Chen, X. Wei, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, B. Lin, Z. Tang, et al. (2024) Sharegpt4video: improving video understanding and generation with better captions. Advances in Neural Information Processing Systems 37, p. 19472–19495. Cited by: §2.3. [5] L. Chen, Y. Niu, B. Chen, X. Lin, G. Han, C. Thomas, H. Ayyubi, H. Ji, and S. Chang (2022) Weakly-supervised temporal article grounding. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 9402–9413. Cited by: §2.2. [6] A. Gao and J. Liu (2025) STEAD: spatio-temporal efficient anomaly detection for time and compute sensitive applications. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 6256–6263. Cited by: §1. [7] A. Garg, A. Kumar, and Y. S. Rawat (2025) Stpro: spatial and temporal progressive learning for weakly supervised spatio-temporal grounding. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 3384–3394. Cited by: Table 8, §1, §2.2, Table 1, §4.3.1. [8] X. Gu, B. Fan, J. Yao, Z. Zhang, Y. Huang, C. Han, H. Fan, and L. Zhang (2026) Towards long-form spatio-temporal video grounding. arXiv preprint arXiv:2602.23294. Cited by: §2.1, Table 1, §4.3.1. [9] X. Gu, H. Fan, Y. Huang, T. Luo, and L. Zhang (2024) Context-guided spatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 18330–18339. Cited by: §1, §2.1, Table 1, §4.3.1. [10] X. Gu, Y. Shen, C. Luo, T. Luo, Y. Huang, Y. Lin, H. Fan, and L. Zhang (2025) Knowing your target: target-aware transformer makes better spatio-temporal video grounding. International Conference on Learning Representations (ICLR). Cited by: §1, §2.1, §2.1, Table 1, §4.3.1. [11] T. Gupta, A. Vahdat, G. Chechik, X. Yang, J. Kautz, and D. Hoiem (2020) Contrastive learning for weakly supervised phrase grounding. In European Conference on Computer Vision, p. 752–768. Cited by: §2.2. [12] Y. Jin and Y. Mu (2024) Weakly-supervised spatio-temporal video grounding with variational cross-modal alignment. In European Conference on Computer Vision, p. 412–429. Cited by: §2.2. [13] Y. Jin, Z. Yuan, Y. Mu, et al. (2022) Embracing consistency: a one-stage approach for spatio-temporal video grounding. Advances in Neural Information Processing Systems 35, p. 29192–29204. Cited by: §2.1. [14] A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion (2021) Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF international conference on computer vision, p. 1780–1790. Cited by: Appendix A, §3.4, §3.5.1, §3.5.1. [15] D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diego, CA. Cited by: §3.5.2. [16] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L. Li, D. A. Shamma, et al. (2017) Visual genome: connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123 (1), p. 32–73. Cited by: §3.5.1. [17] A. Kumar, Z. Kira, and Y. S. Rawat (2025) Contextual self-paced learning for weakly supervised spatio-temporal video grounding. International Conference on Learning Representations (ICLR). Cited by: Table 8, §2.2, Table 1, §4.1, §4.3.1. [18] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, et al. (2024) Llava-onevision: easy visual task transfer. arXiv preprint arXiv:2408.03326. Cited by: §2.3. [19] J. Li, Y. Zhang, J. Hu, C. Tan, T. Liang, and B. Xia (2026) TubeRMC: tube-conditioned reconstruction with mutual constraints for weakly-supervised spatio-temporal video grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 6253–6261. Cited by: §2.2, Table 1, §4.3.1. [20] M. Li, H. Wang, W. Zhang, J. Miao, Z. Zhao, S. Zhang, W. Ji, and F. Wu (2023) Winner: weakly-supervised hierarchical decomposition and alignment for spatio-temporal video grounding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 23090–23099. Cited by: Table 8, §1, §2.2, Table 1, §4.3.1. [21] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In European conference on computer vision, p. 740–755. Cited by: §3.5.1. [22] Z. Lin, C. Tan, J. Hu, Z. Jin, T. Ye, and W. Zheng (2023) Collaborative static and dynamic vision-language streams for spatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 23100–23109. Cited by: §2.1. [23] K. Liu, M. Qu, Y. Liu, Y. Wei, W. Zhe, Y. Zhao, and W. Liu (2024) Single-frame supervision for spatio-temporal video grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (7), p. 5177–5191. Cited by: §1. [24] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. (2024) Grounding dino: marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, p. 38–55. Cited by: §1, §2.2, §2.3. [25] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) RoBERTa: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: §3.4. [26] Y. Liu, B. Wan, L. Ma, and X. He (2021) Relation-aware instance refinement for weakly supervised visual grounding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5612–5621. Cited by: §2.2. [27] Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu (2022) Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 3202–3211. Cited by: §2.1. [28] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik (2015) Flickr30k entities: collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, p. 2641–2649. Cited by: Appendix A, §3.5.1. [29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1. [30] N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. (2024) Sam 2: segment anything in images and videos. arXiv preprint arXiv:2408.00714. Cited by: §1, §2.3. [31] A. Shtedritski, C. Rupprecht, and A. Vedaldi (2023) What does clip know about a red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 11987–11997. Cited by: Table 8, §2.3, Table 1, §4.3.1. [32] R. Su, Q. Yu, and D. Xu (2021) Stvgbert: a visual-linguistic transformer based framework for spatio-temporal video grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 1533–1542. Cited by: §2.1. [33] S. Subramanian, W. Merrill, T. Darrell, M. Gardner, S. Singh, and A. Rohrbach (2022) Reclip: a strong zero-shot baseline for referring expression comprehension. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5198–5215. Cited by: Table 8, §2.3, Table 1, §4.3.1. [34] M. Tan and Q. Le (2019) EfficientNet: rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, p. 6105–6114. Cited by: §3.5.1. [35] Z. Tang, Y. Liao, S. Liu, G. Li, X. Jin, H. Jiang, Q. Yu, and D. Xu (2021) Human-centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology 32 (12), p. 8238–8249. Cited by: §1, §2.1, §4.1. [36] X. Tu, J. Wu, F. Kang, Q. Nong, K. Zhang, C. Niu, and F. Wu (2026) Bridging time and space: decoupled spatio-temporal alignment for video grounding. arXiv preprint arXiv:2604.08014. Cited by: Table 8, §2.1, Table 1, §4.3.1. [37] P. K. A. Vasu, J. Gabriel, J. Zhu, O. Tuzel, and A. Ranjan (2023) Fastvit: a fast hybrid vision transformer using structural reparameterization. In Proceedings of the IEEE/CVF international conference on computer vision, p. 5785–5795. Cited by: §3.3.1. [38] P. K. A. Vasu, J. Gabriel, J. Zhu, O. Tuzel, and A. Ranjan (2023) Mobileone: an improved one millisecond mobile backbone. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 7907–7917. Cited by: §3.3.1. [39] P. K. A. Vasu, H. Pouransari, F. Faghri, R. Vemulapalli, and O. Tuzel (2024) Mobileclip: fast image-text models through multi-modal reinforced training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15963–15974. Cited by: §1, §3.3.1. [40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §1. [41] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. (2024) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Cited by: §2.3. [42] Q. Wang, H. Tan, S. Shen, M. Mahoney, and Z. Yao (2020) Maf: multimodal alignment framework for weakly-supervised phrase grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 2030–2038. Cited by: §2.2. [43] R. Wang, J. Huang, J. Zhang, X. Liu, X. Zhang, Z. Liu, P. Zhao, S. Chen, and X. Sun (2024) Facialpulse: an efficient rnn-based depression detection via temporal facial landmarks. In Proceedings of the 32nd ACM international conference on multimedia, p. 311–320. Cited by: §1. [44] S. T. Wasim, M. Naseer, S. Khan, M. Yang, and F. S. Khan (2024) Videogrounding-dino: towards open-vocabulary spatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 18909–18918. Cited by: §2.1. [45] H. Wei and Z. Chen (2025) RealVG: unleashing mllms for training-free spatio-temporal video grounding in the wild. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 4271–4280. Cited by: Table 1, §4.3.1. [46] M. Yamaguchi, K. Saito, Y. Ushiku, and T. Harada (2017) Spatio-temporal person retrieval via natural language queries. In Proceedings of the IEEE international conference on computer vision, p. 1453–1462. Cited by: §1, §2.1. [47] A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid (2022) Tubedetr: spatio-temporal video grounding with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16442–16453. Cited by: §1, §2.1, §2.1, Table 1, §4.1, §4.3.1. [48] C. Yang, Z. An, L. Huang, J. Bi, X. Yu, H. Yang, B. Diao, and Y. Xu (2024) Clip-kd: an empirical study of clip model distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15952–15962. Cited by: §3.5.1. [49] M. Yang, Z. Jia, Z. Dai, S. Guo, and L. Wang (2025) Mobileviclip: an efficient video-text model for mobile devices. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 20824–20835. Cited by: §1, §3.3.1. [50] Z. Yang, Y. Liu, G. Hancke, and R. W. Lau (2025) Unleashing the potential of multimodal llms for zero-shot spatio-temporal video grounding. Advances in Neural Information Processing Systems. Cited by: Table 8, §1, §2.3, Table 1, §4.3.1. [51] S. Zeng, X. Chang, M. Xie, X. Liu, Y. Bai, Z. Pan, M. Xu, X. Wei, and N. Guo (2025) Futuresightdrive: thinking visually with spatio-temporal cot for autonomous driving. arXiv preprint arXiv:2505.17685. Cited by: §1. [52] Z. Zhang, Z. Zhao, Y. Zhao, Q. Wang, H. Liu, and L. Gao (2020) Where does it exist: spatio-temporal video grounding for multi-form sentences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 10668–10677. Cited by: §1, §2.1, §4.1. [53] H. Zhao, Y. Ong, and J. T. Zhou (2026) Agentic spatio-temporal grounding via collaborative reasoning. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 2475–2485. Cited by: Table 8, §2.3, Table 1, §4.3.1. [54] M. Zheng, Y. Huang, Q. Chen, and Y. Liu (2022) Weakly supervised video moment localization with contrastive negative sample mining. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, p. 3517–3525. Cited by: §2.2. Supplementary Material Appendix A Distillation in MDETR tuning On table 6 we have results on Flickr test dataset [28], in terms of recall@k with k = 1,5,10, i.e. the percentage of queries for which at least one relevant item is found in the top-k results ranked by the model. Light-blue row refers to the results from the teacher network, which is the EfficientNet-B3 based model trained by [14]. As it is possible to observe, the major gain in our results happens when we fine-tune starting from the teacher itself, meaning that a good starting point is vital to align the model to a new text encoder. On the other hand, the contribution that the distillation term gave is marginal, even if present to a certain level; this suggests that this can be a promising direction to follow in future developments. Table 6: Results of MDETR fine-tuning for text encoder alignment in terms of Recall@1,5,10. Light blue line represents performance of the teacher. Starting Net λ R@1 R@5 R@10 Scratch 1 72.27 86.11 90.84 Scratch 0 72.09 85.55 89.66 Teacher 1 79.41 91.84 94.48 Teacher 0 79.28 91.65 94.29 - - 82.9 93.8 95.6 Appendix B Qualitative samples Fig. 5 presents qualitative examples in which thresholding outperforms U-Net temporal grounding. Note that the spatial grounding may differ even when the same method is applied, as the frames considered in each case are different. Fig. 6 shows the same thing, but for a sample where our temporal grounding method outperformed plain thresholding. Appendix C Zero-shot quantitative results using different threshold In tab. 7 we analyze results in terms of m_tIoU considering our proposed method in a zero-shot configuration, i.e. by applying thresholding with different τ. We consider HC-STVG v2 dataset and we just shot temporal results, since we are focusing only on temporal decoder module. We can observe that best results have been obtained when the threshold is greater than 0.35. Table 7: Performance on HCSTVG-v2 considering our zero-shot methods with different threshold. Light blue indicates configuration chosen for the main paper. threshold τ m_tIoU tIoU@3 tIoU@5 0.65 45.44 71.55 44.55 0.60 45.56 71.55 44.55 0.55 45.57 70.9 46 0.5 45.19 70.15 46.2 0.45 44.81 69.65 45.75 0.4 44.76 69.5 44.7 0.35 44.18 68.5 44.7 0.3 43.8 67.95 44.45 0.25 43.59 67.2 43.95 0.2 42.9 65.85 43.6 0.15 42.6 65.4 43.6 0.10 42.24 64.6 43.6 Appendix D Estimation of state-of-the-art parameters In tab. 8 we show our parameters estimation for some SOTA methods. We highlight the fact that this estimation can be not exact, but it is has still a good level of precision useful to capture the order of magnitude of each method. Table 8: Parameter analysis of state-of-the-art method for zero-shot and weak-shot STVG. Method Component Model used Parameters State Unleash [50] Vision Encoder CLIP-ViT-L/14 ∼ 300M Frozen Language Model LLaVA-v1.5-7B ∼ 6.7B Frozen Projection Layer MLP Adapter ∼ 20M Frozen TOTAL - ∼ 7B Frozen E3M [2] Vision Encoder CLIP-ViT-L/14 ∼ 300M Frozen Language Model Video-LLaVA-7B ∼ 7B Frozen Projection Layer MLP Adapter ∼ 20M Frozen Object refiner Grounding DINO ∼ 172M Frozen TOTAL - ∼ 7.5B Frozen RedCircle [31] Vision Encoder CLIP (Vit-B/16 orViT-L/14) ∼ 86-300M Frozen Text encoder CLIP ∼ 63-123M Frozen TOTAL - ∼ 149-423M Frozen ReCLIP [33] Vision Encoder CLIP (Vit-B/16 orViT-L/14) ∼ 86-300M Frozen Text encoder CLIP ∼ 63-123M Frozen TOTAL - ∼ 149-423M Frozen WINNER [20] Vision Encoder Video-Swin / SlowFast ∼ 30-120M Frozen Text encoder RoBERTa / BERT ∼ 110-125M Frozen Hierarchical Decomposition Action-Object decoder ∼ 15M Trainable Alignment Module Spatio-Temporal Head ∼ 5-10M Trainable Refinement Module BBox Regression Head ∼ 2-5M Trainable TOTAL - ∼ 162-275M Frozen CoSPal [17] Vision Encoder Video-Swin / C3D ∼ 30-120M Frozen Text encoder BERT ∼ 110M Frozen Contextual Scoring Self-placed Head ∼ 10-15M Trainable Interaction Module Cross-Modal Attn ∼ 8-12M Trainable BBox Predictor Regression Head ∼ 2-5M Trainable TOTAL - ∼ 160-262M Frozen STPro [7] Vision Encoder Video-Swin / I3D ∼ 30-120M Frozen Text encoder BERT / RoBERTa ∼ 110-125M Frozen Spatial Selector Multiple Instance Learning ∼ 8-15M Trainable Temporal Module Temporal decoder ∼ 12-20M Trainable BBox Predictor MLP Head ∼ 2-5M Trainable TOTAL - ∼ 162-285M Frozen ASTG [53] Spatial Reasoning Agent Qwen3-VL-Plus (MLLM) — Frozen Temporal Reasoning Agent Qwen3-VL-Plus (MLLM) — Frozen Query Parser (Controller) Qwen3-Plus (LLM) — Frozen Object Tracker SAM2 (Hiera-L) ∼ 224M Frozen TOTAL – ≥∼30B≥ 30B Frozen Bridge-STG [36] Vision Encoder Qwen3-VL ViT + patch merger ∼ 0.54B Frozen Language Model Qwen3-8B (Qwen3-VL 7B) ∼ 8.2B Frozen LoRA Adapters r=8r=8, α=32α=32 (language tower) ∼ 20M Trainable Bridging Queries STSB (M=8M=8 queries + MLP) ∼ 1M Trainable Spatial Backbone Swin-L (COCO pre-trained) ∼ 197M Trainable Spatial Enc-Decoder QGSL (Deformable DETR, 6+6 layers) ∼ 60M Trainable TOTAL – ∼ 9.0B Trainable Figure 5: Qualitative samples taken from HCSTV-v2 using our method with thresholding temporal grounding (top) and Unet temporal grounding (bottom). Figure 6: Qualitative samples taken from HCSTV-v2 using our method with thresholding temporal grounding (top) and Unet temporal grounding (bottom).