Paper deep dive
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
Yu Chen, Xiaohong Li, Xiaole Wang, Jianjin Zhang, Jun Sun, Yafeng Deng
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In video understanding, vision-language models (VLMs) must ingest massive numbers of visual tokens, causing the computational and memory cost of the prefill stage to rise sharply. Such visual sequences are highly redundant along the spatio-temporal dimension, yet a high compression ratio is often accompanied by the loss of critical details. Existing token-compression methods either employ heuristic, training-free compression with limited content adaptivity or introduce additional modules that require expensive alignment training, leaving the trade-off between efficiency and adaptivity unresolved. To alleviate this limitation, we propose CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens. CRAFT recursively merges tokens by decoupling parameter-free token selection from learnable token fusion: global similarity determines which tokens to merge, while a position-aware weighting module and a content-adaptive channel-wise gate learn how to fuse them. The whole compression pipeline is query-agnostic. Because every retained token is a linear combination of the original tokens, CRAFT preserves their true spatio-temporal coordinates and stays aligned with the pre-trained language model's input distribution. Experiments on multiple representative video benchmarks show that CRAFT consistently outperforms prior state-of-the-art token-compression methods. At about $8\times$ compression, it retains roughly $97\%$ of the backbone's average accuracy and shows significant efficiency improvement.
Tags
Links
- Source: https://arxiv.org/abs/2608.01644v1
- Canonical: https://arxiv.org/abs/2608.01644v1
Trouble viewing inline? Open PDF directly →
Full Text
60,524 characters extracted from source content.
Expand or collapse full text
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models Yu Chen1,2 , Xiaohong Li1,2 , Xiaole Wang1,2,3 , Jianjin Zhang1,2, Jun Sun3, Yafeng Deng1,2 Abstract In video understanding, vision-language models (VLMs) must ingest massive numbers of visual tokens, causing the computational and memory cost of the prefill stage to rise sharply. Such visual sequences are highly redundant along the spatio-temporal dimension, yet a high compression ratio is often accompanied by the loss of critical details. Existing token-compression methods either employ heuristic, training-free compression with limited content adaptivity or introduce additional modules that require expensive alignment training, leaving the trade-off between efficiency and adaptivity unresolved. To alleviate this limitation, we propose CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens. CRAFT recursively merges tokens by decoupling parameter-free token selection from learnable token fusion: global similarity determines which tokens to merge, while a position-aware weighting module and a content-adaptive channel-wise gate learn how to fuse them. The whole compression pipeline is query-agnostic. Because every retained token is a linear combination of the original tokens, CRAFT preserves their true spatio-temporal coordinates and stays aligned with the pre-trained language model’s input distribution. Experiments on multiple representative video benchmarks show that CRAFT consistently outperforms prior state-of-the-art token-compression methods. At about 8×8× compression, it retains roughly 97%97\% of the backbone’s average accuracy and shows significant efficiency improvement. Figure 1: Left: six-benchmark accuracy comparison at a fixed 8×8× compression ratio, normalized per benchmark to the best score among the six methods shown (outer vertex == best). Right: benchmark-averaged accuracy; the dashed line marks the uncompressed backbone (67.467.4). 1 Introduction Video understanding has become an important frontier direction for multimodal large language models (MLLMs) (Bai et al. 2025; Chen et al. 2025; Shu et al. 2025). However, as visual tokens grow with the product of video frame count and spatial resolution, reaching tens of thousands of tokens for a single video, the prefill self-attention cost, which grows quadratically, and the decoding-stage KV-cache memory footprint, which accumulates linearly, become serious bottlenecks (Shao et al. 2026; Kong et al. 2025). Therefore, substantially shrinking the scale of visual tokens while preserving semantics is a straightforward path to improving video reasoning efficiency. Masked autoencoders (He et al. 2022; Tong et al. 2022) and attention analysis (Chen et al. 2024) confirm that visual sequences contain highly redundant information, and that this redundancy is mainly distributed across frames. However, eliminating this redundancy must take into account the structural characteristics of video, so merging only within local temporal windows or within a single frame cannot eliminate the bulk of the redundancy, indicating the necessity of global-level merging. Existing video token compression research has evolved along two routes: the first is training-free heuristic methods, which perform unconditional reduction via hard dropping (Chen et al. 2024; Xing et al. 2025) or equal-weight averaging (Bolya et al. 2023). Such methods incur no extra training cost, but their combination weights are fixed, lack the ability to adapt to different scene content, and tend to impair the model’s understanding of complex dynamic features. The second is learnable compression methods, which introduce additional parametric modules to re-encode the visual input (Li et al. 2023; Alayrac et al. 2022). Such methods have strong fitting capacity, but often face heavy alignment training, and the reconstructed representation deviates from the original distribution the multimodal model was pre-trained on. In addition, a few query-conditioned compression attempts (Huang et al. 2025; Li et al. 2025b) can improve accuracy under a single question, but their compressed representation is tied to a specific query, preventing the KV cache from being reused across multi-turn dialogue. In summary, how to endow the compression operator with content adaptivity while maintaining low training overhead and general-purpose reusability remains a challenge that this field urgently needs to solve. To alleviate the above dilemma, we propose our solution CRAFT (Compression via Recursive Adaptive Fusion of Video Tokens), which balances adaptive aggregation and distribution preservation while keeping a low level of computational complexity. Specifically, CRAFT adopts a multi-round iterative “select-then-fuse” mechanism, decoupling the compression process within each round into two parts: which tokens to merge, which is dynamically decided by training-free global feature similarity. It introduces a process that is parameter-free and query-agnostic, ensuring the cross-round robustness of the compressed representation; and how to combine, which is adaptively determined by an end-to-end learnable gated merging mechanism, including a self-attention global weight for all selected tokens and a per-channel local gate for each token individually. Our contributions are as follows: (1) To the best of our knowledge, this is the first time an end-to-end iterative trainable “select-then-fuse” approach has been introduced for general-purpose visual token compression. The combination of parameter-free selection and parametric aggregation ensures the method’s adaptability, robustness, and low computational complexity. (2) We design a dual-path trainable aggregation mechanism. For a specific selected cluster, this mechanism simultaneously incorporates adaptive capabilities across both token and channel dimensions through a combination of coarse-grained global attention weighting and fine-grained local gating control. (3) Systematic experiments on six representative video-understanding benchmarks show that CRAFT achieves state-of-the-art performance among the compared methods across compression ratios from 2×2× to 32×32×, while delivering the lowest prefill latency and the largest FLOPs reduction among all compared methods. 2 Related Work 2.1 Training-Free Token Reduction Training-free compression methods shrink the visual token sequence with zero added parameters and zero training cost, offering strong plug-and-play deployability. By how they treat the information of removed tokens, they split into two routes: selective pruning and merging-based aggregation. Selective pruning focuses on hard-dropping redundant tokens. In terms of pruning schedule, FastV (Chen et al. 2024) performs one-shot pruning using attention scores at a shallow layer of the model; PyramidDrop (Xing et al. 2025) instead improves this into staged, progressive dropping along network depth. In terms of the ranking criterion, DivPrune (Alvar et al. 2025) introduces max–min diversity to improve the spatial coverage of retained tokens; VidCom2 (Liu et al. 2025b) further extends the uniqueness measure to the temporal dimension, achieving dual screening by both frame-level and intra-frame uniqueness. Merging-based aggregation mitigates information loss by merging similar tokens. The foundational ToMe (Bolya et al. 2023) uses bipartite matching to merge the most similar token pairs in linear time. For video tasks, subsequent work organizes merging within different spatio-temporal scopes: DyCoke (Tao et al. 2025) and FastVID (Shen et al. 2025a) focus on dynamic merging along the temporal dimension; DynTok (Zhang et al. 2025a) and LLaVA-Scissor (Sun et al. 2025) perform adaptive regional aggregation based on information density and semantic connected components, respectively; STTM (Hyun et al. 2025), HoliTom (Shao et al. 2025), and VisionZip (Yang et al. 2025b) extend the merging scope to the full spatio-temporal domain, coordinating merging inside and outside the language model with multi-granularity or redundancy-aware mechanisms. Although the above work keeps refining the scope of selection, the information of dropped tokens is always lost outright, and fusion weights are typically fixed as static rules at design time, limiting content-adaptive trade-offs, and risking irreversible semantic loss at high compression ratios. In contrast, while retaining the low-overhead advantage of training-free similarity-based selection, CRAFT builds a multi-round iterative “select-then-fuse” mechanism that shifts the core toward learnable, channel-wise gated fusion weights. 2.2 Learnable Compression and Conditioned Methods Learnable compression methods raise the ceiling of visual representation by introducing trainable parameters (Rao et al. 2021; Liang et al. 2022). Early connectors such as Q-Former (Li et al. 2023), the Perceiver Resampler (Alayrac et al. 2022), and TokenPacker (Li et al. 2025a) project arbitrary-length visual sequences into a fixed number of latent tokens. Later designs such as PVC (Yang et al. 2025a), LaCo (Liu et al. 2025a), and InternVL-X (Lu et al. 2025) embed the compressor deep inside the vision encoder or the shallow language-model layers. These designs deviate from the original pre-training input distribution, requiring costly re-alignment training. Beyond such heavyweight re-encoders, recent learnable compression is predominantly query-conditioned, and lightweight query-agnostic learnable compressors have received far less attention: even when token selection is learned with lightweight differentiable modules, as in VisionSelector (Zhu et al. 2025), fusion still follows a static, equal-weight rule. Query-conditioned methods instead treat the question as a prior, filtering the most task-relevant visual tokens with text-guided attention (Zhang et al. 2025b; Huang et al. 2025; Li et al. 2025b; Shen et al. 2025b) or retrieving question-relevant clips via reinforcement learning (Wu et al. 2026). Such compression works well within a single question-answering instance, but the representation is tightly bound to that query, so the compressed KV cache cannot be reused across questions or multi-turn dialogue. Unlike the methods above, CRAFT restricts the learning target to linear combinations of the original tokens, using iterative selection and fusion to keep the compressed representation within the original pre-training space, avoiding costly re-alignment overhead. Furthermore, CRAFT adopts a query-agnostic compression scheme, allowing the compressed visual representation to be computed once and reused for general purposes, making it effective for multi-turn inference. Figure 2: CRAFT performs Training-Free Global Selection and Learnable Gated Merging iteratively. Note that in Training-Free Global Selection, previously merged tokens are shown as white rectangles. For clarity, only one representative token xtx_t and its associated token set xit\x_i\_t are illustrated. 3 Method 3.1 Problem Definition and Overview As shown in Fig. 2, given a video input, the vision encoder and projector project it into the language model’s embedding space, producing a visual feature sequence X∈ℝNf×DX ^N_f× D. In video scenarios, NfN_f is typically on the order of 10410^4. CRAFT acts on the feature sequence X, iteratively identifying pairs of similar tokens and adaptively merging them, ultimately producing a compressed sequence XC∈ℝNc×DX_C ^N_c× D. The iterative process supports two budget modes: 1) an adaptive similarity-threshold mode, which stops once the maximum pairwise similarity among retained tokens falls below a threshold sθs_θ, so the realized ratio ρ=Nf/Ncρ=N_f/N_c adapts to the video content, with a budget floor on the retained token count, Nmin=max(1,⌊Nf/128⌋)N_ = (1, N_f/128 ) in our experiments, preventing over-merging; 2) a fixed-ratio mode, which continues until ρ reaches a target ρθρ_θ (Algorithm 1 details the former; the latter is in Appendix A.1). Notably, CRAFT performs iterative token merging without conditioning on textual queries. As a result, a video needs to be compressed only once, and the resulting compact visual sequence XCX_C can be directly cached as visual key-value representations and reused across multiple dialogue turns and diverse user queries: p(∣XC,)=∏ℓ=1Lp(yℓ∣y<ℓ,XC,)p(y X_C,q)= _ =1^Lp(y_ y_< ,X_C,q) (1) 3.2 Training-Free Global Selection Since redundancy in videos often spans across frames, intra-frame or local sliding-window merging strategies cannot fundamentally eliminate the majority of redundant information. Therefore, token selection must be performed at a global scale. To maintain extremely low computational overhead on high-dimensional sequences at the Nf∼104N_f 10^4 level, we exploit the inner-product/cosine-similarity-preserving property of random projections (Johnson and Lindenstrauss 1984; Achlioptas 2003), computing similarity in a fixed low-dimensional random projection space: Esim=XWrand¯,Wrand∈ℝD×dsim.E_sim= XW_rand, W_rand ^D× d_sim. (2) Here ⋅¯ · denotes ℓ2 _2 normalization, and dsimd_sim is typically set to 128. The projection matrix WrandW_rand is generated once with a fixed random seed and column-normalized. The similarity score between any two tokens i,ji,j is given directly by the inner product of their projected unit vectors: sij=Esim,iEsim,j⊤≈cos(xi,xj)s_ij=E_sim,iE_sim,j ≈ (x_i,x_j) (3) Similarity computation is carried out via blockwise matrix multiplication with block size 20482048 to prevent extreme GPU memory peaks. Each iteration consists of four stages: pairing, budget truncation, orientation, and chain prevention (Algorithm 1). In the pairing stage, each active token nominates its most similar token, p(i)=argmaxj≠isijp(i)= _j≠ is_ij, yielding one candidate edge (i,p(i))(i,p(i)) per token with similarity si⋆=si,p(i)s_i =s_i,p(i). The budget-truncation stage then merges only the k=min(|active|−Nmin,⌊|active|/2⌋)k= (|active|-N_ , |active|/2 ) highest-similarity edges in the current round (edges with si⋆<sθs_i <s_θ are further excluded in the adaptive threshold mode), keeping the retained count above the budget floor NminN_ . Next, during the orientation stage, the token with the larger ℓ2 _2-norm in each selected edge is retained as the representative token, denoted by xtx_t, based on the assumption that a larger norm indicates more salient semantic information. As a result, multiple tokens may be associated with the same representative token xtx_t, forming a group xit\x_i\_t. However, some representative tokens may themselves be selected as the source of another edge whose partner has an even larger ℓ2 _2-norm. To avoid such chain-style aggregation, these cases are resolved in the chain-prevention stage: rather than being discarded, a conflicting representative token simply drops its outgoing edge, is excluded from merging in the current iteration, and remains active, re-entering the pairing stage in the next iteration. The surviving representative token preserves its original spatio-temporal coordinates (t,h,w)(t,h,w), thereby maintaining positional information throughout the merging process. Algorithm 1 CRAFT Select-then-Fuse (adaptive mode) 0: Features X, coords (t,h,w)\(t,h,w)\, threshold sθs_θ, budget floor NminN_ 0: Compressed sequence XCX_C, updated coords 1: active←1,…,Nfactive←\1,…,N_f\; m←0m← 0; Esim←XWrand¯E_sim← XW_rand ⊳ Eq. 2 2: while |active|>Nmin|active|>N_ do 3: m←m+1m← m+1 4: p(i)←argmaxj≠isijp(i)← _j≠ is_ij; si⋆←si,p(i)s_i ← s_i,p(i), ∀i∈active∀ i ⊳ Pairing (Eq. 3) 5: C←(i,p(i)):si⋆≥sθC←\(i,p(i)):s_i ≥ s_θ\; if C=∅C= then break ⊳ Adaptive stop 6: k←min(|active|−Nmin,⌊|active|/2⌋,|C|)k← \! (|active|-N_ ,\, |active|/2 ,\,|C| ) 7: P←P← the k edges of C with highest si⋆s_i ⊳ Budget truncation 8: Orient edges (ℓ2 _2-larger endpoint → target), drop chain-conflicting edges, group into (t,it)(t,\i\_t) ⊳ §3.2 9: for all merge group (t,it)(t,\i\_t) do 10: xtnew←GatedMerge(xt,xit)x_t^new (x_t,\x_i\_t) ⊳ Eqs. 4–9 11: end for 12: X[targets]←xtnewX[targets]← x_t^new; active←active∖sourcesactive \sources\ 13: Esim[targets]←X[targets]Wrand¯E_sim[targets]← X[targets]W_rand ⊳ Refresh (Eq. 2) 14: end while 15: return XC←X[active]X_C← X[active] with updated coordinates 3.3 Learnable Gated Merging How are the selected pairs combined? For each representative xtx_t and its source set xi\x_i\, CRAFT fuses by xtnew=xt−∑iwiGi⊙(xt−xi).x_t^new=x_t- _iw_i\,G_i (x_t-x_i). (4) wiw_i is the weight among the sources, satisfying ∑iwi=1 _iw_i=1, thereby enabling the aggregation of information from a global perspective. Gi∈(0,1)DG_i∈(0,1)^D denotes the per-channel gating vector for the local perspective, and ⊙ denotes the Hadamard product. When Gi≡1G_i≡ 1, Eq. (4) reduces to xtnew=(1−∑iwi)xt+∑iwixi=∑iwixix_t^new= (1- _iw_i )x_t+ _iw_ix_i= _iw_ix_i , i.e. it degenerates into a weighted average of the source tokens; if wiw_i is further taken to be uniform, this becomes the traditional equal-weight average. When Gi≡0G_i≡ 0, ∑iwiGi⊙(xt−xi)≡0 _iw_iG_i (x_t-x_i)≡ 0, and Eq. (4) degenerates into hard-dropping the sources and keeping only the representative. This design allows the contribution of each token to the merging process to be adaptively regulated from both global and local perspectives. Global position-aware weighting (source weights wiw_i). The weighting scorer first models spatio-temporal relations through low-rank projection and 3D positional rotation: Q^=RoPE3d(XWq¯),K^=RoPE3d(XWk¯). Q=RoPE_3d( XW_q), K=RoPE_3d( XW_k). (5) Here Wq,Wk∈ℝD×rW_q,W_k ^D× r, r=128r=128, with the RoPE3dRoPE_3d following M-RoPE (Wang et al. 2024). The positional score is computed by the projected dot product between the query and key vectors on the selected merge pair. Then the similarity score stis_ti from selection is added to the positional score. The weights among sources are adaptively assigned via softmax normalization over the representative token’s source set: wi=exp((Q^iK^t⊤+sti)/τ)∑jexp((Q^jK^t⊤+stj)/τ).w_i= (( Q_i K_t +s_ti)/τ) _j (( Q_j K_t +s_tj)/τ). (6) Local content-conditioned gating (per-channel GiG_i). The gating network produces GiG_i of the same size as xtx_t, enabling fine-grained control over the contribution of each channel during token merging. It takes the concatenation of the representative feature and the source feature as input, and, via adaptive layer normalization (adaLN) incorporating the merge round m, outputs a multi-channel adaptive gating vector: hi=GELU(Wdown[xt;xi])∈ℝdh,h_i=GELU (W_down\,[\,x_t;\,x_i\,] ) ^d_h, (7) h~i=(1+γi)⊙LN(hi)+βi,(γi,βi)=Wmodϕ(m)+bmod, h_i=(1+ _i) (h_i)+ _i, ( _i, _i)=W_mod\,φ(m)+b_mod, (8) Gi=σ(Wuph~i)∈(0,1)D.G_i=σ (W_up\, h_i )∈(0,1)^D. (9) Here dhd_h is typically 512, σ denotes the Sigmoid activation function, and LNLN denotes layer normalization without affine parameters. ϕ(m)φ(m) denotes the sine-cosine positional encoding of round m, which provides the network with awareness of the current merging stage. Notably, the linear projection WupW_up, as well as the modulation projections Wmod,bmodW_mod,b_mod, adopt zero initialization (adaLN-Zero (Peebles and Xie 2023)), so that the network’s initial output is γi=βi=0 _i= _i=0 and Gi=0.5G_i=0.5. Overall procedure. The training-free selection of §3.2 and the learnable fusion above are not two sequential stages but two sub-steps within the same iterative loop, in which each round first selects the pairs to be merged, then performs the weighted fusion, and refreshes the similarities before entering the next round based on the merged tokens, until the compression budget is reached. Algorithm 1 gives the complete pseudocode of this iterative “select-then-fuse” loop (adaptive threshold mode; the fixed-ratio variant is in Appendix A.1). 3.4 Two-Stage Training Strategy We freeze the vision encoder, multimodal projector, and large language model, and optimize only the weighting scorer and gating network under the standard cross-entropy objective, so training stays lightweight and the backbone’s input distribution remains untouched. Training follows a two-stage curriculum. The first stage pre-trains the gate on multi-source video-captioning data, encouraging it to retain globally relevant semantic information during compression. The second stage continues on video question-answering (QA) data, aligning the compressor with downstream reasoning objectives and steering it toward fine-grained visual details. The data composition of both stages is detailed in Appendix B.3. 4 Experiments Backbone Method Comp. Video-MME LongVideoBench MVBench EgoSchema MLVU TempCompass Avg. A. Main Comparison Qwen3.5-4B Base 1×1× 67.0 (100%) 58.4 (100%) 68.4 (100%) 67.0 (100%) 69.3 (100%) 74.3 (100%) 67.4 (100%) VisionZip 8×8× 61.5 (91.8%) 52.1 (89.2%) 59.1 (86.4%) 63.4 (94.6%) 63.3 (91.3%) 67.9 (91.3%) 61.2 (90.8%) FastVID 8×8× 60.2 (89.9%) 54.2 (92.8%) 59.9 (87.5%) 60.0 (89.6%) 64.8 (93.4%) 69.4 (93.4%) 61.4 (91.1%) DivPrune 8×8× 63.7 (95.1%) 54.2 (92.8%) 61.2 (89.5%) 63.0 (94.0%) 66.2 (95.4%) 68.2 (91.7%) 62.7 (93.1%) VidCom2 8×8× 62.5 (93.3%) 56.0 (95.8%) 59.9 (87.5%) 63.8 (95.2%) 64.3 (92.8%) 65.5 (88.2%) 62.0 (92.0%) LLaVA-Scissor 8×8× 62.3 (93.0%) 53.2 (91.0%) 62.8 (91.7%) 63.4 (94.6%) 64.9 (93.6%) 69.3 (93.3%) 62.6 (92.9%) CRAFT (Ours) × 8× 64.6 (96.4%) 58.2 (99.6%) 63.0 (92.1%) 65.8 (98.2%) 70.0 (101.0%) 70.0 (94.2%) 65.3 (96.8%) CRAFT+ (Ours)‡ 10×10× 64.0 (95.5%) 59.5 (101.8%) 66.2 (96.8%) 70.8 (105.7%) 69.0 (99.5%) 71.8 (96.6%) 66.9 (99.2%) B. Cross-Backbone Transfer Qwen2.5-VL-7B Base 1×1× 64.1 (100%) 60.3 (100%) 67.8 (100%) 64.6 (100%) 67.4 (100%) 72.6 (100%) 66.1 (100%) TimeChat-Online 7.3×7.3× 59.9 (93.4%) 54.4 (90.2%) 63.9 (94.2%) 50.0 (77.4%) 62.3 (92.4%) 68.3 (94.1%) 59.8 (90.5%) VisionSelector 8×8× 60.0 (93.6%) 54.2 (89.9%) 62.9 (92.8%) 65.8 (101.9%) 59.6 (88.4%) 65.4 (90.1%) 61.3 (92.7%) CRAFT (Ours) × 8× 59.9 (93.4%) 57.0 (94.5%) 63.8 (94.1%) 61.8 (95.7%) 66.2 (98.2%) 69.3 (95.5%) 63.0 (95.3%) LLaVA-OV-7B Base 1×1× 59.0 (100%) 57.0 (100%) 56.9 (100%) 64.4 (100%) 67.2 (100%) 63.7 (100%) 61.4 (100%) CRAFT (Ours) × 8× 55.5 (94.1%) 52.4 (91.9%) 56.0 (98.4%) 63.4 (98.4%) 63.9 (95.1%) 62.9 (98.7%) 59.0 (96.1%) Table 1: Main results on six video benchmarks. Below each accuracy is its retention relative to the corresponding uncompressed backbone. The “Comp.” column is the average realized token compression ratio across benchmarks. Block A compares CRAFT under adaptive mode with training-free token-reduction methods on Qwen3.5-4B; Block B transfers CRAFT to Qwen2.5-VL-7B and LLaVA-OV-7B. Bold/underline mark the best/second-best compressed result per column. CRAFT is our original implementation. ‡CRAFT++ additionally adapts the LLM with LoRA and is reported in gray for reference only. Full accuracy–compression trade-off from 2×2× to 32×32× on Qwen3.5-4B is given in Fig. 3. 4.1 Experimental Setup Benchmarks. We evaluate on six video-understanding benchmarks that span short clips to long-form temporal reasoning, including Video-MME (Fu et al. 2025) (w/o subtitles), LongVideoBench (Wu et al. 2024), MVBench (Li et al. 2024b), EgoSchema (Mangalam et al. 2023), MLVU (Zhou et al. 2025), and TempCompass (Liu et al. 2024). Unless otherwise noted, all numbers are top-1 accuracy under greedy decoding, sampling frames at fps=2fps=2 up to a maximum of 6464 frames, on the full test/validation splits; benchmark-specific evaluation details for MVBench and TempCompass are given in Appendix B.1. Metrics. Besides raw accuracy we report the retention acc/accbaseacc/acc_base relative to the corresponding uncompressed backbone, shown as the small parenthetical below each accuracy in Table 1, and the realized token-compression ratio ρ=Nf/Ncρ=N_f/N_c (computed per video and averaged over each dataset), shown as the “Comp.” column. Table 1, the efficiency analysis of §4.4, and Fig. 4 evaluate CRAFT under the adaptive similarity-threshold mode, whereas the remaining experiments (Fig. 1 and Table 3) use the fixed-ratio mode to keep the token budget identical across configurations. Backbones. Qwen3.5-4B (Qwen Team 2026) is our primary backbone. To demonstrate that CRAFT is architecture-agnostic, we transfer it to Qwen2.5-VL-7B (Bai et al. 2025) and LLaVA-OV-7B (Li et al. 2024a), the two most widely adopted architectures in token-compression studies, which differ in vision encoder and token layout. Baselines. On Qwen3.5-4B we compare against five representative training-free token-reduction methods covering both the pruning and the merging routes, DivPrune (Alvar et al. 2025), VidCom2 (Liu et al. 2025b), LLaVA-Scissor (Sun et al. 2025), VisionZip (Yang et al. 2025b), and FastVID (Shen et al. 2025a), all transferred plug-and-play onto our backbone so that every baseline is compared on an identical footing. On Qwen2.5-VL-7B we additionally compare against two learnable compressors, TimeChat-Online (Yao et al. 2025) and VisionSelector (Zhu et al. 2025). Because VisionSelector relies on a trainable compressor module and TimeChat-Online fine-tunes its backbone on dedicated data, porting them to another backbone would introduce confounds, so both are kept on their native Qwen2.5-VL-7B. All methods are run under the same frame budget for fairness. Implementation. We build CRAFT, and its LoRA-adapted variant CRAFT++, on Qwen3.5-4B and transfer the same compressor design unchanged to Qwen2.5-VL-7B and LLaVA-OV-7B. The training curriculum and hyperparameters are detailed in Appendix B.2. 4.2 Main Results Comparison with training-free methods. Table 1 (block A) reports the head-to-head comparison of CRAFT against a range of state-of-the-art training-free methods on Qwen3.5-4B, and the results confirm the advantage of a learnable gated fusion over the fixed, heuristic rules that every training-free baseline relies on. At ∼8× 8× compression CRAFT retains 96.8%96.8\% of the backbone’s average accuracy (65.365.3 vs. 67.467.4), outperforming every training-free baseline by a clear margin. The strongest competitor, DivPrune, reaches only 93.1%93.1\%, while VisionZip and FastVID fall to ∼91% 91\%. Notably, CRAFT even surpasses the uncompressed backbone on MLVU, reaching 101.0%101.0\% of the original accuracy. Adaptive thresholding grants CRAFT no extra budget: the dataset-averaged realized ratio is 8.0×8.0×, matching the baselines, and tokens are merely reallocated from redundant clips to information-dense ones—itself an advantage of content-aware compression. Even at a strictly fixed 8×8× budget, CRAFT averages 64.864.8 (Table 3, last row), still clearly ahead of the best training-free baseline (62.762.7). The superiority of CRAFT extends beyond any single benchmark. Under a fixed 8×8× compression budget, it also achieves the best or equal-level accuracy across all six benchmarks in Fig. 1. Accuracy–compression trade-off. While a single operating point may favor a particular method, the rate of accuracy degradation under tighter token budgets is more telling. Fig. 3 sweeps the compression ratio from 2×2× to 32×32× on Qwen3.5-4B. Up to 4×4× every method stays close to the uncompressed base of 67.41%67.41\%, but beyond ∼8× 8× the training-free baselines decline sharply toward 53.553.5–58%58\% at 32×32×, whereas CRAFT degrades gracefully, widens its margin as the ratio grows, and still reaches nearly 59%59\%. The steadily growing margin shows that learnable gated fusion, rather than any static merging rule, is what preserves critical information when the token budget becomes severely constrained. Figure 3: Accuracy–compression trade-off on Qwen3.5-4B: benchmark-averaged top-1 accuracy vs. realized compression ratio (2×2× to 32×32×). 4.3 Cross-Backbone Transfer Without any change to the compressor design, CRAFT transfers to two further backbones (Table 1, block B; porting details in Appendix A.2). On Qwen2.5-VL-7B, it retains 95.3%95.3\% of the full-model performance (63.063.0) at an 8×8× compression ratio, outperforming compressors trained on the same backbone, including VisionSelector (92.7%92.7\%) and TimeChat-Online (90.5%90.5\%), while requiring the training of only a lightweight ∼5 5M-parameter module. On LLaVA-OV-7B, CRAFT retains 96.1%96.1\% of the full-model performance. That the same design generalizes across three architectures with distinct vision encoders and token layouts indicates that its effectiveness arises from the decoupling of training-free selection and learnable, position-aware fusion, rather than architecture-specific tuning. 4.4 Efficiency Analysis Token compression aims to cut the cost of the prefill stage, where visual tokens dominate the input at Nf∼104N_f 10^4. CRAFT itself adds little overhead. Pair selection is linear in the surviving tokens per round, and similarity is computed in a fixed dsim=128d_sim=128 random-projection space with blockwise products, so the full Nf×NfN_f× N_f matrix is never materialized. Its two learnable modules hold only ∼5 5M parameters, negligible beside a 44B-parameter LLM, so the select-then-fuse step costs far less than it saves downstream. Method Acc.(%) Prefill (ms)↓ KV (MB)↓ FLOPs↑ Base 68.4 249.2 162.5 1×1× DivPrune 61.2 72.3 (↑ 11.6%) 26.0 6.75×6.75× VidCom2 59.9 72.6 (↑ 12.0%) 26.0 6.75×6.75× VisionZip 59.1 72.5 (↑ 11.9%) 26.0 6.75×6.75× FastVID 59.9 72.8 (↑ 12.3%) 26.0 6.75×6.75× LLaVA-Scissor 62.8 70.5 (↑ 8.8%) 26.2 6.71×6.71× CRAFT 63.2 64.8 19.0 9.30×9.30× Table 2: Prefill-stage efficiency and analytical FLOPs reduction on MVBench, at the operating point where CRAFT’s accuracy first matches the strongest merging-based baseline, LLaVA-Scissor. Table 2 reports this at the point where CRAFT first matches LLaVA-Scissor, the strongest merging-based baseline on this benchmark. CRAFT reaches 63.2%63.2\%, surpassing every baseline while leading on all three efficiency axes. Time. CRAFT reaches the lowest prefill latency of any method compared, 8.88.8–12.3%12.3\% faster than every training-free baseline, whose own mutual spread is only 3.4%3.4\%. Space. The per-sample KV cache shrinks to 19.019.0MB, 2727–28%28\% smaller than every training-free baseline’s 26.026.0–26.226.2MB—the budget every additional dialogue turn reuses without recomputation. FLOPs. The analytical reduction of FLOPs, computed purely from the surviving token count and architecture, reaches 9.30×9.30× for CRAFT versus 6.716.71–6.75×6.75× for training-free baselines—the largest computational saving of any method compared. Weight Reduce Stage Avg. Δ mean mean 1 61.81 – mean target 1 62.81 +1.00+1.00 mean gate 1 63.62 +1.81+1.81 pos-aware gate 1 63.92 +2.11+2.11 pos-aware gate 2 64.75 +2.94+2.94 Table 3: Progressive ablation of CRAFT on Qwen3.5-4B at a fixed 8×8× ratio, 66-benchmark average. Weight is how the merged neighbors are weighted across tokens (mean = uniform; pos-aware = the global position-aware weighting of §3.3); Reduce is how the weighted neighbors are fused into the surviving token; Stage: 1 = caption pretraining; 2 = + stage-2 QA. 4.5 Ablation Experiment Table 3 enables one design choice at a time on Qwen3.5-4B, and the six-benchmark average rises monotonically at every step. Notably, the two fixed reduction rules are exactly the two degenerate extremes of the gate in Eq. (4). The mean rule corresponds to Gi≡1G_i≡ 1, i.e., equal-weight averaging of the group; the target rule corresponds to Gi≡0G_i≡ 0, i.e., pruning. Switching from the former extreme to the latter lifts the average by 1.001.00, validating our assumption that tokens with larger ℓ2 _2 norm carry more salient semantics. Replacing the two fixed endpoints with the learned gate adds a further 0.810.81. The global position-aware weighting of §3.3 contributes another 0.300.30 by reweighting neighbor tokens with content and relative t,h,wt,h,w position. The stage-2 QA curriculum adds a final 0.830.83, yielding the full compressor at 64.7564.75. The two learnable components act at different granularities: the gate locally per channel, and the weighting scorer globally across tokens. Together they recover most of the accuracy that fixed-rule merging discards. 4.6 Evaluation of Scaffold and Learnable Merging The compressor of CRAFT involves two separable design choices. The training-free selection scaffold of §3.2 decides by global feature similarity which tokens are grouped in each round, while the learnable gated merging of §3.3 decides how each group is fused. To attribute accuracy to each part and to test whether the merging module depends on the scaffold it is paired with, we compare four configurations at a shared ∼8× 8× budget on six benchmarks in Fig. 4. Besides the full CRAFT and a selection-only CRAFT whose learnable merging is replaced by equal-weight averaging, we evaluate LLaVA-Scissor and a Scissor-Merge variant that feeds our merging module with LLaVA-Scissor’s selection module. Scissor-Merge is trained under exactly the same data and hyperparameters as CRAFT, so the two differ only in which scaffold feeds the shared merging module. Figure 4: Selection scaffold vs. learnable merging, at ∼8× 8×. Scissor-Merge couples LLaVA-Scissor’s scaffold with our learnable merging module. Scissor is the original LLaVA-Scissor, and CRAFT (selection only) is our scaffold with the learnable merging replaced by equal-weight averaging. All methods run in their threshold-driven adaptive mode, and all trained variants are stage-1-only checkpoints. The results support three observations. The scaffold alone is already competitive, as the selection-only variant reaches an average of 62.462.4, on par with original LLaVA-Scissor (62.562.5). The learnable merging likewise generalizes to a different scaffold, since grafting it onto LLaVA-Scissor’s selection lifts the average from 62.562.5 to 63.463.4, showing that learnable, content-adaptive fusion also improves an established scaffold with a fixed fusion rule. Finally, the two act as complements rather than substitutes: the same merging module that adds 0.90.9 on LLaVA-Scissor’s scaffold adds 2.02.0 on ours (62.4→64.462.4→ 64.4), and pairing it with our own scaffold reaches the best of the four configurations at 64.464.4 on average. 5 Conclusion In this paper, we present CRAFT, a query-agnostic visual token compressor for video vision-language models that recursively merges tokens through a decoupled select-then-fuse mechanism, where training-free global similarity decides which tokens to merge and a learnable position-aware weighting with a content-adaptive per-channel gate decides how to fuse them. Extensive experiments show that CRAFT consistently outperforms state-of-the-art compressors and transfers unchanged across three backbones, retaining about 97%97\% of the backbone’s average accuracy at roughly 8×8× compression. CRAFT offers a practical path toward efficient long-video understanding beyond hand-written rules and heavy re-encoding. References D. Achlioptas (2003) Database-friendly random projections: Johnson-Lindenstrauss with binary coins. Journal of Computer and System Sciences 66 (4), p. 671–687. Cited by: §3.2. J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan (2022) Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2204.14198, Link Cited by: §1, §2.2. S. R. Alvar, G. Singh, M. Akbari, and Y. Zhang (2025) DivPrune: diversity-based visual token pruning for large multimodal models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: 2503.02175, Link Cited by: §2.1, §4.1. S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. External Links: 2502.13923, Link Cited by: §1, §4.1. D. Bolya, C. Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman (2023) Token merging: your vit but faster. In International Conference on Learning Representations (ICLR), External Links: 2210.09461, Link Cited by: §1, §2.1. L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang (2024) An image is worth 1/2 tokens after layer 2: plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision (ECCV), External Links: 2403.06764, Link Cited by: §1, §1, §2.1. Y. Chen, F. Xue, D. Li, Q. Hu, L. Zhu, X. Li, Y. Fang, H. Tang, S. Yang, Z. Liu, E. He, H. Yin, P. Molchanov, J. Kautz, L. Fan, Y. Zhu, Y. Lu, and S. Han (2025) LongVILA: scaling long-context visual language models for long videos. In International Conference on Learning Representations (ICLR), External Links: 2408.10188, Link Cited by: §1. C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, P. Chen, Y. Li, S. Lin, S. Zhao, K. Li, T. Xu, X. Zheng, E. Chen, C. Shan, R. He, and X. Sun (2025) Video-mme: the first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: 2405.21075, Link Cited by: §4.1. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick (2022) Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: 2111.06377, Link Cited by: §1. X. Huang, H. Zhou, and K. Han (2025) PruneVid: visual token pruning for efficient video large language models. In Findings of the Association for Computational Linguistics (ACL Findings), External Links: 2412.16117, Link Cited by: §1, §2.2. J. Hyun, S. Hwang, S. H. Han, T. Kim, I. Lee, D. Wee, J. Lee, S. J. Kim, and M. Shim (2025) Multi-granular spatio-temporal token merging for training-free acceleration of video llms. In IEEE/CVF International Conference on Computer Vision (ICCV), External Links: 2507.07990, Link Cited by: §2.1. W. B. Johnson and J. Lindenstrauss (1984) Extensions of Lipschitz mappings into a Hilbert space. Contemporary Mathematics 26, p. 189–206. Cited by: §3.2. Z. Kong, Y. Li, F. Zeng, L. Xin, S. Messica, X. Lin, P. Zhao, M. Kellis, H. Tang, and M. Zitnik (2025) Token reduction should go beyond efficiency in generative models – from vision, language to multimodality. arXiv preprint arXiv:2505.18227. External Links: 2505.18227, Link Cited by: §1. B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, and C. Li (2024a) LLaVA-onevision: easy visual task transfer. External Links: 2408.03326, Link Cited by: §4.1. J. Li, D. Li, S. Savarese, and S. Hoi (2023) BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML), External Links: 2301.12597, Link Cited by: §1, §2.2. K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, L. Wang, and Y. Qiao (2024b) MVBench: a comprehensive multi-modal video understanding benchmark. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: 2311.17005, Link Cited by: §4.1. W. Li, Y. Yuan, J. Liu, D. Tang, S. Wang, J. Qin, J. Zhu, and L. Zhang (2025a) TokenPacker: efficient visual projector for multimodal llm. International Journal of Computer Vision (IJCV). External Links: 2407.02392, Link Cited by: §2.2. Y. Li, H. Gui, Z. Fan, J. Wang, B. Kang, B. Chen, and Z. Tian (2025b) Less is more, but where? dynamic token compression via llm-guided keyframe prior. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2512.06866, Link Cited by: §1, §2.2. Y. Liang, C. Ge, Z. Tong, Y. Song, J. Wang, and P. Xie (2022) Not all patches are what you need: expediting vision transformers via token reorganizations. In International Conference on Learning Representations (ICLR), External Links: 2202.07800, Link Cited by: §2.2. J. Liu, L. Niu, W. Chen, J. Zhou, and F. Meng (2025a) LaCo: efficient layer-wise compression of visual tokens for multimodal large language models. arXiv preprint arXiv:2507.02279. External Links: 2507.02279, Link Cited by: §2.2. X. Liu, Y. Wang, J. Ma, and L. Zhang (2025b) Video compression commander: plug-and-play inference acceleration for video large language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: 2505.14454, Link Cited by: §2.1, §4.1. Y. Liu, S. Li, Y. Liu, Y. Wang, S. Ren, L. Li, S. Chen, X. Sun, and L. Hou (2024) TempCompass: do video llms really understand videos?. In Findings of the Association for Computational Linguistics (ACL Findings), External Links: 2403.00476, Link Cited by: §4.1. D. Lu, Y. Sun, Z. Zhang, L. Huang, J. Zeng, M. Shu, and H. Cao (2025) InternVL-x: advancing and accelerating internvl series with efficient visual token compression. arXiv preprint arXiv:2503.21307. External Links: 2503.21307, Link Cited by: §2.2. K. Mangalam, R. Akshulakov, and J. Malik (2023) EgoSchema: a diagnostic benchmark for very long-form video language understanding. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, External Links: 2308.09126, Link Cited by: §4.1. W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In IEEE/CVF International Conference on Computer Vision (ICCV), External Links: 2212.09748, Link Cited by: §3.3. Qwen Team (2026) Qwen3.5-omni technical report. External Links: 2604.15804, Link Cited by: §4.1. Y. Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C. Hsieh (2021) DynamicViT: efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2106.02034, Link Cited by: §2.2. K. Shao, K. Tao, C. Qin, H. You, Y. Sui, and H. Wang (2025) HoliTom: holistic token merging for fast video large language models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2505.21334, Link Cited by: §2.1. K. Shao, K. Tao, K. Zhang, S. Feng, M. Cai, Y. Shang, H. You, C. Qin, Y. Sui, and H. Wang (2026) When tokens talk too much: a survey of multimodal long-context token compression across images, videos, and audios. Transactions on Machine Learning Research (TMLR). External Links: 2507.20198, Link Cited by: §1. L. Shen, G. Gong, T. He, Y. Zhang, P. Liu, S. Zhao, and G. Ding (2025a) FastVID: dynamic density pruning for fast video large language models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2503.11187, Link Cited by: §2.1, §4.1. X. Shen, Y. Xiong, C. Zhao, L. Wu, J. Chen, C. Zhu, Z. Liu, F. Xiao, B. Varadarajan, F. Bordes, Z. Liu, H. Xu, H. J. Kim, B. Soran, R. Krishnamoorthi, M. Elhoseiny, and V. Chandra (2025b) LongVU: spatiotemporal adaptive compression for long video-language understanding. In International Conference on Machine Learning (ICML), External Links: 2410.17434, Link Cited by: §2.2. Y. Shu, Z. Liu, P. Zhang, M. Qin, J. Zhou, Z. Liang, T. Huang, and B. Zhao (2025) Video-xl: extra-long vision language model for hour-scale video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: 2409.14485, Link Cited by: §1. B. Sun, J. Zhao, X. Wei, and Q. Hou (2025) LLaVA-scissor: token compression with semantic connected components for video llms. arXiv preprint arXiv:2506.21862. External Links: 2506.21862, Link Cited by: §2.1, §4.1. K. Tao, C. Qin, H. You, Y. Sui, and H. Wang (2025) DyCoke: dynamic compression of tokens for fast video large language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: 2411.15024, Link Cited by: §2.1. Z. Tong, Y. Song, J. Wang, and L. Wang (2022) VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2203.12602, Link Cited by: §1. P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin (2024) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. External Links: 2409.12191, Link Cited by: §3.3. H. Wu, D. Li, B. Chen, and J. Li (2024) LongVideoBench: a benchmark for long-context interleaved video-language understanding. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, External Links: 2407.15754, Link Cited by: §4.1. P. Wu, Z. Yu, Y. Liu, C. Wu, E. Zhou, and J. Shen (2026) MARC: memory-augmented rl token compression for efficient video understanding. In International Conference on Learning Representations (ICLR), External Links: 2510.07915, Link Cited by: §2.2. L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang, Y. Zang, Y. Cao, C. He, J. Wang, F. Wu, and D. Lin (2025) PyramidDrop: accelerating your large vision-language models via pyramid visual redundancy reduction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: 2410.17247, Link Cited by: §1, §2.1. C. Yang, X. Dong, X. Zhu, W. Su, J. Wang, H. Tian, Z. Chen, W. Wang, L. Lu, and J. Dai (2025a) PVC: progressive visual token compression for unified image and video processing in large vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: 2412.09613, Link Cited by: §2.2. S. Yang, Y. Chen, Z. Tian, C. Wang, J. Li, B. Yu, and J. Jia (2025b) VisionZip: longer is better but not necessary in vision language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: 2412.04467, Link Cited by: §2.1, §4.1. L. Yao, Y. Li, Y. Wei, L. Li, S. Ren, Y. Liu, K. Ouyang, L. Wang, S. Li, S. Li, L. Kong, Q. Liu, Y. Zhang, and X. Sun (2025) TimeChat-online: 80% visual tokens are naturally redundant in streaming videos. In Proceedings of the 33rd ACM International Conference on Multimedia (ACM M), External Links: 2504.17343, Link Cited by: §4.1. H. Zhang, J. Zhang, X. Ji, Q. Wang, and F. Zhang (2025a) DynTok: dynamic compression of visual tokens for efficient and effective video understanding. arXiv preprint arXiv:2506.03990. External Links: 2506.03990, Link Cited by: §2.1. Y. Zhang, C. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. Gudovskiy, T. Okuno, Y. Nakata, K. Keutzer, and S. Zhang (2025b) SparseVLM: visual token sparsification for efficient vision-language model inference. In International Conference on Machine Learning (ICML), External Links: 2410.04417, Link Cited by: §2.2. J. Zhou, Y. Shu, B. Zhao, B. Wu, Z. Liang, S. Xiao, M. Qin, X. Yang, Y. Xiong, B. Zhang, T. Huang, and Z. Liu (2025) MLVU: benchmarking multi-task long video understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: 2406.04264, Link Cited by: §4.1. J. Zhu, Y. Zhu, X. Lu, W. Yan, D. Li, K. Liu, X. Fu, and Z. Zha (2025) VisionSelector: end-to-end learnable visual token compression for efficient multimodal llms. arXiv preprint arXiv:2510.16598. External Links: 2510.16598, Link Cited by: §2.2, §4.1. Appendix A Method Details A.1 Algorithm Algorithm 1 presents the adaptive similarity-threshold mode of CRAFT. Algorithm 2 below gives the counterpart for the fixed-compression-ratio mode: merging continues until the retained token count reaches the fixed target Ntgt=max(1,⌊Nf/ρθ⌋)N_tgt= (1, N_f/ _θ ), i.e., until the compression ratio ρ=Nf/Ncρ=N_f/N_c reaches ρθρ_θ. The per-round pairing, orientation, chain prevention, and gated fusion are identical to the threshold mode; the only differences are that no similarity threshold is applied when truncating candidate edges, and the loop terminates solely on the token budget NtgtN_tgt rather than on the similarity level, so NtgtN_tgt itself plays the role of the budget floor NminN_ of the threshold mode. Algorithm 2 CRAFT Select-then-Fuse (fixed-ratio mode) 0: Features X, coords (t,h,w)\(t,h,w)\, target ratio ρθρ_θ 0: Compressed sequence XCX_C, updated coords 1: active←1,…,Nfactive←\1,…,N_f\; m←0m← 0; Esim←XWrand¯E_sim← XW_rand ⊳ Eq. 2 2: Ntgt←max(1,⌊Nf/ρθ⌋)N_tgt← (1, N_f/ _θ ) ⊳ Fixed token budget 3: while |active|>Ntgt|active|>N_tgt do 4: m←m+1m← m+1 5: p(i)←argmaxj≠isijp(i)← _j≠ is_ij; si⋆←si,p(i)s_i ← s_i,p(i), ∀i∈active∀ i ⊳ Pairing (Eq. 3) 6: k←min(|active|−Ntgt,⌊|active|/2⌋)k← \! (|active|-N_tgt,\, |active|/2 ) 7: P←P← the k edges (i,p(i))(i,p(i)) with highest si⋆s_i ⊳ Budget truncation 8: Orient each edge in P (ℓ2 _2-larger endpoint → target), drop chain-conflicting edges, and group into (t,it)(t,\i\_t) ⊳ §3.2 9: for all merge group (t,it)(t,\i\_t) do 10: xtnew←GatedMerge(xt,xit)x_t^new (x_t,\x_i\_t) ⊳ Eqs. 4–9 11: end for 12: X[targets]←xtnewX[targets]← x_t^new; active←active∖sourcesactive \sources\ 13: Esim[targets]←X[targets]Wrand¯E_sim[targets]← X[targets]W_rand ⊳ Refresh (Eq. 2) 14: end while 15: return XC←X[active]X_C← X[active] with updated coordinates A.2 Cross-Backbone Transfer Details CRAFT interacts with the host VLM through only two interfaces: (i) the visual token sequence after the vision encoder and multimodal projector, i.e., already in the language model’s embedding space, and (i) the spatio-temporal coordinates (t,h,w)(t,h,w) of each visual token. The compression core—the training-free selection of Algorithm 1 and the learnable gated merging of §3.3—is implemented as a single backbone-agnostic module shared verbatim by all three backbones, so porting CRAFT to a new architecture only requires locating the visual span in the input sequence and supplying per-token coordinates. Because the weighting scorer and gate operate on the backbone’s hidden width D, the lightweight merge module is re-instantiated at the target width and trained with exactly the same two-stage curriculum and hyperparameters as on Qwen3.5-4B (Table 4); no architecture-specific tuning is performed. Qwen2.5-VL-7B. The visual token layout matches Qwen3.5 (patch embedding with 2×22×2 spatial merging, M-RoPE), so the port is direct: 3D position ids are obtained from the backbone’s native get_rope_index (including Qwen2.5-VL’s per-video temporal scaling), compression runs unchanged on the video span, and after merging the position ids are index-selected onto the surviving positions, so retained tokens keep their original 3D rotary coordinates. LLaVA-OV-7B. This backbone differs in both the vision tower and the position encoding. Videos are encoded by SigLIP into a 27×2727×27 grid per frame and pooled to 14×14=19614×14=196 tokens per frame, followed by one image_newline token per video; compression is applied to the F×196F×196 feature tokens only, and the newline token is always retained. The language model uses standard 1D RoPE rather than M-RoPE, so the (t,h,w)(t,h,w) coordinates required by the position-aware weighting scorer (Eq. 5) are constructed explicitly from the known frame/grid layout and consumed only inside the scorer, while the language model keeps the original 1D positions: each surviving token inherits its own absolute position in the uncompressed sequence and decoding continues from the original sequence length, leaving all relative rotary distances identical to the uncompressed model. That the same design and training recipe works across distinct vision encoders (ViT with 2×22×2 merging vs. SigLIP with pooling), token layouts, and position encodings (M-RoPE vs. 1D RoPE) substantiates the claim of §4.3 that CRAFT’s effectiveness does not rely on architecture-specific tuning. Appendix B Experiment Details B.1 Benchmark Details MVBench. MVBench comprises 4,0004,000 QA pairs across 2020 temporal-understanding tasks. We evaluate on 3,8003,800 of them and skip the 200200 samples of the episodic_reasoning task, which are provided as pre-extracted frame folders sampled at fps=3fps=3 rather than as video files, inconsistent with the fps=2fps=2 sampling used in our evaluation. TempCompass. TempCompass offers several answer formats, including multiple-choice, yes/no, caption matching, and caption generation. We report on its full multiple-choice test set only. Figure 5: Training-data composition of the two-stage curriculum (§3.4). Left: source composition as a nested ring—the outer ring (blues) is Stage-1 caption pre-training (≈122≈122K video–caption pairs over eight public corpora), the inner ring (oranges) is Stage-2 QA (120120K pairs over five sources); percentages are within-stage shares and absolute counts are listed in the legend. Right: the Stage-2 temporal profile, resampled to a target long / medium / short mix of ≈33/ 35/ 31%≈33\,/\,35\,/\,31\%. Stage-2 sources play distinct roles: LongViTU (long-form), YouTube (open-domain breadth), NExT-QA (everyday), Academic (world knowledge), and PerceptionTest (fine-grained temporal). B.2 Training Hyperparameters Hyperparameter Stage 1 Stage 2 Caption QA CRAFT LR 4e−34e-3 6e−66e-6 LoRA LR (CRAFT++) 1e−41e-4 1e−41e-4 Gate initialization from scratch inherit Stage 1 Optimizer AdamW LR schedule cosine, warmup ratio 0.020.02 Weight decay 1e−21e-2 Gradient clipping 1.01.0 Epochs 11 Per-device batch size 11 Gradient accumulation 44 Global batch size 3232 (8×8×A800) Precision bf16 DeepSpeed ZeRO-2 Gradient checkpointing enabled Sampling rate fps=2fps=2 Max frames 768768 Max sequence length 3232K Table 4: Training hyperparameters for the two-stage curriculum on Qwen3.5-4B. The upper block lists the settings that differ between stages; the lower block lists shared settings. Only the lightweight weighting scorer and gate are trained in both stages; CRAFT++ additionally trains LoRA adapters on the language model. All experiments, including both training stages and all evaluations, are conducted on a single node with 88 NVIDIA A800 (80GB) GPUs. CRAFT is trained with a two-stage captioning-to-QA curriculum. Both stages fine-tune only the lightweight merge module—the position-aware weighting scorer and the content-adaptive gating network, a negligible fraction of the backbone parameters—inserted between the vision encoder and the language model, while the vision encoder, projector, and language-model backbone stay frozen throughout. We report two variants: CRAFT, which trains only this merge module, and CRAFT++, which additionally inserts LoRA adapters (rank 3232, α=64α=64, dropout 0.050.05) into the language-model attention projections (q,k,v,oq,k,v,o), trained with a separate learning rate. Stage 2 warm-starts from the Stage 1 checkpoint and inherits its learned gate, so it adapts the module with a much smaller learning rate. Table 4 lists the full configuration. During training the compression budget is not fixed. At every optimizer step we sample one compression setting and apply it consistently across the batch: with probability 0.50.5 a similarity threshold drawn from 0.75,0.8,0.85,0.9,0.95\0.75,0.8,0.85,0.9,0.95\, and otherwise a fixed ratio drawn from 2,4,8,16,32\2,4,8,16,32\. This exposes the compressor to a wide range of budgets so that a single checkpoint generalizes across operating points. B.3 Training Data Distribution CRAFT’s merge module is optimized with the two-stage curriculum of §3.4: a caption pre-training stage followed by a video question-answering (QA) stage. Figure 5 summarizes the composition of both stages; all data are drawn from public video corpora. To rule out data contamination, the training pool is audited against the six evaluation benchmarks and de-duplicated at the video level: videos overlapping any benchmark are removed, so the training set shares no videos—and hence no QA pairs—with the evaluation data; moreover, the compressed representation is query-agnostic, so no benchmark-specific data are used. Stage 1: caption pre-training. We aggregate ≈122≈122K video–caption pairs from eight public sources that span web video (WebVid-10M), bilingual described events (VATEX-en/zh, DiDeMo), short clips (MSR-VTT, MSVD), long-range video (LOVR), and surveillance (UCA-Crime). To keep the mixture broad and prevent high-volume corpora from swamping the long tail, each source is capped at 3030K pairs; WebVid-10M and LOVR are subsampled from much larger pools to this cap, while smaller sources are kept in full. This stage teaches the gate to preserve global semantics under aggressive merging. Stage 2: QA curriculum. We then align the compressor on 120120K video-QA pairs drawn from five sources with complementary roles: world knowledge (Academic), open-domain breadth (YouTube), everyday activities (NExT-QA), long-form video (LongViTU), and fine-grained temporal perception (PerceptionTest); videos in Academic that overlap the existing NExT-QA/ActivityNet pools are de-duplicated. This mixture is deliberately shifted toward longer clips than Stage 1: measured durations rise from a Stage-1 median of 1010 s (96.5%96.5\% under 6060 s) to a Stage-2 median of 6262 s, with about half of the clips falling in the 11–33 min range (Fig. 5, right). This exposes the compressor to substantially more temporal content than the near-static short clips of Stage 1, steering it toward the fine-grained cues needed to answer questions.