Paper deep dive
VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs
Yuan Feng, Qize Yang, Ruizhe Chen, Sibo Song, Haolin He, Muzhi Zhu, Zihan Liu, Yunfei Chu, Xize Cheng, Yuxuan Wang, Jin Xu, Xike Xie
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Downsampling sacrifices this detail, while existing token pruning and adaptive approaches remain limited in content-adaptive granularity, task generalization, and integration with modern MLLMs and serving infrastructure. Overcoming these limitations calls for foundation models that learn, end to end, where-and at what granularity-to allocate visual representations, a native capability we term elastic visual representation weaving. We introduce VisionWeave, establishing this capability in frontier-level MLLMs through large-scale training. It combines two components: a gated spatial pooler constructs coarse-grained representations alongside native fine-grained representations within a shared MRoPE coordinate, while a granularity router learns their content-adaptive allocation. Through self-distillation alone, we validate this capability on Qwen3.5-4B and scale to Qwen3.8-27B with over 30K A100 GPU-hours. Based on Qwen3.8-27B, VisionWeave adaptively adjusts token savings to visual content, saving 43.0% tokens on average while retaining 98.9% native performance across eight benchmarks, versus only 88% performance preserved for token pruning baselines with a fixed 50% savings target. Extensive evaluations confirm robust efficiency-quality trade-offs across diverse tasks, resolutions and video frames. When deployed on SGLang serving engine, our method achieves a 2.3x throughput gain while reducing mean TTFT by 54.4% and mean TPOT by 60.6%. Together, we believe these results position elastic visual weaving as a promising capability for next-generation multimodal models.
Tags
Links
- Source: https://arxiv.org/abs/2610.07987v1
- Canonical: https://arxiv.org/abs/2610.07987v1
Trouble viewing inline? Open PDF directly →
Full Text
67,292 characters extracted from source content.
Expand or collapse full text
Preprint VISIONWEAVE: WEAVING ELASTIC VISUAL REPRE- SENTATIONS AS A NATIVE CAPABILITY OF MLLMS Yuan Feng 1,2∗ , Qize Yang 2∗ , Ruizhe Chen 2 , Sibo Song 2 , Haolin He 2,3 , Muzhi Zhu 2,4 , Zihan Liu 2 , Yunfei Chu 2 , Xize Cheng 2 , Yuxuan Wang 2 , Jin Xu 2,† , Xike Xie 1,† 1 University of Science and Technology of China 2 Alibaba Token Hub, Alibaba Group 3 The Chinese University of Hong Kong 4 Zhejiang University ABSTRACT Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed- size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Downsampling sacrifices this detail, while existing token pruning and adaptive approaches remain limited in content-adaptive granularity, task generalization, and integration with modern MLLMs and serving infrastructure. Overcoming these limitations calls for foundation models that learn, end to end, where—and at what granularity—to allocate visual representations, a native capability we term elastic visual representation weaving. We introduce VisionWeave, establishing this capability in frontier-level MLLMs through large-scale training. It com- bines two components: a gated spatial pooler constructs coarse-grained represen- tations alongside native fine-grained representations within a shared MRoPE coor- dinate, while a granularity router learns their content-adaptive allocation. Through self-distillation alone, we validate this capability on Qwen3.5-4B and scale to Qwen3.8-27B with over 30K A100 GPU-hours. Based on Qwen3.8-27B, Vision- Weave adaptively adjusts token savings to visual content, saving 43.0% tokens on average while retaining 98.9% native performance across eight benchmarks, ver- sus only 88% performance preserved for token pruning baselines with a fixed 50% savings target. Extensive evaluations confirm robust efficiency–quality trade-offs across diverse tasks, resolutions and video frames. When deployed on SGLang serving engine, our method achieves a 2.30× end-to-end throughput gain while reducing mean TTFT by 54.4% and mean TPOT by 60.6%. Together, we believe these results position elastic visual weaving as a promising capability for next- generation multimodal foundation models, natively modulating computational ef- fort by information content. Our SGLang code and other details will be available at https://github.com/FFY0/VisionWeave. (a) RealWorldVisual Tokens 1728 → 867 (−49.8%)(b) TextVisual Tokens 1976 → 1577 (−20.2%) (c) DrivingVisual Tokens 1380 → 840 (−39.1%)(d) ChartVisual Tokens 1900 → 1054 (−44.5%) Fine-grained visual representationsCoarse-grained visual representation Figure 1: Weaving elastic visual representations. More examples appear in Appendix C. ∗ Equal contribution. † Corresponding authors . Work done as an intern at Qwen Team, Alibaba Token Hub. 1 arXiv:2610.07987v1 [cs.CV] 6 Oct 2026 Preprint 0255075 RealWorldQA DocVQA InfoVQA ScreenSpot V2 HallusionBench VideoOCR Video-MME LongVideoBench Fixed 50% Qwen3.8-27BQwen3.5-4B RealWorldQA DocVQA InfoVQA ScreenSpot V2 HallusionBench VideoOCR Video- MME LongVideo Bench 75% 80% 85% 90% 95% 100% 105% 50% RealWorldQA DocVQA InfoVQA ScreenSpot V2 HallusionBench VideoOCR Video- MME LongVideo Bench 75% 80% 85% 90% 95% 100% 105% 50% (b) Qwen3.8-27B(c) Qwen3.5-4B(a) Token savings (%) Median; P25–P75 (thick); P5–P95 (thin)Task score / same-model Native (%) Native Qwen model (512 tokens/image or frame)VisionWeave FastV † VisionZip Figure 2: VisionWeave achieves content-adaptive token savings with near-native quality, whereas fix-budget baselines degrade sharply on challenging tasks. Visual case studies on ScreenSpotV2 and DocVQA illustrate this difference (Appendix B). Our VisionWeave still retains higher quality even when baselines are granted its adaptive saving ratios (Appendix E.3). 1INTRODUCTION Multimodal large language models (MLLMs) have become the dominant paradigm for visual under- standing, supporting diverse tasks such as visual question answering, document analysis, and video understanding within a unified framework (Bai et al., 2025b;a). As these tasks demand finer spatial detail and broader temporal coverage, MLLMs must process higher-resolution images and longer sequences of video frames. However, their visual inputs are typically partitioned into fixed-size patches and encoded into dense sequences of visual tokens. The resulting growth in sequence length places substantial computational and memory demands, making visual efficiency a central challenge in scaling multimodal capabilities (Shao et al., 2025a; Tao et al., 2024; Shao et al., 2025b). Visual downsampling offers a straightforward solution but sacrifices fine-grained detail, motivat- ing extensive research into visual token pruning and merging (Chen et al., 2024; Yang et al., 2024; Ye et al., 2024; Guo et al., 2026). However, both heuristic and learned methods introduce a train- ing–inference mismatch and discard task-specific information. Our evaluations of FastV and Vi- sionZip reveal severe degradation on information-dense inputs and grounding tasks. Common de- sign choices in these methods, including attention-weight extraction and in-LLM pruning, also com- plicate integration with modern serving engines such as SGLang and vLLM, where implementations remain rare and practical efficiency gains insufficiently validated. Preset compression ratios further limit adaptivity to diverse visual inputs. These limitations motivate native, content-adaptive visual processing for general-purpose MLLMs: learning where to retain fine-grained detail and where— and to what extent—to use compact coarse representations. Recent work has explored content-adaptive visual representations, but realizing this adaptivity as a native capability of general-purpose MLLMs remains challenging. One line of work varies ViT patch sizes using low-level image statistics, such as edge density and entropy (Yu et al., 2025; Choudhury et al., 2025). These methods primarily target standalone vision tasks, and their granular- ity criteria are not optimized for diverse downstream multimodal objectives. ViCO (Cui et al., 2025) is particularly close to our approach: it partitions an image into tiles and adaptively assigns each a visual token resolution. However, its tile-based design limits applicability to modern MLLMs such as Qwen (Bai et al., 2025a) and Kimi (Team, 2026), which use native-resolution visual processing. Moreover, detail-rich regions and low-information backgrounds are often interleaved (Figure 1), requiring finer-grained spatial allocation than a single resolution per tile can provide. 1 1 See related work section for further discussion in Appendix D and Figure 16 for a visual comparison. 2 Preprint Granularity Router Internal visual features Context-adaptive decision Image / Frame ViT Backbone Fine-grained Representation Merger Granularity Selection Gather Mixed-granularity LLM Gated Spatial Pooler Coarse-grained Representation Native Position Center Position X P M V fine V coarse Figure 3: Overall architecture of VisionWeave. The gated spatial pooler complements native fine- grained tokens V fine with coarse-grained representations V coarse within the same MRoPE coordinate, while the granularity router adaptively weaves them into a mixed-granularity sequence for the LLM. We envision elastic visual representation weaving as a native capability of foundamental MLLMs that should satisfy three requirements: 1. Autonomously content-adaptive token savings while preserving quality. 2. Consistently favorable trade-off than downsampling across diverse visual task inputs. 3. Compatibility with frontier-level MLLM architectures and modern serving infrastructure. To this end, we introduce VisionWeave, which enables MLLMs to operate natively on elastic visual representations, adaptively weaving together fine- and coarse-grained representations according to visual content. To our knowledge, this is the first effort to establish elastic visual representation weaving as a native capability of frontier MLLMs through large-scale training, accompanied by extensive evaluation to assess its viability as an integral capability of foundation models. The method comprises two core components. First, the elastic representation itself: a gated spatial pooler constructs a learned coarse-grained representation that complements the native fine-grained tokens within the same MRoPE frame. Second, learned granularity allocation: a granularity router probes ViT features to determine the appropriate granularity for each spatial block. Through self- distillation alone, we make elastic visual processing a native capability, validating on Qwen3.5-4B and then scaling to frontier Qwen3.8-27B with over 30K A100 GPU-hours. With this native capability, VisionWeave preserves fine-grained detail where needed and uses com- pact representations elsewhere (Figure 1), achieving content-adaptive token savings with minimal impact on task quality (Figure 2). At an input budget of 512 tokens per image, VisionWeave on Qwen3.8-27B saves 16.5–55.1% of visual tokens across eight benchmarks, averaging 43.0% savings while preserving 98.9% native performance. In contrast, FastV † and VisionZip retain only 88.2% and 87.5%, respectively, at a fixed 50% savings target. Broader evaluations across input resolutions and video frame budgets demonstrate consistently more favorable efficiency–quality trade-offs than input downsampling. Crucially, these token savings translate into practical serving gains build upon modern SGLang serving engine, achieving around 2× throughput improvements. Together, content- adaptive savings, cross-task robustness, and practical deployability support elastic visual processing as a native capability of next-general efficient MLLMs. 2METHOD In this work, we realize elastic visual representation weaving through two complementary compo- nents. A gated spatial pooler constructs learned coarse-grained representations that complement native fine-grained tokens (§2.2.1). A granularity router probes intermediate ViT features to deter- mine the granularity for each spatial block (§2.2.2). Through self-distillation alone, we make elastic visual processing a native capability of MLLMs (§2.3). 3 Preprint 2.1PRELIMINARIES For modern MLLMs, the ViT produces patch features X ∈R H×W×d . A native mergerM concate- nates each 2×2 neighborhood and applies an MLP to produce fine-grained visual tokens: V fine =M(X)∈R H 2 × W 2 ×D ,(1) where d and D are the ViT and LLM hidden sizes, respectively. MRoPE assigns each token coor- dinates π ij = (τ,i,j), with temporal coordinate τ and spatial indices (i,j). The flattened visual tokens and textual context c condition autoregressive response generation, p θ (y | c,V fine ). 2.2ARCHITECTURE 2.2.1GATED SPATIAL POOLER Pooling adjacent patch features is a natural compression primitive: the patches of a block cover contiguous regions whose content is typically correlated—increasingly so wherever the image is locally homogeneous. Leveraging this correlation, Our gated spatial pooler P maps each 2×2 neighborhood of ViT patch features to a single pooled feature. Passing these features through the native merger yields the coarse-grained representation in the same embedding space. V coarse =M P(X) ∈R H 4 × W 4 ×D , P(X)∈R H 2 × W 2 ×d .(2) Each block (I,J ) on the H 4 × W 4 grid covers 4×4 patches and admits either four native fine-grained tokens V fine,2I+a,2J+b a,b∈0,1 or one coarse-grained token V coarse,IJ . We use positional inter- polation (Peng et al., 2023; Chen et al., 2023) to place each coarse token at the geometric center of its four corresponding fine-grained tokens. π IJ = τ, 2I + 1 2 , 2J + 1 2 .(3) We implement the gated spatial pooler P as a learnable extension of mean pooling, combining content-dependent and channel-wise weighting with feature refinement. For each 2×2 neighbor- hood, we normalize its four patch features x k 4 k=1 using learnable layer normalization 2 and ag- gregate them into a shared global context g k : u k = LN(x k )∈R d , g k = f glb [u 1 ;u 2 ;u 3 ;u 4 ] + b k ∈R d ,(4) where [·;·] denotes concatenation and b k ∈R d is a learnable position-specific bias (DeepSeek-AI, 2026). The gate then computes channel-wise scores from patch feature u k and its context g k : α k = softmax k=1,...,4 f gate [u k ; g k ] ∈ (0, 1) d ,(5) where the softmax normalizes over the four spatial positions channel-wise, i.e. P 4 k=1 α k = 1 d . Finally, a value net f val refines each feature, and the pooled output is the gated sum, v k = u k + f val [u k ; g k ] ∈R d , P(x 1:4 ) = X 4 k=1 α k ⊙ v k ∈R d ,(6) where f glb , f gate , f val are two-layer MLPs (each mapping intoR d ) and⊙ is the elementwise prod- uct. Beyond its greate expressiveness 3 , the pooler offers a favorable starting point: zero-initializing the final layers of f gate and f val reducesP to mean pooling over normalized patch features. 2.2.2GRANULARITY ROUTER With both granularities available, our granularity router learns their content-adaptive allocation across spatial blocks 4 . It probes ViT features at multiple depths with a learnable query, combining low-level detail from early layers with semantic information from later layers. We implement this 2 ViT patch features exhibit large magnitude variation; LayerNorm prevents high-magnitude tokens from dominating the pooling output. 3 Ablation study in Section 4.1 shows gated pooler consistently outperforms the widely used pixel-unshuffle. 4 Ablation studies in Section 4.1 validate the effectiveness of the learned router. 4 Preprint Algorithm 1 VisionWeave inference with elastic visual representations Require: An H×W patch grid for an image; textual context c; routing threshold t (default 0.5) 1: Obtain X (0) , X (L/2) , X (L) in a single ViT forward pass 2: V fine ←M(X (L) ); V coarse ←M P(X (L) ) Eqs. (1), (2) 3: Initialize h IJ from the shared query with block-center RoPE coordinates 4: for ℓ∈0, L//2, L do 5: h IJ ← LocalCrossAttn (ℓ) h IJ , X (ℓ) IJ 6: h IJ ← GlobalCrossAttn (ℓ) h IJ , X (ℓ) Eq. (8) 7: end for 8: Compute p IJ using the calibrated router head 9: for each block (I,J ) do 10: if p IJ > t then 11:Select V coarse,IJ with its block-center coordinate π IJ from Eq. (3) 12: else 13:SelectV fine,2I+a,2J+b a,b∈0,1 with their native coordinates from Section 2.1 14: end if 15: end for 16: Gather selected tokens into V routed in native raster order 17: return p θ y | c, V routed multi-depth probing with one query state h IJ ∈R d per spatial block, initialized from a shared learn- able vector. Each query uses the block-center MRoPE coordinate 4I + 3 2 , 4J + 3 2 on the H×W patch grid, corresponding to the same spatial center as Eq. (3). For a ViT with L layers, we denote the features after layer ℓ by X (ℓ) ∈R H×W×d , with X (0) the patch embeddings and X (L) = X . Each query state is updated through cross-attention to ViT features, followed by an MLP, both with residual connections: h ′ = LN h + CrossAttn h, LN(X (ℓ) ) , h← LN h ′ + MLP(h ′ ) ,(7) where CrossAttn takes its query from h and keys/values from LN(X (ℓ) ), with MRoPE applied to queries and keys; the KV-side LN keeps features tapped at different depths on a comparable scale. Each query sequentially probes depths ℓ = 0, L//2, L. At each depth, a local block is followed by a global block, yielding six independently parameterized blocks in total. The local block attends to the spatial block’s own 4×4 patch features X (ℓ) IJ ∈R 4×4×d ; a global block then opens them to the whole frame: h IJ ← LocalCrossAttn (ℓ) h IJ , X (ℓ) IJ ; h IJ ← GlobalCrossAttn (ℓ) h IJ , X (ℓ) .(8) Finally, a linear head maps the normalized query state to two logits, giving the probability of using the coarse-grained representation: p IJ = softmax W r LN(h IJ ) coarse , W r ∈R 2×d .(9) For each spatial block, we use its coarse-grained token if the routing probability exceeds a threshold 5 and retain its four native fine-grained tokens otherwise. The selected tokens are gathered into a mixed-granularity sequence for the LLM backbone within the shared MRoPE frame. Algorithm 1 summarizes the complete inference path. 2.3TRAINING We develop elastic visual weaving through three-stage self-distillation (Table 1). 6 The native for- ward pass provides output-distribution supervision. Distillation is off-policy 7 , with teacher and 5 We set t = 0.5 by default, equivalent to selecting the granularity with the higher predicted probability. Section 3.5 further shows how varying this threshold controls the compression level at inference time. 6 Section 2.4 details the training data and settings and analyzes loss and routing dynamics. 7 We try switching to on-policy self-distillation in Stage 3 of Qwen3.5-4B but observe no consistent gains (Appendix E.2). We hypothesize that elastic visual weaving keeps the student’s output distribution close to the teacher’s, limiting the benefit of on-policy sampling; we therefore use off-policy training for efficiency. 5 Preprint Table 1: Three-stage self-distillation for elastic visual processing. Stage 1Stage 2Stage 3 Trainable components Gated Spatial PoolerGranularity RouterPooler, LLM TeacherNative modelNative modelNative model RoutingBypassed (all coarse) Soft-mixing routing Hard routing Loss termsL distill L distill + 0.02L bal L distill Student visual tokens25%100%Content-adaptive student conditioned on the same teacher rollout. The student progresses from coarse-grained repre- sentations to a differentiable soft-mixing surrogate, and finally to the hard-routed mixed-granularity sequences used at inference. 2.3.1STAGE 1: COARSE-GRAINED SELF-DISTILLATION We first train the pooler with routing bypassed, using the coarse-grained representation for every block: V student = flatten(V coarse ). All other components remain frozen. The model’s native for- ward pass serves as the teacher, yielding the distillation objective: L distill = 1 |y| X t KL bp ̄ θ ·| y <t , c, V fine bp θ ·| y <t , c, V student ,(10) where y is a response generated by the teacher and ̄ θ denotes its parameters. Both bp ̄ θ and bp θ are renormalized over the teacher’s top-512 vocabulary entries at each response position. Later stages reuse this objective with different constructions of V student . 2.3.2STAGE 2: SOFT-MIXING SELF-DISTILLATION We next train only the granularity router using a differentiable soft-mixing surrogate for hard rout- ing. We interpolate feature directions and magnitudes separately to prevent norm differences from biasing the directional mixture. Let b V fine and b V coarse denote the token-wise L2-normalized repre- sentations underN (v) = v/ max(∥v∥ 2 ,ε). We compute m ij = (1− p IJ )∥V fine,ij ∥ 2 + p IJ ∥V coarse,IJ ∥ 2 , V mix,ij = m ij N (1− p IJ ) b V fine,ij + p IJ b V coarse,IJ . (11) where I = ⌊i/2⌋, J = ⌊j/2⌋, and ε ensures numerical stability. Each coarse token is broadcast to its four corresponding native positions. The resulting sequence, V student = flatten(V mix ), is used in the distillation objective of Eq. (10). We add a per-sample balance loss borrow from MoE training: L bal = F ρ G + 1− F 1− ρ (1− G),(12) where F is the fraction of the sample’s blocks with p IJ ≥ 0.5, G is its mean routing probability, ρ is the target coarse-routing fraction. The penalty encourages more coarse routing when F < ρ and less when F > ρ. Per-sample statistics prevent larger inputs from dominating the estimate and batch averages from masking input-level imbalance. We set ρ = 0.8 and optimizeL distill + 0.02L bal . 2.3.3STAGE 3: MIXED-GRANULARITY SELF-DISTILLATION Stage 3 performs self-distillation on the hard-routed mixed-granularity sequences used at inference. 8 The router head includes an offline-calibrated scalar coarse-routing bias, held fixed during training and inference, without auxiliary balancing losses or online adjustments. 9 The student gathers V student from V fine and V coarse using the same token selection and RoPE coordinates as at inference. The native forward pass of a frozen teacher provides supervision through the distillation objective. 8 Appendix E.1 shows that Stage 3 improves performance across diverse tasks, especially on grounding. 9 In experiments, we calibrate on a training subset targeting 50% visual-token savings. 6 Preprint Table 2: Training-data composition for Qwen3.8-27B. SourceDomainApprox. training samples LLaVA-OneVision (An et al., 2025)Image understanding246K LLaVA-Video-178K (Zhang et al., 2025b)Video understanding281K UGround (Gou et al., 2025)GUI grounding250K Total777K 2.4TRAINING DETAILS AND DYNAMICS Training data. All three stages of Qwen3.8-27B training use the same mixture of approximately 0.78 million samples covering image understanding, video understanding, and GUI grounding (Ta- ble 2). Counts refer to samples before sequence packing. The visual input budgets are 4,096 native visual tokens per image and 512 per video frame, with up to 64 sampled frames. Training settings. The granularity router samples ViT features at depths [0, 13, 27], with one local–global cross-attention pair per depth. Its six layers use 16 attention heads, an MLP expansion ratio of 2, and key/value layer normalization; the scoring head is kept in FP32. All stages use BF16 training with Adam (β 1 = 0.9, β 2 = 0.999, ε = 10 −8 ), zero weight decay and cosine learning-rate decay. Table 3 summarizes the stage-specific settings. Distillation is off-policy at temperature 1, using forward KL divergence over top-512 vocabulary entries. Stage 1 weights each sample’s mean token loss by the square root of its supervised response length. We observed training instability with this weighting in Stage 2 and therefore use a global mean over supervised response tokens in Stages 2 and 3. Stage 3 updates the pooler and LLM, while keeping the ViT, native merger, and router frozen. Global batch sizes count packed sequences in Stages 1 and 2 and individual samples in Stage 3. Table 3: Training hyperparameters for Qwen3.8-27B. SettingStage 1Stage 2Stage 3 Peak learning rate1× 10 −4 1× 10 −5 3×10 −6 Minimum learning rate5×10 −6 00 Warmup fraction5%5%2% Global batch size3264512 Training steps5,2572,5001,000 Gradient clipping1.01.00.5 Student maximum sequence length16,38424,57624,576 Packed sequence length16,38424,576Disabled Training dynamics. Figure 4 traces the three self-distillation stages on Qwen3.8-27B, showing raw losses and exponential moving averages (EMA, α = 0.02, initialized at the first step). The routing distributions reveal how granularity decisions develop. In Stage 2, mean router entropy falls from 0.658 to 0.427 nats between the first and last 100 steps. From step 50 to 2,500, the validation share of blocks with p IJ ∈ [0.4, 0.6) decreases from 64.8% to 14.6%, while the share with p IJ ≥ 0.9 reaches 28.3%. These changes indicate increasingly decisive allocation; Section 4.1 tests its contribution to task quality. 2.5DEPLOY VISIONWEAVE ON MODERN SERVING INFRASTRUCTURE To demonstrate how native elastic visual weaving can be deployed on modern serving infrastructure, we integrate VisionWeave into SGLang (Zheng et al., 2023). This requires addressing two core challenges: supporting block-center MRoPE coordinates and mixed-granularity visual sequences. Block-center RoPE support. Coarse-grained tokens use the block-center RoPE coordinates de- fined in Eq. (3). To support these half-integer positions in SGLang’s integer-indexed RoPE caches, we double all position coordinates and halve rotary frequencies, preserving rotary phases. We apply 7 Preprint 020004000 0.05 0.10 0.15 0.20 Training loss Stage 1 010002000 0.04 0.06 0.08 Stage 2 05001000 0.02 0.04 Stage 3 Training step Raw loss Smoothed loss (EMA, α= 0.02) 0100020002500 0.3 0.4 0.5 0.6 0.7 Entropy (nats) Training router entropy 50100020002500 [0.0, 0.1) [0.1, 0.2) [0.2, 0.3) [0.3, 0.4) [0.4, 0.5) [0.5, 0.6) [0.6, 0.7) [0.7, 0.8) [0.8, 0.9) [0.9, 1.0] Coarse-routing probability p IJ Validation probability bins 0 25 50 Blocks (%) Training step Raw entropy Smoothed entropy (EMA, α= 0.02) Figure 4: Training dynamics of three-stage self-distillation. this rescaling to cache construction and growth and to position updates during prefill and decode, reusing the existing rotary-embedding and attention kernels. Mixed-granularity sequence support. We extend SGLang’s encoder-disaggregated stage by at- taching layout metadata to visual responses. This enables pre-admission embedding slicing and RoPE construction, finalizing sequence layouts before scheduling while preserving dynamic batch- ing, KV-cache management, and CUDA-graph decoding. Serving gains are reported in Section 3.4. 3EXPERIMENTS We organize our evaluation around the three requirements for elastic visual weaving as a native MLLM capability: quality-preserving, content-adaptive token savings (Section 3.2), robust trade- offs across resolutions and frame budgets (Section 3.3), and practical serving gains on SGLang (Section 3.4). We then examine inference-time control (Section 3.5) and analyze component contri- butions and granularity allocation (Section 4). 3.1EXPERIMENTAL SETUP Models and baselines. We evaluate with two frontier open-source MLLMs, the Qwen3.5-4B and Qwen3.8-27B. We compare VisionWeave against two heuristic visual-token reduction methods: Vi- sionZip Yang et al. (2024) and FastV Chen et al. (2024), representing token merging and pruning, respectively. To enable a consistent evaluation within SGLang and support its modern serving fea- tures, we use a FastV variant, denoted FastV † , that selects tokens after the vision encoder rather than within the LLM. Unless otherwise stated, VisionWeave uses a default routing threshold of t = 0.5. Tasks. We assess cross-domain robustness on eight benchmarks spanning natural-image (Real- WorldQA), document and infographic understanding (DocVQA and InfoVQA), GUI grounding (ScreenSpotV2), visual hallucination (HallusionBench), video details (VideoOCR), and video un- 8 Preprint Table 4: Content-adaptive token savings with near-native quality. Native ModelFixed Savings (s=50%)Content-adaptive Savings Qwen3.5-4B512 token/ImageDownsamplingFastV † VisionZipVisionWeaveSavings (%) RealWorldQA74.6472.16(↓3.32%)70.98 (↓4.90%)70.72 (↓5.25%) 73.46 (↓1.58%)43.2 DocVQA91.9282.71(↓10.02%)78.69 (↓14.39%)76.57 (↓16.70%) 91.03 (↓0.97%)16.0 InfoVQA67.4452.87(↓21.60%)51.75 (↓23.27%)50.89 (↓24.54%) 66.11 (↓1.97%)10.6 ScreenSpotV289.8680.42(↓10.51%)72.56 (↓19.25%)70.44 (↓21.61%) 89.47 (↓0.43%)24.1 HallusionBench69.8869.09(↓1.13%) 69.26 (↓0.89%)68.64 (↓1.77%) 69.26 (↓0.89%)44.0 VideoOCR64.5159.74 (↓7.39%) 63.90 (↓0.95%)63.18 (↓2.06%)63.79 (↓1.12%)47.0 Video-MME67.7867.07 (↓1.05%)67.70 (↓0.12%) 67.74 (↓0.06%)67.22 (↓0.83%)51.4 LongVideoBench59.9958.86 (↓1.88%)60.06 (↑0.12%)59.61 (↓0.63%) 60.36 (↑0.62%)49.8 Average73.2567.87(↓7.11%)66.86 (↓7.96%)65.97 (↓9.08%) 72.59 (↓0.90%)35.8 Qwen3.8-27B RealWorldQA78.8274.64 (↓5.30%) 78.43 (↓0.49%)77.65 (↓1.48%)77.91(↓1.15%)43.1 DocVQA92.4783.19(↓10.04%)77.05 (↓16.68%)76.01 (↓17.80%) 91.33 (↓1.23%)27.3 InfoVQA74.1358.48(↓21.11%)55.49 (↓25.15%)53.96 (↓27.21%) 71.61 (↓3.40%)16.5 ScreenSpotV294.4291.04 (↓3.58%)49.53 (↓47.54%)49.21 (↓47.88%) 91.43 (↓3.17%)55.1 HallusionBench73.2572.10(↓1.57%)71.92 (↓1.82%)71.66 (↓2.17%) 73.60 (↑0.48%)44.9 VideoOCR70.7265.59 (↓7.25%)69.69(↓1.46%)69.54 (↓1.67%) 70.51 (↓0.30%)49.6 Video-MME70.8970.41 (↓0.68%)70.70 (↓0.27%)70.96(↑0.10%) 71.04 (↑0.21%)54.2 LongVideoBench66.4964.70 (↓2.69%)65.74(↓1.13%)65.45 (↓1.56%) 66.42 (↓0.11%)53.0 Average77.6572.52(↓6.53%)67.32 (↓11.82%)66.81 (↓12.46%) 76.73 (↓1.08%)43.0 derstanding (Video-MME and LongVideoBench). All task scores are reported on a 0–100 scale, using ANLS for DocVQA and InfoVQA, aAcc for HallusionBench, and the task-specific metric for each remaining benchmark. 3.2CONTENT-ADAPTIVE TOKEN SAVINGS WITH PRESERVED QUALITY With a default routing threshold t = 0.5, VisionWeave achieves content-adaptive token savings while maintaining near-native quality (Table 4). Across all eight benchmarks, VisionWeave incurs an average performance loss of only 0.90% on Qwen3.5-4B and 1.08% on Qwen3.8-27B, relative to the corresponding native models. Specifically, on Qwen3.8-27B, VisionWeave saves 16.5% of tokens on InfoVQA and 27.3% on DocVQA, compared with 49.6–54.2% across the three video benchmarks, where performance losses remain at or below 0.30%. Qwen3.5-4B shows a similar pattern, saving 10.6–16.0% on the two document benchmarks and 47.0–51.4% on video. In contrast, FastV † and VisionZip incur average losses of 11.82% and 12.46% on Qwen3.8-27B at a fixed 50% token-savings target. Although losses are modest on video, they degrade sharply on grounding and information-dense tasks: VisionZip incurs losses of 47.88% on ScreenSpotV2 and 17.80% on DocVQA. VisionWeave, by contrast, loses only 3.17% on ScreenSpotV2 despite more aggressive token savings of 55.1%, while adaptively reducing its savings to 27.3% on DocVQA with only a 1.23% loss. These results highlight the advantage of content-adaptive savings: reducing visual tokens according to input demands while preserving quality. To isolate the effect of dataset-level savings, we also assign FastV † and VisionZip VisionWeave’s observed savings ratio for each backbone and dataset. VisionWeave retains the highest average score: 72.59 versus 72.20/71.90 on Qwen3.5-4B, and 76.73 versus 70.10/70.33 on Qwen3.8-27B. Rankings vary across tasks, but matching the dataset-level savings alone does not reproduce Vision- Weave’s average quality. Complete results appear in Appendix E.3. 3.3ROBUSTNESS ACROSS INPUT RESOLUTIONS AND FRAME BUDGETS We next demonstrate the robustness of VisionWeave’s efficiency–quality trade-offs across tasks, spa- tial resolutions, and frame budgets. Consistently improving upon input downsampling is nontrivial: a method that works well in one setting may fail in another. Figure 5 shows that VisionWeave con- sistently maintains favorable trade-offs across all eight benchmarks and the evaluated resolutions. 9 Preprint 00.5K1K 70 75 128 256 512 1024 1024 128256512 1024 Tokens/image 0 25 50 Savings (%) 38.7 47.2 Qwen3.5-4B 00.5K1K 70 75 128 256 512 1024 1024 128256512 1024 Tokens/image 0 25 50 Savings (%) 31.5 45.6 Qwen3.8-27B 00.5K1K 40 60 80 128 256 512 1024 1024 128256512 1024 Tokens/image 0 25 50 Savings (%) 6.7 25.1 Qwen3.5-4B 00.5K1K 50 75 128 256 512 1024 1024 128256512 1024 Tokens/image 0 25 50 Savings (%) 5.6 39.4 Qwen3.8-27B 00.5K1K 50 75 128 256 512 1024 1024 128256512 1024 Tokens/image 0 25 50 Savings (%) 10.4 30.2 Qwen3.5-4B 00.5K1K 50 75 128 256 512 1024 1024 128256512 1024 Tokens/image 0 25 50 Savings (%) 33.9 56.8 Qwen3.8-27B 06K12K 55 60 65 128 256 512 1024 1024 128256512 1024 Tokens/frame 0 25 50 Savings (%) 39.9 48.1 Qwen3.5-4B 06K12K 60 70 128 256 512 1024 1024 128256512 1024 Tokens/frame 0 25 50 Savings (%) 41.0 52.7 Qwen3.8-27B 00.2K0.4K 40 60 64 128 256 512 512 64 128256512 Tokens/image 0 25 50 Savings (%) 10.0 10.6 Qwen3.5-4B 00.2K0.4K 40 60 64 128 256 512 512 64 128256512 Tokens/image 0 25 50 Savings (%) 18.7 16.5 Qwen3.8-27B 00.15K0.3K 65 70 64 128 256 512 512 64 128256512 Tokens/image 0 25 50 Savings (%) 29.6 44.0 Qwen3.5-4B 00.15K0.3K 65 70 64 128 256 512 512 64 128256512 Tokens/image 0 25 50 Savings (%) 26.6 44.9 Qwen3.8-27B 06K12K 64 66 68 64 128 256 512 512 64 128256512 Tokens/frame 0 25 50 Savings (%) 41.4 51.4 Qwen3.5-4B 06K12K 65.0 67.5 70.0 64 128 256 512 512 64 128256512 Tokens/frame 0 25 50 Savings (%) 44.2 54.2 Qwen3.8-27B 05K10K 55.0 57.5 60.0 64 128 256 512 512 64 128256512 Tokens/frame 0 25 50 Savings (%) 39.1 49.8 Qwen3.5-4B 05K10K 60.0 62.5 65.0 64 128 256 512 512 64 128256512 Tokens/frame 0 25 50 Savings (%) 41.0 53.0 Qwen3.8-27B (a) RealWorldQA(b) DocVQA (c) ScreenSpotV2(d) VideoOCR (e) InfoVQA(f) HallusionBench (g) Video-MME(h) LongVideoBench Score (%) Visual tokens per sample Native Qwen modelVisionWeave (default t=0.5) FastV † VisionZip Figure 5: VisionWeave achieves consistently favorable trade-offs across tasks and input resolutions. Video resolution sweeps fix the frame cap at 64 and the sampling rate at 2 FPS. FastV † and VisionZip remain competitive on VideoOCR yet degrade sharply on ScreenSpotV2. Their effectiveness also varies with resolution within the same task. On Qwen3.8-27B DocVQA, for example, VisionZip offers a competitive trade-off at a 256-token input budget but falls behind simple downsampling at higher resolutions. By contrast, VisionWeave consistently offers a more favorable efficiency–quality trade-off than input downsampling across the evaluated tasks and resolutions. The same advantage extends to temporal coverage (Figure 6). With a 256-frame cap, Qwen3.8-27B VisionWeave out- performs the native model capped at 128 frames by 1.20 points on LongVideoBench while using fewer visual tokens. Together, these results support the robust performance of native elastic visual weaving across diverse input conditions. 10 Preprint 025K50K 60 65 32 64 128 256 256 Qwen3.5-4B 3264 128256 Max frames 0 25 50 Token savings (%) 49.6 50.5 025K50K 65.0 67.5 32 64 128 256 256 Qwen3.8-27B 3264 128256 Max frames 0 25 50 Token savings (%) 53.1 52.7 Accuracy (%) Visual tokens per sample Native Qwen FastV † VisionWeave (t= 0.5) VisionZip Figure 6: VisionWeave maintains favorable trade-offs across frame budgets on LongVideoBench. DocVQA InfoVQA ScreenSpotV2 RealWorldQA HallusionBench VideoOCR Video-MME LongVideoBench 40 60 80 Score 65.46 45.42 67.53 68.89 67.05 57.69 65.89 57.44 68.10 46.81 69.42 69.15 68.20 60.46 66.93 58.71 Pixel-unshuffleGated spatial pooler Figure 7: Gated pooling outperforms pixel- unshuffle across eight benchmarks. Table 5: VisionWeave accelerates Qwen3.8-27B serving on SGLang. ModelThroughput (req/min)↑Mean TTFT (s)↓P95 TTFT (s)↓Mean TPOT (ms)↓P95 TPOT (ms)↓ Native Model0.90169.90281.41355.64715.58 VisionWeave2.07 (2.30×)77.54 (↓54.4%) 120.59 (↓57.1%) 140.29 (↓60.6%)289.19 (↓59.6%) 128256512 70.0 72.5 75.0 RealWorldQA 128256512 60 80 DocVQA 128256512 45 60 InfoVQA 128256512 60 75 90 ScreenSpotV2 128256512 67.5 69.0 HallusionBench 128256512 55 60 65 VideoOCR 128256512 66 67 68 Video-MME 128256512 58 60 LongVideoBench 128256512 72 76 80 RealWorldQA 128256512 60 80 DocVQA 128256512 45 60 75 InfoVQA 128256512 50 75 100 ScreenSpotV2 128256512 70 72 74 HallusionBench 128256512 60 66 72 VideoOCR 128256512 70.5 72.0 Video-MME 128256512 64.5 66.0 67.5 LongVideoBench Thresholds, left to right: 0.1, 0.25, 0.5, 1.0 (a) Qwen3.5-4B (b) Qwen3.8-27B Score Equivalent token budget VisionWeave VisionWeave default (t= 0.5) Qwen 512 TokenQwen 128 TokenFastV 256 TokenVisionZip 256 Token Figure 8: VisionWeave enables flexible inference-time control of token savings while closely match- ing native-model performance when savings are disabled. 3.4END-TO-END SERVING EFFICIENCY ON SGLANG To evaluate practical deployability, we benchmark native Qwen3.8-27B and VisionWeave on SGLang using our integration (Section 2.5). Both run on two A100-80GB GPUs (BF16, TP=2), with chunked prefill and CUDA-graph decoding enabled. We use the LongVideoBench with fps2, 256 max frames per video, a spatial budget of 1024 tokens/frame. Client concurrency is four, with a server limit of eight. More details appear in Appendix E.4. Overall, VisionWeave delivers a 2.30× end-to-end throughput speedup (Table 5). Both mean and tail latency improve: P95 time to first token (TTFT) and time per output token (TPOT) decrease by 57.1% and 59.6%, respectively. These results demonstrate that native elastic visual processing translates into higher serving throughput and lower response latency under this long-video workload. 11 Preprint 3.5INFERENCE-TIME CONTROL OF TOKEN SAVINGS The default threshold t = 0.5 already provides content-adaptive token savings without task-specific tuning. We further show that varying this threshold enables flexible inference-time control without retraining (Figure 8). At the full-retention endpoint, labeled t = 1 in the figure, every block uses its fine-grained representation, with no token savings. These results suggest that native visual weaving expands the model’s efficiency–quality operating range while preserving its fully fine-grained per- formance. Lowering the threshold then favors coarse-grained representations and increases token savings. On Video-MME, reducing t from 0.5 to 0.25 increases savings from 54.2% to 66.0% on Qwen3.8-27B, with only a 0.37-point performance loss. On Qwen3.5-4B, the same adjustment in- creases savings from 51.4% to 62.1% while preserving the score of 67.22. These results highlight a key practical advantage of native visual weaving: a single model can accommodate different quality and cost requirements without retraining, while closely matching the native model’s performance ceiling when token savings are disabled. 4ANALYSIS OF ELASTIC VISUAL WEAVING We examine the contributions of individual components and the spatial allocation of visual repre- sentations. 4.1ABLATION STUDIES Gated spatial pooler. We compare our pooler with a pixel-unshuffle-based projector on Qwen3.5- 4B after Stage 1, both at 75% token savings. Our pooler achieves higher scores on all eight bench- marks (Figure 7), including gains of 2.64 points on DocVQA and 2.77 points on VideoOCR. This supports gated spatial pooling for constructing coarse-grained visual representations. Learned granularity allocation. Using the main comparison’s 512-token image/frame input bud- gets, we shuffle block-wise routing scores within each image or video before applying t = 0.5. This preserves the number of coarse selections for each input, including input-dependent token savings, but breaks their alignment with visual content. Mean scores decrease by 2.31 and 5.36 points on Qwen3.5-4B and Qwen3.8-27B, respectively (Table 6). Even excluding ScreenSpotV2, the mean decreases remain 1.30 and 1.53 points. Thus, learning where to allocate each granularity matters beyond choosing how many tokens to retain. Table 6: Learned granularity allocation improves performance over shuffled routing. Qwen3.5-4BQwen3.8-27B BenchmarkVisionWeaveShuffled∆VisionWeaveShuffled∆ RealWorldQA73.4673.460.0077.9176.60 −1.31 DocVQA91.0388.70 −2.3291.3388.35 −2.98 InfoVQA66.1164.83 −1.2871.6169.04 −2.58 ScreenSpotV289.4780.03 −9.4391.4359.28 −32.15 HallusionBench69.2667.32 −1.9573.6071.66 −1.95 VideoOCR63.7961.85 −1.9570.5169.44 −1.08 Video-MME67.2266.78 −0.4471.0470.67 −0.37 LongVideoBench60.3659.24 −1.1266.4265.97 −0.45 Average72.5970.28 −2.3176.7371.37 −5.36 4.2VISUAL ANALYSIS OF GRANULARITY ALLOCATION Figure 9 compares VisionWeave and FastV † on GUI grounding and DocVQA examples using Qwen3.8-27B at a 512-token input budget. In the web interface, FastV † drops tokens covering click targets, including the star button, whereas VisionWeave keeps every region represented. In the document, VisionWeave saves 45.2% of tokens while representing larger text coarsely; FastV † removes tokens covering informative text, leaving some characters only partially represented. These 12 Preprint Original image FastV † VisionWeave (a) DesktopVisual tokens: 510 → 255Visual tokens: 480 → 183 (b) TabletVisual tokens: 486 → 243Visual tokens: 468 → 261 (c) WebVisual tokens: 480 → 240Visual tokens: 480 → 207 VisionWeave: Fine-grained visual representationsVisionWeave: Coarse-grained visual representations FastV † : Retained visual tokens FastV † : Dropped visual tokens Original image FastV † VisionWeave (a) DesktopVisual tokens: 510 → 255Visual tokens: 480 → 183 (b) TabletVisual tokens: 486 → 243Visual tokens: 468 → 261 (c) WebVisual tokens: 480 → 240Visual tokens: 480 → 207 VisionWeave: Fine-grained visual representationsVisionWeave: Coarse-grained visual representations FastV † : Retained visual tokens FastV † : Dropped visual tokens Original image FastV † VisionWeave (a) DesktopVisual tokens: 510 → 255Visual tokens: 480 → 183 (b) TabletVisual tokens: 486 → 243Visual tokens: 468 → 261 (c) WebVisual tokens: 480 → 240Visual tokens: 480 → 207 VisionWeave: Fine-grained visual representationsVisionWeave: Coarse-grained visual representations FastV † : Retained visual tokens FastV † : Dropped visual tokens Original image FastV † VisionWeave (a) Printed and handwritten formVisual tokens: 506 → 253Visual tokens: 484 → 265 (b) Dense numeric tableVisual tokens: 494 → 247Visual tokens: 468 → 468 VisionWeave: Fine-grained visual representationsVisionWeave: Coarse-grained visual representations FastV † : Retained visual tokens FastV † : Dropped visual tokens Original image FastV † VisionWeave (a) Printed and handwritten formVisual tokens: 506 → 253Visual tokens: 484 → 265 (b) Dense numeric tableVisual tokens: 494 → 247Visual tokens: 468 → 468 VisionWeave: Fine-grained visual representationsVisionWeave: Coarse-grained visual representations FastV † : Retained visual tokens FastV † : Dropped visual tokens Original image FastV † VisionWeave (a) DesktopVisual tokens: 510 → 255Visual tokens: 480 → 183 (b) TabletVisual tokens: 486 → 243Visual tokens: 468 → 261 (c) WebVisual tokens: 480 → 240Visual tokens: 480 → 207 VisionWeave: Fine-grained visual representationsVisionWeave: Coarse-grained visual representations FastV † : Retained visual tokens FastV † : Dropped visual tokens (a) GUI grounding (b) Document understanding Figure 9: VisionWeave maintains spatial coverage through mixed granularity, whereas FastV † drops tokens covering task-relevant content. examples illustrate how changing granularity maintains spatial coverage instead of leaving gaps, helping explain the robustness on grounding and information-dense inputs. Additional cases appear in Appendix B. 5CONCLUSION We introduced VisionWeave, establishing elastic visual representation weaving as a native capability of MLLMs through self-distillation alone. It combines content-adaptive token savings with robust efficiency–quality trade-offs across tasks, resolutions, and frame budgets. Our SGLang integration further translates these token savings into higher throughput and lower latency on modern serving infrastructure. These findings establish representational granularity as a learnable degree of freedom, validating content-adaptive computational effort as a feasible architectural principle. 6ACKNOWLEDGE This work was supported by Alibaba Research Intern Program. We would like to thank the Qwen Team at Alibaba Token Hub (ATH), Alibaba Group, for providing the computational resources and foundation models (Qwen) used in this research. REFERENCES Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, Chunsheng Wu, Huajie Tan, Chunyuan Li, Jing Yang, 13 Preprint Jie Yu, Xiyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, Ziyong Feng, Ziwei Liu, Bo Li, and Jiankang Deng. Llava-onevision-1.5: Fully open framework for democratized multimodal training, 2025. URL https://arxiv.org/abs/2509.23661. Shuai Bai, Yuxuan Cai, Rui-Zhe Chen, Ke qin Chen, Xiong-Hui Chen, Zesen Cheng, Liang-Hao Deng, Wei Ding, Rongyao Fang, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Qidong Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Ming-Sheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xue-Jing Liu, Jiawei Liu, Cheng-Long Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Lingchen Meng, Xin yi Ren, Sibo Song, Yu chen Sun, Jun Tang, Jianhong Tu, Jian-Qiang Wan, Peng Wang, Pengfei Wang, Qiu-Yue Wang, Yuxuan Wang, Tianbao Xie, Yihe Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bo-Wen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Botao Zheng, Humen Zhong, Jingren Zhou, Fanxi Zhou, Jingren Zhou, Yuanzhi Zhu, and Keming Zhu. Qwen3-vl technical report. ArXiv, abs/2511.21631, 2025a. URL https: //api.semanticscholar.org/CorpusID:283262018. Shuai Bai, Ke qin Chen, Xue-Jing Liu, Jia-Lin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jian- Qiang Wan, Peng-Fei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl techni- cal report. ArXiv, abs/2502.13923, 2025b. URL https://api.semanticscholar.org/ CorpusID:276449796. Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models, 2024. URL https://arxiv.org/abs/2405.17430. Junjie Chen, Xuyang Liu, Zichen Wen, Yiyu Wang, Siteng Huang, and Honggang Chen. Variation- aware vision token dropping for faster large vision-language models, 2026. URL https:// arxiv.org/abs/2509.01552. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, 2024. URL https://api. semanticscholar.org/CorpusID:268358224. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. ArXiv, abs/2306.15595, 2023. URL https: //api.semanticscholar.org/CorpusID:259262376. Rohan Choudhury, Jungeun Kim, Jinhyung Park, Eunho Yang, László A. Jeni, and Kris M. Kitani. Accelerating vision transformers with adaptive patch sizes. ArXiv, abs/2510.18091, 2025. URL https://api.semanticscholar.org/CorpusID:282246197. Long Cui, Weiyun Wang, Jie Shao, Zichen Wen, Gen Luo, Linfeng Zhang, Yanting Zhang, Yu Qiao, and Wenhai Wang. Vico: A training strategy towards semantic aware dynamic high- resolution. ArXiv, abs/2510.12793, 2025. URL https://api.semanticscholar.org/ CorpusID:282064724. DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence. 2026. URL https://api.semanticscholar.org/CorpusID:289623513. Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference. ArXiv, abs/2407.11550, 2024. URL https://api.semanticscholar.org/CorpusID:271218006. Yuan Feng, Hao-Yu Guo, Junlin Lv, S. Kevin Zhou, and Xike Xie.Taming the fragility of kv cache eviction in llm inference. ArXiv, abs/2510.13334, 2025a. URL https://api. semanticscholar.org/CorpusID:282102254. Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou.Criticalkv: Optimizing kv cache eviction from an output perturbation perspective. 2025b. URL https://api. semanticscholar.org/CorpusID:276161406. 14 Preprint Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents, 2025. URL https://arxiv.org/abs/2410.05243. Haoyu Guo, Yuan Feng, Junlin Lv, Mingjun Xiao, S Kevin Zhou, and Xike Xie. Videomm: Adaptive macro-micro inference for efficient video mllms, 2026. URL https://arxiv.org/abs/ 2609.16722. Yuhang Han, Xuyang Liu, Zihan Zhang, Pengxiang Ding, Junjie Chen, Donglin Wang, Honggang Chen, Qingsen Yan, and Siteng Huang. Filter, correlate, compress: Training-free token reduction for mllm acceleration, 2025. URL https://arxiv.org/abs/2411.17686. Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Ma- tryoshka query transformer for large vision-language models, 2024. URL https://arxiv. org/abs/2405.19315. Ahmadreza Jeddi, Negin Baghbanzadeh, Elham Dolatabadi, and Babak Taati. Similarity-aware token pruning: Your vlm but faster. ArXiv, abs/2503.11549, 2025. URL https://api. semanticscholar.org/CorpusID:277043961. Selim Kuzucu, Alessio Tonioni, Vasile Lup, Bernt Schiele, Federico Tombari, and Muhammad Fer- jad Naeem. Parcel: Pool-anchored resampling with conditioned elastic queries for efficient vision- language understanding, 2026. URL https://arxiv.org/abs/2605.30126. Xu-Yang Liu, Xiyan Gui, Yuchao Zhang, and Linfeng Zhang. Mixing importance with diver- sity: Joint optimization for kv cache compression in large vision-language models. ArXiv, abs/2510.20707, 2025.URL https://api.semanticscholar.org/CorpusID: 282304428. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.Yarn: Efficient context window extension of large language models. ArXiv, abs/2309.00071, 2023. URL https: //api.semanticscholar.org/CorpusID:261493986. Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, and Huan Wang. A survey of token compression for efficient multimodal large language models. Trans. Mach. Learn. Res., 2026, 2025a. URL https://api.semanticscholar. org/CorpusID:280323457. Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, and Huan Wang. When tokens talk too much: A survey of multimodal long- context token compression across images, videos, and audios. ArXiv, abs/2507.20198, 2025b. URL https://api.semanticscholar.org/CorpusID:290088866. Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke : Dynamic compres- sion of tokens for fast video large language models. 2025 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), p. 18992–19001, 2024. URL https://api. semanticscholar.org/CorpusID:274192345. Kimi Team.Kimi k3:Open frontier intelligence.2026.URL https://api. semanticscholar.org/CorpusID:290625162. Jiahui Wang, Zuyan Liu, Yongming Rao, and Jiwen Lu. Sparsemm: Head sparsity emerges from visual concept responses in mllms. 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 23177–23187, 2025. URL https://api.semanticscholar.org/ CorpusID:279244559. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. 2025 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), p. 19792–19802, 2024. URL https://api.semanticscholar.org/CorpusID:274514545. 15 Preprint Xubing Ye, Yukang Gan, Yi-Xiao Ge, Xiao-Ping Zhang, and Yansong Tang. Atp-llava: Adap- tive token pruning for large vision language models. 2025 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), p. 24972–24982, 2024. URL https://api. semanticscholar.org/CorpusID:274436316. Qiyang Yu, Yu Fang, Tianrui Li, Xuemei Cao, Yan Chen, Jianghao Li, and Fan Min.Dy- namic granularity matters: Rethinking vision transformers beyond fixed patch splitting. ArXiv, abs/2511.19021, 2025.URL https://api.semanticscholar.org/CorpusID: 283244585. Qizhe Zhang, Meng-Zhen Liu, Li-Chen Li, Ming Lu, Yuan Zhang, Junwen Pan, Qi She, and Shang- hang Zhang. Beyond attention or similarity: Maximizing conditional diversity for token pruning in mllms. ArXiv, abs/2506.10967, 2025a. URL https://api.semanticscholar.org/ CorpusID:279318547. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava- video: Video instruction tuning with synthetic data, 2025b. URL https://arxiv.org/ abs/2410.02713. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chu-Yue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christoforos E. Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems 37, 2023. URL https://api.semanticscholar.org/ CorpusID:266174771. 16 Preprint APPENDIX CONTENTS A Limitations and Future Works18 B Case Studies18 B.1 Case Study in GUI Grounding Tasks . . . . . . . . . . . . . . . . . . . . . . . . .18 B.2 Case Study in DocVQA Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 C More Visualizations of Elastic Visual Weaving18 D Related Work18 E Additional Experimental Details21 E.1 Ablation of Mixed-Granularity Self-Distillation . . . . . . . . . . . . . . . . . . .21 E.2 Ablation of Distillation Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . .23 E.3 Performance at Matched Token Savings . . . . . . . . . . . . . . . . . . . . . . .24 E.4 SGLang Serving Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 17 Preprint ALIMITATIONS AND FUTURE WORKS Our ablations examine on- versus off-policy self-distillation, the contribution of Stage 3, and the pooler architecture. However, the substantial computational cost of training limits a more compre- hensive exploration of architectural choices, including alternative router designs. These unexplored choices offer opportunities to further improve elastic visual weaving. This work focuses on two granularities, while we do not foresee significant barriers to extending our approach to diverse lev- els. We therefore expect that moving beyond the current two-level design to richer multi-scale representations could yield further gains. Moreover, whereas we currently implant this capability via post-training self-distillation, future work could explore introducing elastic visual weaving dur- ing base model pre-training. Our long-term vision is a new generation of multimodal foundation models that jointly develop visual understanding and efficiency, natively modulating computational effort by information content. BCASE STUDIES B.1CASE STUDY IN GUI GROUNDING TASKS Figure 10 compares visual token allocation on desktop, tablet, and web interfaces using Qwen3.8- 27B at a 512-token input budget. VisionWeave interleaves fine- and coarse-grained representations, keeping every region represented within a shared spatial frame. In contrast, FastV † drops tokens covering critical click targets, such as the star button in the web interface, leaving gaps in token coverage that can hinder accurate grounding. This spatially coherent coverage helps explain Vision- Weave’s substantial advantage over FastV † and VisionZip on grounding tasks. B.2CASE STUDY IN DOCVQA TASKS Figure 11 compares token allocation on two DocVQA examples using Qwen3.8-27B at a 512-token input budget. At the default thresholdt = 0.5, VisionWeave saves 45.2% of tokens on the form while retaining all native fine-grained tokens for the dense table. FastV † instead removes 50% of tokens in both cases, including tokens covering text. Even in example (a), FastV † drops tokens covering informative text, leaving some characters only partially represented. In contrast, VisionWeave uses coarse-grained representations for larger text while maintaining full spatial coverage. This also helps explain the robustness of VisionWeave across diverse inputs. CMORE VISUALIZATIONS OF ELASTIC VISUAL WEAVING Figures 12–15 show additional examples of elastic visual representations on Qwen3.5-4B and Qwen3.8-27B, illustrating how fine- and coarse-grained representations are interleaved according to visual content. DRELATED WORK Visual token pruning and merging. Visual token pruning and merging shorten visual sequences through heuristic or learned policies (Jeddi et al., 2025; Ye et al., 2024; Chen et al., 2026; Han et al., 2025; Zhang et al., 2025a), as exemplified by FastV and VisionZip (Chen et al., 2024; Yang et al., 2024). Direct pruning removes tokens deemed unimportant by the selection policy and can sub- stantially alter the visual context available to a pretrained MLLM. Our evaluations of FastV † and VisionZip show modest performance losses on video tasks but substantial degradation on grounding and information-dense inputs. Most methods also rely on preset compression ratios that cannot ac- commodate the varying information density of visual inputs. In addition, despite extensive research, visual token-pruning methods rarely provide implementations in modern serving engines such as SGLang or vLLM, leaving their practical efficiency gains insufficiently validated. Common algo- rithmic choices further hinder deployment: some methods require explicit attention weights, which standard FlashAttention kernels do not expose, while others prune tokens within the LLM, compli- cating engine integration and support for dynamic batching and chunked prefill. Consequently, the 18 Preprint Original image FastV † VisionWeave (a) DesktopVisual tokens: 510 → 255Visual tokens: 480 → 183 (b) TabletVisual tokens: 486 → 243Visual tokens: 468 → 261 (c) WebVisual tokens: 480 → 240Visual tokens: 480 → 207 VisionWeave: Fine-grained visual representationsVisionWeave: Coarse-grained visual representations FastV † : Retained visual tokens FastV † : Dropped visual tokens Figure 10: VisionWeave retains full spatial coverage through mixed-granularity representations, whereas FastV † drops visual tokens. growing body of token-pruning research has yet to translate into widespread adoption in practical LLM serving. In contrast, VisionWeave improves visual efficiency through content-adaptive se- lection of fine- and coarse-grained representations, using the same representation construction and routing during final-stage training and inference. Extensive evaluations on SGLang engine across tasks, input resolutions, and video frame budgets demonstrate consistently favorable efficiency– quality trade-offs, supporting elastic visual weaving as a robust native capability of MLLMs. Content-adaptive visual representations. At the vision-encoder level, prior work varies patch sizes using low-level image statistics, such as edge density and entropy (Yu et al., 2025; Choud- hury et al., 2025). These approaches primarily target standalone vision tasks rather than learn- ing granularity for diverse downstream multimodal objectives. At the MLLM level, Matryoshka Multimodal Models (Cai et al., 2024), Matryoshka Query Transformer (Hu et al., 2024) and PAR- CEL (Kuzucu et al., 2026) support multiple token budgets, but require the budget to be specified externally. ViCO (Cui et al., 2025) goes further by adaptively assigning a visual token resolution to each image tile. However, its tile-based design limits direct applicability to frontier MLLMs such as Qwen and Kimi, which use native-resolution visual processing. Moreover, interleaved detail-rich regions and low-information backgrounds call for finer spatial allocation than a single resolution per tile provides (Figure 16). VisionWeave instead learns block-wise allocation of fine- and coarse-grained representations within the native-resolution pipeline. Our contribution is to es- 19 Preprint Original image FastV † VisionWeave (a) Printed and handwritten formVisual tokens: 506 → 253Visual tokens: 484 → 265 (b) Dense numeric tableVisual tokens: 494 → 247Visual tokens: 468 → 468 VisionWeave: Fine-grained visual representationsVisionWeave: Coarse-grained visual representations FastV † : Retained visual tokens FastV † : Dropped visual tokens Figure 11: VisionWeave adapts visual granularity to document content rather than imposing fixed token savings. (a) RealWorldVisual Tokens 1728 → 765 (−55.7%)(b) TextVisual Tokens 1976 → 1346 (−31.9%) (c) DrivingVisual Tokens 1380 → 651 (−52.8%)(d) ChartVisual Tokens 1900 → 979 (−48.5%) Fine-graned representationCoarse-gained representation Figure 12: Elastic visual representations on Qwen3.5-4B at a 2048-token input budget. tablish this content-adaptive visual processing as a native capability of frontier MLLMs, combining autonomous token savings, robust efficiency–quality trade-offs, and compatibility with modern serv- ing infrastructure. 20 Preprint (a) RealWorldVisual Tokens 936 → 480 (−48.7%)(b) TextVisual Tokens 988 → 952 (−3.6%) (c) DrivingVisual Tokens 912 → 522 (−42.8%)(d) ChartVisual Tokens 1008 → 582 (−42.3%) Fine-graned representationCoarse-gained representation Figure 13: Elastic visual representations on Qwen3.8-27B at a 1024-token input budget. KV-cache compression. KV-cache compression addresses a related but distinct aspect of infer- ence efficiency. Visual token efficiency shortens the sequence processed by the LLM, reducing both attention and feed-forward computation during prefill, as well as KV-cache memory, thereby en- abling faster time to first token Feng et al. (2024; 2025b;a); Liu et al. (2025); Wang et al. (2025). In contrast, KV-cache compression compacts the stored key–value states without shortening the input token sequence. When applied after prefill, it primarily reduces memory usage and decod- ing bandwidth rather than prompt-processing computation. The distinction is therefore between processing fewer tokens and retaining more compact states for those tokens. Learned KV com- pression, exemplified by DeepSeek-V4 (DeepSeek-AI, 2026), extends this direction by integrating compression into attention itself. Combining such mechanisms with elastic visual representation weaving is a promising direction: fewer visual tokens would reduce computation throughout the LLM, while learned KV compression could further lower the cache memory footprint of dense- or sparse-attention layers. EADDITIONAL EXPERIMENTAL DETAILS E.1ABLATION OF MIXED-GRANULARITY SELF-DISTILLATION We compare Qwen3.8-27B before and after Stage 3 (Table 7). Stage 3 improves seven of eight benchmarks and raises the average from 73.15 to 76.73. ScreenSpotV2 shows the largest gain (66.59 to 91.43); excluding it, the average still improves by 0.54 points, despite a 1.18-point decrease on RealWorldQA. This supports self-distillation on the hard-routed mixed-granularity sequences used at inference. Table 7: Stage 3 improves Qwen3.8-27B performance on seven of eight benchmarks. BenchmarkWithout Stage 3With Stage 3∆ (points) RealWorldQA79.0877.91−1.18 DocVQA90.7691.33+0.56 InfoVQA70.6671.61+0.96 ScreenSpotV266.5991.43+24.84 HallusionBench72.1073.60+1.51 VideoOCR69.8570.51+0.67 Video-MME70.4471.04+0.59 LongVideoBench65.7466.42+0.67 Average73.1576.73+3.58 21 Preprint Fine-graned representationCoarse-gained representation OriginalAdaptive · Visual tokens: 900 → 384OriginalAdaptive · Visual tokens: 832 → 343 OriginalAdaptive · Visual tokens: 1976 → 1055OriginalAdaptive · Visual tokens: 1280 → 578 OriginalAdaptive · Visual tokens: 1380 → 672OriginalAdaptive · Visual tokens: 1380 → 579 OriginalAdaptive · Visual tokens: 1380 → 573OriginalAdaptive · Visual tokens: 1380 → 636 OriginalAdaptive · Visual tokens: 1932 → 885OriginalAdaptive · Visual tokens: 576 → 339 OriginalAdaptive · Visual tokens: 1728 → 744OriginalAdaptive · Visual tokens: 1728 → 774 OriginalAdaptive · Visual tokens: 504 → 210OriginalAdaptive · Visual tokens: 252 → 99 OriginalAdaptive · Visual tokens: 196 → 70OriginalAdaptive · Visual tokens: 1900 → 916 Figure 14: Additional Qwen3.5-4B visualizations at a 2048-token input budget. 22 Preprint Fine-graned representationCoarse-gained representation OriginalAdaptive · Visual tokens: 900 → 525OriginalAdaptive · Visual tokens: 832 → 463 OriginalAdaptive · Visual tokens: 1976 → 848OriginalAdaptive · Visual tokens: 1280 → 725 OriginalAdaptive · Visual tokens: 1380 → 840OriginalAdaptive · Visual tokens: 1380 → 705 OriginalAdaptive · Visual tokens: 1380 → 669OriginalAdaptive · Visual tokens: 1380 → 774 OriginalAdaptive · Visual tokens: 1932 → 963OriginalAdaptive · Visual tokens: 576 → 366 OriginalAdaptive · Visual tokens: 1728 → 918OriginalAdaptive · Visual tokens: 1728 → 969 OriginalAdaptive · Visual tokens: 504 → 273OriginalAdaptive · Visual tokens: 252 → 189 OriginalAdaptive · Visual tokens: 196 → 100OriginalAdaptive · Visual tokens: 1900 → 1171 Figure 15: Qwen3.8-27B visualizations corresponding to Figure 14, with the same 2048-token input budget. E.2ABLATION OF DISTILLATION POLICY We also compare on- and off-policy self-distillation in Stage 3 of Qwen3.5-4B (Figure 17). The two variants achieve similar mean scores across eight benchmarks, each leading on four. We therefore use off-policy self-distillation for training efficiency. 23 Preprint Where is visual granularity allocated? ViCOVisionWeave InternVL3.5-8B | tile-level selectionQwen3.8-27B | block-level selection 256 or 64 tokens per tile4 fine-grained tokens or 1 coarse-grained token Document (a) RealWorldVisual Tokens 1728 → 867 (−49.8%)(b) TextVisual Tokens 1976 → 1577 (−20.2%) (c) DrivingVisual Tokens 1380 → 840 (−39.1%)(d) ChartVisual Tokens 1900 → 1054 (−44.5%) Fine-grained visual representationsCoarse-gained representation Fig. 4, row 6, column 2Figure 1(b), Text, routed panel Driving (a) RealWorldVisual Tokens 1728 → 867 (−49.8%)(b) TextVisual Tokens 1976 → 1577 (−20.2%) (c) DrivingVisual Tokens 1380 → 840 (−39.1%)(d) ChartVisual Tokens 1900 → 1054 (−44.5%) Fine-grained visual representationsCoarse-gained representation Fig. 3, row 2, column 4Figure 1(c), Driving, routed panel Shaded: 64 tokens; unshaded: 256 tokensBlue grid: fine-grained; gray grid: coarse-grained Different inputs, backbones, and preprocessing; a granularity illustration, not a matched performance comparison. ViCO panels reproduced from Cui et al. (2025), Figs. 4 and 3, C BY 4.0; cropped, annotations unchanged. Figure 16: VisionWeave allocates visual granularity at a finer spatial scale than tile-level routing. ViCO panels are cropped from Cui et al. (2025) DocVQA InfoVQA ScreenSpotV2 RealWorldQA HallusionBench VideoOCR Video-MME LongVideoBench 60 70 80 90 100 Score 90.95 66.51 88.76 71.90 66.52 65.03 67.48 60.51 91.03 66.11 89.47 73.46 69.26 63.79 67.22 60.36 On-policyOff-policy Figure 17: On- and off-policy self-distillation yield comparable performance on Qwen3.5-4B. E.3PERFORMANCE AT MATCHED TOKEN SAVINGS Table 8 gives the per-task results for the matched-savings comparison in Section 3.2. FastV † and VisionZip match VisionWeave’s dataset-level savings at t = 0.5 for each backbone and dataset. 24 Preprint Table 8: Performance at matched dataset-level token savings. Upper boundMatched savings Qwen3.5-4B512 tokensFastV † VisionZipVisionWeave RealWorldQA74.6471.7672.4273.46 DocVQA91.9291.1691.0991.03 InfoVQA67.4466.6665.6866.11 ScreenSpotV289.8687.3486.0889.47 HallusionBench69.8869.0969.4469.26 VideoOCR64.5163.5963.2863.79 Video-MME67.7867.5667.48 67.22 LongVideoBench59.9960.4359.7660.36 Average73.2572.2071.9072.59 Qwen3.8-27B RealWorldQA78.8278.1778.4377.91 DocVQA92.4788.2688.5191.33 InfoVQA74.1371.2971.0771.61 ScreenSpotV294.4243.7945.68 91.43 HallusionBench73.2572.5472.5473.60 VideoOCR70.7269.7969.4470.51 Video-MME70.8971.1571.1571.04 LongVideoBench66.4965.8265.8266.42 Average77.6570.1070.3376.73 ScreenSpotV2 contributes substantially to the 27B average gap; excluding it, mean scores remain 74.63 for VisionWeave versus 73.86/73.85 for FastV † /VisionZip. E.4SGLANG SERVING CONFIGURATION The experiments in Section 3.4 use our SGLang integration, Python 3.12.3, PyTorch 2.13.0+cu129, Transformers 5.12.1, and Triton 3.7.1. Each model runs on two A100-SXM4-80GB GPUs with LLM tensor parallelism of two and one TP=1 vision encoder per GPU, providing encoder data parallelism of two. Both models use BF16, Triton full/linear/ViT attention, and NCCL with custom all-reduce disabled. The context limit is 147,456 tokens, with 4,096-token prefill chunks. Static memory fractions are 0.55 for the language service and 0.06 for each encoder. FCFS and overlap scheduling are enabled; mixed prefill/decode chunks are disabled. Padded decode CUDA graphs capture batch sizes 1, 2 and 4; prefill graphs are disabled. Client concurrency is capped at four outstanding requests. 25