Paper deep dive
ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning
Huanxuan Liao, Zhongtao Jiang, Yupu Hao, Yuqiao Tan, Shizhu He, Jun Zhao, Kun Xu, Kang Liu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/31/2026, 2:46:56 AM
Summary
ResAdapt is an input-side adaptation framework for Multimodal Large Language Models (MLLMs) that optimizes visual token efficiency by dynamically allocating visual budgets per frame before encoding. It uses a lightweight Allocator trained via Cost-Aware Policy Optimization (CAPO) and a temporal-similarity regularizer to achieve an improved efficiency-accuracy Pareto frontier, allowing for significantly longer video context processing without modifying the backbone architecture.
Entities (4)
Relation Signals (2)
ResAdapt → utilizes → CAPO
confidence 95% · train the Allocator with Cost-Aware Policy Optimization (CAPO)
ResAdapt → improves → MLLM
confidence 90% · ResAdapt improves low-budget operating points and often lies on or near the efficiency-accuracy frontier
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal Large Language Models (MLLMs) achieve stronger visual understanding by scaling input fidelity, yet the resulting visual token growth makes jointly sustaining high spatial resolution and long temporal context prohibitive. We argue that the bottleneck lies not in how post-encoding representations are compressed but in the volume of pixels the encoder receives, and address it with ResAdapt, an Input-side adaptation framework that learns how much visual budget each frame should receive before encoding. ResAdapt couples a lightweight Allocator with an unchanged MLLM backbone, so the backbone retains its native visual-token interface while receiving an operator-transformed input. We formulate allocation as a contextual bandit and train the Allocator with Cost-Aware Policy Optimization (CAPO), which converts sparse rollout feedback into a stable accuracy-cost learning signal. Across budget-controlled video QA, temporal grounding, and image reasoning tasks, ResAdapt improves low-budget operating points and often lies on or near the efficiency-accuracy frontier, with the clearest gains on reasoning-intensive benchmarks under aggressive compression. Notably, ResAdapt supports up to 16x more frames at the same visual budget while delivering over 15% performance gain. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.28610v1
- Canonical: https://arxiv.org/abs/2603.28610v1
Trouble viewing inline? Open PDF directly →
Full Text
109,898 characters extracted from source content.
Expand or collapse full text
ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Huanxuan Liao τ, μ , Zhongtao Jiang, Yupu Hao τ, μ , Yuqiao Tan τ, μ , Shizhu He τ, μ , Jun Zhao τ, μ , Kun Xu † , Kang Liu τ, μ,∗ τ Institute of Automation, Chinese Academy of Sciences , μ University of Chinese Academy of Sciences , † Project Leader ∗ Corresponding author: kliu@nlpr.ia.ac.cn Multimodal Large Language Models (MLLMs) achieve stronger visual understanding by scaling input fidelity, yet the resulting visual token growth makes jointly sustaining high spatial resolution and long temporal context prohibitive. Existing efficiency strategies only partially resolve this tension: model-side token compression discards fine-grained evidence after encoding and can disrupt optimized inference kernels, whereas output-side agentic reasoning adds iterative latency and can still miss decisive cues when the initial view is too coarse. We argue that the bottleneck lies not in how post-encoding representations are compressed but in the volume of pixels the encoder receives, and address it with ResAdapt, an Input-side adaptation framework that learns how much visual budget each frame should receive before encoding. ResAdapt couples a lightweight Allocator with an unchanged MLLM backbone, so the backbone retains its native visual-token interface while receiving an operator-transformed input. We formulate allocation as a contextual bandit and train the Allocator with Cost-Aware Policy Optimization (CAPO), which converts sparse rollout feedback into a stable accuracy–cost learning signal. We further introduce a temporal-similarity regularizer that suppresses redundant high-budget allocation on adjacent similar frames, encouraging differentiated, content-aware allocation in a single forward pass. Across budget-controlled video QA, temporal grounding, and image reasoning tasks, ResAdapt improves low-budget operating points and often lies on or near the efficiency–accuracy frontier, with the clearest gains on reasoning-intensive benchmarks under aggressive compression. Notably, ResAdapt supports up to16×more frames at the same visual budget while delivering over 15% performance gain. The learned policy exhibits open-loop active perception, concentrating visual budget on information-dense content without modifying the backbone architecture. These results position Input-side adaptation as a practical and effective route to long-context video reasoning under tight visual budgets. Project Page: https://xnhyacinth.github.io/projects/ResAdapt Code Repository: https://github.com/Xnhyacinth/ResAdapt Contact: liaohuanxuan2023@ia.ac.cn 1. Introduction Multimodal Large Language Models (MLLMs) achieve stronger visual understanding by scaling input fidelity, yet the resulting visual-token growth makes jointly sustaining high spatial resolution and long temporal context prohibitive (Guo et al., 2025a, Bai et al., 2025a, Liu et al., 2025a, Shu et al., 2025, Shao et al., 2025b). In practice, this trade-off is central to video reasoning: reducing resolution risks losing the small visual cues that determine the answer, whereas shortening the clip removes the temporal context needed for long-horizon inference. Even architecturally efficient encoders (Zhang et al., 2026, Liu et al., 2025b) do not remove this tension; they merely shift where it becomes painful. arXiv:2603.28610v1 [cs.CV] 30 Mar 2026 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Merge / Prune / Sparse Question: Why did the boy lift up the towel near the end? A. pick up coat. B. put on the plate. C. lose balance. Tool Call Multi-turn InteractionZoom-in Search ... Full Visual Tokens Compressed Visual Tokens Allocations Input-side Adaptive Sampling (Source-Level) Model-Side Token Economy (Representation-Level) Output-side Agentic Reasoning (Behavior-Level) Allocator (a) Input-side Adaptation vs. Existing Paradigms(b) ResAdapt achieves competitive performance Operator Resize Frame Select Figure 1: Input-side Adaptation improves the visual-token efficiency frontier. (a) Three efficiency paradigms for video reasoning. Model-side methods compress tokens after encoding; output-side methods iteratively retrieve or zoom; ResAdapt reallocates per-frame visual budget before encoding, preserving the backbone’s native token interface and compatibility with optimized inference engines. (b) Qwen2.5-VL-7B results with 32 frames at∼10% visual retention, where ResAdapt lies on or near the Pareto frontier and shows its largest gain on the reasoning-heavy benchmark. Mainstream efficiency methods largely fall into two paradigms (Figure 1a), both of which intervene too late. Model-side approaches prune or merge tokens after visual encoding (Khaki et al., 2025, Xu et al., 2025, Bolya et al., 2022, Tao et al., 2025), so once fine-grained evidence is discarded, it cannot be recovered. They also alter the token layout expected by the backbone, which can complicate direct use of optimized attention kernels and inference engines (Dao, 2024, Kwon et al., 2023, Zheng et al., 2024). Output-side agentic reasoning instead adds iterative retrieval or zoom steps (Zhang et al., 2025b, Yang et al., 2025d, Shen et al., 2025b, Zheng et al., 2025b). This strategy can improve coverage, but it introduces multi-turn latency and still depends on an initial coarse view that may undersample the decisive cue. These limitations point to a different bottleneck: not how post-encoding representations are compressed, but how many pixels the encoder receives in the first place. This paper studies that perspective through Input- side adaptation, which reallocates visual budget before encoding rather than compressing representations afterward. Our method, ResAdapt, introduces a lightweight Allocator that predicts a per-frame visual allocation from coarse visual features and the query, then realizes that allocation through a visual budget operator, such as resolution resizing or frame selection. The backbone therefore processes a standard visual- token sequence in a single pass, preserving its native interface and compatibility with existing optimized inference stacks (Dao, 2024, Kwon et al., 2023). Compared with prior slow–fast pipelines (Yang et al., 2025a, Zhang et al., 2026), which route frames using query-agnostic heuristics or fixed resolution tiers, ResAdapt learns a query-aware allocation policy directly from task reward. Making this idea work requires more than a front-end allocator. The allocation space is continuous, the operator is non-differentiable, and naive accuracy–cost penalties collapse toward uniformly tiny budgets. We address these difficulties with Cost-Aware Policy Optimization (CAPO), which converts sparse rollout 2 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning feedback into a stable asymmetric learning signal, and with a temporal-similarity regularizer that discourages redundant high-budget allocation on adjacent similar frames. Together, these components turn Input-side adaptation into a trainable and content-aware policy rather than a handcrafted compression rule. Across video QA and temporal grounding benchmarks on multiple backbones and temporal horizons, ResAdapt improves or closely tracks the efficiency–accuracy Pareto frontier. ResAdapt matches or surpasses other token economy methods while compressing over 90% of visual tokens (Figure 1b), and the saved compute reinvests as temporal coverage: under equivalent budgets, ResAdapt processes16×more frames with>15% relative gains. The learned policy exhibits open-loop active perception, concentrating visual budget on information-dense frames in a single backbone pass without explicit saliency supervision. Our main contributions are: 1.We introduce ResAdapt, an input-side adaptation framework that formulates dynamic per-frame vi- sual budgeting as a contextual bandit problem, fully preserving the native architecture and hardware optimizations of MLLMs. 2.We propose CAPO with a temporal similarity regularizer, providing a stable, asymmetric learning signal to jointly optimize accuracy and cost without hand-crafted heuristics. 3.Through extensive experiments and ablations, we show that ResAdapt achieves better efficiency–accuracy Pareto frontier across video QA and temporal grounding tasks. 2. Background and Problem Formulation 2.1. Preliminaries Given a text queryqand a videoV = f t T t=1 , letx = (q,V)denote the full input. A backbone policyπ φ encodes every frame at fixed fidelity and autoregressively generates a rollout y = (y 1 , . . . , y L ): π φ (y| x) = L ∏︁ j=1 π φ (y j | y <j , x).(1) When useful, we writey = (r, o)for a reasoning tracerand a final answero. The inefficiency is immediate: visual cost scales with total pixel volume, whereas answer-critical evidence is sparse in time. To control pre-encoding cost, we introduce an Allocator policy π θ that emits a per-frame allocation vector s = (s 1 , . . . , s T )∼ π θ (·| x),s t ∈ [s min , s max ],(2) and applies a visual budget operatorOto each frame: ̃ f t =O( f t , s t ) . The backbone then generates from the transformed input ̃x = (q, ̃ f t T t=1 ): π φ (y| ̃x) = L ∏︁ j=1 π φ (y j | y <j , ̃x).(3) We keepOabstract only to state the decision problem cleanly. The framework is operator-agnostic:Omay implement resizing, frame selection, or other pre-encoding budget controls. 3 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning 2.2. Problem Formulation Because the Allocator acts once before decoding, the outer problem is a Contextual Bandit (equivalently, a one-step contextual MDP). The context is the raw inputx∈X, and the action is the continuous allocation vector s∈ [s min , s max ] T . For joint training, it is convenient to write the induced two-stage policy as p θ,φ (s, y| x) = π θ (s| x) π φ (y| ̃x),(4) where ̃x = (q,O( f t , s t ) T t=1 )is the deterministically transformed input. The immediate reward is response quality r(x, s, y) = Q(x, y). Let C(s) denote the visual cost induced by allocation s. The ideal budgeted objective is max θ,φ E x∼D, s∼π θ (·|x), y∼π φ (·| ̃x) [ Q(x, y) ] s.t. E x∼D, s∼π θ (·|x) [ C(s) ] ≤ τ, (5) where τ is the target budget. Lagrangian relaxation yields the unconstrained utility max θ,φ E x,s,y [ U(x, s, y) ] , U(x, s, y) = Q(x, y)− λ C(s), (6) for trade-off coefficient λ≥ 0. Equations(5)–(6)define the target trade-off but not yet a stable optimizer. Section 3 instantiates this objective with an Input-side adaptation policy, CAPO, temporal regularization, and PPO-style surrogate losses; the experiments use resize as the concrete operator. Detailed derivations are deferred to Appendix C. 3. Method Figure 2 summarizes the Input-side adaptation framework. At inference, the Allocator predicts one allocation per frame and applies a pre-encoding operator before the video reaches the backbone in a single pass. In the experimental instantiation studied here,Ois bilinear resizing, so the allocation becomes a resize factors t and ̃ f t =R( f t , s t ). At training, rollout feedback updates the Allocator and, optionally, the backbone. 3.1. Joint RL Optimization Framework Section 2.2 defines allocation as a contextual bandit and states the ideal accuracy–cost trade-off. Appendix C starts from the marginal probability of the correct answer under the transformed input and motivates a one-step expected-reward objective by abstracting the resulting scalar quality term as a rollout utilityQ(x, y) that is treated as parameter-independent after samplingy. For a fixed contextx, the induced joint policy factorizes as p θ,φ (s, y| x) = π θ (s| x) π φ (y| ̃x).(7) Hereπ θ (s| x)is the density onsinduced by the latent Beta policyq θ (a| x)in Eq.(10)(change of variables; the per-frame affine map has aθ-independent Jacobian, so∇ θ log π θ (s| x)matches∇ θ log q θ (a| x)). The PPO ratios in Eq.(21)are therefore computed on the latent actionsa t as in Eq.(11). The corresponding ideal rollout reward is R ideal s,y = Q(x, y)− λ C(s),(8) 4 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Allocator Visual Input Low Resolution Text Query Operator ... s (1) s (2) s (M) MLLM ... o (1,1) o (1,N) o (1,2) MLLM ... o (M,1) o (M,N) o (M,2) MLLM ... o (2,1) o (2,N) o (2,2) ... ... R (1,1) R (1,N) R (1,2) ... R (2,1) R (2,N) R (2,2) ... R (M,1) R (M,N) R (M,2) ... A (1,1) A (1,N) A (1,2) ... A (2,1) A (2,N) A (2,2) ... A (M,1) A (M,N) A (M,2) ... c (M) c (2) c (1) A (M) A (2) A (1) Base Advantage for MLLM Efficiency Shaping Regularization Long Video Inputs Allocator Text Encoder Visual Encoder Spatial Attention Temporal Attention Cross Attention MLP K V Q Predictor Block Action Head Beta Dist. Linear Transform Sample ~ Beta Mean Text Query Cost-Aware Policy Optimization Dynamic Pivot New Visual Question: Why did the boy lift up the towel near the end? A. pick up coat. B. put on the plate. C. lose balance. Info Fusion Entropy / Sim... Allocations Figure 2: ResAdapt framework. (a) At inference, a lightweight Allocatorπ θ maps coarse visual features and the query to latent actionsa t ∼ Beta(α t , β t ), which parameterize per-frame input allocations. In the resize instantiation used in our experiments, these allocations are realized as scaless t ∈ [s min , s max ], and the resized frames are processed by the MLLM in a single call. (b) During training, CAPO reshapes group-relative advantages with a dynamic cost pivot τ dyn , while temporal-similarity regularization suppresses redundant high-budget allocation on adjacent similar frames. and the ideal optimization target is max θ,φ J (θ, φ) = E π θ (s|x) [︁ E π φ (y| ̃x) [︁ R ideal s,y ]︁]︁ .(9) Equation(9)conditions on a fixedx; training averages overx ∼ D. Although its gradients follow the standard score-function form detailed in Appendix C, direct optimization remains brittle for three reasons: 1.Policy parameterization.π θ must emit aT-dimensional continuous action with negligible overhead relative to the backbone. 2.Credit assignment. Delayed rollout reward must yield low-variance, cost-aware gradient estimates; the raw Lagrangian rewardQ(x, y)− λC(s)is high-variance and often collapses the policy to minimum budget. 3.Local temporal structure. Rollout-level reward carries no explicit temporal signal, so near-duplicate adjacent frames can still receive redundant allocation. The next three subsections resolve these bottlenecks in order. 3.2. Allocator Architecture This subsection resolves Bottleneck 1. Equation(9)requires a continuous allocation policy whose overhead is negligible relative to the backbone it controls. The framework itself only assumes a policy over operator 5 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning parameters; below we describe the continuous resize instantiation used for training and evaluation. Each framef t ∈ R 3×H t ×W t is encoded by a frozen lightweight visual encoder, while the query is encoded separately. Both are projected to a shared dimensionD. A shallow decoder alternates temporal self-attention over f t T t=1 with gated cross-attention to the query, producing hidden statesh t T t=1 . This design exposes both temporal redundancy and query dependence at low cost. To preserve exploration, we parameterize each latent action with a Beta distribution whose bounded support maps naturally to [s min , s max ]: a t ∼ Beta(α t , β t ),s t = s min + a t (s max − s min ).(10) Becausea t ∈ (0, 1), the allocation satisfiess t ∈ (s min , s max )almost surely; in our experiments,0< s min < 1< s max permits both downscaling and selective upscaling. Letq θ (a| x)denote the latent Beta policy over a = (a 1 , . . . , a T ). Conditioned onh t , it factorizes across frames: log q θ (a| x) = T ∑︁ t=1 log Beta(a t ; α t , β t ).(11) The affine mapa7→ sinduces the corresponding policyπ θ (s| x); the exact change-of-variables details are deferred to Appendix C. The result is a continuous, query-aware allocation policy. 3.3. Cost-Aware Policy Optimization (CAPO) This subsection resolves Bottleneck 2. The Allocator emits a structured allocation, but the optimizer needs a scalar cost signal compatible with rollout reward. A flat penalty onC(s)collapses the policy toward uniformly tiny budgets. CAPO avoids this failure by replacing the raw penalty with a shaped surrogate learning signal. Compute metric. For the resize instantiation used in our experiments, if framef t ∈ R 3×H t ×W t is resized by s t , its visual token count isn t (s t )∝⌈s t H t /P⌉⌈s t W t /P⌉for patch sizeP. Physical compute is measured by the token retention ratio ρ(s) = ∑︀ T t=1 n t (s t ) ∑︀ T t=1 n t (1) ≈ ∑︀ T t=1 s 2 t H t W t ∑︀ T t=1 H t W t .(12) In our implementation, frames are normalized to a common base resolution before allocation, soρ(s)is well approximated by the average quadratic scale. Proxy cost. The quadratic dependence ofρons t over-emphasizes a few large allocations and produces high-variance updates. We therefore introduce the smoother proxy c(s) = ̄ s− s min s max − s min , ̄ s = 1 T T ∑︁ t=1 s t ,(13) used only inside the optimizer; ρ(s) remains the efficiency metric reported in experiments. Notation bridge. During training,R task m,n denotes the concrete rollout score,A base m,n the GRPO-normalized advantage, andA m,n the final CAPO-shaped advantage. Appendix C relates these practical quantities to the ideal reward in Eq. (8). Base advantage. For each promptx, letR task m,n denote the scalar task reward of rollout(m, n),A base m,n the corresponding GRPO group-normalized advantage,c m = c(s m )the proxy cost of allocationm, andu m,n ∈ 0, 1 a binary correctness indicator (exact-match for QA; thresholded success for continuous metrics). 6 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Dynamic cost pivot. A fixed threshold does not track the evolving policy, whereas a purely group-dependent statistic is too noisy. CAPO therefore interpolates between a fixed target and the prompt-local mean: τ dyn = κ mix ̄ c group + (1− κ mix ) τ fix ,(14) where ̄ c group = 1 M ∑︀ M m=1 c m is the prompt-local cost mean,τ fix ∈ [0, 1]is a fixed target proxy budget corresponding to Eq. (5), and κ mix ∈ [0, 1] controls adaptivity. Asymmetric shaping. With τ dyn as pivot, CAPO applies a correctness-dependent bonus or penalty: S m,n = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ λ + σ (︂ τ dyn − c m τ s )︂ if u m,n = 1, −λ − σ (︂ c m − τ dyn τ s )︂ if u m,n = 0, ,(15) withλ − > λ + > 0. Efficient correct rollouts receive a moderate bonus, whereas costly incorrect rollouts receive a stronger penalty. The sigmoid temperature τ s smooths the transition around the pivot. Final CAPO advantage. Let ̃ A m,n = A base m,n + λ capo S m,n − γ c m .(16) The final advantage is A m,n = ⎧ ⎨ ⎩ max (︀ ̃ A m,n , ε + )︀ if u m,n = 1, ̃ A m,n if u m,n = 0, (17) whereλ capo > 0scales CAPO shaping,γ ≥ 0applies a residual global cost penalty, and the floorε + > 0 ensures that correct low-cost rollouts retain a positive learning signal. The dominant anti-collapse term is the pivoted asymmetric shaping in S m,n . 3.4. Regularization and Training Objective This subsection resolves Bottleneck 3 and assembles the final optimizer. Temporal similarity loss (L sim ). CAPO optimizes the global quality–efficiency trade-off but does not penalize redundant high-budget allocation on near-duplicate adjacent frames. Reusing the coarse featuresf t from Sec. 3.2, we penalize such pairs by L sim = 1 T− 1 T−1 ∑︁ t=1 w t · max ( 0, log s t + log s t+1 + η sim ) ,(18) where the similarity-gated weight w t = σ (︂ cos( f t , f t+1 )− τ sim γ sim )︂ (19) activates only when adjacent frames exceed a cosine-similarity thresholdτ sim ∈ (0, 1), with temperature γ sim . No penalty is incurred when s t s t+1 ≤ e −η sim . Concentration loss (L con ). To prevent the Beta distributions from collapsing to near-deterministic spikes, we softly cap the total concentration at κ max > 0: L con = 1 T T ∑︁ t=1 max ( 0, α t + β t − κ max ) .(20) 7 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Together,L sim encourages differentiated allocation across redundant neighbors, whileL con preserves explo- ration. Practical training objective. We optimize both policies in a single GRPO-style loop (Zheng et al., 2025a, Yu et al., 2025). For each promptx, the Allocator samplesMallocation trajectoriess 1:M ; each transformed input ̃x (m) then producesNresponse rollouts. CAPO computes rollout advantagesA m,n , which serve as the shared learning signal for both policies. The exact PPO approximation is deferred to Appendix C. Allocator objective. We first aggregate rollout advantages per allocation,A CAPO m = 1 N ∑︀ n A m,n ,and optimize the per-frame PPO surrogate L θ =− 1 MT M ∑︁ m=1 T ∑︁ t=1 min (︁ r (m) θ,t A CAPO m , clip (︀ r (m) θ,t , 1−ε, 1+ε )︀ A CAPO m )︁ ,(21) where the per-frame importance ratio is r (m) θ,t = q θ (a (m) t | x) q θ old (a (m) t | x) .(22) The full Allocator loss is L alloc =L θ + λ sim L sim + λ con L con .(23) Backbone update. Conditioned on the sampled allocations, the backbone is updated with the standard token-level PPO approximation L φ =− 1 MN M ∑︁ m=1 N ∑︁ n=1 1 L m,n L m,n ∑︁ j=1 min (︁ r (m,n) φ,j A m,n , clip (︀ r (m,n) φ,j , 1−ε, 1+ε )︀ A m,n )︁ ,(24) where L m,n is the rollout length and r (m,n) φ,j = π φ (y (m,n) j | y (m,n) <j , ̃x (m) ) π φ old (y (m,n) j | y (m,n) <j , ̃x (m) ) .(25) If the backbone is frozen, we omitL φ ; jointly updating both networks corresponds to ResAdapt-RL. In practice,L alloc andL φ are optimized alternately within the same loop. 4. Experiments 4.1. Setup Implementation. The Allocatorπ θ uses the SmolVLM architecture (Marafioti et al., 2025) for high- throughput front-end prediction. Throughout, we instantiate input-side allocation with resize, so the learned allocations are realized as per-frame resize factors. We train the Allocator on Qwen2.5-VL-7B-Instruct (Bai et al., 2025b) and additionally test transfer to Qwen3-VL-8B-Instruct (Bai et al., 2025a). We report two settings: ResAdapt-RL, obtained by jointly updating the Allocator and the backbone, and ResAdapt, which directly reuses the trained Allocator with a frozen backbone to evaluate plug-and-play generalization. Resize is used during training because it provides the continuous action space required by our optimizer; thresholded 8 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Table 1: Evaluation Results on Video QA Benchmarks. Retention ratioRreflects visual token count; Reasoning (✓/✗) indicates chain-of-thought use; bold marks the best result. ResAdapt yields larger gains on the reasoning benchmark than on the perception benchmarks. Backbone Method Retention Ratio R Reasoning Video Perception BenchmarkVideo Reasoning Benchmark VideoMME LongVideoBench MMVU MLVU VideoMMMULVBench Qwen2.5-VL-7B 32 Frames Vanilla100%✗62.058.952.763.149.638.6 Random Drop25.0%✗58.957.849.658.345.336.7 ToMe (Bolya et al., 2022)25.0%✗58.758.051.058.741.837.7 VisionZip (Yang et al., 2025c)25.0%✗59.457.149.857.942.436.5 FlashVid (Fan et al., 2026)29.3%✗60.258.651.159.246.336.9 FixedScale25.0%✗60.056.851.259.846.737.3 ResAdapt (Ours)23.8%✗60.358.251.960.148.837.9 Random Drop10.0%✗56.155.647.156.539.835.2 ToMe (Bolya et al., 2022)10.0%✗56.455.248.958.039.233.6 VisionZip (Yang et al., 2025c)10.0%✗55.554.547.657.339.135.3 FlashVid (Fan et al., 2026)10.4%✗57.956.847.957.739.436.5 FixedScale12.3%✗58.055.147.757.544.335.4 ResAdapt (Ours)11.4%✗59.455.449.258.445.735.9 VideoAuto-R1 (Liu et al., 2026)100%✓63.258.955.060.053.641.5 + ResAdapt (Ours)23.8%✓60.457.153.261.151.238.7 + ResAdapt (Ours)11.4%✓59.356.351.859.349.136.7 128 Frames Vanilla100%✗65.360.353.166.547.942.0 Random Drop25.0%✗64.961.250.864.848.141.3 ToMe (Bolya et al., 2022)25.0%✗65.161.651.963.146.642.1 VisionZip (Yang et al., 2025c)25.0%✗64.861.351.164.547.341.5 ResAdapt (Ours)22.9%✗65.660.252.865.951.142.1 Random Drop10.0%✗63.059.045.863.446.738.0 ToMe (Bolya et al., 2022)10.0%✗60.656.344.263.541.839.5 VisionZip (Yang et al., 2025c)10.0%✗61.856.144.863.242.138.2 FixedScale12.3%✗64.160.949.664.546.940.3 ResAdapt (Ours)11.1%✗63.858.649.064.349.239.9 VideoAuto-R1 (Liu et al., 2026)100%✓64.759.156.765.152.241.2 + ResAdapt (Ours)23.8%✓66.260.253.566.052.641.8 + ResAdapt (Ours)11.4%✓64.757.852.464.651.339.5 Qwen3-VL-8B 32 Frames Vanilla100%✗65.058.657.564.060.840.2 Random Drop25.0%✗61.358.457.160.253.437.8 ToMe (Bolya et al., 2022)25.0%✗62.457.456.060.849.136.4 VisionZip (Yang et al., 2025c)25.0%✗61.857.254.460.651.537.3 FlashVid (Fan et al., 2026)30.0%✗63.959.054.861.955.138.5 ResAdapt (Ours)23.8%✗62.657.555.361.058.438.5 Random Drop10.0%✗58.854.753.256.647.135.5 ToMe (Bolya et al., 2022)10.0%✗59.255.553.158.542.735.8 VisionZip (Yang et al., 2025c)10.0%✗59.955.453.758.845.835.4 FlashVid (Fan et al., 2026)12.2%✗61.057.154.859.147.837.1 FixedScale12.3%✗60.854.953.858.452.637.1 ResAdapt (Ours)11.4%✗60.756.654.659.656.137.3 128 Frames Vanilla100%✗69.464.358.572.763.045.7 Random Drop25.0%✗67.261.356.867.455.342.4 ToMe (Bolya et al., 2022)25.0%✗67.262.055.970.453.543.1 VisionZip (Yang et al., 2025c)25.0%✗67.161.355.769.256.841.2 ResAdapt (Ours)22.9%✗67.461.956.370.859.643.3 Random Drop10.0%✗64.158.355.462.455.538.8 ToMe (Bolya et al., 2022)10.0%✗64.758.655.167.346.340.5 VisionZip (Yang et al., 2025c)10.0%✗64.259.154.266.847.639.4 FixedScale12.3%✗66.759.554.467.756.341.7 ResAdapt (Ours)11.1%✗66.860.255.469.458.242.6 9 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning 102550 Retention Ratio (%) 40.0 42.5 45.0 47.5 50.0 VideoMMMU Accuracy(%) 46.1 49.2 51.6 45.7 48.8 50.0 44.3 46.7 39.1 42.4 48.5 39.4 46.3 49.0 39.2 41.8 48.7 39.8 45.3 47.7 20.4: avg scale 0.4544.9: avg scale 0.67 102550 Retention Ratio (%) 56 58 60 62 VideoMME Accuracy(%) 58.1 60.5 61.7 59.4 60.5 61.2 58.0 59.3 55.5 59.4 60.3 57.9 60.2 61.1 56.4 58.7 60.7 56.1 58.9 60.1 22.3: avg scale 0.4839.0: avg scale 0.63 8 / 100%32 / 25%64 / 12.5%128 / 6.2% #Frames / Retention Ratio (%) Relative Accuracy (%) ResAdapt-RL (Ours)ResAdapt (Ours)FixedScaleVisionZipFlashVidToMeRandomVanilla Figure 3: Efficiency–accuracy trade-offs and temporal reallocation. (a,b) VideoMMMU and VideoMME versus visual-token retention ratioR. ResAdapt is on or near the Pareto frontier, with the clearest advantage on reasoning- heavy settings at low retention. (c) Relative gain from trading spatial resolution for temporal coverage under a fixed 8-frame-equivalent budget. frame selection is treated only as the conceptual zero-budget limit of the same pre-encoding interface. Full hyperparameters, hardware, prompts, and reward definitions are deferred to Appendix A. Baselines. We compare against three classes of methods: heuristic baselines (Random Drop, FixedScale), model-side compression (ToMe (Bolya et al., 2022), FlashVid (Fan et al., 2026), VisionZip (Yang et al., 2025c)), and reasoning-time inference augmentation (VideoAuto-R1 (Liu et al., 2026)). We use visual- token retention ratioRas the primary budget descriptor and report the exact retained budget for every method. For reasoning-time baselines,Rmeasures only visual encoder tokens; unless latency is reported separately, these comparisons should therefore be read as visual-budget comparisons rather than total- inference-budget matches. Because several baselines admit only discrete operating points, some comparisons are only approximately budget matched and should be interpreted relative to the explicit trade-offs shown in each table. Benchmarks. For video QA, we report results on VideoMME (Fu et al., 2025a), LongVideoBench (Wu et al., 2024), MMVU (Zhao et al., 2025b), MLVU (Zhou et al., 2025), VideoMMMU (Hu et al., 2025), and LVBench (Wang et al., 2025b). For temporal grounding, we report Recall@0.3, 0.5, 0.7and mIoU on Charades-STA (Gao et al., 2017) and ActivityNet (Fabian et al., 2015), plus grounding QA on NExT-GQA (Xiao et al., 2024). For image understanding, we evaluate on MathVista (Lu et al., 2023), MMMU (Yue et al., 2024), OCRBench (Liu et al., 2024), ChartQA (Masry et al., 2022), AI2D (Kembhavi et al., 2016), and TextVQA (Singh et al., 2019). Unless stated otherwise, figures and analyses use Qwen2.5-VL-7B with 32 input frames. All evaluations use lmms-eval (Zhang et al., 2024a); the exact token budgets and decoding limits are reported in Appendix A. 4.2. Main Results We organize the evaluation around two primary claims and one exploratory question. First, resize-based input-side allocation should improve low-budget operating points on video QA. Second, allocating pixels before encoding should be clearly more robust than frame dropping and often competitive with model-side compression on temporal grounding. Third, we ask how far the learned policy transfers beyond the training setting; we treat this last analysis as exploratory rather than as a headline claim. Video QA. This experiment tests the first claim. We emphasize low-budget operating points rather than a universal frontier statement, since several baselines are available only at discrete budgets (Table 1). 10 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Table 2: Evaluation Results on Temporal Grounding Benchmarks. Grounding is much more compression-sensitive. Backbone Method Retention Ratio R Reasoning Temporal Grounding Benchmark Charades-STAActivityNetNExT-GQA 0.30.50.7 mIoU0.30.50.7 mIoUAcc mIoU Qwen2.5-VL-7B 32 Frames Vanilla100%✗71.0 51.4 26.0 47.330.4 18.0 8.922.678.9 28.0 Random Drop25.0%✗39.4 23.2 11.0 25.715.2 8.13.711.777.5 16.6 ToMe (Bolya et al., 2022)25.0%✗39.5 23.9 11.4 26.016.0 8.44.012.177.8 16.3 FlashVid (Fan et al., 2026)31.3%✗40.7 24.2 11.3 26.615.8 8.43.812.078.1 16.5 FixedScale25.0%✗36.7 24.7 12.3 24.918.6 9.44.314.177.7 12.3 ResAdapt (Ours)16.2%✗53.834.817.035.619.810.85.215.376.623.2 Random Drop10.0%✗36.9 23.2 11.6 24.614.3 7.53.611.176.3 15.4 ToMe (Bolya et al., 2022)10.0%✗41.3 26.9 14.1 27.416.0 8.4 4.0 12.277.3 15.7 FlashVid (Fan et al., 2026)12.6%✗38.2 22.9 11.1 25.115.4 8.13.711.877.4 16.1 FixedScale12.3%✗48.0 31.5 15.4 32.017.5 8.9 4.0 13.376.1 13.7 FixedScale6.3%✗39.9 26.8 13.3 26.715.2 8.13.911.974.1 15.4 ResAdapt (Ours)6.8%✗41.027.814.027.216.38.53.912.574.320.4 VideoAuto-R1 (Liu et al., 2026)100%✓60.0 48.3 27.2 41.550.8 34.1 17.4 34.473.6 33.8 + ResAdapt (Ours)6.8%✓43.530.115.830.035.421.510.024.474.724.7 128 Frames Vanilla100%✗77.5 60.3 34.1 52.847.9 30.9 17.5 34.479.8 29.9 Random Drop25.0%✗32.3 19.6 7.920.726.7 13.9 6.318.880.3 10.7 ToMe (Bolya et al., 2022)25.0%✗32.4 19.8 7.920.727.2 14.4 6.419.180.3 10.9 ResAdapt (Ours)16.1%✗63.543.621.342.033.119.310.224.378.127.2 Random Drop10.0%✗37.8 23.8 11.2 24.723.8 12.0 5.317.079.4 12.8 ToMe (Bolya et al., 2022)10.0%✗27.9 16.2 7.317.922.9 11.8 5.516.479.1 11.1 FixedScale12.3%✗34.7 22.3 10.5 22.725.0 13.8 5.9 18.377.9 11.3 FixedScale6.3%✗42.6 28.4 14.3 28.322.8 12.8 5.717.175.7 12.9 ResAdapt (Ours)6.8%✗43.529.815.028.923.512.96.117.276.223.9 VideoAuto-R1 (Liu et al., 2026)100%✓40.3 33.7 22.1 28.949.4 34.3 18.5 33.568.0 31.0 + ResAdapt (Ours)16.1%✓72.853.027.549.165.844.923.844.779.335.3 + ResAdapt (Ours)6.8%✓50.133.216.634.253.434.016.435.776.629.4 Qwen3-VL-8B 32 Frames Vanilla100%✗73.0 49.0 21.4 46.444.6 28.3 15.5 31.878.7 34.2 Random Drop25.0%✗16.2 8.63.812.112.4 6.73.210.077.2 15.6 ToMe (Bolya et al., 2022)25.0%✗68.7 42.1 17.6 43.145.9 28.8 15.6 32.677.1 31.7 FlashVid (Fan et al., 2026)31.3%✗72.9 52.3 25.1 47.751.9 33.4 19.0 36.877.8 33.9 ResAdapt (Ours)16.2%✗64.437.316.339.940.024.413.028.575.130.2 Random Drop10.0%✗4.11.80.74.44.72.41.05.074.3 11.3 ToMe (Bolya et al., 2022)10.0%✗67.6 39.3 16.6 41.846.3 31.0 19.2 34.179.2 34.0 FlashVid (Fan et al., 2026)12.6%✗68.8 46.9 22.9 44.649.9 31.5 17.4 35.275.6 31.8 FixedScale12.3%✗61.3 34.3 14.6 37.939.6 24.2 13.1 28.474.2 29.9 FixedScale6.3%✗52.7 28.2 11.3 33.237.0 22.3 12.0 27.071.5 28.0 ResAdapt (Ours)6.8%✗53.629.011.833.637.522.512.327.271.828.2 128 Frames Vanilla100%✗72.8 46.0 20.1 45.645.8 31.1 19.2 33.981.1 36.6 Random Drop25.0%✗41.6 25.2 10.6 27.436.1 21.1 12.7 26.379.3 22.4 ResAdapt (Ours)16.1%✗64.437.015.939.840.626.715.730.076.833.3 Random Drop10.0%✗32.6 19.0 7.821.933.5 18.6 11.5 24.876.9 19.9 ToMe (Bolya et al., 2022)10.0%✗61.6 33.8 13.3 38.142.4 27.6 16.6 31.477.4 31.5 FixedScale12.3%✗61.7 34.9 14.7 38.139.9 26.2 15.3 29.575.4 32.6 FixedScale6.3%✗53.7 28.2 11.8 33.637.9 24.3 14.3 28.173.0 39.1 ResAdapt (Ours)6.8%✗54.328.011.733.738.324.514.428.473.243.9 11 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Table 3: Latency breakdown (ms,↓) on Qwen2.5-VL-7B with single-GPU Allocator and 4-GPU vLLM engine. Averaged over 200 runs after 5 warm-up; E2E latency = Scale Time + Gen. Time. Method #Frames Retention Ratio R ScaleInferenceTotal TFLOPs Text Enc. Visual Enc. Scale Pred. Scale Apply Scale Time TFLOPsTTFT Gen. Time TFLOPs E2E Time Vanilla16100%–111.4378.9527.9111.4527.9 ResAdapt1676.3%1.519.894.185.66.3205.877.2 (↓30.7%)272.5 (↓28.1%)370.7 (↓29.8%)80.1 (↓28.1%)576.5 (↑9.2%) ResAdapt1652.8%1.519.9102.994.58.4225.751.5 (↓53.8%)261.5 (↓31.0%)313.1 (↓40.7%)54.4 (↓51.2%)538.8 (↑2.1%) ResAdapt1628.9%1.520.4103.492.29.0225.031.0 (↓72.2%)227.2 (↓40.0%)237.9 (↓54.9%)33.9 (↓69.6%)462.9 (↓12.3%) Vanilla32100%–222.5723.3881.9222.5881.9 ResAdapt3274.4%2.919.9204.197.414.4335.9153.9 (↓30.8%)589.4 (↓18.5%)627.6 (↓28.8%)159.7 (↓28.2%)963.5 (↑9.2%) ResAdapt3251.5%2.920.0193.292.016.2321.4102.4 (↓54.0%)505.0 (↓30.2%)467.1 (↓47.0%)108.2 (↓51.4%)788.5 (↓10.6%) ResAdapt3228.2%2.920.3190.490.317.3318.361.4 (↓72.4%)451.8 (↓37.5%)332.6 (↓62.3%)67.2 (↓69.8%)650.9 (↓26.2%) Vanilla64100%–444.61457.52059.6444.62059.6 ResAdapt6473.2%5.819.8389.595.826.4531.5307.3 (↓30.9%)1093.1 (↓25.0%)1327.0 (↓35.6%)318.9 (↓28.3%)1858.5 (↓9.8%) ResAdapt6450.7%5.820.1382.194.929.9527.0204.3 (↓54.0%)991.8 (↓31.9%)740.5 (↓64.0%)215.9 (↓51.4%)1267.5 (↓38.5%) ResAdapt6427.8%5.820.0371.690.234.8516.6122.2 (↓72.5%)899.2 (↓38.3%)511.4 (↓75.2%)133.8 (↓69.9%)1028.0 (↓50.1%) Vanilla128100%–888.92936.34877.0888.94877.0 ResAdapt12874.2%11.620.1766.395.053.1934.5614.1 (↓30.9%)2286.6 (↓22.1%)2323.6 (↓52.4%)637.3 (↓28.3%)3258.1 (↓33.2%) ResAdapt12851.4%11.620.2755.393.859.4928.7408.0 (↓54.1%)2071.0 (↓29.5%)1496.0 (↓69.3%)431.2 (↓51.5%)2424.7 (↓50.3%) ResAdapt12828.2%11.620.4734.592.068.6915.5243.9 (↓72.6%)1766.7 (↓39.8%)1061.8 (↓78.2%)267.1 (↓70.0%)1977.3 (↓59.5%) Disproportionate gains on multi-step reasoning. Under aggressive compression (∼10% retention), content- agnostic methods discard sparse but decisive evidence. On Qwen2.5-VL with 32 frames, ResAdapt reaches 45.7 on VideoMMMU at 11.4% retention, improving over ToMe (39.2), VisionZip (39.1), FlashVid (39.4), and FixedScale (44.3) while remaining competitive on the perception benchmarks. The gain is therefore concentrated on the reasoning-heavy benchmark rather than being uniform across tasks. The transferred Allocator remains strong on Qwen3-VL, reaching 56.1 on VideoMMMU at the same 11.4% retention, but we treat the cross-backbone comparison as supportive rather than definitive because the available operating points are not perfectly matched. Spatial savings reinvested as temporal coverage. Extending context from 32 to 128 frames amplifies the advantage. At 22.9% retention on Qwen2.5-VL, ResAdapt reaches 51.1 on VideoMMMU versus 47.9 for the 128-frame uncompressed model while recovering most of the perception performance at far lower visual cost. Even at 11.1% retention, ResAdapt attains 49.2 on VideoMMMU, again slightly exceeding the 128-frame vanilla score. The effect is task-dependent rather than uniform, but it shows that saved spatial budget can be reinvested as temporal headroom: the model observes4×more frames without paying native-resolution cost (Figure 3). Temporal Grounding. This experiment tests the second claim. Grounding is markedly more compression- sensitive than QA, so we interpret Table 2 as a comparison of operating points rather than as a perfectly budget-matched frontier. Pre-encoding allocation is more robust than frame dropping. On Qwen2.5-VL (32F), Random Drop, ToMe, FlashVid, and FixedScale reduce Charades-STA mIoU from 47.3 to 25.7, 26.0, 26.6, and 24.9, respectively, at ≈25–31% retention. At a lower 16.2% budget, ResAdapt reaches 35.6. The point is therefore not a universal matched-budget win over every grounding baseline; rather, allocating pixels before encoding is substantially more robust than frame dropping and remains competitive with model-side compression despite operating at a smaller budget. On the reported Qwen3 rows, the same pattern holds most clearly against frame dropping, while comparisons to model-side baselines are more mixed. Reasoning without temporal anchors regresses. The strongest grounding result is a single but important reasoning-augmented operating point. On VideoAuto-R1 (Qwen2.5-VL), extending from 32 to 128 frames 12 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning 051015202530 0.2 0.4 0.6 0.8 1.0 Scale Factor Case 338 (=0.214) 051015202530 0.2 0.4 0.6 0.8 1.0 Case 1640 (=0.247) 051015202530 0.2 0.4 0.6 0.8 1.0 Case 1302 (=0.251) 051015202530 Frame Index 0.2 0.4 0.6 0.8 1.0 Scale Factor Case 1166 (=0.002) 051015202530 Frame Index 0.2 0.4 0.6 0.8 1.0 Case 2435 (=0.019) 051015202530 Frame Index 0.2 0.4 0.6 0.8 1.0 Case 1770 (=0.009) Figure 4: Emergent active perception. Per-frame scales t over frame index for six VideoMME videos, grouped by intra-video scale diversityσ. High-diversity videos show localized scale spikes on scene changes, text overlays, and rapid motion; low-diversity videos remain near-uniform. degrades Charades-STA mIoU from 41.5 to 28.9, indicating that longer reasoning chains alone do not preserve localization quality in this setting. Adding ResAdapt at 16.1% visual-token retention lifts the 128-frame score to 49.1, showing that allocation can materially improve this particular long-context reasoning regime. Emergent denoising. On NExT-GQA (Qwen3-VL, 128F), ResAdapt improves mIoU from 36.6 to 43.9 at only 6.8% retention, suggesting that suppressing question-irrelevant frames can sharpen localization even under extreme compression. Exploratory image transfer. Transfer beyond video is mixed and not central to our claims. The clearest positive result is ChartQA on Qwen2.5-VL, but it is obtained at 105% visual-token retention, whereas text- heavy tasks degrade once resolution becomes too low. We therefore treat image results as a robustness check and report them in Appendix D.4, Table 7. 4.3. Runtime Overhead This experiment is a pipeline-latency case study: it measures when the front-end cost of allocation is amortized by downstream token savings. Table 3 reports latency against vanilla using a dedicated single-GPU Allocator and a separate 4-GPU vLLM engine; it should therefore be read as a vanilla-versus-ResAdapt deployment profile rather than as a same-hardware comparison against ToMe, FlashVid, or VisionZip. By adjusting the maximum allowed scale, ResAdapt spans retention ratios from conservative (R≈74%) to aggressive (R≈28%) compression, providing a single controllable accuracy–speed knob. The key trade-off is when token reduction outweighs the front-end scaling cost. AtR≈74%, generation time drops 29–52% but end-to-end (E2E) savings appear only at≥64 frames (−9.8%), growing to−33.2%at 128 frames. AtR≈51%, the break-even shifts to 32 frames (−10.6%E2E); atR≈28%, wall-clock savings 13 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning 051015202530 Frame Index 0.24 0.26 0.28 0.30 0.32 0.34 0.36 0.38 0.40 Scale Factor (a) Aggregate Scale by Frame Position (VideoMME) IQR Mean Median 08162431 Frame Index 0 25 50 75 100 125 150 175 Case ID (b) Per-Frame Scale Heatmap (First 200 Cases) 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Scale Factor Figure 5: Global allocation statistics on VideoMME. (a) Aggregate predicted scale by frame position. (b) Case×frame heatmap for the first 200 videos. High-scale allocation appears as localized bursts rather than a fixed positional pattern. Table 4: Distribution family ablation for CAPO. The two variants follow the same training protocol. Variant ̄ sVideoMME LongVideoBench MMVUVideoMMMULVBench Per. Comp. Adap. β-CAPO 0.5460.358.251.265.0 54.328.737.6 N -CAPO 0.6061.057.451.8 66.050.030.337.2 emerge even at 16 frames (−12.3%), reaching−59.5%at 128 frames with 78% generation-time reduction. This scaling reflects the quadratic cost of attention: backbone savings compound faster than the lower-order Allocator overhead as sequences grow, making ResAdapt most impactful in the long-context regime. 4.4. Analysis and Ablation Emergent active perception. These analyses explain why the main results hold. The Allocator learns a strongly sparse temporal allocation rather than a near-uniform compression policy. Figure 4 shows this clearly: many videos are compressed almost everywhere, with short bursts of higher resolution around text overlays, scene transitions, or other brief informative events. This is precisely the behavior we would want from input-side allocation: the Allocator spends pixels where the answer is likely to be decided, rather than distributing them uniformly across the clip. Figure 5 shows that this behavior is not a trivial positional prior. The median scale stays close to the low end of the range, while the mean is lifted by localized peaks, implying that high-resolution allocation is the exception rather than the default. The per-video heatmap further confirms that these peaks appear as content-dependent segments rather than as a fixed bias toward the beginning or end of the sequence. CAPO ablation. Two questions arise: how cost should enter optimization, and what induces the policy to vary across neighboring frames rather than collapse to a uniform scaler. Table 4 suggests that the exact policy family is secondary:β-CAPO andN-CAPO trade marginal advantages 14 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning 0100200300400500 Training Step 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Mean Predicted Scale ̄ s (a) Training 0100200300400500 Training Step (b) Validation β-Dist. w/o Cost β-Dist. w Direct Cost β-Dist. w CAPO -Dist. w CAPO Figure 6: Reward-design ablation. Mean predicted scale ̄ sduring training and validation. Direct cost penalties collapse to the minimum scale, whereas CAPO variants converge to stable intermediate operating points. across benchmarks, with neither variant consistently dominating. Because the realized budgets are not exactly matched, we interpret this as a directional trend rather than as a perfectly controlled ablation. The shared ingredient that appears to matter is CAPO’s asymmetric cost shaping, not the specific parametric form of the scale distribution. Figure 6 makes this more explicit from a training-dynamics perspective. Direct cost penalties drive the policy rapidly toward the minimum-scale boundary, while removing cost altogether pushes it toward the upper bound. CAPO is effective precisely because it stabilizes an intermediate operating point where the model is still rewarded for being selective rather than merely cheap or merely accurate. We provide further analysis of per-sample scale adaptivity and convergence behavior in Appendix D.2.2. Table 5: Operator generalization. Zero-shot transfer of ResAdapt scores to frame selection. Combining top-Kselection with adaptive resizing from 128 candidate frames outperforms uniform sampling baselines at lower token budgets. MethodVideoMME LongVideoBench LVBenchMMVU Budget: 8 frames Vanilla54.053.933.348.9 Top-8 Select52.251.132.049.2 Budget: 16 frames Vanilla58.956.036.150.9 Threshold Select58.057.436.451.0 Avg. Budget (Retention Ratio)12.2f (9.5%)23.2f (18.1%)16.7f (13.0%)17.2f (13.4%) Top-32 Select + Resize60.657.238.950.2 Avg. Budget (Retention Ratio)11.7f (9.1%)16.9f (13.2%)13.7f (10.7%)14.1f (11.0%) Budget: 32 frames Vanilla62.358.739.552.0 Top-32 Select59.755.737.051.2 Top-64 Select + Resize62.558.440.052.3 Avg. Budget (Retention Ratio)23.8f (18.6%)36.2f (28.3%)24.1f (18.8%)32.5f (25.4%) Operator generalization. Although Re- sAdapt is trained exclusively for adap- tive resizing, its learned policy general- izes zero-shot to other input-side opera- tors. We repurpose the Allocator’s pre- dicted scales as frame importance scores to rank and filter 128 candidate frames. Ta- ble 5 demonstrates that while pure frame selection (top-Kor threshold-based) pro- vides a viable strategy, combining selec- tion with resizing yields the best efficiency– accuracy trade-off. Specifically, selecting and resizing the top-32 or top-64 frames consistently outperforms the vanilla 16- frame and 32-frame baselines, respectively, despite consuming fewer tokens on aver- age. This confirms that the Allocator learns a robust, operator-agnostic measure of visual importance that seamlessly cascades with discrete frame dropping. 15 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning 0.25 0.00 0.25 0.50 0.75 1.00 Scale s t =0.0032 Video 125 0.25 0.00 0.25 0.50 0.75 1.00 =0.0019 Video 288 0.25 0.00 0.25 0.50 0.75 1.00 =0.0019 Video 483 51015202530 Frame Index 0.25 0.00 0.25 0.50 0.75 1.00 Scale s t =0.2421 51015202530 Frame Index 0.25 0.00 0.25 0.50 0.75 1.00 =0.2481 51015202530 Frame Index 0.25 0.00 0.25 0.50 0.75 1.00 =0.2114 w/o sim w/ sim Figure 7:L sim ablation: per-frame scale profiles. Without temporal-similarity regularization, the Allocator ap- proaches near-uniform scaling; with it, the policy concentrates resolution on selected frames and suppresses redundant neighbors. Temporal regularization complements CAPO. CAPO alone determines how cost enters learning, but it does not by itself force the Allocator to distinguish among visually redundant neighbors. Figure 7 shows that onceL sim is removed, the learned scale traces collapse toward a near-constant profile that behaves much like FixedScale. ReintroducingL sim restores sharp frame-level differentiation, which confirms that CAPO and the temporal regularizer are complementary: one stabilizes the accuracy–cost objective, and the other breaks the symmetry that would otherwise favor uniform allocation. Robustness and failure modes. Adaptive allocation does not act as a lossless compression layer. In practice, ResAdapt usually preserves many originally correct predictions, but it can still miss decisive evidence, especially when the relevant cue is visually simple and appears only briefly. Because the policy is open-loop, it cannot revise allocations after reasoning begins or recover evidence that was undersampled in the initial pass. We therefore interpret its gains as selective redistribution of visual budget rather than as guaranteed preservation of all useful information. 5. Related Work Input-side adaptation before visual encoding. A growing body of work reduces visual cost before or during input construction. Early approaches primarily perform temporal downsampling through keyframe selection or clip condensation (Liang et al., 2024, Zhu et al., 2025, Sun et al., 2025, Tang et al., 2025). More recent methods incorporate query awareness and iterative search, tailoring frame selection to question types or intermediate evidence (Zou et al., 2025, Li et al., 2025a, Guo et al., 2025b, He et al., 2025). Beyond selecting which frames to process, several works allocate perceptual budgets via multi-resolution encoding. Slow–fast pipelines (Yang et al., 2025a, Zhang et al., 2026) use inter-frame similarity to route frames to high- or low-resolution paths, but their binary, query-agnostic routing cannot adapt to the downstream question. Query-aware multi-resolution strategies (Zhang et al., 2025d) and early truncation of less informative visual tokens (Chen et al., 2026) go further by conditioning on the query, yet still rely on handcrafted rules or fixed resolution bins. In contrast, ResAdapt is an Input-side adaptation framework: it learns input-side allocations from task reward via RL and can realize them through different pre-encoding operators, including resizing and frame selection; the experiments in this paper study the continuous resize instantiation. 16 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Model-side token economy after encoding. Post-encoding methods prune, merge, or redistribute visual tokens in embedding space. For images, representative approaches include token merging (Bolya et al., 2022), attention- or saliency-guided pruning (Chen et al., 2024, Yang et al., 2025c, Shang et al., 2025, Zhang et al., 2025c), progressive dropping (Xing et al., 2024, Zhang et al., 2024b), and diversity-based budget allocation (Alvar et al., 2025, Yang et al., 2025b, Zhang et al., 2025a). Video-specific extensions exploit spatiotemporal redundancy via static/dynamic token separation (Huang et al., 2025, Shen et al., 2025a), hierarchical merging (Hyun et al., 2025), and segment-level fusion or budget allocation (Tao et al., 2025, Fu et al., 2024, Shao et al., 2025a). These methods are complementary to ResAdapt: they operate after visual encoding and cannot recover high-frequency details lost to undersampling before encoding. Our focus is earlier in the pipeline, deciding how many pixels to encode in the first place. Output-side agentic reasoning. Another strategy leaves the input fixed and recovers efficiency through iterative reasoning: retrieve candidate frames, zoom into regions, then re-query the model. Approaches range from static toolsets with predefined cropping or clipping operators (Zheng et al., 2025b, Wang et al., 2025a, Song et al., 2026) to dynamic tooling via code-generation primitives (Zhang et al., 2025e, Zhao et al., 2025a, Hong et al., 2025), often exposed through executable interfaces (Wang et al., 2024). While these methods can target hard evidence precisely, they are multi-pass by construction and rely on an initial coarse view to trigger subsequent refinement. ResAdapt instead studies whether a single-pass pre-encoding allocation policy can recover much of this benefit without the latency and control overhead of iterative interaction. RL for multimodal reasoning and perception control. Recent work has extended RL post-training from language models (Shao et al., 2024, Guo et al., 2025a) to multimodal reasoning and video understanding. Algorithmic refinements include improved advantage estimation and PPO-style stabilization (Liu et al., 2025c, Yu et al., 2025, Zheng et al., 2025a), while video-domain extensions strengthen reasoning through iterative frame selection and evidence refinement (Feng et al., 2025, Li et al., 2025b, Liu et al., 2026, Yang et al., 2025d, Chen et al., 2025, Wang et al., 2025c, Fu et al., 2025b). Our use of RL is orthogonal: we apply it to input-side perception control—learning frame-level visual allocations under an explicit accuracy–cost trade-off—rather than output-side reasoning policies. CAPO is designed for this setting, where naive cost penalties drive the policy to a degenerate low-budget solution. 6. Conclusion We study Input-side adaptation through ResAdapt, a framework that shifts part of the efficiency burden from post-encoding token compression to pre-encoding visual budget control. A lightweight Allocator, trained with Cost-Aware Policy Optimization (CAPO) and temporal-similarity regularization, predicts per-frame allocations before visual encoding, and the downstream backbone continues to operate on its native token interface. Empirically, the framework is most convincing in low-visual-token video QA and in one reasoning- augmented long-context grounding regime where saved spatial compute is reinvested as temporal coverage. The learned policy is sparse and content-dependent, but it remains open-loop: once the initial allocations are set, the model cannot recover missed evidence within the same pass. More broadly, the results suggest that pre-encoding allocation is a promising direction for long-context video reasoning, while the present evidence remains specific to the resize instantiation studied here and transfers beyond the training domain only unevenly. 17 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning References Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity-based visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9392–9401, 2025. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report, 2025a. URL https://arxiv.org/abs/2511.21631. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b. Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer, 2024. Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, et al. Scaling rl to long videos. arXiv preprint arXiv:2507.07966, 2025. Zeyuan Chen, Kai Zhang, Zhuowen Tu, and Yuanjun Xiong. Soft tail-dropping for adaptive visual tokenization. arXiv preprint arXiv:2601.14246, 2026. Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, 2024. Caba Heilbron Fabian, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015. Ziyang Fan, Keyu Chen, Ruilong Xing, Yulin Li, Li Jiang, and Zhuotao Tian. Flashvid: Efficient video large lan- guage models via training-free tree-based spatiotemporal token merging. arXiv preprint arXiv:2602.08024, 2026. Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776, 2025. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24108–24118, 2025a. 18 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Shenghao Fu, Qize Yang, Yuan-Ming Li, Xihan Wei, Xiaohua Xie, and Wei-Shi Zheng. Love-r1: Advancing long video understanding with an adaptive zoom-in mechanism via multi-step reasoning. arXiv preprint arXiv:2509.24786, 2025b. Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. Framefusion: Combining similarity and importance for video token reduction on large vision language models. arXiv preprint arXiv:2501.01986, 2024. Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pages 5267–5275, 2017. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025a. Weiyu Guo, Ziyang Chen, Shaoguang Wang, Jianxiang He, Yijie Xu, Jinhui Ye, Ying Sun, and Hui Xiong. Logic- in-frames: Dynamic keyframe search via visual semantic-logical verification for long video understanding. arXiv preprint arXiv:2503.13139, 2025b. Zefeng He, Xiaoye Qu, Yafu Li, Siyuan Huang, Daizong Liu, and Yu Cheng. Framethinker: Learning to think with long videos via multi-turn frame spotlighting. arXiv preprint arXiv:2509.24304, 2025. Jack Hong, Chenxiao Zhao, ChengLin Zhu, Weiheng Lu, Guohai Xu, and Xing Yu. Deepeyesv2: Toward agentic multimodal model. arXiv preprint arXiv:2511.05271, 2025. Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video- mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826, 2025. Xiaohu Huang, Hao Zhou, and Kai Han. Prunevid: Visual token pruning for efficient video large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19959–19973, 2025. Jeongseok Hyun, Sukjun Hwang, Su Ho Han, Taeoh Kim, Inwoong Lee, Dongyoon Wee, Joon-Young Lee, Seon Joo Kim, and Minho Shim. Multi-granular spatio-temporal token merging for training-free acceleration of video llms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23990– 24000, 2025. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European conference on computer vision, pages 235–251. Springer, 2016. Samir Khaki, Junxian Guo, Jiaming Tang, Shang Yang, Yukang Chen, Konstantinos N Plataniotis, Yao Lu, Song Han, and Zhijian Liu. Sparsevila: Decoupling visual sparsity for efficient vlm inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23784–23794, 2025. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. Jialuo Li, Bin Li, Jiahao Li, and Yan Lu. Divide, then ground: Adapting frame selection to query types for long-form video understanding. arXiv preprint arXiv:2512.04000, 2025a. 19 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958, 2025b. Hao Liang, Jiapeng Li, Tianyi Bai, Xijie Huang, Linzhuang Sun, Zhengren Wang, Conghui He, Bin Cui, Chong Chen, and Wentao Zhang. Keyvideollm: Towards large-scale video keyframe selection. arXiv preprint arXiv:2407.03104, 2024. Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407, 2025a. Shuming Liu, Mingchen Zhuge, Changsheng Zhao, Jun Chen, Lemeng Wu, Zechun Liu, Chenchen Zhu, Zhipeng Cai, Chong Zhou, Haozhe Liu, et al. Videoauto-r1: Video auto reasoning via thinking once, answering twice. arXiv preprint arXiv:2601.05175, 2026. Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102, 2024. Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4122–4134, 2025b. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025c. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, et al. Smolvlm: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299, 2025. Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, pages 2263–2279, 2022. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505–3506, 2020. Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22857–22867, 2025. Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Holitom: Holistic token merging for fast video large language models. arXiv preprint arXiv:2505.21334, 2025a. 20 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, and Huan Wang. When tokens talk too much: A survey of multimodal long-context token compression across images, videos, and audios. arXiv preprint arXiv:2507.20198, 2025b. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Leqi Shen, Guoqiang Gong, Tao He, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, and Guiguang Ding. Fastvid: Dynamic density pruning for fast video large language models. arXiv preprint arXiv:2503.11187, 2025a. Xiaoqian Shen, Min-Hung Chen, Yu-Chiang Frank Wang, Mohamed Elhoseiny, and Ryo Hachiuma. Zoom-zero: Reinforced coarse-to-fine video understanding via temporal zoom-in. arXiv preprint arXiv:2512.14273, 2025b. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297, 2025. Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 26160–26169, 2025. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. Mingyang Song, Haoyu Sun, Jiawei Gu, Linjie Li, Luxin Xu, Ranjay Krishna, and Yu Cheng. Adareasoner: Dynamic tool orchestration for iterative visual reasoning. arXiv preprint arXiv:2601.18631, 2026. Guangyu Sun, Archit Singhal, Burak Uzkent, Mubarak Shah, Chen Chen, and Garin Kessler. From frames to clips: Training-free adaptive key clip selection for long-form video understanding. arXiv preprint arXiv:2510.02262, 2025. Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. arXiv preprint arXiv:2502.21271, 2025. Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18992–19001, 2025. Haozhe Wang, Alex Su, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning. arXiv preprint arXiv:2505.15966, 2025a. Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, et al. Lvbench: An extreme long video understanding benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22958–22967, 2025b. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning, 2024. 21 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, et al. Time-r1: Post-training large vision language model for temporal video grounding. arXiv preprint arXiv:2503.13377, 2025c. Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828–28857, 2024. Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13204–13214, 2024. Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247, 2024. Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, and Song Han. Streamingvlm: Real-time understanding for infinite video streams. arXiv preprint arXiv:2510.09608, 2025. Biao Yang, Bin Wen, Boyang Ding, Changyi Liu, Chenglong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, et al. Kwai keye-vl 1.5 technical report. arXiv preprint arXiv:2509.01563, 2025a. Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Ponnuswamy Sadayappan, Xia Hu, et al. Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19803–19813, 2025b. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19792–19802, 2025c. Zuhao Yang, Sudong Wang, Kaichen Zhang, Keming Wu, Sicong Leng, Yifan Zhang, Bo Li, Chengwei Qin, Shijian Lu, Xingxuan Li, and Lidong Bing. Longvt: Incentivizing "thinking with long videos" via native tool calling. arXiv preprint arXiv:2511.20785, 2025d. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024. Boqiang Zhang, Lei Ke, Ruihan Yang, Qi Gao, Tianyuan Qu, Rossell Chen, Dong Yu, et al. Penguin-vl: Exploring the efficiency limits of vlm with llm-based vision encoders. arXiv preprint arXiv:2603.06569, 2026. Ce Zhang, Kaixin Ma, Tianqing Fang, Wenhao Yu, Hongming Zhang, Zhisong Zhang, Yaqi Xie, Katia Sycara, Haitao Mi, and Dong Yu. Vscan: Rethinking visual token reduction for efficient large vision-language models. arXiv preprint arXiv:2505.22654, 2025a. 22 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Congzhi Zhang, Zhibin Wang, Yinchao Ma, Jiawei Peng, Yihan Wang, Qiang Zhou, Jun Song, and Bo Zheng. Rewatch-r1: Boosting complex video reasoning in large vision-language models through agentic data synthesis. arXiv preprint arXiv:2509.23652, 2025b. Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024a. URL https://arxiv.org/abs/2407.12772. Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20857–20867, 2025c. Shaojie Zhang, Jiahui Yang, Jianqin Yin, Zhenbo Luo, and Jian Luan. Q-frame: Query-aware frame selection and multi-resolution adaptation for video-llms. arXiv preprint arXiv:2506.22139, 2025d. Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, et al. Thyme: Think beyond images. arXiv preprint arXiv:2508.11630, 2025e. Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417, 2024b. Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Ming Li, Qilong Wu, Kaipeng Zhang, and Chen Wei. Pyvision: Agentic vision with dynamic tooling. arXiv preprint arXiv:2507.07998, 2025a. Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, et al. Mmvu: Measuring expert-level multi-discipline video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8475–8489, 2025b. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025a. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, 2024. Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deep- eyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505.14362, 2025b. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, et al. Mlvu: Benchmarking multi-task long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13691–13701, 2025. Zirui Zhu, Hailun Xu, Yang Luo, Yong Liu, Kanchan Sarkar, Zhenheng Yang, and Yang You. Focus: Efficient keyframe selection for long video understanding. arXiv preprint arXiv:2510.27280, 2025. Yuanhao Zou, Shengji Jin, Andong Deng, Youpeng Zhao, Jun Wang, and Chen Chen. Air: Enabling adaptive, iterative, and reasoning-based frame selection for video question answering. arXiv preprint arXiv:2510.04428, 2025. 23 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Limitations and future work. ResAdapt improves the efficiency–accuracy trade-off for long-video MLLMs, but the current evidence is still bounded by four concrete design choices. (i) Front-end overhead is amortized only in the long-context regime. The Allocator adds a fixed pre-encoding cost—coarse visual encoding, cross-frame fusion, and distribution prediction—before any backbone savings are realized. When the sequence is short (T≤32), this constant overhead can offset a meaningful fraction of the downstream attention reduction, so the clearest wall-clock gains appear only when temporal context is long (Sec. 4.3). Reducing this fixed cost through cached video features, cheaper front-ends, or distilled allocation rules is therefore an important next step. (i) Allocation is limited by coarse visual evidence. The Allocator observes frozen coarse featuresf t ∈R D rather than the full high-resolution frame. This is sufficient to detect broad redundancy and scene structure, but it is weaker on small text, subtle objects, and brief answer-critical cues embedded in otherwise simple frames (Figure 20). Multi-scale conditioning, motion-aware features, or lightweight local refinement would help close this gap without giving up the speed advantage of the current front-end. (i) The present study validates the framework through one video-centric instantiation. Our formulation is general input-side adaptation, but the experiments instantiate the operator with resizing and train the policy primarily on video tasks. As a result, transfer beyond this regime is uneven: the learned policy sometimes recognizes image inputs that need more fidelity, yet it does not deliver uniformly efficiency-preserving gains on static-image benchmarks (Table 7). Extending training to mixed image–video data and alternative operators such as hard frame selection remains open. (iv) Allocation is open-loop rather than reasoning-aware. All budget decisions are committed before the backbone processes any visual token. The policy therefore cannot revise a mistaken low-resolution choice after partial reasoning or uncertainty signals emerge. A natural extension is closed-loop allocation, where early backbone states trigger re-encoding, budget revision, or a second visual pass only when needed. Software and Data The code for this paper is available at: https://github.com/Xnhyacinth/ResAdapt A. Implementation Details A.1. Training Data Data Composition. We build the training set from the difficulty-filtered data of VideoAuto-R1 (Liu et al., 2026), keeping only image and video samples and discarding pure-text examples. To improve coverage of visually demanding subdomains, we additionally sample 16,500 video instances from Video-R1 (Feng et al., 2025), focusing on OCR, free-form QA, and regression-style tasks. The merged pool contains approximately 93.4K training samples. We manually remove all evaluation examples from our benchmark suite to avoid leakage. 24 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning A.2. Training Configuration Unless otherwise noted, training runs for one epoch with global batch size 128 and AdamW. The learning rate is2× 10 −5 for the Allocator and1× 10 −6 for the backbone, with weight decay0.01and gradient clipping at1.0. We set the maximum video token budget to8,192, useT=128frames during training, and allow scales in the range[s min , s max ] = [0.2, 1.8], which permits both downscaling and selective upscaling. CAPO samplesM=16allocation trajectories per prompt andN=1rollout per trajectory. Training is conducted on 32 H100 GPUs with VeRL (Sheng et al., 2025), DeepSpeed (Rasley et al., 2020), and vLLM (Kwon et al., 2023). Evaluation uses lmms-eval (Zhang et al., 2024a); unless stated otherwise, we cap response length at 256 tokens and increase it to 4,096 for reasoning models. A.3. Reward Design We provide full details complementing Sec. 3.3. The base scalar rewardR task m,n is task-specific; efficiency enters later through CAPO advantage shaping rather than through a raw additive reward term. Base Task Reward (R task m,n ). We consider four task types: • Question Answering. For math problems, we extract the numeric answer and compare it to the ground truth within a tolerance of10 −2 . For multiple-choice questions, we extract the option letter. For other QA tasks, we compare normalized strings (e.g., case-folded, whitespace-stripped). This yields the binary reward R QA ( ˆ o, o)∈0, 1. • Free-form Generation. For open-ended tasks, we compute the ROUGE-L score between the generated answer ˆ o and the reference o: R Gen ( ˆ o, o) = ROUGE-L( ˆ o, o)∈ [0, 1]. • Temporal Grounding. Let the ground-truth segments beG =[s j , e j ] j and the predicted segments be ̂︀ G = [ ˆ s k , ˆ e k ] k (each set may contain one or multiple intervals). We compute the temporal IoU and select the best-matching pair: R TG ( ̂︀ G,G) =max [ ˆ s, ˆ e]∈ ̂︀ G, [s,e]∈G tIoU ( [ ˆ s, ˆ e], [s, e] ) ∈ [0, 1]. If no valid segment can be parsed from the output, we assign R TG ( ̂︀ G,G) = 0. • Grounding QA. We parse both the textual answer and the predicted temporal segments from the model output, compute R QA ( ˆ o, o) and R TG ( ̂︀ G,G), and sum them: R GQA ( ˆ o, ̂︀ G; o,G) = R QA ( ˆ o, o) + R TG ( ̂︀ G,G) ∈ [0, 2]. These task-specific metrics define the scalar base rewardR task m,n . CAPO additionally uses a binary success indicatoru m,n ∈0, 1: for exact-match QA tasks we use the binary correctness outcome directly, whereas for continuous metrics (ROUGE-L, temporal IoU, and their grounding-QA combination) we threshold the scalar score at0.35, matching the implementation. When format validation is enabled, a weighted format term is added before GRPO normalization, but u m,n is computed from the task metric alone. 25 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Table 6: Prompt template used for CAPO training. The template presents video frames and the task question, requires intermediate reasoning inside<think>tags, and places the final answer in within<answer>tags. This structure enables automatic reward extraction from MLLM outputs. Prompt Template for Training with Thinking System Prompt: You are a helpful assistant. You FIRST think about the reasoning process as an internal monologue and then provide the final answer. The reasoning process MUST BE enclosed within<think> </think>tags and the answer MUST BE enclosed within <answer> </answer> tags. The final answer MUST BE put in and the expression MUST BE contained entirely within the <answer> </answer> tags. Do not include any reasoning or explanations outside these tags. Format Reward. We employ a binary format rewardR fmt ( ˆ o)∈0, 1enforced via strict regex validation. The output must contain exactly one<think>...</think>block and one<answer>...</answer>block, with the final answer enclosed in \ ... within the <answer> tags: R fmt ( ˆ o) = ︃ 1 if format matches regex, 0 otherwise. In the implementation, malformed outputs receive a penalty before weighting, and the format term enters the scalar reward with weight 0.2. A.4. Prompt Template We employ the standard prompt for GRPO training, shown in Table 6. The model generates a reasoning trace within<think> </think>tags (optional for ResAdapt since reasoning is handled by the MLLMπ φ , but maintained for compatibility with reasoning-based baselines), followed by the final answer enclosed in \ . B. Complexity Analysis We derive formal computational bounds for ResAdapt to clarify when Allocator overhead is negligible relative to the savings induced in the backbone. For readability, we assume a standard Transformer backbone with quadratic self-attention and a uniform native resolutionH× WoverTframes; the extension to heterogeneous resolutions is immediate by replacing HW with per-frame products H t W t . Baseline cost. LetPdenote the ViT patch size. A vanilla MLLM encodingTframes at full resolution incurs a total visual token count of: N 0 = T· ⌈︂ H P ⌉︂⌈︂ W P ⌉︂ ≈ T HW P 2 .(26) 26 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Adaptive cost and token retention ratio. For the resize instantiation analyzed in this paper, framef t is rescaled by factors t ∈ [s min , s max ], producingn t (s t ) =⌈s t H/P⌉⌈s t W/P⌉≈ s 2 t · HW/P 2 tokens. Summing over the sequence and normalizing by N 0 yields the token retention ratio: N adapt = T ∑︁ t=1 n t (s t ) ≈ HW P 2 T ∑︁ t=1 s 2 t ,ρ ≜ N adapt N 0 = 1 T T ∑︁ t=1 s 2 t .(27) Because the learned Beta policy places most redundant frames nears min (Figure 5),ρis much smaller than1 in practice; across our evaluation suite, ρ∈ [0.06, 0.16]. Quadratic FLOPs reduction. For anL mllm -layer MLLM with hidden dimensionD mllm , self-attention cost scales quadratically in the visual sequence length:Φ(N) = O(L mllm N 2 D mllm ). SubstitutingN adapt = ρ· N 0 gives: Φ adapt mllm = O (︀ L mllm · ρ 2 N 2 0 · D mllm )︀ ,(28) a reduction by a factor ofρ 2 relative to full-resolution processing. At the representative operating point ρ = 0.11, we obtain ρ 2 ≈ 0.012, corresponding to roughly 83× fewer backbone attention FLOPs. Allocator overhead. The Allocator processes N c = T·⌈H/P c ⌉⌈W/P c ⌉ coarsely pooled tokens across L pred layers with dimension D pred , where P c ≫ P is the coarse spatial stride. Its cost and relative overhead are: Φ pred = O (︀ L pred · N 2 c · D pred )︀ , Φ pred Φ base mllm = O (︃ L pred D pred L mllm D mllm · (︂ P P c )︂ 4 )︃ ≪ 1.(29) Substituting our implementation parameters (P c =14,L pred =4,D pred =1,024versusL mllm =28,D mllm =3,584), the Allocator accounts for less than 3% of inference FLOPs. The decision stage is therefore small compared with the backbone computation it helps eliminate. Net speedup. Combining the above under the first-order approximationΦ base mllm ≫Φ pred : Speedup ≈ Φ base mllm Φ adapt mllm +Φ pred ≈ N 2 0 (N adapt ) 2 = 1 ρ 2 .(30) At ρ = 0.11, this again yields a theoretical reduction of roughly 83× in backbone attention computation. Temporal context scaling. The same savings admit a second interpretation in terms of temporal coverage. Under a fixed token budgetB, a vanilla MLLM can process onlyT 0 = BP 2 /(HW)full-resolution frames, whereas the resize instantiation of ResAdapt used in our experiments can processT 0 /ρadaptively resized frames. This yields an effective1/ρ≈ 6–16×increase in temporal horizon at comparable compute, which is exactly the trade-off exploited by the long-context experiments in Sec. 4.2. Remark (acceleration transparency). A practical consequence of Input-side adaptation is that the backbone still receives an ordinary visual-token sequence, only shorter. As a result, ResAdapt remains compatible with optimized attention stacks such as FlashAttention, vLLM (Kwon et al., 2023), and SGLang (Zheng et al., 2024) without kernel-level modifications. By contrast, model-side pruning and merging often create irregular token layouts that are harder to route through the same optimized kernels and may require fallback implementations or architecture-specific engineering. 27 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning C. Derivation of Joint RL Formulation This appendix collects derivations omitted from Sec. 3 for space and clarifies how the one-step contextual MDP (Contextual Bandit) introduced in Sec. 2.2 motivates the practical surrogate objectives optimized by ResAdapt. All derivations are stated for a single context (video and query); the full objective is the expectation over the datasetD. Notation. The prompt context isx = (q,V). The Allocator first samples latent actionsafrom a Beta policyq θ (a | x)(Sec. 3.2); the continuous allocationsis the deterministic image ofaunder Eq.(10), andπ θ (s | x)denotes the induced density (pushforward). A deterministic transformation constructs the operator-transformed input ̃x = (q,O( f t , s t ) T t=1 ); in the experimental instantiation,Ois bilinear resizing. The MLLM backbone policyπ φ (y| ̃x)then samples a full response rollouty = (r, o), whereris the reasoning trace and o is the final answer. C.1. One-Step Contextual MDP and the Joint Objective As defined in Sec. 2.2, the system is a one-step contextual MDP. In this setting, there are no sequential state transitions across time stepst; the episode terminates after the allocationsis sampled and the corresponding rolloutyis produced. Consequently, the value functions collapse to the immediate reward, and the standard Policy Gradient Theorem simplifies drastically without requiring temporal discount factors or credit assignment across Markov states. The joint distribution of the allocation and the rollout factorizes conditionally: p θ,φ (s, y| x) = π θ (s| x) π φ (y| ̃x).(31) For a single context with ground-truth answero ⋆ , the marginal answer probability under the transformed input is p θ,φ (o ⋆ | x) = E π θ (s|x) [︁ E π φ (r| ̃x) [︀ π φ (o ⋆ | ̃x, r) ]︀ ]︁ .(32) Equation(32)is the law of total expectation under an autoregressive factorizationπ φ (y | ̃x) = π φ (r | ̃x) π φ (o| ̃x, r): the inner term is the conditional probability (mass or density) of the ground-truth answer o ⋆ given the prefixr. Summing/integrating overryields the marginalP(o ⋆ | x)only under this generative ordering; the subsequent RL objective does not require Eq. (32) to hold in closed form. Becauselog(·)is monotone, maximizinglog p θ,φ (o ⋆ | x)would be equivalent, but the RL derivation below does not require introducing the logarithm. It only requires a scalar utility evaluated after sampling(s, y). We therefore abstract the answer-quality term as a rollout utilityQ(x, y), wherey = (r, o), and treat it as parameter-independent once the rollout is sampled. This is a modeling abstraction rather than an exact reformulation: whenQis chosen as an answer-aligned task score, the resulting RL problem is a surrogate to likelihood maximization. This lets us define the ideal rollout reward R ideal s,y = Q(x, y)− λ C(s),(33) and optimize the one-step expected return max θ,φ J (θ, φ) = E x∼D E π θ (s|x) [︁ E π φ (y| ̃x) [︁ R ideal s,y ]︁]︁ .(34) 28 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning C.2. Policy Gradient and Alternating Optimization Because the objective involves two distinct parameterized policies, its gradients follow the score-function estimator (the likelihood-ratio / REINFORCE identity). This is the underlying policy-gradient structure; GRPO/PPO does not change that structure, but replaces the raw reward with normalized advantages and clipped surrogates for practical optimization. Taking the gradient ofJ (θ, φ)with respect to the backbone parameters φ: ∇ φ J (θ, φ) = E x E π θ (s|x) [︂ ∇ φ ∫︁ π φ (y| ̃x)R ideal s,y dy ]︂ = E x E π θ (s|x) E π φ (y| ̃x) [︁ R ideal s,y ∇ φ log π φ (y| ̃x) ]︁ .(35) Similarly, the gradient with respect to the Allocator parametersθrelies on the marginalized rewardR ideal s = E π φ (y| ̃x) [R ideal s,y ]: ∇ θ J (θ, φ) = E x E π θ (s|x) [︁ R ideal s ∇ θ log π θ (s| x) ]︁ .(36) To optimize this objective with GRPO/PPO, we introduce importance sampling from behavior policiesπ θ old andπ φ old . A naive joint importance weight π θ π φ π θ old π φ old suffers from compounded variance. We therefore use an alternating block-coordinate ascent approximation. When updating the MLLM (φ), we fix the Allocator to its behavior policy (π θ = π θ old ), making its importance ratio exactly1. The off-policy surrogate gradient for φ becomes: ∇ φ J surr (φ) = E π θ old E π φ old [︂ π φ (y| ̃x) π φ old (y| ̃x) R ideal s,y ∇ φ log π φ (y| ̃x) ]︂ .(37) Using the log-derivative identity∇ φ r φ = r φ ∇ φ log π φ wherer φ = π φ /π φ old , this motivates the surrogate objective: L ideal φ = E π θ old E π φ old [︁ r φ (y| ̃x)R ideal s,y ]︁ .(38) Policy-gradient ascent onφincreasesL ideal φ (equivalently, training minimizes its negative); Sec. 3.4 implements the clipped PPO surrogate with advantages in place of R ideal . Conversely, when updating the Allocator (θ), we fix the backbone to its behavior policy (π φ = π φ old ). The corresponding ideal allocator surrogate is L ideal θ = E π θ old [︁ r θ (s| x) R ideal s ]︁ ,r θ (s| x) = π θ (s| x) π θ old (s| x) ,(39) whereR ideal s = E π φ old (y| ̃x) [R ideal s,y ] . In practice, this expectation is approximated by Monte Carlo rollouts under the frozen backbone. Sequential allocator–backbone updates within one iteration. The alternating derivation above fixes one policy while updating the other, so the importance ratio for the inactive policy is unity. In implementations that first update the Allocator fromθ old toθ ′ and then update the MLLM on the same rollout batch, trajectories are still drawn from the behavior pair(θ old , φ old )while the MLLM gradient is evaluated underφat fixed 29 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning VideoMME LongVideoBench MMVU MMMU-P MMMU-C MMMU-A LVBench 0.2 0.4 0.6 0.8 1.0 Mean Scale Factor per Video Predicted Scale Distribution Across Benchmarks Perception Reasoning Figure 8: Per-video mean scale across benchmarks. Kernel density estimates of the per-video mean scale ̄ s. Reasoning- heavy benchmarks shift toward larger ̄ sthan perception-heavy ones, indicating that the learned policy spends more fidelity where fine-grained evidence is more likely to matter. (x, a, y) . The change in the marginal over allocations between behavior and the post-allocator policy is corrected by the importance weightρ θ = q θ ′ (a | x)/q θ old (a | x) = π θ ′ (s | x)/π θ old (s | x)(affine map, Jacobian factor cancels in the ratio). Multiplying rollout-level advantages byρ θ before the token-level PPO surrogate forφimplements the standard importance-sampling correction when reusing trajectories drawn underq θ old while evaluating gradients atq θ ′ (support and clipping caveats as in ordinary PPO); this matches the practical “ispred” path in the codebase. C.3. Advantage Shaping and Monte Carlo Surrogates The ideal linear penalty−λC(s)insideR ideal often causes catastrophic collapse to minimum budgets. CAPO therefore replaces the raw reward with a cost-shaped, group-normalized advantageA s,y (denotedA m,n in the main text). This replacement is not an unbiased baseline transformation ofR ideal s,y : the CAPO signal depends on the sampled allocation, the rollout outcome, and the within-group cost statistics. Instead, it defines a deliberately biased surrogate objective that trades exact fidelity to the Lagrangian reward for lower variance and stronger budget control in practice. Applying PPO clipping to the exact joint ratios would couple all frame- and token-level factors, which is prohibitively noisy in practice. We therefore arrive at practical decoupled objectives. For a batch ofM allocations and N rollouts per allocation, the MLLM sequence-level surrogate is: L seq φ =− 1 MN M ∑︁ m=1 N ∑︁ n=1 min (︁ r (m,n) φ A m,n , clip(r (m,n) φ , 1−ε, 1+ε) A m,n )︁ .(40) This sequence-level loss is already approximate because it uses the CAPO-shaped advantage in place of the ideal reward. To achieve finer credit assignment for the autoregressive MLLM, we further factorizeπ φ (y| ̃x) into token-level probabilities, distribute the same rollout-level advantageA m,n to all tokens, and average 30 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning 0.30.40.50.60.70.80.9 Mean Scale Factor 0 50 100 150 200 250 Count (a) Scale Distribution by Duration Short (n=900) Medium (n=900) Long (n=900) 0.000.050.100.150.200.250.30 Intra-Video Scale Std 0 20 40 60 80 100 120 Count (b) Scale Diversity by Duration Short Medium Long ShortMedium Long 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy 67.8% 55.3% 50.8% (c) Accuracy by Duration Figure 9: VideoMME broken down by video duration. As clip duration grows, the policy lowers the average scale, increases within-video scale diversity, and faces lower task accuracy. Longer clips are therefore processed more aggressively and more selectively. over the sequence lengthL m,n . Equation(24)should therefore be read as the standard token-level PPO approximation to this sequence-level surrogate, not as an exact decomposition of the clipped joint ratio. Conversely, when updating the Allocator (θ), we fix the MLLM (π φ = π φ old ) and use the aggregated advantageA CAPO m = 1 N ∑︀ n A m,n . Because the Allocator’s output distribution factorizes conditionally across frames (Eq. 11), its score function decomposes additively: ∇ θ log π θ (s (m) | x) = T ∑︁ t=1 ∇ θ log Beta(a (m) t ; α t , β t ).(41) This additive log-probability structure supports low-variance frame-level credit assignment. Nevertheless, Eq.(21)remains a practical approximation to a trajectory-level clipped objective: conditional independence justifies decomposition oflog π θ , but not exact factorization of the nonlinear PPO clipping term. We use the per-frame surrogate because it is substantially more stable in large-scale training. D. Supplementary Experiments and Analysis This section first analyzes the learned allocation policy, then studies the two key ablation axes, and finally reports representative qualitative cases and a boundary-case transfer test beyond video. Unless otherwise noted, all plots use Qwen2.5-VL-7B with 32 uniformly sampled frames. D.1. Behavioral Analysis of the Learned Policy D.1.1. Benchmark-Level Budget Allocation Figure 8 shows a clear benchmark-level ordering even though the policy never observes benchmark labels during training. Averaged across datasets, reasoning-oriented tasks use slightly higher mean scales than perception-oriented ones (0.435 vs. 0.417), with MMMU-Adaptation at the high end and VideoMME at the low end. The pattern is consistent with the main claim of the paper: the policy is not enforcing a fixed compression rule, but adapting its operating point to the expected visual difficulty of the task family. 31 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Artistic Performance Life Record Film & Television Knowledge Multilingual Sports Competition 0.0 0.1 0.2 0.3 0.4 Mean Scale Factor 58% 53% 61% 60% 53% 59% VideoMME: Scale Allocation by Task Category (accuracy annotated) Figure 10: Scale allocation by VideoMME task category. Mean ̄ svaries substantially across categories, with larger budgets assigned to categories that contain crowded motion or finer local evidence. Accuracy annotations show that allocation is not a trivial proxy for which category is easiest. 0.000.050.100.150.200.250.30 Gini Coefficient (Selectivity) 0 2 4 6 8 Density p=2.762e-01 VideoMME Correct (n=1565) Incorrect (n=1135) 0.000.050.100.150.200.25 Gini Coefficient (Selectivity) 0 2 4 6 8 Density p=2.979e-02 MMMU-P Correct (n=198) Incorrect (n=102) 0.000.050.100.150.200.250.300.35 Gini Coefficient (Selectivity) 0 1 2 3 4 5 6 Density p=1.046e-01 LongVidBench Correct (n=740) Incorrect (n=597) Scale Selectivity (Gini Coefficient): Correct vs. Incorrect Predictions Figure 11: Selectivity versus prediction correctness on three representative benchmarks. Per-video Gini coefficients of the frame-level scales. Correct predictions tend to have higher Gini than incorrect ones, linking success to sharper concentration of resolution rather than merely larger average budgets. D.1.2. Long-Context and Semantic Structure Figure 9 is consistent with the long-context gains in the main paper. From short to long clips, the mean scale drops (0.342→0.336→0.332), but the within-video diversity rises (0.085→∼0.095). In other words, the policy does not merely compress longer videos more; it also becomes more selective inside them, which is exactly the regime where uniform resizing is least satisfactory. Figure 10 refines the same story within a single benchmark. The policy spends the most budget on Sports Competition and the least on Artistic Performance, suggesting that even within VideoMME it distinguishes categories that are dense and spatially demanding from those that are visually simpler. This complements the main benchmark tables: the appendix focuses on why retained budgets differ, while the main text already reports the exact realized retention ratios. 32 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Video-MMELongVideo Bench LVBenchMMVUV-MMMU (Perc.) V-MMMU (Comp.) V-MMMU (Adapt.) 0 20 40 60 80 100 Proportion (%) 57.4% 53.3% 33.6% 43.7% 57.0% 46.3% 20.7% 4.9% 4.5% 7.5% 9.0% 8.0% 8.0% 4.7% 5.6% 5.0% 9.0% 8.3% 8.3% 9.0% 34.3% 36.2% 56.9% 39.8% 25.7% 37.3% 62.3% Correctness Transition Distribution (Vanilla → ResAdapt) CCWCCWWW 01020304050 Error Introduction Rate (%) 0 10 20 30 40 50 Error Correction Rate (%) Video-MME LongVideo Bench LVBench MMVU V-MMMU (Perc.) V-MMMU (Comp.) V-MMMU (Adapt.) Net Beneficial Region Net Harmful Region Error Correction vs. Introduction Trade-off Figure 12: Sample-level robustness at 25% retention. Most originally correct predictions remain correct, but corrected and newly introduced errors are of comparable magnitude. Adaptive allocation is therefore selective rather than lossless. D.1.3. Selectivity and Success We next ask whether successful samples allocate budget more selectively within a clip. We quantify frame-level selectivity with the Gini coefficient of the predicted scales. High Gini means the policy concentrates budget on a small subset of frames; low Gini means the allocation is nearly uniform. Figure 11 shows that correct predictions consistently lie in the more selective regime, with the clearest separation on MMMU-P. This sharpens the mechanism claim of the appendix: success is associated not merely with keeping more pixels overall, but with concentrating them onto the frames that matter. Robustness and failure modes. A final question is whether adaptive compression preserves existing correct answers or merely swaps one error pattern for another. Figure 12 provides the right robustness interpretation for aggressive compression. Prediction stability remains high overall (about89%of originally correct samples stay correct in the aggregate summary), so the policy is not helping only by randomly perturbing the answer distribution. However, error correction and error introduction are close enough that the effect should be read as selective redistribution: the policy repairs some failures, but it can also lose fine-grained evidence, especially when the decisive cue is brief or visually simple. D.2. Ablation Studies D.2.1. Temporal Similarity Ablation We provide two complementary views of the temporal-similarity ablation: a cross-benchmark summary showing that the effect generalizes, and a single-benchmark diagnostic panel showing exactly how the allocation pattern changes. Figure 13 makes the role ofL sim unusually clear. Without it, the policy collapses to near-uniform scales on every benchmark (σ< 0.003); with it, the same model family recovers substantial within-video variation, with4×–693×larger diversity depending on the benchmark. CAPO therefore controls where the global budget should sit, whereasL sim prevents the trivial fixed-scale solution. 33 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning VideoMME LongVideoBench MMVUMMMU-PLVBench 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Intra-video Scale Std Dev ( ) 0.002 0.091 43× 0.033 0.133 4× 0.045 0.091 2× 0.000 0.095 693× 0.000 0.092 561× w/o sim w/ sim Figure 13: Cross-benchmark scale diversity with and withoutL sim . Per-video scale standard deviationσacross five benchmarks. Without the regularizer, diversity collapses toward zero; addingL sim restores broad within-video variation on every benchmark. Quantitative confirmation. Figure 14 shows that this is not an artifact of any single statistic. The regularizer changes the global histogram, the per-video range, the frame-to-frame variation, and the Gini coefficient in the same direction, confirming that the benefit is structural rather than metric-specific. D.2.2. Reward Design Ablation We next examine whether different reward designs preserve a non-degenerate adaptive regime during training. All plots use EMA smoothing to suppress per-step noise; raw values remain visible as translucent traces. Per-sample scale adaptivity. Figure 15 complements Figure 6 by measuring the per-sample scale range s max − s min rather than the mean. CAPO preserves non-trivial adaptivity on validation, whereas direct cost collapses to the lower boundary and cost-free optimization drifts toward a nearly uniform high-scale policy. Convergence and stability. Figure 16 explains why CAPO works and the simpler baselines do not. The CAPO variants converge to stable interior solutions, whereas accuracy-only training saturates nears max and direct cost collapses tos min . This is consistent with CAPO’s intended role: balancing task reward and budget pressure without falling into either trivial boundary solution. The key result is therefore not merely convergence, but convergence to a non-degenerate operating point where content-adaptive allocation is still available. D.3. Qualitative Case Studies We present four representative case studies that complement the aggregate analysis above: two task-contrast examples from Video-MMMU, one evidence-localization success from VideoMME, and one failure case. Each visualization (Figures 17–20) renders 32 uniformly sampled frames at their assigned scale inside a fixed grid; warmer borders indicate larger predicted scales. 34 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning 0.20.40.60.81.0 Frame Scale s t 0 5 10 15 20 25 30 Density w/o: =0.376, =0.176 w/: =0.337, =0.169 (a) Frame-Level Scale Distribution w/o sim w/ sim 0.00.20.40.60.8 Per-Video Scale Range (maxs t mins t ) 0 10 20 30 40 50 60 Density w/o med: 0.000 w/ med: 0.361 (b) Intra-Video Scale Range w/o sim w/ sim 0.000.050.100.150.200.25 Mean Adjacent-Frame Scale ( |s t s t + 1 | ) 0 25 50 75 100 125 150 175 200 Density w/o med: 0.0000 w/ med: 0.0479 (c) Frame-to-Frame Scale Variation w/o sim w/ sim 0.000.050.100.150.200.250.300.35 Gini Coefficient (Scale Selectivity) 0 20 40 60 80 100 120 140 Density w/o med: 0.0000 w/ med: 0.0992 (d) Scale Selectivity (Gini) w/o sim w/ sim Figure 14: Four diagnostics of theL sim ablation on VideoMME. With the regularizer, the frame-scale histogram becomes bimodal, the per-video range expands, adjacent-frame variation increases, and the Gini coefficient rises. The policy moves from near-uniform allocation to a genuinely selective regime. Task-dependent operating regimes. Figures 17 and 18 contrast two Video-MMMU tasks from a visually similar educational domain that nevertheless demand very different allocations. In the comprehension example, the relevant evidence is concentrated in a small set of diagram-bearing slides, so the policy adopts a sparse operating regime and suppresses the explicitly irrelevant quiz frame. In the adaptation example, the downstream reasoning depends on reading a dense numeric table, so the same policy shifts to a much higher-budget regime and preserves high fidelity much more broadly. The contrast shows that the policy responds to what the task will require, not just to generic visual clutter. Evidence localization and failure. The VideoMME success case in Figure 19 shows a more local version of the same phenomenon: the answer depends on short text overlays embedded in otherwise repetitive footage, and the policy magnifies only those evidence-bearing moments. Figure 20 shows the failure mode that remains. The decisive cue is temporally brief and visually simple, so the policy enlarges a nearby frame but compresses the frame that actually contains the fork. This diagnosis matches the quantitative robustness analysis: ResAdapt is strong at concentrating budget, but still vulnerable when the decisive evidence is both 35 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning 0100200300400500 Training Step 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Scale Range s max − s min (a) Training: Per-Sample Adaptivity 0100200300400500 Training Step (b) Validation: Per-Sample Adaptivity β-Dist. w/o Cost β-Dist. w Direct Cost β-Dist. w CAPO -Dist. w CAPO Figure 15: Per-sample scale adaptivity under different reward designs. Scale ranges max − s min over training on (a) training and (b) validation splits. CAPO keeps a non-trivial adaptive range, whereas direct cost collapses and cost-free training saturates. 0100200300400500 Training Step 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Rolling Mean of ̄ s ( w =5) (a) Smoothed Convergence (Validation) β-Dist. w/o Cost β-Dist. w Direct Cost β-Dist. w CAPO -Dist. w CAPO 0100200300400500 Training Step 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Rolling Std of ̄ s ( w =5) (b) Scale Stability (Validation) β-Dist. w/o Cost β-Dist. w Direct Cost β-Dist. w CAPO -Dist. w CAPO Figure 16: Validation-time convergence under different reward designs. CAPO variants converge to stable intermediate operating points, while cost-free training saturates at the upper boundary and direct cost collapses to the lower boundary. Stability alone is not sufficient; the key is where the policy stabilizes. subtle and short-lived. Summary. Together, these case studies support the same three conclusions as the quantitative appendix: the policy changes its operating regime with the task, concentrates fidelity on evidence-bearing frames, and fails in interpretable ways when subtle cues are missed. The qualitative examples therefore reinforce the claim that ResAdapt learns a meaningful input-allocation strategy rather than a fixed compression heuristic. D.4. Boundary-Case Transfer Beyond Video The paper’s main claims target video QA and temporal grounding, so we place image transfer at the end of the appendix as a boundary-case analysis rather than as supporting evidence for the main contribution. Table 7 is still informative: the learned video policy sometimes identifies image inputs that warrant additional fidelity, as in ChartQA, but it does not yet yield reliable efficiency-preserving transfer on text-dense image tasks. The result is therefore best read as scope clarification. It suggests that input-side adaptation is broader than the resize-on-video setting studied here, while also showing that a video-trained policy should not be assumed to transfer cleanly to static images. 36 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Table 7: Exploratory zero-shot transfer to image benchmarks. Parenthetical values denote per-task retention ratio R, and ResAdapt-RL additionally fine-tunes the MLLM via RL. Model MathVista testmini MMMU val OCRBench ChartQAAI2D TextVQA val Qwen2.5-VL-7B49.1(100%) 50.9(100%) 84.2(100%) 83.9(100%) 82.5(100%) 82.9(100%) Random Drop44.8(50%) 49.0(50%) 74.8(50%) 71.6(50%) 80.3(50%) 78.1(50%) ToMe (Bolya et al., 2022)46.2(50%) 49.6(50%) 79.3(50%) 78.1(50%) 81.9(50%) 81.2(50%) VisionZip (Yang et al., 2025c) 47.2(50%) 48.6(50%) 79.6(50%) 77.9(50%) 81.9(50%) 81.3(50%) ResAdapt(Qwen2.5-VL-7B)45.5(42%)51.0(29%)80.0(64%)85.9(105%)81.4(41%)69.6(30%) ResAdapt-RL(Qwen2.5-VL-7B)46.7(42%)50.9(29%)80.8(64%)86.6(105%)81.1(41%)70.1(30%) Qwen3-VL-8B56.1(100%) 53.4(100%) 85.0(100%) 84.0(100%) 83.5(100%) 82.1(100%) Random Drop47.3(50%) 48.7(50%) 62.9(50%) 70.2(50%) 79.7(50%) 76.6(50%) VisionZip (Yang et al., 2025c) 47.8(50%) 50.3(50%) 70.5(50%) 75.0(50%) 80.5(50%) 79.3(50%) ToMe (Bolya et al., 2022)49.6(50%) 50.6(50%) 70.3(50%) 75.2(50%) 80.5(50%) 79.4(50%) ResAdapt(Qwen3-VL-8B)52.5(42%)50.9(29%)82.7(64%)83.2(105%)81.2(41%)67.8(30%) 37 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Q: Evaluate five statements about Urban Geography City Models (concentric zone, Hoyt sector, multiple nuclei, galactic, Latin American); identify which are correct. Please ignore the Quiz question in last frame of the video. 0.250.300.350.400.450.500.550.600.65 Adaptive Scale Factor (cell = 0.65×) Figure 17: Case 1: Video-MMMU Comprehension (Hu et al., 2025) (Vanilla× →ResAdapt✓). The policy concentrates resolution on diagram-bearing slide frames, compresses lecturer-only frames, and suppresses the final quiz frame that the prompt explicitly marks as irrelevant. 38 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Q: Watch and learn the video content. Then apply what you learned to answer: Table 11.47 provides a survey of the youngest online entrepreneurs (ages 17–30) whose net worth≥$1M. We want to know whether ages and net worth are independent. χ 2 test statistic = ______ 0.30.40.50.60.70.80.9 Adaptive Scale Factor (cell = 0.95×) Figure 18: Case 2: Video-MMMU Adaptation (Hu et al., 2025) (Vanilla× →ResAdapt✓). When the answer depends on reading a numeric table and performing aχ 2 computation, the policy keeps a much higher global budget and strongly upscales the table-bearing frames. 39 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Q: When is the zodiacal light visible from the video? (A) Mar. 19, (B) Mar. 24, (C) Mar. 25, (D) Mar. 29. 0.250.300.350.400.450.500.550.60 Adaptive Scale Factor (cell = 0.60×) Figure 19: Case 3: VideoMME (Fu et al., 2025a) (Vanilla× →ResAdapt✓). Frames containing the decisive date overlays are enlarged, while the largely homogeneous sky footage is compressed. The policy spends budget on answer-bearing evidence rather than on the surrounding context. 40 ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning Q: Which item does the man throw into the trash at the beginning of the video? (A) A fork, (B) A pair of chopsticks, (C) A box of noodles, (D) A spoon. 0.30.40.50.60.7 Adaptive Scale Factor (cell = 0.75×) Figure 20: Case 4: VideoMME (Fu et al., 2025a) (Vanilla✓ →ResAdapt×; failure case). A nearby frame is enlarged, but the actual fork-bearing frame is compressed. The decisive fine detail is therefore lost at exactly the wrong moment. 41