Paper deep dive

Not All Visual Tokens Are Equally Safe to Remove:Consequence-Sensitive Visual Token Compression

Jingbo Wen, Liang He, Mingyu Cao, Haoyu Wang, Minxuan Hu, Kangning Cui, Xilu Wang

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 64

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Visual token compression for vision--language models (VLMs) has largely relied on criteria such as attention, redundancy, and uncertainty to maximize average accuracy under a fixed compute budget, implicitly assuming that all errors carry equal cost. However, the consequence of an incorrect prediction on downstream tasks is rarely symmetric: misreading an invoice amount can be far more costly than misclassifying a background color. Motivated by this, we introduce consequence-sensitive visual token compression, which allocates visual computation across requests according to their potential error costs. Our method follows a calibrate-then-allocate procedure, estimating consequence-specific error-budget curves offline and applying the calibrated token budgets online using consequence signals available from question or task information. On a controlled within-task benchmark, high- and low-consequence questions are drawn from the same document images, so content alone cannot reveal which questions are costly to get wrong. In this setting, our method reduces high-stakes errors from 0.300 to 0.133 under the same total token budget, whereas a content-driven allocator performs no better than uniform allocation. Measuring how error rates change with token budget across different cost ratios, we derive an allocation frontier: uniform allocation is optimal when errors are equally costly, and token transfer toward high-consequence questions becomes increasingly beneficial as the cost gap grows. This allocation principle generalizes well across three dense vision-language benchmarks, two budget realization mechanisms (token deletion and resolution reallocation), two VLM architectures, and multiple token selection strategies. On a realistic mixed workload, consequence-sensitive allocation reduces cost-weighted error by 38% while achieving approximately 21% lower latency than full-resolution inference.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

62,333 characters extracted from source content.

Expand or collapse full text

Not All Visual Tokens Are Equally Safe to Remove: Consequence-Sensitive Visual Token Compression Jingbo Wen1 Liang He2,∗ Mingyu Cao3 Haoyu Wang4 Minxuan Hu5 Kangning Cui6 Xilu Wang3 1The University of Sydney 2Tongji University 3University of Surrey 4Nankai University 5Cornell University 6City University of Hong Kong hel9919@163.com ∗Corresponding author Abstract Visual token compression for vision–language models (VLMs) has largely relied on criteria such as attention, redundancy, and uncertainty to maximize average accuracy under a fixed compute budget, implicitly assuming that all errors carry equal cost. However, the consequence of an incorrect prediction on downstream tasks is rarely symmetric: misreading an invoice amount can be far more costly than misclassifying a background color. Motivated by this, we introduce consequence-sensitive visual token compression, which allocates visual computation across requests according to their potential error costs. Our method follows a calibrate-then-allocate procedure, estimating consequence-specific error-budget curves offline and applying the calibrated token budgets online using consequence signals available from question or task information. On a controlled within-task benchmark, high- and low-consequence questions are drawn from the same document images, so content alone cannot reveal which questions are costly to get wrong. In this setting, our method reduces high-stakes errors from 0.300 to 0.133 under the same total token budget, whereas a content-driven allocator performs no better than uniform allocation. Measuring how error rates change with token budget across different cost ratios, we derive an allocation frontier: uniform allocation is optimal when errors are equally costly, and token transfer toward high-consequence questions becomes increasingly beneficial as the cost gap grows. This allocation principle generalizes well across three dense vision-language benchmarks, two budget realization mechanisms (token deletion and resolution reallocation), two VLM architectures, and multiple token selection strategies. On a realistic mixed workload, consequence-sensitive allocation reduces cost-weighted error by 38% while achieving approximately 21% lower latency than full-resolution inference. 1 Introduction Vision tokens dominate the inference cost of modern vision–language models (VLMs), motivating a growing body of work on visual token compression. Existing approaches reduce token computation using signals derived from the input itself, including attention-based importance (Chen et al. 2024), feature redundancy (Zhang et al. 2025; Yang et al. 2025), and layer-wise reduction schedules (Xing et al. 2025). These methods have achieved strong accuracy–efficiency trade-offs by optimizing a common objective, i.e., preserving average accuracy while reducing the number of processed visual tokens. This objective implicitly assumes that all errors have equal consequences on downstream tasks. Figure 1: Why consequence matters. Existing compression treats all errors equally, whereas consequence-sensitive compression prioritizes samples where mistakes are more costly. However, real-world deployments rarely satisfy this assumption, as illustrated in Figure 1. In document, chart, and infographic understanding, VLM outputs often support downstream decisions and workflows. Misreading an invoice amount, transaction identifier, or medical value can trigger different consequences than misreading a section title or a decorative attribute, so the cost of an error is inherently asymmetric across samples. Importantly, such asymmetry is often available before inference through request-level information, such as question type, task identity, or application workflow. Existing visual token allocation strategies cannot exploit this signal: attention, redundancy, and uncertainty describe properties of the visual input, but do not capture the downstream consequence of being wrong. This observation leads to a different perspective on visual token compression. Instead of asking which visual tokens are redundant, we ask where a limited visual computation budget should be allocated when different errors have different consequences. Given a set of requests with per-sample consequence weights cic_i and a fixed total vision-token budget, we study how to assign per-image token budgets to minimize cost-weighted error (CWE). This formulation raises three fundamental questions: 1) Does consequence information provide value beyond content-based allocation signals? To answer this, we construct a within-task benchmark where high- and low-consequence questions share the same distribution of dense document images. Since visual content is identical across consequence tiers, content-based allocators cannot exploit consequence information, allowing the effect of cost-aware allocation to be isolated. 2) When does consequence-sensitive allocation outperform uniform allocation, and how should tokens be transferred under different levels of error asymmetry? We characterize the relationship between token budgets, error reduction, and consequence ratios through measured error–budget curves. 3) Does this effect depend on a particular token reduction mechanism, model architecture, or token selector, or does it reflect a general allocation principle? We test this by varying each factor independently while holding the allocation principle fixed. Our contributions are summarized as follows: • We introduce consequence-sensitive visual token compression, which allocates visual computation by minimizing cost-weighted error under a strict equal total token budget. Unlike existing adaptive pruning approaches that determine budgets from content-derived signals, our new formulation incorporates inference-time-observable consequence information to guide computation allocation. • We design a within-task evaluation protocol where high- and low-consequence questions are paired with the same image distribution, eliminating content-based shortcuts. Under this setting, consequence-sensitive allocation reduces high-stakes error by 2.25×2.25× under the same realized token budget. • We characterize how the optimal visual token allocation changes with error asymmetry. By analyzing measured error–budget curves, we identify a transition regime where uniform allocation is optimal under symmetric costs and token transfer toward high-consequence samples becomes increasingly beneficial as cost asymmetry grows. A calibrate-then-allocate procedure recovers near-optimal operating points on held-out data. • The allocation principle generalizes across information-dense benchmarks, i.e., DocVQA, ChartQA, and InfographicVQA; two budget realization mechanisms, i.e., token deletion and resolution reallocation; two VLM architectures; and multiple token selection strategies. On a realistic mixed workload, it reduces cost-weighted error by 38%38\% while achieving substantial latency reduction compared with full-resolution inference. 2 Related Work Visual token pruning and selection. Visual token compression has become an important direction for efficient VLM inference. Various methods, such as FastV (Chen et al. 2024), SparseVLM (Zhang et al. 2025), PyramidDrop (Xing et al. 2025), and VisionZip (Yang et al. 2025), reduce visual computation by identifying less informative tokens using signals such as attention, redundancy, or similarity patterns. These approaches primarily address the within-image question of which tokens should be retained under a given compression ratio. In contrast, we study a complementary problem: given a global token budget across requests, how should it be distributed across samples with different error consequences? We show this allocation principle is orthogonal to token selection and holds across different selectors. Adaptive visual token allocation. Recent methods further adapt the amount of visual computation per input. AdaptVision learns per-sample visual acquisition policies using reinforcement learning (Lin et al. 2025); AVIS adapts visual context and reasoning scale using redundancy and difficulty signals (Jeddi et al. 2026); and PSCA and COAST adjust budgets according to information density or uncertainty-related cues (Fang et al. 2026; Ma et al. 2026). These methods allocate computation based on input properties or prediction difficulty. Orthogonally, consequence-sensitive allocation considers varying consequences by using inference-time-observable cost signals, such as question type or task identity, and we separate this axis from content-driven allocation in controlled within-task experiments. Cost-sensitive computation. Cost-sensitive learning incorporates unequal error costs into prediction objectives (Elkan 2001). Related ideas appear in adaptive inference, including cost-sensitive early exit and selective prediction, where computation or coverage is adjusted according to risk and budget constraints (Valade et al. 2024). Recent consequence-aware reasoning compute allocation further shows that language-model reasoning budgets can be allocated by downstream error consequence rather than difficulty alone (Wen et al. 2026). We extend this principle to VLM inference, where the controllable resource is visual token computation and consequence must be disentangled from visual redundancy and token sensitivity. Vision token deletion with multimodal positional encoding. Practical evaluation of visual token compression requires faithful removal of visual information rather than masking or replacement. This is challenging for modern VLMs with multimodal positional encoding mechanisms, such as Qwen2.5-VL, where visual positions encode spatial information. Recent studies have analyzed positional integrity issues introduced by token pruning (Li and Zhang 2025; Huang et al. 2025). We implement exact visual token deletion by preserving the original spatial coordinates of retained tokens and performing decoding with explicitly controlled embeddings and positions. This implementation enables faithful evaluation of token allocation strategies and is provided as a reproducibility artifact. 3 Consequence-Sensitive Token Compression Figure 2: Overview of consequence-sensitive visual token compression. During offline calibration, validation outcomes are used to estimate tier-specific error–budget curves and select calibrated budgets for a given total token budget. During online inference, each request is assigned to a consequence tier from question or task information, and the calibrated budget is applied before VLM inference. The feedback loop updates error–budget curves only during offline calibration. Figure 2 summarizes the proposed framework. The method consists of an offline calibration stage and an online inference stage. During the offline stage, consequence signals and validation outcomes are used to estimate tier-specific error–budget curves and select a calibrated budget policy. During the online stage, the policy is fixed: each request is assigned to a consequence tier using inference-time-observable question or task information, and the corresponding visual-token budget is realized before VLM inference. 3.1 Problem Each sample i carries a consequence weight ci>0c_i>0, the cost incurred if the model answers it incorrectly, and receives a vision-token budget bib_i. Samples are grouped into tiers t∈t with common weight ctc_t, tier size ntn_t, and a tier-level error-vs-budget curve et(b)e_t(b), measured with the token selector held fixed. Given a hard total budget B, we seek minbt∑tntctet(bt)s.t.∑tntbt≤B,bt≥bmin, _\b_t\\; _tn_t\,c_t\,e_t(b_t) .t. _tn_tb_t≤ B,\;\;b_t≥ b_ , (1) where bminb_ is the smallest measured budget (a grid edge, not a known system floor). In our two-tier instantiation (nhi=nlo=n_hi=n_lo=n, chi=rc_hi=r, clo=1c_lo=1, Bhi+Blo=608B_hi+B_lo=608, i.e., 304 tokens/image on average), the objective equals n(r+1)n(r+1) times the cost-weighted error J(Bhi)=rEhi(Bhi)+Elo(608−Bhi)r+1,J(B_hi)= r\,E_hi(B_hi)+E_lo(608-B_hi)r+1, (2) so minimizing either is equivalent. 3.2 Allocation Principle For a continuous, differentiable relaxation, the Lagrangian ℒ=∑tntctet(bt)+λ(∑tntbt−B)−∑tμtnt(bt−bmin)L= _tn_tc_te_t(b_t)+λ( _tn_tb_t-B)- _t _tn_t(b_t-b_ ) with λ,μt≥0λ, _t≥ 0 gives ctet′(bt)=μt−λc_te_t (b_t)= _t-λ; since every ete_t is decreasing, λ>0λ>0 and the budget constraint is tight, and for tiers with inactive floor (μt=0 _t=0) the cost-weighted marginal error reduction ct|et′(bt)|=λc_t|e_t (b_t)|=λ is equalized—the classical water-filling rule in the convex case. Our measured problem violates both premises of that rule: budgets live on a discrete 13-point grid, which invalidates derivative-based conditions outright, and the curves are non-convex (finite-difference marginals of EhiE_hi are non-monotone: −0.043-0.043, −0.023-0.023, −0.064-0.064 per 32 tokens across 32→64→96→12832→64→96→128), so even on the relaxation stationarity does not imply local minimality. The appropriate discrete first-order condition is exchange-based: at a grid optimum, moving one grid step of budget from either tier to the other does not decrease J. Our allocator therefore minimizes the measured J by exhaustive search over the feasible pairs; the swap condition holds at every reported argmin by construction, and the continuous marginal condition is used only descriptively, where it happens to hold. 3.3 Calibrate-Then-Allocate The allocation procedure follows a calibrate-then-allocate strategy. First, we estimate the tier-specific error–budget curves e^t(b) e_t(b) on a calibration split by evaluating each candidate budget in the predefined grid. We then select the optimal allocation by minimizing the estimated cost-weighted objective: (Bhi,Blo)=arg⁡minBhi,Blo⁡J^(Bhi,Blo),(B_hi,B_lo)= _B_hi,B_lo J(B_hi,B_lo), (3) subject to the fixed total budget constraint. During inference, each query is assigned to a consequence tier using an inference-time-observable signal. In our experiments, this signal is obtained either from a frozen keyword rule over the question text (e.g., amounts, dates, counts, and identifiers ⇒ high consequence; descriptive queries ⇒ low consequence) or from task identity in the mixed-workload setting. The corresponding tier budget is then applied to the input. The calibration stage requires O(|grid|×|calibrationset|)O(|grid|×|calibration\ set|) offline forward passes, while online inference overhead is limited to lightweight tier assignment. 3.4 Two Allocation Mechanisms We adopt absolute token transfer rather than proportional allocation because preliminary analysis shows proportional shifting suffers from token dilution in dense visual inputs (detailed in Technical Supplement Section 2: Proportional Allocation and Token Dilution). A per-image budget can be realized by (a) true token deletion: encode at full resolution, rank tokens with a fixed selector, and delete the rest exactly (shortened embeddings and position ids; zero-masking instead of deletion produces out-of-distribution zero vectors and collapses generation); or (b) resolution reallocation: resize the image so the encoder emits approximately the budgeted token count, then run the standard generate path unmodified. Mechanism (b) requires no model-specific code, applies to any VLM, and—as we show—dominates (a) on document images for Qwen2.5-VL; we therefore treat (b) as the deployment mechanism and (a) as the analysis instrument that additionally enables selector-level control. 3.5 A Two-Axis Map of Allocation Regimes Question classes can differ along two orthogonal axes: a sensitivity gap (do their marginal error reductions per token differ?) and a cost gap (do their errors cost differently?). In cross-task benchmarks (DocVQA vs. synthetic solid-color images), there is a sensitivity gap and no cost gap: a content-driven allocator, reading only the image, routes tokens to the sensitive class and wins; cost labels carry no additional information by definition. Our within-task benchmark works differently by design: both classes see the same dense-document images and their measured token sensitivities are statistically indistinguishable across the budget grid, so an image-only allocator is blind by design; a cost-blind allocator is also definitionally handicapped on a cost-weighted metric, so the empirical content of that comparison is the gap’s magnitude, not its sign. When neither gap exists (saturated synthetic images) no allocator helps; the both-gaps cell is untested in this paper, and composition there is a hypothesis, not a result. In the regimes we test, consequence-sensitive allocation is necessary exactly when a cost gap exists without a sensitivity gap that image content could proxy. 4 Experimental Setup Model and mechanisms. Qwen2.5-VL-7B (Qwen Team 2025) (bf16, A100) is the primary model, with LLaVA-OneVision-7B (Li et al. 2024) as the second architecture. Unless stated otherwise, budgets are realized by true deletion with a fixed redundancy selector (keep the most mutually diverse tokens); the resolution mechanism and alternative selectors are evaluated in the invariance study. Greedy decoding, ≤ 16 new tokens, questions suffixed with "Answer briefly." Benchmarks. Within-task (main): DocVQA validation (Mathew et al. 2021), images resized to max side 1008 (≥ 640 native tokens), split by a frozen keyword rule on the question text into 300 transactional (high, c=5c=5) and 300 descriptive (low, c=1c=1) questions, drawn disjointly from a 120-sample pilot. Cross-task (diagnostic): 60 DocVQA questions vs. 60 color-identification questions on 100821008^2 solid-color images (1296 native tokens each, so every strategy’s budget is feasible). Generalization: ChartQA test (Masry et al. 2022) and InfographicVQA validation (Mathew et al. 2022), 200/200 each, same frozen rule verbatim. Mixed workload: 120 VQAv2 (Goyal et al. 2017) (low, c=1c=1), 120 TextVQA (Singh et al. 2019) (medium, c=3c=3), 120 DocVQA (high, c=5c=5), tiers given by task identity. Equal-budget protocol. The cost-aware assignment fixes the total budget B; uniform gives every image B/NB/N; anti reverses the tiers (direction control); content allocates B proportionally to each image’s feature-level information density (1 −- mean pairwise cosine similarity), clamped and redistributed to preserve B exactly. Every run asserts, per image, that the assigned budget does not exceed the native token count (no silent clamping) and, per strategy, that realized totals are exactly equal (deviations for the discrete-resolution mechanisms are reported: ≤ 0.4% for resolution, ≤ 0.02% for LLaVA rung alignment). Statistics. Per benchmark, the pre-registered primary comparison is high-tier error, uniform vs. cost-aware, by exact McNemar on paired correctness; ΔCWE _CWE is assessed by paired bootstrap (10k resamples) with the same weights used for allocation; answers are graded by a normalized containment match shared by all arms. We report discordant counts b:cb:c (uniform-wrong/cost-right vs. converse) throughout. Additional implementation details, including dataset construction, budget verification, and statistical protocols, are provided in the Technical Supplement, Section 1 (Full Experimental Setup and Implementation Details). 5 Results 5.1 Attribution: When Do Cost Labels Matter? Cross-Task Analysis: Separating Content Effects from Consequence Effects. Table 1 and Table 2 realize the two tested cells of the map in Section 3.5. We first examine a cross-task setting where content information can reveal token sensitivity differences. In this setting, the content allocator squeezes saturated solid-color images to the floor and outperforms even our cost-aware assignment, showing that content signals alone are sufficient when visual sensitivity differs across tasks. Therefore, gains observed in such settings cannot be attributed to consequence information, motivating a controlled within-task evaluation. Additional task-level sensitivity diagnostics and failure cases are analyzed in the Technical Supplement, Section 6 (Task-Level Sensitivity Diagnostics). Within-Task Evaluation: Isolating Consequence-Aware Allocation. To isolate the effect of consequence information, we construct a within-task setting where high- and low-consequence questions share the same document image distribution. Under this setting, the content allocator is blind to consequence differences because visual inputs are identical across tiers. It therefore assigns nearly uniform budgets and becomes statistically indistinguishable from uniform allocation, while consequence-aware allocation converts 54 high-stakes errors against only 4 reversals. The trade-off introduced by consequence-aware allocation is explicit: descriptive-tier error increases to 0.843, resulting in worse unweighted accuracy (0.488 vs. 0.337 for uniform). This demonstrates that the method is optimizing the intended cost-weighted objective rather than simply recovering an accuracy-maximizing allocation. Two further checks support this attribution. First, the two tiers’ error-vs-budget curves are statistically indistinguishable in slope across the 13-point grid (central-difference marginals at the uniform split: 0.001730.00173 vs. 0.001710.00171, slope SE ≈2×10−4≈2×10^-4), indicating that the observed improvement is not caused by high-consequence samples being inherently more token-sensitive. Second, our earlier proportional-budget benchmarks primarily reflected token sensitivity differences; the absolute token-transfer design adopted here removes this confound and isolates the contribution of consequence information. For attribution, Table 2 uses a pre-specified absolute-transfer allocation rather than the post-hoc grid optimum; the calibrated optimum is studied separately in Section 5.2. Strategy High err. Low err. CWE (r=5r=5) uniform (304/img) 0.400 0.000 0.333 content 0.050 0.000 0.042 cost-aware (512/96) 0.117 0.000 0.097 anti 0.783 0.000 0.653 Table 1: Cross-task diagnostic (DocVQA vs. solid-color, N=120N=120, equal realized budget 36,480). With a sensitivity gap and no real cost gap, the content allocator wins without cost labels—this setting cannot attribute gains to consequence. Strategy High Low Avg CWE McNemar err. err. err. (r=5r=5) (b:cb:c, p) uniform 0.300 0.373 0.337 0.312 — content 0.300 0.383 0.342 0.314 4:4, 1.01.0 cost-aware 0.133 0.843 0.488 0.252 54:4, ≈3×10−12≈3×10^-12 anti 0.837 0.130 0.484 0.719 2:163 Table 2: Within-task attribution result (DocVQA, N=600N=600, equal realized budget 182,400, pre-specified budgets (512, 96)). The image distribution is identical across tiers: the content allocator assigns ≈uniform budgets (305/303) and performs like uniform, while consequence-sensitive allocation cuts high-stakes error 2.25×2.25×. Avg err. (unweighted, equal tier sizes) rises for cost-aware (0.488 vs. 0.337), showing that the method optimizes cost-weighted error rather than average accuracy. The calibrated optimum under r=5r=5 is analyzed separately in the allocation frontier. Do the tiers track real consequence? The frozen rule is a keyword proxy, so we validate it two ways on the confirmatory 600. Additional validation of the consequence signals is provided in the Technical Supplement, Section 7 (Validation of Consequence Signals). Against gold-answer content, 90.3%90.3\% of high-tier questions have a quantitative gold answer (amount, date, count, code) versus 32%32\% of low-tier ones (κ=0.58κ=0.58); against an independent frontier-LLM consequence judge, blind to the rule, on a stratified 200-question sample, agreement is 72.5%72.5\% (κ=0.45κ=0.45). Disagreements are systematic: the rule over-labels navigational numbers (page/table/chapter) as high and, lacking the keywords budget/tel/zip, leaks some genuinely transactional questions into low—which makes the reported effect conservative, as high-value questions are then under-served. Deployment does not require the hand-written rule: a lightweight classifier predicts the tier from question text with 94%94\% 5-fold accuracy and better matches the judge’s consequence labels than the rule itself. Additional details are provided in the Technical Supplement, Section 9 (Automatic Consequence Signal Prediction). 5.2 The Allocation Frontier Figure 3: The allocation frontier (within-task DocVQA, N=600N=600). (a) High- and low-consequence questions have nearly identical error–budget curves, showing that the allocation gain is not caused by different token sensitivity. (b) Cost-weighted error as a function of the high-tier budget BhiB_hi under different consequence ratios with a fixed total budget of 304 tokens per image on average. The optimum remains uniform at r=1r=1 and shifts toward high-consequence samples as the consequence ratio increases. Token-Response Curves (Equal Sensitivity). (Bhi,Blo)(B_hi,B_lo) EhiE_hi EloE_lo CWE at cost ratio r r=1r=1 r=3r=3 r=5r=5 r=10r=10 (304, 304) 0.300 0.373 0.337 0.318 0.312 0.307 (384, 224) 0.230 0.520 0.375 0.302 0.278 0.256 (448, 160) 0.160 0.687 0.423 0.292 0.248 0.208 (480, 128) 0.143 0.790 0.467 0.305 0.251 0.202 (512, 96) 0.133 0.843 0.488 0.311 0.252 0.198 (544, 64) 0.120 0.873 0.497 0.308 0.246 0.188 (576, 32) 0.100 0.910 0.505 0.302 0.235 0.174 Table 3: Measured frontier (within-task DocVQA, N=600N=600; all allocations use the same total budget). Bold values indicate the grid optimum under each consequence ratio: uniform allocation for r≤2r≤ 2, an interior optimum for r=3r=3, and boundary solutions when the low-tier budget becomes limited. Figure 3(a) visualizes the tier-specific error–budget curves on the confirmatory set; the full tabulated values are reported in the Technical Supplement, Section 3 (Additional Allocation Frontier Details). The two tiers’ error rates remain nearly identical across the entire budget range; the central-difference marginals at the uniform split, computed from the tabulated values, are 0.001730.00173 for high-consequence samples and 0.001710.00171 for low-consequence samples, with a slope standard error of approximately 2×10−42× 10^-4. Therefore, high- and low-consequence questions exhibit statistically indistinguishable token sensitivity. This result isolates the source of the allocation gain. The improvement is not caused by high-consequence samples being inherently more sensitive to additional visual tokens. Under symmetric cost (r=1r=1), where both tiers have equal importance, the optimal allocation remains uniform. Only when consequence asymmetry is introduced does the optimal allocation shift toward high-consequence samples. Allocation Frontier. We next characterize how the optimal allocation changes as consequence asymmetry increases. Figure 3 visualizes the allocation frontier, and Table 3 reports the measured cost-weighted error for different high-to-low consequence ratios while keeping the total token budget fixed. When the consequence ratio is symmetric (r=1r=1), uniform allocation is optimal because the two tiers exhibit equivalent token-response behavior. As the consequence ratio increases, the optimal allocation progressively shifts tokens toward high-consequence samples. At moderate asymmetry (r=3r=3), the optimal solution lies inside the allocation space, where the weighted benefits of transferring additional tokens are approximately balanced. When the consequence ratio becomes larger (r≥5r≥ 5), the optimal allocation moves toward the boundary of the evaluated budget grid, indicating that additional computation should continue to be transferred toward high-consequence samples. The anti-direction allocation provides a control condition. Moving tokens away from high-consequence samples consistently increases cost-weighted error in the frontier (Figure 3(b)). At the extreme anti-direction point, the high-tier budget is reduced to Bhi=32B_hi=32, where the high-tier error reaches 0.9030.903 (Figure 3(a)). These results show that the allocation frontier is determined by consequence asymmetry rather than average accuracy optimization. The measured frontier is obtained directly from empirical error–budget curves and therefore represents a discrete allocation boundary rather than an analytical optimum. We further examine whether the selected allocations generalize beyond the calibration samples. Benchmark N High-tier err. McNemar Break-even unif. cost (b:cb:c, p) r InfoVQA 400 0.660 0.410 57:7, <10−9<10^-9 0.7 DocVQA 600 0.300 0.133 54:4, 3×10−123×10^-12 2.8 ChartQA 400 0.335 0.235 22:2, 4×10−54×10^-5 3.6 Table 4: Cross-dataset replication with the tier rule frozen verbatim. The absolute high-tier reduction is monotone in information density (0.250 / 0.167 / 0.100), and the break-even cost ratio falls as density rises. Split-half calibration fits the two error-budget curves on odd-indexed samples and evaluates the predicted allocation on the even-indexed samples. The predicted optimum achieves rank 1/13 at r∈1,10,20r∈\1,10,20\ and rank 3/13 at r∈2,5r∈\2,5\ (held-out CWE gaps +0.007+0.007 and +0.003+0.003), but only rank 6/13 at r=3r=3 (gap +0.023+0.023), where the optimum lies in the interior regime and is more sensitive to curve estimation errors. The deployed optimum selected at r=5r=5, (576,32)(576,32), improves high-tier error over uniform allocation with discordant pairs 65:565:5 (exact p≈2×10−14p≈ 2× 10^-14) and achieves a ΔCWE _CWE 95% confidence interval of [0.032,0.122][0.032,0.122]. 5.3 Generalization Across Dense Benchmarks Table 4 shows the same frozen question rule transferring across three benchmarks. The gradient is the applicability boundary made quantitative: the denser the task, the larger the absolute gain and the lower the cost asymmetry needed to justify transfer—to the point that on InfographicVQA reallocation wins even on plain accuracy (break-even 0.7<10.7<1). We state the CWE-level statistics plainly: at r=5r=5 the paired bootstrap CI on ΔCWE _CWE excludes zero for DocVQA ([0.018,0.102][0.018,0.102]) and InfographicVQA ([0.117,0.238][0.117,0.238]) but not for ChartQA ([−0.019,0.065][-0.019,0.065]), whose break-even (3.6) sits close to r=5r=5; ChartQA’s allocation direction is nonetheless decisively significant (22:2), and its CWE margin grows with r. The high-tier reduction is not an artifact of the lenient scorer: it persists on all three datasets under strict exact match and the official DocVQA ANLS@0.50.5 metric; see the Technical Supplement, Section 8 (Metric Robustness), e.g., DocVQA McNemar 64:564:5 and 44:344:3. Full cross-dataset results, including all allocation strategies and detailed statistical comparisons, are reported in the Technical Supplement, Section 4 (Full Cross-Dataset Results). 5.4 Invariance: Mechanism, Model, Selector Axis Variant High-tier err. McNemar unif. cost (b:cb:c, p) Mechanism resolution 0.250 0.123 47:9, <10−6<10^-6 Model LLaVA-OV 0.417 0.330 40:14, 5×10−45×10^-4 Selector random 0.497 0.247 80:5, 2×10−182×10^-18 Selector saliency 0.377 0.127 80:5, 2×10−182×10^-18 Selector FastV 0.560 0.297 84:5, <10−18<10^-18 Table 5: Invariance study (within-task DocVQA, same 600 samples). Rows vary one axis at a time from the main setting (deletion / Qwen2.5-VL / redundancy). Mechanism. Table 5 summarizes the invariance results across different budget realization mechanisms, model architectures, and token selectors. Realizing budgets by resolution instead of deletion improves every arm (uniform 0.250 vs. 0.300)—consistent with Qwen2.5-VL’s native-dynamic-resolution training— while preserving the allocation gain (47:9). The consequence-sensitive gain is a property of the allocation, not of any one token-reduction mechanism, and its strongest instantiation requires no model surgery at all. Model. LLaVA-OneVision shares no encoder, positional scheme, or tiling strategy with Qwen2.5-VL, and its AnyRes ladder quantizes budgets to coarse rungs (aligned to equal totals within 0.02%). The ordering replicates (anti: 14:90, p≈9×10−15p\!≈\!9× 10^-15), with a smaller absolute gain (its minimum rung already spends 1306 tokens), and a break-even ratio of 2.8—nearly identical to Qwen’s 2.8, an invariance we report as an observation, not a claim. As with ChartQA, the r=5r=5 CWE margin is directionally positive but its CI includes zero ([−0.009,0.073][-0.009,0.073]). Selector Robustness. To examine whether consequence-sensitive allocation depends on a specific token selector, we evaluate the same allocation policy with different visual token selection strategies, including random selection, saliency ranking, FastV (Chen et al. 2024), SparseVLM (Zhang et al. 2025), and VisionZip (Yang et al. 2025). Table 6 summarizes the complete selector-level results. The allocation gain consistently persists across all selection mechanisms. Random and saliency provide simple selection controls, while FastV, SparseVLM, and VisionZip represent stronger training-free pruning strategies based on attention-guided pruning, text-guided relevance estimation, and informative token selection, respectively. Although these selectors produce different absolute error levels under uniform allocation, consequence-sensitive allocation consistently reduces high-consequence errors under the same total token budget. These results show that consequence-sensitive allocation operates at the budget allocation level and is orthogonal to the token selection mechanism. Therefore, existing token-selection methods such as FastV, SparseVLM, and VisionZip can be viewed as composable components rather than competing alternatives. Selector Arm High Low CWEr5 McNemar redundancy unif. 0.300 0.373 0.312 — cost 0.133 0.843 0.252 54:454:4 random unif. 0.497 0.470 0.492 — cost 0.247 0.753 0.331 80:580:5 saliency unif. 0.377 0.457 0.390 — cost 0.127 0.840 0.246 80:580:5 FastV unif. 0.560 0.450 0.542 — cost 0.297 0.710 0.366 84:584:5 SparseVLM unif. 0.460 0.373 0.446 — cost 0.257 0.657 0.323 67:667:6 VisionZip unif. 0.397 0.353 0.389 — cost 0.253 0.607 0.312 45:245:2 Table 6: Full selector robustness (within-task DocVQA, N=600N=600). Consequence-sensitive allocation remains effective across diverse token selection strategies, including redundancy-based selection, random selection, saliency ranking, FastV, SparseVLM, and VisionZip. 5.5 Mixed-Workload Deployment Strategy High Med Low CWE (5/3/1) uniform (288/img) 0.475 0.183 0.183 0.345 content 0.500 0.175 0.167 0.355 tiered (512/256/96) 0.192 0.200 0.350 0.212 anti 0.825 0.200 0.125 0.539 Table 7: Three-tier mixed workload (DocVQA / TextVQA / VQAv2, N=360N=360, equal total budget). Tiers come from task identity—metadata that is free at deployment. McNemar (high tier): tiered 36:2 (p≈5×10−9p≈5×10^-9); content 2:5 (p=0.45p=0.45). On a realistic three-task mixture (Table 7), tier-based allocation cuts CWE by 38% (ΔCWE _CWE CI [0.080,0.188][0.080,0.188]). The content allocator fails informatively: real photographs, scene-text images, and documents have similar feature-level diversity, so its allocation collapses to near-uniform (273/291/299 tokens per tier)—the clean separation it enjoyed against synthetic solid colors (Table 1) does not survive contact with real workloads, while task identity remains free and effective. 5.6 Efficiency Using the resolution-based deployment mechanism, reducing the average visual tokens from full resolution to 304 tokens lowers latency from 482.1 ms to 356.1 ms (−26.1%-26.1\%). The deployed two-tier allocation (576,32)(576,32) achieves about 21%21\% lower latency than full-resolution inference while improving CWE from 0.312 to 0.235. True deletion is used only as an analysis mechanism and is not itself an accelerator because it extracts full-resolution embeddings before deletion. Full latency measurements are reported in the Technical Supplement, Section 10 (Full Latency Sweep). 6 Conclusion We formulate visual token compression as cost-weighted error minimization under a strict token budget, introduce a controlled evaluation that disentangles consequence information from content-derived signals, and characterize when consequence-sensitive allocation is beneficial. The resulting allocation frontier shows that uniform allocation is optimal when error costs are similar, whereas increasingly asymmetric costs justify shifting more visual computation toward high-consequence requests. On DocVQA, this transition occurs at a consequence ratio of approximately 2.8:12.8:1, with break-even ratios ranging from 0.70.7 to 3.63.6 across benchmarks of different information density. The framework is lightweight and deployment-oriented, inferring consequence tiers from question or task information, calibrating tier-specific error–budget curves offline, and realizing the resulting budgets through token deletion or resolution reallocation. More broadly, our results suggest that visual token compression should be evaluated not only by how much accuracy it preserves, but also by which errors it prevents. References L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang (2024) An image is worth 1/2 tokens after layer 2: plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision (ECCV), Cited by: §1, §2, §5.4. C. Elkan (2001) The foundations of cost-sensitive learning. In International Joint Conference on Artificial Intelligence (IJCAI), Cited by: §2. Z. Fang, P. Lyu, C. Zhang, G. Lu, J. Yu, and W. Pei (2026) Prune redundancy, preserve essence: vision token compression in VLMs via synergistic importance-diversity. arXiv preprint arXiv:2603.09480. Cited by: §2. Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh (2017) Making the v in vqa matter: elevating the role of image understanding in visual question answering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4. M. Huang, B. Jiang, D. Zheng, H. Hu, K. Han, and X. Chen (2025) PPE: positional preservation embedding for token compression in multimodal large language models. arXiv preprint arXiv:2510.22936. Cited by: §2. A. Jeddi, M. N. Le, A. Kazerouni, H. C. Karaimer, H. Nguyen, I. Mohomed, M. Brudno, A. Levinshtein, K. G. Derpanis, B. Taati, and R. Grzeszczuk (2026) AVIS: adaptive test-time scaling for vision-language models. arXiv preprint arXiv:2606.11576. Cited by: §2. B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y. Li, Z. Liu, and C. Li (2024) LLaVA-onevision: easy visual task transfer. arXiv preprint arXiv:2408.03326. Cited by: Appendix A, §4. B. Li and W. Zhang (2025) ID-align: rope-conscious position remapping for dynamic high-resolution adaptation in vision-language models. arXiv preprint arXiv:2505.21465. Cited by: §2. Z. Lin, Y. Liu, Y. Yang, L. Tao, and D. Ye (2025) AdaptVision: efficient vision-language models via adaptive visual acquisition. arXiv preprint arXiv:2512.03794. Cited by: §2. J. Ma, Y. Liu, Z. Qiu, J. Ji, and X. Sun (2026) Evading visual aphasia: contrastive adaptive semantic token pruning for vision-language models. arXiv preprint arXiv:2605.09429. Cited by: §2. A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque (2022) ChartQA: a benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics (ACL), Cited by: §4. M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. V. Jawahar (2022) InfographicVQA. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Cited by: §4. M. Mathew, D. Karatzas, and C. V. Jawahar (2021) DocVQA: a dataset for vqa on document images. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Cited by: §4. Qwen Team (2025) Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: Appendix A, §4. A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach (2019) Towards vqa models that can read. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4. F. Valade, M. Hebiri, and P. Gay (2024) EERO: early exit with reject option for efficient classification with limited budget. arXiv preprint arXiv:2402.03779. Cited by: §2. J. Wen, L. He, and Z. He (2026) Not all errors are equal: consequence-aware reasoning compute allocation. arXiv preprint arXiv:2606.04402. Cited by: §2. L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang, Y. Zang, Y. Cao, C. He, J. Wang, F. Wu, and D. Lin (2025) PyramidDrop: accelerating your large vision-language models via pyramid visual redundancy reduction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2. S. Yang, Y. Chen, Z. Tian, C. Wang, J. Li, B. Yu, and J. Jia (2025) VisionZip: longer is better but not necessary in vision language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2, §5.4. Y. Zhang, C. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. Gudovskiy, T. Okuno, Y. Nakata, K. Keutzer, and S. Zhang (2025) SparseVLM: visual token sparsification for efficient vision-language model inference. In International Conference on Machine Learning (ICML), Cited by: §1, §2, §5.4. Technical Supplement This Technical Supplement provides additional experimental details, analyses, and robustness studies supporting the main paper "Not All Visual Tokens Are Equally Safe to Remove: Consequence-Sensitive Visual Token Compression." The Full Experimental Setup and Implementation Details section describes the complete experimental setup, including model configurations, dataset construction, consequence assignment rules, budget constraints, scoring, and statistical protocols. The Proportional Allocation and Token Dilution section analyzes why earlier ratio-based allocation was insufficient and motivates the absolute token-transfer design used in the main paper. The Additional Allocation Frontier Details section reports the full token-response curves underlying the allocation frontier and supports the claim that the within-task high- and low-consequence tiers have nearly identical token sensitivity. The Full Cross-Dataset Results section provides comprehensive per-arm evaluations on dense vision-language benchmarks, validating that the proposed allocation principle generalizes beyond the primary DocVQA setting. The Additional Model and Scale Generalization section further evaluates the method across different model architectures and scales, including LLaVA-OneVision and Qwen2.5-VL-3B. The Task-Level Sensitivity Diagnostics section presents task-level sensitivity analyses that motivate the choice of dense visual understanding tasks and clarify when consequence information provides additional value beyond content-based allocation. The Validation of Consequence Signals section validates the consequence signals used for allocation through both objective answer-based analysis and independent LLM-based judgment. The Metric Robustness section verifies that the observed improvements remain consistent across multiple evaluation metrics, including strict exact match and the official ANLS metric. The Automatic Consequence Signal Prediction section demonstrates that consequence signals can be automatically predicted from question text using a lightweight classifier, supporting practical deployment without manually designed rules. The Full Latency Sweep section reports detailed latency measurements for the resolution-based deployment mechanism. Finally, the Limitations section discusses the main assumptions and remaining scope of the study. Together, these analyses provide additional evidence that consequence-sensitive visual token compression improves cost-weighted inference efficiency by allocating limited visual computation toward samples where errors are more costly. Appendix A Full Experimental Setup and Implementation Details Models and environment. The primary model is Qwen2.5-VL-7B (Qwen Team 2025) in bf16 on a single A100-80GB; the second model is LLaVA-OneVision-Qwen2-7B-ov-hf (Li et al. 2024). Decoding is greedy (do_sample=False, argmax), ≤16≤ 16 new tokens, with every question suffixed by “ Answer briefly.” Datasets and disjoint sampling. All samples are drawn by streaming. The within-task pilot (N=120N=120) uses the first 257 scanned DocVQA validation items; the pre-registered confirmatory set (N=600N=600) skips the first 257 items and scans 1184 items, so the pilot and confirmatory sets are strictly disjoint. ChartQA test scans 568 samples and InfographicVQA validation scans 601 samples from parquet files, supplying 200 high-consequence and 200 low-consequence samples each. The mixed workload uses 120 samples each from VQAv2, TextVQA, and DocVQA, with the DocVQA subset skipping the first 1184 scanned items. Images are resized to a maximum side of 1008 and filtered to have at least 640 native tokens so that every grid budget is feasible without clamping; InfographicVQA additionally up-scales extreme aspect ratios up to a 1512 maximum side. Frozen cost-label rule. A tier is assigned from the question text only, never from the image or answer, using a keyword rule frozen verbatim across all datasets. A question is assigned to the high-consequence tier (c=5c=5) if it matches any of the following keywords: how much, how many, amount, total, number, no., date, when, year, price, cost, percent, percentage, %, phone, fax, time, quantity, rate, value, figure, sum, id, code. Otherwise, it is assigned to the low-consequence tier (c=1c=1). In the mixed workload, the tier is assigned by task identity: VQAv2 is low consequence, TextVQA is medium consequence (c=3c=3), and DocVQA is high consequence. Scoring. The function is_correct normalizes both prediction and reference strings by stripping whitespace, lowercasing, and dropping a trailing period. It accepts exact match or bidirectional substring containment. The identical scorer is shared by all arms. Equal-budget protocol. Each run asserts, per image, that the assigned budget does not exceed the native token count, preventing silent clamping. Each run also asserts, per arm, that realized totals are exactly equal; uniform allocation assigns ⌊B/N⌋ B/N tokens to every image, with the remainder given to the first samples. Realized-total deviations for discrete mechanisms are reported rather than hidden: they are at most 0.4%0.4\% for the resolution mechanism and at most 0.02%0.02\% for the LLaVA AnyRes ladder. Cost-weighted error is computed as CWE=∑ici[wrongi]∑ici.CWE= _ic_i\,1[wrong_i] _ic_i. The cost ratio r∈1,2,3,5,10,20r∈\1,2,3,5,10,20\ is applied by zero-GPU post-hoc reweighting of the same per-sample correctness vectors. Statistics. The pre-registered primary test is high-tier error, comparing uniform allocation against cost-aware allocation by two-sided exact McNemar test on discordant pairs b:cb:c. The cost-weighted error difference ΔCWE _CWE is assessed using a paired bootstrap with 10k resamples and BCa intervals. Split-half calibration fits both tier-specific error–budget curves on odd-indexed samples and evaluates the predicted argmin on the even-indexed samples. Greedy decoding is deterministic, so the only randomness is bootstrap resampling and selector tie-breaking. Appendix B Proportional Allocation and Token Dilution Our earlier design allocated budgets by a ratio: the mean keep-ratio was fixed at 0.50.5, the high-side ratio was chosen from 0.55,0.65\0.55,0.65\, and the low-side ratio was solved to preserve the equal-budget constraint. This design produced only weak, non-significant gains. On 120 DocVQA-vs-color samples, cost-aware allocation achieved CWE 0.1250.125 versus uniform CWE 0.1530.153 (1.22×1.22×). The larger 1.75×1.75× gain measured at N=48N=48 was small-sample optimism, corresponding to only a two-sample high-tier difference. The main cause is token dilution. Dense document images carry a large native token count, so a proportional shift transfers only a small number of absolute tokens and the intended consequence-aware transfer is swamped. An early synthetic-image bug further compounded this issue: 336×336336×336 images contain only 144 native tokens, so uniform and anti ratio budgets silently exceeded the native token count and broke the equal-budget protocol. These observations motivate the absolute token-transfer design used throughout the main paper. Appendix C Additional Allocation Frontier Details Table 8 reports the full token-response curves used to construct the allocation frontier in the main paper. The high- and low-consequence tiers respond almost identically to additional visual tokens across the evaluated budget grid. This supports the within-task attribution claim that the observed allocation gain is driven by consequence asymmetry rather than by a difference in token sensitivity. Budget 32 64 96 128 160 High err. 0.903 0.860 0.837 0.773 0.670 Low err. 0.910 0.873 0.843 0.790 0.687 Budget 224 304 448 512 576 High err. 0.507 0.300 0.160 0.133 0.100 Low err. 0.520 0.373 0.163 0.130 0.093 Table 8: Token-response curves (within-task DocVQA, N=600N=600). The two tiers respond to additional visual tokens almost identically. Appendix D Full Cross-Dataset Results Table 9 reports all four arms on the three dense benchmarks. Content allocation is indistinguishable from uniform on all three datasets: DocVQA 4:44:4, p=1.0p=1.0; ChartQA 5:15:1, p=0.22p=0.22; and InfographicVQA 3:13:1, p=0.63p=0.63. Anti allocation is catastrophic everywhere. The break-even cost ratios are 2.822.82 for DocVQA, 3.603.60 for ChartQA, and 0.740.74 for InfographicVQA. Paired-bootstrap ΔCWE _CWE 95% confidence intervals at r=5r=5 are [0.018,0.102][0.018,0.102] for DocVQA and [0.117,0.238][0.117,0.238] for InfographicVQA, both excluding zero; ChartQA has interval [−0.019,0.065][-0.019,0.065], which does not exclude zero because its break-even ratio 3.603.60 sits close to r=5r=5. Data Strategy High Low CWEr5 DocVQA uniform 0.300 0.373 0.312 cost 0.133 0.843 0.252 anti 0.837 0.130 0.719 content 0.300 0.383 0.314 ChartQA uniform 0.335 0.365 0.340 cost 0.235 0.725 0.317 anti 0.650 0.275 0.588 content 0.315 0.370 0.324 InfoVQA uniform 0.660 0.670 0.662 cost 0.410 0.855 0.484 anti 0.865 0.455 0.797 content 0.650 0.675 0.654 Table 9: Full cross-dataset results (N=600/400/400N=600/400/400; 300/300, 200/200, 200/200 high/low). McNemar tests for uniform vs. cost-aware allocation on the high tier: DocVQA 54:454:4 (3×10−123×10^-12), ChartQA 22:222:2 (4×10−54×10^-5), and InfographicVQA 57:757:7 (<10−9<10^-9). Appendix E Additional Model and Scale Generalization LLaVA-OneVision shares no encoder, positional scheme, or tiling strategy with Qwen2.5-VL. Its AnyRes ladder quantizes budgets to coarse rungs 1269,1793,∼2900,4854,5589\1269,1793, 2900,4854,5589\, realized as low→ 1306 tokens and high→ 4732 tokens, with uniform and anti greedily aligned to equal totals within 0.02%0.02\%. Table 10 reports the three arms. The ordering replicates: cost-aware allocation gives 40:1440:14, p≈5×10−4p≈5×10^-4, and anti allocation gives 14:9014:90, p≈9×10−15p≈9×10^-15. The absolute gain is smaller because the minimum LLaVA rung already spends 1306 tokens. The break-even ratio is 2.792.79, near-identical to Qwen’s 2.822.82; we report this as an observation, not a claim. The r=5r=5 ΔCWE _CWE confidence interval [−0.009,0.073][-0.009,0.073] includes zero, as with ChartQA. Strategy High Low Avg CWEr5 uniform 0.417 0.440 0.428 0.421 cost 0.330 0.683 0.507 0.389 anti 0.670 0.383 0.527 0.622 Table 10: LLaVA-OneVision-7B (within-task DocVQA, N=600N=600, resolution mechanism). Smaller scale (Qwen2.5-VL-3B). The effect also holds at a smaller model size. On Qwen2.5-VL-3B with the resolution mechanism on the same within-task 600 samples, the ordering replicates, as shown in Table 11. Cost-aware allocation cuts high-tier error from 0.2770.277 to 0.1470.147 with McNemar 49:1049:10, p≈3×10−7p≈3×10^-7; anti allocation yields 0:1650:165. The CWE decreases from 0.2890.289 to 0.2560.256 at r=5r=5. The break-even ratio lies between r=3r=3 and r=5r=5, and the r=5r=5 ΔCWE _CWE confidence interval [−0.008,0.078][-0.008,0.078] includes zero, as with ChartQA and LLaVA-OneVision. The gain therefore survives across model scale, from 3B to 7B, as well as across architecture. Strategy High Low Avg CWEr5 uniform 0.277 0.353 0.315 0.289 cost 0.147 0.800 0.473 0.256 anti 0.827 0.187 0.507 0.720 Table 11: Qwen2.5-VL-3B (within-task DocVQA, N=600N=600, resolution mechanism; realized budgets within 0.4%0.4\%). Appendix F Task-Level Sensitivity Diagnostics Before the within-task design, we ran cross-task probes whose task confounds make them unsuitable as primary evidence but useful for motivating the density boundary. On synthetic solid-color images, which have low token sensitivity, budgeting is inert: true-deletion cost-aware allocation equals uniform allocation, with both achieving CWE 0.0690.069, because the task saturates by 25–50% of tokens. On TextVQA, which shows weak sensitivity, cost-aware allocation did not help: CWE is 0.1740.174 for cost-aware allocation versus 0.1390.139 for uniform allocation, additionally confounded by unequal image sizes. Only on dense DocVQA does the premise that high-cost questions need more tokens hold clearly: full-budget high-tier error is 0.0830.083, rising to 0.5420.542 at 25% tokens. These diagnostics explain why the main evaluation uses within-task DocVQA and why the generalization set focuses on dense document-like tasks. The cross-task diagnostic in the main paper makes the complementary point: when a content signal can observe a sensitivity gap, it can win without any cost label. Therefore, only the within-task setting attributes the observed gain to consequence information rather than to visual token sensitivity differences. Appendix G Validation of Consequence Signals The tier rule described in the Full Experimental Setup and Implementation Details section is a keyword proxy; we assess how well it tracks error consequence with two checks on the confirmatory 600 samples. Objective check (gold-answer content). A question’s gold answer is labeled quantitative if any reference answer contains a digit or a currency/percent marker. Cross-tabulated against the tier, as shown in Table 12, 90.3%90.3\% of high-tier questions have a quantitative answer, indicating that the high tier is genuinely transactional. The low tier is noisier: 68%68\% are non-quantitative, while 96 transactional questions leak into the low tier. Agreement is 0.7920.792 with κ=0.583κ=0.583. LLM-judge check (consequence rubric). On a seed-fixed stratified sample of 200 questions, with 100 from each tier, an independent frontier LLM is shown only the question and is blind to the keyword rule. It rates each question under the following rubric: high consequence if a wrong answer is a specific monetary amount, date, count, rate, or formal record identifier, such as invoice, voucher, fund, serial, phone, fax, zip, or code, that propagates into a decision or record; low consequence if the answer is a name, title, type, heading, description, navigational locator such as a page/table/chapter number, or trivial schedule time. Agreement is 0.7250.725 with κ=0.450κ=0.450. Disagreement taxonomy. Both checks fail in the same two directions. First, in over-labeled high cases, keywords such as number match page, table, or chapter numbers and expressions such as “the number at the bottom right,” while time matches coffee-break or lunch schedule times. These are navigational or trivial values and account for approximately 60% of the rule-high/judge-low cell. Second, in under-labeled low cases, questions whose answers are dollar amounts or formal contact fields fall into the low tier because their stems lack the listed keywords, for example “the budget for…”, “Tel no”, or “zipcode”. Because these under-labeled cases place genuinely high-value questions in the low tier, the equal-budget experiment under-serves them. The measured cost-weighted gain is therefore a lower bound on what a cleaner consequence labeler could yield. Check Agree κ Confusion H/H , L/L H/L , L/H Gold-answer (N=600N=600) 0.792 0.583 271 , 204 29 , 96 LLM judge (N=200N=200) 0.725 0.450 70 , 75 30 , 25 Table 12: Label validation. H/H = rule-high and check-high, L/L = rule-low and check-low, H/L = rule-high but check-low, and L/H = rule-low but check-high. The high tier is clean; residual noise concentrates in the low tier and is conservative for the main claim. Appendix H Metric Robustness Our grader is a lenient normalized-containment match. To verify that the allocation gain is not an artifact of this scorer, we re-score the saved per-sample predictions, without re-inference, under two stricter metrics: strict exact match, which uses normalized equality with no substring containment, and the official DocVQA ANLS@0.50.5 metric, which thresholds average normalized Levenshtein similarity at 0.50.5. Table 13 shows that the high-tier reduction and its significance survive on all three datasets under all three metrics. The reproduced relaxed column matches the main-text numbers exactly, confirming the re-scoring pipeline. Data Metric High (u→ ) CWE (u→ ) McN. DocVQA relaxed 0.300→ 0.133 0.312→ 0.252 54:4 exact 0.360→ 0.163 0.382→ 0.288 64:5 ANLS 0.247→ 0.110 0.259→ 0.227 44:3 ChartQA relaxed 0.335→ 0.235 0.340→ 0.317 22:2 exact 0.470→ 0.370 0.468→ 0.443 21:1 ANLS 0.405→ 0.325 0.398→ 0.397 17:1 InfoVQA relaxed 0.660→ 0.410 0.662→ 0.484 57:7 exact 0.675→ 0.445 0.677→ 0.517 53:7 ANLS 0.570→ 0.355 0.562→ 0.421 52:9 Table 13: Metric robustness (uniform→ -aware, high tier and CWE at r=5r=5). All McNemar tests have p<10−4p<10^-4. The direction and significance hold under strict exact match and official ANLS@0.50.5. Appendix I Automatic Consequence Signal Prediction The consequence tier is cheaply predictable from question text, so deployment need not rely on the hand-written keyword rule. We fit a lightweight TF-IDF uni/bi-gram logistic-regression classifier and evaluate it by 5-fold cross-validation, as shown in Table 14. When predicting the frozen rule’s own tier, it reaches 94.3%94.3\% accuracy with κ=0.887κ=0.887. This means the rule is recovered almost exactly from text, so an online predictor can stand in for the keyword list. When predicting the independent LLM-judge consequence described in the Validation of Consequence Signals section, it reaches 83.5%83.5\% accuracy with κ=0.672κ=0.672, which is higher than the fixed rule’s own 72.5%72.5\% accuracy and κ=0.45κ=0.45 agreement with the judge. This improvement occurs because a learned predictor generalizes to unlisted transactional cues such as budget, tel, and zip. The frontier-LLM judge described in the Validation of Consequence Signals section is itself a second, training-free predictor. Prediction target N Acc F1 κ Frozen rule tier 600 0.943 0.942 0.887 LLM-judge consequence 200 0.835 0.839 0.672 Table 14: Automatic consequence prediction from question text (TF-IDF + logistic regression, 5-fold CV). A lightweight learned predictor recovers the deployed tier and, on the judge’s labels, exceeds the fixed rule’s agreement with the judge. Appendix J Full Latency Sweep Table 15 reports the full latency sweep for the resolution-based deployment mechanism. Latency decreases as the number of visual tokens is reduced, but the gain plateaus near 160 tokens because decoding and fixed overhead begin to dominate. Reducing visual tokens from full resolution to 304 tokens lowers mean latency from 482.1 ms to 356.1 ms, a 26.1% reduction. The deployed two-tier point (576,32)(576,32) achieves approximately 21% lower latency than full-resolution inference. Setting Mean (ms) Tokens Latency ↓ full generate 482.1 959 — resolution 576 415.8 572 13.8% resolution 448 391.1 450 18.9% resolution 304 356.1 304 26.1% resolution 160 353.3 162 26.7% resolution 96 362.6 96 24.8% Table 15: Full latency sweep (resolution mechanism). Latency decreases with token reduction but plateaus near 160 tokens due to decoding and fixed computation overhead. The deployed two-tier point (576,32)(576,32) achieves approximately 21%21\% lower latency than full-resolution inference. Appendix K Limitations Our study has several limitations. The consequence weights are predefined design parameters rather than measured downstream costs, and consequence categories are assumed to be available at inference time. Although we evaluate cost-ratio sweeps and show that lightweight predictors can approximate consequence labels, real deployment would require workflow-specific cost estimation and uncertainty-aware consequence prediction. The allocation frontier is estimated from discrete empirical error–budget curves, so the optimum depends on the evaluated grid and may not coincide with a continuous optimum. Finally, our evaluation focuses on dense vision-language workloads where consequence effects can be isolated from visual sensitivity; broader workloads with simultaneously varying difficulty, token sensitivity, and error cost remain open.

Instant research discovery