Paper deep dive
Can a Lightweight Multimodal Model Estimate LLM Reasoning Performance? A Study for Compute-Optimal Document Inference
Zishan Ahmad, Vishal Vaddina
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/20/2026, 4:49:39 AM
Summary
This paper introduces BudgetDoc, a multimodal benchmark for evaluating LLM performance across variable reasoning budgets, and DRB (Document-Reasoning Balancer), a lightweight ~1B-parameter multimodal estimator. DRB predicts ordinal performance classes for document tasks, enabling dynamic budget allocation that matches or improves F1 scores while drastically reducing costs compared to maximum-budget baselines. The study highlights the 'over-thinking penalty' where excessive reasoning tokens degrade performance or increase cost unnecessarily.
Entities (11)
Relation Signals (9)
DRB → evaluatedon → BudgetDoc
confidence 98% · achieving a 0.753 weighted F1 on the BudgetDoc test set
DRB → usescomponent → SigLIP-2
confidence 95% · DRB (Document-Reasoning Balancer), an approx. 1B-parameter pre-flight estimator (SigLIP-2 + Qwen3-0.6B)
DRB → usescomponent → Qwen3-0.6B
confidence 95% · DRB (Document-Reasoning Balancer), an approx. 1B-parameter pre-flight estimator (SigLIP-2 + Qwen3-0.6B)
BudgetDoc → containstasks → CheckboxQA
confidence 90% · BudgetDoc spans three document-centric tasks... RVL-CDIP... TAT-DQA... CheckboxQA
BudgetDoc → containstasks → RVL-CDIP
confidence 90% · BudgetDoc spans three document-centric tasks... RVL-CDIP... TAT-DQA... CheckboxQA
BudgetDoc → containstasks → TAT-DQA
confidence 90% · BudgetDoc spans three document-centric tasks... RVL-CDIP... TAT-DQA... CheckboxQA
DRB → optimizes → reasoning_budget
confidence 90% · using DRB predictions to select the minimum budget expected to achieve near-peak performance
Gemini 2.5 Flash → suffersfrom → Over-thinking penalty
confidence 85% · For Gemini 2.5 Flash on RVL-CDIP, F1 decreases monotonically from 0.86 (budget=0) to 0.81 (budget=2048), a 6% penalty from over-allocation.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Uniformly allocating inference reasoning budgets to LLMs is expensive and prone to over-thinking penalties; especially in document tasks where visual layouts drive complexity. To address this, we introduce BudgetDoc, the first multimodal benchmark providing explicit supervision for model-budget-performance trade-offs across three document tasks. Using BudgetDoc, we train DRB (Document-Reasoning Balancer), an approx. 1B-parameter pre-flight estimator (SigLIP-2 + Qwen3-0.6B) that predicts ordinal model performance across budget levels, achieving a 0.753 weighted F1. When dynamically allocating reasoning budgets across five frontier models and three datasets, DRB matches or improves F1 scores compared to always-maximum-budget baselines in 9 of 15 configurations while drastically reducing cost. Finally, preliminary evaluations demonstrate DRB's potential to generalize to cross-model selection.
Tags
Links
- Source: https://arxiv.org/abs/2608.18591v1
- Canonical: https://arxiv.org/abs/2608.18591v1
Trouble viewing inline? Open PDF directly →
Full Text
45,163 characters extracted from source content.
Expand or collapse full text
Published as a conference paper at COLM 2026 Can a Lightweight Multimodal Model Estimate LLM Reason- ing Performance? A Study for Compute-Optimal Document Inference Zishan AhmadVishal Vaddina Phi Labs, Quantiphi zishan.ahmad, vishal.vaddina@quantiphi.com Abstract Modern frontier LLMs expose controllable reasoning budgets at inference time. However, allocating these budgets uniformly across all inputs is both expensive and often counterproductive, a phenomenon commonly known as the over-thinking penalty. Frontier providers offer in-flight controls to mit- igate this, such as Gemini’s dynamic auto-thinking, but these mechanisms rely on the model’s own self-assessment of difficulty. Recent work shows this is insufficient: 32% of nominally cheaper model pairs cost more in prac- tice due to thinking-token overrun, with up to 9.7×per-query cost variance for identical prompts (Chen et al., 2026). For document tasks, where diffi- culty is driven by visual layout, table structure, and multi-modal content, a pre-flight estimator that reads the document directly is better positioned to make this judgment than a model reasoning in-flight. Resolving this requires an external pre-flight estimator that can predict, per sample, how well a given model will perform at each budget level before committing to expensive inference. In this work, we explore if a lightweight multimodal model can accurately estimate the reasoning performance of a much larger LLM across variable reasoning budgets. To study this, we introduce Bud- getDoc, the first multimodal benchmark providing explicit supervision for model-budget-performance trade-offs across three document-centric tasks. We train DRB (Document-Reasoning Balancer), a∼1B-parameter estimator (SigLIP-2 + Qwen3-0.6B) that predicts a 7-class ordinal performance label for any (document, prompt, model, budget) configuration, achieving a weighted F1 of 0.753 on the BudgetDoc test set. We then apply DRB’s esti- mates to optimize reasoning budgets across five frontier models and three datasets: in 9 of 15 model-dataset configurations, DRB matches or improves F1 over always-maximum-budget inference while mostly reducing cost. As a secondary case study, we probe whether the same estimator can general- ize to model selection across intra- and inter-family configurations, finding encouraging preliminary results and identifying dedicated multi-provider training as an important direction for future work. 1 Introduction A defining trend in large language model (LLM) development is the shift toward scaling compute at test time (Snell et al., 2024; Wu et al., 2024). Modern frontier models, including OpenAI’s GPT-5 family and Google’s Gemini series, expose user-controllable reasoning budgets, enabling chains of thought (Wei et al., 2022) of variable depth and cost. The implicit assumption is that more reasoning tokens yield better answers. In practice, this is only conditionally true. Consider a document classification query that requires visual pattern matching rather than multi-step deduction: forcing the model through thousands of reasoning tokens does not help and may hurt, by encouraging spurious over-confident reasoning chains. We call this the over-thinking penalty: the phenomenon where additional test-time compute leads to performance degradation or merely wastes money relative to what a lower-cost 1 arXiv:2608.18591v1 [cs.AI] 19 Aug 2026 Published as a conference paper at COLM 2026 configuration would have produced. Our empirical analysis across three benchmarks and five frontier models confirms this penalty is systematic and significant (Figure 1). Providers have introduced in-flight controls to mitigate over-spending: Gemini 2.5 Flash offers a dynamic thinking mode that automatically scales its internal reasoning chain to perceived task complexity (enabled by default), while the GPT-5 family exposes a reasoningeffortparameter with discrete tiers (low,medium,high,xhigh). However, these controls are fundamentally in-flight and self-assessed: the model calibrates its own reasoning based on its own perception of difficulty, without an external estimate of its likely perfor- mance. Recent work shows this is insufficient to eliminate cost pathologies. Chen et al. (2026) evaluate eight frontier reasoning models and find that 32% of nominally cheaper model-pair comparisons exhibit a pricing reversal: the cheaper-listed model incurs higher actual cost due to uncontrolled thinking-token consumption, with per-query cost variance reaching 9.7×for identical prompts even within a single model. Bai et al. (2026) document up to 30×within-problem token variance in agentic coding tasks, finding that models’ self- predictions of their own usage achieve only weak correlation with realized consumption (Kendallτ ≤ 0.39). These findings establish that in-flight self-assessment is insufficient for reliable budget control, motivating an external, pre-flight approach. Addressing the over-thinking penalty via external estimation requires knowing, before inference, how well a given model will perform at a given reasoning budget for a specific input. This is a non-trivial prediction problem: the performance surface over budgets varies with document complexity, query type, and model inductive biases. This raises a natural scientific question: can a lightweight multimodal model learn to estimate the reasoning performance of a much larger LLM on a per-sample basis, enabling hard budget caps to be imposed at API-call time? If the answer is yes, a cheap pre-flight estimation pass could select the minimum sufficient reasoning budget for each input, achieving near-oracle performance at a fraction of the cost. The key challenge is that the estimator must be small enough that its own cost is negligible relative to the savings it enables. To study this question rigorously, we make three contributions: 1.BudgetDoc Dataset. We introduce the first multimodal benchmark designed specifically for supervised LLM performance estimation. BudgetDoc spans three document-centric tasks, two frontier model families, and variable reasoning bud- gets. By exhaustively evaluating a core set of 360 unique base samples (120 per task) across all 25 model-budget configurations, we generate a complete, dense combinatorial matrix of 9,000 labeled pairs that capture exactly how model accuracy varies with compute. 2.Document-Reasoning Balancer (DRB). We train a∼1B-parameter multimodal estimator composed of a SigLIP-2 vision encoder (Tschannen et al., 2025), a Qwen3- 0.6B language encoder (Yang et al., 2025), a cross-page fusion transformer, and a lightweight MLP head. DRB predicts a 7-class ordinal performance label for any (document, prompt, model, budget) tuple, achieving a weighted F1 of 0.753 on the held-out BudgetDoc test set, demonstrating that small models can learn meaningful performance estimates for much larger reasoning models. 3.Downstream Applications. We demonstrate two applications of DRB’s estimates: (i) per-sample reasoning budget optimization (primary): using DRB predictions to select the minimum budget expected to achieve near-peak performance, delivering upto 99% cost reductions across five frontier models; and (i) model selection (ex- ploratory case study): probing whether the same estimator generalizes to selecting among models within and across provider families, finding encouraging results that motivate future dedicated work. Our central finding is that a∼1B-parameter multimodal model can meaningfully estimate the reasoning performance of frontier LLMs on document tasks, with estimation quality sufficient to drive substantial cost savings through budget optimization. The over-thinking penalty is universal across modern model families: maximum reasoning budgets are rou- 2 Published as a conference paper at COLM 2026 0 512 102415362048 Reasoning Budget (tokens) 0.78 0.80 0.82 0.85 F1 Score Gemini Family RVL-CDIP 0 512 102415362048 Reasoning Budget (tokens) 0.30 0.40 0.50 0.60 0.70 CheckboxQA (OOD) 0 512 102415362048 Reasoning Budget (tokens) 0.80 0.82 0.84 0.86 TAT-DQA Gemini 2.5 Flash Gemini 3 Flash Gemini 3.1 Flash-Lite NoneLowMed.High Reasoning Effort 0.80 0.82 0.84 0.86 F1 Score GPT Family NoneLowMed.High Reasoning Effort 0.60 0.65 NoneLowMed.High Reasoning Effort 0.76 0.78 0.80 GPT-5.1 GPT-5.2 Over-Thinking Penalty: F1 vs. Reasoning Budget Across Models and Datasets Figure 1: The over-thinking penalty across model families and datasets. Each line shows mean F1 as a function of reasoning budget (Gemini, top row) or reasoning effort tier (GPT, bottom row). For Gemini 2.5 Flash on RVL-CDIP, F1 decreases monotonically from 0.86 (budget=0) to 0.81 (budget=2048), a 6% penalty from over-allocation. GPT-5.2 exhibits the same pattern on CheckboxQA (0.68 atlow→0.58 athigh). The optimal budget is task- and model-specific, motivating per-sample estimation rather than a fixed ceiling. tinely suboptimal in both accuracy and cost, and sample-level performance estimation is a practical lever to address this. 2 Related Work Test-time compute scaling.Wei et al. (2022) demonstrated that eliciting reasoning chains at inference time substantially improves LLM accuracy on complex tasks. Subsequent work on scaling laws for test-time compute (Snell et al., 2024; Wu et al., 2024) showed that increased reasoning tokens correlate with better performance, but only for tasks of sufficient complexity. Our work identifies the complementary regime: for many inputs, over-allocated reasoning is harmful or wasteful. We frame this as an estimation problem: predicting, per sample, how performance varies with budget, rather than as a fixed-scaling problem. In-flight reasoning controls and pricing reversals. Frontier providers have introduced mechanisms to reduce unnecessary reasoning spend. Google’s Gemini 2.5 Flash uses dynamic auto-thinking (defaultthinkingbudget=-1) to scale reasoning tokens to perceived task complexity. OpenAI’s GPT-5 family exposes areasoningeffortparameter with levels fromminimaltoxhigh. Despite these controls, Chen et al. (2026) show they do not eliminate cost pathologies: across 8 reasoning models and 12 tasks, 32% of nominally cheaper model pairs cost more in practice due to thinking-token overrun, a “pricing reversal” driven by the fact that thinking tokens account for 98%+ of cost differences on single-turn tasks and exhibit up to 9.7×per-query variance for identical prompts. Bai et al. (2026) document the same phenomenon in agentic coding tasks (up to 30×within-problem variance), showing that self-prediction by the model itself achieves only weak correlation (τ ≤0.39) with actual consumption. In our work, rather than improving in-flight self-assessment, we train an external pre-flight estimator that imposes a hard budget cap before inference begins. 3 Published as a conference paper at COLM 2026 LLM performance prediction and evaluation. Predicting the output quality of large models is an emerging concern in efficient inference. Prior work has largely focused on post- hoc evaluation, self-consistency scoring (Wei et al., 2022), or learned reward models trained for alignment. Our work is distinct: DRB is a pre-hoc estimator that predicts performance before inference runs, conditioned on a configuration (model, reasoning budget) the system has not yet executed. To our knowledge, BudgetDoc is the first benchmark designed specifically for this LLM performance estimation. LLM routing and model cascades.A related line of work addresses routing queries across multiple models to balance cost and quality (Ong et al., 2406). These approaches typically route to either a strong expensive model or a weak cheap one based on query difficulty. Our primary contribution is distinct in two ways: (i) we study performance estimation as the core task, and (i) our main application is budget selection within a fixed model rather than model selection. We additionally probe our estimator as a model selector in an exploratory case study, but a dedicated routing architecture is beyond the scope of this work. Document understanding with VLMs.Document intelligence has benefited from vision- language models capable of jointly processing rendered page images and text (Harley et al., 2015; Zhu et al., 2021; Turski et al., 2025). Our work builds on these benchmarks to study inference efficiency in the document understanding setting, where inputs are multi-page, visually rich, and paired with structured or open-ended queries. These properties make document tasks a particularly interesting testbed for performance estimation. Backbone architectures. DRB draws on the success of large vision encoders (SigLIP- 2 (Tschannen et al., 2025)) and efficient language models (Qwen3 (Yang et al., 2025)). The Transformer architecture (Vaswani et al., 2017) underpins the cross-page fusion module. 3 Methodology 3.1 BudgetDoc: A Benchmark for LLM Performance Estimation BudgetDoc is a supervised estimation dataset designed to answer the question: given a (document, prompt, model, reasoning budget) configuration, what performance class will the model achieve? We build it by exhaustively evaluating five frontier models from two provider families across three source benchmarks and multiple discrete reasoning budget levels, yielding a labeled dataset of configuration-outcome pairs. Source benchmarks. (i) RVL-CDIP (Multi-class classification): We use the Ryerson Vision Lab Complex Document Information Processing dataset (Harley et al., 2015). To increase task difficulty, we modify the single-class task by concatenating three PDFs from different categories into one multi-page document, requiring page-level multi-class prediction. (i) TAT-QA (Arithmetic reasoning): The Tabular And Textual dataset (Zhu et al., 2021) evaluates hybrid numerical reasoning over semi-structured tables and associated text in financial reports. (i) CheckboxQA (Form-checkbox extraction): We use CheckboxQA (Turski et al., 2025) to test performance on form understanding and checkbox-state extraction. Inputs are scanned form images; outputs are structured key-value pairs derived from checkbox regions, making this a visually structured extraction task distinct from classification and arithmetic reasoning. Data collection and labeling. To construct the benchmark pairs, we select a core set of exactly 120 unique base samples (documents and accompanying prompts) from each of the three source tasks, totaling 360 unique base contexts. We subject these samples to an exhaustive combinatorial evaluation across all five target models (three Gemini models: gemini-2.5-flash,gemini-3-flash-preview,gemini-3.1-flash-lite-preview; two GPT models:gpt-5.1,gpt-5.2) and five token reasoning budgetsb∈0, 512, 1024, 1536, 2048. For the GPT family, budget levels map directly to effort tiers (0→ none, 512→ low, 1024/1536→ medium, 2048→ high/xhigh). This exhaustive profiling sweeps the com- plete model-budget performance surface for every underlying document, yielding a dense, 4 Published as a conference paper at COLM 2026 closed combinatorial grid of exactly 9,000 total labeled pairs (360base contexts×5models× 5budgets). Prompts are paraphrased usinggemini-2.5-flash-liteto prevent syntactic overfitting. Per-sample F1 scores for each configuration are discretized into seven ordinal performance classesY ∈0, . . . , 6 based on operational accuracy thresholds: y = 0if f 1 < 0.5 k∈1, . . . , 4if f 1 ∈ [0.4 + 0.1k, 0.5 + 0.1k) 5if f 1 ∈ [0.9, 0.99) 6if f 1 ≥ 0.99 (1) This discretization allows the estimator to distinguish total failures (Class 0) from strong partial successes (Classes 1–5) and perfect outputs (Class 6). To prevent data leakage and ensure rigorous generalization to completely unseen text and layouts, the dataset split is strictly enforced at the unique base sample (document) level rather than via random row splitting. Out of the 360 total base contexts, 240 documents are assigned to the training set (yielding 240×25=6, 000 training pairs), 20 documents are assigned to validation (20×25=500 validation pairs), and the remaining 100 core documents are strictly held out as the test set (100×25=2, 500 test pairs). Because the test split preserves the complete, unbroken combinatorial grid for these 100 entirely unobserved documents, it provides a leak-free empirical ground truth to validate our out-of-sample hierarchical scanning trajectories. The 2,500-instance test split is used strictly to evaluate the estimator’s raw classification performance (reporting precision, recall, and F1-score). For the downstream deployment testing, we evaluate the pipeline at the document level using the held-out base of the 100 unique test documents. Crucially, the downstream deployment pipeline requires no pre-existing configuration matrices to be present. 3.2 DRB: A Lightweight Multimodal Performance Estimator The DRB model is a lightweight multimodal estimator that maps a (document, prompt, model, budget) configuration to a predicted performance class. It consists of four modality- specific encoders fused via a late-fusion MLP classification head. Visual encoding. Document pages are encoded with a SigLIP-2-Large-patch16-512 back- bone (Tschannen et al., 2025) (428M parameters). For multi-page documents, page-level embeddings are processed by a 12-layer decoder-style Llama fusion transformer with hid- den dimension 1152, which performs cross-page attention and mean-pooling to produce a fixed document representation z doc ∈R D . Textual encoding. The task prompt is encoded by Qwen3-0.6B (Yang et al., 2025), pro- ducingz prompt . A linear projectionW p aligns the textual and visual dimensions:z ′ prompt = W p z prompt . Configuration conditioning. Learned embedding layers map the target model namem and reasoning budgetbto continuous vectorse model ,e budget ∈R D , allowing the estimator to condition its performance prediction on the specific configuration being queried. Fusion and classification. The four representations are concatenated and passed to a 2048-unit MLP head with GELU activation: z final = [z doc ∥ z ′ prompt ∥ e budget ∥ e model ], ˆ y = MLP(z final )(2) We optimize with standard cross-entropy loss over the seven performance classes. 3.3 From Estimates to Budget Decisions: Hierarchical Scanning At inference time, DRB’s predictions are used to select a reasoning budget via hierarchical scanning. For a fixed target model, DRB queries each budget level from lowest to highest, 5 Published as a conference paper at COLM 2026 predicting the expected performance class for each. Scanning terminates early when DRB predicts Class 6 (perfect performance), as there is no benefit to testing higher budgets. The budget with the highest predicted performance class is selected, with lower cost as a tie- breaker. In the exploratory model-selection case study (Section 4.4), DRB scores all (model, budget) pairs jointly, though this extends the estimator beyond its training distribution. 4 Experiments 4.1 Setup Models evaluated.We evaluate five frontier models spanning two provider families: Gem- ini (gemini-2.5-flash,gemini-3-flash-preview,gemini-3.1-flash-lite-preview) across five token budgetsb ∈ 0, 512, 1024, 1536, 2048; and GPT (gpt-5.1,gpt-5.2) across the same five budget classes mapped to effort tiers (four distinct levels, since classes 3 and 4 both map to medium; see Section 3.1). Evaluation benchmarks.We evaluate downstream performance on the same three task do- mains used to construct BudgetDoc: (i) RVL-CDIP (document classification), (i) TAT-DQA (tabular QA), and (i) CheckboxQA (form-checkbox extraction). Crucially, all experimental evaluations are conducted strictly using the 100 held-out base documents quarantined in our test split, ensuring that the target models are tasked with optimizing compute for entirely unseen visual layouts and structural configurations. Metrics.We report F1-score as the primary quality metric, total API cost per sample (USD), and latency (seconds) for baseline results. For DRB based experiments, we additionally report an adjusted cost that factors in the estimator’s infrastructure compute overhead. This overhead is calculated dynamically for each model-dataset configuration by multiplying its empirical average scanning latency by the hardware operational cost ($0.00015 per second for a GCP T4 GPU instance, derived from standard market rates of $0.54/hour). This total system cost ensures that all comparisons reflect the actual local resource expenditure. Baselines. For single-model experiments, the baseline is the same model run always at maximum reasoning budget, the natural default when no estimator is available. For the model-selection case study, the baseline is the best single model at maximum budget per dataset, representing an oracle that already knows which model to deploy. 4.2 How Well Does DRB Estimate LLM Performance? Before applying DRB’s estimates downstream, we characterize estimation quality on the BudgetDoc test set (n =2500). DRB achieves a weighted F1 of 0.753, demonstrating that a ∼1B-parameter model can meaningfully estimate the performance of frontier LLMs across variable reasoning budgets. Table 3 (Appendix A) provides a per-class breakdown. The most salient finding is the strong Class 6 recall of 0.932. Class 6 represents samples where the model achieves near-perfect output regardless of budget. High recall on this class means DRB reliably detects “easy” samples, specifically inputs where inference will succeed without extended reasoning. This is the most actionable signal for budget optimization: if DRB is confident a sample belongs to Class 6 at a low budget, it can assign zero additional thinking tokens without sacrificing accuracy. Estimation is hardest at the intermediate classes (F1 of 0.46–0.74 for Classes 1–5), reflecting the stochastic nature of partial reasoning successes where small differences in chain length can produce large accuracy swings. This difficulty is acceptable for the budget optimization application: even imperfect middle-class estimates allow cost reductions, as long as the system avoids misclassifying Class 6 samples as failures. 6 Published as a conference paper at COLM 2026 Table 1: Reasoning budget optimization via DRB estimation on held-out test documents. Baseline: auto-budget for Gemini and high/xhigh for GPT models (the default without an estimator).∆F1 and Adj.∆Cost are relative to the baseline, where the adjusted cost dynamically factors in the configuration-level estimator compute overhead.∆Lat shows absolute seconds followed by percentage change. Green = improvement, red = degradation. DatasetModelDRB F1 BL F1∆F1Adj. DRB CostBL CostAdj.∆Cost∆Lat (s, %) CDIP Gemini-2.5-Flash0.8880.840+5.7%$0.00169$0.00427 −48.0%+3.6s (+27.6%) Gemini-3-Flash-Preview0.8900.920 −3.3%$0.00965$0.01292 −21.3%+3.8s (+25.5%) Gemini-3.1-Flash-Lite0.7990.762+4.9%$0.00482$0.00752 −29.0% +10.3s (+76.4%) GPT-5.10.8150.816 −0.1%$0.00096$0.01896 −92.2% −15.5s (−40.7%) GPT-5.20.8790.869+1.1%$0.00093$0.09611 −99.0% −39.7s (−62.3%) CheckboxQA Gemini-2.5-Flash0.5590.556+0.4%$0.00301$0.00515 −31.4%+3.8s (+31.9%) Gemini-3-Flash-Preview0.7050.670+5.1%$0.01746$0.02126 −15.4%+3.5s (+19.4%) Gemini-3.1-Flash-Lite0.6680.642+4.0%$0.00873$0.01080 −14.3% +22.0s (+146.9%) GPT-5.10.5030.478+5.2%$0.00274$0.00202+35.6%+6.5s (+45.5%) GPT-5.20.6510.673 −3.2%$0.00428$0.31805 −98.5% −144.3s (−85.4%) TAT-DQA Gemini-2.5-Flash0.5140.504+2.1%$0.00088$0.00149 −5.2% −0.9s (−26.4%) Gemini-3-Flash-Preview0.5420.515+5.2%$0.00080$0.00372 −64.3% −2.1s (−29.2%) Gemini-3.1-Flash-Lite0.5360.568 −5.7%$0.00040$0.00156 −40.6% −0.6s (−15.5%) GPT-5.10.5040.519 −2.9%$0.00032$0.00031+3.2%+0.8s (+35.3%) GPT-5.20.5310.547 −2.9%$0.00057$0.06084 −98.2% −29.2s (−90.5%) 4.3 Application I: Per-Sample Reasoning Budget Optimization We now use DRB’s performance estimates to dynamically select reasoning budgets per sample across the held-out test documents, replacing the naive default of always allocating the maximum budget ceiling. Table 1 reports these out-of-sample optimization results for all five models across the three benchmark tasks. DRB estimates drive consistent cost reductions.In 13 of the 15 model-dataset configura- tions, DRB-guided budget selection reduces cost relative to the maximum-budget baseline. Reductions range from 5.2% to 99%, with the largest savings on GPT models where extended thinking is especially expensive. This confirms that DRB’s estimates are actionable: across diverse models and tasks, they reliably identify when less reasoning is sufficient. Accuracy is maintained or improved. In 9 of 15 configurations, DRB-guided selection matches or improves F1 over the maximum-budget baseline, a direct consequence of the over-thinking penalty. The gains are largest forgemini-2.5-flashon CDIP (+5.7%), on TAT-DQA (+5.2%) and CheckboxQA (+5.1%). This confirms that DRB’s estimation of when additional budget hurts is well-calibrated. GPT cost savings are extreme. GPT models with extended thinking enabled are dispro- portionately expensive at maximum budget (c.f Table 1, 4):gpt-5.2costs $0.096/sample on CDIP and $0.318/sample on CheckboxQA (10–60×above Gemini). DRB correctly estimates that most samples do not require this compute, selecting zero or minimal budgets and achieving 99.0% and 98.5% cost reductions with F1 changes of≤3.2% on CheckboxQA. This is the most compelling demonstration that estimation-guided budget selection has practical value. TAT-DQA is the hardest estimation target.Across all models, TAT-DQA yields the weak- est estimation-to-accuracy transfer: some configurations show F1 losses of up to−5.7% under DRB-guided selection. GPT effort-level sensitivity. Table 1 compares DRB against the maximum-effort GPT baseline (high/xhigh). Table 4 (Appendix C) extends this comparison across all four GPT effort tiers, confirming that DRB’s F1 almost matches the performance at all the effort tiers. The only constant exception being TAT-DQA where the performance is marginally but consistently lower. 7 Published as a conference paper at COLM 2026 Table 2: Exploratory case study: applying DRB estimates to cross-model selection on held-out test documents. Oracle baseline represents the best single model deployed at default auto-budget (Gemini) or maximum reasoning effort (GPT). Adjusted DRB cost dynamically incorporates the configuration-level estimator compute overhead. While cross- family selection triggers expected F1 degradation under this unobserved distribution shift, cost savings remain substantial across all scopes by routing away from expensive high-effort tiers. Selection ScopeDatasetDRB F1Oracle F1∆F1Oracle CostAdj. DRB CostAdj.∆Cost Gemini Family CDIP0.8970.920 −2.5%$0.01292$0.00593 −54.1% CheckboxQA0.6440.670 −3.9%$0.02126$0.00513 −75.9% TAT-DQA0.5120.568 −9.9%$0.00156$0.00141 −9.6% GPT Family CDIP0.7810.869 −10.1%$0.09611$0.00121 −98.7% CheckboxQA0.6220.673 −9.3%$0.31805$0.00183 −99.4% TAT-DQA0.4970.545 −9.5%$0.06084$0.00061 −99.0% Cross-Family CDIP0.8610.920 −6.4%$0.01292$0.00476 −63.2% CheckboxQA0.6020.676 −10.9%$0.02825$0.00133 −95.2% TAT-DQA0.5050.569 −11.2%$0.00156$0.00063 −59.6% 4.4 Application I: Model Selection (Case Study) DRB was designed and trained to estimate performance for a fixed model across budgets, not to select among models. As a secondary case study, we probe whether its learned utility surface generalizes to the more challenging task of cross-model selection: given an unobserved document input from the held-out test split and a target candidate pool, can DRB accurately identify the most resource-efficient model configuration to deploy? This question is exploratory; we do not claim DRB is a routing system, and we identify dedicated multi-provider routing architectures as important future work. Table 2 reports results for three scopes of model selection: within the Gemini family, within the GPT family, and across both families. The baseline is the oracle best single model at maximum budget (for GPT-family) or auto-budget (for Gemini-family). Gemini-family selection: estimation transfers well.Selecting among three Gemini mod- els, the family closest to DRB’s training distribution, reduces cost by 10–76% (adjusted) with F1 losses of−2.5% to−9.9%. On CheckboxQA, DRB falls−3.9% below oracle F1 while cutting cost by 75.9%; on CDIP it stays within−2.5% of the oracle while halving cost. These results are encouraging given that DRB was not explicitly trained for model selection. GPT-family selection: large F1 loss, extreme cost savings. GPT-family selection shows larger F1 degradation:−10.1% on CDIP,−9.3% on CheckboxQA, and−9.5% on TAT- DQA relative to the oracle. For GPT-family, adjusted DRB cost reduces expenditure by 98–99% across all three datasets, because DRB systematically routes GPT inputs to low-effort configurations. The result is a stark trade-off: DRB captures nearly all available cost savings while sacrificing 9–10% in F1. Cross-family selection: the limits of estimation transfer.Expanding the pool to all Gemini and GPT models compounds the calibration mismatch. F1 drops by 6.4% (CDIP) and 11.2% (TAT-DQA).Despite this, the cost savings remain large (59–94% adjusted), suggesting that DRB captures enough signal to avoid expensive configurations even when fine-grained accuracy prediction drops. These results confirm that cross-family model selection requires estimators trained with explicit multi-provider supervision. 5 Analysis Small models can estimate LLM performance. Figure 1 confirms the over-thinking penalty is systematic and universal: Gemini 2.5 Flash on RVL-CDIP loses 6% F1 when the budget rises from 0 to 2048 tokens; GPT-5.2 loses 12% F1 on CheckboxQA going from 8 Published as a conference paper at COLM 2026 lowtohigheffort. The core question of this paper, whether a lightweight multimodal model can estimate frontier LLM reasoning performance, has an affirmative answer, with important nuances. DRB achieves 0.753 weighted F1 on BudgetDoc (c.f Table 3) and reliably identifies the most actionable classe i.e Class 6 (perfect output, F1 = 0.860 with recall 0.932). This is meaningful as it enables the model to predict (with high precision and recall), the tasks that are completely solvable at every budget, helping the DRB assign lower budgets with high confidence when needed. What the estimation quality enables ?Estimation quality of 0.75 weighted F1 is sufficient to drive large practical gains because the savings mechanism is robust to prediction errors. Even when DRB mispredicts a middle class (e.g., predicts Class 4 when the true class is Class 5), the selected budget is unlikely to be catastrophically wrong. The dominant win comes from correctly identifying high-confidence Class 6 samples, which DRB does reliably, and routing them to zero or minimal reasoning budgets. DRB complements, rather than replaces, in-flight reasoning controls. Frontier models increasingly offer built-in budget mechanisms: Gemini 2.5 Flash’s auto-thinking and GPT- 5’sreasoningefforttiers adjust compute in-flight based on the model’s self-assessment of difficulty. DRB is architecturally complementary. In-flight controls calibrate how much the model thinks; DRB determines whether to allocate extended thinking at all by predicting expected performance before the API call is issued. Chen et al. (2026) demonstrate that Gemini 3 Flash, even with dynamic auto-thinking, generated over 60,000 thinking tokens on select MMLU-Pro problems while GPT-5.4 needed only 25, resulting in a realized cost 35% higher despite a lower listed price. The auto-thinking mechanism could not override the model’s internally perceived need to reason at length. An external pre-flight estimator like DRB can: if DRB predicts Class 6 at budgetb =512, the API call is issued with thinkingbudget=512, enforcing a hard cap that the model’s own mechanism would not have respected. This makes external estimation a necessary complement to in-flight controls, not a redundancy. When estimation is most valuable ?The value of estimation scales with the gap between baseline cost and optimal cost. For GPT-5.2 on CheckboxQA, this gap is enormous ($0.318 vs. $0.004 per sample); DRB’s estimates are highly actionable even when imperfect. For cheap Gemini-Lite models on short documents, the absolute gap is small and the DRB inference step’s latency overhead can dominate. The practical takeaway: estimation-guided budget selection is most valuable for high-cost models on complex documents. Estimation fails gracefully on hard tasks. TAT-DQA consistently represents the hardest estimation target across all settings. For hybrid numerical reasoning over financial tables, the performance surface over budgets is irregular: small changes in chain length can produce large accuracy swings, making it difficult for any pre-hoc estimator to learn a reliable mapping. Even here, DRB achieves substantial cost savings (40–98%) because it can still avoid extremely expensive configurations, even when fine-grained budget ranking is imperfect. Estimation generalizes uniformly across document types. DRB achieves strong budget optimization on all three datasets (c.f Table 1), including the visually distinctive Check- boxQA: Gemini-3-Flash-Preview improves by +5.1% F1 at−15.4% adjusted cost, and GPT- 5.2 achieves 98.5% cost savings with only−3.2% F1. These results confirm that DRB’s joint visual and textual representations are sufficient to capture performance variation across structurally diverse document types, ranging from dense financial tables (TAT-DQA) to scanned form images (CheckboxQA), within a single estimator trained on the BudgetDoc benchmark. Early stopping reflects estimation confidence.In single-model settings, DRB’s hierarchical scanning tests on average 1.0–3.9 of the 5 available budget levels per sample. Although the full sample-by-sample scanning trajectories are omitted from the primary tables for brevity, a comprehensive analysis of our runtime execution logs reveals highly efficient search 9 Published as a conference paper at COLM 2026 dynamics. On CDIP, all Gemini models achieve a 100% early-stop rate, where the estimator is confident enough on the very first budget level to immediately terminate the scan and bypass higher tiers. This high-confidence conservative behavior holds globally: across the remaining dataset-model configurations, ensuring that the estimator ’s operational overhead remains modest relative to the downstream API savings it enables. 6 Conclusion We asked whether a lightweight multimodal model can estimate the reasoning performance of frontier LLMs on document tasks. The answer is yes: DRB, a∼1B-parameter estimator, achieves 0.753 weighted F1 on the BudgetDoc benchmark and, crucially, 0.932 recall on perfect-performance class. These estimates are actionable: applying them for per-sample reasoning budget selection delivers up to 99% cost reductions across five frontier models and three datasets, while maintaining or improving F1 in 9 of 15 configurations. The over- thinking penalty, where maximum reasoning budgets produce worse accuracy at higher cost, is systematic and universal across modern model families. We additionally showed that DRB’s learned performance surface partially transfers to model selection (an application it was not designed for), with moderate F1 losses (−2.5% to −9.9%) and strong cost savings (10–76%) for intra-family Gemini selection, and degrading F1 accuracy as the scope widens to GPT and cross-family settings (though cost savings remain large at 59–99% due to avoidance of expensive high-effort configurations). We identify multi-provider performance estimation as an important direction for future work, alongside dedicated routing architectures trained with explicit cross-provider supervision. Our results also establish that external pre-flight estimation is necessary even as frontier providers offer increasingly sophisticated in-flight reasoning controls. Built-in dynamic thinking modes (Gemini auto-thinking, GPT reasoningeffort tiers) adjust compute based on the model’s self-assessment, but Chen et al. (2026) show this does not prevent pricing reversals: 32% of nominally cheaper models still cost more in practice due to thinking-token overrun. Bai et al. (2026) confirm that models systematically underestimate their own token consumption. An external estimator like DRB provides a hard pre-flight cap that in-flight mechanisms cannot. BudgetDoc and the DRB baseline are released to facilitate future research into sample-level LLM performance estimation and compute-optimal inference. Limitations While our framework demonstrates robust performance across both Gemini and GPT model families, several limitations govern its deployment. First, expanding or adapting this architecture to entirely new model providers or unobserved model architectures introduces a data-gathering requirement; a small calibration sample of configuration-outcome pairs must be collected through a standard evaluation process to fine-tune the estimator on the new target domain. Second, our evaluation is strictly restricted to document-centric tasks where visual layout and structural complexity dictate reasoning success. Extending pre-flight performance estimation to non-visual, text-dominant applications such as general-purpose QA, code generation, or open-ended creative tasks remains an open direction for future work. Finally, the hierarchical budget scanning process could introduce an operational latency overhead depending on the task. This computational overhead dictates that our framework is highly favorable for token-heavy, high-cost frontier model inference, but less suitable for ultra-low-latency or real-time application pipelines. Ethics Statement This work uses publicly available benchmarks (RVL-CDIP, TAT-QA, CheckboxQA) and frontier model APIs accessed under standard commercial terms. No personal data was collected or used. By enabling estimation-guided budget selection, our work reduces unnecessary reasoning token expenditure, which has a positive environmental impact. 10 Published as a conference paper at COLM 2026 References Longju Bai, Zhemin Huang, Xingyao Wang, Jiao Sun, Rada Mihalcea, Erik Brynjolfsson, Alex Pentland, and Jiaxin Pei. How do AI agents spend your money? Analyzing and predicting token consumption in agentic coding tasks. arXiv preprint arXiv:2604.22750, 2026. Lingjiao Chen, Chi Zhang, Yeye He, Ion Stoica, Matei Zaharia, and James Zou. The price reversal phenomenon: When cheaper reasoning models cost more. arXiv preprint arXiv:2603.23971, 2026. Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convo- lutional nets for document image classification and retrieval. In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), p. 991–995. IEEE, 2015. Isaac Ong, Amjad Almahairi, Vincent Vincent, Srinivasan NMS, Ion Stoica, et al. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665, 2406. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time com- pute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Al- abdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. Michał Turski, Mateusz Chili ́ nski, andŁukasz Borchmann. Unchecked and overlooked: Addressing the checkbox blind spot in large language models with checkboxqa, 2025. URL https://arxiv.org/abs/2504.10419. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,Łukasz Kaiser, and Illia Polosukhin.Attention is all you need.In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.URLhttps://proceedings.neurips.c/paperfiles/paper/2017/file/ 3f5e243547dee91fbd053c1c4a845a-Paper.pdf. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. Yifan Wu, Charlie Snell, Edouard Oyallon, and Sergey Levine. Inference scaling laws: An empirical analysis of compute-optimal inference for problem solving with language models. arXiv preprint arXiv:2408.00724, 2024. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. arXiv preprint arXiv:2105.07624, 2021. A DRB Per-Class Estimation Performance on BudgetDoc Table 3 provides the full per-class breakdown of DRB’s estimation performance on the BudgetDoc test set (n =2500). Classes 0–6 correspond to F1-score bins defined in Section 3.1, where Class 0 represents total task failure and Class 6 represents perfect output. The strong Class 6 recall (0.932) is the most consequential metric for the budget optimization application: it governs how reliably DRB can assign minimal reasoning budgets to samples where any budget would yield correct output, without sacrificing accuracy. Low Class 1 11 Published as a conference paper at COLM 2026 Table 3: DRB (SigLIP-2 + Qwen3-0.6B) per-class estimation performance on the BudgetDoc test set. Class 6 (perfect performance) has the highest F1, driven by strong recall, meaning DRB reliably identifies samples where any budget will succeed. Class (F1 bin)PrecisionRecallF1Support 0( f 1 < 0.5)0.6230.4910.549448 1( f 1 ∈ [0.5, 0.6))0.6460.3610.46387 2( f 1 ∈ [0.6, 0.7))0.7160.6710.693143 3( f 1 ∈ [0.7, 0.8))0.7440.5690.645174 4( f 1 ∈ [0.8, 0.9))0.8490.6560.740154 5( f 1 ∈ [0.9, 0.99))0.7810.6530.71198 6( f 1 ≥ 0.99)0.7980.9320.8601396 Weighted Avg0.7560.7650.7532500 recall (0.361) reflects the inherent difficulty of distinguishing near-total failures from total failures, a fine-grained distinction that matters less in practice, since both map to low-cost budget assignments. B Implementation Details The DRB estimator uses aSigLIP-2-large-patch16-512vision backbone and aQwen3-0.6B prompt encoder. Training was conducted for 20 epochs on an NVIDIA A100 (80GB) using the AdamW optimizer with learning rate 5×10 −5 and batch size 4. The 12-layer fusion transformer uses hidden dimension 1152. DRB compute overhead. The DRB estimator is deployed on GCPn1-standard-4with one NVIDIA T4 GPU (us-central1 on-demand pricing: $0.19/hr VM + $0.35/hr GPU = $0.54/hr total, reducing to $0.00015 per second of compute). Because our hierarchical budget scanning uses an early-stopping mechanism, the execution latency scales dynamically based on the complexity of the document-query pair. To ensure a rigorous financial comparison, the adjusted system cost (Adj. DRB Cost) reported in paper across tables is computed dynamically at the configuration level: Adj. DRB Cost = API Cost RB + (∆t config × $0.00015) where∆t config represents the empirical mean scanning latency for that specific model- dataset group. This approach guarantees that the local computational burden of running the pre-flight estimator is fully captured and charged against the downstream API savings. C GPT Effort-Level Sensitivity Table 4 extends the comparison from Table 1 to all four GPT reasoning effort tiers (none,low, medium, high/xhigh). 12 Published as a conference paper at COLM 2026 Table 4: DRB versus GPT models across all four reasoning effort tiers. Adj.∆Cost factors in the dynamic configuration-level compute overhead. Green = improvement, red = degrada- tion. DatasetModel (Effort)DRB F1 BL F1∆F1Adj. DRB CostBL CostAdj.∆Cost CDIP GPT-5.1 (Low)0.8150.814 ±0.0%$0.00096$0.01814 −91.8% GPT-5.1 (Medium)0.8150.813 +0.2%$0.00096$0.02148 −93.1% GPT-5.1 (High)0.8150.816 −0.1%$0.00096$0.02900 −94.9% GPT-5.2 (Low)0.8790.873 +0.7%$0.00093$0.05738 −97.5% GPT-5.2 (Medium)0.8790.876 +0.3%$0.00093$0.05851 −97.5% GPT-5.2 (High)0.8790.869 +1.1%$0.00093$0.06073 −97.6% CheckboxQA GPT-5.1 (Low)0.5030.491 +2.4%$0.00274$0.02418 −86.5% GPT-5.1 (Medium)0.5030.488 +3.1%$0.00274$0.03014 −89.2% GPT-5.1 (High)0.5030.478 +5.2%$0.00274$0.05040 −93.5% GPT-5.2 (Low)0.6510.663 −1.8%$0.00428$0.07439 −93.5% GPT-5.2 (Medium)0.6510.658 −1.1%$0.00428$0.07751 −93.8% GPT-5.2 (High)0.6510.673 −3.2%$0.00428$0.12162 −96.0% TAT-DQA GPT-5.1 (Low)0.5040.513 −1.8%$0.00032$0.00210 −59.8% GPT-5.1 (Medium)0.5040.508 −0.8%$0.00032$0.00276 −69.4% GPT-5.1 (High)0.5040.519 −2.9%$0.00032$0.00547 −84.5% GPT-5.2 (Low)0.5310.540 −1.7%$0.00057$0.00511 −78.6% GPT-5.2 (Medium)0.5310.535 −0.7%$0.00057$0.00605 −81.9% GPT-5.2 (High)0.5310.547 −2.9%$0.00057$0.00700 −84.4% 13