Paper deep dive
Empirical Recipes for Efficient and Compact Vision-Language Models
Jiabo Huang, Zhizhong Li, Sina Sajadmanesh, Weiming Zhuang, Lingjuan Lyu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/22/2026, 5:46:54 AM
Summary
The paper presents a systematic study of inference bottlenecks in compact vision-language models (VLMs) and introduces 'ArgusVLM', a model family designed for efficiency and structured perception. The authors identify that CPU-side operations, such as image preprocessing and tokenization, are the primary latency drivers in compact VLMs, rather than GPU compute. They provide actionable optimization recipes that significantly reduce time-to-first-token (TTFT) and end-to-end latency, and demonstrate that ArgusVLM achieves superior performance on VQA and dense captioning benchmarks compared to existing compact models.
Entities (6)
Relation Signals (3)
ArgusVLM â optimizedfor â Inference Efficiency
confidence 95% ¡ ARGUSVLM achieves strong performance while maintaining a compact and efficient design.
austin â profiles â CPU-side operations
confidence 95% ¡ We therefore recommend austin for CPU-side profiling.
vLLM â serves â ArgusVLM
confidence 90% ¡ We build upon the efficient serving framework vLLM
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deploying vision-language models (VLMs) in resource-constrained settings demands low latency and high throughput, yet existing compact VLMs often fall short of the inference speedups their smaller parameter counts suggest. To explain this discrepancy, we conduct an empirical end-to-end efficiency analysis and systematically profile inference to identify the dominant bottlenecks. Based on these findings, we develop optimization recipes tailored to compact VLMs that substantially reduce latency while preserving accuracy. These techniques cut time to first token (TTFT) by 53% on InternVL3-2B and by 93% on SmolVLM-256M. Our recipes are broadly applicable across both VLM architectures and common serving frameworks, providing practical guidance for building efficient VLM systems. Beyond efficiency, we study how to extend compact VLMs with structured perception outputs and introduce the resulting model family, ArgusVLM. Across diverse benchmarks, ArgusVLM achieves strong performance while maintaining a compact and efficient design.
Tags
Links
- Source: https://arxiv.org/abs/2603.16987v1
- Canonical: https://arxiv.org/abs/2603.16987v1
Trouble viewing inline? Open PDF directly â
Full Text
48,089 characters extracted from source content.
Expand or collapse full text
Empirical Recipes for Efficient and Compact Vision-Language Models Jiabo Huang 1 * Zhizhong Li 1 * Sina Sajadmanesh 1 Weiming Zhuang 1 Lingjuan Lyu 1 Abstract Deploying vision-language models (VLMs) in resource-constrained settings demands low la- tency and high throughput, yet existing compact VLMs often fall short of the inference speedups their smaller parameter counts suggest. To ex- plain this discrepancy, we conduct an empiri- cal end-to-end efficiency analysis and systemati- cally profile inference to identify the dominant bottlenecks. Based on these findings, we de- velop optimization recipes tailored to compact VLMs that substantially reduce latency while pre- serving accuracy. These techniques cut time to first token (TTFT) by 53% on InternVL3-2B and by 93% on SmolVLM-256M. Our recipes are broadly applicable across both VLM architec- tures and common serving frameworks, provid- ing practical guidance for building efficient VLM systems. Beyond efficiency, we study how to ex- tend compact VLMs with structured perception outputs and introduce the resulting model fam- ily, ARGUSVLM. Across diverse benchmarks, ARGUSVLM achieves strong performance while maintaining a compact and efficient design. 1. Introduction Building on the success of large language models, vision lan- guage models (VLMs) have advanced rapidly, enabling vi- sual understanding through free-form, instruction-following text generation (Zhu et al., 2025; Bai et al., 2025b). This progress has sparked growing interest in compact VLMs for deployment in resource-constrained settings, such as edge devices (Korrapati; Chu et al., 2023; 2024), due to their lower compute and memory footprint. However, we find that these smaller models often do not deliver the expected end-to-end efficiency gains. For instance, despite having far fewer parameters, SmolVLM-256M (Marafioti et al., 2025) has a longer time-to-first-token (TTFT; 344.7 ms) than InternVL3-8B (Zhu et al., 2025) (177.4 ms). * Equal contribution 1 Sony AI. Correspondence to: Lingjuan Lyu <Lingjuan.Lv@sony.com>. Preprint. March 19, 2026. Š Sony AI Confidential | 10 93% TTFT â 42% performance â 53% TTFT â 5% performance â Figure 1. ARGUSVLM excels at both performance and inference efficiency with a faster time-to-first-token (TTFT) compared to existing compact VLMs. Each bubble represents a model variant, where the area indicates model size. In this work, we conduct a comprehensive study of compact VLMs with a focus on their end-to-end efficiency. Through systematic profiling, we find that in the compact regime, previously overlooked CPU-side operations, such as image processing and text tokenization, often dominate latency. In contrast, GPU-side computations contribute less to over- all latency, due to extensive prior optimizations such as FlashAttention (Dao et al., 2022), CUDA graphs (NVIDIA Corporation, 2024), and kernel fusion (Filipovi Ë c et al., 2015). Motivated by these findings, we develop targeted optimiza- tions that substantially reduce inference latency without compromising accuracy. As a result, we reduce TTFT by 53% on InternVL3-2B (124.0â 57.7ms) and by 93% on SmolVLM-256M (344.7â 22.8ms), as shown in Figure 1. Beyond inference efficiency, we investigate how compact VLMs can be extended from multimodal understanding to visual perception tasks that require structured, region-level outputs. We focus on dense image captioning (Johnson et al., 2016), a practical and challenging task that pairs localized image regions with semantically rich captions. While prior VLM approaches generate bounding boxes either as numeric coordinates in text or via specialized location tokens, it remains unclear which design is more effective, especially in the compact regime. To clarify this, we conduct a controlled comparison of these two formulations and identify the most 1 arXiv:2603.16987v1 [cs.CV] 17 Mar 2026 Empirical Recipes for Efficient and Compact Vision-Language Models Figure 2. ARGUSVLM-2B achieves strong performance on image understanding and captioning tasks across five benchmarks com- pared with leading vision-language models such as QwenVL (Bai et al., 2025b) and InternVL (Zhu et al., 2025). suitable approach for enabling compact VLMs to perform structured prediction under a language-modeling objective. Our contributions are threefold. (i) We present a systematic end-to-end profiling study of compact VLMs, pinpointing the primary latency bottlenecks and deriving actionable guidance for practical deployment. (i) We introduce AR- GUSVLM, a family of compact VLMs spanning 256Mâ2B parameters, and conduct a controlled study of bounding- box prediction formulations to better unify structured visual perception with text-based multimodal understanding. (i) ARGUSVLM achieves strong performance on diverse VQA and captioning benchmarks (Figure 2), as well as dense image captioning, delivering up to 93% and 80% reductions in TTFT and end-to-end latency, respectively, compared to SmolVLM-256M. 2. Related Works 2.1. Vision-language Models Vision-language models (VLMs) have rapidly evolved over the past few years. Pioneering methods such as CLIP (Rad- ford et al., 2021; Zhai et al., 2023a; Sun et al., 2023) adopt a dual-encoder design to learn discriminative image repre- sentations by aligning image and text embeddings using lan- guage supervision. Inspired by the success of generative lan- guage modeling (Radford et al., 2018; Brown et al., 2020), later works adopt an encoder-decoder architecture (Raffel et al., 2020) that unifies diverse computer vision tasks as a token generation problem with an end-to-end training (Lu et al., 2022a; 2024b; Xiao et al., 2024). Following the suc- cess of LLaVA (Liu et al., 2023) on visual instruction tuning, recent developments (Li et al., 2024a; Chen et al., 2024c; Zhu et al., 2025; Bai et al., 2025b; Team et al., 2025) have gradually converged to the decoder-only architecture, where visual embeddings from a pretrained vision encoder are used as conditions for text generation via a pre-trained large language model (LLM) (Yang et al., 2025; Cai et al., 2024; Allal et al., 2025). This design greatly improves flexibility and generalization across tasks. 2.2. Compact Vision-Language Models There is a growing interest in building compact VLMs, usu- ally with fewer than 2 billion parameters, that can deliver strong multimodal performance under limited resources. Moondream (Korrapati) emphasizes both efficiency and per- formance but remains closed-source. H2OVL (Galib et al., 2024) explores compact VLMs optimized for OCR-related tasks, yet its capability to generalize to broader visual do- mains is underexplored. SmolVLM (Marafioti et al., 2025) highlights the importance of memory footprint by an exten- sive empirical study of architectural trade-offs for compact models. However, it fails to achieve the expected inference speed up compared to larger models and are primarily eval- uated on language-based understanding tasks, overlooking structured perception problems such as dense captioning. 2.3. Efficiency of VLMs Beyond reducing model size, recent VLMs improve runtime efficiency through techniques such as visual token compres- sion (Zhu et al., 2025; Marafioti et al., 2025; Chu et al., 2023; 2024) and more efficient multimodal fusion (Zhang et al., 2025), which alleviate the computation overhead in- duced by long visual token sequences. In parallel, serving systems such as vLLM (Kwon et al., 2023) provide practical, general-purpose optimizations (e.g., kernel-level speedups and scheduling) that significantly improve throughput and latency. Our methods target end-to-end bottlenecks and identify overlooked bottlenecks on top of generic serving frameworks. We observe additional inference speed im- provements even when deploying models with vLLM. 2.4. Perception Support of VLMs Visual perception capabilities are increasingly recognized as essential and have been integrated into several state-of- the-art VLMs. Most existing approaches represent object locations either as plain-text numerical coordinates (Zhu et al., 2025; Bai et al., 2025b) or via dedicated special to- kens (Xiao et al., 2024; Lu et al., 2022a). However, it remains unclear which formulation is more effective, par- ticularly for compact VLMs, where limited model capacity may amplify representational bottlenecks. In this work, we provide a controlled comparison of these design choices and study how to best equip compact VLMs with structured perception outputs under a language-modeling objective. 2 Empirical Recipes for Efficient and Compact Vision-Language Models Figure 3. Flame graph from profiling VLM inference with austin (Tornetta, 2026) to identify CPU-side bottlenecks. The visual- ization separates processes for easier analysis: Process 2953917 performs multimodal preprocessing, while Process 2954325 runs model inference. Best viewed digitally with zoom. 3. Recipes to Improve Inference Efficiency To improve the inference efficiency of compact VLMs, we build upon the efficient serving framework vLLM (Kwon et al., 2023). vLLM is a highly optimized system for serving large language models with low latency and state-of-the-art throughput (vLLM Team, 2026). 3.1. Inference Computation Profiling Accurately identifying bottlenecks is crucial for improving VLM efficiency and requires reliable profilers. We therefore evaluated several profilers to measure the time of both CPU- and GPU-side operations in end-to-end inference. Profilers.We choose austin (Tornetta, 2026) for CPU-side profiling and NVIDIA Nsight Systems (NVIDIA Corpora- tion, 2026) after evaluating the following profilers: â˘cProfile (Python Software Foundation, 2026) is a built-in, deterministic Python profiler that provides function-level measurements and call counts. However, it incurs higher overhead than sampling profilers and offers limited sup- port for visualizing complex codebases such as vLLM. â˘py-spy (Frederickson, 2026) and austin (Tornetta, 2026) are low-overhead sampling profilers that can attach to a running Python process without code changes and provide line-level attribution. Both can generate flame graphs to surface hotspots. Compared to py-spy, austin better supports multi-process applications, which is im- portant for profiling vLLM because it uses multiple pro- cesses to handle requests (see Figure 3). We therefore recommend austin for CPU-side profiling. â˘NVIDIA Nsight Systems (NVIDIA Corporation, 2026) provides a system-level view of GPU activity, includ- ing kernel timings, memory transfers, and CPUâGPU interactions. We use it to diagnose GPU-side bottlenecks, particularly to understand why quantization may not yield speedups for compact VLMs. (a) SmolVLM-256M with bfloat16 precision. (b) SmolVLM-256M with W8A8 quantization. Figure 4. GPU profiling of SmolVLM-256M inference under vLLM with (a) bfloat16 and (b) W8A8 quantization. The timelines show GPU kernels within a transformer layer during decoding, with the average execution time (Îźs) annotated below each kernel. Despite reduced precision, the quantized model is slower. Quantization Remains Ineffective for Compact VLMs. We quantize our ARGUSVLM-256M (introduced in Sec- tion 4) using llmcompressor (vLLM Project, 2024) with FP8 W8A8 quantization, and evaluate inference efficiency under vLLM against the bfloat16 baseline. Surprisingly, the quantized model achieves 6.4% lower decoding throughput than bfloat16 (499.08 vs. 533.28 tokens/s on an H100 GPU). To identify the cause, we profile both settings and inspect the GPU timelines for one transformer layer during decod- ing (Figure 4). Quantized matrix multiplications, including QKV projection, output projection, gate-up projection, and down projection, are indeed faster (-1.39Îźs) by1.1Ă. How- ever, activation quantization introduces substantial overhead (+5.50Îźs), despite being fused into the layernorm and silu kernels. Overall, this overhead outweighs the gains from quantized matrix multiplications, leading to lower through- put. These results suggest that fully FP8 execution (avoiding bfloat16âFP8 conversions) is a promising direction for im- proving quantized inference in compact VLMs. CPU-Side Operations are the Primary Bottleneck.Pro- filing in Figure 4 also highlights the extent of existing GPU optimizations in vLLM: (1) compute-intensive operators leverage optimized kernels such as FlashAttention (Dao et al., 2022); (2) kernels are fused where possible (e.g., lay- ernorm and rotary embedding) to reduce memory traffic; and (3) CUDA Graphs reduce kernel-launch overhead, re- sulting in near-continuous kernel execution. Together, these optimizations push GPU-side performance close to satura- tion. As a result, end-to-end latency in compact VLMs is often dominated by CPU-side workload, which is frequently overlooked in prior optimization efforts. We therefore focus on CPU-side bottlenecks, which directly impact TTFT â a key metric for interactive user experience. 3 Empirical Recipes for Efficient and Compact Vision-Language Models Table 1. Impact of incremental optimizations on InternVL3-2B inference. The first row is baseline performance under vLLM. Optimizations TTFT Throughput E2E Latency (ms)â (tokens/s)â(ms)â InternVL3-2B124.0270.86180.85 + Reduce img transformâ & 80.9279.14127.20Tensor img process⥠& GPU preprocess⢠+ Pin memoryâŁ72.8278.43119.31 + Reduce PIL decodingâ¤71.7280.30109.52 + BF16 img normalizeâĽ68.4282.87113.40 + Tokenizer⌠& UInt8â§58.3281.94105.82 + Pack H2D transfer 12 57.7292.66101.26 Table 2. Impact of incremental optimizations on SmolVLM-256M inference. The first row is baseline performance under vLLM. Optimizations TTFT Throughput E2E Latency (ms)â (tokens/s)â(ms)â SmolVLM-256M344.7460.71427.59 + Pillow-SIMDâ¨241.6461.12325.20 + Reduce img transformâ & 71.9420.91151.34Tensor img process⥠& Reduce PIL decoding⤠+ GPU preprocessâ˘59.7466.21131.65 + UInt8â§47.4471.77118.32 + BF16 img normalizeâĽ38.3470.40109.71 + Remove pixel mask 10 29.5494.5196.54 + Pin memoryâŁ28.7497.9595.26 + TokenizerâŚ27.4498.8693.91 + Avoid split 11 23.5500.7489.67 + Pack H2D transfer 12 22.8533.2885.07 3.2. Case Studies: InternVL3-2B and SmolVLM-256M We next present two case studiesâInternVL3-2B (Zhu et al., 2025) and SmolVLM-256M (Marafioti et al., 2025)âto illustrate common implementation inefficiencies in compact VLM inference and the impact of targeted fixes. Settings. We use LMMs-Eval (Zhang et al., 2024) to run the COCO2017 (Lin et al., 2014) image captioning bench- mark with vLLM as the serving backend. The benchmark contains 5,000 multimodal requests, evaluated with a batch size of 1. During inference, we profile CPU-side execution using austin and inspect flame graphs (e.g., Figure 3) to locate hotspots. We then apply targeted optimizations, re- measure latency, and iterate until the remaining costs are no longer dominated by obvious implementation bottlenecks. InternVL3-2B. As summarized in Table 1, the optimiza- tions reduce TTFT from124.0to57.7ms (-53%). End-to- end latency (from request arrival to the full response) drops from 180.85 to 101.26 ms (-44%). SmolVLM-256M. We apply the same workflow to SmolVLM-256M and report results in Table 2. TTFT de- creases from344.7to22.8ms (-93%), and end-to-end la- tency descends from 427.59 to 85.07 ms (-80%). 3.3. Practical Recipes to Reduce Latency Based on these case studies, we distill the following practi- cal recipes for reducing latencies in compact VLMs: â˘Minimize image processing on the critical path. (a) Avoid repeated image decode/encodeâ¤, resizing, crop- ping and paddingâ . (b) Use optimized image libraries e.g., Pillow-SIMD (Uploadcare, 2026) for decodingâ¨, torchvision for transformsâĄ. (c) When possible, offload preprocessing (resize/normalize) to GPUâ˘. (d) Simplify the image processing logic wherever possible⼠10 11 . â˘Reduce CPUâGPU communication.(a) Transfer smaller dtypes (e.g., UInt8 images instead of Float32) and convert on-deviceâ§. (b) Use pinned (page-locked) memory to accelerate H2D (host-to-device) transfersâŁ. (c) Packing small H2D transfers into one transfer 12 . â˘Tokenization can be slow. The large number of repeated image token placeholders (e.g.,<IMGCONTEXT>) in VLMs leads to extremely long prompts, which can slow down tokenization. Using a more compact placeholder scheme yields measurable TTFT gainsâŚ. â˘Profile first, optimize later. Use profilers such as austin and Nsight Systems to identify real bottlenecks before optimization and avoid premature optimizations. 4. Recipes for Building Compact VLMs Beyond efficiency, a practical challenge for VLMs is adapt- ing them to downstream tasks that are not explicitly covered during pre-training. For example, although several recent models support visual grounding (Zhu et al., 2025; Qwen et al., 2024), few can perform dense captioning, a useful capability that requires structured, region-level outputs. To best enable dense captioning in the compact regime, we con- duct a controlled comparison of two common formulations for learning regional labels under a next-token prediction objective. Based on the most effective design, we introduce ARGUSVLM, a family of compact VLMs that supports dense captioning while maintaining strong performance on conventional text-based image understanding tasks. In this section, we first describe the ARGUSVLM architecture and then detail the two bounding-box prediction formulations. 4.1. Model Architecture We follow recent decoder-only VLMs (Marafioti et al., 2025; Zhu et al., 2025) and adopt the âViTâMLPâLLMâ architec- ture (Chen et al., 2024d) shown in Figure 5. A pretrained Vision Transformer (ViT) (Vaswani et al., 2017) encodes an input image into a sequence of visual embeddings. In parallel, the textual instruction (or question) is tokenized into text tokens and embedded using the vocabulary of a pre- trained large language model (LLM). An MLP then projects 4 Empirical Recipes for Efficient and Compact Vision-Language Models mythology Greek in giant seeing all- the is is What ... Resize to the closest aspect ratios Split into fixed-size tiles Resize Pixel Shuffle What is shown in this image? LLM MLP ? Argus ... ... ViT Tokenizer Figure 5. Overview of the ARGUSVLM architecture. ARGUSVLM consists of a Vision Transformer (ViT), a single-layer MLP, and a large language model (LLM). Given an input image, we first resize it to the closest aspect ratio from a predefined set, then split it into square tiles that are resized to the ViT input resolution; we also generate a global thumbnail to provide scene-level context. Each tile (and the thumbnail) is independently encoded by the ViT into patch tokens, which are then compressed by concatenating features from spatially adjacent patches along the channel dimension. In parallel, the user instruction is tokenized into text tokens. Finally, the visual and text tokens are concatenated and fed to the LLM, which generates the response autoregressively. the visual embeddings into the LLM embedding space to align feature dimensions across modalities. Finally, the pro- jected visual tokens are concatenated with the text tokens and passed to the LLM for autoregressive generation. Image Encoding. Inspired by UReader (Ye et al., 2023), we adopt an image-tiling strategy to preserve fine-grained visual details. Most off-the-shelf ViTs are pretrained on fixed-resolution square images, so na Ě Äąvely resizing high- resolution inputs with arbitrary aspect ratios often intro- duces distortion and obscures important visual details. To mitigate this issue, we first resize each image to the closest aspect ratio from a predefined set (minimizing geometric distortion), then split it into square tiles and resize each tile to the ViTâs native input resolution. In addition, we cre- ate a global thumbnail to provide scene-level context. All tiles and the thumbnail are encoded independently by the vision encoder, and their resulting visual token sequences are concatenated to form the visual input to the LLM. This design enables robust handling of diverse aspect ratios while retaining both local detail and global context. Visual Token Reduction. Image tiling preserves fine- grained details but increases the number of visual tokens, which can slow down the LLM. To reduce this overhead, ARGUSVLM applies pixel-unshuffle (space-to-depth) (Shi et al., 2016; Marafioti et al., 2025) to the ViT patch grid. Specifically, we merge eachrĂrneighborhood of spatially adjacent patch tokens into a single token by concatenating their features along the channel dimension. If the original visual token sequence has shape(N,D), this produces a compressed sequence of shape( N /r 2 ,D¡r 2 ), yielding anr 2 reduction in token count while largely preserving local vi- sual information. This design provides a favorable trade-off between visual fidelity and inference efficiency. Training and Inference. We train ARGUSVLM with teacher forcing (Williams & Zipser, 1989), predicting each output token conditioned on the preceding ground-truth to- kens. All training samples are reformatted into dialogues between a âUserâ and an âAssistantâ using a unified chat template. Given an input image and instruction, we tokenize the dialogue and concatenate the resulting text tokens with the corresponding visual tokens to form an input sequence X âR TĂd , with sequence lengthTand hidden dimension d. We construct supervision by shifting the sequence by one position so that the model learns next-token prediction. We optimize a categorical cross-entropy loss computed only over the assistant response tokens: L(θ) =â T X i=t logP θ (x i | x <i ),(1) whereθdenotes the model parameters,tis the index of the first assistant token, andP θ (x i | x <i )is the predicted probability of the ground-truth tokenx i given all previous tokens. During inference, we apply the same chat template to the userâs instruction and image(s) and leave the assistant response empty for autoregressive generation. 4.2. Structured Label Learning Text-based image understanding provides a flexible interface for interacting with VLMs. However, many real-world ap- plications require structured visual perception outputs. We focus on dense image captioning, which associates localized image regions with semantically rich captions. For example, in autonomous driving, dense captioning can describe fine- grained scene elements (e.g., road type and location), while the model then performs higher-level reasoning over these elements. We study two common formulations for learning bounding boxes under a next-token prediction objective. 5 Empirical Recipes for Efficient and Compact Vision-Language Models Table 3. Comparison to the leading vision-language models on text-based image understanding benchmarks. Results of models marked with * are measured by their open-source checkpoints provided on Huggingface (Wolf et al., 2019) using LMMs-Eval (Zhang et al., 2024). Model#Parameters VQACaptioning GQAPOPEVQAv2COCO2017NoCaps Exact MatchâAccuracyâAccuracyâBLEU4âCIDErâBLEU4âCIDErâ SmolVLM* (Marafioti et al., 2025)256M41.580.166.00.0560.1180.0870.182 ARGUSVLM (Ours)256M55.387.370.40.3471.1910.4091.014 SmolVLM* (Marafioti et al., 2025)500M45.385.970.40.1840.7120.2730.722 ARGUSVLM (Ours)500M58.286.374.10.3691.2630.4381.086 MobileVLM (Chu et al., 2023)1.7B56.184.5----- MobileVLM-V2 (Chu et al., 2024)1.7B59.384.3----- SmolVLM* (Marafioti et al., 2025)2B49.287.775.30.2260.8890.3510.932 InternVL3* (Zhu et al., 2025)2B58.989.672.50.3941.3710.4721.208 DeepSeekVL (Lu et al., 2024a)2B59.388.3----- MiniGemini (Li et al., 2024c)2.3B59.983.9----- Qwen2.5-VL* (Bai et al., 2025b)3B60.288.172.70.2761.0140.3991.047 Qwen3-VL* (Bai et al., 2025a)2B59.489.474.10.1270.2200.1710.194 ARGUSVLM (Ours)2B63.389.380.70.4221.4200.4771.187 Plain-Text Coordinates. A straightforward approach se- rializes a box as numeric coordinates, e.g.,[x 1 ,y 1 ,x 2 ,y 2 ], and trains the model to generate them as text (Zhu et al., 2025; Bai et al., 2025b). While simple and compatible with standard LLM vocabularies, each coordinate may be split into multiple subword tokens, which can weaken the alignment between textual symbols and spatial locations. Special Location Tokens. An alternative discretizes the image into aKĂ Kgrid and introduces dedicated location tokensloc i K i=1 (Xiao et al., 2024; Lu et al., 2024b). A po- sition is represented by predicting two tokens corresponding to its row and column indices. This avoids fragmenting nu- meric coordinates into multiple tokens and provides a more explicit, semantically grounded representation of spatial positions. We conduct a controlled comparison of these two formula- tions on compact VLMs. Our results (see Section 5.2) show that special location tokens consistently yield better dense captioning performance. We hypothesize that discretized location tokens provide a clearer mapping between output symbols and image regions, which is particularly beneficial in the compact regime. 5. Experiments This section describes our experimental setup and evalu- ates compact vision-language models (VLMs) on both text- based image understanding and dense image captioning (structured perception). Datasets. We train ARGUSVLM on a mixture of image- caption pairs, visual instruction data, and localization data with bounding-box annotations. To support reproducibil- ity, we rely exclusively on publicly available datasets. For image captioning, we use 1.2M image-caption pairs from ShareGPT4V (Chen et al., 2024b) and 591K pairs from the COCO2017 training split (Lin et al., 2014). For instruc- tion following, we combine 3.55M curated examples from ShareGPT4V (Chen et al., 2024b), WiT (Srinivasan et al., 2021), VFLAN (Chen et al., 2024a), ScienceQA (Lu et al., 2022b), and MGM-Instruct (Li et al., 2024b). To enable structured bounding-box prediction, we leverage box annota- tions from Objects365 (Shao et al., 2019), COCO2017 (Lin et al., 2014), and Visual Genome (Krishna et al., 2017). Evaluation Metrics. We evaluate ARGUSVLM against leading compact VLMs on standard image understanding benchmarks: VQAv2 (Goyal et al., 2017) for general vi- sual question-answering, POPE (Li et al., 2023) with a focus on object hallucination detection, and GQA (Hudson & Manning, 2019) for compositional reasoning and rela- tional understanding. For captioning, we report results on COCO2017 (Lin et al., 2014) and NoCaps (Agrawal et al., 2019). For structured perception, we evaluate dense im- age captioning following the protocol of Wu et al. (2024). We also measure inference efficiency via time-to-first-token (TTFT) and decoding throughput (tokens/second). Implementation Details.We build ARGUSVLM at three scales, from 256M to 2B parameters. We bootstrap from publicly available pretrained checkpoints and apply super- vised fine-tuning to both (i) improve efficiency and (i) adapt the model to dense captioning. Specifically, AR- GUSVLM-256M and ARGUSVLM-500M are based on SmolVLM (Marafioti et al., 2025), using a single-layer MLP to connect a SigLIP-B/16 (Zhai et al., 2023b) vision en- coder (93M parameters) to SmolLM2-135M and SmolLM2- 360M (Allal et al., 2025) text decoders, respectively. AR- 6 Empirical Recipes for Efficient and Compact Vision-Language Models Š Sony AI 22 InternVL3-2B ArgusVLM-2B Qwen3-VL-2B (Patch=1) ArgusVLM-2B (Patch=1) SmolVLM-500M ArgusVLM-500M SmolVLM-256M ArgusVLM-256M (a) Time-to-first-token (ms)â Š Sony AI 23 InternVL3-2B ArgusVLM-2B Qwen3-VL-2B (Patch=1) ArgusVLM-2B (Patch=1) SmolVLM-500M ArgusVLM-500M SmolVLM-256M ArgusVLM-256M (b) Decoding throughput (tokens/s)â Figure 6. Efficiency comparison between ARGUSVLM and leading VLMs. ARGUSVLM outperforms both SmolVLM and InternVL3 on TTFT and throughput. As Qwen3-VL-2B does not use tiling, we compare it to ARGUSVLM-2B under the patch=1 setting. Table 4. Comparison of bounding-box representations for dense captioning on Visual Genome. ModelBbox FormatmAP (%) ARGUSVLM-256MPlain-text1.89 ARGUSVLM-256MLocation Tokens1.98 Florence2-base (0.23B)Location Tokens1.67 ARGUSVLM-500MPlain-text2.18 ARGUSVLM-500MLocation Tokens3.21 Florence2-large (0.77B)Location Tokens1.48 GUSVLM-2B follows the same design but uses InternViT- 300M (Chen et al., 2024d) and Qwen2.5-1.5B (Qwen et al., 2024) as the vision encoder and text decoder. The input resolution is512Ă 512for ARGUSVLM-256M and AR- GUSVLM-500M, and448Ă 448for ARGUSVLM-2B. Af- ter encoding image tiles and the thumbnail into visual em- beddings, we apply pixel unshuffle (Shi et al., 2016) to reduce the number of visual tokens to 64 for ARGUSVLM- 256M/500M (r = 4) and 256 for ARGUSVLM-2B (r = 2). During inference, we apply the optimizations introduced in Section 3.2 to all model variants. 5.1. Comparison on Visual Understanding As shown in Table 3, ARGUSVLM achieves strong perfor- mance on diverse VQA and captioning benchmarks com- pared to existing compact VLMs with fewer than 2B pa- rameters. Compared to compact variants of recent state- of-the-art VLM families such as InternVL3 (Zhu et al., 2025) and Qwen3-VL-2B (Bai et al., 2025a), ARGUSVLM- 2B achieves the best performance on 4 out of 5 image understanding tasks. Notably, ARGUSVLM-2B outper- forms Qwen2.5-VL across all reported metrics with fewer parameters. Restricting to models under 1B parameters, ARGUSVLM-256M and ARGUSVLM-500M improve over SmolVLM counterparts by an average of 16% and 11% on the three VQA benchmarks. Overall, ARGUSVLM pro- vides state-of-the-art accuracy in the sub-2B regime while remaining compact and efficient. 5.2. Comparison on Dense Captioning We evaluate dense image captioning on Visual Genome (Kr- ishna et al., 2017), which requires predicting a set of regions together with free-form captions describing each region. Following prior work (Johnson et al., 2016; Xiao et al., 2024), we compute a joint mean Average Precision (mAP) over regionâcaption pairs by treating each paired prediction as a retrieval target. We compare two ways of representing bounding boxes within the language modeling space: (i) plain-text coordinates and (i) special location tokens. Both keep the rest of the output in natural language, maintain- ing compatibility with standard next-token prediction. We evaluate ARGUSVLM at the 256M and 500M scales and compare against Florence2-base and Florence2-large (Xiao et al., 2024), which also use location tokens. As shown in Table 4, special location tokens consistently out- perform plain-text coordinates for all ARGUSVLM variants. We attribute the gap primarily to reduced token fragmenta- tion: location tokens replace multi-token numeric substrings with single, semantically homogeneous units. This simpli- fies optimization during fine-tuning, reduces exposure bias from partially generated numbers, and helps the model learn stable joint distributions overposition, caption. The gains increase with model scale, suggesting that larger models benefit more from discrete spatial representations, while the 256M model remains capacity-limited. 5.3. Comparison on Inference Efficiency We compare ARGUSVLM with InternVL3-2B (Zhu et al., 2025), Qwen3-VL-2B (Bai et al., 2025a), and multiple SmolVLM variants (Marafioti et al., 2025) in Figure 6. For fair comparison, we use batch size 1, run all meth- ods on an NVIDIA H100 GPU, and average over the same COCO2017 image-captioning requests. Baselines use their official vLLM implementations without applying our op- timizations (Section 3). With our proposed optimizations, ARGUSVLM achieves the lowest TTFT and among the best 7 Empirical Recipes for Efficient and Compact Vision-Language Models User: What brand of dogs are these? Assistant: Beagle User: What is keeping the dogs from running away? Assistant: Fence User: What is the painting on above the doors called? Assistant: Graffiti User: Is the door closed? Assistant: Yes User: Provide a one-sentence description of the provided image. Assistant: A group of people sitting at a table under a roof. User: Provide a one-sentence description of the provided image. Assistant: A restaurant with tables and chairs in front of a window. User: What brand of dogs are these? Assistant: Beagle User: What is keeping the dogs from running away? Assistant: Fence User: What is the painting on above the doors called? Assistant: Graffiti User: Is the door closed? Assistant: Yes User: What color is the sign? Assistant: Red and yellow User: What color is the plant Assistant: Green User: How many skyscrapers are there? Assistant: 2 User: What is on the boy's back walking? Assistant: Backpack A group of people sitting at a table under a roof. A restaurant with tables and chairs in front of a window. A dessert with blueberries and mint on a plate. An old ambulance is parked in a field. Figure 7. Qualitative results from ARGUSVLM-2B. Top: VQA and image captioning. Bottom: dense captioning. decoding throughput across model sizes. 5.4. Applying the Optimization Recipes Beyond vLLM The optimization recipes in Section 3.3 also apply to serving stacks beyond vLLM. Following the same iterative profil- ing workflow, we optimized the Hugging Face (Wolf et al., 2019) implementations of InternVL3-2B and SmolVLM- 256M. Optimizing image processing and text tokenization reduces TTFT by 28.3% (145â104 ms) for InternVL3-2B and by 63.3% (332â122 ms) for SmolVLM-256M. We also identified and improved slow GPU-side components (notably RMSNorm and rotary positional embeddings) and removed redundant operations. These changes improve throughput by 47.5% (51.6â98.2 tokens/s) and 43.3% (53.2â 93.9 tokens/s), respectively. 5.5. Visualization Figure 7 provides qualitative examples of ARGUSVLM-2B. The examples in the top row show visual question answer- ing and captioning predictions across diverse scenes. AR- GUSVLM correctly answers user queries, and produces co- herent captions conditioned on the images. The bottom row shows ARGUSVLMâs capability on predicting bounding boxes along with description for dense captioning. These examples demonstrate that ARGUSVLM supports struc- tured perception outputs while retaining the flexible natural- language interface of text-based image understanding. 6. Conclusion We introduced ARGUSVLM, a compact and efficient vision- language model series that unify text-based visual under- standing with fine-grained perception. Through end-to-end profiling of compact VLM inference, we identified key im- plementation bottlenecks and proposed targeted optimiza- tions that significantly reduce latency without degrading accuracy. To extend compact VLMs to structured percep- tion, we further studied effective formulations for learning spatially grounded outputs (e.g., bounding boxes) under a next-token prediction objective. Across extensive exper- iments, ARGUSVLM delivers strong performance while remaining compact, and achieves substantially faster infer- ence than existing compact VLMs. Overall, our results provide practical guidance and empirical evidence for build- ing efficient and capable vision-language systems. 8 Empirical Recipes for Efficient and Compact Vision-Language Models References Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., John- son, M., Batra, D., Parikh, D., Lee, S., and Anderson, P. nocaps: novel object captioning at scale. In Proceed- ings of the IEEE International Conference on Computer Vision, p. 8948â8957, 2019. Allal, L. B., Lozhkov, A., Bakouch, E., Bl Ě azquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydl Ě Äą Ë cek, H., Lajar Ě Äąn, A. P., Srivastav, V., et al. Smollm2: When smol goes bigâdata-centric training of a small language model. arXiv preprint arXiv:2502.02737, 2025. Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, J., Tu, J., Wan, J., Wang, P., Wang, P., Wang, Q., Wang, Y., Xie, T., Xu, Y., Xu, H., Xu, J., Yang, Z., Yang, M., Yang, J., Yang, A., Yu, B., Zhang, F., Zhang, H., Zhang, X., Zheng, B., Zhong, H., Zhou, J., Zhou, F., Zhou, J., Zhu, Y., and Zhu, K. Qwen3-vl technical report, 2025a. URL https://arxiv.org/abs/2511.21631. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877â1901, 2020. Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024. Chen, G. H., Chen, S., Zhang, R., Chen, J., Wu, X., Zhang, Z., Chen, Z., Li, J., Wan, X., and Wang, B. Allava: Har- nessing gpt4v-synthesized data for lite vision-language models. arXiv preprint arXiv:2402.11684, 2024a. Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, p. 370â387. Springer, 2024b. Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024c. Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 24185â24198, 2024d. Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y., Hu, Y., Wei, F., Zhang, X., Zhang, B., Wei, X., et al. Mobilevlm: A fast, strong and open vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y., Sun, X., Hu, Y., Lin, X., Zhang, B., et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and R Ě e, C. FlashAt- tention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022. Filipovi Ë c, J., Madzin, M., Fousek, J., and Matyska, L. Op- timizing cuda code by kernel fusion: application on blas. The Journal of Supercomputing, 71(10):3934â3957, 2015. Frederickson, B. py-spy: A sampling profiler for python programs.https://github.com/benfred/ py-spy, 2026. GitHub repository. Galib, S., Wang, S., Xu, G., Pfeiffer, P., Chesler, R., Landry, M., and Ambati, S. S. H2ovl-mississippi vi- sion language models technical report. arXiv preprint arXiv:2410.13611, 2024. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 6904â6913, 2017. Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6700â 6709, 2019. Johnson, J., Karpathy, A., and Fei-Fei, L. Densecap: Fully convolutional localization networks for dense caption- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 4565â4574, 2016. Korrapati, V. URL https://moondream.ai/. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. Visual genome: Connecting language and 9 Empirical Recipes for Efficient and Compact Vision-Language Models vision using crowdsourced dense image annotations. In- ternational journal of computer vision, 123(1):32â73, 2017. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024a. Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305.10355, 2023. Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., Liu, S., and Jia, J. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814, 2024b. Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., Liu, S., and Jia, J. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814, 2024c. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Doll Ě ar, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European conference on computer vision, p. 740â755. Springer, 2014. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tun- ing. Advances in neural information processing systems, 36:34892â34916, 2023. Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Yang, H., et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024a. Lu, J., Clark, C., Zellers, R., Mottaghi, R., and Kembhavi, A. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022a. Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, p. 26439â26455, 2024b. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 35:2507â2521, 2022b. Marafioti, A., Zohar, O., Farr Ě e, M., Noyan, M., Bakouch, E., Cuenca, P., Zakka, C., Allal, L. B., Lozhkov, A., Tazi, N., et al. Smolvlm: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299, 2025. NVIDIA Corporation. CUDA Graphs.NVIDIA, 2024.URLhttps://docs.nvidia. com/cuda/cuda-programming-guide/ 04-special-topics/cuda-graphs.html. NVIDIA Corporation. Nvidia nsight systems.https: //developer.nvidia.com/nsight-systems , 2026. Accessed: 2026-01-27. Python Software Foundation.The Python profil- ers.https://docs.python.org/3/library/ profile.html, 2026. Accessed: 2026-01-27. Qwen, A. Y., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint, 2024. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748â8763. PmLR, 2021. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1â67, 2020. Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., and Sun, J. Objects365: A large-scale, high- quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, p. 8430â8439, 2019. Shi, W., Caballero, J., Husz Ě ar, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., and Wang, Z. Real-time sin- gle image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 1874â1883, 2016. Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Najork, M. Wit: Wikipedia-based image text dataset for 10 Empirical Recipes for Efficient and Compact Vision-Language Models multimodal multilingual machine learning. In Proceed- ings of the 44th international ACM SIGIR conference on research and development in information retrieval, p. 2443â2449, 2021. Sun, Q., Fang, Y., Wu, L., Wang, X., and Cao, Y. Eva- clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. Team, K., Du, A., Yin, B., Xing, B., Qu, B., Wang, B., Chen, C., Zhang, C., Du, C., Wei, C., et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025. Tornetta, G. N.Austin: Python frame stack sampler for cpython.https://github.com/P403n1x87/ austin, 2026. Accessed: 2026-01-27. Uploadcare. uploadcare/pillow-simd.https://github. com/uploadcare/pillow-simd , 2026. Accessed: 2026-01-28. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser,Ĺ., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017. vLLM Project.LLM Compressor, 8 2024.URL https://github.com/vllm-project/ llm-compressor. vLLM Team. vllm documentation.https://docs. vllm.ai/en/latest/, 2026.URLhttps:// docs.vllm.ai/en/latest/. Accessed: 2026-01- 27. Williams, R. J. and Zipser, D. A learning algorithm for con- tinually running fully recurrent neural networks. Neural computation, 1(2):270â280, 1989. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingfaceâs transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. Wu, J., Wang, J., Yang, Z., Gan, Z., Liu, Z., Yuan, J., and Wang, L. Grit: A generative region-to-text transformer for object understanding. In European Conference on Computer Vision, p. 207â224. Springer, 2024. Xiao, B., Wu, H., Xu, W., Dai, X., Hu, H., Lu, Y., Zeng, M., Liu, C., and Yuan, L. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4818â4829, 2024. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Xu, G., Li, C., Tian, J., Qian, Q., Zhang, J., et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. In Findings of the As- sociation for Computational Linguistics: EMNLP 2023, p. 2841â2858, 2023. Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proceed- ings of the IEEE/CVF international conference on com- puter vision, p. 11975â11986, 2023a. Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training, 2023b. Zhang, K., Li, B., Zhang, P., Pu, F., Cahyono, J. A., Hu, K., Liu, S., Zhang, Y., Yang, J., Li, C., and Liu, Z. Lmms-eval: Reality check on the evaluation of large mul- timodal models, 2024. URLhttps://arxiv.org/ abs/2407.12772. Zhang, S., Fang, Q., Yang, Z., and Feng, Y. Llava-mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.03895, 2025. Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 11