Paper deep dive
Detached Skip-Links and $R$-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR
Ziye Yuan, Ruchang Yao, Chengxin Zheng, Yusheng Zhao, Daxiang Dong, Ming Zhang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/23/2026, 12:10:51 PM
Summary
The paper introduces 'Detached Skip-Links' to address gradient interference in multi-layer feature fusion for MLLMs, specifically improving OCR performance by stopping gradients through skip branches during training. It also proposes 'RR-Probe', a reconstruction-based diagnostic tool that uses a shallow decoder initialized from LLM layers to measure the preservation and decodability of fine-grained visual information in projected tokens.
Entities (5)
Relation Signals (3)
Detached Skip-Links → improves → OCR
confidence 95% · our approach consistently improves OCR-centric benchmarks
Detached Skip-Links → mitigates → gradient interference
confidence 95% · To mitigate this gradient interference, we propose Detached Skip-Links
RR-Probe → measures → fine-grained information
confidence 92% · RR-Probe, which measures pixel-level reconstructability of projected visual tokens
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models (MLLMs) excel at high-level reasoning yet fail on OCR tasks where fine-grained visual details are compromised or misaligned. We identify an overlooked optimization issue in multi-layer feature fusion. Skip pathways introduce direct back-propagation paths from high-level semantic objectives to early visual layers. This mechanism overwrites low-level signals and destabilizes training. To mitigate this gradient interference, we propose Detached Skip-Links, a minimal modification that reuses shallow features in the forward pass while stopping gradients through the skip branch during joint training. This asymmetric design reduces gradient interference, improving stability and convergence without adding learnable parameters. To diagnose whether fine-grained information is preserved and usable by an LLM, we introduce $R$-Probe, which measures pixel-level reconstructability of projected visual tokens using a shallow decoder initialized from the first quarter of the LLM layers. Across multiple ViT backbones and multimodal benchmarks, and at scales up to 7M training samples, our approach consistently improves OCR-centric benchmarks and delivers clear gains on general multimodal tasks.
Tags
Links
- Source: https://arxiv.org/abs/2603.20020v1
- Canonical: https://arxiv.org/abs/2603.20020v1
Trouble viewing inline? Open PDF directly →
Full Text
80,040 characters extracted from source content.
Expand or collapse full text
Detached Skip-Links and R-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR Ziye Yuan Ruchang Yao Chengxin Zheng Yusheng Zhao Daxiang Dong Ming Zhang Abstract Multimodal large language models (MLLMs) excel at high-level reasoning yet fail on OCR tasks where fine-grained visual details are compromised or misaligned. We identify an overlooked optimization issue in multi-layer feature fusion. Skip pathways introduce direct back-propagation paths from high-level semantic objectives to early visual layers. This mechanism overwrites low-level signals and destabilizes training. To mitigate this gradient interference, we propose Detached Skip-Links, a minimal modification that reuses shallow features in the forward pass while stopping gradients through the skip branch during joint training. This asymmetric design reduces gradient interference, improving stability and convergence without adding learnable parameters. To diagnose whether fine-grained information is preserved and usable by an LLM, we introduce R-Probe, which measures pixel-level reconstructability of projected visual tokens using a shallow decoder initialized from the first quarter of the LLM layers. Across multiple ViT backbones and multimodal benchmarks, and at scales up to 7M training samples, our approach consistently improves OCR-centric benchmarks and delivers clear gains on general multimodal tasks. Multimodal Learning,Optical Character Recognition (OCR),Feature Fusion,Probing / Representation Analysis 1 Introduction Multimodal large language models (MLLMs) have rapidly advanced the integration of vision and language, showing strong performance in high-level semantic reasoning and dialogue.(Team et al., 2023; Achiam et al., 2023; Wang et al., 2025; Bai et al., 2025) However, they still exhibit a clear performance gap on low-level perception tasks, particularly Optical Character Recognition (OCR) and fine-grained visual grounding. Existing benchmarks show that even state-of-the-art models often hallucinate text in dense documents or fail to resolve small objects in high-resolution scenes(Fu et al., 2024; Liu et al., 2024; Kanade and Ganu, 2025). Previous studies identify the pre-trained Vision Transformer (ViT) as a primary bottleneck (Liu et al., 2025a). By design, contrastive objectives (e.g., CLIP) encourage semantic alignment, pushing the encoder to abstract away spatial details to align with global text descriptions (Tong et al., 2024; Bolya et al., 2025). To mitigate this information loss, recent approaches either incorporate auxiliary objectives such as reconstruction losses (Fini et al., 2025; Tschannen et al., 2025), or rely on end-to-end generative supervision from the multimodal LLM through its next-token prediction (NTP) loss (Guo et al., 2024; Chen et al., 2024b). In parallel, architectural designs increasingly adopt multi-layer fusion to incorporate shallow features that retain geometric and pixel-level information (Yao et al., 2024; Wei et al., 2024; Lin et al., 2025). While intuitively promising, we identify a previously under-discussed optimization issue in this fusion-based architectures. Straightforward fusion establishes direct backpropagation paths from the LLM’s semantic objectives to early visual blocks. This subjects shallow layers originally optimized for low-level patterns to conflicting high-level supervision, resulting in gradient interference and training instability, as in Figure 2. To address this trade-off between spatial detail and training stability, we propose Detached Skip-Links (Fig. 1). By stopping gradients on shallow features before fusion, we decouple feature aggregation from gradient optimization. This mechanism passes fine-grained visual details to the LLM while preventing semantic-heavy gradients from destabilizing shallow layers, without sacrificing the simplicity of skip-connections. Both theoretical analysis and empirical results show that this operation reduces gradient conflict, leading to improved training stability and faster convergence. These effects are consistent across different ViT backbones and remain stable when scaling training to 7M samples. Figure 1: Overview of Detached Skip Links. Intermediate features are concatenated with the final output along the channel dimension, here S denotes stride. We apply a stop-gradient operation to shallow skip features before fusion. This design effectively delivers fine-grained details to the LLM while shielding early layers from optimization conflicts. Beyond optimization, a major challenge in improving fine-grained perception is the lack of a reliable diagnostic metric. Standard downstream benchmarks are often noisy proxies for visual capability, as MLLMs can bypass perception by exploiting language priors or parametric knowledge(He et al., 2025). Existing probing methods typically use linear classifiers(Pagh et al., 2007; Alain and Bengio, 2016; Dosovitskiy, 2020) to assess representation quality, but such tasks are insufficient for measuring fine-grained visual information. To systematically quantify the visual signal effectively transmitted to the LLM, we introduce the Reconstruction Probe (R-Probe). Moving beyond traditional linear separability tests, R-Probe assesses the recoverability of visual details through pixel-level reconstruction. By initializing the reconstruction head with the first quarter layers of the target LLM, we simulate the actual information injection process, probing features exactly as they are ”perceived” by the language model. We posit that minimal reconstruction loss from the projected tokens serves as a proxy for effective information preservation. Our main contributions are summarized as follows: • Introduce Detached Skip-Links: reuse shallow ViT features for fusion while stopping gradients through the selected skip branch, reducing gradient interference and improving training stability without additional parameters. • Propose R-Probe: a reconstruction-based diagnostic that measures whether projected visual tokens retain fine-grained information and remain directly decodable by an LLM-initialized shallow decoder. • Demonstrate effectiveness at scale: extensive experiments across ViT backbones and 22 benchmarks show strong gains on OCR-centric tasks and consistent improvements on general multimodal evaluation. 2 Related Work OCR-Centric Multimodal Large Language Models. Recent advances have shifted from modular OCR pipelines to end-to-end MLLMs that internalize text recognition. While some systems (Cui et al., 2025b) focused on localized extraction, recent large-scale MLLMs (Chen et al., 2024a; Lu et al., 2024; Wu et al., 2024b; Cui et al., 2025a; Team et al., 2025a, b; Bai et al., 2025; Dong et al., 2025; Wang et al., 2025) demonstrate that massive pretraining with OCR-synthesized data yields strong character-level capabilities. To capture fine-grained details, modern architectures often employ dynamic resolution or image-splitting strategies (Chen et al., 2024a; Bai et al., 2025). However, effectively injecting these high-fidelity features into the LLM context remains an open challenge. While earlier general-purpose models relied on heavy bridging modules like (Alayrac et al., 2022; Li et al., 2023; Guo et al., 2024), recent OCR-specific approaches favor multi-scale fusion (Ye et al., 2023; Li et al., 2024) or deep cross-attention (Bai et al., 2025; Chen et al., 2026). Representative examples include TextHawk (Yu et al., 2024), with token compression and multi-level cross-attention, and VLM-FO1 (Liu et al., 2025b), which strengthens regional OCR using auxiliary high-resolution encoders and explicit region tokens. In contrast to complicated architectures, our work provides a lightweight training-time solution without introducing new modules, and is orthogonal to these architectural designs. Multi-layer Fusion Strategies. To mitigate information loss in deep ViTs, researchers have explored leveraging intermediate features. Methods like DeepStack (Meng et al., 2024), DenseConnector (Yao et al., 2024) and (Gao et al., 2022; Lin et al., 2025) aggregate representations from multiple depths, while others combine signals from distinct vision backbones (Shi et al., 2024; Kar et al., 2024; Wu et al., 2024b) or use hierarchical schemes (Zhang et al., 2024). While conceptually sound, such heterogeneous fusion often exhibits unstable optimization dynamics in practice. In the broader optimization literature, controlling or blocking gradient flow has been explored as a strategy to improve training stability in settings such as recurrent networks and representation learning (Arpit et al., 2018; Yu et al., 2020). A plausible source of instability is the interference between high-level semantic gradients from the LLM and the shallow layers’ role in preserving fine-grained visual details, such as character strokes. As a result, naive fusion can lead to gradient conflicts and impair training stability. We therefore investigate whether decoupling feature propagation from gradient flow can alleviate such interference, while retaining the benefits of shallow visual cues. Detail Preservation and Diagnostics. Several works preserve or recover fine detail by introducing specialized tokens and reconstruction/decoding objectives, including AURORA/Perception Tokens (Bigverdi et al., 2025), Morph-Tokens (Pan et al., 2024) and SeTok (Wu et al., 2024a). Our R-Probe is aligned in spirit as a fidelity assessment, but differs by being probe-only. Specifically, it reuses a shallow decoder initialized from the quater of LLM layers to probe representations, without introducing additional heavyweight decoding modules. 3 Detached Skip Links 3.1 Motivation Skip-based multi-scale fusion is a common strategy for injecting fine-grained visual details into vision–language models. However, straightforward skip connections introduce a direct gradient path from the language modeling objective to shallow visual blocks. These shallow blocks primarily encode low-level geometric structures, which differ from the high-level semantic objectives of the LLM. Empirically, we observe that allowing full gradient backpropagation through skip connections leads to diffused and structurally inconsistent attention patterns in shallow layers (Fig. 2). The visualization follows prior work (Darcet et al., 2023), with implementation details provided in the appendix A. As shown in Fig. 2 (Middle), gradients dominated by semantic objectives disrupt pre-trained spatial priors, degrading the encoder’s ability to localize fine-grained features. In contrast, detaching gradients (Fig. 2, Right) preserves baseline-like structural consistency. This observation motivates Detached Skip-Links, a mechanism designed to decouple feature aggregation from gradient propagation. Figure 2: Impact of gradient backpropagation on shallow-layer representations. We visualize [CLS] attention maps of the 4th ViT block. Left: Original frozen attention patterns. Middle (Full Gradients): Backpropagation from the LLM leads to diffused and structurally inconsistent attention, as semantic-heavy gradients disrupt pre-trained spatial priors. Right (Detached): Detaching gradients preserves fine-grained structural consistency. 3.2 Architecture To resolve the conflict described above, we adopt a selective detachment strategy based on feature depth. To formalize this, we partition the intermediate skip features into two sets: a shallow group shallowh_shallow (e.g., blocks 6, 12) and a deep group deeph_deep (e.g., blocks 18, 23). The input to the fusion adapter is constructed as: =MLP([main,;,deep,;,sg(shallow)]),z=MLP ( [h_main,;,h_deep,;,sg(h_shallow) ] ), (1) where [⋅][·] denotes concatenation and sg(⋅)sg(·) is the stop-gradient operator. This selective detachment establishes a dual-pathway optimization landscape, grounded in the hypothesis that feature depth dictates optimization compatibility. Deep features inherently align with the output and benefit from joint optimization. In contrast, shallow features primarily capture low-level geometry and are more susceptible to distortion under direct supervision. By allowing gradients to back-propagate only through the deep branch, our design enables semantic alignment while simultaneously shielding early layers from interference. This ensures that low-level cues remain robust and fully available for the forward pass (as supported by Proposition 4.1). Comprehensive ablations analyzing which layers to detach and how to configure fusion strides are presented in Section 6.3. An overview of the training pipeline is illustrated in Fig. 1. 3.3 Gradient Dynamics Analysis To further justify the necessity of detachment, we analyze gradient statistics during the early phase of joint training (i.e., the first 1.3k steps). We validate Assumption 4.2 by monitoring the gradient flow at the first skip-link block (Block 6) during the the early phase of the joint training stage. As illustrated in Fig. 3(b), the empirical results are consistent with our hypothesis. The skip path is initially dominated by high-variance noise rather than coherent signal, remains approximately orthogonal to the main path, and exhibits negligible cross-covariance cancellation. Detailed experimental settings and further analysis are provided in Appendix C.3. (a) Grad. norms (gm/gs, MA). (b) Trace (tr(Σm)tr( _m), tr(Σs)tr( _s)). Figure 3: Gradient Analysis on the First Skip-Link Block. We visualize the training dynamics during the early phase of the joint training stage. (a) Gradient norms of the skip and main branches, measured by [‖2]E[\|g\|^2], together with their MA(Moving Average) counterparts, where the MA serves as an estimation to ‖[]‖2\|E[g]\|^2. (b) Trace statistics computed as [‖2]−‖[]‖2E[\|g\|^2]-\|E[g]\|^2, characterizing the variance of the gradients over training. (c) The cosine similarity between gskipg^skip and gmaing^main remains close to zero, indicating approximate orthogonality. (d) Scatter plot of δ defined in Assumption 4.2. Additional analysis is provided in Appendix C. (c) Similarity cos(gskip,gmain) (g^skip,g^main). (d) δ defined in Assumption 4.2. 4 Theoretical Analysis We provide a simplified analysis to build intuition for the proposed detachment strategy. Our analysis focuses on the local optimization behavior during the early stage of joint training, with the goal of explaining why gradient detachment leads to improved empirical stability. 4.1 Why skip-links help: a Bayes-risk decomposition We first justify the forward benefit of fusing shallow skip features B=b(X)B=b(X) with the deep main-path representation A=a(X)A=a(X). While deep networks are theoretically universal approximators, in practice, they act as information bottlenecks that may attenuate local details essential for dense understanding. We quantify this benefit by comparing the optimal population risk of a fusion predictor fskip(X)=G([A;B])f_skip(X)=G([A;B]) versus a main-only predictor fmain(X)=F(A)f_main(X)=F(A). Proposition 4.1 (Skip features provide complementary predictive information). Consider the squared loss ℛ(f)≜(Y−f(X))2R(f) (Y-f(X))^2. The reduction in Bayes risk achievable by incorporating the skip feature B is exactly the conditional variance explained by B given A: Δℛ≜infFℛ(F(A))−infGℛ(G([A;B]))=[Var([Y∣A,B]∣A)]=[([Y∣A,B]−[Y∣A])2]≥0. split & _FR (F(A) )- _GR (G([A;B]) )\\ =&E\! [Var\! (E[Y A,B] A ) ]\\ =&E\! [ (E[Y A,B]-E[Y A] )^2 ]≥ 0. split (2) Moreover, Δℛ>0 >0 whenever B provides additional predictive information beyond A, i.e., whenever ℙ([Y∣A,B]≠[Y∣A])>0P\! (E[Y A,B] [Y A] )>0 (equivalently, Y⟂̸B∣AY B A). Interpretation. Shallow features can complement deep representations. Eq. (2) formalizes a simple condition under which fusing B with A strictly improves the optimal achievable risk: the skip feature B must contain task-relevant information that is not already captured by the deep representation A. In practice, deep encoders are shaped by architectural and training biases (e.g., reduced spatial resolution due to striding/pooling and a tendency to emphasize coarse semantic abstractions), which can reduce sensitivity to fine-grained local cues that are useful for dense grounding. Thus, even when the model class is expressive, the learned deep feature A may not retain all predictive cues needed for the downstream objective. Since B is extracted from earlier layers, it can preserve complementary localized or high-frequency information and thereby reduce the irreducible error captured by Δℛ . (See Appendix B.1 for the detailed derivation.) 4.2 Gradient Estimation: Pathwise Decomposition and Variance Structure Consider the training objective ℒ(θ)L(θ). In architectures with skip-based fusion, the stochastic gradient on the shared parameters at iteration t admits a pathwise decomposition: t=tmain+tskip,g_t=g_t^main+g_t^skip, (3) where tmaing_t^main is propagated through the main visual backbone, and tskipg_t^skip is induced by the skip-fusion pathway. Empirical motivation. In the early stage of joint optimization (e.g., the initial phase of FFT(Full Fine-Tuning) after warmup), we observe two recurring patterns (Fig. 3(b)): (i) gradient misalignment: cos(main,skip) (g^main,g^skip) is often near-zero or negative; (i) variance dominance: the second moment (or a running variance proxy) of skipg^skip is substantially larger than that of maing^main. Mean–covariance decomposition. Let ≜[main]m [g^main] and ≜[skip]s [g^skip], and define Σm≜Cov(main) _m (g^main), Σs≜Cov(skip) _s (g^skip), and Σms≜Cov(main,skip) _ms (g^main,g^skip). Then the second moment of the full estimator full≜main+skipg_full ^main+g^skip can be written as [‖full‖2]=‖+‖2+tr(Σm+Σs+Σms+Σms⊤),E\! [\|g_full\|^2 ]=\|m+s\|^2+tr\! ( _m+ _s+ _ms+ _ms ), (4) where the trace terms summarize the total variance and cross-covariance contributions. (We include the short derivation in Appendix B.2 for completeness.) Assumption 4.2 (Early-phase pathwise gradient statistics). During the early stage of joint optimization, the skip-path gradient is noise-dominant and only weakly beneficial in expectation: 1. Variance dominance: tr(Σs)≥c⋅tr(Σm)tr( _s)≥ c·tr( _m) for some c≫1c 1. 2. Weak (or adverse) mean alignment: ⟨,⟩≤0 ,s ≤ 0 and ‖≤ρ‖\|s\|≤ρ\|m\| for a small ρ. 3. Limited cross-covariance cancellation (mild): |tr(Σms+Σms⊤)|≤δ⋅tr(Σs) |tr( _ms+ _ms ) |≤δ·tr( _s) for some δ∈[0,1)δ∈[0,1). SNR and the role of detachment. We measure the quality of a stochastic gradient estimator g via a directional signal-to-noise ratio (SNR), η()≜‖[]‖2[‖2]=‖[]‖2‖[]‖2+tr(Cov()).η(g) \|E[g]\|^2E[\|g\|^2]= \|E[g]\|^2\|E[g]\|^2+tr(Cov(g)). (5) Eq. (4) shows that fullg_full inherits a potentially large variance term tr(Σs)tr( _s) from the skip path. Under Assumption 4.2, this variance dominates the denominator in (5), while the skip mean s provides little (or even adverse) contribution. Detaching the skip pathway on shared parameters corresponds to using detach≜maing_detach ^main, which removes this dominant source of stochastic variability and increases the effective directional SNR in the early phase. One-step progress under smoothness. To connect the estimator-level view to optimization progress, assume ℒL is L-smooth and consider the update θ+=θ−γθ^+=θ- . A standard smoothness argument yields (see Appendix B.3) [ℒ(θ+)]≤ℒ(θ)−γ⟨∇ℒ(θ),[]⟩+Lγ22[‖2].E [L(θ^+) ] (θ)-γ (θ),E[g] + Lγ^22E [\|g\|^2 ]. (6) Proposition 4.3 (When detachment improves early-phase stability). Let full=main+skipg_full=g^main+g^skip and detach=maing_detach=g^main. If ⟨∇ℒ(θ),⟩≤Lγ2(‖full‖2−‖detach‖2), (θ),s \;≤\; Lγ2 (E\|g_full\|^2-E\|g_detach\|^2 ), (7) then [ℒ(θ−γdetach)]≤[ℒ(θ−γfull)]E[L(θ- _detach)] [L(θ- _full)]. Interpretation. Condition (7) makes the bias–variance tradeoff explicit. The left-hand side measures the additional expected descent contributed by the skip mean gradient s, while the right-hand side captures the extra second-moment penalty incurred when adding the skip component. In the early phase, Assumption 4.2 suggests that s is small and weakly aligned, whereas ‖full‖2−‖detach‖2E\|g_full\|^2-E\|g_detach\|^2 is dominated by the high-variance skip term, making detachment more stable and often yielding better expected one-step progress. This perspective is consistent with analyses of biased SGD (Ajalloeian and Stich, 2020), where a small early-phase bias can be beneficial if it substantially reduces effective gradient noise. 5 Empirical Analysis: R-probe Hallucinations in OCR, such as misrecognizing “appie” as “apple”, can originate from two distinct failure modes: (i) loss of fine-grained visual information during visual tokenization (Wei et al., 2024), or (i) representational misalignment, where visual tokens are projected into a space that is poorly used by downstream language models(Huang et al., 2024). Metrics based solely on final textual outputs conflate these factors, making it difficult to distinguish visual encoding errors from language-side inference failures (He et al., 2025). To disentangle these effects, we introduce the Reconstruction Probe (R-Probe), a reconstruction-based diagnostic designed to evaluate whether visual tokens preserve sufficient information and are aligned with an LLM-style decoding regime. Crucially, R-Probe operationalizes reconstructability under a constrained, LLM-aligned decoder, using reconstruction performance as a proxy for visual fidelity and representational alignment, rather than as a general-purpose auto-encoding objective. 5.1 R-Probe as an LLM-Aligned Diagnostic Head Figure 4 (right) illustrates the R-Probe architecture. The probe attaches a lightweight reconstruction head to a pre-trained multimodal backbone, while strictly freezing the ViT encoder and the adapter. These frozen components constitute the subject of evaluation and define the vision-language bridging mechanism under analysis. The probe itself consists of a shallow Transformer decoder followed by an MLP projector that maps decoder states back to pixel space, as in Fig 4 right branch. The decoder is initialized from the first quarter of layers of a pre-trained language model(e.g LLaMA-3.1-8B). This choice intentionally restricts the probe’s expressive capacity: early LLM layers operate in a relatively modality-agnostic regime, whereas deeper layers increasingly encode language-specific abstractions and priors (Liang et al., 2022). By restricting depth and freezing the backbone, successful reconstruction is only possible if visual tokens are both information-rich and projected into a subspace directly used by an LLM-style decoder. Figure 4: Overview of the R-Probe and its use as an auxiliary training signal. The R-Probe corresponds to the right reconstruction branch, which evaluates information retention and LLM-aligned consumability of visual tokens under frozen encoders. When attached to the full model (left), the same reconstruction head supplies an auxiliary loss during training, encouraging visually faithful representations without modifying the primary LLM decoding objective. 5.2 Context-Aware Sequence Modeling Figure 5: Sequence Construction Strategy. For reconstruction used in R-probe, we prioritize context (background image and text prompts) before the target image tokens. This forces the probe to verify if the adapter successfully projects visual features into a space that the LLM can utilize for context-dependent reconstruction. To reflect the conditional nature of OCR inference, we adopt a context-aware reconstruction protocol (Figure 5). Input images are resized to 448×448448× 448, and consistent with the adapter design, 2×22× 2 adjacent ViT patch embeddings are merged into a single visual token to match the LLM hidden dimension. Rather than reconstructing images in isolation, we construct the input sequence as =[context_img,text,target_img],S=[E_context\_img,E_text,E_target\_img], (8) where target_imgE_target\_img corresponds to the image region containing the text of interest. Global 2D RoPE is applied prior to reordering, ensuring that absolute spatial relationships are preserved across visual tokens. This formulation enforces conditional reconstruction: the probe must recover the target region while attending to both surrounding visual context and textual prompts, rather than performing unconditional image reconstruction. We further analyze R-Probe behavior under missing textual or visual inputs in the Appendix D.2, to isolate the respective roles of language context and visual evidence. 5.3 Diagnostic Rationale: Why Reconstruction Matters The diagnostic value of R-Probe relies on the premise that effective visual tokens must be compatible with the LLM’s initialization space to support reconstruction. Importantly, R-Probe is not a pixel-level autoencoder; it serves as a semantic consistency check between visual and language representations. We assess its validity and sensitivity through the following analyses. Semantic Dependency Verification. We examine whether reconstruction relies on the vision–language interface rather than visual input alone. As shown by the modality ablation study (Appendix D.2), reconstruction quality degrades substantially when visual regions are masked. However, providing textual descriptions reduces the reconstruction loss from 1.980 to 1.103 (MSE), despite the absence of visual signals. This result indicates that the decoder exploits semantic information from language tokens to guide reconstruction, confirming that R-Probe measures cross-modal alignment. Sensitivity to Feature Quality. We examine whether R-Probe reflects differences in visual representation quality. We evaluate optimization efficiency by the number of steps to reach MSE<0.75MSE<0.75 and report the final reconstruction loss, where 0.75 is an empirically chosen threshold indicating good reconstruction quality. As shown in Appendix D.3, detached multi-layer aggregation reaches the target loss faster (2158 → 1689 steps) and achieves a lower final error (0.698 vs. 0.724) than the baseline, indicating that R-Probe is sensitive to feature quality. Robustness and Correlation with Downstream Performance. We further test whether this sensitivity is consistent across architectures and aligned with downstream tasks. Across four MLLM backbones, the same configuration consistently reduces optimization steps and final reconstruction loss (Table 7), demonstrating robust and architecture-agnostic behavior. Overall, reconstruction loss rankings induced by R-Probe show a clear association with downstream performance within each backbone, (Appendix D.4), supporting its use as a predictive diagnostic for comparing model configurations under a fixed backbone. 6 Experiments: large-scale In this section, we evaluate Detached Skip-Links at scale. We first describe the experimental setup and benchmark protocol, and then present ablation studies, comparisons with state-of-the-art fusion methods, and evaluations across different ViT backbones. 6.1 Experimental Setup We adopt a two-stage training pipeline: (i) adapter pre-training (warm-up on the adapter only, then FFT) and (i) supervised fine-tuning (SFT). Unless otherwise specified, we use LLaMA-3.1-8B as the base LLM and a 300M–400M parameter Vision Transformer as the visual encoder. To evaluate scalability, we pre-train on 5M multimodal samples and further fine-tune on 2M task-specific samples. Full details on data sources, task composition, and hyperparameters are provided in Appendix E. As illustrated in Fig. 1, during adapter pre-training we freeze both the ViT encoder and the LLM, and optimize only the adapter. During FFT and SFT, we fine-tune the full model for OCR- and VQA-centric tasks using the same training recipe, with stage-specific learning rates and sequence lengths reported in Appendix E.3. 6.2 Benchmark Evaluation We evaluate on 22 benchmarks spanning four categories: STEM Puzzle, General, Alignment, and OCR, covering a broad range of multimodal reasoning and perception tasks. For compactness, the main text reports the average score of each group, while per-benchmark results are deferred to Appendix G. We use a customized VLMEvalKit pipeline; implementation details are included in Appendix F. 6.3 Ablation Studies Detachment configuration. In this section, we study how fusion density and gradient-flow control affect multi-layer visual feature fusion. We introduce two key hyperparameters: the sampling stride (S), which controls how densely intermediate ViT layers are selected, and the number of detached layers (D). Specifically, we extract feature maps every S blocks from shallow to deep. Let ℒ=ℓ1,ℓ2,…,ℓKL=\ _1, _2,…, _K\ denote the selected blocks ordered by depth, where ℓ1 _1 is the shallowest. We apply stop-gradient to the shallowest D layers ℓ1,…,ℓD\ _1,…, _D\, preventing gradients from the LLM objective from updating these layers through the skip-fusion path. Figure 6 visualizes the ablation results, where bubble size and color indicate performance relative to the baseline (yellow). Two clear findings emerge. (i) An intermediate fusion density is optimal: denser fusion with smaller strides (e.g., S=3S=3 or 44) consistently outperforms sparse fusion (e.g., S=12S=12), highlighting the benefit of incorporating multi-level visual features. (i) Shallow-only detachment is robust: Detaching shallow layers while keeping deeper layers trainable yields strong performance across fusion settings, whereas detaching deeper layers leads to instability. This suggests that our method is robust to detachment hyperparameters. Training with R-Probe as an Auxiliary Loss We optionally employ the R-Probe as a self-supervised auxiliary objective to impose a structural consistency constraint on visual tokens (Figure 4). While this enhances OCR performance by preserving fine-grained details, it introduces slight trade-offs in abstract reasoning. We attribute this to distributional bias from the OCR-centric auxiliary data (see Appendix D.1). Figure 6: Ablation study on feature sampling stride (S) and the number of detached layers (D). The chart visualizes the OCR performance (top) and the Average score across all benchmarks (bottom).Green nodes indicate improvement, while red nodes indicate degradation. 6.4 Comparison with State-of-the-Art Methods We compare our method with three representative multi-layer visual feature fusion methods: Dense Connector for MLLMs (DC) (Yao et al., 2024), Deepstack (Meng et al., 2024), and Multi-Layer Visual Feature Fusion (ML) (Lin et al., 2025). For fair comparison, all methods are trained from the same initialization on the same dataset, using identical settings. As shown in Table 1, our method achieves the strongest overall performance under matched training budgets. Following the official best settings of each baseline, we apply the detach operation to selected layers, which consistently improves performance and demonstrates its generalization. Specifically, we use DC with layer groups [1–12] and [12–23], DeepStack with Starting Layers=4, Interval=2, and N-layers=4, and ML with External Direct Fusion, applying detach to DC’s [1–12] group and ML’s last layer. Table 1: Performance Comparison (Categorized Averages). Full results in Table 9 Setting STEM General Align. OCR Overall PE-baseline 63.0 53.2 72.6 65.2 61.1 Ours (PE-best) 64.1 54.6 73.6 68.3 63.0 DC 63.2 54.0 72.5 66.7 62.0 DC-detached 64.2 54.4 72.8 67.6 62.6 ML 63.5 54.1 72.6 66.9 62.1 ML-detached 63.1 54.0 73.2 68.1 62.5 DeepStack 63.8 54.5 73.2 67.6 62.6 6.5 Experiments Across Different ViTs To evaluate generalizability, we test Detached Skip-Links across diverse Vision Transformer backbones with different architectures and pre-training objectives. Specifically, beyond the Perception Encoder used in ablations, we evaluate InternViT-300M (448px), AimV2-L (patch14-224), and SigLip2-So400M (patch14-384). For each ViT, we compare the baseline w/o our proposed Detached Skip-Links method. Experimental results (categorized averages) are illustrated in Table 2. Consistent performance improvements across all tested ViTs demonstrate that our method possesses broad adaptability to different visual encoding architectures. Table 2: Performance Across Different ViT Backbones (Categorized Averages). Full results in Table 10 Setting STEM General Align. OCR Overall PE-baseline 63.0 53.2 72.6 65.2 61.1 PE-ours 64.1 54.6 73.6 68.3 63.0 Δ +1.1 +1.4 +1.0 +3.1 +1.9 InternViT-baseline 58.7 49.0 64.0 60.3 56.2 InternViT-ours 61.1 50.8 71.4 62.2 58.7 Δ +2.4 +1.8 +7.4 +1.9 +2.5 AimV2L_L-baseline 60.5 51.0 69.5 63.0 58.8 AimV2L_L-ours 62.2 52.7 71.5 64.8 60.6 Δ +1.7 +1.7 +2.0 +1.8 +1.8 SigLip2so400_so400-baseline 57.1 47.4 69.6 58.1 55.0 SigLip2so400_so400-ours 59.7 49.8 69.4 60.6 57.3 Δ +2.6 +2.4 -0.2 +2.5 +2.3 7 Conclusions We study an optimization challenge in OCR-centric ViT–LLM fusion, where shallow visual features are underutilized or distorted during joint training. To address this issue, we introduce a detached gradient strategy for multi-layer fusion, together with R-Probe, a reconstruction-based diagnostic for assessing fine-grained visual information preservation and LLM-aligned consumability. Across large-scale experiments, our approach consistently improves OCR-centric performance while maintaining or improving general multimodal capabilities, and demonstrates robust transfer across diverse ViT backbones. Beyond empirical gains, our analysis highlights the importance of decoupling feature aggregation from gradient propagation when integrating heterogeneous visual representations. We hope these findings provide practical guidance for designing and diagnosing ViT–LLM bridging mechanisms, particularly in document-level and fine-grained multimodal understanding settings. Impact Statement This work focuses on improving the reliability of OCR-centric multimodal models through better optimization and diagnostic tools. All training and evaluation data are desensitized and sourced from publicly available or synthetic datasets, and do not involve personally identifiable information. While our methods may benefit document understanding applications, including large-scale information processing systems, they do not introduce new data collection mechanisms or user-facing decision-making components. We do not foresee significant negative societal impacts arising directly from this work. References J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1. A. Ajalloeian and S. U. Stich (2020) On the convergence of sgd with biased gradients. arXiv preprint arXiv:2008.00051. Cited by: §4.2. G. Alain and Y. Bengio (2016) Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644. Cited by: §1. J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, p. 23716–23736. Cited by: §2. D. Arpit, B. Kanuparthi, G. Kerg, N. R. Ke, I. Mitliagkas, and Y. Bengio (2018) H-detach: modifying the lstm gradient towards better optimization. arXiv preprint arXiv:1810.03023. Cited by: §2. S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. (2025) Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §1, §2. M. Bigverdi, Z. Luo, C. Hsieh, E. Shen, D. Chen, L. G. Shapiro, and R. Krishna (2025) Perception tokens enhance visual reasoning in multimodal language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 3836–3845. Cited by: §2. D. Bolya, P. Huang, P. Sun, J. H. Cho, A. Madotto, C. Wei, T. Ma, J. Zhi, J. Rajasegaran, H. Rasheed, et al. (2025) Perception encoder: the best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504.13181. Cited by: §1. C. Chen, Y. Guo, P. Zeng, J. Song, P. Di, H. Yu, and L. Gao (2026) From one-to-one to many-to-many: dynamic cross-layer injection for deep vision-language fusion. arXiv preprint arXiv:2601.10710. Cited by: §2. Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, et al. (2024a) Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271. Cited by: §2. Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024b) Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 24185–24198. Cited by: §1. C. Cui, T. Sun, S. Liang, T. Gao, Z. Zhang, J. Liu, X. Wang, C. Zhou, H. Liu, M. Lin, et al. (2025a) Paddleocr-vl: boosting multilingual document parsing via a 0.9 b ultra-compact vision-language model. arXiv preprint arXiv:2510.14528. Cited by: §2. C. Cui, T. Sun, M. Lin, T. Gao, Y. Zhang, J. Liu, X. Wang, Z. Zhang, C. Zhou, H. Liu, et al. (2025b) Paddleocr 3.0 technical report. arXiv preprint arXiv:2507.05595. Cited by: §2. T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski (2023) Vision transformers need registers. arXiv preprint arXiv:2309.16588. Cited by: Appendix A, §3.1. D. Dong, M. Zheng, D. Xu, B. Zhuang, W. Zhang, C. Luo, H. Wang, Z. Zhao, J. Li, Y. Li, et al. (2025) Qianfan-vl: domain-enhanced universal vision-language models. arXiv preprint arXiv:2509.18189. Cited by: §2. A. Dosovitskiy (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Cited by: §1. E. Fini, M. Shukor, X. Li, P. Dufter, M. Klein, D. Haldimann, S. Aitharaju, V. G. T. da Costa, L. Béthune, Z. Gan, et al. (2025) Multimodal autoregressive pre-training of large vision encoders. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 9641–9654. Cited by: §1. L. Fu, Z. Kuang, J. Song, M. Huang, B. Yang, Y. Li, L. Zhu, Q. Luo, X. Wang, H. Lu, et al. (2024) Ocrbench v2: an improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.00321. Cited by: §1. Y. Gao, J. Liu, Z. Xu, J. Zhang, K. Li, R. Ji, and C. Shen (2022) Pyramidclip: hierarchical feature alignment for vision-language model pretraining. Advances in neural information processing systems 35, p. 35959–35970. Cited by: §2. Z. Guo, R. Xu, Y. Yao, J. Cui, Z. Ni, C. Ge, T. Chua, Z. Liu, and G. Huang (2024) Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision, p. 390–406. Cited by: §1, §2. Z. He, C. Zhang, Z. Wu, Z. Chen, Y. Zhan, Y. Li, Z. Zhang, X. Wang, and M. Qiu (2025) Seeing is believing? mitigating ocr hallucinations in multimodal large language models. arXiv preprint arXiv:2506.20168. Cited by: §1, §5. Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. Yu (2024) Opera: alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13418–13427. Cited by: §5. A. Kanade and T. Ganu (2025) Do you see me: a multidimensional benchmark for evaluating visual perception in multimodal llms. arXiv preprint arXiv:2506.02022. Cited by: §1. O. F. Kar, A. Tonioni, P. Poklukar, A. Kulshrestha, A. Zamir, and F. Tombari (2024) Brave: broadening the visual encoding of vision-language models. In European Conference on Computer Vision, p. 113–132. Cited by: §2. J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, p. 19730–19742. Cited by: §2. Z. Li, B. Yang, Q. Liu, Z. Ma, S. Zhang, J. Yang, Y. Sun, Y. Liu, and X. Bai (2024) Monkey: image resolution and text label are important things for large multi-modal models. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 26763–26773. Cited by: §2. V. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou (2022) Mind the gap: understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems 35, p. 17612–17625. Cited by: §5.1. J. Lin, H. Chen, Y. Fan, Y. Fan, X. Jin, H. Su, J. Fu, and X. Shen (2025) Multi-layer visual feature fusion in multimodal llms: methods, analysis, and best practices. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 4156–4166. Cited by: §1, §2, §6.4. J. Liu, W. Zeng, X. Zhang, Y. Wang, Z. Shan, and J. He (2025a) On the perception bottleneck of vlms for chart understanding. p. 10829–10841. External Links: Document Cited by: §1. P. Liu, H. Shen, C. Fang, Z. Sun, J. Liao, and T. Zhao (2025b) VLM-fo1: bridging the gap between high-level reasoning and fine-grained perception in vlms. arXiv preprint arXiv:2509.25916. Cited by: §2. Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. Liu, L. Jin, and X. Bai (2024) Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences 67 (12), p. 220102. Cited by: §1. H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yang, et al. (2024) Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525. Cited by: §2. L. Meng, J. Yang, R. Tian, X. Dai, Z. Wu, J. Gao, and Y. Jiang (2024) Deepstack: deeply stacking visual tokens is surprisingly simple and effective for lmms. Advances in Neural Information Processing Systems 37, p. 23464–23487. Cited by: §2, §6.4. A. Pagh, R. Pagh, and M. Ruzic (2007) Linear probing with constant independence. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing, p. 318–327. Cited by: §1. K. Pan, S. Tang, J. Li, Z. Fan, W. Chow, S. Yan, T. Chua, Y. Zhuang, and H. Zhang (2024) Auto-encoding morph-tokens for multimodal llm. arXiv preprint arXiv:2405.01926. Cited by: §2. M. Shi, F. Liu, S. Wang, S. Liao, S. Radhakrishnan, Y. Zhao, D. Huang, H. Yin, K. Sapra, Y. Yacoob, et al. (2024) Eagle: exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998. Cited by: §2. G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. (2023) Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: §1. G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. (2025a) Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Cited by: §2. H. V. Team, P. Lyu, X. Wan, G. Li, S. Peng, W. Wang, L. Wu, H. Shen, Y. Zhou, C. Tang, et al. (2025b) HunyuanOCR technical report. arXiv preprint arXiv:2511.19575. Cited by: §2. S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie (2024) Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9568–9578. Cited by: §1. M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, et al. (2025) Siglip 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786. Cited by: §1. W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao, et al. (2025) Internvl3. 5: advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265. Cited by: §1, §2. H. Wei, L. Kong, J. Chen, L. Zhao, Z. Ge, J. Yang, J. Sun, C. Han, and X. Zhang (2024) Vary: scaling up the vision vocabulary for large vision-language model. In European Conference on Computer Vision, p. 408–424. Cited by: §1, §5. S. Wu, H. Fei, X. Li, J. Ji, H. Zhang, T. Chua, and S. Yan (2024a) Towards semantic equivalence of tokenization in multimodal llm. arXiv preprint arXiv:2406.05127. Cited by: §2. Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao, Y. Ma, C. Wu, B. Wang, et al. (2024b) Deepseek-vl2: mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302. Cited by: §2, §2. H. Yao, W. Wu, T. Yang, Y. Song, M. Zhang, H. Feng, Y. Sun, Z. Li, W. Ouyang, and J. Wang (2024) Dense connector for mllms. Advances in Neural Information Processing Systems 37, p. 33108–33140. Cited by: §1, §2, §6.4. J. Ye, A. Hu, H. Xu, Q. Ye, M. Yan, G. Xu, C. Li, J. Tian, Q. Qian, J. Zhang, et al. (2023) Ureader: universal ocr-free visually-situated language understanding with multimodal large language model. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 2841–2858. Cited by: §2. T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn (2020) Gradient surgery for multi-task learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. External Links: ISBN 9781713829546 Cited by: §2. Y. Yu, M. Liao, J. Wu, Y. Liao, X. Zheng, and W. Zeng (2024) TextHawk: exploring efficient fine-grained perception of multimodal large language models. CoRR abs/2404.09204. Cited by: §2. Y. Zhang, Y. Liu, Z. Guo, Y. Zhang, X. Yang, X. Zhang, C. Chen, J. Song, B. Zheng, Y. Yao, et al. (2024) LLaVA-uhd v2: an mllm integrating high-resolution semantic pyramid via hierarchical window transformer. arXiv preprint arXiv:2412.13871. Cited by: §2. Appendix A Attention Visualization Details We visualize shallow-layer attention maps following the protocol of (Darcet et al., 2023). Given an input image, we forward it through the vision encoder and extract the self-attention weights from a shallow transformer block. Here, we use the 4th transformer block as a representative shallow layer. We use the [CLS] token attention to patch tokens: we average attention weights over all heads and take the row corresponding to the [CLS] query, excluding the [CLS]→[CLS] entry. The resulting patch-level importance scores are reshaped into a 2D grid according to the ViT patch layout. For visualization, we apply min–max normalization to map the scores to [0,1][0,1] and upsample the grid to the input image resolution using nearest-neighbor interpolation, producing a discrete patch-aligned heatmap. Unless otherwise specified, we enable the [CLS]-based attention mode by default. We repeat the same procedure for three checkpoints and concatenate the resulting heatmaps with the original image into a single panel for side-by-side comparison (original ∥ ori || baseline || detached), using a white background and fixed spacing. Appendix B Additional Proofs for Section 4 B.1 Proof of Proposition 4.1 Proof. We analyze the gap between the optimal risks of the main-only path and the fused path. Recall that under the squared loss ℛ(f)=[(Y−f(X))2]R(f)=E[(Y-f(X))^2], the Bayes optimal predictor is the conditional expectation of the target given the inputs. Therefore, the minimum achievable risks for the two scenarios are: ℛmain∗ ^*_main =infFℛ(F(A))=[(Y−[Y∣A])2], = _FR(F(A))=E [(Y-E[Y A])^2 ], ℛfuse∗ ^*_fuse =infGℛ(G([A;B]))=[(Y−[Y∣A,B])2]. = _GR(G([A;B]))=E [(Y-E[Y A,B])^2 ]. We can decompose the risk of the main branch by adding and subtracting the term [Y∣A,B]E[Y A,B] inside the quadratic expectation: ℛmain∗ ^*_main =[((Y−[Y∣A,B])+([Y∣A,B]−[Y∣A]))2] =E [ ((Y-E[Y A,B])+(E[Y A,B]-E[Y A]) )^2 ] =[(Y−[Y∣A,B])2]⏟=ℛfuse∗+[([Y∣A,B]−[Y∣A])2]⏟Gap term = E [(Y-E[Y A,B])^2 ]_=R^*_fuse+ E [(E[Y A,B]-E[Y A])^2 ]_Gap term +2[(Y−[Y∣A,B])⋅([Y∣A,B]−[Y∣A])]⏟Cross term. +2 E [(Y-E[Y A,B])·(E[Y A,B]-E[Y A]) ]_Cross term. (9) Now, we show that the cross term vanishes. By the Law of Iterated Expectations, conditioning on A,BA,B inside the expectation: Cross term =A,B[[Y−[Y∣A,B]∣A,B]⋅([Y∣A,B]−[Y∣A])] =E_A,B [E [Y-E[Y A,B] A,B ]·(E[Y A,B]-E[Y A]) ] =A,B[([Y∣A,B]−[Y∣A,B]⏟0)⋅(…)]=0. =E_A,B [( E[Y A,B]-E[Y A,B]_0)·(…) ]=0. Substituting this back into Eq. (9), we obtain the risk difference: ℛmain∗−ℛfuse∗=[([Y∣A,B]−[Y∣A])2].R^*_main-R^*_fuse=E [(E[Y A,B]-E[Y A])^2 ]. Using the definition of conditional variance Var(Z∣A)=[Z2∣A]−([Z∣A])2Var(Z A)=E[Z^2 A]-(E[Z A])^2, and setting Z=[Y∣A,B]Z=E[Y A,B], we observe that: [Z∣A]=[[Y∣A,B]∣A]=[Y∣A].E[Z A]=E[E[Y A,B] A]=E[Y A]. Thus, the gap term is exactly the expected conditional variance of the predictor: [([Y∣A,B]−[Y∣A])2]=A[Var([Y∣A,B]|A)].E [(E[Y A,B]-E[Y A])^2 ]=E_A [Var (E[Y A,B]\, |\,A ) ]. This confirms Eq. (2). Since the squared term is always non-negative, ℛmain∗≥ℛfuse∗R^*_main ^*_fuse. Condition for Equality. The gap is zero if and only if [Y∣A,B]=[Y∣A]E[Y A,B]=E[Y A] almost surely. This occurs when Y is conditionally independent of B given A (i.e., Y⟂B∣AY B A). In our context, this would imply that the deep feature A has preserved all information from B relevant to Y, meaning no information bottleneck exists. Conversely, if the main path attenuates relevant information (as discussed in Section 4.1), strict inequality holds. □ B.2 Derivation of Eq. (4) Let full=main+skipg_full=g^main+g^skip with means =[main]m=E[g^main], =[skip]s=E[g^skip]. Write main=+ϵmg^main=m+ ε_m and skip=+ϵsg^skip=s+ ε_s, where [ϵm]=[ϵs]=0E[ ε_m]=E[ ε_s]=0. Then ‖full‖2=‖+‖2+‖ϵm+ϵs‖2=‖+‖2+‖ϵm‖2+‖ϵs‖2+2⟨ϵm,ϵs⟩.E\|g_full\|^2=\|m+s\|^2+E\| ε_m+ ε_s\|^2=\|m+s\|^2+E\| ε_m\|^2+E\| ε_s\|^2+2E ε_m, ε_s . Using ‖ϵ‖2=tr(Cov(⋅))E\| ε\|^2=tr(Cov(·)) and 2⟨ϵm,ϵs⟩=tr(Σms+Σms⊤)2E ε_m, ε_s =tr( _ms+ _ms ) yields Eq. (4). □ B.3 One-step smoothness bound Assume ℒL is L-smooth, i.e., ℒ(θ′)≤ℒ(θ)+⟨∇ℒ(θ),θ′−θ⟩+L2‖θ′−θ‖2L(θ ) (θ)+ (θ),θ -θ + L2\|θ -θ\|^2. Substitute θ′=θ−γθ =θ- and take expectation to obtain Eq. (6). □ B.4 Proof of Proposition 4.3 Apply Eq. (6) to fullg_full and detachg_detach and subtract: ℒ(θ−γfull)−ℒ(θ−γdetach)≤−γ⟨∇ℒ(θ),⟩+Lγ22(‖full‖2−‖detach‖2),EL(θ- _full)-EL(θ- _detach)≤-γ (θ),s + Lγ^22 (E\|g_full\|^2-E\|g_detach\|^2 ), where =[skip]s=E[g^skip]. Rearranging gives the sufficient condition (7) for detachment to yield no worse expected one-step loss. □ Appendix C Details on Gradient Statistics Measurement This section details the methodology used to separate and analyze the skip-path and main-path gradients presented in Figure 3(b). C.1 Online Measurement Strategy In standard backpropagation, gradients from the skip and main branches are aggregated automatically. To decouple them for analysis without disrupting the training graph, we employed a Double Backward strategy with random state preservation: 1. State Checkpointing: We save the current state of the Random Number Generator (RNG) to ensure consistent dropout masks and stochastic operations. 2. Main-Path Isolation: We perform a forward pass where the skip connection is detached (y=ℱ(x)+x.detach()y=F(x)+x.detach()). The subsequent backward pass yields gmain=∇θℒdetachg^main= _θL_detach. 3. State Restoration: The RNG state is restored. 4. Full Gradient Computation: A standard forward and backward pass is executed to obtain the total gradient gfullg^full. 5. Decomposition: The skip-path gradient is derived via subtraction: gskip=gfull−gmaing^skip=g^full-g^main. C.2 Metric Definitions To empirically verify the statistical assumptions in Section 4, we compute the following window-based gradient statistics. Signal–Noise Decomposition. For a random gradient vector g, its second moment admits the standard decomposition [‖g‖2]=‖[g]‖2+tr(Var(g)).E\! [\|g\|^2 ]=\|E[g]\|^2+tr\! (Var(g) ). (10) In practice, expectations are approximated using a sliding window of length K. Specifically, given gradients gt−K+1,…,gt\g_t-K+1,…,g_t\, we define the window mean g^t=1K∑i=1Kgt−K+i, g_t= 1K _i=1^Kg_t-K+i, (11) and the corresponding empirical variance trace tr(Σ^t)=1K∑i=1K‖gt−K+i‖2−‖g^t‖2.tr( _t)= 1K _i=1^K\|g_t-K+i\|^2-\| g_t\|^2. (12) Accordingly, the instantaneous squared norm ‖gt‖2\|g_t\|^2 serves as a proxy for the total gradient energy, while ‖g^t‖2\| g_t\|^2 approximates the signal component. The pronounced gap between these quantities in Figure 3(a) indicates that gradient variance dominates the optimization dynamics in the early training stage. Cross-Covariance Strength (δ). To quantify the relative magnitude of the cross-covariance term between the main and skip branches, we consider windowed gradients gimain,giskipi=1K\g^main_i,g^skip_i\_i=1^K. Let m^t=1K∑i=1Kgt−K+imain,s^t=1K∑i=1Kgt−K+iskip, m_t= 1K _i=1^Kg^main_t-K+i, s_t= 1K _i=1^Kg^skip_t-K+i, denote their respective window means. We estimate the trace of the symmetric cross-covariance as |tr(Σms+Σms⊤)|≈ 2|1K∑i=1K⟨gt−K+imain,gt−K+iskip⟩−⟨m^t,s^t⟩|. |tr( _ms+ _ms ) |\;≈\;2 | 1K _i=1^K g^main_t-K+i,g^skip_t-K+i - m_t, s_t |. (13) Similarly, the variance trace of the skip branch is estimated by tr(Σs)≈1K∑i=1K‖gt−K+iskip‖2−‖s^t‖2.tr( _s)\;≈\; 1K _i=1^K\|g^skip_t-K+i\|^2-\| s_t\|^2. (14) We then define the empirical cross-covariance ratio δt=|tr(Σms+Σms⊤)|tr(Σs)+ϵ,ϵ=10−12, _t= |tr( _ms+ _ms ) |tr( _s)+ε, ε=10^-12, (15) which serves as a practical proxy for the constant δ in Assumption 4.2. As shown in Figure 3(d), δt _t remains small throughout training, indicating that the cross-covariance is negligible relative to the variance of the skip-path gradients. C.3 Detailed Empirical Analysis of Gradient Statistics In Section 3.3, we summarized the gradient dynamics of the first skip-link block. Here, we provide a detailed interpretation of the empirical observations shown in Figure 3(b). • Variance Dominance and Phase Transition (Fig. 3(a)): During the early stage of training (approximately the first 300 steps), the gradient norm of the skip branch, ‖gskip‖\|g^skip\|, is consistently larger than that of the main branch, ‖gmain‖\|g^main\|. More importantly, the instantaneous skip-path norm significantly exceeds the squared norm of its short-horizon moving average, i.e., ‖gskip‖2≫‖g^skip‖2\|g^skip\|^2 \| g^skip\|^2, indicating that the skip-path gradients are dominated by high-variance fluctuations rather than a coherent mean signal. As training progresses, we observe a clear regime transition in which ‖gmain‖\|g^main\| surpasses ‖gskip‖\|g^skip\|, marking the shift from initialization-driven dynamics to effective feature learning. • Approximate Orthogonality (Fig. 3(c)): The cosine similarity between gskipg^skip and gmaing^main fluctuates tightly around zero throughout training. This behavior empirically supports the weak mean-alignment assumption, suggesting that the stochastic noise introduced by the skip branch is approximately orthogonal to the effective gradient direction of the main branch. • Negligible Cross-Covariance Cancellation (Fig. 3(d)): The empirical cross-covariance ratio δt _t remains consistently small (typically below 0.10.1) over the entire training trajectory. This observation indicates that the interaction between the main and skip branches is insufficient to offset the variance of the skip-path gradients. Consequently, the cross-covariance term plays a negligible role in practice, justifying its omission in our theoretical analysis. C.4 Robustness to learning rate. LR ttranst_trans median(cos)( ) median(δ)(δ) 8e-5 400 -0.0011 0.0577 2e-5 900 -0.0005 0.0624 5e-6 / 0.0008 0.0315 Table 3: Learning-rate robustness summary of early-phase gradient statistics We further evaluate the robustness of our gradient-statistics analysis under different learning rates, while keeping the model architecture and data pipeline fixed. Unless otherwise specified, all main experiments are conducted with a learning rate of 1×10−51× 10^-5, with global batch size of 128, using Adam as optimizer. Across a wide range of learning rates, we observe qualitatively consistent behaviors during the early phase of training: (i) variance dominance in the skip-path gradients, evidenced by the instantaneous norm ‖gtskip‖\|g_t^skip\| being substantially larger than the squared norm of its short-horizon window mean, and typically comparable to or larger than ‖gtmain‖\|g_t^main\|; (i) near-orthogonality between the main and skip branches, with cosine similarity fluctuating tightly around zero; and (i) weak cross-covariance cancellation, with the empirical ratio δt _t remaining small (e.g., below 0.10.1 in our measurements). The primary effect of changing the learning rate is a rescaling of the time axis, manifested as a shift in the step at which the training dynamics transition between regimes. Using a reproducible definition of the transition step ttranst_trans—defined as the first step after which a running-window median of ‖gskip‖/‖gmain‖\|g^skip\|/\|g^main\| remains below 11 for consecutive windows—we find that larger learning rates generally induce earlier transitions, while smaller learning rates delay this transition (Table 3). At sufficiently large learning rates, we observe a qualitative inversion of the relative gradient magnitudes, where ‖gmain‖\|g^main\| can dominate ‖gskip‖\|g^skip\| even in the initial training steps. Notably, this inversion does not contradict our analysis, as Assumption 4.2 is intended as a local characterization of the early optimization regime under standard training settings, rather than a global statement valid for arbitrarily large step sizes. Appendix D R-probe Results D.1 Training with R-Probe as Auxiliary Loss To investigate the cross-modal interaction between the R-probe reconstruction mechanism and Optical Character Recognition (OCR) capabilities, we conducted a specialized ablation study. We constructed a dataset with bounding box annotations for text regions across various domains, including scene text, documents, and charts. By training the R-probe jointly with Multimodal Large Language Model (MLLM) OCR tasks, we aimed to determine if visual reconstruction objectives could synergize with text recognition objectives. As shown in Table 4, we observe a mutual enhancement between the two tasks, particularly for benchmarks that rely heavily on pure visual perception and text recognition. For instance, OCRBench scores improved from 714.0 to 721.0, and DocVQA improved from 71.4 to 72.1. However, for tasks requiring complex reasoning over visual elements (e.g., CharXiv_RQ), we observed a slight performance trade-off, where scores decreased from 43.9 to 42.8. This suggests that while R-probe significantly strengthens low-level visual grounding and recognition features, it may introduce a slight interference in high-level semantic reasoning pathways when trained conjointly. Table 4: Comparison of OCR capabilities with and without R-probe. “Baseline+R-probe” indicates the model jointly trained with the R-probe reconstruction objective. Note the improvement in pure OCR tasks (OCRBench, DocVQA) versus the trade-off in reasoning-heavy tasks (CharXiv_RQ). Method OCRBench DocVQA TextVQA ChartQA OCRVQA AI2D CharXiv_DQ CharXiv_RQ Baseline 714.0 71.4 68.7 70.0 55.0 72.2 69.7 43.9 Baseline + R-probe 721.0 72.1 68.9 70.1 55.4 69.2 68.3 42.8 D.2 Ablation Study of R-probe Modalities To quantify the information contribution of language tokens versus image tokens in the reconstruction process, we designed a set of ablation experiments involving masking image regions and removing language tokens (Table 5). In the “Masked” setting, the bounding box regions in the input image are replaced with black pixels (R=G=B=0R=G=B=0). In the “w/o Lang” setting, all language tokens are replaced with a special [UNK] token. The results demonstrate that the presence of language tokens significantly aids visual reconstruction. Comparing Setting 1 (Masked, w/o Lang) and Setting 2 (Masked, w/ Lang), the reconstruction loss drops from 1.980 to 1.103, indicating that the language description provides crucial cues for reconstructing the missing visual information. This provides evidence that the R-probe Transformer possesses a degree of semantic understanding, effectively bridging the modality gap between text and image. Table 5: Ablation study on input modalities for the R-probe. Lower Final Loss indicates better reconstruction quality. ID Setting Final Loss (MSE) 1 Masked Image, w/o Language 1.980 2 Masked Image, w/ Language 1.103 3 Full Image, w/o Language 0.757 4 Full Image, w/ Language 0.724 D.3 Convergence Analysis and Architecture Configuration We further analyzed the convergence behavior and reconstruction quality of the R-probe under different layer configurations and backbone architectures. Convergence is defined as the number of steps required for the per-token MSE loss to drop below 0.75, representing a visually coherent reconstruction threshold. Table 6 presents the impact of feature selection from different ViT layers. The “Detached” setting, where gradients are not backpropagated to the main ViT backbone, combined with multi-layer feature aggregation, yields the fastest convergence (1689 steps) and the lowest final loss (0.698). This suggests that aggregating hierarchical features from multiple depths provides a richer representation for reconstruction than using only superficial or deep layers alone. Table 6: Impact of layer selection and gradient detachment on R-probe convergence and performance. ”Steps” denotes steps to reach MSE << 0.75. ID Configuration Steps (<<0.75) Final Loss 1 Baseline 2158 0.724 2 Ours Stride=12 (w/o Detach) 1935 0.719 3 Ours Stride=12 (Detached) 1956 0.716 4 Ours Stride=6 (w/o Detach) 1834 0.709 5 Ours Stride=6 (Detached) 1689 0.698 D.4 Correlation Between R-Probe Reconstruction Loss and Downstream Performance As discussed in the main text, R-Probe is designed not only to provide stable measurements across architectures, but also to reflect downstream perceptual performance. To support this claim, we analyze the relationship between R-Probe reconstruction loss and downstream benchmark results. We first report the robustness of the optimized R-Probe configuration across different MLLM backbones (AimV2, InternViT, SigLip2). As shown in Table 7, the detached multi-layer configuration consistently reduces both the required optimization steps and the final reconstruction loss across all tested architectures, indicating architecture-agnostic behavior. Table 7: Performance of R-probe across different MLLM backbones. ”Ours” refers to the best detached multi-layer configuration (Setting ID=5 from Table 6). Backbone Setting Steps (<<0.75) Final Loss PE Original 2158 0.724 Ours 1689 0.698 AimV2 Original 2301 0.741 Ours 1889 0.706 InternViT Original 2189 0.735 Ours 1835 0.703 SigLip2 Original 2535 0.748 Ours 2089 0.721 Building on the robustness results, we examine whether reconstruction loss aligns with downstream task performance under a fixed backbone. Within each backbone, configurations with lower reconstruction loss consistently achieve higher OCR and general benchmark scores. This consistent within-backbone ranking indicates that R-Probe serves as a predictive diagnostic for comparing perceptual quality across model variants sharing the same architecture. Table 8: Relationship between R-Probe reconstruction loss and downstream benchmark performance within each backbone. Backbone Setting Recon. Loss ↓ OCR Score ↑ General Score ↑ PE Original 0.724 65.2 61.1 Ours Stride=12 (w/o Detach) 0.719 66.1 61.1 Ours Stride=12 (Detached) 0.716 67.0 61.7 Ours Stride=6 (w/o Detach) 0.709 66.4 61.8 Ours Stride=6 (Detached) 0.698 68.3 63.0 AimV2 Original 0.741 63.0 58.8 Ours 0.706 64.8 60.6 InternViT Original 0.735 60.3 56.2 Ours 0.703 62.2 58.7 SigLip2 Original 0.748 58.1 55.0 Ours 0.721 60.6 57.3 Appendix E Training Details E.1 Overview For all model configurations, we adopt a two-stage training paradigm consisting of (i) adapter pre-training and (i) supervised fine-tuning (SFT). The training data is sourced from a mixture of internal collections and publicly available datasets.111Due to licensing and privacy constraints, the internal portion cannot be released. We provide the composition, task taxonomy, and full evaluation protocol to facilitate reproducibility of trends. Unless otherwise specified, we use LLaMA3.1-8B as the base language model and a ViT visual encoder with 300M–400M parameters. We sample 5M multimodal examples for adapter pre-training and 2M examples for SFT. Both stages share the same core optimization recipe for consistency, with minor stage-specific adjustments described in Appendix E.4. E.2 Data Composition Adapter pre-training. We employ a warm-up strategy on the first 10% of pre-training data, mainly consisting of high-quality image-caption pairs and basic visual question-answering (VQA) tasks. The remaining 90% focuses on General Knowledge Injection, which draws from multiple public datasets, including InternVL-Chat-V1-2-SFT-Data, GRIT, LLaVAR, A-OKVQA, geo170K, LNQA, MAVIS, Screen2Words, and MMDU. The category composition of this subset is shown in Figure 7(a). Supervised fine-tuning. For the SFT stage, we curate a task-specific dataset with a balanced distribution of task types (Figure 7(b)), covering OCR, document understanding, captioning, math-centric multimodal reasoning, and pure-text instruction-following. 43%40%17%OCR & OCRQACaptionOthers (a) Pre-training: General Knowledge Injection DistributionOthers: Grounding; Chart Understanding; Doc Understanding; GUI; Knowledge; Math; VQA 34%33%14%10%9%OCRDoc UnderstandingCaptionMathPure Text (b) SFT: Task Distribution Figure 7: Data distribution of the pre-training and SFT stages. E.3 Optimization and Systems We use the same optimizer and training infrastructure for both stages unless stated otherwise. The common settings are: • Optimizer: AdamW • Weight decay: 0.05 • Warmup ratio: 0.03 • LR schedule: Cosine decay • Precision and acceleration: bf16 training, DeepSpeed ZeRO-3, FlashAttention-2 • Global batch size: 256 E.4 Stage-specific Settings Adapter pre-training. We freeze the parameters of the ViT and the LLM, and optimize only the adapter modules. We set the learning rate to 2×10−52× 10^-5 and use a sequence length of 16k. Supervised fine-tuning (SFT). For OCR- and VQA-centric tasks, we perform full-parameter fine-tuning following the same training recipe, with the following adjustments: we set the learning rate to 1×10−51× 10^-5 and use a sequence length of 8k. Appendix F Benchmarks and Evaluation Protocol F.1 Benchmark Suite We evaluate our models on a suite of 22 benchmarks using a customized evaluation pipeline based on VLMEvalKit. For systematic analysis, we group all benchmarks into four categories: STEM Puzzle. MMMUval_val, MathVistamini_mini, ScienceQATEST_TEST, ScienceQAVAL_VAL. General. A-BenchVAL_VAL, CCBench, SEEDBenchIMG_IMG, SEEDBench2Plus_Plus, MMVet, MMStar, BLINK, RealWorldQA. Alignment. HallusionBench, POPE. OCR. DocVQAtest_test, AI2Dtest_test, ChartQAtest_test, OCRBenchdiv10_div10, CharXivDQ_DQ, CharXivRQ_RQ, OCRVQAtestscore_testscore, TextVQAval_val. F.2 Evaluation Pipeline We adopt an automatic evaluation pipeline built upon VLMEvalKit, with minor adaptations to match our model I/O format and to unify prompting templates across benchmarks. Unless a benchmark provides an official evaluation server, we follow the official released splits and compute the corresponding metrics locally. Prompting and formatting. We use a unified prompt wrapper with a task-agnostic system instruction and a single-turn user query containing the image and the benchmark-specific question. For multiple-choice benchmarks, we format the candidate options verbatim and instruct the model to output the option letter only. For open-ended benchmarks, we instruct the model to output a concise final answer without additional explanations unless the benchmark explicitly requires rationales. All prompts and answer parsers used in our evaluation are included in the supplementary code release. Decoding. We use greedy decoding (temperature =0=0) by default to reduce evaluation variance. When a benchmark requires longer generations (e.g., certain reasoning-style QA), we increase the maximum generation length accordingly while keeping other decoding parameters unchanged. We apply no external tools (e.g., OCR engines, retrieval, or calculators) during evaluation unless the benchmark protocol explicitly assumes them. F.3 Per-benchmark Results For completeness, we provide the full per-benchmark scores of all compared methods in Appendix G. These tables include (i) the raw score on each of the 22 benchmarks, (i) the category-level macro averages reported in the main text, and (i) the final overall average aggregated across all 22 benchmarks (rounded to one decimal place), and (iv) any benchmark-specific evaluation notes (e.g., answer normalization rules). Appendix G Full Results G.1 Comparison We provide detailed per-benchmark results to complement the category-level averages in Table 9. The table shows that gradient detachment consistently benefits OCR-centric benchmarks while preserving general reasoning performance. Table 9: Detailed performance comparison across all benchmarks. We report results for PE-baseline, our proposed Ours (PE-best), and other fusion strategies including DenseConnector (DC), Multi-Layer Fusion (ML), and DeepStack. For DC and ML, we also show results with our detached gradient strategy applied (-detached). All scores are rounded to one decimal place. Benchmark PE-baseline Ours DC DC-detached ML ML-detached DeepStack STEM Puzzle !10 STEM Puzzle Avg 63.0 64.1 63.2 64.2 63.5 63.1 63.8 MMMUval_val 41.0 42.8 42.3 43.3 42.6 42.0 43.0 MathVistamini_mini 43.3 45.2 44.3 45.1 44.9 44.4 44.8 ScienceQATEST_TEST 83.9 83.9 82.9 84.2 83.0 82.8 83.6 ScienceQAVAL_VAL 83.8 84.7 83.5 83.9 83.4 83.2 84.0 General !10 General Avg 53.2 54.6 54.0 54.4 54.1 54.0 54.5 A-BenchVAL_VAL 72.2 72.6 72.3 72.6 72.0 72.4 72.9 CCBench 38.6 39.4 39.0 39.4 38.2 38.3 38.9 SEEDBenchIMG_IMG 71.3 72.5 72.1 72.0 72.3 72.0 72.0 SEEDBench2Plus_Plus 59.4 60.2 59.5 59.9 60.0 59.8 60.1 MMVet 40.8 42.2 41.0 42.2 41.8 41.2 42.0 MMStar 46.7 46.7 46.9 46.1 46.9 47.0 46.9 BLINK 43.7 47.3 46.2 46.8 47.1 47.0 47.2 RealWorldQA 52.7 56.1 55.1 56.1 54.3 54.6 56.0 Alignment !10 Alignment Avg 72.6 73.6 72.5 72.8 72.6 73.2 73.2 HallusionBench 59.0 60.4 59.0 59.0 58.9 59.8 59.6 POPE 86.2 86.9 86.0 86.5 86.2 86.6 86.7 OCR !10 OCR Avg 65.2 68.3 66.7 67.6 66.9 68.1 67.6 DocVQAtest_test 71.4 73.2 73.0 72.9 72.6 73.1 72.9 AI2Dtest_test 71.9 73.3 73.6 73.8 73.5 73.2 73.6 ChartQAtest_test 70.0 74.2 72.3 73.8 72.1 72.2 73.8 OCRBench 714 731 701 711 723 735 718 CharXivDQ_DQ 69.7 73.0 70.3 71.2 72.0 72.3 70.9 CharXivRQ_RQ 43.9 45.2 44.7 45.3 45.2 45.6 45.0 OCRVQAtestscore_testscore 55.0 62.8 58.9 61.3 59.4 63.3 62.8 TextVQAval_val 68.7 71.3 70.3 71.1 68.0 71.3 69.8 !10 Overall Avg 61.1 63.0 62.0 62.6 62.1 62.6 62.7 G.2 Eval on Different ViTs We report results across four ViT backbones to assess architectural generality in Table 10. Consistent gains across backbones indicate that Detached Skip-Links are robust to different visual encoders. Table 10: Detailed ablation results across different ViT backbones: PE, InternViT, AimV2, and SigLip2. For each backbone, we compare the Baseline with our proposed Detached Skip-Links method (Ours). All scores are rounded to one decimal place. Benchmark PE InternViT AimV2 SigLip Baseline Ours Baseline Ours Baseline Ours Baseline Ours STEM Puzzle !10 STEM Puzzle Avg 63.0 64.1 58.7 61.1 60.5 62.2 57.1 59.7 MMMUval_val 41.0 42.8 39.0 40.7 40.2 41.5 37.5 40.8 MathVistamini_mini 43.3 45.2 38.6 40.3 40.5 42.1 37.8 39.2 ScienceQATEST_TEST 83.9 83.9 78.0 82.1 80.5 82.5 76.3 79.4 ScienceQAVAL_VAL 83.8 84.7 79.0 81.3 80.8 82.9 76.8 79.3 General !10 General Avg 53.2 54.6 49.0 50.8 51.0 52.7 47.4 49.8 A-BenchVAL_VAL 72.2 72.6 69.6 71.3 70.8 71.9 62.5 65.9 CCBench 38.6 39.4 24.9 27.3 33.5 36.5 27.8 28.6 SEEDBenchIMG_IMG 71.3 72.5 67.7 69.5 69.5 70.8 67.2 69.1 SEEDBench2Plus_Plus 59.4 60.2 61.5 61.7 58.5 60.1 57.7 59.6 MMVet 40.8 42.2 34.9 35.3 37.6 39.4 35.8 37.6 MMStar 46.7 46.7 42.9 46.2 44.5 46.5 39.9 43.9 BLINK 43.7 47.3 39.2 40.8 41.5 42.8 37.1 41.6 RealWorldQA 52.7 56.1 51.1 54.4 52.5 53.8 50.7 52.6 Alignment !10 Alignment Avg 72.6 73.6 64.0 71.4 69.5 71.5 69.6 69.4 HallusionBench 59.0 60.4 48.3 57.9 54.5 57.2 50.7 50.4 POPE 86.2 86.9 79.6 84.9 84.5 85.8 88.5 88.4 OCR !10 OCR Avg 65.2 68.3 60.3 62.2 63.0 64.8 58.1 60.6 DocVQAtest_test 71.4 73.2 71.4 73.2 71.8 73.2 65.8 68.4 AI2Dtest_test 71.9 73.3 68.6 71.0 71.2 72.4 69.3 72.5 ChartQAtest_test 70.0 74.2 73.3 74.9 69.5 72.0 73.5 75.2 OCRBench 714 731 656 664 685 702 608 641 CharXivDQ_DQ 69.7 73.0 57.7 63.7 64.5 67.5 51.4 53.4 CharXivRQ_RQ 43.9 45.2 43.0 43.3 43.5 44.1 36.2 37.6 OCRVQAtestscore_testscore 55.0 62.8 38.3 39.5 48.5 51.5 41.9 44.6 TextVQAval_val 68.7 71.3 64.6 66.0 66.2 67.8 65.6 69.4 !10 Overall Avg 61.1 63.0 56.2 58.7 58.8 60.6 55.0 57.3