Paper deep dive

Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders

Qingtao Xia, Jiahua Bao, Siyao Cheng, Jie Liu

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 68

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We propose a pre-fine-tuning probing method for Parameter-Efficient Fine-Tuning (PEFT) layer selection, aiming to obtain more stable and higher gains with fewer trainable parameters when adapting large vision--language models (VLMs). Unlike the common practice of applying LoRA and other adapters to all layers at once---where layer selection often relies on heuristic rules---we focus on the vision encoder and directly evaluate the "adaptability'' of each Transformer layer. Specifically, we characterize each layer from two perspectives: (i) the statistical properties of its Q/K/V projection weights (e.g., norms and condition numbers); (ii) robustness under controlled parameter perturbations. We then systematically compare these indicators with the downstream performance gains brought by applying PEFT to a single layer. Across experiments covering seven benchmarks and five PEFT variants, we observe a consistent correlation: layers (or matrices) with larger weight norms and higher condition numbers are usually more robust to perturbations and are more likely to yield larger fine-tuning gains. These results show that distribution-statistics analysis and perturbation tests before fine-tuning can provide practical signals for adaptation-layer selection, thereby maintaining or improving performance while reducing trainable parameters.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

64,431 characters extracted from source content.

Expand or collapse full text

Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders Qingtao Xia Affiliation: Faculty of Computing Harbin Institute of Technology Harbin, China 25s103308@stu.hit.edu.cn Jiahua Bao Affiliation: Faculty of Computing Harbin Institute of Technology Harbin, China bjh_samsara@163.com Siyao Cheng1 †thanks: 1 Corresponding author. Affiliation: Faculty of Computing Harbin Institute of Technology Harbin, China csy@hit.edu.cn Jie Liu Affiliation: Faculty of Computing Harbin Institute of Technology Harbin, China jieliu@hit.edu.cn Abstract We propose a pre-fine-tuning probing method for Parameter-Efficient Fine-Tuning (PEFT) layer selection, aiming to obtain more stable and higher gains with fewer trainable parameters when adapting large vision–language models (VLMs). Unlike the common practice of applying LoRA and other adapters to all layers at once—where layer selection often relies on heuristic rules—we focus on the vision encoder and directly evaluate the “adaptability” of each Transformer layer. Specifically, we characterize each layer from two perspectives: (i) the statistical properties of its Q/K/V projection weights (e.g., norms and condition numbers); (i) robustness under controlled parameter perturbations. We then systematically compare these indicators with the downstream performance gains brought by applying PEFT to a single layer. Across experiments covering seven benchmarks and five PEFT variants, we observe a consistent correlation: layers (or matrices) with larger weight norms and higher condition numbers are usually more robust to perturbations and are more likely to yield larger fine-tuning gains. These results show that distribution-statistics analysis and perturbation tests before fine-tuning can provide practical signals for adaptation-layer selection, thereby maintaining or improving performance while reducing trainable parameters. Index Terms: Parameter-Efficient Fine-Tuning,Vision-Language Models, Layer Selection, LoRA, Perturbation Analysis, Vision Encoder I Introduction Vision–Language Models (VLMs) combine vision encoders and language decoders for multimodal understanding and generation, and models such as InternVL, LLaVA, and OpenFlamingo have shown strong image–text capabilities [1, 2, 3]. Most VLMs follow a modular design in which a pre-trained vision encoder is connected to an LLM through a lightweight connector [4, 5, 6]. For downstream adaptation, Parameter-Efficient Fine-Tuning (PEFT) is widely used because it updates only a small subset of parameters while often retaining competitive performance relative to full fine-tuning [7, 8, 9]. However, PEFT remains sensitive to where trainable modules are inserted, and layer selection is still largely heuristic. Even so, PEFT still faces a key bottleneck in practice: hyperparameters and insertion positions are highly sensitive, and there is a lack of explainable and predictable selection criteria. For example, which layers LoRA should be inserted into, whether all layers need to be covered, and how to choose layers when “only tuning part of the layers” can often significantly affect the final performance, while the trial-and-error cost is high. Moreover, many fine-tuning methods introduce additional trainable parameters (e.g., low-rank adapters or gating vectors). With standard random initialization and early-stage updates, these added degrees of freedom perturb the target layer computation in a structured manner. If, before fine-tuning starts, we can pre-judge which layers are more “worth fine-tuning” by analyzing in-layer weight statistics and controlled perturbation responses, then there is an opportunity to improve both the efficiency and the effectiveness of PEFT. Figure 1 provides an overview of our pre-PEFT probing pipeline; perturbation details are given in Sec. I-B. Fig. 1: Overview of our pre-PEFT probing pipeline for layer selection in the vision encoder. We quantify layer-wise weight statistics and perturbation robustness before fine-tuning, and use them as signals to guide PEFT layer selection. To reduce PEFT training parameters and improve model performance, it is crucial to identify which layers will have the most significant fine-tuning effects before starting the process. In both traditional Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), deeper layers are generally more effective at learning complex features [10]. Based on this, we quantify the parameter distribution for Query (Q), Key (K), and Value (V) across each ViT layer, as well as their robustness to perturbations. We then compare the fine-tuning effectiveness of each layer using five common PEFT methods. Our empirical results indicate a stable correspondence between these pre-tuning indicators and layer-wise PEFT gains across the vision encoders of InternVL2-1B (24 layers) and Qwen2.5-VL-3B (32 layers). For each experiment, we have conducted three trials and reported the median as the result; across both backbones, run-to-run variance is small (on the order of 0.3p). Our contributions are as follows: • We provide a unified layer-wise study linking perturbation robustness, Q/K/V weight statistics, and PEFT gains in VLM vision encoders. • Across InternVL2-1B and Qwen2.5-VL-3B, more robust layers often coincide with larger norms, higher condition numbers, and better layer-wise PEFT gains. • We show that selective tuning of a small subset of layers can remain competitive with broader tuning, providing a practical pre-screening signal for PEFT layer selection. The code and scripts used in this work are publicly available at https://github.com/atoz03/prepeft-probing. I Related Work I-A VLMs VLMs align a vision encoder with a language decoder for multimodal understanding and generation. Representative systems such as LLaVA connect CLIP ViT-L/14 to an LLM through a lightweight projector [2, 11], while InternVL2 and Qwen2.5-VL adopt ViT-style vision encoders, lightweight connectors, and staged training recipes [12, 1, 13, 14]. I-B PEFT with VLMs To adapt VLMs under limited budgets, PEFT updates only a small subset of parameters, with common choices including LoRA, adapters, and prompt tuning [7, 8, 15]. These methods can be viewed as introducing structured perturbations to the original network during adaptation. Prior work has studied perturbation and robustness in CNN/Transformer models [16, 17, 18], but their connection to layer-wise PEFT effectiveness in VLM vision encoders remains underexplored. Our work studies this connection through weight statistics, perturbation robustness, and layer-wise PEFT gains. Fig. 2: Perturbation operators and injection points for pre-PEFT probing, including element-wise and activation perturbations. I Datasets and Perturbation Settings We study pre-PEFT layer selection in the vision encoder. Our hypothesis is that robustness to controlled perturbations before fine-tuning correlates with subsequent PEFT effectiveness. Accordingly, we probe each layer without updating the base model, and later compare the probing signals with layer-wise PEFT results. I-A Datasets Preparation We select 48 datasets that cover diverse VLM capabilities (image captioning, OCR, chart and table understanding, scientific image question answering, math question answering, etc.), including Screen2Words [19], OCR-VQA [20], Chart2Text [21], CLEVR-Math [22], and ScienceQA [23]. Since most downstream fine-tuning tasks have small data sizes and there is some overlap among datasets, the training distribution may be dominated by a few datasets. To more clearly observe the correlation between “perturbation scenarios–PEFT methods” and to better match real-world settings, we sample 27,000 pairs from 1.8 million ⟨image,text⟩ image,text pairs to construct a training mixture for subsequent perturbation probing and PEFT training. To evaluate model performance after PEFT, we choose seven benchmarks for comprehensive evaluation: MMStar [24] as the primary benchmark for overall comparison and consistency checking of basic capabilities, along with six additional benchmarks—ChartQA [25], OCR-VQA [20], HallusionBench [26], ScienceQA [23], RealWorldQA [27], and MMBench [28]—to evaluate diverse downstream tasks (English dialogue, form understanding, OCR, hallucination detection, science question answering, real-world question answering, etc.). I-B Perturbation Settings Figure 2 summarizes these perturbation operators and injection points. To align perturbation design with layer-wise controllability, we focus on five PEFT methods that can be naturally compared at the layer level: LoRA [7], VeRA [29], IA3 [30], LayerNorm-Tuning, and Partial Fine-Tuning. We design five perturbation methods: Additive Noise (AddNoise), Multiplicative Noise (MulNoise), Linear Noise (LinearNoise), Activation Noise (ActNoise), and Path Noise (PathNoise). AddNoise and MulNoise sample from a normal distribution and apply element-wise noise scaled by a noise weight; LinearNoise and ActNoise introduce perturbations via a Kaiming-initialized linear layer [31] and ReLU [32]; PathNoise explicitly simulates the LoRA-style “added bypass” structure: we perform Singular Value Decomposition (SVD) on the original QKV generation linear layer to obtain U, S, and V, and then construct a new QKV generation matrix by down-sampling with U⊗SuU S_u and up-sampling with Sv⊗VS_v V (where Su=Sv=S_u=S_v= S). The final output is a weighted sum of the original path and the new path. In preliminary tests, we find that inserting a trainable linear layer (LinearNoise) into the vision encoder collapses the VLM (all benchmark scores drop to 0.0); thus LinearNoise is omitted from the reported tables. Formalization Let e denote the input embedding of a target layer and qkv(⋅)qkv(·) denote the original QKV projection. We instantiate perturbations by modifying the QKV output: AddNoise: AddNoise: qkv(e)+αϵ,ϵ∼⁡(0,I) qkv(e)+α\,ε,\ \ ε (0,I) (1) MulNoise: MulNoise: qkv(e)⊙(1+αϵ),ϵ∼⁡(0,I) qkv(e) (1+α\,ε),\ \ ε (0,I) ActNoise: ActNoise: ϕ⁡(qkv(e)),ϕ⁡(⋅)=ReLU⁡(W⁡(⋅)) φ\! (qkv(e) ),\ \ φ(·)=ReLU(W(·)) PathNoise: PathNoise: (1−α)qkv(e)+αqkv′(e) (1-α)\,qkv(e)+α\,qkv (e) where α is the perturbation weight and qkv′(⋅)qkv (·) is the SVD-constructed low-rank bypass path. We measure robustness by the change of downstream scores under the same evaluation protocol: layers with smaller drops are considered more robust. Layer selection by probing Given an adaptation budget of k layers, we rank candidate layers on a held-out split D by a probing score computed from ActNoise and PathNoise drops (smaller is better). Concretely, we define: Rℓ=12(dℓ,act+dℓ,path),dℓ,path=α∈path[s0−sℓ,α],R_ = 12 (d_ ,act+d_ ,path ), d_ ,path=E_α _path\! [s_0-s_ ,α ], (2) where s0s_0 is the unperturbed score and sℓ,αs_ ,α is the score after perturbing layer ℓ with strength α. We then select the top-k layers with the smallest RℓR_ for subsequent multi-layer joint PEFT. Hyperparameter Settings (Two Backbones) • Backbones: InternVL2-1B (24L ViT + MLP); Qwen2.5-VL-3B (32L ViT) [14] • Training: lr=4e−54e-5, wd=0.01, warmup=0.03, bf16 • LoRA: r∈32,128,512r∈\32,128,512\, α=256, dropout=0.05 • Perturbation: Add/Mul/Act: α=0.05; PathNoise: base α=0.5 (rank=32), sweep α=0.1–1.0 • Inference: temp=1.0, max_len=32768, sliding-window (Transformers 4.37.2) IV Experiments and Analysis IV-A Distribution of Model Parameters We analyze the Q/K/V projection matrices in the 24-layer InternVL2-1B and 32-layer Qwen2.5-VL-3B vision encoders. For coarse cross-backbone comparisons, we segment the encoders into contiguous 8-layer blocks and refer to higher-index blocks/layers as deeper. To quantify differences in the weight distributions of the Q/K/V projection layers, we compute six metrics for each layer and each projection matrix: variance, L1L_1 norm, L2L_2 norm, Frobenius norm, infinity norm, and condition number. For each layer ℓ , we compute the statistics on the raw projection weights Wq(ℓ),Wk(ℓ),Wv(ℓ)W_q^( ),W_k^( ),W_v^( ). We define the condition number as κ⁡(W)=σmax(W)/σmin(W)κ(W)= _ (W)/ _ (W), where σmax _ and σmin _ are the largest and smallest singular values of W; κ⁡(⋅)κ(·) is computed on the raw weights without any attention-time scaling (e.g., 1/dk1/ d_k) or additional normalization/reparameterization. For visualization and cross-layer comparison, we apply per-metric min–max scaling across layers within the same backbone to map each metric to [0,1][0,1]. Fig. 3: Layer-wise statistics of the Q/K/V projection linear weights in the vision encoders of InternVL2-1B (top, 24 layers) and Qwen2.5-VL-3B (bottom, 32 layers). We report six metrics (variance, L1L_1, L2L_2, Frobenius, infinity norm, and condition number), each min–max normalized across layers within the same backbone. TABLE I: Block-level perturbation results on MMStar (left) and six benchmarks (right) for InternVL2-1B. ♠ /♣ /♡ /♢ : column-best; ↑ /↓ : vs. unperturbed. (a) MMStar (InternVL2-1B) Layers Perturbation Overall CP FGP IR LR MA S&T 0-7 (Block-1) PathNoise 37.61↓8.06_ 8.06 49.33↓11.87_ 11.87 28.22↓7.38_ 7.38 40.53↓13.07_ 13.07 27.96↓7.24_ 7.24 ♠ 44.13↓3.87_ 3.87 35.47↓4.93_ 4.93 ActNoise 40.80↓4.87_ 4.87 54.40↓6.80_ 6.80 32.40↓3.20_ 3.20 ♣ 48.00↓5.60_ 5.60 31.20↓4.00_ 4.00 44.00↓4.00_ 4.00 ♣ 34.80↓5.60_ 5.60 AddNoise 45.07↓0.60_ 0.60 61.200.00_0.00 33.60↓2.00_ 2.00 53.20↓0.40_ 0.40 34.80↓0.40_ 0.40 47.60↓0.40_ 0.40 40.00↓0.40_ 0.40 MulNoise 45.13↓0.53_ 0.53 61.200.00_0.00 34.80↓0.80_ 0.80 52.80↓0.80_ 0.80 33.60↓1.60_ 1.60 ♢ 48.80↑0.80_ 0.80 39.60↓0.80_ 0.80 8-15 (Block-2) PathNoise 36.12↓9.55_ 9.55 44.22↓16.98_ 16.98 28.98↓6.62_ 6.62 39.07↓14.53_ 14.53 27.69↓7.51_ 7.51 43.78↓4.22_ 4.22 32.98↓7.42_ 7.42 ActNoise 29.87↓15.80_ 15.80 37.20↓24.00_ 24.00 22.00↓13.60_ 13.60 29.60↓24.00_ 24.00 20.40↓14.80_ 14.80 40.40↓7.60_ 7.60 29.60↓10.80_ 10.80 AddNoise 45.33↓0.33_ 0.33 61.200.00_0.00 34.00↓1.60_ 1.60 53.20↓0.40_ 0.40 35.200.00_0.00 ♡ 48.40↑0.40_ 0.40 40.00↓0.40_ 0.40 MulNoise 45.40↓0.27_ 0.27 61.60↑0.40_ 0.40 ♢ 35.20↓0.40_ 0.40 52.80↓0.80_ 0.80 34.80↓0.40_ 0.40 47.20↓0.80_ 0.80 ♢ 40.80↑0.40_ 0.40 16-23 (Block-3) PathNoise ♠ 39.88↓5.79_ 5.79 ♠ 54.13↓7.07_ 7.07 ♠ 30.44↓5.16_ 5.16 ♠ 45.07↓8.53_ 8.53 ♠ 30.13↓5.07_ 5.07 43.20↓4.80_ 4.80 ♠ 36.31↓4.09_ 4.09 ActNoise ♣ 41.13↓4.53_ 4.53 ♣ 56.00↓5.20_ 5.20 ♣ 33.60↓2.00_ 2.00 45.20↓8.40_ 8.40 ♣ 34.00↓1.20_ 1.20 ♣ 44.80↓3.20_ 3.20 33.20↓7.20_ 7.20 AddNoise ♡ 45.60↓0.07_ 0.07 ♡ 61.60↑0.40_ 0.40 ♡ 34.80↓0.80_ 0.80 ♡ 53.600.00_0.00 ♡ 36.00↑0.80_ 0.80 47.20↓0.80_ 0.80 ♡ 40.400.00_0.00 MulNoise ♢ 45.53↓0.13_ 0.13 ♢ 62.00↑0.80_ 0.80 34.80↓0.80_ 0.80 ♢ 53.600.00_0.00 ♢ 35.60↑0.40_ 0.40 47.60↓0.40_ 0.40 39.60↓0.80_ 0.80 (b) Benchmarks (InternVL2-1B) Layers Perturbation CQA HB MMB OCRVQA RWQA SQA 0-7 (Block-1) PathNoise 28.79↓29.09_ 29.09 29.59↓5.62_ 5.62 47.69↓19.49_ 19.49 31.98↓11.53_ 11.53 ♠ 42.99↓5.77_ 5.77 78.83↓8.58_ 8.58 ActNoise 27.84↓30.04_ 30.04 ♣ 30.82↓4.39_ 4.39 ♣ 60.91↓6.27_ 6.27 ♣ 35.37↓8.13_ 8.13 ♣ 44.31↓4.44_ 4.44 82.30↓5.11_ 5.11 AddNoise ♡ 58.16↑0.28_ 0.28 35.49↑0.29_ 0.29 ♡ 67.35↑0.17_ 0.17 ♡ 43.54↑0.03_ 0.03 ♡ 48.760.00_0.00 ♡ 87.36↓0.05_ 0.05 MulNoise 58.04↑0.16_ 0.16 31.27↓3.93_ 3.93 67.180.00_0.00 43.510.00_0.00 ♢ 48.89↑0.13_ 0.13 87.26↓0.15_ 0.15 8-15 (Block-2) PathNoise 24.57↓33.31_ 33.31 ♠ 30.52↓4.68_ 4.68 42.50↓24.68_ 24.68 27.96↓15.55_ 15.55 42.67↓6.09_ 6.09 78.09↓9.32_ 9.32 ActNoise 12.88↓45.00_ 45.00 26.22↓8.98_ 8.98 28.69↓38.49_ 38.49 20.64↓22.87_ 22.87 37.52↓11.24_ 11.24 73.18↓14.23_ 14.23 AddNoise 57.96↑0.08_ 0.08 ♡ 36.14↑0.94_ 0.94 67.180.00_0.00 43.53↑0.03_ 0.03 48.24↓0.52_ 0.52 87.31↓0.10_ 0.10 MulNoise ♢ 58.08↑0.20_ 0.20 35.10↓0.10_ 0.10 ♢ 67.70↑0.52_ 0.52 ♢ 43.54↑0.04_ 0.04 47.84↓0.92_ 0.92 ♢ 87.46↑0.05_ 0.05 16-23 (Block-3) PathNoise ♠ 36.39↓21.49_ 21.49 29.77↓5.44_ 5.44 ♠ 53.00↓14.18_ 14.18 ♠ 33.74↓9.77_ 9.77 42.95↓5.81_ 5.81 ♠ 82.23↓5.17_ 5.17 ActNoise ♣ 49.52↓8.36_ 8.36 30.41↓4.79_ 4.79 51.46↓15.72_ 15.72 34.76↓8.74_ 8.74 ♣ 44.31↓4.44_ 4.44 ♣ 82.90↓4.51_ 4.51 AddNoise ♡ 58.16↑0.28_ 0.28 31.56↓3.64_ 3.64 67.27↑0.09_ 0.09 43.50↓0.01_ 0.01 48.63↓0.13_ 0.13 ♡ 87.36↓0.05_ 0.05 MulNoise 57.76↓0.12_ 0.12 ♢ 35.68↑0.47_ 0.47 67.27↑0.09_ 0.09 43.500.00_0.00 48.50↓0.26_ 0.26 87.410.00_0.00 TABLE I: Block-level perturbation results on MMStar (left) and six benchmarks (right) for Qwen2.5-VL-3B. ♠ /♣ /♡ /♢ : column-best; ↑ /↓ : vs. unperturbed. (a) MMStar (Qwen2.5-VL-3B) Layers Perturbation Overall CP FGP IR LR MA S&T 0-7 (Block-1) PathNoise ♠ 54.64↓0.26_ 0.26 ♠ 66.40↓1.20_ 1.20 46.40↑0.40_ 0.40 60.80↑0.40_ 0.40 ♠ 57.43↑2.41_ 2.41 ♠ 60.80↓1.60_ 1.60 36.00↓2.00_ 2.00 ActNoise ♣ 55.57↑0.67_ 0.67 ♣ 68.00↑0.40_ 0.40 46.000.00_0.00 60.400.00_0.00 ♣ 57.43↑2.41_ 2.41 ♣ 63.20↑0.80_ 0.80 38.40↑0.40_ 0.40 AddNoise 55.30↑0.40_ 0.40 67.600.00_0.00 44.80↓1.20_ 1.20 60.00↓0.40_ 0.40 ♡ 57.43↑2.41_ 2.41 ♡ 63.60↑1.20_ 1.20 38.40↑0.40_ 0.40 MulNoise 55.10↑0.20_ 0.20 ♢ 67.600.00_0.00 ♢ 47.60↑1.60_ 1.60 60.400.00_0.00 55.42↑0.40_ 0.40 61.60↓0.80_ 0.80 38.000.00_0.00 8-15 (Block-2) PathNoise 53.57↓1.33_ 1.33 65.20↓2.40_ 2.40 46.000.00_0.00 58.80↓1.60_ 1.60 54.62↓0.40_ 0.40 60.80↓1.60_ 1.60 36.00↓2.00_ 2.00 ActNoise 55.44↑0.54_ 0.54 68.00↑0.40_ 0.40 ♣ 46.40↑0.40_ 0.40 60.80↑0.40_ 0.40 57.03↑2.01_ 2.01 62.00↓0.40_ 0.40 38.40↑0.40_ 0.40 AddNoise ♡ 55.97↑1.07_ 1.07 ♡ 68.00↑0.40_ 0.40 ♡ 47.60↑1.60_ 1.60 61.20↑0.80_ 0.80 56.22↑1.20_ 1.20 62.80↑0.40_ 0.40 ♡ 40.00↑2.00_ 2.00 MulNoise 55.24↑0.34_ 0.34 65.60↓2.00_ 2.00 46.000.00_0.00 60.80↑0.40_ 0.40 ♢ 57.83↑2.81_ 2.81 ♢ 62.00↓0.40_ 0.40 39.20↑1.20_ 1.20 16-23 (Block-3) PathNoise 52.30↓2.60_ 2.60 65.20↓2.40_ 2.40 44.80↓1.20_ 1.20 60.400.00_0.00 52.61↓2.41_ 2.41 58.40↓4.00_ 4.00 32.40↓5.60_ 5.60 ActNoise 55.10↑0.20_ 0.20 67.20↓0.40_ 0.40 45.20↓0.80_ 0.80 ♣ 61.60↑1.20_ 1.20 55.020.00_0.00 62.80↑0.40_ 0.40 38.80↑0.80_ 0.80 AddNoise 55.37↑0.47_ 0.47 68.00↑0.40_ 0.40 45.20↓0.80_ 0.80 ♡ 61.60↑1.20_ 1.20 55.82↑0.80_ 0.80 61.60↓0.80_ 0.80 40.00↑2.00_ 2.00 MulNoise 54.57↓0.33_ 0.33 66.80↓0.80_ 0.80 46.40↑0.40_ 0.40 60.00↓0.40_ 0.40 55.020.00_0.00 61.20↓1.20_ 1.20 38.000.00_0.00 24-31 (Block-4) PathNoise 53.77↓1.13_ 1.13 64.80↓2.80_ 2.80 ♠ 46.80↑0.80_ 0.80 ♠ 62.00↑1.60_ 1.60 52.61↓2.41_ 2.41 59.60↓2.80_ 2.80 ♠ 36.80↓1.20_ 1.20 ActNoise 55.17↑0.27_ 0.27 67.600.00_0.00 46.000.00_0.00 61.20↑0.80_ 0.80 54.62↓0.40_ 0.40 61.60↓0.80_ 0.80 ♣ 40.00↑2.00_ 2.00 AddNoise 55.17↑0.27_ 0.27 67.20↓0.40_ 0.40 45.20↓0.80_ 0.80 61.60↑1.20_ 1.20 54.62↓0.40_ 0.40 62.80↑0.40_ 0.40 39.60↑1.60_ 1.60 MulNoise ♢ 55.37↑0.47_ 0.47 66.80↓0.80_ 0.80 46.80↑0.80_ 0.80 ♢ 61.20↑0.80_ 0.80 55.82↑0.80_ 0.80 62.00↓0.40_ 0.40 ♢ 39.60↑1.60_ 1.60 (b) Benchmarks (Qwen2.5-VL-3B) Layers Perturbation CQA HB MMB OCRVQA RWQA SQA 0-7 (Block-1) PathNoise ♠ 84.28↑0.20_ 0.20 ♠ 61.09↑0.73_ 0.73 ♠ 82.27↓0.16_ 0.16 ♠ 76.80↓0.50_ 0.50 ♠ 65.23↓1.18_ 1.18 74.81↓0.75_ 0.75 ActNoise 84.12↑0.04_ 0.04 60.15↓0.21_ 0.21 ♣ 82.53↑0.10_ 0.10 77.300.00_0.00 ♣ 65.75↓0.66_ 0.66 ♣ 75.46↓0.10_ 0.10 AddNoise ♡ 84.16↑0.08_ 0.08 ♡ 61.20↑0.84_ 0.84 ♡ 82.59↑0.16_ 0.16 77.00↓0.30_ 0.30 65.49↓0.92_ 0.92 75.06↓0.50_ 0.50 MulNoise 83.80↓0.28_ 0.28 ♢ 60.78↑0.42_ 0.42 82.53↑0.10_ 0.10 76.90↓0.40_ 0.40 65.23↓1.18_ 1.18 75.16↓0.40_ 0.40 8-15 (Block-2) PathNoise 83.48↓0.60_ 0.60 60.46↑0.10_ 0.10 80.89↓1.54_ 1.54 76.60↓0.70_ 0.70 63.53↓2.88_ 2.88 74.71↓0.85_ 0.85 ActNoise 84.080.00_0.00 59.73↓0.63_ 0.63 82.20↓0.23_ 0.23 77.20↓0.10_ 0.10 65.23↓1.18_ 1.18 75.01↓0.55_ 0.55 AddNoise 84.04↓0.04_ 0.04 60.78↑0.42_ 0.42 82.41↓0.02_ 0.02 76.70↓0.60_ 0.60 ♡ 65.88↓0.53_ 0.53 75.11↓0.45_ 0.45 MulNoise 83.84↓0.24_ 0.24 60.67↑0.31_ 0.31 82.74↑0.31_ 0.31 77.10↓0.20_ 0.20 65.10↓1.31_ 1.31 74.91↓0.65_ 0.65 16-23 (Block-3) PathNoise 82.44↓1.64_ 1.64 60.15↓0.21_ 0.21 81.63↓0.80_ 0.80 75.40↓1.90_ 1.90 63.40↓3.01_ 3.01 74.47↓1.09_ 1.09 ActNoise 84.12↑0.04_ 0.04 ♣ 60.57↑0.21_ 0.21 82.47↑0.04_ 0.04 76.60↓0.70_ 0.70 65.62↓0.79_ 0.79 75.26↓0.30_ 0.30 AddNoise 83.96↓0.12_ 0.12 60.78↑0.42_ 0.42 82.33↓0.10_ 0.10 76.90↓0.40_ 0.40 65.62↓0.79_ 0.79 ♡ 75.21↓0.35_ 0.35 MulNoise ♢ 84.00↓0.08_ 0.08 60.15↓0.21_ 0.21 ♢ 82.78↑0.35_ 0.35 77.10↓0.20_ 0.20 ♢ 66.410.00_0.00 75.16↓0.40_ 0.40 24-31 (Block-4) PathNoise 82.16↓1.92_ 1.92 59.10↓1.26_ 1.26 81.96↓0.47_ 0.47 76.80↓0.50_ 0.50 62.48↓3.93_ 3.93 ♠ 74.91↓0.65_ 0.65 ActNoise ♣ 84.16↑0.08_ 0.08 60.57↑0.21_ 0.21 82.49↑0.06_ 0.06 ♣ 77.70↑0.40_ 0.40 65.62↓0.79_ 0.79 75.01↓0.55_ 0.55 AddNoise 83.96↓0.12_ 0.12 60.78↑0.42_ 0.42 82.35↓0.08_ 0.08 ♡ 77.10↓0.20_ 0.20 65.88↓0.53_ 0.53 75.01↓0.55_ 0.55 MulNoise 84.00↓0.08_ 0.08 60.46↑0.10_ 0.10 82.61↑0.18_ 0.18 ♢ 77.20↓0.10_ 0.10 65.36↓1.05_ 1.05 ♢ 75.31↓0.25_ 0.25 In Figure 3, deeper layers generally exhibit larger variance and norm-related metrics, while condition-number fluctuations are more concentrated in earlier layers. IV-B Segmented Analysis of Vision Encoder Layers To estimate promising PEFT layers before tuning, we inject perturbations into the predefined 8-layer blocks and evaluate MMStar plus six benchmarks; PathNoise is used as the LoRA-like perturbation. Tables I(a) and I(a) report the block-level MMStar results for InternVL2-1B and Qwen2.5-VL-3B, respectively. Across both backbones, ActNoise and PathNoise produce the largest degradation, whereas AddNoise and MulNoise are much milder; moreover, block-level robustness is not monotonic with depth (Tables I and I). This motivates the finer layer-wise analysis below. IV-C Layer-Wise Analysis of Vision Encoder Table I shows substantial within-block variation on InternVL2-1B, with several middle layers consistently more fragile under both ActNoise and PathNoise, suggesting layer-level PEFT selection rather than contiguous block selection. IV-D PEFT Analysis of Vision Encoder We evaluate layer-wise PEFT on InternVL2-1B with LoRA (r=32/128/512r=32/128/512), VeRA, IA3, LayerNorm-Tuning, and Partial-Tuning, and on Qwen2.5-VL-3B with LoRA (r=32r=32) plus the same baselines. Figure 4 and Table IV show that the best PEFT gains cluster in early and late layers, broadly overlapping with perturbation-robust, high-statistics layers. Strong same-budget baselines. Under the same adaptation budget (k=5k=5), budget-matched heuristics yield only marginal and unstable gains. On InternVL2-1B, first-k LoRA reaches 43.70% versus the 43.63% base, while random subsets and single-component tuning are weaker; on Qwen2.5-VL-3B, middle-k LoRA reaches 55.24% versus the 54.90% base, whereas attention-only and MLP-only tuning drop to 54.70% and 54.50%. TABLE I: Layer-wise perturbation on MMStar (a) and six benchmarks (b) for InternVL2-1B. Best in bold; worst underlined. (a) MMStar (InternVL2-1B) L ActNoise PathNoise Overall CP FGP IR LR MA S&T Overall CP FGP IR LR MA S&T 0 45.6↓0.1_ 0.1 60.0↓1.2_ 1.2 36.4↑0.8_ 0.8 53.60.0_0.0 37.2↑2.0_ 2.0 47.6↓0.4_ 0.4 38.8↓1.6_ 1.6 44.5↓1.1_ 1.1 61.6↑0.4_ 0.4 34.0↓1.6_ 1.6 51.6↓2.0_ 2.0 33.6↓1.6_ 1.6 46.4↓1.6_ 1.6 40.0↓0.4_ 0.4 1 45.0↓0.7_ 0.7 60.0↓1.2_ 1.2 36.0↑0.4_ 0.4 53.60.0_0.0 34.0↓1.2_ 1.2 45.6↓2.4_ 2.4 40.8↑0.4_ 0.4 44.2↓1.5_ 1.5 57.6↓3.6_ 3.6 36.0↑0.4_ 0.4 51.6↓2.0_ 2.0 35.20.0_0.0 46.4↓1.6_ 1.6 38.4↓2.0_ 2.0 2 44.9↓0.7_ 0.7 60.0↓1.2_ 1.2 33.6↓2.0_ 2.0 53.2↓0.4_ 0.4 34.8↓0.4_ 0.4 47.6↓0.4_ 0.4 40.40.0_0.0 45.3↓0.4_ 0.4 61.20.0_0.0 36.4↑0.8_ 0.8 51.6↓2.0_ 2.0 36.8↑1.6_ 1.6 47.2↓0.8_ 0.8 38.4↓2.0_ 2.0 3 45.5↓0.1_ 0.1 62.0↑0.8_ 0.8 35.2↓0.4_ 0.4 51.6↓2.0_ 2.0 34.0↓1.2_ 1.2 49.6↑1.6_ 1.6 40.8↑0.4_ 0.4 43.8↓1.9_ 1.9 58.8↓2.4_ 2.4 33.2↓2.4_ 2.4 51.2↓2.4_ 2.4 36.0↑0.8_ 0.8 44.4↓3.6_ 3.6 39.2↓1.2_ 1.2 4 45.1↓0.5_ 0.5 60.4↓0.8_ 0.8 36.0↑0.4_ 0.4 53.2↓0.4_ 0.4 33.2↓2.0_ 2.0 49.2↑1.2_ 1.2 38.8↓1.6_ 1.6 44.9↓0.8_ 0.8 59.6↓1.6_ 1.6 36.4↑0.8_ 0.8 51.2↓2.4_ 2.4 36.0↑0.8_ 0.8 47.2↓0.8_ 0.8 38.8↓1.6_ 1.6 5 45.4↓0.3_ 0.3 60.0↓1.2_ 1.2 36.4↑0.8_ 0.8 52.4↓1.2_ 1.2 35.6↑0.4_ 0.4 47.2↓0.8_ 0.8 40.8↑0.4_ 0.4 44.5↓1.2_ 1.2 61.20.0_0.0 35.60.0_0.0 51.2↓2.4_ 2.4 37.2↑2.0_ 2.0 45.6↓2.4_ 2.4 36.0↓4.4_ 4.4 6 44.6↓1.1_ 1.1 60.4↓0.8_ 0.8 36.0↑0.4_ 0.4 51.2↓2.4_ 2.4 36.0↑0.8_ 0.8 45.6↓2.4_ 2.4 38.4↓2.0_ 2.0 42.7↓3.0_ 3.0 57.2↓4.0_ 4.0 33.2↓2.4_ 2.4 51.2↓2.4_ 2.4 33.2↓2.0_ 2.0 46.8↓1.2_ 1.2 34.4↓6.0_ 6.0 7 44.7↓0.9_ 0.9 60.0↓1.2_ 1.2 33.6↓2.0_ 2.0 53.2↓0.4_ 0.4 34.8↓0.4_ 0.4 47.2↓0.8_ 0.8 39.6↓0.8_ 0.8 45.3↓0.4_ 0.4 60.0↓1.2_ 1.2 35.60.0_0.0 51.2↓2.4_ 2.4 36.4↑1.2_ 1.2 46.4↓1.6_ 1.6 42.0↑1.6_ 1.6 8 44.8↓0.9_ 0.9 60.4↓0.8_ 0.8 36.0↑0.4_ 0.4 52.0↓1.6_ 1.6 34.4↓0.8_ 0.8 46.4↓1.6_ 1.6 39.6↓0.8_ 0.8 44.6↓1.1_ 1.1 58.0↓3.2_ 3.2 37.2↑1.6_ 1.6 53.60.0_0.0 34.8↓0.4_ 0.4 46.8↓1.2_ 1.2 37.2↓3.2_ 3.2 9 44.5↓1.1_ 1.1 59.6↓1.6_ 1.6 35.2↓0.4_ 0.4 52.0↓1.6_ 1.6 36.0↑0.8_ 0.8 45.2↓2.8_ 2.8 39.2↓1.2_ 1.2 44.6↓1.1_ 1.1 58.0↓3.2_ 3.2 37.2↑1.6_ 1.6 53.60.0_0.0 34.8↓0.4_ 0.4 46.8↓1.2_ 1.2 37.2↓3.2_ 3.2 10 44.7↓1.0_ 1.0 60.4↓0.8_ 0.8 33.2↓2.4_ 2.4 52.8↓0.8_ 0.8 35.6↑0.4_ 0.4 46.0↓2.0_ 2.0 40.0↓0.4_ 0.4 44.5↓1.1_ 1.1 59.6↓1.6_ 1.6 35.60.0_0.0 52.0↓1.6_ 1.6 34.8↓0.4_ 0.4 45.2↓2.8_ 2.8 40.0↓0.4_ 0.4 11 45.1↓0.6_ 0.6 62.4↑1.2_ 1.2 36.8↑1.2_ 1.2 52.4↓1.2_ 1.2 34.8↓0.4_ 0.4 45.2↓2.8_ 2.8 38.8↓1.6_ 1.6 44.5↓1.1_ 1.1 59.6↓1.6_ 1.6 35.60.0_0.0 52.0↓1.6_ 1.6 34.8↓0.4_ 0.4 45.2↓2.8_ 2.8 40.0↓0.4_ 0.4 12 43.9↓1.8_ 1.8 59.2↓2.0_ 2.0 32.4↓3.2_ 3.2 50.4↓3.2_ 3.2 34.8↓0.4_ 0.4 47.2↓0.8_ 0.8 39.2↓1.2_ 1.2 44.1↓1.5_ 1.5 60.4↓0.8_ 0.8 35.60.0_0.0 51.2↓2.4_ 2.4 35.6↑0.4_ 0.4 47.2↓0.8_ 0.8 34.8↓5.6_ 5.6 13 44.0↓1.7_ 1.7 59.6↓1.6_ 1.6 32.4↓3.2_ 3.2 52.8↓0.8_ 0.8 34.4↓0.8_ 0.8 47.6↓0.4_ 0.4 37.2↓3.2_ 3.2 42.7↓3.0_ 3.0 56.4↓4.8_ 4.8 32.8↓2.8_ 2.8 49.6↓4.0_ 4.0 34.0↓1.2_ 1.2 46.0↓2.0_ 2.0 37.2↓3.2_ 3.2 14 44.0↓1.7_ 1.7 60.4↓0.8_ 0.8 32.4↓3.2_ 3.2 50.8↓2.8_ 2.8 34.4↓0.8_ 0.8 48.4↑0.4_ 0.4 37.6↓2.8_ 2.8 43.9↓1.7_ 1.7 59.6↓1.6_ 1.6 31.2↓4.4_ 4.4 53.2↓0.4_ 0.4 34.8↓0.4_ 0.4 47.2↓0.8_ 0.8 37.6↓2.8_ 2.8 15 44.4↓1.3_ 1.3 57.2↓4.0_ 4.0 33.6↓2.0_ 2.0 53.60.0_0.0 36.8↑1.6_ 1.6 47.2↓0.8_ 0.8 38.0↓2.4_ 2.4 44.5↓1.1_ 1.1 59.2↓2.0_ 2.0 34.4↓1.2_ 1.2 52.0↓1.6_ 1.6 35.20.0_0.0 47.2↓0.8_ 0.8 39.2↓1.2_ 1.2 16 44.5↓1.2_ 1.2 60.4↓0.8_ 0.8 34.4↓1.2_ 1.2 50.4↓3.2_ 3.2 35.6↑0.4_ 0.4 46.8↓1.2_ 1.2 39.2↓1.2_ 1.2 44.4↓1.3_ 1.3 58.8↓2.4_ 2.4 31.6↓4.0_ 4.0 52.8↓0.8_ 0.8 36.4↑1.2_ 1.2 48.00.0_0.0 38.8↓1.6_ 1.6 17 44.1↓1.6_ 1.6 59.6↓1.6_ 1.6 32.0↓3.6_ 3.6 52.0↓1.6_ 1.6 36.4↑1.2_ 1.2 45.2↓2.8_ 2.8 39.2↓1.2_ 1.2 44.1↓1.5_ 1.5 56.8↓4.4_ 4.4 34.4↓1.2_ 1.2 51.2↓2.4_ 2.4 35.6↑0.4_ 0.4 46.0↓2.0_ 2.0 40.8↑0.4_ 0.4 18 45.3↓0.3_ 0.3 61.6↑0.4_ 0.4 34.8↓0.8_ 0.8 52.4↓1.2_ 1.2 36.4↑1.2_ 1.2 46.8↓1.2_ 1.2 40.0↓0.4_ 0.4 44.7↓1.0_ 1.0 60.0↓1.2_ 1.2 33.2↓2.4_ 2.4 52.8↓0.8_ 0.8 35.20.0_0.0 46.0↓2.0_ 2.0 40.8↑0.4_ 0.4 19 45.0↓0.7_ 0.7 61.20.0_0.0 35.2↓0.4_ 0.4 53.60.0_0.0 34.0↓1.2_ 1.2 46.0↓2.0_ 2.0 40.0↓0.4_ 0.4 45.0↓0.7_ 0.7 60.0↓1.2_ 1.2 35.60.0_0.0 52.4↓1.2_ 1.2 35.6↑0.4_ 0.4 45.6↓2.4_ 2.4 40.8↑0.4_ 0.4 20 44.8↓0.9_ 0.9 62.0↑0.8_ 0.8 35.2↓0.4_ 0.4 52.0↓1.6_ 1.6 34.0↓1.2_ 1.2 47.6↓0.4_ 0.4 38.0↓2.4_ 2.4 44.9↓0.8_ 0.8 60.4↓0.8_ 0.8 36.0↑0.4_ 0.4 51.6↓2.0_ 2.0 36.0↑0.8_ 0.8 44.8↓3.2_ 3.2 40.40.0_0.0 21 44.6↓1.1_ 1.1 60.0↓1.2_ 1.2 33.2↓2.4_ 2.4 51.6↓2.0_ 2.0 35.6↑0.4_ 0.4 48.4↑0.4_ 0.4 38.8↓1.6_ 1.6 44.1↓1.6_ 1.6 59.6↓1.6_ 1.6 33.2↓2.4_ 2.4 52.8↓0.8_ 0.8 35.6↑0.4_ 0.4 44.8↓3.2_ 3.2 38.4↓2.0_ 2.0 22 44.9↓0.7_ 0.7 60.8↓0.4_ 0.4 36.0↑0.4_ 0.4 51.2↓2.4_ 2.4 32.8↓2.4_ 2.4 47.6↓0.4_ 0.4 41.2↑0.8_ 0.8 44.5↓1.1_ 1.1 62.0↑0.8_ 0.8 35.2↓0.4_ 0.4 54.0↑0.4_ 0.4 36.4↑1.2_ 1.2 42.0↓6.0_ 6.0 37.6↓2.8_ 2.8 23 44.7↓1.0_ 1.0 60.4↓0.8_ 0.8 37.6↑2.0_ 2.0 51.2↓2.4_ 2.4 35.6↑0.4_ 0.4 46.0↓2.0_ 2.0 37.2↓3.2_ 3.2 44.1↓1.5_ 1.5 59.6↓1.6_ 1.6 33.6↓2.0_ 2.0 51.2↓2.4_ 2.4 33.2↓2.0_ 2.0 46.8↓1.2_ 1.2 40.40.0_0.0 (b) Benchmarks (InternVL2-1B) Layer ActNoise PathNoise CQA HB MMB OCRVQA RWQA SQA CQA HB MMB OCRVQA RWQA SQA 0 55.8↓2.0_ 2.0 31.5↓3.7_ 3.7 67.4↑0.3_ 0.3 43.3↓0.2_ 0.2 48.4↓0.4_ 0.4 87.50.0_0.0 53.7↓4.2_ 4.2 32.3↓2.9_ 2.9 67.0↓0.2_ 0.2 42.8↓0.7_ 0.7 48.1↓0.7_ 0.7 87.8↑0.3_ 0.3 1 57.4↓0.5_ 0.5 31.8↓3.4_ 3.4 67.7↑0.5_ 0.5 43.50.0_0.0 48.5↓0.3_ 0.3 87.6↑0.1_ 0.1 54.0↓3.9_ 3.9 33.2↓2.0_ 2.0 65.6↓1.5_ 1.5 42.7↓0.8_ 0.8 47.2↓1.6_ 1.6 86.9↓0.5_ 0.5 2 58.2↑0.3_ 0.3 32.4↓2.8_ 2.8 67.8↑0.6_ 0.6 43.50.0_0.0 48.4↓0.4_ 0.4 87.6↑0.1_ 0.1 54.3↓3.6_ 3.6 32.9↓2.3_ 2.3 67.0↓0.2_ 0.2 42.8↓0.7_ 0.7 46.0↓2.7_ 2.7 86.7↓0.7_ 0.7 3 55.5↓2.4_ 2.4 35.6↑0.4_ 0.4 67.7↑0.5_ 0.5 43.3↓0.2_ 0.2 47.3↓1.4_ 1.4 87.6↑0.1_ 0.1 47.5↓10.4_ 10.4 34.2↓1.0_ 1.0 65.4↓1.8_ 1.8 42.3↓1.2_ 1.2 45.2↓3.5_ 3.5 87.0↓0.4_ 0.4 4 57.1↓0.8_ 0.8 31.9↓3.3_ 3.3 66.7↓0.5_ 0.5 43.4↓0.1_ 0.1 48.2↓0.5_ 0.5 87.50.0_0.0 54.9↓3.0_ 3.0 31.8↓3.4_ 3.4 67.3↑0.1_ 0.1 43.0↓0.5_ 0.5 48.4↓0.4_ 0.4 87.1↓0.3_ 0.3 5 56.6↓1.3_ 1.3 35.7↑0.5_ 0.5 67.3↑0.1_ 0.1 43.3↓0.2_ 0.2 47.5↓1.3_ 1.3 87.5↑0.1_ 0.1 55.0↓2.8_ 2.8 34.3↓1.0_ 1.0 67.6↑0.4_ 0.4 43.1↓0.4_ 0.4 45.9↓2.9_ 2.9 87.5↑0.1_ 0.1 6 52.8↓5.0_ 5.0 33.4↓1.8_ 1.8 67.6↑0.4_ 0.4 43.0↓0.5_ 0.5 48.1↓0.7_ 0.7 87.5↑0.1_ 0.1 50.2↓7.6_ 7.6 32.0↓3.2_ 3.2 65.0↓2.1_ 2.1 41.8↓1.7_ 1.7 45.4↓3.4_ 3.4 85.9↓1.5_ 1.5 7 57.8↓0.1_ 0.1 35.3↑0.1_ 0.1 67.20.0_0.0 43.4↓0.2_ 0.2 47.3↓1.4_ 1.4 87.7↑0.2_ 0.2 56.8↓1.1_ 1.1 31.5↓3.7_ 3.7 67.4↑0.3_ 0.3 43.3↓0.2_ 0.2 46.1↓2.6_ 2.6 87.3↓0.1_ 0.1 8 56.9↓1.0_ 1.0 32.5↓2.7_ 2.7 66.7↓0.5_ 0.5 43.2↓0.3_ 0.3 45.2↓3.5_ 3.5 88.1↑0.6_ 0.6 55.7↓2.2_ 2.2 31.2↓4.0_ 4.0 67.20.0_0.0 43.4↓0.1_ 0.1 43.0↓7.2_ 7.2 87.50.0_0.0 9 56.6↓1.3_ 1.3 35.1↓0.1_ 0.1 67.0↓0.2_ 0.2 43.1↓0.4_ 0.4 47.8↓0.9_ 0.9 87.9↑0.4_ 0.4 55.3↓2.7_ 2.7 32.7↓2.5_ 2.5 67.3↑0.1_ 0.1 43.2↓0.3_ 0.3 46.5↓2.2_ 2.2 87.8↑0.3_ 0.3 10 56.5↓1.4_ 1.4 31.6↓3.7_ 3.7 67.5↑0.3_ 0.3 43.3↓0.2_ 0.2 45.6↓3.1_ 3.1 87.9↑0.4_ 0.4 55.1↓2.8_ 2.8 34.7↓0.5_ 0.5 67.4↑0.3_ 0.3 43.3↓0.2_ 0.2 47.0↓1.7_ 1.7 87.7↑0.2_ 0.2 11 56.9↓1.0_ 1.0 32.4↓2.8_ 2.8 67.7↑0.5_ 0.5 43.2↓0.3_ 0.3 45.2↓3.6_ 3.6 87.8↑0.3_ 0.3 55.2↓2.7_ 2.7 33.2↓2.0_ 2.0 67.6↑0.4_ 0.4 43.0↓0.5_ 0.5 45.9↓2.8_ 2.8 87.6↑0.1_ 0.1 12 57.4↓0.5_ 0.5 33.3↓1.9_ 1.9 67.20.0_0.0 43.3↓0.2_ 0.2 47.8↓0.9_ 0.9 87.6↑0.1_ 0.1 54.9↓3.1_ 3.1 35.0↓0.3_ 0.3 67.20.0_0.0 43.3↓0.2_ 0.2 47.6↓1.2_ 1.2 87.50.0_0.0 13 51.9↓6.0_ 6.0 35.5↑0.3_ 0.3 67.6↑0.4_ 0.4 42.5↓1.0_ 1.0 46.9↓1.8_ 1.8 87.0↓0.4_ 0.4 43.6↓14.2_ 14.2 30.6↓4.6_ 4.6 65.5↓1.6_ 1.6 39.9↓3.6_ 3.6 47.2↓1.6_ 1.6 87.40.0_0.0 14 52.5↓5.4_ 5.4 31.4↓3.9_ 3.9 67.3↑0.1_ 0.1 42.6↓0.9_ 0.9 44.3↓4.4_ 4.4 87.40.0_0.0 55.3↓2.6_ 2.6 34.4↓0.8_ 0.8 66.5↓0.7_ 0.7 43.2↓0.3_ 0.3 47.2↓1.6_ 1.6 87.5↑0.1_ 0.1 15 57.2↓0.7_ 0.7 31.2↓4.0_ 4.0 67.4↑0.2_ 0.2 43.0↓0.6_ 0.6 46.9↓1.8_ 1.8 87.50.0_0.0 56.0↓1.9_ 1.9 36.0↑0.8_ 0.8 67.4↑0.3_ 0.3 42.8↓0.7_ 0.7 49.5↑0.8_ 0.8 87.40.0_0.0 16 56.9↓1.0_ 1.0 35.3↑0.1_ 0.1 67.9↑0.7_ 0.7 42.9↓0.6_ 0.6 48.5↓0.3_ 0.3 87.5↑0.1_ 0.1 58.0↑0.1_ 0.1 31.4↓3.8_ 3.8 66.8↓0.4_ 0.4 43.1↓0.4_ 0.4 47.7↓1.0_ 1.0 87.1↓0.3_ 0.3 17 58.5↑0.6_ 0.6 35.20.0_0.0 67.0↓0.2_ 0.2 43.3↓0.2_ 0.2 48.5↓0.3_ 0.3 87.0↓0.4_ 0.4 57.90.0_0.0 36.2↑1.0_ 1.0 66.7↓0.5_ 0.5 43.3↓0.2_ 0.2 47.1↓1.7_ 1.7 87.6↑0.2_ 0.2 18 57.5↓0.4_ 0.4 34.1↓1.1_ 1.1 67.4↑0.2_ 0.2 43.3↓0.2_ 0.2 47.3↓1.4_ 1.4 87.3↓0.1_ 0.1 56.9↓1.0_ 1.0 31.5↓3.7_ 3.7 66.1↓1.1_ 1.1 43.2↓0.3_ 0.3 47.3↓1.4_ 1.4 87.3↓0.1_ 0.1 19 57.5↓0.4_ 0.4 34.9↓0.3_ 0.3 67.0↓0.2_ 0.2 43.3↓0.2_ 0.2 47.3↓1.4_ 1.4 87.6↑0.1_ 0.1 55.6↓2.3_ 2.3 31.2↓4.0_ 4.0 67.20.0_0.0 43.4↓0.1_ 0.1 46.8↓2.0_ 2.0 87.2↓0.2_ 0.2 20 58.7↑0.8_ 0.8 31.7↓3.6_ 3.6 67.1↓0.1_ 0.1 43.3↓0.2_ 0.2 47.8↓0.9_ 0.9 87.40.0_0.0 56.9↓1.0_ 1.0 32.2↓3.0_ 3.0 67.1↓0.1_ 0.1 43.3↓0.2_ 0.2 48.1↓0.7_ 0.7 87.1↓0.3_ 0.3 21 57.7↓0.2_ 0.2 31.7↓3.5_ 3.5 66.8↓0.4_ 0.4 43.4↓0.1_ 0.1 49.2↑0.4_ 0.4 87.0↓0.4_ 0.4 56.4↓1.5_ 1.5 32.9↓2.3_ 2.3 65.9↓1.3_ 1.3 43.50.0_0.0 47.5↓1.3_ 1.3 87.3↓0.1_ 0.1 22 58.0↑0.1_ 0.1 30.3↓4.9_ 4.9 67.1↓0.1_ 0.1 43.3↓0.2_ 0.2 48.5↓0.3_ 0.3 87.8↑0.3_ 0.3 55.1↓2.8_ 2.8 30.1↓5.1_ 5.1 64.9↓2.3_ 2.3 42.9↓0.6_ 0.6 47.2↓1.6_ 1.6 86.9↓0.5_ 0.5 23 58.7↑0.8_ 0.8 30.2↓5.0_ 5.0 67.6↑0.4_ 0.4 43.50.0_0.0 47.5↓1.3_ 1.3 87.6↑0.2_ 0.2 56.4↓1.4_ 1.4 30.7↓4.5_ 4.5 67.5↑0.3_ 0.3 43.6↑0.1_ 0.1 48.80.0_0.0 87.6↑0.1_ 0.1 (a) All (b) Partial (c) IA3 (d) LayerNorm (e) LoRA r=32r=32 (f) LoRA r=128r=128 (g) LoRA r=512r=512 (h) VeRA Fig. 4: Benchmark scores for all-layer tuning (a) and per-layer PEFT (b–h): Partial-Tuning, IA3, LayerNorm-Tuning, LoRA (r=32/128/512r=32/128/512), and VeRA. TABLE IV: MMStar results of LayerNorm-Tuning, LoRA (r=32r=32), and Partial-Tuning for each layer and all-layer tuning (ALL). Layer Layernorm-Tuning LoRA (r=32r=32) Partial-Tuning OR CP FGP IR LR MA S&T OR CP FGP IR LR MA S&T OR CP FGP IR LR MA S&T 0 43.2 64.0 32.4 50.8 32.9 42.4 36.4 43.4 62.8 32.8 52.0 34.5 40.8 37.6 43.7 64.0 33.2 52.0 34.5 40.8 37.6 1 43.8 63.2 32.4 53.2 34.5 42.4 36.8 43.1 62.8 32.8 51.6 34.5 41.2 35.6 43.3 62.8 32.4 52.8 32.9 41.2 37.6 2 43.6 63.2 34.4 52.0 34.1 40.8 37.2 43.7 62.4 32.0 53.2 34.9 41.6 38.0 43.3 63.6 32.4 52.8 34.5 41.2 35.2 3 43.6 63.2 33.6 51.6 34.5 42.0 36.8 43.4 63.6 32.8 51.6 34.1 42.0 36.4 43.6 64.0 31.2 53.6 34.5 41.2 37.2 4 43.6 63.2 33.2 52.8 34.1 41.2 36.8 43.7 62.4 35.2 52.8 34.1 40.4 37.2 42.8 62.0 31.6 52.8 33.7 40.4 36.4 5 43.8 62.0 32.4 52.8 34.5 42.0 38.8 43.9 63.6 34.0 53.6 33.7 41.2 37.2 43.8 64.0 32.8 52.8 34.5 41.6 37.2 6 43.2 63.2 32.4 52.4 34.5 40.8 35.6 43.6 63.6 34.4 52.0 34.1 41.2 36.4 43.7 63.2 34.0 52.8 33.7 40.4 38.0 7 43.4 63.6 32.0 52.0 34.5 41.2 36.8 43.8 62.8 33.2 52.0 34.5 42.0 38.0 43.4 63.6 33.6 51.2 34.1 40.0 38.0 8 43.4 63.6 32.8 53.2 33.7 41.6 35.6 43.5 63.2 33.6 52.0 34.1 41.2 36.8 43.1 63.2 32.8 51.6 34.5 40.0 36.4 9 43.3 62.8 32.8 52.0 34.5 41.6 36.0 44.1 64.0 34.4 52.4 34.1 41.6 38.0 43.7 62.8 33.2 52.4 34.5 40.8 38.4 10 43.3 63.2 32.4 51.6 34.1 42.0 36.4 43.4 62.8 33.2 52.0 33.7 40.4 38.0 43.4 63.2 32.0 52.4 33.7 40.8 38.0 11 43.7 63.2 32.8 53.6 34.5 41.2 36.8 43.3 63.2 33.6 51.6 34.1 41.6 35.6 43.5 63.2 32.8 52.8 35.3 39.6 37.2 12 43.4 63.6 32.8 52.4 34.1 41.2 36.4 43.4 62.8 33.2 52.4 34.1 40.4 37.6 43.6 63.6 34.0 52.0 34.1 40.4 37.6 13 43.2 62.0 33.2 51.6 33.7 41.2 37.6 43.8 63.2 33.6 53.6 34.1 41.2 36.8 43.4 63.2 32.4 52.8 33.3 41.6 36.8 14 43.3 63.6 32.4 51.6 33.7 40.8 37.6 43.4 62.8 33.2 52.4 34.5 41.6 36.0 43.4 63.2 33.2 52.8 33.7 40.4 36.8 15 43.3 63.2 33.2 51.6 33.7 40.8 37.2 43.9 63.6 33.2 52.8 33.7 42.4 37.6 43.8 62.8 33.2 52.4 34.9 41.6 37.6 16 43.3 63.2 32.4 52.4 32.9 42.0 36.8 43.6 63.6 33.6 52.4 34.1 40.8 37.2 43.5 63.6 33.2 52.0 34.1 41.6 36.4 17 42.9 63.2 32.0 52.0 32.9 40.8 36.4 43.6 63.6 33.6 51.6 34.1 41.6 37.2 42.9 62.8 32.0 51.6 33.3 40.8 36.8 18 43.7 63.2 32.8 52.4 34.9 41.2 37.6 43.8 63.6 32.8 52.8 34.5 40.8 38.0 42.9 62.8 32.0 52.0 34.1 41.2 35.2 19 43.6 64.0 32.0 52.4 34.1 41.2 37.6 43.5 62.4 34.0 51.2 34.1 42.0 37.2 43.6 63.6 32.4 53.2 34.1 42.0 36.0 20 43.4 63.2 31.6 52.8 34.5 40.4 38.0 43.4 63.2 32.8 51.2 34.9 41.6 36.8 43.7 63.2 33.2 53.2 33.7 40.8 38.0 21 43.5 62.8 32.8 52.4 33.7 42.0 37.2 43.6 62.8 33.6 51.2 34.1 41.2 38.8 43.4 62.8 33.2 52.0 34.9 40.8 36.4 22 43.4 63.2 32.8 53.2 33.7 40.8 36.8 43.6 64.0 33.2 51.6 34.5 41.2 36.8 43.5 63.6 32.4 52.0 33.7 41.6 37.6 23 43.6 62.0 34.0 52.0 34.5 41.6 37.2 43.4 62.0 32.8 52.0 34.1 42.0 37.2 43.4 63.6 30.8 52.0 34.1 41.2 38.4 ALL 43.2 63.6 32.8 52.4 32.9 40.8 36.4 43.6 62.8 33.2 53.6 34.1 41.6 36.0 43.6 62.4 33.6 51.6 35.7 41.6 36.8 IV-E Paired Significance Testing on Aggregated Benchmarks We conduct significance testing on the pooled six-benchmark set (ChartQA, HallusionBench, MMBench, OCRVQA, RealWorldQA, and ScienceQA) using a two-sided McNemar test and a paired percentile bootstrap 95% CI (27k resamples) for the accuracy difference Δ . VeRA (layer 1) improves pooled accuracy by Δ=+2.14 =+2.14p (95% CI [1.63, 2.64][1.63,\,2.64]p; p=1.2×10−16p=1.2× 10^-16), and LoRA (layer 2) by Δ=+2.08 =+2.08p (95% CI [1.57, 2.58][1.57,\,2.58]p; p=8.236×10−16p=8.236× 10^-16), mainly driven by HallusionBench and RealWorldQA; OCRVQA slightly decreases for VeRA (−1.23-1.23p; p=0.1038p=0.1038). Partial-Tuning (layer 13) also yields a small HallusionBench gain (Δ=+1.58 =+1.58p; p=0.02007p=0.02007). In summary, probing-selected layers provide statistically significant gains on the pooled benchmarks, with improvements concentrated on HallusionBench and RealWorldQA. Across both backbones and multiple PEFT families, perturbation-robust layers are distributed in both shallow and deep blocks and usually deliver stronger PEFT gains, supporting pre-PEFT probing as a practical layer-selection signal. V Conclusion We study pre-PEFT layer selection for VLM vision encoders through Q/K/V weight statistics and perturbation probing. Across two backbones, layers that are more robust to perturbations tend to yield better PEFT outcomes, often together with larger weight norms and higher condition numbers in our evaluated settings. These results suggest that robustness-guided selection of a small subset of layers can reduce trainable parameters while remaining competitive with broader tuning in limited-data regimes. Acknowledgment This work is partly supported by the Project granted by the Ministry of Agriculture and Rural Affairs, the Key Research and Development Program of Heilongjiang Province under Grant Nos. 2022ZX01A22, 2021ZXJ05A03, the National Natural Science Foundation of China under Grant Nos. 62350710797, 61972114, 62106061, the National Science and Technology Major Project of China under Grant Nos. 2021ZD0110901, the collaborative innovation and promotion system of the modern agricultural industry technology for watermelon and melon in Heilongjiang Province. References [1] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024) InternVL: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24185–24198. Cited by: §I, §I-A. [2] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2024) Visual instruction tuning. Advances in neural information processing systems 36. Cited by: §I, §I-A. [3] A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa, et al. (2023) OpenFlamingo: an open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390. Cited by: §I. [4] S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao (2024) Large language models: a survey. arXiv preprint arXiv:2402.06196. Cited by: §I. [5] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen (2023) A survey on multimodal large language models. arXiv preprint arXiv:2306.13549. Cited by: §I. [6] J. Zhang, J. Huang, S. Jin, and S. Lu (2024) Vision-language models for vision tasks: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §I. [7] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §I, §I-B, §I-B. [8] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019) Parameter-efficient transfer learning for nlp. In International conference on machine learning, p. 2790–2799. Cited by: §I, §I-B. [9] R. Zhang, J. Han, C. Liu, P. Gao, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, and Y. Qiao (2023) LLaMA-adapter: efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199. Cited by: §I. [10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, External Links: Link Cited by: §I. [11] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §I-A. [12] OpenGVLab Team (2024) InternVL2. Note: Online External Links: Link Cited by: §I-A. [13] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, et al. (2024) Qwen2 technical report. arXiv preprint arXiv:2407.10671. Cited by: §I-A. [14] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. (2025) Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923. Cited by: §I-A, 1st item. [15] M. Jia, L. Tang, B. Chen, C. Cardie, S. Belongie, B. Hariharan, and S. Lim (2022) Visual prompt tuning. In European Conference on Computer Vision, p. 709–727. Cited by: §I-B. [16] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30, p. 5998–6008. External Links: Link Cited by: §I-B. [17] D. Coppola and H. K. Lee (2024) Investigating and unmasking feature-level vulnerabilities of cnns to adversarial perturbations. arXiv preprint arXiv:2405.20672. Cited by: §I-B. [18] F. N. Nokabadi, J. Lalonde, and C. Gagné (2024) Reproducibility study on adversarial attacks against robust transformer trackers. arXiv preprint arXiv:2406.01765. Cited by: §I-B. [19] B. Wang, G. Li, X. Zhou, Z. Chen, T. Grossman, and Y. Li (2021) Screen2words: automatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, p. 498–510. Cited by: §I-A. [20] A. Mishra, S. Shekhar, A. K. Singh, and A. Chakraborty (2019) OCR-VQA: visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), p. 947–952. Cited by: §I-A, §I-A. [21] S. Kantharaj, R. T. K. Leong, X. Lin, A. Masry, M. Thakkar, E. Hoque, and S. Joty (2022) Chart-to-text: a large-scale benchmark for chart summarization. arXiv preprint arXiv:2203.06486. Cited by: §I-A. [22] A. D. Lindström and S. S. Abraham (2022) CLEVR-Math: a dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358. Cited by: §I-A. [23] P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan (2022) Learn to explain: multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, Vol. 35. External Links: Link Cited by: §I-A, §I-A. [24] L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, et al. (2024) Are we on the right way for evaluating large vision-language models?. arXiv preprint arXiv:2403.20330. Cited by: §I-A. [25] A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque (2022) ChartQA: a benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244. Cited by: §I-A. [26] T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, et al. (2024) HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14375–14385. Cited by: §I-A. [27] xAI (2024) RealWorldQA. Note: Hugging Face External Links: Link Cited by: §I-A. [28] Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. (2023) MMBench: is your multi-modal model an all-around player?. arXiv preprint arXiv:2307.06281. Cited by: §I-A. [29] D. J. Kopiczko, T. Blankevoort, and Y. M. Asano (2023) VeRA: vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454. Cited by: §I-B. [30] H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel (2022) Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems 35, p. 1950–1965. Cited by: §I-B. [31] K. He, R. Girshick, and P. Dollár (2019) Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4918–4927. Cited by: §I-B. [32] X. Glorot, A. Bordes, and Y. Bengio (2011) Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, p. 315–323. Cited by: §I-B.

Instant research discovery