Paper deep dive

EigenShield: Causal Subspace Filtering via Random Matrix Theory for Adversarially Robust Vision-Language Models

Nastaran Darabi, Devashri Naik, Sina Tayebati, Dinithi Jayasuriya, Ranganath Krishnan

Year: 2025Venue: arXiv preprintArea: Multimodal SafetyType: EmpiricalEmbeddings: 134

Models: Florence-2-large, InstructBLIP, LLaVA-v1.5-7B, MiniGPT-4, Qwen2-VL

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/12/2026, 5:32:52 PM

Summary

EigenShield is an inference-time defense framework for Vision-Language Models (VLMs) that utilizes Random Matrix Theory (RMT) to filter adversarial noise. By employing the spiked covariance model and a Robustness-based Nonconformity Score (RbNS), it identifies and projects embeddings onto a causal subspace, effectively separating semantic information from adversarial artifacts without requiring model retraining or parameter modification.

Entities (5)

EigenShield · defense-framework · 100%Random Matrix Theory · mathematical-framework · 100%Vision-Language Models · ai-model-architecture · 100%Robustness-based Nonconformity Score · algorithmmetric · 95%Spiked Covariance Model · statistical-model · 95%

Relation Signals (3)

EigenShield → leverages → Random Matrix Theory

confidence 100% · EigenShield, an inference-time defense leveraging Random Matrix Theory

EigenShield → employs → Robustness-based Nonconformity Score

confidence 95% · Using a Robustness-based Nonconformity Score (RbNS) and quantile-based thresholding, it separates causal eigenvectors

EigenShield → protects → Vision-Language Models

confidence 95% · EigenShield: Causal Subspace Filtering via Random Matrix Theory for Adversarially Robust Vision-Language Models

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Vision-Language Models (VLMs) inherit adversarial vulnerabilities of Large Language Models (LLMs), which are further exacerbated by their multimodal nature. Existing defenses, including adversarial training, input transformations, and heuristic detection, are computationally expensive, architecture-dependent, and fragile against adaptive attacks. We introduce EigenShield, an inference-time defense leveraging Random Matrix Theory to quantify adversarial disruptions in high-dimensional VLM representations. Unlike prior methods that rely on empirical heuristics, EigenShield employs the spiked covariance model to detect structured spectral deviations. Using a Robustness-based Nonconformity Score (RbNS) and quantile-based thresholding, it separates causal eigenvectors, which encode semantic information, from correlational eigenvectors that are susceptible to adversarial artifacts. By projecting embeddings onto the causal subspace, EigenShield filters adversarial noise without modifying model parameters or requiring adversarial training. This architecture-independent, attack-agnostic approach significantly reduces the attack success rate, establishing spectral analysis as a principled alternative to conventional defenses. Our results demonstrate that EigenShield consistently outperforms all existing defenses, including adversarial training, UNIGUARD, and CIDER.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

133,963 characters extracted from source content.

Expand or collapse full text

EigenShield: Causal Subspace Filtering via Random Matrix Theory for Adversarially Robust Vision-Language Models Nastaran Darabi1, Devashri Naik1, Sina Tayebati1, Dinithi Jayasuriya1, Ranganath Krishnan2, Amit Ranjan Trivedi1 1Department of Electrical and Computer Engineering, University of Illinois at Chicago, IL, USA 2Intel Labs, Hillsboro, OR, USA We acknowledge Tejaswi Tripathi from the Department of Mathematics, University of Michigan, for his valuable insights. This work was supported in part by COGNISENSE, one of seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA, and NSF funding #2235207. Corresponding Authors Email: ndarab2@uic.edu, amitrt@uic.edu Abstract Vision-Language Models (VLMs) inherit adversarial vulnerabilities of Large Language Models (LLMs), which are further exacerbated by their multimodal nature. Existing defenses, including adversarial training, input transformations, and heuristic detection, are computationally expensive, architecture-dependent, and fragile against adaptive attacks. We introduce EigenShield, an inference-time defense leveraging Random Matrix Theory to quantify adversarial disruptions in high-dimensional VLM representations. Unlike prior methods that rely on empirical heuristics, EigenShield employs the spiked covariance model to detect structured spectral deviations. Using a Robustness-based Nonconformity Score (RbNS) and quantile-based thresholding, it separates causal eigenvectors, which encode semantic information, from correlational eigenvectors that are susceptible to adversarial artifacts. By projecting embeddings onto the causal subspace, EigenShield filters adversarial noise without modifying model parameters or requiring adversarial training. This architecture-independent, attack-agnostic approach significantly reduces the attack success rate, establishing spectral analysis as a principled alternative to conventional defenses. Our results demonstrate that EigenShield consistently outperforms all existing defenses including adversarial training, UNIGUARD, and CIDER. Warning: This paper contains data, prompts, and model outputs that are offensive in nature. I Introduction and Related Works Vision-Language Models (VLMs) have extended Large Language Models (LLMs) by integrating visual understanding, thereby enabling applications like image captioning and visual question answering [1, 2]. Despite these advancements, VLMs remain highly vulnerable to adversarial “jailbreak” attacks [3, 4], where imperceptible perturbations in images or text can manipulate outputs, leading to harmful, undesirable, or policy-violating responses [5]. Their multimodal nature specifically expands this attack surface [5], as the fusion of continuous visual signals and discrete linguistic representations introduces exploitable inconsistencies. This interplay enables novel attacks, such as cross-modal misalignment, where conflicting vision and text inputs induce failure modes. Moreover, securing one modality does not necessarily mitigate vulnerabilities in the other, making defense challenging. VLM jailbreaks can generate toxic content, biased outputs, or execute unsafe instructions, posing risks for real-world deployment [6]. Prior works have demonstrated attacks in three main categories: adversarial perturbation-based attacks [7], prompt manipulation [8], and proxy model transfer jailbreaks [5, 4]. Adversarial perturbation-based attacks introduce subtle modifications to images or text to bypass alignment mechanisms, leading to harmful outputs [6, 9]. Prompt manipulation strategically modifies text inputs to exploit model biases and evade safety constraints [8]. Proxy model transfer jailbreaks leverage vulnerabilities in surrogate models to generate adversarial examples that generalize to stronger VLMs [5, 4]. Moreover, these attack vectors are not isolated; adversaries can strategically combine them to create more potent, adaptive jailbreaks by compounding vulnerabilities. Current defense strategies fall into proactive (training-based) and reactive (inference-time) approaches. Adversarial training improves robustness proactively by exposing models to adversarial examples but is computationally demanding and requires careful parameter tuning [3]. UNIGUARD [10], a state-of-the-art proactive defense, establishes universal safety guardrails for Multimodal LLMs (MLLMs) by optimizing modality-specific risk mitigation. Inference-time reactive defenses operate without modifying the model, using methods such as image transformations (e.g., blurring, compression) or diffusion-based purification (e.g., DiffPure [11]), though these can degrade clean input performance or be computationally expensive. Detection mechanisms, such as Jailguard [12] and CIDER [13], identify adversarial inputs by analyzing perturbations or cross-modal inconsistencies. However, existing defenses face several fundamental limitations. They focus on individual samples rather than modeling adversarial patterns globally, making them ineffective against adaptive attacks that exploit multimodal inconsistencies [5]. Moreover, heuristic defenses relying on fine-tuned hyperparameters often fail under evolving threats, as seen in adversarial transferability across models [4]. Studies have shown that adversarial perturbations introduce structured noise that systematically alters the spectral properties of model activations [6]. Crucially, these perturbations do not behave randomly but follow measurable statistical patterns—suggesting that their spectral signatures can be leveraged to design a principled approach to distinguishing clean from adversarial data. Building on this insight, we introduce EigenShield, a defense framework rooted in Random Matrix Theory (RMT) to model spectral behavior in high-dimensional systems by distinguishing signal-dominant regimes from noise-induced disruptions. Leveraging spectral distribution principles from RMT, EigenShield systematically detects and mitigates adversarial perturbations by analyzing spectral deviations in feature representations. Unlike adversarial training, which enforces robustness through injected perturbations, EigenShield operates on first principles rather than empirical heuristics. Specifically, we present EigenShield’s framework by leveraging the spiked covariance model to separate adversarial noise from meaningful signal components in VLM feature representations. By introducing a Robustness-based Nonconformity Score (RbNS) and quantile-based thresholding, EigenShield distinguishes causal eigenvectors, which are semantically meaningful, from correlational eigenvectors that encode adversarial artifacts. The framework operates entirely at the input level, projecting embeddings onto the subspace of causal eigenvectors to filter adversarial noise without modifying the VLM’s architecture or requiring adversarial training. This makes EigenShield architecture-independent, requiring no retraining, theoretically grounded with guarantees via eigenvalue distributions, attack-agnostic, and computationally efficient. Figure 1: Overview of EigenShield: Jailbreak attacks attempt to produce harmful responses from VLMs by manipulating inputs. EigenShield intervenes by filtering input embeddings via a causal subspace. EigenShield’s defense mechanisms are architecture-independent, requiring no retraining, theoretically grounded in Random Matrix Theory with guarantees via eigenvalue distributions, attack-agnostic, and computationally efficient. I Foundations from Random Matrix Theory RMT provides a powerful analytical framework for characterizing the spectral properties of large random matrices, particularly suitable in high-dimensional data regimes where the disentanglement of genuine signals from pervasive noise forms a fundamental challenge [14, 15, 16]. We first briefly discuss the key concepts from RMT that form the basis of EigenShield. Wigner Semicircle Law and the Asymptotic Noise Floor: Wigner Semicircle Law characterizes the asymptotic eigenvalue distribution of large symmetric random matrices with independent and identically distributed (i.i.d.) entries [17]. Let WW be a p×p× p × p real symmetric random matrix, where the upper triangular entries Wi⁢jsubscriptW_ijWitalic_i j (for 1≤i≤j≤p11≤ i≤ j≤ p1 ≤ i ≤ j ≤ p) are i.i.d. with zero mean and variance σ2superscript2σ^2σ2, and the lower triangular entries are defined by symmetry, Wj⁢i=Wi⁢jsubscriptsubscriptW_ji=W_ijWitalic_j i = Witalic_i j. As the matrix size p approaches infinity, the empirical spectral distribution (ESD) of WW, representing the eigenvalues λ1,λ2,…,λpsubscript1subscript2…subscript _1, _2,…, _pλ1 , λ2 , … , λitalic_p, almost surely converges to the Wigner Semicircle distribution. The probability density function (PDF) of this limiting distribution is given by: fWigner⁢(λ)=12⁢π⁢σ2⁢4⁢σ2−λ2,if ⁢|λ|≤2⁢σ0,otherwisesubscriptWignercases12superscript24superscript2superscript2if 20otherwisef_Wigner(λ)= cases 12πσ^2 4σ^% 2-λ^2,&if |λ|≤ 2σ\\ 0,&otherwise casesfWigner ( λ ) = start_ROW start_CELL divide start_ARG 1 end_ARG start_ARG 2 π σ2 end_ARG square-root start_ARG 4 σ2 - λ2 end_ARG , end_CELL start_CELL if | λ | ≤ 2 σ end_CELL end_ROW start_ROW start_CELL 0 , end_CELL start_CELL otherwise end_CELL end_ROW (1) This result demonstrates that for purely random matrices, the eigenvalues are not arbitrarily distributed but are confined within a bounded interval [−2⁢σ,2⁢σ]22[-2σ,2σ][ - 2 σ , 2 σ ], forming a well-defined “noise floor” in the eigenvalue spectrum. This theoretical noise floor serves as a critical benchmark for identifying eigenvalues that significantly deviate, potentially indicating the presence of non-random signal components. A simplified proof for this is provided in Appendix A. For the sample covariance matrix, which is central to many statistical applications, a related result is the Marchenko-Pastur distribution. Consider an n×pn× pn × p data matrix XX with i.i.d. entries of zero mean and variance σ2superscript2σ^2σ2. In the high-dimensional limit, where n,p→∞→n,p→∞n , p → ∞ with a fixed aspect ratio c=p/n>00c=p/n>0c = p / n > 0, the empirical spectral distribution (ESD) of the sample covariance matrix =1n⁢T⁢1superscriptC= 1nX^TXC = divide start_ARG 1 end_ARG start_ARG n end_ARG Xitalic_T X converges to the Marchenko-Pastur distribution. The distribution support lies within [σ2⁢(1−c)2,σ2⁢(1+c)2]superscript2superscript12superscript2superscript12[σ^2(1- c)^2,σ^2(1+ c)^2][ σ2 ( 1 - square-root start_ARG c end_ARG )2 , σ2 ( 1 + square-root start_ARG c end_ARG )2 ] [18]. Spiked Covariance Model and Signal Detection via Eigenvalue Outliers: To model scenarios where a low-rank informative signal is embedded within high-dimensional noise, we use the spiked covariance model [19]. This model assumes the true covariance matrix Σ is decomposed into: =signal+noisesubscriptsignalsubscriptnoise = _signal+ _noiseΣ = Σsignal + Σnoise. Here, signalsubscriptsignal _signalΣsignal represents a structured, low-rank signal component of rank k≪pmuch-less-thank pk ≪ p, and noise=σ2⁢psubscriptnoisesuperscript2subscript _noise=σ^2I_pΣnoise = σ2 Iitalic_p represents isotropic noise, where psubscriptI_pIitalic_p is the p×p× p × p identity matrix and σ2superscript2σ^2σ2 is the noise variance. The term “spikes” refers to eigenvalues of the sample covariance matrix CC that deviate significantly from the bulk distribution described by the Wigner Semicircle Law or Marchenko-Pastur distribution, indicating the presence of signalsubscriptsignal _signalΣsignal. If signalsubscriptsignal _signalΣsignal has the eigendecomposition signal=⁢Tsubscriptsignalsuperscript _signal=V ^TΣsignal = V Λ Vitalic_T, where VV is a p×kp× kp × k matrix with orthonormal columns spanning the signal subspace, and =diag⁢(λ1′,…,λk′)diagsubscriptsuperscript′1…subscriptsuperscript′ =diag(λ _1,…,λ _k)Λ = diag ( λ′1 , … , λ′italic_k ) contains the signal eigenvalues (λ1′≥…≥λk′>0subscriptsuperscript′1…subscriptsuperscript′0λ _1≥…≥λ _k>0λ′1 ≥ … ≥ λ′italic_k > 0), then under certain conditions, RMT predicts the emergence of outlier eigenvalues. Specifically, when λi′>σ2⁢csubscriptsuperscript′2λ _i>σ^2 cλ′italic_i > σ2 square-root start_ARG c end_ARG for i=1,…,k1…i=1,…,ki = 1 , … , k, asymptotic theory guarantees the existence of k outlier eigenvalues λ^1,…,λ^ksubscript^1…subscript λ_1,…, λ_kover start_ARG λ end_ARG1 , … , over start_ARG λ end_ARGk of CC. These outliers lie outside the upper edge of the Marchenko-Pastur bulk and are closely related to the underlying signal eigenvalues λi′subscriptsuperscript′λ _iλ′italic_i. These “spiked” eigenvalues and their corresponding eigenvectors capture the dominant signal directions, enabling the separation of signal from noise in high-dimensional data. We provide a more detailed discussion in Appendix B. RMT-based Decomposition for Causal Subspace Extraction: To extract “causal” subspaces using the spiked covariance principle, we formulate an optimization problem to decompose the empirical covariance matrix ΣnsubscriptΣ _nΣitalic_n into a low-rank signal component and an isotropic noise component. This is achieved by minimizing the RMT Loss function: ℒRMT⁢(U,Λ)=∥Σn−(U⁢Λ⁢U⊤+σ2⁢IP)∥22.subscriptℒRMTΛsuperscriptsubscriptdelimited-∥subscriptΣΛsuperscripttopsuperscript2subscript22L_RMT(U, )\;=\; \|\, _n\;-\; (U% \, \,U \;+\;σ^2I_P ) \|_2^2.Lroman_RMT ( U , Λ ) = ∥ Σitalic_n - ( U Λ U⊤ + σ2 Iitalic_P ) ∥22 . (2) Here, U∈ℝP×rsuperscriptℝU ^P× rU ∈ blackboard_RP × r is a matrix with orthonormal columns representing the top-r principal (causal) directions, Λ∈ℝr×rΛsuperscriptℝ ^r× rΛ ∈ blackboard_Rr × r is a diagonal matrix of signal eigenvalues, and σ2⁢IPsuperscript2subscriptσ^2I_Pσ2 Iitalic_P represents the isotropic noise covariance. The Euclidean norm ∥⋅∥2\|·\|_2∥ ⋅ ∥2 quantifies the difference between the empirical covariance and the model. Minimizing ℒRMTsubscriptℒRMTL_RMTLroman_RMT performs a principled decomposition of ΣnsubscriptΣ _nΣitalic_n into a dominant signal subspace U⁢Λ⁢U⊤ΛsuperscripttopU U U Λ U⊤ and a residual noise complement. This process identifies components that are causal based on their signal strength rather than correlational or noise-driven influences. Specifically, we designate the eigenvalues with the highest magnitudes as “causal,” under the premise that these eigenvalues, derived from disentangled causal factors, are most strongly linked to the output label and exert the greatest influence on output class determination. We provide a more detailed discussion in Appendix C. Figure 2: EigenShield’s effect on adversarial representations. (a) Eigenvalue spectrum comparison shows restoration of adversarial images to clean-like distributions. (b) Heatmap visualization depicts eigenvalue variability across 100 images. (c) Eigenvector classification via RbNS separates robust causal (blue) from spurious components (red) using threshold τ∗superscriptτ^*τ∗. I Random Matrix Theory–Rooted Safety of Vision-Language Models: EigenShield In adversarial jailbreak attacks against VLMs, we hypothesize that adversarial perturbations, though visually imperceptible, introduce statistically anomalous signal components into the input image, perturbing the eigenvalue spectrum and deviating from expected noise characteristics defined by RMT. In Fig. 1, EigenShield leverages these insights through three key steps: (i) Eigenvalue Decomposition of Input Images: Each input undergoes eigenvalue decomposition, extracting principal spectral components. (i) Causal Eigenvalue Hypothesis and Thresholding via RbNS: Higher-magnitude eigenvalues correspond to semantically relevant components, while lower-magnitude ones likely represent noise or adversarial artifacts. Using the Robustness-based Nonconformity Score (RbNS) and quantile-based thresholding, we determine an optimal threshold to separate causal from correlational eigenvalues. (i) Filtering via Projection onto the Causal Subspace: At inference, eigenvalue decomposition identifies causal eigenvalues exceeding the threshold, whose eigenvectors define a causal subspace. The input is then projected onto this subspace, preserving causal components while filtering spurious correlations. This approach selectively amplifies input image components associated with high-magnitude, causally relevant eigenvalues while mitigating adversarial noise and spurious correlations. Based on this, the proposed methodology operates in two sequential phases. Phase-1: Threshold Determination via RbNS and RMT Analysis and Phase-2: Inference-Time Jailbreak Defense. Key components are discuss below: Phase 1: Threshold Determination via Robustness-based Nonconformity Score (RbNS) This phase introduces a data-driven approach to determine the optimal threshold τ∗superscriptτ^*τ∗, which separates causal and correlational eigenvalues. The threshold is derived using the Robustness-based Nonconformity Score (RbNS), which evaluates the stability and predictive relevance of eigenvector directions using the following procedures: (i) Extracting Outlier Directions via RMT: We first apply RMT principles to identify outlier eigenvectors =v1,v2,…,v||subscript1subscript2…subscriptO=\v_1,v_2,…,v_|O|\O = v1 , v2 , … , v| O | that deviate from the expected Marchenko-Pastur bulk. Each eigenvector vj∈subscriptv_j _j ∈ O is associated with an eigenvalue λjsubscript _jλitalic_j exceeding the noise floor, which indicates a potential signal-bearing component. (i) RbNS Calculation: For each vj∈subscriptv_j _j ∈ O, we compute an RbNS score αjsubscript _jαitalic_j to quantify its robustness. A lower αjsubscript _jαitalic_j indicates a causal direction, while a higher αjsubscript _jαitalic_j suggests a spurious/noisy component. The calculation follows as: A. 1-D Projection and Distribution Learning: We perform eigenvalue decomposition on each input image isubscriptx_ixitalic_i in the validation set valsubscriptvalD_valDval, obtaining eigenvectors vxi,1,vxi,2,…,vxi,Psubscriptsubscript1subscriptsubscript2…subscriptsubscript\v_x_i,1,v_x_i,2,…,v_x_i,P\ vitalic_x start_POSTSUBSCRIPT i , 1 end_POSTSUBSCRIPT , vitalic_x start_POSTSUBSCRIPT i , 2 end_POSTSUBSCRIPT , … , vitalic_x start_POSTSUBSCRIPT i , P end_POSTSUBSCRIPT . Each eigenvector vxi,jsubscriptsubscriptv_x_i,jvitalic_x start_POSTSUBSCRIPT i , j end_POSTSUBSCRIPT is projected onto the outlier direction vjsubscriptv_jvitalic_j: ui,j=vxi,j⊤⁢vjsubscriptsuperscriptsubscriptsubscripttopsubscriptu_i,j=v_x_i,j v_juitalic_i , j = vitalic_x start_POSTSUBSCRIPT i , j end_POSTSUBSCRIPT⊤ vitalic_j. We then use a Variational Autoencoder (VAE) g^j⁢(ui,j)subscript^subscript g_j(u_i,j)over start_ARG g end_ARGj ( uitalic_i , j ) to estimate the conditional distribution of reconstructed images given ui,jsubscriptu_i,juitalic_i , j. Directions associated with high eigenvalues are expected to yield lower entropy in the distributions, indicating a stronger causal relevance. B. Cross-Validation Subsampling: We assess the robustness of the VAE predictor g^jsubscript g_jover start_ARG g end_ARGj using 10-fold cross-validation. The validation set valsubscriptvalD_valDval is split into K=1010K=10K = 10 folds, with each g^j(k)superscriptsubscript g_j^(k)over start_ARG g end_ARGj( k ) trained on K−11K-1K - 1 folds and evaluated on the held-out fold. Performance is measured using the Kullback-Leibler (KL) divergence between the distribution of the original input features and the distribution of reconstructed features from a given VAE. More formally, we pass input images through a pre-trained encoder to get the feature representations zisubscriptz_izitalic_i. We denote the distribution of the features zisubscriptz_izitalic_i on the validation set as PzisubscriptsubscriptP_z_iPitalic_z start_POSTSUBSCRIPT i end_POSTSUBSCRIPT, and the reconstructed features from the VAE as z^isubscript z_iover start_ARG z end_ARGi. The VAE provides a model for the distribution of z^isubscript z_iover start_ARG z end_ARGi, namely Qz^isubscriptsubscript^Q_ z_iQover start_ARG z end_ARG start_POSTSUBSCRIPT i end_POSTSUBSCRIPT, given the projection ui,j(k)superscriptsubscriptu_i,j^(k)uitalic_i , j( k ). We then calculate KL divergence using Perfj,k=KL⁢(Pzi∥Qz^i|ui,j(k)).subscriptPerfKLconditionalsubscriptsubscriptsubscriptconditionalsubscript^superscriptsubscriptPerf_j,k=KL(P_z_i Q_ z_i|u_i,j^(k)).Perfj , k = KL ( Pitalic_z start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ∥ Qover start_ARG z end_ARG start_POSTSUBSCRIPT i | uitalic_i , j( k ) end_POSTSUBSCRIPT ) . C. Robustness Statistic Computation: To quantify the stability of each eigenvector direction vjsubscriptv_jvitalic_j, we compute its robustness ρjsubscript _jρitalic_j using the KL divergence performance distribution. Specifically, ρjsubscript _jρitalic_j is calculated as the difference between the median and a lower quantile (q=0.100.10q=0.10q = 0.10) of the KL divergence distribution over the cross-validation folds: ρj=Median⁢(Perfj,kk=1K)−LQuantileq⁢(Perfj,kk=1K).subscriptMediansuperscriptsubscriptsubscriptPerf1subscriptLQuantilesuperscriptsubscriptsubscriptPerf1 _j=Median(\Perf_j,k\_k=1^K)-LQuantile_q(% \Perf_j,k\_k=1^K).ρitalic_j = Median ( Perfj , k k = 1K ) - LQuantileq ( Perfj , k k = 1K ) . (3) In this context, a smaller ρjsubscript _jρitalic_j indicates a more stable and robust direction. This implies greater consistency in distribution matching across different data partitions. D. Nonconformity Score Computation: Finally, the nonconformity score αjsubscript _jαitalic_j is computed as the exponent of ρjsubscript _jρitalic_j, αj=eρjsubscriptsuperscriptsubscript _j=e _jαitalic_j = eitalic_ρitalic_j. Smaller αjsubscript _jαitalic_j values indicate causal directions, while larger αjsubscript _jαitalic_j values correspond to spurious or noisy signal components. (i) Quantile-Based Thresholding for Causal Eigenvalue Selection: Given the computed nonconformity scores αjvj∈subscriptsubscriptsubscript\ _j\_v_j αitalic_j v start_POSTSUBSCRIPT j ∈ O end_POSTSUBSCRIPT for all outlier directions, we employ quantile-based thresholding to determine the optimal eigenvalue classification threshold τ∗superscriptτ^*τ∗. A coverage parameter γ∈[0,1]01γ∈[0,1]γ ∈ [ 0 , 1 ] controls the retention of “causal” directions. Fig. 3 presents an evaluation of γ values ranging from 0.5 to 0.9, with γ=0.750.75γ=0.75γ = 0.75 selected as the optimal choice. The figure demonstrates that as γ increases, more eigenvalues are incorporated into the causal subspace, leading to an overall improvement in EigenShield’s performance. However, a higher γ also results in an increased number of trainable parameters. The nonconformity threshold t^ tover start_ARG t end_ARG is set as the γ-th quantile of the nonconformity score distribution: t^=quantile(γ)⁢(αjvj∈).^subscriptquantilesubscriptsubscriptsubscript t=quantile_(γ) (\ _j\_v_j % ).over start_ARG t end_ARG = quantile( γ ) ( αitalic_j v start_POSTSUBSCRIPT j ∈ O end_POSTSUBSCRIPT ) . A direction vjsubscriptv_jvitalic_j (with eigenvalue λjsubscript _jλitalic_j) is classified as “causal” if its nonconformity score satisfies αj≤t^subscript _j≤ tαitalic_j ≤ over start_ARG t end_ARG. The final threshold τ∗superscriptτ^*τ∗ is then set as the minimum eigenvalue among all causal directions. We prioritize higher-magnitude eigenvalues, aligning with our hypothesis that these correspond to disentangled causal components with the strongest influence on the output label. Phase 2: Inference-Time Jailbreak Defense Once the optimal threshold τ∗superscriptτ^*τ∗ is determined in Phase 1, we deploy EigenShield at inference to protect the VLM from adversarial jailbreak attacks. Given an input image xinputsubscriptinputx_inputxinput, we first compute its eigenvalue decomposition, obtaining the eigenvalue set ΛxinputsubscriptΛsubscriptinput _x_inputΛitalic_x start_POSTSUBSCRIPT input end_POSTSUBSCRIPT. Each eigenvalue λxinput,i∈ΛxinputsubscriptsubscriptinputsubscriptΛsubscriptinput _x_input,i∈ _x_inputλitalic_x start_POSTSUBSCRIPT input , i end_POSTSUBSCRIPT ∈ Λitalic_x start_POSTSUBSCRIPT input end_POSTSUBSCRIPT is classified as causal if λxinput,i>τ∗subscriptsubscriptinputsuperscript _x_input,i>τ^*λitalic_x start_POSTSUBSCRIPT input , i end_POSTSUBSCRIPT > τ∗, while lower values correspond to correlational or noise components. Input image embedding xinputsubscriptsubscriptinpute_x_inputeitalic_x start_POSTSUBSCRIPT input end_POSTSUBSCRIPT is then projected onto the causal subspace causal∗superscriptsubscriptcausalE_causal^*Ecausal∗, spanned by eigenvectors associated with causal eigenvalues, yielding the filtered embedding: xinputfiltered=causal∗⁢xinput,wherecausal∗=causal∗⁢(causal∗)T.formulae-sequencesuperscriptsubscriptsubscriptinputfilteredsuperscriptsubscriptcausalsubscriptsubscriptinputwheresuperscriptsubscriptcausalsuperscriptsubscriptcausalsuperscriptsuperscriptsubscriptcausale_x_input^filtered=P_causal^*% e_x_input, _causal% ^*=E_causal^*(E_causal^*)^T.eitalic_x start_POSTSUBSCRIPT input end_POSTSUBSCRIPTfiltered = Pcausal∗ eitalic_x start_POSTSUBSCRIPT input end_POSTSUBSCRIPT , where Pcausal∗ = Ecausal∗ ( Ecausal∗ )T . (4) Finally, the filtered embedding xinputfilteredsuperscriptsubscriptsubscriptinputfilterede_x_input^filteredeitalic_x start_POSTSUBSCRIPT input end_POSTSUBSCRIPTfiltered is passed to the VLM alongside the original text prompt for inference. By projecting embeddings onto the causal subspace defined by τ∗superscriptτ^*τ∗, EigenShield effectively removes spurious correlations and adversarial perturbations while preserving dominant semantic information. This input-level filtering enhances VLM robustness against jailbreak attacks without modifying the model architecture or incurring significant computational overhead, making it a practical and efficient defense. Fig. 2 presents a comprehensive analysis of EigenShield’s processing through eigenvalue and robustness-based evaluations. Fig. 2(a) compares the eigenvalue spectra of clean, adversarial, and EigenShield-processed images, revealing how the method modifies eigenvalue distributions to mitigate adversarial perturbations. The log-scale representation highlights differences in magnitude across sorted eigenvalue indices. Fig. 2(b) visualizes the variability of eigenvalues across 100 randomly selected EigenShield-processed images, demonstrating the inherent distributional differences between images. Finally, Fig. 2(c) classifies eigenvectors using the Robustness-Based Nonconformity Score (RbNS), where causal eigenvectors (blue) exhibit lower RbNS values, indicating higher robustness, while correlational eigenvectors (red) have higher RbNS values, suggesting susceptibility to spurious correlations. The threshold τ∗superscriptτ^*τ∗ (dashed black line) effectively distinguishes robust causal components from spurious ones. Together, these results highlight how EigenShield enhances robustness by selectively retaining causal eigenvectors while filtering out correlational noise. Table I: Effectiveness of EigenShield against multimodal jailbreak attacks on VLMs, evaluated using Perspective API metrics. Dataset: HarmBench [20]. Lower is better. Model No Defense EigenShield ASR Identity Profanity Sexually Threat Toxicity ASR Identity Profanity Sexually Threat Toxicity Attack Explicit Attack Explicit LLaVA-v1.5-7B 65.72 18.59 49.61 30.24 27.32 63.30 24.37 6.53 16.12 7.94 4.18 23.11 MiniGPT-4 55.24 7.11 34.82 20.04 6.24 51.94 22.46 2.81 14.55 9.12 2.01 19.84 InstructBLIP 10.63 1.35 7.95 3.46 0.88 9.06 4.52 0.61 3.43 1.42 0.35 3.92 Qwen2-VL 7.57 0.82 5.14 1.54 0.89 6.91 2.66 0.39 1.79 0.63 0.47 2.03 Florence-2-large 15.24 2.15 9.78 5.28 1.72 13.31 3.51 0.73 3.72 1.54 0.73 3.46 Table I: EigenShield evaluated on clean images from ImageNet and adversarial texts from RTP [21] and HarmBench [20]. Lower is better. Model No Defense EigenShield RTP HarmBench RTP HarmBench LLaVA-v1.5-7B 1.1 0.7 0.3 0.2 MiniGPT-4 0.6 0.4 0.1 0.1 InstructBLIP 0.7 0.5 0.4 0.1 Qwen2-VL 0.2 0.4 0.2 0.1 Florence-2-large 0.5 0.4 0.1 0.2 Table I: Comparison of EigenShield with state-of-the-art defenses against multimodal jailbreak attacks using Perspective API metrics. The dataset includes unconstrained adversarial visual attacks [9] and RTP [21] adversarial text on LLaVA-v1.5-7B, MiniGPT-4, InstructBLIP, and Qwen2-VL. Lower values indicate better performance. Best and second-best results are highlighted in bold and underlined. Defense Attack Identity Profanity Sexually Threat Toxicity Attack Identity Profanity Sexually Threat Toxicity Success Attack Explicit Success Attack Explicit LLaVA-v1.5-7B MiniGPT-4 No Defense 81.61 25.41 67.22 39.38 40.64 77.93 37.20 2.94 26.53 12.76 2.10 31.57 Adv. Training 28.21 3.71 25.05 10.90 1.87 28.44 29.82 2.66 22.41 10.15 1.63 23.20 UNIGUARD 25.17 2.06 22.34 7.99 0.86 19.16 24.98 1.37 16.42 10.69 1.80 18.73 BLURKERNEL 39.03 3.92 30.61 14.10 3.17 32.28 38.92 2.28 28.34 13.79 2.12 33.08 COMP-DECOMP 37.70 2.67 29.02 13.26 3.59 31.94 35.21 2.31 25.56 11.97 1.54 29.06 DIFFPURE 40.42 3.01 30.89 14.48 3.35 34.06 41.32 2.12 29.89 15.24 2.12 35.65 CIDER 24.73 2.88 25.80 9.79 2.53 17.49 22.74 1.68 17.15 9.82 1.93 20.04 EigenShield 19.20 1.76 15.31 5.16 1.01 13.28 20.37 1.39 12.80 8.53 0.96 15.77 InstructBLIP Qwen2-VL No Defense 59.8 6.51 44.95 19.02 4.92 54.55 30.38 2.13 27.65 12.53 1.08 29.48 UNIGUARD 43.79 5.09 34.36 13.43 2.42 39.95 19.87 1.85 17.98 7.20 0.91 18.52 BLURKERNEL 69.31 9.26 56.96 23.85 6.42 66.22 35.17 3.54 30.11 14.52 1.59 34.27 COMP-DECOMP 69.22 8.17 56.13 23.69 6.17 65.72 34.85 3.19 29.74 14.35 1.29 33.82 DIFFPURE 68.31 8.76 52.79 24.35 5.09 63.47 36.15 3.89 31.47 16.09 1.66 35.57 CIDER 44.28 5.69 36.52 12.96 3.05 42.09 21.72 2.63 19.75 6.81 0.97 20.14 EigenShield 41.21 4.89 32.11 11.75 2.38 40.69 17.25 1.51 14.82 5.99 0.83 16.68 Figure 3: Attack Success Rate (%) vs. coverage parameter γ for two VLMs: LLaVA-v1.5 (blue) and MiniGPT-4 (red). Higher values of γ generally correspond to stricter causal inclusion, leading to a reduction in attack success for both models. The top axis indicates the number of trainable parameters (in thousands) for each γ. As γ increases, more eigenvalue directions are retained, effectively enhancing the model’s capacity. IV Experiments IV-A Model, Datasets, and Evaluation Metrics Models: We evaluated the generalizability of EigenShield on diverse VLMs, including LLaVA-v1.5-7B [22], MiniGPT-4 [23], InstructBLIP [24], Qwen2-VL [25], and Florence-2-large [26]. LLaVA-v1.5-7B fine-tunes LLaMA/Vicuna with multimodal instruction data for vision-language reasoning. MiniGPT-4 aligns a pretrained vision encoder with a large language model for multimodal interaction. InstructBLIP applies instruction tuning. Qwen2-VL integrates advanced multimodal fusion techniques with extensive pretraining on both textual and visual data. Florence-2-large leverages large-scale supervised learning for high-resolution image understanding and captioning. The comprehensive set of test models ensured a robust assessment of EigenShield across a range of VLM characteristics. Datasets: To set up the baseline accuracy without adversarial attacks, we used a subset of the ImageNet dataset [27], randomly selecting a thousand images. To evaluate EigenShield’s robustness against adversarial threats, we employed the HarmBench dataset [20], a benchmark for testing VLMs against harmful content generation. Specifically, we focused on two-hundred harmful queries in HarmBench and generated a thousand adversarial text-image pairs using a Projected Gradient Descent (PGD) attack [28] with varying perturbation rates. Additionally, we scaled our evaluation using the RealToxicityPrompts benchmark [21]. Following [9] and [29], we used the challenging subset containing 1,225 text prompts designed to trigger toxic continuations. Visual adversarial examples were paired with each text prompt to serve as VLM inputs. To evaluate EigenShield’s generalization under various real-world threat models, we generated adversarial examples using five distinct attack methods: FGSM [30], PGD [28], MIM [31], CW [32], and Square Attack [33]. Each method targets unique optimization objectives or utilizes distinct information (e.g., gradients vs. query-based sampling), covering a wide range of adversarial strategies. We randomly sampled images from our validation set and applied each attack to create five adversarial subsets, which were combined into a unified “generalization” dataset. Defense Baselines: We compared EigenShield against the following defense baselines: No Defense (Baseline Vulnerability Assessment): We establish a baseline by evaluating VLMs without any defense mechanism, highlighting their inherent vulnerabilities to adversarial jailbreak attacks and quantifying the security gap targeted by defense strategies. Image Transformation Defenses: We utilize two image pre-processing techniques: BLURKERNEL, which uses small average convolution kernels to smooth input images, reducing high-frequency noise linked to adversarial attacks, and COMP-DECOMP, which lowers image quality to obscure adversarial features embedded in the images. DIFFPURE [11]: DIFFPURE leverages diffusion models for adversarial purification by introducing controlled noise to adversarial images and applying a reverse diffusion process to effectively remove perturbations while preserving the original image’s semantics. CIDER [13]: Cross-modality Information DEtectoR (CIDER) is an inference-time detector that identifies adversarial images by analyzing semantic shifts between text and image modalities before and after denoising. Adversarial Training [3]: Adversarial training enhances robustness by training models on both clean and adversarial inputs to strengthen internal representations. This approach, however, requires modifications to the training pipeline and is computationally demanding. UNIGUARD [10]: UNIGUARD is a multimodal safety framework that minimizes harmful responses by optimizing modality-specific guardrails using a small corpus of harmful content. It can be seamlessly integrated at inference time and demonstrates strong multimodal defense. Evaluation Metrics: To quantify the harmfulness of generated responses, we leverage the Perspective API111https://perspectiveapi.com/. The Perspective API analyzes text across various attributes indicative of online abuse and harmful discourse. Specifically, we utilize the API to evaluate each VLM response across five key dimensions: toxicity (average of “overall toxicity” and “severe toxicity” scores), identity attack, threat, profanity, and sexually explicit content. For each attribute, the Perspective API returns a score ranging from 0 to 1, representing the probability that the text is perceived as exhibiting the respective attribute. Attack Success Rate (ASR): To quantify the effectiveness of defense mechanisms in mitigating jailbreak attacks, we employ ASR. ASR is formally defined as the proportion of adversarial attack attempts, represented as pairs of adversarial input (xadv)subscriptadv(x_adv)( xadv ) and a corresponding harmful prompt (prompt)prompt(prompt)( prompt ), that result in the VLM generating a disallowed or harmful response despite the application of the defense mechanism. Similar to [13], ASR is calculated using the following equation as the average of the harmfulness indicator function, harmsubscriptharmI_harmblackboard_Iharm, over a dataset DD of adversarial examples. For each adversarial example, consisting of a harmful query prompt and an adversarial image xadvsubscriptadvx_advxadv, we evaluate the response generated by the Vision-Language Model ℱFF using an LLM-based classifier GG (implemented via the Perspective API). The indicator function harmsubscriptharmI_harmblackboard_Iharm then flags whether the response is harmful based on predefined criteria. By averaging this indicator across all examples in DD, the ASR provides a quantifiable measure of the defense’s failure rate against jailbreak attacks as: ASR=def1||⁢∑(prompt,xadv)∈harm⁢(⁢(ℱ⁢(prompt,xadv)))superscriptdefASR1subscriptpromptsubscriptadvsubscriptharmℱpromptsubscriptadvASR = 1|D| _% (prompt,x_adv) I_harm(% G(F(prompt,x_adv)))ASR start_RELOP SUPERSCRIPTOP start_ARG = end_ARG start_ARG def end_ARG end_RELOP divide start_ARG 1 end_ARG start_ARG | D | end_ARG ∑( prompt , x start_POSTSUBSCRIPT adv ) ∈ D end_POSTSUBSCRIPT blackboard_Iharm ( G ( F ( prompt , xadv ) ) ) (5) A lower ASR indicates a more robust and effective defense, i.e., reduced likelihood of adversarial inputs successfully producing harmful or policy-violating outputs from the VLM when protected by the defense. IV-B Performance Characterizations In Table I, ASR drops significantly across models with EigenShield, LLaVA-v1.5-7B improving from 65.72% to 24.37%, MiniGPT-4 from 55.24% to 22.46%, InstructBLIP from 10.63% to 4.52%, Qwen2-VL from 7.57% to 2.66%, and Florence-2-large from 15.24% to 3.51%. Toxicity is also markedly reduced, with LLaVA-v1.5-7B decreasing from 63.30 to 23.11, MiniGPT-4 from 51.94 to 19.84, and Florence-2-large from 13.31 to 3.46. Similar improvements are observed across identity attacks, profanity, sexually explicit content, and threats. Likewise, in Table I on RealToxicityPrompts dataset (RTP), EigenShield reduces ASR for LLaVA-v1.5-7B from 81.61% to 19.20%, marking a 76.5% relative reduction. MiniGPT-4 sees ASR drop from 37.20% to 20.37%, InstructBLIP from 59.80% to 41.21%, and Qwen2-VL from 30.38% to 17.25%. Notably, unlike adversarial training and UNIGUARD, which require costly retraining, EigenShield operates entirely at inference time, filtering adversarial noise without modifying model parameters, thus maintaining computational efficiency. Tables V and VI in Appendix D present a further analysis using constrained adversarial images. In Table I, when combining clean images from ImageNet with adversarial text inputs from RTP and HarmBench, EigenShield consistently lowers ASR across all models compared to the no-defense baseline. For LLaVA-v1.5-7B, ASR drops from 1.1% to 0.3% on RTP and from 0.7% to 0.2% on HarmBench, demonstrating a substantial improvement in adversarial resilience. MiniGPT-4 exhibits a similar trend, with ASR decreasing from 0.6% to 0.1% on RTP and from 0.4% to 0.2% on HarmBench. InstructBLIP also benefits significantly, with ASR reductions from 0.7% to 0.4% on RTP and from 0.5% to 0.1% on HarmBench. Qwen2-VL and Florence-2-large, which already exhibit lower baseline ASR values, still show meaningful reductions with EigenShield. Qwen2-VL maintains an ASR of 0.2% on RTP while improving from 0.4% to 0.1% on HarmBench. Florence-2-large experiences ASR reductions from 0.5% to 0.1% on RTP and from 0.4% to 0.2% on HarmBench. These improvements indicate that EigenShield enhances robustness across diverse architectures, even for models with relatively lower initial vulnerability. Fig. 4 presents a comparative analysis of attack success rates (%) across five adversarial attack methods on two vision-language models: LLaVA-v1.5-7B [22] and MiniGPT-4 [23]. The spider plot represents each attack along different axes, with the radial distance indicating the attack success rate—where a larger radius corresponds to higher vulnerability. The results demonstrate that EigenShield consistently achieves the lowest attack success rates against a diverse range of adversarial strategies. In contrast, the effectiveness of other defense methods varies, with some showing stronger resilience to specific attacks but failing against others. Figure 4: Attack Success Rates (%) across five adversarial attacks on (a) LLaVA-v1.5-7B [22] and (b) MiniGPT-4 [23]. Radial axes represent attack methods, with larger radii indicating higher vulnerability. EigenShield consistently achieves the lowest attack success rates, while other defenses vary in effectiveness. IV-C Comparison with State-of-the-Art Defenses In Table I, EigenShield consistently achieves the lowest ASR and toxicity while reducing harmful content more effectively than existing methods. For LLaVA-v1.5-7B, ASR drops from 81.61% (no defense) to 19.20%, outperforming adversarial training (28.21%), UNIGUARD (25.17%), and inference-time defenses like CIDER (24.73%). It also significantly reduces profanity (67.22 → 15.31), explicit content (39.38 → 5.16), and toxicity (77.93 → 13.28), demonstrating strong mitigation of harmful language. Similarly, on MiniGPT-4, EigenShield lowers ASR from 37.20% to 20.37%, surpassing UNIGUARD (24.98%) and CIDER (22.74%), while achieving the lowest toxicity (15.77). It also reduces identity attacks (2.94 → 1.39), profanity (26.53 → 12.80), and explicit content (12.76 → 8.53). For InstructBLIP, EigenShield cuts ASR from 59.80% to 41.21%, offering stronger defense than UNIGUARD (43.79%) and CIDER (44.28%), while also lowering toxicity from 54.55 to 40.69 and reducing explicit content and profanity more effectively than diffusion-based purification methods like DIFFPURE. Qwen2-VL, already relatively robust, still benefits from EigenShield, with ASR dropping from 30.38% to 17.25%, outperforming UNIGUARD (19.87%) and CIDER (21.72%). Similar trends are seen in reducing identity attacks (2.13 → 1.51) and explicit content (12.53 → 5.99). For Florence-2-large, ASR falls from 36.47% to 18.80%, while toxicity reaches the lowest level (16.95), outperforming all baselines in profanity reduction. Table IV in Appendix D provides further analysis using constrained adversarial images. Additionally, example prompts and the corresponding responses before and after applying EigenShield can be found in Appendix D. V Conclusions In this work, we have introduced EigenShield, a novel defense for Vision-Language Models (VLMs) that leverages Random Matrix Theory (RMT) to mitigate adversarial vulnerabilities. Unlike adversarial training and heuristic defenses, EigenShield operates entirely at inference time, making it architecture-independent, attack-agnostic, and computationally efficient. Using the spiked covariance model and Robustness-Based Nonconformity Score (RbNS), it effectively separates causal eigenvectors from adversarial noise, preserving semantic integrity. Extensive evaluations on state-of-the-art VLMs, including LLaVA-v1.5-7B, MiniGPT-4, InstructBLIP, Qwen2-VL, and Florence-2-large, demonstrate EigenShield’s ability to significantly reduce attack success rates (ASR) across various adversarial strategies. It consistently outperforms adversarial training, UNIGUARD, CIDER, and diffusion-based defenses in reducing ASR and harmful content while preserving clean input integrity. Our findings establish spectral analysis as a principled alternative to conventional defenses. Future directions could include exploring the applicability of spectral analyses in broader multimodal AI architectures such as vision and speech beyond VLMs. References [1] S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,” arXiv preprint arXiv:2402.06196, 2024. [2] J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. [3] A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, “Adversarial training for free!” Advances in neural information processing systems, vol. 32, 2019. [4] H. Jin, L. Hu, X. Li, P. Zhang, C. Chen, J. Zhuang, and H. Wang, “Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models,” arXiv preprint arXiv:2407.01599, 2024. [5] D. Liu, M. Yang, X. Qu, P. Zhou, Y. Cheng, and W. Hu, “A survey of attacks on large vision-language models: Resources, advances, and future trends,” arXiv preprint arXiv:2407.07403, 2024. [6] E. Shayegani, Y. Dong, and N. Abu-Ghazaleh, “Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models,” in The Twelfth International Conference on Learning Representations, 2023. [7] Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N.-M. M. Cheung, and M. Lin, “On evaluating adversarial robustness of large vision-language models,” Advances in Neural Information Processing Systems, vol. 36, 2024. [8] J. Yi, Y. Xie, B. Zhu, E. Kiciman, G. Sun, X. Xie, and F. Wu, “Benchmarking and defending against indirect prompt injection attacks on large language models,” arXiv preprint arXiv:2312.14197, 2023. [9] X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal, “Visual adversarial examples jailbreak aligned large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 19, 2024, p. 21 527–21 536. [10] S. Oh, Y. Jin, M. Sharma, D. Kim, E. Ma, G. Verma, and S. Kumar, “Uniguard: Towards universal safety guardrails for jailbreak attacks on multimodal large language models,” arXiv preprint arXiv:2411.01703, 2024. [11] W. Nie, B. Guo, Y. Huang, C. Xiao, A. Vahdat, and A. Anandkumar, “Diffusion models for adversarial purification,” arXiv preprint arXiv:2205.07460, 2022. [12] X. Zhang, C. Zhang, T. Li, Y. Huang, X. Jia, X. Xie, Y. Liu, and C. Shen, “A mutation-based method for multi-modal jailbreaking attack detection,” arXiv preprint arXiv:2312.10766, 2023. [13] Y. Xu, X. Qi, Z. Qin, and W. Wang, “Defending jailbreak attack in vlms via cross-modality information detector,” arXiv e-prints, p. arXiv–2407, 2024. [14] A. Edelman and N. R. Rao, “Random matrix theory,” Acta numerica, vol. 14, p. 233–297, 2005. [15] T. Tao, Topics in random matrix theory. American Mathematical Soc., 2012, vol. 132. [16] A. R. Feier, Methods of proof in random matrix theory. Citeseer, 2012, vol. 9. [17] T. Chan, “The wigner semi-circle law and eigenvalues of matrix-valued diffusions,” Probability theory and related fields, vol. 93, no. 2, p. 249–272, 1992. [18] P. Yaskov, “A short proof of the marchenko–pastur theorem,” Comptes Rendus Mathematique, vol. 354, no. 3, p. 319–322, 2016. [19] D. Paul, “Asymptotics of sample eigenstructure for a large dimensional spiked covariance model,” Statistica Sinica, p. 1617–1642, 2007. [20] M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li et al., “Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,” arXiv preprint arXiv:2402.04249, 2024. [21] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, “Realtoxicityprompts: Evaluating neural toxic degeneration in language models,” arXiv preprint arXiv:2009.11462, 2020. [22] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, 2024. [23] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592, 2023. [24] W. Dai, J. Li, D. LI, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, p. 49 250–49 267. [25] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin, “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191, 2024. [26] B. Xiao, H. Wu, W. Xu, X. Dai, H. Hu, Y. Lu, M. Zeng, C. Liu, and L. Yuan, “Florence-2: Advancing a unified representation for a variety of vision tasks,” arXiv preprint arXiv:2311.06242, 2023. [27] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, p. 248–255. [28] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” arXiv preprint arXiv:1706.06083, 2017. [29] N. Mehrabi, A. Beirami, F. Morstatter, and A. Galstyan, “Robust conversational agents against imperceptible toxicity triggers,” arXiv preprint arXiv:2205.02392, 2022. [30] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014. [31] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, p. 9185–9193. [32] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in 2017 ieee symposium on security and privacy (sp). Ieee, 2017, p. 39–57. [33] M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein, “Square attack: a query-efficient black-box adversarial attack via random search,” in European conference on computer vision. Springer, 2020, p. 484–501. [34] I. M. Johnstone, “On the distribution of the largest eigenvalue in principal components analysis,” The Annals of statistics, vol. 29, no. 2, p. 295–327, 2001. [35] Z. Bai and J. W. Silverstein, Spectral analysis of large dimensional random matrices. Springer, 2010, vol. 20. Appendix A Spectral Confinement in Random Matrices Setting. Consider an N×N× N × N real symmetric random matrix M whose entries satisfy: 1. mi⁢jsubscriptm_ijmitalic_i j (for 1≤i≤j≤N11≤ i≤ j≤ N1 ≤ i ≤ j ≤ N) are independent random variables; 2. ⁢[mi⁢j]=0delimited-[]subscript0E[m_ij]=0blackboard_E [ mitalic_i j ] = 0 and ⁢[mi⁢j2]=σ2delimited-[]superscriptsubscript2superscript2E[m_ij^2]=σ^2blackboard_E [ mitalic_i j2 ] = σ2, with finite higher moments; 3. mi⁢j=mj⁢isubscriptsubscriptm_ij=m_jimitalic_i j = mitalic_j i, so M is symmetric. Let λ1,λ2,…,λNsubscript1subscript2…subscript _1, _2,…, _Nλ1 , λ2 , … , λitalic_N be the eigenvalues of M, and define the empirical spectral distribution (ESD) by μN=1N⁢∑k=1Nδλk.subscript1superscriptsubscript1subscriptsubscript _N\;=\; 1N _k=1^N _ _k.μitalic_N = divide start_ARG 1 end_ARG start_ARG N end_ARG ∑k = 1N δitalic_λ start_POSTSUBSCRIPT k end_POSTSUBSCRIPT . (6) By Wigner Semicircle Law, in the limit N→∞→N→∞N → ∞, the eigenvalues of M become densely packed in the interval [−2⁢σ,2⁢σ]22[-2σ,2σ][ - 2 σ , 2 σ ]. Equivalently, μNsubscript _Nμitalic_N converges in distribution to the so-called semicircle law supported on [−2⁢σ,2⁢σ]22[-2σ,2σ][ - 2 σ , 2 σ ]. A standard way to prove this spectral confinement is via the method of moments. Define the n-th moment of μNsubscript _Nμitalic_N as mn⁢(μN)=∫xn⁢μN⁢(x)=1N⁢∑k=1Nλkn=1N⁢Tr⁢(Mn).subscriptsubscriptsuperscriptdifferential-dsubscript1superscriptsubscript1superscriptsubscript1Trsuperscriptm_n( _N)\;=\; x^n\,d _N(x)\;=\; 1N _k=1^N% _k^n\;=\; 1N\,Tr(M^n).mitalic_n ( μitalic_N ) = ∫ xitalic_n d μitalic_N ( x ) = divide start_ARG 1 end_ARG start_ARG N end_ARG ∑k = 1N λitalic_kitalic_n = divide start_ARG 1 end_ARG start_ARG N end_ARG Tr ( Mitalic_n ) . (7) We aim to show that for each fixed n, 1N⁢[Tr⁢(Mn)]⟶κnas ⁢N→∞,formulae-sequence⟶1delimited-[]Trsuperscriptsubscript→as 1N\,E\! [Tr(M^n) ]\;\; % \;\; _n N→∞,divide start_ARG 1 end_ARG start_ARG N end_ARG blackboard_E [ Tr ( Mitalic_n ) ] ⟶ κitalic_n as N → ∞ , (8) where κnsubscript _nκitalic_n is the n-th moment of the semicircle distribution with radius 2⁢σ22 2 σ: ρsc⁢(x)=12⁢π⁢σ2⁢ 4⁢σ2−x2⁢ 1|x|≤2⁢σ,κn=∫−2⁢σ2⁢σxn⁢ρsc⁢(x)⁢x.formulae-sequencesubscriptsc12superscript24superscript2superscript2subscript12subscriptsuperscriptsubscript22superscriptsubscriptscdifferential-d _sc(x)\;=\; 12πσ^2\, \,4σ^2-x^2\,% \,1_\|x|≤ 2σ\, _n= _-2σ^2σx^% n\, _sc(x)\,dx.ρsc ( x ) = divide start_ARG 1 end_ARG start_ARG 2 π σ2 end_ARG square-root start_ARG 4 σ2 - x2 end_ARG 1 | x | ≤ 2 σ , κitalic_n = ∫- 2 σ2 σ xitalic_n ρsc ( x ) d x . (9) (a) Expanding the Trace. We write Tr⁢(Mn)=∑i1=1N(Mn)i1,i1=∑i1,i2,…,in=1Nmi1⁢i2⁢mi2⁢i3⁢⋯⁢min−1⁢in⁢min⁢i1.Trsuperscriptsuperscriptsubscriptsubscript11subscriptsuperscriptsubscript1subscript1superscriptsubscriptsubscript1subscript2…subscript1subscriptsubscript1subscript2subscriptsubscript2subscript3⋯subscriptsubscript1subscriptsubscriptsubscriptsubscript1Tr(M^n)\;=\; _i_1=1^N(M^n)_i_1,i_1\;=\; _i_1% ,i_2,…,i_n=1^Nm_i_1i_2\,m_i_2i_3·s m_i_n-1i_n% \,m_i_ni_1.Tr ( Mitalic_n ) = ∑i start_POSTSUBSCRIPT 1 = 1 end_POSTSUBSCRIPTN ( Mitalic_n )i start_POSTSUBSCRIPT 1 , i1 end_POSTSUBSCRIPT = ∑i start_POSTSUBSCRIPT 1 , i2 , … , iitalic_n = 1 end_POSTSUBSCRIPTN mitalic_i start_POSTSUBSCRIPT 1 i2 end_POSTSUBSCRIPT mitalic_i start_POSTSUBSCRIPT 2 i3 end_POSTSUBSCRIPT ⋯ mitalic_i start_POSTSUBSCRIPT n - 1 iitalic_n end_POSTSUBSCRIPT mitalic_i start_POSTSUBSCRIPT n i1 end_POSTSUBSCRIPT . (10) Thus, mn⁢(μN)=1N⁢Tr⁢(Mn)=1N⁢∑i1,…,in=1Nmi1⁢i2⁢⋯⁢min⁢i1.subscriptsubscript1Trsuperscript1superscriptsubscriptsubscript1…subscript1subscriptsubscript1subscript2⋯subscriptsubscriptsubscript1m_n( _N)\;=\; 1N\,Tr(M^n)\;=\; 1N _i_1,% …,i_n=1^Nm_i_1i_2·s m_i_ni_1.mitalic_n ( μitalic_N ) = divide start_ARG 1 end_ARG start_ARG N end_ARG Tr ( Mitalic_n ) = divide start_ARG 1 end_ARG start_ARG N end_ARG ∑i start_POSTSUBSCRIPT 1 , … , iitalic_n = 1 end_POSTSUBSCRIPTN mitalic_i start_POSTSUBSCRIPT 1 i2 end_POSTSUBSCRIPT ⋯ mitalic_i start_POSTSUBSCRIPT n i1 end_POSTSUBSCRIPT . (11) (b) Taking Expectations and Pairings. Because mi⁢jsubscript\m_ij\ mitalic_i j are independent mean-zero random variables, any product mi1⁢i2⁢mi2⁢i3⁢⋯⁢min⁢i1subscriptsubscript1subscript2subscriptsubscript2subscript3⋯subscriptsubscriptsubscript1m_i_1i_2m_i_2i_3·s m_i_ni_1mitalic_i start_POSTSUBSCRIPT 1 i2 end_POSTSUBSCRIPT mitalic_i start_POSTSUBSCRIPT 2 i3 end_POSTSUBSCRIPT ⋯ mitalic_i start_POSTSUBSCRIPT n i1 end_POSTSUBSCRIPT with an odd number of distinct mi⁢jsubscriptm_ijmitalic_i j factors has zero expectation. Nonzero terms arise only when each factor appears an even number of times, so that factors “pair up.” In the large-N limit, the combinatorial counting of these pairings is closely related to the Catalan numbers. One finds that the leading-order terms in ⁢[Tr⁢(Mn)]/Ndelimited-[]TrsuperscriptE [Tr(M^n) ]/Nblackboard_E [ Tr ( Mitalic_n ) ] / N match exactly the moments of the semicircle distribution: 1N⁢[Tr⁢(Mn)]→κnas ⁢N→∞.formulae-sequence→1delimited-[]Trsuperscriptsubscript→as 1N\,E\! [Tr(M^n) ]\;→\; _n% N→∞.divide start_ARG 1 end_ARG start_ARG N end_ARG blackboard_E [ Tr ( Mitalic_n ) ] → κitalic_n as N → ∞ . (12) Moreover, one can show (through variance bounds or concentration inequalities) that mn⁢(μN)subscriptsubscriptm_n( _N)mitalic_n ( μitalic_N ) converges almost surely to the same value, implying that μNsubscript _Nμitalic_N converges weakly to the semicircle law on [−2⁢σ,2⁢σ]22[-2σ,2σ][ - 2 σ , 2 σ ]. It remains to show that with high probability, no eigenvalue λksubscript _kλitalic_k lies far beyond ±(2⁢σ+ε)plus-or-minus2±(2σ+ )± ( 2 σ + ε ). Concentration inequalities (e.g., Hanson–Wright) show that ‖M‖norm\|M\|∥ M ∥ is almost surely of order 2⁢σ⁢N22σ N2 σ square-root start_ARG N end_ARG, thus once normalized by N Nsquare-root start_ARG N end_ARG, all eigenvalues remain near ±2⁢σplus-or-minus2± 2σ± 2 σ. We conclude that for large N, all eigenvalues reside in the interval [−2⁢σ,2⁢σ]22[-2σ,2σ][ - 2 σ , 2 σ ] with overwhelming probability. Thus, the empirical spectral distribution cannot place mass arbitrarily far out on the real line. Rhe above arguments establish that for symmetric random matrices with independent, mean-zero, variance-σ2superscript2σ^2σ2 entries, (1) the eigenvalues are confined to ⁢[−2⁢σ,2⁢σ],and(2) they become densely packed in this interval as ⁢N→∞.→(1) the eigenvalues are confined to 22and(2) they become densely packed in this interval as (1) the eigenvalues are confined to [-2σ,2σ], % (2) they become densely packed in this interval as N→∞.(1) the eigenvalues are confined to [ - 2 σ , 2 σ ] , and (2) they become densely packed in this interval as N → ∞ . Any deviation from the semicircle shape in the empirical spectral distribution typically indicates extra structure (e.g. correlation, signal components (or causal components), or low-rank perturbations) rather than pure noise. Appendix B Formal Theorems on the Spiked Covariance Model As discussed in Section I, we leverage the spiked covariance model to detect and isolate “signal-bearing” directions in high-dimensional data. This section states the formal conditions under which certain eigenvalues of the sample covariance matrix separate from the bulk, thereby indicating low-rank “spike” components rather than pure noise. We adapt classical results from the Random Matrix Theory (RMT) literature—particularly [19] and subsequent works on spiked models [34]. B-A Setup and Definitions We assume the population covariance matrix takes the form =signal+σ2⁢p,subscriptsignalsuperscript2subscript \;=\; _signal\;+\;σ^2\,% I_p,Σ = Σroman_signal + σ2 Iitalic_p , (13) where signal=⁢⊤subscriptsignalsuperscripttop _signal=V\, \,% V Σroman_signal = V Λ V⊤ is a matrix of rank r≪pmuch-less-thanr pr ≪ p containing the “spike” signals, and σ2⁢psuperscript2subscriptσ^2I_pσ2 Iitalic_p denotes isotropic noise of variance σ2superscript2σ^2σ2. Here, ∈ℝr×rsuperscriptℝ ^r× rΛ ∈ blackboard_Rr × r stores the positive eigenvalues of the signal component, and ∈ℝp×rsuperscriptℝV ^p× rV ∈ blackboard_Rp × r has orthonormal columns (the corresponding eigenvectors). Given i.i.d. samples ii=1n⊂ℝpsuperscriptsubscriptsubscript1superscriptℝ\x_i\_i=1^n ^p xitalic_i i = 1n ⊂ blackboard_Rp with true covariance Σ, we form the sample covariance matrix n=1n⁢∑i=1ni⁢i⊤∈ℝp×p.subscript1superscriptsubscript1subscriptsuperscriptsubscripttopsuperscriptℝC_n\;=\; 1n\, _i=1^nx_i\,x_i^% \;∈\;R^p× p.Citalic_n = divide start_ARG 1 end_ARG start_ARG n end_ARG ∑i = 1n xitalic_i xitalic_i⊤ ∈ blackboard_Rp × p . (14) We operate in a high-dimensional limit, where both n and p grow to infinity at a fixed ratio c=p/nc=p/nc = p / n. In the absence of spikes, the eigenvalues of nsubscriptC_nCitalic_n concentrate near the classical Marchenko–Pastur (MP) bulk [18], spanning from σ2⁢(1−c)2superscript2superscript12σ^2(1- c)^2σ2 ( 1 - square-root start_ARG c end_ARG )2 to σ2⁢(1+c)2superscript2superscript12σ^2(1+ c)^2σ2 ( 1 + square-root start_ARG c end_ARG )2. The spiked covariance model investigates how large population eigenvalues in Σ cause outlier eigenvalues in nsubscriptC_nCitalic_n. B-B Signal-Noise Separation via Outlier Eigenvalues Let λ1′≥λ2′≥⋯≥λr′>0subscriptsuperscript′1subscriptsuperscript′2⋯subscriptsuperscript′0λ _1≥λ _2≥…≥λ _r>0λ′1 ≥ λ′2 ≥ ⋯ ≥ λ′italic_r > 0 be the non-noise eigenvalues of signalsubscriptsignal _signalΣroman_signal. Let λ^1≥λ^2≥⋯≥λ^rsubscript^1subscript^2⋯subscript λ_1≥ λ_2≥…≥ λ_rover start_ARG λ end_ARG1 ≥ over start_ARG λ end_ARG2 ≥ ⋯ ≥ over start_ARG λ end_ARGr denote the top-r sample eigenvalues of nsubscriptC_nCitalic_n. We restate a known result (adapted from [19]) to illustrate when outlier eigenvalues appear in the sample covariance. Theorem 1 (Spiked Covariance Model, adapted from [19]) Suppose ii=1nsuperscriptsubscriptsubscript1\x_i\_i=1^n xitalic_i i = 1n are i.i.d. with mean 000 and covariance Σ as above, and let c=p/n>00c=p/n>0c = p / n > 0 be fixed as p,n→∞→p,n→∞p , n → ∞. Define βj=λj′σ2,j=1,…,r,formulae-sequencesubscriptsubscriptsuperscript′21… _j\;=\; λ _jσ^2, j=1,…,r,βitalic_j = divide start_ARG λ′italic_j end_ARG start_ARG σ2 end_ARG , j = 1 , … , r , (15) where λj′subscriptsuperscript′λ _jλ′italic_j is an eigenvalue of signalsubscriptsignal _signalΣroman_signal. Then: 1. (Bulk Behavior.) If βj≤(1+c)2subscriptsuperscript12 _j≤(1+ c)^2βitalic_j ≤ ( 1 + square-root start_ARG c end_ARG )2, the corresponding sample eigenvalue λ^jsubscript λ_jover start_ARG λ end_ARGj remains inside the Marchenko–Pastur bulk supported on [σ2⁢(1−c)2,σ2⁢(1+c)2]superscript2superscript12superscript2superscript12 [σ^2(1- c)^2,\,σ^2(1+ c)^2 ][ σ2 ( 1 - square-root start_ARG c end_ARG )2 , σ2 ( 1 + square-root start_ARG c end_ARG )2 ]. In other words, the spike is not large enough to detach from the noise region. 2. (Outlier Behavior.) If βj>(1+c)2subscriptsuperscript12 _j>(1+ c)^2βitalic_j > ( 1 + square-root start_ARG c end_ARG )2, there is an outlier sample eigenvalue λ^jsubscript λ_jover start_ARG λ end_ARGj that separates from the MP bulk. Moreover, λ^jsubscript λ_jover start_ARG λ end_ARGj converges almost surely to a limit Λ⁢(λj′)Λsubscriptsuperscript′ (λ _j)Λ ( λ′italic_j ) satisfying Λ⁢(λj′)=σ2⁢(βj+c⁢βjβj−1),Λsubscriptsuperscript′2subscriptsubscriptsubscript1 (λ _j)\;=\;σ^2 ( _j+ c\, _j% \, _j-1\, ),Λ ( λ′italic_j ) = σ2 ( βitalic_j + divide start_ARG c βitalic_j end_ARG start_ARG βitalic_j - 1 end_ARG ) , (16) and the associated sample eigenvector aligns closely with the corresponding population eigenvector in VV. Discussion. Theorem 1 formally characterizes how strong signal eigenvalues in Σ lead to isolated sample eigenvalues (outliers) in nsubscriptC_nCitalic_n. If the spike λj′subscriptsuperscript′λ _jλ′italic_j is large enough (relative to σ2superscript2σ^2σ2 and the aspect ratio c), the sample eigenvalue λ^jsubscript λ_jover start_ARG λ end_ARGj is pulled away from the main bulk, allowing us to detect it under typical large-p conditions. B-C Outlier Location and the Bulk Edge A key quantity that emerges in Theorem 1 is the Marchenko–Pastur upper edge, namely: λbulk,edge=σ2⁢(1+c)2.subscriptbulkedgesuperscript2superscript12 _bulk,edge\;=\;σ^2\,(1+ c)^2.λroman_bulk , edge = σ2 ( 1 + square-root start_ARG c end_ARG )2 . (17) Any eigenvalue of nsubscriptC_nCitalic_n that exceeds λbulk,edgesubscriptbulkedge _bulk,edgeλroman_bulk , edge is likely to be a spike. Formally, let λ^jsubscript λ_jover start_ARG λ end_ARGj be the j-th largest sample eigenvalue; if λ^j>λbulk,edgesubscript^subscriptbulkedge λ_j> _bulk,edgeover start_ARG λ end_ARGj > λroman_bulk , edge with high probability, then λ^jsubscript λ_jover start_ARG λ end_ARGj converges almost surely to a specific function of the corresponding population spike λj′subscriptsuperscript′λ _jλ′italic_j. Lemma 1 (Asymptotic Location of Spiked Sample Eigenvalues) Under the assumptions of Theorem 1, suppose λ^jsubscript λ_jover start_ARG λ end_ARGj is an outlier eigenvalue of nsubscriptC_nCitalic_n corresponding to a population spike λj′subscriptsuperscript′λ _jλ′italic_j. Then with probability tending to 1 as n,p→∞→n,p→∞n , p → ∞, λ^j→a.s.Λ⁢(λj′)=σ2⁢(βj+c⁢βjβj−1),whereβj=λj′σ2.formulae-sequencea.s.→subscript^Λsubscriptsuperscript′2subscriptsubscriptsubscript1wheresubscriptsubscriptsuperscript′2 λ_j\; a.s.\; (λ _j)\;=% \;σ^2\, ( _j+ c\, _j _j-1 ), % where _j\;=\; λ _jσ^2.over start_ARG λ end_ARGj start_ARROW overa.s. → end_ARROW Λ ( λ′italic_j ) = σ2 ( βitalic_j + divide start_ARG c βitalic_j end_ARG start_ARG βitalic_j - 1 end_ARG ) , where βitalic_j = divide start_ARG λ′italic_j end_ARG start_ARG σ2 end_ARG . (18) In particular, if βj≤(1+c)2subscriptsuperscript12 _j≤(1+ c)^2βitalic_j ≤ ( 1 + square-root start_ARG c end_ARG )2, then λ^jsubscript λ_jover start_ARG λ end_ARGj merges with the MP bulk, and no outlier emerges in that direction. B-D Implications for EigenShield EigenShield employs these results as a principled mechanism to separate “causal” signal components from noise-based or weakly relevant components. In practice, any sample eigenvalue above the upper MP edge, λbulk,edge=σ2⁢(1+c)2subscriptbulkedgesuperscript2superscript12 _bulk,edge=σ^2(1+ c)^2λroman_bulk , edge = σ2 ( 1 + square-root start_ARG c end_ARG )2, is treated as a candidate signal direction, which we retain in the “causal” subspace. By projecting onto these top outlier eigenvectors, EigenShield discards the bulk (noise) portion of the covariance and thereby filters out adversarial perturbations that primarily reside in lower-eigenvalue directions. These theorems thus ground our inference-phase defense in well-established RMT principles, ensuring that separated sample eigenvalues correspond to genuinely meaningful—rather than random or purely noise-driven—directions in the embedding space. B-E Proof of Theorem 1 and Lemma 1 Below, we present a concise proof sketch for Theorem 1(the spiked covariance result) and Lemma 1 (the asymptotic location of spiked sample eigenvalues). These arguments follow classical techniques in Random Matrix Theory (RMT), particularly from [19], [34], and [35]. Notation and Statement Recap. We have i.i.d. samples ii=1nsuperscriptsubscriptsubscript1\x_i\_i=1^n xitalic_i i = 1n in ℝpsuperscriptℝR^pblackboard_Rp with mean 000 and population covariance =signal+σ2⁢p,subscriptsignalsuperscript2subscript \;=\; _signal\;+\;σ^2\,% I_p,Σ = Σroman_signal + σ2 Iitalic_p , (19) where signalsubscriptsignal _signalΣroman_signal is low-rank and contains r “spikes.” Denote the non-noise eigenvalues of Σ by λj′j=1r>0superscriptsubscriptsubscriptsuperscript′10\λ _j\_j=1^r>0 λ′italic_j j = 1r > 0. We define the sample covariance n=1n⁢∑i=1ni⁢i⊤,c=pn.formulae-sequencesubscript1superscriptsubscript1subscriptsuperscriptsubscripttopC_n\;=\; 1n _i=1^nx_i\,x_i^% , c\;=\; pn.Citalic_n = divide start_ARG 1 end_ARG start_ARG n end_ARG ∑i = 1n xitalic_i xitalic_i⊤ , c = divide start_ARG p end_ARG start_ARG n end_ARG . (20) Let λ^jj=1psuperscriptsubscriptsubscript^1\ λ_j\_j=1^p over start_ARG λ end_ARGj j = 1p be the eigenvalues of nsubscriptC_nCitalic_n, arranged in nonincreasing order. Theorem 1 states that if λj′/σ2>(1+c)2subscriptsuperscript′2superscript12λ _j/σ^2>(1+ c)^2λ′italic_j / σ2 > ( 1 + square-root start_ARG c end_ARG )2, then λ^jsubscript λ_jover start_ARG λ end_ARGj emerges as an outlier above the main Marchenko–Pastur (MP) bulk. Otherwise, it remains inside the bulk. Lemma 1 refines this result by providing the exact limit of such an outlier, namely λ^j→a.s.σ2⁢(βj+c⁢βjβj−1),where ⁢βj=λj′σ2.formulae-sequencea.s.→subscript^superscript2subscriptsubscriptsubscript1where subscriptsubscriptsuperscript′2 λ_j\; a.s.\;σ^2 ( _j+% c\, _j _j-1 ), _j= % λ _jσ^2.over start_ARG λ end_ARGj start_ARROW overa.s. → end_ARROW σ2 ( βitalic_j + divide start_ARG c βitalic_j end_ARG start_ARG βitalic_j - 1 end_ARG ) , where βitalic_j = divide start_ARG λ′italic_j end_ARG start_ARG σ2 end_ARG . (21) Proof Sketch of Theorem 1 Step 1: Preliminaries and Transformation. Define =[1⁢2⁢⋯⁢n]∈ℝp×n,delimited-[]subscript1subscript2⋯subscriptsuperscriptℝX\;=\; [x_1\;x_2\;·s\;x_n% ]\;∈\;R^p× n,X = [ x1 x2 ⋯ xitalic_n ] ∈ blackboard_Rp × n , (22) so that n=1n⁢⊤.subscript1superscripttopC_n= 1nXX .Citalic_n = divide start_ARG 1 end_ARG start_ARG n end_ARG X⊤ . Let the population covariance be =⁢⊤+σ2⁢p,superscripttopsuperscript2subscript \;=\;V\, \,V \;+% \;σ^2I_p,Σ = V Λ V⊤ + σ2 Iitalic_p , (23) where =diag⁢(λ1′,…,λr′)diagsubscriptsuperscript′1…subscriptsuperscript′ =diag(λ _1,…,λ % _r)Λ = diag ( λ′1 , … , λ′italic_r ) and ∈ℝp×rsuperscriptℝV ^p× rV ∈ blackboard_Rp × r is orthonormal in its columns. Step 2: Decomposition into Spikes + Noise. One may write =1/2⁢superscript12X= ^1/2\,ZX = Σ1 / 2 Z, where ∈ℝp×nsuperscriptℝZ ^p× nZ ∈ blackboard_Rp × n has i.i.d. entries of mean zero and variance 1. Then n=1n⁢⊤=1/2⁢(1n⁢⊤)⁢1/2.subscript1superscripttopsuperscript121superscripttopsuperscript12C_n\;=\; 1n\,XX \;=\; % ^1/2\, ( 1nZZ )\,% ^1/2.Citalic_n = divide start_ARG 1 end_ARG start_ARG n end_ARG X⊤ = Σ1 / 2 ( divide start_ARG 1 end_ARG start_ARG n end_ARG Z⊤ ) Σ1 / 2 . (24) The matrix 1n⁢⊤1superscripttop 1nZZ divide start_ARG 1 end_ARG start_ARG n end_ARG Z⊤ is well known (by classical MP theory) to have its spectrum within [(1−c)2,(1+c)2]superscript12superscript12[(1- c)^2,\,(1+ c)^2][ ( 1 - square-root start_ARG c end_ARG )2 , ( 1 + square-root start_ARG c end_ARG )2 ] with high probability, when p,n→∞→p,n→∞p , n → ∞ at ratio c=p/nc=p/nc = p / n. Step 3: Outlier Condition. Focus on a single spike λj′subscriptsuperscript′λ _jλ′italic_j in the signal covariance. Define βj=λj′σ2subscriptsubscriptsuperscript′2 _j= λ _jσ^2βitalic_j = divide start_ARG λ′italic_j end_ARG start_ARG σ2 end_ARG. We can write =σ2⁢(p+),where⁢=σ−2⁢signal.formulae-sequencesuperscript2subscriptwheresuperscript2subscriptsignal \;=\;σ^2\, (I_p\;+\;Q % ), \;\;Q\;=\;σ^-2\, _% signal.Σ = σ2 ( Iitalic_p + Q ) , where Q = σ- 2 Σroman_signal . (25) When βjsubscript _jβitalic_j exceeds (1+c)2superscript12(1+ c)^2( 1 + square-root start_ARG c end_ARG )2, the eigenvalue perturbation arguments (via either the Stieltjes transform or matrix determinant lemmas) show that a corresponding sample eigenvalue λ^jsubscript λ_jover start_ARG λ end_ARGj must leave the main bulk. If βj≤(1+c)2subscriptsuperscript12 _j≤(1+ c)^2βitalic_j ≤ ( 1 + square-root start_ARG c end_ARG )2, we show there is not enough spectral “pull” for an outlier to form, so λ^jsubscript λ_jover start_ARG λ end_ARGj remains inside the bulk. Step 4: Technical Tools (Stieltjes Transform or Determinant Identities). A standard approach is to use the determinant characterization det(n−ℓ⁢p)=det(1/2⁢[1n⁢⊤−ℓσ2⁢p+…]⁢1/2).subscriptℓsubscriptsuperscript12delimited-[]1superscripttopℓsuperscript2subscript…superscript12 (C_n- _p )\;=\; (% ^1/2 [ 1nZZ -% σ^2I_p+… ]\, ^1/2% ).det ( Citalic_n - ℓ Iitalic_p ) = det ( Σ1 / 2 [ divide start_ARG 1 end_ARG start_ARG n end_ARG Z⊤ - divide start_ARG ℓ end_ARG start_ARG σ2 end_ARG Iitalic_p + … ] Σ1 / 2 ) . (26) One can track how eigenvalues ℓ=λ^jℓsubscript = λ_jℓ = over start_ARG λ end_ARGj appear as zeros of this determinant and then isolate the spike contributions via rank-one updates if signalsubscriptsignal _signalΣroman_signal is rank 1 (extended to rank r by block-decomposition). The condition βj>(1+c)2subscriptsuperscript12 _j>(1+ c)^2βitalic_j > ( 1 + square-root start_ARG c end_ARG )2 emerges from analyzing the perturbed Stieltjes transform inside or outside the main spectral arc. Step 5: Almost-Sure Convergence. Lastly, to promote these claims from expectations or weak convergence to almost-sure statements, one typically applies standard concentration bounds (matrix Chernoff bounds) to ensure that with probability approaching 1, the eigenvalue shifts cannot exceed small ε ε thresholds. Thus, λ^jsubscript λ_jover start_ARG λ end_ARGj indeed separates from the MP bulk almost surely when βj>(1+c)2subscriptsuperscript12 _j>(1+ c)^2βitalic_j > ( 1 + square-root start_ARG c end_ARG )2, concluding the main proof. This completes the proof of Theorem 1. Proof Sketch of Lemma 1(Asymptotic Outlier Location) Lemma D.2 refines the outlier result by showing that once a spiked sample eigenvalue λ^jsubscript λ_jover start_ARG λ end_ARGj emerges above the bulk, it converges to the specific function Λ⁢(λj′)=σ2⁢(βj+c⁢βjβj−1),whereβj=λj′σ2.formulae-sequenceΛsubscriptsuperscript′2subscriptsubscriptsubscript1wheresubscriptsubscriptsuperscript′2 (λ _j)\;=\;σ^2 ( _j\;+\; c\,% _j\, _j-1\, ), _j\;=\; % λ _jσ^2.Λ ( λ′italic_j ) = σ2 ( βitalic_j + divide start_ARG c βitalic_j end_ARG start_ARG βitalic_j - 1 end_ARG ) , where βitalic_j = divide start_ARG λ′italic_j end_ARG start_ARG σ2 end_ARG . (27) Step 1: Preliminary Rank-r Expansion. Let signalsubscriptsignal _signalΣroman_signal have the eigen-decomposition signal=⁢⊤subscriptsignalsuperscripttop _signal=V\, \,% V Σroman_signal = V Λ V⊤ with =diag⁢(λ1′,…,λr′)diagsubscriptsuperscript′1…subscriptsuperscript′ =diag(λ _1,…,λ % _r)Λ = diag ( λ′1 , … , λ′italic_r ). One typically starts by examining a rank-1 perturbation (say λ1′subscriptsuperscript′1λ _1λ′1, with eigenvector v1subscript1v_1v1), proving the location of the resulting outlier in the limit. The rank-r case follows by induction or block matrix arguments, as the presence of multiple spikes yields multiple outliers under similar reasoning. Step 2: Characterizing Outlier via Stieltjes Transform. Define the Stieltjes transform of the empirical spectral distribution (ESD) by mn⁢(z)=1p⁢Tr⁢((n−z⁢p)−1),z∈ℂ+.formulae-sequencesubscript1Trsuperscriptsubscriptsubscript1superscriptℂm_n(z)\;=\; 1p\,Tr ((C_n-z\,I_p% )^-1 ), z ^+.mitalic_n ( z ) = divide start_ARG 1 end_ARG start_ARG p end_ARG Tr ( ( Citalic_n - z Iitalic_p )- 1 ) , z ∈ blackboard_C+ . (28) Its limit m⁢(z)m(z)m ( z ) for the pure noise case σ2⁢psuperscript2subscriptσ^2I_pσ2 Iitalic_p is known from Marchenko–Pastur theory. In the presence of a spike λj′subscriptsuperscript′λ _jλ′italic_j, one can analyze how mn⁢(z)subscriptm_n(z)mitalic_n ( z ) (or the companion determinant) shifts. By matching singularities of the limiting transform with the location of λ^jsubscript λ_jover start_ARG λ end_ARGj, one obtains an explicit algebraic equation whose solution is precisely Λ⁢(λj′)=σ2⁢(βj+c⁢βjβj−1).Λsubscriptsuperscript′2subscriptsubscriptsubscript1 (λ _j)\;=\;σ^2 ( _j\;+\; c\,β% _j\, _j-1\, ).Λ ( λ′italic_j ) = σ2 ( βitalic_j + divide start_ARG c βitalic_j end_ARG start_ARG βitalic_j - 1 end_ARG ) . (29) (See [19] for the full derivation, which involves solving a quadratic equation for the limiting outlier position.) Step 3: Convergence and Eigenvector Alignment. After establishing that λ^j→Λ⁢(λj′)→subscript^Λsubscriptsuperscript′ λ_j→ (λ _j)over start_ARG λ end_ARGj → Λ ( λ′italic_j ) in probability (or in distribution), one again applies high-probability concentration results to lift it to almost-sure convergence. A related argument with the resolvent or rank-1 update formulas also shows that the sample eigenvector associated with λ^jsubscript λ_jover start_ARG λ end_ARGj converges to the population spike direction (i.e., the corresponding column of VV). Conclusion. Hence, whenever βj=λj′/σ2subscriptsubscriptsuperscript′2 _j=λ _j/σ^2βitalic_j = λ′italic_j / σ2 satisfies βj>(1+c)2subscriptsuperscript12 _j>(1+ c)^2βitalic_j > ( 1 + square-root start_ARG c end_ARG )2, a distinct sample outlier λ^jsubscript λ_jover start_ARG λ end_ARGj emerges and converges almost surely to σ2⁢(βj+c⁢βjβj−1),superscript2subscriptsubscriptsubscript1σ^2 ( _j+ c\, _j\, _j-1\, ),σ2 ( βitalic_j + divide start_ARG c βitalic_j end_ARG start_ARG βitalic_j - 1 end_ARG ) , exactly matching the statement of Lemma 1. This completes our proof sketches for Theorem 1 and Lemma 1. Appendix C Mathematical Justification for Information Gain from Causal Directions To confirm our central claim that projections onto causal eigenvector directions enhance the fidelity of input data reconstruction and yield significant information gain about the input data distribution, we now present a justification rooted in the principles of information theory. This section will formally demonstrate how a reduction in entropy, when conditioning on projections along causal directions, directly translates to an increase in information gain regarding the input data distribution and improved reconstruction. Entropy as a Fundamental Measure of Uncertainty in Data Representation We begin by recalling the foundational concept of Shannon entropy, now interpreting it as a measure of uncertainty associated with representing the input data distribution. Let us formally define the random variable X to represent the high-dimensional input data (or its feature representation), taking values from a continuous space XX with a probability density function P⁢(X)P(X)P ( X ). For simplicity in this derivation, we will consider a discretized version, where X takes values from a discrete set x1,…,xNsubscript1…subscript\x_1,…,x_N\ x1 , … , xitalic_N , each occurring with a probability P⁢(X=xi)=pisubscriptsubscriptP(X=x_i)=p_iP ( X = xitalic_i ) = pitalic_i, where ∑i=1Npi=1superscriptsubscript1subscript1 _i=1^Np_i=1∑i = 1N pitalic_i = 1. The Shannon entropy of X, denoted as H⁢(X)H(X)H ( X ), is then mathematically expressed as: H⁢(X)=−∑i=1Npi⁢log2⁡(pi).superscriptsubscript1subscriptsubscript2subscriptH(X)=- _i=1^N\,p_i\, _2 (p_i ).H ( X ) = - ∑i = 1N pitalic_i log2 ( pitalic_i ) . (30) Intuitively, H⁢(X)H(X)H ( X ) captures the inherent randomness or complexity within the input data distribution. High entropy signifies a distribution that is spread out, representing high variability and uncertainty in the data. Conversely, low entropy indicates a distribution that is concentrated, implying a more structured and predictable data representation. Entropy Reduction as a Signature of Enhanced Data Representation In the context of unsupervised reconstruction, predictability now relates to how well we can reconstruct or represent the input data distribution. Lower entropy in the representation implies a more efficient and less uncertain representation, which should facilitate better reconstruction. Consider the limiting case where H⁢(X)=00H(X)=0H ( X ) = 0. This would ideally occur if the data was perfectly deterministic and concentrated at a single point in the representation space, implying maximal predictability and minimal uncertainty, although practically such a scenario is less relevant for real-world data. As entropy increases, the data representation becomes more complex and less predictable, potentially hindering faithful reconstruction. From an information-theoretic viewpoint, entropy here represents the average number of bits needed to encode data samples from X. Lower entropy suggests that fewer bits are needed to represent the data distribution, indicating a more structured and less random data representation, and thus, potentially enhanced reconstructability. Conditional Entropy and Quantifying Information Gain for Data Reconstruction To quantify the information gained about the input data distribution through directional projections, we again introduce the concept of conditional entropy. Let U represent a random variable denoting the projection of input data onto a specific direction, such as a causal eigenvector, resulting in a random variable U. The conditional entropy H⁢(X∣U)conditionalH(X U)H ( X ∣ U ) measures the remaining uncertainty about the input data distribution X after we have observed the value of the projection U. It is mathematically defined as: H⁢(X∣U)=∑uP⁢(U=u)⁢H⁢(X∣U=u),conditionalsubscriptconditionalH(X U)= _uP(U=u)\,H(X U=u),H ( X ∣ U ) = ∑u P ( U = u ) H ( X ∣ U = u ) , (31) where the inner term, H⁢(X∣U=u)conditionalH(X U=u)H ( X ∣ U = u ), represents the entropy of the input data distribution given a specific value u of the projection: H⁢(X∣U=u)=−∑i=1NP⁢(X=xi∣U=u)⁢log2⁡(P⁢(X=xi∣U=u)).conditionalsuperscriptsubscript1conditionalsubscriptsubscript2conditionalsubscriptH(X U=u)=- _i=1^N\,P(X=x_i U=u)\, _2 (P(X=x_i% U=u) ).H ( X ∣ U = u ) = - ∑i = 1N P ( X = xitalic_i ∣ U = u ) log2 ( P ( X = xitalic_i ∣ U = u ) ) . (32) The crucial metric for quantifying information gain in the context of data reconstruction is the reduction in entropy of the input data distribution, which is the difference between the marginal entropy H⁢(X)H(X)H ( X ) and the conditional entropy H⁢(X∣U)conditionalH(X U)H ( X ∣ U ): H⁢(X)−H⁢(X∣U)conditionalH(X)-H(X U)H ( X ) - H ( X ∣ U ). This difference represents the amount of uncertainty about X that is eliminated by knowing U, and thus, the information gained about X from U relevant to data reconstruction. In information theory, this is directly related to Mutual Information, I⁢(X;U)=H⁢(X)−H⁢(X∣U)conditionalI(X;U)=H(X)-H(X U)I ( X ; U ) = H ( X ) - H ( X ∣ U ). A positive value for this difference indicates that U provides non-zero information about X, enhancing our ability to represent and reconstruct the input data distribution. Mathematical Proof of Information Gain for Enhanced Reconstruction from Causal Directions Our core hypothesis asserts that for a “causal” direction vjsubscriptv_jvitalic_j, the projection UjsubscriptU_jUitalic_j will yield a conditional entropy H⁢(X∣Uj)conditionalsubscriptH(X U_j)H ( X ∣ Uitalic_j ) that is strictly less than the marginal entropy H⁢(X)H(X)H ( X ): H⁢(X∣Uj)<H⁢(X).conditionalsubscriptH(X U_j)<H(X).H ( X ∣ Uitalic_j ) < H ( X ) . (33) By definition, the mutual information between X and UjsubscriptU_jUitalic_j is given by: I⁢(X;Uj)=H⁢(X)−H⁢(X∣Uj).subscriptconditionalsubscriptI(X;U_j)=H(X)-H(X U_j).I ( X ; Uitalic_j ) = H ( X ) - H ( X ∣ Uitalic_j ) . (34) Since we have hypothesized H⁢(X∣Uj)<H⁢(X)conditionalsubscriptH(X U_j)<H(X)H ( X ∣ Uitalic_j ) < H ( X ), it follows that their difference is strictly positive: H⁢(X)−H⁢(X∣Uj)>0.conditionalsubscript0H(X)-H(X U_j)>0.H ( X ) - H ( X ∣ Uitalic_j ) > 0 . (35) Therefore, we conclude that the mutual information is strictly positive: I⁢(X;Uj)>0.subscript0I(X;U_j)>0.I ( X ; Uitalic_j ) > 0 . (36) This derivation formally demonstrates that if the conditional entropy of the input data distribution, given the projection onto a direction vjsubscriptv_jvitalic_j, is less than the marginal entropy, then the mutual information between the input data distribution and the projection is strictly positive. This positive mutual information signifies that knowing UjsubscriptU_jUitalic_j, the projection onto the “causal” direction, indeed decreases the uncertainty about the input data distribution X, enhancing the quality of data representation and facilitating improved reconstruction, as evaluated by metrics like KL divergence. Direct Connection between Mutual Information and KL Divergence in VAE Setup To establish a more direct connection between maximizing mutual information and minimizing KL Divergence within our VAE framework, let’s consider the VAE objective function and its relation to these information-theoretic quantities. Recall that in our RbNS framework, we train a conditional VAE g^j⁢(ui,j)subscript^subscript g_j(u_i,j)over start_ARG g end_ARGj ( uitalic_i , j ) for each outlier direction vjsubscriptv_jvitalic_j. The VAE is trained to reconstruct input features zisubscriptz_izitalic_i (derived from input images xisubscriptx_ixitalic_i via a pre-trained encoder) from the 1D projection ui,j=vxi,j⊤⁢vjsubscriptsuperscriptsubscriptsubscripttopsubscriptu_i,j=v_x_i,j v_juitalic_i , j = vitalic_x start_POSTSUBSCRIPT i , j end_POSTSUBSCRIPT⊤ vitalic_j. The standard VAE objective function maximizes the Evidence Lower Bound (ELBO), which can be written as: ℒVAE=q⁢(z|x)[logp(x|z)]−KL(q(z|x)||p(z)),L_VAE=E_q(z|x)[ p(x|z)]-KL(q(z|x)||p(% z)),LVAE = blackboard_Eq ( z | x ) [ log p ( x | z ) ] - KL ( q ( z | x ) | | p ( z ) ) , (37) where q⁢(z|x)conditionalq(z|x)q ( z | x ) is the encoder’s approximate posterior, p⁢(x|z)conditionalp(x|z)p ( x | z ) is the decoder’s likelihood, and p⁢(z)p(z)p ( z ) is the prior distribution. In our conditional VAE setup, we are interested in the conditional distribution given the projection ui,jsubscriptu_i,juitalic_i , j. Let’s adapt the ELBO for our conditional VAE g^j⁢(ui,j)subscript^subscript g_j(u_i,j)over start_ARG g end_ARGj ( uitalic_i , j ) that reconstructs features zisubscriptz_izitalic_i from projection ui,jsubscriptu_i,juitalic_i , j. Ideally, we want to maximize the conditional mutual information I⁢(Z;X|Uj)conditionalsubscriptI(Z;X|U_j)I ( Z ; X | Uitalic_j ), which represents the information about the input features Z (derived from X) that is preserved in the projection UjsubscriptU_jUitalic_j. While a direct identity equating the VAE loss to I⁢(Z;Uj)subscriptI(Z;U_j)I ( Z ; Uitalic_j ) or I⁢(Z;X|Uj)conditionalsubscriptI(Z;X|U_j)I ( Z ; X | Uitalic_j ) isn’t straightforward due to the approximate inference in VAEs, we can highlight the intuitive link to KL Divergence. Our performance metric Perfj,k=KL(Pzi||Qz^i|ui,j(k))Perf_j,k=KL(P_z_i||Q_ z_i|u_i,j^(k))Perfj , k = KL ( Pitalic_z start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | | Qover start_ARG z end_ARG start_POSTSUBSCRIPT i | uitalic_i , j( k ) end_POSTSUBSCRIPT ) directly measures how well the VAE decoder Qz^i|ui,j(k)subscriptconditionalsubscript^superscriptsubscriptQ_ z_i|u_i,j^(k)Qover start_ARG z end_ARG start_POSTSUBSCRIPT i | uitalic_i , j( k ) end_POSTSUBSCRIPT approximates the true distribution of input features PzisubscriptsubscriptP_z_iPitalic_z start_POSTSUBSCRIPT i end_POSTSUBSCRIPT, conditioned on the projection ui,j(k)superscriptsubscriptu_i,j^(k)uitalic_i , j( k ). Consider the ideal scenario where a direction vjsubscriptv_jvitalic_j is highly ”causal” and thus, the projection UjsubscriptU_jUitalic_j captures most of the essential information about the input features Z. In this case, the conditional entropy H⁢(Z|Uj)conditionalsubscriptH(Z|U_j)H ( Z | Uitalic_j ) would be low, and the mutual information I⁢(Z;Uj)=H⁢(Z)−H⁢(Z|Uj)subscriptconditionalsubscriptI(Z;U_j)=H(Z)-H(Z|U_j)I ( Z ; Uitalic_j ) = H ( Z ) - H ( Z | Uitalic_j ) would be high. A well-trained VAE g^j⁢(ui,j)subscript^subscript g_j(u_i,j)over start_ARG g end_ARGj ( uitalic_i , j ) in this scenario, aiming to minimize reconstruction error and regularize the latent space, will learn to make the decoder distribution Qz^i|ui,j(k)subscriptconditionalsubscript^superscriptsubscriptQ_ z_i|u_i,j^(k)Qover start_ARG z end_ARG start_POSTSUBSCRIPT i | uitalic_i , j( k ) end_POSTSUBSCRIPT a close approximation of the true conditional distribution Pzi|ui,j(k)subscriptconditionalsubscriptsuperscriptsubscriptP_z_i|u_i,j^(k)Pitalic_z start_POSTSUBSCRIPT i | uitalic_i , j( k ) end_POSTSUBSCRIPT. The KL divergence KL(Pzi||Qz^i|ui,j(k))KL(P_z_i||Q_ z_i|u_i,j^(k))KL ( Pitalic_z start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | | Qover start_ARG z end_ARG start_POSTSUBSCRIPT i | uitalic_i , j( k ) end_POSTSUBSCRIPT ) then precisely quantifies the dissimilarity between these distributions. Therefore, while not a strict mathematical identity, the following intuitive connection emerges: Directions vjsubscriptv_jvitalic_j that maximize mutual information I⁢(Z;Uj)subscriptI(Z;U_j)I ( Z ; Uitalic_j ) (and thus minimize conditional entropy H⁢(Z|Uj)conditionalsubscriptH(Z|U_j)H ( Z | Uitalic_j )) are expected to enable VAE decoders g^j⁢(ui,j)subscript^subscript g_j(u_i,j)over start_ARG g end_ARGj ( uitalic_i , j ) to learn more accurate approximations of the true conditional feature distribution Pzi|ui,j(k)subscriptconditionalsubscriptsuperscriptsubscriptP_z_i|u_i,j^(k)Pitalic_z start_POSTSUBSCRIPT i | uitalic_i , j( k ) end_POSTSUBSCRIPT. This, in turn, should empirically lead to a minimization of the KL Divergence metric KL(Pzi||Qz^i|ui,j(k))KL(P_z_i||Q_ z_i|u_i,j^(k))KL ( Pitalic_z start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | | Qover start_ARG z end_ARG start_POSTSUBSCRIPT i | uitalic_i , j( k ) end_POSTSUBSCRIPT ), indicating a more faithful reconstruction of the input feature distribution and enhanced data representation. This connection provides a theoretical rationale for using KL Divergence as a performance metric in our RbNS framework. By selecting eigenvector directions that lead to lower KL Divergence in the VAE reconstruction task, we are implicitly prioritizing directions that are more informative about the input data distribution, aligning with the principle of maximizing information gain from causal directions. Appendix D Additional Experimental Results D-A Example Prompts and EigenShield’s Effect on Responses Figure 5: Example prompts and corresponding model responses before and after applying EigenShield. The images represent various sensitive topics, including religion, sexuality, and social issues. EigenShield effectively mitigates adversarial vulnerabilities by filtering out harmful or policy-violating responses while preserving semantic coherence. D-B Evaluating EigenShield Against Constrained Adversarial Attacks The results presented in Tables IV, V, and VI highlight the effectiveness of EigenShield in mitigating multimodal jailbreak attacks on Vision-Language Models (VLMs) under constrained adversarial settings (ϵ=64255italic-ϵ64255ε= 64255ϵ = divide start_ARG 64 end_ARG start_ARG 255 end_ARG. EigenShield consistently achieves the lowest attack success rates (ASR) and toxicity scores across all evaluated models, outperforming existing defenses such as adversarial training, UNIGUARD, CIDER, and image transformation-based approaches. Table IV demonstrates that EigenShield significantly reduces ASR and harmful content generation across multiple VLMs, including LLaVA-v1.5-7B, MiniGPT-4, InstructBLIP, Qwen2-VL, and Florence-2-large. Notably, EigenShield achieves a dramatic drop in ASR for LLaVA-v1.5-7B (73.73% to 17.87%) and MiniGPT-4 (41.77% to 14.82%), while also reducing toxicity and explicit content scores by a large margin. These improvements suggest that EigenShield effectively isolates and filters adversarial perturbations by projecting embeddings onto a causal subspace, preserving semantically meaningful representations while eliminating adversarial artifacts. Tables V and VI provide a comparative analysis against state-of-the-art defenses on constrained adversarial visual and textual attacks. EigenShield consistently achieves the best or second-best performance across all metrics. On LLaVA-v1.5-7B, EigenShield reduces ASR to 14.19%, outperforming adversarial training (25.48%) and inference-time defenses like CIDER (16.42%). Similarly, for MiniGPT-4, EigenShield attains the lowest ASR (15.61%) while also minimizing toxicity (13.78%). This trend continues for InstructBLIP and Qwen2-VL, where EigenShield outperforms or matches the best-performing defenses in mitigating adversarial vulnerabilities. Unlike adversarial training, which requires costly retraining and fine-tuning, EigenShield operates entirely at inference time, making it a lightweight, architecture-agnostic solution. Furthermore, while detection-based methods like CIDER rely on heuristics and predefined thresholds, EigenShield leverages Random Matrix Theory (RMT) to systematically separate causal and correlational eigenvectors, resulting in more robust and generalizable defense mechanisms. The ability to significantly reduce harmful content generation across diverse attack strategies and model architectures underscores EigenShield’s efficacy in strengthening the robustness of VLMs against adversarial threats. Table IV: RMT-Based Defense Against Multimodal Jailbreak Attacks Using Perspective API Metrics. Dataset: constrained adversarial visual attack [9] and two adversarial texts, RTP [21] and HarmBench [20] on VLMs. ‘↓ ↓’ means lower is better. Model RTP [21] HarmBench [20] Attack Identity Profanity Sexually Threat Toxicity Attack Identity Profanity Sexually Threat Toxicity Success Attack Explicit Success Attack Explicit LLaVA-v1.5-7B 73.73 16.76 59.55 30.28 34.70 69.47 58.31 12.84 42.73 22.05 20.57 54.90 +EigenShield 17.87 0.94 13.46 3.57 0.80 11.93 21.43 3.08 14.77 5.91 1.68 19.84 MiniGPT-4 41.77 2.92 29.16 13.45 2.51 36.01 39.26 5.10 26.82 14.02 3.19 33.41 +EigenShield 14.82 1.12 10.49 5.11 1.05 13.69 15.94 2.26 11.27 6.32 1.52 14.81 InstructBLIP 58.47 7.34 43.62 19.60 4.42 55.55 9.82 1.15 6.26 2.79 0.72 8.73 +EigenShield 39.06 4.61 28.39 9.07 1.96 37.83 3.21 0.48 2.08 0.97 0.24 2.71 Qwen2-VL 12.41 1.06 8.94 3.82 1.09 10.01 6.37 0.83 4.86 1.82 0.48 5.84 +EigenShield 3.78 0.42 2.21 1.02 0.20 3.05 1.92 0.27 1.31 0.58 0.14 1.75 Florence-2-large 22.68 2.71 14.41 6.28 2.17 19.35 10.34 1.64 6.72 2.84 1.21 9.27 +EigenShield 6.41 0.92 3.85 1.54 0.63 5.87 3.06 0.52 2.02 0.88 0.31 2.86 Table V: Comparison of EigenShield with state-of-the-art defenses against multimodal jailbreak attacks using Perspective API metrics. The dataset includes constrained adversarial visual attacks [9] and RTP [21] adversarial text on LLaVA-v1.5-7B and MiniGPT-4. Lower values are better. The best and second-best performances for each model are highlighted in bold and underlined. Defense LLaVA-v1.5-7B MiniGPT-4 Attack Identity Profanity Sexually Threat Toxicity Attack Identity Profanity Sexually Threat Toxicity Success Attack Explicit Success Attack Explicit No Defense 73.73 16.76 59.55 30.28 34.7 69.47 41.77 2.92 29.16 13.45 2.51 36.01 Adv. Training 25.48 2.05 21.72 8.46 1.17 19.39 25.27 2.04 19.4 9.71 1.38 21.5 UNIGUARD 19.95 1.17 17.23 5.69 0.68 13.33 21.02 1.33 14.93 7.42 0.92 16.18 BLURKERNEL 31.53 1.58 25.6 10.51 2.61 26.86 36.35 2.28 26.29 12.43 1.94 30.85 COMP-DECOMP 34.11 2.17 26.52 11.76 2.7 31.94 34.35 2.28 24.2 12.1 1.78 29.78 DIFFPURE 30.27 2.51 23.08 9.28 3.34 26.59 42.56 3.2 29.69 14.38 2.61 36.42 CIDER 16.42 1.29 15.80 5.04 0.75 12.70 18.93 1.58 15.72 6.44 1.26 16.03 EigenShield 14.19 0.88 12.6 4.34 0.59 10.93 15.61 1.29 12.83 5.11 0.84 13.78 Table VI: Comparison of EigenShield with state-of-the-art defenses against multimodal jailbreak attacks using Perspective API metrics. The dataset includes constrained adversarial visual attacks [9] and RTP [21] adversarial text on InstructBLIP and Qwen2-VL. Lower values are better. The best and second-best performances for each model are highlighted in bold and underlined. Defense InstructBLIP Qwen2-VL Attack Identity Profanity Sexually Threat Toxicity Attack Identity Profanity Sexually Threat Toxicity Success Attack Explicit Success Attack Explicit No Defense 58.47 7.34 43.62 19.6 4.42 55.55 12.41 1.06 8.94 3.82 1.09 10.01 UNIGUARD 41.03 4.92 33.11 13.68 1.83 37.86 8.72 0.83 5.97 2.45 0.53 6.78 BLURKERNEL 55.55 6.34 42.2 18.93 5.42 51.88 11.32 1.74 8.26 3.6 0.91 9.17 COMP-DECOMP 57.8 7.51 44.54 19.52 5.09 54.88 11.2 1.09 8.16 3.57 0.99 9.31 DIFFPURE 56.13 7.09 43.37 18.68 4.34 53.38 10.71 1.16 7.45 3.72 0.87 8.74 CIDER 44.51 5.38 37. 19 11.93 1.75 39.83 9.35 0.97 6.49 2.73 0.66 7.18 EigenShield 39.06 4.61 28.39 9.07 1.96 37.83 3.78 0.42 2.21 1.02 0.20 3.05

Instant research discovery