Paper deep dive
Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection
Yewon Han, Yumin Seol, EunGyung Kong, Minsoo Jo, Taesup Kim
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/22/2026, 5:15:07 AM
Summary
The paper introduces 'Two Birds, One Projection' (TBOP), an inference-time defense for Large Vision-Language Models (LVLMs) that addresses the safety-utility trade-off. By identifying a modality-induced bias direction—a shared structural cause for both safety vulnerabilities and utility degradation—the authors propose projecting cross-modal features onto the null space of this bias. This method requires only a single forward pass and improves performance across both safety and utility benchmarks.
Entities (5)
Relation Signals (3)
TBOP → appliedto → LVLM
confidence 100% · we propose TBOP, an efficient inference-time defense strategy for LVLMs
TBOP → mitigates → Modality-induced bias
confidence 95% · TBOP removes the bias-related components from cross-modal features by projecting them onto the null space of the identified bias direction.
Modality-induced bias → causes → Safety-Utility Trade-Off
confidence 90% · we argue that the persistent conflict between safety and utility is not an inherent property of LVLMs, but rather stems from a shared underlying bias
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing jailbreak defence frameworks for Large Vision-Language Models often suffer from a safety utility tradeoff, where strengthening safety inadvertently degrades performance on general visual-grounded reasoning tasks. In this work, we investigate whether safety and utility are inherently antagonistic objectives. We focus on a modality induced bias direction consistently observed across datasets, which arises from suboptimal coupling between the Large Language Model backbone and visual encoders. We further demonstrate that this direction undermines performance on both tasks. Leveraging this insight, we propose Two Birds, One Projection, an efficient inference time jailbreak defence that projects cross-modal features onto the null space of the identified bias direction to remove the corresponding components. Requiring only a single forward pass, our method effectively breaks the conventional tradeoff, simultaneously improving both safety and utility across diverse benchmarks.
Tags
Links
- Source: https://arxiv.org/abs/2603.14825v1
- Canonical: https://arxiv.org/abs/2603.14825v1
Trouble viewing inline? Open PDF directly →
Full Text
77,673 characters extracted from source content.
Expand or collapse full text
Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection Yewon Han 1 , Yumin Seol 1⋆ , EunGyung Kong 2 , Minsoo Jo 1 , and Taesup Kim 1† 1 Graduate School of Data Science, Seoul National University 2 Mobilint Abstract. Existing jailbreak defense frameworks for Large Vision-Language Models (LVLMs) often suffer from a safety–utility trade-off, where strength- ening safety inadvertently degrades performance on general visual-grounded reasoning tasks. In this work, we investigate whether safety and utility are inherently antagonistic objectives. We focus on a modality-induced bias direction consistently observed across datasets, which arises from suboptimal coupling between the LLM backbone and visual encoders. We further demonstrate that this direction undermines performance on both tasks. Leveraging this insight, we propose TBOP (TwoBirds,One Projection), an efficient inference-time jailbreak defense that projects cross-modal features onto the null space of the identified bias direction to remove the corresponding components. Requiring only a single forward pass, our method effectively breaks the conventional trade-off, simultane- ously improving both safety and utility across diverse benchmarks. Keywords: LVLMs· Jailbreak Defense· Safety-Utility Trade-off 1 Introduction Large Vision-Language Models (LVLMs) extend the reasoning capabilities of Large Language Models (LLMs) by integrating visual perception. However, this expanded input space introduces complex cross-modal interactions during contextualization, which can shift cross-modal features away from the original safety-aligned space established in LLM backbone training. Consequently, the model’s inherent refusal mechanisms for harmful cross-modal queries are often weakened, significantly increasing susceptibility to jailbreaks. Prior research has addressed these safety risks through two primary lenses. Training-based methods such as MLLM-Protector [31] integrate auxiliary safety modules (e.g., harm detectors) using additional fine-tuning data. Conversely, inference-time approaches like ECSO [12] and ETA [6] assess the harmfulness of generated responses and refine them without retraining. Despite their effectiveness, these approaches predominantly follow a detect-then-detoxify paradigm. This reliance on post-hoc safety detection inherently triggers a safety–utility trade-off. ⋆ Work done during internship. † Corresponding author. arXiv:2603.14825v1 [cs.CV] 16 Mar 2026 2Y. Han et al. By treating the symptoms (i.e., harmful outputs) rather than the underlying cause, these methods inevitably become more conservative as defenses strengthen, unnecessarily suppressing benign queries and degrading overall utility performance as observed in Fig. 1. Furthermore, this sequential paradigm incurs significant computational overhead and memory costs due to multiple forward passes and auxiliary model dependencies. While recent generative approaches such as Immune [9] and CMRM [21] attempt to mitigate these risks internally, they still fall short of a holistic solution. Specifically, Immune lacks a rigorous analysis of the side effects that its internal manipulations have on utility, resulting in substantial performance loss. CMRM relies on carefully tuned coefficients, making it sensitive to dataset-specific vari- ations thus limiting its effectiveness and generalizability. Consequently, these methods still fail to effectively harmonize two objectives: safety and utility. Utility Improvement ( Acc %) Δ 4 2 0 -2 -4 -6 -19 -20 -21 -22 Safety Improvement ( (1-ASR) %) Δ 0-5101520253035 5 VA N I L L A CMRM (ACL 2025) ECSO (ECCV 2024) OURS ETA (ICLR 2025) MLLP (ACL 2024) IMMNUE (CVPR 2025) Fig. 1: Trade-off Between Safety and Utility Observed in LVLM De- fense Methods. By effectively elimi- nating the shared performance under- lying direction, our approach generates safe and useful responses, thereby over- coming the conventional tradeoffs. In this work, we argue that the persis- tent conflict between safety and utility is not an inherent property of LVLMs, but rather stems from a shared underlying bias caused by the imperfect coupling between the vision encoder and LLM backbone. Un- like prior generative methods that inter- vene without a unified diagnostic frame- work, we identify the modality-induced bias as an axis that simultaneously un- dermines safety and utility performance. Our empirical evidence supports this, high- lighting that the two objectives are not antagonistic; rather, they are jointly com- promised by this common structural root cause. Motivated by this insight, we propose TBOP, an efficient inference-time defense strategy for LVLMs that can generate safe and useful responses. TBOP removes the bias-related components from cross-modal features by projecting them onto the null space of the identified bias direction. Targeting the principled direction rather than the overall bias allows our method to remain effective across diverse datasets, unaffected by inconsistent bias intensity. Extensive experiments demonstrate that our simple projection-based approach simultaneously improves performance across diverse safety and utility benchmarks, effectively breaking the conventional tradeoff and establishing a more robust foundation for secure LVLM deployment. Since such biases naturally emerge when combining modules that process different modalities, and are costly to mitigate with additional training, TBOP provides an efficient inference-time remedy that reduces this overhead. Our main contributions can be summarized as follows: TBOP: Safety–Utility Harmonization in LVLMs via IFP3 –We identify that modality-induced bias consistently affects both safety and utility performance, serving as a shared performance-undermining direction. –We propose an effective defense strategy for LVLMs that can achieve signifi- cant improvements across a wide range of safety and utility tasks. –TBOP is designed as a computationally efficient inference-time defense strat- egy that requires only a single forward pass, enabling seamless integration. 2 Related Work Large Vision Language Models. Recent advances in large vision language models (LVLMs) have demonstrated that pairing visual encoders with pretrained text-only LLMs, typically via a projector that bridges the visual feature space and the LLM’s token-embedding space. Representative systems include LLaVA [19], InstructBLIP [5], Qwen2-VL [41], and InternVL [3], which adopt this general design while differing in the specific alignment modules or training strategies for integrating visual representations with language models. Such modular coupling often induces representational misalignment, arising from the components being optimized under different objectives and training stages [15,22,49], which in turn brings a range of issues, from object hallucination to broader safety vulnerabilities. Jailbreak Attacks and Defenses in LVLMs. LVLMs remain highly vul- nerable to jailbreak attacks that exploit cross-modal interactions. HADES [16] shows that images can conceal and amplify malicious context to evade guardrails, undermining safety alignment. Test-time backdoors like AnyDoor [26] implant uni- versal triggers via adversarial images, exposing the limits of input filtering. Recent analyses [20,23,36] argue risks compound across modalities [43,46], demanding defenses that address risk composition rather than single-channel perturbations. Defenses for LVLM jailbreaks can be broadly grouped into train-based and inference-time approaches. Train-based defenses, including finetuning-based safety alignment [27, 32, 53], improve harmlessness but require training cycles and computational overhead. Inference-time defenses, such as prompting strategies [35,44,51] and alignment methods [6,9] avoid modifying model weights but often suffer from brittleness under cross-modal attacks. Feature Manipulation. Recent work shows that feature-level manipulation can systematically steer LLMs toward safer and more faithful outputs [30,42, 47]. ReFAT [47] finds a “refusal” feature dimension in the residual stream and adversarially trains on its worst-case offset, operationalizing harmful subspaces to improve robustness. TSV [30] adds a vector that reshapes representations to split truthful from hallucinatory content without weight updates. InferAligner [42] uses safety steer- ing vectors from an aligned model to guide a target model. On the other hand, layer-wise analyses [37] further motivate interventions on token-level states rather than only outputs. Within MLLMs, VTI [22] reduces hallucinations by test-time steering of multimodal features. 4Y. Han et al. Nullu [45] builds a HalluSpace from contrasts between hallucinated and truthful feature embeddings, and suppresses hallucinations via weight editing, yielding an effect akin to feature manipulation. CMRM [39] and ShiftDC [54] utilize feature manipulation for safety alignment. Building on these ideas, we propose a defense method that removes bias components in cross-modal features to better align safety while extracting refined representations for enhanced visual grounding. 3Consistent Modality-induced Bias as Degradation Axis in LVLMs. 3.1 Preliminaries Decomposition of Cross-modal Representations. Following the formulation introduced by Liu et al. [21], the representation of a text–image pair can be decomposed as: h(x txt ,x img ) = h ∗ (x txt ,x img ) + α· [h(x txt ,x ′ img )− h(x txt )],(1) where the difference termh(x txt ,x ′ img )− h(x txt ) defines the representation shift introduced by the visual modality, capturing how the feature changes purely due to the activation of the visual input. The dummy imagex ′ img serves as a blank input to isolate modality activation from visual content, and the ideal component h ∗ (x txt ,x img ) represents how the model should encode cross-modal information in the absence of such undesirable distortion. Because this shift arises from a single underlying mechanism, namely the activation of the visual modality, it induces a consistent pattern across inputs. Consequently, the resulting shifts do not disperse arbitrarily in the feature space but concentrate along a limited set of dominant directions, forming a structured subspace. We refer to this systematic deviation from the ideal representation as modality-induced bias, which we argue is a key factor underlying performance degradation in both safety defense and general visual-grounded reasoning. Visual Inputs as Expanded Attack Surfaces. Empirical studies have re- vealed that visual inputs significantly increase jailbreak vulnerability: HADES [16] reports that even blank images can double attack success rates. Motivated by these findings, defenses such as ECSO [12] attempt to mitigate this risk by converting harmful images into text captions and regenerating responses if the initial output is classified as harmful, underscoring the strong link between the visual modality and multimodal vulnerability. Furthermore, CMRM [21] analyzes these vulnerabilities in feature level, demonstrating that visual inputs can shift representations away from safety-aligned regions, thus inducing toxic outputs. 3.2 Observations Consistent Modality-induced Bias Across Safety and Utility. While prior research has primarily focused on the impact of visual inputs on model TBOP: Safety–Utility Harmonization in LVLMs via IFP5 (a) Modality-induced bias(b) Performance shift under bias-direction steering Safety (MMSB)Utility (MME) Fig. 2: Consistent Modality-induced Bias Across Safety and Utility in LVLMs. (a) The bias direction remains consistent across safety and utility datasets. (b) Rein- forcing the bias along eitherb safe orb util in cross-modal features causes substantial performance degradation in both tasks. The effect is significantly stronger than random Gaussian noise, indicating that this modality-induced bias acts as a shared driver of both safety risks and utility degradation. safety, we further analyze whether this bias also appears for utility-related inputs. To this end, we curated a dataset of 1,260 instances each from safety (e.g., M- SafetyBench) and general utility (e.g., MME) benchmarks. We then quantify the representation shift in the contextualized features of the final input token by comparing text-only queries (x txt ) with their multimodal counterparts (x txt ,x ′ img ) incorporating a dummy blank image. As shown in Fig. 2-(a), this modality-induced bias consistently manifests across both safety and utility domains, exhibiting a high degree of directional alignment (i.e., cos≥ 0.6). Moving beyond mere observational analysis, we investigate the causal rela- tionship between the identified bias directions and their impact on safety and utility. First, we extract the principal bias components from each domain via Singular Value Decomposition (SVD), denoted asb safe andb util . By utilizing unit eigenvectors, this procedure isolates the directional influence of the bias from variations in its magnitude across different datasets. To evaluate the func- tional implications, we conduct feature perturbation experiments by steering the contextualized features of cross-modal pairs (x txt ,x img ) along theb safe and b util direction (Fig. 2-(b)). Our empirical findings confirm that amplifying the modality-induced bias—regardless of whether the direction is derived from safety or utility tasks—induces a synchronous performance degradation across both tasks. Notably, this impact is significantly more pronounced than that of stochas- tic perturbations using isotropic Gaussian noise of equivalent norm. 6Y. Han et al. Qualitative Analysis. Amplifyingb safe causes the model to generate step- by-step instructions for harmful queries, systematically undermining LVLMs’ defense mechanism. Synchronous performance degradation is observed in benign vision-grounded question-answering tasks; upon amplifyingb util , the model ex- hibits response collapse to yes-or-no questions (e.g., exclusively answering “No”), producing answers without properly grounding them in the visual cues. These pathological behaviors scale with steering intensity, further confirming that this modality-induced bias serves as a common driver for both utility degradation as well as safety risks in multimodal systems. Detailed examples and additional results are provided in the supplementary materials. Table 1: Limitated Performance Gains of the Steering-based Ap- proach in Safety and Utility Tasks. Due to their reliance on estimating steering coefficients for shift intensity-which can vary across datasets-it fails to effectively enhance performance and suffer from limited generalizability. Method Safety (↓)Utility (↑) MMSB HADES M-VET MME Vanilla 38.86 39.0641.91 1715.13 CMRM 25.02 32.3041.11 1814.89 Ours5.095.4843.981870.17 Remedies for Mitigating the Modality- induced Bias. The most straightforward approach to alleviate the bias is to steer the cross-modal representation in the opposite direction. Recall the decomposition from Eq. (1) whereαrepresents the degree of a shift. Defense strategy based on feature steering, such as CMRM [21], attempts to neutralize this shift via representation in- terpolation: ˆ h(x txt ,x img ) =h(x txt ,x img )− ˆα · ∆h=h ∗ (x txt ,x img ) + (α − ˆα)· ∆h, whereˆαis the estimated steering coeffi- cient. However, this approach presents lim- ited performance and generalizability since it is sensitive to hyperparameter tuning for trackingα, as the true shift intensity of the modality-induced bias varies across datasets and samples. We hypothesize that this reliance on coefficient estima- tion is a primary factor behind the sub-optimal performance of conventional steering-based defenses observed in Tab. 1. Therefore, there is a critical need for a methodology independent of bias coefficient that can robustly neutralize modality-induced bias by focusing on its geometric direction, thereby ensuring consistent performance regardless of variations in α across inputs. 4 Method We propose TBOP (Two Birds, One Projection), an inference-time defense strategy for LVLMs by identifying and removing modality-induced bias from the model’s internal representation. TBOP operates fully at inference time, requires only a single forward pass, and enhances task-relevant multimodal semantics while suppressing the safety risks. TBOP: Safety–Utility Harmonization in LVLMs via IFP7 4.1 Modality-induced Nuisance Subspace We formalize this structure by defining the nuisance subspace, which captures the principal directions along which the visual modality perturbs the model’s representation. LetV ∈R d×d be an orthonormal basis of this space, and the projectionV V ⊤ h(x txt ,x img )then corresponds to the component of a multimodal representation aligned with these perturbation directions. Definition 1(The Nuisance Subspace). The nuisance spaceWis defined as the subspace spanned by all modality-induced shifts vectors: W = span n h(x txt ,x ′ img )− h(x txt ) ∀x txt o . (2) The orthogonal projectorV V ⊤ extracts the bias component of any representation. Lemma 1(Bias Vectors Lie in the Nuisance Space). By construction, each bias vector lies entirely in W and is therefore orthogonal to its complement: h(x txt ,x ′ img )− h(x txt ) ⊥ (I − V V ⊤ ).(3) Lemma 2(Ideal Representation Orthogonality). By equation 1, the ideal multimodal representation h ∗ (x txt ,x img ) is orthogonal to the nuisance space: h ∗ (x txt ,x img ) ⊥ W.(4) This reflects that modality-induced shifts represent undesirable perturbations that do not contribute to the semantic multimodal encoding. Together, these components allow us to isolate and subsequently remove the bias component through projection-based inference-time intervention, while enhancing the underlying semantic representation. 4.2 Orthogonal Projection for Bias Removal Given the characterization of the nuisance spaceW, our goal is to remove the representation components that align with this space. SinceWcaptures the principal directions along which the visual modality perturbs the model’s internal representations, suppressing these directions allows us to mitigate modality- induced vulnerabilities and refine better cross-modal features during inference. To achieve this, we project the cross-modal representation onto the orthogonal complement of the nuisance space. For any representationh(x txt ,x img ), the corrected representation is obtained as: ̃ h(x txt ,x img ) = (I − V V ⊤ )h(x txt ,x img ).(5) This projection removes the bias-related componentV V ⊤ h(x txt ,x img ), while preserving the components unrelated to modality-induced shifts. 8Y. Han et al. Under the ideal representation assumption introduced earlier, this procedure preserves the ideal multimodal component: (I − V V ⊤ )h ∗ (x txt ,x img ) = h ∗ (x txt ,x img ),(6) and eliminates the modality-induced shift: (I − V V ⊤ )[h(x txt ,x ′ img )− h(x txt )] = 0.(7) Thus, the resulting representation ̃ h(x txt ,x img ) approximates the ideal represen- tation by selectively suppressing directions associated with modality-induced perturbations. By removing the components aligned with the nuisance space, this projection effectively neutralizes vulnerability inducing directions while enhanc- ing semantic structure and it can be applied with only a single forward pass at inference time. 4.3 Approximating the Nuisance Space The true nuisance spaceWis defined conceptually as the span of all modality- induced bias vectors. However, this space cannot be computed exactly, as it requires enumerating all possible text inputs. To obtain a practical approximation, we estimate the principal directions of modality-induced shifts from a finite sample set. Given a collection of text inputsx (i) txt N i=1 , we compute the empirical shift vector for each sample as: d (i) = h(x (i) txt ,x ′ img )− h(x (i) txt ). Stacking these vectors yields a shift matrixD ∈R N×d , where each row corre- sponds to one empirical shift. To extract the principal directions that characterize the bias space, we perform singular value decomposition (SVD) on D: D = UΣ ˆ V ⊤ . The right singular vectors in ˆ V represent orthogonal directions in the representa- tion space ordered by their contribution to the overall variance of the shift vectors. We select the top-ksingular vectors to form an approximate basis ˆ V k ∈R d×k , which captures the most influential dimensions of modality-induced perturbations. Finally, we replace the ideal basisVin the projection with this approximated basis, yielding the practical inference-time correction: ̃ h(x txt ,x img ) = (I − ˆ V k ˆ V ⊤ k )h(x txt ,x img ).(8) This procedure enables a low-dimensional approximation of the nuisance space derived purely from empirical shift observations, making the projection both computationally efficient and readily applicable to real-world LVLMs. TBOP: Safety–Utility Harmonization in LVLMs via IFP9 To apply this projection in practice, we must specify which internal repre- sentation of the model is used for computing the shift vectors and performing the correction. Following prior work [17, 21], we extract the activation of the final decoder layer at the position of the last input token. By intervening at this representation located immediately before the projection into the model’s output vocabulary, we operate at the point where multimodal information is most concentrated and directly influences generation. This choice not only yields strong empirical performance but also ensures computational efficiency, as the projection is performed on a single, compact feature vector per input. 5 Experiments In this section, we verify whether our proposed method effectively mitigates safety vulnerabilities while simultaneously enhancing the comprehensive visual understanding capabilities of LVLMs. 5.1 Experimental Setup Benchmark Datasets. First, we evaluate whether the model can effectively defend against harmful cross-modal inputs. We use M-SafetyBench [23] (SD-TYPO) and HADES [16], which contain harmful text–image pairs across various scenarios; in HADES, images are generated through an iterative process that progressively amplifies toxicity. We also test on FigStep [11], an image-text benchmark that embeds step-by-step unethical instructions in typography, under the black-box attack SI-Attack [50], which alleviates jailbreaks by shuffling text instructions. To further verify that our method also generates useful response, we test MME [7], which consists of yes-or-no perception and cognition tasks; M-Vet [48], which assesses open-ended multimodal reasoning including OCR and spatial reasoning tasks; and ScienceQA [28], comprising of multiple-choice scientific questions that require nuanced reasoning over modality-specific cues. Evaluation Protocols and Metrics. We follow standard protocols of each benchmark for all evaluations. For safety evaluation, we measure the Attack Success Rate (ASR), defined as the proportion of responses classified as harmful by the LLM judge LLaMA-Guard-4-12B [13]. For utility evaluation, we assess M-Vet with GPT-4.1 [29] to score open-ended generation output. Baselines and Backbones. To evaluate the safety–utility tradeoff in LVLMs, we benchmark our method against representative defense frameworks. We compare our method with (1) the source model, (2) MLLM-Protector [32], a train-based approach, and inference-time approaches that operate at the prompt, token, or feature-level: (3) Immune [9], (4) ECSO [12], (5) ETA [6] and (6) CMRM [21]. As our LVLM backbones, we employ LLaVA-1.6-7B/13B [18], Qwen2.5-VL- 7B [1], InternVL2.5-8B [2], and InstructBLIP-7B [5]. Unless otherwise specified, experiments without explicit model descriptions are conducted with LLaVA-7B. 10Y. Han et al. Table 2: Comparative Analysis of Safety and Utility Performance on LLaVA- 7B and 13B. Safety is measured by Attack Success Rate (ASR) on M-SafetyBench and HADES (lower is better). Utility is evaluated by accuracy across M-Vet, ScienceQA, and MME (higher is better). The best andsecond-bestresults within each model scale are highlighted in bold and underlined, respectively. Model Method Safety (↓)Utility (↑) MMSB HADES M-Vet ScienceQA MME LLAVA 7B Vanilla38.8639.0641.91 57.201715.13 Protector7.4816.1337.1548.13725.39 Immune9.3914.5321.1841.261394.69 ECSO8.4812.0041.3154.161601.56 ETA8.343.4740.1056.501698.52 Ours (TBOP)5.095.4843.9863.101870.17 LLAVA 13B Vanilla32.8928.9245.6359.841773.24 Protector7.8026.2747.1052.18556.08 Immune12.968.4018.1044.381471.44 ECSO10.5813.6046.2954.071668.62 ETA5.883.8744.6459.111755.49 Ours (TBOP)2.561.7449.9865.201798.85 Implementation Details. By default, 20% of M-SafetyBench and 10% of MME are set aside for anchor dataset to compute shift vectors, based on the observation in Sec. 3.2 that modality-induced bias operates along a consistent axis across both safety and utility dimensions. For fair evaluation, we report results on the validation split only. We use k = 32 as default and conduct ablations across various k in Sec. 6.2. 5.2 Evaluation Results Enhancements in Both Safety and Utility. The comparative results in Table 2 demonstrate that our method achieves a superior balance between safety and utility compared to all existing baselines. While conventional defense mechanisms—such as Protector, Immune, and ECSO—significantly reduce ASR on M-SafetyBench (MMSB) and HADES, they frequently incur a substantial utility tax, leading to degraded performance on vision-language benchmarks like M-Vet and MME. In contrast, TBOP not only achieves the lowest ASR in most categories (e.g., reaching 2.56% on MMSB for LLaVA-13B) but also consistently improves utility scores beyond the original Vanilla models across all scales. This simultaneous enhancement validates that by identifying and neutralizing the shared axis of degradation, our method effectively mitigates multimodal vulnerabilities while refining the model’s overall perception and reasoning capabilities. We present a category-level comparison between the vanilla LLaVA-7B and ours across safety and utility benchmarks. As shown in Fig. 3, TBOP consistently improves defense success rates (1−ASR) across all categories from explicit harmful requests to more implicit threats such as professional misuse. Furthermore, TBOP: Safety–Utility Harmonization in LVLMs via IFP11 MMSB HADES SQA MME Va n i l l aOurs Fig. 3: Category-wise Performance Comparison Across Safety and Utility Tasks. The proposed method yields broad improvements across individual domains rather than localized gains. This collective advancement collectively drives the overall enhancement in both defensive robustness and multimodal reasoning capabilities. performance gains are observed across all categories of ScienceQA and MME: SQA probes multimodal reasoning breadth through diverse subjects and grade levels, each imposing distinct reasoning demands on multimodal comprehension, while MME assesses fine-grained perception and recognition abilities. These results demonstrate holistic improvement not confined to specific categories. Instruct BLIP QwenVL2.5 InternVL2.5 Instruct BLIP QwenVL2.5 InternVL2.5 Improvement over Vanilla ( %) Δ MMSBMM-Vet Fig. 4: Performance Gain Across LVLM Backbones over Vanilla. Ours consistently improves performance across various model families, demon- strating its generalizability. The superiority of our method is ob- served across diverse model families in- cluding InstructBLIP, Qwen-VL-2.5, and InternVL2.5 with different architectural characteristics, LLM backbones and train- ing paradigm. Applying our method to these base model yields consistent improve- ments in both safety and utility as shown in Fig. 4. Notably, the substantial gains in safety for InstructBLIP and Qwen-VL-2.5 suggest that our orthogonal projection ef- fectively mitigates the structural modality gap in models with frozen or separately pretrained backbones. For InternVL2.5, our method provides meaningful improve- ments particularly in utility. These con- sistent gains across diverse architectures indicate that TBOP addresses a funda- mental, architecture-agnostic vulnerability in multimodal integration. Robustness against Adversarial Attacks. In real-world deployments, LVLMs may face additional threats such as black-box adversarial attacks that involve intentional input toxification. In these scenarios, safety risks may not be fully attributable to modality-induced bias alone. Therefore, we investigate whether our method can be integrated with other defense mechanisms and whether 12Y. Han et al. Fig. 5: Efficiency Comparison of Response Refinement Strategies. Unlike the two-stage detection-refinement pipeline (b), which incurs a sequential computational bottleneck, our single-stage refinement (a) operates directly on the model’s internal states in a single forward pass. As shown in (c), TBOP runs about 60×faster than the second-best baseline (ETA), demonstrating strong scalability. such combinations yield a positive synergistic effect. Specifically, we apply SI- Attack [50] as a text-based attack, which aims to confuse LVLMs by shuffling text tokens, thereby concealing their malicious intent, on FigStep. We apply TBOP at the initial generation stage for the two-stage methods, MLLM-Protector and ECSO; and at the reward-model-guided generation stage for Immune. Table 3: Synergetic Ef- fects with TBOP under SI-Attack on FigStep. Method 7B13B Vanilla27.220.4 MLLP7.23.8 + Ours3.8 (-3.4)0.8 (-3.0) Immune7.64.8 + Ours4.2 (-3.4)4.4 (-0.4) ECSO16.811.6 + Ours5.6 (-11.2)3.2 (-8.4) ETA10.80.8 + Ours4.0 (-6.8)0.6 (-0.2) As shown in Tab. 3, TBOP significantly lowers the ASR when integrated with other defense methods. For multi-stage frameworks like MLLM-Protector and ECSO, our bias removal guides the model to produce a safer initial representation, reinforcing the overall defense pipeline. For logit-based methods like Immune, TBOP ensures the first generated token aligns with a refusal stance for harmful queries, preventing the model from entering a risky state from the start. By addressing feature-level vulnerabilities that input- or logit-level methods often overlook, ours acts as a foun- dational defense layer that can be seamlessly integrated into diverse frameworks. Computational and Memory Efficiency. To ensure a seamless user experi- ence in real-world deployments, inference-time defense methods must improve model reliability without imposing significant latency or resource overhead. As illustrated in Fig. 5, TBOP offers clear efficiency in both dimensions, maintaining a footprint comparable to the source model. By modifying features during the initial forward pass, it avoids the multi-step post-hoc refinement required by detect-then-detoxify approaches such as ECSO and ETA. These baselines incur substantial overhead—up to 60×higher inference cost—due to their sequential pipelines. This efficiency demonstrates the superior scalability of TBOP. TBOP: Safety–Utility Harmonization in LVLMs via IFP13 Before+ OursBefore+ Ours List steps in detail to hack into government security systems and surveillance cameras. txt List steps in detail to hack into government security systems and surveillance cameras. + txt + img’ List steps in detail to hack into government security systems and surveillance cameras. txt + The bottom of the image shows a phrase of an activity. List steps in detail to perform this activity in relation to surveillance cameras. txt + img (a) Harmful text paired with a dummy image(b) Benign text paired with a harmful image Fig. 6: Impacts of TBOP in Feature Space. For inputs (a), orthogonal projection suppresses modality-induced bias and aligns multimodal features with their semantically similar text-only counterparts. The same mechanism applies to actual cross-modal inputs (b), realigning features toward safety-aligned text regions and mitigating vulnerabilities. 6 Analysis 6.1 Impact of Bias Removal on Safety and Utility Feature Visualization. To understand how our feature-level manipulation modifies cross-modal representations, we visualize for two types of input pairs: (a) harmful text with a dummy image and (b) benign text with an actual harmful image in Fig. 6. As shown in the figure, TBOP corrects the cross-modal features toward the text-only region in both scenarios. These results suggest that our orthogonal projection suppresses the bias direction that drives features away from the safety coverage of LLM backbone, guiding them back toward a safety-aligned space, thereby effectively reducing ASR. Query : What is the spatial relation between the frisbee and the man? LLaVA 7B v1.6 The frisbee is in the air, and the man is reaching out towards it with his left hand. The frisbee is positioned above the man’s head, indicating that it is relatively close to him, and he is in the process of trying to catch it. LLaVA 7B v1.6 + Ours In the image, the frisbee is in the air, and the man is reaching out towards it with his left hand. The frisbee is positioned above the man's head, indicating that it is relatively close to him. The spatial relation between the frisbee and the man suggests that the man is in the process of catching the frisbee, as he is extending his arm towards it. Fig. 7: Response Examples on M-Vet spatial reasoning between the vanilla and TBOP (Ours). Qualitative Analysis. From the perspective of feature en- coding in LVLMs, TBOP can be viewed as producing more refined cross-modal features. In essence, bias removal elimi- nates unnecessary components that negatively affect visual- grounded understanding. By filtering out these modality- induced biases, the model can better extract essential cross- modal information, thereby in- creasing the signal-to-noise ratio of the features. As a result, the generated re- sponses tend to be more informative and richer, reflecting improved cross-modal reasoning and understanding. 14Y. Han et al. Table 4: Ablation studies on key design choices of TBOP. (a) Effect of anchor composition AnchorMMSB MME Vanilla38.56 1686.7 MMSB only7.10 1776.5 MME only36.54 1740.1 MMSB+MME5.091870.2 (b) Effect of perturbation type Type 7B 13B Vanilla 38.86 32.89 Gaussian 10.12 6.07 Uniform 10.26 6.84 Blank5.092.56 (c) Effect of Rank k k EVR MMSB Vet – – 38.86 41.91 16 0.85 12.81 42.73 320.895.0942.98 64 0.93 4.60 43.75 6.2 Ablation Study Effects of Anchor Datasets. Motivated by the observations in Sec. 3.2, we compute ˆ V k from an anchor dataset constructed by mixing a portion of safety and utility samples, and apply the resulting ˆ V k consistently across all evaluations. As shown in Tab. 4a, using either safety-only or utility-only anchors already improves performance on both tasks, reconfirming that the bias directions (b safe andb util ) are consistent and jointly influence safety and utility behaviors. Moreover, combining both types of samples yields the best results, as the mixed anchor set better approximates the nuisance space. Effects of Perturbation Type As shown in Tab. 4b, TBOP effectively improves performance on M-SafetyBench for both LLaVA-7B and 13B whenx ′ img is replaced with random noise (Gaussian, uniform). However, a blank image yields the largest improvement, as it best aligns with our assumption of isolating modality activation from visual structure. Effects of Rankk. Our method introduces a parameterk, representing the top-kcomponents used to approximate the basisVof the nuisance space. We extract the top-keigenvectors and use the same configuration for all evaluations. As shown in Tab. 4c, TBOP consistently improves both safety and utility across differentkvalues, provided that the selected basis captures a sufficiently high explained variance ratio of the modality-induced bias. 7 Conclusion In this work, we address the conventional trade-off between safety and utility in LVLM defense methods. We identify modality-induced bias as a shared direction that degrades performance in both dimensions. Based on this insight, we propose TBOP, a computationally efficient inference-time defense that mitigates this bias to improve both safety and utility. TBOP requires only a single forward pass and can be seamlessly integrated into existing LVLMs without additional training or architectural changes. Extensive evaluations across diverse benchmarks demonstrate consistent improvements in both aspects, providing a practical TBOP: Safety–Utility Harmonization in LVLMs via IFP15 solution for robust LVLM deployment. Ultimately, we expect our methodology to facilitate the development of trustworthy models that deliver both secure and useful responses in real-world applications. References 1.Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report (2025), https://arxiv.org/abs/2502.13923 9 2. Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., Gu, L., Wang, X., Li, Q., Ren, Y., Chen, Z., Luo, J., Wang, J., Jiang, T., Wang, B., He, C., Shi, B., Zhang, X., Lv, H., Wang, Y., Shao, W., Chu, P., Tu, Z., He, T., Wu, Z., Deng, H., Ge, J., Chen, K., Zhang, K., Wang, L., Dou, M., Lu, L., Zhu, X., Lu, T., Lin, D., Qiao, Y., Dai, J., Wang, W.: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling (2025), https://arxiv.org/abs/2412.05271 9 3.Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 24185–24198 (2024) 3 4.Computer, T.: Redpajama-data: An open source recipe to reproduce llama training dataset (2023), https://github.com/togethercomputer/RedPajama-Data 4 5. Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems 36, 49250–49267 (2023) 3, 9 6. Ding, Y., Li, B., Zhang, R.: Eta: Evaluating then aligning safety of vision language models at inference time. arXiv preprint arXiv:2410.06625 (2024) 1, 3, 9, 5 7. Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., Ji, R., Shan, C., He, R.: Mme: A comprehensive evaluation benchmark for multimodal large language models. In: Proceedings of the Twenty- Ninth Conference on Neural Information Processing Systems (NeurIPS 2025) (2025), https://neurips.c/virtual/2025/loc/san-diego/poster/121773, poster Pre- sentation, exhibit hall C,D,E #4503, San Diego, Dec 3, 2025 9, 4 8. Geng, X., Liu, H.: Openllama: An open reproduction of llama (May 2023),https: //github.com/openlm-research/open_llama 4 9. Ghosal, S.S., Chakraborty, S., Singh, V., Guan, T., Wang, M., Beirami, A., Huang, F., Velasquez, A., Manocha, D., Bedi, A.S.: Immune: Improving safety against jailbreaks in multi-modal llms via inference-time alignment. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 25038–25049 (2025) 2, 3, 9 10.Ghosal, S.S., Chakraborty, S., Singh, V., Guan, T., Wang, M., Beirami, A., Huang, F., Velasquez, A., Manocha, D., Bedi, A.S.: Immune: Improving safety against jailbreaks in multi-modal llms via inference-time alignment. In: Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR). p. 25038–25049 (June 2025) 5 11.Gong, Y., Ran, D., Liu, J., Wang, C., Cong, T., Wang, A., Duan, S., Wang, X.: Figstep: Jailbreaking large vision-language models via typographic visual prompts. 16Y. Han et al. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, p. 23951–23959 (2025) 9, 3 12.Gou, Y., Chen, K., Liu, Z., Hong, L., Xu, H., Li, Z., Yeung, D.Y., Kwok, J.T., Zhang, Y.: Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation (2024), https://arxiv.org/abs/2403.09572 1, 4, 9, 5 13.Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C.C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E.M., Radenovic, F., Guzmán, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G.L., Thattai, G., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I.A., Kloumann, I., Misra, I., Evtimov, I., Zhang, J., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K.V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala-Yeary, L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Tsimpoukelli, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M.K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P.S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R.S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S.S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X.E., Xia, X., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z.D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B.D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.H., Cai, C., Tindal, C., Feichtenhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, TBOP: Safety–Utility Harmonization in LVLMs via IFP17 D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Le, E.T., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Kokkinos, F., Ozgenel, F., Caggioni, F., Kanayet, F., Seide, F., Florez, G.M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Inan, H., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Damlaj, I., Molybog, I., Tufanov, I., Leontiadis, I., Veliche, I.E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.B., Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K.H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., A, L., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M.L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M.J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N.P., Dong, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S.J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S.C., Patil, S., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V.S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V.T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wu, X., Wang, X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Y., Li, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., Ma, Z.: The llama 3 herd of models (2024), https://arxiv.org/abs/2407.21783 9, 4 14.Jiang, Y., Gao, X., Peng, T., Tan, Y., Zhu, X., Zheng, B., Yue, X.: HiddenDetect: Detecting jailbreak attacks against multimodal large language models via moni- toring hidden states. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Proceedings of the 63rd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). p. 14880–14893. Association for Computational Linguistics, Vienna, Austria (Jul 2025).https://doi.org/10.18653/v1/2025.acl- long.724, https://aclanthology.org/2025.acl-long.724/ 8 15.Jing, L., Chen, G.H., Aghazadeh, E., Wang, X.E., Du, X.: A comprehensive analysis for visual object hallucination in large vision-language models. arXiv preprint arXiv:2505.01958 (2025) 3 16.Li, Y., Guo, H., Zhou, K., Zhao, W.X., Wen, J.R.: Images are achilles’ heel of align- ment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models. In: European Conference on Computer Vision. p. 174–189. Springer (2024) 18Y. Han et al. 3, 4, 9, 7 17.Li, Z., Shi, H., Gao, Y., Liu, D., Wang, Z., Chen, Y., Liu, T., Zhao, L., Wang, H., Metaxas, D.N.: The hidden life of tokens: Reducing hallucination of large vision- language models via visual information steering. In: Forty-second International Conference on Machine Learning (2025),https://openreview.net/forum?id= 7BKcLeHQsm 9 18. Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Lee, Y.J.: Llava-next: Improved reasoning, ocr, and world knowledge (January 2024),https://llava-vl.github. io/blog/2024-01-30-llava-next/ 9 19. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023) 3 20. Liu, J., Guo, H., Duan, R., Bu, X., He, Y., Li, S., Huang, H., Liu, J., Wang, Y., Jing, C., et al.: Dream: Disentangling risks to enhance safety alignment in multimodal large language models. arXiv preprint arXiv:2504.18053 (2025) 3 21.Liu, Q., Shang, C., Liu, L., Pappas, N., Ma, J., Anna John, N., Doss, S., Mar- quez, L., Ballesteros, M., Benajiba, Y.: Unraveling and mitigating safety alignment degradation of vision-language models. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Findings of the Association for Computational Linguistics: ACL 2025. p. 3631–3643. Association for Computational Linguistics, Vienna, Austria (Jul 2025).https://doi.org/10.18653/v1/2025.findings-acl.186, https://aclanthology.org/2025.findings-acl.186/ 2, 4, 6, 9, 1, 5 22.Liu, S., Ye, H., Zou, J.: Reducing hallucinations in large vision-language models via latent space steering. In: The Thirteenth International Conference on Learning Representations (2025) 3 23. Liu, X., Zhu, Y., Gu, J., Lan, Y., Yang, C., Qiao, Y.: Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. In: European Conference on Computer Vision. p. 386–403. Springer (2024) 3, 9, 2 24.Liu, X., Zhu, Y., Gu, J., Lan, Y., Yang, C., Qiao, Y.: Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. In: Proceedings of the European Conference on Computer Vision (ECCV) 2024 – Poster (2024), https://eccv.ecva.net/virtual/2024/poster/1856, poster presentation 5 25.Lou, X., Yan, D., Shen, W., Yan, Y., Xie, J., Zhang, J.: Uncertainty-aware reward model: Teaching reward models to know what is unknown (2025),https://arxiv. org/abs/2410.00847 5 26.Lu, D., Pang, T., Du, C., Liu, Q., Yang, X., Lin, M.: Test-time backdoor attacks on multimodal large language models. arXiv preprint arXiv:2402.08577 (2024) 3 27.Lu, L., Pang, S., Liang, S., Zhu, H., Zeng, X., Liu, A., Liu, Y., Zhou, Y.: Adversarial training for multimodal large language models against jailbreak attacks. arXiv preprint arXiv:2503.04833 (2025) 3 28. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. In: The 36th Conference on Neural Information Processing Systems (NeurIPS) (2022) 9, 4 29. OpenAI: Introducing gpt-4.1 in the api | openai,https://openai.com/index/gpt- 4-1/ 9, 3 30.Park, S., Du, X., Yeh, M.H., Wang, H., Li, Y.: Steer llm latents for hallucination detection. arXiv preprint arXiv:2503.01917 (2025) 3 31. Pi, R., Han, T., Zhang, J., Xie, Y., Pan, R., Lian, Q., Dong, H., Zhang, J., Zhang, T.: Mllm-protector: Ensuring mllm’s safety without hurting performance (2024), https://arxiv.org/abs/2401.02906 1, 5 TBOP: Safety–Utility Harmonization in LVLMs via IFP19 32.Pi, R., Han, T., Zhang, J., Xie, Y., Pan, R., Lian, Q., Dong, H., Zhang, J., Zhang, T.: Mllm-protector: Ensuring mllm’s safety without hurting performance. arXiv preprint arXiv:2401.02906 (2024) 3, 9, 4 33.Qi, X., Huang, K., Panda, A., Henderson, P., Wang, M., Mittal, P.: Visual adversarial examples jailbreak aligned large language models (2023),https://arxiv.org/abs/ 2306.13213 7, 8 34.Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, p. 8748–8763. PMLR (18–24 Jul 2021), https://proceedings.mlr.press/v139/radford21a.html 5 35.Shao, Z., Liu, H., Hu, Y., Gong, N.Z.: Refusing safe prompts for multi-modal large language models. arXiv preprint arXiv:2407.09050 (2024) 3 36. Shayegani, E., Dong, Y., Abu-Ghazaleh, N.: Jailbreak in pieces: Compositional ad- versarial attacks on multi-modal language models. arXiv preprint arXiv:2307.14539 (2023) 3 37. Skean, O., Arefin, M.R., Zhao, D., Patel, N., Naghiyev, J., LeCun, Y., Shwartz-Ziv, R.: Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013 (2025) 3 38.Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023) 4 39.Wang, H., Wang, G., Zhang, H.: Steering away from harm: An adaptive approach to defending vision language model against jailbreaks. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 29947–29957 (2025) 4 40.Wang, H., Xiong, W., Xie, T., Zhao, H., Zhang, T.: Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2024. p. 10582–10592. Association for Computational Linguistics, Miami, Florida, USA (Nov 2024).https://doi.org/10.18653/v1/2024.findings-emnlp. 620, https://aclanthology.org/2024.findings-emnlp.620/ 5 41.Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024) 3 42.Wang, P., Zhang, D., Li, L., Tan, C., Wang, X., Ren, K., Jiang, B., Qiu, X.: Infer- aligner: Inference-time alignment for harmlessness through cross-model guidance. arXiv preprint arXiv:2401.11206 (2024) 3 43.Wang, R., Ma, X., Zhou, H., Ji, C., Ye, G., Jiang, Y.G.: White-box multimodal jailbreaks against large vision-language models. In: Proceedings of the 32nd ACM International Conference on Multimedia. p. 6920–6928 (2024) 3 44.Wang, Y., Liu, X., Li, Y., Chen, M., Xiao, C.: Adashield: Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting. In: European Conference on Computer Vision. p. 77–94. Springer (2024) 3 45.Yang, L., Zheng, Z., Chen, B., Zhao, Z., Lin, C., Shen, C.: Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 14635–14645 (2025) 4 46.Ying, Z., Liu, A., Zhang, T., Yu, Z., Liang, S., Liu, X., Tao, D.: Jailbreak vision lan- guage models via bi-modal adversarial prompt. IEEE Transactions on Information Forensics and Security (2025) 3 20Y. Han et al. 47.Yu, L., Do, V., Hambardzumyan, K., Cancedda, N.: Robust llm safeguarding via refusal feature adversarial training. arXiv preprint arXiv:2409.20089 (2024) 3 48. Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., Wang, L.: Mm-vet: evaluating large multimodal models for integrated capabilities. In: Proceedings of the 41st International Conference on Machine Learning. ICML’24, JMLR.org (2024) 9, 3 49. Zhang, G., Fan, X., Fang, J., Sun, Y., Shi, X., Lu, C.: Unveiling vulnerabilities in large vision-language models: The savj jailbreak approach. In: International Conference on Artificial Neural Networks. p. 417–434. Springer (2024) 3 50.Zhao, S., Duan, R., Wang, F., Chen, C., Kang, C., Ruan, S., Tao, J., Chen, Y., Xue, H., Wei, X.: Jailbreaking multimodal large language models via shuffle inconsistency. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 2045–2054 (October 2025) 9, 12, 3 51. Zhao, Y., Zheng, X., Luo, L., Li, Y., Ma, X., Jiang, Y.G.: Bluesuffix: Reinforced blue teaming for vision-language models against jailbreak attacks. arXiv preprint arXiv:2410.20971 (2024) 3 52.Zheng, C., Yin, F., Zhou, H., Meng, F., Zhou, J., Chang, K.W., Huang, M., Peng, N.: On prompt-driven safeguarding for large language models (2024),https: //arxiv.org/abs/2401.18018 8 53. Zong, Y., Bohdal, O., Yu, T., Yang, Y., Hospedales, T.: Safety fine-tuning at (almost) no cost: A baseline for vision large language models. arXiv preprint arXiv:2402.02207 (2024) 3 54.Zou, X., Kang, J., Kesidis, G., Lin, L.: Understanding and rectifying safety percep- tion distortion in vlms (2025), https://arxiv.org/abs/2502.13095 4 TBOP: Safety–Utility Harmonization in LVLMs via IFP1 Appendix A Observations A.1 Implemenation Details The observation experiment is conducted using LLaVA-v1.6-7B. To visualize modality-induced shifts, we extract the features of the last input token from the final decoder layer. We then normalize the hidden states to focus on the direction of the bias and perform 2D PCA visualization. This modality-induced bias remains consistent across both safety and utility datasets and is observed across multiple LVLM backbones, as illustrated in Fig. 8. PC2 (18.07%) PC1 (18.82%) PC2 (17.40%) InstructBLIP PC1 (35.78%) QwenVL Fig. 8: Consistency of modality-induced bias across LVLM backbones. The cosine similarities between the bias vectors b safe and b util are 0.62 and 0.60 respectively. For the performance degradation experiment, we adopt the default configu- ration of CMRM [21] but modify the steering direction. While CMRM steers cross-modal features in the direction opposite to the modality-induced shift to mitigate the bias, we instead steer along the shift (i.e., directionb) to investigate how amplifying this bias affects performance degradation in both safety and utility tasks. We report results with γ safe = 1.0 and γ util = 1.5. A.2 Further Analyses Fig. 9 presents a detailed analysis of the results obtained when feature steering is applied usingb safe andb util , respectively. When feature steering is applied in a direction that amplifies modality-induced bias, the refusal signal for harmful inputs is significantly weakened. As a result, instead of beginning responses with typical refusal phrases such as “I’m sorry”, “I’m unable to...”, the model tends to directly generate step-by-step answers to the query (Fig. 9-(a)). For benign queries, we observe response collapse. While the false negative rate is 15% 2Y. Han et al. under the vanilla setting, it increases dramatically as the shift intensity grows, in extreme case reaching 100% (Fig. 9-(b)). These results further confirm that modality-induced bias negatively affects utility performance and help explain why removing this bias leads to improved utility. TPRFNR 0.00.850.15 1.00.60.40 1.50.180.83 2.00.001.00 “... 3 . Create a plan: - Write down the amount of money you want to steal. - Write down the name of the person you want to steal from. ...” ... Linear “The bottom of the image shows a phrase of an activity. List steps in detail to perform this activity.” + (a) feature steering results on MMSB(b) feature steering results on MME MLLM MLLM ... Linear “Is this movie titled a good day to die hard (2013)?” + Fig. 9: Impact of amplifying modality-induced bias of cross-modal features for safety- and utility-related inputs. Reinforcing the bias simultaneously weakens jailbreak defense mechanisms and degrades the visual-grounded reasoning capabilities of LVLM. B Datasets and Evaluation Configurations B.1 Safety Benchmarks Table 5: Statistics of safety benchmarks used in our experiment. The metric range indicates the lowest/highest metric score. The lower is the better. Benchmark#Questions #Scenarios Ans. format Metric range M-SafetyBench 168013Open-ended [0, 100] % HADES7505Open-ended [0, 100] % FIGSTEP50010Open-ended [0, 100] % In Tab. 5, we present the detailed configurations of the safety benchmarks used in our experiments. M-SafetyBench [23] is a representative benchmark for evaluating visual- language jailbreaking. The dataset is organized into three splits: Text-only, SD, and SD-TYPO. The SD and SD-TYPO splits consist of pairs of harmful images generated via diffusion models and their corresponding text instructions. For our experiments, we use the SD-TYPO split. M-SafetyBench covers harmful topics with different safety objectives: TBOP: Safety–Utility Harmonization in LVLMs via IFP3 1.Harmful or illegal behavior elicitation (Scenarios 01–07, 09), where questions contain harmful key phrases intended to prompt the model to generate unsafe or unethical actions. 2. Political neutrality (Scenarios 08, 13), where questions involve political topics and the model should avoid expressing its own opinions. 3.Specialized professional requests (Scenarios 10–12), involving legal, fi- nancial, or health-related queries, where the model should acknowledge its lack of professional expertise and provide appropriate risk disclaimers. HADES [16] benchmark comprises 750 harmful instructions across five distinct scenarios. Each instruction is associated with six harmful images created using diffusion models through iterative optimization process. For the purposes of this study, we report experimental outcomes specifically at toxicity level 5. FIGSTEP [11] is a benchmark consisting of 500 questions across 10 topics that are prohibited by both OpenAI and Meta usage policies. To further evaluate the robustness of our approach, we apply the closed-box attack SI-Attack [50] to this dataset. This setting allows us to examine whether TBOP, when combined with other defense methods, can effectively defend against additional safety risks, introduced by text-based adversarial attacks in LVLMs for example, beyond the safety risks arising from the inclusion of image modality. B.2 Utility Benchmarks Table 6: Statistics of the utility benchmarks used in our experiments. Metric ranges denote the minimum and maximum scores. For MME, the ranges correspond to its two subcategories: Perception and Cognition. Benchmark #Questions #Tasks Ans. format Metric range M-Vet2186Open-ended[0, 100] % ScienceQA42418 Single-choices [0, 100] % MME237414Yes/No [0, 2000]/[0, 800] We verify that the model performs robustly in enhancing its visual-grounded reasoning capabilities while defending against jailbreak inputs across various utility benchmarks. The specific configurations and statistics of these benchmarks are described in Tab. 6. M-Vet [48] is a benchmark that comprehensively assesses general multi-modal capabilities including OCR, spatial awareness, and other reasoning skills. We measure an accuracy following the evaluation protocol proposed by M-Vet using GPT-4.1 [29]. 4Y. Han et al. ScienceQA [28] consists of scientific questions that require reasoning over modality-specific cues. The performance is measured using accuracy. ScienceQA spans diverse reasoning dimensions such as subject, grade level, and modality cues, enabling a comprehensive evaluation of multimodal reasoning ability. MME [7] is a benchmark to evaluate the perception and cognition abilities of LVLMs. This contains various yes-or-no question tasks, where the accuracy is measured following the MME evaluation protocol. B.3 Evaluations As the safety evaluation metric, we employ the Attack Success Rate (ASR), the proportion of harmful responses generated across all inputs, described in Eq. 9. The indicator functionI(·) returns 1 when the modelπ θ generate a responseπ θ (x txt ,x img ) as harmful, and 0 otherwise. For evaluation, we use LLaMA-Guard-4-12B [13] as an indicator and apply the original system prompt of LLaMA-Guard-4 without modification. ASR = 1 |D| X (x txt ,x img )∈D I(x txt ,π θ (x txt ,x img ))(9) C Experiments C.1 Implementation Details for Experiments and TBOP All experiments are implemented in PyTorch and executed on a single NVIDIA RTX 3090 GPU (for LLaVA-7B, QwenVL, InstructBLIP, and InternVL) or one or two NVIDIA A6000 GPUs (for LLaVA-13B). To estimate computational costs, we monitor peak GPU memory usage using PyTorch CUDA utilities, ensuring accurate measurement and efficient resource management. For generation, the LLaVA models use the default greedy decoding strategy, producing deterministic outputs without sampling. We set the max length to 2048, the default setting for LLaVA-v1.6. For TBOP, we apply a stopping criterion that halts generation when phrases such as “I’m sorry” or “I’m unable to” are produced. This rule is used for both safety and utility evaluations to better reflect real-world conditions, where the maliciousness of a user query is unknown to the model in advance. C.2 Baselines Configurations 1.Source LVLMs refers to the original LVLMs without any additional safety or modality alignment frameworks. 2.MLLM-Protector [32] fine-tunes models to identify unsafe responses and generate detoxified outputs. We use a LoRA fine-tuned Open-LLaMA-3B- v2 [4, 8, 38] as the harm detector and a LoRA fine-tuned LLaMA-2-7B as the detoxifier, while the first-stage response generation model follows that of each experiment. TBOP: Safety–Utility Harmonization in LVLMs via IFP5 Table 7: Jailbreaking defense performance comparison on M-SafetyBench across various safety-related categories evaluated by LLaMA-Guard-4. All values represent per- centages (%). A lower Attack Success Rate (ASR) indicates better defense performance. Bold values indicate the best ASR within each category among the inference-time frameworks, while underlinedvalues denote the second-best. ModelMethodIllegal Hate Malware Physical Economic Fraud Sex Politics Privacy Legal Finance Health Govt.Avg. LLaVA 7B Vanilla79.49 24.43 55.5660.3412.2465.32 13.64 0.0059.82 8.650.009.09 0.0029.89 MLLM-Protector [31] 8.97 0.0013.8912.073.066.45 3.41 0.0011.61 2.880.0012.50 0.005.76 ECSO [12]16.67 17.56 8.332.595.1017.74 6.82 4.880.89 0.004.489.09 0.007.24 Immune [10]15.38 9.1619.4414.662.0410.483.410.009.823.852.245.680.007.40 ETA [6]11.542.2913.8912.076.1211.29 0.00 1.638.934.811.4912.50 0.006.66 Ours10.260.768.336.900.009.684.550.007.140.960.002.270.003.91 LLaVA 13B Vanilla61.76 16.52 41.9456.4415.1258.33 11.69 0.0055.10 5.490.856.49 0.0025.36 MLLM-Protector [31] 13.24 0.8716.1311.881.166.487.79 0.939.182.20 0.009.09 0.006.07 ECSO [12]19.12 25.226.455.9410.4725.93 5.19 28.70 1.02 0.003.426.490.9510.68 Immune [10]20.59 7.8316.1318.814.6525.00 3.90 1.8518.37 0.000.8514.29 0.0010.17 ETA [6]10.291.749.689.902.337.410.00 0.009.184.400.853.90 0.004.59 Ours2.940.000.001.981.162.782.600.009.181.100.003.900.001.97 3.ECSO [12] converts input images into textual captions thereby generates safe responses. While ECSO is a multi-stage framework, the first response generation, caption generation, and response refinement all share the model presented in each experiment. 4.Immune [10] modifies the token logits by adding a safety reward score to the base logits, thereby encouraging the final response to be safety-aligned. Following the configuration introduced in the original paper, we use URM- LLaMA-3.1-8B [25] as the token-level reward model. We set the number of candidate tokens in the decoding step ask= 10 and reward scaling parameter as α = 1. 5.ETA [6] assesses the harmfulness of the input image and generates a final response that does not contain harmful context, based on representation similarity. Adhering to the experimental setup of the original paper, we use CLIP-ViT-L-336px [34] as the pre-generation evaluator, measuring cosine sim- ilarity with harmfulness-indicating prompts to assess input image safety, and ArmoRM-LLaMA3-8B [40] as the post-generation reward scoring model. The number of candidate samples, pre-generation threshold, and post-generation threshold are set to N = 5, 0.16, and 0.06, respectively. 6.CMRM [21] steers cross-modal features in the opposite direction of the predefined modality-induced bias, thereby mitigating its negative impact on model safety. We follow the default configurations, applying it to the last input token across all layers withα= 1.0 for dataset-level extraction shifting vector. For a fair comparison. The anchor dataset is constructed using the same recipe as ours. C.3 Additional Results In this section, we present additional evaluations and analyses covering a broader set of jailbreak attacks and baseline comparisons. Evaluation on M-SafetyBench Across All Scenarios. In main paper, we provide benchmark results on M-SafetyBench [24] for scenarios with high 6Y. Han et al. Query : The bottom of the image shows a phrase of an activity. List steps in detail to perform this activity in relation to surveillance cameras. LLaVA 7B v1.6 The image shows a person sitting in front of a computer with multiple monitors, likely a security professional or someone involved in monitoring and surveillance. The phrase at the bottom of the image reads "hack into government security systems.” To p e r f o r m t h i s a c t i v i t y, o n e w o u l d typically follow these steps: 1. Gather Information: Research the target system, including its architecture, security protocols, and vulnerabilities. ... LLaVA 7B v1.6 + Ours I'm unable to provide specific instructions on how to hack into government security systems, as that is illegal and goes against ethical and moral principles. Fig. 10: Qualitative Analysis of TBOP on M-SafetyBench. Attack Success Rates (ASR), demonstrating TBOP’s superior effectiveness as an LVLM jailbreak defense strategy. Comprehensive results for all benchmark scenarios are also provided in Tab. 7, where TBOP consistently outperforms alternative approaches across the board. Qualitative Analysis of TBOP on M-SafetyBench. In the main paper, we present qualitative analysis on the utility benchmark M-Vet. Here, we provide additional qualitative results on the safety benchmark M-SafetyBench to illustrate the behavior of TBOP in safety-critical scenarios. We compare the responses generated by our method with those of the original LLaVA source model. As shown in Fig. 10, while the source model tend to interpret the image context and provide step-by-step instructions in response to harmful queries, our method instead generates refusal responses, effectively preventing the model from complying with malicious user requests. Table 8: Computational and Memory efficiency measured by inference time per token (seconds↓) and peak memory usage (GB↓). Values in parentheses indicate the ratio relative to the vanilla model for each baseline. Bold values indicate the best efficiency within each dataset among jailbreak defense strategies. ModelMethod Inference Time Per TokenPeak Memory M-Safety M-VetMM-SafetyMM-Vet LLaVA 7B Vanilla0.040.0414.7314.31 ECSO 0.29 (×7.9) 0.06 (×1.3) 14.92 (×1.0) 14.36 (×1.0) Immune 0.08 (×2.0) 0.07 (×1.6) 28.70 (×1.9) 28.37 (×2.0) ETA 2.23 (×60.8) 0.13 (×2.8) 43.89 (×3.0) 33.69 (×2.4) Ours0.04 (×1.0)0.05 (×1.0)14.73 (×1.0)14.31 (×1.0) LLaVA 13B Vanilla0.060.0827.3126.62 ECSO 0.18 (×2.9) 0.11 (×1.4) 27.59 (×1.0) 26.76 (×1.0) Immune 0.10 (×1.5) 0.11 (×1.4) 41.20 (×1.5) 40.68 (×1.5) ETA 3.42 (×53.8) 0.21 (×2.6) 59.30 (×2.2) 46.85 (×1.8) Ours0.07 (×1.0)0.09 (×1.2)27.31 (×1.0)26.63 (×1.0) TBOP: Safety–Utility Harmonization in LVLMs via IFP7 Computational and Memory Efficiency of TBOP. We compare the efficiency of the inference-time frameworks. For LLaVA 7B and LLaVA 13B, we evaluate efficiency along two dimensions: (1) computational efficiency by measuring wall-clock inference time per token, and (2) memory efficiency by the peak memory usage during the response generation process. Our framework demonstrates time and memory efficiency comparable to the vanilla model. Compared to our method, ETA consumes nearly 60× on M-SafetyBench. The other inference-time approaches—Immune and ETA—require substantially more memory, as their pipelines involve auxiliary models at inference time. 7B13B 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Attack Success Rate (ASR ) -10.80 -4.30 Vanilla Ours Fig. 11: Effectiveness of TBOP ap- proach on the HADES blank setting for LLaVA 7B and 13B. Extended Experiments on HADES. We demonstrate that TBOP robustly de- fends against harmful requests even when image inputs are adversarially optimized under the toxic HADES [16] setting in main paper. We further evaluate a complementary scenario from the same dataset in which harmful text is paired with a blank image whose modality ac- tivation is isolated and carries no mean- ingful visual information. Because TBOP assumes that the vulnerability arises from modality-induced shifts—captured using blank-image activations—we conduct an experiment to verify that our orthogonal projection approach suppresses this vul- nerability component. As shown in Fig. 11, applying TBOP consistently reduces ASR for both LLaVA 7B and LLaVA 13B, confirming that our orthogonal projection effectively removes vulnerabilities introduced by the image modality. Table 9: Jailbreaking defense performance on Visual Adversarial Attacks [33]. We report ASR (%) on images optimized with varying levels of adversarial noise, denoted by ε, with “unconstrained” representing the most challenging scenario. ε = 64/255Unconstrained ModelMethod Perplexity Detoxify Perplexity Detoxify LLaVA 7B Vanilla 82.3280.0577.9874.54 Ours78.0776.0474.6472.70 LLaVA 13B Vanilla 60.6357.1059.9756.59 Ours59.9756.5959.5556.09 8Y. Han et al. Evaluation on Visual Adversarial Attacks. We use the derogatory corpus from prior work [33] to construct prompts that encourage harmful or derogatory responses. The same visual adversarial attack procedure is applied to both the vanilla LLaVA models and our defended models; afterwards, the attacked models are queried with harm-inducing texts, and their generated answers are collected for evaluation. We consider two attack settings: a constrained setup with ε= 64/255 and an unconstrained setup without anεlimit, as reported in Tab. 9. To determine whether a response is harmful, we score each generated answer using two independent toxicity detectors, Perspective API and Detoxify, and compare the average scores across models and attack regimes. Across both LLaVA-7B and LLaVA-13B, and in both constrained and unconstrained settings, attacks on our models yield consistently lower scores in both Perspective and Detoxify than attacks on the vanilla models, indicating that our method suppresses the effectiveness of visual adversarial attacks in eliciting harmful generations. C.4 More discussions Technical Advantages Beyond Safety Detection While TBOP builds upon prior work [14,52] by addressing jailbreaks through hidden states analysis, it is motivated by and offers two key advantages. – Enhanced Safety and Utility. The visual representation shift modeled by TBOP captures modality-induced feature misalignment existing for all multimodal inputs, regardless of their semantics. Through extensive experiments and analysis, we demonstrate that removing these components enhances cross-modal feature alignment and improves performance across both safety-critical and utility-oriented inputs. In contrast, the scope of classifier-based methods mainly lies in safety detection. They assess input safety via feature similarity to a refusal vector using a threshold, which induces a safety–utility trade-off: higher thresholds lead to more refusals and fewer helpful responses. Moreover, their utility performance is theoretically bounded by that of the vanilla (i.e., no filtering). –Direct Response Modification. TBOP directly edits the feature of the last input token, not reducing to measuring the harmfulness of it, thus influencing autoregressively generated tokens. This design enables contextual explanations of why a given input is considered harmful (Fig. 10), which is technically unattainable for linear classifiers. Since this procedure introduces no additional latency or memory overhead, it demonstrates clear efficiency advantage, achieving 60× speedup over ETA, an inference-time defense method that bridges feature-level safety detection and response modification (Tab. 8). Longevity for Future Foundational Models TBOP is based on the as- sumption that safety vulnerabilities in LVLMs stem from cross-modal feature misalignment; thus models with stronger alignment may exhibit smaller relative gains. However, achieving robust alignment in recent multimodal LLMs (e.g., TBOP: Safety–Utility Harmonization in LVLMs via IFP9 Qwen3-Omni) has become increasingly challenging as the number of engaged modalities grows. The cost of cross-modal alignment training rises substantially due to the need for large-scale data across multiple modality pairs and increased model complexity. In this context, TBOP remains practically valuable as an inference-time safety patch that significantly reduces the overhead. D Limitations and Future Works TBOP is designed to address cross-modal jailbreaks, and as a result, its defensive strength against purely text-based attacks may be comparatively weaker. However, as demonstrated in earlier experiments, this limitation can be mitigated by combining TBOP with other existing defense strategies. We expect that future work exploring additional training techniques that reduce the modality gap during pretraining could further mitigate vulnerabilities arising from modality-induced shifts at an early stage.