Paper deep dive
Towards Mechanistic Defenses Against Typographic Attacks in CLIP
Lorenz Hufe, Constantin Venhoff, Maximilian Dreyer, Sebastian Lapuschkin, Wojciech Samek
Models: CLIP ViT-B, CLIP ViT-BigG, CLIP ViT-L
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 5:34:04 PM
Summary
The paper introduces 'Dyslexify', a gradient-free, mechanistic defense against typographic attacks in CLIP models. By identifying and ablating specific 'typographic circuits'—attention heads that causally transmit typographic information to the cls token—the method improves robustness against adversarial text injections by up to 22.06% on ImageNet-100-typo while maintaining standard accuracy within 1%. The approach is validated across various model scales and in safety-critical medical applications like skin lesion diagnosis.
Entities (5)
Relation Signals (3)
Dyslexify → defendsagainst → Typographic attacks
confidence 100% · we introduce Dyslexify - a method to defend CLIP models against typographic attacks
Typographic Attention Score → guides → Circuit construction
confidence 95% · we introduce the Typographic Attention Score T i,ℓ to guide the circuit construction
Dyslexify → appliedto → WhyLesionCLIP
confidence 90% · we analyze WhyLesionCLIP... and demonstrate that Dyslexify can substantially mitigate this vulnerability
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Typographic attacks exploit multi-modal systems by injecting text into images, leading to targeted misclassifications, malicious content generation and even Vision-Language Model jailbreaks. In this work, we analyze how CLIP vision encoders behave under typographic attacks, locating specialized attention heads in the latter half of the model's layers that causally extract and transmit typographic information to the cls token. Building on these insights, we introduce Dyslexify - a method to defend CLIP models against typographic attacks by selectively ablating a typographic circuit, consisting of attention heads. Without requiring finetuning, dyslexify improves performance by up to 22.06% on a typographic variant of ImageNet-100, while reducing standard ImageNet-100 accuracy by less than 1%, and demonstrate its utility in a medical foundation model for skin lesion diagnosis. Notably, our training-free approach remains competitive with current state-of-the-art typographic defenses that rely on finetuning. To this end, we release a family of dyslexic CLIP models which are significantly more robust against typographic attacks. These models serve as suitable drop-in replacements for a broad range of safety-critical applications, where the risks of text-based manipulation outweigh the utility of text recognition.
Tags
Links
- Source: https://arxiv.org/abs/2508.20570
- Canonical: https://arxiv.org/abs/2508.20570
Trouble viewing inline? Open PDF directly →
Full Text
65,908 characters extracted from source content.
Expand or collapse full text
DYSLEXIFY: A MECHANISTIC DEFENSE AGAINST TYPOGRAPHIC ATTACKS IN CLIP Lorenz Hufe 1,2 , Constantin Venhoff 2 , Erblina Purelku 1 , Maximilian Dreyer 1 , Sebastian Lapuschkin 1,3 , Wojciech Samek 1,4 1 Fraunhofer Heinrich Hertz Institute, 2 University of Oxford 3 Technological University Dublin, 4 Technische Universität Berlin lorenz.hufe@hhi.fraunhofer.de ABSTRACT Typographic attacks exploit multi-modal systems by injecting text into images, leading to targeted misclassifications, malicious content generation and even Vision- Language Model jailbreaks. In this work, we analyze how CLIP vision encoders behave under typographic attacks, locating specialized attention heads in the latter half of the model’s layers that causally extract and transmit typographic informa- tion to theclstoken. Building on these insights, we introduce Dyslexify – a method to defend CLIP models against typographic attacks by selectively ablating a typographic circuit, consisting of attention heads. Without requiring finetuning, Dyslexify improves performance by up to 22.06% on a typographic variant of ImageNet-100, while reducing standard ImageNet-100 accuracy by less than 1%, and demonstrate its utility in a medical foundation model for skin lesion diagnosis. Notably, our gradient-free approach remains competitive with current state-of-the- art typographic defenses that rely on finetuning. To this end, we release a family of dyslexic CLIP models which are significantly more robust against typographic attacks. These models serve as suitable drop-in replacements for a broad range of safety-critical applications, where the risks of text-based manipulation outweigh the utility of text recognition. Poject page: https://hufe.info/dyslexify/ Code: https://github.com/lowlorenz/dyslexify 1INTRODUCTION CLIP models are increasingly adopted as general-purpose vision–language representations, enabling applications in zero-shot classification, retrieval, diffusion-based generative models, and large-scale vision–language models (VLMs). Their versatility has further driven adoption in safety-relevant domains such as healthcare (Yang et al., 2024; Wang et al., 2022; Eslami et al., 2023), remote sensing (Liu et al., 2024a; Vivanco et al., 2023; Li et al., 2023), and content moderation (Schuhmann et al., 2022; Liu et al., 2025; Reyes et al., 2025). However, despite their widespread use, CLIP models remain vulnerable to typographic attacks: inserting text into an image can mislead classification, trigger malicious generations, or even jailbreak multi-modal systems (see Fig. 1). Existing defenses against typographic attacks require gradient-based optimization. While effective to some extent, these methods require substantial computational resources and lack interpretability into the mechanisms underlying CLIP’s behavior. In this work, we introduce Dyslexify a gradient-free defense that directly targets model circuits responsible for the vulnerability to typographic attacks. By identifying and ablating a set of attention heads with demonstrable causal effects, we construct dyslexic CLIP models that are substantially more robust to typographic attacks. Our method scales seamlessly to billion-parameter models, making it applicable to state-of-the-art multi-modal systems. Beyond improving robustness, our approach also enhances interpretability of CLIP models, enabling targeted intervention that are computationally efficient and easily integrated into existing pipelines without additional overhead. The contributions of this work are: 1 arXiv:2508.20570v2 [cs.CV] 26 Feb 2026 Figure 1: Defending CLIP against typographic attacks with Dyslexify a) Adversarial text in images can dominate CLIP’s representation and lead to misclassification. b) We construct a circuit of attention heads responsible for transmitting typographic information. c) By suppresses the typographic circuit, we defend against typographic attacks without a single gradient step. • Mechanistic Understanding: We present the Typographic Attention Score to locate spe- cialized typographic attention heads and demonstrate their causal role in typographic attacks within CLIP models through controlled interventions. • Gradient-Free Defense: We introduce a method that utilizes circuit ablation to effectively defend against typographic attacks, while maintaining general visual capabilities. Due to its gradient-free nature, Dyslexify seamlessly scales to billion-parameter models on consumer grade hardware. • Empirical Validation: We validate Dyslexify across a diverse set of zero-shot classification tasks, demonstrating that our approach improves robustness to typographic attacks by up to 22.06% on a typographic version of Imagenet-100 while maintaining high accuracy on non-typographic benchmarks. •Medical Use Case: We show that typographic attacks pose a tangible risk to safety-critical medical foundation models, and demonstrate that Dyslexify can substantially mitigate this vulnerability. •Model Release: To facilitate safer deployment, we release a family of dyslexic CLIP models with reduced typographic sensitivity, suitable for use in safety-critical applications. Our approach provides a practical, interpretable, and computationally efficient typographic defense, paving the way for safer multimodal systems without the need for fine-tuning. 2RELATED WORK CLIP models (Radford et al., 2021) are pretrained on large-scale image–text datasets such as Laion-5b (Schuhmann et al., 2022), aligning global image features with textual descriptions for strong zero-shot transfer. This reliance on textual supervision also makes them vulnerable to typographic attacks. Typographic attacks (Goh et al., 2021) insert written text into an image to maliciously alter a model’s behavior. Recent work demonstrates that typographic attacks can degrade model performance, bypass safety filters (jailbreaking) and hijack goals in Vision Language Models (VLMs) (Qraitem et al., 2024; Kimura et al., 2024; Gong et al., 2025; Cao et al., 2024; Westerhoff et al., 2025), trigger harmful content generation in image-to-image pipelines (Cheng et al., 2024), and cause targeted misclassification in object detection and zero-shot classification settings (Materzy ́ nska et al., 2022; Ilharco et al., 2022; Azuma & Matsui, 2023; Westerhoff et al., 2025; Dreyer et al., 2025). Several defenses have been proposed, they rely either on fine-tuning the model (Ilharco et al., 2022), learning a projection matrix (Materzy ́ nska et al., 2022), incorporating a learnable text-token called Defense-Prefix (Azuma & Matsui, 2023), or employing Sparse Autoencoders (Joseph et al., 2025). Crucially, none of these approaches offer a gadient-free method for mitigating typographic attacks. 2 In contrast, our work introduces a controllable intervention at inference time by directly locating and suppressing the components responsible for typographic sensitivity, without requiring gradients or fine-tuning. 3MOTIVATION: LOCATING LAYERS OF TYPOGRAPHIC UNDERSTANDING To better understand CLIP’s vulnerability to typographic attacks and to motivate our method, we begin by investigating which layers and components are responsible for typographic understanding using linear probes. Typographic datasets: We further construct typographic attack datasets from standard image classi- fication datasetsD =(x i ,y i ) n i=1 by assigning each input examplex i an additional typographic labelz i ̸= y i different from the original labely i , and overlaying a corresponding textual description ofz i onto the original imagex i at a random location as shown in Fig. 2a. We denote these modified datasets with the suffix “-typo”. More information on the datasets is provided Section F. Extending the ImageNet-100 dataset into the ImageNet-100-typo dataset, we can evaluate typographic understanding by training linear probes on theclstoken embedding at each layer of OpenCLIP models, ranging in scale from ViT-B to ViT-bigG (Ilharco et al., 2021). Formally, for a layer ℓ we define a linear probe P ℓ ˆy ℓ (x) = w ⊤ h ℓ cls + b ,(1) whereh ℓ cls ∈R d denotes the activation of the model at layerℓon theclstoken for an input samplex, andwandbare the probe’s weight vector and bias term, respectively. We train two types of probes: P img,ℓ , which predicts the object labelyandP typo,ℓ , which predicts the typographic labelz. The accuracy of a probe P is denoted by Acc(P ). Fig. 2b shows thatAcc(P typo,ℓ ) > 0.99at the final layer for all tested models, indicating that CLIP models can distinguish the typographic classes. Furthermore it shows, thatAcc(P typo,ℓ )is low in early layers, but exhibit a sharp increase in the latter half of the model. Fig. 2c shows thatAcc(P img,ℓ )improves gradually over the layers, whileAcc(P typo,ℓ )exhibits a sharp performance rise around second half of the models layers. Fig. 2d highlights the effect of the attention and the MLP blocks ontoAcc(P img,ℓ )andAcc(P typo,ℓ ). Attention layers consistently improve accuracy indicating that they add linearly decodable information to theclstoken. In contrast, the MLP layers tend to reduce accuracy. We show in Section B, that the Intrinsic Dimensionality (ID) of the signal decreases after the MLP blocks, suggesting that the MLP compresses or discards information. 4DYSLEXIFY: A DEFENSE AGAINST TYPOGRAPHIC ATTACKS To defend CLIP against typographic attacks we present Dyslexify: a framework for detecting and suppressing typographic circuits. Based on the finding in Section 3 that attention heads are responsible for adding typographic information, we only consider attention heads for the circuit construction. We define a circuit as a subsetC ⊆ Ψ, where Ψ denotes the set of attention heads in CLIP Ψ =H i,ℓ | i∈0,...,I, ℓ∈0,...,L(2) with I heads per layer and L layers in total, whereH i,ℓ denote the ith attention head in layer ℓ. To robustify a modelMagainst typographic attacks, we conduct circuit-ablation of a typographic circuitC. Circuit-ablation modifies here only the residual stream of theclstoken, leaving all other computations intact. Specifically, the residual update of the cls token is given by z ℓ cls = h ℓ cls + MLP(h ℓ cls )(3) h ℓ+1 cls = z ℓ cls + H X i=1 H i,ℓ,cls (z ℓ cls )(4) wherez ℓ cls is theclsactivation after the layer-ℓMLP block is applied toh ℓ cls , andH i,ℓ,cls (z ℓ cls )is the contribution of headH i,ℓ to the cls token. 3 Figure 2: Investigating where typographic understanding emerges in CLIP. a) We train two linear probes on all layers of CLIP models. ProbeP img,ℓ is used to predict the text label of each sample whileP typo,ℓ is trained to predict the typographic class. b)P typo,ℓ shows a consistent pattern across all model sizes: typographic information emerges abruptly in the second half of the models layers. c) This trend is not true for the object probesP img,ℓ . Object specific information builds gradually over the layers. Each line in the shaded area represents one CLIP model. d) While attention layers seem to add linearly decodable information to theclstoken, MLP layers remove or obscure information. We define ablation of a typographic circuitC as H i,ℓ,cls (z ℓ cls )← 0 for allH i,ℓ ∈C,(5) while leaving all spatial contributions unchanged. We useM C to denote a modelMin which circuit C is ablated. 4.1TYPOGRAPHIC ATTENTION SCORE Building on Hung et al. (2024) we introduce the Typographic Attention ScoreT i,ℓ to guide the circuit construction. Intuitively the score measures the amount of spatial attention a headH i,ℓ dedicates to typographic content. Given a headH i,ℓ and an inputx, we writeA i,ℓ (x)∈ [0, 1] T+1 to denote theclstokens attention pattern, whereTis the number of spatial tokens and the additional entry corresponds to thecls token. We further define the spatialcls-attention patternA ∗ i,ℓ (x) ∈ [0, 1] T which excludes the cls-to-cls entry A i,ℓ,cls (x), such that A i,ℓ (x) = A i,ℓ,cls (x), A ∗ i,ℓ (x) .(6) We write A ∗ i,ℓ,t (x) to index the tth element of the pattern. Formally the score is given by: T i,ℓ = X x∈D P T t=1 1(t)A ∗ i,ℓ,t (x) P T t=1 A ∗ i,ℓ,t (x) .(7) 4 Wherexis a data point,Dis the dataset, and1is an indicator function so that1(t) = 1if the input patch associated with the token at index t corresponds to typographic content and is zero otherwise. 4.2TYPOGRAPHIC CIRCUIT CONSTRUCTION To defend against typographic attacks without degrading zero-shot classification, Dyslexify constructs a typographic circuitC. The circuit is built iteratively while monitoring the accuracy of the circuit- ablated modelM C on a non-typographic benchmarkD img and a typographic benchmarkD typo , ensuring that the accuracy on D img never decreases by more than a threshold ε∈R. Our procedure consists of two steps: (i) rank all attention headsH i,ℓ by their typographic scoreT i,ℓ ; (i) add heads toC in descending order of T i,ℓ , evaluating accuracy after each addition. LetAcc(M,D)denote the accuracy of modelMon datasetD. For each candidate headH, we compute ∆Acc img = Acc(M,D img )− Acc(M C∪H ,D img ),(8) ∆Acc typo = Acc(M C∪H ,D typo )− Acc(M C ,D typo ),(9) where∆Acc img measures the accuracy drop onD img relative to the base model, and∆Acc typo measures the incremental gain on D typo from adding headH to the current circuitC. If∆Acc typo ≤ 0, the headHis skipped, as it does not improve robustness to typographic attacks. If the head is not skipped and∆Acc img < ε, the head is added toC; otherwise, the algorithm terminates. In addition, if more than k ∈N heads are skipped consecutively, the algorithm also terminates. We refer to the final modelM C , equipped with the constructed circuitC, as the dyslexic model. Algorithm 1 Dyslexify 1: Initialize circuitC ←∅, skip counter s← 0. 2: Set hyperparameters: tolerance ε, max skips k. 3: Rank headsH i,ℓ by score T i,ℓ . 4: for headH i,ℓ in descending order of T i,ℓ do 5:Compute ∆Acc img , ∆Acc typo . 6: if ∆Acc typo ≤ 0 then 7: s← s + 1; 8:if s≥ k then break; end if 9:continue; 10: end if 11: if ∆Acc img ≥ ε then break; end if 12:AddH i,ℓ toC; s← 0. 13: end for 14: Return final circuitC. 5EXPERIMENTS 5.1EVALUATING THE TYPOGRAPHIC ATTENTION SCORE We construct localized typographic dataset consisting of 10,000 natural images from Unsplash 1 , originally containing minimal typographic content. To efficiently analyze the attention patterns, we synthetically introduce typographic content at the bottom center of the image, responding to the lowest two token rows in the spatial grid. For each attention headH i,ℓ we extract the attention patternA ∗ i,ℓ (x)over this localized dataset. We use the known spatial bias to define the indicator mask1(t) ∈ 0, 1, where1(t) = 1at the the typographic region and 1(t) = 0 elsewhere. Fig. 3 shows the resulting Typographic Attention ScoresT i,ℓ for ViT-B. A small subset of heads shows high scores of up toT i,ℓ ≥ μ(T ) + 2σ(T ), revealing a strong spatial bias towards typography. 1 https://huggingface.co/datasets/wtcherr/unsplash 5 Figure 3: Analysis of the Typographic Attention Score. a) For each head in the model we calculate the Typographic Attention ScoreT i,ℓ , utilizing the spatial bias in the Unsplash-typo dataset. b) Depiction of ViT-B’sT i,ℓ scores. While most attention heads do not show any spatial bias in their attention patterns, a few attention heads indicate significantly elevated scores, exceedingT i,ℓ ≥ μ(T ) + 2σ(T ). Those heads only occur in the second half of the models layers c) Overlaying the linear probes with significantly elevatedT i.ℓ scores , highlights an interesting correlation. Only after the attention heads with exceptionally highT i,ℓ scores are passed the model the accuracy ofP typo,ℓ begins to increase rapidly. Furthermore we observe, that the spike in Acc(P typo,ℓ ) only occurs after layers with high T i,ℓ heads. More results can be found in Section K. 5.2EVALUATING THE TYPOGRAPHIC CIRCUITS To construct the circuitsC, we set the tolerance toε = 0.01, set the maximum number of consecutive skips tok = 10, use the ImageNet-100 training split asD img , and ImageNet-100-typo asD typo . Table 5 shows that the resulting circuits are sparse, covering at most 10.1% of Ψ. Fig. 4 plotsAcc(M C ,D img )andAcc(M C ,D typo )as heads are added. Dyslexify, improves accuracy on ImageNet-100-typo’s train set by more than 20% across all evaluated models, while limiting the drop in ImageNet-100 accuracy to below 1%. More results can be found in Fig. 9. 012345678 Number of Ablated Heads 45 50 55 60 65 70 75 80 85 90 Zeroshot Accuracy (%) ViT-B ImageNet-100 ImageNet-100-Typo 0510152025 Number of Ablated Heads 45 50 55 60 65 70 75 80 85 90 Zeroshot Accuracy (%) ViT-L ImageNet-100 ImageNet-100-Typo 05101520 Number of Ablated Heads 45 50 55 60 65 70 75 80 85 90 Zeroshot Accuracy (%) ViT-H ImageNet-100 ImageNet-100-Typo Figure 4: Tradeoff between general accuracy and typographic robustness as a function of the number of ablated heads. Ablations are applied in decreasing order of T i,ℓ . 5.2.1DEMONSTRATING CAUSALITY OF TYPOGRAPHIC CIRCUITS We observe that attention heads inCutilize theirclsself-attention as attention sinks (Xiao et al., 2023) depending on the presence of typography in the image. Further details are provided in Section C. Building on those findings we showcase the casual nature of these attention heads in this section. 6 Figure 5: Controlling typographic vulnerability by manipulating attention sinks in circuit heads. a) We set theclstoken attention toαand rescale the spatial token attentions to sum to1− α. b) Increasingαraises attention to spatial tokens, amplifying typographic understanding. c) Decreasing α increases typographic robustness, increasing the probability of predicting the true object class. To demonstrate the causal role of these heads in typographic vulnerability, we manipulate their attention patterns. Specifically, we construct A α i,ℓ = α, A ∗ i,ℓ · (1− α)/∥A ∗ i,ℓ ∥ ,(10) whereα ∈ 0.0, 0.1,..., 0.9, 1.0. The scaling factor(1− α)/∥A ∗ i,ℓ ∥ensures that the attention distribution remains normalized. We then evaluate the effectiveness of typographic attacks under these manipulations by tracing the predicted label probabilitiesp(y text )andp(y typo )as a function ofαon the ImageNet-100-typo dataset. Results: Fig. 5 shows that increasingαcausally reduces the effectiveness of typographic attacks, while decreasingαamplifies it. This provides direct causal evidence: as circuit heads allocate more weight to theclssink, the attack signal is suppressed; conversely, whenαis low and spatial attention dominates,p(y typo )increases, indicating that typographic information is transferred from spatial tokens into the cls representation. 5.3EVALUATING THE DYSLEXIC MODELS To evaluate Dyslexify we construct dyslexic OpenClip model variants: ViT-B, L, H, G, and BigG and conduct zero-shot classification experiments. Concretely we first evaluate the effectiveness of our defense against typographic attacks and secondly measure the zero-shot object classification capabilities on non-typographic datasets. For each model, we record the accuracy difference between the original model and dyslexic model. A detailed description of the datasets is given in Section F. Results: Table 1 shows that Dyslexify yields consistent robustness improvements across both real- world typographic attack datasets and synthetic benchmarks. The observed gains are substantial – up to +31% accuracy – and occur across all evaluated datasets, suggesting that the identified typographic circuits capture generalizable failure modes rather than dataset-specific artifacts. Table 2 further demonstrates that Dyslexify preserves performance on standard vision datasets. In nearly all cases, deviations remain within±1% of the base model, the only exception being ViT-L showing the largest decline (−1.74%) on Aircraft and (−1.17%) on Food-101, which is close to the tolerance boundε = 1%. This indicates that Dyslexify achieves a favorable robustness–accuracy trade-off: substantial robustness gains are obtained while standard zero-shot performance is essentially maintained. 7 Table 1: Comparison of dyslexic model performance on datasets of typographic attacks across model sizes, showing accuracy changes relative to the base model, with improvements (↑) or declines (↓). IN denotes ImageNet, and the suffix -T indicates the corresponding typographic version of the dataset. Real TypographicSynthetic Typographic ModelRTA-100DisentanglingPaintIN-100-TFood-101-TAircraft-T B68.30↑12.0085.00↑31.1172.73↑14.55 66.84↑19.9078.27↑22.6416.23↑5.91 L71.00↑16.6060.56↑10.0076.36↑14.55 72.22↑20.3282.15↑26.5523.34↑9.51 H68.30↑15.2072.22↑26.6770.91↑21.82 75.34↑21.2683.01↑28.6829.40↑8.07 G62.00↑12.0067.22↑9.4471.82↑16.36 68.76↑22.0673.05↑20.2127.69↑3.45 Big-G72.90↑11.9068.33↑20.0069.09↑21.82 78.64↑16.7484.69↑25.9841.61↑16.29 Table 2: Comparison of dyslexic model performance on non-typographic datasets across model sizes, showing accuracy changes relative to the base model, with improvements (↑) or declines (↓). Not Typographic ModelAircraftFood-101 ImageNet-100 B27.72↓0.12 84.97↓0.9975.00↑0.64 L34.62↓1.74 89.31↓1.1779.52↓0.24 H43.98↑0.12 92.29↓0.2483.40↓0.34 G44.07↓0.30 91.47↓0.7082.58↓0.66 Big-G50.47↓0.39 92.55↓0.4284.72↓0.34 5.4COMPARING TO BASELINES We compare Dyslexify to Defense-Prefix (DP) (Azuma & Matsui, 2023), which introduces a learnable prefix token for CLIP’s language transformer on OpenCLIP ViT-L without fine-tuning the full ViT. Following their setup, we train the DP on the ImageNet-100-typo training split with a learning rate of 0.002, batch size of 64, and hyperparameters γ = 3.0 and η = 1.0 for 6 epochs. Table 3 shows that Dyslexify outperforms DP on two out of three typographic benchmarks, while DP retains slightly higher performance on two out of three non-typographic benchmarks. Notably, DP yields a modest accuracy improvement on the corresponding non-typographic ImageNet-100, likely caused by the choice of ImageNet-100-typo as the training set for DP. We hypothesize that the black-box optimization in DP captures features relevant not only to typographic defense but also to ImageNet-100 classification, thereby limiting its generalization. In contrast, our method focuses on key mechanisms relevant to typographic attacks, leading to more robust transfer across non-typographic benchmarks. Table 3: Performance comparison of Dyslexify and Defense-Prefix (DP) on ViT-L across typographic and non-typographic datasets. For each method we show the accuracy followed by the deviation from the baseline, (↑) for improvement, (↓) for decline. Real TypographicTrainingNon-Typographic MethodRTA-100DisentanglingPAINTImageNet-100 Food-101Aircraft Baseline54.4050.5661.8179.7690.4836.36 DP62.20↑ 7.8082.78↑32.2071.82↑10.0181.70↑1.9489.83↓0.65 32.94↓3.42 Dyslexify 71.00↑16.6060.56↑10.0076.36↑14.5579.52↓0.2489.31↓1.17 34.62↓1.74 5.5DEFENDING AGAINST TYPOGRAPHIC ATTACKS IN MELANOMA DETECTION Safety-critical domains such as medicine are particularly vulnerable, as AI decisions may impact human lives. Therefore, we investigate whether typographic attacks transfer to this setting and 8 Figure 6: Typographic attacks in melanoma detection. (a) Adding adversarial text may cause CLIP to misdiagnose a malignant lesion as benign. (b) Applying Dyslexify mitigates these failures, increasing robustness to typographic attacks and even improving accuracy in several non-attacked cases. whether our defense remains effective. Specifically, we analyze WhyLesionCLIP, a foundation model for skin lesion classification (Yang et al., 2024), i.e., melanoma detection, based on OpenClip-ViT-L. We utilize the same setting as in Section 5.1 retrieving the typographic attention scores, but deviate from Section 5.2 in that we constructC by setting ISIC2019 as D img and ISIC2019-Typo as D typo . The results in Fig. 6 and in Table 4 reveal two key insights: (i) Typographic attacks reduce the zero- shot accuracy of melanoma detection by up to 22% and (i) Dyslexify proves effective in defending a medical foundation model from a relevant attack vector. Not only does Dyslexify increase the accuracy under typographic attack by up to 19.3%, but it also increases the base models accuracy in three out of the four datasets. An additional medical use-case is analyzed in Section E. 6CONCLUSION We present a mechanistic defense against typographic attack in CLIP using an interpretability-first approach. We reveal that a small number of attention heads located in the later layers of the vision encoder are responsible for the effectiveness of typographic attacks. By selectively ablating a typographic circuit, Dyslexify is able to defend CLIP against typographic attacks without requiring fine-tuning steps, offering a practical and interpretable method for controlling model behavior. To our knowledge, this is the first work to address typographic attacks in CLIP through causal interventions. Dyslexify demonstrates that fine-grained control over model capabilities is achievable through targeted architectural manipulations without retraining, and thus paves the way for more robust and modular deployment of multimodal models. Beyond standard benchmarks, we further show that typographic attacks constitute a realistic threat vector in the medical domain, where they can mislead safety-critical models, and that Dyslexify substantially mitigates this vulnerability. We believe this work motivates a broader shift toward mechanistic interpretability as a tool not only for understanding, but for controlling safety-relevant behaviors in deep transformer models. Finally, we release a family of dyslexic CLIP models that are significantly more robust against typographic attacks. These models serve as drop-in replacements for safety-critical applications where the risks posed by adversarial text manipulation outweigh the benefits of typographic understanding. Limitations and future work: Dyslexify enhances the typographic robustness of theclstoken by preventing specialized typographic attention heads from writing to it. However, many multimodal applications, such as LLaVA and IP adapters (Ye et al., 2023; Liu et al., 2023; 2024b), leverage not only theclstoken but also spatial tokens, allowing typographic information to propagate into downstream tasks. This might limits the impact of Dyslexify on improving robustness in applications, and calls for further investigation into its generalizability to VLM setups. Furthermore is it standard practice to evaluate adversarial defenses against adaptive attacks (Tramer et al., 2020), i.e., attacks that are explicitly optimized to circumvent the defense mechanism. In our 9 case, however, such an evaluation is not feasible: typographic attacks are inherently non-differentiable, which prevents constructing adaptive variants that directly optimize against Dyslexify. Misuse Potential: While we aim to enhance the safety of multimodal systems, we acknowledge that our insights into CLIP’s behaivor under typographic attacks could be exploited by attackers. Specifically, adversarial inputs might be crafted to increase the spatial attention of heads inC, making typographic attacks even more effective. ACKNOWLEDGEMENTS This work was supported by the German Research Foundation (DFG) as research unitDeSBi [KI-FOR 5363] (459422098). ETHICS STATEMENT We affirm that this work adheres to the ICLR Code of Ethics. All authors have read and agree with the Code. We note that our study does not involve human subjects or sensitive personal data. While insights into typographic attacks may inform adversarial strategies, our primary aim is to strengthen robustness and safety of multimodal models. REPRODUCIBILITY STATEMENT We have taken efforts to make our results reproducible. Our code is open-sourced and provided to reviewers in anonymized form. All results and the majority of plots in the paper can be reproduced with the released code. REFERENCES Hiroki Azuma and Yusuke Matsui. Defense-prefix for preventing typographic attacks on clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3644–3653, 2023. Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative compo- nents with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, p. 446–461. Springer, 2014. G. Bradski. The OpenCV Library. Dr. Dobb’s Journal of Software Tools, 2000. Yue Cao, Yun Xing, Jie Zhang, Di Lin, Tianwei Zhang, Ivor Tsang, Yang Liu, and Qing Guo. Scenetap: Scene-coherent typographic adversarial planner against vision-language models in real-world environments. arXiv preprint arXiv:2412.00114, 2024. Hao Cheng, Erjia Xiao, Jiayan Yang, Jiahang Cao, Qiang Zhang, Jize Zhang, Kaidi Xu, Jindong Gu, and Renjing Xu. Uncovering vision modality threats in image-to-image tasks. arXiv preprint arXiv:2412.05538, 2024. Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), p. 168–172. IEEE, 2018. Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288, 2019. Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, Yue Zhang, Wenyu Lv, Kui Huang, Yichao Zhang, Jing Zhang, Jun Zhang, Yi Liu, Dianhai Yu, and Yanjun Ma. Paddleocr 3.0 technical report, 2025. URL https://arxiv.org/abs/2507.05595. 10 Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, p. 248–255. Ieee, 2009. Maximilian Dreyer, Lorenz Hufe, Jim Berend, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. From what to how: Attributing clip’s latent components reveals unexpected semantic reliance. arXiv preprint arXiv:2505.20229, 2025. Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, p. 1151–1163, 2023. Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.23915/distill.00030. https://distill.pub/2021/multimodal-neurons. Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typographic visual prompts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, p. 23951–23959, 2025. Kuo-Han Hung, Ching-Yun Ko, Ambrish Rawat, I Chung, Winston H Hsu, Pin-Yu Chen, et al. Attention tracker: Detecting prompt injection attacks in llms. arXiv preprint arXiv:2411.00348, 2024. Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URLhttps://doi.org/10.5281/ zenodo.5143773. Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. Advances in Neural Information Processing Systems, 35:29262–29277, 2022. Sonia Joseph, Praneet Suresh, Ethan Goldfarb, Lorenz Hufe, Yossi Gandelsman, Robert Graham, Danilo Bzdok, Wojciech Samek, and Blake Aaron Richards. Steering clip’s vision transformer with sparse autoencoders. arXiv preprint arXiv:2504.08729, 2025. Subaru Kimura, Ryota Tanaka, Shumpei Miyawaki, Jun Suzuki, and Keisuke Sakaguchi. Empirical analysis of large vision-language models against goal hijacking via visual prompt injection. arXiv preprint arXiv:2408.03554, 2024. Xiang Li, Congcong Wen, Yuan Hu, and Nan Zhou. Rs-clip: Zero shot remote sensing scene classification via contrastive vision-language supervision. International Journal of Applied Earth Observation and Geoinformation, 124:103497, 2023. Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Remoteclip: A vision language foundation model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 62:1–16, 2024a. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URLhttps: //llava-vl.github.io/blog/2024-01-30-llava-next/. Mingrui Liu, Sixiao Zhang, and Cheng Long. Wukong framework for not safe for work detection in text-to-image systems. arXiv preprint arXiv:2508.00591, 2025. S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013. 11 Joanna Materzy ́ nska, Antonio Torralba, and David Bau. Disentangling visual and written concepts in clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16410–16419, 2022. A. Mishra, K. Alahari, and C. V. Jawahar. Scene text recognition using higher order language priors. In BMVC, 2012. Maan Qraitem, Nazia Tasnim, Piotr Teterwak, Kate Saenko, and Bryan A Plummer. Vision-llms can fool themselves with self-generated typographic attacks. arXiv preprint arXiv:2402.00626, 2024. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. PMLR, 2021. Camilo Carvajal Reyes, Joaquín Fontbona, and Felipe Tobar. Towards sfw sampling for diffusion models via external conditioning. arXiv preprint arXiv:2505.08817, 2025. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278–25294, 2022. Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in neural information processing systems, 33:1633–1645, 2020. Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1–9, 2018. Vicente Vivanco, Gaurav Kumar Nayak, and Mubarak Shah. Geoclip: Clip-inspired alignment between locations and images for effective worldwide geo-localization. In Advances in Neural Information Processing Systems, 2023. Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2022, p. 3876, 2022. Justus Westerhoff, Erblina Purellku, Jakob Hackstein, Leo Pinetzki, and Lorenz Hufe. Scam: A real-world typographic robustness evaluation for multimodal foundation models. arXiv preprint arXiv:2504.04893, 2025. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. Yue Yang, Mona Gandhi, Yufei Wang, Yifan Wu, Michael Yao, Chris Callison-Burch, James Gee, and Mark Yatskar. A textbook remedy for domain shifts: Knowledge priors for medical image analysis. Advances in neural information processing systems, 37:90683–90713, 2024. Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721, 2023. 12 CONTENTS OF APPENDIX A Use of Large Language Models13 B MLP layers compress information13 C Attention sinks for typographic attention heads14 D Ablation Curves15 E Further details on Medical Application16 F Datasets17 G Circuit size17 H OCR capabilities17 I Runtime and memory requirements18 J Blurring baseline18 K Relationship between probes and T i,ℓ scores19 AUSE OF LARGE LANGUAGE MODELS Large Language Models (LLMs) were used in the preparation of this work. Specifically, they assisted in polishing the text, generating code snippets, and conducting literature searches. All conceptual contributions, experimental designs, analyses, and conclusions are the sole work of the authors. BMLP LAYERS COMPRESS INFORMATION In Section 3, we observe thatAcc(P img,ℓ )andAcc(P typo,ℓ )increases after attention layers, but consistently drops following MLP blocks. Thus we estimate the intrinsic dimensionality (ID) of cls-token representations across layers using PCA. From a 5% split of the ImageNet-100-Typo training set, we extract residual stream activations for each layer. For each embeddingh ℓ ∈R d , we fit PCA and define ID as the smallest number of principal componentsksuch that the cumulative explained variance exceeds 95%: ID = min ( k : P k j=1 λ j P d j=1 λ j ≥ 0.95 ) , whereλ j are the PCA eigenvalues. A larger ID indicates higher representational complexity. We report ID across layers and token types to analyze how typographic inputs affect the geometry of CLIP representations. ResultsFig. 7 compares linear probe accuracy (left) and ID (right) across layers of a ViT-B model. We observe a consistent pattern: attention layers tend to increase both probe accuracy and intrinsic dimensionality, while MLP layers reduce them. This trend is most prominent in the middle layers (3–7), suggesting that attention blocks introduce linearly accessible information, whereas MLPs compress or remove it. This pattern is not consistent throughout the model. The MLPs at layers 9 and 11 exhibit increases in ID, deviating from the overall compression trend. Conversely, the attention block at layer 11 causes a sharp drop in ID. These exceptions indicate that deeper layers may serve more specialized roles. 13 Figure 7: Attention layers increase, MLP layers reduceclstoken information. (a) Linear probe accuracy rises after attention blocks and drops after MLPs, indicating improved linear accessibility followed by compression. (b) Intrinsic dimensionality shows a matching trend, especially in layers 3–7. Exceptions include MLPs at layers 9 and 11 (ID increase) and the attention block at layer 11 (ID drop), suggesting deeper-layer specialization. CATTENTION SINKS FOR TYPOGRAPHIC ATTENTION HEADS We analyze the attention patterns of head 5 in layer 6 (H 5,6 ) in ViT-B, which has the highestT i,ℓ score in the model. More specifically we evaluate its spatial attention norm∥A ∗ 5,6 ∥on ImageNet-100 and ImageNet-100-Typo. As shown in Fig. 8,∥A ∗ 5,6 ∥is systematically higher on ImageNet-100-Typo than on ImageNet-100. While the distribution on typopgrahic images is unimodal, the distribution on the original dataset is bimodal. Manual inspection reveals that the high-norm mode in ImageNet-100 contains incidental text in, such as watermarks or copyright tags. One possible interpretation of these results is thatH 5,6 uses thecls-to-clsattention as an attention sink (Xiao et al., 2023), to selectively adjust the impact of this specialized typographic attention head. Building on these finding we evaluateH 5,6 ’s capabilities to predict if a samplexoriginates from ImageNet-100 or ImageNet-100-typo. The score∥A ∗ 5,6 ∥ROC-AUC of 0.887, indicating that this attention signal can be used as a robust classifier. In comparison, a linear classifier trained on the same task reaches an ROC-AUC of 1.0, but overfits to superficial typographic features specific to the ImageNet-100-Typo construction. Many images in the original dataset that contain real-world typography are still correctly classified as non-typographic by this probe - supporting the conclusion that it does not generalize beyond the synthetic intervention. 14 Figure 8: Distribution of the spatial attention norm of ViT-B headH 5,6 across ImageNet-100 and ImageNet-100-Typo. The norm is consistently higher for typographic images, while the non- typographic distribution is bimodal. Manual inspection links the higher mode to incidental text, suggesting that the head selectively activates in response to typography, regardless of its origin. DABLATION CURVES 012345678 Number of Ablated Heads 45 50 55 60 65 70 75 80 85 90 Zeroshot Accuracy (%) ViT-B ImageNet-100 ImageNet-100-Typo 0510152025 Number of Ablated Heads 45 50 55 60 65 70 75 80 85 90 Zeroshot Accuracy (%) ViT-L ImageNet-100 ImageNet-100-Typo 05101520 Number of Ablated Heads 45 50 55 60 65 70 75 80 85 90 Zeroshot Accuracy (%) ViT-H ImageNet-100 ImageNet-100-Typo 0510152025 Number of Ablated Heads 45 50 55 60 65 70 75 80 85 90 Zeroshot Accuracy (%) ViT-G ImageNet-100 ImageNet-100-Typo 0510152025 Number of Ablated Heads 45 50 55 60 65 70 75 80 85 90 Zeroshot Accuracy (%) ViT-BigG ImageNet-100 ImageNet-100-Typo 012345678 Number of Ablated Heads 45 50 55 60 65 70 75 80 85 90 Zeroshot Accuracy (%) WhylesionCLIP ISIC2019 ISIC2019-Typo Figure 9: Tradeoff between general accuracy and typographic robustness as a function of the number of ablated heads. Ablations are applied in decreasing order of T i,ℓ . 15 EFURTHER DETAILS ON MEDICAL APPLICATION We evaluated Dyslexify in the safety-critical context of melanoma detection, using WhyLesionCLIP based on OpenCLIP ViT-L as the underlying foundation model. To construct typographic attack datasets, we introduced textual labels into ISIC2019, HAM10k, and BCN20k, resulting in paired typographic variants. The typographic circuit was selected using the same procedure as in Section 5.2, with tolerance ε = 0.01 and maximum skips k = 10. Table 4 reports detailed results. We find that typographic attacks reduce zero-shot accuracy by up to 22.1%, while Dyslexify recovers up to 19.3% accuracy, and even improves performance on non-attacked datasets in some cases. These results confirm that typographic attacks pose a realistic failure mode for medical AI systems. Fig. 10 further illustrates that typographic probes perform more strongly in WhyXrayCLIP than in WhyLesionCLIP, which may be explained by frequent typographic artifacts (e.g., “R” markers) in X-ray training data. This suggests that the degree of vulnerability depends on the presence of typographic features in the training distribution. Table 4: Comparison of dyslexic model performance on WhylesionCLIP dataset, showing accuracy changes relative to the base model, with improvements (↑) or declines (↓). The suffix -T indicates the corresponding typographic version of the dataset. Not TypographicSynthetic Typographic ModelISIC2019MelanomaBCN20kHAM10kISIC2019-T Melanoma-T BCN20k-T HAM10k-T Base77.8084.1031.9935.6655.1962.1019.6519.48 Ours77.47↓0.33 84.40↑0.30 32.49↑0.50 40.20↑4.5470.92↑15.7381.40↑19.3026.46↑6.8124.42↑4.94 Figure 10: (a) Linear probes on CLS embeddings show that typographic probes perform less well in WhyLesionCLIP compared to WhyXrayCLIP, indicating weaker encoding of typographic features in the lesion model. (b) Examples of typographic artifacts (e.g., “R” markers) commonly found in X-ray training data, which may influence probe performance. 16 FDATASETS For the zero-shot evaluation we tested a variety of datasets, grouped below by purpose. RTA-100 (Azuma & Matsui, 2023) consists of 1,000 handcrafted typographic attacks, each written on a Post-it note and overlaid onto natural images. Disentangling (Materzy ́ nska et al., 2022) includes 180 typographic attacks, also written on Post-its, designed to probe the separation of visual and textual features in multimodal models. PAINT-DS (Ilharco et al., 2022) comprises 110 Post-it-based typographic attacks and serves to evaluate patch-level vulnerabilities in vision-language models. Food-101 (Bossard et al., 2014) is a standard image classification dataset containing 101 food categories, each with 1,000 images. FGVC-Aircraft (Maji et al., 2013) (referred to as Aircraft in our paper) is a fine-grained classification benchmark consisting of 10,000 images across 100 aircraft variants. ImageNet-100 2 is a subset of the ImageNet-1k dataset (Deng et al., 2009), containing 100 object and animal classes with 1,000 images per class. ISIC2019 3 contains 25,331 images available for the classification of dermoscopic images among nine different diagnostic Tschandl et al. (2018); Combalia et al. (2019); Codella et al. (2018). In this paper we evaluate the binary classification task into the classes "Benign" and "Malignant". HAM10k (Tschandl et al., 2018) includes 10,015 dermatoscopic images labeled into seven diagnostic categories: Actinic Keratoses, Basal Cell Carcinoma, Benign Keratosis-like Lesions, Dermatofibroma, Melanoma, Melanocytic Nevi, and Vascular Lesions. BCN20k (Combalia et al., 2019) comprises 19,424 dermatoscopic images collected at the Hospital Clínic de Barcelona, annotated into eight categories: Actinic Keratoses, Basal Cell Carcinoma, Benign Keratosis-like Lesions, Dermatofibroma, Melanocytic Nevi, Melanoma, Squamous Cell Carcinoma, and Vascular Lesions. For the generation of the -typo dataset we used the following fonts: Times New Roman, Georgia, Arial. GCIRCUIT SIZE Table 5: Number of selected and total headsH i,ℓ in circuitC per model ModelSelectedTotalPercentage (%) B81445.6 L2928810.1 H243846.2 G294806.0 Big-G265764.5 HOCR CAPABILITIES Dyslexify is designed to suppress typographic understanding in favor of robustness to typographic attacks. As a consequence, we expect a degradation on tasks that require optical character recognition (OCR). To quantify this effect, we evaluate the base and dyslexic OpenCLIP models on the IIIT5K word recognition benchmark (Mishra et al., 2012), using the standard small and medium lexicon settings. 2 https://w.kaggle.com/datasets/ambityga/imagenet100 3 https://w.kaggle.com/datasets/salviohexia/isic-2019-skin-lesion-images-for-classification 17 For each image, we treat the lexicon entries as candidate labels and perform zero-shot classification with CLIP, reporting the top-1 word accuracy. Table 6 reports the results. Across all model sizes, Dyslexify substantially reduces OCR performance, with drops between 8 and 30 percentage points depending on the model and lexicon size. The effect is strongest for the smaller ViT-B and ViT-L models and remains pronounced even for the larger ViT-bigG. This confirms that dyslexic models are not appropriate for applications that benefit from text recogni- tion. Instead, they are intended as drop-in replacements in safety-critical settings where the risks of adversarial text manipulation outweigh the utility of typographic understanding. Table 6: Comparison of dyslexic model performance on IIIT5K OCR evaluation, showing accuracy changes relative to the base model, with improvements (↑) or declines (↓). Model Small Lexicon Medium Lexicon B25.03↓30.5011.37↓17.87 L35.50↓30.2715.30↓20.47 H51.27↓12.2326.40↓8.57 G51.40↓13.7027.27↓10.03 Big-G54.47↓20.3729.40↓15.67 IRUNTIME AND MEMORY REQUIREMENTS We benchmark the computational cost of Dyslexify against Defense-Prefix (DP) in terms of wall-clock runtime and memory requirements. All experiments were run on a single NVIDIA H200 GPU using identical data pipelines and batch sizes. Importantly, both methods were evaluated using the entire ImageNet-100 training split for circuit construction and prefix optimization, respectively. Table 7: Runtime comparison (in seconds) of Dyslexify and Defense-Prefix (DP) in seconds on a single NVIDIA H200 GPU. Lower is better. Model DyslexifyDP ViT-B3 818 14 598 ViT-L23 518 23 145 ViT-H24 237 39 483 Dyslexify is3.8×faster than DP on ViT-B,1.6×faster on ViT-H, and comparable on ViT-L. The cost gap reflects the fact that Dyslexify requires only forward passes with circuit ablations, whereas DP performs gradient-based optimization over a learnable prefix. Runtime reduction through subset evaluation. The reported Dyslexify runtimes use the full ImageNet-100 training set. In practice, Dyslexify does not require the entire dataset: we observe that using a significantly smaller random subset yields nearly identical circuits and comparable robustness gains. VRAM requirements. Dyslexify further has a low memory footprint. It runs on all evaluated models, including those exceeding 1B parameters, on a single NVIDIA Titan RTX (24 GB). Defense- Prefix cannot be trained beyond ViT-L due to CUDA out-of-memory errors. This makes Dyslexify suitable for low-resource environments where finetuning-based defenses are infeasible. JBLURRING BASELINE Following the reviewer’s suggestion, we implemented a simple OCR-based defense baseline that removes typographic content directly from the image. We use the PaddleOCR text detector (Cui et al., 18 2025) to localize text regions and then apply a Gaussian blur to each detected bounding box using OpenCV (Bradski, 2000).. In brief, for each input image we (1) run an OCR pass to obtain polygonal text boxes; (2) compute the pixel bounds of each box; and (3) blur only the corresponding subregion with a large-kernel Gaussian filter (kernel size 51,σ = 25). No other pixels are modified. This produces a text-ablated variant of ImageNet-100-Typo that preserves the underlying image content while suppressing overlaid words. This baseline allows direct comparison between Dyslexify (a mechanistic intervention) and a purely image-level text-removal defense. The OCR+blur baseline is strong: it consistently outperforms Dyslexify on ImageNet-100-Typo across all model sizes. This confirms that removing text at the pixel level is an effective strategy for mitigating typographic attacks. On non-typographic datasets, the blur baseline has minimal impact - the accuracy remains within±0.3%of the original models - indicating that the image modifications do not substantially harm object-centric features. Dyslexify shows similarly small degradation (<1%), but delivers lower absolute robustness than OCR+blur on this benchmark. While OCR-based preprocessing is competitive in absolute robustness, it comes with clear limita- tions. First, OCR+blur must be applied at every inference step, adding recurring computational overhead, whereas Dyslexify is executed once per model and introduces no inference-time cost. Second, OCR+blur removes visual information rather than modifying internal causal pathways; it therefore does not provide mechanistic insight into model behavior or contribute to understanding the underlying failure modes. Overall, the results highlight that OCR-based preprocessing is a strong baseline, but Dyslexify serves a different purpose: it demonstrates that mechanistic analysis can yield actionable interventions that improve robustness without retraining and without modifying the input image. We hope that this comparison highlights a broader point: while preprocessing-based defenses can be strong, they do not replace the need for mechanistic approaches. Our results show that even simple circuit-level interventions can yield meaningful robustness gains without retraining, without modifying inputs, and without inference-time overhead. We hope that this work encourages further research on actionable interpretability - using mechanistic understanding not only to analyze models, but to directly improve their safety and reliability. Table 8: Comparison of dyslexic model performance on ImageNet-100 with and without blur. ImageNet-100BLHGBig-G Normal74.36 79.76 83.7483.24 85.06 Blurred74.32 79.48 83.88 83.2084.94 Dyslexify75.00 79.5283.40 82.5884.72 Both74.6479.50 83.50 82.5284.60 Table 9: Comparison of dyslexic model performance on ImageNet-100-Typo with and without blur. ImageNet-100-TypoBLHGBig-G Normal46.94 51.90 54.08 46.7061.90 Blurred68.52 75.12 79.98 79.56 80.84 Dyslexify66.84 72.22 75.34 68.7678.64 Both67.8674.3080.00 78.4880.52 KRELATIONSHIP BETWEEN PROBES ANDT i,ℓ SCORES As discussed in Section 3, we observe a strong correspondence between layers with elevatedT i,ℓ scores and those where the typographic probeP typo,ℓ exhibits sharp increases in accuracy. In this section, we extend this analysis across all evaluated model sizes to support the trends previously 19 shown for ViT-B. Fig. 11 to Fig. 15 visualize this relationship for each model. To improve readability, we transpose the probe accuracy plots: they-axis denotes the layer index, while thex-axis indicates probe accuracy. H0 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 Head 18.529.813.320.314.730.312.67.518.76.29.735.3 21.716.211.612.216.613.537.315.28.233.89.069.0 39.318.617.214.422.518.637.410.34.011.311.167.4 23.519.314.620.034.518.216.833.36.86.234.052.5 11.213.719.416.923.015.816.27.95.410.814.746.2 18.618.517.314.215.410.072.819.715.94.536.740.0 16.67.415.716.78.921.112.69.17.87.121.635.6 14.519.222.116.111.015.413.88.07.624.048.550.6 25.721.414.233.314.930.422.534.170.825.123.050.2 19.212.919.921.112.220.433.36.110.559.69.154.0 32.421.621.213.414.422.218.18.07.49.118.658.3 13.715.020.817.315.012.732.910.914.05.218.064.0 Typographic Scores L0L1L2L3L4L5L6L7L8L9L10L11 Layer H0 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 Head Thresholded Scores 0.0 0.2 0.4 0.6 0.8 1.0 Probe Accuracy Typographic Probe Heads with T i, > 2 T + T Figure 11: ViT-B typographic attention scoresT i,ℓ (top), thresholded and compared with linear probe accuracies (bottom). Layers with higher typographic attention scores align with increased probe accuracy, highlighting a correspondence between attention patterns and typographic feature decodability. 20 H0 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 H12 H13 H14 H15 Head 14.811.59.412.212.217.07.812.714.318.434.514.913.08.417.842.35.862.210.516.629.241.336.447.8 20.113.314.717.415.923.429.314.719.010.633.135.39.213.113.511.414.519.814.114.321.337.941.314.1 18.77.28.412.916.67.017.715.430.522.414.311.510.117.49.39.613.020.835.913.226.741.344.332.0 20.415.22.016.922.710.815.722.614.512.413.67.815.39.714.167.315.013.816.48.845.836.847.353.9 11.112.69.89.216.818.217.624.214.314.919.711.28.039.410.919.713.610.423.632.229.943.143.355.3 24.215.811.119.68.614.521.513.212.516.831.58.313.715.448.26.116.819.321.88.322.750.757.547.4 35.99.611.119.59.918.113.314.541.29.620.514.150.231.24.77.214.78.440.142.632.240.045.136.4 19.111.820.821.723.817.217.918.519.512.513.246.76.97.521.514.415.120.18.518.524.833.341.140.6 38.415.310.312.034.526.99.615.817.522.233.820.120.243.514.215.712.113.812.457.222.234.452.352.3 10.611.716.217.518.015.48.521.317.936.920.517.016.012.910.236.211.119.411.824.226.739.850.843.6 17.813.316.910.815.110.717.112.110.515.019.817.210.49.79.049.922.926.710.118.228.726.444.433.3 11.38.29.212.921.19.919.016.718.513.712.513.711.711.810.212.324.012.214.126.416.142.839.958.0 18.811.010.714.79.917.620.38.924.420.411.824.916.014.59.68.852.021.414.814.016.130.440.937.1 16.38.920.722.921.315.410.517.613.616.116.020.89.831.811.810.116.710.810.030.030.344.136.748.8 18.513.412.011.314.116.017.319.214.419.313.219.734.53.79.210.015.68.116.76.335.455.942.346.3 23.95.810.915.613.415.028.313.315.316.827.814.05.717.19.18.813.317.429.415.921.538.155.758.4 Typographic Scores L0L1L2L3L4L5L6L7L8L9L10L11L12L13L14L15L16L17L18L19L20L21L22L23 Layer H0 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 H12 H13 H14 H15 Head Thresholded Scores 0.0 0.2 0.4 0.6 0.8 1.0 Probe Accuracy Typographic Probe Heads with T i, > 2 T + T Figure 12: ViT-L typographic attention scoresT i,ℓ (top), thresholded and compared with linear probe accuracies (bottom). Layers with higher typographic attention scores align with increased probe accuracy, highlighting a correspondence between attention patterns and typographic feature decodability. 21 H0 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 H12 H13 H14 H15 Head 29.512.020.831.212.514.815.713.717.420.38.630.39.918.615.151.616.57.918.016.622.460.715.830.422.820.027.539.059.138.939.967.0 16.410.810.517.017.021.628.216.016.711.632.343.817.322.718.48.912.014.320.922.117.014.721.317.121.529.528.041.731.729.135.438.2 12.017.420.912.916.213.79.914.710.810.230.117.813.120.374.110.136.08.221.146.517.018.718.931.615.723.632.333.040.668.838.842.3 24.89.625.421.09.016.08.010.413.014.99.423.018.316.919.118.716.910.225.351.417.622.47.823.617.817.235.239.029.640.026.155.4 21.313.59.117.540.524.415.815.116.220.421.916.721.66.716.222.120.573.822.930.815.730.318.426.727.816.718.935.029.649.939.841.3 16.010.713.412.717.99.412.917.013.521.822.014.338.116.019.313.315.424.716.418.115.214.621.945.241.617.532.935.229.347.043.245.4 21.212.321.813.024.917.317.718.716.39.130.344.517.119.415.312.070.111.015.215.218.916.630.319.721.928.230.540.231.954.035.036.8 14.510.09.519.413.07.415.39.575.19.318.815.920.718.121.319.313.380.017.412.017.151.319.813.720.926.340.431.333.840.840.149.3 13.914.221.715.231.721.115.425.416.816.010.015.48.621.618.015.932.419.236.723.823.027.215.319.314.035.811.930.935.532.251.837.5 20.911.714.619.211.318.014.58.926.820.412.810.68.814.713.317.114.425.88.314.370.419.721.525.728.518.235.633.330.350.856.831.8 32.513.915.913.69.212.019.012.218.216.819.513.848.717.414.111.111.612.314.225.79.420.528.424.423.524.531.729.927.631.032.345.9 18.22.410.911.417.617.915.94.617.611.020.814.162.714.96.737.69.319.99.525.420.518.421.819.924.427.630.644.233.934.436.261.3 26.111.320.314.015.013.934.717.318.016.813.324.513.612.113.715.418.019.515.713.821.517.718.722.724.225.322.728.941.235.135.736.9 16.613.915.917.010.07.112.68.110.99.518.910.116.714.225.716.823.830.118.626.663.522.616.320.619.726.539.932.130.735.537.445.2 32.09.512.614.019.718.415.513.36.514.918.214.519.024.038.513.423.714.514.722.315.411.538.526.222.318.441.432.230.249.049.643.6 26.313.422.114.410.17.618.013.919.215.515.722.621.314.116.312.814.912.613.011.321.018.117.719.723.615.930.239.323.339.735.616.6 Typographic Scores L0L1L2L3L4L5L6L7L8L9L10L11L12L13L14L15L16L17L18L19L20L21L22L23L24L25L26L27L28L29L30L31 Layer H0 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 H12 H13 H14 H15 Head Thresholded Scores 0.0 0.2 0.4 0.6 0.8 1.0 Probe Accuracy Typographic Probe Heads with T i, > 2 T + T Figure 13: ViT-H typographic attention scoresT i,ℓ (top), thresholded and compared with linear probe accuracies (bottom). Layers with higher typographic attention scores align with increased probe accuracy, highlighting a correspondence between attention patterns and typographic feature decodability. 22 H0 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 H12 H13 H14 H15 Head 13.68.212.319.314.28.410.89.59.720.67.723.617.117.321.924.124.227.412.617.329.718.416.826.922.231.230.542.433.422.428.716.628.411.219.428.659.241.644.647.9 23.38.412.012.813.98.516.58.610.48.511.521.29.717.748.218.611.012.916.314.914.023.88.417.224.519.531.918.835.526.521.433.730.020.323.535.237.439.344.147.6 25.39.411.015.515.315.812.015.814.66.68.021.030.013.541.421.620.037.916.416.572.720.123.823.218.543.619.520.122.021.226.320.923.412.421.560.730.229.836.948.4 24.28.69.714.68.57.27.114.815.917.210.417.38.617.613.020.544.636.220.122.711.07.313.611.317.540.120.317.561.423.317.421.422.819.744.435.837.035.950.545.9 16.18.614.210.915.014.614.98.321.613.97.814.412.25.118.114.510.821.614.28.823.916.628.022.721.721.122.819.219.624.529.139.019.016.023.836.930.361.438.151.6 21.49.310.613.317.87.712.28.115.411.019.117.98.711.820.234.716.059.421.58.913.314.02.374.024.718.124.126.029.629.821.331.521.419.129.134.122.454.735.657.0 35.09.115.110.720.725.26.913.85.912.28.917.99.211.014.812.913.910.230.513.428.212.121.326.016.99.821.524.025.316.019.317.552.338.228.829.834.955.644.741.7 31.09.610.411.516.18.514.010.19.214.514.524.124.518.427.123.020.623.518.021.110.813.240.313.420.526.919.539.119.620.223.632.68.616.521.220.945.339.151.645.1 8.57.314.515.024.527.517.812.19.616.625.834.814.617.618.918.310.916.611.068.916.913.225.632.836.670.772.920.318.630.021.921.110.021.337.628.519.566.338.249.2 20.612.214.911.910.86.916.018.324.711.08.56.812.415.437.929.221.214.321.620.919.513.216.721.823.319.824.623.022.649.622.126.322.114.831.332.546.837.643.740.7 30.97.811.816.19.014.611.014.516.58.711.16.265.716.511.013.359.119.027.410.910.817.715.226.218.916.238.818.07.926.734.821.318.526.230.341.241.731.448.514.4 22.18.810.517.522.217.86.612.69.57.82.513.510.416.342.115.324.333.514.215.638.23.715.615.713.516.568.427.115.817.737.721.218.624.625.338.639.259.044.949.0 21.77.812.410.314.57.818.111.217.410.415.719.615.516.126.418.327.324.311.936.917.552.013.918.216.123.237.841.521.517.627.815.423.327.736.827.352.641.059.139.4 14.68.912.09.614.121.315.213.317.111.017.418.814.627.312.711.217.216.831.822.020.513.410.314.513.820.627.720.828.017.641.820.46.131.536.345.249.137.161.947.1 33.210.014.415.816.412.19.77.98.618.323.211.46.424.723.618.79.320.623.221.546.510.528.017.016.416.318.724.519.610.523.66.635.828.020.219.040.933.846.549.6 18.28.511.810.518.511.07.221.214.814.114.77.517.118.110.111.526.947.320.514.323.516.126.78.620.233.524.547.122.510.153.022.819.811.432.822.032.947.443.164.2 Typographic Scores L0L1L2L3L4L5L6L7L8L9L10L11L12L13L14L15L16L17L18L19L20L21L22L23L24L25L26L27L28L29L30L31L32L33L34L35L36L37L38L39 Layer H0 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 H12 H13 H14 H15 Head Thresholded Scores 0.0 0.2 0.4 0.6 0.8 1.0 Probe Accuracy Typographic Probe Heads with T i, > 2 T + T Figure 14: ViT-G typographic attention scoresT i,ℓ (top), thresholded and compared with linear probe accuracies (bottom). Layers with higher typographic attention scores align with increased probe accuracy, highlighting a correspondence between attention patterns and typographic feature decodability. 23 H0 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 H12 H13 H14 H15 Head 14.48.611.712.215.419.48.722.019.015.126.317.714.49.17.714.835.68.712.229.721.621.040.320.931.525.133.010.662.711.218.777.712.725.117.116.59.815.446.623.218.413.219.851.222.026.341.346.3 9.014.211.316.611.58.613.715.07.77.513.012.59.76.67.917.612.311.318.013.512.915.124.024.94.89.110.411.840.066.08.620.468.135.051.716.927.58.316.016.013.136.344.015.336.940.041.542.6 15.99.010.514.17.511.29.115.39.716.410.09.310.010.77.613.611.610.717.121.122.119.123.415.924.226.710.719.69.520.015.418.513.647.235.726.313.715.818.115.429.132.842.724.031.240.742.556.5 11.27.213.48.515.124.37.337.911.47.517.212.320.214.27.613.88.216.317.931.424.116.518.919.413.923.418.928.28.922.738.511.88.510.88.325.121.110.625.019.719.820.342.342.939.028.840.343.0 6.38.28.412.27.89.910.630.616.79.858.511.214.732.47.412.311.410.618.525.716.617.413.428.133.630.622.914.614.224.612.825.012.525.228.927.614.023.015.217.419.818.732.827.033.229.728.939.9 11.88.511.410.011.39.39.414.611.612.39.64.712.06.410.011.191.311.911.96.716.416.820.126.910.215.518.731.816.513.335.012.720.514.315.828.822.529.631.918.728.322.722.218.338.355.543.350.5 11.67.711.99.610.834.921.513.018.915.137.110.211.715.016.511.410.612.419.326.020.325.223.218.131.037.114.915.416.913.111.915.623.012.19.324.322.228.417.126.822.940.811.450.735.642.944.142.9 34.89.211.012.617.312.913.849.823.312.013.013.513.716.616.39.112.58.932.715.519.518.043.431.413.851.87.019.210.813.483.518.468.68.67.319.716.519.021.318.618.415.327.844.037.832.643.947.6 4.08.912.315.59.913.47.59.216.316.39.68.67.68.011.821.210.19.716.67.236.221.96.615.618.813.823.624.913.78.134.415.713.762.122.313.315.327.925.117.517.734.242.731.047.124.548.059.0 10.99.112.413.310.911.813.019.212.316.611.712.812.28.521.410.615.613.511.715.624.524.628.112.715.18.124.714.118.517.918.539.428.250.917.68.868.916.516.323.325.220.345.743.030.324.022.840.8 10.511.011.46.59.415.010.718.316.310.410.213.810.625.811.313.615.212.614.420.78.821.821.628.648.576.69.818.29.58.315.06.214.636.125.916.923.320.216.713.815.416.846.841.038.130.446.010.5 8.013.39.516.412.518.09.117.310.112.372.013.911.617.07.032.810.813.415.58.118.916.99.834.534.36.65.830.420.757.111.29.120.616.915.615.418.915.715.422.515.119.725.226.768.442.456.159.0 22.78.211.211.78.527.78.87.414.718.111.013.69.07.55.712.520.016.210.928.49.412.429.525.810.66.069.517.530.217.117.423.567.211.76.414.19.816.325.530.918.821.041.642.421.442.413.147.2 15.99.212.310.38.918.58.515.413.414.812.87.318.67.69.57.18.012.818.912.714.016.917.516.818.437.219.016.413.458.89.218.833.28.214.318.122.312.420.713.212.815.645.137.527.529.333.241.2 14.111.211.313.111.411.08.822.514.67.714.614.812.215.36.212.922.218.317.140.719.718.231.79.116.47.613.125.923.66.720.68.210.324.921.315.59.113.928.522.618.421.222.655.224.532.739.840.6 11.69.210.613.24.518.79.110.312.812.112.017.210.57.719.710.112.912.211.013.041.452.725.219.123.227.812.962.022.811.352.477.523.465.814.513.935.619.217.231.421.727.025.541.351.334.429.042.3 Typographic Scores L0L1L2L3L4L5L6L7L8L9L10L11L12L13L14L15L16L17L18L19L20L21L22L23L24L25L26L27L28L29L30L31L32L33L34L35L36L37L38L39L40L41L42L43L44L45L46L47 Layer H0 H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 H11 H12 H13 H14 H15 Head Thresholded Scores 0.0 0.2 0.4 0.6 0.8 1.0 Probe Accuracy Typographic Probe Heads with T i, > 2 T + T Figure 15: ViT-BigG typographic attention scoresT i,ℓ (top), thresholded and compared with linear probe accuracies (bottom). Layers with higher typographic attention scores align with increased probe accuracy, highlighting a correspondence between attention patterns and typographic feature decodability. 24