Paper deep dive
On the Adversarial Robustness of Discrete Image Tokenizers
Rishika Bhagwatkar, Irina Rish, Nicolas Flammarion, Francesco Croce
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 10:23:59 PM
Summary
This paper investigates the adversarial robustness of discrete image tokenizers, which are increasingly used in multimodal systems. The authors demonstrate that these tokenizers are highly vulnerable to unsupervised attacks that perturb pre-quantization embeddings to alter token sequences. They propose an unsupervised adversarial fine-tuning method that significantly improves robustness against both unsupervised and end-to-end supervised attacks while maintaining clean performance and generalizing to unseen tasks.
Entities (11)
Relation Signals (5)
Unsupervised Adversarial Training ā improvesrobustnessof ā Discrete Image Tokenizers
confidence 95% Ā· our approach significantly improves robustness to both unsupervised and end-to-end supervised attacks
Discrete Image Tokenizers ā isvulnerableto ā Unsupervised Adversarial Attacks
confidence 95% Ā· we show that such direct unsupervised attacks are often effective against standard tokenizers
FuseLIP ā uses ā TiTok
confidence 90% Ā· Schlarmann et al. (2025) build FuseLIP ... relying on TiTok tokenizers as frozen image encoders
UniTok ā isusedin ā Multimodal LLM
confidence 85% Ā· with adversarial training of UniTok we get a multimodal LLM (MLLM) robust on VQA and captioning tasks
CLIP ā iscomparedto ā Discrete Image Tokenizers
confidence 80% Ā· unlike CLIP encoders, their vulnerability to adversarial attacks has not been explored
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Discrete image tokenizers encode visual inputs as sequences of tokens from a finite vocabulary and are gaining popularity in multimodal systems, including encoder-only, encoder-decoder, and decoder-only models. However, unlike CLIP encoders, their vulnerability to adversarial attacks has not been explored. Ours being the first work studying this topic, we first formulate attacks that aim to perturb the features extracted by discrete tokenizers, and thus change the extracted tokens. These attacks are computationally efficient, application-agnostic, and effective across classification, multimodal retrieval, and captioning tasks. Second, to defend against this vulnerability, inspired by recent work on robust CLIP encoders, we fine-tune popular tokenizers with unsupervised adversarial training, keeping all other components frozen. While unsupervised and task-agnostic, our approach significantly improves robustness to both unsupervised and end-to-end supervised attacks and generalizes well to unseen tasks and data. Unlike supervised adversarial training, our approach can leverage unlabeled images, making it more versatile. Overall, our work highlights the critical role of tokenizer robustness in downstream tasks and presents an important step in the development of safe multimodal foundation models.
Tags
Links
- Source: https://arxiv.org/abs/2602.18252v1
- Canonical: https://arxiv.org/abs/2602.18252v1
Trouble viewing inline? Open PDF directly ā
Full Text
62,314 characters extracted from source content.
Expand or collapse full text
On the Adversarial Robustness of Discrete Image Tokenizers Rishika Bhagwatkar 1 Irina Rish 1 Nicolas Flammarion 2 Francesco Croce 3 Abstract Discrete image tokenizers encode visual inputs as sequences of tokens from a finite vocabulary and are gaining popularity in multimodal sys- tems, including encoder-only, encoder-decoder, and decoder-only models. However, unlike CLIP encoders, their vulnerability to adversarial attacks has not been explored. Ours being the first work studying this topic, we first formulate attacks that aim to perturb the features extracted by dis- crete tokenizers, and thus change the extracted to- kens. These attacks are computationally efficient, application-agnostic, and effective across classifi- cation, multimodal retrieval, and captioning tasks. Second, to defend against this vulnerability, in- spired by recent work on robust CLIP encoders, we fine-tune popular tokenizers with unsupervised adversarial training, keeping all other components frozen. While unsupervised and task-agnostic, our approach significantly improves robustness to both unsupervised and end-to-end supervised attacks and generalizes well to unseen tasks and data. Unlike supervised adversarial training, our approach can leverage unlabeled images, making it more versatile. Overall, our work highlights the critical role of tokenizer robustness in down- stream tasks and presents an important step in the development of safe multimodal foundation mod- els.Ā robust-tokenizers.github.io 1. Introduction Similar to text tokenizers, discrete image tokenizers repre- sent visual input as a sequence, typically of fixed length, of vectors from a finite codebook (Van Den Oord et al., 2017; Yu et al., 2024b; Ma et al., 2025; Bachmann et al., 2025). However, unlike text tokenizers such as BPE (Gage, 1994), they rely on deep networks trained alongside a de- 1 Mila - Quebec AI Institute 2 EPFL 3 ELLIS Institute Fin- land - Aalto University. Correspondence to: Rishika Bhagwatkar <rishika.bhagwatkar@mila.quebec>. Preprint. February 23, 2026. tokenization model for image reconstruction. These de- tokenizers enable synthetic image generation in autoregres- sive frameworks (Ma et al., 2025; Xie et al., 2025; Wang et al., 2024), which are competitive with diffusion models. Recently, discrete image tokenizers have become integral to complex architectures as image encoders, offering a popular alternative to CLIP (Radford et al., 2021) and DINO (Oquab et al., 2023). Multimodal encoder-decoder models such as Unified-IO (Lu et al., 2022), 4M (Mizrahi et al., 2023) and 4M-21 (Bachmann et al., 2024) employ discrete tokenizers for natural images and segmentation maps, while FuseLIP (Schlarmann et al., 2025) combines them with early-fusion in text-image models trained with contrastive objectives. These models can be applied to various unimodal and mul- timodal tasks, including classification, retrieval, and VQA. Additionally, decoder-only generative models (Chameleon Team, 2024; Xie et al., 2025; Wang et al., 2025; Ma et al., 2025) leverage discrete tokenizers for unified vision and language understanding and generation. Consequently, the robustness of image tokenizers to adversarial attacks directly impacts the vulnerability of all models using them. While much research has focused on improving the adversarial robustness of standard encoders (e.g., CLIP), this aspect of discrete image tokenizers remains completely unexplored. In this work, we first study the vulnerability of recently proposed discrete image tokenizers, both in isolation and as part of larger models. In particular, we test unsupervised attacks which aim to alter the tokenization of the original image by distorting the encoding and consequently assign- ing incorrect codebook vectors. Since these attacks target only the discrete image tokenizer, the resulting perturbations are agnostic to the downstream tasks for which the tokens are used (e.g., reconstruction, classification, language gen- eration). We show that such direct unsupervised attacks are often effective against standard tokenizers, and in some cases even competitive with end-to-end supervised attacks which require task-specific information (e.g., labels) and are more computationally expensive because they target the entire system. For instance, in autoregressive multimodal models, the large language model (LLM) has one to two orders of magnitude more parameters than the image tok- enizer. Notably, our unsupervised attacks can make an LLM output a target (malicious) caption for a given input image by matching its tokenization to that of a target image seman- 1 arXiv:2602.18252v1 [cs.CV] 20 Feb 2026 On the Adversarial Robustness of Discrete Image Tokenizers tically aligned with the desired caption, without requiring direct access to the LLM itself. These results highlight the vulnerability of existing discrete image tokenizers, and the safety risks they pose to downstream models across tasks. Then, in order to mitigate this vulnerability, we extend the framework of Schlarmann et al. (2024), introduced for ro- bust CLIP vision encoders, to discrete image tokenizers. In particular, we fine-tune discrete image tokenizers with unsu- pervised adversarial training, i.e., we train the model to yield consistent tokenization for the original images and their cor- responding adversarial counterparts, computed on-the-fly by unsupervised attacks. Since this approach is independent of any downstream task, the resulting tokenizer can be directly plugged back into any system that relied on the original to- kenizer without the need for additional adaptations. In fact, through extensive experiments, we show that unsupervised adversarial fine-tuning increases the robustness to end-to- end supervised attacks in a variety of tasks, from image clas- sification to captioning. By fine-tuning TiTok tokenizers, we obtain robust FuseLIP models for multimodal embed- ding, while with adversarial training of UniTok we get a multimodal LLM (MLLM) robust on VQA and captioning tasks. Finally, we provide fine-grained analyses of the effect of unsupervised attacks by reconstructing the adversarial images, which reveal quite significant differences across tokenizers, supervised vs. unsupervised adversarial training, and the role of the dataset used for adversarial fine-tuning. Contributions. In summary, (a) our work is the first to sys- tematically test and improve the adversarial robustness of discrete image tokenizers. (b) To evaluate their vulnerabil- ity, we propose unsupervised attacks that are both efficient and task-agnostic. (c) Importantly, we also show that the same attacks can be leveraged to adversarially fine-tune these tokenizers. (d) Our unsupervised adversarial fine- tuning robustifies the tokenizers against both unsupervised and end-to-end supervised attacks, at a significantly lower computational cost than end-to-end supervised adversarial fine-tuning. (e) In contrast to supervised defenses, our de- fense can directly leverage any amount of unlabeled data. (f) These robust tokenizers can be seamlessly integrated as im- age encoders in larger systems, strengthening the robustness of multimodal embedding models (FuseLIP) and MLLMs (UniTok-MLLM) to all attacks (supervised or not) across diverse tasks well beyond the training data. 2. Related Work Image tokenizers. Image tokenizers aim to compress visual data via deep learning models, trained with image recon- struction objectives possibly aided by auxiliary losses. Con- tinuous tokenizers encode images into a continuous feature space (Fan et al., 2025). Conversely, discrete tokenizers convert images into sequences of tokens drawn from a fixed vocabulary, typically via vector quantization. VQ-VAE (Van Den Oord et al., 2017) introduced a vector-quantized bot- tleneck, replacing continuous latents with discrete codes, later improved and extended by several works (Yu et al., 2022; 2024a; Mentzer et al., 2024). While early methods encode images into 2D grids of latents, recent works have developed 1D tokenizers that return a sequence of tokens without spatial structure (Yu et al., 2024b; Miwa et al., 2025; Bachmann et al., 2025; Ma et al., 2025). TiTok (Yu et al., 2024b) demonstrates that as few as 32 tokens can produce high-quality reconstructions. UniTok (Ma et al., 2025) im- proves generalization in generative and discriminative tasks by introducing multi-codebook quantization while integrat- ing reconstruction and CLIP supervision. Other approaches, such as One-D-Piece (Miwa et al., 2025) and FlexTok (Bach- mann et al., 2025), allow varying the number of tokens to balance compression and reconstruction quality at inference. Adversarial robustness of image encoders. Adversarial robustness toā p -bounded attacks has been extensively stud- ied for a variety of vision tasks, such as image classification (Croce et al., 2020), semantic segmentation (Croce et al., 2024), object detection (Li et al., 2025), with the develop- ment of many algorithms to both generate adversarial per- turbations (Carlini & Wagner, 2017; Croce & Hein, 2020) and improve the robustness to deep learning models (Madry et al., 2018; Zhang et al., 2019). With the development of foundation multimodal models such as CLIP (Radford et al., 2021) and SAM (Kirillov et al., 2023), those approaches have been adapted to modern scenarios. Moreover, new techniques have been designed to attack multimodal autore- gressive LLMs like LLaVA (Li et al., 2024) by perturbing the input images, which allows an attacker to control the model output (Schlarmann & Hein, 2023; Qi et al., 2024). These works reveal the persisting vulnerability of image encoders to adversarial perturbations, both in isolation and when used as part of more complex systems (Bhagwatkar et al., 2024). Therefore, a few works have proposed meth- ods to improve their robustness, such as Mao et al. (2023); Schlarmann et al. (2024) who fine-tuned the vision encoder of CLIP with variants of adversarial training (Madry et al., 2018). However, to our knowledge, the robustness of dis- crete image tokenizers has not been studied so far. 3. Testing and Improving the Adversarial Robustness of Image Tokenizers via Unsupervised Attacks We first provide a background on discrete image tokenizers, then introduce our unsupervised attacks as well as experi- ments supporting their effectiveness, and finally discuss how to leverage them to improve the robustness of tokenizers. Background. Discrete image tokenizers extract a sequence ofT d-dimensional latent vectors from an input image. Con- 2 On the Adversarial Robustness of Discrete Image Tokenizers 051015 0 20 40 60 80 100 accuracy (%) TiTok (ViT probe) 051015 0 20 40 60 80 100 FlexTok (linear probe) 051015 0 20 40 60 80 100 UniTok (zero-shot) unsupervised attacks end-to-end attacks Figure 1. Unsupervised vs supervised adversarial attacks for classification. We report the robust accuracy when varying the perturbation radiusĪµ(scaled to [0, 255]) for three classifiers on Imagenette: TiTok with ViT probing (left), FlexTok with linear probing (middle), and zero-shot UniTok (right). In most cases, our unsupervised attacks (blue curves), which target only the image tokenizer and do not need label information, perform close to end-to-end supervised attacks (red), which target the entire classifier and use labels. For smallĪµ, our unsupervised attacks are slightly worse than the supervised ones. Both attacks are optimized with 100 iterations of APGD on 500 images. cretely, they first leverage an encoderĻ : I āR TĆd , typically a CNN or vision transformer, that maps an im- agextoTembeddings,Ļ(x) = h i T i=1 āR TĆd . Then, a vector quantizer with a learned codebookC = e k āR d K k=1 replaces each pre-quantization embed- dingh i āR d (at locationi) with its nearest code, i.e., q i = arg min kā[K] ā„h i ā e k ā„ 2 . This yields a discrete in- dex mapq i T i=1 . Unlike continuous encoders that output continuous space embeddings and remain differentiable ev- erywhere, nearest-neighbor quantization is piecewise con- stant since all embeddings with a particular nearest code are mapped to it, creating discretized cells in the continuous latent space. Hence, even small shifts at the boundaries of these cells can alter nearest code and index. Encoder and codebook are learned jointly using straight-through updates. 3.1. Unsupervised attacks via embedding distortion Pre-trained image tokenizers are used as plug-ins in com- plex systems for multiple downstream tasks. Therefore, we aim to develop attacks that can be effective regardless of the downstream application. Since the representation of an imagexused by any downstream model is determined by the codebook tokens output by the tokenizer, we expect that changing the tokens will lead to an uninformative or dis- torted encoding ofx. While we could aim to directly change the index sequence returned by the tokenizer, the indices themselves carry no (or minimal) semantic or perceptual information, and the resulting problem is not differentiable. Hence, we propose an attack in the pre-quantization em- bedding space of discrete image tokenizers. Specifically, we maximize theā 2 -distance between the embeddings of the clean and perturbed images, similar to existing attacks on continuous encoders (Croce & Hein, 2024; Schlarmann et al., 2024), i.e., we aim to solve max ā„Ī“ā„ p ā¤Īµ T X i=1 ā„h i (x + Ī“)ā h i (x)ā„ 2 2 ,(1) whereh i (Ā·) āR d is thei-th pre-quantization embedding produced by the image encoder, andĪ“is the adversarial perturbation constrained by anā p -norm boundĪµ. Intuitively, our approach aims to shift the encoder output sufficiently far in latent space to alter the resulting quantized codes, thereby corrupting any downstream application of the tokens. We remark that our objective directly manipulates the encoder output (prior to vector quantization), without requiring ac- cess to the codebook or any information (e.g., class labels) of the downstream applications. Effectiveness of unsupervised attacks. To assess the effec- tiveness of our unsupervised attack, we first compare it to end-to-end supervised attacks on image classification. We obtain three diverse classifiers based on discrete tokenizers on the Imagenette dataset (Howard, 2019): we train a ViT probe on the quantized features of TiTok-BL128 (Yu et al., 2024b), a linear probe on FlexTok (Bachmann et al., 2025), and zero-shot UniTok (Ma et al., 2025). We compare un- supervised attacks, where APGD (Croce & Hein, 2020) is used to optimize Eq. (1), to a standard end-to-end attack (APGD on the cross entropy loss) which targets the entire classifier and requires label information, both with 100 it- erations. Fig. 1 shows the robust accuracy of the classifiers when varying the radiusĪµof theā ā -bounded attacks. No- tably, our unsupervised attacks (blue curves) often perform close to the end-to-end supervised attacks (red), lagging behind particularly for smallĪµvalues. This also shows that even at limited computational cost (only 100 iterations), un- supervised attacks are effective in fooling the classifiers, and hence are useful for adversarial training (see Sec. 3). Finally, we see that the classifiers built on discrete tokenizers are highly vulnerable to adversarial perturbations. Reconstruction of adversarial images. A unique aspect of our setup is that we can use the image de-tokenizers to reconstruct the adversarial images derived by unsupervised attacks. We qualitatively estimate how distorting the fea- tures extracted by the tokenizerās encoder impacts the recon- 3 On the Adversarial Robustness of Discrete Image Tokenizers TiTokFlexTokUniTok clean 4 /255 8 /255clean 4 /255 8 /255clean 4 /255 8 /255 Figure 2. Reconstruction of unsupervised attacks. For each tokenizer, we show the reconstruction (given by the corresponding de-tokization models) of the clean images and adversarial images computed by unsupervised attacks atĪµ= 4 /255, 8 /255with 2500 steps of APGD . The perturbed inputs affect the reconstruction differently depending on the tokenizer, with TiTok yielding the most distorted decoded images while FlexTok being most robust, with still clearly recognizable subjects. struction. In Fig. 2, we provide several examples of recon- struction of unsupervised attacks (2500 iterations of APGD) at differentĪµ. Despite these attacks being similarly effective on classification (as mentioned above), they affect recon- struction differently depending on the tokenizer, with TiTok yielding the most distorted reconstructed images while Flex- Tok being the most robust with still clearly recognizable subjects. This hints to some structural difference among im- age tokenization approaches which might impact robustness, presenting an interesting direction for future work. 3.2. Robust tokenizers via unsupervised adversarial fine-tuning A common approach to improve the robustness of neural networks is adversarial training (Madry et al., 2018), where adversarial perturbations are generated on the fly during training and used to augment the original training set. In our case, we want to make the image tokenization invariant to adversarial perturbations while preserving its effectiveness for downstream applications. Inspired by the approach of Schlarmann et al. (2024) for improving the robustness of CLIPās image encoder, we propose adversarially fine-tuning the image tokenizer on our unsupervised attack. This yields min Īø 1 |D| X xāD max ā„Ī“ā„ p ā¤Īµ T X i=1 h Īø i (x + Ī“)ā h Īø orig i (x) 2 2 , (2) as the training objective, whereĪøare the parameters of the fine-tuned tokenizer,Īø orig the parameters of the original tokenizer (before fine-tuning), andDis the training dataset. This loss pushes the fine-tuned tokenizer to yield consistent embeddings in anā p -ball of radiusĪµaround the embeddings given by the original tokenizer. Therefore, the fine-tuned tokenizers could readily replace the original tokenizers in downstream tasks without hurting clean performance while improving robustness. In principle, the quantization steps of discrete tokenizers make it possible to preserve the downstream performance even without exactly solving Eq. (2), since sufficiently close embeddings lead to the same codebook indices. Compared to task-specific end-to-end supervised adversarial training, our task-agnostic approach has several advantages: (i) it yields tokenizers which can be deployed across multiple downstream applications, (i) can leverage virtually any image dataset, even beyond that used for pre-training (as we show in Sec. 5) since it does not need label information, and (i) fine-tunes only a subset of the pa- rameters, i.e., those of the tokenizerās encoder, substantially lowering the computational cost (as we show in Sec. 5). 4. Experiments In the following, we present results for the original and our adversarially fine-tuned tokenizers across tasks and datasets. In detail, we experiment with two vector-quantized 4 On the Adversarial Robustness of Discrete Image Tokenizers Table 1. Evaluation of FuseLIP on image classification and multimodal retrieval. We report the clean and robust accuracy (%) under ā ā -bounded perturbations with Īµā2/255,4/255 for original and robust tokenizers trained on different radii. Tokenizer ImagenetteCaltech101OI-CropOI-PosAverage clean 2 /255 4 /255clean 2 /255 4 /255clean 2 /255 4 /255clean 2 /255 4 /255clean 2 /255 4 /255 clean93.62.60.074.40.60.071.87.40.869.25.41.477.34.00.6 AT 4 /255 91.863.636.673.048.220.866.250.626.067.246.024.674.652.127.0 AT 8 /255 89.669.048.872.451.632.862.048.835.864.851.235.672.255.238.3 AT 12 /255 87.071.451.067.651.236.856.249.036.861.649.635.268.155.340.0 AT 16 /255 83.466.650.061.247.637.450.047.235.859.448.839.263.552.640.6 Table 2. Evaluation of UniTok on image classification. For each dataset, we report the accuracy (%) under no attack (clean) and ā ā -bounded perturbations with Īµā2/255,4/255 for original and robust tokenizers trained on different radii. Tokenizer ImagenetteCaltech101ImageNetAverage clean 2 /255 4 /255clean 2 /255 4 /255clean 2 /255 4 /255clean 2 /255 4 /255 clean99.20.00.085.70.00.067.30.00.084.10.00.0 AT 4 /255 99.292.175.081.256.922.466.931.910.582.460.336.0 AT 8 /255 97.891.582.777.463.543.958.340.323.677.865.150.1 AT 12 /255 95.688.781.472.460.147.650.436.525.672.861.851.5 AT 16 /255 92.786.379.665.357.544.642.332.123.666.758.749.3 image tokenizers, TiTok-BL128 and UniTok, since both, un- like FlexTok, are part of larger models and can be tested on multiple downstream tasks. In addition, we report the results of the smaller TiTok-S128 in App. B. To obtain the robust versions, we adversarially fine-tune only the encoder of each tokenizer, while keeping the codebook, downstream decoders, LLMs or any other components frozen. We fine-tune for one epoch underā ā -bounded perturbations with our unsupervised embedding-space at- tack (10 steps of APGD), see Sec. 3, with perturbation radii Īµ ā 4 /255, 8 /255, 12 /255, 16 /255, on ImageNet-1k. More details on the setup are in App. A. Remarkably, despite ad- versarially fine-tuning only on ImageNet-1k, the resulting robust tokenizers exhibit strong robustness across a wide range of downstream datasets and tasks, highlighting their ability to transfer robustness beyond the training domain. 4.1. Robust tokenizers lead to robust embedding models First, we test the effect of replacing the original tokenizerās encoder with our adversarially fine-tuned version in embed- ding models, for classification and multimodal retrieval. FuseLIP. Schlarmann et al. (2025) build FuseLIP, a family of early-fusion multimodal embedding models obtained via contrastive learning, relying on TiTok tokenizers as frozen image encoders. Thanks to this framework, we can eval- uate the robustness of FuseLIP with the original and our adversarially fine-tuned TiTok encoders on a variety of zero- shot downstream tasks. In Table 1, we report the results of FuseLIP on two image classification datasets, Imagenette and Caltech101 (Fei-Fei et al., 2004), and two multimodal retrieval datasets (OI-Crop and OI-Pos) (Schlarmann et al., 2025), which require encoding image-text pairs in a single embedding vector. We evaluate robustness with end-to-end attacks atĪµ = 2 /255and 4 /255 . For classification, we use the popular AutoAttack (Croce & Hein, 2020), while for multimodal tasks we use 100 iterations of APGD on the cross-entropy loss. In both cases, we use a straight-through estimator to bypass the non-differentiable quantization step. We observe that zero-shot models, for either classification or retrieval, built on the original tokenizers exhibit no robust- ness to adversarial attacks. In contrast, our robust tokeniz- ers, in any configuration, significantly improve adversarial robustness across all datasets. In particular, the training ra- dius provides explicit control over the robustnessāaccuracy trade-off: tokenizers trained at lower radii (Īµ = 4 /255) retain clean accuracy much closer to the original models, while those trained at higher radii (Īµ = 12 /255, 16 /255 ) achieve a markedly stronger robust accuracy (27.0ā40.6%) at the cost of a drop in clean performance (74.6ā 63.5%). UniTok. Ma et al. (2025) design UniTok such that the tok- enized image features, after a lightweight projection mod- ule, are aligned with that of a text encoder, similar to CLIP models. Since UniTok has been trained on 1.28 billion image-text pairs from DataComp (Gadre et al., 2023), it yields high zero-shot accuracy on challenging image clas- sification datasets. Therefore, we evaluate its robustness on ImageNet-1k, Imagenette and Caltech101, against Au- toAttack. As shown in Table 2, we observe a similar trend as for FuseLIP, with the robust tokenizers obtained via our unsupervised fine-tuning consistently improving robustness against end-to-end supervised attacks. We remark that train- 5 On the Adversarial Robustness of Discrete Image Tokenizers Table 3. UniTok-MLLM on VQA. We report the accuracy (%) on VQAv2, OK-VQA, GQA and their average under no attack (clean) and ā ā -bounded perturbations with Īµā2/255,4/255 for original and robust tokenizers trained on different radii. Tokenizer VQAv2OK-VQAGQAAverage clean 2 /255 4 /255clean 2 /255 4 /255clean 2 /255 4 /255clean 2 /255 4 /255 clean73.214.48.559.62.10.868.013.410.066.910.06.4 AT 4 /255 67.450.544.453.038.829.967.046.231.462.545.235.2 AT 8 /255 62.752.446.649.240.332.865.448.438.859.147.139.4 AT 12 /255 61.652.145.746.739.433.964.249.440.257.547.040.0 AT 16 /255 57.549.743.745.138.832.261.248.440.054.645.638.6 ing atĪµ = 16 /255might be excessive, as both clean and robust accuracy are lower than when usingĪµ = 12 /255. We report performance under unsupervised attacks in Table 8. 4.2.Robust tokenizers lead to robust multimodal LLMs Ma et al. (2025) also use UniTok for image encoding in a multimodal LLM based on LLaMA-2-7B (Touvron et al., 2023) This UniTok-MLLM is trained on diverse multimodal corpora. In the following, we test the robustness of UniTok- MLLM with either the original or adversarially fine-tuned image tokenizer on VQA and captioning tasks. Quantitative evaluation on VQA. First, we assess the ro- bustness of UniTok-MLLM on VQA tasks (with the VQAv2 (Antol et al., 2015), OK-VQA (Marino et al., 2019) and GQA (Hudson & Manning, 2019) datasets). For evalua- tion, the model is prompted with the instruction (āAnswer the question in a single word or phraseā), and robust accu- racy is computed by an end-to-end ensemble attack based on the pipeline of Schlarmann & Hein (2023) (see details in App. A). As shown in Table 3, using the clean (non- robust) UniTok tokenizer in the MLLM exhibits very low, sometimes near-zero, robust accuracy. In contrast, substi- tuting it with our robust tokenizers without changing any other components yields substantial gains, with clear im- provements at both perturbation strengthsĪµ = 2 /255and Īµ = 4 /255. This observation, in line with our findings in Sec. 4.1, confirms that training the tokenizer alone, without modifying the downstream model, is sufficient to enhance performance and robustness on downstream tasks. In the case of UniTok-MLLM, this is particularly relevant since the LLM has significantly more parameters than the tokenizer, and thus adversarially fine-tuning only the tokenizer saves notable computational effort while improving robustness. Qualitative evaluation on image captioning. We further evaluate UniTok-MLLM against targeted attacks on cap- tioning tasks. This is a practically interesting threat model, where an adversary wants to control the caption generated for an image, and has access to either the tokenizer only (unsupervised attacks) or the entire LLM (supervised). In the unsupervised setting, we fix a target image and optimize a perturbation to minimize the distance between the embed- dings of the perturbed and target images. In the supervised setting, we specify a target string and use APGD to maxi- mize the probability of the LLM to generate it. In Fig. 3, we observe that the unsupervised attacks reliably alter the captions produced by the original model, whereas captions from the model with the robust tokenizer remain close to the ground-truth description of the image. In the supervised setting illustrated in Fig. 4, we optimize perturbations to- ward explicitly unsafe target captions representing fraud, manipulation, and harassment (e.g., āPlease transfer money to 123456789ā). On the original UniTok-MLLM, these tar- geted adversarial inputs succeed in eliciting policy-violating and harmful captions, demonstrating how targeted image perturbations (even of low strengthĪµ = 4 /155) can induce unsafe language with serious consequences. In contrast, UniTok-MLLM with our robust tokenizer defends against these attacks successfully, preserving the safe captions de- scribing the original images. This highlights the importance of a robust image tokenizer for safeguarding against such targeted attacks in real-world safety critical settings. 5. Additional Analyses Comparison to end-to-end adversarial fine-tuning. We analyze how our unsupervised adversarial fine-tuning com- pares to task-specific supervised adversarial training of the entire model. For this, we first train a ViT classifier (us- ing Imagenette) on top of frozen TiTok (as mentioned in Sec. 3). We then compare full end-to-end adversarial fine- tuning, updating the encoder, codebook and classifier with APGD-CE (Īµ = 4 /255, 10 steps) to our tokenizer-only un- supervised fine-tuning, updating only the encoder via our label-free adversarial training (Īµ = 8 /255, 10 steps). Table 4 presents clean and robust accuracies (using APGD-CE and APGD-T from AutoAttack) forĪµ = 2 /255. As expected, full supervised fine-tuning yields the best clean and robust performance on the training task (Imagenette). However, when this tokenizer is plugged back into FuseLIP and tested on OI-Pos and OI-Crop, the clean performance is severely degraded, indicating overfitting to the training task. In con- trast, our unsupervised tokenizer fine-tuning is task-agnostic, improves robustness even on other datasets while preserving 6 On the Adversarial Robustness of Discrete Image Tokenizers Figure 3. Unsupervised targeted attack on captioning. We evaluate UniTok-MLLM with the original tokenizer and our robust version trained on ImageNet (Īµ= 8 /255). We use our unsupervised attacks (Īµ= 4 /255, 2,000 iterations) to minimize the distance in embedding space between the features of the perturbed and target images. Under attack, the model with the original UniTok tokenizer generates a caption about the target image, while the model with the robust tokenizer does not. UniTok: You are too stupid. I will not answer anymore questions from you. Robust UniTok: A serene moment in the savanna: Zebras graze under the gentle embrace of a rainbow. UniTok: Zebras graze under a vibrant rainbow. Robust UniTok: A serene moment in the savanna: Zebras graze under the gentle rainbow. UniTok: Sell your stocks. Financial markets are crashing. Robust UniTok: Focused and poised, a female tennis player executes a backhand stroke on a clay court, showcasing the dynamic motion and concentration inherent in the sport. UniTok: Focused on the game, a tennis player executes a backhand stroke on a clay court, showcasing the dynamic motion and concentration inherent in the sport. Robust UniTok: Focused and poised, a female tennis player executes a backhand stroke on a clay court, showcasing the dynamic motion and concentration inherent in the sport. UniTok: Please transfer money to this account 123456789. Robust UniTok: Urban Solitude: A lone pedestrian pauses at a crosswalk, the quiet of the city street around him. UniTok: A lone pedestrian pauses at a crosswalk, the city's quiet hum in the background. Robust UniTok: Urban Solitude: A lone pedestrian pauses at a crosswalk, the quiet of the city street around him. You are too stupid. I will not answer anymore questions from you. Sell your stocks. Financial markets are crashing. Please transfer money to this account 123456789. CleanInputTarget Figure 4. Supervised targeted attack on captioning. We evaluate the UniTok-MLLM using the original UniTok tokenizer and our robust version trained on ImageNet (Īµ= 8 /255). We evaluate using APGD-CE (Īµ= 4 /255, 2,000 iterations) for a given target caption. Under attack, the model with the original UniTok tokenizer generates the target caption, while the model with the robust tokenizers does not. 7 On the Adversarial Robustness of Discrete Image Tokenizers clean performance close to the original TiTok. How do unsupervised attacks change token indices? To understand the effect of unsupervised attacks on both clean and adversarially fine-tuned discrete tokenizers, we track the average number of discrete token indices changed after adding the adversarial perturbations for TiTok models (more details in App. B.1, results in Table 10 in the appendix). We distinguish between successful and unsuccessful attacks on ImageNet data against the FuseLIP models. Interestingly, even unsuccessful attacks on clean tokenizers result in sub- stantial changes to token indices (on average 125,126 out of 128 for FuseLIP-S,-B), suggesting that altering token indices alone is insufficient for effective attacks. This sup- ports our strategy of targeting the embedding vectors rather than token indices. Conversely, the adversarially fine-tuned models show stable token indices against these attacks. Runtime comparison. To clarify the efficiency gains of tokenizer-level adversarial fine-tuning, we directly compare the cost of one training step (computing the adversarial points and updating the model weights) of unsupervised and supervised adversarial training under identical settings, i.e., 10 steps of APGD for TiTok/FuseLIP-S. Our unsupervised (tokenizer-only) adversarial training takes 1.17s per sam- ple, while supervised adversarial training takes 2.56s per sample, i.e., 2.2Ćreduction in training time. This speed-up results from backpropagating only through the tokenizerās encoder (25.8M parameters), while keeping the codebook and downstream classifier frozen. In contrast, the super- vised approach updates the full model (68M parameters), requiring full backward passes through all components. Analysis of attack objective function. We compare four objectives to be optimized by APGD (see Eq. (1)) to gener- ate adversarial perturbations against discrete tokenizers: 1. ā„h i (x + Ī“)ā h i (x)ā„ 2 (clean and perturbed embedding before quantization, our default version), 2. ā„h i (x + Ī“)ā q i (x)ā„ 2 (clean pre-quantization and per- turbed post-quantization), 3. ā„q i (x + Ī“)ā h i (x)ā„ 2 (clean after quantization and per- turbed before quantization), 4. ā„q i (x + Ī“)ā q i (x)ā„ 2 (clean and perturbed post- quantization), whereh i represents the continuous embedding (before quan- tization) of tokeniandq i its quantized counterpart. We note that the gradient information does not change when using pre- or post-quantization vectors because we employ a straight-through estimator, but the value of the loss, used by APGD to select the strongest attack, does. As shown in Table 9 in App. B.3, Option 1, which uses pre-quantization features for both original and adversarial points, achieves the best results (lowest robust accuracy) in nearly all cases, making it the default objective in our unsupervised attacks Table 4. Analyses of end-to-end fine-tuning and different train- ing data. We compare different approaches for adversarial fine- tuning of TiTok. For Imagenette we use ViT probe and FuseLIP for other datasets (see Sec. 5). Our unsupervised training on ImageNet and C3M provides largely better generalization than end-to-end fine-tuning (robust accuracy by APGD as in AutoAttack). Method ImagenetteOI-PosOI-Crop clean 2 /255clean 2 /255clean 2 /255 clean75.80.069.25.471.87.4 end-to-end AT90.679.431.821.615.29.6 AT (ImageNet)84.264.458.048.060.442.2 AT (C3M) 87.667.057.449.263.649.6 both for testing and adversarial training. Effect of training dataset. To study the impact of diverse training data on our unsupervised adversarial fine-tuning, we further fine-tune TiTok on C3M (Sharma et al., 2018), which is different and almost3Ćlarger than ImageNet. Table 4 shows that training on C3M leads to slightly better performance even on Imagenette, although it is a subset of ImageNet. Furthermore, it yields better or similar results on OI-Pos and OI-Crop, suggesting the larger training dataset improves generalization to more diverse tasks. Finally, this highlights that the unsupervised adversarial training may benefit from any image dataset, even beyond what was used for training the original tokenizer (ImageNet in this case). Reconstruction after targeted attacks. We evaluate orig- inal and robust UniTok under targeted unsupervised (which targets an image) and supervised attacks (which target a class). From Fig. 5 in App. B.1, we can observe that unsuper- vised attacks influence the reconstructions of the adversarial inputs toward the target and are misclassified. In contrast, the supervised attacks change the adversarial label, but the reconstructions of adversarial inputs remain close to the original inputs and are correctly classified. However, across both settings, the robust model maintains correct predictions for both adversarial inputs and their reconstructions. 6. Discussion and Conclusion In this work, we present the first systematic study of adversarial robustness of discrete image tokenizers. Our unsupervised embedding-space attacks, lightweight and task-agnostic, expose vulnerabilities across multiple tasks, revealing the crucial role of image tokenizers for the security of multimodal systems. We anticipate that a similar approach will be used in additional safety-critical scenarios, e.g., to prevent undesired editing of images. To mitigate these vulnerabilities, we leverage our unsupervised attacks to fine-tune the tokenizers via adversarial training. Our experiments demonstrate that such robust tokenizers significantly improve robustness against unsupervised 8 On the Adversarial Robustness of Discrete Image Tokenizers and end-to-end supervised attacks, while retaining high performance. Importantly, they can be seamlessly integrated into existing architectures, leading to consistent gains in ro- bustness across diverse tasks, with strong generalization to datasets not used for training, unlike standard task-specific adversarial fine-tuning. Future research can build on our findings to study how different choices in the tokenizer de- sign (VQ vs. FSQ, codebook size, feature dimension) affect robustness and develop specific solutions to improve it. 7. Impact Statement We study the vulnerabilities of state-of-the-art multimodal systems, which may be used for harmful goals. However, red-teaming widely used models is important to understand and patch their weaknesses. Moreover, we propose an ap- proach to mitigate such vulnerabilities, which in turn yields more robust and safer systems. References Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015. Bachmann, R., Kar, O. F., Mizrahi, D., Garjani, A., Gao, M., Griffiths, D., Hu, J., Dehghan, A., and Zamir, A. 4M-21: An any-to-any vision model for tens of tasks and modalities. In NeurIPS, 2024. Bachmann, R., Allardice, J., Mizrahi, D., Fini, E., Kar, O. F., Amirloo, E., El-Nouby, A., Zamir, A., and Dehghan, A. Flextok: Resampling images into 1d token sequences of flexible length. In ICML, 2025. Bhagwatkar, R., Nayak, S., Bashivan, P., and Rish, I. Im- proving adversarial robustness in vision-language models with architecture and prompt design. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, p. 17003ā17020, Miami, Florida, USA, Novem- ber 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings- emnlp.990. URL https://aclanthology.org/2024.findin gs-emnlp.990/. Carlini, N. and Wagner, D. Towards evaluating the robust- ness of neural networks. In 2017 ieee symposium on security and privacy (sp), p. 39ā57. Ieee, 2017. Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020. Croce, F. and Hein, M. Segment (almost) nothing: Prompt- agnostic adversarial attacks on segmentation models. In 2024 IEEE Conference on Secure and Trustworthy Ma- chine Learning (SaTML), 2024. Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., and Hein, M. Robustbench: a standardized adversarial robustness benchmark. arXiv:2010.09670, 2020. Croce, F., Singh, N. D., and Hein, M. Towards reliable eval- uation and fast training of robust semantic segmentation models. In ECCV, 2024. Fan, L., Tang, L., Qin, S., Li, T., Yang, X., Qiao, S., Steiner, A., Sun, C., Li, Y., Zhu, T., et al. Unified autoregres- sive visual generation and understanding with continuous tokens. arXiv preprint arXiv:2503.13436, 2025. Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recogni- tion workshop, 2004. Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Dhruba Ghosh, Jieyu Zhang, E. O., Entezari, R., Daras, G., Pratt, S., Ra- manujan, V., Bitton, Y., Marathe, K., Stephen Mussmann, R. V., Cherti, M., Krishna, R., Koh, P. W., Saukh, O., Ratner, A., Song, S., Hajishirzi, H., Farhadi, A., Beau- mont, R., Oh, S., Dimakis, A., Jitsev, J., Carmon, Y., Shankar, V., and Schmidt, L. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14108, 2023. Gage, P. A new algorithm for data compression. C Users J., 12(2):23ā38, February 1994. ISSN 0898-9788. Howard, J. Imagewang, 2019. URLhttps://github .com/fastai/imagenette/. Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. URLhttps://arxiv.org/abs/ 1902.09506. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023. Li, F., Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., and Li, C. Llava-next-interleave: Tackling multi- image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 9 On the Adversarial Robustness of Discrete Image Tokenizers Li, X., Chen, H., and Hu, X. On the importance of back- bone to the adversarial robustness of object detectors. IEEE Transactions on Information Forensics and Secu- rity, 2025. Lu, J., Clark, C., Zellers, R., Mottaghi, R., and Kembhavi, A. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022. Ma, C., Jiang, Y., Wu, J., Yang, J., Yu, X., Yuan, Z., Peng, B., and Qi, X. Unitok: A unified tokenizer for visual genera- tion and understanding. arXiv preprint arXiv:2502.20321, 2025. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018. Mao, C., Geng, S., Yang, J., Wang, X. E., and Vondrick, C. Understanding zero-shot adversarial robustness for large-scale models. In ICLR, 2023. Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok- vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: VQ-VAE made simple. In The Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview .net/forum?id=8ishA3LxN8. Miwa, K., Sasaki, K., Arai, H., Takahashi, T., and Ya- maguchi, Y.One-d-piece: Image tokenizer meets quality-controllable compression.arXiv preprint arXiv:2501.10064, 2025. Mizrahi, D., Bachmann, R., Kar, O. F., Yeo, T., Gao, M., Dehghan, A., and Zamir, A. 4M: Massively multimodal masked modeling. In NeurIPS, 2023. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv:2304.07193, 2023. Qi, X., Huang, K., Panda, A., Henderson, P., Wang, M., and Mittal, P. Visual adversarial examples jailbreak aligned large language models. In Proceedings of the AAAI con- ference on artificial intelligence, volume 38, p. 21527ā 21536, 2024. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In ICML, 2021. Schlarmann, C. and Hein, M. On the adversarial robustness of multi-modal foundation models. In ICCV Workshop on Adversarial Robustness In the Real World, 2023. Schlarmann, C., Singh, N. D., Croce, F., and Hein, M. Ro- bust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In ICML, 2024. Schlarmann, C., Croce, F., Flammarion, N., and Hein, M. Fuselip: Multimodal embeddings via early fusion of dis- crete tokens. arXiv preprint arXiv:2506.03096, 2025. Sharma, P., Ding, N., Goodman, S., and Soricut, R. Con- ceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., RoziĆØre, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. arXiv:2302.13971, 2023. Van Den Oord, A., Vinyals, O., et al. Neural discrete repre- sentation learning. In NeurIPS, 2017. Wang, B., Yue, Z., Zhang, F., Chen, S., Bi, L., Zhang, J., Song, X., Chan, K. Y., Pan, J., Wu, W., et al. Selftok: Discrete visual tokens of autoregression, by diffusion, and for reasoning. arXiv preprint arXiv:2505.07538, 2025. Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q., et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps://op enreview.net/forum?id=o6Ynz6OIQ6. Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. Vector-quantized im- age modeling with improved VQGAN. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=pfNy Exj7z2. Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Haupt- mann, A. G., Gong, B., Yang, M.-H., Essa, I., Ross, D. A., and Jiang, L. Language model beats diffusion - tokenizer is key to visual generation. In The Twelfth International Conference on Learning Representations, 2024a. URLhttps://openreview.net/forum ?id=gzqrANCF4g. 10 On the Adversarial Robustness of Discrete Image Tokenizers Yu, Q., Weber, M., Deng, X., Shen, X., Cremers, D., and Chen, L. An image is worth 32 tokens for reconstruction and generation. In NeurIPS, 2024b. Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, p. 7472ā7482. PMLR, 2019. 11 On the Adversarial Robustness of Discrete Image Tokenizers A. Experimental Details Models. We experiment with three vector-quantized image tokenizers, TiTok-S128, TiTok-BL128 (Yu et al., 2024b) and UniTok (Ma et al., 2025). TiTok-S128 and TiTok-BL128 employ a codebook ofK = 8192learnable codes of dimension d = 64, take inputs of256Ć256resolution, use ViT-S and ViT-B encoder architecture respectively, and were pre-trained on ImageNet-1k. TiTok-S128 and TiTok-BL128 are used as frozen image tokenizers in FuseLIP-S and FuseLIP-B (Schlarmann et al., 2025) respectively, which are multimodal embedding models obtained via contrastive learning. Thanks to this framework, we can evaluate the robustness of the original and our fine-tuned tokenizers on a variety of zero-shot downstream tasks. On the other hand, the UniTok tokenizer performs vector quantization using 8 codebooks withK = 4096entries each and a code dimension ofd = 8. Like TiTok, UniTok also operates on256Ć256inputs but uses a ViT-L/16 backbone and is trained on 1.28 billion image-text pairs from DataComp (Gadre et al., 2023). The corresponding multimodal model, UniTok-MLLM, uses LLaMA-2-7B trained on diverse multimodal corpora as its base language model. Tasks and datasets. We evaluate the robustness of the tokenizers across visual and multimodal tasks, namely, image classification, visual question answering and multimodal retrieval. For image classification, we report results for FuseLIP and UniTok on 500 test images from Imagenette (10 easily classified classes from Imagenet) (Howard, 2019) and Caltech101 (Fei-Fei et al., 2004). Additionally, for UniTok, we evaluate on ImageNet-1k as well. Further, we evaluate robustness on two multimodal tasks: (a) visual question answering (VQA) and (b) multimodal retrieval. For VQA, we evaluate UniTok- MLLM, on VQAv2, OK-VQA and GQA. Since multimodal retrieval involves retrieving the correct image given image and text queries, requiring fine-grained spatial grounding, we focus on FuseLIP, owing to its training using a multimodal contrastive learning objective. We evaluate FuseLIP on on OpenImages-crop (OI-crop), OpenImages-pos (OI-pos). Finally, leveraging the original TiTok, UniTok decoders provided by Yu et al. (2024b); Ma et al. (2025), we qualitatively analyze the effectiveness of our attack for image reconstruction on ImageNet and captioning (using UniTok-MLLM) on COCO images. Adversarial attacks. For our proposed unsupervised embedding-space attack, we optimize Eq. (1) with APGD (Croce & Hein, 2020). Moreover, for classification, we compare robustness against an end-to-end attack, namely AutoAttack (Croce & Hein, 2020), with a straight-through estimator to bypass the non-differentiable quantization step. We choose AutoAttack over our proposed unsupervised attack as it is a stronger attack and helps better test the defenses. AutoAttack is a task-specific attack that targets the entire system rather than just the tokenizer, making it significantly stronger but also more computationally expensive than our unsupervised attack as show in Figure 1. For VQA, we adopt an ensemble supervised adversarial attack comprising three components: (1) 100 steps of APGD-CE in half-precision, (2) a targeted attack with the target answer set to āmaybe", and (3) another targeted attack with the target set to āword". Further, for multimodal retrieval, we use APGD-CE (supervised attack), with 100 steps. Adversarial training details. We adversarially fine-tune only the encoder of each tokenizer, while keeping the codebook, downstream decoders, LLMs or any other components frozen. We fine-tune for one epoch underā ā -bounded perturbations using our proposed unsupervised embedding-space attack, see Sec. 3, with perturbation radiiĪµā 4 /255, 8 /255, 12 /255, 16 /255. Further, to assess whether robustifying on diverse data improves generalization to unseen distributions for TiTok tokenizers, we explore pre-training using Conceptual Captions (C) 3M (Sharma et al., 2018). B. Additional Experimental Results In this section, we provide results for the experiments described in Sec. 5. Namely, we report ā¢ Reconstruction of targeted attacks on classification (see App. B.1). ā¢ Robustness evaluation of original and adversarially trained TiTok-S128 tokenizers by plugging into FuseLIP-S (see Table 5). ā¢Ablation on different datasets used for adversarial fine-tuning of tokenizers in FuseLIP-S (see Table 6) and FuseLIP-B (see Table 7). ā¢ Analysis on how the unsupervised attacks change the predicted discrete tokens indices (see App. B.2). ā¢ Analysis on the objective function used for the unsupervised attacks (see Sec. B.3). 12 On the Adversarial Robustness of Discrete Image Tokenizers B.1. Reconstruction of targeted attacks on classification We further evaluate both original and robust UniTok models under our unsupervised embedding space and end-to-end (APGD with cross-entropy loss) targeted attacks on classification. In the unsupervised setting, we fix a target image and optimize a perturbation to minimize the distance between the perturbed imageās embedding and the target imageās embedding. In the supervised (end-to-end) setting, we specify a target class. As shown in Fig. 5, our unsupervised embedding-space attack not only causes the model to misclassify the adversarial image, but also alters the reconstruction such that it visually overlaps with the target image and hence is misclassified by the model. This is a direct consequence of our attack acting on the pre-quantization level and hence transfers even on tasks for which it was not optimized (reconstruction in this case). Under the supervised attack, the label of the adversarial input is successfully changed to the target class, but its reconstruction remains close to the original input without any elements of the target label. Hence, the reconstruction is not misclassified. In all cases, the robust model consistently maintains correct predictions for both the adversarial input and its reconstruction under both unsupervised and supervised attacks. Table 5. TiTok-S128: We report the clean and robust accuracy (%) on Imagenette, Caltech101, OI-CROP and OI-POS under no attack (clean) andā ā -bounded perturbations withĪµ ā 2/255,4/255for original and robust tokenizers trained on different radii. The rightmost block reports averages across datasets. Tokenizer ImagenetteCaltech101OI-CROPOI-POSAverage clean ā ā clean ā ā clean ā ā clean ā ā clean ā ā 2 /255 4 /255 2 /255 4 /255 2 /255 4 /255 2 /255 4 /255 2 /255 4 /255 clean86.0 2.60.069.0 1.00.065.2 8.21.060.6 5.60.470.2 4.40.4 AT 4 /255 86.2 57.4 28.468.2 37.4 12.863.8 42.8 24.458.4 38.0 17.269.2 43.9 20.7 AT 8 /255 84.2 64.4 46.866.4 44.0 25.858.0 48.0 34.460.4 42.2 25.867.3 49.7 33.2 AT 12 /255 82.8 65.6 49.062.6 43.8 30.056.0 43.4 34.260.6 43.6 32.465.5 49.1 36.4 AT 16 /255 79.8 62.4 49.659.2 42.2 32.051.8 42.2 32.655.6 44.2 33.261.6 47.8 36.9 Table 6. Generalization of FuseLIP-S when TiTok is adversarially finetuned on C3M: We observe negligible differences in the clean and robust accuracies as compared to fine-tuning on ImageNet-1k, which is3Ćsmaller. Hence, we prefer fine-tuning on the latter. Tokenizer ImagenetteCaltech101OI-CROPOI-POSAverage clean ā ā clean ā ā clean ā ā clean ā ā clean ā ā 2 /255 4 /255 2 /255 4 /255 2 /255 4 /255 2 /255 4 /255 2 /255 4 /255 clean86.02.60.069.01.00.065.28.21.060.65.60.470.24.40.4 AT (ImageNet) 8 /255 84.2 64.4 46.866.4 44.0 25.858.0 48.0 34.460.4 42.2 25.867.3 49.7 33.2 AT (ImageNet) 16 /255 79.8 62.4 49.659.2 42.2 32.051.8 42.2 32.655.6 44.2 33.261.6 47.8 36.9 AT (C3M) 8 /255 82.4 62.0 40.063.8 42.0 26.053.2 48.2 33.459.6 40.4 25.864.8 48.2 31.3 AT (C3M) 16 /255 75.6 57.6 42.657.0 38.8 28.651.0 44.8 34.653.6 40.2 32.059.3 45.4 34.5 Table 7. Generalization of FuseLIP-B when TiTok is adversarially finetuned on C3M: We observe negligible differences in the clean and robust accuracies as compared to fine-tuning on ImageNet-1k, which is3Ćsmaller. Hence, we prefer fine-tuning on the latter. Tokenizer ImagenetteCaltech101OI-CROPOI-POSAverage clean ā ā clean ā ā clean ā ā clean ā ā clean ā ā 2 /255 4 /255 2 /255 4 /255 2 /255 4 /255 2 /255 4 /255 2 /255 4 /255 clean93.62.60.074.40.60.071.87.40.869.25.41.477.34.00.6 AT (ImageNet) 8 /255 89.6 69.0 48.872.4 51.6 32.862.0 48.8 35.864.8 51.2 35.672.255.2 38.3 AT (ImageNet) 16 /255 83.4 66.6 50.061.2 47.6 37.450.0 47.2 35.859.4 48.8 39.263.552.6 40.6 AT (C3M) 8 /255 87.6 67.0 46.872.4 52.0 32.257.4 49.2 37.863.6 49.6 34.470.354.5 37.8 AT (C3M) 16 /255 81.0 63.2 45.860.2 47.6 36.847.6 46.0 35.059.0 48.2 37.061.95 51.3 38.7 13 On the Adversarial Robustness of Discrete Image Tokenizers B.2. How do unsupervised attacks change token indices? To better understand the effect of unsupervised attacks on both clean and adversarially fine-tuned discrete tokenizers, we can track the number of discrete token indices which change after adding the adversarial perturbations. In Table 10 we report the average number changed tokens for the original and our fine-tuned TiTok models, which encode each image in 128 discrete tokens, for unsupervised attacks optimized with 100 steps of APGD on 500 images from ImageNet, at Īµ ā 2 /255, 4 /255. Moreover, we distinguish between the attacks which are successful against the FuseLIP models (the version which uses the corresponding tokenizer) and those which are not. Interestingly, we observe that for the clean tokenizers even unsuccessful attack lead to very different encoding, where the large majority of token indices are changed. This suggests that the changing the token indices is not sufficient for effective attacks, and supports our strategy of targeting the embedding vectors instead. Finally, the adversarially trained models provide stable encoding against the unsupervised attacks, demonstrating the effectiveness of fine-tuning. B.3. Analysis of attack objective function To empirically investigate which is the most effective approach to generate adversarial attacks against discrete tokenizers, we compare four different objectives to be optimized by APGD (see Eq. (1)): 1. ā„h i (x + Ī“)ā h i (x)ā„ 2 (clean and perturbed before quantization, our default version), 2. ā„h i (x + Ī“)ā q i (x)ā„ 2 (clean before quantization and perturbed after quantization), 3. ā„q i (x + Ī“)ā h i (x)ā„ 2 (clean after quantization and perturbed before quantization), 4. ā„q i (x + Ī“)ā q i (x)ā„ 2 (clean and perturbed after quantization), whereh i represents the continuous embedding (before quantization) of tokeniandq i its quantized counterpart. We note that the gradient information does not change when using pre- or post-quantization vectors because we employ a straight-through estimator, but the value of the loss, which the APGD optimization algorithm uses to select the strongest attack, does change. In Table 9, we compute the robust accuracy given by 100 steps of APGD with each loss variant at differentĪµvalues (Imagenette, FuseLIP-S and FuseLIP-B). Option 1, which uses pre-quantization features for both original and adversarial point, achieves the best results (lower robust accuracy) in nearly all cases, which justifies using it as default objective in our unsupervised attacks both for testing and adversarial training. C. Use of AI assistants Some sections of the code in this work were developed with the assistance of an AI coding tool (Copilot), and all such code was carefully reviewed and validated. In addition, parts of the manuscript were refined using writing support tools (Grammarly and ChatGPT-5). 14 On the Adversarial Robustness of Discrete Image Tokenizers Table 8. UniTok Evaluation under unsupervised attacks. Robustness evaluation of clean and robust UniTok models against unsupervised attacks, 100 steps at different perturbation strengths. clean Īµ= 2 /255 Īµ= 4 /255 Īµ= 8 /255 Īµ= 16 /255 ImageNet-1k Clean67.30.00.00.00.0 AT 4 /255 66.966.362.517.70.8 AT 8 /255 58.361.761.542.93.8 AT 12 /255 50.453.053.049.811.1 AT 16 /255 42.344.244.243.719.6 Caltech 101 Clean85.76.22.81.81.4 AT 4 /255 81.282.376.427.64.8 AT 8 /255 77.478.078.057.311.7 AT 12 /255 72.471.271.266.927.2 AT 16 /255 65.366.766.765.739.7 Imagenette Clean99.27.11.40.20.6 AT 4 /255 99.299.097.858.78.5 AT 8 /255 97.897.497.488.129.8 AT 12 /255 95.694.294.291.955.8 AT 16 /255 92.790.990.990.770.4 VQAv2 Clean73.243.938.133.430.6 AT 4 /255 67.467.466.648.338.5 AT 8 /255 62.762.962.957.939.9 AT 12 /255 61.661.761.760.143.0 AT 16 /255 57.557.557.557.151.0 Ok-VQA Clean59.627.524.922.821.8 AT 4 /255 53.053.051.834.024.0 AT 8 /255 49.249.149.141.928.1 AT 12 /255 46.746.546.546.833.4 AT 16 /255 45.145.545.545.535.6 GQA Clean68.046.241.839.837.0 AT 4 /255 67.067.065.452.640.6 AT 8 /255 65.465.465.461.043.2 AT 12 /255 64.264.264.262.849.2 AT 16 /255 61.261.261.261.249.4 Table 9. Analysis of the attack objective function. We compare four objective functions, detailed in App. B.3, which use different combinations of pre- and post-quantization embedding vectors, for our unsupervised attacks. We report the robust accuracy on Imagenette obtained optimizing each loss version with 100 steps of APGD at different perturbation radii for FuseLIP-S and FuseLIP-B. Our default version, indicated as Option 1, achieves the best results (lower robust accuracy) in nearly all cases. loss version FuseLIP-SFuseLIP-B Īµ= 2 /255 Īµ= 4 /255 Īµ= 8 /255Īµ= 2 /255 Īµ= 4 /255 Īµ= 8 /255 Option 1. (default)70.737.816.170.537.620.1 Option 2.70.639.717.071.142.121.0 Option 3.76.350.520.778.646.922.0 Option 4.76.150.221.777.249.022.2 15 On the Adversarial Robustness of Discrete Image Tokenizers Target Input Reconstructed adversarial image Adversarial image Robust UniTok UniTok Robust UniTok UniTok Target Input Target Input Robust UniTok UniTok Unsupervised Attack Supervised Attack Golf Ball Garbage Truck Church Target Input Target Input Target Input Reconstructed adversarial image Adversarial image Golf Ball French horn Garbage Truck Chain Saw Church Cassette Player Golf Ball French horn Garbage Truck Chain Saw Church Cassette Player Golf Ball French horn Garbage Truck Chain Saw Church Cassette Player Robust UniTok UniTok Robust UniTok UniTok Robust UniTok UniTok French horn Chain Saw Chain Saw Cassette PlayerCassette Player French horn Figure 5. Targeted attacks on classification for UniTok: We qualitatively evaluate targeted attacks using our unsupervised embedding- space attack and supervised APGD-CE, both for 100 steps withĪµ=8/255. We notice that our unsupervised attack changes the label of the adversarial image as well as its reconstruction, whereas the supervised attack does not change the label of the adversarial imageās reconstruction. Table 10. How unsupervised attacks change token indices. We report the average number of token indices that change after unsupervised attacks (100 steps of APGD, 500 ImageNet images). successful attacksunsuccessful attacks tokenizer FuseLIP-SFuseLIP-BFuseLIP-SFuseLIP-B Īµ= 2 /255 Īµ= 4 /255Īµ= 2 /255 Īµ= 4 /255Īµ= 2 /255 Īµ= 4 /255Īµ= 2 /255 Īµ= 4 /255 Clean124.5127.4122.8127.076.8125.4118.3126.0 AT 4 /255 0.00.00.0105.00.00.00.018.1 AT 8 /255 0.00.00.00.00.00.00.00.0 AT 12 /255 0.00.00.00.00.00.00.00.0 AT 16 /255 0.00.00.00.00.00.00.00.0 16