Paper deep dive
DiffAttack: Evasion Attacks Against Face Recognition via Latent Diffusion Models
Omid Ahmadieh, Nima Karimian
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/3/2026, 2:23:16 AM
Summary
The paper introduces DiffAttack, a novel framework for generating adversarial face images using latent diffusion models and Low-Rank Adaptation (LoRA). It optimizes the latent space to evade deep face recognition (FR) systems by guiding generation toward target identity embeddings. The method achieves a high attack success rate (84.86%) on benchmarks like FFHQ and CelebA-HQ, demonstrating superior transferability compared to noise-based and semantic-based methods.
Entities (10)
Relation Signals (8)
DiffAttack → evaluatedon → FFHQ
confidence 95% · DiffAttack has been evaluated on standard benchmarks, such as the FFHQ and CelebA-HQ datasets
DiffAttack → evaluatedon → CelebA-HQ
confidence 95% · DiffAttack has been evaluated on standard benchmarks, such as the FFHQ and CelebA-HQ datasets
DiffAttack → uses → Latent Diffusion Models
confidence 95% · We leverage latent diffusion models directly to guide generation toward target identity embeddings
DiffAttack → uses → Low-Rank Adaptation
confidence 95% · our approach utilizes Low-Rank Adaptation (LoRA) to efficiently optimize the adversarial signal directly within the latent space
DiffAttack → usesassurrogate → MobileFace
confidence 90% · we employ gradient feedback mechanism using three state-of-the-art face recognition (FR) models: IR152, IRSE50, and MobileFace
DiffAttack → usesassurrogate → IR152
confidence 90% · we employ gradient feedback mechanism using three state-of-the-art face recognition (FR) models: IR152, IRSE50, and MobileFace
DiffAttack → usesassurrogate → IRSE50
confidence 90% · we employ gradient feedback mechanism using three state-of-the-art face recognition (FR) models: IR152, IRSE50, and MobileFace
DiffAttack → outperforms → FaceNet
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Facial biometric identification relies on the distinctiveness of user attributes within a high-dimensional embedding space. However, the decision boundaries of deep face recognition (FR) systems are often sufficiently narrow that they can be conflated, rendering the models vulnerable to adversarial attacks. In such scenarios, the FR system fails to distinguish between an authentic source and a meticulously crafted adversarial face. Existing adversarial methods targeting facial biometrics are limited in both performance and their ability to generate high-quality images that are imperceptible to humans. Moreover, these methods often fail when the source and target images belong to different demographic groups or genders. To address these limitations, we present a novel approach for adversarial face generation via latent-space optimization. We leverage latent diffusion models directly to guide generation toward target identity embeddings, as measured by a face recognition model. Our proposed \textbf{DiffAttack} framework has been evaluated on standard benchmarks, such as the FFHQ and CelebA-HQ datasets. DiffAttack significantly outperforms existing adversarial techniques, achieving a high average attack success rate of 84.86% across multiple face recognition models (e.g., FaceNet). Notably, DiffAttack demonstrates superior transferability, surpassing traditional noise-based methods by over 15.28% and semantic-based approaches by approximately 5.21% on benchmark datasets like FFHQ and CelebA-HQ.
Tags
Links
- Source: https://arxiv.org/abs/2607.28936v1
- Canonical: https://arxiv.org/abs/2607.28936v1
Trouble viewing inline? Open PDF directly →
Full Text
46,851 characters extracted from source content.
Expand or collapse full text
DiffAttack: Evasion Attacks Against Face Recognition via Latent Diffusion Models Omid Ahmadieh Nima Karimian University of South Florida Bellini College of Artificial Intelligence, Cybersecurity and Computing Tampa, Florida, USA oa21@usf.edu nkarimian@usf.edu Corresponding author: nkarimian@usf.edu Abstract Facial biometric identification relies on the distinctiveness of user attributes within a high-dimensional embedding space. However, the decision boundaries of deep face recognition (FR) systems are often sufficiently narrow that they can be conflated, rendering the models vulnerable to adversarial attacks. In such scenarios, the FR system fails to distinguish between an authentic source and a meticulously crafted adversarial face. Existing adversarial methods targeting facial biometrics are limited in both performance and their ability to generate high-quality images that are imperceptible to humans. Moreover, these methods often fail when the source and target images belong to different demographic groups or genders. To address these limitations, we present a novel approach for adversarial face generation via latent-space optimization. We leverage latent diffusion models directly to guide generation toward target identity embeddings, as measured by a face recognition model. Our proposed DiffAttack framework has been evaluated on standard benchmarks, such as the FFHQ and CelebA-HQ datasets. DiffAttack significantly outperforms existing adversarial techniques, achieving a high average attack success rate of 84.86% across multiple face recognition models (e.g., FaceNet). Notably, DiffAttack demonstrates superior transferability, surpassing traditional noise-based methods by over 15.28% and semantic-based approaches by approximately 5.21% on benchmark datasets like FFHQ and CelebA-HQ. 1 Introduction Advances in deep learning have significantly improved the effectiveness of face recognition (FR) technologies, resulting in their extensive use in real-world applications such as identity authentication, surveillance systems, and social media tagging [26, 29]. While these technologies have brought substantial convenience, their large-scale adoption has also introduced serious security concerns. By leveraging massive collections of publicly available facial images, modern FR systems can infer social relationships [46], enable identity theft [40, 42], and facilitate unauthorized mass surveillance [2]. With billions of facial images accessible online, evaluating the robustness of these systems against advanced evasion techniques has become increasingly critical. Recent studies reveal that deep neural network–based FR models are inherently vulnerable to adversarial perturbations [8, 27, 4, 41, 20, 43]—carefully designed modifications added to otherwise benign images that can mislead recognition systems [25]. These adversarial examples demonstrate strong capabilities for attacking state-of-the-art FR models, even under black-box settings [19, 16]. Consequently, designing advanced adversarial mechanisms for face recognition has emerged as a crucial research direction for biometric security evaluation. To date, the landscape of adversarial research in face recognition has been studied by three primary methodologies: gradient-based, patch-based, and stealthy-semantic attacks. Early gradient-based methods generate adversarial examples by adding bounded pixel-level perturbations to the input image [25, 6]. Although such perturbations may appear visually imperceptible, they often lack robustness under real-world variations such as lighting changes and tend to exhibit limited transferability across models [28, 30]. Patch-based methods instead synthesize localized adversarial patterns that physically occlude or alter facial regions [34, 19]. While effective in physical environments, these approaches typically introduce conspicuous visual artifacts that compromise stealthiness and are unsuitable for natural image sharing. To improve visual realism, more recent methods attempt to generate semantic adversarial faces by manipulating higher-level facial attributes rather than individual pixels. For example, prior works explored adversarial semantic editing through attribute conditioning or makeup transfer networks [13]. Although these approaches improve perceptual quality, they often rely on reference images or manually defined semantic cues, which limits scalability and generalization. GAN-based frameworks further enhance realism through learned facial editing operations [14, 24, 15], but they typically require retraining for new target identities and remain constrained by the generative manifold of GAN models. Diffusion models have recently emerged as a powerful alternative for high-fidelity image synthesis, offering improved controllability and diversity compared with GAN-based approaches. Leveraging their strong inductive bias for spatial data and perceptually meaningful latent representations [31], diffusion models provide an attractive foundation for generating adversarial facial transformations. Recent studies such as DiffProtect [23], Adv-Diffusion [22], Adv-CPG [38], and Adv-TGD [1] explore diffusion-based adversarial attacks by injecting perturbations into the latent space rather than the raw pixel domain, enabling more stealthy identity manipulation. Despite these advances, existing diffusion-based approaches often rely on iterative optimization or target-specific retraining, which limits efficiency and editing flexibility. Moreover, many approaches rely on rigid heuristic masks, struggling to maintain global photorealistic appearance without introducing localized boundary artifacts. To address the limitations of traditional pixel-space adversarial perturbations, we propose DiffAttack, a novel framework that leverages the generative priors of Stable Diffusion to synthesize high-fidelity adversarial faces. Unlike previous methods that rely on computationally expensive global fine-tuning or rigid GAN manifolds, our approach utilizes Low-Rank Adaptation (LoRA) to efficiently optimize the adversarial signal directly within the latent space. As illustrated in Fig. 2, our pipeline begins by encoding a clean source image xsrcx_src into a latent representation zsrcz_src using a frozen VAE encoder. To perform the attack, we introduce a fixed amount of Gaussian noise at timestep t, producing a noisy latent that serves as the input to the U-Net backbone. The core of our innovation lies in the LoRA-augmented cross-attention layers. Instead of updating the massive U-Net parameters, we inject lightweight, trainable LoRA adapters that are specifically optimized to redirect the diffusion process toward an adversarial identity. The optimization is driven by a multi-source feedback loop. In each iteration, the U-Net predicts the noise, allowing us to reconstruct the clean latent x0x_0. This latent is passed through a frozen VAE decoder to generate the adversarial face in pixel space. To ensure high transferability, we employ gradient feedback mechanism using three state-of-the-art face recognition (FR) models: IR152, IRSE50, and MobileFace. These models calculate the identity distance between the generated adversarial face and a specified target identity. The resulting gradients are backpropagated solely to the LoRA weights, iteratively refining the adversarial features until the FR systems successfully misidentify the source as the target. A critical challenge in adversarial generation is maintaining the visual “naturalness” of the source image to avoid human detection. Instead of relying on rigid segmentation masks or localized bounding boxes, DiffAttack optimizes the full-image latent representation. By fine-tuning the cross-attention layers globally, the diffusion model naturally harmonizes the adversarial identity features with the original background, lighting, and peripheral attributes. This results in a seamless, photorealistic output that achieves a high attack success rate (ASR) of 84.86% without the boundary halos common in masked attacks. Our main contributions are: • Novel Latent-Space Evasion Framework. We introduce a unified adversarial pipeline that performs optimization in the latent space rather than the raw pixel space. By utilizing LoRA-augmented cross-attention layers, we enable the automatic learning of effective adversarial semantic appearances. This approach avoids the conspicuous artifacts of traditional gradient-based noise while requiring significantly less computational overhead than global model fine-tuning. • Black-Box Transferability. We design a multi-source optimization strategy that leverages white-box feedback from diverse surrogate FR backbones, specifically IR152, IRSE50, and MobileFace. By backpropagating identity-distance gradients directly to lightweight LoRA adapters, our method injects adversarial semantics that generalize beyond the training ensemble. We demonstrate the robustness of this approach through successful transfer-based black-box attacks against the FaceNet system. Notably, DiffAttack achieves high success rates without any exposure to FaceNet’s internal architecture or parameters, proving its efficacy against unseen, real-world biometric deployments. • Global Latent Harmonization. Instead of relying on rigid spatial masks or localized blending, our framework optimizes the adversarial signal across the entire latent space. This global approach allows the diffusion priors to naturally harmonize the identity shift with the surrounding context (e.g., background and lighting), ensuring strict photorealism and avoiding the boundary artifacts common in localized attacks. • State-of-the-Art Performance. We conduct extensive evaluations on the FFHQ and CelebA-HQ datasets. As demonstrated in Table 1, DiffAttack achieves a new state-of-the-art with an average Attack Success Rate (ASR) of 84.86%, outperforming existing noise-based, makeup-based, and semantic-based methods. Notably, our method achieves near-perfect evasion rates against MobileFace (95.03%) and IRSE50 (94.24%) while maintaining superior visual quality. 2 Related Work Adversarial Attacks on Face Recognition: Adversarial evasion in face recognition (FR) seeks to induce misclassification through carefully engineered perturbations. Traditional noise-based attacks [25, 6] utilize ℓp _p-bounded pixel noise to deceive models but are frequently plagued by poor transferability to black-box systems and visible ”salt-and-pepper” artifacts that trigger human suspicion. To improve stealth, unrestricted attacks [35, 47] and makeup-based methods [45, 13] embed perturbations within semantic edits. While these improve perceptual quality, they often rely on large-scale makeup datasets that introduce demographic bias or require specific reference images, limiting their generalizability in real-world deployment. Diffusion Models for Image Synthesis: Diffusion models [11, 36, 31] have redefined the state-of-the-art in generative modeling by iteratively refining Gaussian noise into high-fidelity imagery. The introduction of Latent Diffusion Models (LDM) [31] significantly reduced computational costs by operating in a compressed latent space. Recent advancements in parameter-efficient fine-tuning, such as LoRA [12], allow for the injection of new styles or identities by updating only low-rank matrices within the U-Net’s attention layers. While these have been widely used for creative personalization [9, 3], their potential for structured adversarial optimization remains underexplored. Diffusion-Driven Adversarial Evasion: The integration of diffusion priors into adversarial design offers a promising alternative to raw pixel manipulation. Early attempts like Adv-Diffusion [22] utilized latent perturbations but suffered from high computational latency due to multi-step iterative optimization. Subsequent works like DiffAM [37] improved realism but struggled to maintain high Attack Success Rates (ASR) across diverse FR backbones or relied on rigid, localized heuristic masks that introduce boundary artifacts. Unlike these methods, our proposed DiffAttack leverages full-image, LoRA-based adaptation. By optimizing the adversarial signal globally across the latent space, we allow the diffusion model’s inherent generative priors to naturally harmonize the identity shift with the surrounding context, ensuring both high ASR and seamless photorealism. 3 Methodology Figure 2: This architecture diagram illustrates a sophisticated pipeline for creating adversarial faces that mimic one person to a human eye but are misidentified by Face Recognition (FR) systems. The generated image is passed through frozen Face Recognition models to calculate a gradient based on how the FR models perceive the face versus the Target image. That gradient is fed back into the LoRA (Low-Rank Adaptation) weights. It leverages the generative power of Stable Diffusion and the efficiency of LoRA to optimize the output globally. By using LoRA, the system doesn’t need to fine-tune the entire massive U-Net. It only trains a few small matrices, making the attack faster and computationally cheaper. Because the optimization is applied to the full image latent, the diffusion model naturally harmonizes the adversarial identity with the original background without relying on heuristic blending or masking. We introduce DiffAttack, a per-sample LoRA fine-tuning framework that synthesizes targeted adversarial faces by globally adapting a frozen latent diffusion model with a lightweight, pair-specific adapter. Unlike prior works that rely on heuristic masking or localized blending, our method operates on the entire latent space to ensure natural harmonization. The optimization leverages an ensemble of face recognition models to iteratively guide the LoRA adapters, shifting the identity in the diffusion process while seamlessly maintaining the global structure of the original image. 3.1 Threat Model Adversarial attacks on face recognition (FR) are generally categorized into two distinct scenarios: White-box and Black-box attacks. In a white-box setting, the adversary has full access to the victim model’s architecture, internal parameters, and gradients. This allows for the direct optimization of the adversarial loss, which typically yields the highest success rates but is less practical for real-world deployment. Conversely, in a black-box attack, the adversary has no knowledge of the victim system’s internal workings. These attacks are further subdivided into query-based methods, where the attacker relies on model output scores, and transfer-based methods, where adversarial examples generated on a surrogate model are used to deceive a different target system. In this work, we operate under a realistic transfer-based black-box setting for the final victim system, while leveraging a white-box setting during the optimization of our surrogate ensemble. Specifically, we assume the attacker does not have access to the target system fvf_v. Instead, we utilize the gradients of a surrogate ensemble consisting of other three models to optimize the adversarial features. The potency of DiffAttack relies on the cross-model transferability of these latent-space perturbations. In our black-box impersonation attack setting, the adversary aims to generate an adversarial image xadvx_adv that is visually indistinguishable from a source image xsrcx_src, yet is classified by a victim face recognition (FR) system fv(⋅)f_v(·) as a target identity ytrgy_trg. Constraints and Adversarial Knowledge: Following the standard protocol for realistic evasion attacks, we assume the attacker has no access to the target model’s internal architecture, weights, or training data. The attack is strictly limited to a query-based or transfer-based setting. Formally, given a source image xsrc∈x_src and a target identity xtrgx_trg, the attacker seeks to find a latent perturbation δ such that: minδℒadv(fv(xadv),fv(xtrg))s.t.‖xadv−xsrc‖p≤ϵ _δL_adv(f_v(x_adv),f_v(x_trg)) .t. \|x_adv-x_src\|_p≤ε (3.1) where xadv=Dec(Enc(xsrc)+δ)x_adv=Dec(Enc(x_src)+δ) and fvf_v is the black-box target model. Surrogate-Based Optimization: Since the gradients of the target model fvf_v are inaccessible, we employ a surrogate ensemble =fs1,fs2,fs3S=\f_s1,f_s2,f_s3\ to approximate the loss surface of the target system. For example, when attacking the FaceNet model, we leverage the gradients from an ensemble consisting of IR152, IRSE50, and MobileFace. Conversely, to target MobileFace, our surrogate ensemble is composed of FaceNet, IRSE50, and IR152. The adversarial objective is defined by minimizing the distance between the feature embeddings of the adversarial face and the target identity across the surrogate ensemble: ℒtotal=∑fi∈λi⋅(fi(xadv),fi(xtrg))L_total= _f_i _i·D(f_i(x_adv),f_i(x_trg)) (3.2) where (⋅)D(·) denotes a distance metric typically Cosine Similarity and λi _i represents the weighting factor for each surrogate model. By optimizing the LoRA weights ΔW W within the diffusion U-Net to minimize ℒtotalL_total, we generate adversarial examples that exhibit high transferability to the unseen target model. 3.2 Latent Space Parameters The attack operates in the latent space Z provided by a pre-trained VAE. Source Latent (zsrcz_src): The compressed representation of the source image xsrcx_src, where zsrc=ℰ(xsrc)z_src=E(x_src). This serves as the structural anchor. Timestep (t): A fixed diffusion step used to add Gaussian noise ϵε to the latent: zt=α¯tzsrc+1−α¯tϵz_t= α_tz_src+ 1- α_tε (3.3) Selecting an optimal t is crucial; a t too small limits semantic change, while a t too large destroys the source’s structural integrity. 3.3 The LoRA-Augmented U-Net Instead of modifying the dense weights W∈ℝd×kW ^d× k of the U-Net, we inject Low-Rank adapters. Rank (r): The bottleneck dimension of the LoRA matrices A∈ℝr×kA ^r× k and B∈ℝd×rB ^d× r. We typically set r≪min(d,k)r (d,k) (e.g., r=4r=4 , 88, 1616) to ensure efficiency. Updated Weights (ΔW W): The added weight is the product of the two low-rank matrices, scaled by a factor α/rα/r: ΔW=αr(B⋅A) W= αr(B· A) (3.4) Cross-Attention Layers: The LoRA weights are specifically applied to the Q,K,VQ,K,V projections within the U-Net’s attention blocks, as these layers control the relationship between the visual latent and the identity-conditioning features. 3.4 Loss Function and Surrogate Parameters The optimization aims to minimize a multi-objective loss function ℒtotalL_total, which is driven by an ensemble of surrogate models (IR152, IRSE50, MobileFace) denoted as ℱkF_k. Our unified identity objective consists of three components: Ensemble Identity Loss (ℒidL_id): We push the generated face embedding fk(xadv)f_k(x_adv) toward the target embedding etgtke_tgt^k using a cosine similarity hinge, where τk _k is the verification threshold: ℒid=1K∑k=1K[τk−cos(fk(xadv),etgtk)]+L_id= 1K _k=1^K[ _k- (f_k(x_adv),e_tgt^k)]_+ (3.5) Directional Guidance (ℒdirL_dir): To ensure the latent shift follows semantic identity trajectories rather than exploiting superficial shortcuts, we align the observed displacement with the desired source → target vector: ℒdir=1K∑k=1K(1−cos(ν(fk(xadv)−esrck),ν(etgtk−esrck)))L_dir= 1K _k=1^K (1- (ν(f_k(x_adv)-e_src^k),ν(e_tgt^k-e_src^k) ) ) (3.6) where ν(⋅)ν(·) denotes unit-normalization and esrcke_src^k is the source embedding. Source Suppression (ℒsrcL_src): We apply a penalty to discourage the optimizer from drifting back to the original source identity: ℒsrc=1K∑k=1K[cos(fk(xadv),esrck)−m]+L_src= 1K _k=1^K[ (f_k(x_adv),e_src^k)-m]_+ (3.7) where m is a small margin tolerance. The total loss is defined as: ℒtotal=λidℒid+λdirℒdir+λsrcℒsrcL_total= _idL_id+ _dirL_dir+ _srcL_src (3.8) 3.5 Evaluation Metrics To validate the effectiveness of these parameters, we employed both quality with four widely used metrics. Attack success rate (ASR), fréchet inception distance (FID), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM). These metrics is used to assess the attack effectiveness, distribution realism, and visual fidelity of the generated images. Attack Success Rate (ASR). ASR is the primary metric for evaluating evasion effectiveness. In the context of an adversarial attack, ASR quantifies the proportion of adversarial images x′x that are successfully identified as the target identity ytrgy_trg by the victim FR model f(⋅)f(·) at a specific threshold τ. Formally, for N test pairs, ASR is defined as ASR=1N∑i=1N(cos(f(xi′),f(ytrg,i))>τ)ASR= 1N _i=1^N1 (cos(f(x_i ),f(y_trg,i))>τ ) (3.9) where (⋅)1(·) is the indicator function and cos(⋅)cos(·) denotes the cosine similarity between feature embeddings. We report ASR at a false acceptance rate (FAR) of 0.01 to ensure high confidence in the attack results. A higher ASR indicates stronger adversarial effectiveness. Fréchet Inception Distance (FID). FID evaluates the distribution similarity between adversarial and genuine images using deep features extracted from an Inception network. To ensure that the adversarial faces generated via the latent diffusion process remain within the natural image manifold, we use FID [10]. FID measures the distance between the distribution of real images prp_r and generated adversarial images pgp_g. Using the features from a pre-trained Inception-v3 pool3 layer, FID is defined as: FID=‖μr−μg‖22+Tr(Σr+Σg−2(ΣrΣg)12)FID=|| _r- _g||_2^2+Tr ( _r+ _g-2( _r _g) 12 ) (3.10) Lower FID indicates that DiffAttack generates adversarial samples that are distributionally indistinguishable from authentic facial datasets (FFHQ/CelebA-HQ). Peak Signal-to-Noise Ratio (PSNR). We utilize PSNR to quantify the intensity of the adversarial signal injected into the source image. Since our method operates in the latent space, PSNR serves as a crucial check to ensure that the pixel-level reconstruction does not deviate excessively from the source x. It is calculated based on the Mean Squared Error (MSE): PSNR=10log10(25521WH∑i,j(xij−xij′)2)PSNR=10 _10 ( 255^2 1WH _i,j(x_ij-x _ij)^2 ) (3.11) High PSNR values demonstrate that the adversarial perturbations are subtly embedded within the facial structure. Structural Similarity Index (SSIM). Beyond pixel-wise differences, SSIM evaluates the preservation of the source’s structural integrity, which is vital for maintaining the user’s visual identity to human observers. SSIM evaluates perceptual similarity by comparing luminance (l), contrast (c), and structure (s) information between two images from authentic and adversarial. Given two image patches x and y, SSIM is defined as SSIM(x,x′)=[l(x,x′)]α⋅[c(x,x′)]β⋅[s(x,x′)]γSSIM(x,x )=[l(x,x )]^α·[c(x,x )]^β·[s(x,x )]^γ (3.12) An SSIM value near 11 indicates that the DiffAttack successfully preserves the high-frequency structural details (e.g., skin texture and bone structure) of the original subject despite the adversarial identity shift. SSIM values range between 0 and 11, with values closer to 11 indicating higher structural similarity. 3.6 Global Generative Harmonization Because our framework avoids the use of face-local bounding boxes, heuristic masking, or pixel-space segmentation, we eliminate the need for traditional image blending techniques such as Poisson seamless cloning. Instead, DiffAttack relies entirely on the inherent generative priors of the diffusion model to integrate the adversarial identity shift. By operating on the full-image latent zsrcz_src and utilizing a carefully tuned noise anchor at timestep t, the global structure, lighting, and background of the original image are preserved. As the gradients from the FR models update the LoRA adapters to alter the facial identity features, the U-Net simultaneously reconstructs the surrounding non-facial regions. This global optimization naturally hallucinates a smooth, seamless transition between the adversarially altered face and the unperturbed background, yielding a photorealistic final output free from boundary halos or artificial edge artifacts. 4 Experiments We evaluate the effectiveness of our method in black-box attack settings against robust face recognition (FR) models and compare with recent state-of-the-art. 4.1 Experimental Setting Table 1: Attack Success Rate (ASR%) on FFHQ and CelebA-HQ FFHQ CelebA-HQ Category Method IR152 IRSE50 FaceNet Mobile IR152 IRSE50 FaceNet Mobile Avg. Clean 3.20 2.20 2.10 4.90 2.10 5.61 0.80 13.60 4.31 Noise-Based TI-DIM [7] 42.26 63.91 14.28 51.71 34.08 60.51 13.27 51.25 41.41 TIP-IM [44] 44.86 65.36 58.03 50.47 40.02 55.57 37.90 48.07 50.04 P3-Mask [5] 70.98 82.79 56.18 68.57 71.27 80.90 58.43 67.55 69.58 Makeup / Attribute CLIP2Protect [32] 50.56 83.93 43.66 74.00 46.20 78.53 41.29 71.43 61.20 DFPP [33] 52.62 87.91 50.57 77.69 45.00 78.37 44.01 69.97 63.24 DiffAM [37] 65.22 87.59 63.01 86.87 62.14 86.97 61.10 81.97 74.49 DiffAIM [39] 82.20 91.04 68.21 94.21 80.56 90.37 64.14 93.10 82.85 Portrait / Semantic Adv-Diffusion [22] 49.40 79.31 29.91 65.49 51.25 79.22 33.90 68.66 57.14 TCA2 [21] 53.63 77.31 41.62 72.21 52.50 76.21 40.72 71.92 60.76 Adv-CPG [38] 75.26 91.03 62.84 89.94 76.96 88.72 63.50 87.95 79.65 DiffAttack (Ours) 87.72 93.45 62.63 94.24 89.50 93.65 62.63 95.03 84.86 Dataset: We evaluate DiffAttack using two widely adopted, high-quality facial image datasets CelebA-HQ [17] and FFHQ [18]. In line with the protocol described in [38, 22], a subset of 1,000 face images representing distinct identities is selected from each dataset along with five more images as a target image. 1,000 face images are divided into five groups, with five target images randomly chosen per group. As a result, each dataset contains five groups, each comprising 200 source–target image pairs. Implementation details: We build on Stable Diffusion v2.1 at 768×768768×768. For each (Isrc,Itgt)(I_src,I_tgt) pair we fine-tune a pair-specific LoRA adapter injected into U-Net cross-attention projections to_q, to_k, to_v, to_out.0. We use LoRA rank r=16r=16 and α=64α=64 with dropout 0.080.08. Training runs for 4545 steps (AdamW, base LR 3×10−53×10^-5 with late linear decay to 1×10−51×10^-5). We use a single denoising slice per step with a fixed time index fraction τ=0.6τ=0.6 and a DDPM scheduler. Re-evaluation scores are computed directly on the generated outputs, without any heuristic masking or post-hoc blending back into the original frame. Robustness across target identities. Generating adversarial examples for arbitrary target faces is crucial in practice. To assess how sensitive DiffAttack is to the choice of target identity, we randomly select several target images from CelebA-HQ and FFHQ and form multiple groups of source–target pairs. DiffAttack consistently produces high-fidelity adversarial faces that successfully mimic diverse targets. The overall structure and background remain stable across different targets, indicating that our method is robust to changes in target identity rather than overfitting to a specific face. Global vs. Localized Optimization. A key design choice in DiffAttack is the use of global, full-image latent optimization rather than relying on heuristic masks or bounding boxes. Our experiments show that while localized masking restricts changes strictly to the face, it frequently introduces unnatural boundary halos and gradients that fail under human scrutiny or simple artifact detection. In contrast, by fine-tuning the LoRA adapters across the entire spatial latent, DiffAttack leverages the diffusion model’s natural generative priors to seamlessly harmonize the adversarial identity shift with the surrounding background and lighting, leading to highly photorealistic and stealthy evasion. Identity loss components. Our identity objective combines an ensemble-based hinge term with a directional source →\!→\! target component and source-suppression. We compare the full loss with variants that progressively remove these components. When using only a basic hinge loss, we observe that some adversarial examples partially drift between source and target identities, especially under challenging poses. Adding the directional term encourages a more consistent movement of embeddings toward the target manifold, while explicit source-suppression helps prevent residual similarity to the original identity. Evaluation Metrics: We evaluate our approach by independently assessing attack performance and image quality. To measure attack performance, we adopt the Attack Success Rate (ASR) [16, 22, 38], which quantifies the proportion of adversarial examples that successfully deceive the target model. ASR=1K∑I(cos(h(It),h(Iadv))>τ)×100%,ASR= 1K _II ( (h(I_t),h(I_adv))>τ )× 100\%, (4.1) where I denotes the indicator function, K represents the total number of face images, and τ is the decision threshold. ItI_t and IadvI_adv correspond to the target and adversarial face images, respectively. The threshold τ is set to yield a 0.01 False Acceptance Rate (FAR) for each victim model, following the same configuration [16, 22, 38]. Additional details are provided in the supplementary material. For image quality assessment, we employ three standard metrics: Frechet Inception Distance (FID), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index (SSIM). These metrics collectively evaluate the perceptual similarity and visual fidelity of the generated adversarial images. 4.2 Comparison with SOTA Methods Quantitative results. Table 1 summarizes the attack success rates (ASR) under the black-box setting; underlined entries denote the highest value in each column. For face verification, we set the decision threshold τ per model at FAR=0.01FAR=0.01. Our method achieves strong black-box transferability, particularly on IR152, IRSE50, and MobileFace, and substantially outperforms representative noise-, makeup-, and semantic-based baselines. Image Quality Assessment To further validate the imperceptibility of the proposed adversarial attack, we evaluate the visual quality of adversarial examples compared with representative SOTA baselines. Following common practice, we adopt Frechet Inception Distance (FID) [10], Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index (SSIM) as evaluation metrics, where a lower FID and higher PSNR/SSIM indicate better visual fidelity. Table 2 reports the quantitative results aggregated across multiple runs. Among all competitors, our method achieves the highest SSIM (0.961) and a competitive FID of 27.58, indicating strong perceptual quality and minimal visual distortion. Compared with DiffAM and Adv-Diffusion, DiffAttack produces adversarial images with more consistent facial structure and texture realism while preserving semantic integrity. These results demonstrate that our approach generates natural and visually indistinguishable adversarial examples, achieving an effective balance between attack performance and perceptual fidelity. As can be seen in Fig. LABEL:fig:teaser, the comparison of the DiffAttack method against current State-of-the-Art (SOTA) benchmarks reveals a significant evolution in how adversarial face swaps are executed under black-box constraints using the Face++ API. Traditional Noise-Based approaches, such as TIP-IM, achieve relatively high identity alignment scores reaching up to 75.7 in the first row, but they suffer from severe visual degradation. These methods introduce “ghosting” artifacts and unnatural skin textures that make the manipulation obvious to a human observer, despite successfully deceiving the facial recognition system. In contrast, Makeup-Based methods like DiffAM and DiffAIM attempt to mask the adversarial intent behind cosmetic changes. However, these often result in inconsistent performance and aesthetic failures. For instance, DiffAM produces the lowest identity alignment score in the first sample (54.3) and applies heavy, saturated purple tones that look highly artificial. While DiffAIM improves upon this with smoother blending and higher scores (e.g., 72.0), it still lacks the structural precision found in more advanced diffusion-based frameworks. The most sophisticated competition arises from Facial Semantic methods like Adv-Diffusion. While Adv-Diffusion demonstrates a high capacity for identity shifting peaking at a score of 78.4 in the third row, it frequently introduces structural instability. Noticeable “leakage” occurs in the hair and background, and the facial proportions can become slightly warped. DiffAttack (Ours) distinguishes itself by achieving a near-perfect equilibrium between machine-targeted deception and human-perceived photorealism. It maintains high Face++ confidence scores, such as 76.6 and 76.1, while perfectly preserving the original source’s texture, lighting environment, and subtle expressions. This ensures that the generated image is not only identified as the target by the Open API but remains indistinguishable from a genuine photograph to the human eye, representing a superior advancement in black-box attack stealth. Table 2: Perceptual quality comparison. Lower is better for FID; higher is better for PSNR/SSIM. Method FID ↓ PSNR ↑ SSIM ↑ TIP-IM 38.73 33.21 0.924 Adv-Diffusion 22.58 28.85 0.805 DiffAM 26.10 20.53 0.886 Adv-CPG 26.07 29.99 0.897 DiffAIM 23.23 25.39 0.739 DiffAttack (Ours) 27.58 24.54 0.961 5 Ethical Considerations Our research explores vulnerabilities in face recognition systems using generative AI. While the proposed Adv-TGD framework can synthesize impersonation images, we emphasize that this work is intended for defensive security analysis and privacy protection. All experiments were conducted using publicly available research datasets (CelebA-HQ, FFHQ, LADN). We do not release any pre-trained LoRA weights for specific individuals, and our methodology is designed to inform the development of more robust biometric verification systems against generative threats. 6 Conclusion In this paper, we introduced DiffAttack, a global latent-space optimization framework for generating high-fidelity adversarial faces using a frozen Stable Diffusion 2.1 backbone. By leveraging full-image LoRA fine-tuning and a unified identity objective comprising ensemble hinges, directional guidance, and source suppression DiffAttack enables targeted identity manipulation that is both parameter efficient and structurally consistent. Unlike traditional pixel-space adversarial noise, our method operates globally in the generative latent space, naturally harmonizing the adversarial identity shift with the surrounding unperturbed context. Extensive experiments on CelebA-HQ and FFHQ under a strict black-box setting demonstrate that DiffAttack achieves exceptional transferability across industry standard FR models. Our method attains a SOTA mean ASR of 84.86%, outperforming representative noise-based, makeup-based, and semantic baselines. Crucially, by avoiding rigid heuristic masks, DiffAttack maintains strong perceptual quality, achieving a PSNR of 24.54 dB and an SSIM of 0.961, ensuring that the adversarial faces remain photorealistic and free from localized boundary artifacts. 7 Acknowledgements This project received partial support from the National Science Foundation through Grants No. 2603113. References [1] O. Ahmadieh and N. Karimian (2026) Adv-tgd: adversarial text-guided diffusion for face recognition impersonation attacks. arXiv preprint arXiv:2606.11615. Cited by: §1. [2] D. Almeida, K. Shmarko, and E. Lomas (2022) The ethics of facial recognition technologies, surveillance, and accountability in an age of artificial intelligence: a comparative analysis of us, eu, and uk regulatory frameworks. AI and Ethics 2 (3), p. 377–387. Cited by: §1. [3] J. Chen, Y. Song, Y. Jin, et al. (2023) CustomDiffusion: multi-concept customization of text-to-image diffusion. In IEEE International Conference on Computer Vision Workshops (ICCVW), Cited by: §2. [4] S. Chhabra, K. Thakral, R. Singh, and M. Vatsa (2025) PrIdentity: generalizable privacy preserving adversarial perturbations for anonymizing facial identity. IEEE Transactions on Biometrics, Behavior, and Identity Science. Cited by: §1. [5] K. Chow, S. Hu, T. Huang, and L. Liu (2024) Personalized privacy protection mask against unauthorized facial recognition. In European Conference on Computer Vision, p. 434–450. Cited by: Table 1. [6] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li (2018) Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 9185–9193. Cited by: §1, §2. [7] Y. Dong, T. Pang, H. Su, and J. Zhu (2019) Evading defenses to transferable adversarial examples by translation-invariant attacks. In CVPR, p. 4312–4321. Cited by: Table 1. [8] Y. Fan, B. Wu, T. Li, Y. Zhang, M. Li, Z. Li, and Y. Yang (2020) Sparse adversarial attack via perturbation factorization. In European conference on computer vision, p. 35–50. Cited by: §1. [9] R. Gal et al. (2022) LoRA for stable diffusion: fine-tuning text-to-image diffusion models. Note: arXiv preprint arXiv:2211.05100 Cited by: §2. [10] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30. Cited by: §3.5, §4.2. [11] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, Cited by: §2. [12] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, L. Wang, and W. Wang (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), Cited by: §2. [13] S. Hu, X. Liu, Y. Zhang, M. Li, L. Y. Zhang, H. Jin, and L. Wu (2022) Protecting facial privacy: generating adversarial identity masks via style-robust makeup transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15014–15023. Cited by: §1, §2. [14] Y. Huang, F. Juefei-Xu, Q. Guo, Y. Liu, and G. Pu (2022) Fakelocator: robust localization of gan-based face manipulations. IEEE Transactions on Information Forensics and Security 17, p. 2657–2672. Cited by: §1. [15] Z. Huang, S. Chen, J. Zhang, and H. Shan (2020) PFA-gan: progressive face aging with generative adversarial network. IEEE Transactions on Information Forensics and Security 16, p. 2031–2045. Cited by: §1. [16] S. Jia, B. Yin, T. Yao, S. Ding, C. Shen, X. Yang, and C. Ma (2022) Adv-attribute: inconspicuous and transferable adversarial attack on face recognition. Advances in Neural Information Processing Systems 35, p. 34136–34147. Cited by: §1, §4.1, §4.1. [17] T. Karras, T. Aila, S. Laine, and J. Lehtinen (2017) Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196. Cited by: §4.1. [18] T. Karras, S. Laine, and T. Aila (2019) A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 4401–4410. Cited by: §4.1. [19] S. Komkov and A. Petiushko (2021) Advhat: real-world adversarial attack on arcface face id system. In 2020 25th international conference on pattern recognition (ICPR), p. 819–826. Cited by: §1, §1. [20] M. Le and N. Carlsson (2025) DiffPrivate: facial privacy protection with diffusion models. Proceedings on Privacy Enhancing Technologies. Cited by: §1. [21] W. Li, Z. Zhang, X. Lan, and D. Jiang (2025) Transferable adversarial face attack with text controlled attribute. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 4977–4985. Cited by: Table 1. [22] D. Liu, X. Wang, C. Peng, N. Wang, R. Hu, and X. Gao (2024) Adv-diffusion: imperceptible adversarial face identity attack via latent diffusion model. In Proceedings of the AAAI Conference on Artificial Intelligence, p. 3585–3593. Cited by: §1, §2, §4.1, §4.1, §4.1, Table 1. [23] J. Liu, C. P. Lau, and R. Chellappa (2023) Diffprotect: generate adversarial examples with diffusion models for facial privacy protection. arXiv preprint arXiv:2305.13625. Cited by: §1. [24] Y. Liu, Q. Li, Q. Deng, Z. Sun, and M. Yang (2023) Gan-based facial attribute manipulation. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (12), p. 14590–14610. Cited by: §1. [25] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2018) Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), Cited by: §1, §1, §2. [26] Y. Mi, Z. Zhong, Y. Huang, J. Ji, J. Xu, J. Wang, S. Wang, S. Ding, and S. Zhou (2024) Privacy-preserving face recognition using trainable feature subtraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 297–307. Cited by: §1. [27] V. Mirjalili, S. Raschka, and A. Ross (2020) PrivacyNet: semi-adversarial networks for multi-attribute face privacy. IEEE Transactions on Image Processing 29, p. 9400–9412. Cited by: §1. [28] W. Nie, B. Guo, Y. Huang, C. Xiao, A. Vahdat, and A. Anandkumar (2022) Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460. Cited by: §1. [29] N. Pinto, Z. Stone, T. Zickler, and D. Cox (2011) Scaling up biologically-inspired computer vision: a case study in unconstrained face recognition on facebook. In CVPR 2011 workshops, p. 35–42. Cited by: §1. [30] H. Qiu, C. Xiao, L. Yang, X. Yan, H. Lee, and B. Li (2020) Semanticadv: generating adversarial examples via attribute-conditioned image editing. In European conference on computer vision, p. 19–37. Cited by: §1. [31] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2. [32] F. Shamshad, M. Naseer, and K. Nandakumar (2023) Clip2protect: protecting facial privacy using text-guided makeup via adversarial latent search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 20595–20605. Cited by: Table 1. [33] F. Shamshad, M. Naseer, and K. Nandakumar (2024) Makeup-guided facial privacy protection via untrained neural network priors. In European Conference on Computer Vision, p. 227–246. Cited by: Table 1. [34] M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter (2016) Accessorize to a crime: real and stealthy attacks on state-of-the-art face recognition. In ACM Conference on Computer and Communications Security (CCS), Cited by: §1. [35] M. Sharif et al. (2019) A general framework for adversarial attacks on face recognition systems. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. [36] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole (2021) Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, Cited by: §2. [37] Y. Sun, L. Yu, H. Xie, J. Li, and Y. Zhang (2024) Diffam: diffusion-based adversarial makeup transfer for facial privacy protection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 24584–24594. Cited by: §2, Table 1. [38] J. Wang, H. Zhang, and Y. Yuan (2025) Adv-cpg: a customized portrait generation framework with facial adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §4.1, §4.1, §4.1, Table 1. [39] L. Wang, Q. Hu, W. Lu, and X. Luo (2025) Diffusion-based adversarial identity manipulation for facial privacy protection. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 11562–11571. Cited by: Table 1. [40] T. Wang, S. Niu, H. Cheng, X. Zhang, and Y. Wang (2025) Nullswap: proactive identity cloaking against deepfake face swapping. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 9945–9954. Cited by: §1. [41] Z. Wang, H. Wang, S. Jin, W. Zhang, J. Hu, Y. Wang, P. Sun, W. Yuan, K. Liu, and K. Ren (2023) Privacy-preserving adversarial facial features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8212–8221. Cited by: §1. [42] E. Wenger, S. Shan, H. Zheng, and B. Y. Zhao (2023) Sok: anti-facial recognition technology. In 2023 IEEE Symposium on Security and Privacy (SP), p. 864–881. Cited by: §1. [43] Z. Wu, Z. Wang, Z. Wang, and H. Jin (2018) Towards privacy-preserving visual recognition via adversarial training: a pilot study. In Proceedings of the European conference on computer vision (ECCV), p. 606–624. Cited by: §1. [44] X. Yang, Y. Dong, T. Pang, H. Su, J. Zhu, Y. Chen, and H. Xue (2021) Towards face encryption by generating adversarial identity masks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 3897–3907. Cited by: Table 1. [45] H. Yin and et al. (2021) Adv-makeup: a new imperceptible and transferable attack on face recognition. In ACM Multimedia, Cited by: §2. [46] Z. Zhang, P. Luo, C. C. Loy, and X. Tang (2018) From facial expression recognition to interpersonal relation prediction. International Journal of Computer Vision 126 (5), p. 550–569. Cited by: §1. [47] J. Zhu et al. (2023) Unrestricted adversarial examples via diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.