Paper deep dive
Probabilistic Adversarial Training
Andi Zhang, Xingyu Zhao, Siddartha Khastgir
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Building on a probabilistic perspective in which adversarial examples arise from the overlap between a distance-based distribution $p_{\mathrm{dis}}$ and a victim-classifier-induced distribution $p_{\mathrm{vic}}$, we start from a simple intuition: adversarial examples become harder to generate when these two distributions are pushed apart, as their overlap becomes smaller, thereby increasing robustness. This intuition naturally motivates a KL-based robustness objective. We then prove that $\mathrm{KL}(p_{\mathrm{dis}}\|p_{\mathrm{vic}})-\log Z_{\mathrm{vic}}$ is a lower bound on probabilistic robustness (PR), where $Z_{\mathrm{vic}}$ denotes the normalizing constant of $p_{\mathrm{vic}}$. Since PR is generally intractable to compute directly, maximizing this KL-based lower bound provides a tractable surrogate objective for improving PR. We further show that this objective recovers a scaled form of adversarial training, offering a probabilistic interpretation of adversarial training and a principled route to robustness improvement. We call the resulting method probabilistic adversarial training. Experiments show that it consistently improves PR, and ablation studies demonstrate that the induced scaling factor can even enhance the PR of non-probabilistic adversarial training methods.
Tags
Links
- Source: https://arxiv.org/abs/2609.39798v1
- Canonical: https://arxiv.org/abs/2609.39798v1
Trouble viewing inline? Open PDF directly →
Full Text
96,174 characters extracted from source content.
Expand or collapse full text
Probabilistic Adversarial Training Andi Zhang †thanks: Corresponding Author. Affiliation: WMG, University of Warwick Email: andi.zhang@warwick.ac.uk Xingyu Zhao Affiliation: WMG, University of Warwick Affiliation: Wuhan University Siddartha Khastgir Affiliation: WMG, University of Warwick Abstract Building on a probabilistic perspective in which adversarial examples arise from the overlap between a distance-based distribution pdisp_dis and a victim-classifier-induced distribution pvicp_vic, we start from a simple intuition: adversarial examples become harder to generate when these two distributions are pushed apart, as their overlap becomes smaller, thereby increasing robustness. This intuition naturally motivates a KL-based robustness objective. We then prove that KL(pdis∥pvic)−logZvicKL(p_dis\|p_vic)- Z_vic is a lower bound on probabilistic robustness (PR), where ZvicZ_vic denotes the normalizing constant of pvicp_vic. Since PR is generally intractable to compute directly, maximizing this KL-based lower bound provides a tractable surrogate objective for improving PR. We further show that this objective recovers a scaled form of adversarial training, offering a probabilistic interpretation of adversarial training and a principled route to robustness improvement. We call the resulting method probabilistic adversarial training. Experiments show that it consistently improves PR, and ablation studies demonstrate that the induced scaling factor can even enhance the PR of non-probabilistic adversarial training methods. 1 Introduction Adversarial robustness refers to a model’s ability to maintain correct predictions when its inputs are subjected to small perturbations. This property is important in the era of deep learning, as a substantial body of work (Goodfellow et al., 2015; Kurakin et al., 2017; Madry et al., 2018; Moosavi-Dezfooli et al., 2016; Moosavi-Dezfooli et al., 2017; Papernot et al., 2016; Szegedy et al., 2014) has shown that deep neural networks can be highly vulnerable to adversarial perturbations. To better understand and evaluate this vulnerability, Zhang et al. (2024a) proposed probabilistic adversarial attack. By applying Langevin Dynamics, they demonstrated that adversarial examples can be viewed as samples drawn from an adversarial distribution. This distribution is essentially the product of a “victim” distribution pvicp_vic that encourages misclassification, and a “distance” distribution pdisp_dis that constrains the perturbation magnitude. However, their framework is strictly confined to targeted attacks and focuses exclusively on the generation of adversarial examples, leaving a systematic approach to model defense unexplored. On the other side, to formally quantify model security, Webb et al. (2018) introduced the concept of probabilistic robustness (PR), defined as the probability that a classifier successfully resists perturbations sampled from a specific distribution. In this work, we unify the probabilistic attack perspective (Zhang et al., 2024a) with the probabilistic robustness framework (Webb et al., 2018). We theoretically prove that KL(pdis∥pvic)−logZvicKL(p_dis\|p_vic)- Z_vic is a lower bound on probabilistic robustness, where ZvicZ_vic denotes the normalizing constant of pvicp_vic. Since computing and optimizing PR directly is generally intractable, maximizing this KL-based lower bound provides a tractable surrogate objective to effectively enhance PR. This theoretical result aligns with a simple yet profound intuition. In essence, adversarial examples naturally arise from the overlap between the distance distribution pdisp_dis (which confines the perturbation to be close to the original image) and the victim distribution pvicp_vic (which actively drives the model toward misclassification). By maximizing the KL divergence KL(pdis∥pvic)KL(p_dis\|p_vic), we are pushing these two distributions apart. As their overlap shrinks, it becomes inherently more difficult for an attacker to sample an adversarial example that simultaneously satisfies the distance constraint and triggers a misclassification. We further show that maximizing this lower bound essentially recovers a scaled form of adversarial training. This provides a probabilistic interpretation of adversarial training itself, and provides a principled route to robustness improvement, which we term Probabilistic Adversarial Training (PAT). Empirically, PAT consistently improves probabilistic robustness. Moreover, our ablation studies reveal an intriguing byproduct: the theoretical scaling factor derived from our framework can plug into and enhance traditional, non-probabilistic defense methods, such as PGD-based adversarial training. To summarize, our main contributions are as follows: • We formalize the untargeted version of probabilistic adversarial attacks. (Section 3) • We prove that KL(pdis∥pvic)−logZvicKL(p_dis\|p_vic)- Z_vic is a lower bound on probabilistic robustness (PR). (Section 4) • By maximizing this KL-based lower bound, we propose Probabilistic Adversarial Training (PAT), which is theoretically guaranteed to improve PR. Moreover, its importance sampling formulation makes the estimation of the bound tractable (Section 5). • We empirically demonstrate that PAT consistently improves PR, and its derived scaling factor can even enhance non-probabilistic methods like PGD-based adversarial training. (Section 6) 2 Preliminaries 2.1 Adversarial Attack Let C:[0,1]d→C:[0,1]^d be a classifier with input dimension d and label space Y. Let xori∈[0,1]dx_ori∈[0,1]^d be a clean (original) image with its corresponding true label yori∈y_ori . Adversarial attacks generally fall into two categories. The goal of an untargeted adversarial attack is to construct an adversarial example xadvx_adv that simply causes the classifier to misclassify the image, meaning C(xadv)≠yoriC(x_adv)≠ y_ori. Alternatively, given a specific target label ytar∈y_tar (where ytar≠yoriy_tar≠ y_ori), a targeted adversarial attack aims to force the classifier to predict that exact label, such that C(xadv)=ytarC(x_adv)=y_tar. In both scenarios, the distance between xadvx_adv and xorix_ori must remain small. The corresponding optimization problem can be formulated as: min(xori,xadv)subject toC(xadv)∈andxadv∈[0,1]d, (x_ori,x_adv) to C(x_adv) x_adv∈[0,1]^d, where =ytarT=\y_tar\ for a targeted attack and =∖yoriT=Y \y_ori\ for an untargeted attack, and D measures the distance (similarity) between xorix_ori and xadvx_adv, typically via an L1L_1, L2L_2, or L∞L_∞ norm. Directly solving this constrained optimization can be challenging. To address this, Szegedy et al. (2014) propose relaxing it into the following optimization problem: minxadv∈[0,1]dc1(xori,xadv)+c2L(xadv) _x_adv∈[0,1]^d\;c_1\,D(x_ori,x_adv)\;+\;c_2\,L(x_adv) (1) where c1>0c_1>0 and c2>0c_2>0 are trade-off constants. For a targeted attack, we set L(xadv)=f(xadv,ytar)L(x_adv)=f(x_adv,y_tar) to minimize the loss with respect to the target label; for an untargeted attack, we set L(xadv)=−f(xadv,yori)L(x_adv)=-f(x_adv,y_ori) to maximize the loss with respect to the true label, thereby pushing the prediction away from the original class. In Szegedy et al. (2014)’s work, f is taken to be the cross-entropy loss11 1 When we say that f(x,y)f(x,y) is the cross-entropy loss, we mean the cross-entropy between the Dirac categorical distribution concentrated at label y and the categorical distribution output by the neural network given input x.; Carlini & Wagner (2017) present additional choices for f. 2.2 Probabilistic Adversarial Attack Zhang et al. (2024a) derive a probabilistic perspective on adversarial attacks by applying Langevin Dynamics as an optimizer for equation 1. Notably, their framework focuses exclusively on targeted attack. Under this specific setting (where L(xadv)=f(xadv,ytar)L(x_adv)=f(x_adv,y_tar)), they introduce the adversarial distribution: padv(xadv,xori,ytar)∝pvic(xadv,ytar)pdis(xadv,xori),p_adv(x_adv;x_ori,y_tar) p_vic(x_adv;y_tar)\,p_dis(x_adv;x_ori), where pvic(xadv,ytar)∝exp(−c2f(xadv,ytar))p_vic(x_adv;y_tar) (-c_2\,f(x_adv,y_tar) ) is the “victim” distribution emphasizing misclassification toward the target label ytary_tar, and pdis(xadv,xori)∝exp(−c1(xori,xadv))p_dis(x_adv;x_ori) (-c_1\,D(x_ori,x_adv) ) is the “distance” distribution around xorix_ori. This formulation leverages the fact that Langevin Dynamics converges to the corresponding Gibbs distribution (Lamperski, 2021), thereby providing a probabilistic interpretation of adversarial examples. This probabilistic perspective aligns with traditional geometry-based adversarial attacks. For example, if D is the L1L_1 norm, then pdis(xadv,xori)∝exp(−∥xadv−xori∥1)p_dis(x_adv;x_ori) (- x_adv-x_ori _1) takes the form of a Laplace distribution; if D is the squared L2L_2 norm, then pdis(xadv,xori)∝exp(−∥xadv−xori∥22)p_dis(x_adv;x_ori) (- x_adv-x_ori _2^2) is a Gaussian distribution; if D is L∞L_∞ norm, then pdisp_dis is an L∞L_∞-spherical distribution (Iglesias et al., 1998). 2.3 Probabilistic Robustness Webb et al. (2018) introduced a statistical framework to assess neural network robustness by evaluating the probability of an attack’s success. Let p denote the probability density function of perturbations around a clean input xorix_ori. To quantify misclassification, let z(x)∈ℝ||z(x) ^|Y| denote the pre-softmax logits output by the classifier. For an untargeted attack, the margin function s(⋅)s(·) is formally defined as the logit margin violation: s(x)=maxy≠yori[zy(x)−zyori(x)].s(x)= _y≠ y_ori [z_y(x)-z_y_ori(x) ]. The event s(X)≥0s(X)≥ 0 strictly indicates that at least one incorrect class logit matches or exceeds the true class logit, signifying a successful adversarial attack (i.e., the classifier is fooled). Consequently, the attack success probability, or probabilistic vulnerability, is defined as: ℐ[p,s]=ℙX∼p(s(X)≥0)=∫s(x)≥0p(x)dx,I[p,s]=P_X p(s(X)≥ 0)= _\s(x)≥ 0\p(x)dx, where I is the indicator function. While directly minimizing ℐ[p,s]I[p,s] reduces vulnerability, framing robustness as an objective to be maximized provides a more consistent narrative for our subsequent theoretical derivations. Therefore, throughout this paper, we formally define Probabilistic Robustness (PR) as the complementary probability of an attack failing: PR(p,s):=1−ℐ[p,s]=ℙX∼p(s(X)<0).PR(p,s):=1-I[p,s]=P_X p(s(X)<0). Under this definition, enhancing the intrinsic security of a model corresponds directly to maximizing the value of PR. 2.4 Adversarial Training Adversarial training (AT) is a widely adopted empirical method to defend against adversarial attack described above (Madry et al., 2018). The core idea of AT is to augment the training data with adversarial examples during the learning process. Formally, let θ denote the parameters of the classifier, and let pdatap_data be the underlying data distribution. Adversarial training formulates the robustness objective as a min-max optimization problem: minθ(xori,yori)∼pdata[maxxadv∈(xori)f(xadv,yori,θ)], _θE_(x_ori,y_ori) p_data [ _x_adv (x_ori)f(x_adv,y_ori;θ) ], where (xori)=xadv∈[0,1]d:(xori,xadv)≤ϵS(x_ori)=\x_adv∈[0,1]^d:D(x_ori,x_adv)≤ε\ defines the set of allowed adversarial perturbations bounded by a small budget ϵε (e.g., an L∞L_∞ or L2L_2 norm ball). 3 An Untargeted Version of Probabilistic Adversarial Attack As introduced in Section 2.2, an untargeted adversarial attack can be formulated as the following optimization problem: minxadv∈[0,1]dc1(xori,xadv)−c2f(xadv,yori) _x_adv∈[0,1]^d\;c_1\,D(x_ori,x_adv)\;-\;c_2\,f(x_adv,y_ori) Similar to the work of Zhang et al. (2024a), applying Langevin dynamics to this optimization problem yields convergence to a Gibbs distribution concentrated around the solutions: padv(xadv,xori,yori)∝pvic(xadv,yori)pdis(xadv,xori),p_adv(x_adv;x_ori,y_ori) p_vic(x_adv;y_ori)\,p_dis(x_adv;x_ori), where pvic(xadv,yori)∝exp(c2f(xadv,yori))p_vic(x_adv;y_ori) (c_2\,f(x_adv,y_ori) ) is the “victim” distribution and pdis(xadv,xori)∝exp(−c1(xori,xadv))p_dis(x_adv;x_ori) (-c_1\,D(x_ori,x_adv) ) is the “distance” distribution around xorix_ori. For typical choices of D, it is clear that pdisp_dis is well-defined (Zhang et al., 2024a). The following proposition shows that pvicp_vic is also well-defined in the untargeted setting: Proposition 1 (Well-definedness of the untargeted victim distribution). Let K=[0,1]dK=[0,1]^d and let f(⋅,yori):K→ℝf(·,y_ori):K be continuous. For any c>0c>0, define pvic(x,yori)=exp(cf(x,yori))∫Kexp(cf(u,yori))u,x∈K.p_vic(x;y_ori)= (cf(x,y_ori) ) _K (cf(u,y_ori) )\,du, x∈ K. Then pvic(⋅,yori)p_vic(·;y_ori) is a well-defined probability density on K. The proof is in Appendix B. In this paper, we take D to be the squared L2L_2 norm, so that pdisp_dis is a Gaussian distribution, and choose f to be the cross-entropy loss. Under this setting, the Langevin-dynamics-based algorithm for sampling adversarial examples from padvp_adv is presented in Algorithm 1 (See Appendix A). 4 A KL-Based Lower Bound on Probabilistic Robustness Recall that PR(p,s)PR(p,s) (introduced in Section 2.3) evaluates the model’s security against perturbations drawn from a distribution p around each clean (original) input xorix_ori. We observe that the distance distribution pdis(⋅,xori)p_dis(·;x_ori) introduced in Section 2.2 naturally serves as the evaluation distribution p. Consequently, our objective formally becomes maximizing PR(pdis,s)PR(p_dis,s). To establish a mathematical lower bound on PR, we must connect the attack-success event, characterized by the logit margin violation s(x)≥0s(x)≥ 0 (Section 2.3), with our optimization objective, which relies on the continuous cross-entropy loss f(x,yori)f(x,y_ori). However, establishing a direct, tractable algebraic mapping between the margin s(x)s(x) and the loss f is highly challenging due to softmax non-linearities and complex multi-class dynamics. To circumvent this, we define the attack-success region Eadv=x∈:s(x)≥0E_adv=\x :s(x)≥ 0\ as the vulnerable space where the classifier is fooled. We then introduce a connecting scalar γ∗ _* representing the infimum of the scaled cross-entropy loss over this region: γ∗:=infx∈Eadvc2f(x,yori,θ). _*:= _x∈ E_advc_2f(x,y_ori;θ). Since the cross-entropy loss is inherently non-negative and the temperature scaling factor c2>0c_2>0, it strictly holds that γ∗≥0 _*≥ 0. By definition, any successful adversarial example x∈Eadvx∈ E_adv rigorously guarantees c2f(x,yori,θ)≥γ∗c_2f(x,y_ori;θ)≥ _*. This property is pivotal, as it allows us to formally link the expected loss under pdisp_dis to the probability mass of the vulnerable region, paving the way for the following theorem. Theorem 2 (KL-Based Lower Bound on Probabilistic Robustness). Let pdis(⋅,xori)p_dis(·;x_ori) be a probability density on [0,1]d[0,1]^d. Suppose the victim distribution parameterized by θ is defined as pvic(x,yori,θ)=exp(c2f(x,yori,θ))Zvic,p_vic(x;y_ori,θ)= (c_2f(x,y_ori;θ))Z_vic, where Zvic=∫exp(c2f(u,yori,θ))uZ_vic= _X (c_2f(u,y_ori;θ))du is the normalizing constant. Let γ∗ _* be defined as above and assume γ∗>0 _*>0. Then, the probabilistic robustness satisfies: PR(pdis,s)≥1−logZvic−H(pdis)−KL(pdis∥pvic)γ∗,PR(p_dis,s)≥ 1- Z_vic-H(p_dis)-KL(p_dis\|p_vic) _*, where H(pdis)=−∫pdis(x)logpdis(x)dxH(p_dis)=- _Xp_dis(x) p_dis(x)dx is the differential entropy of the distance distribution. The proof is in Appendix B. Crucially, Theorem 2 formally unifies the probabilistic attack perspective (Zhang et al., 2024a) with the probabilistic robustness framework (Webb et al., 2018). By expressing the lower bound on PR using the statistical divergence between pdisp_dis and pvicp_vic, this theorem directly translates our intuition of “pushing the distributions apart” into a rigorous mathematical objective. Notice that once the clean input xorix_ori and the perturbation budget are fixed, the distance distribution pdisp_dis and its differential entropy H(pdis)H(p_dis) are constants. Furthermore, since γ∗ _* acts as a positive scaling factor, maximizing the lower bound on PR requires maximizing the term KL(pdis∥pvic)−logZvicKL(p_dis\|p_vic)- Z_vic. This fundamental insight serves as the theoretical bedrock for the tractable surrogate objective derived in the next section. While intuitively one might assume that maximizing KL(pdis||pvic)KL(p_dis||p_vic) alone is sufficient to separate the distributions, the −logZvic- Z_vic term holds crucial geometric and physical significance. We defer a detailed discussion on this to Appendix D. 5 Probabilistic Adversarial Training Based on Section 4, maximizing the lower bound KL(pdis||pvic)−logZvicKL(p_dis||p_vic)- Z_vic acts as a surrogate objective that indirectly improves probabilistic robustness. However, calculating the exact gradients of this theoretical bound is challenging due to intractable normalizing constants. In this section, we derive a computable empirical loss function through a structured, three-step derivation. We then explicitly reveal how this empirical formulation recovers a scaled, probabilistic counterpart to standard adversarial training, before formally proving its convergence. Algorithm 2 in Appendix A presents the pseudocode for the final practical implementation of probabilistic adversarial training. 5.1 Deriving the Empirical Loss Function Step 1: The Idealized Objective. We first attempt to formulate an objective by directly applying importance sampling to the theoretical lower bound. Proposition 3 (Idealized Objective). Let pdisp_dis, pvicp_vic, and padvp_adv be defined as in Section 3. Maximizing the probabilistic robustness lower bound with respect to the classifier parameters θ is mathematically equivalent to minimizing the following idealized expected loss: ideal(θ)=X∼padv(⋅,xori,yori,θ)[Zadvpvic(X,yori,θ)c2f(X,yori,θ)]J_ideal(θ)=E_X p_adv(·;x_ori,y_ori,θ) [ Z_advp_vic(X;y_ori,θ)c_2f(X,y_ori;θ) ] See Appendix B for the proof. While Proposition 3 demonstrates that the intractable logZvic Z_vic cancels out, directly optimizing ideal(θ)J_ideal(θ) is highly problematic. The sampling distribution padv(X,xori,yori,θ)∝pdis(X,xori)pvic(X,yori,θ)p_adv(X;x_ori,y_ori,θ) p_dis(X;x_ori)p_vic(X;y_ori,θ) depends on the parameters θ. Differentiating through this expectation is notoriously unstable and requires complex score-function estimators. Step 2: The Gradient-First Approach. To circumvent the challenge of differentiating through a parameterized distribution, we propose computing the gradient of the lower bound before applying importance sampling. Proposition 4 (Gradient of the Objective). The exact gradient of the negative lower bound with respect to θ can be expressed as an expectation over the adversarial distribution padvp_adv: ∇θ(θ)=X∼padv(⋅,xori,yori,θ)[Zadvpvic(X,yori,θ)∇θ(c2f(X,yori,θ))] _θJ(θ)=E_X p_adv(·;x_ori,y_ori,θ) [ Z_advp_vic(X;y_ori,θ) _θ(c_2f(X,y_ori;θ)) ] See Appendix B for the proof. This formulation decouples the sampling distribution from the gradient computation. Because the gradient operator ∇θ _θ only applies to the loss term f, the importance weight w(X,xori,yori,θ)=Zadvpvic(X,yori,θ)w(X;x_ori,y_ori,θ)= Z_advp_vic(X;y_ori,θ) acts purely as a scaling constant for the gradients of individual samples. Step 3: Self-Normalized Importance Sampling. Although Proposition 4 provides a relatively tractable gradient direction, evaluating the theoretical weight w(X,xori,yori,θ)w(X;x_ori,y_ori,θ) is still impossible. By substituting pvic(X,yori,θ)=1Zvicexp(c2f(X,yori,θ))p_vic(X;y_ori,θ)= 1Z_vic (c_2f(X,y_ori;θ)), we reveal the hidden constants: w(X,xori,yori,θ)=ZadvZvicexp(−c2f(X,yori,θ))w(X;x_ori,y_ori,θ)=Z_advZ_vic (-c_2f(X,y_ori;θ)) Both ZadvZ_adv and ZvicZ_vic are intractable to compute. However, for a given clean image xorix_ori and a specific set of model parameters θ, their product is strictly a constant across different sampled perturbations. To circumvent the intractability of these constants, we employ Self-Normalized Importance Sampling (SNIS). Given a mini-batch of adversarial examples X(1),…,X(B)X^(1),...,X^(B) sampled from padvp_adv, we normalize the weights across the batch: w^(i) w^(i) =w(X(i),xori,yori,θ)∑j=1Bw(X(j),xori,yori,θ)=ZadvZvicexp(−c2f(X(i),yori,θ))∑j=1BZadvZvicexp(−c2f(X(j),yori,θ)) = w(X^(i);x_ori,y_ori,θ) _j=1^Bw(X^(j);x_ori,y_ori,θ)= Z_advZ_vic (-c_2f(X^(i),y_ori;θ)) _j=1^BZ_advZ_vic (-c_2f(X^(j),y_ori;θ)) =exp(−c2f(X(i),yori,θ))∑j=1Bexp(−c2f(X(j),yori,θ)) = (-c_2f(X^(i),y_ori;θ)) _j=1^B (-c_2f(X^(j),y_ori;θ)) The intractable terms ZadvZ_adv and ZvicZ_vic cancel out exactly. This self-normalized weighting scheme naturally corresponds to applying a Softmax function. Consequently, the true gradient derived in Proposition 4 can be estimated by backpropagating through the following empirical loss: ℒemp(θ)=∑i=1Bw^(i)f(X(i),yori,θ)L_emp(θ)= _i=1^B w^(i)f(X^(i),y_ori;θ) where w^(i) w^(i) are treated as fixed constants (i.e., stop-gradient) during optimization, as explicitly implemented in Lines 16-18 of Algorithm 2 (Appendix A). 5.2 Connection to Standard Adversarial Training Standard adversarial training (AT) methods, such as PGD with random restarts, can be viewed probabilistically as drawing samples from an implicit adversarial distribution padvp_adv to minimize the unweighted expected loss X∼padv(⋅|xori,yori)[f(X,yori;θ)]E_X p_adv(·|x_ori,y_ori)[f(X,y_ori;θ)]. Empirically, this directly corresponds to optimizing a uniformly weighted batch loss 1B∑i=1Bf(X(i),yori,θ) 1B _i=1^Bf(X^(i),y_ori;θ). In this work, both our idealized objective idealJ_ideal and empirical loss ℒempL_emp recover this exact AT paradigm, differing solely by a principled scaling factor. Instead of treating all samples equally, Probabilistic Adversarial Training (PAT) optimizes a weighted loss using the self-normalized importance weights w^(i)∝exp(−c2f(X(i),yori,θ)) w^(i) (-c_2f(X^(i),y_ori;θ)). This negative exponent acts as a direct penalty on extreme adversarial examples. While standard AT strictly focuses on worst-case perturbations, PAT deliberately down-weights extreme outliers. This weighting mechanism prevents the model from over-optimizing for worst-case attacks, thereby improving overall probabilistic robustness. 5.3 Convergence of the Empirical Gradient Estimator We present a convergence theorem demonstrating that our self-normalized importance sampling (SNIS) estimator is asymptotically consistent. Theorem 5 (Asymptotic Consistency of the SNIS Gradient Estimator). Let X(1),…,X(B)X^(1),…,X^(B) be i.i.d. samples drawn from the adversarial distribution padv(⋅,xori,yori,θ)p_adv(·;x_ori,y_ori,θ). Define the unnormalized importance weights as w~(X,yori,θ)=exp(−c2f(X,yori,θ)) w(X;y_ori,θ)= (-c_2f(X,y_ori;θ)). The SNIS empirical gradient estimator is given by: g^B(θ)=∑i=1Bw^(i)∇θf(X(i),yori,θ)=∑i=1Bw~(X(i),yori,θ)∇θf(X(i),yori,θ)∑i=1Bw~(X(i),yori,θ) g_B(θ)= _i=1^B w^(i) _θf(X^(i),y_ori;θ)= _i=1^B w(X^(i);y_ori,θ) _θf(X^(i),y_ori;θ) _i=1^B w(X^(i);y_ori,θ) Assume the expected gradient under the distance distribution is finite, i.e., X∼pdis(⋅,xori)[|∇θf(X,yori,θ)|]<∞E_X p_dis(·;x_ori)[| _θf(X,y_ori;θ)|]<∞. As the batch size B→∞B→∞, the estimator g^B(θ) g_B(θ) converges almost surely to the true expected gradient under pdis(⋅,xori)p_dis(·;x_ori): g^B(θ)→a.s.X∼pdis(⋅,xori)[∇θf(X,yori,θ)] g_B(θ) a.s.E_X p_dis(·;x_ori)[ _θf(X,y_ori;θ)] The proof is provided in Appendix B. 5.4 Batch-Level Approximation and Implicit Regularization While Theorem 5 assumes multiple perturbations per single image xorix_ori, this is computationally expensive. For efficiency, our practical implementation (Algorithm 2 in Appendix A) applies SNIS across a mini-batch of clean images (x(i),y(i))∼pdata\(x^(i),y^(i))\ p_data, generating only one perturbation per image. This batch-level formulation alters the weighting scheme. A strict estimator over the joint distribution requires weighting each sample by an intractable local adversarial volume: w¯true(i)∝Zadv(i)Zvic(i)exp(−c2f(X(i),y(i),θ)) w_true^(i) Z_adv^(i)Z_vic^(i) (-c_2f(X^(i),y^(i);θ)) By definition, Zadv(i)Zvic(i)=U∼pdis(⋅,x(i))[exp(c2f(U,y(i),θ))]Z_adv^(i)Z_vic^(i)=E_U p_dis(·;x^(i))[ (c_2f(U,y^(i);θ))]. For highly vulnerable samples, this large expected volume theoretically compensates for the sharp decay of exp(−c2f) (-c_2f). Naively omitting the volume term while retaining a large attack temperature c2c_2 leads to catastrophic weight collapse: the weights of crucial hard samples are driven to near zero, forcing the Softmax to collapse onto the easiest samples and defeating the purpose of adversarial training. To counteract this without calculating the intractable volume, we introduce a small smoothing factor β (e.g., β=0.001β=0.001) to replace c2c_2 in the final empirical weighting: w~emp(i)=exp(−βf(X(i),y(i),θ)) w_emp^(i)= (-β f(X^(i),y^(i);θ)) (2) This β acts as a conceptual temperature scaling that prevents weight collapse for valid hard samples. Concurrently, retaining the negative exponential structure preserves the algorithmic benefit of omitting Zadv(i)Zvic(i)Z_adv^(i)Z_vic^(i): implicit regularization against robust overfitting. Outliers or mislabeled data, which theoretically possess exploding local volumes, are penalized. This extends our core philosophy of down-weighting extreme worst-case outliers to the global data manifold. The theoretical analysis of this batch-level estimator is provided in Appendix C. Table 1: Comprehensive evaluation of Probabilistic Robustness (PR) and test accuracies across CIFAR-10 and CIFAR-100 datasets. PR is evaluated under escalating perturbation scales D. M. Attack Acc. % PGD % CW % PR(⋅·, 0.1) % PR(⋅·, 0.12) % PR(⋅·, 0.15) % PR(⋅·, 0.2) % CIFAR-10 ResNet-18 Clean 93.75 0.00 0.00 49.82 36.62 24.19 16.36 FGSM 87.16 0.01 0.05 88.88 84.23 75.83 59.91 PGD 82.00 45.80 44.94 95.26 92.55 87.72 77.03 ALP 81.24 46.65 45.56 95.36 92.88 88.16 77.94 CLP 83.23 44.11 43.34 95.37 93.03 88.47 78.77 TRADES 79.62 48.78 45.91 94.33 91.97 87.33 77.33 MART 78.56 51.30 47.02 94.26 91.86 87.36 78.29 AT-PR 80.33 40.58 39.23 95.25 92.80 88.10 78.22 PAT 82.44 32.53 30.51 95.53 93.43 89.26 80.62 WRN-50-2 Clean 91.89 0.00 0.00 56.99 44.66 31.52 20.85 FGSM 87.60 0.00 0.00 79.79 67.51 44.35 21.04 PGD 76.49 46.54 44.33 95.59 93.56 89.44 79.98 ALP 72.45 44.42 41.50 95.71 93.76 89.71 81.23 CLP 79.61 37.71 36.41 95.86 93.69 88.94 78.70 TRADES 74.03 44.38 40.82 94.72 92.62 88.48 79.89 MART 69.87 46.83 42.29 94.93 92.83 89.26 81.93 AT-PR 75.93 40.03 39.21 95.47 93.61 89.61 80.23 PAT 78.26 31.94 29.91 95.98 93.96 90.96 82.19 CIFAR-100 ResNet-18 Clean 74.61 0.00 0.00 27.45 19.20 11.93 7.08 FGSM 59.56 0.00 0.00 64.50 53.45 38.65 21.62 PGD 54.50 22.48 21.18 89.27 83.31 71.01 46.90 ALP 54.88 23.83 21.98 90.45 84.69 72.95 49.35 CLP 55.71 22.08 19.28 90.05 84.80 74.50 54.53 TRADES 53.56 26.74 22.66 89.07 84.23 74.74 56.24 MART 52.13 26.88 23.51 90.02 84.86 74.66 51.76 AT-PR 52.67 21.48 20.28 89.09 83.14 71.18 47.64 PAT 53.54 17.50 14.87 90.77 86.58 78.07 59.78 WRN-50-2 Clean 70.41 0.00 0.00 25.51 17.95 11.23 6.24 FGSM 51.04 0.00 0.00 39.19 25.26 13.12 5.26 PGD 49.55 24.04 22.00 92.06 87.94 78.85 58.99 ALP 44.38 23.10 20.80 92.07 88.07 79.92 62.17 CLP 52.95 19.15 16.94 90.91 86.36 76.20 56.99 TRADES 49.85 23.01 19.18 90.64 86.46 78.32 62.03 MART 42.61 24.67 21.15 92.03 88.52 81.41 66.51 AT-PR 48.47 22.02 20.98 92.11 88.08 79.87 61.58 PAT 49.95 17.26 15.98 92.28 89.49 82.18 67.12 6 Experiments Due to space constraints, this section focuses on the main evaluation results. Comprehensive details regarding the experimental settings, datasets, and hyperparameters are deferred to Appendix E. 6.1 Probabilistic Robustness Table 1 summarizes the performance of PAT and baselines (PGD (Madry et al., 2018), ALP (Kannan et al., 2018), CLP (Kannan et al., 2018), TRADES (Zhang et al., 2019), MART (Wang et al., 2019) and AT-PR (Zhang et al., 2025)) on CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009). PAT consistently achieves superior Probabilistic Robustness (PR) across all evaluated architectures (WRN-50-2, ResNet-18) and datasets, with the advantage becoming more pronounced at larger perturbation scales. While PAT shows a nominal decrease in strictly worst-case robustness (PGD/CW Acc.) compared to unweighted AT, this directly aligns with our theory. Unlike standard AT which over-fits to anomalous worst-case boundaries, PAT’s theoretically derived importance weight naturally penalizes extreme outliers. This optimizes overall distributional security, effectively pushing the distance and victim distributions apart. 6.2 Ablation Study Table 2 isolates the impact of our theoretically derived importance weight through two variants: PAT(WOS) (Langevin sampling without the importance weight) and PGD-COR/FGSM-COR (standard attacks equipped with our theoretical scaling factor). PAT consistently outperforms PAT(WOS). On CIFAR-100 (WRN-50-2), removing the importance weight drops PR(0.2) from 67.12%67.12\% to 63.98%63.98\%. This confirms that sampling from padvp_adv alone is insufficient; down-weighting extreme outliers via our theoretical formulation is an essential requirement for maximizing probabilistic robustness. Our importance weight acts as a versatile plug-in regularizer. Applying it to standard PGD (PGD-COR) improves PR(0.2) on CIFAR-100 from 58.99%58.99\% to 62.21%62.21\%. Notably, the performance of FGSM-COR exhibits more irregularity. This discrepancy is theoretically expected: as a deterministic, single-step attack, FGSM lacks the iterative exploration and stochasticity required for the importance weight to effectively evaluate the local adversarial distribution. Nonetheless, the consistent gains in PGD-COR demonstrate the broader utility of our weighting scheme in improving the distributional robustness of traditional iterative defense methods. Table 2: Ablation results for the importance weight across various architectures. PAT(WOS) removes the weighting factor; COR denotes the application of our scaling factor to standard baselines. D. M. Attack Acc. % PGD % CW % PR(⋅·, 0.1) % PR(⋅·, 0.12) % PR(⋅·, 0.15) % PR(⋅·, 0.2) % CIFAR-10 ResNet-18 FGSM 87.16 0.01 0.05 88.88 84.23 75.83 59.91 FGSM-COR 89.04 0.02 0.03 89.03 84.53 75.33 56.45 PGD 82.00 45.80 44.94 95.26 92.55 87.72 77.03 PGD-COR 81.75 45.81 44.72 95.03 92.38 87.33 75.97 PAT(WOS) 82.96 32.15 30.34 94.86 92.56 88.36 79.81 PAT 82.44 32.53 30.51 95.53 93.43 89.26 80.62 WRN-50-2 FGSM 87.60 0.00 0.00 79.79 67.51 44.35 21.04 FGSM-COR 76.46 42.63 41.01 93.42 90.26 84.74 74.40 PGD 76.49 46.54 44.33 95.59 93.56 89.44 79.98 PGD-COR 73.45 45.11 42.71 95.79 93.85 89.79 80.62 PAT(WOS) 79.06 31.99 29.63 94.63 92.29 87.91 79.25 PAT 78.26 31.94 29.91 95.98 93.96 90.96 82.19 CIFAR-100 ResNet-18 FGSM 59.56 0.00 0.00 64.50 53.45 38.65 21.62 FGSM-COR 56.20 0.00 0.01 62.68 51.06 35.49 18.52 PGD 54.50 22.48 21.18 89.27 83.31 71.01 46.90 PGD-COR 55.02 23.42 22.06 90.11 84.23 72.39 47.92 PAT(WOS) 53.21 16.84 14.53 90.47 86.24 78.07 59.83 PAT 53.54 17.50 14.87 90.77 86.58 78.07 59.78 WRN-50-2 FGSM 51.04 0.00 0.00 39.19 25.26 13.12 5.26 FGSM-COR 52.50 0.00 0.00 41.66 26.31 11.75 3.88 PGD 49.55 24.04 22.00 92.06 87.94 78.85 58.99 PGD-COR 44.87 22.74 20.46 92.66 88.79 80.55 62.21 PAT(WOS) 50.17 16.92 14.96 91.65 87.91 80.22 63.98 PAT 49.95 17.26 15.98 92.28 89.49 82.18 67.12 7 Related Work Adversarial Training for Worst-Case Robustness. Traditional adversarial training (AT) formulates a min-max optimization problem to defend against worst-case perturbations (Madry et al., 2018). A vast literature has developed around improving this worst-case robustness (WCR), including regularization techniques like TRADES (Zhang et al., 2019) and MART (Wang et al., 2019), logit pairing methods such as ALP and CLP (Kannan et al., 2018), and efficient training variants like Fast-AT (Wong et al., 2020) and Free-AT (Shafahi et al., 2019). Complementary to our work, Li & Li (2024) theoretically analyzed the feature learning dynamics of AT. However, these WCR-focused methods inherently optimize for extreme, deterministic adversarial outliers, which frequently compromises standard accuracy and broader distributional security (Zhao, 2026). Probabilistic Robustness Assessment. To bridge the gap between extreme-case vulnerability and practical reliability, Webb et al. (2018) formally defined Probabilistic Robustness (PR) to evaluate the overall likelihood of adversarial examples within a local neighborhood. Following this, significant progress has been made in PR assessment, including scalable statistical estimators (Tit et al., 2021; Karim et al., 2023), probabilistic verification approaches (Weng et al., 2019), and extensions to functional perturbations (Zhang et al., 2022). A comprehensive survey of these evaluation metrics is provided by Zhao (Zhao, 2026). While evaluating PR has become increasingly sophisticated, dedicated training frameworks designed to directly optimize it remain sparse. Defenses Targeting Probabilistic Robustness. Only a few recent works have explicitly aimed at improving PR during training. Wang et al. (2021) demonstrated that adding random perturbations enhances PR, though it yields near-zero worst-case robustness. Other approaches (Robey et al., 2022; Zhang et al., 2024b) proposed training with risk measures such as Conditional Value-at-Risk (CVaR) or Entropic Value-at-Risk (EVaR), but also struggled to maintain competitive standard adversarial robustness. The most direct comparison to our work is the recent AT-PR method (Zhang et al., 2025), which explicitly alters the inner maximization of AT to target PR. However, their approach relies on a heuristic, gradient-based boundary search to empirically find the “widest peak” in the loss landscape. In contrast, our Probabilistic Adversarial Training (PAT) establishes a rigorous theoretical foundation: we formalize untargeted probabilistic attacks via Langevin Dynamics, prove a KL-based lower bound for PR, and employ Self-Normalized Importance Sampling (SNIS) to derive a fully tractable, principled optimization objective. 8 Conclusion In this work, we provided a unified probabilistic perspective on adversarial defense by proving that KL(pdis||pvic)−logZvicKL(p_dis||p_vic)- Z_vic serves as a lower bound for probabilistic robustness (PR). By maximizing this tractable surrogate objective via SNIS applied to gradient, we introduced Probabilistic Adversarial Training (PAT). Our theoretical framework not only offers a principled probabilistic interpretation of standard adversarial training but also yields a natural importance-weighting mechanism that penalizes extreme worst-case outliers. Empirically, PAT consistently enhances PR across various architectures. We hope this probabilistic perspective of adversarial vulnerabilities inspires more robust and theoretically grounded defense strategies in the future. Acknowledgments The work presented in this paper has been supported by UKRI Future Leaders Fellowship (Grant MR/S035176/1) and funded by the European Union under the Horizon Europe project AIGGREGATE (AI-enhanced collective intelligence for resilient, ethical and user-centric awareness and decision making in CCAM applications, Grant Agreement No. 101202457). Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them. References Carlini & Wagner (2017) Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), p. 39–57. Ieee, 2017. Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6572. Iglesias et al. (1998) Pilar L Iglesias, Carlos AB Pereira, and Nelson I Tanaka. Characterizations of multivariate spherical distributions in l infinity norm. Test, 7(2):307–324, 1998. Kannan et al. (2018) Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv:1803.06373, 2018. Karim et al. (2023) TIT Karim, Teddy Furon, and Mathias Rousset. Gradient-informed neural network statistical robustness estimation. In International Conference on Artificial Intelligence and Statistics, p. 323–334. PMLR, 2023. Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. Kurakin et al. (2017) Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=BJm4T4Kgx. Lamperski (2021) Andrew Lamperski. Projected stochastic gradient langevin algorithms for constrained sampling and non-convex learning. In Conference on Learning Theory, p. 2891–2937. PMLR, 2021. Li & Li (2024) Binghui Li and Yuanzhi Li. Adversarial training can provably improve robustness: Theoretical analysis of feature learning process under structured data. arXiv preprint arXiv:2410.08503, 2024. Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=rJzIBfZAb. Moosavi-Dezfooli et al. (2016) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2574–2582, 2016. Moosavi-Dezfooli et al. (2017) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 1765–1773, 2017. Papernot et al. (2016) Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (EuroS&P), p. 372–387. IEEE, 2016. Robey et al. (2022) Alexander Robey, Luiz Chamon, George J Pappas, and Hamed Hassani. Probabilistically robust learning: Balancing average and worst-case performance. In International Conference on Machine Learning, p. 18667–18686. PMLR, 2022. Shafahi et al. (2019) Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in neural information processing systems, 32, 2019. Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http://arxiv.org/abs/1312.6199. Tit et al. (2021) Karim Tit, Teddy Furon, and Mathias Rousset. Efficient statistical assessment of neural network corruption robustness. Advances in Neural Information Processing Systems, 34:9253–9263, 2021. Wang et al. (2021) Benjie Wang, Stefan Webb, and Tom Rainforth. Statistically robust neural network classification. In Uncertainty in Artificial Intelligence, p. 1735–1745. PMLR, 2021. Wang et al. (2019) Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International conference on learning representations, 2019. Webb et al. (2018) Stefan Webb, Tom Rainforth, Yee Whye Teh, and M Pawan Kumar. A statistical approach to assessing neural network robustness. arXiv preprint arXiv:1811.07209, 2018. Weng et al. (2019) Lily Weng, Pin-Yu Chen, Lam Nguyen, Mark Squillante, Akhilan Boopathy, Ivan Oseledets, and Luca Daniel. Proven: Verifying robustness of neural networks with a probabilistic approach. In International Conference on Machine Learning, p. 6727–6736. PMLR, 2019. Wong et al. (2020) Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994, 2020. Zhang et al. (2024a) Andi Zhang, Mingtian Zhang, and Damon Wischik. Constructing semantics-aware adversarial examples with a probabilistic perspective. Advances in Neural Information Processing Systems, 37:136259–136285, 2024a. Zhang et al. (2019) Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, p. 7472–7482. PMLR, 2019. Zhang et al. (2022) Tianle Zhang, Wenjie Ruan, and Jonathan E Fieldsend. Proa: A probabilistic robustness assessment against functional perturbations. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, p. 154–170. Springer, 2022. Zhang et al. (2024b) Tianle Zhang, Yanghao Zhang, Ronghui Mu, Jiaxu Liu, Jonathan E Fieldsend, and Wenjie Ruan. Prass: Probabilistic risk-averse robust learning with stochastic search. In IJCAI, p. 559–567, 2024b. Zhang et al. (2025) Yi Zhang, Yuhang Chen, Zhen Chen, Wenjie Ruan, Xiaowei Huang, Siddartha Khastgir, and Xingyu Zhao. Adversarial training for probabilistic robustness. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 1675–1685, 2025. Zhao (2026) Xingyu Zhao. Probabilistic robustness in deep learning: A concise yet comprehensive guide. In Adversarial Example Detection and Mitigation Using Machine Learning, p. 209–222. Springer, 2026. Appendix Appendix A Pseudo Code of the Algorithms Algorithm 1 Untargeted Probabilistic Adversarial Attack Input: Original image xori∈[0,1]dx_ori∈[0,1]^d, original class yoriy_ori, cross-entropy loss f corresponding to the victim classifier, parameters c1,c2>0c_1,c_2>0, step size η, noise scale σ, gradient clipping threshold ρ>0ρ>0, total timesteps T. Output: Adversarial example xadvx_adv. xadv∼Uniform[0,1]dx_adv [0,1]^d ⊳ Initialize for t=1t=1 to T do zt∼(0,I)z_t (0,I) ⊳ Generate noise for Langevin Dynamics xadv←Π[0,1]d(xadv+σzt)x_adv← _[0,1]^d (x_adv+σ z_t )⊳ Apply noise ℰ←c1‖xadv−xori‖22−c2f(xadv,yori)E← c_1\,\|x_adv-x_ori\|_2^2-c_2\,f(x_adv,y_ori)⊳ Calculate energy vt←clip(∇xadvℰ,−ρ,ρ)v_t ( _x_advE,-ρ,ρ)⊳ Compute and clip gradient xadv←Π[0,1]d(xadv−ηvt)x_adv← _[0,1]^d (x_adv-η v_t )⊳ update xadvx_adv end for return xadvx_adv. Algorithm 2 Probabilistic Adversarial Training 1: Input: Training dataset D, initial model parameters θ, cross-entropy loss f, learning rate α. 2: Hyperparameters: Total epochs EtotalE_total, batch size B, inner steps T, step size η, noise scale σ, gradient clip ρ, energy weights c1,c2c_1,c_2, smoothing factor β. 3: Output: Robust model parameters θ. 4: for epoch e=0e=0 to Etotal−1E_total-1 do 5: for each minibatch (,)∼(X,Y) with size B do 6: ⊳ Phase 1: Probabilistic Adversarial Example Generation 7: adv∼Uniform[0,1]B×dX_adv [0,1]^B× d⊳ Initialize 8: for t=1t=1 to T do 9: t∼(,)B×dZ_t (0,I)^B× d⊳ Sample noise for Langevin Dynamics 10: adv←Π[0,1]B×d(adv+σt)X_adv← _[0,1]^B× d (X_adv+ _t )⊳ Apply noise 11: ℰ←∑i=1B(c1‖adv(i)−(i)‖22−c2fθ(adv(i),(i)))E← _i=1^B (c_1\,\|X_adv^(i)-X^(i)\|_2^2-c_2\,f_θ(X_adv^(i),Y^(i)) ) ⊳ Calculate batch energy 12: t←clip(∇advℰ,−ρ,ρ)V_t ( _X_advE,-ρ,ρ) ⊳ Compute and clip gradient 13: adv←Π[0,1]B×d(adv−ηt)X_adv← _[0,1]^B× d (X_adv- _t ) ⊳ Update advX_adv 14: end for 15: ⊳ Phase 2: Model Parameter Update via Gradient Descent 16: ℓi←fθ(adv(i),(i)) _i← f_θ(X_adv^(i),Y^(i))⊳ Compute sample losses ℓ∈ℝB ^B 17: ~pvic←Detach(−βℓ) w_pvic (-β\, )⊳ Apply smoothing factor and stop gradient 18: ^pvic←Softmax(~pvic) w_pvic ( w_pvic) ⊳ Softmax over batch dimension 19: ℒ←ℓ⊤^pvicL← w_pvic ⊳ Final weighted loss 20: θ←θ−α∇θℒθ←θ-α _θL ⊳ Update model parameters 21: end for 22: end for 23: return θ. Appendix B Proofs of Theorems in the Main Text Proposition 1 (Well-definedness of the untargeted victim distribution). Let K=[0,1]dK=[0,1]^d and let f(⋅,yori):K→ℝf(·,y_ori):K be continuous. For any c>0c>0, define pvic(x,yori)=exp(cf(x,yori))∫Kexp(cf(u,yori))u,x∈K.p_vic(x;y_ori)= (cf(x,y_ori) ) _K (cf(u,y_ori) )\,du, x∈ K. Then pvic(⋅,yori)p_vic(·;y_ori) is a well-defined probability density on K. Proof. Since K=[0,1]dK=[0,1]^d is compact and f(⋅,yori)f(·,y_ori) is continuous, exp(cf(⋅,yori)) (cf(·,y_ori)) is also continuous on K. Hence it is bounded and measurable. Moreover, it is strictly positive on K, so the normalizing constant Zvic=∫Kexp(cf(u,yori))uZ_vic= _K (cf(u,y_ori) )\,du satisfies 0<Zvic<∞0<Z_vic<∞. Therefore pvicp_vic is non-negative, measurable, and integrates to one over K. Thus it is a well-defined probability density. ∎ Theorem 2 (KL-Based Lower Bound on Probabilistic Robustness). Let pdis(⋅,xori)p_dis(·;x_ori) be a probability density on [0,1]d[0,1]^d. Suppose the victim distribution parameterized by θ is defined as pvic(x,yori,θ)=exp(c2f(x,yori,θ))Zvic,p_vic(x;y_ori,θ)= (c_2f(x,y_ori;θ))Z_vic, where Zvic=∫exp(c2f(u,yori,θ))uZ_vic= _X (c_2f(u,y_ori;θ))du is the normalizing constant. Let γ∗ _* be defined as above and assume γ∗>0 _*>0. Then, the probabilistic robustness satisfies: PR(pdis,s)≥1−logZvic−H(pdis)−KL(pdis∥pvic)γ∗,PR(p_dis,s)≥ 1- Z_vic-H(p_dis)-KL(p_dis\|p_vic) _*, where H(pdis)=−∫pdis(x)logpdis(x)dxH(p_dis)=- _Xp_dis(x) p_dis(x)dx is the differential entropy of the distance distribution. Proof. By the definition of γ∗ _*, for all x∈Eadvx∈ E_adv, we have c2f(x,yori)≥γ∗c_2f(x,y_ori)≥ _*. Thus, we can lower bound the expected loss under pdisp_dis: X∼pdis[c2f(X,yori)]≥∫Eadvpdis(x)c2f(x,yori)x≥γ∗∫Eadvpdis(x)x=γ∗ℐ[pdis,s]E_X p_dis[c_2f(X,y_ori)]≥ _E_advp_dis(x)c_2f(x,y_ori)dx≥ _* _E_advp_dis(x)dx= _*I[p_dis,s] On the other hand, expanding the KL divergence with pvic(x,yori)=exp(c2f(x,yori))Zvicp_vic(x;y_ori)= (c_2f(x,y_ori))Z_vic yields: KL(pdis||pvic) KL(p_dis||p_vic) =∫pdis(x)logpdis(x)x−∫pdis(x)logpvic(x,yori)x = p_dis(x) p_dis(x)dx- p_dis(x) p_vic(x;y_ori)dx =−H(pdis)−X∼pdis[c2f(X,yori)−logZvic] =-H(p_dis)-E_X p_dis[c_2f(X,y_ori)- Z_vic] =−H(pdis)+logZvic−X∼pdis[c2f(X,yori)] =-H(p_dis)+ Z_vic-E_X p_dis[c_2f(X,y_ori)] ≤−H(pdis)+logZvic−γ∗ℐ[pdis,s] ≤-H(p_dis)+ Z_vic- _*I[p_dis,s] Rearranging the inequality gives: ℐ[pdis,s]≤logZvic−H(pdis)−KL(pdis||pvic)γ∗I[p_dis,s]≤ Z_vic-H(p_dis)-KL(p_dis||p_vic) _* Finally, by the definition of probabilistic robustness PR(pdis,s)=1−ℐ[pdis,s]PR(p_dis,s)=1-I[p_dis,s], we conclude: PR(pdis,s)≥1−logZvic−H(pdis)−KL(pdis||pvic)γ∗PR(p_dis,s)≥ 1- Z_vic-H(p_dis)-KL(p_dis||p_vic) _* ∎ Proposition 3 (Idealized Objective). Let pdisp_dis, pvicp_vic, and padvp_adv be defined as in Section 3. Maximizing the probabilistic robustness lower bound with respect to the classifier parameters θ is mathematically equivalent to minimizing the following idealized expected loss: ideal(θ)=X∼padv(⋅,xori,yori,θ)[Zadvpvic(X,yori,θ)c2f(X,yori,θ)]J_ideal(θ)=E_X p_adv(·;x_ori,y_ori,θ) [ Z_advp_vic(X;y_ori,θ)c_2f(X,y_ori;θ) ] Proof. By expanding the KL divergence and changing the measure from pdisp_dis to padvp_adv, we have: KL(pdis(⋅;xori)||pvic(⋅;yori,θ))−logZvic KL(p_dis(·;x_ori)||p_vic(·;y_ori,θ))- Z_vic =X∼pdis(⋅,xori)[logpdis(X,xori)]−∫pdis(x,xori)logpvic(x,yori,θ)x−logZvic =E_X p_dis(·;x_ori)[ p_dis(X;x_ori)]- p_dis(x;x_ori) p_vic(x;y_ori,θ)dx- Z_vic =−H(pdis)−∫padv(x,xori,yori,θ)Zadvpdis(x,xori)pdis(x,xori)pvic(x,yori,θ)logpvic(x,yori,θ)x−logZvic =-H(p_dis)- p_adv(x;x_ori,y_ori,θ) Z_advp_dis(x;x_ori)p_dis(x;x_ori)p_vic(x;y_ori,θ) p_vic(x;y_ori,θ)dx- Z_vic =−H(pdis)−X∼padv(⋅,xori,yori,θ)[Zadvpvic(X,yori,θ)(c2f(X,yori,θ)−logZvic)]−logZvic =-H(p_dis)-E_X p_adv(·;x_ori,y_ori,θ) [ Z_advp_vic(X;y_ori,θ)(c_2f(X,y_ori;θ)- Z_vic) ]- Z_vic =−H(pdis)−X∼padv(⋅,xori,yori,θ)[Zadvpvic(X,yori,θ)c2f(X,yori,θ)] =-H(p_dis)-E_X p_adv(·;x_ori,y_ori,θ) [ Z_advp_vic(X;y_ori,θ)c_2f(X,y_ori;θ) ] +logZvicX∼padv(⋅,xori,yori,θ)[Zadvpvic(X,yori,θ)]⏟=1−logZvic + Z_vic E_X p_adv(·;x_ori,y_ori,θ) [ Z_advp_vic(X;y_ori,θ) ]_=1- Z_vic =−H(pdis)−X∼padv(⋅,xori,yori,θ)[Zadvpvic(X,yori,θ)c2f(X,yori,θ)] =-H(p_dis)-E_X p_adv(·;x_ori,y_ori,θ) [ Z_advp_vic(X;y_ori,θ)c_2f(X,y_ori;θ) ] Since H(pdis)H(p_dis) is constant with respect to θ, maximizing the lower bound is equivalent to minimizing ideal(θ)J_ideal(θ). ∎ Proposition 4 (Gradient of the Objective). The exact gradient of the negative lower bound with respect to θ can be expressed as an expectation over the adversarial distribution padvp_adv: ∇θ(θ)=X∼padv(⋅,xori,yori,θ)[Zadvpvic(X,yori,θ)∇θ(c2f(X,yori,θ))] _θJ(θ)=E_X p_adv(·;x_ori,y_ori,θ) [ Z_advp_vic(X;y_ori,θ) _θ(c_2f(X,y_ori;θ)) ] Proof. We return to the negative lower bound before importance sampling: −KL(pdis||pvic)+logZvic=H(pdis)+X∼pdis(⋅,xori)[c2f(X,yori;θ)].-KL(p_dis||p_vic)+ Z_vic=H(p_dis)+E_X p_dis(·;x_ori)[c_2f(X,y_ori;θ)]. Since pdis(⋅,xori)p_dis(·;x_ori) depends only on the clean image xorix_ori and is entirely independent of θ, the gradient operator can pass directly inside the expectation: ∇θ(X∼pdis(⋅,xori)[c2f(X,yori,θ)]) _θ (E_X p_dis(·;x_ori)[c_2f(X,y_ori;θ)] ) = = X∼pdis(⋅,xori)[∇θc2f(X,yori,θ)] _X p_dis(·;x_ori)[ _θc_2f(X,y_ori;θ)] = = ∫padv(x,xori,yori,θ)pdis(x,xori)padv(x,xori,yori,θ)∇θc2f(x,yori,θ)x p_adv(x;x_ori,y_ori,θ) p_dis(x;x_ori)p_adv(x;x_ori,y_ori,θ) _θc_2f(x,y_ori;θ)dx = = X∼padv(⋅,xori,yori,θ)[Zadvpvic(X,yori,θ)∇θc2f(X,yori,θ)] _X p_adv(·;x_ori,y_ori,θ) [ Z_advp_vic(X;y_ori,θ) _θc_2f(X,y_ori;θ) ] ∎ Theorem 5 (Asymptotic Consistency of the SNIS Gradient Estimator). Let X(1),…,X(B)X^(1),…,X^(B) be i.i.d. samples drawn from the adversarial distribution padv(⋅,xori,yori,θ)p_adv(·;x_ori,y_ori,θ). Define the unnormalized importance weights as w~(X,yori,θ)=exp(−c2f(X,yori,θ)) w(X;y_ori,θ)= (-c_2f(X,y_ori;θ)). The SNIS empirical gradient estimator is given by: g^B(θ)=∑i=1Bw^(i)∇θf(X(i),yori,θ)=∑i=1Bw~(X(i),yori,θ)∇θf(X(i),yori,θ)∑i=1Bw~(X(i),yori,θ) g_B(θ)= _i=1^B w^(i) _θf(X^(i),y_ori;θ)= _i=1^B w(X^(i);y_ori,θ) _θf(X^(i),y_ori;θ) _i=1^B w(X^(i);y_ori,θ) Assume the expected gradient under the distance distribution is finite, i.e., X∼pdis(⋅,xori)[|∇θf(X,yori,θ)|]<∞E_X p_dis(·;x_ori)[| _θf(X,y_ori;θ)|]<∞. As the batch size B→∞B→∞, the estimator g^B(θ) g_B(θ) converges almost surely to the true expected gradient under pdis(⋅,xori)p_dis(·;x_ori): g^B(θ)→a.s.X∼pdis(⋅,xori)[∇θf(X,yori,θ)] g_B(θ) a.s.E_X p_dis(·;x_ori)[ _θf(X,y_ori;θ)] Proof. By the Strong Law of Large Numbers (SLLN), as B→∞B→∞, the sample averages in the numerator and denominator of g^B(θ) g_B(θ) converge almost surely to their respective expectations under padv(⋅,xori,yori,θ)p_adv(·;x_ori,y_ori,θ): 1B∑i=1Bw~(X(i),yori,θ)∇θf(X(i),yori,θ) 1B _i=1^B w(X^(i);y_ori,θ) _θf(X^(i),y_ori;θ) →a.s.X∼padv[w~(X,yori,θ)∇θf(X,yori,θ)] a.s.E_X p_adv [ w(X;y_ori,θ) _θf(X,y_ori;θ) ] 1B∑i=1Bw~(X(i),yori,θ) 1B _i=1^B w(X^(i);y_ori,θ) →a.s.X∼padv[w~(X,yori,θ)] a.s.E_X p_adv [ w(X;y_ori,θ) ] Recall the definitions from Section 3. We can express the product of the sampling density and the unnormalized weight as: padv(X,xori,yori,θ)w~(X,yori,θ) p_adv(X;x_ori,y_ori,θ) w(X;y_ori,θ) =pdis(X,xori)exp(c2f(X,yori,θ))ZadvZvicexp(−c2f(X,yori,θ)) = p_dis(X;x_ori) (c_2f(X,y_ori;θ))Z_advZ_vic (-c_2f(X,y_ori;θ)) =pdis(X,xori)ZadvZvic = p_dis(X;x_ori)Z_advZ_vic Substituting this identity into the expectation of the numerator yields: X∼padv[w~(X,yori,θ)∇θf(X,yori,θ)] _X p_adv [ w(X;y_ori,θ) _θf(X,y_ori;θ) ] =∫pdis(x,xori)ZadvZvic∇θf(x,yori,θ)x = p_dis(x;x_ori)Z_advZ_vic _θf(x,y_ori;θ)dx =1ZadvZvicX∼pdis[∇θf(X,yori,θ)] = 1Z_advZ_vicE_X p_dis [ _θf(X,y_ori;θ) ] Applying the same substitution to the denominator: X∼padv[w~(X,yori,θ)]=∫pdis(x,xori)ZadvZvicx=1ZadvZvicE_X p_adv [ w(X;y_ori,θ) ]= p_dis(x;x_ori)Z_advZ_vicdx= 1Z_advZ_vic By the Continuous Mapping Theorem (or Slutsky’s Theorem), the limit of the ratio is the ratio of their limits. Notice that the intractable constant term perfectly cancels out: g^B(θ)→a.s.1ZadvZvicX∼pdis[∇θf(X,yori,θ)]1ZadvZvic=X∼pdis(⋅,xori)[∇θf(X,yori,θ)] g_B(θ) a.s. 1Z_advZ_vicE_X p_dis [ _θf(X,y_ori;θ) ] 1Z_advZ_vic=E_X p_dis(·;x_ori) [ _θf(X,y_ori;θ) ] This confirms that the self-normalized gradient is asymptotically unbiased and converges to the true gradient of our theoretical objective. ∎ Appendix C Theoretical Analysis of the Batch-Level Estimator Theorem 5 establishes consistency of the SNIS gradient estimator in an idealized regime: a single clean image xorix_ori with label yoriy_ori, a batch of B perturbations from its adversarial distribution padv(⋅,xori,yori,θ)p_adv(·;x_ori,y_ori,θ). Algorithm 2 differs in two respects: it draws a mini-batch of distinct clean images with one perturbation each, and it uses the smoothing factor β in place of c2c_2 in the empirical weights (Eq. (2)). Sec. 5.4 observed that restoring exactness of the joint estimator would require the intractable volume weights wtrue(i)w_true^(i). This section analyses the estimator Algorithm 2 actually computes and proves: • it is consistent, with an explicitly characterized limit (Theorem 7); • the limit is the exact gradient of an entropic-risk objective under a vulnerability-based reweighting of the data (Proposition 8, Corollary 9); • this objective still lower-bounds probabilistic robustness for every admissible β, with Theorem 2 recovered as the tightest member of the resulting family (Proposition 10, Corollary 11); Setup. Let =[0,1]dX=[0,1]^d. For each y∈y , the loss f(⋅,y,θ)f(·,y;θ) is continuous on the compact set X, as in Proposition 1, and 0≤f(u,y,θ)≤Fmax(θ):=maxy∈maxu∈f(u,y,θ)<∞.0≤ f(u,y;θ)≤ F_ (θ):= _y _u f(u,y;θ)<∞. For a clean pair (x,y)∈×(x,y) ×Y and t≥0t≥ 0, let Mt(x,y,θ):=U∼pdis(⋅,x)[exp(tf(U,y,θ))].M_t(x,y;θ):=E_U p_dis(·;x) [ (tf(U,y;θ) ) ]. With β∈[0,c2]β∈[0,c_2] and λ:=c2−βλ:=c_2-β, define the tilted distribution, volume ratio, and reweighted data distribution: pλ(u,x,y,θ) p_λ(u;x,y,θ) :=pdis(u,x)exp(λf(u,y,θ))Mλ(x,y,θ), := p_dis(u;x) (λ f(u,y;θ) )M_λ(x,y;θ), r(x,y,θ) r(x,y;θ) :=Mλ(x,y,θ)Mc2(x,y,θ), := M_λ(x,y;θ)M_c_2(x,y;θ), p~(x,y,θ) p(x,y;θ) ∝pdata(x,y)r(x,y,θ). p_data(x,y)\,r(x,y;θ). The last is well-defined since (x,y)∼pdata[r(x,y,θ)]∈[exp(−βFmax(θ)),1]E_(x,y) p_data [r(x,y;θ) ]∈ [ (-β F_ (θ) ),1 ] by Lemma 6 (i). Define the entropic risk Rλ(x,y,θ):=1λlogMλ(x,y,θ),λ>0,R_λ(x,y;θ):= 1λ M_λ(x,y;θ), λ>0, and R0(x,y,θ):=X∼pdis(⋅,x)[f(X,y,θ)].R_0(x,y;θ):=E_X p_dis(·;x) [f(X,y;θ) ]. The map λ↦Rλ(x,y,θ)λ R_λ(x,y;θ) is continuous at 00. As in Theorem 5, the Langevin sampler is treated as an exact sampler of padv(⋅,x,y,θ)p_adv(·;x,y,θ) (non-asymptotic guarantees for projected Langevin dynamics); all statements are at a fixed θ. Lemma 6 (Tilting and volume identities). Fix (x,y)∈×(x,y) ×Y, θ, and β∈[0,c2]β∈[0,c_2], and let λ=c2−βλ=c_2-β. Then: (i) For every t≥0t≥ 0, 1≤Mt(x,y,θ)≤exp(tFmax(θ)),1≤ M_t(x,y;θ)≤ (tF_ (θ) ), and pt(⋅,x,y,θ)p_t(·;x,y,θ) is a valid probability density. In particular, p0(⋅,x,y,θ)=pdis(⋅,x),pc2(⋅,x,y,θ)=padv(⋅,x,y,θ),p_0(·;x,y,θ)=p_dis(·;x), p_c_2(·;x,y,θ)=p_adv(·;x,y,θ), with ZadvZvic=Mc2(x,y,θ),Z_advZ_vic=M_c_2(x,y;θ), which is the identity used in Section 5.4. (i) Pointwise on X, padv(u,x,y,θ)exp(−βf(u,y,θ))=r(x,y,θ)pλ(u,x,y,θ).p_adv(u;x,y,θ) (-β f(u,y;θ) )=r(x,y;θ)\,p_λ(u;x,y,θ). (i) exp(−βFmax(θ))≤r(x,y,θ)≤1, (-β F_ (θ) )≤ r(x,y;θ)≤ 1, and r(x,y,θ)−1=U∼pλ(⋅,x,y,θ)[exp(βf(U,y,θ))].r(x,y;θ)^-1=E_U p_λ(·;x,y,θ) [ (β f(U,y;θ) ) ]. Proof. (i) The function exp(tf(⋅,y,θ)) (tf(·,y;θ) ) is continuous on the compact set X, with values in [1,exp(tFmax(θ))][1, (tF_ (θ))]. Hence 1≤Mt(x,y,θ)≤exp(tFmax(θ)),1≤ M_t(x,y;θ)≤ (tF_ (θ) ), and pt(⋅,x,y,θ)p_t(·;x,y,θ) integrates to one. The case t=0t=0 gives p0(⋅,x,y,θ)=pdis(⋅,x).p_0(·;x,y,θ)=p_dis(·;x). Since padv(u,x,y,θ)=pdis(u,x)exp(c2f(u,y,θ))ZadvZvic,p_adv(u;x,y,θ)= p_dis(u;x) (c_2f(u,y;θ) )Z_advZ_vic, integrating over X gives ZadvZvic=Mc2(x,y,θ),Z_advZ_vic=M_c_2(x,y;θ), and therefore padv(⋅,x,y,θ)=pc2(⋅,x,y,θ).p_adv(·;x,y,θ)=p_c_2(·;x,y,θ). (i) Using (i) and λ=c2−βλ=c_2-β, padv(u,x,y,θ)exp(−βf(u,y,θ)) p_adv(u;x,y,θ) (-β f(u,y;θ) ) =pdis(u,x)exp(λf(u,y,θ))Mc2(x,y,θ) = p_dis(u;x) (λ f(u,y;θ) )M_c_2(x,y;θ) =Mλ(x,y,θ)Mc2(x,y,θ)⋅pdis(u,x)exp(λf(u,y,θ))Mλ(x,y,θ) = M_λ(x,y;θ)M_c_2(x,y;θ)· p_dis(u;x) (λ f(u,y;θ) )M_λ(x,y;θ) =r(x,y,θ)pλ(u,x,y,θ). =r(x,y;θ)\,p_λ(u;x,y,θ). (i) Since f(u,y,θ)≥0f(u,y;θ)≥ 0 and λ≤c2λ≤ c_2, exp(λf(u,y,θ))≤exp(c2f(u,y,θ)). (λ f(u,y;θ) )≤ (c_2f(u,y;θ) ). Thus Mλ(x,y,θ)≤Mc2(x,y,θ),r(x,y,θ)≤1.M_λ(x,y;θ)≤ M_c_2(x,y;θ), r(x,y;θ)≤ 1. Conversely, Mc2(x,y,θ) M_c_2(x,y;θ) =U∼pdis(⋅,x)[exp(λf(U,y,θ))exp(βf(U,y,θ))] =E_U p_dis(·;x) [ (λ f(U,y;θ) ) (β f(U,y;θ) ) ] ≤exp(βFmax(θ))Mλ(x,y,θ), ≤ (β F_ (θ) )M_λ(x,y;θ), so r(x,y,θ)≥exp(−βFmax(θ)).r(x,y;θ)≥ (-β F_ (θ) ). Finally, U∼pλ(⋅,x,y,θ)[exp(βf(U,y,θ))]=Mc2(x,y,θ)Mλ(x,y,θ)=r(x,y,θ)−1.E_U p_λ(·;x,y,θ) [ (β f(U,y;θ) ) ]= M_c_2(x,y;θ)M_λ(x,y;θ)=r(x,y;θ)^-1. ∎ Part (i) makes the mechanism transparent: the local adversarial volume ZadvZvic=Mc2(x,y,θ)Z_advZ_vic=M_c_2(x,y;θ) omitted from the empirical weights resurfaces in the large-batch limit as the per-image factor r(x,y,θ)=Mλ(x,y,θ)Mc2(x,y,θ),r(x,y;θ)= M_λ(x,y;θ)M_c_2(x,y;θ), which down-weights a clean pair exactly according to the exponential moment of its loss over its perturbation neighborhood. This is the precise form of the implicit regularization described informally in Section 5.4. Theorem 7 (Batch-level consistency of the empirical gradient estimator). Let (x(i),y(i))∼i.i.d.pdata,i=1,…,B,(x^(i),y^(i)) i.i.d. p_data, i=1,…,B, and, conditionally on these clean pairs, let X(i)∼padv(⋅,x(i),y(i),θ)X^(i) p_adv (·;x^(i),y^(i),θ ) independently across i. Equivalently, the tuples (x(i),y(i),X(i))(x^(i),y^(i),X^(i)) are i.i.d. from the joint density π(x,y,u):=pdata(x,y)padv(u,x,y,θ).π(x,y,u):=p_data(x,y)\,p_adv(u;x,y,θ). Let g^Bemp(θ)=∑i=1Bw^emp(i)∇θf(X(i),y(i),θ) g_B^emp(θ)= _i=1^B w_emp^(i) _θf(X^(i),y^(i);θ) be the update of Lines 16-19 of Algorithm 2, with self-normalized weights w^emp(i)=w~emp(i)∑j=1Bw~emp(j),w~emp(i):=exp(−βf(X(i),y(i),θ)), w_emp^(i)= w_emp^(i) _j=1^B w_emp^(j), w_emp^(i):= (-β f(X^(i),y^(i);θ) ), treated as constants during backpropagation (stop-gradient, Line 17). If (x,y)∼pdata[X∼padv(⋅,x,y,θ)[‖∇θf(X,y,θ)‖]]<∞,E_(x,y) p_data [E_X p_adv(·;x,y,θ) [ \| _θf(X,y;θ) \| ] ]<∞, then, as B→∞B→∞, g^Bemp(θ)→a.s.(x,y)∼p~(⋅,⋅,θ)[X∼pλ(⋅,x,y,θ)[∇θf(X,y,θ)]]. g_B^emp(θ) a.s.E_(x,y) p(·,·;θ) [E_X p_λ(·;x,y,θ) [ _θf(X,y;θ) ] ]. Proof. Write g^Bemp(θ) g_B^emp(θ) as a ratio of sample means: g^Bemp(θ)=1B∑i=1Bw~emp(i)∇θf(X(i),y(i),θ)1B∑j=1Bw~emp(j). g_B^emp(θ)= 1B _i=1^B w_emp^(i) _θf(X^(i),y^(i);θ) 1B _j=1^B w_emp^(j). Only independence across tuples is used. It holds because the clean pairs are i.i.d. and, conditionally on them, the B Langevin chains evolve independently: the batch energy in Line 11 is separable across i, so its gradient decouples blockwise, and the noise in Line 9 is independent across i. The within-tuple dependence is exactly what the factorization π(x,y,u)=pdata(x,y)padv(u,x,y,θ)π(x,y,u)=p_data(x,y)\,p_adv(u;x,y,θ) encodes. Denominator. Since w~emp(j)∈[exp(−βFmax(θ)),1], w_emp^(j)∈ [ (-β F_ (θ) ),1 ], the weights are integrable. By the Strong Law of Large Numbers (SLLN), the tower property, and Lemma 6 (i), 1B∑j=1Bw~emp(j) 1B _j=1^B w_emp^(j) →a.s.(x,y,X)∼π[exp(−βf(X,y,θ))] a.s.E_(x,y,X) π [ (-β f(X,y;θ) ) ] =(x,y)∼pdata[X∼padv(⋅,x,y,θ)[exp(−βf(X,y,θ))]] =E_(x,y) p_data [E_X p_adv(·;x,y,θ) [ (-β f(X,y;θ) ) ] ] =(x,y)∼pdata[r(x,y,θ)] =E_(x,y) p_data [r(x,y;θ) ] ≥exp(−βFmax(θ))>0, ≥ (-β F_ (θ) )>0, where the lower bound follows from Lemma 6 (i). Numerator. Since ‖w~emp(i)∇θf(X(i),y(i),θ)‖≤‖∇θf(X(i),y(i),θ)‖, \| w_emp^(i) _θf(X^(i),y^(i);θ) \|≤ \| _θf(X^(i),y^(i);θ) \|, the integrability assumption allows the SLLN to be applied componentwise: 1B∑i=1Bw~emp(i)∇θf(X(i),y(i),θ) 1B _i=1^B w_emp^(i) _θf(X^(i),y^(i);θ) →a.s.(x,y,X)∼π[exp(−βf(X,y,θ))∇θf(X,y,θ)]. a.s.E_(x,y,X) π [ (-β f(X,y;θ) ) _θf(X,y;θ) ]. By the tower property and Lemma 6 (i), X∼padv(⋅,x,y,θ)[exp(−βf(X,y,θ))∇θf(X,y,θ)] _X p_adv(·;x,y,θ) [ (-β f(X,y;θ) ) _θf(X,y;θ) ] =r(x,y,θ)X∼pλ(⋅,x,y,θ)[∇θf(X,y,θ)]. =r(x,y;θ)\,E_X p_λ(·;x,y,θ) [ _θf(X,y;θ) ]. Thus the numerator converges almost surely to (x,y)∼pdata[r(x,y,θ)X∼pλ(⋅,x,y,θ)[∇θf(X,y,θ)]].E_(x,y) p_data [r(x,y;θ)\,E_X p_λ(·;x,y,θ) [ _θf(X,y;θ) ] ]. Ratio. On the probability-one event where both limits hold, the denominator tends to a strictly positive constant by Lemma 6 (i). The continuous mapping theorem therefore gives g^Bemp(θ) g_B^emp(θ) →a.s.(x,y)∼pdata[r(x,y,θ)X∼pλ(⋅,x,y,θ)[∇θf(X,y,θ)]](x,y)∼pdata[r(x,y,θ)] a.s. E_(x,y) p_data [r(x,y;θ)\,E_X p_λ(·;x,y,θ) [ _θf(X,y;θ) ] ]E_(x,y) p_data [r(x,y;θ) ] =∫×pdata(x,y)r(x,y,θ)(x′,y′)∼pdata[r(x′,y′,θ)] = _X×Y p_data(x,y)\,r(x,y;θ)E_(x ,y ) p_data [r(x ,y ;θ) ] ×X∼pλ(⋅,x,y,θ)[∇θf(X,y,θ)]d(x,y) ×E_X p_λ(·;x,y,θ) [ _θf(X,y;θ) ]\,d(x,y) =(x,y)∼p~(⋅,⋅,θ)[X∼pλ(⋅,x,y,θ)[∇θf(X,y,θ)]], =E_(x,y) p(·,·;θ) [E_X p_λ(·;x,y,θ) [ _θf(X,y;θ) ] ], where the last equality holds because the normalizing constant of p~(x,y,θ)∝pdata(x,y)r(x,y,θ) p(x,y;θ) p_data(x,y)\,r(x,y;θ) is exactly (x,y)∼pdata[r(x,y,θ)].E_(x,y) p_data [r(x,y;θ) ]. ∎ Relation to Theorem 5. (i) The integrability assumption is implied by the data-averaged assumption of Theorem 5. By Lemma 6 (i), padv(u,x,y,θ)pdis(u,x)=exp(c2f(u,y,θ))Mc2(x,y,θ)≤exp(c2Fmax(θ)), p_adv(u;x,y,θ)p_dis(u;x)= (c_2f(u,y;θ) )M_c_2(x,y;θ)≤ (c_2F_ (θ) ), since Mc2(x,y,θ)≥1M_c_2(x,y;θ)≥ 1. Hence (x,y,X)∼π[‖∇θf(X,y,θ)‖] _(x,y,X) π [ \| _θf(X,y;θ) \| ] ≤exp(c2Fmax(θ))(x,y)∼pdata[X∼pdis(⋅,x)[‖∇θf(X,y,θ)‖]]. ≤ (c_2F_ (θ) )\,E_(x,y) p_data [E_X p_dis(·;x) [ \| _θf(X,y;θ) \| ] ]. (i) Theorem 7 strictly generalizes Theorem 5: with a single clean pair (xori,yori)(x_ori,y_ori) and β=c2β=c_2, the factor r(xori,yori,θ)r(x_ori,y_ori;θ) is a common constant that cancels in the self-normalized ratio. Moreover, λ=0λ=0 and p0(⋅,xori,yori,θ)=pdis(⋅,xori).p_0(·;x_ori,y_ori,θ)=p_dis(·;x_ori). The limit therefore reduces to X∼pdis(⋅,xori)[∇θf(X,yori,θ)],E_X p_dis(·;x_ori) [ _θf(X,y_ori;θ) ], which is the conclusion of Theorem 5. (i) At the other endpoint β=0β=0, the weights are uniform: w^emp(i)=1B. w_emp^(i)= 1B. In this case, r(x,y,θ)=1,p~(x,y,θ)=pdata(x,y),r(x,y;θ)=1, p(x,y;θ)=p_data(x,y), and pλ(⋅,x,y,θ)=padv(⋅,x,y,θ).p_λ(·;x,y,θ)=p_adv(·;x,y,θ). This is unweighted adversarial training on Langevin samples, i.e., exactly the PAT(WOS) variant of Section 6.2. (iv) Had the intractable weights wtrue(i)∝Zadv(i)Zvic(i)exp(−c2f(X(i),y(i),θ))w_true^(i) Z_adv^(i)Z_vic^(i) (-c_2f(X^(i),y^(i);θ) ) of Section 5.4 been used, the same argument, using the identity padv(u,x,y,θ)ZadvZvicexp(−c2f(u,y,θ))=pdis(u,x)p_adv(u;x,y,θ)\,Z_advZ_vic\, (-c_2f(u,y;θ) )=p_dis(u;x) from Lemma 6, would give the limit (x,y)∼pdata[X∼pdis(⋅,x)[∇θf(X,y,θ)]],E_(x,y) p_data [E_X p_dis(·;x) [ _θf(X,y;θ) ] ], the exact joint analogue of Theorem 5. Proposition 8 (Entropic-risk gradient identity). Fix (x,y)∈×(x,y) ×Y. Assume, as in Proposition 4, that ∇θ _θ may be exchanged with integration over X. Then, for λ>0λ>0, ∇θRλ(x,y,θ)=X∼pλ(⋅,x,y,θ)[∇θf(X,y,θ)]. _θR_λ(x,y;θ)=E_X p_λ(·;x,y,θ) [ _θf(X,y;θ) ]. The identity extends to λ=0λ=0, with R0(x,y,θ)=X∼pdis(⋅,x)[f(X,y,θ)],p0(⋅,x,y,θ)=pdis(⋅,x).R_0(x,y;θ)=E_X p_dis(·;x) [f(X,y;θ) ], p_0(·;x,y,θ)=p_dis(·;x). Proof. Since pdis(⋅,x)p_dis(·;x) does not depend on θ, ∇θlogMλ(x,y,θ) _θ M_λ(x,y;θ) =λMλ(x,y,θ)∫pdis(u,x)∇θf(u,y,θ)exp(λf(u,y,θ))u = λM_λ(x,y;θ) _Xp_dis(u;x)\, _θf(u,y;θ)\, (λ f(u,y;θ) )\,du =λX∼pλ(⋅,x,y,θ)[∇θf(X,y,θ)]. =λ\,E_X p_λ(·;x,y,θ) [ _θf(X,y;θ) ]. Dividing by λ gives the claim for λ>0λ>0. ∎ Corollary 9 (What PAT optimizes in practice). Combining Theorem 7 and Proposition 8, as B→∞B→∞, g^Bemp(θ)→a.s.∇θ′(x,y)∼p~(⋅,⋅,θ)[Rλ(x,y,θ′)]|θ′=θ, g_B^emp(θ) a.s. . _θ E_(x,y) p(·,·;θ) [R_λ(x,y;θ ) ] |_θ =θ, where the gradient passes inside the outer expectation because p~(⋅,⋅,θ) p(·,·;θ) does not depend on θ′θ . Proposition 10 (The smoothed objective preserves the lower bound on probabilistic robustness). Fix a clean pair (xori,yori)(x_ori,y_ori), and let s, EadvE_adv, and γ∗>0γ^*>0 be as in Theorem 2. Then: (i) Monotone family. The map λ↦Rλ(xori,yori,θ)λ R_λ(x_ori,y_ori;θ) is continuous on [0,∞)[0,∞), and, for λ>0λ>0, dλRλ(xori,yori,θ)=1λ2KL(pλ(⋅,xori,yori,θ)∥pdis(⋅,xori))≥0. ddλR_λ(x_ori,y_ori;θ)= 1λ^2KL\! (p_λ(·;x_ori,y_ori,θ)\, \|\,p_dis(·;x_ori) )≥ 0. Hence, for every λ∈[0,c2]λ∈[0,c_2], R0(xori,yori,θ)≤Rλ(xori,yori,θ)≤Rc2(xori,yori,θ),R_0(x_ori,y_ori;θ)≤ R_λ(x_ori,y_ori;θ)≤ R_c_2(x_ori,y_ori;θ), with the exact integral representation Rλ(xori,yori,θ)−R0(xori,yori,θ)=∫0λ1t2KL(pt(⋅,xori,yori,θ)∥pdis(⋅,xori))t. R_λ(x_ori,y_ori;θ)-R_0(x_ori,y_ori;θ)= _0^λ 1t^2KL\! (p_t(·;x_ori,y_ori,θ)\, \|\,p_dis(·;x_ori) )\,dt. (i) A family of lower bounds. For every β∈[0,c2]β∈[0,c_2], i.e., every λ=c2−β∈[0,c2]λ=c_2-β∈[0,c_2], PR(pdis(⋅,xori),s)≥1−c2Rλ(xori,yori,θ)γ∗.PR\! (p_dis(·;x_ori),s )≥ 1- c_2R_λ(x_ori,y_ori;θ)γ^*. At λ=0λ=0 (i.e., β=c2β=c_2), this coincides with the bound of Theorem 2, since logZvic−H(pdis(⋅,xori)) Z_vic-H\! (p_dis(·;x_ori) ) −KL(pdis(⋅,xori)∥pvic(⋅,yori,θ)) -KL\! (p_dis(·;x_ori)\, \|\,p_vic(·;y_ori,θ) ) =X∼pdis(⋅,xori)[c2f(X,yori,θ)] =E_X p_dis(·;x_ori) [c_2f(X,y_ori;θ) ] =c2R0(xori,yori,θ). =c_2R_0(x_ori,y_ori;θ). For λ>0λ>0, the bound is looser by exactly c2γ∗(Rλ(xori,yori,θ)−R0(xori,yori,θ)), c_2γ^* (R_λ(x_ori,y_ori;θ)-R_0(x_ori,y_ori;θ) ), which is quantified by the integral representation in (i). Proof. (i) The boundedness of f(⋅,yori,θ)f(·,y_ori;θ) justifies differentiation under the integral: dλlogMλ(xori,yori,θ)=X∼pλ(⋅,xori,yori,θ)[f(X,yori,θ)]. ddλ M_λ(x_ori,y_ori;θ)=E_X p_λ(·;x_ori,y_ori,θ) [f(X,y_ori;θ) ]. Moreover, KL(pλ(⋅,xori,yori,θ)∥pdis(⋅,xori)) \! (p_λ(·;x_ori,y_ori,θ)\, \|\,p_dis(·;x_ori) ) =X∼pλ(⋅,xori,yori,θ)[λf(X,yori,θ)−logMλ(xori,yori,θ)] =E_X p_λ(·;x_ori,y_ori,θ) [λ f(X,y_ori;θ)- M_λ(x_ori,y_ori;θ) ] =λdλlogMλ(xori,yori,θ)−logMλ(xori,yori,θ)≥0. =λ ddλ M_λ(x_ori,y_ori;θ)- M_λ(x_ori,y_ori;θ)≥ 0. The quotient rule therefore gives dλ(logMλ(xori,yori,θ)λ)=1λ2KL(pλ(⋅,xori,yori,θ)∥pdis(⋅,xori)). ddλ ( M_λ(x_ori,y_ori;θ)λ )= 1λ^2KL\! (p_λ(·;x_ori,y_ori,θ)\, \|\,p_dis(·;x_ori) ). Continuity at λ=0λ=0 follows from the expansion logMλ(xori,yori,θ)=λX∼pdis(⋅,xori)[f(X,yori,θ)]+O(λ2). M_λ(x_ori,y_ori;θ)=λ\,E_X p_dis(·;x_ori) [f(X,y_ori;θ) ]+O(λ^2). The integral representation follows from the fundamental theorem of calculus, with the integrand integrable at 00 since 1t2KL(pt(⋅,xori,yori,θ)∥pdis(⋅,xori))→t↓012VarX∼pdis(⋅,xori)[f(X,yori,θ)]. 1t^2KL\! (p_t(·;x_ori,y_ori,θ)\, \|\,p_dis(·;x_ori) ) [t 0] 12Var_X p_dis(·;x_ori) [f(X,y_ori;θ) ]. (i) Jensen’s inequality gives Mλ(xori,yori,θ)≥exp(λX∼pdis(⋅,xori)[f(X,yori,θ)]),M_λ(x_ori,y_ori;θ)≥ \! (λ\,E_X p_dis(·;x_ori) [f(X,y_ori;θ) ] ), and hence Rλ(xori,yori,θ)≥R0(xori,yori,θ).R_λ(x_ori,y_ori;θ)≥ R_0(x_ori,y_ori;θ). From the proof of Theorem 2, X∼pdis(⋅,xori)[c2f(X,yori,θ)]≥γ∗I[pdis(⋅,xori),s].E_X p_dis(·;x_ori) [c_2f(X,y_ori;θ) ]≥γ^*\,I\! [p_dis(·;x_ori),s ]. Therefore, I[pdis(⋅,xori),s]≤c2R0(xori,yori,θ)γ∗≤c2Rλ(xori,yori,θ)γ∗,I\! [p_dis(·;x_ori),s ]≤ c_2R_0(x_ori,y_ori;θ)γ^*≤ c_2R_λ(x_ori,y_ori;θ)γ^*, and PR(pdis(⋅,xori),s) \! (p_dis(·;x_ori),s ) =1−I[pdis(⋅,xori),s]≥1−c2Rλ(xori,yori,θ)γ∗. =1-I\! [p_dis(·;x_ori),s ]≥ 1- c_2R_λ(x_ori,y_ori;θ)γ^*. ∎ Corollary 11 (Aggregate control of average probabilistic robustness). For each y∈y , define the per-label quantities sy(u) s_y(u) :=maxy′≠y(zy′(u)−zy(u)), := _y ≠ y (z_y (u)-z_y(u) ), Eadv(y) E_adv(y) :=u∈:sy(u)≥0, := \u :s_y(u)≥ 0 \, γ∗(y,θ) γ^*(y;θ) :=infu∈Eadv(y)c2f(u,y,θ), := _u∈ E_adv(y)c_2f(u,y;θ), with the convention inf∅=+∞ =+∞. Assume γ∗(y,θ)>0γ^*(y;θ)>0 for every y∈y . Since Y is finite, γmin∗(θ):=miny∈γ∗(y,θ)>0. _ ^*(θ):= _y γ^*(y;θ)>0. Define the average probabilistic robustness PR¯(θ):=(x,y)∼pdata[PR(pdis(⋅,x),sy)]. PR(θ):=E_(x,y) p_data [PR\! (p_dis(·;x),s_y ) ]. Then, for every β∈[0,c2]β∈[0,c_2], PR¯(θ)≥1−c2exp(βFmax(θ))γmin∗(θ)(x,y)∼p~(⋅,⋅,θ)[Rλ(x,y,θ)]. PR(θ)≥ 1- c_2 (β F_ (θ) ) _ ^*(θ)E_(x,y) p(·,·;θ) [R_λ(x,y;θ) ]. Proof. Apply Proposition 10(i) to each clean pair. Using γ∗(y,θ)≥γmin∗(θ)γ^*(y;θ)≥ _ ^*(θ) and Rλ(x,y,θ)≥0R_λ(x,y;θ)≥ 0 (since Mλ(x,y,θ)≥1M_λ(x,y;θ)≥ 1), and taking expectations over (x,y)∼pdata,(x,y) p_data, gives PR¯(θ)≥1−c2γmin∗(θ)(x,y)∼pdata[Rλ(x,y,θ)]. PR(θ)≥ 1- c_2 _ ^*(θ)E_(x,y) p_data [R_λ(x,y;θ) ]. By Lemma 6(i), exp(−βFmax(θ))≤r(x,y,θ)≤1. (-β F_ (θ) )≤ r(x,y;θ)≤ 1. Hence, (x,y)∼p~(⋅,⋅,θ)[Rλ(x,y,θ)] _(x,y) p(·,·;θ) [R_λ(x,y;θ) ] =(x,y)∼pdata[r(x,y,θ)Rλ(x,y,θ)](x,y)∼pdata[r(x,y,θ)] = E_(x,y) p_data [r(x,y;θ)\,R_λ(x,y;θ) ]E_(x,y) p_data [r(x,y;θ) ] ≥exp(−βFmax(θ))(x,y)∼pdata[Rλ(x,y,θ)]. ≥ (-β F_ (θ) )\,E_(x,y) p_data [R_λ(x,y;θ) ]. Equivalently, (x,y)∼pdata[Rλ(x,y,θ)]≤exp(βFmax(θ))(x,y)∼p~(⋅,⋅,θ)[Rλ(x,y,θ)]. _(x,y) p_data [R_λ(x,y;θ) ]≤ (β F_ (θ) )\,E_(x,y) p(·,·;θ) [R_λ(x,y;θ) ]. Combining the two bounds completes the proof. ∎ Appendix D Further Discussion on the Global Vulnerability Capacity ZvicZ_vic While our theoretical results establish KL(pdis||pvic)−logZvicKL(p_dis||p_vic)- Z_vic as a surrogate for probabilistic robustness, a deeper understanding of the term −logZvic- Z_vic reveals a fundamental trade-off in robust optimization. From the perspective of distribution expansion, the normalization constant Zvic=∫exp(c2f(u,yori,θ))uZ_vic= _X (c_2f(u,y_ori;θ))du represents the total “volume” of the model’s vulnerability across the entire input hypercube =[0,1]dX=[0,1]^d. Geometrically, the probability density pvicp_vic is a normalized representation of the classification loss. Because the domain X is compact and the integral of pvicp_vic is constrained to 11, any attempt to minimize the density (i.e., increase robustness) in the local neighborhood of xorix_ori via KL(pdis||pvic)KL(p_dis||p_vic) must be compensated by an increase in density elsewhere in the domain. This is analogous to the “water level” in a closed pool: pushing the water away from one area (the distance distribution) necessarily causes the overall water level (ZvicZ_vic) to rise if the container’s volume is fixed. This perspective implies that −logZvic- Z_vic is not merely a mathematical residue but a global regularizer that penalizes the “compression” of vulnerabilities. If the optimization focuses solely on the KL term, the model might achieve high local robustness by creating extremely sharp decision boundaries just outside the pdisp_dis support, leading to a surge in ZvicZ_vic and potential global instability. Therefore, PAT’s inherent structure forces a principled balance: it requires the model to not only push vulnerabilities away from the clean data but also to suppress the total growth of the global vulnerability volume. This justifies the use of the smoothing factor β in Equation equation 2 as a mechanism to manage this pressure and prevent catastrophic weight collapse during the mass redistribution process. Appendix E Details of the Experiments In this section, we provide the comprehensive experimental settings, hyperparameters, and evaluation protocols required to reproduce the results presented in Section 6. E.1 Basic Training Setup For all experiments conducted on the CIFAR-10 and CIFAR-100 datasets, we apply a standard data augmentation pipeline to the training set: a 4-pixel reflection padding (Pad(4, reflect)), followed by a random crop to 32×3232× 32 pixels, and a random horizontal flip with a probability of p=0.5p=0.5. We train all models for a total of Etotal=100E_total=100 epochs using a batch size of B=256B=256. The network parameters are optimized using Stochastic Gradient Descent (SGD) with a Nesterov momentum of 0.90.9 and a weight decay of 5×10−45× 10^-4. The initial learning rate is set to α=0.01α=0.01 and is scheduled using a MultiStepLR decay strategy, which multiplies the learning rate by a decay factor of γ=0.1γ=0.1 at milestones of epoch 7575 and epoch 9090. E.2 Hyperparameters of Probabilistic Adversarial Training For our proposed Probabilistic Adversarial Training (PAT) detailed in Algorithm 2, the generation of the adversarial distribution via Langevin Dynamics requires careful calibration. The precise hyperparameter configurations used across our main experiments are summarized below: • Langevin Dynamics Steps (T): 100100 iterations. • Langevin Step Size (η): 0.30.3. • Noise Scale (σ): 0.0010.001, which ensures sufficient stochastic exploration within the local neighborhood. • Gradient Clipping Threshold (ρ): 1.01.0, applied to stabilize the energy landscape exploration. • Energy Weights: We set the distance distribution weight to c1=0.3c_1=0.3 and the victim distribution (loss) weight to c2=0.42c_2=0.42. • Smoothing Factor (β): As extensively discussed in Section 5.4 and Appendix D, we instantiate the smoothing factor as β=0.001β=0.001 to prevent global weight collapse during the Self-Normalized Importance Sampling (SNIS). E.3 Baselines and Evaluation Protocol Standard Robustness Evaluation: To assess standard worst-case robustness, we evaluate the models using the well-established Projected Gradient Descent (PGD) and Carlini-Wagner (CW) attacks. Both attacks are executed for 20 iterative steps (PGD-20 and CW-20). ϵ=8/255,α=2/255ε=8/255,α=2/255. Probabilistic Robustness Evaluation: For the Probabilistic Robustness (PR) reported in Table 1, the test-time distance distributions are configured with escalating perturbation scales ϵ∈0.1,0.12,0.15,0.2ε∈\0.1,0.12,0.15,0.2\. We sample N=100N=100 perturbations per image to empirically estimate the PR objective. Baseline Training Settings: For standard adversarial training baselines such as PGD-AT and TRADES, we strictly follow their original implementations. The baselines generate training adversaries using a 10-step attack with a maximum perturbation bounded by L∞≤8/255L_∞≤ 8/255. E.4 Compute Resources All experiments, including baseline training and PAT, were implemented in PyTorch. The models were trained and evaluated on a compute cluster equipped with NVIDIA RTX 5090 GPUs. A single complete training run for PAT on CIFAR-10 takes approximately 7 minutes per epoch on a single GPU. Appendix F Limitations Probabilistic adversarial attacks generate adversarial examples from noise and therefore typically require more generation steps than conventional adversarial attacks. This leads to a higher computational cost compared with standard attack methods.