Paper deep dive
Improving Certified Robustness via Adversarial Distillation
Matteo Melis, Jesus Martinez Del Rincon, Vishal Sharma
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 7/5/2026, 7:36:31 AM
Summary
The paper introduces AD-CERT (Adversarial Distillation for Certification), a novel certified training objective designed to improve the trade-off between standard accuracy and certified robustness in neural networks. AD-CERT combines adversarial distillation (AD) from an empirically robust teacher with Interval Bound Propagation (IBP) bounds. The method uses the teacher's clean distribution as a surrogate for the adversarial lower bound, effectively interpolating between the empirical strengths of adversarial training and the formal guarantees of IBP. Experimental results demonstrate that AD-CERT achieves state-of-the-art certified performance and outperforms feature-space distillation methods.
Entities (8)
Relation Signals (4)
Matteo Melis → affiliatedwith → Queen’s University Belfast
confidence 100% · Matteo Melis School of EEECS Queen’s University Belfast
AD-CERT → combines → Adversarial Distillation
confidence 100% · we introduce AD-CERT, a certified training objective that combines adversarial distillation with an IBP upper bound.
AD-CERT → combines → Interval Bound Propagation
confidence 100% · AD-CERT, a certified training objective that combines adversarial distillation with an IBP upper bound.
Adversarial Distillation → uses → Empirically Robust Teacher
confidence 100% · distilling adversarial information over the logit space from an empirically robust teacher
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Certified training aims to produce models whose predictions can be formally verified against adversarial perturbations, typically by optimising upper bounds on the worst-case loss over an allowed perturbation set. For neural networks, certified training methods based purely on tight relaxation bounds produce networks that are amenable to certification, but sacrifice standard accuracy. Conversely, adversarial training often yields stronger empirical robustness and standard accuracy, but the resulting models are generally difficult to certify with neural network verifiers. Recently, the literature has shown that better standard-certified accuracy trade-offs can be achieved by combining adversarial training objectives with loose over-approximations based on Interval Bound Propagation (IBP), effectively interpolating between lower and upper bounds of the worst-case loss. Building on this, we introduce AD-CERT, a certified training objective that combines adversarial distillation with an IBP upper bound. We show that distilling adversarial information over the logit space from an empirically robust teacher provides an effective lower bound surrogate for certified training, with AD-CERT achieving state-of-the-art certified performance on several robustness benchmarks. Furthermore, in a unified setup, distilling adversarial information at the logit-level is shown to improve certified accuracy over a robust feature-space distillation objective by up to 5.40 percentage points.
Tags
Links
- Source: https://arxiv.org/abs/2606.31653v1
- Canonical: https://arxiv.org/abs/2606.31653v1
Trouble viewing inline? Open PDF directly →
Full Text
94,411 characters extracted from source content.
Expand or collapse full text
Improving Certified Robustness via Adversarial Distillation Matteo Melis School of EEECS Queen’s University Belfast mmelis01@qub.ac.uk &Jesus Martinez Del Rincon School of EEECS Queen’s University Belfast j.martinez-del-rincon@qub.ac.uk &Vishal Sharma School of EEECS Queen’s University Belfast V.Sharma@qub.ac.uk Corresponding author. Abstract Certified training aims to produce models whose predictions can be formally verified against adversarial perturbations, typically by optimising upper bounds on the worst-case loss over an allowed perturbation set. For neural networks, certified training methods based purely on tight relaxation bounds produce networks that are amenable to certification, but sacrifice standard accuracy. Conversely, adversarial training often yields stronger empirical robustness and standard accuracy, but the resulting models are generally difficult to certify with neural network verifiers. Recently, the literature has shown that better standard-certified accuracy trade-offs can be achieved by combining adversarial training objectives with loose over-approximations based on Interval Bound Propagation (IBP), effectively interpolating between lower and upper bounds of the worst-case loss. Building on this, we introduce AD-CERT, a certified training objective that combines adversarial distillation with an IBP upper bound. We show that distilling adversarial information over the logit space from an empirically robust teacher provides an effective lower bound surrogate for certified training, with AD-CERT achieving state-of-the-art certified performance on several robustness benchmarks. Furthermore, in a unified setup, distilling adversarial information at the logit-level is shown to improve certified accuracy over a robust feature-space distillation objective by up to 5.40 percentage points. 1 Introduction While deep neural networks have seen great success across various disciplines, adversarial examples Biggio et al. (2013); Szegedy et al. (2014); Goodfellow et al. (2015) raise important questions about their adversarial robustness, i.e., a network’s ability to preserve its prediction under small input perturbations. This is of particular concern in safety-critical domains such as autonomous driving and medical diagnostics. Motivated by this, neural network verification methods Katz et al. (2017); Ehlers (2017) aim to provide formal certificates of robustness for a given network. Neural network verifiers broadly fall into two categories, complete verifiers Katz et al. (2017); Tjeng et al. (2019), which compute exact bounds but have an exponential worst-case runtime, and incomplete verifiers Zhang et al. (2018); Singh et al. (2019), which rely on convex relaxations to obtain approximate bounds. Modern state-of-the-art verifiers Ferrari et al. (2022); Xu et al. (2021); Wang et al. (2021); Zhang et al. (2022c) typically combine the two, using convex relaxations to accelerate complete verification within a branch-and-bound framework Bunel et al. (2020). Since verification only certifies robustness after training, robust training methods are needed to produce networks that are both accurate and certifiable. Training neural networks for adversarial robustness is typically approached either empirically, through adversarial examples, or through certified training, using over-approximations of the network’s reachable set over a perturbation region. Adversarial training Madry et al. (2018) is an empirical approach that aims to improve a network’s adversarial robustness by augmenting the training objective with adversarial examples, thereby optimising a lower bound on the worst-case loss. However, adversarially trained networks are hard to formally verify and often fall short when faced with stronger or adaptive attack strategies Croce and Hein (2020); Tramer et al. (2020). In contrast, certified training directly optimises an upper bound on the worst-case loss, producing networks that are more amenable to verification, often at the cost of standard accuracy. While earlier certified training methods Mirman et al. (2018); Gowal et al. (2018); Zhang et al. (2020); Shi et al. (2021) trained networks against sound upper bounds of the robust loss, recent state-of-the-art methods Mao et al. (2023); Müller et al. (2023); De Palma et al. (2024); De Palma (2026) effectively optimise unsound approximations of the worst-case loss by combining adversarial training with loose IBP-based over-approximations Gowal et al. (2018). Due to IBP’s favourable optimisation properties Jovanović et al. (2022); Mao et al. (2024) and adversarial training’s empirical strengths, such certified training methods have been shown to provide state-of-the-art standard-certified accuracy trade-offs in recent benchmarks Mao et al. (2025). Motivated by the success of robust objectives that combine adversarial and certified training, we hypothesise that transferring adversarial knowledge from an empirically robust teacher to a certified student using knowledge distillation could further improve certified performance when paired with IBP bounds. This leads to the following contributions: (1−α)ℒKL(Pθt()∥Pθs())(1-α)\,L_KL\! (P_ _t(x)\,\|\,P_ _s(x_adv) )++αℒCE(−¯θsΔ(,y),y)α\,L_CE\! (- z _ _s(x,y),\,y )ℒAD-CERTL_AD -CERTPθs()P_ _s(x_adv)Pθt()P_ _t(x)Studentfθsf_ _sFrozen teacherfθtf_ _tPGD attack xAdversarial distillation¯θsΔ(,y) z _ _s(x,y)IBPStudentfθsf_ _sℬϵ()=[−ϵ,+ϵ]B_ε(x)=[x-ε,x+ε]Bound propagation Figure 1: Overview of AD-CERT (§3.1) loss formulation, which combines adversarial distillation (blue) with IBP (orange). • We introduce Adversarial Distillation for Certification (AD-CERT) (§3.1), a novel certified training objective that combines adversarial logit-level distillation from an empirically robust teacher with IBP bounds (depicted in Figure 1). • We show that AD-CERT is a scalarisation between a teacher-guided lower bound surrogate and a certified IBP upper bound on the robust loss (§3.2). • We present extensive experimental evaluations of AD-CERT across standard certified training benchmarks (§4.2), showing that it achieves state-of-the-art certified accuracy against current certified training methods. • We provide a systematic analysis of multiple knowledge distillation approaches for empirical endpoints (§4.3), showing that adversarial logit-level distillation preserves teacher robustness more effectively than clean or feature-space distillation over the same architecture, and translates best into certified training when combined with an IBP loss (§4.2). 2 Background Here, we discuss some necessary background for this work. Let fθf_θ denote a multi-class neural network classifier parametrised by θ such that fθ:ℝd↦ℝCf_θ:R^d R^C maps an input ∈⊆ℝdx ^d to a numerical prediction score [fθ()]i[f_θ(x)]_i for each class i∈1,2,…,Ci∈\1,2,…,C\. Let (,y)∼(x,y) denote a sample input and true label from a data distribution D and let ℬεp()B_ _p(x) denote a perturbation set based on an ℓp _p norm threat model. In this work, as standard with certified training Gowal et al. (2018); Balunovic and Vechev (2020); De Palma et al. (2022); Mao et al. (2023); Müller et al. (2023); De Palma et al. (2024); De Palma (2026), we only consider the ℓ∞ _∞ case, and therefore put ℬε():=′:‖′−‖∞≤εB_ (x):=\x :\|x -x\|_∞≤ \, where ′x denotes the perturbed input and ε is the perturbation radius. 2.1 Neural Network Verification A network is said to be adversarially robust for an input x if for all ′∈ℬε()x _ (x), we have [fθ(′)]i<[fθ(′)]y∀i≠y[f_θ(x )]_i<[f_θ(x )]_y\ ∀\ i≠ y. Let θΔ∈ℝC−1z _θ ^C-1 denote the logit-difference between the true class and all other classes, i.e., θΔ(,y):=([fθ()]y−[fθ()]i)i≠yz _θ(x,y):=([f_θ(x)]_y-[f_θ(x)]_i)_i≠ y. Then, a network’s adversarial robustness can be verified by solving: min′∈ℬε() mini≠y[θΔ(′,y)]i, _x _ (x) _i≠ y[z _θ(x ,y)]_i, (1) and checking if the solution to Equation (1) is greater than zero. Finding an exact solution to Equation (1) was shown to be NP-hard Katz et al. (2017), thus, neural network verifiers either approximate the bounds Zhang et al. (2018); Singh et al. (2019), or perform complete verification at a larger computational cost Katz et al. (2017); Tjeng et al. (2019); Ferrari et al. (2022); Xu et al. (2021); Wang et al. (2021); Zhang et al. (2022c). 2.2 Training for Robustness Standard training of neural network classifiers optimises network parameters θ by minimising the expected cross-entropy loss, i.e., minθ(,y)∼[ℒCE(fθ(),y)], _θE_(x,y) [L_CE (f_θ(x),y ) ], (2) where ℒCE(fθ(),y)=log(1+∑i≠yCexp([fθ()]i−[fθ()]y))L_CE(f_θ(x),y)= (1+ _i≠ y^C ([f_θ(x)]_i-[f_θ(x)]_y)). In contrast, training for robustness can be viewed as the following min-max objective: minθ(,y)∼[max′∈ℬε()ℒCE(fθ(′),y)]. _θE_(x,y) [ _x _ (x)L_CE (f_θ(x ),y ) ]. (3) Concretely, for an input x, the inner maximisation finds the most malicious perturbation ′∈ℬε()x _ (x) that maximises the loss. The outer minimisation seeks the optimal network parameters θ such that these threats are less effective against the network predictions. Since the inner maximisation problem is non-convex, we generally solve this by under- or over-approximating the worst-case loss. In general, if we let ℒrob∗(fθ(),y):=max′∈ℬε()ℒ(fθ(′),y)L^*_rob(f_θ(x),y):= _x _ (x)L (f_θ(x ),y ) denote a network objective solved for the optimisation problem in Equation (3), then we have: ℒ¯rob(fθ(),y)≤ℒrob∗(fθ(),y)≤ℒ¯rob(fθ(),y), L_rob(f_θ(x),y) ^*_rob(f_θ(x),y)≤ L_rob(f_θ(x),y), (4) with ℒ¯rob(fθ(),y) L_rob(f_θ(x),y) and ℒ¯rob(fθ(),y) L_rob(f_θ(x),y) corresponding to adversarial and certified training, respectively, as described below. Adversarial Training (AT) AT first approximates a solution to the inner maximisation problem of Equation (3) by performing an adversarial attack, such as projected gradient descent (PGD) (Madry et al., 2018), within ℬε()B_ (x) for each input x. Following this, the outer minimisation problem of Equation (3) is solved by augmenting the network’s training objective with the adversarial example ∈ℬε()x_adv _ (x) obtained by the attack. In its standard form, AT aims to minimise: ℒAT:=ℒCE(fθ(),y).L_AT:=L_CE (f_θ(x_adv),y ). (5) This can be seen as optimising a lower bound of the worst-case loss, i.e., ℒ¯rob(fθ(),y) L_rob(f_θ(x),y) from Equation (4). While AT leads to strong empirical robustness, it produces complex networks that are difficult to formally verify and remain susceptible to stronger attacks Tramer et al. (2020); Croce and Hein (2020). Certified Training and Bound Propagation Certified training optimises an upper bound of the inner-maximisation problem in Equation (3) by over-approximating the worst-case loss. A simple approach is Interval Bound Propagation (IBP) Gowal et al. (2018), leveraging interval arithmetic to approximate the output range of a given layer. Given an input x, IBP propagates the upper and lower bounds of the perturbed input space, ℬε()B_ (x), (corresponding to ¯:=+ε x:=x+ and ¯:=−ε x:=x- , respectively) through the entire network layer by layer, providing a hyper-rectangle that is monotonically increasing every layer. The final layer output of this propagation, which we denote with Box, is an over-approximation of the possible output space, i.e., Box ⊇fθ(ℬε()) f_θ(B_ (x)). Hence, using the Box relaxation, we can compute the worst-case logit difference between the true class and all other classes by putting ¯θΔ(,y):=([fθ(′)]¯y−[fθ(′)]¯i)i≠y z_θ (x,y):= ( [f_θ(x )]_y- [f_θ(x )]_i )_i≠ y and obtain a robust loss given by: ℒIBP:=ℒCE(−¯θΔ(,y),y),L_IBP:=L_CE(- z_θ (x,y),y), (6) where ℒIBPL_IBP is a sound upper bound, ℒ¯rob(fθ(),y) L_rob(f_θ(x),y), from Equation (4). Further details on IBP can be found in Appendix A. Interestingly, alternative certified training methods that utilise tighter relaxations, e.g., linear relaxations Wong and Kolter (2018); Zhang et al. (2020), consistently yield worse performance than imprecise IBP-based training. Jovanović et al. (2022) attribute this phenomenon to tighter relaxations adding discontinuity and difficulty to the optimisation objective. Moreover, Mao et al. (2024) explicitly show that IBP-based training leads to better propagation tightness, while non-IBP-based methods do not. Additionally, they note that over-regularisation provides worse standard-certified accuracy trade-offs. In line with these findings, all current state-of-the-art certified training methods Müller et al. (2023); Mao et al. (2023); De Palma et al. (2024); De Palma (2026) move away from previous over-regularised approaches Gowal et al. (2018); Balunovic and Vechev (2020); De Palma et al. (2022) and instead shift their focus to combining AT with IBP-based training, achieving state-of-the-art standard-certified accuracy trade-offs while preserving certification properties. De Palma et al. (2024) further elucidate this in their definition of expressive losses whereby expressivity is attained through interpolation between lower and upper bounds of Equation (4) during training, providing an unsound but favourable solution to Equation (3). 2.3 Knowledge Distillation Hinton et al. (2015) introduced knowledge distillation as a method for transferring knowledge from a large teacher model, TθtT_ _t, to a smaller, more compact student model, SθsS_ _s. This allows the student to retain much of the teacher’s performance while reducing computational cost. Knowledge distillation utilises Kullback-Leibler (KL) divergence to compare the predictive distributions of the student and teacher, which is defined by: KL(Pθtτ()∥Pθsτ())=∑i=1C[Pθtτ()]ilog([Pθtτ()]i[Pθsτ()]i),KL\! (P^τ_ _t(x)\,\|\,P_ _s^τ(x) )=Σ^C_i=1[P_ _t^τ(x)]_i ( [P_ _t^τ(x)]_i[P_ _s^τ(x)]_i ), (7) where τ is a temperature applied to the softmax operation and Pθtτ():=softmax(Tθt()τ)P_ _t^τ(x):=softmax ( T_ _t(x)τ ), Pθsτ():=softmax(Sθs()τ)P_ _s^τ(x):=softmax ( S_ _s(x)τ ) are the respective predictive distributions of the teacher and student after softmax. Knowledge distillation then optimises the following objective: minθs(,y)∼[(1−α)ℒCE(Sθs(),y)+ατ2KL(Pθtτ()∥Pθsτ())], _ _sE_(x,y) [(1-α)L_CE (S_ _s(x),y )+ατ^2KL\! (P_ _t^τ(x)\,\|\,P_ _s^τ(x) ) ], (8) where α∈[0,1]α∈[0,1] is a balancing factor between standard training and learning the teacher’s predictive distribution. Knowledge distillation has since been applied in the AT setting to boost empirical robustness of a student using an adversarially trained teacher Goldblum et al. (2020); Zi et al. (2021); Zhu et al. (2022); Cui et al. (2024). This was first explored by Goldblum et al. (2020), through adversarially robust distillation (ARD), which injects adversarial examples into the KL term from Equation (8). More specifically: ℒARD:=(1−α)ℒCE(Sθs(),y)+ατ2KL(Pθtτ()∥Pθsτ()).L_ARD:=(1-α)L_CE (S_ _s(x),y )+ατ^2KL\! (P_ _t^τ(x)\,\|\,P_ _s^τ(x_adv) ). (9) Additionally, De Palma (2026) recently introduced knowledge distillation within the field of certified training, where knowledge is distilled from an empirically robust teacher over the feature space (further details in Appendix G). In the coming section, we build on previous certified training Gowal et al. (2018); Balunovic and Vechev (2020); De Palma et al. (2022); Mao et al. (2023); Müller et al. (2023); De Palma et al. (2024); De Palma (2026) and adversarial knowledge distillation Goldblum et al. (2020); Zi et al. (2021) works, presenting a novel certified training objective that distils the predictive distribution of an empirically robust teacher onto a student training for certifiable robustness. 3 Methodology In this section, we introduce Adversarial Distillation for Certification (AD-CERT), a novel certified training scheme that interpolates between adversarial distillation (AD) and IBP Gowal et al. (2018). We leverage the key insight that replacing a hard-label adversarial lower bound with a teacher-guided surrogate could further improve certified training. In this way, the empirical teacher shapes the student on concrete adversarial examples, while certification is enforced directly through IBP. Full proofs and additional theoretical results are deferred to Appendices B and C, respectively. Pseudocode for the complete training procedure is provided in Appendix E and further details on AD-CERT’s computational cost can be found in Appendix F.5, where AD-CERT corresponds to performing IBP and PGD training, with the slight added cost of a single forward pass on the detached teacher. 3.1 AD-CERT Formulation Going forward, we denote by fθtf_ _t a fixed teacher network trained via AT, and let fθsf_ _s be the student model that we aim to certify. Note that both the student and teacher share the same architecture in this distillation setting. Moreover, for all experiments, we use a temperature setting of τ=1τ=1, and henceforth, omit the τ notation from all softmax operations, writing Pθ():=softmax(fθ())P_θ(x):=softmax(f_θ(x)) for the predictive distribution of a network. Excluding additional regularisation, AD-CERT attains the following loss objective: ℒAD-CERT:=(1−α)KL(Pθt()∥Pθs())⏟Adversarial Distillation+αℒCE(−¯θsΔ(,y),y)⏟IBP,α∈[0,1].L_AD -CERT:=(1-α) KL (P_ _t(x)\,\|\,P_ _s(x_adv) )_Adversarial Distillation+α L_CE (- z_ _s (x,y),y )_IBP, α∈[0,1]. (10) The AD component of Equation (10) is exactly the KL portion of Equation (9), where the student is evaluated on an adversarially perturbed input, and tasked to match the clean distribution of an adversarially trained teacher, aiming to preserve standard and empirically robust accuracy. On an intuitive level, AD-CERT is designed to keep the teacher signal completely outside of verified bounds, using it purely for its empirical strengths, while leveraging IBP to tighten the worst-case logit differences. This aims to make the student amenable to verification, while still retaining a large proportion of the teacher’s adversarial and standard accuracy. 3.2 Teacher-Guided Lower Bound Surrogates In this subsection, we take an analytical view of the objective in Equation (10) and formalise the role of AD within certified training. Unlike AT and IBP, which, respectively, provide formal lower and upper bounds from Equation (4), the AD term in AD-CERT is used as a teacher-guided surrogate for the adversarial lower bound. This is an intentional relaxation of the lower bound objective, using an empirical teacher only to improve the adversarial endpoint, while leaving the verifiable IBP upper bound objective unchanged. We begin by showing that, up to a teacher-dependent constant, the distillation component of AD-CERT is exactly AT with soft teacher labels. Proposition 3.1 (Equivalence of AD and soft-label AT). Let x_adv be fixed, and let the teacher predictive distribution Pθt()P_ _t(x) be fixed. Then the AD objective: ℒAD:=KL(Pθt()∥Pθs()),L_AD:=KL\! (P_ _t(x)\,\|\,P_ _s(x_adv) ), (11) is equal, up to an additive constant independent of the student parameters, to AT with soft teacher labels, namely: ℒsoft-AT:=ℒCE(fθs(),Pθt()).L_soft -AT:=L_CE\! (f_ _s(x_adv),\,P_ _t(x) ). (12) Moreover, both objectives induce identical backpropagation updates with respect to the student logits. In particular, for each class index i: ∂ℒAD∂zθsi=[Pθs()]i−[Pθt()]i=∂ℒsoft-AT∂zθsi. _AD∂ z_ _s^i=[P_ _s(x_adv)]_i-[P_ _t(x)]_i= _soft -AT∂ z_ _s^i. (13) Proof. See Appendix B. ∎ Given Proposition 13, we have shown that the distillation component of AD-CERT preserves AT at the gradient level, while being conditioned by an empirical teacher. We further elucidate this connection through an analytical view of our loss function from Equation (10). Proposition 3.2 (Analytical form of AD-CERT). Let H(P)H(P) and H(Q,P)H(Q,P) denote the entropy and cross entropy respectively for probability distributions P and Q. That is: H(P)=−∑i=1Cpilog(pi),H(Q,P)=−∑i=1Cqilog(pi).H(P)=- _i=1^Cp_i (p_i), H(Q,P)=- _i=1^Cq_i (p_i). (14) Then, ℒAD-CERTL_AD -CERT from Equation (10) can be re-written as: ℒAD-CERT=(1−α)H(Pθt(),Pθs())+αℒIBP−(1−α)H(Pθt()).L_AD -CERT=(1-α)H(P_ _t(x),P_ _s(x_adv))+ _IBP-(1-α)H(P_ _t(x)). (15) Therefore, up to a constant independent of θs _s, AD-CERT is a scalarisation between (i) a soft-label adversarial lower bound surrogate and (i) a certified IBP upper bound. Proof. See Appendix B. ∎ Given Proposition 15, AD-CERT fits naturally within the broader class of modern certified training objectives Müller et al. (2023); Mao et al. (2023); De Palma et al. (2024); De Palma (2026) that combine empirical and certifiable endpoints. Its distinguishing feature is that the adversarial endpoint is no longer a formal lower bound of Equation (3), but a relaxed surrogate induced by AD from an empirically robust teacher. Additionally, this positions AD-CERT as distinct and complementary to the recent work of C-DIST De Palma (2026), which computes both IBP bounds and adversarial examples within their robust feature-space distillation objective. Instead, AD-CERT integrates teacher distillation directly into the lower bound endpoint, while keeping IBP bounds purely for the certified endpoint. While Proposition 15 characterises AD-CERT at the level of its objective, the next result highlights a key geometric difference between soft-label and hard-label adversarial supervision. In particular, the AD branch of AD-CERT induces a finite target in the logit space, whereas standard AT only approaches its optimum as the correct class margin grows indefinitely. Lemma 3.3. For AD, the optimal logit margin, Δzij∗ z_ij^*, between any two classes i,ji,j is: Δzij∗=log([Pθt()]i[Pθt()]j). z_ij^*= ( [P_ _t(x)]_i[P_ _t(x)]_j ). (16) Moreover, standard AT has no finite minimiser, and its infimum 0 is approached only as [fθs()]y−[fθs()]j→+∞,∀j≠y[f_ _s(x_adv)]_y-[f_ _s(x_adv)]_j→+∞, ∀ j≠ y. Proof. See Appendix B. ∎ Lemma 16 suggests that teacher-guided soft-label supervision may be a principled approach to enhance certified training, since it arguably induces a smoother lower bound objective, not pushing for arbitrarily large class margins. We evaluate our hypotheses experimentally in the next section. 4 Experimental Evaluation In this section, we conduct experiments to assess the effectiveness of our proposed certified training method, AD-CERT (§3.1). Across our experiments, we report standard, certified and AutoAttack (A) Croce and Hein (2020) accuracy. Certified accuracy is the primary metric used in certified training literature Gowal et al. (2018); Balunovic and Vechev (2020); De Palma et al. (2022); Müller et al. (2023); Mao et al. (2023); De Palma et al. (2024); Mao et al. (2025); De Palma (2026), defined as the proportion of the validation set for which a verifier proves robustness within ℬε()B_ (x). Standard accuracy is the clean accuracy on the validation set without perturbation. A accuracy refers to the proportion of the validation set that could not be attacked by the A strategy. As such, A accuracy is strictly greater than or equal to certified accuracy. 4.1 Settings We implement AD-CERT on top of CTBench Mao et al. (2025), a unified certified training library using PyTorch Paszke et al. (2019). We adopt three datasets, MNIST LeCun et al. (2010), CIFAR-10 Krizhevsky and Hinton (2009) and TinyImageNet Le and Yang (2015). We train on the corresponding training sets and certify on the validation sets. To align with recent literature Shi et al. (2021); Müller et al. (2023); Mao et al. (2023); De Palma et al. (2024); Mao et al. (2025); De Palma (2026), we use a 7-layer convolutional network (CNN7CNN7) for both the teacher and student models, and the perturbation radii applied are ε=0.1,0.3 =\0.1,0.3\ for MNIST, ε=2255,8255 =\ 2255, 8255\ for CIFAR-10 and ε=1255 =\ 1255\ for TinyImageNet. For verification, we use the α,β-CROWN library Zhang et al. (2018); Xu et al. (2020, 2021); Wang et al. (2021), which combines linear bound propagation with branch-and-bound Bunel et al. (2020) for complete neural network verification. Full details on training of both student and teacher models, certification and the experimental setup can be found in Appendix F. 4.2 Comparison with State-of-the-Art Table 1 compares the performance of AD-CERT (§3.1) to current state-of-the-art certified training methods Gowal et al. (2018); Zhang et al. (2022b); Müller et al. (2023); Mao et al. (2023); De Palma et al. (2024); De Palma (2026). For each method, we report the standard and certified accuracy on the validation set. When available, we use the benchmark results from CTBench Mao et al. (2025), as they generally report the strongest certified accuracies of each reported certified training method compared to the original paper Gowal et al. (2018); Zhang et al. (2022b); Müller et al. (2023); Mao et al. (2023); De Palma et al. (2024); De Palma (2026), while also providing a fair comparison to AD-CERT, which is built on top of the unified library. Overall, AD-CERT achieves state-of-the-art certified accuracy for ReLU-based architectures across all evaluated settings, while retaining competitive standard accuracy. On MNIST, AD-CERT obtains modest but consistent improvements over prior methods at both perturbation radii. While the gains are small, this is expected given the relative simplicity of MNIST and the already saturated performance of existing methods. For CIFAR-10 and TinyImageNet, AD-CERT improves the best prior ReLU-based certified accuracy by an average of 0.370.37 percentage points, with the largest gain observed at ε=2255 = 2255 on CIFAR-10. For CIFAR-10 at ε=8255 = 8255, SORTNET Zhang et al. (2022b) obtains higher certified accuracy, however, it relies on a specialised 1-Lipschitz architecture known to perform particularly well on this setting. We therefore distinguish it from the ReLU-based methods, with further discussion in related work (§5). On TinyImageNet, AD-CERT achieves the highest certified accuracy among all compared methods, while substantially improving standard accuracy compared to previous non-distillation-based certified training methods Gowal et al. (2018); Müller et al. (2023); Mao et al. (2023); De Palma et al. (2024). This suggests that AD-CERT scales favourably beyond the smaller MNIST and CIFAR-10 settings, where certified training is often closer in performance. An interesting comparison is that of AD-CERT and C-DIST De Palma (2026), the only distillation-based certified training methods. In general, C-DIST retains higher standard accuracy, while AD-CERT obtains higher certified accuracy. We investigate this trade-off further through a controlled comparison in Table 2. Table 1: Comparison of AD-CERT with prior state-of-the-art certified training methods. Bold indicates the best standard or certified accuracy on each benchmark. When the best ReLU-based architecture result differs from the best overall result, the top ReLU-based accuracy is bold-underlined. Dataset ε Method Source Std. acc. [%] Cert. acc. [%] MNIST 0.10.1 AD-CERT Ours 99.25 98.53 IBP Mao et al. (2025) 98.87 98.26 SORTNET† Zhang et al. (2022b) 99.01 98.14 SABR Mao et al. (2025) 99.08 98.40 TAPS/STAPS‡ Mao et al. (2025) 99.16 98.52 MTL-IBP Mao et al. (2025) 99.18 98.37 0.30.3 AD-CERT Ours 98.85 93.98 IBP Mao et al. (2025) 98.54 93.80 SORTNET† Zhang et al. (2022b) 98.46 93.40 SABR Mao et al. (2025) 98.66 93.68 TAPS/STAPS‡ Mao et al. (2025) 98.56 93.95 MTL-IBP Mao et al. (2025) 98.74 93.90 CIFAR-10 2255 2255 AD-CERT Ours 79.21 65.08 IBP Mao et al. (2025) 67.49 55.99 SORTNET† Zhang et al. (2022b) 67.72 56.94 SABR Mao et al. (2025) 77.86 63.61 TAPS/STAPS‡ Mao et al. (2025) 77.05 64.21 MTL-IBP Mao et al. (2025) 78.82 64.41 C-DIST De Palma (2026) 81.55 64.60 8255 8255 AD-CERT Ours 53.69 35.84 IBP Mao et al. (2025) 48.51 35.28 SORTNET† Zhang et al. (2022b) 54.84 40.39 SABR Mao et al. (2025) 52.71 35.34 TAPS/STAPS‡ Mao et al. (2025) 49.96 35.25 MTL-IBP Mao et al. (2025) 54.28 35.41 C-DIST De Palma (2026) 55.13 35.52 TinyImageNet 1255 1255 AD-CERT Ours 40.06 28.19 IBP Mao et al. (2025) 26.77 19.82 SORTNET† Zhang et al. (2022b) 25.69 18.18 SABR Mao et al. (2025) 30.58 20.96 TAPS/STAPS‡ Mao et al. (2025) 30.63 22.31 MTL-IBP Mao et al. (2025) 35.97 27.73 C-DIST De Palma (2026) 43.78 27.88 † Non-ReLU-based architecture. ‡ We report the stronger of TAPS and STAPS based on certified accuracy. Comparison of Distillation-based Certified Training Methods Table 2 compares AD-CERT against alternative distillation-based certified training methods under the exact same experimental setup for a direct comparison. This controlled comparison allows us to isolate the effect of the distillation mechanism itself by training and evaluating all methods in the CTBench library Mao et al. (2025) with the same teacher model, training pipeline and verifier. We compare AD-CERT with C-DIST De Palma (2026), which performs robust feature-space distillation. Additionally, we introduce RSLAD-CERT, an alternative logit-level objective obtained by replacing the adversarial distillation term in AD-CERT with RSLAD Zi et al. (2021), but retaining the same IBP interpolation, i.e., LRSLAD-CERT:=(1−α)LRSLAD+αLIBPL_RSLAD -CERT:=(1-α)L_RSLAD+α L_IBP. For all three methods, we use the exact same hyperparameter values as those used for AD-CERT in Table 1, which we report in detail in Appendix F.3, Table 6. The only exception is wrobw_rob for TinyImageNet, where we used an earlier untuned value of 0.70.7, as per MTL-IBP’s De Palma et al. (2024) reported value in CTBench Mao et al. (2025). Overall, the results support the design choice of AD-CERT, showing that directly distilling the teacher’s predictive distribution at the adversarial endpoint provides a simple and effective empirical branch for certified training. The addition of the clean distillation term in RSLAD-CERT reduces certifiability by 0.60.6 percentage points on average, while only improving standard accuracy by an average of 0.380.38 percentage points compared to AD-CERT. Consistent with the pattern observed in Table 1, C-DIST is generally strongest at transferring standard accuracy, but shows weaker certified accuracy under the controlled setup. This results in worse standard-certified accuracy trade-offs, with C-DIST improving standard accuracy over AD-CERT by an average of 1.691.69 percentage points, but reducing certifiability by 3.523.52 percentage points. Both logit-level objectives, i.e., RSLAD-CERT and AD-CERT, improve certified accuracy over the feature-space distillation objective of C-DIST, despite not matching the formal definition of expressivity De Palma et al. (2024). One possible explanation is that C-DIST induces a more regularised objective, since it computes adversarial examples and IBP bounds during distillation, before being coupled with another expressive loss C-IBP De Palma et al. (2024). This suggests that tightly coupling distillation-based certified training objectives with the formal definition of expressivity may not translate into stronger certification properties. In contrast, AD-CERT keeps the distillation branch as a pure adversarial endpoint, while leaving the certified component as the standard IBP loss. This relaxes the definition of expressivity by extending it to surrogate lower bounds (§3.2), prioritising simplicity and smoothness in the loss objective. Finally, we acknowledge that further exhaustive hyperparameter and teacher tuning on C-DIST could potentially reduce the gap between the results reported in Table 2 and those in Table 1. However, the most important shared hyperparameters, namely the IBP coefficient α and L1L_1 regularisation, coincide with the original C-DIST setup De Palma (2026) (Table 3). The main remaining difference is therefore the teacher model. For completeness, we include an additional ablation on the sensitivity of AD-CERT and C-DIST to different teachers in Appendix D, which shows a similar trend to the above ablation. Table 2: Comparison of AD-CERT against distillation-based certified training methods under the same experimental setup. Dataset ε Method Method source† Dist. space Std. [%] A [%] Cert. [%] CIFAR-10 2255 2255 AD-CERT Ours Logits 79.21 68.38 65.20 RSLAD-CERT Ours Logits 79.87 68.27 64.36 C-DIST De Palma (2026) Features 81.04 69.97 59.80 8255 8255 AD-CERT Ours Logits 53.49 36.46 35.60 RSLAD-CERT Ours Logits 53.95 36.50 35.19 C-DIST De Palma (2026) Features 54.32 35.69 34.55 TinyImageNet 1255 1255 AD-CERT Ours Logits 41.38 30.68 27.68 RSLAD-CERT Ours Logits 41.39 30.90 27.14 C-DIST De Palma (2026) Features 43.79 32.66 23.57 † The “Method source” column indicates the origin of the training objective, all results are from our implementation and evaluation in the CTBench library Mao et al. (2025). 4.3 Ablations & Further Discussions Teacher-Student Comparison Table 3 compares each AD-CERT student with its corresponding empirically robust teacher. Overall, AD-CERT converts models that are empirically robust but not certifiable into students with strong certified accuracy, while retaining a substantial fraction of the teacher’s standard and adversarial performance. This effect is particularly clear on MNIST, where AD-CERT stays within one percentage point of the teacher in both standard and A accuracy across both perturbation radii, while introducing strong certifiability. On TinyImageNet, the teacher-student gap remains moderate, with losses below seven percentage points in standard accuracy and close to two percentage points in A accuracy, though notably achieving 28.19%28.19\% in certified accuracy. The largest gap occurs on CIFAR-10 at ε∞=8255 _∞= 8255, where the student sees a decrease of almost twenty percentage points in standard accuracy relative to the teacher, although the drop in A accuracy is much smaller. Nevertheless, AD-CERT still achieves the strongest certified accuracy among ReLU-based architectures in this setting. However, this highlights that closing the remaining gap between empirically robust teachers and certifiably robust students at larger perturbation radii remains an important direction for future work. Motivated by this observation, we next ablate different pure AD techniques on CIFAR-10, aiming to better understand the difficulty of transferring empirical robustness into certifiable robustness and to identify potential directions for stronger distillation-based certified training objectives in future. Table 3: Comparison between teacher and student models of AD-CERT for each dataset entry in Table 1. Deltas are reported relative to the teacher at the same perturbation radius. Dataset ε∞ _∞ Model Std. acc. [%] Δ Std. A acc. [%] Δ A Cert. acc. [%] MNIST 0.10.1 Teacher 99.47 – 98.97 – ≈0†≈ 0 Student 99.25 -0.22 98.55 -0.42 98.53 0.30.3 Teacher 99.43 – 93.83 – ≈0†≈ 0 Student 98.85 -0.58 94.64 +0.81 93.98 CIFAR-10 2255 2255 Teacher 88.67 – 72.41 – ≈0†≈ 0 Student 79.21 -9.46 68.39 -4.02 65.08 8255 8255 Teacher 73.55 – 41.99 – ≈0†≈ 0 Student 53.69 -19.86 36.69 -5.30 35.84 TinyImageNet 1255 1255 Teacher 46.78 – 33.16 – ≈0†≈ 0 Student 40.06 -6.72 31.14 -2.02 28.19 † None of the first 10 samples were verified within a 1000 second time-limit. Exploring Adversarial Distillation Techniques Table 4 compares several distillation objectives on CIFAR-10 Krizhevsky and Hinton (2009) after removing any certified training component and trained against a PGD-trained teacher on the same architecture. This ablation isolates the empirical distillation endpoint, allowing us to study which teacher-student objectives best preserve the robustness of the PGD teacher before introducing certifiability. Natural knowledge distillation Hinton et al. (2015), denoted Nat-KL, transfers the teacher’s clean predictive distribution to the student. AD corresponds to removing the IBP term from AD-CERT (§3.1), while RSLAD Zi et al. (2021) extends adversarially robust distillation Goldblum et al. (2020) using soft-labels. Finally, adversarial feature-space knowledge distillation (AF-KD) corresponds to setting α=0α=0 in C-DIST De Palma (2026), thereby removing the IBP components from the objective. For RSLAD and AF-KD, we use λ=56λ= 56 and β=5kβ= 5k, respectively, following the suggestions of the original works Zi et al. (2021); De Palma (2026), where k denotes the dimensionality of the feature space. From the results, it can be observed that Nat-KL improves standard accuracy at both perturbation radii, but dramatically fails to transfer adversarial robustness, losing more than 4141 percentage points in A accuracy in both settings. This confirms that matching an adversarially trained teacher only on clean inputs is insufficient for transferring adversarial robustness. In contrast, objectives that expose the student to adversarial inputs preserve teacher-level A accuracy much more effectively. Across both perturbation radii, AD and RSLAD remain close to the teacher’s A accuracy, while AF-KD is competitive at the smaller radius but degrades more noticeably at ε∞=8255 _∞= 8255. Among these methods, AD provides the most direct and consistently effective adversarial endpoint, improving teacher-level A accuracy at the smaller radius and remaining within one percentage point at the larger radius. These results further motivate our use of AD as the empirical endpoint of AD-CERT, since it provides a simple but effective mechanism for retaining adversarial accuracy from an empirically robust teacher, while remaining straightforward to combine with an IBP-certified upper bound. This yields a loss objective that resembles previous state-of-the-art expressive losses De Palma et al. (2024), but in a distillation setting with lower bound surrogates, as discussed in §3.2. Table 4: Effect of different AD techniques without a certification component on CIFAR-10. Method Objective Nat-KL KL(Pθt()∥Pθs())KL\! (P_ _t(x)\,\|\,P_ _s(x) ) AD† KL(Pθt()∥Pθs())KL\! (P_ _t(x)\,\|\,P_ _s(x_adv) ) RSLAD‡ (1−λ)KL(Pθt()∥Pθs())+λKL(Pθt()∥Pθs())(1-λ)KL\! (P_ _t(x)\,\|\,P_ _s(x) )+ \! (P_ _t(x)\,\|\,P_ _s(x_adv) ) AF-KD† ℒCE(−θs(,y),y)+β‖hθs()−hθt()‖22L_CE\! (-z_ _s(x_adv,y),y )+β \|h_ _s(x_adv)-h_ _t(x) \|_2^2 Dataset ε∞ _∞ Method Std. acc. [%] Δ Std. A acc. [%] Δ A CIFAR-10 2255 2255 PGD Teacher 88.67 – 72.41 – Nat-KL 89.89 +1.22 30.72 -41.69 AD† 88.64 -0.03 73.62 +1.21 RSLAD‡ 87.88 -0.79 73.32 +0.91 AF-KD† 89.52 +0.85 73.55 +1.14 8255 8255 PGD Teacher 78.95 – 42.48 – Nat-KL 81.32 +2.37 0.52 -41.96 AD† 77.08 -1.87 41.64 -0.84 RSLAD‡ 76.47 -2.48 42.38 -0.10 AF-KD† 78.95 0.00 37.49 -4.99 † IBP terms removed from AD-CERT/C-DIST, i.e., α=0α=0. ‡ Adversarial examples are generated against soft-labels as in original work Zi et al. (2021). IBP Coefficient Sensitivity Analysis In certified training, it is well known that higher certified accuracy often comes at the cost of standard accuracy. Recent methods fall into a family of expressive losses De Palma et al. (2024), e.g., SABR Müller et al. (2023), C-IBP & MTL-IBP De Palma et al. (2024), and C-DIST De Palma (2026), including hyperparameters that regulate this trade-off by interpolating between empirical and certifiable training objectives. Our goal in this study is to empirically support our methodology claim (§3.2) that the AD term in AD-CERT provides a sensible teacher-guided surrogate for the lower bound endpoint used in prior expressive losses Müller et al. (2023); De Palma et al. (2024); De Palma (2026). Figure 2 shows the trade-off between standard, A, and certified accuracy for AD-CERT on TinyImageNet with ε=1255 = 1255 as we decrease the IBP coefficient, α, from Equation 10. Consistent with the sensitivity analyses of Müller et al. (2023) (Figure 7), De Palma et al. (2024) (Figure 1) and Mao et al. (2025) (Figure 7), standard and A accuracy increase as the strength of IBP regularisation is reduced. Up to a point, this also benefits certified accuracy, before the model starts to dramatically lose its certification properties when α is decreased beyond this. This behaviour suggests that the adversarial distillation endpoint in AD-CERT acts analogously to the hard-label adversarial endpoints used in prior expressive losses, while also achieving state-of-the-art certified accuracy (§4.2). Figure 2: Sensitivity analysis of AD-CERT with respect to the IBP coefficient, α, on TinyImageNet at ϵ=1/255ε=1/255. We report standard, A, and certified accuracy over eight values of α∈[5×10−4,5×10−2]α∈[5× 10^-4,5× 10^-2]. 5 Related Work Certified Training As previously mentioned (§2.2), IBP Mirman et al. (2018); Gowal et al. (2018) performs interval arithmetic over the bounds of a simple bounding-box relaxation that over-approximates the possible output ranges of each layer, before computing an upper bound of the worst-case loss based on the output range of the final layer. Shi et al. (2021) later improved IBP-based training by introducing a parameter initialisation technique that induces a constant growth rate of IBP bounds, along with specially designed regularisation to stabilise initial phases of training and improve overall performance. Müller et al. (2023) propose SABR, which computes the IBP bounds over a small adversarially selected region rather than over the full perturbation set. It defines a smaller box of radius τ=λτετ= _τ , with λτ∈[0,1] _τ∈[0,1], centred around a projected adversarial point, ensuring the local region ℬτ(τ)=′:‖′−τ‖∞≤τB_τ(x_τ)=\x :\|x -x_τ\|_∞≤τ\ is fully contained within ℬε()B_ (x). Final IBP bounds are then computed over ℬτ(τ)B_τ(x_τ) and used to calculate the final IBP loss, i.e., ℒSABR=ℒ(−¯θΔτ(,y),y)L_SABR=L(- z _τ_θ(x,y),y). The intuition is that propagating a smaller adversarially selected region reduces the approximation error induced by IBP bounds, which grows with respect to network depth Mao et al. (2024), thereby providing an unsound but effective approximation of the robust loss over the full perturbation set. Mao et al. (2023) introduced TAPS, which splits a network into two sub-parts before training the first part through IBP and the second through AT. The idea is that the PGD under-approximation can compensate for some of the over-approximation error induced by IBP bounds. STAPS Mao et al. (2023) carries out an identical procedure, only changing the perturbation region to a smaller adversarially selected one as in SABR. De Palma et al. (2024) formalise a family of expressive losses, which interpolate between empirical lower bounds and certified upper bounds of the worst-case robust loss. MTL-IBP is one such method that achieved relative state-of-the-art performance through leveraging a task balancing coefficient α∈[0,1]α∈[0,1] for fine-grained control over a linear interpolation of PGD and IBP based training, i.e., ℒMTL−IBP=(1−α)ℒPGD+αℒIBPL_MTL-IBP=(1-α)L_PGD+ _IBP. De Palma (2026) introduced C-DIST, which splits the network into a feature extractor and classification head, using an empirically trained teacher for feature-space distillation before learning final classifications through their prior C-IBP objective De Palma et al. (2024) on the full model. C-DIST is closest to our work, since it distils knowledge from an empirically robust teacher. However, C-DIST is tightly coupled with the notion of expressivity De Palma et al. (2024), combining a robust feature-space distillation loss with the expressive C-IBP objective. Moreover, C-DIST calculates both adversarial examples and IBP bounds for distillation. In contrast, AD-CERT uses a simple logit-level AD branch as the empirical endpoint, while keeping the certified branch as the baseline IBP loss. All of the previously mentioned works train standard feed-forward ReLU networks. A related but distinct line of work instead studies robustness by construction. Sortnet Zhang et al. (2022a) is a notable such model that utilises ℓ∞ _∞-distance functions to provide a 1-Lipschitz architecture. While Sortnet performs particularly well on the CIFAR-10 dataset with ε=8255 = 8255, it is less competitive with standard certified training methods on ReLU architectures, particularly against smaller ε values (see §4.2). The non-smoothness and gradient sparsity introduced by 1-Lipschitz architectures leave room for exploration in their practical applicability to certified training but remains outside the scope of this work. 6 Conclusions In this paper, we introduced AD-CERT, a novel certified training algorithm that combines the empirical strengths of an adversarially trained teacher with the verifiability of IBP bounds through a simple linear combination of adversarial distillation over the logit-level and an IBP loss objective. AD-CERT achieves overall state-of-the-art certified accuracy, outperforming all prior methods across five standard certified training benchmarks, while remaining competitive in standard accuracy. Our ablations further suggest that adversarial logit-level distillation provides an effective empirical endpoint for certified training, acting analogously to previous hard-label lower bound endpoints while improving certified performance. Nevertheless, a substantial gap remains between empirically robust teachers and certifiably robust students in terms of standard and adversarial accuracy, particularly on harder settings, highlighting the need for future work on stronger mechanisms for transferring empirical robustness into certifiable robustness. References M. Balunovic and M. Vechev (2020) Adversarial training and provable defenses: bridging the gap. In International Conference on Learning Representations, Cited by: §2.2, §2.3, §2, §4. B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, and F. Roli (2013) Evasion attacks against machine learning at test time. In Advanced Information Systems Engineering, Cited by: §1. R. Bunel, J. Lu, I. Turkaslan, P. H. S. Torr, P. Kohli, and M. P. Kumar (2020) Branch and bound for piecewise linear neural network verification. J. Mach. Learn. Res.. Cited by: §1, §4.1. F. Croce and M. Hein (2020) Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning, Cited by: §F.4, §1, §2.2, §4. J. Cui, Z. Tian, Z. Zhong, X. Qi, B. Yu, and H. Zhang (2024) Decoupled kullback-leibler divergence loss. In Advances in Neural Information Processing Systems, Cited by: §2.3. A. De Palma, R. Bunel, K. Dvijotham, M. P. Kumar, R. Stanforth, and A. Lomuscio (2024) Expressive losses for verified robustness via convex combinations. In International Conference on Machine Learning, Cited by: Appendix A, §F.1, §F.2, §F.3, §F.5, Appendix G, Appendix G, §1, §2.2, §2.3, §2, §3.2, §4.1, §4.2, §4.2, §4.3, §4.3, §4, §5. A. De Palma, R. Bunel, K. Dvijotham, M. P. Kumar, and R. Stanforth (2022) IBP regularization for verified adversarial robustness via branch-and-bound. In International Conference on Machine Learning, Cited by: Appendix A, §2.2, §2.3, §2, §4. A. De Palma (2026) Learning better certified models from empirically-robust teachers. External Links: Link Cited by: Appendix A, Appendix D, §F.1, §F.2, §F.3, §F.5, Appendix G, §1, §2.2, §2.3, §2, §3.2, §4.1, §4.2, §4.2, §4.3, §4.3, Table 1, Table 1, Table 1, Table 2, Table 2, Table 2, §4, §5. R. Ehlers (2017) Formal verification of piece-wise linear feed-forward neural networks. Automated Technology for Verification and Analysis. Cited by: §1. C. Ferrari, M. N. Mueller, N. Jovanović, and M. Vechev (2022) Complete verification via multi-neuron relaxation guided branch-and-bound. In International Conference on Learning Representations, Cited by: §1, §2.1. M. Goldblum, L. Fowl, S. Feizi, and T. Goldstein (2020) Adversarially robust distillation. In Association for the Advancement of Artificial Intelligence (AAAI) Conference on Artificial Intelligence, Cited by: §2.3, §2.3, §4.3. I. J. Goodfellow, J. Shlens, and C. Szegedy (2015) Explaining and harnessing adversarial examples. In International Conference on Learning Representations, Cited by: §1. S. Gowal, K. Dvijotham, R. Stanforth, R. Bunel, C. Qin, J. Uesato, R. Arandjelovic, T. Mann, and P. Kohli (2018) On the effectiveness of interval bound propagation for training verifiably robust models. In Advances in Neural Information Processing Systems, Cited by: Appendix A, §F.4, §F.5, §1, §2.2, §2.2, §2.3, §2, §3, §4.2, §4, §5. K. He, X. Zhang, S. Ren, and J. Sun (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In International Conference on Computer Vision, Cited by: §F.3. G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. In Advances in Neural Information Processing Systems, Cited by: §2.3, §4.3. N. Jovanović, M. Balunović, M. Baader, and M. Vechev (2022) On the paradox of certified training. Transactions on Machine Learning Research. Cited by: §1, §2.2. G. Katz, C. Barrett, D. Dill, K. Julian, and M. Kochenderfer (2017) Reluplex: an efficient SMT solver for verifying deep neural networks. In Computer Aided Verification, Lecture Notes in Computer Science, Vol. 10426, p. 97–117. Cited by: §1, §2.1. D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In International Conference on Learning Representations, Cited by: §F.3. A. Krizhevsky and G. Hinton (2009) Learning multiple layers of features from tiny images. Technical report Technical Report , Technical report, University of Toronto, University of Toronto. Cited by: §F.1, §4.1, §4.3. Y. Le and X. Yang (2015) Tiny imagenet visual recognition challenge. CS 231N 7 (7), p. 3. Cited by: §F.1, §4.1. Y. LeCun, C. Cortes, and C. J. C. Burges (2010) MNIST handwritten digit database. External Links: Link Cited by: §F.1, §4.1. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2018) Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, Cited by: Appendix D, §F.5, §1, §2.2. Y. Mao, S. Balauca, and M. Vechev (2025) CTBENCH: a library and benchmark for certified training. In International Conference on Machine Learning, Cited by: Appendix D, §F.1, §F.2, §F.3, §F.4, §F.5, Table 8, §1, §4.1, §4.2, §4.2, §4.3, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 2, §4. Y. Mao, M. N. Müller, M. Fischer, and M. Vechev (2023) Connecting certified and adversarial training. In Advances in Neural Information Processing Systems, Cited by: Appendix A, §F.1, §F.2, §F.5, §1, §2.2, §2.3, §2, §3.2, §4.1, §4.2, §4, §5. Y. Mao, M. N. Müller, M. Fischer, and M. Vechev (2024) Understanding certified training with interval bound propagation. In International Conference on Learning Representations, Cited by: §1, §2.2, §5. M. Mirman, T. Gehr, and M. Vechev (2018) Differentiable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning, Cited by: §1, §5. M. N. Müller, F. Eckert, M. Fischer, and M. Vechev (2023) Certified training: small boxes are all you need. In International Conference on Learning Representations, Cited by: Appendix A, §F.1, §F.2, §F.5, §1, §2.2, §2.3, §2, §3.2, §4.1, §4.2, §4.3, §4, §5. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala (2019) PyTorch: an imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, Cited by: §4.1. Z. Shi, Y. Wang, H. Zhang, J. Yi, and C. Hsieh (2021) Fast certified robust training with short warmup. In Advances in Neural Information Processing Systems, Cited by: Appendix A, §F.1, §F.2, §F.3, §F.3, §1, §4.1, §5, 4. G. Singh, T. Gehr, M. Püschel, and M. Vechev (2019) An abstract domain for certifying neural networks. Proc. ACM Program. Lang. 3 (POPL). Cited by: §1, §2.1. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus (2014) Intriguing properties of neural networks. In International Conference on Learning Representations, Cited by: §1. V. Tjeng, K. Xiao, and R. Tedrake (2019) Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations, Cited by: §1, §2.1. F. Tramer, N. Carlini, W. Brendel, and A. Madry (2020) On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems, Cited by: §1, §2.2. S. Wang, H. Zhang, K. Xu, X. Lin, S. Jana, C. Hsieh, and J. Z. Kolter (2021) Beta-CROWN: efficient bound propagation with per-neuron split constraints for complete and incomplete neural network verification. In Advances in Neural Information Processing Systems, Cited by: §F.4, §1, §2.1, §4.1. E. Wong and J. Z. Kolter (2018) Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference of Machine Learning, Cited by: §2.2. K. Xu, Z. Shi, H. Zhang, Y. Wang, K. Chang, M. Huang, B. Kailkhura, X. Lin, and C. Hsieh (2020) Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Processing Systems. Cited by: §4.1. K. Xu, H. Zhang, S. Wang, Y. Wang, S. Jana, X. Lin, and C. Hsieh (2021) Fast and Complete: enabling complete neural network verification with rapid and massively parallel incomplete verifiers. In International Conference on Learning Representations, Cited by: §F.4, §1, §2.1, §4.1. B. Zhang, D. Jiang, D. He, and L. Wang (2022a) Boosting the certified robustness of l-infinity distance nets. In International Conference on Learning Representations, Cited by: §5. B. Zhang, D. Jiang, D. He, and L. Wang (2022b) Rethinking lipschitz neural networks and certified robustness: a boolean function perspective. In Advances in Neural Information Processing Systems, Cited by: §4.2, Table 1, Table 1, Table 1, Table 1, Table 1. H. Zhang, H. Chen, C. Xiao, S. Gowal, R. Stanforth, B. Li, D. Boning, and C. Hsieh (2020) Towards stable and efficient training of verifiably robust neural networks. In International Conference on Learning Representations, Cited by: §1, §2.2. H. Zhang, S. Wang, K. Xu, L. Li, B. Li, S. Jana, C. Hsieh, and J. Z. Kolter (2022c) General cutting planes for bound-propagation-based neural network verification. In Advances in Neural Information Processing Systems, Cited by: §1, §2.1. H. Zhang, T. Weng, P. Chen, C. Hsieh, and L. Daniel (2018) Efficient neural network robustness certification with general activation functions. In Advances in Neural Information Processing Systems, Cited by: §F.4, §1, §2.1, §4.1. M. Zhang, M. Backes, and X. Zhang (2025) Generating less certain adversarial examples improves robust generalization. Transactions on Machine Learning Research. Cited by: Appendix D, §F.3, §F.5. J. Zhu, J. Yao, B. Han, J. Zhang, T. Liu, G. Niu, J. Zhou, J. Xu, and H. Yang (2022) Reliable adversarial distillation with unreliable teachers. In International Conference on Learning Representations, Cited by: §2.3. B. Zi, S. Zhao, X. Ma, and Y. Jiang (2021) Revisiting adversarial robustness distillation: robust soft labels make student better. In International Conference on Computer Vision, Cited by: §2.3, §2.3, §4.2, §4.3, Table 4. Appendix A Interval Bound Propagation (IBP) IBP has been at the core of all recent state-of-the-art certified training methods Shi et al. [2021], De Palma et al. [2022], Müller et al. [2023], Mao et al. [2023], De Palma et al. [2024], De Palma [2026]. Here, we formalise the bound propagation for a single layer, as proposed by Gowal et al. [2018]. For an affine layer (e.g., convolutional or linear layer) represented by hk(zk−1)=Wzk−1+bh_k(z_k-1)=Wz_k-1+b, we can define the bound calculation as follows: μk−1=z¯k−1+z¯k−12 _k-1= z_k-1+ z_k-12 ,μk=Wμk−1+b, , _k=W _k-1+b, (17) rk−1=z¯k−1−z¯k−12 r_k-1= z_k-1- z_k-12 ,rk=|W|rk−1, , r_k=|W|r_k-1, (18) z¯k=μk−rk z_k= _k-r_k ,z¯k=μk+rk, , z_k= _k+r_k, (19) where μ denotes the centre of the interval box, r its radius, and z¯k,z¯k z_k, z_k denote lower and upper bounds of the layer output, respectively. When hkh_k is an element-wise increasing activation function, e.g., ReLU, sigmoid, or tanh, we instead trivially propagate the bounds as z¯k=hk(z¯k−1) z_k=h_k( z_k-1) and z¯k=hk(z¯k−1) z_k=h_k( z_k-1). For ReLU networks, an unstable ReLU activation implies the ReLU activation may be either active or inactive given the input interval, i.e., z¯k−1<0<z¯k−1 z_k-1<0< z_k-1. Hence, incomplete verifiers must over-approximate the possible activation outputs. Figure 3 compares the loose IBP box relaxation with the tighter convex hull relaxation for an unstable ReLU. zzReLU(z)ReLU(z)z¯k−1 z_k-1z¯k−1 z_k-1z¯k z_k zzReLU(z)ReLU(z)z¯k−1 z_k-1z¯k−1 z_k-1z¯k z_k (a) IBP Relaxation (b) Convex Hull Figure 3: Comparison of IBP relaxation and the tighter convex hull relaxation. The shaded regions represent convex over-approximations of an unstable ReLU function. The dashed line represents the actual possible activation outputs. Appendix B Deferred Proofs of Theoretical Results In this section, we provide proofs for each of the theoretical results from our methodology section (§3) of the main text. In each case, we restate the claim for the ease of the reader. Proposition B.1 (Equivalence of AD and soft-label AT). Let x_adv be fixed, and let the teacher predictive distribution Pθt()P_ _t(x) be fixed. Then the AD objective: ℒAD:=KL(Pθt()∥Pθs()),L_AD:=KL\! (P_ _t(x)\,\|\,P_ _s(x_adv) ), (20) is equal, up to an additive constant independent of the student parameters, to AT with soft teacher labels, namely: ℒsoft-AT:=ℒCE(fθs(),Pθt()).L_soft -AT:=L_CE\! (f_ _s(x_adv),\,P_ _t(x) ). (21) Moreover, both objectives induce identical backpropagation updates with respect to the student logits. In particular, for each class index i: ∂ℒAD∂zθsi=[Pθs()]i−[Pθt()]i=∂ℒsoft-AT∂zθsi. _AD∂ z_ _s^i=[P_ _s(x_adv)]_i-[P_ _t(x)]_i= _soft -AT∂ z_ _s^i. (22) Proof. For brevity, we let zθsi:=[fθs()]iz^i_ _s:=[f_ _s(x_adv)]_i and Pθsi:=[Pθs]iP^i_ _s:=[P_ _s]_i denote the i-th entry of the raw logit outputs and predictive distributions of the student classifier, respectively. First, recall the derivative of the softmax function: ∂Pj∂zi=Pi(1−Pi),i=j,−PiPj,i≠j. ∂ P^j∂ z^i= casesP^i(1-P^i),&i=j,\\ -P^iP^j,&i≠ j. cases (23) For ease of notation, we can rewrite this in terms of the Kronecker delta function: ∂Pj∂zi=Pj(δij−Pi),where δij=1,i=j,0,i≠j. ∂ P^j∂ z^i=P^j( _ij-P^i), _ij= cases1,&i=j,\\ 0,&i≠ j. cases (24) Now, we compare the derivatives of ℒADL_AD and ℒsoft-ATL_soft -AT with respect to zθsiz^i_ _s. Given that ℒAD=∑j=1CPθtj()⋅log(Pθtj()Pθsj())=∑j=1CPθtj()⋅log(Pθtj())−∑j=1CPθtj()⋅log(Pθsj())L_AD=Σ^C_j=1P_ _t^j(x)· ( P_ _t^j(x)P_ _s^j(x_adv) )= _j=1^CP_ _t^j(x)· (P_ _t^j(x))- _j=1^CP_ _t^j(x)· (P_ _s^j(x_adv)), we note that the ∑j=1CPθtj()⋅log(Pθtj()) _j=1^CP_ _t^j(x)· (P_ _t^j(x)) term is independent of student parameters, so: ∂ℒAD∂zθsi _AD∂ z_ _s^i =∂zθsi[−∑j=1CPθtj()log(Pθsj())], = ∂ z_ _s^i [- _j=1^CP_ _t^j(x) (P_ _s^j(x_adv)) ], (25) =−∑j=1CPθtj()1Pθsj()∂Pθsj()∂zsi, =- _j=1^CP_ _t^j(x) 1P_ _s^j(x_adv) ∂ P_ _s^j(x_adv)∂ z_s^i, (26) =−∑j=1CPθtj()Pθsj()Pθsj()(δij−Pθsi()), =- _j=1^C P_ _t^j(x)P_ _s^j(x_adv)P_ _s^j(x_adv)( _ij-P_ _s^i(x_adv)), (27) =−∑j=1CPθtj()(δij−Pθsi()), =- _j=1^CP_ _t^j(x)( _ij-P_ _s^i(x_adv)), (28) =−Pθti()+Pθsi()∑j=1CPθtj(), =-P_ _t^i(x)+P_ _s^i(x_adv) _j=1^CP_ _t^j(x), (29) =Pθsi()−Pθti(). =P_ _s^i(x_adv)-P_ _t^i(x). (30) Finally, recall that ℒsoft-AT=ℒCE(fθs(),Pθt())L_soft -AT=L_CE(f_ _s(x_adv),P_ _t(x)), and, ℒCE(fθs(),Pθt())=−∑j=1CPθtj()log(Pθsj()).L_CE(f_ _s(x_adv),P_ _t(x))=-Σ^C_j=1P_ _t^j(x) (P_ _s^j(x_adv)). (31) Hence, the result falls immediately from Equations (25)-(30), and by extension, the two objectives must only differ by an additive constant independent of θs _s. ∎ Proposition B.2 (Analytical form of AD-CERT). Let H(P)H(P) and H(Q,P)H(Q,P) denote the entropy and cross entropy respectively for probability distributions P and Q. That is: H(P)=−∑i=1Cpilog(pi),H(Q,P)=−∑i=1Cqilog(pi).H(P)=- _i=1^Cp_i (p_i), H(Q,P)=- _i=1^Cq_i (p_i). (32) Then, ℒAD-CERTL_AD -CERT from Equation (10) can be re-written as: ℒAD-CERT=(1−α)H(Pθt(),Pθs())+αℒIBP−(1−α)H(Pθt()).L_AD -CERT=(1-α)H(P_ _t(x),P_ _s(x_adv))+ _IBP-(1-α)H(P_ _t(x)). (33) Therefore, up to a constant independent of θs _s, AD-CERT is a scalarisation between (i) a soft-label adversarial lower bound surrogate and (i) a certified IBP upper bound. Proof. The proof is immediate since we can write: ℒKL(Pθt()∥Pθs()) _KL\! (P_ _t(x)\,\|\,P_ _s(x_adv) ) =∑i=1CPθti()⋅log(Pθti()Pθsi()), =Σ^C_i=1P_ _t^i(x)· ( P_ _t^i(x)P_ _s^i(x_adv) ), (34) =∑i=1CPθti()⋅(log(Pθti())−log(Pθsi())), =Σ^C_i=1P_ _t^i(x)· ( (P_ _t^i(x))- (P_ _s^i(x_adv)) ), (35) =−∑i=1CPθti()log(Pθsi())+∑i=1CPθti()log(Pθti()), =-Σ^C_i=1P_ _t^i(x) (P_ _s^i(x_adv))+Σ^C_i=1P_ _t^i(x) (P_ _t^i(x)), (36) =H(Pθt(),Pθs())−H(Pθt()). =H(P_ _t(x),P_ _s(x_adv))-H(P_ _t(x)). (37) Hence, substituting this identity into Equation (10) gives: ℒAD-CERT=(1−α)H(Pθt(),Pθs())+αℒIBP−(1−α)H(Pθt()),L_AD -CERT=(1-α)H(P_ _t(x),P_ _s(x_adv))+ _IBP-(1-α)H(P_ _t(x)), (38) as required. ∎ Lemma B.3. For AD, the optimal logit margin, Δzij∗ z_ij^*, between any two classes i,ji,j is: Δzij∗=log([Pθt()]i[Pθt()]j). z_ij^*= ( [P_ _t(x)]_i[P_ _t(x)]_j ). (39) Moreover, standard AT has no finite minimiser, and its infimum 0 is approached only as [fθs()]y−[fθs()]j→+∞,∀j≠y[f_ _s(x_adv)]_y-[f_ _s(x_adv)]_j→+∞, ∀ j≠ y. Proof. First note that, from Equation (30), ℒADL_AD is minimised iff Pθs=PθtP_ _s=P_ _t, or equivalently, Pθsi()=Pθti()P_ _s^i(x_adv)=P_ _t^i(x) ∀ i∈1,…,C.i∈\1,…,C\. Rewriting this using the definition of softmax, we have: ezsi∑j=1Cezsj=Pθti()∀i∈1,…,C, e^z_s^i _j=1^Ce^z_s^j=P_ _t^i(x) ∀\ i∈\1,…,C\, (40) where zθsi=[fθs()]iz^i_ _s=[f_ _s(x_adv)]_i, which implies: zsi=log(Pθti()⋅∑j=1Cezsj)=log(Pθti())+log(∑j=1Cezsj)z_s^i= (P_ _t^i(x)· _j=1^Ce^z_s^j )= (P_ _t^i(x) )+ ( _j=1^Ce^z_s^j ) (41) Then, we can write the optimal logit for a given class i as zsi∗:=log(Pθti())+cz_s^i*:= (P_ _t^i(x))+c for some shared constant, c∈ℝc . Since softmax is invariant to adding a constant to each logit, the optimal logit margin between any two classes i and j is: Δzij∗:=zsi∗−zsj∗=log(Pθti())−log(Pθtj())=log(Pθti()Pθtj()), z_ij^*:=z_s^i*-z_s^j*= (P_ _t^i(x))- (P_ _t^j(x))= ( P_ _t^i(x)P_ _t^j(x) ), (42) which is finite since Pθti()>0P_ _t^i(x)>0 ∀ i. In contrast, standard AT loss is defined by ℒAT=−log(Pθsy())L_AT=- (P_ _s^y(x_adv)), with y denoting the index of the true class label here. Hence, we have: ℒAT=−log(ezsy∑j=1Cezsj)=log(∑j=1Cezsj−zsy)=log(1+∑j≠yezsj−zsy).L_AT=- ( e^z_s^y _j=1^Ce^z_s^j )= ( _j=1^Ce^z_s^j-z_s^y )= (1+ _j≠ ye^z_s^j-z_s^y ). (43) For any finite logits, we have ezsj−zsy>0e^z_s^j-z_s^y>0, and hence 1+∑j≠yezsj−zsy>1⟹ℒAT>0.1+ _j≠ ye^z_s^j-z_s^y>1 _AT>0. (44) Therefore, ℒATL_AT has no finite minimiser. Since we have established that ℒAT>0L_AT>0 for all finite logits, its global minimum is never attained. However, we can show that its infimum is exactly 0. Let m=minj≠y(zsy−zsj)m= _j≠ y(z_s^y-z_s^j) denote the minimum logit margin of the correct class over all incorrect classes. We can bound the loss as follows: 0<ℒAT=log(1+∑j≠ye−(zsy−zsj))≤log(1+(C−1)e−m),0<L_AT= (1+ _j≠ ye^-(z_s^y-z_s^j) )≤ (1+(C-1)e^-m ), (45) where C is the total number of classes. Taking the limit as the minimum margin grows to infinity (m→+∞m→+∞), we have: limm→+∞log(1+(C−1)e−m)=log(1+0)=0. _m→+∞ (1+(C-1)e^-m )= (1+0)=0. (46) Thus, infℒAT=0 _AT=0. This infimum is not reachable for any finite network weights, but is approached asymptotically as the correct-class logit margins tend to positive infinity. ∎ Appendix C Additional Results In this section, we present additional theoretical results complementary to those in the main text. Proposition C.1 (Teacher-margin transfer in logit space). Assume the teacher fθtf_ _t predicts the true class y on the natural input x. Then, if: ∥fθs()−fθt()∥∞<12minj≠y[θt(,y)]j, \|f_ _s(x_adv)-f_ _t(x) \|_∞< 12 _j≠ y[z_ _t(x,y)]_j, (47) it follows that argmaxi[fθs()]i=y *argmax_i[f_ _s(x_adv)]_i=y. Proof. First, define δ:=‖fθs(adv)−fθt()‖∞δ:= \|f_ _s(x_adv)-f_ _t(x) \|_∞. Then, −δ≤[fθs()]i−[fθt()]i≤δ,-δ≤[f_ _s(x_adv)]_i-[f_ _t(x)]_i≤δ, for any class index i. Which means, [fθs()]y≥[fθt()]y−δ[f_ _s(x_adv)]_y≥[f_ _t(x)]_y-δ and [fθs()]i≤[fθt()]i+δ[f_ _s(x_adv)]_i≤[f_ _t(x)]_i+δ for any i≠yi≠ y. Moreover: ⟹[fθs()]y−[fθs()]i [f_ _s(x_adv)]_y-[f_ _s(x_adv)]_i ≥[fθt()]y−[fθt()]i−2δ, ≥[f_ _t(x)]_y-[f_ _t(x)]_i-2δ, (48) ⟹[θs(,y)]i [z_ _s(x_adv,y)]_i ≥[θt(,y)]i−2δ. ≥[z_ _t(x,y)]_i-2δ. (49) Hence, if δ<12mini≠y[θt(,y)]iδ< 12 _i≠ y[z_ _t(x,y)]_i, then clearly [θt(,y)]i−2δ>0[z_ _t(x,y)]_i-2δ>0, which implies [θs(,y)]i>0[z_ _s(x_adv,y)]_i>0 and argmaxi[fθs(adv)]i=y *argmax_i[f_ _s(x_adv)]_i=y. ∎ Lemma C.2. Let ℒATL_AT be bounded by some U∈ℝU and U>0U>0, i.e., ℒAT=ℒCE(fθs(),y)=−log(Pθsy)≤U,L_AT=L_CE(f_ _s(x_adv),y)=- (P_ _s^y)≤ U, (50) where y∈1,…,Cy∈\1,…,C\ is the true label index and Pθsy=[softmax(fθs())]yP_ _s^y= [softmax(f_ _s(x_adv)) ]_y. Then, for every j≠yj≠ y, the corresponding true-vs-false logit margin satisfies zsy−zsj≥−log(eU−1).z_s^y-z_s^j≥- (e^U-1). (51) Proof. Notice that AT loss can be written as: ℒAT=−log(ezsy∑k=1Cezsk)=log(1+∑j≠yezsj−zsy).L_AT=- ( e^z_s^y _k=1^Ce^z_s^k )= (1+ _j≠ ye^z_s^j-z_s^y ). (52) Now assume that ℒAT≤UL_AT≤ U. Then, log(1+∑j≠yezsj−zsy) (1+ _j≠ ye^z_s^j-z_s^y ) ≤U, ≤ U, (53) ⟹1+∑j≠yezsj−zsy 1+ _j≠ ye^z_s^j-z_s^y ≤eU, ≤ e^U, (54) ⟹∑j≠yezsj−zsy _j≠ ye^z_s^j-z_s^y ≤eU−1. ≤ e^U-1. (55) Since each term in the sum is non-negative, it follows that for every j≠yj≠ y, we have: ezsj−zsy e^z_s^j-z_s^y ≤eU−1, ≤ e^U-1, (56) ⟹zsj−zsy z_s^j-z_s^y ≤log(eU−1), ≤ (e^U-1), (57) ⟹zsy−zsj z_s^y-z_s^j ≥−log(eU−1),∀j≠y, ≥- (e^U-1), ∀ j≠ y, (58) as required. ∎ Appendix D Additional Experiments In this section, we evaluate the sensitivity of AD-CERT and C-DIST De Palma [2026] to teachers trained under different adversarial training methods and schedules. The sensitivity results from Table 5 suggest that the choice of teacher can affect the standard and adversarial accuracy of the resulting students, but the certified accuracy is comparatively stable across teacher variants, particularly for AD-CERT. Here, short-cycle refers to the training procedure described in De Palma [2026], where we use the SGD optimiser with momentum set to 0.9, a 30-epoch training cycle with a cyclic learning rate, which linearly increases from 0 to 0.2 during the first half of the training, and then decreases back to 0. Long-cycle corresponds to the standard AT settings used in the CTBench library Mao et al. [2025], where PGD Madry et al. [2018] and EDAC Zhang et al. [2025] models are trained for 240 epochs using the Adam optimiser. Short-cycle teachers tend to slightly improve the certified results of AD-CERT and C-DIST, which coincides with the observations of De Palma [2026] in their work. In general, C-DIST appears to benefit more noticeably from short-cycle teachers, with AD-CERT being slightly less sensitive. Nevertheless, AD-CERT remains consistently stronger in certified and A accuracy across all teacher protocols, while C-DIST tends to achieve slightly higher standard accuracy in most settings, consistent with previous observations in §4.2. Table 5: Sensitivity analysis of AD-CERT and C-DIST on CIFAR-10 at ε=8255 = 8255 with different teacher training protocols. Teacher Teach. Std. [%] Teach. A [%] Method Std. [%] A [%] Cert. [%] PGD-Long 78.71 35.93 AD-CERT 53.17 35.95 35.21 C-DIST 52.35 34.80 33.75 PGD-Short 76.14 42.55 AD-CERT 53.90 36.52 35.61 C-DIST 54.44 36.05 34.81 EDAC-Long 78.95 42.48 AD-CERT 53.49 36.46 35.60 C-DIST 54.32 35.69 34.55 EDAC-Short 73.55 41.99 AD-CERT 53.69 36.69 35.84 C-DIST 54.33 35.76 34.68 Appendix E Pseudo-code In this section, we outline the pseudocode for the training procedure of AD-CERT (§3.1) in Algorithm 1. Algorithm 1 AD-CERT Training Procedure 1:PGD teacher fθtf_ _t, data D, perturbation radius ϵtrainε_train, epochs Nwarm,N_warm,N, IBP coefficient α 2: 3:Initialisation 4:Initialise student fθsf_ _s with parameters θs _s ⊳ Shi et al. [2021] initialisation 5:Detach teacher fθtf_ _t gradients from training 6: 7:Training 8:for epoch=1epoch=1 to N do 9: if epoch≤Nwarmepoch≤ N_warm then 10: Update ϵcurr _curr and αcurr _curr ⊳ ramp-up to ϵtrain,α _train,α 11: end if 12: for (,y)(x,y) in D do 13: Compute x_adv via PGD attack 14: Compute IBP bounds ¯θsΔ(,y) z _ _s(x,y) using ϵcurr _curr 15: ℒdist←KL(softmax(fθt())∥softmax(fθs()))L_dist (softmax(f_ _t(x))\,\|\,softmax(f_ _s(x_adv)) ) 16: ℒrobust←ℒCE(−¯θsΔ(,y),y)L_robust _CE(- z _ _s(x,y),y) 17: ℒtotal←(1−αcurr)ℒdist+αcurrℒrobustL_total←(1- _curr)L_dist+ _currL_robust 18: Update θs←θs−η∇θsℒtotal _s← _s-η _ _sL_total 19: end for 20:end for 21:return fθsf_ _s Appendix F Experimental Setup F.1 Dataset We conduct experiments on MNIST LeCun et al. [2010], CIFAR-10 Krizhevsky and Hinton [2009], and TinyImageNet Le and Yang [2015]. These datasets are open-source and freely available, with unspecified licenses. We follow the data preprocessing directly available in the CTBench library Mao et al. [2025]. No preprocessing is applied to MNIST. CIFAR-10 and TinyImageNet are normalised using the dataset mean and standard deviation, and random horizontal flips are applied. For CIFAR-10, additional random cropping to 32×3232× 32 is applied after zero-padding each image by 2 pixels on all sides. For TinyImageNet, random cropping to 64×6464× 64 is applied after zero-padding each image by 4 pixels on all sides. We train on the corresponding training sets and certify on the validation sets, following common practice in the certified training literature Shi et al. [2021], Müller et al. [2023], Mao et al. [2023], De Palma et al. [2024], Mao et al. [2025], De Palma [2026]. F.2 Model Architectures We use the standard CNN7 architecture, a convolutional network consisting of seven convolutional and linear layers. Each layer, except for the final linear layer, is followed by Batch Normalisation and a ReLU activation. This architecture has been shown to perform consistently well across settings Shi et al. [2021], Mao et al. [2025], and is therefore widely adopted in the certified training literature Shi et al. [2021], Müller et al. [2023], Mao et al. [2023], De Palma et al. [2024], Mao et al. [2025], De Palma [2026]. For TinyImageNet, we double the stride of the final convolutional layer to reduce computational cost. F.3 Training Details Implementation We implement AD-CERT in CTBench using PyTorch. The training loss follows the CTBench decomposition into a natural loss, a robust loss, and regularisation terms. For AD-CERT, the robust loss is given by Equation 10, where the AD term uses a fixed adversarially trained teacher, and the certified term is computed with IBP bounds. Unless stated otherwise, the teacher and student use the same CNN7 architecture. Initialisation Adversarial teacher models are initialised by Kaiming uniform He et al. [2015], while certified student models are initialised using the IBP initialisation from Shi et al. [2021]. Training Schedule We follow the standard CTBench training schedule. Certified models are first trained for one epoch with ε=0 =0, and then use a warm-up phase where ε is smoothly increased from 0 to the target value. The warmup phase is 20 epochs for MNIST with ε=0.1 =0.1 and ε=0.3 =0.3, 80 epochs for CIFAR-10 with ε=2255 = 2255, 120 epochs for CIFAR-10 with ε=8255 = 8255, and 80 epochs for TinyImageNet with ε=1255 = 1255. We use the IBP regularisation proposed by Shi et al. [2021], with weight 0.50.5 on MNIST and CIFAR-10, and 0.20.2 on TinyImageNet, during warmup. In total, we train for 70 epochs on MNIST, 160 epochs on CIFAR-10 with ε=2255 = 2255, 240 epochs on CIFAR-10 with ε=8255 = 8255, and 160 epochs on TinyImageNet. Optimisation In general, we follow the exact optimisation setup from CTBench. We use Adam Kingma and Ba [2015] with a learning rate of 5×10−45× 10^-4. The learning rate is decayed by a factor of 0.20.2 at epochs 50 and 60 for MNIST, epochs 120 and 140 for CIFAR-10 with ε=2255 = 2255, epochs 200 and 220 for CIFAR-10 with ε=8255 = 8255, and epochs 120 and 140 for TinyImageNet. We use a batch size of 256 for MNIST and 128 for CIFAR-10 and TinyImageNet. Gradients are clipped to 1010 in ℓ2 _2 norm before every optimiser step. No weight decay is applied, and L1L_1 regularisation is applied only to the weights of linear and convolutional layers. Batch normalisation follows the CTBench population-statistics setting. We apply Stochastic Weight Averaging (SWA) using the settings in accordance with MTL-IBP’s training setup in CTBench since both methods use an unsound robust objective, making direct model selection on the robust loss impractical. Teacher Models We use the pre-trained PGD checkpoints provided by CTBench as teacher models, except for CIFAR-10 with ε=8255 = 8255, where our ablation teacher sensitivity analysis from Table 5 showed that a 30-epoch short-cycle PGD model trained with a cyclic learning-rate schedule exactly as described by De Palma [2026] alongside an EDAC Zhang et al. [2025] step size of 0.30.3, gives marginally better performance. All teachers are kept fixed during AD-CERT training. Tuning of Hyperparameters In general, we perform minimal hyperparameter tuning due to the computational overhead of running complete verification. We ran initial experiments following the exact hyperparameters and settings used in CTBench Mao et al. [2025] for the MTL-IBP De Palma et al. [2024] method. For the tuning of α, we use a manually selected search range guided by some reported quantities in the CTBench library, namely, unstable ReLU ratio, IBP certification rate, and standard, adversarial, and certified accuracy. These metrics help indicate whether the objective places too much or too little weight on the IBP branch, allowing us to adjust α accordingly. We find that slightly reducing α is beneficial on MNIST at ε=0.1 =0.1 and on TinyImageNet at ε=1255 = 1255. For TinyImageNet, we also observe improvements from increasing wrobw_rob and enlarging the PGD attack region used in the empirical branch. The final hyperparameters used for AD-CERT across all settings are reported in Table 6. Table 6: Best hyperparameters for AD-CERT across dataset settings. MNIST CIFAR-10 TinyImageNet Hyperparameter 0.10.1 0.30.3 2255 2255 8255 8255 1255 1255 L1L_1 regularisation 1×10−61× 10^-6 1×10−61× 10^-6 3×10−63× 10^-6 0.0 5×10−55× 10^-5 wrobw_rob 0.7 1.0 1.0 1.0 0.9 IBP coefficient (α) 7.5×10−37.5× 10^-3 0.5 0.01 0.5 5×10−35× 10^-3 Train ε 0.2 0.3 1255 1255 8255 8255 1255 1255 PGD steps 10 1 8 1 1 Attack range scale 1.0 1.0 2.0 1.0 2.0 F.4 Certification Details Certification is performed within CTBench Mao et al. [2025] using its wrapper for the α,βα,β-CROWN verification library. For each example, we first check whether the model predicts the correct clean label. Incorrectly classified examples are marked as uncertified. For correctly classified examples, we run AutoAttack Croce and Hein [2020] and if an adversarial counterexample is not found, we attempt certification using a sequence of progressively stronger verifiers. Namely, IBP Gowal et al. [2018], CROWN Zhang et al. [2018], α-CROWN Xu et al. [2021], and finally, the complete α,βα,β-CROWN branch-and-bound verifier Wang et al. [2021]. Each stage is only invoked when the previous stage fails to certify the example. We use a timeout of 400 seconds for the α,βα,β-CROWN branch-and-bound stage and an overall timeout of 1000 seconds per example. F.5 Computational Setup & Cost Table 7 reports the training and certification runtime of AD-CERT under the training and certification setups described in Sections F.3 and F.4, respectively. All experiments were run on an internal cluster with access to NVIDIA V100, A100, A100 MIG slices, and H100 GPUs. Training jobs were run on either 2g.20GB or 3g.40GB A100 MIG slices, using 8 CPU cores. For certification, we used MIG slices in most cases, but occasionally used full A100 or H100 GPUs for harder settings, such as CIFAR-10 with ε=2255 = 2255 and TinyImageNet. For clarity, Table 7 reports runtimes for experiments run on a 3g.40GB A100 MIG slice. Training experiments used 64GB of RAM, while certification experiments used 128GB. The training times range from ∼ 1 hour on the simplest MNIST setting, to ∼ 1 day on TinyImageNet. Certification times range from ∼ 12 minutes to over 3 days. Additionally, we report the training cost complexity associated with an AD-CERT student, assuming a pre-trained teacher, compared to previous adversarial and certified training methods Madry et al. [2018], Zhang et al. [2025], Gowal et al. [2018], Müller et al. [2023], Mao et al. [2023], De Palma et al. [2024], De Palma [2026] in Table 8. On a high level, AD-CERT corresponds to performing IBP and PGD, with the small added cost of a single forward pass of a teacher model that is detached from training. Note that Table 8 is a direct extrapolation of the table provided in the original CTBench paper Mao et al. [2025] with added entries for AD-CERT and C-DIST. Table 7: Training and certification time for AD-CERT on different datasets and ε . Dataset ε Train Time (seconds) Certification Time (seconds) MNIST 0.10.1 1.20×1041.20× 10^4 7.67×1027.67× 10^2 0.30.3 4.08×1034.08× 10^3 5.32×1045.32× 10^4 CIFAR-10 2255 2255 2.66×1042.66× 10^4 2.61×1052.61× 10^5 8255 8255 1.59×1041.59× 10^4 3.58×1043.58× 10^4 TinyImageNet 1255 1255 8.97×1048.97× 10^4 1.54×1051.54× 10^5 Table 8: Detailed breakdown of training costs for each certified training method. Method Training cost per batch Details Standard T Forward + backward PGD / EDAC (M+1)T(M+1)T M attack steps + standard loss computation IBP 2T2T Lower and upper bounds propagation SABR (M+2)T(M+2)T IBP + PGD MTL-IBP (M+2)T(M+2)T IBP + PGD AD-CERT† (M+2)T+F(M+2)T+F IBP + PGD + detached teacher forward C-DIST† (M+2)T+Fh(M+2)T+F_h C-IBP + PGD + detached teacher forward on feature split TAPS 2t+K(M+1)(T−t)2t+K(M+1)(T-t) IBP for first split and PGD for second split for each class STAPS 2t+K(M+1)(T−t)+(M+1)T2t+K(M+1)(T-t)+(M+1)T TAPS + PGD Legend T Time cost for standard training, including forward + backward pass F Time cost for a forward pass through the teacher network FhF_h Time cost for a forward pass through the teacher feature extractor network M Number of adversarial attack steps, including repeats K Number of classes t Time cost for standard training in the first network split in TAPS † Added entries from original table in CTBench Mao et al. [2025]. Appendix G C-IBP Distillation (C-DIST) De Palma [2026] formulate the C-DIST loss function as a linear combination of an expressive robust distillation loss in the feature space, ℛfθ(α;,y)R_f_θ(α;x,y), and their prior expressive loss formulation, C-IBP De Palma et al. [2024]. Let fθ:ℝd↦ℝCf_θ:R^d R^C be a classification model as expressed earlier in §2. Then, we can rewrite fθf_θ as a composition of a feature extractor hθ:ℝd↦ℝkh_θ:R^d R^k and a classification head gθ:ℝk↦ℝCg_θ:R^k R^C, yielding fθ=gθ∘hθf_θ=g_θ h_θ (note that we abuse notation of θ for brevity). Then, the expressive robust distillation loss over the feature space is defined by: ℛfθ(α;,y):=∑i=1kmax([h¯θ(α;)]i−[hθt()]i)2,([h¯θ(α;)]i−[hθt()]i)2,R_f_θ(α;x,y):=Σ^k_i=1 \ ([ h_θ(α;x)]_i-[h_ _t(x)]_i )^2, ([ h_θ(α;x)]_i-[h_ _t(x)]_i )^2 \, (59) where h¯θ(α;):=(1−α)hθ()+αh¯θ() h_θ(α;x):=(1-α)h_θ(x_adv)+α h_θ(x) and h¯θ(α;):=(1−α)hθ()+αh¯θ() h_θ(α;x):=(1-α)h_θ(x_adv)+α h_θ(x), with h¯θ() h_θ(x) and h¯θ() h_θ(x) denoting the IBP upper and lower bounds of hθ()h_θ(x), respectively. Note that hθth_ _t denotes the feature extractor of a pre-trained PGD teacher model. Now, let β be the distillation coefficient, determining the relative weight of ℛfθ(α;,y)R_f_θ(α;x,y) from Equation (59). Omitting any regularisation, the training loss for C-DIST takes the following form: ℒfθC-DIST(α,β;,y):=ℒfθC-IBP(α;,y)+βℛfθ(α;,y),L^C-DIST_f_θ(α,β;x,y):=\ L^C-IBP_f_θ(α;x,y)+β\ R_f_θ(α;x,y), (60) where ℒfθC-IBP(α;,y)L^C-IBP_f_θ(α;x,y) is the C-IBP loss from De Palma et al. [2024]. Denoting k as the dimensionality of the feature space, β=5/kβ=5/k was used for the majority of experiments.