Paper deep dive
Interpretable but Fragile? Robustness of Concept Bottlenecks under Geometric-Semantic Perturbations
Hanwei Zhang, Tianma Hu, Gaojie Jin, Xu Cheng, Ronghui Mu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 10/1/2026, 8:44:50 AM
Summary
This paper investigates the robustness of Concept Bottleneck Models (CBMs) compared to standard classifiers under geometric and semantic perturbations. The authors introduce a generator-based evaluation framework to disentangle robustness notions, revealing that interpretability does not inherently confer robustness. Instead, CBMs redistribute sensitivity, with robustness outcomes depending on perturbation type, class semantic similarity, and concept vocabulary size.
Entities (10)
Relation Signals (7)
Concept Bottleneck Models → isevaluatedon → RIVAL-10
confidence 98% · experiments on CUB and RIVAL10 variants
Concept Bottleneck Models → isevaluatedon → CUB-200-2011
confidence 98% · experiments on CUB and RIVAL10 variants
Interpretability → doesnotimply → Robustness
confidence 97% · interpretability does not inherently confer robustness
Randomized Smoothing → isusedfor → Certified Robustness
confidence 96% · certifiably, using randomized smoothing in latent and concept spaces
Concept Bottleneck Models → isevaluatedagainst → Standard Classifiers
confidence 95% · controlled comparisons between standard classifiers and CBMs
Concept Bottleneck Models → issubjectto → Semantic Perturbations
confidence 93% · discrete semantic interventions in concept space
Concept Bottleneck Models → issubjectto → Geometric Perturbations
confidence 93% · evaluate robustness... under two distinct perturbation types: continuous geometric perturbations
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Concept Bottleneck Models (CBMs) are designed to provide interpretable intermediate representations, yet how such bottlenecks affect robustness remains unclear, with existing studies reporting mixed and sometimes contradictory findings. We argue that these discrepancies arise from conflating different robustness notions and perturbation regimes, rather than from fundamental disagreements about CBMs themselves. To disentangle these factors, we introduce a generator-based evaluation framework that enables controlled comparisons between standard classifiers and CBMs under two distinct perturbation types: continuous geometric perturbations in latent space and discrete semantic interventions in concept space. Within this framework, we evaluate robustness both empirically, via prediction and concept-level sensitivity metrics, and certifiably, using randomized smoothing in latent and concept spaces. Across experiments, we reconcile previously conflicting findings by clarifying when, and in what sense, concept bottlenecks do or do not improve robustness. By further analyzing robustness under varying task conditions, including class semantic similarity and concept vocabulary size, we show that interpretability does not inherently confer robustness. Instead, concept bottlenecks shift where and how sensitivity manifests, revealing a nuanced interpretability robustness trade off that depends critically on the perturbation regime and task structure. Together, our results show that interpretability and robustness are distinct objectives: interpretable intermediate representations do not uniformly improve robustness, but instead redistribute sensitivity across perturbation spaces and model families.
Tags
Links
- Source: https://arxiv.org/abs/2609.38625v1
- Canonical: https://arxiv.org/abs/2609.38625v1
Trouble viewing inline? Open PDF directly →
Full Text
110,081 characters extracted from source content.
Expand or collapse full text
Interpretable but Fragile? Robustness of Concept Bottlenecks under Geometric-Semantic Perturbations Hanwei Zhang Affiliation: Department of Computer Science Affiliation: Saarland University Affiliation: Saarbrücken, Germany Email: zhang@depend.uni-saarland.de Tianma Hu Affiliation: School of Computer Science and Engineering Affiliation: Tianjin University of Technology Affiliation: Tianjin, China Email: hutianma15299@stud.tjut.edu.cn Gaojie Jin Affiliation: Department of Artificial Intelligence & Affiliation: Institute of Artificial Intelligence and Brain Sciences Affiliation: University of Macau Affiliation: Macau SAR, China Email: gaojiejin@um.edu.mo Xu Cheng Affiliation: School of Computer Science and Engineering Affiliation: Tianjin University of Technology Affiliation: Tianjin, China Email: xu.cheng@ieee.org Ronghui Mu †thanks: corresponding author Affiliation: Department of Computer Science Affiliation: University of Exeter Affiliation: Exeter,UK Email: r.mu2@exeter.ac.uk Abstract Concept Bottleneck Models (CBMs) are designed to provide interpretable intermediate representations, yet how such bottlenecks affect robustness remains unclear, with existing studies reporting mixed and sometimes contradictory findings. We argue that these discrepancies arise from conflating different robustness notions and perturbation regimes, rather than from fundamental disagreements about CBMs themselves. To disentangle these factors, we introduce a generator-based evaluation framework that enables controlled comparisons between standard classifiers and CBMs under two distinct perturbation types: continuous geometric perturbations in latent space and discrete semantic interventions in concept space. We further evaluate a prototype-based interpretable model, PixPNet, showing that the same perturbation-to-prediction evaluation and certification pipeline extends beyond concept bottlenecks without requiring alignment between heterogeneous internal representations. Within this framework, we evaluate robustness both empirically, via prediction and concept-level sensitivity metrics, and certifiably, using randomized smoothing in latent and concept spaces. Across experiments on CUB and RIVAL10 variants, we reconcile previously conflicting findings by clarifying when, and in what sense, concept bottlenecks do or do not improve robustness. By further analyzing robustness under varying task conditions, including class semantic similarity and concept vocabulary size, we show that interpretability does not inherently confer robustness. Instead, concept bottlenecks shift where and how sensitivity manifests, revealing a nuanced interpretability robustness trade off that depends critically on the perturbation regime and task structure. Together, our results show that interpretability and robustness are distinct objectives: interpretable intermediate representations do not uniformly improve robustness, but instead redistribute sensitivity across perturbation spaces and model families. 1 Introduction Concept Bottleneck Models (CBMs) (Koh et al., 2020) are a prominent paradigm of interpretable learning, explicitly structuring predictions around human-interpretable concepts. By exposing intermediate concept representations, CBMs provide semantically grounded explanations, support targeted interventions, and offer a modular design that extends naturally in vision and multimodal settings. These properties make CBMs attractive, particularly in high-stakes and regulated domains such as those governed by the EU AI Act (Hermanns et al., 2024; Parliament and of the EU, 2024), where interpretability is often believed to promote robustness. However, the robustness implications of introducing a concept bottleneck remain underexplored, and existing evidence is mixed. Prior work reaches conflicting conclusions regarding the robustness of CBMs. Rasheed et al. (2024) report robustness gains from concept bottlenecks, attributing them to the removal of non-essential variation, with robustness evaluated at the level of final class predictions. In contrast, Sinha et al. (2023) argues that concept bottlenecks can reduce robustness: concept representations themselves are often unstable under perturbations, and exposing an explicit concept interface introduces additional attack surfaces. As illustrated in Figure 1(c), we observe this tension directly: under different task settings of the same dataset, evaluating the same standard classifier and its concept bottleneck counterpart can lead to opposite conclusions about whether concept bottlenecks improve robustness. (a) Evaluation framework (b) Robustness gap summary (c) Concrete contrast Figure 1: Evaluation framework and representative results. (a) Evaluation robustness across geometric vs. semantic perturbations and empirical vs. certified criteria. (b) The heatmap reports representative fcbmf_cbm–fstdf_std robustness gaps using ϵ=0.3ε=0.3, τ=5τ=5, σ=0.10σ=0.10, and ρ=0.10ρ=0.10 for the four columns, respectively. All gaps are oriented so that positive values indicate greater CBM robustness. Detailed computations in Appendix C.1 (c) Certified concept-radius examples at ρ=0.10ρ=0.10 show opposite outcomes across task settings. At first glance, these findings appear irreconcilable. We argue that they instead stem from different notions of robustness and perturbation regimes. In practice, robustness is still most often assessed using small pixel-level perturbations, which lack semantic meaning and fail to reflect changes in underlying concepts. Whether a concept bottleneck improves or degrades robustness depends on where robustness is measured (class vs. concept level), how perturbations are applied, and the task conditions under which the model operates. Making these distinctions explicit is key to understanding the role of concept bottlenecks in robust learning. This leads to the central question of this work: How does introducing an interpretable intermediate representation reshape robustness relative to a standard classifier, and how does this depend on the perturbation space and robustness criterion? To study this systematically, we adopt a concept bottleneck generator (Kulkarni et al., 2025) that maps latent representations to an explicit concept space, enabling controlled concept-level interventions before image generation. The generator is frozen and shared across classifiers, serving only as an experimental instrument. This design allows us to disentangle two fundamentally different types of perturbations: (i) geometric perturbations, which induce continuous variations in the latent embedding space, and (i) semantic perturbations, which directly modify individual concepts. Within this framework, we evaluate robustness empirically by measuring changes in class predictions and intermediate representations, and certifiably by deriving guarantees via randomized smoothing. Standard models and their corresponding CBMs share the same backbone, ensuring a fair comparison. Our analysis focuses on generator-based perturbations to enable controlled evaluation of the robustness induced by concept bottlenecks, without claiming universal robustness guarantees. Our results show that concept bottlenecks do not exhibit a uniform robustness behavior. Instead, they can either improve or degrade robustness, depending on the setting. In particular, we find that the effect of the bottleneck is strongly influenced by the semantic similarity between classes, and the number and relevance of concepts used by the model. These findings provide a unifying explanation for previously conflicting results and clarify the conditions under which concept bottlenecks act as beneficial regularizers versus when they introduce fragility. Overall, our contributions are threefold: (i) A unified robustness perspective for concept bottlenecks. We reconcile previously conflicting findings on CBM robustness by showing that they often arise from conflating different perturbation spaces and robustness criteria. By distinguishing prediction-level and concept-level sensitivity under geometric and semantic perturbations, as well as empirical, certified, and adversarial robustness, we clarify when concept bottlenecks suppress sensitivity and when they instead introduce additional vulnerabilities. Our results emphasize that interpretability and robustness are distinct objectives, and that concept bottlenecks reshape robustness rather than uniformly improving it. (i) A controlled evaluation and certification framework for geometric and semantic robustness. We introduce a generator-based framework (Figure 1(a)) that separates continuous latent-space geometric perturbations from discrete concept-level semantic interventions. The framework enables matched comparisons under a common perturbation-to-prediction interface, together with empirical robustness metrics and perturbation-specific randomized-smoothing certificates in latent and concept spaces. Importantly, the evaluation interface does not require different classifier architectures to share or align their internal representations. (i) Empirical and certified analysis of when robustness gains emerge. Through extensive experiments on CUB-200-2011 and RIVAL-10 variants, we show that concept bottlenecks do not provide an inherent robustness advantage (e.g., Figure 1(b)). Instead, robustness depends strongly on the perturbation regime, robustness criterion, class semantic similarity, and concept vocabulary design. These findings provide practical guidance for designing CBMs that better balance interpretability and robustness. We further validate the broader applicability of the evaluation pipeline on PixPNet, a prototype-based interpretable model, showing that the same generator-defined evaluation and certification procedure can be applied beyond CBMs without requiring alignment between generator concepts and model-specific prototypes. 2 Related Work Robustness of Concept Bottlenecks. Early work on interpretable models highlighted connections between interpretability and robustness, with Self-Explaining Neural Networks (SENN) enforcing robustness as the stability of explanations under small perturbations (Sawada and Nakamura, 2022). More recent studies directly examine robustness in Concept Bottleneck Models (CBMs). Sinha et al. (2023) show that interpretability alone does not guarantee robustness, demonstrating that concept representations can be fragile under adversarial perturbations and that explicit concept interfaces introduce new attack surfaces, revealing a tension between concept-level stability and adversarial robustness. In contrast, Rasheed et al. (2024) find that CBMs, particularly sequentially trained variants, can achieve higher adversarial accuracy than end-to-end models under weak to moderate attacks, suggesting that conceptual bottlenecks may act as an information-filtering regularizer at the prediction level. Related observations beyond CBMs further indicate that highly interpretable representations can remain non-robust (Li et al., 2026), while recent extensions such as language-guided and flexible CBMs improve scalability and adaptability without addressing robustness explicitly (Yu et al., 2025; Du et al., 2026). Overall, prior work reports seemingly conflicting conclusions, which we argue arise from differences in robustness definitions, perturbation models, and task settings rather than fundamental inconsistencies. Randomized Smoothing for Certified Robustness. Randomized smoothing provides a general framework for certifying robustness by constructing a smoothed classifier through noise injection and aggregation, yielding probabilistic guarantees against bounded perturbations (Cohen et al., 2019). While most work applies smoothing in pixel space, recent studies have explored extensions to learned representations or structured perturbations (Hao et al., 2022; Wang et al., 2026; Mu et al., 2023; Mu et al., 2024). In this work, randomized smoothing is used primarily as a model-agnostic certification and comparison tool. The resulting smoothed predictor can also improve empirical robust accuracy in some adversarial settings, but defense is not the primary objective of our analysis. Interpretable intermediate architectures beyond CBMs. Interpretable intermediate representations are also exposed by post-hoc concept methods (Yuksekgonul et al., 2022) and prototype-based architectures (Zhang et al., 2021; Carmichael et al., 2024; Sicre et al., 2023). PCBM (Yuksekgonul et al., 2022) constructs concept representations using CAV-style directions, while prototype-based models such as PixPNet (Carmichael et al., 2024) ground predictions in localized parts. Although these models expose different internal representations, our evaluation does not require these representations to be aligned: the generator defines a common external perturbation interface, and robustness is measured at the resulting prediction level. 3 Methodology 3.1 Perturbation Models Let g:d→g: ^d denote a pretrained generator that maps a latent noise vector ∈dz∈ ^d to an image =g()x=g(z). We assume that g admits a decomposition g=g2∘g1g=g_2 g_1, where g1g_1 maps the latent code to an intermediate representation and g2g_2 maps this representation to the image space. Following concept bottleneck generators, an encoder–decoder pair (E,D)(E,D) is intermediate layer, including an explicit concept space representation =E(g1())∈ℝKc=E(g_1(z)) ^K such that image generation can be written as =g()=g2(D(E(g1())))x=g(z)=g_2(D(E(g_1(z)))). Each concept cic_i corresponds to a semantically meaningful attribute and is associated with a binary state indicating the presence or absence of the concept. This representation allows us to distinguish two fundamentally different perturbation regimes: Geometric perturbations are applied in latent space by adding noise to the latent code, ~=+δ,‖δ‖2≤ϵ z=z+δ,\|δ\|_2≤ε, which induces a continuous variation along the generator manifold and results in a geometrically perturbed image ~=g(~) x=g( z). Semantic perturbations are applied directly in concept space by modifying the concept vector c. We uniformly sample a subset S⊆1,⋯,KS \1,·s,K\ of size |S|=τ|S|=τ and, for each i∈Si∈ S, swap the binary representation of corresponding concept from [ci+,ci−][c_i^+,c_i^-] to [ci−,ci+][c_i^-,c_i^+], simulating changes in high-level semantic attributes while preserving the underlying geometric structure. To detect concept changes under perturbation, we train a set of binary concept classifiers ()∈0,1KC(c)∈\0,1\^K that map a concept representation to a binary concept state. We then define a pseudo-labeling function ϕi(,~):=[(ci)≠(ci~)] _i(c, c) := 1[C(c_i) ( c_i)] to determine whether a given concept cic_i has flipped under perturbation. 3.2 Statistical Robustness Analysis Let f:→f:X denote a classifier operating on generated images. We consider two classes of models: standard classifier fstdf_std directly maps the input image to a label; concept bottleneck model is defined as a tuple fcbm:=(r,h)f_cbm :=(r,h), where r:→Kr:X→ ^K predicts a concept representation from image and h:K→h: ^K maps concepts to the final prediction. This setup enables a controlled comparison of how standard and concept-based classifiers respond to geometric perturbations in latent space and semantic perturbations in concept space. Further methodological details are provided in Appendix B.1. Empirical Robustness. To quantify empirical robustness under geometric and semantic perturbations, we measure changes at the concept and prediction levels using the following metrics: 1. Concept Flip Rate (CFR) measures the average fraction of concepts whose predicted states change under perturbation: CFR:==E()[1K∑i=1Kϕi(,~)];CFR\; :=\; E_\c=E(z)\ [ 1K _i=1^K _i(c, c) ]; 2. Prediction Flip Rate (PFR) measures the probability that the classifier’s predicted label changes under perturbation: PFR(f):=[[f()≠f(~)]];PFR(f)\; :=\; E_x [ 1 [f(x)≠ f( x) ] ]; 3. Decision Function Sensitivity (DFS) quantifies the sensitivity of a concept bottleneck classifier fcbmf_cbm by measuring the change in its internal concept representation: DFS:=[‖r(g())−r(g(~))‖2].DFS\; :=\; E_z [ \|r (g(z) )-r (g( z) ) \|_2 ]. CFR and DFS characterize concept-level robustness, capturing both true concept flips induced by the generator and the sensitivity of the concept bottleneck classifier, while PFR measures robustness at the prediction level. Together, these metrics allow us to study how concept instability propagates to model predictions. Certified Robustness. We use randomized smoothing to certify robustness in the two perturbation spaces induced by the generator. Latent-space smoothing. For geometric robustness, we apply Gaussian smoothing in latent space: f^G()=argmaxy∈ℙδ∼(0,σ2I)[f(g(+δ))=y]. f_G(z)= _y P_δ (0,σ^2I)\! [f(g(z+δ))=y ]. (1) Theorem 3.1 (Latent-space Gaussian certificate (Cohen et al., 2019)). Let A and B be the top and runner-up classes of f^G f_G with probabilities pA>pBp_A>p_B. Then f^G f_G is constant for all latent perturbations ‖ϵ‖2≤RG\|ε\|_2≤ R_G, where RG=σ2(Φ−1(pA)−Φ−1(pB)).R_G= σ2 ( ^-1(p_A)- ^-1(p_B) ). (2) Concept-space smoothing. For semantic robustness, we sample independent concept flips ηi∼Bernoulli(ρ) _i (ρ) and write ℱ(,η)F(c,η) for the flipped concept representation. The smoothed classifier is f^C()=argmaxy∈ℙη∼Bernoulli(ρ)K[f(g2(D(ℱ(,η))))=y]. f_C(c)= _y P_η (ρ)^K\! [f\! (g_2(D(F(c,η))) )=y ]. (3) Theorem 3.2 (Concept-space ℓ0 _0 certificate (Lee et al., 2019)). Let y⋆y be the top class of f^C f_C, and let p¯y⋆ p_y and p¯y p_y be conservative confidence bounds on the top and competing class probabilities. For any concept-flip budget Δ , if ℒΔ(p¯y⋆)>Δ(p¯y),∀y≠y⋆,L_ ( p_y )>U_ ( p_y), ∀ y≠ y , (4) then, with probability at least 1−α1-α over Monte Carlo estimation, f^C(~)=y⋆ f_C( c)=y for all ‖(~)−()‖0≤Δ\|C( c)-C(c)\|_0≤ . We report the largest certified semantic radius Δ⋆()=maxΔ:ℒΔ(p¯y⋆)>Δ(p¯y),∀y≠y⋆. (c)= \ :L_ ( p_y )>U_ ( p_y),\ ∀ y≠ y \. (5) All confidence bounds use Clopper–Pearson intervals; full transfer-function definitions, proofs, and algorithms are in Appendix B.2. 4 Experiments In this section, we empirically and certifiably evaluate the robustness of fstdf_std and fcbmf_cbm under controlled geometric and semantic perturbations. Due to page limitations, we report a subset of the experimental results in the main paper; the complete results, including additional statistical analyses and backbone comparisons, are provided in Appendix C. Across perturbation regimes, task conditions, and evaluation metrics, our experiments reveal a consistent pattern: Concept bottlenecks do not induce monotonic robustness improvements; instead, they shift where and how sensitivity manifests. 4.1 Experimental Setup We conduct experiments on two datasets with rich concept annotations: CUB-200-2011 and RIVAL-10. CUB-200-2011 (Wah et al., 2011) is a fine-grained bird classification benchmark consisting of 11,788 images from 200 species, each annotated with 112 expert-defined visual concepts.11 1 https://w.vision.caltech.edu/datasets/cub_200_2011/ RIVAL-10 (Moayeri et al., 2022) aligns CIFAR-10 (Krizhevsky et al., 2009) classes with ImageNet (Krizhevsky et al., 2012) categories and contains approximately 26,000 images annotated with 18 visual attributes and corresponding segmentations. For CUB-200-2011, we select K=15K=15 concepts by default based on empirical frequency. We adopt a post-hoc concept bottleneck autoencoder (CB-AE) (Kulkarni et al., 2025) built on StyleGAN (Karras et al., 2020) as the generator, and train it on CUB-200-2011 to induce an explicit concept space aligned with these concepts. For RIVAL-10, models are trained on the official training split of 21,098 images and evaluated on the test split of 5,286 images, with all reported results computed on the test set. We adopt BigGAN (Brock et al., 2018) as the base image generator and treat all 18 annotated attributes as concepts by default. For classification, all compared classifiers share the same pretrained backbone to ensure a fair and controlled comparison. For both datasets, we use a ResNet-50 (He et al., 2016) for the standard classifier fstdf_std and a ResNet-50 version of the post-hoc concept bottleneck model (PCBM) (Yuksekgonul et al., 2022) for the concept-based classifier fcbmf_cbm. We restrict evaluation to generated inputs on which fstdf_std and fcbmf_cbm agree, denoted by agreeS_agree, and use the shared prediction as a reference label. We use 100 images per run to balance computational cost with repeated-subset statistical testing. 4.1.1 Empirical robustness results 0.050.10.30.50.8002020404060608080100100ϵ PFR (%) ↓ perturbations001122⋅10−2· 10^-2RIVAL PFR (%) ↓ 1122551010002020404060608080100100τ PFR (%) ↓ perturbations00224466RIVAL PFR (%) ↓ Color code: CUB-200-2011 RIVAL-10 Line code: fstdf_std fcbmf_cbm Figure 2: Empirical robustness under geometric and semantic perturbations. Orange (left axis): CUB-200-2011; blue (right axis): RIVAL-10. Dataset-specific y-axis scales are used for clarity. 11223344552020404060608080100100Decision Function Sensitivity (DFS)PFR % ↓ 005510102020404060608080100100Concept Flip Rate (CFR)PFR % ↓ Color code: Geometric Semantic Figure 3: Prediction sensitivity in concept-level measures on CUB-200-2011. To quantify variability due to subset selection, we repeat the evaluation on 55 independent random subsets of size 200 drawn uniformly without replacement from the pool (subsets may overlap). For each subset, both fstdf_std and fcbmf_cbm are evaluated on the same inputs (paired design), yielding five paired measurements per metric. We then test H0:Δ=0H_0: =0 using a paired two-sided test across the 55 subset-level measurements, where Δ denotes the per-subset difference between methods. When multiple budgets and metrics are tested, we additionally report Holm–Bonferroni corrected p-values in Appendix C.3. As shown in Figure 2, the prediction flip rate (PFR) increases with perturbation strength. On CUB-200-2011, fcbmf_cbm exhibits slightly higher PFR than fstdf_std across most geometric and semantic budgets, indicating greater sensitivity. In contrast, on RIVAL-10, both models remain nearly perfectly robust over the same perturbation ranges. As shown in Figure 3, raw concept flip rates (CFR) alone do not fully account for prediction instability. In contrast, decision function sensitivity (DFS) shows a more consistent relationship with PFR, suggesting that instability in the CBM decision representation better explains prediction flips than the number of concept changes. This arises because CBMs primarily rely on task-relevant concepts and are relatively insensitive to perturbations of irrelevant ones, aligning with the intuition that concept bottlenecks can act as a form of regularization under structured semantic noise. Additional analysis of DFS, CFR, and PFR, along with the complete numerical results, is provided in Appendix C.3. 4.2 Randomised Smoothing Experiments Empirical perturbations characterize sensitivity under sampled inputs but do not provide guarantees over all perturbations within a neighborhood. To assess whether these trends persist under worst-case perturbations, we evaluate certified robustness using randomized smoothing in latent (ℓ2 _2) and concept (ℓ0 _0) spaces. Unless otherwise stated, we use n=200n=200 inputs sampled from agreeS_agree and estimate smoothed class probabilities with N=1000N=1000 Monte Carlo samples. Confidence bounds are computed using one-sided Clopper–Pearson intervals at level 1−α=95%1-α=95\%. All smoothed and certified accuracies are measured with respect to the reference labels defined by the agreement set. To quantify variability, we repeat the evaluation over five independently sampled subsets of size 200200. For latent-space smoothing, we use Gaussian perturbations δ∼(0,σ2I)δ (0,σ^2I) with σ∈0.01,0.05,0.10,0.30σ∈\0.01,0.05,0.10,0.30\ and report the average certified ℓ2 _2 radius R over correctly classified, non abstained samples. For concept space smoothing, each concept is independently flipped with probability ρ, and we report the average certified ℓ0 _0 concept flip radius Δ over correctly classified, non-abstained samples. In both settings, smoothed accuracy (SmAcc) counts abstentions as incorrect. Figure 4summarizes the results. Under latent-space smoothing, the fstdf_std generally achieves higher SmAcc and larger certified radii than the fcbmf_cbm on CUB-200-2011, especially as σ increases. This indicates that, on generator native inputs, fcbmf_cbm does not improve certifiable robustness to continuous latent perturbations. On the original RIVAL-10 split, both models are saturated: SmAcc remains at 100% and certified radii are nearly identical across all σ, reflecting the strong class separability of this benchmark and indicating that samples remain far from decision boundaries in latent space. Under concept-space smoothing, increasing the flip probability ρ reduces both SmAcc and the certified ℓ0 _0 radius, indicating that stronger semantic randomization reduces the probability margin between the top class and competing classes. On CUB-200-2011, the fcbmf_cbm does not consistently improve certified concept robustness: although it slightly improves SmAcc for some noise levels, its certified radius is comparable to or smaller than that of the ResNet baseline. On RIVAL-10, the certificates are again saturated, motivating the harder RIVAL-10 variants studied in section 5. 0.010.050.100.3060608080100100σ (%) ↑ (a) Latent0.010.050.100.30000.20.20.40.40.60.60.80.8σ ↑ (b) Latent5⋅10−25· 10^-20.10.10.20.2404060608080100100ρ (%) ↑ (c) Concept5⋅10−25· 10^-20.10.10.20.200551010ρΔ ↑ (d) Concept Color code: CUB-200-2011 RIVAL-10 Line code: fstdf_std fcbmf_cbm Figure 4: Certified robustness under latent-space and concept-space randomized smoothing. 4.3 Adversarial Evaluation We apply the APGD-CE attack (Croce and Hein, 2020) to the entire pipeline, including the generator, for a practical stress test of adversarial robustness rather than a head-to-head robustness comparison. As shown in Figure 5, randomized smoothing consistently improves robust accuracy under both latent-space and concept-space attacks, with larger gains at higher attack strengths. While subsection 4.2 indicates that fcbmf_cbm does not outperform fstdf_std under randomized smoothing in either latent or concept space, adversarial evaluations reveal a complementary pattern: fcbmf_cbm is more robust to latent-space attacks but less robust to concept-space attacks. This reflects the dual role of concept bottlenecks as both a new attack interface and a regularizing mechanism against geometric perturbations, helping resolve prior conflicting conclusions about whether concept bottlenecks introduce robustness. latent space zzconcept space c002020404060608080Robust Accuracy (%) ↑ (a) ϵ=0.03ε=0.03latent space zzconcept space c002020404060608080Robust Accuracy (%) ↑ (b) ϵ=0.01ε=0.01Color code: Vanilla Smoothed Fill code: fstdf_std fcbmf_cbm Figure 5: Adversarial robustness under APGD-CE (Croce and Hein, 2020) attacks on the CUB generated dataset. Takeaway. (1) Concept bottlenecks do not uniformly improve prediction-level robustness, but they can offer increased robustness to large semantic perturbations by down-weighting irrelevant concepts, acting as a form of regularization. (2) Concept bottlenecks introduce a robustness trade-off: they improve robustness against geometric adversarial perturbations while increasing vulnerability to concept-space attacks. Our unified framework explains previously contradictory observations and clarifies how future designs can better exploit the robustness trade-off. 5 Robustness under Task Conditions The preceding sections show that concept bottleneck robustness varies substantially across datasets, despite using identical model architectures. This suggests that robustness is shaped not only by the presence of a bottleneck, but also by task-level factors. To isolate these effects, we conduct controlled studies on RIVAL-10 by varying (i) class structure and semantic similarity, and (i) concept vocabulary size, while keeping the backbone and training protocol fixed. 5.1 Effect of Class Structure and Semantic Similarity 0.050.050.10.10.30.30.50.50.80.8000.50.5111.51.522ϵ (%) ↓ (a) Dissimilar0.050.050.10.10.30.30.50.50.80.8005510101515ϵ (%) ↓ (b) Similar+10 dissim+5 dissim+5 sim +10 sim−4-4−2-20022Δ ↓ (c) fcbm−fstdf_cbm-f_stdGeometricSemanticColor code: +5 classes +10 classes Line code: fstdf_std fcbmf_cbm Figure 6: Effect of class structure and semantic similarity on task-conditional robustness under geometric perturbations. (a,b) PFR v.s. ϵε for dissimilar and similar class settings. (c) Difference in PFR (denoted as Δ PFR) between fcbmf_cbm and fstdf_std when ϵ=0.8ε=0.8. Lower Δ PFR favors fcbmf_cbm. To study the effect of class similarity on robustness, we augment the original RIVAL-10 classification task with additional classes designed to vary semantic proximity to the original categories. Specifically, we construct two groups of new classes: similar classes, which are semantically close to the original RIVAL-10 categories, and dissimilar classes, which are semantically distant. Using these groups, we define four task configurations by adding 5 (or 10) similar (or dissimilar) classes, to the original 10-class task, yielding classification problems of increasing size and semantic difficulty. Across all configurations, we retain the original 18 concept attributes provided by RIVAL-10, ensuring that only the class structure changes while the concept vocabulary remains fixed. We then evaluate how robustness varies across these controlled settings. Across all settings, class semantic similarity emerges as the dominant factor shaping robustness differences between fstdf_std and fcbmf_cbm. Figure 6 shows that semantic class dissimilarity generally improves robustness, with classifiers exhibiting lower prediction instability when classes are well separated. Across configurations, fcbmf_cbm tends to match or slightly outperform fstdf_std under geometric perturbations and mild semantic changes, but becomes increasingly sensitive under large semantic perturbations, reflecting greater vulnerability at the concept level. We observe consistent trends across both empirical robustness evaluations under geometric and semantic perturbations and certified robustness analyses in latent and concept spaces. In addition, randomized smoothing yields larger robustness gains for fcbmf_cbm in several settings, suggesting that smoothing can interact favorably with structured concept representations, albeit without providing a uniform robustness advantage. 5.2 Effect of Concept Vocabulary Size We study the effect of concept set size on robustness using the original RIVAL-10 task. Starting from the original set of 18 annotated concepts, we use a large language model to generate an additional 18 concepts that are semantically relevant to these classes. This yields three concept configurations with increasing granularity: 27 and 36 concepts. For each configuration, we evaluate robustness while keeping the classification task fixed, allowing us to isolate the impact of concept vocabulary size on model robustness. Figure 7shows that changing the concept vocabulary affects the robustness behavior of both fstdf_std and fcbmf_cbm, but the direction of the effect depends on the perturbation setting. Under geometric perturbations, the original RIVAL-10 setting is nearly saturated. Thus, geometric perturbations provide little evidence that either model is more robust. Semantic concept flips are more diagnostic. With 27 concepts, fcbmf_cbm is slightly more stable at the largest flip budget (9.299.29 vs. 9.429.42 PFR), whereas with 36 concepts the trend reverses (3.113.11 vs. 2.652.65 PFR). This suggests that the effect of the concept vocabulary is not monotonic. One possible explanation is that, in the 27 concept setting, some flipped concepts are redundant or weakly used by the fcbmf_cbm decision head, allowing the model to absorb large semantic flips slightly better. In the 36 concept setting, however, the additional concepts may introduce more correlated or unevenly useful attributes; perturbing them can destabilize the fcbmf_cbm concept representation without yielding a consistent semantic robustness benefit. The randomized smoothing certified results support this interpretation. In latent space, the fstdf_std usually obtains larger certified radii than fcbmf_cbm across both 27 and 36 concept settings. This is consistent with the fact that latent smoothing perturbs the generated image manifold continuously; fcbmf_cbm must first recover a stable concept representation from each perturbed image, so instability in this intermediate representation can reduce the smoothed class margin. In concept space, where perturbations are applied directly at the concept interface, the two models are much closer: fcbmf_cbm is sometimes marginally better and sometimes marginally worse. Overall, concept vocabulary size changes the measured robustness profile, but it does not induce a uniform robustness trend for fcbmf_cbm relative to the fstdf_std. Color code: K=27 K=36 Line code: fstdf_std fcbmf_cbm Color: K=27 K=36 Fill: fstdf_std fcbmf_cbm 0.050.100.300.500.8000224466⋅10−2· 10^-2ϵ % ↓ (a) Geometric perturbation 112255101000224466881010τ % ↓ (b) Semantic perturbation 0.010.050.100.30000.50.511σ ↑ (c) Certified in Latent space 0.050.100.20005510101515ρΔ⋆ ↑ (d) Certified in Concept space Figure 7: Impact of concept vocabulary size on robustness. Concept-vocabulary ablation on RIVAL-10. (a,b) Empirical PFR under geometric latent perturbations and semantic concept flips. (c,d) Certified robustness under Gaussian latent smoothing and Bernoulli concept smoothing. Takeaway. (1) Lower inter-class similarity is generally associated with higher classifier robustness. (2) As class similarity increases, concept bottlenecks tend to remain more stable and exhibit a slower degradation in robustness. (3) Increasing the number of concepts does not monotonically improve robustness: additional concepts can help when perturbations target redundant attributes, but may also introduce instability when concepts are correlated or weakly grounded. Design Implications for Robust Concept Bottlenecks. These results indicate that robustness differences between fstdf_std and fcbmf_cbm are primarily governed by task structure. This suggests that concept bottlenecks are particularly beneficial for classification tasks with high inter-class similarity, provided that the concept vocabulary is chosen carefully. Our findings further highlight that robustness should be treated as a key criterion when selecting and designing concept sets. While concept bottlenecks reduce sensitivity to semantic perturbations and improve robustness to adversarial geometric perturbations, they are less robust to adversarial semantic attacks. Consequently, improving the robustness of concept bottlenecks requires focusing on enhancing robustness within the concept space itself. 6 Limitation and Conclusion Limitation. Our analysis relies on generator-native inputs to disentangle geometric and semantic perturbations. While this enables controlled robustness evaluation, results may differ for real-world inputs outside the generator manifold. We focus on post-hoc CBMs with frozen backbones; end-to-end CBMs or alternative concept learning paradigms may exhibit different behaviors. Finally, our conclusions are limited to robustness under structured geometric and semantic perturbations and do not replace standard pixel-space adversarial robustness evaluations. Conclusion. We introduced a generator-based framework that disentangles continuous latent/geometric perturbations from discrete concept/semantic interventions and used it to systematically evaluate robustness under empirical, certified, and adversarial criteria. Across CUB-200-2011 and RIVAL-10 variants, we find that concept bottlenecks do not confer uniform robustness improvements. Instead, they reshape where and how sensitivity emerges, with robustness depending strongly on the perturbation space, robustness criterion, task structure, and concept vocabulary design. Within our controlled evaluation setting, these results clarify when concept bottlenecks suppress irrelevant variation and when they introduce additional sensitivity through the semantic interface. In particular, class semantic similarity and concept granularity emerge as important design factors governing this trade-off. Our findings reinforce that interpretability and robustness are distinct objectives: concept bottlenecks should not be treated as an implicit robustness mechanism, but as a modeling choice whose robustness consequences depend on how concepts are defined and used. Finally, although CBMs provide our primary analytical setting, the additional PixPNet experiments show that the same generator-defined perturbation-to-prediction evaluation and randomized-smoothing certification pipeline can be applied to a prototype-based interpretable classifier without requiring alignment between heterogeneous internal representations. This broader applicability suggests a path toward evaluating robustness across a wider family of interpretable architectures while preserving the controlled perturbation framework developed here. Acknowledgment This work received support from DFG under grant No. 389792660 (TRR 248), and No. 547583482. This work was also supported by The Royal Society Grant (Ensuring Trustworthy AI: Robustness Certification for Large Language Models)[Reference RGS 2\252444]. GJ is supported by the University of Macau under grants MYRG-SRG2026-00033-FIC. This work was supported by the National Natural Science Foundation of China (NSF) (T2422015, 62306212) and the Beijing-Tianjin-Hebei Natural Science Foundation Cooperation Project (25JJJJC0009). References Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018. Carlini et al. (2019) Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705, 2019. Carmichael et al. (2024) Zachariah Carmichael, Suhas Lohit, Anoop Cherian, Michael J Jones, and Walter J Scheirer. Pixel-grounded prototypical part networks. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4756–4767. IEEE, 2024. Chauhan et al. (2023) Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. In Proceedings of the aaai conference on artificial intelligence, volume 37, pages 5948–5955, 2023. Chen et al. (2018) Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018. Cohen et al. (2019) Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pages 1310–1320. PMLR, 2019. Croce and Hein (2020) Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206–2216. PMLR, 2020. Ding et al. (2020) Zheng Ding, Yifan Xu, Weijian Xu, Gaurav Parmar, Yang Yang, Max Welling, and Zhuowen Tu. Guided variational autoencoder for disentanglement learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7920–7929, 2020. Du et al. (2026) Xingbo Du, Qiantong Dou, Lei Fan, and Rui Zhang. Flexible concept bottleneck model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 20932–20940, 2026. Hao et al. (2022) Zhongkai Hao, Chengyang Ying, Yinpeng Dong, Hang Su, Jian Song, and Jun Zhu. Gsmooth: Certified robustness against semantic transformations via generalized randomized smoothing. In International Conference on Machine Learning, pages 8465–8483. PMLR, 2022. Havasi et al. (2022) Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems, 35:23386–23397, 2022. He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. Hermanns et al. (2024) Holger Hermanns, Anne Lauber-Rönsberg, Philip Meinel, Sarah Sterz, and Hanwei Zhang. Ai act for the working programmer. In International Conference on Bridging the Gap between AI and Reality, pages 74–98. Springer, 2024. Higgins et al. (2017) Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017. Hu et al. (2024) Lijie Hu, Chenyang Ren, Zhengyu Hu, Hongbin Lin, Cheng-Long Wang, Hui Xiong, Jingfeng Zhang, and Di Wang. Editable concept bottleneck models. arXiv preprint arXiv:2405.15476, 2024. Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017. Huang et al. (2020) Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review, 37:100270, 2020. Ismail et al. (2023) Aya Abdelsalam Ismail, Julius Adebayo, Hector Corrada Bravo, Stephen Ra, and Kyunghyun Cho. Concept bottleneck generative models. In The Twelfth International Conference on Learning Representations, 2023. Jin et al. (2022) Gaojie Jin, Xinping Yi, Wei Huang, Sven Schewe, and Xiaowei Huang. Enhancing adversarial training with second-order statistics of weights. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15273–15283, 2022. Jin et al. (2025) Gaojie Jin, Sihao Wu, Jiaxu Liu, Tianjin Huang, and Ronghui Mu. Enhancing robust fairness via confusional spectral regularization. In International Conference on Learning Representations, volume 2025, pages 52945–52966, 2025. Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020. Kim et al. (2023) Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. arXiv preprint arXiv:2306.01574, 2023. Koh et al. (2020) Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pages 5338–5348. PMLR, 2020. Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012. Kulkarni et al. (2025) Akshay Kulkarni, Ge Yan, Chung-En Sun, Tuomas Oikarinen, and Tsui-Wei Weng. Interpretable generative models through post-hoc concept bottlenecks. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8162–8171, 2025. Lee et al. (2019) Guang-He Lee, Yang Yuan, Shiyu Chang, and Tommi Jaakkola. Tight certificates of adversarial robustness for randomly smoothed classifiers. Advances in Neural Information Processing Systems, 32, 2019. Li et al. (2026) Aaron J Li, Suraj Srinivas, Usha Bhalla, and Himabindu Lakkaraju. Evaluating adversarial robustness of concept representations in sparse autoencoders. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5940–5957, 2026. Meng et al. (2022) Mark Huasong Meng, Guangdong Bai, Sin Gee Teo, Zhe Hou, Yan Xiao, Yun Lin, and Jin Song Dong. Adversarial robustness of deep neural networks: A survey from a formal verification perspective. IEEE Transactions on Dependable and Secure Computing, 2022. Moayeri et al. (2022) Mazda Moayeri, Phillip Pope, Yogesh Balaji, and Soheil Feizi. A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19087–19097, 2022. Mu et al. (2023) Ronghui Mu, Wenjie Ruan, Leandro Soriano Marcolino, Gaojie Jin, and Qiang Ni. Certified policy smoothing for cooperative multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15046–15054, 2023. Mu et al. (2024) Ronghui Mu, Leandro Soriano Marcolino, Yanghao Zhang, Tianle Zhang, Xiaowei Huang, and Wenjie Ruan. Reward certification for policy smoothed reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 21429–21437, 2024. Parliament and of the EU (2024) European Parliament and Council of the EU. Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence and amending regulations (ec) no 300/2008, (eu) no 167/2013, (eu) no 168/2013, (eu) 2018/858, (eu) 2018/1139 and (eu) 2019/2144 and directives 2014/90/eu, (eu) 2016/797 and (eu) 2020/1828 (artificial intelligence act), 2024. URL https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=OJ:L_202401689. Rasheed et al. (2024) Bader Rasheed, Mohamed Abdelhamid, Adil Khan, Igor Menezes, and Asad Masood Khatak. Exploring the impact of conceptual bottlenecks on adversarial robustness of deep neural networks. IEEE Access, 12:131323–131335, 2024. Sawada and Nakamura (2022) Yoshihide Sawada and Keigo Nakamura. C-senn: Contrastive self-explaining neural network. arXiv preprint arXiv:2206.09575, 2022. Shang et al. (2024) Chenming Shang, Shiji Zhou, Hengyuan Zhang, Xinzhe Ni, Yujiu Yang, and Yuwang Wang. Incremental residual concept bottleneck models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11030–11040, 2024. Sicre et al. (2023) Ronan Sicre, Hanwei Zhang, Julien Dejasmin, Chiheb Daaloul, Stéphane Ayache, and Thierry Artières. Dp-net: learning discriminative parts for image recognition. In 2023 IEEE international conference on image processing (ICIP), pages 1230–1234. IEEE, 2023. Sinha et al. (2023) Sanchit Sinha, Mengdi Huai, Jianhui Sun, and Aidong Zhang. Understanding and enhancing robustness of concept-based models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15127–15135, 2023. Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. Vandenhirtz et al. (2024) Moritz Vandenhirtz, Sonia Laguna, Ričards Marcinkevičs, and Julia Vogt. Stochastic concept bottleneck models. Advances in Neural Information Processing Systems, 37:51787–51810, 2024. Wagh et al. (2022) Neeraj Wagh, Jionghao Wei, Samarth Rawal, Brent M Berry, and Yogatheesan Varatharajah. Evaluating latent space robustness and uncertainty of eeg-ml models under realistic distribution shifts. Advances in Neural Information Processing Systems, 35:21142–21156, 2022. Wah et al. (2011) C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. Wang et al. (2021) Binghui Wang, Jinyuan Jia, Xiaoyu Cao, and Neil Zhenqiang Gong. Certified robustness of graph neural networks against adversarial structural perturbation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1645–1653, 2021. Wang et al. (2026) Zixia Wang, Gaojie Jin, Jia Hu, and Ronghui Mu. Clucert: Certifying llm robustness via clustering-guided denoising smoothing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 37998–38006, 2026. Yu et al. (2025) Lu Yu, Haoyu Han, Zhe Tao, Hantao Yao, and Changsheng Xu. Language guided concept bottleneck models for interpretable continual learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14976–14986, 2025. Yuksekgonul et al. (2022) Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint arXiv:2205.15480, 2022. Zhang et al. (2026) Hanwei Zhang, Luo Cheng, Rui Wen, Yang Zhang, Lijun Zhang, and Holger Hermanns. Sl-cbm: Enhancing concept bottleneck models with semantic locality for better interpretability, 2026. URL https://arxiv.org/abs/2601.12804. Zhang et al. (2021) Xing-Xing Zhang, Zhen-Feng Zhu, Ya-Wei Zhao, and Yao Zhao. Prototype learning in machine learning: A literature review. Journal of Software, 33(10):3732–3753, 2021. To support reproducibility and completeness, the supplementary material provides extended background and related work (Appendix A), additional methodological details (Appendix B), including full randomized smoothing proofs and algorithms, comprehensive experimental results (Appendix C), and cross-family applicability experiments (Appendix D). Appendix A Background and Related Work Concept Bottleneck Models. Concept Bottleneck Models (CBMs) were introduced by Koh et al. [2020] as a framework that decomposes prediction into concept extraction followed by concept-based decision making. Building on this idea, subsequent work has explored CBMs along several directions. Post-hoc CBMs (PCBM) leverage frozen vision or multimodal backbones and add lightweight projection layers to obtain concept representations [Yuksekgonul et al., 2022]. Other approaches focus on improving the quality, uncertainty, or robustness of learned concepts [Vandenhirtz et al., 2024, Kim et al., 2023, Havasi et al., 2022, Zhang et al., 2026], while additional work aims to increase the flexibility and adaptability of concept representations through interactive or editable mechanisms [Chauhan et al., 2023, Hu et al., 2024, Shang et al., 2024]. Robustness of Concept Bottlenecks. Early work on interpretable models highlighted connections between interpretability and robustness, with Self-Explaining Neural Networks (SENN) enforcing robustness as the stability of explanations under small perturbations [Sawada and Nakamura, 2022]. More recent studies directly examine robustness in Concept Bottleneck Models (CBMs). Sinha et al. [2023] show that interpretability alone does not guarantee robustness, demonstrating that concept representations can be fragile under adversarial perturbations and that explicit concept interfaces introduce new attack surfaces, revealing a tension between concept-level stability and adversarial robustness. In contrast, Rasheed et al. [2024] find that CBMs, particularly sequentially trained variants, can achieve higher adversarial accuracy than end-to-end models under weak to moderate attacks, suggesting that conceptual bottlenecks may act as an information-filtering regularizer at the prediction level. Related observations beyond CBMs further indicate that highly interpretable representations can remain non-robust [Li et al., 2026], while recent extensions such as language-guided and flexible CBMs improve scalability and adaptability without addressing robustness explicitly [Yu et al., 2025, Du et al., 2026]. Overall, prior work reports seemingly conflicting conclusions, which we argue arise from differences in robustness definitions, perturbation models, and task settings rather than fundamental inconsistencies. Concept-Based Generative Models. Prior work explores disentangled or concept-based generative models for controllable generation [Chen et al., 2018, Ding et al., 2020, Higgins et al., 2017, Ismail et al., 2023], while CB-AE [Kulkarni et al., 2025] enables concept-level control by augmenting frozen pretrained generators with minimal supervision. In this work, we use CB-AE [Kulkarni et al., 2025] solely as an experimental instrument to enable controlled geometric and semantic perturbations. Robustness Beyond Pixel Space. Deep neural networks are known to be highly sensitive to small input perturbations, motivating extensive work on adversarial robustness and robustness evaluation [Szegedy et al., 2013, Carlini et al., 2019, Meng et al., 2022, Huang et al., 2020]. However, most studies focus on pixel-level perturbations, which often lack semantic meaning and do not reflect changes in underlying factors of variation. Recent work has therefore explored robustness beyond the pixel space, including robustness to perturbations in learned representations or structured transformations [Hao et al., 2022, Wang et al., 2021, Wagh et al., 2022, Jin et al., 2022, Jin et al., 2025]. Despite this progress, how architectural choices such as concept-aligned or interpretable representations affect robustness under semantic perturbations remains underexplored, motivating our focus on concept-level robustness. Randomized Smoothing for Certified Robustness. Randomized smoothing [Cohen et al., 2019] was developed to evaluate probabilistic certified robustness for classification tasks. By constructing a smoothed classifier f^(x) f(x) through aggregation of predictions over randomized perturbations, it can produce the most probable prediction of the base classifier f(x)f(x) over perturbed inputs from Gaussian noise in a test instance. The smoothed classifier f^(x) f(x) is supposed to be provably robust to l2l_2-norm bounded perturbations within a certain radius R: Theorem A.1. [Cohen et al., 2019] For a classifier f:ℝ→f:R , suppose y∈y , let δ∼(0,σ2I)δ (0,σ^2I). Suppose the top class A is predicted with probability pAp_A, and the runner-up prediceted class B, with probability pBp_B. The smoothed classifier be f^():=argmaxyℙ(f(+δ)=y) f(x):= _yP(f(x+δ)=y), suppose pA¯,pB¯∈[0,1] p_A, p_B∈[0,1], if ℙ(f(+δ)=yA)≥pA¯≥pB¯≥maxy≠yAℙ(f(+δ)=y),P(f(x+δ)=y_A)≥ p_A≥ p_B≥ _y≠ y_AP(f(x+δ)=y), (6) then f^(+ϵ)=yA f(x+ε)=y_A for all ‖ϵ‖2≤R||ε||_2≤ R, where R=σ2(Φ−1(pA¯)−Φ−1(pB¯)).R= σ2( ^-1( p_A)- ^-1( p_B)). (7) Here Φ−1 ^-1 is the inverse cumulative distribution function (CDF) of the normal distribution. This approach has been applied primarily in pixel space to certify robustness against ℓp _p-bounded adversarial perturbations, and has been extended to different noise distributions and norm settings. While most work applies smoothing in pixel space, recent studies have explored extensions to learned representations or structured perturbations [Hao et al., 2022]. In this work, we use randomized smoothing not as a defense mechanism, but as a comparative evaluation tool to assess certified robustness of concept-based and standard classifiers under controlled geometric and semantic perturbations. Appendix B Methodology B.1 Empirical Robustness Characterization As illustrated in Figure 8, we introduce a generator-based evaluation framework that disentangles continuous latent-space perturbations from discrete concept-level interventions. In the latent space, geometric perturbations are applied by modifying the corresponding latent vector z, while in the concept space, semantic interventions are performed by flipping entries in the concept representation c. These operations yield controlled sets of generated images reflecting geometric and semantic perturbations, which we use to systematically analyze the robustness of classifiers with and without concept bottlenecks. Figure 8: Generator-based evaluation framework for geometric vs. semantic perturbations. (i) geometric perturbations by adding latent-space noise within an ℓ2 _2 budget ϵε, and (i) semantic perturbations by flipping up to τ binary concepts in concept space before image generation. B.2 Certified Robustness via Random Smoothing B.3 Latent-Space Gaussian Smoothing Given a latent representation z, the latent-smoothed classifier is f^G()=argmaxy∈ℙδ∼(0,σ2I)[f(g(+δ))=y]. f_G(z)= _y P_δ (0,σ^2I) [f(g(z+δ))=y ]. (8) Let A and B denote the top and runner-up classes with probabilities pA=ℙδ[f(g(+δ))=A],pB=ℙδ[f(g(+δ))=B].p_A=P_δ [f(g(z+δ))=A ], p_B=P_δ [f(g(z+δ))=B ]. (9) By the Gaussian randomized smoothing certificate of Cohen et al. [2019], if pA>pBp_A>p_B, then f^G(+ϵ)=A f_G(z+ε)=A for all ‖ϵ‖2≤σ2(Φ−1(pA)−Φ−1(pB)).\|ε\|_2≤ σ2 ( ^-1(p_A)- ^-1(p_B) ). (10) In practice, pAp_A and pBp_B are estimated by Monte Carlo sampling. Given N i.i.d. samples δ1,…,δN∼(0,σ2I) _1,…, _N (0,σ^2I), we estimate p^y()=1N∑i=1N[f(g(+δi))=y]. p_y(z)= 1N _i=1^NI [f(g(z+ _i))=y ]. (11) We compute one-sided Clopper–Pearson confidence bounds at level 1−α1-α: a lower bound p¯A p_A for the top class and an upper bound p¯B p_B for the runner-up class. The empirical certified radius is R^G=σ2(Φ−1(p¯A)−Φ−1(p¯B)). R_G= σ2 ( ^-1( p_A)- ^-1( p_B) ). (12) The certificate holds with probability at least 1−α1-α over the sampling procedure. B.4 Concept-Space Bernoulli Flip Smoothing Let ∈ℝKc ^K denote a concept representation and let ()∈0,1KC(c)∈\0,1\^K be its binarized concept vector. For paired concept coordinates [ci+,ci−][c_i^+,c_i^-], flipping the i-th concept swaps [ci+,ci−][c_i^+,c_i^-] into [ci−,ci+][c_i^-,c_i^+]. We define Bernoulli concept-flip noise by sampling ηi∼Bernoulli(ρ),i=1,…,K, _i (ρ), i=1,…,K, (13) independently across concepts. Let ℱ(,η)F(c,η) denote the concept representation obtained after flipping all coordinates with ηi=1 _i=1. The concept-smoothed class probabilities are py()=ℙη∼Bernoulli(ρ)K[f(g2(D(ℱ(,η))))=y],p_y(c)=P_η (ρ)^K [f\! (g_2(D(F(c,η))) )=y ], (14) and the concept-smoothed classifier is f^C()=argmaxy∈py(). f_C(c)= _y p_y(c). (15) We certify robustness to adversarial concept flips satisfying ‖(~)−()‖0≤Δ.\|C( c)-C(c)\|_0≤ . (16) Because randomized concept flipping modifies semantic values rather than masking them, sparse-ablation certificates do not directly apply. We instead use the Neyman–Pearson smoothing framework of Lee et al. [2019] on the Hamming cube. For a candidate adversarial budget Δ , let ℒΔL_ and ΔU_ denote the Neyman–Pearson transfer functions induced by the Bernoulli smoothing distribution. Intuitively, ℒΔ(p)L_ (p) gives the smallest possible probability of an event after any Δ -bit adversarial shift, assuming that the event has clean smoothing probability at least p. Similarly, Δ(p)U_ (p) gives the largest possible perturbed probability of an event whose clean smoothing probability is at most p. Let y⋆=argmaxypy()y = _yp_y(c) be the top class. Suppose conservative confidence bounds satisfy py⋆()≥p¯y⋆(),py()≤p¯y(),∀y≠y⋆.p_y (c)≥ p_y (c), p_y(c)≤ p_y(c), ∀ y≠ y . (17) If ℒΔ(p¯y⋆())>Δ(p¯y()),∀y≠y⋆,L_ ( p_y (c) )>U_ ( p_y(c) ), ∀ y≠ y , (18) then with probability at least 1−α1-α over Monte Carlo estimation, f^C(~)=y⋆,∀~:‖(~)−()‖0≤Δ. f_C( c)=y , ∀ c:\|C( c)-C(c)\|_0≤ . (19) Certified concept-flip radius. The certified adversarial concept-flip radius is Δ⋆()=maxΔ∈0,1,…,K: (c)= \ ∈\0,1,…,K\: ℒΔ(p¯y⋆())>Δ(p¯y()), _ ( p_y (c) )>U_ ( p_y(c) ), ∀y∈∖y⋆. ∀ y \y \ \. (20) B.5 Monte Carlo Estimation and Certification Algorithm For concept-space smoothing, we estimate the smoothed probabilities using N i.i.d. Bernoulli flip samples. Abstentions occur when the top-class lower confidence bound does not exceed the runner-up upper confidence bound. The procedure for estimating the smoothed class probabilities is presented in Algorithm 1. Algorithm 1 Concept-space ℓ0 _0 certification via Bernoulli concept flips 0: Concept representation c, base classifier f, generator mapping g2(D(⋅))g_2(D(·)), flip probability ρ, samples N, confidence level 1−α1-α. 1: for i=1i=1 to N do 2: Sample ηi∼Bernoulli(ρ)K _i (ρ)^K. 3: Compute i′=ℱ(,ηi)c_i =F(c, _i). 4: Compute y^i=f(g2(D(i′))) y_i=f(g_2(D(c_i ))). 5: end for 6: Compute class counts count[y]=|i:y^i=y|count[y]=|\i: y_i=y\| for all y∈y . 7: Compute Clopper–Pearson confidence bounds p¯y() p_y(c) and p¯y() p_y(c) from count[y]count[y]. 8: Let y⋆=argmaxyp¯y()y = _y p_y(c). 9: Return Δ⋆() (c) using Eq. (20). Appendix C Experiments C.1 Computation of the robustness-gap heatmap. Each entry in Figure 1(b) is computed from the corresponding representative experimental setting in the main results. For empirical geometric robustness we use ϵ=0.3ε=0.3; for empirical semantic robustness we use τ=5τ=5; for latent-space randomized smoothing we use σ=0.10σ=0.10; and for concept-space randomized smoothing we use ρ=0.10ρ=0.10. Since lower PFR indicates greater robustness, empirical gaps are computed as GapPFR=PFR(fstd)−PFR(fcbm).Gap_PFR=PFR(f_std)-PFR(f_cbm). Thus, a positive PFR gap means that the CBM has fewer prediction flips. For certified robustness, larger radii indicate greater robustness. Therefore, latent-space certified gaps are computed as GapR=R(fcbm)−R(fstd),Gap_R=R(f_cbm)-R(f_std), and concept-space certified gaps are computed as GapΔ⋆=Δ⋆(fcbm)−Δ⋆(fstd).Gap_ = (f_cbm)- (f_std). Thus, positive values consistently indicate that the CBM is more robust. PFR gaps are reported in percentage points, while certified-radius gaps are reported in absolute radius units. C.2 More Details on Experiments Setups Hareware. We profiled the GPU memory usage of the two main experimental components under the default setting of batch size 8. All experiments were run on a single NVIDIA RTX 4090 GPU with 24GB memory. For the latent-space perturbation experiment, each epsilon setting involves approximately 400×1000=400,000400× 1000=400,000 latent perturbations, with a peak allocated GPU memory of about 5.3GB and a PyTorch reserved memory of about 6.6GB. For the concept-space flipping experiment, the default setting involves approximately 4×1000×100=1,600,0004× 1000× 100=1,600,000 concept perturbations, with a peak allocated GPU memory of about 5.0GB and a PyTorch reserved memory of about 6.3GB. These measurements indicate that, under the reported experimental scale and batch-size setting, a single RTX 4090 24GB GPU is sufficient for the memory requirements of these experiments. To ensure stable execution, we recommend keeping at least 8GB of free GPU memory, and preferably 10–12GB. Each experiment typically uses a single GPU; multiple experiments are run in parallel via separate CUDA_VISIBLE_DEVICES configurations C.3 Empirical Robustness Results Perturbation CUB-200-2011 RIVAL-10 Type Budget CFRCFR PFR (%) ↓ DFS↓DFS CFRCFR PFR (%) ↓ DFS↓DFS fstdf_std fcbmf_cbm fstdf_std fcbmf_cbm Geo. ϵ=0.05ε=0.05 0.40± 0.01 27.1± 2.1 31.4± 1.5 1.28± 0.03 0.32 ± 0.14 0.00 0.00 0.48± 0.01 ϵ=0.1ε=0.1 0.79± 0.02 42.2± 2.5 46.6± 1.4 1.59± 0.01 0.67 ± 0.22 0.00 0.00 0.75± 0.01 ϵ=0.3ε=0.3 2.14± 0.04 74.6± 1.6 76.6± 0.8 3.26± 0.05 2.11 ± 0.37 0.00 0.00 1.28± 0.03 ϵ=0.5ε=0.5 3.15± 0.04 86.3± 1.0 87.8± 1.3 4.08± 0.04 4.09± 0.47 0.00 0.00 1.62± 0.02 ϵ=0.8ε=0.8 4.10± 0.04 92.6± 0.6 93.2± 0.3 4.84± 0.03 8.36 ± 0.55 0.02± 0.01 0.02± 0.02 1.93± 0.04 Sem. τ=1τ=1 1 42.7± 1.6 45.0± 0.8 1.87± 0.48 1 0.00 0.00 1.84± 0.02 τ=2τ=2 2 58.4± 1.3 60.9± 0.7 2.57± 0.50 2 0.07± 0.02 0.08± 0.03 2.28± 0.03 τ=5τ=5 5 77.7± 0.9 80.1± 0.6 3.61± 0.60 5 0.72± 0.07 0.73± 0.09 3.14± 0.05 τ=10τ=10 10 90.0± 0.6 90.4± 0.4 4.46± 0.81 10 6.22± 0.62 6.13± 0.70 5.20± 0.11 Table 1: Robustness comparison under geometric (latent ℓ2 _2 noise) and semantic (concept flip) perturbations. Table 1reports the complete empirical robustness results on both CUB-200-2011 and RIVAL-10. Here, CFR reflects the surrogate ground-truth rate of concept flips induced by the generator. We then measure prediction instability via the classifier’s prediction flip rate. For concept bottleneck models, we additionally evaluate robustness in the intermediate concept space. The results show that, although fcbmf_cbm is less robust than fstdf_stdat the prediction level, its concept representation is comparatively stable: even when up to 10 concepts are semantically flipped, changes in the concept space remain limited. Statistical Analysis. We test for differences in robustness between fstdf_std and fcbmf_cbm using a paired design: for each repeat k, both classifiers are evaluated on the same subset, yielding paired measurements. For each perturbation budget, we define Δk=PFRk(fstd)−PFRk(fcbm) _k=PFR_k(f_std)-PFR_k(f_cbm) and test the null hypothesis H0:[Δ]=0H_0:E[ ]=0 with a two-sided paired test across the K=5K=5 repeats. Since we test multiple budgets, we additionally report Holm–Bonferroni corrected p-values. The per-budget p-values and corrected values are reported in Table 2. Type Budget Δ¯±σΔ ± _ (p) p pHolmp_Holm Geo. ϵ=0.05ε=0.05 −4.25±2.43-4.25± 2.43 0.017 0.098 Geo. ϵ=0.10ε=0.10 −4.39±2.26-4.39± 2.26 0.012 0.086 Geo. ϵ=0.30ε=0.30 −2.00±1.21-2.00± 1.21 0.021 0.098 Geo. ϵ=0.50ε=0.50 −1.12±0.63-1.12± 0.63 0.016 0.098 Geo. ϵ=0.80ε=0.80 −0.51±0.31-0.51± 0.31 0.022 0.098 Sem. τ=1τ=1 −2.33±1.49-2.33± 1.49 0.025 0.098 Sem. τ=2τ=2 −2.83±1.34-2.83± 1.34 0.009 0.073 Sem. τ=5τ=5 −2.33±0.84-2.33± 0.84 0.004 0.032 Sem. τ=10τ=10 −0.84±0.49-0.84± 0.49 0.018 0.098 Table 2: Paired significance tests for PFR differences between fstdf_std and fcbmf_cbm across K=5K=5 subset repeats (each repeat uses n=200n=200 images). Δ is measured in percentage points (p). We report raw two-sided paired t-test p-values and Holm–Bonferroni corrected pHolmp_Holm across all 9 budgets in the table. Across perturbation budgets, stronger perturbations increase PFR for both classifiers and increase DFS, indicating that the perturbations induce larger changes in the CBM concept representation. The repeated-subset statistics show that the reported trends are stable across different random selections of evaluation images, and the paired tests quantify when the observed differences in PFR between fstdf_std and fcbmf_cbm are statistically significant. Additional Analysis of CFR, PFR, and DFS 22442020404060608080DFSPFR (%) 005510102244CFRDFS Figure 9: mean PFR vs mean DFS (left) and mean DFS vs mean CFR (right). Red: Geometric, Blue: Semantic. Solid: fstdf_std; Dashed: fcbmf_cbm. As shown in Figure 9, PFR generally increases with CFR, but the relationship is not strictly monotonic. For example, when the semantic budget is τ=5τ=5 (CFR = 5) and the geometric budget is ϵ=0.5ε=0.5 (CFR = 3.15), the corresponding PFR values are similar, indicating that higher CFR does not always imply higher prediction instability. DFS exhibits a more consistent relationship with PFR. Geometric perturbations tend to induce larger DFS values than semantic perturbations at comparable CFR levels, suggesting that latent-space perturbations can distort the model decision representation more strongly than direct concept flips. This may be due to the nonlinear CB-AE mapping between latent, concept, and image spaces. C.4 Randomised Smoothing Experiments Type Budget Model CUB-200-2011 RIVAL-10 (10-class) SmAcc (%) ↑ R/Δ↑R/ SmAcc (%) ↑ R/Δ↑R/ Latent RM σ=0.01σ=0.01 fstdf_std 96.20 ± 0.84 0.0228 ± 0.0005 100.00 ± 0.00 0.0246 ± 0.0000 fcbmf_cbm 97.40 ± 0.55 0.0215 ± 0.0004 100.00 ± 0.00 0.0246 ± 0.0000 σ=0.05σ=0.05 fstdf_std 88.40 ± 1.39 0.0747 ± 0.0031 100.00 ± 0.00 0.1232 ± 0.0000 fcbmf_cbm 84.90 ± 1.67 0.0672 ± 0.0026 100.00 ± 0.00 0.1232 ± 0.0000 σ=0.10σ=0.10 fstdf_std 81.70 ± 2.39 0.1034 ± 0.0030 100.00 ± 0.00 0.2463 ± 0.0000 fcbmf_cbm 77.20 ± 2.14 0.0918 ± 0.0038 100.00 ± 0.00 0.2463 ± 0.0000 σ=0.30σ=0.30 fstdf_std 62.40 ± 3.65 0.1263 ± 0.0058 100.00 ± 0.00 0.7376 ± 0.0004 fcbmf_cbm 57.10 ± 4.57 0.1183 ± 0.0045 100.00 ± 0.00 0.7375 ± 0.0003 Concept RM ρ=0.05ρ=0.05 fstdf_std 76.08 ± 0.28 5.890 ± 0.074 99.97 ± 0.00 13.502 ± 0.002 fcbmf_cbm 75.24 ± 0.47 5.671 ± 0.124 99.97 ± 0.00 13.502 ± 0.002 ρ=0.10ρ=0.10 fstdf_std 59.13 ± 1.46 0.909 ± 0.151 99.90 ± 0.01 6.560 ± 0.002 fcbmf_cbm 57.60 ± 0.59 0.751 ± 0.061 99.89 ± 0.01 6.558 ± 0.002 ρ=0.20ρ=0.20 fstdf_std 40.34 ± 0.50 0.000 ± 0.000 99.52 ± 0.07 3.064 ± 0.006 fcbmf_cbm 38.18 ± 1.00 0.000 ± 0.000 99.52 ± 0.08 3.063 ± 0.007 Table 3: Unified randomized smoothing results in latent and concept space. Latent RM uses Gaussian noise δ∼(0,σ2I)δ (0,σ^2I) and reports certified ℓ2 _2 radius R. Concept RM uses Bernoulli concept flips with rate ρ and reports certified ℓ0 _0 radius Δ . SmAcc denotes smoothed accuracy. Values are mean ± std over five random subsets. Bold indicates the better value between fstdf_std and fcbmf_cbm for CUB-200-2011 only; we omit highlighting on RIVAL-10 due to saturation. Table 3reports the complete set of randomized smoothing experimental results. Under latent-space smoothing, the ResNet baseline generally achieves higher SmAcc and larger certified radii than the PCBM on CUB-200-2011, especially as σ increases. This indicates that, on generator native inputs, the concept bottleneck does not improve certifiable robustness to continuous latent perturbations. On the original RIVAL-10 split, both models are saturated: SmAcc remains at 100% and certified radii are nearly identical across all σ, reflecting the strong class separability of this benchmark and indicating that samples remain far from decision boundaries in latent space. C.4.1 Statistical Analysis for Latent-space Randomized Smoothing We evaluate latent-space randomized smoothing on a pool of 1000 generator-native inputs. To quantify sampling variability from subset selection, we form K=5K=5 independent random subsets of size n=200n=200 drawn uniformly without replacement from the 400-sample pool (subsets may overlap). We use the same subsets for both methods and all noise scales. For each input, we estimate smoothed class probabilities using N=1000N=1000 i.i.d. Gaussian perturbations, and compute one-sided Clopper–Pearson confidence bounds at α=0.05α=0.05. Following multi-class confidence separation, the smoothed classifier abstains unless the top-class lower bound exceeds the runner-up upper bound (pA>pBp_A>p_B). Metrics. Smoothed accuracy (SmAcc) counts abstentions as incorrect. The average certified radius R is computed over inputs that are both non-abstained and correct (with respect to the reference/no-noise label). Significance testing. For each σ and each metric, we compare methods using a paired two-sided test across the K=5K=5 subset-level measurements (paired because both methods are evaluated on the same subset). We additionally report Holm–Bonferroni corrected p-values across all hypothesis tests. σ Δ (p) p ΔR R p 0.01 −1.90±1.56-1.90± 1.56 0.053 (Holm: 0.210) 0.0010±0.00040.0010± 0.0004 0.004 (Holm: 0.028) 0.05 0.80±2.590.80± 2.59 0.528 (Holm: 0.917) 0.0061±0.00120.0061± 0.0012 2.93×10−42.93× 10^-4 (Holm: 0.002) 0.10 3.60±3.453.60± 3.45 0.080 (Holm: 0.240) 0.0078±0.00300.0078± 0.0030 0.004 (Holm: 0.028) 0.30 1.30±3.551.30± 3.55 0.458 (Holm: 0.917) 0.0089±0.00640.0089± 0.0064 0.036 (Holm: 0.182) Table 4: Significance analysis over K=5K=5 random 200-sample subsets (mean ± std across subsets). Differences are reported as Δ=fstd−fcbm =f_std-f_cbm (p = percentage points). We report paired two-sided p-values across the five subset-level measurements; Holm denotes Holm–Bonferroni correction over all tests. C.4.2 Statistical Analysis for Concept-space Randomized Smoothing m Δ (p) p (Holm) Δτ τ p (Holm) 1 -1.50 ± 2.09 0.1841 (0.5334) 0.174 ± 0.051 0.0016 (0.0094) 2 1.80 ± 2.46 0.1778 (0.5334) 0.020 ± 0.069 0.5501 (0.5501) 4 -2.90 ± 2.61 0.0677 (0.3386) 0.019 ± 0.020 0.0950 (0.3798) Table 5: Significance analysis for concept-space (ℓ0 _0) randomized smoothing over K=5K=5 random 200-sample subsets. Differences are Δ=fstd−fcbm =fstd-fcbm (p = percentage points), reported as mean ± std across subsets. We report paired two-sided p-values across the five subset-level measurements; Holm denotes Holm–Bonferroni correction over all 66 tests (3 flip budgets × 2 metrics). We begin from a pool of 1000 generator-native inputs. To quantify variability due to subset selection, we generate K=5K=5 independent random subsets of size n=200n=200 by uniform sampling without replacement from the pool (subsets may overlap). For each subset, we evaluate both methods on the same 200 inputs (paired design), and repeat this for all concept flip budgets m∈1,2,4m∈\1,2,4\. Smoothed prediction and abstention. For each input we estimate class probabilities under fixed-count concept flips using N=1000N=1000 Monte Carlo samples. We compute one-sided Clopper–Pearson confidence bounds at α=0.05α=0.05 for the top class A and runner-up B. The smoothed classifier outputs y^=A y=A only if p¯A>p¯B p_A> p_B; otherwise it abstains. Metrics. Smoothed accuracy (SmAcc) counts abstentions as incorrect: a sample is correct iff the smoothed prediction matches the reference label and does not abstain. The certified flip radius τ is computed using the ℓ0 _0 smoothing certificate (Def. 5.7 / Cor. 5.6) and averaged over samples that are both correct and non-abstained. Significance testing. For each flip budget m and each metric (SmAcc and τ), we obtain one measurement per subset for each method, yielding five paired measurements. We compare methods using a paired two-sided test across the five subset-level measurements at α=0.05α=0.05. Since we test 3 flip budgets and 2 metrics (6 hypotheses), we additionally report Holm–Bonferroni corrected p-values across all 6 tests. C.5 Robustness under Task Conditions C.5.1 Effect of Class Structure and Semantic Similarity To construct additional classes for RIVAL-10, we use a large language model22 2 https://deploymentsafety.openai.com/gpt-5-4-thinking to identify two sets of ImageNet classes: ten classes that are semantically similar to the original RIVAL-10 categories and ten classes that are semantically dissimilar. The selected classes are shown in Figure 10. Class similarity is estimated using feature distances extracted from a ResNet-50 backbone: for each class, we compute a centroid feature via K-means clustering and measure pairwise similarity using Euclidean distances between these class centroids. Figure 10: Distance heatmap among 20 classes under similar and dissimilar settings. Class distances are computed as Euclidean distances between class-level centroid features extracted using a ResNet-50 backbone. Table 6and Table 7 present the complete set of empirical and certified robustness experiments. Table 6: Robustness on RIVAL-10 (18 concepts, varying class count). Backbone: ResNet50. Geo.: latent ℓ2 _2 perturbation (values mean ± SE). Sem.: concept number flip, τ concepts flipped (global average). CFR (%): concept flip rate / τ; PFR (%): prediction flip rate; DFS: mean PCBM concept distance. Perturbation 10+ 5 dissimilar classes 10+ 5 similar classes 10+ 10 dissimilar classes 10+ 10 similar classes Type Budget CFR PFR (%) ↓ DFS ↓ CFR PFR (%) ↓ DFS ↓ CFR PFR (%) ↓ DFS ↓ CFR PFR (%) ↓ DFS ↓ fstdf_std fcbmf_cbm fstdf_std fcbmf_cbm fstdf_std fcbmf_cbm fstdf_std fcbmf_cbm Geo. ε=0.05 =0.05 0.81 0.29±0.160.29± 0.16 0.06±0.040.06± 0.04 0.37±0.010.37± 0.01 0.56 3.72±0.643.72± 0.64 2.06±0.472.06± 0.47 0.60±0.010.60± 0.01 0.44 0.06±0.040.06± 0.04 0.08±0.050.08± 0.05 0.48±0.010.48± 0.01 0.14 1.86±0.351.86± 0.35 1.85±0.431.85± 0.43 0.45±0.010.45± 0.01 ε=0.1 =0.1 1.77 0.46±0.190.46± 0.19 0.15±0.080.15± 0.08 0.60±0.020.60± 0.02 1.11 5.96±0.865.96± 0.86 3.61±0.653.61± 0.65 0.92±0.020.92± 0.02 0.93 0.17±0.080.17± 0.08 0.25±0.110.25± 0.11 0.75±0.020.75± 0.02 0.18 3.81±0.573.81± 0.57 3.40±0.653.40± 0.65 0.72±0.020.72± 0.02 ε=0.3 =0.3 5.77 0.80±0.240.80± 0.24 0.33±0.110.33± 0.11 1.10±0.031.10± 0.03 3.85 10.06±1.2010.06± 1.20 6.94±0.986.94± 0.98 1.60±0.031.60± 0.03 3.38 0.47±0.140.47± 0.14 0.61±0.170.61± 0.17 1.32±0.031.32± 0.03 1.02 8.01±0.988.01± 0.98 6.10±0.936.10± 0.93 1.29±0.031.29± 0.03 ε=0.5 =0.5 9.83 1.03±0.291.03± 0.29 0.53±0.160.53± 0.16 1.39±0.041.39± 0.04 7.17 11.71±1.3311.71± 1.33 8.34±1.118.34± 1.11 1.95±0.041.95± 0.04 6.89 0.59±0.160.59± 0.16 0.81±0.180.81± 0.18 1.67±0.031.67± 0.03 2.42 9.91±1.129.91± 1.12 7.37±1.037.37± 1.03 1.62±0.031.62± 0.03 ε=0.8 =0.8 15.73 1.33±0.361.33± 0.36 0.83±0.230.83± 0.23 1.69±0.051.69± 0.05 12.37 12.80±1.4112.80± 1.41 9.55±1.229.55± 1.22 2.31±0.042.31± 0.04 13.04 0.78±0.160.78± 0.16 1.23±0.211.23± 0.21 2.05±0.042.05± 0.04 5.86 11.59±1.1911.59± 1.19 8.63±1.108.63± 1.10 1.99±0.041.99± 0.04 Sem. τ=1τ=1 1 0.99 ± 0.22 0.46 ± 0.14 1.40 ± 0.01 1 11.07 ± 0.89 7.87 ± 1.09 1.92 ± 0.03 1 0.29 ± 0.11 0.56 ± 0.11 1.69 ± 0.03 1 9.64 ± 1.72 6.93 ± 1.67 1.62 ± 0.05 τ=2τ=2 2 1.28 ± 0.29 0.77 ± 0.18 1.89 ± 0.05 2 12.70 ± 1.31 9.34 ± 1.10 2.33 ± 0.04 2 0.78 ± 0.20 1.36 ± 0.18 2.19 ± 0.03 2 9.72 ± 0.69 7.03 ± 0.31 1.89 ± 0.02 τ=5τ=5 5 1.09 ± 0.22 0.89 ± 0.15 2.95 ± 0.03 5 13.76 ± 1.17 10.48 ± 1.17 3.11 ± 0.04 5 6.77 ± 0.31 7.66 ± 0.39 3.48 ± 0.05 5 13.73 ± 1.13 9.61 ± 1.29 2.50 ± 0.04 τ=7τ=7 7 1.93 ± 0.15 2.34 ± 0.12 3.76 ± 0.07 7 16.01 ± 0.73 13.05 ± 0.63 3.67 ± 0.03 7 15.08 ± 0.32 17.03 ± 0.49 4.56 ± 0.07 7 17.65 ± 1.40 13.14 ± 1.31 3.02 ± 0.07 τ=10τ=10 10 6.32 ± 0.63 8.27 ± 0.82 5.42 ± 0.16 10 20.91 ± 1.04 18.07 ± 0.99 4.78 ± 0.05 10 34.66 ± 1.34 37.86 ± 1.26 6.59 ± 0.07 10 21.97 ± 2.56 17.85 ± 2.26 3.76 ± 0.12 Type Budget Model 10 + 5 dissimilar classes 0 + 5 similar classes 10 + 10 dissimilar classes 10 + 10 similar classes SmAcc (%) ↑ R/Δτ↑R/ _τ SmAcc (%) ↑ R/Δτ↑R/ _τ SmAcc (%) ↑ R/Δτ↑R/ _τ SmAcc (%) ↑ R/Δτ↑R/ _τ σ=0.01σ=0.01 fstdf_std 100.00 ± 0.00 0.0699 ± 0.0001 99.50 ± 0.00 0.0659 ± 0.0007 100.00 ± 0.00 0.0699 ± 0.0002 99.70 ± 0.24 0.0660 ± 0.0006 fcbmf_cbm 100.00 ± 0.00 0.0703 ± 0.0000 100.00 ± 0.00 0.0681 ± 0.0006 100.00 ± 0.00 0.0700 ± 0.0002 99.70 ± 0.24 0.0677 ± 0.0009 σ=0.05σ=0.05 fstdf_std 99.70 ± 0.24 0.3446 ± 0.0019 95.70 ± 1.50 0.3112 ± 0.0065 100.00 ± 0.00 0.3401 ± 0.0017 98.60 ± 0.49 0.2973 ± 0.0087 Latent RM fcbmf_cbm 100.00 ± 0.00 0.3475 ± 0.0006 98.30 ± 0.93 0.3179 ± 0.0037 100.00 ± 0.00 0.3401 ± 0.0012 97.50 ± 0.84 0.3218 ± 0.0035 σ=0.10σ=0.10 fstdf_std 100.00 ± 0.00 0.6808 ± 0.0056 92.20 ± 1.12 0.6368 ± 0.0140 100.00 ± 0.00 0.6686 ± 0.0046 95.30 ± 0.68 0.5909 ± 0.0146 fcbmf_cbm 100.00 ± 0.00 0.6878 ± 0.0021 97.10 ± 1.07 0.6334 ± 0.0089 100.00 ± 0.00 0.6539 ± 0.0050 96.30 ± 1.47 0.6286 ± 0.0095 σ=0.30σ=0.30 fstdf_std 99.70 ± 0.24 1.9922 ± 0.0097 87.10 ± 1.20 1.8504 ± 0.0336 100.00 ± 0.00 1.7956 ± 0.0165 90.00 ± 1.87 1.6167 ± 0.0354 fcbmf_cbm 100.00 ± 0.00 2.0335 ± 0.0135 92.80 ± 0.24 1.7937 ± 0.0346 100.00 ± 0.00 1.6745 ± 0.0200 94.10 ± 1.96 1.6606 ± 0.0278 Concept RM ρ=0.05ρ=0.05 fstdf_std 99.34 ± 0.26 13.259 ± 0.098 92.82 ± 0.72 10.898 ± 0.245 99.50 ± 0.77 13.322 ± 0.296 94.42 ± 0.37 0.92 ± 0.01 fcbmf_cbm 99.60 ± 0.14 13.357 ± 0.056 94.79 ± 0.61 11.583 ± 0.217 99.50 ± 0.32 13.320 ± 0.122 95.89 ± 0.69 0.94 ± 0.01 ρ=0.10ρ=0.10 fstdf_std 99.18 ± 0.15 6.424 ± 0.028 90.09 ± 2.00 4.868 ± 0.316 98.60 ± 0.86 6.318 ± 0.158 90.88 ± 0.52 1.44 ± 0.03 fcbmf_cbm 99.50 ± 0.11 6.485 ± 0.020 92.58 ± 1.59 5.268 ± 0.262 98.70 ± 0.68 6.336 ± 0.125 93.43 ± 0.53 1.47 ± 0.03 ρ=0.20ρ=0.20 fstdf_std 98.81 ± 0.20 3.001 ± 0.017 88.97 ± 1.17 2.213 ± 0.085 96.30 ± 0.75 2.787 ± 0.063 88.08 ± 0.93 1.96 ± 0.03 fcbmf_cbm 99.08 ± 0.15 3.025 ± 0.013 91.42 ± 1.02 2.397 ± 0.077 94.90 ± 0.97 2.672 ± 0.079 91.72 ± 0.72 1.99 ± 0.03 Table 7: Randomized smoothing ablation on RIVAL-10 variants after removing the saturated 10-class setting. Latent RM uses Gaussian latent noise η∼(0,σ2I)η (0,σ^2I) and reports certified ℓ2 _2 radius R. Concept RM uses Bernoulli concept flips with probability ρ and reports certified ℓ0 _0 concept-flip radius Δτ _τ. SmAcc denotes smoothed accuracy. All variants use 18 concepts. C.5.2 Effect of Concept Vocabulary Size Table 8and Table 9 presents the complete set of empirical and certified robustness experiments. Table 8: Empirical robustness under geometric (latent ℓ2 _2) and semantic (concept flip) perturbations on RIVAL-10 with varying concept vocabulary size. CFR: concept flip rate; PFR (%): prediction flip rate; DFS: PCBM concept distance. Perturbation 18 concepts 27 concepts 36 concepts Type Budget CFR PFR (%) ↓ DFS ↓ CFR PFR (%) ↓ DFS ↓ CFR PFR (%) ↓ DFS ↓ fstdf_std fcbmf_cbm fstdf_std fcbmf_cbm fstdf_std fcbmf_cbm Geo. ε=0.05 =0.05 0.32 0.00 0.00 0.48±0.010.48± 0.01 0.15 0.00 0.00 0.71±0.020.71± 0.02 1.0 0.00 0.00 0.77±0.030.77± 0.03 ε=0.10 =0.10 0.67 0.00 0.00 0.74±0.010.74± 0.01 0.56 0.00 0.01±0.010.01± 0.01 1.10±0.021.10± 0.02 1.67 0.00 0.00 1.21±0.041.21± 0.04 ε=0.30 =0.30 2.11 0.00 0.00 1.29±0.021.29± 0.02 2.58 0.00 0.01±0.010.01± 0.01 1.91±0.041.91± 0.04 4.09 0.00 0.00 2.20±0.062.20± 0.06 ε=0.50 =0.50 4.09 0.00 0.00 1.61±0.031.61± 0.03 5.16 0.01±0.000.01± 0.00 0.01±0.010.01± 0.01 2.36±0.042.36± 0.04 6.94 0.01±0.000.01± 0.00 0.00 2.86±0.082.86± 0.08 ε=0.80 =0.80 8.36 0.02±0.010.02± 0.01 0.03±0.020.03± 0.02 1.95±0.031.95± 0.03 9.92 0.04±0.010.04± 0.01 0.04±0.010.04± 0.01 2.85±0.052.85± 0.05 11.97 0.04±0.010.04± 0.01 0.06±0.010.06± 0.01 3.67±0.103.67± 0.10 Sem. τ=1τ=1 1 0.00 ± 0.00 0.00 ± 0.00 1.82 ± 0.02 1 0.03 ± 0.02 0.04 ± 0.02 2.38 ± 0.06 1 0.01 ± 0.01 0.00 ± 0.00 2.25 ± 0.05 τ=2τ=2 2 0.07 ± 0.02 0.09 ± 0.02 2.29 ± 0.05 2 0.16 ± 0.05 0.18 ± 0.04 3.03 ± 0.06 2 0.02 ± 0.01 0.02 ± 0.01 3.05 ± 0.07 τ=5τ=5 5 0.75 ± 0.07 0.77 ± 0.07 3.17 ± 0.05 5 1.42 ± 0.20 1.44 ± 0.20 4.73 ± 0.07 5 0.29 ± 0.05 0.36 ± 0.05 4.67 ± 0.17 τ=10τ=10 10 5.79 ± 0.52 5.74 ± 0.52 5.13 ± 0.10 10 8.74 ± 0.99 8.69 ± 0.88 9.10 ± 0.24 10 2.62 ± 0.18 3.06 ± 0.19 7.99 ± 0.14 Table 9: RIVAL-10 10-class results with 27 vs 36 concepts (ResNet50), recomputed with image-level 5×200 subset aggregation (mean± ). Better value is bolded only when strictly better (ties are not bolded). Type Budget Model 27 concepts 36 concepts SmAcc (%)↑ R↑R SmAcc (%)↑ R↑R Latent σ=0.01σ=0.01 fstdf_std 100.00 ± 0.00 0.0703 ± 0.0000 100.00 ± 0.00 0.0703 ± 0.0000 fcbmf_cbm 100.00 ± 0.00 0.0703 ± 0.0000 100.00 ± 0.00 0.0703 ± 0.0000 σ=0.05σ=0.05 fstdf_std 100.00 ± 0.00 0.3517 ± 0.0000 100.00 ± 0.00 0.2876 ± 0.0786 fcbmf_cbm 100.00 ± 0.00 0.2607 ± 0.0747 100.00 ± 0.00 0.1996 ± 0.0107 σ=0.10σ=0.10 fstdf_std 100.00 ± 0.00 0.4756 ± 0.1141 100.00 ± 0.00 0.4052 ± 0.0261 fcbmf_cbm 99.99 ± 0.01 0.4509 ± 0.1287 100.00 ± 0.00 0.3936 ± 0.0092 σ=0.30σ=0.30 fstdf_std 99.99 ± 0.00 1.0960 ± 0.0128 99.99 ± 0.00 1.1438 ± 0.0316 fcbmf_cbm 99.98 ± 0.01 1.0571 ± 0.0216 99.99 ± 0.00 1.1218 ± 0.0291 Concept ρ=0.05ρ=0.05 fstdf_std 99.88 ± 0.02 13.466 ± 0.009 99.96 ± 0.01 13.499 ± 0.002 fcbmf_cbm 99.87 ± 0.02 13.464 ± 0.008 99.96 ± 0.00 13.496 ± 0.002 ρ=0.10ρ=0.10 fstdf_std 99.47 ± 0.06 6.479 ± 0.010 99.80 ± 0.03 6.542 ± 0.005 fcbmf_cbm 99.47 ± 0.05 6.479 ± 0.010 99.76 ± 0.04 6.534 ± 0.008 ρ=0.20ρ=0.20 fstdf_std 97.58 ± 0.09 2.895 ± 0.008 98.79 ± 0.12 2.999 ± 0.011 fcbmf_cbm 97.61 ± 0.07 2.897 ± 0.006 98.59 ± 0.14 2.982 ± 0.012 C.5.3 Experiments on DenseNet-161. We test similar experiments on DenseNet-161 [Huang et al., 2017] and the results are shown in Table 10 and Table 11. 10 class;K=18 10 + 10 similar classes; K=18 10 + 10 dissimilar classes; K=18 10 + 10 dissimilar classes; K=27 Type Budget PFRfdn_f_dn PFRfcbm_f_cbm DFS PFRfdn_f_dn PFRfcbm_f_cbm DFS PFRfdn_f_dn PFRfcbm_f_cbm DFS PFRfdn_f_dn PFRfcbm_f_cbm DFS Geo. (ℓ2 _2 latent) ε=0.05 =0.05 0.00±0.000.00± 0.00 0.000.00 0.391±0.0070.391± 0.007 1.72±0.411.72± 0.41 4.21±0.614.21± 0.61 0.537±0.0040.537± 0.004 0.03±0.040.03± 0.04 0.07±0.050.07± 0.05 0.491±0.0120.491± 0.012 0.22±0.090.22± 0.09 0.30±0.180.30± 0.18 0.710±0.0130.710± 0.013 ε=0.1 =0.1 0.000.00 0.00±0.000.00± 0.00 0.663±0.0120.663± 0.012 3.39±0.523.39± 0.52 6.02±0.756.02± 0.75 0.895±0.0230.895± 0.023 0.10±0.050.10± 0.05 0.25±0.130.25± 0.13 0.830±0.0240.830± 0.024 0.49±0.170.49± 0.17 0.36±0.220.36± 0.22 1.192±0.0351.192± 0.035 ε=0.3 =0.3 0.09±0.030.09± 0.03 0.00±0.000.00± 0.00 1.343±0.0101.343± 0.010 6.68±1.046.68± 1.04 9.95±1.819.95± 1.81 1.775±0.0441.775± 0.044 0.12±0.070.12± 0.07 0.29±0.150.29± 0.15 1.547±0.0371.547± 0.037 0.66±0.170.66± 0.17 0.30±0.140.30± 0.14 2.230±0.0612.230± 0.061 ε=0.5 =0.5 0.10±0.060.10± 0.06 0.01±0.000.01± 0.00 1.761±0.0211.761± 0.021 9.07±1.259.07± 1.25 12.24±1.7012.24± 1.70 2.341±0.0642.341± 0.064 0.18±0.080.18± 0.08 0.43±0.150.43± 0.15 2.054±0.0492.054± 0.049 1.04±0.151.04± 0.15 0.43±0.080.43± 0.08 2.879±0.0852.879± 0.085 ε=0.8 =0.8 0.16±0.050.16± 0.05 0.01±0.010.01± 0.01 2.219±0.0252.219± 0.025 8.51±0.808.51± 0.80 11.51±1.0411.51± 1.04 2.913±0.0592.913± 0.059 0.29±0.070.29± 0.07 0.51±0.070.51± 0.07 2.588±0.0352.588± 0.035 1.24±0.211.24± 0.21 0.45±0.140.45± 0.14 3.594±0.0863.594± 0.086 Sem. (Concept flip) τ=1τ=1 0.02±0.010.02± 0.01 0.02±0.020.02± 0.02 2.020±0.0192.020± 0.019 5.50±0.875.50± 0.87 9.12±1.429.12± 1.42 2.253±0.0232.253± 0.023 0.15±0.040.15± 0.04 0.28±0.080.28± 0.08 2.141±0.0212.141± 0.021 0.94±0.110.94± 0.11 0.30±0.090.30± 0.09 3.054±0.0433.054± 0.043 τ=2τ=2 0.03±0.010.03± 0.01 0.03±0.010.03± 0.01 2.583±0.0172.583± 0.017 7.36±1.327.36± 1.32 11.07±1.2711.07± 1.27 2.765±0.0302.765± 0.030 0.46±0.060.46± 0.06 0.86±0.090.86± 0.09 2.835±0.0252.835± 0.025 1.59±0.341.59± 0.34 0.43±0.130.43± 0.13 3.965±0.0573.965± 0.057 τ=5τ=5 0.60±0.050.60± 0.05 0.66±0.040.66± 0.04 3.622±0.0413.622± 0.041 9.53±0.939.53± 0.93 14.04±1.5414.04± 1.54 3.724±0.0593.724± 0.059 5.31±0.335.31± 0.33 7.08±0.347.08± 0.34 4.890±0.0774.890± 0.077 4.13±0.274.13± 0.27 1.96±0.111.96± 0.11 5.781±0.0905.781± 0.090 τ=7τ=7 2.20±0.212.20± 0.21 2.17±0.242.17± 0.24 4.349±0.0234.349± 0.023 10.08±0.2710.08± 0.27 16.30±1.3116.30± 1.31 4.327±0.0474.327± 0.047 14.50±0.3014.50± 0.30 17.31±0.3017.31± 0.30 6.818±0.0606.818± 0.060 7.04±0.467.04± 0.46 4.28±0.164.28± 0.16 7.144±0.0517.144± 0.051 τ=10τ=10 7.06±0.587.06± 0.58 6.83±0.786.83± 0.78 5.744±0.1565.744± 0.156 14.14±0.6814.14± 0.68 22.17±0.9222.17± 0.92 5.264±0.0765.264± 0.076 35.51±1.0935.51± 1.09 38.38±1.3338.38± 1.33 10.098±0.21110.098± 0.211 14.24±0.3414.24± 0.34 10.30±0.2810.30± 0.28 9.318±0.0879.318± 0.087 Table 10: DenseNet-161 direct robustness on RIVAL-10 variants. PFR (%): prediction flip rate for fdnf_dn (DenseNet-161) and fcbmf_cbm; DFS: average PCBM concept distance. Mean ± std over 5 random subsets of 200 images. 10 class; K=18 10 + 10 similar classes; K=18 10 + 10 dissimilar classes; K=18 10 + 10 dissimilar classes; K=27 σ Model SmAcc (%) ↑ R↑R\, SmAcc (%) ↑ R↑R\, SmAcc (%) ↑ R↑R\, SmAcc (%) ↑ R↑R\, 0.01 fdnf_dn 100.00100.00 0.02350.0235 99.60±0.2299.60± 0.22 0.02310.0231 100.00100.00 0.02350.0235 100.00100.00 0.02340.0234 fcbmf_cbm 100.00100.00 0.02350.0235 99.60±0.4299.60± 0.42 0.02270.0227 100.00100.00 0.02340.0234 100.00100.00 0.02350.0235 0.05 fdnf_dn 100.00100.00 0.11710.1171 98.10±0.4298.10± 0.42 0.1118±0.00080.1118± 0.0008 100.00100.00 0.11700.1170 99.70±0.2799.70± 0.27 0.1149±0.00050.1149± 0.0005 fcbmf_cbm 100.00100.00 0.11730.1173 95.60±0.8295.60± 0.82 0.1112±0.00090.1112± 0.0009 100.00100.00 0.11640.1164 100.00100.00 0.11700.1170 0.10 fdnf_dn 100.00100.00 0.2335±0.00060.2335± 0.0006 96.80±0.7696.80± 0.76 0.2196±0.00340.2196± 0.0034 100.00100.00 0.2330±0.00080.2330± 0.0008 99.60±0.2299.60± 0.22 0.2271±0.00150.2271± 0.0015 fcbmf_cbm 100.00100.00 0.23460.2346 91.30±2.1791.30± 2.17 0.2216±0.00170.2216± 0.0017 100.00100.00 0.2310±0.00140.2310± 0.0014 100.00100.00 0.2332±0.00070.2332± 0.0007 0.30 fdnf_dn 100.00100.00 0.6938±0.00230.6938± 0.0023 90.10±1.1490.10± 1.14 0.6719±0.00510.6719± 0.0051 100.00100.00 0.6910±0.00240.6910± 0.0024 99.70±0.2799.70± 0.27 0.6525±0.00320.6525± 0.0032 fcbmf_cbm 100.00100.00 0.70320.7032 88.90±1.2488.90± 1.24 0.6376±0.00710.6376± 0.0071 100.00100.00 0.6778±0.00300.6778± 0.0030 100.00100.00 0.6866±0.00170.6866± 0.0017 Table 11: DenseNet-161 latent Gaussian randomized smoothing. z~=z+η z=z+η, η∼(0,σ2I)η (0,σ^2I). SmAcc: smoothed accuracy; R: average certified radius over correctly classified and certified samples. Mean ± std over five subsets of 200 images. C.5.4 Experiments on Attacks. We evaluate adversarial robustness by attacking the entire pipeline using APGD-CE [Croce and Hein, 2020]; the corresponding results are reported in Table 12. Table 12: APGD-CE attack results on the CUB generated dataset based on backbone ResNet-50. Attack-ϵε Attack Space Model Clean Acc. Robust Acc. 0.030.03 latent z fstdf_std 100% 8% f^std f_std (smoothed) 100% 16.5% fcbmf_cbm 100% 23.5% f^cbm f_cbm (smoothed) 100% 33.5% concept c fstdf_std 100% 44% f^std f_std (smoothed) 88.5% 58.5% fcbmf_cbm 100% 41% f^cbm f_cbm (smoothed) 82.5% 55.5% 0.01 latent z fstdf_std 100% 29.5% f^std f_std (smoothed) 100% 33.5% fcbmf_cbm 100% 37.5% f^cbm f_cbm (smoothed) 100% 47% concept c fstdf_std 100% 72% f^std f_std (smoothed) 88.5% 80.5% fcbmf_cbm 100% 60.5% f^cbm f_cbm (smoothed) 82.5% 71% Appendix D Cross-Family Applicability Beyond CBMs Our primary analysis focuses on Concept Bottleneck Models (CBMs), since their explicit concept interface provides a particularly clean setting for separating geometric and semantic robustness. To examine whether the proposed evaluation procedure is restricted to CBMs, we additionally evaluate a prototype-based interpretable architecture, PixPNet, together with ResNet and PCBM under the same generator-defined perturbation protocol. Importantly, we do not assume that different model families share a common internal representation. In particular, the concepts exposed by the generator are not assumed to align one-to-one with the localized prototypes learned by PixPNet. Instead, what is shared across models is the external perturbation-to-prediction interface: the generator defines a controlled perturbation space, and each downstream classifier is evaluated on the resulting perturbed inputs using its own internal representation. This distinction allows us to compare standard, concept-based, and prototype-based classifiers under matched perturbations without requiring their heterogeneous internal representations to be aligned. D.1 Models and Evaluation Protocol We consider three representative model families: • Standard classifier: ResNet; • Concept-based interpretable classifier: PCBM, using CAV-style concept representations; • Prototype-based interpretable classifier: PixPNet, using localized part prototypes. For each classifier fmf_m, we compose it with the same fixed generator g and define Fm(z)=fm(g(z)).F_m(z)=f_m(g(z)). Therefore, the generator and perturbation process are shared across model families, while only the downstream classifier changes. We evaluate two perturbation geometries: 1. Gaussian latent perturbations, which induce continuous geometric variation through z′=z+δ,δ∼(0,σ2I);z =z+δ, δ (0,σ^2I); 2. Bernoulli concept interventions, which independently flip binary generator concepts and therefore induce discrete semantic variation. For empirical evaluation, we measure prediction flip rate (PFR). For certified evaluation, we use a perturbation-specific randomized-smoothing certificate: an ℓ2 _2 latent-space certificate for Gaussian perturbations and a Hamming-space certificate for discrete concept perturbations. All cross-family experiments use the same CUB-200-2011 generator-defined perturbation spaces and the same evaluation and certification procedures as the corresponding experiments in the main paper. The purpose of this section is therefore to vary the downstream classifier family while keeping the perturbation protocol fixed. D.2 Empirical Robustness of a Prototype-Based Classifier We first evaluate whether the same generator-defined perturbations can be used to characterize the prediction stability of a prototype-based classifier. Table 13 reports PFR for ResNet and PixPNet under Gaussian latent perturbations and Bernoulli semantic interventions. Table 13: Prediction flip rate (PFR) for ResNet and PixPNet under the same generator-defined perturbations. The semantic perturbations are defined in the generator concept space; they are not prototype-space interventions. Perturbation Strength ResNet PFR (%) PixPNet PFR (%) Gaussian latent σ=0.05σ=0.05 23.99 28.71 Gaussian latent σ=0.10σ=0.10 39.21 44.03 Bernoulli semantic ρ=0.10ρ=0.10 33.97 36.58 Bernoulli semantic ρ=0.20ρ=0.20 54.31 57.40 The purpose of this experiment is not to establish a robustness ranking between ResNet and PixPNet. Rather, it demonstrates that the same generator-defined perturbation protocol produces measurable and systematic robustness trends for a prototype-based interpretable classifier. In particular, increasing either the latent perturbation strength or the semantic intervention probability increases prediction instability for both architectures. This suggests that the proposed perturbation-to-prediction evaluation interface can be applied to a non-CBM interpretable architecture without modifying its internal prototype representation. D.3 Cross-Family Gaussian Randomized Smoothing We next evaluate whether the randomized-smoothing component of the framework transfers across classifier architectures. For each model fmf_m, we smooth the composite classifier Fm(z)=fm(g(z))F_m(z)=f_m(g(z)) under Gaussian latent perturbations. Because the perturbation distribution is defined before the downstream classifier, the same smoothing and certification procedure can be applied to ResNet, PCBM, and PixPNet without requiring their internal representations to be aligned. We report two quantities: • Smoothed accuracy (SmAcc): classification accuracy of the smoothed classifier, with abstentions counted as incorrect; • Average certified radius (ACR): the average certified latent-space radius over correctly classified, non-abstaining samples. Table 14: Gaussian randomized smoothing across standard, concept-based, and prototype-based classifiers. SmAcc denotes smoothed accuracy (%), with abstentions counted as incorrect. ACR denotes the average certified latent-space radius over correctly classified, non-abstaining samples. ResNet PCBM PixPNet σ SmAcc ACR SmAcc ACR SmAcc ACR 0.01 96.20 0.0228 97.40 0.0215 95.00 0.0199 0.05 88.40 0.0747 84.90 0.0672 88.90 0.0553 0.10 81.70 0.1034 77.20 0.0918 82.90 0.0682 0.30 62.40 0.1263 57.10 0.1183 56.20 0.0839 All three model families achieve non-trivial smoothed accuracy and certified latent-space radii across the tested noise levels. The purpose of Table 14 is therefore not to identify a uniformly most robust architecture. Instead, it demonstrates that the same randomized-smoothing procedure and certificate can be applied to standard, concept-based, and prototype-based classifiers through the common composite interface Fm(z)=fm(g(z))F_m(z)=f_m(g(z)). The results also reinforce the main conclusion of the paper: robustness differences are regime-dependent rather than determined solely by whether a model exposes an interpretable intermediate representation. D.4 Discrete Semantic Perturbations Across Model Families We further examine whether the same evaluation pipeline extends from continuous latent perturbations to discrete semantic interventions. We use the same CUB-200-2011 generator, concept representation, and Bernoulli concept-space perturbation protocol as in the main experiments. Specifically, each binary concept in the generator concept representation is independently flipped with probability ρ. The perturbed concept representation is then decoded into an image, which is subsequently evaluated by the downstream classifier. Thus, the perturbation process is identical across classifier families; only the downstream classifier is changed. Importantly, for PixPNet this should not be interpreted as directly perturbing or removing PixPNet prototypes. The perturbation is defined entirely in the generator concept space. PixPNet is evaluated as a prototype-based classifier under the resulting generator-defined semantic changes, without assuming any one-to-one correspondence between generator concepts and PixPNet prototypes. Table 15 reports the smoothed accuracy under the same Bernoulli smoothing procedure used in the main experiments. Table 15: Smoothed accuracy under Bernoulli generator-concept interventions on CUB-200-2011. The same generator-defined perturbation protocol is applied to ResNet and PixPNet. ρ ResNet SmAcc (%) PixPNet SmAcc (%) 0.05 100.00 99.75 0.10 75.25 71.25 0.20 28.75 23.75 0.30 12.50 9.00 As ρ increases, smoothed accuracy decreases for both classifiers, reflecting the increasing severity of the generator-defined semantic perturbations. The purpose of this experiment is not to compare concept representations across architectures, but to test whether the same semantic perturbation and evaluation procedure can be applied to a prototype-based classifier without modifying its internal prototype representation. D.5 Certified Robustness under Discrete Concept Interventions For discrete semantic perturbations, we use the same Bernoulli randomized-smoothing certification procedure described in Sec. B.4. Since the perturbation space is discrete, robustness is certified in Hamming space rather than using the Gaussian ℓ2 _2 radius employed for latent perturbations. For each downstream classifier, the certificate is defined with respect to the same generator concept space. Therefore, the certification procedure does not require the downstream model itself to expose concepts or prototypes that are aligned with the generator concepts. We report two complementary quantities: • ACR-H: the average certified Hamming radius over correctly classified, non-abstaining samples; • CA@1: the percentage of all evaluated images that are correctly classified and certified against at least one additional generator-concept flip. Table 16: Certified robustness under Bernoulli generator-concept interventions on CUB-200-2011. ACR-H denotes the average certified Hamming radius over correctly classified, non-abstaining samples. CA@1 denotes the percentage of all evaluated images certified against at least one additional generator-concept flip. ACR-H CA@1 (%) ρ ResNet PixPNet ResNet PixPNet 0.05 0.0125 0.0150 1.25 1.50 0.10 0.0133 0.0211 1.00 1.50 0.20 0.0609 0.0526 1.50 1.25 0.30 0.2200 0.2222 1.75 1.50 The certified radii are modest in this setting, and we therefore do not interpret these results as evidence that either architecture is uniformly robust to semantic concept changes. Instead, the experiment demonstrates that the same generator-defined semantic perturbation can be paired with a perturbation-specific certificate for different downstream classifier architectures. Together with the Gaussian results, this shows that the framework supports both continuous and discrete perturbation geometries: continuous latent perturbations are paired with a Gaussian ℓ2 _2 certificate, whereas discrete generator-concept interventions are paired with a Hamming-space certificate. D.6 Discussion and Scope The cross-family experiments support two main conclusions. First, the proposed evaluation procedure is not tied to the internal concept representation of a CBM. The same generator-defined perturbations can be propagated through standard, concept-based, and prototype-based classifiers, enabling matched comparisons of prediction stability under a common perturbation process. Second, randomized smoothing can be attached to this common external interface using a certificate matched to the perturbation geometry: Gaussian latent perturbations admit continuous latent-space certificates, whereas Bernoulli concept interventions admit discrete Hamming-space certificates. Importantly, these results do not imply that CBMs, PCBMs, and prototype-based models share a common interpretable representation. Nor do we assume that generator concepts correspond directly to PixPNet prototypes. What is shared across architectures is the external perturbation→generated input→predictionperturbation input evaluation interface. CBMs remain our primary analytical setting because their explicit semantic bottleneck enables a more detailed analysis of concept-level sensitivity, concept vocabulary design, and task-dependent robustness. The PixPNet experiments provide complementary evidence that the same evaluation and certification pipeline can be applied to classifier families with different internal representations.