Paper deep dive
RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations
Woo Jae Kim, Kyle Min, Suhyeon Ha, Joonsung Jeon, Sung-eui Yoon
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/8/2026, 4:48:06 AM
Summary
The paper introduces RoME (Robust Mixture of Low-Rank Experts), a novel framework for Multi-Perturbation Adversarial Training (MAT) that mitigates robustness trade-offs across different adversarial threats. RoME employs low-rank additive experts on a shared backbone to separate threat-common and threat-specific features, utilizes dual-scale gating (local and global features) for threat discrimination, and applies threat-guided gating diversification to enforce distinct routing pathways. Extensive experiments on CIFAR-10 and ImageNet datasets demonstrate superior union robustness, natural accuracy, and generalization to unseen threats compared to state-of-the-art MAT methods.
Entities (10)
Relation Signals (8)
ROME → uses → Low-Rank Experts
confidence 98% · we propose Robust Mixture of Low-Rank Experts (RoME), where each expert is a low-rank additive update to the shared backbone
ROME → evaluatedon → CIFAR-10
confidence 97% · We conduct extensive experiments on CIFAR-10, ImageNet-100 and ImageNet-1K, demonstrating that RoME outperforms existing state-of-the-art MAT methods
ROME → addresses → Multi-Perturbation Adversarial Training
confidence 95% · Multi-perturbation adversarial training (MAT) aims to achieve robustness against multiple ℓp perturbations but suffers from robustness trade-offs between different threats. To address this, we employ a mixture of experts... we propose Robust Mixture of Low-Rank Experts (RoME)
ROME → uses → Dual-Scale Gating
confidence 95% · RoME introduces (i) dual-scale gating that exploits threat-discriminative signals from local and global level features
ROME → uses → Threat-Guided Gating Diversification
confidence 94% · and (ii) threat-guided gating diversification that enforces diverse expert utilization across threats.
Multi-Perturbation Adversarial Training → suffersfrom → Robustness Trade-offs
confidence 92% · Multi-perturbation adversarial training (MAT) aims to achieve robustness against multiple ℓp perturbations but suffers from robustness trade-offs between different threats.
Dual-Scale Gating → combines → Global Gating
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-perturbation adversarial training (MAT) aims to achieve robustness against multiple $\ell_p$ perturbations but suffers from robustness trade-offs between different threats. To address this, we employ a mixture of experts (MoE) to route different threats through distinct model pathways. However, naive application of MoE encounters two critical challenges: experts tend to overlook threat-specific features and redundantly capture features shared across threats, and gating networks suffer from threat-agnostic routing where they learn nearly identical routing patterns across threats, thus preventing the construction of threat-specific model pathways. To this end, we propose Robust Mixture of Low-Rank Experts (RoME), where each expert is a low-rank additive update to the shared backbone, allowing it to capture threat-common features while experts focus on threat-specific information. To address threat-agnostic routing, RoME introduces (i) dual-scale gating that exploits threat-discriminative signals from local and global level features, and (ii) threat-guided gating diversification that enforces diverse expert utilization across threats. Extensive experiments demonstrate that RoME outperforms existing state-of-the-art MAT in union robustness and natural accuracy and improves robustness against unseen threats. Codes are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.06109v1
- Canonical: https://arxiv.org/abs/2607.06109v1
Trouble viewing inline? Open PDF directly →
Full Text
72,193 characters extracted from source content.
Expand or collapse full text
RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations Woo Jae Kim 1 , Kyle Min 2 , Suhyeon Ha 1 , Joonsung Jeon 1 , and Sung-eui Yoon 1 1 KAIST, Daejeon, Korea 2 Oracle, Seattle WA, USA wkim97, suhyeon.ha, mikeraph@kaist.ac.kr, sungeui@kaist.edu kyle.min@oracle.com Abstract. Multi-perturbation adversarial training (MAT) aims to achieve robustness against multiple ℓ p perturbations but suffers from robustness trade-offs between different threats. To address this, we employ a mixture of experts (MoE) to route different threats through distinct model path- ways. However, naïve application of MoE encounters two critical chal- lenges: experts tend to overlook threat-specific features and redundantly capture features shared across threats, and gating networks suffer from threat-agnostic routing where they learn nearly identical routing pat- terns across threats, thus preventing the construction of threat-specific model pathways. To this end, we propose Robust Mixture of Low-Rank Experts (RoME), where each expert is a low-rank additive update to the shared backbone, allowing it to capture threat-common features while experts focus on threat-specific information. To address threat- agnostic routing, RoME introduces (i) dual-scale gating that exploits threat-discriminative signals from local and global level features, and (i) threat-guided gating diversification that enforces diverse expert uti- lization across threats. Extensive experiments demonstrate that RoME outperforms existing state-of-the-art MAT in union robustness and nat- ural accuracy and improves robustness against unseen threats. Codes are available at https://github.com/wkim97/RoME. Keywords: Multi-Perturbation Adversarial Training· Mixture of Ex- perts· Adversarial Robustness 1 Introduction Adversarial training [21,43] has become one of the most effective defenses against adversarial threats. However, traditional adversarial training typically focuses on a single type of threat (e.g., ℓ ∞ perturbations), remaining vulnerable to un- seen threats during inference [44, 59]. Multi-perturbation adversarial training (MAT) [9,28,42,44,59] addresses this by training on diverse ℓ p threats simulta- neously, improving robustness across multiple threats. Despite its effectiveness, MAT faces robustness trade-offs when learning on di- verse threats, suffering from suboptimal robustness on individual threat types [9, arXiv:2607.06109v1 [cs.CV] 7 Jul 2026 2W. Kim et al. Expert # 0 L1 L2 Linf Expert # 1 L1 L2 Linf (a) Conventional MoE Expert # 0 L1 L2 Linf Expert # 1 L1 L2 Linf (b) Ours (Low-rank experts) Fig. 1: t-SNE visualization of expert outputs. (a) In con- ventional MoE, disjoint FFN experts redundantly capture similar threat-common fea- tures, while (b) our low-rank experts effectively capture threat-specific features. Ours: Diverse threat-specific expert usages Conventional Approaches: Similar threat-agnostic expert usage ℓ ∞ threatℓ 2 threatℓ 1 threat Threat-aware routing Threat-agnostic routing ℓ ∞ threatℓ 2 threatℓ 1 threat Failed samples Fig. 2: Conceptual visualization of conventional mixture of experts and our approach. Conventional methods (left) face the threat-agnostic routing issue and route different threats through similar ex- pert combinations, resulting in a single shared model pathway for distinct threats. Our approach (right) effectively routes each threat through distinct expert combinations, constructing multiple threat-specific model pathways. 28,59]. This degradation arises because different adversarial threats induce dis- tinct distributional shifts, creating conflicting optimization objectives during training [9,28]. Existing approaches [9,28,59] force these distinct threats through a single, fixed architectural pathway, where feature representations learned to be robust against one threat may fail to transfer robustness to others. To address this, we employ mixture of experts (MoE) [11,18,56,67] to route different threats through distinct model pathways, each learning representations robust to a different type of threat. However, naïve application of MoE [18,47,56] encounters two critical challenges. First, conventional experts, typically imple- mented as multiple independent FFN layers, struggle to capture features unique to each threat. Adversarial examples crafted from an image naturally share the same underlying image content [26,78], and certain threats even share overlap- ping features [9] (e.g., ℓ 1 and ℓ ∞ robustness partially transfer to ℓ 2 ). Without a shared backbone to separate threat-common from threat-specific features [69], conventional experts redundantly capture the former as shown in Fig. 1a. The shared features among threats further raise a second challenge of threat- agnostic routing, where the gating network routes different threats to similar expert combinations as shown in Fig. 2, failing to construct distinct pathways for each threat. This occurs because the overlapping features across threat types provide insufficient discriminative signal, preventing the gating network from distinguishing one threat from another. This forces a single model pathway to learn robust representations against multiple threats, facing the same multi- threat trade-off in existing MAT methods. To address these challenges, our key insight is two-fold. First, inspired by a recent approach [69] in multi-task learning that handles different tasks using low- RoME: Robust Mixture of Low-Rank Experts3 rank experts on top of a shared backbone, we shift from conventional disjoint experts and implement each expert as a low-rank additive update to the shared backbone weights. This allows the backbone to capture threat-common features, while each expert focuses on learning threat-specific information. To address threat-agnostic routing, we make an observation that different threats exhibit discriminative cues at different levels of granularity (Fig. 5 in Sec. 4.3); ℓ 1 threats are more distinguishable at local patch-level features due to their sparse perturbations, while ℓ ∞ threats show clearer discrimination at global image-level features due to their uniform perturbations across the image. We thus propose threat-distinguishing dual-scale gating that combines both scales to provide richer discriminative signals to gating networks. However, even with richer input signals, the gating network may still converge to threat-agnostic expert assignments without explicit routing supervision. We thus further intro- duce threat-guided gating diversification, which supervises gating networks to enforce diverse routing patterns across threats. Building on these insights, we propose Robust Mixture of Low-Rank Experts (RoME) consisting of: (i) low-rank experts (Sec. 4.2) for learning threat-specific model pathways, (i) threat-distinguishing dual-scale gating (Sec. 4.3) for dis- criminative gating signals, and (i) threat-guided gating diversification (Sec. 4.4) for diverse expert utilization across threats. These components effectively route each threat to distinct expert combinations (Fig. 2), with each expert learning distinct threat-specific features (Fig. 1b), enabling threat-adaptive model path- ways that mitigate cross-threat trade-offs. We conduct extensive experiments on CIFAR-10, ImageNet-100 and ImageNet- 1K, demonstrating that RoME outperforms existing state-of-the-art MAT meth- ods in union robustness and natural accuracy (Sec. 5.2). Diverse representations captured with multiple model pathways also improve robustness against threats unseen during training (Sec. 5.2). Ablation studies (Sec. 5.3) validate the im- portance of each component of RoME, and analysis (Sec. 5.4) shows that our framework enables threat-specific representations for improved performance. In summary, our contributions are as follows: – We identify threat-agnostic routing problem when applying MoE to MAT, where the gating network learns similar expert routing for different threats. – To this end, we introduce RoME, a novel mixture of low-rank experts frame- work for adversarial training on multiple threat types via threat-distinguishing dual-scale gating and threat-guided gating diversification. – Extensive experiments demonstrate that our approach outperforms existing state-of-the-art MAT in multi-threat robustness across multiple benchmarks. 2 Related Works 2.1 Multi-Perturbation Adversarial Training (MAT) Adversarial training, which minimizes the worst-case loss over adversarial ex- amples [21, 43], has become the de facto adversarial defense. Subsequent work 4W. Kim et al. improved trade-off between robustness and natural accuracy [60, 72], enhanced training efficiency [55, 63], and improved scalability to large-scale datasets [57, 61, 66]. Despite these advances, most methods optimize for a single adversarial threat (e.g., ℓ ∞ -bounded), limiting robustness against different threats. To improve robustness across diverse adversarial threats, early work [59] pro- posed training a model on multiple ℓ p threats. Subsequent works improved ro- bustness or training efficiency via worst-case descent directions [44], stochastic adversarial sampling [42], alternation between extreme norms [9], parameter- space interpolation [10], logit-pairing with gradient projection [28], or logit-space regularization [12]. While effective, these methods learn robustness against dis- tinct threats within a single model representation space, where optimizing for one threat can interfere with others [9,28,59] and degrade their robustness. MORE [5] employs mixture-of-experts but relies on gating without explicit threat-aware guidance, which can face difficulty distinguishing between threats. In contrast, our RoME learns diverse model pathways each tailored to different threats for mitigating cross-threat robustness trade-off. 2.2 Mixture of Experts Mixture of Experts (MoE) [27,56] learns multiple experts and a gating network that routes each input to a subset of experts, enabling input-adaptive model specialization. In Transformers, this is typically implemented by replacing feed- forward networks (FFNs) with multiple independent FFN experts [18, 47, 56]. Different variants have enhanced its routing stability [11,18], scalability [34], and expert selection [51,77]. MoE has been applied across multi-task learning [3,40, 69], continual learning [35,70], and large-scale vision models [52]. More recently, MoE has been applied with parameter-efficient adapters [16, 19, 64, 68]. While parallel lines of works explored adversarial robustness of MoE architectures [48, 73] or used MoE for improving robustness against a single threat [45,49,50,74], our work addresses an orthogonal problem of resolving cross-threat robustness trade-offs when training on multiple distinct threat types. To this end, we tackle the redundant features and threat-agnostic routing issues of conventional MoE through low-rank experts, dual-scale gating, and gating diversification. 3 Preliminary We consider a classification model f θ :X →Y parameterized by θ, whereX and Y denote the input and label spaces, and (x,y)∼D is a sample from data dis- tribution D. Given adversarial threats A 1 ,A 2 ,A ∞ with perturbation budgets ε 1 ,ε 2 ,ε ∞ , multi-perturbation adversarial training (MAT) aims to solve: min θ E (x,y)∼D φ(L 1 ,L 2 ,L ∞ ), (1) where L p = max δ p ∈A p (x,ε p ) L(f θ (x + δ p ),y) is adversarial loss [43] under threat A p for p∈1, 2,∞ [9,28,59], and φ : R 3 → R aggregates losses across threats. RoME: Robust Mixture of Low-Rank Experts5 Expert 0Expert 1Expert 2Expert 3 0.00.5 Density 1 threat 0.00.5 Expert activation 2 threat 0.00.5 threat (a) Naïve approach 0.00.5 Density 1 threat 0.00.5 Expert activation 2 threat 0.00.5 threat (b) Load balancing [56] 0.00.5 Density 1 threat 0.00.5 Expert activation 2 threat 0.00.5 threat (c) Ours (RoME) Fig. 3: Gaussian KDE analysis on expert activations across different threat types on CIFAR-10. (a) Naïve MoE suffers from routing collapse (Expert 1 dominates) and threat-agnostic routing (similar routing across threats). (b) Load balancing [56] resolves routing collapse but exhibits threat-agnostic routing. (c) Our RoME addresses threat- agnostic routing issue and learns diverse, threat-specific expert combinations. MAX [59] implements aggregation as φ MAX (L 1 ,L 2 ,L ∞ ) = maxL 1 ,L 2 ,L ∞ to optimize on the worst-case threat at each iteration. RANDOM [42] implements aggregation as φ RANDOM (L 1 ,L 2 ,L ∞ ) = L p , where a threat p ∼ Cat( 1 3 , 1 3 , 1 3 ) is sampled stochastically. 4 RoME 4.1 Limitations of Conventional MoE We employ mixture of experts (MoE) to construct multiple threat-specific model pathways for improved multi-threat robustness. However, unlike prior MoE ap- plications with clearly distinguishable inputs (e.g., different tasks or languages) [3, 4,17,75,76], adversarial threats share the identical semantic content of the under- lying image being perturbed, which creates a strong inductive bias [26,78]. With- out an architectural component to capture these common characteristics [69], experts are prone to capturing redundant similar representations as shown in Fig. 1, thus preventing the construction of threat-specific model pathways. These threat-common features further induce threat-agnostic routing prob- lem, where the gating network receives insufficient discriminative signal and routes different threats to similar expert combinations, failing to mitigate cross- threat trade-offs. To demonstrate this issue, we analyze the gating weight distri- butions across threats in Fig. 3. We train ViT-B [15] on CIFAR-10 [30] with 4 FFN experts [47, 56] for each Transformer block using RANDOM [42] (full de- tails in Sec. A1 of supplementary) and visualize token-averaged gating weights from the final layer over 10K test samples. We first consider a naïve approach that trains the MoE with adversarial loss only (Eq. 1). As shown in Fig. 3a, this approach faces (i) routing collapse, where only a few experts (Expert 1) dominate, and (i) threat-agnostic routing, where all threats exhibit nearly identical gating distributions. To address routing collapse, we apply load balancing loss [56,64] widely used in conventional MoE [11,56,67] to encourage uniform expert utilization, with its loss coefficient set to 0.5 fol- lowing existing protocol [64]. As shown in Fig. 3b, while load balancing resolves 6W. Kim et al. Pre-trained Weights 퐖 Backbone Block Patch embeddings Block Block Output ... Hidden input 퐡 Hidden output 퐨 ... Low-rank experts ⊕ ⊗ Local feature 퐡 푖 (i) Mixture of Robust Low-Rank Experts (i) Threat-Distinguishing Dual-Scale Gating (i) Threat-Guided Gating Diversification Local gating 퐮 푖 Global gating 퐯 ... ... ... ...... Avg. global gating ത 퐯 푝 for Threat 풑 Avg. global gating ത 퐯 푞 for Threat 풒 Avg. local gating ഥ 퐮 풊 푝 for Threat 풑 Avg. local gating ഥ 퐮 풊 푞 for Threat 풒 ...... Local features Global features Total gating 퐠 푖 ℓ ∞ threatℓ 2 threatℓ 1 threat Global feature ҧ 퐡 ⊕ Layer-adaptive combination (Eq. 5) Avg. Avg. Local gating diversification (Eq. 6) Global gating diversification (Eq. 7) Fig. 4: Overview of RoME. (i) We train a mixture of robust low-rank experts, where a shared backbone captures threat-common features while low-rank experts focus on threat-specific information. To address threat-agnostic routing, where the gating as- signs similar expert combinations across threats, we propose (i) threat-distinguishing dual-scale gating, which leverages local patch-level and global image-level features to better distinguish between threats, and (i) threat-guided gating diversification, which ensures different threats are routed to distinct expert combinations. routing collapse, it still suffers from threat-agnostic routing, where the gating distributions remain similar across threats. Evaluation across different loss co- efficients in Fig. A1 also consistently exhibit threat-agnostic routing problem. This reveals that existing MoE techniques alone are insufficient, motivating our framework to learn distinct, threat-aware expert combinations (Fig. 3c). In this work, we propose RoME (Robust Mixture of Low-Rank Experts), illus- trated in Fig. 4, consisting of three components: (i) low-rank experts (Sec. 4.2) to capture threat-specific features, (i) threat-distinguishing dual-scale gating (Sec. 4.3) to provide gating with threat-discriminative signals, and (i) threat- guided gating diversification (Sec. 4.4) to enforce routing diversity across threats. 4.2 Mixture of Robust Low-Rank Experts To prevent experts from redundantly learning shared information common across threats, we implement each expert as a low-rank additive update to the backbone weights. We allow the shared backbone to capture threat-common features, while the low-rank experts focus on capturing threat-specific information. Inspired by its recent success, we implement these low-rank experts as a LoRA [24] module. Each expert k ∈ 1,...,K learns an additive adjustment to the backbone layer with weight matrix W ∈ R d out ×d in via low-rank matrices B k ∈ R d out ×r and A k ∈ R r×d in , where r ≪ d out ,d in . Given input feature h i ∈ R d in at token i, the output o i ∈ R d out is computed as: o i = Wh i + K X k=1 g i,k · B k A k h i .(2) RoME: Robust Mixture of Low-Rank Experts7 ` ∞ threat ` 2 threat ` 1 threat (a) Patch-level features ` ∞ threat ` 2 threat ` 1 threat (b) Image-level features Fig. 5: t-SNE visualization of threat separability at different feature levels. (a) Local patch-level features better separate ℓ 1 threats (silhouette: 0.394) but show overlap for ℓ ∞ and ℓ 2 . (b) Global image-level features show the complementary pattern, better distinguishing ℓ ∞ (silhouette: 0.611) over ℓ 1 and ℓ 2 . Gating weight g i = [g i,1 ,...,g i,K ] ⊤ ∈ R K is predicted for each token i (i.e., patch) in an image by a gating network such that g i,k ∈ [0, 1] and P K k=1 g i,k = 1. 4.3 Threat-Distinguishing Dual-Scale Gating To address threat-agnostic routing, we propose dual-scale gating that leverages both local patch-level and global image-level features to distinguish between threats. This is based on the intuition that adversarial threats exhibit discrimi- native patterns at different feature levels. For instance, ℓ 1 threats create sparse perturbations on specific pixels, best captured by patch-level features, while ℓ ∞ perturbations are spatially uniform and are more evident in image-level features. To verify this, we visualize feature separability using t-SNE [41] (Fig. 5) from ViT-B [15] trained with RANDOM strategy [42] on CIFAR-10. For patch-level features (Fig. 5a), we compute separability for all T token positions through silhouette coefficient score [53]. For image-level features (Fig. 5b), we average features across all patches and compute separability score. Our analysis reveals an interesting observation that local features show substantially higher sepa- rability for ℓ 1 threats (mean silhouette score: 0.394± 0.155) compared to ℓ ∞ (0.125± 0.077) and ℓ 2 (0.078± 0.053). Conversely, global image-level features exhibit the opposite trend (ℓ 1 : 0.303 , ℓ 2 : 0.281, ℓ ∞ : 0.611), indicating that ℓ ∞ threats are more distinguishable compared to ℓ 1 and ℓ 2 threats. Analysis across layers and patches in Fig. A4 verifies the same phenomenon. Based on these observations, we design a dual-scale gating comprised of local gating and global gating, each capturing patch-wise and image-level perturbation patterns, respectively. Given the hidden state h i ∈ R d in at token i, the local gating network MLP local predicts patch-wise gating weight u i as follows: u i = softmax (MLP local (h i ))∈ R K .(3) For image-level information, we average hidden states across all T tokens to obtain ̄ h = 1 T P T i=1 h i and pass it through the global gating network MLP global : v = softmax MLP global ( ̄ h) ∈ R K .(4) 8W. Kim et al. This outputs a single gating weight vector v for an image. Layer-adaptive combination. Early Transformer layers encode token-level information while deeper layers capture global semantic-level features [20, 37], creating a layer-wise bias where global features are under-represented in early layers and local features in deeper layers. Since both features are crucial for threat discrimination (Fig. 5), we compensate for this bias as: g i = (1− β(l))· u i + β(l)· v,(5) where g i = [g 1,i ,...,g K,i ] T is the final gating weight for token i, and g k,i denotes the weight for expert k. The layer-adaptive coefficient β(l) = σ s· 1− l L − b ∈ [0, 1] emphasizes global gating in early layers and local gating in deeper layers, with σ(·) the sigmoid function, L the total number of layers, and s, b controlling transition across layers [37]. Since the structural property of Transformer layer hierarchy is well-established [20,37], we encode it directly as a fixed prior, which empirically outperforms the learnable variant (Table 4 in Sec. 5.3). 4.4 Threat-Guided Gating Diversification While our dual-scale gating captures discriminative cues for each threat, it still requires explicit supervision to learn diverse, threat-specific expert combina- tions. To this end, we introduce a regularization loss that encourages the gating network to assign distinct expert combinations across threats, effectively con- structing separate model pathways for each threat type. Let B p denote adversarial examples under threat A p for p ∈ 1, 2,∞. For each threat p and token i, we compute the average local gating ̄ u p i across B p . We then maximize pairwise Euclidean distances between threat-specific gating patterns, aggregating across all T token positions: L div-local =− 1 T X p̸=q T X i=1 ∥ ̄ u p i − ̄ u q i ∥ 2 2 . (6) Since global gating weights are K-dimensional (number of experts), we project them to a higher K ′ dimension via a linear layer for finer-grained threat discrim- ination. The average projected gating ̄ v p for threat p yields the following loss: L div-global =− X p̸=q ∥ ̄ v p − ̄ v q ∥ 2 2 . (7) This projection is used for computing diversification loss and does not affect routing in Eq. 4. In Table A5, we explore other options for measuring the distance between gating weights, out of which our design leads to the best performance. We aggregate both objectives for each layer l across all L layers to encourage gating networks to learn distinct, threat-specific expert routing: L div = 1 L L X l=1 L (l) div-local +L (l) div-global . (8) RoME: Robust Mixture of Low-Rank Experts9 The diverse model pathways learned by the experts provide broader coverage of the perturbation space, improving robustness against unseen threats compared to existing MAT methods that rely on a single model pathway. Furthermore, since threat labels are only used during training, the learned gating predicts the optimal expert combination at inference without requiring knowledge of the input threat type, allowing our method to generalize to unseen threats. Notably, this generalization extends beyond ℓ p threats and improves robustness also against various non-ℓ p threats (Table 2, Sec. 5.2). 4.5 Training Objective Thanks to its modular design, RoME can be integrated with existing MAT methods. In this work, we apply it to MAX [59] and RANDOM [42]. The training objective combines the base loss (Eq. 1) with our diversification regularization: min θ E (x,y)∼D [φ (L 1 ,L 2 ,L ∞ )] + λL div ,(9) where φ denotes aggregation function φ MAX or φ RANDOM , and λ controls strength of L div . θ denotes parameters for backbone, experts, and gating networks. While this section focuses on ℓ p -MAT, modular design of our RoME makes it widely applicable to other adversarial training methods. In Sec. 5.2, we demon- strate this by applying RoME to non-ℓ p perceptual adversarial training [13,33] and show that RoME further improves their robustness against unseen threats. 5 Experiments 5.1 Experimental Setup Datasets, models, and baselines. We evaluate on CIFAR-10 [30], ImageNet- 100 [6], and ImageNet-1K [14] using standard ImageNet-1K pre-trained [14,62] ViT-B [15], DeiT-B [58], and Swin-B [38]. For MAT, we compare with RAN- DOM [42], AVG [59], MAX [59], MSD [44], MORE [5], E-AT [9], and RAMP [28]. For evaluation against unseen threats, we follow OODRobustBench [36] and also compare with PAT [33] and VR [13], two representative non-ℓ p adversarial train- ing methods trained on AlexNet [31]-based and self-model perceptual threats. For all baselines, we use default hyperparameters from their original manuscripts. Threat setup. Following existing protocols [9, 28, 59], we train and evalu- ate under PGD [43] and APGD [8] with perturbation budgets (ε 1 ,ε 2 ,ε ∞ ) of (12, 0.5, 8 255 ) for CIFAR-10 and (255, 2.0, 4 255 ) for ImageNet-100, 1K. For PGD [42, 44,59], we use steps (n 1 ,n 2 ,n ∞ ) of (20, 20, 10) for CIFAR-10 and (40, 20, 10) for ImageNet-100, 1K. For APGD [9,28], we use steps n = 15 for both CIFAR-10 and ImageNet-100, 1K. For robust fine-tuning, following E-AT [9] and RAMP [28], we use APGD with 10 steps on CIFAR-10 and 5 (ℓ ∞ /ℓ 2 ) and 15 (ℓ 1 ) steps on ImageNet-1K. For evaluation, we use AutoAttack under ℓ 1 , ℓ 2 , and ℓ ∞ norms. We evaluate natural accuracy on clean images, per-threat robustness, their av- erage, and worst-case union robustness across all threats [28]. 10W. Kim et al. Table 1: Comparison with state-of-the-art methods on CIFAR-10 and ImageNet-100 under AutoAttack. We measure natural accuracy on clean images, single-threat robust- ness under ℓ 1 , ℓ 2 , and ℓ ∞ , their average, and worst-case union robustness. Best results are marked in bold. Methods CIFAR-10ImageNet-100 Nat ℓ 1 ℓ 2 ℓ ∞ Avg UnionNat ℓ 1 ℓ 2 ℓ ∞ Avg Union PGD RANDOM86.7 52.3 68.8 40.3 53.8 38.684.4 43.8 67.6 45.2 52.2 39.4 AVG86.3 52.2 67.5 38.9 52.9 37.983.6 44.3 66.4 44.8 51.8 39.3 MAX85.1 46.6 66.8 40.1 51.2 38.579.9 42.2 63.2 47.5 51.0 40.5 MSD84.7 43.9 67.3 40.5 50.6 38.380.2 42.6 64.3 45.8 50.9 40.2 MORE82.6 44.3 42.4 33.7 40.1 31.175.6 34.5 41.2 42.0 39.2 32.8 RoME+RANDOM (Ours)88.156.469.742.156.141.185.246.968.347.854.342.9 RoME+MAX (Ours)85.448.268.243.153.242.380.645.666.050.954.243.7 APGD RANDOM86.5 50.1 69.3 37.5 52.3 37.282.0 38.4 66.0 46.2 50.2 37.1 MAX81.7 45.5 65.3 42.6 51.1 41.479.5 41.2 63.6 44.8 49.9 39.8 E-AT83.2 50.3 68.9 40.6 53.3 39.783.4 40.6 64.2 45.4 50.1 39.2 RAMP82.2 47.1 63.5 43.4 51.3 42.582.2 42.6 63.4 45.1 50.4 41.8 RoME+RANDOM (Ours)87.454.771.640.055.439.483.442.167.848.652.840.5 RoME+MAX (Ours)82.548.967.344.253.543.780.244.264.347.952.142.8 Implementation details. For fair comparison, we follow standard Transformer training recipe [15,58] for all baselines and our method, using AdamW [39] with initial learning rates of 1e-3 for CIFAR-10 and 1e-4 for ImageNet-100, 1K, with warmup and linear decay scheduling (full details in Sec. A1). We train for 20 epochs on CIFAR-10 and 5 epochs on ImageNet-100, 1K. For robust fine-tuning, we fine-tune ℓ ∞ -robust models for 3 epochs on CIFAR-10 and 1 epoch on ImageNet-1K. We use K = 4 experts of rank 16, apply them to QKV and O projection layers in all Transformer blocks, and set λ = 0.1, s = 4, b = 2, and K ′ = 100 for global gating projection. We also apply our method to CNNs (WideResNet [71]), where low-rank experts on convolutional layers are implemented using the official LoRA implementation 3 , and local and global features are obtained per spatial location of the convolutional feature map and by global average pooling over all spatial locations, respectively. 5.2 Multi-Perturbation Adversarial Robustness Robustness against multiple ℓ p perturbations. In Table 1, we report nat- ural accuracy and robustness against ℓ p threats seen during training. Our ap- proach effectively mitigates cross-threat trade-off, achieving the best average and union robustness across all settings. For example, our RoME+MAX im- proves union robustness of baseline MAX by 3.8%p on CIFAR-10 under PGD and 3.0%p on ImageNet-100 under APGD. We further validate on ImageNet- 1K (Table A2), where our method improves union robustness by 1.7%p and 1.7%p over RANDOM and MAX under PGD and by 1.3%p and 1.6%p under APGD, demonstrating its scalability. It also consistently outperforms baselines 3 https://github.com/microsoft/LoRA/blob/main/loralib/layers.py RoME: Robust Mixture of Low-Rank Experts11 Table 2: Comparison with prior methods on CIFAR-10 under unseen common corrup- tions and adversarial threats. Percep. ℓ p is the average of Fog [33], Snow [33], Gabor [33], Elastic [33], and ℓ ∞ -Jpeg [33] robustness. Best results are marked in bold. MethodsCom. Corr. ℓ 0 Percep. ℓ p PPGD LPA Adv. Patch StAdv ReColorAdv GMA PGD RANDOM79.031.953.243.7 32.247.931.170.444.0 AVG78.426.754.542.9 31.048.629.268.843.7 MAX76.828.351.343.2 33.143.934.470.045.4 MSD76.528.050.442.4 33.340.834.970.845.6 MORE74.923.850.742.7 31.545.232.170.043.6 RoME+RANDOM (Ours)80.238.357.851.235.448.836.873.545.6 RoME+MAX (Ours)76.839.456.252.240.747.444.075.447.5 APGD RANDOM78.531.654.549.5 45.549.147.077.840.0 MAX74.233.451.650.5 49.247.254.676.644.9 E-AT74.334.149.749.7 47.948.048.575.542.8 RAMP74.933.151.651.0 50.049.449.976.545.1 RoME+RANDOM (Ours)79.638.858.950.947.050.449.278.243.3 RoME+MAX (Ours)74.537.853.051.551.649.256.078.045.6 PAT77.616.047.155.5 51.541.259.776.449.0 PAT+VR77.79.947.956.6 51.939.358.477.050.1 RoME+PAT (Ours)78.119.848.657.054.945.263.377.549.8 Non-ℓ p RoME+PAT+VR (Ours)78.317.949.159.357.845.765.280.152.6 on DeiT-B [58] and Swin-B [38] (Table A1). Our approach also achieves the high- est natural accuracy. This is because prior MAT methods force all inputs into a single representation space, where the accuracy-robustness trade-off inherent in adversarial training [60,72] degrades performance on clean images. In contrast, our RoME learns a unique model pathway for natural images differently from adversarial examples as shown in Fig. A2 (Sec. A3.2). Robustness against unseen attacks. Following previous benchmarks [9,28], we evaluate on common corruptions [22], ℓ 0 attack [46], ℓ p perceptual attacks [33], and AutoAttack [8]. We also evaluate on non-ℓ p threats, including perceptual PPGD [33] and LPA [33], adversarial patch [2], spatial StAdv [65], and se- mantic ReColorAdv [32] attacks. We also design and evaluate against gating misrouting attack (GMA), a strong white-box adaptive attack with full ac- cess to the gating framework. GMA optimizes L GMA = L CE (f θ (x + δ),y) − 1 L P L l=1 1 T P T i=1 L CE (g (l) i (x + δ),k ∗ i ), where the first term maximizes the classi- fication loss and the second forces misrouting of each token’s gating weight g (l) i (Eq. 5) to the least likely expert k ∗ i = arg min k g i,k . For this attack, we use ℓ ∞ - PGD with ε = 8 255 and 20 steps. Full attack configurations (budget and steps) are provided in Sec. A1. Table 2 shows that our approach outperforms existing MAT methods across all unseen threats. Notably, while MORE [5] also leverages MoE, it shows lower robustness, highlighting that our approach of learning distinct model pathways improves generalization to unseen threats. Our method also outperforms existing methods against non-ℓ p threats (e.g., PPGD and LPA), demonstrating that the diverse representations captured across multiple model pathways generalize 12W. Kim et al. Table 3: Comparison with existing methods on robust fine-tuning scenario across different model architectures pre-trained on ℓ ∞ -adversarial training with different datasets. Best results are marked in bold. ViT-B (CIFAR-10) WRN-28-10 (CIFAR-10) WRN-94-16 (CIFAR-10) XCiT-S (ImageNet-1K) MethodNatUnionNatUnionNatUnionNatUnion E-AT83.641.789.450.392.844.765.025.9 +RoME 84.743.591.351.893.348.466.127.6 MAX82.943.188.751.491.251.464.127.2 +RoME 83.644.490.553.795.257.565.130.5 RAMP83.143.789.252.390.755.862.829.4 +RoME82.746.591.354.293.658.764.232.1 beyond the ℓ p threats seen during training. It maintains highest robustness even against adaptive attacks, verifying the robustness of our gating framework. To demonstrate the modularity of our RoME (Sec. 4.5), we also compare with PAT [33] and VR [13], two representative non-ℓ p adversarial training meth- ods designed for unseen threat robustness. Different from ℓ p -MAT, we follow the protocol of OODRobustBench [36] and train model on AlexNet [31]-based and self-model perceptual threats [33]. Applying our method on these methods con- sistently improves unseen threat robustness with gains of up to 6.8%p against StAdv, demonstrating that RoME is broadly applicable beyond ℓ p -MAT. Robust fine-tuning. In addition to training the model from scratch, we evalu- ate our method on robust fine-tuning scenario following E-AT [9]. For CIFAR-10, we use ViT-B-16 pre-trained with adversarial training on ℓ ∞ threats [43] along with WRN-28-10 [71] and WRN-94-16 [71] from RobustBench [7]. For ImageNet- 1K, we use XCiT-S [1] also from RobustBench. As shown in Table 3, applying RoME to fine-tuning on ℓ ∞ -robust models consistently improves union AutoAt- tack robustness across all models and baseline methods. For example, applying RoME to RAMP reaches 58.7% union robustness on WRN-94-16 (+14.0%p vs. E-AT, +2.9%p vs. RAMP), with similar trends on WRN-28-10 and XCiT-S. Be- yond robustness, RoME also improves natural accuracy in most settings (e.g., +4.0%p on WRN-94-16 with MAX). These consistent gains across ViT, CNN (WRN), and XCiT backbones show that RoME is architecture-agnostic. 5.3 Ablation Studies In Table 4, we provide ablations on individual components of our RoME. Low-rank experts. We verify that FFN experts leads to degraded union ro- bustness, confirming that it is vital to capture threat-common features with a shared backbone. Our low-rank experts also outperforms adapter [23]-based ex- perts, verifying the effectiveness of our design choices. In Table A4, we analyze the effect of applying RoME to different layers and find that applying experts to QKV and O projection layers achieves the best efficiency-robustness trade-off. RoME: Robust Mixture of Low-Rank Experts13 Table 4: Ablation studies on individual components of RoME on CIFAR-10. ‡ indicates using 3 experts for a fair comparison with gating classification, which requires one expert per threat; others use 4 experts. Best results are marked in bold. MethodsClean ℓ 1 ℓ 2 ℓ ∞ Avg. Union Low-rank experts (Sec. 4.2) Expert architectures: FFN-based experts76.546.4 58.6 31.5 45.530.9 Adapter-based experts88.055.2 70.6 40.2 55.338.6 Ours (Low-rank experts)88.1 56.4 69.7 42.1 56.1 41.1 Dual-scale gating (Sec. 4.3) No learned gating: No gating (uniform weights)87.153.7 69.1 38.3 53.738.1 Single-scale gating: Local (token-level) only87.553.1 69.6 39.9 54.239.4 Global (image-level) only87.055.0 69.3 40.2 54.839.8 Dual-scale gating variants: Inverted layer weights87.655.2 69.4 39.7 54.839.3 Learned layer weights87.653.7 69.5 40.3 54.539.6 CLS token for global87.854.5 69.2 39.0 54.238.7 Ours (avg. + adaptive)88.1 56.4 69.7 42.1 56.1 41.1 Threat-guided gating diversification loss (Sec. 4.4) Conventional MoE regularization: No regularization86.752.9 69.3 38.7 53.638.1 Load balancing [56]86.955.0 69.5 38.8 54.438.4 Importance [56]87.053.5 69.3 39.0 53.938.6 Threat-aware regularization: Gating classification ‡ 87.542.9 69.8 34.4 49.031.4 Global gating projection: w/o projection layer87.756.7 69.9 41.6 56.140.6 Ours (diversification) ‡ 87.9 57.4 69.5 41.9 56.341.0 Ours (diversification)88.156.4 69.7 42.1 56.1 41.1 Dual-scale gating. Removing learned gating degrades robustness, showing that threat-specific expert selection is critical. Single-scale gatings underperform our approach, verifying that different granularities are required for threat discrimi- nation (Fig. 5). While inverting or learning the layer-adaptive weights (Eq. 5) improves robustness thanks to dual-scale gating, both remain suboptimal, verify- ing that encoding the Transformer layer hierarchy [20,37] as a fixed prior is more effective than making it learnable. Replacing averaged global feature with CLS token degrades performance, verifying the effectiveness of our design choices. Gating diversification. Applying no regularization, load balancing [56], or im- portance loss [56] leads to suboptimal robustness due to threat-agnostic routing. Gating classification, which trains the gating network as a threat classifier with cross-entropy loss, remains suboptimal as it maps each threat to a single fixed expert unlike our flexible diversification loss. Removing the projection layer in 14W. Kim et al. ` ∞ threat ` 2 threat ` 1 threat (a) Backbone ` ∞ threat ` 2 threat ` 1 threat (b) Backbone + experts Fig. 6: t-SNE visualization of (a) backbone features and (b) backbone + expert outputs from the final ViT-B layer. Table 5: Analysis on learned experts and gating during inference with RAN- DOM and PGD training on CIFAR-10. Best results are marked in bold. MethodsClean ℓ 1 ℓ 2 ℓ ∞ Avg. Union Random gating88.0 55.1 69.5 39.3 54.6 38.4 Inverse gating87.8 55.4 69.0 39.6 54.7 38.9 w/o experts 88.1 54.9 69.4 39.9 54.7 39.2 Ours88.1 56.4 69.7 42.1 56.1 41.1 global diversification reduces union robustness to 40.6%, and alternative distance choices in Table A5 all underperform our method. Fig. A3 further shows that our method is not overly sensitive to hyperparameters including expert rank, number of experts, loss weight λ, and s, b for layer-adaptive weighting. 5.4 Analysis on Experts and Threat-Aware Gating Learned features on backbone vs. experts. To verify whether low-rank ex- perts learn threat-specific features, we analyze t-SNE visualizations of backbone features and expert outputs added to the backbone in Fig. 6. Backbone captures similar features regardless of threat type as shown in Fig. 6a, showing that it has captured features common among threats. In contrast, features passed through our low-rank experts capture decoupled representations for each threat as shown in Fig. 6b, showing that our experts have learned threat-specific features. Learned routing for ℓ p threats. To understand how threats affect expert routing, we apply different threats to various spatial regions and visualize the resulting expert activations for each image patch. As shown in Fig. 7, standard MoE shows similar expert activations for all image patches (Expert 1 in row 1, left), verifying that experts are not specialized for a specific threat type. In contrast, our RoME learns threat-specific routing; for example, Expert 1 (row 2) is highly activated for ℓ ∞ threats while barely activated for ℓ 1 threats, demonstrating that each expert specializes in a distinct threat type. Analysis on experts and gating. We analyze learned experts and gating through inference-time variants (Table 5). Random weights, inverted gating 1−g, or removing experts degrades robustness, verifying that gating learns threat- specific combinations and that experts learn specialized representations. Computational analysis. In Table 6, we compare computational costs with baseline RANDOM [42]. MORE [5] and conventional MoE [56] with FFN ex- perts incur 4× and 3× more parameters, with 8.25× and 2.75× more train time compared to RANDOM. In contrast, our method achieves SoTA union robust- ness (Table 1) with only 1.04× more parameters and 1.17× more train time. Our method also shows superior optimization efficiency, consistently outperforming robustness of RANDOM given the same wall-clock training time (Fig. 8). It also reaches RANDOM’s peak robustness in 13.2k sec of training versus 23.9k sec for RoME: Robust Mixture of Low-Rank Experts15 21 Std. MoE Adversarial Image Expert 0Expert 1Expert 2Expert 3 0.00 0.25 0.50 0.75 1.00 Activation 21 Ours (RoME) 0.00 0.15 0.30 Activation 21 Std. MoE Adversarial Image Expert 0Expert 1Expert 2Expert 3 0.00 0.25 0.50 0.75 1.00 Activation 21 Ours (RoME) 0.00 0.15 0.30 Activation Fig. 7: Expert activations for various ℓ p threats applied to different regions of im- age. While standard MoE (rows 1, 3) routes threats to same experts, RoME (rows 2, 4) utilizes different experts for each threat. Table 6: Computational cost compari- son. Best results are marked in bold. Methods # Params FLOPs Train time Latency (M)(G)(sec/iter) (ms/img) RANDOM [42]85.155.613.050.53 MORE [5]340.6022.4325.161.70 MoE (FFN) [56]255.209.298.381.36 Ours88.855.843.560.60 0100002000030000 Training time (sec) 10 20 30 40 Union Rob. RANDOM + Ours Fig. 8: Training time vs. robustness. Ours consistently outperforms baseline given the same wall-clock training time. RANDOM. Baselines with additional expert and gating layers but without our training components in Table A3 show barely improved robustness, showing that simply increasing model capacity does not address the cross-threat trade-off. 6 Conclusion We introduce RoME, a novel framework that addresses cross-threat robustness trade-off in multi-perturbation adversarial training through mixture of low-rank experts. We identify the threat-agnostic routing problem where conventional mixture of experts faces difficulty distinguishing between threats and learns sim- ilar routing. To address this, we propose threat-distinguishing dual-scale gating and threat-guided gating diversification that enable distinct expert combinations for each threat. Extensive experiments demonstrate that RoME achieves state- of-the-art union robustness across diverse benchmarks, while also improving ro- bustness against a diverse set of unseen threats. Ablation studies and analysis demonstrate the ability of our method to effectively distinguish between threats and learn diverse threat-specific pathways, leading to improved robustness. Acknowledgement. Sung-eui Yoon is the corresponding author. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2023- 00237965, Recognition, Action and Interaction Algorithms for Open-world Robot Service) and the IITP(Institute of Information & Communications Technology Planning & Evaluation)-ITRC(Information Technology Research Center) grant funded by the Korea government(Ministry of Science and ICT)(IITP-2026-RS- 2020-I201460). 16W. Kim et al. References 1. Ali, A., Touvron, H., Caron, M., Bojanowski, P., Douze, M., Joulin, A., Laptev, I., Neverova, N., Synnaeve, G., Verbeek, J., et al.: Xcit: Cross-covariance image transformers. In: NeurIPS (2021) 2. Brown, T.B., Mané, D., Roy, A., Abadi, M., Gilmer, J.: Adversarial patch. In: NeurIPS (2017) 3. Chen, T., Chen, X., Du, X., Rashwan, A., Yang, F., Chen, H., Wang, Z., Li, Y.: Adamv-moe: Adaptive multi-task vision mixture-of-experts. In: ICCV (2023) 4. Chen, Z., Shen, Y., Ding, M., Chen, Z., Zhao, H., Learned-Miller, E.G., Gan, C.: Mod-squad: Designing mixtures of experts as modular multi-task learners. In: CVPR (2023) 5. Cheng, H., Xu, K., Wang, C., Kailkhura, B., Lin, X., Goldhahn, R.: Mixture of robust experts (more): A robust denoising method towards multiple perturbations. arXiv preprint arXiv:2104.10586 (2021) 6. Chun-Hsiao Yeh, Y.C.: IN100pytorch: Pytorch implementation: Training resnets on imagenet-100. https://github.com/danielchyeh/ImageNet-100-Pytorch (2022), accessed: 2026-03-04 7. Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chi- ang, M., Mittal, P., Hein, M.: Robustbench: a standardized adversarial robustness benchmark. In: NeurIPS Datasets and Benchmarks Track (2021) 8. Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: ICML (2020) 9. Croce, F., Hein, M.: Adversarial robustness against multiple and single l_p-threat models via quick fine-tuning of robust classifiers. In: ICML (2022) 10. Croce, F., Rebuffi, S.A., Shelhamer, E., Gowal, S.: Seasoning model soups for robustness to adversarial and natural distribution shifts. In: CVPR (2023) 11. Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., et al.: Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.06066 (2024) 12. Dai, S., Cianfarani, C., Bhagoji, A., Sehwag, V., Mittal, P.: Adapting to evolving adversaries with regularized continual robust training. In: ICML (2025) 13. Dai, S., Mahloujifar, S., Mittal, P.: Formulating robustness against unforeseen at- tacks. In: NeurIPS (2022) 14. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009) 15. Dosovitskiy, A.: An image is worth 16x16 words: Transformers for image recogni- tion at scale. In: ICLR (2021) 16. Dou, S., Zhou, E., Liu, Y., Gao, S., Shen, W., Xiong, L., Zhou, Y., Wang, X., Xi, Z., Fan, X., et al.: Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In: ACL (2024) 17. Fan, Z., Sarkar, R., Jiang, Z., Chen, T., Zou, K., Cheng, Y., Hao, C., Wang, Z., et al.: M 3 vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design. In: NeurIPS (2022) 18. Fedus, W., Zoph, B., Shazeer, N.: Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research (2022) 19. Gao, C., Chen, K., Rao, J., Liu, R., Sun, B., Zhang, Y., Peng, D., Guo, X., Subrah- manian, V.: Mola: Moe lora with layer-wise expert allocation. In: NAACL (2025) RoME: Robust Mixture of Low-Rank Experts17 20. Geva, M., Schuster, R., Berant, J., Levy, O.: Transformer feed-forward layers are key-value memories. In: EMNLP (2021) 21. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: ICLR (2015) 22. Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common corruptions and perturbations. In: ICLR (2019) 23. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Ges- mundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: ICML (2019) 24. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. In: ICLR (2022) 25. Huang, C., Liu, Q., Lin, B.Y., Pang, T., Du, C., Lin, M.: Lorahub: Efficient cross- task generalization via dynamic lora composition. In: COLM (2023) 26. Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial examples are not bugs, they are features. In: NeurIPS (2019) 27. Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural computation (1991) 28. Jiang, E., Singh, G.: Ramp: Boosting adversarial robustness against multiple l_p perturbations for universal robustness. In: NeurIPS (2024) 29. Jung, M.J., Kim, J.: Pmoe: Progressive mixture of experts with asymmetric trans- former for continual learning. arXiv preprint arXiv:2407.21571 (2024) 30. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009) 31. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep con- volutional neural networks. In: NeurIPS (2012) 32. Laidlaw, C., Feizi, S.: Functional adversarial attacks. In: NeurIPS (2019) 33. Laidlaw, C., Singla, S., Feizi, S.: Perceptual adversarial robustness: Defense against unseen threat models. In: ICLR (2021) 34. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., Chen, Z.: Gshard: Scaling giant models with conditional computation and automatic sharding. In: ICLR (2021) 35. Li, H., Lin, S., Duan, L., Liang, Y., Shroff, N.B.: Theory on mixture-of-experts in continual learning. In: ICLR (2025) 36. Li, L., Wang, Y., Sitawarin, C., Spratling, M.: Oodrobustbench: a benchmark and large-scale analysis of adversarial robustness under distribution shift. In: ICML (2024) 37. Liao, M., Chen, W., Shen, J., Guo, S., Wan, H.: Hmora: Making llms more effective with hierarchical mixture of lora experts. In: ICLR (2025) 38. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin trans- former: Hierarchical vision transformer using shifted windows. In: ICCV (2021) 39. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019) 40. Ma, J., Zhao, Z., Yi, X., Chen, J., Hong, L., Chi, E.H.: Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In: KDD (2018) 41. Maaten, L.v.d., Hinton, G.: Visualizing data using t-sne. In: JMLR (2008) 42. Madaan, D., Shin, J., Hwang, S.J.: Learning to generate noise for multi-attack robustness. In: ICML (2021) 43. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: ICLR (2018) 44. Maini, P., Wong, E., Kolter, Z.: Adversarial robustness against the union of multiple perturbation models. In: ICML (2020) 18W. Kim et al. 45. Meymani, M., Razavi-Far, R.: Defending against adversarial attacks using mixture of experts. arXiv preprint arXiv:2512.20821 (2025) 46. Modas, A., Moosavi-Dezfooli, S.M., Frossard, P.: Sparsefool: a few pixels make a big difference. In: CVPR (2019) 47. Mu, S., Lin, S.: A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07137 (2025) 48. Pavlitska, S., Eisen, E., Zöllner, J.M.: Towards adversarial robustness of model- level mixture-of-experts architectures for semantic segmentation. In: 2024 Interna- tional Conference on Machine Learning and Applications (ICMLA) (2024) 49. Pavlitska, S., Fan, H., Ditschuneit, K., Zöllner, J.M.: Robust experts: the effect of adversarial training on cnns with sparse mixture-of-experts layers. In: ICCV (2025) 50. Puigcerver, J., Jenatton, R., Riquelme, C., Awasthi, P., Bhojanapalli, S.: On the adversarial robustness of mixture of experts. In: NeurIPS (2022) 51. Puigcerver, J., Riquelme, C., Mustafa, B., Houlsby, N.: From sparse to soft mixtures of experts. In: ICLR (2024) 52. Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Su- sano Pinto, A., Keysers, D., Houlsby, N.: Scaling vision with sparse mixture of experts. In: NeurIPS (2021) 53. Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics (1987) 54. Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016) 55. Shafahi, A., Najibi, M., Ghiasi, M.A., Xu, Z., Dickerson, J., Studer, C., Davis, L.S., Taylor, G., Goldstein, T.: Adversarial training for free! In: NeurIPS (2019) 56. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In: ICLR (2017) 57. Singh, N.D., Croce, F., Hein, M.: Revisiting adversarial training for imagenet: Architectures, training and generalization across threat models. In: NeurIPS (2023) 58. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: ICML (2021) 59. Tramer, F., Boneh, D.: Adversarial training and robustness for multiple perturba- tions. In: NeurIPS (2019) 60. Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., Gu, Q.: Improving adversarial ro- bustness requires revisiting misclassified examples. In: ICLR (2020) 61. Wang, Z., Li, X., Zhu, H., Xie, C.: Revisiting adversarial training at scale. In: CVPR (2024) 62. Wightman, R.: Pytorch image models. https://github.com/rwightman/pytorch- image-models (2019). https://doi.org/10.5281/zenodo.4414861 63. Wong, E., Rice, L., Kolter, J.Z.: Fast is better than free: Revisiting adversarial training. In: ICLR (2020) 64. Wu, X., Huang, S., Wei, F.: Mixture of lora experts. In: ICLR (2024) 65. Xiao, C., Zhu, J.Y., Li, B., He, W., Liu, M., Song, D.: Spatially transformed ad- versarial examples. In: ICLR (2018) 66. Xie, C., Yuille, A.: Intriguing properties of adversarial training at scale. In: ICLR (2020) 67. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025) RoME: Robust Mixture of Low-Rank Experts19 68. Yang, Y., Qin, Y., Wu, T., Xu, Z., Li, G., Guo, P., Shao, H., Shi, Y., Li, K., Sun, X., et al.: Leveraging open knowledge for advancing task expertise in large language models. arXiv preprint arXiv:2408.15915 (2024) 69. Yang, Y., Jiang, P.T., Hou, Q., Zhang, H., Chen, J., Li, B.: Multi-task dense prediction via mixture of low-rank experts. In: CVPR (2024) 70. Yu, J., Zhuge, Y., Zhang, L., Hu, P., Wang, D., Lu, H., He, Y.: Boosting contin- ual learning of vision-language models via mixture-of-experts adapters. In: CVPR (2024) 71. Zagoruyko, S., Komodakis, N.: Wide residual networks. In: BMVC (2016) 72. Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: ICML (2019) 73. Zhang, X., Xu, K., Hu, Z., Wang, R.: Optimizing robustness and accuracy in mixture of experts: A dual-model approach. In: ICML (2025) 74. Zhang, Y., Cai, R., Chen, T., Zhang, G., Zhang, H., Chen, P.Y., Chang, S., Wang, Z., Liu, S.: Robust mixture-of-expert training for convolutional neural networks. In: ICCV (2023) 75. Zhao, X., Chen, X., Cheng, Y., Chen, T.: Sparse moe with language guided routing for multilingual machine translation. In: ICLR (2024) 76. Zhou, H., Wang, Z., Huang, S., Huang, X., Han, X., Feng, J., Deng, C., Luo, W., Chen, J.: Moe-lpr: Multilingual extension of large language models through mixture-of-experts with language priors routing. In: AAAI (2025) 77. Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A.M., Le, Q.V., Laudon, J., et al.: Mixture-of-experts with expert choice routing. In: NeurIPS (2022) 78. Zhou, Z., Firestone, C.: Humans can decipher adversarial images. Nature commu- nications (2019) RoME: Robust Mixture of Low-Rank Experts1 In this appendix, we first elaborate on additional implementation details (Sec. A1). We report results of applying our RoME to other Transformer-based vision models and ImageNet-1K (Sec. A2). We also provide additional analysis on our RoME (Sec. A3), including analysis on gating weights, effects of applying our method on different layers, ablations on our threat-guided gating diversification, and hyperparameter analysis. Lastly, we provide limitations and future work (Sec. A4). A1 Additional Experimental Setups In this section, we elaborate on additional implementation details. Implementation details. The learning rate schedule consists of warmup train- ing for first 4 epochs (1 epoch for ImageNet-100 and ImageNet-1K), linear decay to 0.05 of initial learning rate for next 12 epochs (3 epochs for ImageNet-100 and ImageNet-1K), then final linear decay to 0.001 of initial learning rate for the last 4 epochs (1 epoch for ImageNet-100 and ImageNet-1K). We set batch size to 64, 128, and 128 for CIFAR-10 [30], ImageNet-100 [6], and ImageNet-1K [14] datasets, respectively. We set patch size for vision Transformer models to 4 for CIFAR-10, 16 for ImageNet-100, and 16 for ImageNet-1K. Setup for unseen threats. For ℓ 0 attack, we use SparseFool [46] with 10 iterations and λ = 3. Following existing benchmarks [28], we use common cor- ruptions [22] across 5 levels of severity across all corruption types. For perceptual adversarial threats [33], we set ε = 12 for the fog attack, ε = 0.5 for the snow at- tack, ε = 60 for the gabor attack, ε = 0.125 for the elastic attack, and ε = 0.125 for the Jpeg-ℓ ∞ attack with 100 iterations. For PPGD [33] and LPA [33], we use LPIPS bound of 0.25 with 20 iterations. For adversarial patch [2], we use a square shaped patch of size 8× 8 with 50 iterations and step size of 2 255 . For spatial StAdv [65] attack, we set flow regularization loss weight τ to 0.05. For semantic ReColorAdv [32] attack, we use bound of 0.06 with 20 iterations. For gating misrouting attack (GMA), we combine the original classification loss with a loss that minimizes cross-entropy loss between gating outputs and least-likely expert with the same weight for each loss. Setup for conventional MoE in Fig. 3. To demonstrate the threat-agnostic issue with conventional mixture of experts when training against multiple adver- sarial threats, we train ViT-B [15] on CIFAR-10 [30]. We apply experts imple- mented as FFN layers at each layer of the Transformer block. We train the entire model using RANDOM strategy [42] for 20 epochs with learning rate 1e-3. Setup for feature analysis in Fig. 5. To analyze feature separability across different scales, we train ViT-B [15] without any experts using RANDOM strat- egy [42] on CIFAR-10 [30]. We train the model using RANDOM strategy [42] for 20 epochs with learning rate 1e-3. 2W. Kim et al. Table A1: Comparison with MAT methods using DeiT-B and Swin-B on CIFAR- 10 under AutoAttack. We measure natural accuracy on clean images, single-threat robustness under ℓ 1 , ℓ 2 , and ℓ ∞ , their average, and worst-case union robustness. Best results are marked in bold. Methods DeiT-BSwin-B Nat ℓ 1 ℓ 2 ℓ ∞ Avg UnionNat ℓ 1 ℓ 2 ℓ ∞ Avg Union PGD RANDOM87.2 53.4 68.0 39.5 53.6 38.981.1 49.6 62.0 31.5 47.7 31.4 AVG86.7 52.5 67.8 39.2 53.2 38.180.9 49.2 61.4 31.6 47.4 31.4 MAX84.8 43.1 62.5 40.3 48.6 39.578.5 45.1 59.7 37.1 47.3 36.2 MSD84.9 43.3 63.0 40.7 49.0 40.179.7 44.7 60.0 37.0 47.2 36.0 RoME+RANDOM (Ours)87.654.370.540.555.139.781.451.563.435.050.035.0 RoME+MAX (Ours)85.445.363.942.850.741.279.747.260.439.248.938.5 APGD RANDOM83.5 51.5 68.9 39.6 53.3 38.777.6 47.4 62.5 31.9 47.3 31.3 MAX79.3 45.8 64.1 43.9 51.3 41.973.1 42.8 58.5 36.2 45.8 35.6 E-AT80.5 52.1 68.3 40.2 53.5 39.074.5 47.7 62.1 32.1 47.3 31.5 RAMP79.7 46.9 63.8 44.0 51.6 42.374.2 43.3 58.8 37.1 46.4 36.1 RoME+RANDOM (Ours)84.553.970.340.654.939.978.950.063.134.749.334.6 RoME+MAX (Ours)80.149.065.444.352.943.375.144.660.339.848.239.3 Table A2: Comparison with MAT methods using ViT-B on ImageNet-1K under Au- toAttack. We measure natural accuracy on clean images, single-threat robustness under ℓ 1 , ℓ 2 , and ℓ ∞ , their average, and worst-case union robustness. Best results are marked in bold. MethodsNat ℓ 1 ℓ 2 ℓ ∞ Avg Union RANDOM64.9 18.6 41.3 20.1 26.7 15.3 MAX60.5 19.4 39.4 22.3 27.0 18.6 RoME+RANDOM (Ours)66.220.242.422.028.217.0 PGD RoME+MAX (Ours)61.621.439.624.728.620.3 RANDOM63.3 19.5 41.8 20.5 27.3 15.9 MAX60.3 20.0 39.1 23.1 27.4 19.0 RoME+RANDOM (Ours)63.820.942.822.828.817.2 APGD RoME+MAX (Ours)61.221.640.325.329.120.6 A2 Application on Other Models and Dataset Deit-B and Swin-B. In Table A1, we report comparison of our method with previous multi-perturbation adversarial training methods under DeiT-B [58] and Swin-B [38] on CIFAR-10. Our approach using MAX outperforms the union robustness of existing methods by at least 1.1%p under PGD, and our approach using RANDOM outperforms the average robustness and natural accuracy of existing methods by at least 1.5%p and 0.3%p, respectively, under PGD. ImageNet-1K dataset. In Table A2, we evaluate our method on the large-scale ImageNet-1K [14] dataset. Applying RoME improves union robustness over the baseline RANDOM and MAX methods by 1.7%p and 1.7%p under PGD, and by 1.3%p and 1.6%p under APGD, respectively. These consistent gains even on a large-scale dataset verify the scalability and generalizability of our RoME. RoME: Robust Mixture of Low-Rank Experts3 Table A3: Comparison with baselines trained on the same number of parameters as our method by simply adding expert and gating network layers without our proposed components. We use ViT-B as backbone model. Best results are marked in bold. MethodsNat ℓ 1 ℓ 2 ℓ ∞ Avg Union# Params PGD RANDOM86.7 52.3 68.8 40.3 53.8 38.685.15 + experts & gating layers86.9 52.4 68.8 40.4 53.9 38.688.85 MAX85.1 46.6 66.8 40.1 51.2 38.585.15 + experts & gating layers84.9 46.7 66.7 40.1 51.2 38.588.85 RoME+RANDOM (Ours)88.156.469.742.156.141.188.85 RoME+MAX (Ours)85.448.268.243.153.242.388.85 APGD RANDOM86.5 50.1 69.3 37.5 52.3 37.285.15 + experts & gating layers85.5 50.8 69.3 37.6 52.6 37.388.85 MAX81.7 45.5 65.3 42.6 51.1 41.485.15 + experts & gating layers82.3 45.1 65.5 42.2 50.9 41.288.85 RoME+RANDOM (Ours)87.454.771.640.055.439.488.85 RoME+MAX (Ours)82.548.967.344.253.543.788.85 Baselines with more parameters. In Table A3, we compare with baselines trained on the same model architectures as our method where extra layers for ex- perts and gating networks are applied, but without our core training components. We observe that simply increasing the model capacity from 85.15M to 88.85M parameters by adding experts and gating layers results in negligible performance gains. For example, adding these layers to the RANDOM strategy under APGD training only marginally improves union robustness from 37.2% to 37.3%, which remains lower than the 39.4% achieved by our RoME+RANDOM. In some in- stances, such as the MAX strategy under APGD, the additional layers without our proposed components even lead to a slight degradation in union robustness from 41.4% to 41.2%. These findings demonstrate that simply increasing the network capacity with extra expert and gating network layers does not address the robustness trade-offs in multi-perturbation adversarial training. Instead, the effectiveness of RoME stems specifically from our dual-scale gating and threat- guided gating diversification, which provide the necessary discriminative signals and supervision to enable diverse threat-specific model pathways. A3 Additional Analysis A3.1 Gating Weights Similarity In addition to Fig. 3 in the main paper, we visualize gating weight distributions for different loss coefficients λ lb of the load balancing loss [56] in Fig. A1. As λ lb decreases, the gating distribution (Fig. A1b) degenerates to the naïve approach, thus heavily relying on the same single expert for all threats. As λ lb increases, the gating distribution (Fig. A1d) becomes more uniform throughout the expert, thus making gating weights similar for all threats. Either way, load balancing 4W. Kim et al. 0.00.5 Expert activation Density 1 threat 0.00.5 Expert activation 2 threat 0.00.5 Expert activation threat Expert 0Expert 1Expert 2Expert 3 (a) Naïve approach 0.00.5 Expert activation Density 0.00.5 Expert activation 0.00.5 Expert activation (b) Load balancing loss [56] (λ lb = 0.00005) 0.00.5 Expert activation Density 0.00.5 Expert activation 0.00.5 Expert activation (c) Load balancing loss [56] (λ lb = 0.005) 0.00.5 Expert activation Density 0.00.5 Expert activation 0.00.5 Expert activation (d) Load balancing loss [56] (λ lb = 0.5) 0.00.5 Expert activation Density 0.00.5 Expert activation 0.00.5 Expert activation (e) Ours (RoME) Fig. A1: Analysis on gating weight distributions across different threat types on CIFAR-10 using Gaussian KDE. (a) Naïve MoE training suffers from both routing collapse (Expert 1 dominates) and threat-agnostic routing (similar distributions across threats). (b-d) Load balancing [56] resolves routing collapse but still exhibits threat- agnostic routing regardless of its loss coefficient λ lb . (e) Our RoME framework learns diverse, threat-specific expert combinations. loss fails to address the threat-agnostic issue and highlights the need for our RoME. RoME: Robust Mixture of Low-Rank Experts5 E0E1E2E3 Nat 1 2 0.140.310.450.10 0.130.290.470.11 0.140.290.460.12 0.130.310.440.12 Low-rank MoE + load balancing reg. E0E1E2E3 0.080.000.160.76 0.680.160.050.11 0.020.480.240.26 0.040.260.490.21 RoME (Ours) 0.0 0.2 0.4 0.6 Mean expert activation Fig. A2: Gating activation visualization of low-rank mixture of experts with load bal- ancing loss (left) and our RoME with dual-scale gating and threat-guided gating diver- sification (right). While baseline MoE fails to distinguish between threats and assigns the same expert combinations to different threats, our method effectively decouples model pathways for natural images and different threats. A3.2 Gating Activations In Fig. A2, we visualize the gating activations for each experts. Simply apply- ing low-rank mixture of experts and load balancing loss (left) shows that the gating network fails to distinguish between threats and collapses to using the same expert combinations for all threats. In contrast, our RoME with dual-scale gating and threat-guided gating diversification (right) allocates different expert combinations for each threat, effectively specializing each expert for different threats. This allows mitigation of cross-threat trade-off, thus improving union robustness. This allocation of experts for different threats also provides insight on im- provements in natural accuracy compared to baseline methods (Sec. 5.2). As shown in Fig. A2 (right), our RoME assigns a unique combination of experts for natural images, thus separating the model pathways for natural and adversar- ial images. This prevents a single model pathway from capturing both natural accuracy and adversarial robustness, thus alleviating the well-known accuracy- robustness trade-off in adversarial training [60, 72] and thus improving natural accuracy. A3.3 Analysis on Layers of RoME Application We apply our low-rank experts on different layer types of the backbone model in ViT-B-16. As shown in Table A4, applying experts to all layers – query/key/value projection (Attn-QKV), output projection (Attn-Out), and feed-forward net- works (FFN) – achieves the highest robustness. However, this comes at a greater computational cost. We find that applying RoME to query, key, value, and out- put projection layers achieves comparable performance with fewer parameters, providing a better efficiency-robustness trade-off. 6W. Kim et al. Table A4: Analysis on RoME application across different layers (QKV, output pro- jection, and FFN) in ViT-B-16 [15]. # Params denotes additional parameters incurred with mixture of low-rank experts. Best results are marked in bold. Layers Clean ℓ 1 ℓ 2 ℓ ∞ Avg. Union# Params Attn-QKV Attn-Out FFN ✓×87.4 53.7 70.4 41.4 55.2 41.02.44M ×✓×87.6 53.3 71.4 40.8 55.2 39.91.26M ×✓ 86.7 54.8 71.1 41.2 55.7 40.56.28M ✓×88.1 56.4 69.7 42.1 56.1 41.13.70M ✓×✓86.9 55.2 71.8 39.8 55.6 39.38.82M ×✓ 87.5 55.7 71.3 41.9 56.3 41.07.54M ✓ 86.4 55.9 70.1 42.5 56.2 41.99.98M Table A5: Analysis on varying distance choices for threat-guided gating diversification with RANDOM and PGD training on CIFAR-10. Best results are marked in bold. MethodsClean ℓ 1 ℓ 2 ℓ ∞ Avg. Union No regularization86.7 52.9 69.3 38.7 53.6 38.1 Cosine similarity87.0 54.8 69.6 40.2 54.9 39.4 KL divergence87.5 55.7 70.5 39.8 55.3 39.4 Wasserstein distance87.3 55.4 70.3 40.8 55.5 40.0 Ours (L2)88.1 56.4 69.7 42.1 56.1 41.1 A3.4 Different Distance Functions for Threat-Guided Gating Diversification We replace the L2 distance function in Eqs. 6 and 7 with different distance func- tions in Table A5. We first replace it with cosine similarity (row 2), which maxi- mizes the cosine similarity between the two gating weights to maximize angular separation in the expert usage space. We also replace it with KL divergence (row 3), which maximizes the symmetric KL divergence (i.e., KL(P||Q) + KL(Q||P )) between gating distributions to encourage divergent probability mass allocations across experts. We also try Wasserstein distance (row 4), which maximizes the 1-Wasserstein distance (earth mover’s distance) computed as the L1 distance between cumulative distribution functions of sorted gating scores. All three vari- ants outperform no regularization (row 1), verifying the necessity of learning different gating weights for each threat. However, they underperform our L2 distance. A3.5 Hyperparameter Analysis In Fig. A3, we provide sensitivity analysis on various hyperparameters. Number and capacity of experts. As shown in Figs. A3a and A3b, we analyze on various number of experts K and the rank of individual low-rank RoME: Robust Mixture of Low-Rank Experts7 23456 K (Number of Experts) 38 40 Union Robustness (%) (a) Number of experts K 481632 Rank 38 40 Union Robustness (%) (b) Rank for experts 10 3 10 2 10 1 10 0 (Weight on div ) 38 40 Union Robustness (%) (c) λ for L div 02468 s (b=2) 38 40 Union Robustness (%) (d) s for layer-adaptive coeffi- cient 20246 b (s=4) 38 40 Union Robustness (%) (e) b for layer-adaptive coeffi- cient Fig. A3: Effects of varying hyperparameters (a) number of experts, (b) rank for ex- perts, (c) λ for threat-guided gating diversificationL div , and (d-e) s and b for layer- adaptive coefficient on union robustness with RANDOM and PGD training on CIFAR- 10. experts. Union robustness improves as K increases from 2 to 4, showing that it is vital to capture diverse characteristics of threats with multiple experts, while the performance degrades as K increases beyond 4. We can observe similar behavior of improved robustness when increasing the rank to 16, but degraded performance as we increase it to 32. This is a common phenomenon in mixture of low-rank experts [16, 25, 64] where increasing the computational capacity of experts after a certain amount degrades performance. Gating diversification loss weight λ. As shown in Fig. A3c, we analyze on varying weight λ for threat-guided gating diversification L div (Eq. 8). In gen- eral, higher weight leads to higher union robustness, verifying the importance of our gating diversification loss. Too high λ slightly degrades robustness. This is because excessively large λ creates a strong conflict between the classifica- tion objective and the diversification objective, causing the gating network to prioritize threat-guided gating diversification over task performance. Layer-adaptive coefficient s and b. As shown in Figs. A3d and A3e, we analyze on varying values of s and b that control the layer-adaptive coefficient β(l) (Eq. 5). This schedule balances global and local gating to exploit the ar- chitectural hierarchy of Transformers. As shown in Fig. A3d, union robustness peaks at s = 4, suggesting that a moderately steep transition between global and local signals is optimal. Excessively high values (e.g., s = 8) cause a drop in robustness by forcing a premature transition. Similarly as shown in Fig. A3e, the shift parameter b achieves its highest performance at b = 2, with deviations in either direction leading to suboptimal results. 8W. Kim et al. Layer 0 ` ∞ threat ` 2 threat ` 1 threat (a) First patch ` ∞ threat ` 2 threat ` 1 threat (b) Middle patch ` ∞ threat ` 2 threat ` 1 threat (c) Last patch ` ∞ threat ` 2 threat ` 1 threat (d) Global Layer 11 ` ∞ threat ` 2 threat ` 1 threat (e) First patch ` ∞ threat ` 2 threat ` 1 threat (f) Middle patch ` ∞ threat ` 2 threat ` 1 threat (g) Last patch ` ∞ threat ` 2 threat ` 1 threat (h) Global Fig. A4: t-SNE visualization of threat separability of local patch-level features at token first, middle, and last token positions and the global-level features at layer 0 (top) and layer 11 (bottom). A3.6 Additional Analysis on Dual-Scale Feature Separability In Fig. A4, we show t-SNE visualization for multiple patch-level features in addition to Fig. 5. Specifically, we visualize features from three representative patches: the first, middle, and last patches, and global feature. We also visualize feature separability at layer 0 (top) and 11 (bottom) of ViT-B. Local patch- level features consistently show difficulty separating ℓ ∞ threats, while global image-level features better distinguish ℓ ∞ threats, demonstrating the need for our dual-scale gating strategy. A4 Limitations and Future Work Although our method has shown promising results, we train it on a fixed set of threats. It thus requires fully updating the experts and gating networks when training on new threat types after deployment. Future work could explore con- tinual adaptation such as progressive expert addition [29, 54] or dynamic ex- pansion [70] to incrementally expand threat coverage for long-term deployment scenarios [12] without retraining the entire model.