Paper deep dive
Don't Waste the Noise: Importance-Guided Perturbation Allocation under Joint Global and Local Constraints
Melika Shirian, Kianoosh Vadaei
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/2/2026, 9:44:02 AM
Summary
This paper introduces an importance-guided perturbation allocation mechanism for adversarial attacks under joint global ($\ell_1$) and local magnitude constraints. The method uses a fixed clean-gradient prior to steer perturbation toward model-sensitive regions without altering the feasible perturbation set. Experiments across ten robust model-dataset configurations show that this approach improves attack success rates by 2.52 to 17.70 percentage points over matched APGD and PMA baselines, primarily by redistributing perturbation mass to high-importance regions rather than increasing the total budget.
Entities (14)
Relation Signals (14)
Importance-Guided Perturbation Allocation → evaluatedon → CIFAR-10
confidence 95% · We evaluate the proposed mechanism across ten robust model–dataset configurations spanning CIFAR-10
Importance-Guided Perturbation Allocation → evaluatedon → CIFAR-100
confidence 95% · We evaluate the proposed mechanism across ten robust model–dataset configurations spanning CIFAR-100
Importance-Guided Perturbation Allocation → evaluatedon → ImageNet
confidence 95% · We evaluate the proposed mechanism across ten robust model–dataset configurations spanning ... ImageNet.
Importance-Guided Perturbation Allocation → improves → Attack Success Rate
confidence 95% · the proposed method improves attack success over matched APGD- and PMA-based baselines by 2.52 to 17.70 percentage points.
Importance-Guided Perturbation Allocation → uses → Clean-Gradient Prior
confidence 92% · We introduce an importance-guided allocation mechanism that uses a fixed clean-gradient prior to steer perturbation toward model-sensitive regions
Importance-Guided Perturbation Allocation → outperforms → APGD
confidence 90% · improves attack success over matched APGD- and PMA-based baselines
Importance-Guided Perturbation Allocation → outperforms → PMA
confidence 90% · improves attack success over matched APGD- and PMA-based baselines
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Adversarial optimization under a shared $\ell_1$ budget requires deciding not only how much perturbation to use, but also where that limited budget should be spent. This allocation problem becomes particularly important when individual input coordinates are subject to local magnitude constraints, which restrict the extent to which perturbation can be concentrated on a small number of locations. We introduce an importance-guided allocation mechanism that uses a fixed clean-gradient prior to steer perturbation toward model-sensitive regions while leaving the feasible perturbation set unchanged. A centered allocation objective encourages perturbation at above-average importance locations and discourages unnecessary expenditure elsewhere, thereby redistributing rather than enlarging the available budget. Across ten robust model--dataset configurations under a common capacity-limited threat setting, the proposed method improves attack success over matched APGD- and PMA-based baselines by $2.52$ to $17.70$ percentage points. Allocation analysis shows that these gains are accompanied by substantially greater perturbation mass in high-importance regions without increased global $\ell_1$ consumption. Mechanism ablations further show that centered non-uniform redistribution provides part of the benefit, while model-derived importance yields an additional improvement. These results identify perturbation allocation as a distinct and practically relevant dimension of adversarial optimization under shared-budget, locally constrained threat models.
Tags
Links
- Source: https://arxiv.org/abs/2610.00861v1
- Canonical: https://arxiv.org/abs/2610.00861v1
Trouble viewing inline? Open PDF directly →
Full Text
89,914 characters extracted from source content.
Expand or collapse full text
Don’t Waste the Noise: Importance-Guided Perturbation Allocation under Joint Global and Local Constraints Melika Shirian Kianoosh Vadaei Abstract Adversarial optimization under a shared ℓ1 _1 budget requires deciding not only how much perturbation to use, but also where that limited budget should be spent. This allocation problem becomes particularly important when individual input coordinates are subject to local magnitude constraints, which restrict the extent to which perturbation can be concentrated on a small number of locations. We introduce an importance-guided allocation mechanism that uses a fixed clean-gradient prior to steer perturbation toward model-sensitive regions while leaving the feasible perturbation set unchanged. A centered allocation objective encourages perturbation at above-average importance locations and discourages unnecessary expenditure elsewhere, thereby redistributing rather than enlarging the available budget. Across ten robust model–dataset configurations under a common capacity-limited threat setting, the proposed method improves attack success over matched APGD- and PMA-based baselines by 2.522.52 to 17.7017.70 percentage points. Allocation analysis shows that these gains are accompanied by substantially greater perturbation mass in high-importance regions without increased global ℓ1 _1 consumption. Mechanism ablations further show that centered non-uniform redistribution provides part of the benefit, while model-derived importance yields an additional improvement. These results identify perturbation allocation as a distinct and practically relevant dimension of adversarial optimization under shared-budget, locally constrained threat models. 1Independent Researcher melika.shi@mehr.ui.ac.ir, k.vadaei@mehr.ui.ac.ir Figure 1: Overview of importance-guided perturbation budget allocation. The capacity-limited threat model determines which perturbations are feasible through a shared mean-L1L_1 budget, coordinate-wise capacities, and image-range bounds. Independently, a clean-gradient importance prior is normalized and centered to guide where the perturbation budget is spent. The allocation signal changes the optimization preference without changing the feasible perturbation set. Introduction Adversarial attacks under ℓ1 _1-bounded threat models have become increasingly effective at identifying vulnerabilities in image classifiers. Unlike an ℓ∞ _∞ constraint, which independently limits every input coordinate, an ℓ1 _1 constraint provides a shared perturbation budget that may be distributed unevenly across the input (Croce and Hein 2023; Pintor et al. 2021). Specialized methods such as ℓ1 _1-APGD exploit this geometry by concentrating perturbation on influential coordinates (Croce and Hein 2023), while sparse attacks similarly exploit small subsets of sensitive input components (Modas et al. 2019b; Croce and Hein 2019). This flexibility becomes more limited when individual coordinates are also subject to local magnitude constraints. Once attractive coordinates approach their local capacities, useful perturbation must be redirected elsewhere. Adversarial optimization therefore becomes not only a problem of finding a useful update direction, but also one of deciding where the available budget should be spent. This motivates our central question: When a shared ℓ1 _1 perturbation budget must be distributed among locally capacity-limited coordinates, can explicit guidance about where that budget is spent improve adversarial optimization? We study this question by separating two roles: feasibility determines which perturbations are allowed, while allocation preference determines where an optimizer prefers to spend the available budget. Figure 1 summarizes this distinction. Capacity-Limited ℓ1 _1 Optimization We consider a shared global mean-ℓ1 _1 budget together with coordinate-wise magnitude capacities and valid image-range constraints. The global constraint limits the total available perturbation, while the local capacities limit how much of that budget may be concentrated at any individual coordinate (Croce and Hein 2019). When a useful coordinate reaches its local capacity, the optimizer must distribute the remaining budget across other feasible coordinates. This creates a budget-allocation problem that is related to, but distinct from, sparse optimization. Sparse methods primarily determine which coordinates should receive large changes, whereas capacity-limited optimization must also determine how a shared budget should be distributed when individual coordinates cannot absorb arbitrarily large perturbations. The formal feasible set is defined in the Method section. All methods in our main evaluation operate under the same capacity-limited feasible set. Separately parameterized conventional and capacity-limited ℓ1 _1 results are retained as descriptive comparisons in Appendix M; because their global budget parameterizations differ, they are not used to isolate the causal effect of local capacities. Importance-Guided Budget Allocation We address the allocation problem using a fixed, model-derived importance prior. Given a clean input, we construct a spatial importance map from the model gradient (Simonyan et al. 2014), normalize it, and center it around its spatial mean. The resulting allocation reward is added to the adversarial objective to encourage perturbation magnitude to shift toward above-average importance locations. Crucially, the importance prior does not change the threat model. It does not mask low-importance coordinates, enlarge local capacities, or provide additional perturbation budget. Instead, it changes only the optimization preference within the same feasible region. This distinguishes our approach from attacks that explicitly select a sparse support or restrict perturbations to salient regions (Papernot et al. 2015a; Dong et al. 2020c). Centering is important because it prevents the allocation term from simply rewarding larger perturbations everywhere. Instead, it creates a relative preference that encourages perturbation mass to move from below-average toward above-average importance locations. The full importance construction, allocation objective, and optimization procedure are defined in the Method section. Main Findings and Contributions We evaluate the proposed mechanism across ten robust model–dataset configurations spanning CIFAR-10, CIFAR-100, and ImageNet. Under the common capacity-limited threat model, our method improves attack success over the highest-ASR matched APGD/PMA baseline in every configuration, with gains of 2.522.52–17.7017.70 percentage points. These improvements are accompanied by a systematic change in budget allocation. Matched baselines place approximately 10.610.6–11.3%11.3\% of their perturbation mass in the highest-importance 10%10\% of spatial locations, whereas our method places 19.219.2–30.4%30.4\% there while remaining within the same global mean-ℓ1 _1 budget. Mechanism ablations further separate two effects: centered non-uniform redistribution itself improves optimization, while alignment with model-derived importance provides an additional gain. Our main contributions are: • Perturbation allocation as an optimization problem. We study how a shared global mean-ℓ1 _1 budget should be distributed when individual coordinates are subject to local magnitude capacities, explicitly separating feasibility from allocation preference. • Centered importance-guided allocation. We introduce a soft, model-derived allocation objective that guides perturbation toward more informative regions without masking coordinates or changing the feasible perturbation set. • Mechanistic empirical evidence. Across ten robust model–dataset configurations, the mechanism improves matched APGD/PMA-style baselines. Allocation diagnostics and controlled ablations further distinguish the contribution of non-uniform redistribution from the additional benefit of meaningful importance alignment. We do not claim universal superiority over specialized ℓ1 _1 attacks. Adapted ℓ1 _1-APGD remains stronger in the separate reference comparisons reported in Appendix M. Rather, our results identify perturbation allocation as a distinct optimization dimension that can substantially improve matched adversarial optimizers under the capacity-limited setting studied here. Related Work Gradient-based adversarial optimization. Gradient-based attacks have advanced through improvements in optimization procedures, attack objectives, and evaluation protocols (Croce et al. 2021; Cinà et al. 2025). Projected gradient descent (PGD) became a standard approach for norm-constrained adversarial optimization (Madry et al. 2019), while methods such as C&W (Carlini and Wagner 2017) and DeepFool (Moosavi-Dezfooli et al. 2016) explored alternative objective and boundary-based formulations. Auto-PGD (APGD) further improves iterative optimization through adaptive step-size control and related stabilization mechanisms, and forms a core component of AutoAttack (Croce and Hein 2020). Probability Margin Attack (PMA) shows that attack strength can also depend strongly on the adversarial objective itself, using a probability-space margin rather than a logit-space formulation (Xie et al. 2025). Our work complements these directions by studying how a shared perturbation budget is allocated across the input under local capacity constraints. ℓ1 _1 and sparse attacks. Sparse and ℓ1 _1-oriented attacks explicitly exploit the geometry of perturbation concentration (Croce et al. 2022). EAD incorporates an ℓ1 _1 penalty through an elastic-net formulation (Chen et al. 2018), while SparseFool uses local decision-boundary geometry to construct perturbations affecting relatively few input components (Modas et al. 2019a). Croce and Hein studied sparse attacks with additional component-wise magnitude constraints (Croce and Hein 2019), and later introduced ℓ1 _1-APGD, which adaptively controls update sparsity to better exploit ℓ1 _1 geometry (Croce and Hein 2019). Fast Minimum-Norm (FMN) attacks similarly support ℓ1 _1 optimization by adapting the norm constraint while searching for small adversarial perturbations (Pintor et al. 2021). Our setting is closely related because it combines a shared ℓ1 _1-type budget with coordinate-wise magnitude limits. We do not treat these local capacity constraints themselves as the novelty of the work. Instead, we study the optimization problem that arises within such a feasible set: how the shared budget should be distributed when individual coordinates cannot absorb arbitrarily large perturbations. Importance-guided and structured perturbations. Prior attacks have used saliency or spatial structure to influence where perturbations are applied. JSMA constructs a saliency map to identify influential input features (Papernot et al. 2015b); GreedyFool combines gradient information with a distortion map to select candidate pixels for sparse modification (Dong et al. 2020a); and superpixel-guided or saliency-based attacks restrict or prioritize perturbations in informative spatial regions (Dong et al. 2020b; Dai et al. 2022). More generally, gradient-based saliency provides a simple estimate of local model sensitivity (Simonyan et al. 2014). Structured adversarial attacks additionally impose spatial, group-sparse, or related regularities on the perturbation (Xu et al. 2019). Our mechanism differs from hard support selection or importance-dependent constraints. The importance map does not determine which coordinates are feasible and does not increase the admissible magnitude of high-importance locations. Instead, it enters the attack objective as a soft, centered preference that encourages perturbation mass to be redistributed toward more important regions while retaining access to the same feasible set. This positioning makes the contribution narrower than introducing a new saliency attack or a new capacity constraint: we study perturbation allocation as a distinct optimization component under a shared-budget, locally constrained threat model. Importance-Guided Perturbation Budget Allocation We consider adversarial optimization under a shared global perturbation budget and coordinate-wise magnitude constraints. The feasible set specifies which perturbations are admissible, while our allocation objective provides a soft preference over where the available budget should be spent. Model-derived importance affects only this optimization preference and does not modify the admissible perturbation region. Threat Model and Local Capacities Let x∈[0,1]Dx∈[0,1]^D be a correctly classified input with ground-truth label y, where D denotes the number of scalar input coordinates. We seek an adversarial perturbation δ satisfying (x,E,b)=δ:x+δ∈[0,1]D,|δi|≤Ei,‖δ‖1D≤b,C(x,E,b)= \δ:x+δ∈[0,1]^D,\;| _i|≤ E_i,\; \|δ\|_1D≤ b \, (1) where b is a global mean-ℓ1 _1 budget and EiE_i is the local magnitude capacity of coordinate i. The global constraint couples all coordinates through a shared perturbation budget, while the local capacities limit how much of that budget may be absorbed by any individual coordinate. Thus, once locally attractive coordinates approach their capacities, the optimizer must distribute useful perturbation over other feasible coordinates. Heterogeneous local capacities. In the main experiments, capacities are generated spatially as Ep=μE+σEΦ−1(Up),Up∼(umin,umax),E_p= _E+ _E ^-1(U_p), U_p (u_ ,u_ ), (2) where p indexes spatial locations and Φ−1 ^-1 is the inverse CDF of the standard normal distribution. We use umin=10−3u_ =10^-3 and umax=1−10−3u_ =1-10^-3 to avoid extreme tails, and share the sampled capacity across the RGB channels of each spatial location. The capacity map is generated independently of model-derived importance: E specifies how much perturbation is feasible at a location, not whether that location should be preferred by the optimizer. All matched methods use the same realization of E and the same global budget b. Further construction details are provided in Appendix C. Clean-Gradient Importance Prior We construct a fixed spatial importance prior from the clean input. Let zy(x)z_y(x) denote the logit of the ground-truth class. For spatial location p with C input channels, we define sp=1C∑c=1C|∂zy(x)∂xc,p|.s_p= 1C _c=1^C | ∂ z_y(x)∂ x_c,p |. (3) These scores provide a local estimate of model sensitivity (Simonyan et al. 2014). To reduce the influence of isolated gradient spikes, we clip them at an upper quantile q: s^p=minsp,Qq(s), s_p= \s_p,\,Q_q(s) \, (4) where Qq(s)Q_q(s) is the q-th quantile of the spatial scores. We then normalize by the spatial mean, up=s^pmean(s^)+ϵ,u_p= s_pmean( s)+ε, (5) and construct a bounded importance map Mp=clip(aupγ,Mmin,Mmax),M_p=clip (au_p^γ,M_ ,M_ ), (6) where γ controls the sharpness of the importance distribution and a>0a>0 is chosen so that mean(M)=1.mean(M)=1. (7) The resulting map is shared across channels at each spatial location, computed once from the clean input, and held fixed throughout the attack. It is therefore an allocation prior rather than a dynamically updated attack trajectory. Complete importance-map hyperparameters are reported in Appendix F. Centered Allocation Objective We incorporate the importance prior through a soft allocation reward. The attack objective is Jours(δ)=Jadv(x+δ,y)+λRalloc(δ,M),J_ours(δ)=J_adv(x+δ,y)+λ R_alloc(δ,M), (8) where JadvJ_adv is the adversarial classification objective and λ controls the strength of allocation guidance. Because M is normalized to have unit spatial mean, Mi−1M_i-1 distinguishes above-average from below-average importance. We define Ralloc(δ,M)=1Db∑i=1D(Mi−1)δi2+τ2,R_alloc(δ,M)= 1Db _i=1^D(M_i-1) _i^2+τ^2, (9) where δi2+τ2 _i^2+τ^2 is a smooth approximation to |δi|| _i| and τ>0τ>0 is a small smoothing constant. For Mi>1M_i>1, increasing perturbation magnitude contributes positively to the reward; for Mi<1M_i<1, it is penalized. The normalization by DbDb scales the allocation term relative to the available global perturbation budget. Centering is essential to the intended mechanism. Using MiM_i directly would assign a positive magnitude reward at every coordinate, whereas Mi−1M_i-1 creates a relative preference that encourages perturbation mass to shift from below-average toward above-average importance locations. Importantly, RallocR_alloc does not alter (x,E,b)C(x,E,b). Low-importance coordinates remain feasible, high-importance coordinates receive no additional capacity, and no hard support is imposed. The mechanism therefore changes how the optimizer navigates the feasible set rather than changing the set itself. Attack Optimization Starting from a feasible initialization δ0 _0, we directly optimize Equation 8. At iteration t, we compute gt=∇δJours(δt),g_t= _δJ_ours( _t), (10) normalize the gradient by its mean absolute magnitude, and accumulate momentum following momentum-based adversarial optimization (Dong et al. 2018). The perturbation is updated in the sign direction of the accumulated gradient, with a cosine-decayed update magnitude (Loshchilov and Hutter 2017). After each update, feasibility is restored using the same constraint operator employed for the matched attacks. We sequentially 1. clip the perturbation to the coordinate-wise capacities E; 2. enforce the valid image range x+δ∈[0,1]Dx+δ∈[0,1]^D; and 3. project onto the global ℓ1 _1 ball of radius DbDb. The final step uses the standard Euclidean projection onto an ℓ1 _1 ball (Duchi et al. 2008). Initialization, step-size, momentum, restart, and projection details are provided in Appendix B. The corresponding exact-projection control is reported in Appendix B.4. Candidate Selection We retain the strongest feasible candidate encountered during optimization rather than returning only the final iterate. Candidates are ranked by the probability margin (Xie et al. 2025), mprob(x+δ)=maxk≠ypk(x+δ)−py(x+δ),m_prob(x+δ)= _k≠ yp_k(x+δ)-p_y(x+δ), (11) where pk(x+δ)p_k(x+δ) is the predicted probability of class k. The feasible candidate with the largest margin is returned. This criterion is used only for candidate selection and does not replace the optimization objective in Equation 8. Unless explicitly varied in the objective ablations, the attack trajectory is determined by JadvJ_adv together with the centered allocation reward. The resulting construction keeps the two quantities central to our analysis separate: E determines local feasibility, while M supplies an independent preference over budget allocation. The mechanism ablations in the Results and Appendix H test the roles of centering, non-uniform redistribution, and importance quality separately. Figure 2: Qualitative example of importance-aware perturbation allocation. The clean-gradient importance map provides a fixed spatial prior, while the perturbation-magnitude maps show how different mechanism variants distribute perturbation under identical constraints. Ours exhibits stronger alignment with regions emphasized by the importance prior, whereas removing, randomizing, or reversing the guidance changes the resulting allocation pattern. Experimental Setup Our main evaluation compares attacks within a common capacity-limited feasible set. For each model and evaluation sample, all matched methods use the same global perturbation budget, local-capacity realization, image-range constraints, and evaluation population. A separate comparison between conventional and capacity-limited ℓ1 _1 settings is reported descriptively in Appendix M; because those settings use different global budget parameterizations, it is not used to isolate the effect of local capacities. Datasets and Models We evaluate on CIFAR-10, CIFAR-100 (Krizhevsky 2009), and ImageNet (Deng et al. 2009), using ten robust model–dataset configurations spanning convolutional and transformer-based architectures. On CIFAR-10, we evaluate WRN-94-16 from Bartoldson et al. (Bartoldson et al. 2024), WRN-28-10 from Cui et al. (Cui et al. 2024), and XCiT-S12 from Debenedetti et al. (Debenedetti et al. 2023). On CIFAR-100, we evaluate WRN-28-10 (Cui et al. 2024), XCiT-S12 (Debenedetti et al. 2023), and WRN-70-16 from Wang et al. (Wang et al. 2023). On ImageNet, we evaluate ConvNeXt-L (Liu et al. 2023), the robust model of Peng et al. (Peng et al. 2023), ViT-B with a convolutional stem (Singh et al. 2023), and Swin-L from MIMIR (Xu et al. 2025). Table 1 reports the attacked population for each configuration. All methods compared within a row operate on exactly the same inputs and labels. Table 1: Evaluation panel. N is the number of clean-correct inputs attacked in each configuration; all methods within a row use the same evaluation population. Dataset Model N CIFAR-10 WRN-94-16 9,368 CIFAR-10 WRN-28-10 7,500 CIFAR-10 XCiT-S12 9,006 CIFAR-100 WRN-28-10 7,383 CIFAR-100 XCiT-S12 6,734 CIFAR-100 WRN-70-16 7,522 ImageNet ConvNeXt-L 1,000 ImageNet Peng et al. 1,000 ImageNet ViT-B-ConvStem 1,000 ImageNet Swin-L 1,000 For ImageNet, we use 1,000 clean-correct validation images selected in a fixed deterministic order for each model. Complete checkpoint identifiers, preprocessing details, clean-correct population sizes, and sampling information are provided in Appendix E. Capacity-Limited Threat Setting All main comparisons use the feasible set defined in Equation 1. We set the global mean-ℓ1 _1 budget to b=4/255b=4/255. The heterogeneous capacity map follows Equation 2 with μE=12/255 _E=12/255 and σE=2/255 _E=2/255, with each spatial capacity shared across RGB channels. For a given sample, all matched methods receive the same capacity realization and enforce the same global mean-ℓ1 _1, coordinate-wise, and image-range constraints. Where direct initialization matching is applicable, they also use the same initial perturbation. The capacity map is generated independently of the importance prior, so model-derived importance affects optimization preference rather than local feasibility. Capacity construction and constraint-validity checks are reported in Appendix C and Appendix N, respectively. Matched Baselines We compare against three gradient-based baselines adapted to the same capacity-limited feasible set: • APGD-CE, using cross-entropy; • APGD-DLR, using the difference-of-logits-ratio objective (Croce and Hein 2020); and • PMA, using the probability-margin objective (Xie et al. 2025). The adaptations enforce the common constraints while retaining each baseline’s characteristic optimization procedure or objective. Our reference attack uses untargeted cross-entropy augmented with the centered allocation reward in Equation 9. Unless otherwise stated, all attacks use 50 optimization steps and one restart. Full adaptation details and attack hyperparameters are provided in Appendix D and Appendix F. These baselines are used specifically to test allocation guidance under a matched feasible set. Results for specialized ℓ1 _1-APGD and FMN-ℓ1 _1 are reported separately in Appendix M and are not part of the universal baseline comparison. Allocation-Weight Selection We select the allocation strength λ independently for each dataset using 100 held-out clean-correct samples. The candidate set is 0.10,0.20,0.35,0.50,0.75\0.10,0.20,0.35,0.50,0.75\, with ASR used as the selection criterion. This yields λ=0.5λ=0.5 for CIFAR-10, λ=0.5λ=0.5 for CIFAR-100, and λ=0.2λ=0.2 for ImageNet. The selected value is then frozen across models and reported evaluations within that dataset. Complete selection and sensitivity results are given in Appendix G. Evaluation Metrics Our primary effectiveness metric is conditional attack success rate (ASR), computed over inputs correctly classified by the corresponding clean model. To measure perturbation allocation, let α(M)S_α(M) denote the top α fraction of spatial locations ranked by the clean-gradient importance map. We define Massα(δ,M)=∑i∈α(M)|δi|∑i=1D|δi|,Mass_α(δ,M)= _i _α(M)| _i| _i=1^D| _i|, (12) where the spatial ranking is shared across the corresponding RGB coordinates. Our primary allocation statistic uses α=0.10α=0.10. We additionally report cumulative importance-ranked perturbation mass and the realized global magnitude 255‖δ‖1/D255\|δ\|_1/D to verify that differences in attack effectiveness are not explained by greater consumption of the global ℓ1 _1 budget. Complementary allocation statistics are reported in Appendix I, while norm and perceptual diagnostics are reported in Appendix J. Paired Statistical Analysis Because methods within each configuration attack the same clean-correct inputs, performance comparisons are paired. We report paired ASR differences with 95% confidence intervals and evaluate binary success differences using McNemar’s test (McNemar 1947). Multiple comparisons are corrected using Holm’s procedure (Holm 1979). The main text emphasizes effect sizes and confidence intervals; complete pairwise statistics and corrected p-values are reported in Appendix L. Computational Accounting and Reproducibility We measure computational cost using wall-clock runtime and forward/backward model evaluations. Constructing the clean-gradient importance prior requires one additional backward evaluation before iterative optimization. A compute-matched control tests whether this additional evaluation explains the observed gain. Full timing and evaluation counts are reported in Appendix K. Hyperparameters, projection details, and constraint-validity checks are reported in Appendix F, Appendix B, and Appendix N. Results We evaluate whether explicit allocation guidance improves optimization within the common capacity-limited threat model, then examine how the perturbation budget is redistributed and which components of the mechanism account for the observed gains. All main comparisons in this section refer to the matched APGD/PMA-style baselines defined in the Experimental Setup; specialized ℓ1 _1 reference attacks are reported separately in Appendix M. Main Performance under Capacity Limits Table 2 reports conditional attack success rate (ASR) across the ten model–dataset configurations. Table 2: Conditional ASR (%) under the common capacity-limited threat model. Δbest _best is Ours minus the highest-ASR matched APGD/PMA baseline in the same configuration. Dataset Model N APGD-CE APGD-DLR PMA Ours Δbest _best CIFAR-10 WRN-94-16 9,368 8.64 10.28 9.84 17.98 +7.70 CIFAR-10 WRN-28-10 7,500 10.81 12.80 12.47 20.25 +7.45 CIFAR-10 XCiT-S12 9,006 15.08 17.03 17.03 22.52 +5.49 CIFAR-100 WRN-28-10 7,383 22.54 28.44 27.59 35.41 +6.97 CIFAR-100 XCiT-S12 6,734 25.10 30.62 30.83 33.35 +2.52 CIFAR-100 WRN-70-16 7,522 20.15 25.38 24.82 32.53 +7.15 ImageNet ConvNeXt-L 1,000 24.90 24.70 27.00 42.90 +15.90 ImageNet Peng et al. 1,000 35.10 34.80 37.20 54.90 +17.70 ImageNet ViT-B-ConvStem 1,000 31.00 30.80 34.10 44.00 +9.90 ImageNet Swin-L 1,000 25.80 27.30 28.20 36.00 +7.80 Ours achieves higher ASR than all three matched baselines in every configuration. Relative to the highest-ASR matched baseline in each row, the improvement ranges from 2.522.52 percentage points on CIFAR-100/XCiT-S12 to 17.7017.70 points on ImageNet/Peng et al. The pattern is not restricted to one dataset or architecture family. Dataset-level macro-averages for Ours are 20.25%20.25\%, 33.76%33.76\%, and 44.45%44.45\% on CIFAR-10, CIFAR-100, and ImageNet, respectively. The corresponding strongest baseline macro-averages are 13.37%13.37\%, 28.15%28.15\%, and 31.63%31.63\%. Figure 3 provides a compact view of the paired effect in each configuration. All ten differences are positive and their 95% paired confidence intervals remain above zero. Full pairwise statistics and corrected McNemar tests are reported in Appendix L. Figure 3: Paired ASR improvement over the highest-ASR matched APGD/PMA baseline in each configuration. Horizontal bars show 95% paired confidence intervals; full statistics are reported in Appendix L. Where Does the Perturbation Budget Go? The performance gain is accompanied by a clear change in where the shared budget is spent. Table 3 reports the fraction of total absolute perturbation mass placed in the highest-importance 10%10\% of spatial locations. Table 3: Perturbation mass (%) in the highest-importance 10% of spatial locations. Uniform spatial allocation corresponds to approximately 10%10\%. Dataset Model APGD-CE APGD-DLR PMA Ours CIFAR-10 WRN-94-16 11.12 10.84 11.05 30.36 CIFAR-10 WRN-28-10 11.09 10.83 11.05 29.14 CIFAR-10 XCiT-S12 10.72 10.61 10.69 25.19 CIFAR-100 WRN-28-10 11.31 11.08 11.17 24.96 CIFAR-100 XCiT-S12 11.03 11.02 11.17 22.77 CIFAR-100 WRN-70-16 11.20 11.08 11.13 26.22 ImageNet ConvNeXt-L 10.81 10.72 10.66 21.60 ImageNet Peng et al. 11.04 10.72 10.74 20.83 ImageNet ViT-B-ConvStem 10.61 10.56 10.56 19.41 ImageNet Swin-L 11.08 10.76 10.91 19.15 Matched baselines place approximately 10.610.6–11.3%11.3\% of their perturbation mass in the top-10%10\% importance region, close to uniform spatial allocation. Ours increases this share to 19.1519.15–30.36%30.36\%. The dataset-level averages are 28.23%28.23\% on CIFAR-10, 24.65%24.65\% on CIFAR-100, and 20.25%20.25\% on ImageNet. This redistribution does not arise from greater global budget consumption: the realized 255‖δ‖1/D255\|δ\|_1/D averages 3.913.91, 3.963.96, and 3.973.97 on CIFAR-10, CIFAR-100, and ImageNet, respectively, under the shared budget of 4.04.0. Figure 4 shows that the effect is not specific to the Top-10 threshold. Perturbation mass accumulates more rapidly as high-importance locations are included, indicating a systematic shift in allocation rather than a threshold-specific statistic. Complementary importance–perturbation correlation, effective-support, spatial-Gini, and Top-1/5/20 diagnostics are reported in Appendix I. Figure 4: Cumulative perturbation mass over importance-ranked locations. Locations are ordered from highest to lowest clean-gradient importance; the dashed diagonal denotes uniform allocation. The same behavior is visible qualitatively in Figure 2. Under identical constraints, changing or removing the allocation prior changes where perturbation magnitude is concentrated relative to the fixed clean-gradient importance map. Mechanism Ablations We next isolate the roles of centering, non-uniform redistribution, and the quality of the importance prior. Table 4 compares the full method with variants that remove, uncenter, randomize, invert, or smooth the allocation signal. Table 4: Mechanism ablations, conditional ASR (%). RandomMask reports the mean over five spatial permutations. Variant CIFAR-10 CIFAR-100 ImageNet NoMask 10.2 23.9 24.0 Uncentered 3.9 13.7 20.4 RandomMask 14.2 28.2 29.2 InvertedMask 14.5 25.9 23.4 SmoothMask 17.6 34.4 36.2 Ours 19.6 37.9 36.8 Removing the allocation reward reduces ASR by 9.49.4, 14.014.0, and 12.812.8 percentage points on CIFAR-10, CIFAR-100, and ImageNet, respectively. The uncentered variant performs below even the no-allocation control on all three datasets, supporting the reward–penalty role of centering rather than a globally positive magnitude reward. The randomized prior reveals a separate redistribution effect. Despite containing no meaningful model-derived spatial ordering, it improves over NoMask from 10.2%10.2\% to 14.2%14.2\% on CIFAR-10, 23.9%23.9\% to 28.2%28.2\% on CIFAR-100, and 24.0%24.0\% to 29.2%29.2\% on ImageNet. The effect is stable across the five independent spatial permutations reported in Appendix H. Replacing this randomized preference with the model-derived prior provides a further gain, reaching 19.6%19.6\%, 37.9%37.9\%, and 36.8%36.8\%. The ablations therefore support two complementary effects: centered non-uniform redistribution can assist optimization, while importance alignment provides additional benefit. Inverting the prior weakens performance relative to Ours, while smoothing preserves most of the gain. Thus, the mechanism does not appear to depend only on isolated gradient spikes. Figure 5 connects these interventions to their allocation behavior. Figure 5: Mechanism ablation: allocation versus attack effect. Randomized guidance captures part of the benefit of non-uniform redistribution, while the model-derived prior provides an additional gain. Allocation Strength and Sensitivity The allocation weight λ trades off importance-aligned concentration against optimization flexibility. On the CIFAR-10 reference sweep, ASR rises from 10.2%10.2\% at λ=0λ=0 to a maximum of 20.6%20.6\% at λ=0.35λ=0.35, then decreases to 14.7%14.7\% at λ=1.0λ=1.0, even as importance concentration continues to increase. The ImageNet diagnostic is also non-monotonic, and the value selected on the independent calibration set is not the value that maximizes ASR on the reported 500-image diagnostic population. We therefore retain the preselected dataset-specific values rather than retuning on the evaluation runs. These results show that stronger concentration is not itself the objective: the allocation term is most useful when it guides, rather than dominates, the adversarial optimization. The complete CIFAR-10 and ImageNet sensitivity curves and selection protocol are reported in Appendix G. Optimization Controls Several controls test alternative explanations for the observed gain. First, the effect is not specific to cross-entropy. Adding the centered allocation mechanism improves the corresponding unguided attack for all four evaluated objectives—cross-entropy, margin, DLR, and probability margin—on all three reference datasets. Moreover, the strongest guided objective differs across datasets: margin reaches 23.1%23.1\% on CIFAR-10, DLR reaches 42.0%42.0\% on CIFAR-100, and probability margin reaches 43.8%43.8\% on ImageNet. This indicates that allocation guidance is complementary to the choice of adversarial loss rather than specific to the reference cross-entropy objective. Second, compute matching does not explain the result. Replacing one of the 50 iterative attack steps with the clean-gradient computation yields the same reference ASRs as the 50+150+1 configuration: 19.6%19.6\%, 37.9%37.9\%, and 36.8%36.8\% on CIFAR-10, CIFAR-100, and ImageNet. Third, replacing the sequential feasibility operator with the exact projection changes ASR only marginally; full results are reported in Appendix B.4. Finally, on the CIFAR-10 and ImageNet reference controls, replacing the heterogeneous capacity envelope with a uniform 12/25512/255 cap produces the same ASR as the heterogeneous setting. This suggests that the observed allocation gain in these controls is not specific to the sampled capacity heterogeneity. Capacity-structure details are reported in Appendix C, while the remaining mechanism controls are reported in Appendix H. Computational Cost and Perturbation Diagnostics Our method requires one additional backward evaluation to construct the clean-gradient prior: 5151 backward evaluations versus 5050 for the matched baselines. It uses 103103 forward evaluations, compared with 102102 for PMA and 5353 for the APGD variants. Recorded wall-clock runtime is consequently close to PMA, while APGD remains cheaper in forward-evaluation count. Complete timing results are reported in Appendix K. The stronger allocation concentration also changes the local perturbation profile. Although Ours does not consume a larger global mean-ℓ1 _1 budget, it exhibits larger local magnitudes and greater perceptual distortion than the more diffuse matched baselines. The same pattern is observed in paired perceptual comparisons restricted to common-success examples. We therefore interpret the performance gain as an allocation effect, not as a simultaneous improvement in perceptual quality. Full ℓ1 _1, ℓ∞ _∞, high-quantile magnitude, LPIPS, DISTS, and SSIM diagnostics are reported in Appendix J. Scope and Limitations Our results support a specific claim about perturbation allocation within the evaluated capacity-limited threat model. The main comparisons use matched APGD- and PMA-based baselines under the same global budget, local capacities, image-range constraints, and evaluation samples. Within this setting, importance-guided allocation improves attack success across all ten model–dataset configurations. We do not claim universal superiority over specialized ℓ1 _1 attacks: adapted ℓ1 _1-APGD remains stronger in the reference comparisons in Appendix M. The conventional and capacity-limited results in Appendix M are descriptive rather than causal because the settings differ in coordinate-wise capacities and global budget parameterization. Our threat model represents one family of locally constrained ℓ1 _1 settings. Although a uniform-capacity control gives similar results, other capacity structures may behave differently. Our allocation prior is a fixed clean-gradient importance map computed once before the attack, so the study does not establish clean gradients as the optimal allocation signal. Dynamic, learned, or alternative estimators remain possible extensions. The main experiments use 50 optimization steps and one restart, and do not exhaust substantially larger compute budgets. Finally, λ is selected per dataset on a held-out calibration set and then frozen across models. Stronger importance concentration does not monotonically improve attack success, so the selected values are practical operating points rather than globally optimal settings. The method can also increase local perturbation magnitude and perceptual distortion while remaining within the same global mean-ℓ1 _1 budget; corresponding diagnostics are reported in Appendix J. Overall, explicit allocation guidance can materially improve matched adversarial optimization under the evaluated shared-budget, locally constrained setting, without establishing universal superiority over specialized ℓ1 _1 attacks or the causal effect of local capacities across separately parameterized threat regimes. Conclusion We studied adversarial optimization under a shared global mean-ℓ1 _1 budget with coordinate-wise capacity constraints, focusing on the allocation of perturbation within a fixed feasible set. We introduced a centered importance-guided allocation objective that uses a fixed model-derived prior to steer perturbation toward more informative regions without changing the admissible perturbation set. Across ten robust model–dataset configurations, this mechanism improves attack success over matched APGD- and PMA-based baselines under the same capacity-limited constraints. Allocation diagnostics and ablations indicate that both centered non-uniform redistribution and alignment with model-derived importance contribute to the gain. These results identify perturbation allocation as a distinct optimization dimension under shared-budget, locally constrained threat models. They do not imply universal superiority over specialized ℓ1 _1 attacks, but suggest that explicit allocation guidance can be a useful component of future adversarial optimizers. References Bartoldson et al. (2024) B. R. Bartoldson, J. Diffenderfer, K. Parasyris, and B. Kailkhura Adversarial robustness limits via scaling-law and human-alignment studies. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 3046–3072. Cited by: Datasets and Models. Carlini and Wagner (2017) N. Carlini and D. Wagner Towards evaluating the robustness of neural networks. External Links: 1608.04644, Link Cited by: Gradient-based adversarial optimization.. Chen et al. (2018) P. Chen, Y. Sharma, H. Zhang, J. Yi, and C. Hsieh EAD: elastic-net attacks to deep neural networks via adversarial examples. External Links: 1709.04114, Link Cited by: ℓ1 _1 and sparse attacks.. Cinà et al. (2025) A. E. Cinà, J. Rony, M. Pintor, L. Demetrio, A. Demontis, B. Biggio, I. B. Ayed, and F. Roli AttackBench: evaluating gradient-based attacks for adversarial examples. Proceedings of the AAAI Conference on Artificial Intelligence 39 (3), p. 2600–2608. External Links: ISSN 2159-5399, Link, Document Cited by: Gradient-based adversarial optimization.. Croce et al. (2021) F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein RobustBench: a standardized adversarial robustness benchmark. External Links: 2010.09670, Link Cited by: §D.4, Gradient-based adversarial optimization.. Croce et al. (2022) F. Croce, M. Andriushchenko, N. D. Singh, N. Flammarion, and M. Hein Sparse-rs: a versatile framework for query-efficient sparse black-box adversarial attacks. External Links: 2006.12834, Link Cited by: ℓ1 _1 and sparse attacks.. Croce and Hein (2019) F. Croce and M. Hein Sparse and imperceivable adversarial attacks. External Links: 1909.05040, Link Cited by: §A.3, Capacity-Limited ℓ1 _1 Optimization, Introduction, ℓ1 _1 and sparse attacks.. Croce and Hein (2020) F. Croce and M. Hein Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. External Links: 2003.01690, Link Cited by: §D.1, Gradient-based adversarial optimization., 2nd item. Croce and Hein (2023) F. Croce and M. Hein Mind the box: l1l_1-apgd for sparse adversarial attacks on image classifiers. External Links: 2103.01208, Link Cited by: Introduction. Cui et al. (2024) J. Cui, Z. Tian, Z. Zhong, X. Qi, B. Yu, and H. Zhang Decoupled kullback-leibler divergence loss. External Links: 2305.13948, Link Cited by: Datasets and Models. Dai et al. (2022) Z. Dai, S. Liu, K. Tang, and Q. Li Saliency attack: towards imperceptible black-box adversarial attack. External Links: 2206.01898, Link Cited by: Importance-guided and structured perturbations.. Debenedetti et al. (2023) E. Debenedetti, V. Sehwag, and P. Mittal A light recipe to train robust vision transformers. External Links: 2209.07399, Link Cited by: Datasets and Models. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, Vol. , p. 248–255. External Links: Document Cited by: Datasets and Models. Dong et al. (2020a) X. Dong, D. Chen, J. Bao, C. Qin, L. Yuan, W. Zhang, N. Yu, and D. Chen GreedyFool: distortion-aware sparse adversarial attack. External Links: 2010.13773, Link Cited by: Importance-guided and structured perturbations.. Dong et al. (2020b) X. Dong, J. Han, D. Chen, J. Liu, H. Bian, Z. Ma, H. Li, X. Wang, W. Zhang, and N. Yu Robust superpixel-guided attentional adversarial attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Importance-guided and structured perturbations.. Dong et al. (2020c) X. Dong, J. Han, D. Chen, J. Liu, H. Bian, Z. Ma, H. Li, X. Wang, W. Zhang, and N. Yu Robust superpixel-guided attentional adversarial attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Importance-Guided Budget Allocation. Dong et al. (2018) Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li Boosting adversarial attacks with momentum. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 9185–9193. Cited by: Attack Optimization. Duchi et al. (2008) J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra Efficient projections onto the l1-ball for learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, New York, NY, USA, p. 272–279. External Links: ISBN 9781605582054, Link, Document Cited by: §B.3, Attack Optimization. Holm (1979) S. Holm A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6 (2), p. 65–70. Cited by: Appendix L, Paired Statistical Analysis. Krizhevsky (2009) A. Krizhevsky Learning multiple layers of features from tiny images. External Links: Link Cited by: Datasets and Models. Liu et al. (2023) C. Liu, Y. Dong, W. Xiang, X. Yang, H. Su, J. Zhu, Y. Chen, Y. He, H. Xue, and S. Zheng A comprehensive study on robustness of image classification models: benchmarking and rethinking. External Links: 2302.14301, Link Cited by: Datasets and Models. Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter SGDR: stochastic gradient descent with warm restarts. External Links: 1608.03983, Link Cited by: Attack Optimization. Madry et al. (2019) A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu Towards deep learning models resistant to adversarial attacks. External Links: 1706.06083, Link Cited by: Gradient-based adversarial optimization.. McNemar (1947) Q. McNemar Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12 (2), p. 153–157. External Links: Document Cited by: Appendix L, Paired Statistical Analysis. Modas et al. (2019a) A. Modas, S. Moosavi-Dezfooli, and P. Frossard SparseFool: a few pixels make a big difference. External Links: 1811.02248, Link Cited by: ℓ1 _1 and sparse attacks.. Modas et al. (2019b) A. Modas, S. Moosavi-Dezfooli, and P. Frossard SparseFool: a few pixels make a big difference. External Links: 1811.02248, Link Cited by: Introduction. Moosavi-Dezfooli et al. (2016) S. Moosavi-Dezfooli, A. Fawzi, and P. Frossard DeepFool: a simple and accurate method to fool deep neural networks. External Links: 1511.04599, Link Cited by: Gradient-based adversarial optimization.. Papernot et al. (2015a) N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami The limitations of deep learning in adversarial settings. External Links: 1511.07528, Link Cited by: Importance-Guided Budget Allocation. Papernot et al. (2015b) N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami The limitations of deep learning in adversarial settings. External Links: 1511.07528, Link Cited by: Importance-guided and structured perturbations.. Peng et al. (2023) S. Peng, W. Xu, C. Cornelius, M. Hull, K. Li, R. Duggal, M. Phute, J. Martin, and D. H. Chau Robust principles: architectural design principles for adversarially robust CNNs. In British Machine Vision Conference, Cited by: Datasets and Models. Pintor et al. (2021) M. Pintor, F. Roli, W. Brendel, and B. Biggio Fast minimum-norm adversarial attacks through adaptive norm constraints. External Links: 2102.12827, Link Cited by: §D.4, Introduction, ℓ1 _1 and sparse attacks.. Simonyan et al. (2014) K. Simonyan, A. Vedaldi, and A. Zisserman Deep inside convolutional networks: visualising image classification models and saliency maps. External Links: 1312.6034, Link Cited by: Importance-Guided Budget Allocation, Importance-guided and structured perturbations., Clean-Gradient Importance Prior. Singh et al. (2023) N. D. Singh, F. Croce, and M. Hein Revisiting adversarial training for imagenet: architectures, training and generalization across threat models. External Links: 2303.01870, Link Cited by: Datasets and Models. Wang et al. (2023) Z. Wang, T. Pang, C. Du, M. Lin, W. Liu, and S. Yan Better diffusion models further improve adversarial training. External Links: 2302.04638, Link Cited by: Datasets and Models. Xie et al. (2025) Y. Xie, W. Zheng, H. Huang, G. Ye, and X. Ma Towards million-scale adversarial robustness evaluation with stronger individual attacks. External Links: 2411.15210, Link Cited by: §D.2, Gradient-based adversarial optimization., Candidate Selection, 3rd item. Xu et al. (2019) K. Xu, S. Liu, P. Zhao, P. Chen, H. Zhang, Q. Fan, D. Erdogmus, Y. Wang, and X. Lin Structured adversarial attack: towards general implementation and better interpretability. External Links: 1808.01664, Link Cited by: Importance-guided and structured perturbations.. Xu et al. (2025) X. Xu, S. Yu, Z. Liu, and S. Picek MIMIR: masked image modeling for mutual information-based adversarial robustness. External Links: 2312.04960, Link Cited by: Datasets and Models. Appendix A Full Threat-Model Details This appendix specifies the threat model used throughout the main capacity-limited evaluation and clarifies the roles of the global ℓ1 _1 budget, local coordinate-wise capacities, image-range constraints, and model-derived importance. The central distinction is that the threat model defines which perturbations are feasible, whereas the importance mechanism introduced in the Method section affects only which feasible perturbations are preferred by the optimizer. A.1 Capacity-Limited Feasible Set Let x∈[0,1]Dx∈[0,1]^D denote a clean input that is correctly classified by the evaluated model, and let δ∈ℝDδ ^D denote an additive adversarial perturbation. The capacity-limited feasible set is (x,E,b)=δ:x+δ∈[0,1]D,|δi|≤Ei,‖δ‖1D≤b,C(x,E,b)= \δ:x+δ∈[0,1]^D,\;| _i|≤ E_i,\; \|δ\|_1D≤ b \, (13) where: • b is the global mean-ℓ1 _1 perturbation budget; • EiE_i is the maximum admissible perturbation magnitude at coordinate i; and • x+δ∈[0,1]Dx+δ∈[0,1]^D enforces the valid image range. The main experiments use b=4255.b= 4255. (14) The same feasible-set definition is used for all matched methods in the main evaluation. A.2 Global Mean-ℓ1 _1 Budget We parameterize the global constraint using the mean absolute perturbation per scalar input coordinate, ‖δ‖1D≤b. \|δ\|_1D≤ b. (15) Equivalently, the corresponding total ℓ1 _1 radius is ‖δ‖1≤Db.\|δ\|_1≤ Db. (16) Using a mean-ℓ1 _1 parameterization makes the nominal budget independent of the number of scalar input coordinates. Consequently, two datasets with different image resolutions may have different equivalent total ℓ1 _1 radii even when they use the same value of b. For an RGB image of height H and width W, D=3HW,D=3HW, (17) and therefore ϵ1total=3HWb. _1^total=3HWb. (18) Throughout the main capacity-limited experiments, comparisons are made between attacks operating on the same images and under the same value of b. Differences in attack success therefore cannot arise from one matched method receiving a larger global mean-ℓ1 _1 budget. A.3 Local Coordinate-Wise Capacities The local constraint |δi|≤Ei| _i|≤ E_i (19) limits the amount of perturbation that may be assigned to an individual coordinate. This restriction is related to previously studied sparse and component-wise constrained threat models (Croce and Hein 2019), but in our formulation the capacity values do not encode model importance. The main experiments use a spatial capacity map with nominal parameters μE=12255,σE=2255. _E= 12255, _E= 2255. (20) A single capacity value is generated for each spatial location and shared across its RGB channels. Thus, if p denotes a spatial location, E1,p=E2,p=E3,p=Ep.E_1,p=E_2,p=E_3,p=E_p. (21) The exact construction of EpE_p, including the truncated inverse-CDF sampling procedure, is given in Appendix C. A.4 Interaction Between Global and Local Constraints The global and local constraints play different roles. The global mean-ℓ1 _1 constraint limits the total amount of perturbation that may be distributed over the image, whereas EiE_i limits how much of that budget may be placed at any one coordinate. Without local capacities, a large fraction of the shared ℓ1 _1 budget may in principle be concentrated on a comparatively small number of coordinates. Under Equation 13, this is possible only until those coordinates reach their corresponding capacities. For example, if a coordinate reaches |δi|=Ei,| _i|=E_i, (22) additional perturbation magnitude cannot be assigned to that coordinate while remaining feasible. Any remaining useful global budget must instead be allocated among other coordinates. Importantly, the local-capacity constraint does not necessarily reduce the amount of global budget that can be consumed. When sufficiently many coordinates remain below capacity, the full global budget DbDb may still be feasible. Rather, the local constraint changes the set of possible distributions of that budget across the input. This distinction motivates our focus on perturbation allocation. A.5 Feasibility Is Independent of Importance Let M denote the clean-gradient importance map used by the proposed method. The feasible set in Equation 13 contains no dependence on M: =(x,E,b),≠(x,E,b,M).C=C(x,E,b), (x,E,b,M). (23) In particular, model-derived importance does not: • remove low-importance coordinates from the feasible set; • increase the admissible magnitude at high-importance coordinates; • decrease the local capacity of low-importance coordinates; or • modify the global ℓ1 _1 budget. The importance map enters only through the allocation objective Jours(δ)=Jadv(x+δ,y)+λRalloc(δ,M),J_ours(δ)=J_adv(x+δ,y)+λ R_alloc(δ,M), (24) as described in the Centered Allocation Objective subsection. This yields the separation x,E,b⏟define feasibilityversusM,λ⏟define allocation preference. x,\;E,\;b_define feasibility M,\;λ_define allocation preference. (25) This separation is essential to the interpretation of the main experiments: matched baselines and our method receive the same admissible perturbation region, while differing in how they optimize within it. A.6 Matched Threat-Model Protocol For every matched comparison in the Main Performance under Capacity Limits subsection, attacks operate on: • the same clean input x and ground-truth label y; • the same global mean-ℓ1 _1 budget b; • the same realization of the local capacity map E; • the same valid image domain [0,1]D[0,1]^D; and • the same evaluation population. Where the attack construction permits direct matching, the same initial perturbation is also used. Baseline-specific optimization procedures are preserved as described in Appendix D. Constraint-satisfaction diagnostics confirming compliance with the global budget, local capacities, and image range are reported in Appendix N. A.7 Relation to Conventional ℓ1 _1 Constraints For reference, a conventional ℓ1 _1 threat model can be written as ℓ1(x,ϵ1)=δ:x+δ∈[0,1]D,∥δ∥1≤ϵ1,C_ _1(x, _1)= \δ:x+δ∈[0,1]^D,\;\|δ\|_1≤ _1 \, (26) without the additional coordinate-wise capacity constraint E. The conventional-ℓ1 _1 and capacity-limited evaluations included in this work were originally configured using separately chosen global budget parameterizations. They therefore differ in more than the presence or absence of E. For this reason, we do not interpret their difference as a controlled estimate of the isolated effect of local capacities. Those results are retained only as a descriptive reference and are reported separately in Appendix M. All central empirical claims in the main paper instead rely on matched comparisons within the common feasible set of Equation 13. Appendix B Projection and Optimization Details This appendix provides additional details for the optimization procedure introduced in the Attack Optimization subsection. The proposed method optimizes the adversarial objective and allocation reward jointly while restoring feasibility after every update. B.1 Initialization For each attacked input, the perturbation is initialized from a random direction and projected into the capacity-limited feasible set (x,E,b)C(x,E,b). The main experiments use the same initial perturbation for matched methods whenever the corresponding optimization procedure permits direct initialization matching. The recorded initialization mode for the proposed optimizer is random_boundary. The shared experiment seed is s=23.s=23. (27) Using the same initialization and local-capacity realization across matched methods reduces variation arising from stochastic starting points rather than from the attack mechanism itself. B.2 Gradient Update At optimization iteration t, we compute gt=∇δJours(δt),g_t= _δJ_ours( _t), (28) where JoursJ_ours is defined in Equation 8. The gradient is normalized by its mean absolute magnitude, g~t=gtmean(|gt|)+ϵg, g_t= g_tmean(|g_t|)+ _g, (29) and accumulated using momentum, vt=μvt−1+g~t,v_t=μ v_t-1+ g_t, (30) with μ=0.75.μ=0.75. (31) The perturbation is updated in the sign direction of the accumulated gradient, δ~t+1=δt+αtsign(vt), δ_t+1= _t+ _tsign(v_t), (32) where αt _t follows the cosine-decay schedule used in the attack implementation. The recorded reference step scale is αref=2255. _ref= 2255. (33) All main experiments use 50 optimization steps and one restart unless otherwise stated. B.3 Sequential Feasibility Operator After each unconstrained update, feasibility is restored in three stages. First, the perturbation is clipped to the coordinate-wise capacities: δi←clip(δ~i,−Ei,Ei). _i ( δ_i,-E_i,E_i ). (34) Second, the valid image range is enforced: δ←clip(x+δ,0,1)−x.δ (x+δ,0,1)-x. (35) Finally, if necessary, the perturbation is projected onto the global ℓ1 _1 ball of radius DbDb: δ←Π∥⋅∥1≤Db(δ),δ← _\|·\|_1≤ Db(δ), (36) using the Euclidean projection algorithm of (Duchi et al. 2008). We denote the complete sequential operator by Πseq(δ~,x,E,b). _seq( δ;x,E,b). (37) Because the final ℓ1 _1 projection only reduces perturbation magnitude, it does not introduce new violations of the previously enforced local or image-range constraints. B.4 Exact-Projection Control To test whether the main results depend on the sequential feasibility operator, we additionally compare it with direct Euclidean projection onto the full intersection Π(z)=argminδ∈(x,E,b)12‖δ−z‖22. _C(z)= _δ (x,E,b) 12\|δ-z\|_2^2. (38) Table 5 shows that replacing the sequential operator with the exact projection changes ASR only slightly on the three reference datasets. Table 5: Projection-operator control. Values are conditional ASR (%) on the reference ablation populations. Projection CIFAR-10 CIFAR-100 ImageNet Sequential 19.6 37.9 36.8 Exact 19.4 37.7 37.2 The corresponding differences are −0.2-0.2, −0.2-0.2, and +0.4+0.4 percentage points, respectively. This control indicates that the observed allocation effect is not driven by the particular sequential projection used in the main experiments. B.5 Best-Candidate Tracking Optimization does not rely only on the final iterate. At every iteration and restart, feasible candidates are ranked by the probability margin mprob(x+δ)=maxk≠ypk(x+δ)−py(x+δ),m_prob(x+δ)= _k≠ yp_k(x+δ)-p_y(x+δ), (39) and the candidate with the largest observed margin is retained. This ranking criterion is used only for candidate selection. Unless explicitly varied in an objective ablation, the optimization gradient continues to follow the adversarial objective together with the allocation reward. Appendix C Capacity-Map Construction The main threat model assigns a local perturbation capacity to each spatial location. These capacities are generated independently of the importance map used by the proposed allocation mechanism. C.1 Sampling Procedure For spatial location p, we sample Up∼(umin,umax),U_p (u_ ,u_ ), (40) with umin=10−3,umax=1−10−3.u_ =10^-3, u_ =1-10^-3. (41) The spatial capacity is then Ep=μE+σEΦ−1(Up),E_p= _E+ _E ^-1(U_p), (42) where Φ−1 ^-1 is the inverse standard-normal CDF. The main setting uses μE=12255,σE=2255. _E= 12255, _E= 2255. (43) Each sampled spatial capacity is shared across the three RGB channels: E1,p=E2,p=E3,p=Ep.E_1,p=E_2,p=E_3,p=E_p. (44) The truncation of UpU_p avoids extreme Gaussian tails while retaining moderate spatial variation around the nominal 12/25512/255 local scale. C.2 Independence from Importance The capacity-generation process does not use gradients, saliency values, labels, or the allocation map. Consequently, E⟂Mby construction,E M construction, (45) in the sense that the capacity realization is generated independently of the model-derived importance prior. The same realization is supplied to all matched attacks for a given sample. Digest-based consistency checks for the shared capacity maps are reported in Appendix N. C.3 Capacity-Structure Control We also evaluate whether the main result depends specifically on heterogeneous capacities. The reference heterogeneous map is compared with a uniform cap Ep=12255∀p,E_p= 12255 ∀ p, (46) and with a no-local-cap control. Table 6: Local-capacity structure control. Values are conditional ASR (%). The heterogeneous envelope is the main setting. Capacity structure CIFAR-10 ImageNet Heterogeneous envelope 19.6 36.8 Uniform 12/25512/255 cap 19.6 36.8 No local cap 18.3 40.4 The identical ASR obtained by the heterogeneous and uniform-cap variants in these reference runs indicates that the main allocation gain does not depend on the Gaussian heterogeneity itself. We therefore treat the heterogeneous map as one evaluated capacity model rather than as a necessary component of the allocation mechanism. Appendix D Baseline Adaptation Details All main-paper baselines operate under the same capacity-limited feasible set as the proposed method. The adaptation is intended to preserve the characteristic optimization rule or adversarial objective of each baseline while enforcing the shared global budget, local capacities, and image bounds. D.1 APGD-CE and APGD-DLR APGD-CE and APGD-DLR retain the corresponding APGD optimization objectives and adaptive optimization structure (Croce and Hein 2020). Their candidate perturbations are constrained to (x,E,b)C(x,E,b) (47) using the common feasibility operator. APGD-CE optimizes cross-entropy, whereas APGD-DLR uses the difference-of-logits-ratio objective. Both main-paper variants use 50 optimization steps and one restart. D.2 Probability Margin Attack PMA retains its probability-margin adversarial objective (Xie et al. 2025) while operating under the same (x,E,b)C(x,E,b) constraint. The main evaluation uses 50 optimization steps and one restart. As with APGD, the adaptation concerns feasibility rather than introducing the importance prior used by our method. D.3 Shared Assets For the matched APGD/PMA comparisons, the following quantities are shared within each model–sample pair: • the clean image and label; • the global mean-ℓ1 _1 budget; • the local-capacity realization; • the random seed; • the evaluation population; and • the initial perturbation where direct matching is applicable. The main result therefore compares different optimization strategies within a common feasible region. D.4 Specialized ℓ1 _1 Reference Attacks We additionally evaluate ℓ1 _1-APGD (Croce et al. 2021) and FMN-ℓ1 _1 (Pintor et al. 2021) in the descriptive cross-regime experiment of Appendix M. The conventional ℓ1 _1-APGD runs use 50 iterations, one restart, and the standard larger-epsilon initialization option used in the underlying implementation. Their total ℓ1 _1 radii are 1212 on CIFAR and 6060 on ImageNet. The conventional FMN runs use 50 steps. The recorded configuration uses maximum step size 1.01.0, γ=0.05γ=0.05, and Foolbox version 3.3.4. Their nominal total ℓ1 _1 bounds are again 1212 on CIFAR and 6060 on ImageNet. For the capacity-limited reference, adapted versions enforce ‖δ‖1D≤4255 \|δ\|_1D≤ 4255 (48) together with the same heterogeneous local-capacity envelope used by the main experiments. These specialized attacks are retained to document their behavior under the separately configured settings. Because the conventional and capacity-limited experiments do not use identical global-budget parameterizations, they are not treated as controlled local-cap interventions. Appendix E Dataset, Model, and Evaluation Populations Table 7 reports the complete evaluation populations used in the main matched comparison. Table 7: Main evaluation populations. Clean accuracy and clean-correct counts are computed on the full corresponding test or validation set. “Evaluated” denotes the number of clean-correct images included in the reported attack run. Dataset Model Dataset size Clean acc. (%) Clean-correct Evaluated Coverage CIFAR-10 WRN-94-16 10,000 93.68 9,368 9,368 complete CIFAR-10 WRN-28-10 10,000 92.16 9,216 7,500 partial CIFAR-10 XCiT-S12 10,000 90.06 9,006 9,006 complete CIFAR-100 WRN-28-10 10,000 73.83 7,383 7,383 complete CIFAR-100 XCiT-S12 10,000 67.34 6,734 6,734 complete CIFAR-100 WRN-70-16 10,000 75.22 7,522 7,522 complete ImageNet ConvNeXt-L 50,000 78.114 39,057 1,000 fixed subset ImageNet Peng et al. 50,000 73.882 36,941 1,000 fixed subset ImageNet ViT-B-ConvStem 50,000 76.718 38,359 1,000 fixed subset ImageNet Swin-L 50,000 79.180 39,590 1,000 fixed subset For CIFAR-10 and CIFAR-100, the intended population consists of clean-correct examples from the complete 10,000-image test set. The CIFAR-10 WRN-28-10 main run contains 7,500 of the 9,216 clean-correct examples and is therefore explicitly reported as partial coverage rather than as a complete-test evaluation. For ImageNet, each model is evaluated on 1,000 clean-correct validation images selected using the fixed seeded ordering recorded by the evaluation pipeline. Methods compared within a given model use the same 1,000 images. E.1 Reference Populations for Ablations The mechanism ablations use fixed reference populations rather than all ten main configurations. Table 8: Reference populations used for the ablation studies. Dataset Model N CIFAR-10 WRN-28-10 1,000 CIFAR-100 WRN-28-10 1,000 ImageNet Swin-L 500 Within an ablation suite, all variants operate on the same samples and shared attack assets. Appendix F Complete Hyperparameters Table 9 summarizes the reference configuration of the proposed optimizer as recorded by the experiment artifacts. Table 9: Reference hyperparameters for the proposed allocation-guided optimizer. Parameter Value Global mean-ℓ1 _1 budget b 4/2554/255 Capacity mean μE _E 12/25512/255 Capacity std. σE _E 2/2552/255 Capacity mode heterogeneous envelope Capacity quantile range [10−3,1−10−3][10^-3,1-10^-3] Optimization steps 50 Restarts 1 Initialization random boundary / shared initial δ Reference step scale 2/2552/255 Momentum μ 0.750.75 Reference objective cross-entropy Projection sequential Saliency source true-class logit Importance clipping quantile q 0.9950.995 Importance exponent γ 1.01.0 Importance floor MminM_ 0.050.05 Importance ceiling MmaxM_ 8.08.0 Smooth absolute-value τ 0.25/2550.25/255 Reference smoothing σ 00 Candidate selection probability margin Experiment seed 23 The dataset-specific allocation strengths are λ=0.5,CIFAR-10,0.5,CIFAR-100,0.2,ImageNet.λ= cases0.5,&CIFAR-10,\\ 0.5,&CIFAR-100,\\ 0.2,&ImageNet. cases (49) The main APGD-CE, APGD-DLR, and PMA comparisons use 50 optimization steps and one restart under the same threat-model parameters. F.1 Software Environment The ablation result records correspond to the following software environment: Table 10: Recorded experimental software environment. Component Version / configuration Python 3.12.13 PyTorch 2.10.0+cu128 Torchvision 0.25.0+cu128 CUDA 12.8 cuDNN 91002 RobustBench 1.1 LPIPS 0.1.4 PIQ 0.8.0 GPU 2 × Tesla T4 Appendix G Lambda Selection and Sensitivity The allocation weight λ determines the relative contribution of RallocR_alloc to the attack objective. We select one value per dataset on 100 held-out clean-correct examples and freeze it across models within that dataset. The candidate set used for selection is λ∈0.10,0.20,0.35,0.50,0.75.λ∈\0.10,0.20,0.35,0.50,0.75\. (50) The selected values are λC10=0.5,λC100=0.5,λImageNet=0.2. _C10=0.5, _C100=0.5, _ImageNet=0.2. (51) The selected values are not retuned using the reported main-test results. G.1 Sensitivity on the Reference Populations The retained sensitivity runs additionally include λ=0λ=0 and λ=1λ=1 as diagnostic endpoints. Table 11: Lambda sensitivity on the available reference runs. Top-10 mass is the fraction of perturbation magnitude assigned to the highest-importance 10% of spatial locations. λ=0λ=0 0.10.1 0.20.2 0.350.35 Dataset ASR Top-10 ASR Top-10 ASR Top-10 ASR Top-10 CIFAR-10 10.2 10.70 17.9 24.27 20.3 26.57 20.6 28.21 ImageNet 24.0 10.49 32.6 16.35 36.8 18.95 40.8 21.18 λ=0.5λ=0.5 0.750.75 1.01.0 Dataset ASR Top-10 ASR Top-10 ASR Top-10 CIFAR-10 19.6 29.06 16.9 29.81 14.7 30.24 ImageNet 41.2 22.66 43.0 24.49 41.0 25.71 The CIFAR-10 curve illustrates the central trade-off particularly clearly. Increasing λ initially improves both importance-aligned allocation and ASR, but after an intermediate optimum the allocation concentration continues to increase while attack success decreases. The ImageNet diagnostic shows the same non-monotonic relationship between allocation concentration and effectiveness, although the value selected on the independent calibration set is not the value that maximizes ASR on this 500-image diagnostic population. We therefore retain the preselected λ=0.2λ=0.2 rather than retrospectively tuning on the reported evaluation subset. This behavior supports the interpretation that the allocation term should guide, rather than dominate, adversarial optimization. Figure 6: Lambda sensitivity. Left: ASR as a function of allocation weight. Right: perturbation mass assigned to the top-10%10\% importance region. Increasing λ produces progressively stronger importance concentration, but attack effectiveness is non-monotonic. Appendix H Full Mechanism Ablations This section reports the complete mechanism controls underlying the Mechanism Ablations subsection. All variants within an ablation suite use matched samples, threat constraints, and shared assets. H.1 Importance Information and Centering The core variants are: • NoMask: removes the allocation reward (λ=0λ=0); • Uncentered: replaces the centered preference M−1M-1 with positive importance weighting; • RandomMask: replaces model-derived importance with a randomized spatial prior; five random seeds are evaluated; • InvertedMask: reverses the importance ordering; • SmoothMask: spatially smooths the importance prior; and • Ours: uses the centered raw model-derived importance map. Table 12: Core mechanism ablations. ASR is in percent; Top-10 is perturbation-mass share in the highest-importance 10% of locations. RandomMask reports the mean over five randomized priors, with the ASR standard deviation in parentheses. CIFAR-10 CIFAR-100 ImageNet Variant ASR Top-10 ASR Top-10 ASR Top-10 NoMask 10.2 10.70 23.9 10.74 24.0 10.49 Uncentered 3.9 10.31 13.7 10.58 20.4 10.46 RandomMask 14.24 (0.19) 13.68 28.18 (0.16) 13.40 29.24 (0.09) 13.63 InvertedMask 14.5 8.24 25.9 9.26 23.4 10.44 SmoothMask 17.6 24.74 34.4 20.90 36.2 18.03 Ours 19.6 29.06 37.9 24.74 36.8 18.95 The randomized prior consistently improves over NoMask, demonstrating that centered non-uniform redistribution itself contributes to optimization. The true importance prior provides an additional improvement on all three datasets. The Uncentered variant performs below NoMask in all three cases, supporting the use of a reward–penalty formulation rather than a globally positive magnitude reward. H.2 Alternative Adversarial Objectives We next test whether the allocation gain is tied specifically to cross-entropy. Table 13: Objective ablation. Each pair compares an objective without allocation against the same objective augmented by the centered raw importance prior. Values are conditional ASR (%). CE Margin DLR Prob. Margin Dataset NoMask Guided NoMask Guided NoMask Guided NoMask Guided CIFAR-10 10.2 19.6 11.7 23.1 11.9 22.3 11.8 18.6 CIFAR-100 23.9 37.9 28.0 41.5 29.5 42.0 28.5 38.2 ImageNet 24.0 36.8 24.0 30.2 24.4 39.2 26.6 43.8 Allocation guidance improves the corresponding unguided variant for every objective and dataset in this control. H.3 Compute-Matched Control The reference method uses 50 iterative attack steps plus one clean-gradient computation. To test whether the extra backward evaluation explains the gain, we replace one iterative step with the clean-gradient computation. Table 14: Compute-matched control. “Raw 49+1” uses 49 iterative attack gradients plus one clean-gradient computation. Variant CIFAR-10 CIFAR-100 ImageNet NoMask, 50 attack steps 10.2 23.9 24.0 Ours, 50 + 1 19.6 37.9 36.8 Ours, 49 + 1 19.6 37.9 36.8 The identical ASRs of the 50+150+1 and 49+149+1 guided variants indicate that the improvement cannot be explained by receiving one additional backward evaluation. H.4 Optimization Controls Two further controls test projection choice and local-capacity structure. Table 15: Projection control, conditional ASR (%). Variant CIFAR-10 CIFAR-100 ImageNet Sequential projection 19.6 37.9 36.8 Exact projection 19.4 37.7 37.2 Table 16: Local-capacity structure control, conditional ASR (%). Variant CIFAR-10 ImageNet Heterogeneous envelope 19.6 36.8 Uniform 12/25512/255 cap 19.6 36.8 No local cap 18.3 40.4 The uniform-cap result closely matches the heterogeneous reference. The no-cap control should be interpreted as an optimization diagnostic rather than as the controlled cross-regime experiment discussed in Appendix M. Appendix I Allocation Diagnostics We analyze allocation using importance-ranked perturbation mass, importance–perturbation correlation, effective spatial support, and spatial Gini concentration. I.1 Main-Experiment Allocation Statistics Table 17: Dataset-level macro-averaged allocation diagnostics. Top-10 is the fraction of absolute perturbation mass in the highest-importance 10% of spatial locations. Dataset Method Top-10 (%) Importance corr. Effective spatial frac. Spatial Gini CIFAR-10 APGD-CE 10.98 0.096 0.842 0.244 APGD-DLR 10.76 0.064 0.815 0.270 PMA 10.93 0.088 0.842 0.245 Ours 28.23 0.692 0.403 0.623 CIFAR-100 APGD-CE 11.18 0.121 0.839 0.247 APGD-DLR 11.06 0.086 0.792 0.290 PMA 11.16 0.111 0.831 0.255 Ours 24.65 0.665 0.471 0.575 ImageNet APGD-CE 10.89 0.084 0.822 0.264 APGD-DLR 10.69 0.056 0.783 0.298 PMA 10.72 0.069 0.827 0.259 Ours 20.25 0.499 0.560 0.495 The increased Top-10 share and importance correlation directly reflect the intended allocation behavior. The reduced effective spatial fraction and larger spatial Gini additionally show that the resulting perturbations are less spatially diffuse. I.2 Cumulative Importance-Ranked Mass Table 18 reports cumulative perturbation mass at the 1%1\%, 5%5\%, 10%10\%, and 20%20\% importance checkpoints for the reference mechanism variants. Table 18: Cumulative perturbation-mass share (%) after including the most-important spatial locations. A uniform allocation would yield approximately 1%1\%, 5%5\%, 10%10\%, and 20%20\% at the corresponding checkpoints. Dataset Variant Top-1 Top-5 Top-10 Top-20 CIFAR-10 NoMask 1.16 5.44 10.70 21.05 InvertedMask 1.35 5.03 8.24 12.78 SmoothMask 2.71 12.69 24.74 46.23 Ours 3.12 14.72 29.06 57.35 CIFAR-100 NoMask 1.16 5.44 10.74 21.24 InvertedMask 1.29 5.24 9.26 15.64 SmoothMask 2.27 10.67 20.90 40.11 Ours 2.67 12.58 24.74 48.61 ImageNet NoMask 1.07 5.28 10.49 20.79 InvertedMask 1.26 5.70 10.44 18.35 SmoothMask 1.84 9.13 18.03 35.13 Ours 1.93 9.62 18.95 36.91 The cumulative curves therefore show a graded rather than purely binary change in allocation: the difference is already visible in the most important few percent of locations and persists through larger importance regions. Figure 7: Cumulative perturbation mass after ordering locations from highest to lowest clean-gradient importance. The dashed diagonal corresponds to uniform allocation. Appendix J Perceptual and Norm Diagnostics The proposed mechanism is designed to redistribute a fixed global perturbation budget, not to optimize perceptual similarity. We therefore report both global norm utilization and local/perceptual diagnostics. Table 19: Dataset-level macro-averaged perturbation diagnostics. Mean-ℓ1 _1 and ℓ∞ _∞ are multiplied by 255. Lower LPIPS/DISTS and higher SSIM indicate greater perceptual similarity. Dataset Method Mean-ℓ1×255 _1×255 ℓ∞×255 _∞×255 LPIPS DISTS SSIM CIFAR-10 APGD-CE 4.000 11.49 0.00223 0.06037 0.98126 APGD-DLR 3.999 11.86 0.00255 0.06406 0.97999 PMA 4.000 10.81 0.00226 0.06066 0.98217 Ours 3.907 16.78 0.00296 0.07555 0.95456 CIFAR-100 APGD-CE 4.000 11.45 0.00257 0.06383 0.97816 APGD-DLR 3.997 12.27 0.00300 0.06826 0.97576 PMA 4.000 10.89 0.00266 0.06490 0.97871 Ours 3.959 16.18 0.00385 0.07855 0.95442 ImageNet APGD-CE 4.000 13.32 0.04236 0.08282 0.94746 APGD-DLR 3.995 13.67 0.04245 0.08374 0.94562 PMA 4.000 11.54 0.04160 0.08208 0.94995 Ours 3.968 16.94 0.05060 0.10254 0.93714 The realized global mean-ℓ1 _1 magnitude of our method is not larger than that of the matched baselines. However, its realized ℓ∞ _∞ magnitude is consistently larger, reflecting the intended redistribution of perturbation toward selected regions. The perceptual metrics similarly show that the increase in attack effectiveness is not a free improvement in visual similarity. The proposed method generally has larger LPIPS and DISTS and lower SSIM than the matched baselines. These differences are consistent with stronger local concentration while remaining within the same global mean-ℓ1 _1 and coordinate-cap constraints. We therefore use these measurements as diagnostics of the perturbation profile rather than as an additional performance claim. Appendix K Computational Cost Table 20 reports model-evaluation counts and dataset-level macro-averaged attack runtime. Table 20: Computational accounting for the main methods. Runtime is the recorded mean attack runtime in seconds; it is intended for within-experiment comparison rather than as hardware-independent benchmarking. Dataset Method Backward evals. Forward evals. Runtime (s) CIFAR-10 APGD-CE 50 53 2.86 APGD-DLR 50 53 2.88 PMA 50 102 4.08 Ours 51 103 4.06 CIFAR-100 APGD-CE 50 53 2.42 APGD-DLR 50 53 2.44 PMA 50 102 3.43 Ours 51 103 3.42 ImageNet APGD-CE 50 53 3.29 APGD-DLR 50 53 3.29 PMA 50 102 4.73 Ours 51 103 4.71 The additional cost of our method relative to PMA consists primarily of the single clean-gradient computation used to construct the importance map. Accordingly, the recorded runtime is very close to PMA. Relative to the APGD variants, however, our implementation uses substantially more forward evaluations. For this reason we describe the method as approximately PMA-cost, rather than as universally low-overhead. The compute-matched 49+149+1 control in Appendix H shows that the additional clean-gradient backward evaluation does not explain the attack-success gain. Appendix L Statistical Analysis All main attack comparisons are paired because methods within a dataset–model configuration are evaluated on the same clean-correct examples. For each baseline B, we define the paired ASR effect ΔASR=ASRours−ASRB. _ASR=ASR_ours-ASR_B. (52) We report paired 95% confidence intervals produced by the evaluation pipeline. Differences in paired binary success outcomes are tested using the exact McNemar test (McNemar 1947). The family of main comparisons is corrected using Holm’s procedure (Holm 1979). Table 21: Full paired main-result statistics. Δ is ASRours−ASRbaselineASR_ours-ASR_baseline in percentage points. pHolmp_Holm is the globally Holm-corrected exact McNemar p-value. Dataset Model Baseline Δ 95% CI pHolmp_Holm CIFAR-10 WRN-94-16 APGD-CE 9.34 [8.76, 9.93] 2.22×10−2622.22× 10^-262 CIFAR-10 WRN-94-16 APGD-DLR 7.70 [7.15, 8.23] 2.08×10−2082.08× 10^-208 CIFAR-10 WRN-94-16 PMA 8.13 [7.58, 8.70] 4.09×10−2244.09× 10^-224 CIFAR-10 WRN-28-10 APGD-CE 9.44 [8.77, 10.11] 3.86×10−2123.86× 10^-212 CIFAR-10 WRN-28-10 APGD-DLR 7.45 [6.85, 8.05] 3.91×10−1603.91× 10^-160 CIFAR-10 WRN-28-10 PMA 7.79 [7.19, 8.41] 7.86×10−1717.86× 10^-171 CIFAR-10 XCiT-S12 APGD-CE 7.44 [6.91, 7.95] 1.39×10−1961.39× 10^-196 CIFAR-10 XCiT-S12 APGD-DLR 5.49 [5.02, 6.00] 8.70×10−1258.70× 10^-125 CIFAR-10 XCiT-S12 PMA 5.49 [5.01, 5.96] 2.08×10−1272.08× 10^-127 CIFAR-100 WRN-28-10 APGD-CE 12.87 [12.07, 13.61] 1.79×10−2801.79× 10^-280 CIFAR-100 WRN-28-10 APGD-DLR 6.96 [6.31, 7.57] 1.04×10−1221.04× 10^-122 CIFAR-100 WRN-28-10 PMA 7.82 [7.18, 8.48] 2.54×10−1532.54× 10^-153 CIFAR-100 XCiT-S12 APGD-CE 8.26 [7.62, 8.90] 2.92×10−1592.92× 10^-159 CIFAR-100 XCiT-S12 APGD-DLR 2.73 [2.15, 3.34] 1.46×10−191.46× 10^-19 CIFAR-100 XCiT-S12 PMA 2.52 [2.00, 3.06] 1.47×10−191.47× 10^-19 CIFAR-100 WRN-70-16 APGD-CE 12.38 [11.62, 13.13] 7.46×10−2777.46× 10^-277 CIFAR-100 WRN-70-16 APGD-DLR 7.15 [6.57, 7.78] 2.57×10−1372.57× 10^-137 CIFAR-100 WRN-70-16 PMA 7.71 [7.09, 8.34] 3.59×10−1543.59× 10^-154 ImageNet ConvNeXt-L APGD-CE 18.00 [15.60, 20.40] 6.57×10−526.57× 10^-52 ImageNet ConvNeXt-L APGD-DLR 18.20 [15.90, 20.80] 6.85×10−496.85× 10^-49 ImageNet ConvNeXt-L PMA 15.90 [13.70, 18.20] 9.98×10−469.98× 10^-46 ImageNet Peng et al. APGD-CE 19.80 [17.40, 22.20] 6.97×10−596.97× 10^-59 ImageNet Peng et al. APGD-DLR 20.10 [17.60, 22.60] 4.13×10−584.13× 10^-58 ImageNet Peng et al. PMA 17.70 [15.30, 20.00] 1.25×10−521.25× 10^-52 ImageNet ViT-B-ConvStem APGD-CE 13.00 [11.00, 15.00] 1.18×10−381.18× 10^-38 ImageNet ViT-B-ConvStem APGD-DLR 13.20 [11.10, 15.50] 1.59×10−341.59× 10^-34 ImageNet ViT-B-ConvStem PMA 9.90 [8.10, 12.00] 3.31×10−253.31× 10^-25 ImageNet Swin-L APGD-CE 10.20 [8.40, 12.20] 6.21×10−296.21× 10^-29 ImageNet Swin-L APGD-DLR 8.70 [6.80, 10.70] 3.26×10−193.26× 10^-19 ImageNet Swin-L PMA 7.80 [6.00, 9.60] 1.08×10−181.08× 10^-18 All 30 paired ASR differences are positive, all reported confidence intervals exclude zero, and all comparisons remain significant after the global Holm correction. The largest corrected p-value is approximately 1.08×10−181.08× 10^-18. Because the evaluation populations are large, we emphasize the effect sizes and confidence intervals rather than the magnitude of extremely small p-values. Appendix M Descriptive Cross-Regime Results This section preserves the conventional-versus-capacity-limited reference experiment from the earlier study design. These results are intentionally separated from the main empirical evidence. The two regimes use different global-budget parameterizations and therefore do not constitute a controlled intervention in which only the local capacity changes. M.1 Budget Parameterizations For conventional ℓ1 _1-APGD and FMN-ℓ1 _1, the total ℓ1 _1 radius is ϵ1=12 _1=12 (53) on CIFAR and ϵ1=60 _1=60 (54) on ImageNet. The proposed conventional variant instead uses the mean-ℓ1 _1 parameterization ‖δ‖1D≤4255. \|δ\|_1D≤ 4255. (55) Under the capacity-limited setting, all reported methods use ‖δ‖1D≤4255 \|δ\|_1D≤ 4255 (56) together with the heterogeneous local-capacity envelope μE=12255,σE=2255. _E= 12255, _E= 2255. (57) Thus, both the presence of local capacities and the global-budget parameterization differ across portions of the conventional comparison. M.2 Reference Results Table 22: Descriptive results under separately parameterized conventional and capacity-limited ℓ1 _1 threat settings. Because the global budget parameterizations differ, changes between the two column groups must not be interpreted as the isolated causal effect of adding local capacities. Conventional ℓ1 _1 ASR (%) Capacity-limited ASR (%) Dataset Model N ℓ1 _1-APGD FMN-ℓ1 _1 Ours Adapted ℓ1 _1-APGD Adapted FMN-ℓ1 _1 Ours CIFAR-10 WRN-94-16 9,368 84.03 81.18 18.01 32.55 22.88 17.98 CIFAR-100 WRN-70-16 7,522 88.86 82.82 32.03 49.19 43.23 32.53 ImageNet Swin-L 1,000 44.50 33.90 39.50 69.00 16.50 36.00 The table documents substantial method-dependent differences across the two separately defined regimes. For example, ℓ1 _1-APGD decreases on the two CIFAR reference configurations but increases on ImageNet, whereas FMN-ℓ1 _1 decreases on all three. These observations are useful descriptively, but they do not establish that local capacities alone caused the changes. We therefore do not use this experiment to support the main claim of the paper. A second observation is important for the scope of our contribution: adapted specialized ℓ1 _1-APGD is stronger than our reference optimizer on all three capacity-limited configurations in this table. The proposed method should consequently be interpreted as an allocation mechanism that improves the matched APGD/PMA-style optimizers studied in the main experiment, rather than as a universally strongest ℓ1 _1 attack. Appendix N Constraint-Validity Checks We verify the capacity-limited threat constraints for every method in the main ten-configuration evaluation. For every generated perturbation we check ‖δ‖1D \|δ\|_1D ≤4255, ≤ 4255, (58) |δi| | _i| ≤Ei∀i, ≤ E_i ∀ i, (59) 0 0 ≤xi+δi≤1∀i. ≤ x_i+ _i≤ 1 ∀ i. (60) We additionally verify that matched methods use the intended shared capacity and initialization assets through stored digests. Table 23: Constraint-validity summary across the ten main model–dataset configurations. Numerical residuals are maximum observed floating-point excesses above the corresponding constraints. Method Violations Max mean-ℓ1 _1 excess Max cap excess Image-low excess Image-high excess APGD-CE 0 3.07×10−83.07× 10^-8 2.98×10−82.98× 10^-8 0 0 APGD-DLR 0 2.89×10−82.89× 10^-8 2.98×10−82.98× 10^-8 0 0 PMA 0 5.68×10−85.68× 10^-8 2.98×10−82.98× 10^-8 0 0 Ours 0 3.26×10−83.26× 10^-8 2.98×10−82.98× 10^-8 0 0 No substantive constraint violations are observed. The nonzero residuals are on the order of 10−810^-8 and are attributable to floating-point precision. Across the main comparisons, the recorded capacity-digest mismatch count and initialization-digest mismatch count are both zero. The ablation projection preflight checks additionally pass the 2×10−62× 10^-6 numerical tolerance. These diagnostics support the interpretation that differences in attack behavior arise within the intended common feasible region rather than from constraint violations or mismatched capacity realizations. Appendix O Additional Qualitative Examples Qualitative visualizations complement the quantitative allocation measurements by showing how different mechanism variants distribute perturbation under the same threat constraints. For each displayed example, we visualize: 1. the clean input; 2. the fixed clean-gradient importance map; 3. the perturbation magnitude produced without allocation guidance; 4. the perturbation produced using a randomized prior; 5. the perturbation produced using an inverted prior; 6. the perturbation produced using a smoothed importance map; and 7. the perturbation produced by the full centered importance-guided method. Perturbation visualizations are normalized consistently within an example for display purposes. This visualization normalization does not alter the actual perturbations used for evaluation. The representative example used in the main qualitative analysis is selected using the predefined median-LPIPS criterion rather than by manually choosing the most visually favorable case. Additional examples should use the same deterministic selection principle. The qualitative examples should not be interpreted as independent evidence of attack superiority. Their purpose is to make the allocation behavior measured by Top-10 mass, cumulative importance-ranked mass, effective support, and spatial Gini visually interpretable.