Paper deep dive
Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controller
Sagi Habani, Yehonathan Refael, Ofir Lindenbaum
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Releasing a neural network checkpoint can leak information about its training data, and existing reconstruction results rely on gradient-descent training. We ask whether the choice of optimizer affects the ability to recover the training data. Extending work connecting Karush--Kuhn--Tucker conditions to training-data reconstruction, we show that the implicit biases of popular optimizers (SGD, Adam, Muon) leave very different fingerprints of the training data. We argue that the key distinction is the \emph{weight-specificity} of these implicit biases, namely how much continuous information about the final weights the optimizer-dependent target of the stationarity conditions encodes. The limiting stationarity conditions for models trained with SGD and Muon retain enough weight-specific information to reconstruct training examples in practice. For Adam in the $\ell_\infty$ implicit-bias regime, the optimizer-dependent part of these conditions encodes only a discrete pattern of maximal coordinates and their signs, and at nondegenerate KKT points the conditions are too few to identify the training set to first order. Experiments on MNIST and CIFAR-10 support these claims: models trained with Muon can yield stronger reconstructions at much lower training cost than models trained with SGD, although this advantage depends on the training regime and architecture, and models trained with Adam appear consistently more resistant to the evaluated attacks.
Tags
Links
- Source: https://arxiv.org/abs/2610.18294v1
- Canonical: https://arxiv.org/abs/2610.18294v1
Trouble viewing inline? Open PDF directly →
Full Text
117,255 characters extracted from source content.
Expand or collapse full text
Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controller Sagi Habani∗ Yehonathan Refael∗ Ofir Lindenbaum Ben-Gurion University sagihab@post.bgu.ac.il Tel Aviv University refaelkalim@mail.tau.ac.il Bar Ilan University ofir.lindenbaum@biu.ac.il †footnotetext: ∗Equal contribution. Abstract Releasing a neural network checkpoint can leak information about its training data, and existing reconstruction results rely on gradient-descent training (Haim et al., 2022). We ask whether the choice of optimizer affects the ability to recover the training data. Extending work connecting Karush–Kuhn–Tucker conditions to training-data reconstruction, we show that the implicit biases of popular optimizers (SGD, Adam, Muon) leave very different fingerprints of the training data. We argue that the key distinction is the weight-specificity of these implicit biases, namely how much continuous information about the final weights the optimizer-dependent target of the stationarity conditions encodes. The limiting stationarity conditions for models trained with SGD and Muon retain enough weight-specific information to reconstruct training examples in practice. For Adam in the ℓ∞ _∞ implicit-bias regime, the optimizer-dependent part of these conditions encodes only a discrete pattern of maximal coordinates and their signs, and at nondegenerate KKT points the conditions are too few to identify the training set to first order. Experiments on MNIST and CIFAR-10 support these claims: models trained with Muon can yield stronger reconstructions at much lower training cost than models trained with SGD, although this advantage depends on the training regime and architecture, and models trained with Adam appear consistently more resistant to the evaluated attacks. 1 Introduction (a) Muon (b) SGD (c) Adam (d) Muon (e) SGD (f) Adam Figure 1: Training-data reconstructions from networks trained with Muon, SGD and Adam, without an image prior (n=500n=500). The top row is binary MNIST and the bottom row binary CIFAR-10. In each panel, odd rows show reconstructions and the row directly below shows their nearest training images under the one-to-one matching of Section 5. Muon reconstructions are visually closest to their matches, SGD recovers coarser structure, and Adam reconstructions are largely unrecognizable. Over-parameterized networks can perform well while memorizing training examples (Zhang et al., 2017), and good generalization may require memorizing atypical examples (Feldman, 2020), raising a privacy concern: how much memorized data can be recovered? Recently, Haim et al. (2022); Buzaglo et al. (2023) demonstrated the ability to reconstruct a large fraction of the training images used to train a classifier, using only the parameters of the trained classifier (without query access or other data). They achieve this by optimizing the input to satisfy a stationarity condition that relates the weights of the neural network to the training data. Merely sharing a trained model (a checkpoint) may therefore leak its training data. This attack exploits the implicit bias of gradient descent (GD): for a homogeneous network trained with an exponentially tailed loss, the parameter direction converges, under suitable conditions, to that of a KKT point of the maximum-margin problem min12‖22, _ θ 12\| θ\|_2^2, s.t. yif(i,)≥1y_if(x_i; θ)≥ 1 for all i (Lyu and Li, 2020; Ji and Telgarsky, 2020). Its stationarity condition expresses the weights as a nonnegative combination of the signed gradients yi∇f(i,)y_i _ θf(x_i; θ), and Haim et al. (2022) invert it by solving for inputs ix_i with the weights fixed: the optimizer’s bias gives the attacker a system of equations relating the public weights to the training data. Other optimizers induce other margin geometries: under suitable assumptions for smooth homogeneous models, Muon Jordan and others (2024) maximizes a max-spectral-norm margin and Adam (Kingma and Ba, 2015), with a negligible stability constant, an ℓ∞ _∞ margin (Gronich and Vardi, 2026). Different optimizers therefore induce different stationarity equations; we analyze how these differences affect data reconstruction and whether the choice of optimizer can reduce this risk. We generalize the objective of Haim et al. (2022) to the margin geometry of the chosen optimizer (Section 3). A direct extension replaces its Euclidean stationarity term with a condition on the subdifferential of the margin norm. When applied naively, however, a formulation based on a single leading singular-vector pair may fail when the top singular value is repeated (a property encouraged by Muon) or when the relative block scales it fixes differ from those realized at a KKT point. It also breaks the rescaling symmetry of the KKT system, so it can be lowered by sending the multipliers to zero rather than by aligning the candidates with the data (Lemma A.7). To avoid this problem, we derive a scale-invariant penalty that enforces equality in the dual-norm inequality and use the layerwise Euler identity to show that no cross-block coupling term is needed. Our analysis (Section 4) identifies two properties that help explain why the attack works. The first is weight-specificity – the number of independent continuous functionals of θ that the optimizer-dependent stationarity target determines. Equation counting at a generic weight vector does not distinguish the three geometries (there the target set has codimension Θ(p) (p) in all three, where p is the parameter count), but specificity does. At a KKT direction, where every block is active (Corollary A.2), and with simple top singular values (ρk=1 _k=1), S has the values Sℓ2=p−1,Sspectral=∑k(mk+nk−2),Sℓ∞=0,S_ _2=p-1, S_spectral= _k(m_k+n_k-2), S_ _∞=0, where block k has dimensions mk×nkm_k× n_k, so each spectral block contributes a number of functionals proportional to its width. For the ℓ∞ _∞ margin, the optimizer-dependent target depends only on the locations and signs of the coordinates of θ with maximum absolute value, regardless of the multiplicity of the maximum. Moreover, at nondegenerate KKT points, the stationarity constraints impose at most as many equations as there are on-margin examples, far fewer than the unknowns, so the training set is not identifiable from them to first order. The second property is reachability - the speed of convergence to the limiting parameter direction of the optimizer. Combining the two, we find the practical leakage ordering Muon≻SGD≻AdamMuon in our main experiments, where ≻ denotes greater reconstruction leakage (Section 5). The Muon comparison depends on the training budget, sample size, and architecture (SGD becomes comparable to Muon at small sample sizes and matches or exceeds it in some architectures), while Adam remains the most resistant to the evaluated attacks; we do not, however, provide a formal privacy guarantee. Our contributions are: (i) a reconstruction attack that depends on the optimizer’s margin norm, recovers the objective of Haim et al. (2022) in the Euclidean case up to the multiplier scale, and corrects two failure modes of the naive extension (Section 3); (i) a theory of weight-specificity under which Adam’s ℓ∞ _∞ target has zero specificity at every multiplicity of the maximum and its nondegenerate KKT points fail regular first-order identifiability (Section 4); (i) an analysis combining specificity and reachability that explains the ordering Muon≻SGD≻AdamMuon in our main experiments, with experiments showing that the Muon comparison depends on the training regime while Adam remains the most resistant to the evaluated attacks (Section 5). 2 Related Work Reconstruction among privacy attacks. Haim et al. (2022) reconstruct training samples from homogeneous classifiers by inverting the ℓ2 _2-margin KKT conditions; follow-up work extends this to the multiclass setting, weight decay and general losses (Buzaglo et al., 2023), to transfer learning (Oz et al., 2024), and to membership inference (Golbari et al., 2025), and examines the specific reliance of reconstruction attacks on inductive image priors (Refael et al., 2026b). All use the ℓ2 _2 implicit bias of gradient descent; we extend the formulation to other margin geometries and study how the optimizer affects the attack. Implicit bias, and the geometry of Adam and Muon. Adam has a GD-like implicit bias when its stability constant dominates (Wang et al., 2021) and an ℓ∞ _∞ margin bias when it is absent (Zhang et al., 2024); spectral descent and Muon have been analyzed in the multiclass linear setting (Fan et al., 2025). Under suitable assumptions, Gronich and Vardi (2026) extend these results to smooth homogeneous networks, relating idealized Muon to normalized momentum steepest descent under the max-spectral norm and Adam to the global ℓ∞ _∞ margin, and leave open whether KKT-based reconstruction extends to these optimizers. We derive attacks for both geometries: reconstruction succeeds for Muon, while our analysis identifies a first-order identifiability limitation of Adam’s stationarity constraints, consistent with its greater resistance in our experiments. 2.1 Preliminaries: notation and threat model Notation. We write [n]:=1,…,n[n]:=\1,…,n\, (a)+:=maxa,0(a)_+:= \a,0\, je_j for the j-th standard basis vector, and Sd−1S^d-1 for the unit sphere in ℝdR^d. We denote the convex, affine, and conic hulls of a set X by convX X, affX X, and coneX X, respectively. For a matrix A, σ1(A)≥σ2(A)≥⋯ _1(A)≥ _2(A)≥·s are its singular values, A⪰0A 0 means that A is positive semidefinite, and M⪰0:trM=1\M 0:trM=1\ is a spectrahedron. We write S∘:=S∖0S :=S \0\. Parameters, blocks and norms. The network has parameters =(W1,…,WK) θ=(W_1,…,W_K), where Wk∈ℝmk×nkW_k ^m_k× n_k and each weight matrix is a block. We identify θ with the concatenated vector in ℝpR^p, where p=∑kmknkp= _km_kn_k. We write θj _j for its j-th coordinate and Wku_W_k for the k-th block of any ∈ℝpu ^p, and use the Euclidean inner product ⟨,⟩=∑ktr(Wk⊤Wk) ,v = _ktr(u_W_k v_W_k). For each block, we use the Frobenius, spectral, and nuclear norms ‖W‖F\|W\|_F, ‖W‖sp=σ1(W)\|W\|_sp= _1(W), and ‖W‖nuc=∑iσi(W)\|W\|_nuc= _i _i(W), respectively. On the full parameter vector, we use ‖1\| θ\|_1, ‖2=(∑k‖Wk‖F2)1/2\| θ\|_2=( _k\|W_k\|_F^2)^1/2, ‖∞=maxj|θj|\| θ\|_∞= _j| _j|, and the max-spectral norm ‖msp=maxk‖Wk‖sp\| θ\|_msp= _k\|W_k\|_sp of Gronich and Vardi (2026). An unadorned ∥⋅∥\|·\| denotes the margin norm of the optimizer under discussion (Eq. (P)), and ∥⋅∥⋆\|·\|_ denotes its dual norm, ‖⋆=sup⟨,⟩:‖≤1.\|u\|_ = \ ,v :\|v\|≤ 1\. These norms satisfy ⟨,⟩≤‖⋆‖ ,v ≤\|u\|_ \|v\|, which we refer to as the generalized Cauchy–Schwarz inequality. Alignment, gauge, strata. We write ∂g()∂ g( θ) for the subdifferential of a convex function g at θ. Where g is differentiable, this is the singleton ∇g()\∇ g( θ)\. For a norm, ∂∥=:∥⋆≤1,⟨,⟩=∥.∂\| θ\|=\u:\|u\|_ ≤ 1,\ , θ =\| θ\|\. We say that u is dual-aligned with θ when equality holds in generalized Cauchy–Schwarz: ⟨,⟩ , θ =‖⋆‖, =\|u\|_ \| θ\|, (1) equivalently /‖⋆∈∂‖(≠). /\|u\|_ ∈∂\| θ\| (u 0). A gauge is a group of transformations of the unknowns that preserves the constraint system and maps solutions to solutions. Since the constraints cannot distinguish such solutions, we make our objective invariant along gauge directions. Finally, we divide parameter space into strata on which the structural type of ∂‖∂\| θ\| is constant (Appendix F.1). Blockwise homogeneity. Our structural results use homogeneity layer by layer. Assumption 2.1 (Blockwise homogeneity). For each block k there is dk>0d_k>0 such that f(,W1,…,αWk,…,WK)=αdkf(,)f(x;W_1,…,α W_k,…,W_K)=α^d_kf(x; θ) for all α>0α>0, and f(,⋅)f(x;·) is differentiable at the points considered. Then f is L-homogeneous with L=∑kdkL= _kd_k. Optimizer implicit bias. Homogeneous networks trained with logistic or exponential loss converge in direction to a KKT point of a maximum-margin problem; this implicit bias is the key phenomenon underlying the attacks we study. For gradient flow, this follows from Theorem F.1 in Appendix F.3. Under the assumptions of Gronich and Vardi (2026), Adam and Muon converge to KKT directions of (P), with the optimizer-dependent margin norm given in (2). min12‖2s.t.yiΦ(i,)≥1∀i∈[n]. _ θ\ 12\| θ\|^2 .t. y_i (x_i; θ)≥ 1\ \ ∀ i∈[n]. (P) Since every count below depends on how that norm aggregates across blocks, we fix it explicitly: ‖SGD \| θ\|_SGD =(∑k‖Wk‖F2)1/2, = ( _k\|W_k\|_F^2 )^1/2, (2) ‖Muon \| θ\|_Muon =maxk‖Wk‖spck, = _k \|W_k\|_spc_k, ‖Adam \| θ\|_Adam =‖∞. =\| θ\|_∞. with duals ℓ2 _2, ∑kck∥⋅∥nuc _kc_k\|·\|_nuc and ℓ1 _1. For ck≡1c_k≡ 1, the Muon form is ∥⋅∥msp\|·\|_msp used by Gronich and Vardi (2026) (their Corollary 3.3); general ck>0c_k>0 are the per-block weights induced by practical Muon’s per-block step sizes (their Theorem 3.6). All our statements for Muon hold for arbitrary ckc_k, and the objective we construct does not depend on them. The Adam ℓ∞ _∞ norm is global over all p coordinates rather than per tensor. This margin norm is the only property of the optimizer the attack uses, and the only axis along which our three optimizers differ in the analysis. Threat model. Following Haim et al. (2022), the adversary observes the final parameters θ and the architecture and knows the optimizer family, hence ∥⋅∥\|·\|; it observes neither the training inputs nor their labels or count n, nor any intermediate iterate. In particular θ is fixed and given: the attacker may not rescale it (Section 3). As is standard, the attacker posits m candidates (we use m=2nm=2n following Haim et al. (2022), as an upper bound on n suffices) with fixed balanced labels and recovers a subset corresponding to true samples. The gradient-descent attack (Haim et al., 2022). For ℓ2 _2, ∂(12‖22)=∂( 12\| θ\|_2^2)=\ θ\, so stationarity reads =∑iλiyi∇Φ(i,) θ= _i _iy_i _ θ (x_i; θ) with λi≥0 _i≥ 0, and complementary slackness forces λi=0 _i=0 off the margin. With θ known and the inputs unknown, Haim et al. (2022) minimize over candidate inputs and multipliers Lrec L_rec =‖−∑iλiyi∇Φ(i,)‖22⏟Lstat = \| θ- _i _iy_i _ θ (x_i; θ) \|_2^2_L_stat (3) +a2∑i(−λi)+⏟Lλ+a3Lprior, +a_2 _i(- _i)_+_L_λ+a_3\,L_prior, where LλL_λ enforces λi≥0 _i≥ 0 and LpriorL_prior is a weak input prior such as a pixel-range penalty. Only LstatL_stat depends on the optimizer, and only through ∂(12‖2)∂( 12\| θ\|^2). That is the starting point of the next section. 3 A Norm-Parameterized Reconstruction Objective We build the attack in three steps: the stationarity condition of (P) for a general margin norm (§3.1); removal of a non-informative degree of freedom, which yields an alignment condition and our loss (§3.2); and its instantiation for Muon (§3.3) and Adam (§3.4). Throughout, the trained θ is fixed, the unknowns are z:=(ii=1m,λii=1m)z:=(\x_i\_i=1^m,\ _i\_i=1^m), and the residual is r(z):=∑i=1mλiyi∇Φ(i,).r(z):= _i=1^m _iy_i _ θ (x_i; θ). (4) 3.1 Stationarity for a general norm For (S)GD the margin norm is Euclidean, 12‖22 12\| θ\|_2^2 has gradient θ, and the stationarity term of (3) vanishes iff = r= θ. For Muon and Adam, 12‖2 12\| θ\|^2 is not differentiable, so we replace the gradient with the subdifferential; by the chain rule ∂(12‖2)=‖∂‖∂( 12\| θ\|^2)=\| θ\|∂\| θ\| (Lemma B.1), and the stationarity condition for (P) is (z)∈∂(12‖2),λi≥0. r(z)∈∂ ( 12\| θ\|^2 ), _i≥ 0. (5) To obtain a tractable loss, the following lemma characterizes this set-membership condition in terms of two scalar inequalities. Lemma 3.1 (Membership rule). For θ≠0θ≠ 0, ∈∂(12‖2) r∈∂ ( 12\| θ\|^2 ) if and only if ‖⋆≤‖\| r\|_ ≤\| θ\| and ⟨,⟩≥‖2 r, θ ≥\| θ\|^2. Lemma 3.1 splits (5) into a magnitude and an alignment condition (generalizing (1)). Both ‖(z)‖⋆\| r(z)\|_ and ⟨(z),⟩ r(z), θ are computable and almost everywhere differentiable in z, and both inequalities hold with equality for feasible r (see the proof), so (5) can be enforced by penalizing ‖⋆=‖\| r\|_ =\| θ\| and ⟨,⟩=‖2 r, θ =\| θ\|^2. As we show next, all information about i\x_i\ lies in the alignment requirement. 3.2 Two scale freedoms, and why only one may be discarded The magnitude bound of Lemma 3.1 constrains the multipliers but says nothing about the candidate inputs, because the multipliers are themselves free unknowns. Lemma 3.2 (The multiplier scale is a gauge). The residual (⋅) r(·) is positively homogeneous of degree one in λ, and the constraint λ≥0λ≥ 0 is preserved by λ↦cλ cλ for c>0c>0. Hence for any z with (z)≠ r(z) 0 there is a unique c>0c>0 whose rescaled multipliers satisfy ‖⋆=‖\| r\|_ =\| θ\| exactly. The magnitude bound therefore restricts no i\x_i\, and (5) is equivalent, modulo this one-dimensional gauge, to the cone membership (z)∈C() r(z)∈ C( θ) :=cone(∂(12‖2)) := (∂( 12\| θ\|^2) ) (6) =t:∈∂(12∥2),t≥0. =\\,t\, k: k∈∂( 12\| θ\|^2),\,t≥ 0\,\. Thus the magnitude of r is a scale convention, not a property of the data: everything the stationarity condition says about the training set is carried by the direction of r, i.e. by the cone C()C( θ), the central object of Section 4. A reconstruction loss should therefore be invariant under λ↦cλ cλ; otherwise the attack can lower the loss merely by rescaling the multipliers (Lemma A.7). The second rescaling, ↦c θ c\, θ, must be treated differently because the attacker cannot rescale the given θ (Lemma A.1, Appendix A). A layerwise Euler identity. Blockwise homogeneity ties every residual block to one shared scalar, for any candidate and not only at solutions. Lemma 3.3 (Layerwise Euler identity). Under Assumption 2.1, for every z and every block k, ⟨Wk(z),Wk⟩=dkΓ(z),Γ(z):=∑i=1mλiyif(i,). r_W_k(z),W_k =d_k\, (z), (z):= _i=1^m _iy_if(x_i; θ). (7) If moreover (z)∈C∘() r(z)∈ C ( θ), then Γ(z)=⟨(z),⟩/L>0 (z)= r(z), θ /L>0, hence Wk(z)≠ r_W_k(z) 0 and Wk≠W_k 0 for every k; for a solution of (5), Γ(z)=∥/2L (z)=\| θ\|^2/L. For the max-type norms of Muon and Adam, this identity implies that every block is active at a KKT direction (Corollary A.2, Appendix A). From cone membership to a loss. Membership in C()C( θ) is equivalent to tightness in the generalized Cauchy–Schwarz inequality (1), so we penalize the relative Cauchy–Schwarz gap block by block, with block norms ∥⋅∥(k)\|·\|_(k) and their duals, Lalign=∑k=1K(1−⟨Wk,Wk⟩‖Wk‖(k),⋆‖Wk‖(k))2,defined for rWk≠0 for every k. splitL_align&= _k=1^K (1- r_W_k,W_k \| r_W_k\|_(k), \,\|W_k\|_(k) )^2,\\ & for r_W_k≠ 0 for every k. split (8) Each term lies in [0,4][0,4] and vanishes iff Wk r_W_k is dual-aligned with WkW_k, i.e. Wk∈Ck()∖ r_W_k∈ C_k( θ) \0\, where Ck():=cone∂∥Wk∥(k)C_k( θ):= ∂\|W_k\|_(k). The numerator and denominator are positively homogeneous of degree one in each argument, so the loss is invariant under the gauge of Lemma 3.2 and requires no choice of singular vectors. By Lemma 3.3, Wk≠ r_W_k 0 in every block for any cone-feasible z, so this restriction discards no valid solution (for numerical stability we add a small ϵε to the denominator). Plugging in the appropriate ∥⋅∥(k)\|·\|_(k) covers all three attacks with one formulation. Proposition 3.4 (Per-block alignment is exact). Let Assumption 2.1 hold, let ∥⋅∥\|·\| be one of the norms in (2) (Muon with arbitrary ck>0c_k>0), and let (5) have a solution at θ. Then z:(z)∈C∘() \z:\ r(z)∈ C ( θ)\ =z:Wk(z)∈Ck()∖∀k =\z:\ r_W_k(z)∈ C_k( θ) \0\\ \ ∀ k\ =z:Lalign(z)=0. =\z:\ L_align(z)=0\. In particular, the zero set of LalignL_align does not depend on the Muon weights ckc_k. Hence no cross-block coupling term is needed (Remark A.3, Appendix A). To mount the attack we set Lstat:=LalignL_stat:=L_align in (3), keeping LλL_λ and LpriorL_prior unchanged. For ℓ2 _2, Proposition 3.4 gives Lstat=0L_stat=0 iff =c r=c\, θ for a shared c>0c>0, so up to the gauge of Lemma 3.2 the Euclidean case recovers the objective of Haim et al. (2022). 3.3 The Muon (spectral) objective Muon performs steepest descent under the spectral norm: for each block W, ∥⋅∥=∥⋅∥sp=σ1(⋅)\|·\|=\|·\|_sp= _1(·), with dual ∥⋅∥nuc\|·\|_nuc. Its subdifferential is a spectrahedron rather than a point: if the top singular value has multiplicity ρ, with left and right singular subspaces spanned by the columns of U1∈ℝm×ρU_1 ^m×ρ and V1∈ℝn×ρV_1 ^n×ρ, then ∂∥W∥sp=U1MV1⊤:M⪰0,trM=1,∂\|W\|_sp=\\,U_1MV_1 :\ M 0,\ trM=1\,\, (9) which reduces to the rank-one ⊤pq only when ρ=1ρ=1. A natural first instinct is to keep the template of (3) with a spectral target, minimizing the “single-target” loss ‖⋆−W‖22\|k - r_W\|_2^2 for ⋆=∇(12‖W‖sp2)=σ1⊤k =∇( 12\|W\|_sp^2)= _1\,pq . This breaks down for two reasons: within a block, a repeated top singular value makes the target basis-dependent, and across blocks, the relative block scales it fixes need not match those realized at a KKT point (Lemma A.7, Appendix A.8). The Muon instance of (8), in which ckc_k cancels, reads LalignMuon=∑k(1−⟨Wk,Wk⟩‖Wk‖nuc‖Wk‖sp)2.L_align^Muon= _k (1- r_W_k,W_k \| r_W_k\|_nuc\,\|W_k\|_sp )^2. (10) Since every block attains the maximum at a KKT direction (Corollary A.2), the extra tightness condition of the max-type norm is vacuous (Lemma B.3) and (10) is exact for arbitrary per-block weights (Proposition 3.4): no balancing assumption is needed, and every layer carries information. 3.4 The Adam (ℓ∞ _∞) objective Adam with a negligible stability constant maximizes the global ℓ∞ _∞ margin, so ‖=‖∞\| θ\|=\| θ\|_∞ with dual ∥⋅∥1\|·\|_1. With A=argmaxj|θj|A= _j| _j| and s=|A|s=|A|, Hölder tightness holds iff j=0 r_j=0 for j∉Aj∉ A and jθj≥0 r_j _j≥ 0 for j∈Aj∈ A, with ≠0 r≠ 0: the candidate gradient combination must be supported on the largest-magnitude weights with matching signs, and nothing else is asked of it. We use the single global term (1−⟨,⟩/(‖1‖∞))2(1- r, θ /(\| r\|_1\| θ\|_∞))^2 (11) (a per-tensor variant is discussed in Appendix A.11). Why this target is substantially less weight-specific is the subject of the next section. 4 How much do the Optimizers Leak? A Specificity Analysis The attack seeks z whose residual lies in C∘()C ( θ), and succeeds to the extent that this forces the candidates to be the training set. Counting the scalar equations that ∈C() r∈ C( θ) imposes at a generic θ gives Θ(p) (p) for all three geometries and cannot explain the ordering (Appendix A.4); we instead ask how much of θ the optimizer-dependent target C()C( θ) encodes. 4.1 Weight-specificity and target-cone variability Counting tells us how many equations there are, not how much they say about the weights. The constraint references the weights through the residual map z↦(z,)z r(z; θ), fixed by the architecture and θ, and through the target C()C( θ), the only place where the optimizer enters. Definition 4.1 (Weight-specificity). For a margin norm ∥⋅∥\|·\| and a stratum Σ , the weight-specificity SΣ(∥⋅∥)S_ (\|·\|) is the dimension of the image of ↦C() θ C( θ) restricted to the points of Σ at which all blocks are active, i.e. the number of independent continuous functionals of θ that the stationarity target determines. Specificity is a property of the geometry and the stratum, not of a dataset: it counts the continuous degrees of freedom through which the target cone varies with the weights, and is not by itself an information-theoretic bound on input identifiability. Lemma 4.2 (Specificity of the three geometries). On the stratum with multiplicities ρk _k and arg-max set A, at points where all blocks are active, S=p−1,ℓ2,∑kSk(ρk),spectral,0,ℓ∞,S= casesp-1,& _2,\\ _kS_k( _k),&spectral,\\ 0,& _∞, cases (12) since C is the ray through θ, determined by /‖2 θ/\| θ\|_2 (ℓ2 _2); Ck=U1,kMV1,k⊤:M⪰0C_k=\U_1,kMV_1,k :M 0\, determined by U1,kV1,k⊤U_1,kV_1,k (spectral); and C=:j=0 for j∉A,jθj≥0 for j∈AC=\ r: r_j=0 for j∉ A,\ r_j _j≥ 0 for j∈ A\ (ℓ∞ _∞). Here Sk(ρ):=ρ(mk+nk−ρ)−ρ(ρ+1)/2S_k(ρ):=ρ(m_k+n_k-ρ)-ρ(ρ+1)/2 is the dimension of the manifold of rank-ρ partial isometries in ℝmk×nkR^m_k× n_k; Sk(1)=mk+nk−2S_k(1)=m_k+n_k-2. In the ℓ∞ _∞ case C depends on ≠ θ 0 only through the discrete data (A,signθ|A)(A,signθ|_A), for every multiplicity s=|A|s=|A|. 4.2 Consequences, and why Adam resists For SGD the cone is the ray through θ, so the constraint encodes the full direction /‖2 θ/\| θ\|_2: the candidate gradients must reproduce the entire weight vector up to one scalar. This is the maximally specific geometry, and the regime in which Haim et al. (2022) reconstruct. For Muon, CkC_k encodes the top singular structure U1,kV1,k⊤U_1,kV_1,k of every block (all are active at a KKT direction), i.e. Sk(ρk)S_k( _k) numbers per layer, at least Θ(mk+nk) (m_k+n_k) (Appendix A.9). That is far less than ℓ2 _2, but the constraint stays tight: by Lemma A.4 many of each block’s equations survive at any multiplicity in ordinary matrix-block regimes, and it is attached to a weight-specific template that grows with width, so Muon both constrains candidates sharply and carries a genuine data fingerprint. For Adam, with A=argmaxl|θl|A= _l| _l|, ∂(12‖∞2)=‖∞convsign(θj)j:j∈A,∂ ( 12\| θ\|_∞^2 )=\| θ\|_∞\, \sign( _j)e_j:\ j∈ A\, (13) an (s−1)(s-1)-simplex whose interior directions are unconstrained: the condition pins the support and admissible signs of r but never the relative magnitudes of θ. Hence S=0S=0 (cf. Remark A.8). For networks the residual map carries θ-dependence of its own, but the polyhedral geometry has a second consequence that does extend: minimizing ‖∞\| θ\|_∞ under margin constraints drives θ onto a large face of the ℓ∞ _∞ ball, where the constraint has few normal equations. Under a first-order nondegeneracy condition (Assumption A.9), Theorem A.10 (Appendix A.10) shows that codimC()=p−s≤nI C( θ)=p-s≤ n_I, where nIn_I is the number of on-margin examples. Adam’s stationarity system thus has at most nIn_I normal equations for nI(d+1)n_I(d+1) unknowns and, when nId≥2n_Id≥ 2, fails regular first-order identifiability; under additional constant-rank and relative-interior conditions, the stationarity-consistent perturbations of the on-margin candidates form a C1C^1 manifold of dimension at least nIdn_Id. The result concerns the idealized ℓ∞ _∞ bias (Adam with a negligible stability constant) and this stationarity-only attack family only, so it is not a differential-privacy-style guarantee (Abadi et al., 2016). 4.3 Two axes, and the vulnerability ordering Specificity concerns a converged model, but the attack inverts a condition that holds only in the limit, so a second axis matters: reachability, how quickly an optimizer’s parameter direction approaches the KKT direction of its own margin problem. Muon is high on both axes: its specificity is at least Θ(width) (width) in every Muon-trained layer, and per-block orthogonalization escapes the small-initialization stall that traps gradient descent. SGD has the highest specificity, p−1p-1, but the slowest reachability (Haim et al., 2022 train ∼106 10^6 iterations): vulnerable in principle, expensive in practice. Adam has fast reachability but zero continuous target specificity and, at nondegenerate KKT points, too few equations for regular first-order identification (Theorem A.10). Folding in cost, the practical ordering is Muon≻SGD≻Adam. \ Muon\ \ SGD\ \ Adam.\ The Muon order is a prediction from reachability, not a theorem: that gap is about training cost and should narrow wherever SGD has budget to converge, while the Adam gap reflects weaker continuous target dependence and regular underdetermination (summary in Table 3, Appendix B). (a) Muon, binary (b) SGD, binary (c) Adam, binary Figure 2: Per-sample SSIM versus training margin γi _i on CIFAR-10 (n=500n=500, range prior), for binary models. Each point is a training sample, colored by class; the dashed line marks SSIM=0.4SSIM=0.4. Recognizable reconstructions occur near the minimal margin of their class. Muon produces most of them, and Adam essentially none. 5 Experiments We evaluate the attack of Section 3 on networks trained with SGD, Adam, and Muon and test the predictions of Section 4. Implementation details are in Appendix D and full results in Appendix E. Setup. Unless stated otherwise, we use n=500n=500 examples from binary MNIST LeCun et al. (2010) parity, binary CIFAR-10 Krizhevsky et al. (2009) vehicles versus animals, and 10-class CIFAR-10. Models are fully connected ReLU networks d−Cd\!-\!1000\!-\!1000\!-\!C (C=1C=1 binary, C=10C=10 multiclass), with only a first-layer bias, absorbed into W1W_1, so Assumption 2.1 holds. SGD and Adam update all layers; Muon applies five-step Newton–Schulz Muon (Jordan and others, 2024) to hidden-to-hidden matrices and SGD to the input and output layers. We use the binary SGD models of Haim et al. (2022). Models trained by Adam and Muon reach training loss O(10−7)O(10^-7) within 10410^4 iterations, whereas to reach the same order of training loss, SGD requires 10610^6 iterations (Table 4, Appendix D). Attack and evaluation. The attack uses the Euclidean loss (3) for SGD, and the global ℓ∞ _∞ alignment loss (§3.4) for Adam, and a smooth relaxation of (10) for Muon. It optimizes m=2nm=2n candidates with fixed balanced labels and their multipliers; the multiclass extension is described in Appendix D. Unless stated otherwise, no image prior is used. Reconstructions are matched one-to-one to training images by the Hungarian algorithm on centered, normalized ℓ2 _2 distance (more details in Appendix D). We evaluate SSIM@30 as our primary leakage metric. All-pairs averages can favor Adam because its candidates are generic images; @30 measures whether individual samples are recovered. We report SSIM, PSNR, and ℓ1 _1 and ℓ2 _2 distances over all pairs and the best 3030 pairs (@30); SSIM>0.4SSIM>0.4 is considered visually recognizable. 5.1 Main Results Muon-trained models are the most vulnerable. Muon leads to the best reconstructions on every metric and task as visually illustrated for the binary classification setup in Figure 1. Its SSIM@30 is 0.4310.431 on MNIST, 0.5510.551 on binary CIFAR-10 and 0.5340.534 on 10-class CIFAR-10, against 0.3460.346, 0.2290.229 and 0.3830.383 for SGD (the 10-class gap is discussed in Appendix E). The gap is largest on binary CIFAR-10, where Muon more than doubles the SGD score. Since no prior is used, these reconstructions come from the stationarity condition alone, consistent with the fact that the spectral objective (10) inverts the Muon condition and that the per-layer spectral template of Lemma 4.2 is exploitable, with two orders of magnitude fewer training iterations than SGD. This is confirmed by the sorted SSIM curves without priors across both CIFAR-10 and MNIST (Figure 6). Adam-trained models are the most resistant. Adam has the lowest SSIM@30 on every task without a prior, at most 0.2490.249, well below the 0.40.4 recognizability level, consistent with the analysis of §4.2 : Adam’s ℓ∞ _∞ target carries no continuous information about the weight magnitudes, leaving the stationarity system too weakly constrained to identify individual samples. Ablation Studies: The effect of image priors. We add five regularizers to the attack objective: a range prior penalizing pixel values outside [0,1][0,1], total variation (TV), an ℓ1 _1 penalty favoring low-energy backgrounds, TV+ℓ1+ _1, and all three combined. As the only optimizer-independent component of the objective, the prior’s marginal effect measures how much of a reconstruction does not come from the weights (Table 1; full results in Tables 5-7), consistent with recent findings on how priors can dominate checkpoint reconstruction fidelity Refael et al. (2026b). Qualitative comparisons showing reconstructions with and without priors are provided in Figure 3 for MNIST and Figure 5 for CIFAR-10. Table 1: SSIM@30 (↑ ) under different image priors (n=500n=500). Best per task and prior in bold, second best underlined. Muon is best in all 1818 settings, and in all 144144 entries of the full tables (Appendix E). MNIST (binary) CIFAR-10 (binary) CIFAR-10 (10-class) Prior SGD Adam Muon SGD Adam Muon SGD Adam Muon None 0.3463 0.2487 0.4310 0.2287 0.1379 0.5508 0.3827 0.2066 0.5339 Range 0.4057 0.2994 0.6371 0.2336 0.1367 0.6130 0.4515 0.1771 0.5315 TV 0.3768 0.2882 0.5808 0.2574 0.2115 0.6132 0.4332 0.3442 0.5481 ℓ1 _1 0.3614 0.1752 0.5302 0.2375 0.2112 0.5598 0.4141 0.2739 0.5371 TV+ℓ1+ _1 0.3977 0.2311 0.6211 0.2440 0.2369 0.6286 0.4343 0.3050 0.5890 All 0.4511 0.2655 0.6599 0.2536 0.2377 0.6189 0.4343 0.2603 0.5795 Priors do not change the leading optimizer: Muon is best under every prior (SSIM@30 between 0.4310.431 and 0.6600.660), while Adam never exceeds 0.3440.344. On MNIST, combining all priors raises SSIM@30 by 0.2290.229 for Muon and 0.1050.105 for SGD but only 0.0170.017 for Adam: priors sharpen reconstructions when the stationarity condition already constrains them. On binary CIFAR-10, smoothness priors help Adam most (from 0.1380.138 to 0.2380.238 with all priors). We attribute this to smoothness priors making generic natural-image candidates more similar to the training set, rather than to identification of specific samples, and to SGD not having reached its limiting direction on this task (Table 4). Effect of sample size and architecture. To remove the training-budget asymmetry, we retrain all models on binary CIFAR-10 to a training loss below 10−810^-8, varying n∈50,100,500,1000n∈\50,100,500,1000\ or the hidden widths, and attack them with the range prior (Table 2). Table 2: Effect of sample size (top, −10001000\!-\!1000 hidden units) and architecture (bottom, n=500n=500) on binary CIFAR-10. All models are trained to a training loss below 10−810^-8; the attack uses the range prior. Best per configuration in bold. Configuration Optimizer SSIM↑ SSIM@30↑ PSNR↑ PSNR@30↑ ℓ1↓ _1 ℓ1 _1@30↓ ℓ2↓ _2 ℓ2 _2@30↓ n=50n=50 SGD 0.3024 0.4486 13.188 15.066 0.1586 0.1250 0.2191 0.1765 Adam 0.1452 0.1809 11.733 12.663 0.1989 0.1808 0.2590 0.2327 Muon 0.4070 0.5848 14.409 17.390 0.1353 0.0954 0.1904 0.1351 n=100n=100 SGD 0.2949 0.6232 13.499 18.765 0.1554 0.0828 0.2114 0.1153 Adam 0.0976 0.1488 11.214 12.570 0.2087 0.1775 0.2750 0.2352 Muon 0.3501 0.6984 14.257 21.465 0.1404 0.0639 0.1937 0.0845 n=500n=500 SGD 0.0994 0.2336 11.665 13.435 0.2027 0.1634 0.2611 0.2129 Adam 0.1299 0.1367 12.334 13.441 0.1863 0.1624 0.2417 0.2128 Muon 0.1921 0.613 12.8084 19.1901 0.175 0.0825 0.2289 0.1098 n=1000n=1000 SGD 0.0942 0.3001 11.537 14.168 0.2059 0.1476 0.2650 0.1957 Adam 0.0783 0.1202 11.150 11.956 0.2149 0.1968 0.2770 0.2525 Muon 0.1442 0.4186 12.203 15.559 0.1895 0.1252 0.2454 0.1667 −100100\!-\!100 SGD 0.1132 0.1754 11.878 13.373 0.1975 0.1634 0.2547 0.2145 Adam 0.0569 0.0836 10.795 12.129 0.2242 0.1912 0.2886 0.2475 Muon 0.1195 0.1826 11.921 14.001 0.1957 0.1598 0.2535 0.2039 −1001000\!-\!500\!-\!100 SGD 0.1654 0.4170 12.577 15.990 0.1808 0.1197 0.2351 0.1587 Adam 0.0927 0.1226 11.489 12.318 0.2065 0.1836 0.2664 0.2422 Muon 0.1689 0.4234 12.832 16.182 0.1719 0.1170 0.2263 0.1552 −501000\!-\!500\!-\!100\!-\!50 SGD 0.1617 0.3944 12.594 15.882 0.1804 0.1196 0.2346 0.1607 Adam 0.1181 0.1546 11.944 12.736 0.1950 0.1741 0.2528 0.2308 Muon 0.1896 0.4236 13.5685 17.962 0.1622 0.1085 0.2192 0.1396 Across sample sizes, Muon is best on every metric and Adam has the lowest SSIM@30. Once SGD is trained to convergence, however, it becomes competitive at small n: at n=100n=100 its SSIM@30 is 0.6230.623 against 0.6980.698 for Muon, compared with 0.2340.234 against 0.6130.613 at n=500n=500. The Muon gap is therefore partly a gap in training cost, as predicted by the reachability argument of §4.3. Adam is worst on every metric for every architecture, with SSIM@30 below half that of the better optimizer. The Adam separation is thus robust and the Muon comparison regime-dependent, as the theory suggests: Adam’s weakness follows from the structure of the ℓ∞ _∞ target cone, whereas the comparison between the ℓ2 _2 and spectral geometries depends on the residual map and the training budget. Which Samples Are Reconstructed? By complementary slackness, only samples on the margin carry positive multipliers and enter the stationarity condition. Figure 2 tests this by plotting, for each training sample, the SSIM of its matched reconstruction against its margin γi _i. In both the binary and multiclass settings (Figure 4), recognizable reconstructions occur, for every optimizer, only for samples at or near the minimal margin, and samples well inside the margin are never recovered. Muon produces by far the most samples above the threshold, many concentrated at the minimal margin (a narrow vertical band); SGD produces a handful and Adam essentially none. Being on the margin is necessary for recovery under every geometry, but it is sufficient only when the stationarity condition carries weight-specific information, which the ℓ∞ _∞ condition does not. 6 Discussion Our results identify optimizer implicit bias as an overlooked axis of training-data privacy. Even with identical architectures, data, and interpolation behavior, optimizers leave distinct geometric fingerprints: Muon’s spectral bias tightly constrains candidate inputs and yields the strongest reconstructions, SGD leaks less, and Adam is most resistant. This theoretically predicted ordering—driven by the specificity and reachability of stationarity constraints—shows that optimizer choice can serve as a practical privacy control, not merely an optimization decision. Although our checkpoint-only, stationarity-based analysis provides no formal privacy guarantee, it demonstrates that evaluating model privacy independently of the training algorithm can overlook a consequential source of leakage. Finally, while our study focuses on standard implementations of SGD, Adam, and Muon, modern training pipelines increasingly adopt memory-efficient, low-rank, and orthogonalized optimizers. Extending geometric implicit-bias analyses to methods such as AdaRankGrad Refael et al. (2025b), SUMO Refael et al. (2025a), Refael et al. (2026a), and Svirsky et al. (2026) presents a promising direction to understand how adaptive rank reduction and subspace orthogonalization govern training-data leakage. AI use statement In this work, we used generative AI tools for text polishing and obtaining suggestions on document organization and section structure. We have not used generative AI tools for research idea generation, code development, mathematical derivations, or experimental data generation/analysis, and all other disclosure categories are not applicable to this work. We have reviewed all AI-assisted work. Specifically, all AI-suggested structural edits and polished text were thoroughly examined, manually refined, and verified for technical accuracy and consistency by the authors. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI. References Abadi et al. (2016) M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang Deep learning with differential privacy. In ACM SIGSAC Conference on Computer and Communications Security (CCS), Cited by: §A.7, §F.2, §4.2. Battash et al. (2024) B. Battash, L. Wolf, and O. Lindenbaum Revisiting the noise model of stochastic gradient descent. In International Conference on Artificial Intelligence and Statistics, p. 4780–4788. Cited by: Appendix D. Buzaglo et al. (2023) G. Buzaglo, N. Haim, G. Yehudai, G. Vardi, Y. Oz, Y. Nikankin, and M. Irani Deconstructing data reconstruction. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Appendix D, §1, §2. Cortes and Vapnik (1995) C. Cortes and V. Vapnik Support-vector networks. Machine Learning 20 (3), p. 273–297. External Links: Document Cited by: Theorem F.1. Fan et al. (2025) C. Fan, M. Schmidt, and C. Thrampoulidis Implicit bias of spectral descent and muon on multiclass separable data. External Links: 2502.04664 Cited by: §2. Feldman (2020) V. Feldman Does learning require memorization? a short tale about a long tail. In ACM Symposium on Theory of Computing (STOC), Cited by: §1. Fredrikson et al. (2015) M. Fredrikson, S. Jha, and T. Ristenpart Model inversion attacks that exploit confidence information and basic countermeasures. In ACM SIGSAC Conference on Computer and Communications Security (CCS), Cited by: §F.2. Golbari et al. (2025) Y. Golbari, N. Wasserman, G. Vardi, and M. Irani ImpMIA: leveraging implicit bias for membership inference attack. External Links: 2510.10625 Cited by: §2. Gronich and Vardi (2026) E. Gronich and G. Vardi The implicit bias of adam and muon on smooth homogeneous neural networks. In International Conference on Machine Learning (ICML), External Links: 2602.16340 Cited by: §F.1, §1, §2, §2.1, §2.1, §2.1. Gunasekar et al. (2018) S. Gunasekar, J. Lee, D. Soudry, and N. Srebro Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning (ICML), Cited by: §F.4. Haim et al. (2022) N. Haim, G. Vardi, G. Yehudai, O. Shamir, and M. Irani Reconstructing training data from trained neural networks. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2206.07758 Cited by: §A.1, Appendix D, Appendix D, Appendix D, §F.1, §1, §1, §1, §1, §2, §2.1, §2.1, §2.1, §3.2, §4.2, §4.3, §5, Abstract, footnote 1. He et al. (2015) K. He, X. Zhang, S. Ren, and J. Sun Delving deep into rectifiers: surpassing human-level performance on imagenet classification. External Links: 1502.01852, Link Cited by: Appendix D. Ji and Telgarsky (2020) Z. Ji and M. Telgarsky Directional convergence and alignment in deep learning. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 17176–17186. Cited by: §F.4, Theorem F.1, §1. Jordan et al. (2024) K. Jordan et al. Muon: an optimizer for hidden layers in neural networks. Cited by: §1, §5. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), Cited by: §1. Krizhevsky et al. (2009) A. Krizhevsky, V. Nair, and G. Hinton Learning multiple layers of features from tiny images. Technical report University of Toronto. External Links: Link Cited by: §5. LeCun et al. (2010) Y. LeCun, C. Cortes, and C. Burges MNIST handwritten digit database. ATT Labs [Online] 2. Cited by: §5. Lyu and Li (2020) K. Lyu and J. Li Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, Cited by: §F.4, Theorem F.1, §1. Oz et al. (2024) Y. Oz, G. Yehudai, G. Vardi, I. Antebi, M. Irani, and N. Haim Reconstructing training data from real-world models trained with transfer learning. External Links: 2407.15845 Cited by: §2. Refael et al. (2026a) Y. Refael, I. Arbel, O. Lindenbaum, and T. Tirer Lorenza: enhancing generalization in low-rank gradient LLM training via efficient zeroth-order adaptive SAM. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §6. Refael et al. (2026b) Y. Refael, G. Smorodinsky, O. Lindenbaum, and I. Safran No prior, no leakage: revisiting reconstruction attacks in trained neural networks. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2, §5.1. Refael et al. (2025a) Y. Refael, G. Smorodinsky, T. Tirer, and O. Lindenbaum SUMO: subspace-aware moment-orthogonalization for accelerating memory-efficient LLM training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §6. Refael et al. (2025b) Y. Refael, J. Svirsky, B. Shustin, W. Huleihel, and O. Lindenbaum AdaRankGrad: adaptive gradient rank and moments for memory-efficient LLMs training and fine-tuning. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §6. Shokri et al. (2017) R. Shokri, M. Stronati, C. Song, and V. Shmatikov Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy (S&P), Cited by: §F.2. Soudry et al. (2018) D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro The implicit bias of gradient descent on separable data. Journal of Machine Learning Research (JMLR). Cited by: §F.4. Svirsky et al. (2026) J. Svirsky, Y. Refael, and O. Lindenbaum Train less, infer faster: efficient model finetuning and compression via structured sparsity. In The 29th International Conference on Artificial Intelligence and Statistics, External Links: Link Cited by: §6. Vardi (2023) G. Vardi On the implicit bias in deep-learning algorithms. Communications of the ACM 66 (6). Cited by: §F.4. Wang et al. (2021) B. Wang, Q. Meng, W. Chen, and T.-Y. Liu The implicit bias for adaptive optimization algorithms on homogeneous neural networks. In International Conference on Machine Learning (ICML), Cited by: §2. Watson (1992) G. A. Watson Characterization of the subdifferential of some matrix norms. Linear Algebra and its Applications 170, p. 33–45. Cited by: §B.7. Zhang et al. (2017) C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR), Cited by: §1. Zhang et al. (2024) C. Zhang, D. Zou, and Y. Cao The implicit bias of adam on separable data. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Zhu et al. (2019) L. Zhu, Z. Liu, and S. Han Deep leakage from gradients. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §F.2. Supplementary Materials Appendix A Additional Analysis of Reconstruction Objectives and Privacy Leakage A.1 Scale covariance in the trained weights (from §3.2) Besides the multiplier gauge of Lemma 3.2, there is another rescaling invariance, on the weights, that should be treated differently. Lemma A.1 (Scale covariance in θ). If z=(i,λi)z=(\x_i\,\ _i\) satisfies (5) at θ, then for every c>0c>0 the pair (i,c 2−Lλi)(\x_i\,\c^\,2-L _i\) satisfies it at cc\, θ. In line with Haim et al. (2022), Lemma A.1 justifies disregarding the norm of θ in the homogeneous stationarity equation: every θ on the ray remains a valid stationary point up to a rescaling of the multipliers. Importantly, this invariance does not preserve primal feasibility or complementary slackness under the rigid margin constraint “≥1≥ 1” in (P). A.2 Every block is active (from §3.2) For the max-type norms governing Muon and Adam, the Euler identity (Lemma 3.3) enforces a direct structural constraint. Corollary A.2 (Every block is active at a KKT direction). Let Assumption 2.1 hold and let (5) have a solution at θ. (a) For the Muon norm of (2), every block attains the maximum: ∥Wk∥sp/ck=∥\|W_k\|_sp/c_k=\| θ\| for all k. (b) For the Adam norm, every block contains a globally maximal coordinate: A∩Wk≠∅A∩ W_k≠ and ∥Wk∥∞=∥∞\|W_k\|_∞=\| θ\|_∞ for all k, where A=argmaxj|θj|A= _j| _j|. A.3 Why per-block alignment loses nothing (from §3.2) At first glance, optimizing a per-block loss appears to relax the global condition. Under an ℓ2 _2 norm, for example, it permits Wk=tkWk r_W_k=t_kW_k using a distinct scalar tk>0t_k>0 for each block. For max-type norms, it seemingly overlooks which specific blocks actually attain the maximum. However, the Euler identity proves that at the exact weights an attacker encounters in a real scenario, this per-block approach remains completely exact (Proposition 3.4). Remark A.3 (No coupling term is needed). One might add Lcouple=Vark[log(∥Wk∥/(k),⋆∥Wk∥)(k)]L_couple=Var_k [ (\| r_W_k\|_(k), /\|W_k\|_(k)) ] to restore the relative block scales that a per-block loss seems to drop. On the zero set of LalignL_align, (7) gives ∥Wk∥(k),⋆Wk∥=(k)⟨Wk,Wk⟩=dkΓ(z)\| r_W_k\|_(k), \|W_k\|_(k)= r_W_k,W_k =d_k (z), so Lcouple=Vark[log(dk/∥Wk∥)2(k)]L_couple=Var_k [ (d_k/\|W_k\|_(k)^2) ] is a function of θ alone. It therefore cannot discriminate between candidates satisfying Lalign=0L_align=0, and we do not use it. A.4 Equation counting at a generic weight vector does not separate the geometries The constraint ∈C() r∈ C( θ) imposes codimC() C( θ) scalar equalities, plus sign constraints. By Lemma B.4 and Corollary A.2, at a KKT direction on the stratum of θ, codimC()=p−1,ℓ2(C=ray through ),∑k(mknk−ρk(ρk+1)/2),spectral(C=∏kU1,kMV1,k⊤:M⪰0),p−s,ℓ∞(C=simplicial cone on A,s=|A|). C( θ)= casesp-1,& _2\ (C=ray through θ),\\ _k (m_kn_k- _k( _k+1)/2 ),&spectral\ (C= _k\U_1,kMV_1,k :M 0\),\\ p-s,& _∞\ (C=simplicial cone on A,\ s=|A|). cases (14) At a generic θ (ρk=1 _k=1, s=1s=1) these are p−1p-1, p−Kp-K and p−1p-1, all Θ(p) (p) in the usual nontrivial matrix-block regime, so counting at a generic weight vector explains no ordering. But KKT points are not generic, and the geometries respond very differently to degeneracy. For the spectral norm, higher multiplicity costs few equations: Lemma A.4 (The spectral constraint cannot collapse). For every θ and all multiplicities, with rk=min(mk,nk)r_k= (m_k,n_k), ∑k(mknk−ρk(ρk+1)/2)≥12(p−∑krk). _k (m_kn_k- _k( _k+1)/2 )\ ≥\ 12 (p- _kr_k ). For ℓ∞ _∞, in contrast, p−sp-s can be as small as 00, and §4.2 shows that at the KKT points Adam reaches it is at most the number of on-margin examples. The relevant comparison is with the unknowns. Each on-margin candidate contributes d+1d+1 of them (input and multiplier), and the true data can satisfy the regular first-order identifiability criterion only if codimC()≥nI(d+1)−1 C( θ)≥ n_I(d+1)-1, where nIn_I is the number of on-margin examples (Lemma C.2). This is necessary, not sufficient: off-margin candidates contribute nothing (λi=0 _i=0), the prior and λ≥0λ≥ 0 add restrictions, and recovering a subset of the points needs no unique global solution. The ℓ2 _2 and spectral geometries meet it with room to spare in the over-parameterized regime p≫ndp nd, and it is exactly what fails for ℓ∞ _∞. A.5 Target-cone sufficiency for linear-in-parameter models (from §4.1) When the residual map does not depend on θ, the checkpoint affects the stationarity-feasible set only through the target cone. This occurs for models linear in their parameters, f(,)=⟨,ϕ()⟩f(x; θ)= θ,φ(x) for a fixed feature map ϕφ (random-feature, kernel and linearized-network models), where (z)=∑iλiyiϕ(i) r(z)= _i _iy_iφ(x_i). Lemma A.5 (Target-cone sufficiency for linear-in-parameter models). Let f(,)=⟨,ϕ()⟩f(x; θ)= θ,φ(x) . If C()=C(′)C( θ)=C( θ ), then the stationarity-feasible sets z:(z)∈C∘()andz:(z)∈C∘(′)\z: r(z)∈ C ( θ)\ \z: r(z)∈ C ( θ )\ coincide, as do the zero sets of the corresponding global alignment losses. Hence, within the stationarity-only formulation, the checkpoint affects feasibility only through C()C( θ). On each stratum these cones vary in an S-dimensional continuous family. This invariance does not alone imply non-identifiability of the inputs, which also depends on the residual map ϕφ and on the codimension of the cone. For such models C()C( θ) is a sufficient statistic of the weights for this stationarity-only formulation. For networks, (z,)=∑iλiyi∇f(i,) r(z; θ)= _i _iy_i _ θf(x_i; θ) depends on θ as well, so two networks with equal cones but different weights have different residual maps and C()C( θ) is no longer sufficient. Specificity then measures only the part of the constraint that the optimizer controls; the network-level regular first-order result is Theorem A.10. Attackers with extra information (checkpoints, query access, a strong generative prior) are outside the scope of either. A.6 Stratum invariance under ℓ∞ _∞ (from §4.2) Proposition A.6 (Stratum invariance under ℓ∞ _∞). Let f(,)=⟨,ϕ()⟩f(x; θ)= θ,φ(x) . The stationarity-feasible set for the Adam geometry depends on θ only through (A,sign|A)(A,sign θ|_A). Consequently, checkpoints sharing this arg-max/sign pattern induce exactly the same stationarity equations and the same feasible candidate datasets. Thus the target contributes no continuous dependence on the weight magnitudes within a fixed stratum. This invariance does not by itself imply non-identifiability of the inputs, since the equations (z)∈C() r(z)∈ C( θ) may still constrain or identify z through the feature map ϕφ. For these models Adam’s target cone transmits an O(p)O(p)-bit combinatorial arg-max/sign pattern but no continuous information about the weight magnitudes within a fixed stratum. This makes the target less weight-specific, although the resulting stationarity equations may still constrain the inputs through the feature map. A.7 Discussion of Theorem A.10 (from §4.2) On Assumption A.9: The span condition says that θ is first-order isolated given its active set: no direction moves θ while preserving all tied maxima and all active margins to first order. For linear-in-parameter models it holds whenever the ℓ∞ _∞ max-margin linear program has a unique solution, since a unique LP solution is a vertex (Remark B.5). It is checkable on a trained model through its consequence s≥p−nIs≥ p-n_I. Thus, under Assumption A.9, Adam’s stationarity system has at most nIn_I normal equations for nI(d+1)n_I(d+1) unknowns and fails regular first-order identifiability. Under the additional constant-rank and relative-interior hypotheses of part (i), this first-order deficiency becomes an actual positive-dimensional family of stationarity-consistent candidates, whose input projection has dimension at least nI(d−1)n_I(d-1). Without those additional hypotheses, the rank count alone does not exclude an isolated solution. These conclusions concern the stationarity-only attack, not the full KKT system or other reconstruction attacks. The ℓ2 _2 and spectral constraints have many more normal equations in ordinary over-parameterized matrix-block regimes (Lemma A.4). Three caveats remain. The results concern the idealized ℓ∞ _∞ bias, i.e. Adam with a negligible stability constant. The network statement rests on Assumption A.9, a property of the trained model, and actual local ambiguity additionally rests on the hypotheses of Theorem A.10 (i). Finally, they cover this attack family only, so they are not a differential-privacy-style guarantee (Abadi et al., 2016). Specificity and Theorem A.10 explain why Adam is less exposed to this stationarity-only attack, but they do not show that ℓ2 _2 or spectral geometry succeeds. Regular first-order well-posedness in the attacker’s variables is formalized in Appendix C as a rank condition on the Jacobian of the constrained residual at z⋆z . Of the resulting Conjecture C.4, the Adam failure of regular first-order identifiability follows from Theorem A.10 (Proposition C.3); for the positive half we prove only the necessary condition codimC()≥nI(d+1)−1 C( θ)≥ n_I(d+1)-1, which the ℓ2 _2 and spectral geometries satisfy in the stated over-parameterized regime. A.8 Set-valued target and collapse (from §3.3) Degeneracy within blocks: spectral-margin maximization drives blocks toward repeated top singular values, and then ,p,q are defined only up to rotation within the corresponding top singular subspaces U1,V1U_1,V_1. Thus the single-target construction is basis-dependent and is not selected by the KKT condition. Degeneracy across blocks: by Corollary A.2 every block attains the maximum at a KKT direction, so ∂(12‖2)∂( 12\| θ\|^2) is set-valued even when every ρk=1 _k=1. It contains a (K−1)(K-1)-simplex of relative block weights, and the element that solutions actually realize is singled out only through the Euler identity, with relative block scales that depend on the weights ckc_k and the degrees dkd_k. Lemma A.7 (Set-valued target and collapse). Let Assumption 2.1 hold and let (5) have a solution at θ for the Muon norm with weights ckc_k. Then: (a) Each term of (10) is invariant under Wk↦akWk r_W_k a_k r_W_k and Wk↦bkWkW_k b_kW_k for all ak,bk>0a_k,b_k>0; (b) ∂(12∥2)=(∥μkck−1Gk)k:μ∈ΔK,Gk∈∂∥Wk∥sp∂( 12\| θ\|^2)= \(\| θ\|\, _kc_k^-1G_k)_k:\ μ∈ _K,\ G_k∈∂\|W_k\|_sp \, a convex set of dimension (K−1)+∑k(ρk(ρk+1)/2−1)(K-1)+ _k ( _k( _k+1)/2-1 ), where ΔK _K is the probability simplex. Every solution of (5) has Wk=(dkσ1(Wk)/(Lck2))Gk r_W_k= (d_k _1(W_k)/(Lc_k^2) )G_k with Gk∈∂‖Wk‖spG_k∈∂\|W_k\|_sp. Hence if all ρk=1 _k=1, solutions realize a single point ∘k , which is a positive multiple of the naive target ⋆k iff dk/ck2d_k/c_k^2 is constant in k; if some ρk>1 _k>1, the KKT condition selects no unique point target. Although U1,kV1,k⊤/ρkU_1,kV_1,k / _k is a canonical basis-independent choice, imposing it would add a restriction not implied by stationarity; (c) for any fixed target k and direction s, on the ray =t r=ts, t≥0t≥ 0, the fixed-target loss ‖−t‖2\|k-ts\|^2 is minimized at t⋆=(⟨,⟩)+/‖2t =( ,s )_+/\|s\|^2. So if ⟨,⟩≤0 ,s ≤ 0 the minimizer is = r=0, and otherwise the minimum is ‖2sin2∠(,)\|k\|^2 ^2 (k,s), zero iff s is a positive multiple of k. Each term of (10) is constant along the ray. Part (a) shows that the alignment loss is blind to both per-block scales, so by Proposition 3.4 its zero set is the exact solution set, up to the gauge, for any ckc_k, which the attacker need not know. Part (b) identifies what the naive loss gets wrong. Within a block with ρk>1 _k>1 it must select one element of a spectrahedron, even though stationarity does not specify which element the true residual realizes. Across blocks its relative scales are correct only when dk/ck2d_k/c_k^2 is constant; otherwise Lsingle=0L_single=0 has no solution at all, even for the true data. Part (c) is a landscape property shared by every fixed-target loss, including the Euclidean (3): when the residual is negatively correlated with the target, the cheapest descent is to send the multipliers to zero, switching the candidates off rather than aligning them with data. The alignment loss is constant along multiplier rays and has no such incentive. Being restricted to one ray, (c) is a mechanism rather than a proof of failure on the full domain. A.9 Remarks on Lemma 4.2 (from §4.1) Since SkS_k is concave in ρ, Sk(ρk)≥minSk(1),Sk(rk)S_k( _k)≥ \S_k(1),S_k(r_k)\, and Sk(rk)≥rk(rk−1)/2S_k(r_k)≥ r_k(r_k-1)/2. In particular Sk(ρ)≥mk+nk−2S_k(ρ)≥ m_k+n_k-2 for every ρ whenever rk≥4r_k≥ 4: higher multiplicity never erases Muon’s template. Specificity thus separates the continuous target-cone variability of the geometries where generic counting could not. It is also consistent with §3: the magnitudes σ1(Wk) _1(W_k) are absent from (12) because the cone is scale-free, the same fact that makes the block scales of a fixed target a nuisance (Lemma A.7(b)). A loss cannot recover continuous target information the constraint does not contain, although the fixed cone can still constrain inputs through the residual map. Remark A.8 (Linear-in-parameter models). For f(,)=⟨,ϕ()⟩f(x; θ)= θ,φ(x) the residual map does not depend on θ, so the checkpoint affects the stationarity-feasible set only through C()C( θ) (Lemma A.5); under the Adam geometry this set depends on θ only through (A,sign|A)(A,sign θ|_A) (Proposition A.6). Neither statement alone implies non-identifiability of the inputs (Appendix A). A.10 Collapse of the ℓ∞ _∞ constraint (from §4.2) For networks the residual map carries θ-dependence of its own. The polyhedral geometry, however, has a second consequence that does extend. Minimizing ‖∞\| θ\|_∞ under margin constraints drives θ onto a large face of the ℓ∞ _∞ ball (for a⊤≥1a θ≥ 1 with all aj≠0a_j≠ 0, the solution sign(a)/‖a‖1sign(a)/\|a\|_1 has every coordinate maximal), and on such a face the constraint has few normal equations. We make this precise under a first-order nondegeneracy condition. Let z⋆z be the true solution, I=i:λi⋆>0I=\i: _i >0\ and nI=|I|n_I=|I|. Assumption A.9 (Nondegenerate ℓ∞ _∞ KKT point). Strict complementarity holds (the examples with λi⋆>0 _i >0 are exactly those on the margin), and the vectors j:j∈A∪∇f(i⋆,):i∈I\e_j:\ j∈ A\∪\ _ θf(x_i ; θ):\ i∈ I\ span ℝpR^p. Theorem A.10 (Collapse of the ℓ∞ _∞ constraint). Under Assumption A.9, with r continuously differentiable in z near z⋆z : (i) s≥p−nIs≥ p-n_I, hence codimC()=p−s≤nI C( θ)=p-s≤ n_I; (i) let zI=(i,λi)i∈I∈ℝnI(d+1)z_I=(\x_i\,\ _i\)_i∈ I ^n_I(d+1), the remaining candidates being held at z⋆z , and Φ(zI):=PN(z) (z_I):=P_N r(z) with PNP_N the projection onto the coordinates outside A. Then rankDΦ(zI⋆)≤nI D (z_I )≤ n_I; if nId≥2n_Id≥ 2, this is less than nI(d+1)−1n_I(d+1)-1, so z⋆z is not regularly first-order identifiable (Definition C.1); (i) if moreover DΦD has constant rank near zI⋆z_I and j(z⋆)≠0 r_j(z )≠ 0 for j∈Aj∈ A, the set of zIz_I near zI⋆z_I with λi>0 _i>0 and (z)∈C∘() r(z)∈ C ( θ) is a C1C^1 manifold through zI⋆z_I of dimension at least nIdn_Id. Arbitrarily close to zI⋆z_I , its projection onto the input coordinates contains C1C^1 manifolds of dimension at least nI(d−1)n_I(d-1). See Appendix A.7 for a discussion of the assumption and of the result. A.11 Per-tensor form of the Adam objective (from §3.4) Besides the single global term, we also evaluate the per-tensor form (8) with ∥⋅∥(k)\|·\|_(k) the ℓ∞ _∞ norm on block k. For general θ the per-tensor form is a strict relaxation, since it permits a nonzero residual in tensors containing no globally maximal coordinate. At a KKT direction every tensor contains one (Corollary A.2(b)), and the two forms have the same zero set (Proposition 3.4). Appendix B Proofs Throughout, ∥⋅∥\|·\| is the optimizer’s implicit-bias (margin) norm, ∥⋅∥⋆\|·\|_ its dual, and h()=12‖2h( θ)= 12\| θ\|^2. We collect two elementary lemmas (§B.1), then prove the membership test (§B.2), the scale statements and the Euler identity (§B.3), exactness of per-block alignment (§B.4), the set-valued-target lemma (§B.5), the block decomposition (§B.6), and the specificity and collapse results (§B.7). Table 3: Summary. The cone C()C( θ) is the optimizer-dependent part of the constraint and, for linear-in-parameter models, is the part through which the checkpoint affects stationarity feasibility (Lemma A.5). Codimensions are at a KKT direction on the stratum of θ; at a generic θ all three are Θ(p) (p) in the usual nontrivial matrix-block regime and separate nothing. Sk(ρ)=ρ(mk+nk−ρ)−ρ(ρ+1)/2S_k(ρ)=ρ(m_k+n_k-ρ)-ρ(ρ+1)/2, rk=min(mk,nk)r_k= (m_k,n_k). optimizer cone C()C( θ) codim at KKT stratum specificity S reachability SGD ray through θ p−1p-1 p−1p-1 slow Muon ∏kU1,kMV1,k⊤:M⪰0 _k\U_1,kMV_1,k :M 0\ ≥12(p−∑krk)≥ 12(p- _kr_k) ∑kSk(ρk) _kS_k( _k) fast Adam simplicial cone on A, signs of |A θ|_A p−s≤nIp-s≤ n_I (Thm. A.10) 00 (combinatorial only) fast B.1 Two helper lemmas Lemma B.1 (Subdifferential of a squared norm). For any norm ∥⋅∥\|·\| with dual ∥⋅∥⋆\|·\|_ and any θ, ∂h()=∥∂∥,∂∥=:∥⋆≤1,⟨,⟩=∥.∂ h( θ)=\| θ\|\,∂\| θ\|, ∂\| θ\|=\u:\ \|u\|_ ≤ 1,\ , θ =\| θ\|\. Moreover ∂∥⋅∥∂\|·\| is 00-homogeneous, so ∂‖c‖=∂‖∂\|c θ\|=∂\| θ\| and ∂h(c)=c∂h()∂ h(c θ)=c\,∂ h( θ) for every c>0c>0. Proof. Write h=g∘Nh=g N with N=∥⋅∥N=\|·\| and g(t)=12t2g(t)= 12t^2. The outer function g is convex, differentiable and nondecreasing on [0,∞)⊇range(N)[0,∞) (N), so the convex chain rule gives ∂h()=g′(N())∂N()=N()∂N()∂ h( θ)=g (N( θ))\,∂ N( θ)=N( θ)\,∂ N( θ). The displayed form of ∂N∂ N is standard, from N()=sup‖⋆≤1⟨,⟩N( θ)= _\|u\|_ ≤ 1 , θ . For c>0c>0, ∈∂‖c‖u∈∂\|c θ\| iff ‖⋆≤1\|u\|_ ≤ 1 and ⟨,c⟩=‖c‖=c‖ ,c θ =\|c θ\|=c\| θ\|, i.e. iff ∈∂‖u∈∂\| θ\|. Hence ∂h(c)=c‖∂‖=c∂h()∂ h(c θ)=c\| θ\|\,∂\| θ\|=c\,∂ h( θ). ∎ Lemma B.2 (Gradient homogeneity). If f(,⋅)f(x;·) is L-homogeneous and differentiable in θ, then ∇f(,α)=αL−1∇f(,) _ θf(x;α θ)=α^\,L-1 _ θf(x; θ) for all α>0α>0. Proof. Differentiate f(,α)=αLf(,)f(x;α θ)=α^Lf(x; θ) with respect to θ: the left-hand side is α∇f(,α)α\, _ θf(x;α θ) and the right-hand side αL∇f(,)α^L _ θf(x; θ); divide by α>0α>0. For ReLU networks the identity holds wherever the gradient exists, and everywhere for the smooth surrogate used in the attack. ∎ B.2 Membership test (Lemma 3.1) Proof. (⇐ ) The two hypotheses and the generalized Cauchy–Schwarz inequality ⟨,⟩≤‖⋆‖ r, θ ≤\| r\|_ \| θ\| chain into ‖2≤⟨,⟩≤‖⋆‖≤‖2\| θ\|^2≤ r, θ ≤\| r\|_ \| θ\|≤\| θ\|^2, which forces all three to be equalities. Thus ‖⋆=‖\| r\|_ =\| θ\| and ⟨,⟩=‖2 r, θ =\| θ\|^2, so :=/‖v:= r/\| θ\| satisfies ‖⋆=1\|v\|_ =1 and ⟨,⟩=‖ , θ =\| θ\|, that is ∈∂‖v∈∂\| θ\| and =‖∈‖∂‖ r=\| θ\|v∈\| θ\|\,∂\| θ\| by Lemma B.1. (⇒ ) is immediate from the same two defining properties of ∂‖∂\| θ\|. ∎ B.3 Scale statements and the Euler identity Proof of Lemma 3.2. The residual (z)=∑iλiyi∇f(i,) r(z)= _i _iy_i _ θf(x_i; θ) is linear in λ, so replacing λ↦cλ cλ with c>0c>0 gives ↦c r c\, r and preserves λ≥0λ≥ 0; if ≠ r 0, the choice c:=‖/‖⋆c:=\| θ\|/\| r\|_ achieves ‖c‖⋆=‖\|c\, r\|_ =\| θ\|, and it is the only one. For the equivalence, fix the inputs. (⇒ ) If ∈∂h() r∈∂ h( θ) then ∈C() r∈ C( θ), and ≠ r 0 because ⟨,⟩=‖2>0 r, θ =\| θ\|^2>0 by Lemma 3.1. (⇐ ) If ∈C∘() r∈ C ( θ) then =t r=tk with t>0t>0 and ∈∂h()k∈∂ h( θ); replacing λ by λ/t≥0λ/t≥ 0 turns the residual into k, a solution of (5). No homogeneity of f is used. ∎ Proof of Lemma A.1. Fix c>0c>0 and set λi′=c 2−Lλi≥0 _i =c^\,2-L _i≥ 0. By Lemma B.2 with α=cα=c, ′=∑iλi′yi∇f(i,c)=∑ic 2−LλiyicL−1∇f(i,)=c. r = _i _i \,y_i\, _ θf(x_i;c θ)= _ic^\,2-L _i\,y_i\,c^\,L-1 _ θf(x_i; θ)=c\, r. By hypothesis ∈∂h() r∈∂ h( θ), and by Lemma B.1 ∂h(c)=c∂h()∂ h(c θ)=c\,∂ h( θ); hence ′∈∂h(c) r ∈∂ h(c θ), which together with λi′≥0 _i ≥ 0 is (5) at cc θ. This proves covariance of stationarity; primal feasibility and complementary slackness for the fixed unit margin are not asserted. ∎ Proof of Lemma 3.3. Fix x and k. Differentiating α↦f(,W1,…,αWk,…,WK)=αdkf(,)α f(x;W_1,…,α W_k,…,W_K)=α^d_kf(x; θ) at α=1α=1 gives Euler’s identity ⟨∇Wkf(,),Wk⟩=dkf(,) _W_kf(x; θ),W_k =d_kf(x; θ). Multiplying by λiyi _iy_i at =ix=x_i and summing over i gives (7). Summing (7) over k gives ⟨(z),⟩=LΓ(z) r(z), θ =L\, (z). If (z)∈C∘() r(z)∈ C ( θ), write =t r=tk with t>0t>0 and ∈∂h()k∈∂ h( θ); then ⟨,⟩=t‖2>0 r, θ =t\| θ\|^2>0 by Lemma 3.1, so Γ(z)>0 (z)>0 and ⟨Wk,Wk⟩=dkΓ(z)>0 r_W_k,W_k =d_k (z)>0 for every k, which forces Wk≠ r_W_k 0 and Wk≠W_k 0. For a solution of (5), t=1t=1 and Γ=‖2/L =\| θ\|^2/L. ∎ Proof of Corollary A.2. Let z solve (5); by Lemma 3.3, Wk(z)≠ r_W_k(z) 0 for all k. (a) Lemma B.3(b) requires Wk= r_W_k=0 for every block outside argmaxj‖Wj‖sp/cj _j\|W_j\|_sp/c_j, so every block is in the arg-max. (b) By Lemma B.4(i), every element of C()C( θ) is supported on A; since Wk≠ r_W_k 0, block k meets A, so ‖Wk‖∞=‖∞\|W_k\|_∞=\| θ\|_∞. ∎ B.4 Exactness of per-block alignment Proof of Proposition 3.4. The second equality holds for any θ. A summand of (8) vanishes iff ⟨Wk,Wk⟩=‖Wk‖(k),⋆‖Wk‖(k) r_W_k,W_k =\| r_W_k\|_(k), \|W_k\|_(k) with Wk≠ r_W_k 0, i.e. by (1) applied to the block norm, iff Wk/‖Wk‖(k),⋆∈∂‖Wk‖(k) r_W_k/\| r_W_k\|_(k), ∈∂\|W_k\|_(k), i.e. iff Wk∈Ck()∖ r_W_k∈ C_k( θ) \0\. For Muon, ∂(∥⋅∥sp/ck)=ck−1∂∥⋅∥sp∂(\|·\|_sp/c_k)=c_k^-1∂\|·\|_sp, so CkC_k and the summand do not depend on ckc_k. Max-type norms (Muon, Adam). Write ‖=maxkNk(Wk)\| θ\|= _kN_k(W_k) with Nk=∥⋅∥(k)N_k=\|·\|_(k). By Corollary A.2 all blocks attain the maximum, so the subdifferential of a pointwise maximum of convex functions is the convex hull of the active subdifferentials: ∂∥=conv⋃kιk:∈∂Nk(Wk)∂\| θ\|= _k\ _kg:g∈∂ N_k(W_k)\, with ιk _k the embedding of block k. Since ‖>0\| θ\|>0, C()=cone∂∥=∑ktkιk:tk≥0,k∈∂Nk(Wk)=∏kCk()C( θ)= ∂\| θ\|=\ _kt_k _kg_k:t_k≥ 0,\ g_k∈∂ N_k(W_k)\= _kC_k( θ). Hence if ∈C∘() r∈ C ( θ) then Wk∈Ck() r_W_k∈ C_k( θ) for all k, and Wk≠ r_W_k 0 by Lemma 3.3. Conversely, if Wk∈Ck()∖ r_W_k∈ C_k( θ) \0\ for all k, then ∈∏kCk()=C() r∈ _kC_k( θ)=C( θ) and ≠ r 0. ℓ2 _2. Here C()=t:t≥0C( θ)=\t θ:t≥ 0\ and Ck()=tWk:t≥0C_k( θ)=\tW_k:t≥ 0\. Let z⋆z solve (5), i.e. (z⋆)= r(z )= θ; by Lemma 3.3, ‖Wk‖F2=⟨Wk(z⋆),Wk⟩=dkΓ⋆\|W_k\|_F^2= r_W_k(z ),W_k =d_k with Γ⋆=‖2/L>0 =\| θ\|^2/L>0. (⊆ ) If =t r=t θ with t>0t>0, then Wk=tWk≠ r_W_k=tW_k 0. (⊇ ) If Wk=tkWk r_W_k=t_kW_k with tk>0t_k>0 for all k, then (7) gives tk‖Wk‖F2=dkΓ(z)t_k\|W_k\|_F^2=d_k (z), hence tk=Γ(z)/Γ⋆t_k= (z)/ for every k, and =t r=t θ with t=Γ(z)/Γ⋆>0t= (z)/ >0. ∎ Verification of Remark A.3. On the zero set of LalignL_align, every block is tight, so ‖Wk‖(k),⋆‖Wk‖(k)=⟨Wk,Wk⟩=dkΓ(z)\| r_W_k\|_(k), \|W_k\|_(k)= r_W_k,W_k =d_k (z) by (7). Hence log(‖Wk‖(k),⋆/‖Wk‖(k))=logΓ(z)+log(dk/‖Wk‖(k)2) (\| r_W_k\|_(k), /\|W_k\|_(k))= (z)+ (d_k/\|W_k\|_(k)^2), and the variance over k removes the k-independent term logΓ(z) (z). ∎ B.5 Set-valued target and collapse Proof of Lemma A.7. (a) Numerator and denominator are positively 11-homogeneous in each argument: ⟨akWk,bkWk⟩=akbk⟨Wk,Wk⟩ a_k r_W_k,b_kW_k =a_kb_k r_W_k,W_k , ‖akWk‖nuc=ak‖Wk‖nuc\|a_k r_W_k\|_nuc=a_k\| r_W_k\|_nuc and ‖bkWk‖sp=bk‖Wk‖sp\|b_kW_k\|_sp=b_k\|W_k\|_sp, so the ratio is unchanged. (b) By Corollary A.2 all blocks are active, and as in the proof of Proposition 3.4, ∂∥=(μkck−1Gk)k:μ∈ΔK,Gk∈∂∥Wk∥sp∂\| θ\|=\( _kc_k^-1G_k)_k:μ∈ _K,\ G_k∈∂\|W_k\|_sp\; multiplying by ‖\| θ\| gives ∂h()∂ h( θ). On the relative interior (μk>0 _k>0) the parametrization is injective, since ‖Gk‖nuc=trM=1\|G_k\|_nuc=trM=1 recovers μk=ck‖Wk‖nuc/‖ _k=c_k\|u_W_k\|_nuc/\| θ\| from an element u and then GkG_k. Each ∂‖Wk‖sp∂\|W_k\|_sp is a spectrahedron of dimension ρk(ρk+1)/2−1 _k( _k+1)/2-1 (Lemma B.4), which gives the stated dimension. For a solution, Wk=‖μkck−1Gk r_W_k=\| θ\| _kc_k^-1G_k, and since ⟨Gk,Wk⟩=σ1(Wk)=ck‖ G_k,W_k = _1(W_k)=c_k\| θ\|, Lemma 3.3 gives ‖2μk=⟨Wk,Wk⟩=dk‖2/L\| θ\|^2 _k= r_W_k,W_k =d_k\| θ\|^2/L, i.e. μk=dk/L _k=d_k/L. Hence Wk=‖dkGk/(Lck)=dkσ1(Wk)Gk/(Lck2) r_W_k=\| θ\|d_kG_k/(Lc_k)=d_k _1(W_k)G_k/(Lc_k^2), using ‖=σ1(Wk)/ck\| θ\|= _1(W_k)/c_k. If ρk=1 _k=1, Gk=kk⊤G_k=p_kq_k is unique, so Wk∘=(dk/(Lck2))Wk⋆k _W_k=(d_k/(Lc_k^2))k _W_k, which is a common positive multiple of ⋆k iff dk/ck2d_k/c_k^2 is constant. If ρk>1 _k>1, GkG_k ranges over a spectrahedron of positive dimension, so stationarity does not select a unique point. The barycentric choice Gk=U1,kV1,k⊤/ρkG_k=U_1,kV_1,k / _k is basis-independent, but selecting it adds a condition not implied by stationarity. (c) ‖−t‖2=‖2−2t⟨,⟩+t2‖2\|k-ts\|^2=\|k\|^2-2t ,s +t^2\|s\|^2 is a convex quadratic with unconstrained minimizer ⟨,⟩/‖2 ,s /\|s\|^2; projecting onto t≥0t≥ 0 gives t⋆=(⟨,⟩)+/‖2t =( ,s )_+/\|s\|^2. If ⟨,⟩≤0 ,s ≤ 0 then t⋆=0t =0. Otherwise the minimum is ‖2(1−cos2∠(,))\|k\|^2(1- ^2 (k,s)), which vanishes iff s is a positive multiple of k. Each term of (10) is invariant under ↦t r t r, t>0t>0, by (a). ∎ B.6 Per-block versus global tightness for Muon Lemma B.3 (Block decomposition of tightness). With ‖=maxk‖Wk‖sp/ck\| θ\|= _k\|W_k\|_sp/c_k and ∥⋅∥⋆=∑kck∥⋅∥nuc\|·\|_ = _kc_k\|·\|_nuc, global tightness ⟨,⟩=‖⋆‖ r, θ =\| r\|_ \| θ\| holds if and only if (a) ⟨Wk,Wk⟩=‖Wk‖nuc‖Wk‖sp r_W_k,W_k =\| r_W_k\|_nuc\|W_k\|_sp for every k with Wk≠0 r_W_k≠ 0, and (b) Wk=0 r_W_k=0 unless k∈argmaxj‖Wj‖sp/cjk∈ _j\|W_j\|_sp/c_j. Proof. ⟨,⟩=∑k⟨Wk,Wk⟩≤∑k‖Wk‖nuc‖Wk‖sp=∑k(ck‖Wk‖nuc)(‖Wk‖sp/ck)≤(∑kck‖Wk‖nuc)‖ r, θ = _k r_W_k,W_k ≤ _k\| r_W_k\|_nuc\|W_k\|_sp= _k (c_k\| r_W_k\|_nuc ) (\|W_k\|_sp/c_k )≤ ( _kc_k\| r_W_k\|_nuc )\| θ\|. Equality in the first inequality is (a); equality in the second requires ck‖Wk‖nuc(‖−‖Wk‖sp/ck)=0c_k\| r_W_k\|_nuc (\| θ\|-\|W_k\|_sp/c_k )=0 for all k, which is (b). ∎ Condition (a) is what (10) enforces. Condition (b) would be a severe additional requirement if some block failed to attain the maximum, but by Corollary A.2 none does at a KKT direction of a blockwise-homogeneous network, so (b) is vacuous there and (10) is exact (Proposition 3.4). B.7 Specificity and collapse Lemma B.4 (Exact KKT targets). For ≠ θ 0: (i) ℓ2 _2: ∂(12‖22)=∂ ( 12\| θ\|_2^2 )=\ θ\; (i) ℓ∞ _∞: ∂(12‖∞2)=‖∞convsign(θj)j:j∈A∂ ( 12\| θ\|_∞^2 )=\| θ\|_∞ \sign( _j)e_j:j∈ A\ with A=argmaxl|θl|A= _l| _l|, a simplex of dimension |A|−1|A|-1, whose cone is :j=0 for j∉A,jθj≥0 for j∈A\ r: r_j=0 for j∉ A,\ r_j _j≥ 0 for j∈ A\; (i) spectral, for W≠0W≠ 0 whose top singular value has multiplicity ρ: ∂(12∥W∥sp2)=σ1U1MV1⊤:M⪰0,trM=1∂ ( 12\|W\|_sp^2 )= _1\U_1MV_1 :M 0,\ trM=1\, a spectrahedron of dimension ρ(ρ+1)/2−1ρ(ρ+1)/2-1, which for ρ=1ρ=1 is the singleton σ1⊤\ _1pq \. Proof. By Lemma B.1, ∂(12∥⋅∥2)=∥⋅∥∂∥⋅∥∂( 12\|·\|^2)=\|·\|\,∂\|·\| in each case. (i) ∥⋅∥2\|·\|_2 is differentiable at ≠0 θ≠ 0 with ∂‖2=/‖2∂\| θ\|_2=\ θ/\| θ\|_2\. (i) ‖∞=maxj,ϵ∈±1ϵθj\| θ\|_∞= _j,\,ε∈\± 1\ε\, _j is a maximum of linear functions, so Danskin’s theorem gives ∂‖∞=convsign(θj)j:j∈A∂\| θ\|_∞= \sign( _j)e_j:j∈ A\; nonnegative combinations of these vectors are exactly the vectors supported on A whose nonzero coordinates have matching signs, equivalently jθj≥0 r_j _j≥ 0 on A. (i) is the classical spectral-norm subdifferential (Watson, 1992); the map M↦U1MV1⊤M U_1MV_1 is injective, so the dimension is that of M⪰0,trM=1\M 0,trM=1\. For ρ=1ρ=1, Danskin’s theorem applied to ‖W‖sp=max‖=‖=1⊤W\|W\|_sp= _\|u\|=\|v\|=1u Wv gives ∂‖W‖sp=⊤∂\|W\|_sp=\pq \. ∎ Proof of Lemma A.4. Since ρk≤rk _k≤ r_k, mknk−ρk(ρk+1)/2≥mknk−rk(rk+1)/2m_kn_k- _k( _k+1)/2≥ m_kn_k-r_k(r_k+1)/2, and mknk−rk(rk+1)/2−12(mknk−rk)=12(mknk−rk2)≥0m_kn_k-r_k(r_k+1)/2- 12(m_kn_k-r_k)= 12(m_kn_k-r_k^2)≥ 0. Sum over k. ∎ Proof of Lemma A.5. For f(,)=⟨,ϕ()⟩f(x; θ)= θ,φ(x) , ∇f(,)=ϕ() _ θf(x; θ)=φ(x), so (z)=∑iλiyiϕ(i) r(z)= _i _iy_iφ(x_i) does not depend on θ. The stationarity-feasible set z:(z)∈C∘()\z: r(z)∈ C ( θ)\ therefore depends on θ only through C()C( θ), and the global alignment loss vanishes iff (z)∈C∘() r(z)∈ C ( θ) by (1). Hence C()=C(′)C( θ)=C( θ ) implies that the feasible sets and zero sets coincide. Thus the dependence of this stationarity-only formulation on the checkpoint factors through C()C( θ), whose image has dimension S on the stratum by Definition 4.1. No input non-identifiability conclusion follows from this factorization alone. ∎ Proof of Lemma 4.2. ℓ2 _2. By Lemma B.4(i), C()=t:t≥0C( θ)=\t θ:t≥ 0\, which is determined exactly by the direction /‖2∈Sp−1 θ/\| θ\|_2∈ S^p-1, and distinct directions give distinct cones; hence S=p−1S=p-1. Spectral. With all blocks active, C()=∏kCkC( θ)= _kC_k with Ck=U1,kMV1,k⊤:M⪰0C_k=\U_1,kMV_1,k :M 0\ (proof of Proposition 3.4 and Lemma B.4(i)); σ1(Wk) _1(W_k) does not enter. CkC_k determines the partial isometry Pk:=U1,kV1,k⊤P_k:=U_1,kV_1,k : the elements of CkC_k have singular values equal to the eigenvalues of M, so up to positive scale PkP_k is the unique element of rank ρk _k with all nonzero singular values equal (M=cIM=cI). Conversely, PkP_k determines CkC_k, because any two factorizations of PkP_k differ by (U1,V1)↦(U1Q,V1Q)(U_1,V_1) (U_1Q,V_1Q) with Q orthogonal, which leaves U1MV1⊤:M⪰0\U_1MV_1 :M 0\ unchanged. The image of ↦C() θ C( θ) on the stratum is therefore the image of ↦(Pk)k θ (P_k)_k. This image is open in the product of the manifolds kP_k of rank-ρk _k partial isometries: given Pk′P_k near PkP_k, write Pk′=QkPkQk′⊤P_k =Q_kP_kQ_k with orthogonal Qk,Qk′Q_k,Q_k near the identity, and replace WkW_k by QkWkQk′⊤Q_kW_kQ_k , which keeps all singular values, hence the stratum and activity of every block. Finally, dimk=Sk(ρk) _k=S_k( _k): the polar decomposition B=P(B⊤B)1/2B=P\,(B B)^1/2 identifies the manifold of rank-ρ matrices, of dimension ρ(m+n−ρ)ρ(m+n-ρ), with pairs of a rank-ρ partial isometry and a positive definite operator on its ρ-dimensional row space, of dimension ρ(ρ+1)/2ρ(ρ+1)/2. For ρ=1ρ=1, Sk=mk+nk−2S_k=m_k+n_k-2. ℓ∞ _∞. By Lemma B.4(i), C()C( θ) is a function of (A,signθ|A)(A,signθ|_A) alone, which is discrete data; no continuous functional of θ is determined, so S=0S=0, for every |A|=s|A|=s. ∎ Proof of the multiplicity bound in Appendix A.9. Sk(ρ)=ρ(m+n)−32ρ2−12ρS_k(ρ)=ρ(m+n)- 32ρ^2- 12ρ is concave, so on [1,r][1,r] it is at least minSk(1),Sk(r) \S_k(1),S_k(r)\. With r=n≤mr=n≤ m, Sk(r)=nm−n(n+1)/2≥n(n−1)/2S_k(r)=nm-n(n+1)/2≥ n(n-1)/2, and Sk(r)−(m+n−2)=m(n−1)−n(n+3)/2+2≥n(n−1)−n(n+3)/2+2=(n−1)(n−4)/2≥0S_k(r)-(m+n-2)=m(n-1)-n(n+3)/2+2≥ n(n-1)-n(n+3)/2+2=(n-1)(n-4)/2≥ 0 for n≥4n≥ 4. ∎ Proof of Proposition A.6. By Lemma B.4(i), C()C( θ) depends only on (A,sign|A)(A,sign θ|_A). The conclusion about stationarity-feasible sets follows from Lemma A.5. The final qualification is necessary: even a cone that is constant on a stratum can impose equations on (z) r(z) that identify z for a particular feature map. ∎ Remark B.5 (Nondegeneracy for linear-in-parameter models). For f=⟨,ϕ()⟩f= θ,φ(x) , the ℓ∞ _∞ max-margin problem is the linear program mint t s.t. t−θj≥0t- _j≥ 0, t+θj≥0t+ _j≥ 0, yi⟨,ϕ(i)⟩≥1y_i θ,φ(x_i) ≥ 1, in the variables (,t)∈ℝp+1( θ,t) ^p+1. If its solution is unique it is a vertex, so the normals of the active constraints, namely (−sign(θj)j,1)(-sign( _j)e_j,1) for j∈Aj∈ A and (yiϕ(i),0)(y_iφ(x_i),0) for the active margins, span ℝp+1R^p+1, and their θ-components span ℝpR^p. Under strict complementarity the active margins are exactly I, which is the span condition of Assumption A.9. Proof of Theorem A.10. (i) The spanning set of Assumption A.9 has at most s+nIs+n_I elements, so p≤s+nIp≤ s+n_I. By Lemma B.4(i), C()C( θ) is a full-dimensional cone in spanj:j∈Aspan\e_j:j∈ A\, so codimC()=p−s C( θ)=p-s. (i) affC()=spanj:j∈A C( θ)=span\e_j:j∈ A\, so PNP_N is the coordinate projection onto the p−sp-s coordinates outside A, and Φ takes values in ℝp−sR^p-s. Hence rankDΦ(zI⋆)≤p−s≤nI D (z_I )≤ p-s≤ n_I. If nId≥2n_Id≥ 2 then nI<nI(d+1)−1=dimzI−dimn_I<n_I(d+1)-1= z_I- , so the rank condition of Definition C.1 fails. (i) By the constant-rank theorem, Φ−1() ^-1(0) is, near zI⋆z_I , a C1C^1 manifold ℳM of dimension nI(d+1)−rankDΦ≥nIdn_I(d+1)- D ≥ n_Id, and it contains zI⋆z_I because (z⋆)∈C() r(z )∈ C( θ). On ℳM the coordinates of r outside A vanish. The coordinates j r_j, j∈Aj∈ A, are continuous and nonzero with sign signθjsign _j at z⋆z , hence keep that sign nearby, and λi>0 _i>0 persists nearby; so (z)∈C∘() r(z)∈ C ( θ) on a neighbourhood of zI⋆z_I in ℳM. For the projection π onto the input coordinates, ker(Dπ|Tℳ) (Dπ|_TM) lies in the nIn_I multiplier coordinates, so rankD(π|ℳ)≥nId−nI D(π|_M)≥ n_Id-n_I everywhere. Its maximal value is attained on an open set accumulating at zI⋆z_I , and there the constant-rank theorem makes the image a C1C^1 manifold of dimension at least nI(d−1)n_I(d-1). ∎ Remark B.6 (Codimensions in (14)). The cone C has dimension 11 for ℓ2 _2; ∑kρk(ρk+1)/2 _k _k( _k+1)/2 for the spectral norm (a product of per-block spectrahedral cones, all blocks active), which is K when all ρk=1 _k=1; and s for ℓ∞ _∞. The codimensions in (14) follow. Appendix C Local Identifiability Specificity measures the continuous variability of the optimizer-dependent target cone (Lemma A.5). The complementary question, whether the true data is regularly isolated among stationarity-consistent candidates, is one of well-posedness in the attacker’s own variables. Only candidates with positive multipliers enter the local residual analysis, so we restrict to them. Definition C.1 (Regular first-order identifiability). Let z⋆z be the true solution, I=i:λi⋆>0I=\i: _i >0\, nI=|I|n_I=|I|, and zI=(i,λi)i∈I∈ℝnI(d+1)z_I=(\x_i\,\ _i\)_i∈ I ^n_I(d+1), the remaining candidates being held at z⋆z . Let PN()P_N( θ) be the orthogonal projection onto the orthogonal complement of affC() C( θ) and J=∂zI[PN()(z)]J= _z_I [P_N( θ) r(z) ] at z⋆z . Call z⋆z regularly first-order identifiable if rankJ=dimzI−dim=nI(d+1)−1 J= z_I- =n_I(d+1)-1, where G is the one-dimensional multiplier gauge λ↦cλ cλ of the cone formulation (6). Discrete symmetries (permutations of candidates) do not affect the rank. The multiplier gauge always lies in kerJ J: along (,λI⋆)(0, _I ) the derivative of PN()P_N( θ) r is PN()(z⋆)=P_N( θ) r(z )=0. Maximal rank rules out every infinitesimal ambiguity except this gauge and is a sufficient regularity condition for local isolation modulo gauge. Its failure does not, by itself, prove set-theoretic non-identifiability, because a rank-deficient nonlinear equation can still have an isolated zero. Lemma C.2 (A necessary condition). rankJ≤codimC() J≤ C( θ). Hence regular first-order identifiability requires codimC()≥nI(d+1)−1 C( θ)≥ n_I(d+1)-1. Proof. C()C( θ) is a cone containing 0, so affC()=spanC() C( θ)=spanC( θ) has dimension dimC() C( θ), and PN()P_N( θ) r takes values in a space of dimension p−dimC()=codimC()p- C( θ)= C( θ). ∎ By (14) and Lemma A.4, the necessary condition holds for ℓ2 _2 and for the spectral norm whenever p−1p-1, respectively 12(p−∑krk) 12(p- _kr_k), is at least nI(d+1)−1n_I(d+1)-1, which is the over-parameterized regime. For ℓ∞ _∞ it fails at nondegenerate KKT points: Proposition C.3 (Failure of regular first-order identifiability). Under Assumption A.9 and nId≥2n_Id≥ 2, z⋆z is not regularly first-order identifiable for ℓ∞ _∞, at any width. Under the additional hypotheses of Theorem A.10(i), it belongs to a positive-dimensional local family of stationarity-consistent candidates. Proof. Theorem A.10(i)–(i). ∎ Conjecture C.4. For ℓ2 _2, and for the spectral norm, z⋆z is regularly first-order identifiable for sufficiently over-parameterized blockwise-homogeneous f. We prove Adam’s failure of regular first-order identifiability (Proposition C.3) and, under the additional hypotheses of Theorem A.10(i), actual positive-dimensional local ambiguity. For the positive ℓ2 _2 and spectral direction we prove only the necessary condition of Lemma C.2. The conjectured positive half says that high specificity and high codimension are not merely necessary but sufficient for regular first-order well-posedness. It is directly measurable, requiring one singular value decomposition of J at a recovered solution. (a) Muon (b) SGD (c) Adam (d) Muon, range prior (e) SGD, range prior (f) Adam, range prior Figure 3: Reconstructions on binary MNIST (n=500n=500) without a prior (top) and with the range prior (bottom). In each panel, odd rows show reconstructions and the row below shows the matched training images. Muon recovers sharp digit structure even without a prior. Appendix D Experimental Details Multiclass residual. For multiclass models the residual uses one multiplier per sample and competing class, ∑i∑j≠yiλij∇[Φyi(i,)−Φj(i,)] _i _j≠ y_i _ij _ θ[ _y_i(x_i; θ)- _j(x_i; θ)] (Buzaglo et al., 2023). Test data. For 10-class CIFAR-10 we evaluate on a class-balanced test subset of 1,0001,000 images (100100 per class). Initialization. Following Haim et al. (2022), the first layer is initialized with variance 10−410^-4 and the remaining layers with Kaiming normal initialization (He et al., 2015). Trained models & hardware. Table 4 summarizes the trained model configurations used in Section 5. All networks were trained using PyTorch on a single NVIDIA GeForce GTX 1080 Ti GPU. Training required fewer than 22 minutes for 10,00010,000 iterations over N=500N=500 training samples with learning rate 10−210^-2. For the binary classification setup, 250250 samples per class were used; for the 1010-class multiclass setup, 5050 samples per class were used. Code and reproduction scripts will be made publicly available. Table 4: Trained models (n=500n=500). All models interpolate the training set. Because the network is homogeneous, an unnormalized margin depends on the parameter scale and is not comparable across optimizers. Task Optimizer Test acc. (%) Test loss Min. margin MNIST (binary) SGD 90.090.0 0.7900.790 9.219.21 Adam 91.091.0 0.7530.753 16.6416.64 Muon 92.092.0 0.8530.853 16.6516.65 CIFAR-10 (binary) SGD 77.077.0 2.252.25 9.569.56 Adam 79.079.0 3.413.41 16.6416.64 Muon 79.079.0 2.572.57 16.6316.63 CIFAR-10 (10-class) SGD 28.028.0 7.977.97 13.7513.75 Adam 26.026.0 13.513.5 14.1114.11 Muon 29.029.0 7.097.09 15.4615.46 Muon geometry. In our setup, Muon is applied exclusively to internal 2D weight matrices (e.g., hidden-to-hidden layer W2W_2), whereas standard SGD (Battash et al., 2024) is used for both the input layer (W1W_1) and the final output layer (WLW_L). Consequently, the overall implicit bias is spectral norm, with nuclear dual norm on the internal blocks and Euclidean (L2L_2) on the first and last layers. Data reconstruction attack implementation. Data reconstruction was executed by solving the KKT stationarity conditions corresponding to SGD, Adam, and Muon. Specifically, for layers optimized using SGD, the stationarity loss is defined by the stationarity term in Equation 3; for layers utilizing the Adam optimizer, the stationarity loss follows Equation 11; and for Muon optimizer layers, it is given by Equation 10. To facilitate gradient-based optimization during reconstruction without altering forward behavior, we adopted a modified backward pass for the ReLU activation function y=max(0,x)y= (0,x) following Haim et al. (2022). Specifically, while the forward pass remains exact ReLU, the backward gradient is modified as: ∂L∂x~=∂L∂yσ(αx),where σ(z)=11+e−z ∂ L∂ x= ∂ L∂ y\,σ(α x), σ(z)= 11+e^-z (15) Here, α acts as a hyperparameter interpolating between a scaled linear backward pass function (α→0α→ 0) and the standard ReLU derivative (α→∞α→∞). Dual variables (Lagrange multipliers associated with stationarity conditions) and reconstructed input samples xix_i were optimized. For the attack optimization (applied across all three attacks), we followed Haim et al. (2022) using SGD with a momentum of 0.90.9 over 100,000100,000 iterations. Hyperparameter search. We selected reconstruction hyperparameters through a random search maximizing SSIM@30: • Reconstruction learning rate: Log-uniform sampling in [10−4,10−2][10^-4,10^-2]. • Initialization scale (σx _x): Log-uniform sampling in [10−3,1][10^-3,1] for the initial scale of xix_i. • Modified ReLU parameter (α): Uniform sampling in [10,500][10,500]. (a) Muon, 10-class (b) SGD, 10-class (c) Adam, 10-class Figure 4: Per-sample SSIM versus training margin γi _i on CIFAR-10 (n=500n=500, range prior), for multiclass models. Each point is a training sample, colored by class; the dashed line marks SSIM=0.4SSIM=0.4. Recognizable reconstructions occur only near the minimal margin. Muon produces the most of them and Adam essentially none. Post-processing & matching. To evaluate fidelity, reconstructed samples were matched one-to-one to ground-truth training samples using the Hungarian algorithm on centered, normalized ℓ2 _2 distance. The best-matched instances (such as the top 30 pairs) were selected for qualitative and quantitative reporting. Appendix E Additional Experimental Results To complement the qualitative visual comparisons presented in the main text, Tables 5–7 report the complete quantitative reconstruction results across all experimental configurations. We evaluate performance across both binary and multiclass tasks on MNIST and CIFAR-10 under multiple prior regimes: without prior constraints, under range prior constraints [0,1][0,1], and with additional regularized variants including Total Variation (TV) and L1L_1 sparsity priors. Reconstruction fidelity is evaluated using standard pixel-level and structural metrics, specifically Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), L1L_1 distance, and L2L_2 distance. Across all datasets, and prior configurations, Muon consistently achieves the superior value in every single metric entry, outperforming both standard SGD and Adam. Remarkably, even in the completely unconstrained setting (without prior), Muon retains significantly higher structural fidelity, highlighting the high specificity of reconstructions driven by its implicit bias. Multiclass training dynamics. While the overall performance ordering (Muon>SGD>AdamMuon>SGD>Adam) is largely maintained in the 1010-class setting, the quantitative gap between Muon and SGD narrows compared to the binary regime. Specifically, Muon exhibits a higher concentration of samples exceeding the 0.40.4 recognizability threshold compared to SGD, as illustrated in Figure 4. Comparing across sample sizes (Table 2). Evaluating reconstruction fidelity across varying dataset sizes n reveals that SSIM@30 peaks at n=100n=100 and decreases substantially for n≥500n≥ 500. Although selecting the top 30 instances from a larger candidate pool (m=2nm=2n) represents a more selective subset, the observed drop in performance indicates that the underlying inverse problem becomes fundamentally harder to invert accurately as the scale n increases. Computational complexity. Let q be the number of reconstruction candidates, p the number of model parameters, and d the input dimension. Each attack iteration performs work equivalent to q differentiable parameter-gradient evaluations plus O(p)O(p) residual aggregation. The SGD and Adam alignment terms require O(p)O(p) additional work, while Muon’s dual-norm term additionally requires blockwise singular-value computations. Excluding model activations and automatic-differentiation workspace, the explicit attack state requires O(qd+p)O(qd+p) memory. We study the empirical dependence on training-set size in Section 5 and Table 2. (a) Muon (b) SGD (c) Adam (d) Muon, range prior (e) SGD, range prior (f) Adam, range prior Figure 5: Reconstructions on binary CIFAR-10 (n=500n=500), in the layout of Figure 3. Muon reconstructions best preserve object shape and color. Table 5: Reconstruction on binary MNIST (n=500n=500). Best per prior in bold. Prior Optimizer SSIM↑ SSIM@30↑ PSNR↑ PSNR@30↑ ℓ1↓ _1 ℓ1 _1@30↓ ℓ2↓ _2 ℓ2 _2@30↓ None SGD 0.1661 0.3463 10.857 14.867 0.2059 0.1260 0.2865 0.1806 Adam 0.1527 0.2487 10.345 10.903 0.2077 0.1926 0.3039 0.2850 Muon 0.2350 0.4310 11.495 16.997 0.1718 0.0935 0.2662 0.1413 Range SGD 0.2007 0.4057 11.186 15.464 0.1700 0.1008 0.2759 0.1686 Adam 0.1898 0.2994 10.640 11.933 0.1958 0.1415 0.2938 0.2531 Muon 0.3242 0.6371 12.115 20.154 0.1339 0.0517 0.2479 0.0982 TV SGD 0.1811 0.3768 11.042 14.948 0.1964 0.1209 0.2805 0.1789 Adam 0.1739 0.2882 10.593 11.270 0.1986 0.1801 0.2954 0.2732 Muon 0.3288 0.5808 11.781 17.699 0.1550 0.0783 0.2576 0.1303 ℓ1 _1 SGD 0.1688 0.3614 10.894 14.845 0.2007 0.1227 0.2853 0.1810 Adam 0.1445 0.1752 10.340 11.985 0.1829 0.1433 0.3041 0.2516 Muon 0.2849 0.5302 11.610 17.291 0.1563 0.0819 0.2627 0.1366 TV+ℓ1+ _1 SGD 0.1848 0.3977 11.086 15.423 0.1908 0.1120 0.2791 0.1694 Adam 0.1606 0.2311 10.538 12.028 0.1825 0.1487 0.2972 0.2504 Muon 0.3299 0.6211 11.831 18.002 0.1489 0.0728 0.2561 0.1259 All SGD 0.2289 0.4511 11.345 15.657 0.1641 0.0961 0.2709 0.1649 Adam 0.1758 0.2655 10.834 13.006 0.1663 0.1215 0.2873 0.2237 Muon 0.3400 0.6599 11.996 20.126 0.1361 0.0520 0.2513 0.0986 Table 6: Reconstruction on binary CIFAR-10 (n=500n=500). Best per prior in bold. Prior Optimizer SSIM↑ SSIM@30↑ PSNR↑ PSNR@30↑ ℓ1↓ _1 ℓ1 _1@30↓ ℓ2↓ _2 ℓ2 _2@30↓ None SGD 0.0986 0.2287 11.671 13.523 0.2027 0.1623 0.2609 0.2108 Adam 0.1365 0.1379 12.422 12.902 0.1849 0.1743 0.2393 0.2264 Muon 0.1771 0.5508 12.567 17.802 0.1807 0.0974 0.2353 0.1288 Range SGD 0.0994 0.2336 11.665 13.435 0.2027 0.1634 0.2611 0.2129 Adam 0.1299 0.1367 12.334 13.441 0.1863 0.1624 0.2417 0.2128 Muon 0.1921 0.6130 12.808 19.190 0.1750 0.0825 0.2289 0.1098 TV SGD 0.1106 0.2574 11.857 14.099 0.1975 0.1517 0.2554 0.1973 Adam 0.0958 0.2115 11.2531 13.5265 0.2122 0.1789 0.2656 0.2001 Muon 0.2310 0.6132 13.202 18.974 0.1672 0.0852 0.2187 0.1125 ℓ1 _1 SGD 0.0984 0.2375 11.668 13.995 0.2026 0.1528 0.2610 0.1996 Adam 0.0977 0.2112 10.2658 13.2568 0.2471 0.1793 0.2787 0.2323 Muon 0.1933 0.5598 12.798 18.109 0.1750 0.0936 0.2291 0.1243 TV+ℓ1+ _1 SGD 0.1113 0.2440 11.872 13.748 0.1973 0.1581 0.2549 0.2054 Adam 0.1218 0.2369 11.2451 13.1178 0.2284 0.1955 0.2854 0.2323 Muon 0.2305 0.6286 13.216 19.185 0.1667 0.0832 0.2184 0.1098 All SGD 0.1122 0.2536 11.865 14.194 0.1975 0.1502 0.2551 0.1951 Adam 0.1827 0.2377 12.727 14.462 0.1768 0.1407 0.2310 0.1892 Muon 0.2196 0.6189 13.138 18.848 0.1683 0.0852 0.2203 0.1142 Table 7: Reconstruction on 10-class CIFAR-10 (n=500n=500). Best per prior in bold. Prior Optimizer SSIM↑ SSIM@30↑ PSNR↑ PSNR@30↑ ℓ1↓ _1 ℓ1 _1@30↓ ℓ2↓ _2 ℓ2 _2@30↓ None SGD 0.2031 0.3827 11.264 13.435 0.1912 0.1657 0.2535 0.1906 Adam 0.1530 0.2066 12.254 12.740 0.1856 0.1758 0.2440 0.2307 Muon 0.2446 0.5339 13.284 18.826 0.1651 0.0884 0.2167 0.1145 Range SGD 0.2157 0.4515 12.999 16.445 0.1722 0.1158 0.2239 0.1506 Adam 0.1423 0.1771 12.250 13.238 0.1874 0.1653 0.2441 0.2178 Muon 0.2445 0.5315 13.278 18.987 0.1651 0.0863 0.2168 0.1124 TV SGD 0.2112 0.4332 12.977 16.134 0.1729 0.1198 0.2245 0.1561 Adam 0.2310 0.3442 13.520 15.774 0.1603 0.1224 0.2109 0.1627 Muon 0.2761 0.5481 13.880 19.199 0.1537 0.0848 0.2023 0.1097 ℓ1 _1 SGD 0.2081 0.4141 12.952 16.344 0.1731 0.1147 0.2251 0.1523 Adam 0.1382 0.2739 11.940 14.062 0.1913 0.1491 0.2529 0.1981 Muon 0.2529 0.5371 13.436 18.686 0.1615 0.0900 0.2129 0.1163 TV+ℓ1+ _1 SGD 0.2110 0.4343 13.009 16.085 0.1721 0.1197 0.2236 0.1569 Adam 0.1946 0.3050 12.742 14.106 0.1756 0.1499 0.2306 0.1971 Muon 0.2806 0.5890 13.791 19.408 0.1547 0.0821 0.2044 0.1071 All SGD 0.2110 0.4343 13.009 16.085 0.1721 0.1197 0.2236 0.1569 Adam 0.1598 0.2603 12.491 13.821 0.1809 0.1571 0.2374 0.2037 Muon 0.2814 0.5795 13.780 19.279 0.1549 0.0837 0.2046 0.1087 (a) CIFAR-10 (N=500N=500, No Prior) (b) MNIST (N=500N=500, No Prior) Figure 6: Sorted image-level SSIM values (from highest to lowest) for reconstructed samples on CIFAR-10 and MNIST without prior constraints (n=500n=500). The dashed horizontal reference line at SSIM=0.4SSIM=0.4 indicates a threshold for acceptable visual reconstruction quality. Reconstructions under Muon consistently achieve higher SSIM scores, resulting in a substantially larger fraction of images reconstructed above 0.40.4 compared to SGD and Adam (Muon>SGD>AdamMuon>SGD>Adam). Appendix F Further Material F.1 Notation details (from §2.1) For nonzero u and θ, alignment means that u is a positive multiple of θ in the ℓ2 _2 geometry. For a single spectral-norm block, it means that =U1MV1⊤u=U_1MV_1 for some positive semidefinite M, where U1U_1 and V1V_1 span the corresponding top singular subspaces of θ. For the max-spectral norm, aligned blocks can be nonzero only at blocks attaining the maximum spectral norm. For ℓ∞ _∞, u is supported on the largest-magnitude coordinates of θ, with matching signs wherever u is nonzero. On strata: For ℓ∞ _∞, this type is determined by the set A of coordinates attaining the maximum absolute value and their signs. For the spectral norm, it is determined by the multiplicity ρ of the top singular value; for the max-spectral norm, it also depends on which blocks attain the maximum. By “on the generic stratum,” we mean away from ties and other degeneracies: top singular values are simple, the ℓ∞ _∞ maximum is unique, and the relevant active sets are locally constant. Unless stated otherwise, dimension counts are taken on this stratum. On Assumption 2.1: A first-layer bias is absorbed by appending a constant 11 to the input, so that [W11][W_1\ b_1] is a single block (and Muon is understood to act on it). Differentiability holds at almost every θ for ReLU networks, as used by Haim et al. (2022), and everywhere for the smooth homogeneous models covered by the guarantees of Gronich and Vardi (2026). On the generalized Cauchy–Schwarz inequality: The three geometries considered here give familiar instances of this inequality: Cauchy–Schwarz for the (ℓ2,ℓ2)( _2, _2) pair, Hölder for (ℓ∞,ℓ1)( _∞, _1), and a bound obtained from von Neumann’s trace inequality for the max-spectral norm and its dual, the sum-nuclear norm ∑k‖Wk‖nuc _k\|u_W_k\|_nuc. F.2 Other privacy attacks (from §2) Other privacy attacks differ in their goals and requirements. Membership inference determines whether an example belongs to the training set (Shokri et al., 2017), model inversion can recover class-representative inputs (Fredrikson et al., 2015), and gradient leakage reconstructs data from shared gradients (Zhu et al., 2019). KKT-based reconstruction attacks use only the final weights of the trained model. Differentially private training (Abadi et al., 2016) provides a formal guarantee limiting the influence of individual training examples, potentially at a utility cost. Optimizer choice provides another way to affect reconstruction leakage, but does not provide the same guarantee. F.3 Implicit bias of gradient flow (from §2.1) Theorem F.1 (paraphrased version of Lyu and Li (2020), Ji and Telgarsky (2020)). Let Φ(,) ( θ;x) be a normalized homogeneous11 1 A network Φ(,) ( θ;x) is called homogeneous if there exists c>0c>0 such that for every b>0b>0, θ and x, it holds that Φ(b⋅,)=bcΦ(,) (b· θ;x)=b^c ( θ;x) Haim et al. (2022). ReLU neural network. Consider minimizing the logistic (z↦log(1+e−z)z (1+e^-z)) or the exponential (z↦e−z e^-z) loss using gradient flow (which is a continuous-time analog of gradient descent) over a binary classification set (i,yi)i=1n⊆d×−1,1\(x_i,y_i)\_i=1^n ^d×\-1,1\. Assume that there is a time t0t_0 where L((t0))<1nL( θ(t_0))< 1n. Then, gradient flow converges in direction22 2 We say that gradient flow converges in direction to θ if limt→∞(t)‖(t)‖=^‖^‖ _t→∞ θ(t)\| θ(t)\|= θ\| θ\|. to a first order stationary point (KKT point (Cortes and Vapnik, 1995)) of the maximum-margin problem (P). A KKT point of (P) is characterized by the following set of conditions: • Stationarity: =∑i=1nλi∇[yiΦ(,i)] θ= _i=1^n _i _ θ [y_i ( θ;x_i ) ] • Primal feasibility: yiΦ(,i)≥1,∀i∈[n]y_i ( θ;x_i )≥ 1, ∀ i∈[n] • Dual feasibility: λi≥0,∀i∈[n]. _i≥ 0, ∀ i∈[n]. • Comp. slackness: λi=0 if yiΦ(,i)≠1,∀i∈[n]. _i=0 if y_i ( θ;x_i )≠ 1, ∀ i∈[n]. F.4 Implicit-bias background (from §2) The reconstruction attack relies on characterizing which interpolating solution an optimizer selects. For separable linear models, GD converges in direction to the ℓ2 _2 maximum-margin solution (Soudry et al., 2018). Related results establish convergence towards KKT directions of margin problems for homogeneous networks (Lyu and Li, 2020; Ji and Telgarsky, 2020), and connect steepest descent under other norms to the corresponding margin geometry (Gunasekar et al., 2018); see Vardi (2023) for a survey. These results allow us to study optimizer choice through its associated margin geometry.