Paper deep dive
Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression
Akira Sakai, Yuma Ichikawa
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/21/2026, 12:35:09 AM
Summary
This paper introduces 'Sign Lock-In,' a phenomenon where randomly initialized weight signs in deep learning models persist throughout training, creating a 'one-bit wall' that bottlenecks sub-bit model compression. The authors formalize this via a stopping-time analysis showing that sign flips are rare geometric events under SGD noise. They propose a low-rank sign-template training method to overcome this bottleneck by preserving structured sign patterns.
Entities (10)
Relation Signals (8)
Sign Lock-In → causes → One-Bit Wall
confidence 95% · This randomness gives rise to the lower bound of sub-bit model compression -- the one-bit wall.
Sign Lock-In → observedin → Transformer
confidence 95% · Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation...
Sign Lock-In → observedin → CNN
confidence 95% · Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation...
Sign Lock-In → observedin → MLP
confidence 95% · Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation...
Low-Rank Sign-Template → mitigates → One-Bit Wall
confidence 93% · Building on this mechanism, we introduce a from-scratch low-rank sign-template training method that prevents the emergence of this one-bit wall.
Sign Lock-In → analyzedby → Stopping-Time Analysis
confidence 92% · We formalize this behavior with sign lock-in theory, a stopping-time analysis of sign flips under SGD noise.
Sign Lock-In → drivenby → SGD
confidence 90% · sign lock-in theory, a stopping-time analysis of sign flips under SGD noise.
Sign Lock-In → exhibits → Geometric Tail
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Sub-bit model compression targets storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck. Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation and are spectrally indistinguishable from an i.i.d. Rademacher baseline. This randomness gives rise to the lower bound of sub-bit model compression -- the one-bit wall. Despite this apparent randomness, most weights retain their initialization signs; flips primarily occur via rare near-zero boundary crossings, suggesting that sign-pattern randomness is largely inherited from initialization. We formalize this behavior with sign lock-in theory, a stopping-time analysis of sign flips under SGD noise. Under bounded updates and a rare re-entry condition into a small neighborhood of zero, the number of effective sign flips exhibits a geometric tail. Building on this mechanism, we introduce a from-scratch low-rank sign-template training method that prevents the emergence of this one-bit wall.
Tags
Links
- Source: https://arxiv.org/abs/2602.17063v2
- Canonical: https://arxiv.org/abs/2602.17063v2
Trouble viewing inline? Open PDF directly →
Full Text
214,756 characters extracted from source content.
Expand or collapse full text
Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression Akira Sakai Yuma Ichikawa Abstract Sub-bit model compression targets storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck. Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation and are spectrally indistinguishable from an i.i.d. Rademacher baseline. This randomness gives rise to the lower bound of sub-bit model compression—the one-bit wall. Despite this apparent randomness, most weights retain their initialization signs; flips primarily occur via rare near-zero boundary crossings, suggesting that sign-pattern randomness is largely inherited from initialization. We formalize this behavior with sign lock-in theory, a stopping-time analysis of sign flips under SGD noise. Under bounded updates and a rare re-entry condition into a small neighborhood of zero, the number of effective sign flips exhibits a geometric tail. Building on this mechanism, we introduce a from-scratch low-rank sign-template training method that prevents the emergence of this one-bit wall. Machine Learning, ICML UTF8mc @envStartUTF8 1 Introduction Figure 1: One-bit wall. Shannon’s rate-distortion lower bound for binary sign patterns under Hamming distortion evaluated using an entropy-rate proxy estimated from pretrained weights. Across all models, this proxy is close to one, and the bound is nearly indistinguishable from that of an i.i.d. Rademacher baseline, indicating that sign patterns contain little redundancy. The sign is the minimal discrete attribute of a real-valued weight; it maps w∈ℝw to a binary state sign(w)∈±1sign(w)∈\± 1\, carrying one bit of information per scalar weight. Historically, most practical compression pipelines focused on the few-bit regime, where the sign bit constituted a small and nearly constant overhead relative to magnitude storage and, therefore, rarely emerged as a bottleneck. Our study demonstrates that the sub-bit regime is qualitatively distinct. As indicated in Figure 1, once magnitudes are compressed to approximately one bit per weight, the remaining sign becomes a fixed-cost barrier referred to as the one-bit wall. Furthermore, learned sign patterns across architectures are close to i.i.d. Rademacher; they are nearly uniform and only weakly correlated, leaving little redundancy for further compression. However, tracking sign dynamics has revealed that this apparent randomness is largely inherited from initial random weight signs. This empirical picture points to a paradoxical dynamical regime: although the marginal distribution of trained signs is nearly indistinguishable from i.i.d. Rademacher noise, the trajectory of each sign is highly persistent throughout training. Our approach follows the stochastic-process viewpoint for analyzing SGD beyond asymptotic linearization, aiming to explain this sign lock-in phenomenon. When noise drives the dynamics through rare events such as boundary hits and escapes, tracking only the mean flow can miss the key mechanism; localization via stopping times is therefore a central but challenging approach. This stopping-time perspective connects several classical frameworks: the ODE method controls stability by stopping-time localization (KushnerYin2003; Benaim1999), diffusion approximations turn boundary crossings into first-passage problems (LiTaiE2017), and Freidlin–Wentzell theory explains exponentially rare boundary hits via metastable exit times (FreidlinWentzell1998). Recent ML work adopts SDE/Markov-process lenses to study hitting-time-like behavior under realistic schedules and noise levels (MandtHoffmanBlei2017). Following this line, we formalize sign dynamics under schedule-aware SGD as an effective minimal theory built around stopping times. Contributions. The main contributions are as follows: • Empirical discovery. The learned weight signs are much harder to compress than magnitudes in various representative pretrained architectures. In particular, sign matrices exhibit low-rank approximation error decay and behave like an i.i.d. Rademacher baseline. Moreover, sign patterns remain largely inherited from initialization throughout training. The practical implications of this phenomenon are formalized in the sub-bit regime, which is referred to as a one-bit wall. • Sign lock-in theory. An excursion-based effective framework is introduced to characterize sign dynamics. Under two verifiable conditions, a geometric tail law is proved for the effective outer-to-outer sign flip count, which provides a mechanistic explanation for the persistence of noise-like signs in experiments. This theory is also numerically validated. • Lock-in enhancement. Building on the theory, we propose a from-scratch low-rank sign-template method: a re-generable template T=sign(GH⊤)T=sign(GH ) is selected before training, and training is biased to preserve it. Gap initialization and early-phase outer-drift regularization reduce boundary visits and re-entry, preserving the structured sign template during training. Related work is discussed in Appendix B. (a) SVD compressibility. (b) Spectral randomness test. (c) Sign drift during training. Figure 2: Empirical validation of one-bit wall. (a) SVD compressibility (best rank-r approximation error) of the raw weight matrix W, sign matrix S=sign(W)S=sign(W), and magnitude matrix A=|W|A=|W| as a function of rank ratio r/dr/d (d=min(m,n)d= (m,n)). (b) Spectral fit of sign matrices to an i.i.d. Rademacher baseline using a two-sample KS test on normalized singular values. (c) Initialization-to-trained sign drift in a Transformer trained on next-token prediction: flip ratio vs. initialization across layers (input→ ) and pooled. 2 One-Bit Wall of Model Compression We empirically find that learned weight signs are particularly difficult to compress. Across diverse pretrained architectures (MLP, CNN, Transformer) and layers, the sign component exhibits (i) weak low-rank compressibility, (i) spectral statistics close to i.i.d. Rademacher noise, and (i) strong persistence during training. In contrast, the magnitude component is significantly more compressible. Figure 2(a–c) summarizes these signatures. Notation is given in Appendix A. The experimental settings and additional analyses can be found in Appendix C.1–C.3. Modern compression pipelines leverage structure learned during training. For signs, our evidence indicates that training does not yield exploitable structure: the trained pattern remains close to an initialization-level random template, and optimization seldom modifies it. 2.1 Phenomenon: Signs Look Like Noise Yet Persist We investigate both compressibility and randomness of learned weight signs. Our evaluation includes representative pretrained models: MLP (MLP-Mixer-B16), CNN (ResNet18), and Transformer (TinyLlama-1.1B-Chat). To investigate training-time dynamics, we track sign drift in a scratch-trained multi-layer Transformer language model designed for next-token prediction. Sign-magnitude decomposition. Let W∈ℝm×nW∈R^m× n denote a weight matrix and define the components of sign and magnitude as follows: S≔sign(W)∈±1m×n,A≔|W|∈ℝ≥0m×n,S (W)∈\± 1\^m× n,~~A |W|∈R_≥ 0^m× n, so that W=S⊙AW=S A. This isolates the discrete sign pattern S from the nonnegative magnitudes A and allows us to test their structure separately. Compressibility probe: low-rank approximation error. To assess low-rank compressibility, the optimal rank-r approximation error in the Frobenius norm is measured. For a matrix M, let Mr∈argminrank(X)≤r‖M−X‖FM_r∈ *argmin_rank(X)≤ r\|M-X\|_F denote the best rank-r approximation, given by truncated SVD, and define Er(M)≔‖M−Mr‖F/‖M‖FE_r(M) \|M-M_r\|_F\|M\|_F. Since layers have different shapes, set d≔min(m,n)d (m,n) and parameterize r using the rank ratio q≔r/d∈(0,1]q r/d∈(0,1] with r=⌊qd⌉r= qd . Across architectures, Er(S)E_r(S) decays substantially slower than Er(A)E_r(A) at matched q, and the raw matrix error Er(W)E_r(W) tracks the sign-side behavior more closely than the magnitude-side behavior. Thus, the sign matrix is far more resistant to low-rank compression than the magnitudes and largely explains why direct low-rank approximation of W is difficult. Randomness probe. Low-rank error alone does not distinguish structured matrices from those that behave like random noise. To test whether S is spectrally consistent with a random baseline, we sample multiple s×s× s sub-matrices SsubS_sub from S, compute their singular values, and normalize by s s to remove trivial scaling. As a baseline, we use i.i.d. Rademacher matrices R∈±1s×sR∈\± 1\^s× s. Let FsignF_sign and FrandF_rand denote the pooled empirical cumulative distribution functions (ECDFs) of the normalized singular values from SsubS_sub and R, respectively, and measure the discrepancy using the two-sample Kolmogorov–Smirnov statistic D≔supx|Fsign(x)−Frand(x)|.D _x|F_sign(x)-F_rand(x)|. We observe small KS distances D across layers: the singular value statistics of learned sign submatrices closely track the Rademacher baseline. This supports the interpretation that the trained sign patterns exhibit noise-like behavior at the spectral level, consistent with their poor low-rank compressibility. Additional randomness diagnostics are provided in Appendix E.2. Furthermore, Figure 1 presents an information-theoretic evaluation. For a binary sign source with entropy rate HRDH_RD under Hamming distortion, Shannon’s rate–distortion lower bound implies RDlb(ℛRD)=h2−1(max0,HRD−ℛRD)D^lb_RD(R_RD)=h_2^-1( \0,\,H_RD-R_RD\). Estimated HRDH_RD for sign(W)sign(W) in representative pretrained models yields a near-Rademacher regime (HRD≈1H_RD≈ 1), leaving little room for sub-bit sign storage without incurring non-negligible sign distortion. The details of this analysis are provided in Appendix C.2. Dynamics Probe: Sign Drift during Training. To assess whether the noise-like behavior of S is produced during optimization or inherited from initialization, we track the sign mismatch ratio relative to the initial sign pattern during training. Let W(l)(t)l=1L\W^(l)(t)\_l=1^L denote the collection of matrix-shaped weight tensors at training step t, where W(l)(t)∈ℝml×nlW^(l)(t)∈R^m_l× n_l and Nl≔mlnlN_l m_ln_l. Define the sign flip ratio flip(t)≔1∑lNl∑l,i,j [sign(Wij(l)(t))≠sign(Wij(l)(0))].flip(t) 1 _lN_l _l,i,j1 [sign(W^(l)_ij(t)) (W^(l)_ij(0))]. We track matrix-shaped weights and report flip(t)flip(t) over time for token embeddings (input), each Transformer block, and the LM head (output). flip(t)flip(t) increases during the early phase but typically remains well below 0.50.5 throughout training, indicating that most signs are inherited from initialization and remain stable. This connects the two observations above: signs appear noise-like because they remain close to a random initial template, and training rarely alters them. Takeaway: empirical phenomenon Empirically, weight signs are spectrally noise-like yet largely persistent during training. 2.2 Why Noise-Like Signs Create the One-Bit Wall Sub-bit compression targets an average storage cost of less than one bit per parameter. Many successful schemes can reduce the magnitude A to below one bit per weight on average through quantization, low-rank factorization, pruning, and entropy coding. However, if the sign pattern S closely resembles i.i.d. noise, it offers little exploitable structure for such compressors. In that regime, storing S alone costs one bit per weight under any coordinatewise representation, establishing the one-bit wall even when the magnitudes are highly compressible. The empirical phenomenon described translates into a concrete bottleneck: in the sub-bit regime, sign storage becomes the dominant and potentially irreducible cost. This motivates the remainder of the paper: we seek a mechanistic understanding of why signs persist as random-like and stable, and how to transform signs from a bottleneck into a controllable component. Section 3 provides a minimal theory that explains sign persistence as a consequence of rare boundary excursions. Section 4 then converts this understanding into practical interventions that actively promote compressible sign structure. 3 Sign Lock-In Theory The empirical results suggest that optimization rarely changes signs: most coordinates keep the sign they received at initialization. This section formalizes a simple mechanism behind this persistence. A sign flip of a scalar coordinate can occur only if the trajectory crosses the boundary at 0. If typical training dynamics keep coordinates at magnitudes bounded away from 0, then sign flips must be initiated by rare excursions into a narrow boundary neighborhood. We show that, under the standard training setting of deep learning, the number of effective outer-to-outer sign flips admits a geometric-tail bound. 3.1 Problem Setting We analyze a single coordinate because sign patterns are defined entrywise and sign-storage cost is inherently coordinatewise. The theory is stated for a one-dimensional adapted process; in experiments, we aggregate per-coordinate statistics across layers. Let (Ω,ℱ,(ℱt)t≥0,ℙ)( ,F,(F_t)_t≥ 0,P) be a filtered probability space, and let (wt)t≥0(w_t)_t≥ 0 be a one-dimensional (ℱt)(F_t)-adapted process that represents the discrete-time evolution of a scalar parameter over a finite horizon T∈ℕT∈N. We introduce the radii 0<ϵ<ρ0<ε<ρ that separate a sign-stable outer region from a sign-ambiguous boundary neighborhood. Definition 3.1 (Regions). Fix an outer threshold ρ>0ρ>0 and select a base boundary radius ϵ0∈(0,ρ) _0∈(0,ρ). Let Δ be as stated in Assumption 3.3 and define ϵ≔maxϵ0,Δ∈(0,ρ),ε \ _0, \∈(0,ρ), assuming ρ>Δρ> such that ϵ<ρε<ρ. The outer region (ρ) Outer(ρ) and boundary neighborhood (ϵ) Bd(ε) are defined as (ρ)≔|wt|≥ρ,(ϵ)≔|wt|≤ϵ. Outer(ρ) \|w_t|≥ρ\,~~ Bd(ε) \|w_t|≤ε\. Definition 3.2 (Stopping time). With the convention inf∅=∞ =∞, define recursively σ0 _0 ≔inft≥0:|wt|≥ρ, \t≥ 0:\ |w_t|≥ρ\, τk _k ≔inft>σk−1:|wt|≤ϵ,k≥1, \t> _k-1:\ |w_t|≤ε\,~~k≥ 1, σk _k ≔inft>τk:|wt|≥ρ,k≥1. \t> _k:\ |w_t|≥ρ\,~~k≥ 1. (a) Flip-count histogram. (b) Tail probability and geometric fit. Figure 3: Sign lock-in validation. Left: Histogram of the effective outer-to-outer flip count KTeff(ρ)K^eff_T(ρ) across scalar weights; see Appendix C.4 for T, ρ, and ϵε. Right: Tail probability ℙ[KTeff(ρ)≥k]P[K^eff_T(ρ)≥ k] on a log scale for multiple learning rates, with dashed geometric fits of the form h^g^k−1 h\, g^\,k-1. 3.2 Assumption: Bounded Update and Re-Entry The sign lock-in bound is stated for the abstract adapted process (wt)(w_t) and depends on two key components. The first excludes “pathological” one-step sign flips. The second controls the likelihood that the process returns to the boundary neighborhood after moving back into the outer region. Assumption 3.3 (Bounded update). Fix T∈ℕT∈N. There exist deterministic constants Δ>0 >0 and δupd∈[0,1) _upd∈[0,1) such that, for every stopping time θ satisfying θ≤T−1θ≤ T-1, ℙ[ℰΔ(θ)|ℱθ]≥1−δupda.s.,P [E_ (θ)|F_θ ]≥ 1- _upd~~a.s., where the good event ℰΔ(θ)E_ (θ) is ℰΔ(θ)≔maxθ≤t≤T−1|wt+1−wt|≤Δ.E_ (θ) \ _θ≤ t≤ T-1|w_t+1-w_t|≤ \. In particular, on ℰΔ≔ℰΔ(0)E_ E_ (0) we have |wt+1−wt|≤Δ|w_t+1-w_t|≤ for all 0≤t≤T−10≤ t≤ T-1, and the special case δupd=0 _upd=0 yields an almost-sure uniform increment bound. Assumption 3.3 prevents “pathological” one-step sign flips that jump across the origin while remaining in the outer region. With ϵ≥Δε≥ (Definition 3.1), any outer-to-outer sign flip must pass through (ϵ) Bd(ε). Assumption 3.4 (Re-entry bound condition). There exists gT∈(0,1)g_T∈(0,1) such that for all k≥0k≥ 0, ℙ[τk+1≤T|ℱσk]≤gTa.s.onσk≤T.P[ _k+1≤ T|F_ _k]≤ g_T~~a.s.~on~\ _k≤ T\. Rather than being a technical convenience, Assumption 3.4 delineates the boundary between a stable training regime, where meaningful weights persist, and a degenerate floating regime (Appendix D.6) in which the boundary becomes attractive and weights collapse toward zero. The latter requires inward drift strong enough to distort task loss and is therefore non-standard. Under standard smoothness and bounded-noise conditions, this assumption is satisfied for scheduled SGD (Proposition D.10). The proof is based on Lemma D.8, which shows that the inward drift toward the sign boundary—even if present—is bounded by the finite cumulative gradient norm ensured by standard descent theory. Proposition 3.5 (Informal version of Proposition D.10: re-entry bound in SGD). Under the standard bounded-update and descent/noise conditions, there exists an explicit upper bound gTSGDg_T^SGD such that for all k≥0k≥ 0, ℙ[τk+1≤T∣ℱσk]≤gTSGD.P[ _k+1≤ T F_ _k]≤ g_T^SGD. Moreover, gTSGDg_T^SGD decreases when (i) the boundary margin ρ−ϵρ-ε grows, (i) step sizes decay so that ∑t<Tηt2 _t<T _t^2 is small, and (i) mini-batch noise is moderate. Consequently, Assumption 3.4 holds with gT:=gTSGDg_T:=g_T^SGD, yielding the geometric-tail bound in Theorem 3.6. 3.3 Geometric Tail for Effective Sign Flips An outer-to-outer sign flip can occur only if the trajectory (i) exits the outer region, (i) enters a neighborhood of the boundary, and (i) re-enters the outer region from the opposite side. Assumption 3.4 uniformly controls step (i) over time, while Assumption 3.3 prevents single update transitions from (i) to (i). Taken together, these conditions yield a geometric-tail bound on the number of effective outer-to-outer sign flips. Theorem 3.6 (Sign Lock-in Theorem). Under Assumptions 3.3–3.4, define the effective outer-to-outer flip count up to time T by KTeff(ρ)≔∑k≥1[σk≤T,sign(wσk)≠sign(wσk−1)],K^eff_T(ρ) _k≥ 11 [ _k≤ T,\ sign(w_ _k) (w_ _k-1) ], which is well-defined since |wσk|≥ρ>0|w_ _k|≥ρ>0 on σk≤T\ _k≤ T\. Let hT:=ℙ[τ1≤T]∈[0,1]h_T:=P[ _1≤ T]∈[0,1]. Then for all integers k≥1k≥ 1, ℙ[τk≤T]≤hTgTk−1,P[ _k≤ T]≤ h_T\,g_T^k-1, and consequently, ℙ [KTeff(ρ)≥k]≤hTgTk−1+δupd.P [K^eff_T(ρ)≥ k]≤ h_Tg_T^k-1+ _upd. Remark 3.7. The sign lock-in theorem relies solely on Assumptions 3.3 and 3.4, making it optimizer-agnostic. The uniform re-entry condition of Proposition 3.5 is also not tied to SGD. SGD variant optimizers follow the same arguments underlying Proposition D.10. The theorem formalizes the excursion picture: sign changes are initiated only by boundary hits, and repeated effective outer-to-outer sign flips are exponentially unlikely. To connect this excursion-based perspective to the most common empirical notion of sign persistence, we next demonstrate a complementary fact: if the sign at time T differs from the initial sign, then the trajectory must have entered a small Δ -neighborhood of the origin at least once, provided that one-step updates are bounded. Proposition 3.8. Assume Assumption 3.3. Let (wt)t=0T(w_t)_t=0^T be a real-valued process and st≔sign(wt)∈±1s_t (w_t)∈\± 1\ with a fixed tie-break at 0. Define the first hit time of the Δ -band τΔ≔inft∈0,1,…,T:|wt|≤Δ, _ \t∈\0,1,…,T\:|w_t|≤ \, with the convention inf∅=∞ =∞. We have the deterministic implication sT≠s0≤τΔ≤T.1\s_T≠ s_0\ 1\ _ ≤ T\. Consequently, ℙ[sT≠s0]≤ℙ[τΔ≤T]+δupd.P[s_T≠ s_0]≤P[ _ ≤ T]+ _upd. Takeaway: main result of sign lock-in theory Even if a weight sign flips, its effective flip count decays exponentially. 3.4 Empirical Validation of Sign Lock-In We test the core prediction of Theorem 3.6: the effective outer-to-outer sign flip count KTeff(ρ)K^eff_T(ρ) should have a rapidly decaying tail. Figure 3 reports (i) the histogram of KTeff(ρ)K^eff_T(ρ) in baseline training, and (i) the tail probability ℙ[KTeff(ρ)≥k]P[K^eff_T(ρ)≥ k] on a semi-log scale. The distribution is sharply concentrated near small values of KTeff(ρ)K^eff_T(ρ), and the tail exhibits an approximately linear trend on the log scale, consistent with geometric decay. We overlay the fitted geometric form h^g^k−1 h\, g^\,k-1, where (h^,g^)( h, g) are estimated from the empirical distribution under a zero-inflated geometric model. As predicted by the theory, varying the learning rate changes the effective update scale Δ and hence the prefactor and decay rate of the flip-count distribution, while preserving its geometric form. Accordingly, we observe the same qualitative behavior across learning rates, confirming sign lock-in as a robust baseline phenomenon. An analogous experiment on a vision task is also provided in Appendix E.3. (a) Initial-hit factor h h. (b) Re-entry ratio g g. Figure 4: Billion-scale sweep of the lock-in parameters (h^,g^)( h, g). 3.5 Practical Insights of Sign Lock-In Theory Modern deep learning models tend to show stronger lock-in. Appendix D.4 details the theoretical properties of the initial-hit factor h and the re-entry ratio g, derived from sign lock-in theory, and provides experimental validation. (i) The lock-in parameters vary substantially depending on the learning rate and its schedule. Under a fixed training horizon and peak step size, lock-in becomes progressively weaker in the order of inverse decay → cosine decay → exponential decay → constant learning rate. (i) Scale-invariance mechanisms, such as ReLU positive homogeneity and normalization layers, further strengthen lock-in by suppressing effective boundary re-entry. (i) Increasing the batch size or model size enhances lock-in due to reduced stochastic noise and width-induced stabilization effects. Consequently, sign lock-in weakens when small models are trained with a constant learning rate and small batch size, whereas models following modern standard architectures and training recipes, including LLM, tend to exhibit strong sign lock-in. Billion-scale validation. We sweep the size of the model from the ∼30 \!30M to over 1010B weight parameters and estimate the lock-in parameters (h^,g^)( h, g). Appendix C.5 details experiments. As shown in Figure 4, both the initial-hit factor h h and the re-entry ratio g g decrease monotonically with scale. In the largest models, h h becomes very small and g g approaches zero, indicating that sign flips are rarely initiated and almost never repeated. These results demonstrate that sign lock-in is systematically strengthened with model size, consistent with the prediction of sign lock-in theory. Takeaway: effectiveness of sign lock-in theory Sign lock-in theory robustly predicts the histogram of sign flips and demonstrates that lock-in effects are strengthened in modern neural networks. 4 Sign Lock-In Enhancement The sign lock-in theory suggests that effective outer-to-outer sign flips are rare and exhibit a geometric tail. This turns the empirical one-bit wall into a constructive opportunity: if the initial sign pattern is chosen to be compressible and training preserves it, then the trained model can reuse that sign structure rather than store an arbitrary learned sign matrix. Our resulting procedure is therefore a from-scratch training method. It selects the sign structure before optimization begins and trains the model around that structure; it is not intended as a post-training compressor that takes an arbitrary pretrained model and rewrites its signs after the fact. We therefore ask the following question: Can sign lock-in preserve a compressible initialization template while maintaining task quality? To probe this question, we introduce a theory-guided approach that controls the two quantities governing boundary excursions in Theorem 3.6: (i) the probability of reaching the sign boundary, the initial-hit factor hTh_T, and (i) the probability of returning to the boundary after escaping, the re-entry ratio gTg_T. The soft version of this approach tests whether a structured template can survive ordinary training with little quality loss. The hard template-constrained version then tests the compression implication directly: if signs are re-generable, the bit budget can be spent on magnitudes. Srand=[11−11−111−11−11−1−1−111]S_rand= bmatrix1&1&-1&1\\ -1&1&1&-1\\ 1&-1&1&-1\\ -1&-1&1&1 bmatrix (a) A random sign pattern is not low-rank-friendly. Stemp=[11−1−111−1−1−1−111−1−111]=[11−1−1][11−1−1]S_temp= bmatrix1&1&-1&-1\\ 1&1&-1&-1\\ -1&-1&1&1\\ -1&-1&1&1 bmatrix= bmatrix1\\ 1\\ -1\\ -1 bmatrix bmatrix1&1&-1&-1 bmatrix (b) A deliberately chosen template can be rank one. Figure 5: Re-generable sign templates drive the effective sign cost to zero. A random sign matrix such as SrandS_rand has no compact description and costs about one bit per weight. In contrast, StempS_temp is the elementwise sign of a rank-11 outer product and is therefore re-generable from a small specification (a random seed and the rank). Storing only this specification amortizes the per-weight sign cost to ≈0≈ 0 bits as the model grows, freeing the entire bit budget for the magnitudes. Figure 5 illustrates the key point with the example above. If training begins from ordinary random signs, sign lock-in preserves a Rademacher-like, high-rank sign matrix, which is precisely the one-bit wall. Fixing such signs after training would not help. The proposed method changes the initial condition: it starts from a structured, low-rank-friendly sign template such as StempS_temp and uses sign lock-in enhancement to preserve it. Thus sign persistence becomes an advantage rather than a bottleneck. 4.1 Low-Rank Sign Templates for Suppressing the One-Bit Wall Beyond describing a naturally emerging phenomenon, sign lock-in can be actively controlled through artificial interventions. We first choose a low-rank, compressible sign template as an initialization prior, and then use two lightweight mechanisms to keep trajectories away from the sign boundary. Gap initialization reduces initial boundary hits, while outer-drift regularization reduces later re-entry. Appendix D.5 gives the corresponding theoretical support. The complete details of the end-to-end pipeline implementation are provided in Appendix E.4. Low-rank sign template. We begin by specifying a compressible sign template before training starts. For a layer l with a weight matrix W(l)∈ℝm×nW^(l)∈R^m× n, we create a low-rank template by sampling two factor matrices G∈ℝm×rG∈R^m× r and H∈ℝn×rH∈R^n× r, where r≪min(m,n)r (m,n), and then taking the elementwise sign of their product: Gik∼i.i.d.(0,1),Hjk∼i.i.d.(0,1),T(l)≔sign (GH⊤),G_ik i.i.d. N(0,1),H_jk i.i.d. N(0,1),~T^(l) (GH ), The rank parameter r controls the intrinsic degrees of freedom of the template, making T(l)T^(l) clearly specified and straightforward to reuse. We treat T(l)T^(l) as the initial distribution template for the weights in layer l. Let D be any distribution supported on ℝ>0R_>0 and draw magnitudes Aij(l)∼i.i.d.A^(l)_ij i.i.d. D. We initialize the weights as W(l)≔T(l)⊙A(l)W^(l) T^(l) A^(l). This design establishes a priori a sign structure that is easy to store and reuse; in the experiments, we use r=2r=2 as a representative case. Because the template is chosen before optimization, the method targets new training runs and is distinct from post-training quantization or pruning methods that operate on already-trained weights. Figure 6: Flip–quality trade-off with a compressible sign template (r=2r=2). Validation perplexity mean± over three seeds vs. the mean per-step sign flip rate (flip_mean). Each curve fixes the gap threshold ainita_init and sweeps the log-barrier weight λ from left to right (0.5,0.3,0.1,0.05,0.01,0.001,0.00010.5,0.3,0.1,0.05,0.01,0.001,0.0001). Stronger stabilization suppresses flips but can worsen perplexity, while intermediate ainita_init and λ achieve large flip reduction with little loss in validation quality. Gap Initialization. To reduce early sign flips caused by weights drifting near zero, we initialize each weight with a margin explicitly away from the origin. Let σinit>0 _init>0 denote the base initialization scale and define a gap threshold, ainit≔cgapσinit,cgap>0a_init c_gap _init,c_gap>0, where cgapc_gap is a user-chosen constant controlling the gap size. For each entry, we sample z∼(0,σinit2)z N(0, _init^2) and reject the draw if |z|<ainit|z|<a_init, repeating until |z|≥ainit|z|≥ a_init. Let Z(l)∈ℝm×nZ^(l)∈R^m× n be the resulting matrix with entries that are independently and identically distributed according to this rejection-sampling procedure. Equivalently, each Zij(l)Z^(l)_ij follows a two-sided truncated Gaussian supported on ℝ∖[−ainit,ainit]R [-a_init,a_init]. We initialize the layer-l weights as W0(l)≔Z(l)W^(l)_0 Z^(l). By construction, this initialization suppresses early excursions into the near-zero region, where sign changes are most likely. (a) Baseline. (b) Gap initialization. (c) Gap init. + regularization. Figure 7: Sign vs magnitude low-rank compressibility example. Relative Frobenius error Er(M)=∥M−Mr∥F/∥M∥FE_r(M)= M-M_r _F/ M _F as a function of rank ratio r/dr/d (log scale), for the sign matrix S=sign(W)S=sign(W) and magnitude matrix A=|W|A=|W|, evaluated on final trained weights under baseline, gap initialization only, and gap initialization with regularization. Outer-drift regularization. Even with a gap at initialization, individual weights may later drift back to zero, where sign changes are most probable. To discourage such re-entries during early optimization, we introduce a lightweight log-barrier that penalizes small magnitudes. For a weight matrix W∈ℝm×nW∈R^m× n, a gap threshold ainit>0a_init>0, and a numerical stabilizer ϵlb>0 _lb>0, we define the following: RLB(W;ainit,ϵlb)≔1mn∑i,jlog(max1,ainit|Wij|+ϵlb).R_LB(W;a_init, _lb)\\ 1mn _i,j ( \1, a_init|W_ij|+ _lb \ ). This penalty is 0 whenever |Wij||W_ij| is safely outside the near-zero band, i.e., when |Wij|+ϵlb≥ainit|W_ij|+ _lb≥ a_init, and it increases smoothly as |Wij||W_ij| approaches 0. Let θ denote all model weight parameters, and let ℒtask(θ)L_task(θ) be the task loss, e.g., negative log-likelihood. We apply the barrier to a selected set of layers ℳM, resulting in the time-dependent training objective at the optimization step t: ℒtotal(θ;t)≔ℒtask(θ)+λ(t)∑l∈ℳRLB(W(l);ainit,ϵlb),L_total(θ;t) L_task(θ)+λ(t) _l∈MR_LB(W^(l);a_init, _lb), where λ(t)≥0λ(t)≥ 0 controls the strength of the regularizer. In practice, we keep λ(t)λ(t) constant during an initial warmup phase and then reduce it to 0, so that the penalty primarily influences early dynamics. Overall, this regularizer biases optimization away from the near-zero region after weights have moved into the outer region |Wij(l)|≳ainit|W^(l)_ij| a_init, reducing repeated boundary excursions and helping to preserve the intended sign structure. 4.2 Empirical Validation of Sign Lock-In Enhancement We now validate the mechanism in three steps. First, we test whether gap initialization and outer-drift regularization reduce sign flips without substantially degrading task quality. Second, we check whether the resulting trained weights preserve the intended low-rank sign structure while keeping magnitudes compressible. Third, we evaluate the end-to-end sub-bit compression consequence of using a re-generable sign template and magnitude-only SVD storage. All template-based experiments train models from scratch with the template specified at initialization; they should not be interpreted as post-training conversion of an arbitrary pretrained model. The first two tests use a Transformer trained for next-character prediction; detailed settings are provided in Appendix C.4, with an additional vision task in Appendix E.3. Do initial gap and outer drift enhance lock-in? We empirically examine whether gap initialization and outer-drift regularization effectively enhance sign lock-in. Figure 6 shows the trade-off between task quality and the mean sign-flip rate. While the two mechanisms reinforce each other, a combination of a smaller gap and stronger outer-drift regularization consistently lies on the Pareto frontier, suppressing sign flips to ∼10−3 10^-3 with only about a one-point increase in perplexity. These results agree fully with the theoretical prediction. We also report direct control effect of initial hit factor and re-entry ratio in Appendix C.4. Low-rank structure of sign and magnitude preserved? We next investigate whether sign lock-in enhancement preserves the low-rank structure of the magnitude, which is crucial for effective compression. As shown in Figure 7, the magnitude matrix maintains a low-rank structure comparable to the baseline, even under strong sign stabilization. In contrast, the sign matrix becomes substantially more amenable to low-rank approximation due to the preservation of its structured initialization. This result confirms that sign lock-in enhancement stabilizes sign patterns without compromising the compressibility of magnitudes. (a) charlm (b) text8_char (c) dbpedia14 Figure 8: Performance vs. effective bits per weight (bpweffbpw_eff) on benchmark tasks. Markers indicate the mean over three seeds and error bars show one standard deviation. Lower is better for perplexity (CharLM, Text8-Char), while higher is better for accuracy (DBPedia14). Does the low-rank sign template method improve sub-bit compression? We finally evaluate the end-to-end sub-bit compression consequence of using a low-rank sign template and magnitude-only SVD storage. When signs are fixed by the template, the sign component incurs no storage cost, so the remaining budget can be used to compress the nonnegative magnitudes with truncated SVD and quantized factors. Figure 8 reports the resulting performance–bit trade-off on three representative benchmark tasks. In contrast, vanilla SVD on raw weights and explicit 1-bit baselines either degrade sharply near or stall at one bit per weight, directly visualizing the one-bit wall. Across language modeling and classification tasks, SVD |Wlockin||W_lockin| is consistently stronger than applying the same SVD budget directly to raw weights in the sub-bit region, and it remains competitive against hashing, one-bit, and pruning-based baselines. This supports the practical implication of sign lock-in: once a structured sign template is preserved, magnitudes become the main compressible object and the one-bit sign wall can be bypassed. This comparison evaluates the benefit of choosing and preserving a low-rank sign template during training, not a post-training replacement of the sign matrix in pretrained checkpoints. Additional distillation-based variants and bit-accounting details are provided in Appendix E.4.4. 5 Conclusion Learned signs are harder to compress than magnitudes, exhibit near-random spectral statistics, yet remain strongly aligned with initialization. We explain these phenomena through a sign lock-in theory predicting a geometric flip-count tail, and validate it empirically. Based on this mechanism, we propose gap initialization and outer-drift regularization to suppress boundary visits and sign flips. Our results indicate that stabilizing sign structure is a practical prior for sub-bit compression, and that surpassing the one-bit wall requires explicit control and reuse of sign structure. Leveraging these insights to develop a post-training method that extends sign lock-in to arbitrary pretrained checkpoints is an attractive direction for future work. 6 Limitations Our study has several limitations. (i) The low-rank sign-template method is a from-scratch training approach: the template must be selected before optimization and then preserved during training. It is not a post-training compressor for arbitrary pretrained checkpoints, and extending sign-template control to that setting remains future work. (i) While sign lock-in typically emerges under natural training dynamics, the system can be driven into a sign floating mode under extraordinarily strong magnitude-side regularization that induces persistent attraction toward the sign boundary. We also analyze sign floating mode in Appendix D.6. (i) We focus on simple enforcement methods; other strategies remain unexplored. (iv) We have conducted extensive experiments including a billion-scale validation, but broader empirical coverage is left to future work. (v) While we show that sign degrees of freedom are rarely exploited by optimization, we do not analyze the representational role of signs when treated as fixed parameters, nor their potential contribution to expressivity. Acknowledgements This work was partially supported by JST BOOST, Japan (Grant No. JPMJBY24D0). Impact Statement This work aims to improve the scientific understanding and practical efficiency of sub-bit neural-network compression; the proposed method reduces model storage and memory traffic and can lower deployment cost and energy consumption. At the same time, aggressive compression may degrade rare or boundary-case behavior, which is particularly important in safety-critical applications. References Appendix A Notation We summarize the key symbols that appear throughout the paper. Auxiliary variables that are introduced only within a particular proof, such as shifted or stopped processes and intermediate martingales, are defined locally in the corresponding appendix. Symbol Description Weights, signs, and low-rank structure W(ℓ)∈ℝm×nW^( )∈R^m× n weight matrix at layer ℓ S≔sign(W)∈±1m×nS (W)∈\± 1\^m× n sign matrix ties at 0 mapped to +1+1 A≔|W|∈ℝ≥0m×nA |W|∈R_≥ 0^m× n magnitude matrix; W=S⊙AW=S A (ℓ)∈±1m×nT^( )∈\± 1\^m× n re-generable sign template used for sign-template enforcement Er(M)E_r(M) relative rank-r truncated-SVD error: Er(M)=‖M−Mr‖F/‖M‖FE_r(M)=\|M-M_r\|_F/\|M\|_F d:=min(m,n),q:=r/d:= (m,n),\ \ q:=r/d effective dimension and rank ratio Rate–distortion quantities (Figure 1 / Appendix C.2) ℛRDR_RD, RDD_RD rate (bits/sign) and Hamming distortion RD=ℙ[S≠S^]D_RD=P[S≠ S] RDlb(ℛRD)D^lb_RD(R_RD) Shannon lower bound on distortion (inverse view) HRD,H^RDH_RD,\ H_RD entropy-rate parameter and its empirical proxy (estimated from selected layers) h2(p),h2−1(y)h_2(p),\ h_2^-1(y) binary entropy and its inverse on [0,1/2][0,1/2] L,Nℓ,p^ℓL,\ N_ ,\ p_ number of selected layers, layer sizes, and empirical sign frequency used to form H^RD H_RD Sign flips and stopping-time framework ⋅1\·\, N indicator function and number of tracked scalar entries flip(t)flip(t), flip_meanflip\_mean mismatch-to-initialization ratio and mean step-wise flip rate ℰΔ,Δ,δupdE_ ,\ ,\ _upd bounded-update event |wt+1−wt|≤Δ|w_t+1-w_t|≤ and its failure probability ℙ[ℰΔc]≤δupdP[E_ ^c]≤ _upd ρ, ϵε outer threshold and boundary-neighborhood radius. We set ϵ=maxϵ0,Δε= \ _0, \. σk,τk _k,\ _k k-th outer-entry time and boundary-hit time KTeff(ρ)K_T^eff(ρ) effective (outer-to-outer) sign-flip count up to time T hT,gTh_T,\ g_T lock-in parameters: hT=ℙ[τ1≤T]h_T=P[ _1≤ T] and an upper bound on re-entry probability Quantization and bit accounting Qb(x;α)Q_b(x;α), b, α b-bit symmetric uniform quantizer with scale α bpweff,bpwtargetbpw_eff,\ bpw_target effective bits-per-weight and target budget grid Selected appendix-only bundles (defined where used) ℒ,vt,gt,ηt,ξ2L,\,v_t,\,g_t,\, _t,\,ξ^2 objective / iterate / stochastic gradient / step size / noise proxy (App. D.3) ut,rt,w~t,(Δblk,δblk)u_t,\,r_t,\, w_t,\ ( _blk, _blk) BN+ReLU scale-invariance block notation and its bounded-update parameters (App. D.3) ζ,ω,ϵzs,Dzs,(⋅)zsζ,ω, _zs,D_zs,(·)^zs magnitude z-score preconditioning notation (App. E.4.6) ℳfloat,ρf,RID(W;ρf),ℒfloatM_float,\, _f,\,R_ID(W; _f),\,L_float floating-mode regularization objects (App. D.6) To avoid confusion between time indices and discrete counters, we denote time decrements as (t−1)(t-1), e.g., w(t−1)w_(t-1), while counter decrements are indicated using subscripts, e.g., σk−1,τk−1 _k-1, _k-1. Appendix B Additional Related Work Recent work on compressing pretrained models has evolved from few-bit quantization to sub-bit regimes. One line of work represents weight matrices using a small number of binary bases with learned or compactly encoded coefficients, enabling effective storage at one or sub-bit levels and efficient reconstruction (chen2024onebit; boza2025dbf; bulat2024qbb; lee2025littlebit; gu2025btcllm; ichikawa2025more). Pruning and sparsification reduce the number of stored parameters and are often combined with quantization (Han2015LearningWeightsConnections; Frantar2023SparseGPT; Sun2024Wanda). However, these approaches largely treat the sign pattern as a given object to represent, rather than analyzing what makes signs compressible or incompressible. In particular, there is limited research directly studying the statistical structure and training-time dynamics of weight signs, the underlying object that 1-bit bases must ultimately represent. We address this gap by empirically characterizing sign incompressibility and persistence across architectures, providing a minimal stopping-time theory that explains sign stability and offers actionable methods to control sign evolution. Ultra-low-bit training includes binary and ternary networks that constrain weights and/or activations to small discrete sets, typically relying on the straight-through estimator (Bengio2013STE; Courbariaux2015BinaryConnect; Hubara2016BinarizedNeuralNetworks; Li2016TernaryWeightNetworks; Zhu2017TrainedTernaryQuantization). For 1-bit inference, accuracy has improved through scaling strategies and architectural refinements, including XNOR-type formulations and specialized training recipes (Rastegari2016XNORNet; Zhou2016DoReFaNet; Lin2017ABCNet; Liu2018BiRealNet; Qin2020IRNet; Liu2020ReActNet). These studies discuss the optimization of sign-only networks, and several studies (Xiang2024OvSWOS; LIN2025105646; xu2021recurevivingdeadweights) report that typical binary networks do not reach a sign-flip rate of 50%. However, these studies are observation based and there is still no unified explanation of sign dynamics. Our goal is also different from classical binary or ternary QAT such as DoReFa-Net and TTQ: those methods learn low-bit weights or activations but still store the resulting sign or ternary states explicitly, often together with auxiliary scales. In contrast, our sign-template method targets the near- or sub-one-bit storage regime by choosing a compressible sign prior before training, preserving it through sign lock-in, and reallocating the remaining bit budget to magnitudes. More broadly, quantization research includes early analyses of training with limited numerical precision (Gupta2015LimitedPrecision), practical inference pipelines that utilize only integer-arithmetic (Jacob2018IntegerQuantization), and methods for quantization-aware training (QAT) or post-training quantization (PTQ) that learn or parameterize clipping thresholds and step sizes (Choi2018PACT; Esser2019LSQ). For Transformers and large language models, PTQ has matured within the 8-bit regime (Dettmers2022GPT3int8; Xiao2023SmoothQuant) and has recently advanced significantly for settings that use only 4-bit weights (Frantar2023GPTQ; Lin2024AWQ). Complementary compression mechanisms reduce the cost of storing magnitudes via codebooks, hashing, or vector quantization (Chen2015HashedNets; Gong2014VectorQuantization; Ullrich2017SoftWeightSharing). In parallel, communication-efficient optimization compresses gradients into signs or ternary values (Seide20141bitSGD; Alistarh2017QSGD; Wen2017TernGrad; Lin2018DeepGradientCompression; Bernstein2018signSGD), providing further evidence that sign information remains useful even under extreme discretization. Pruning and sparsity methods range from early second-order criteria (LeCun1990OptimalBrainDamage; Hassibi1993OptimalBrainSurgeon) and magnitude-based pruning (Han2015LearningWeightsConnections) to more recent techniques tailored for large language models (Sun2024Wanda; Frantar2023SparseGPT). Structured compression via low-rank and tensorized parameterizations exploits correlations in weights and has a long history in deep learning (Denil2013PredictingParameters; Denton2014ExploitingLinearStructure; Jaderberg2014SpeedingUpConvNetsLowRank; Sainath2013LowRankFactorizationDNN; Novikov2015TensorizingNeuralNetworks). These lines of work complement our approach, which effectively reallocates bits to magnitude factors after rendering sign information free by construction. Theoretical perspectives on near-initialization training in wide networks, including the neural tangent kernel, over-parameterized convergence analyses, and “lazy training,” predict limited parameter movement and therefore offer a plausible explanation for sign persistence during training (Jacot2018NeuralTangentKernel; Du2019GradientDescentGlobalMinima; AllenZhu2019ConvergenceDeepLearningOverparam; Lee2019WideNeuralNetworksLinearModels; Chizat2019Laziness). Finally, our observation that sign matrices exhibit spectral behavior close to i.i.d. Rademacher baselines naturally connects to classical random matrix theory (Marchenko1967DistributionEigenvalues; Wigner1955CharacteristicVectors) and to spectral viewpoints on deep networks (Pennington2017ResurrectingSigmoid; Martin2018ImplicitSelfRegularization; SAKAI2022119). Appendix C Additional Experimental Details This appendix documents experimental settings and implementation details that support the empirical results in the main text. Specifically: • Appendix C.1 reports the hardware and software environment used across experiments. • Appendix C.2 describes the experimental settings for estimating the entropy-rate proxy. • Appendix C.3 details the protocol used for the randomness tests (Figure 2(a–c)). • Appendix C.4 summarizes the full experimental details for the language-task sign lock-in experiments. • Appendix C.5 reports the billion-scale LLM validation setup. C.1 Hardware and Software To ensure reproducibility, we set a global random seed in both NumPy and PyTorch and report the hardware and software configurations used in our experiments. All runs were conducted on a single NVIDIA A100 GPU. Our software environment included Python 3.12.12, PyTorch 2.9.0, torchvision 0.24.0, timm 1.0.22, transformers 4.57.3, and SciPy 1.16.3. C.2 Experimental Settings for Estimating Entropy Proxy We consider three pretrained models: TinyLlama-1.1B-Chat (zhang2024tinyllama), ResNet18 (torchvision; resnet18 with weights="DEFAULT"), and MLP-Mixer-B16 (timm; mixer_b16_224 with pretrained=True). From each model, we extract two-dimensional internal weight tensors and convert them into matrices suitable for unified analysis. For linear layers, we use the native parameter matrix W∈ℝm×nW∈R^m× n. For convolutional layers, we flatten each Conv2d kernel into a matrix of shape W∈ℝCout×(Cinkhkw)W∈R^C_out×(C_in\,k_h\,k_w). For Transformer models, we exclude tensors with parameter names that contain embed or lm_head to avoid including embedding tables and output heads that may exhibit distinct statistical structures. To focus computation on the most informative layers, we prioritize large matrices by sorting all extracted tensors according to d=min(m,n)d= (m,n) in descending order and selecting up to L=6L=6 matrices per model. Let NlN_l denote the number of entries in the selected matrix for layer l. Whenever we compute a statistic per layer, we aggregate it across layers using an entry-weighted average, H^RD:=∑l=1LNlH^RD,l∑l=1LNl, H_RD:= _l=1^LN_l\, H_RD,l _l=1^LN_l, so that larger matrices contribute proportionally to the overall estimate. Given each weight matrix W, we form the sign matrix S=sign(W)S=sign(W) elementwise, adopting the convention sign(0)=+1sign(0)=+1 so that S∈±1m×nS∈\± 1\^m× n. To capture local dependencies beyond the marginal frequency of positive and negative signs, we estimate the entropy of small contiguous sign patches. For each selected sign matrix S(l)∈±1m×nS^(l)∈\± 1\^m× n, we sample contiguous 3×33× 3 patches P∈±13×3P∈\± 1\^3× 3 by choosing a top-left corner (i,j)(i,j) uniformly at random, subject to 1≤i≤m−21≤ i≤ m-2 and 1≤j≤n−21≤ j≤ n-2, and then taking P=S(l)[i:i+2,j:j+2]P=S^(l)[i:i+2,\,j:j+2]. We map each patch to a 9-bit pattern index by converting ±1± 1 to 0,10,1 and packing the resulting bits, yielding a categorical variable over 29=5122^9=512 possible patterns. Let p^(l)(u) p^(l)(u) denote the empirical frequency of pattern u among the sampled patches in layer l. We compute the plug-in Shannon entropy of the patch distribution, H^patch,l≔−∑u=1512p^(l)(u)log2p^(l)(u), H_patch,l - _u=1^512 p^(l)(u) _2 p^(l)(u), and define the corresponding entropy-rate proxy by normalizing per site, H^RD,l≔H^patch,l/9 H_RD,l H_patch,l9. Under an i.i.d. Rademacher field, this quantity approaches 11 (up to finite-sample effects), whereas it decreases when local structure induces predictable, low-entropy patch patterns. We report the entry-weighted aggregate H^RD H_RD in Figure 1. For scalability, when a full enumeration of patch locations is computationally expensive, we estimate p^(l)(u) p^(l)(u) using a uniform random subsample of patch locations within each layer. C.3 Experimental Details for Randomness Test This appendix describes the experimental protocol used to produce Figure 2(a–c), including the sources of the pretrained models, the procedures for selecting and preprocessing weight matrices, and the hyperparameters utilized in each experiment. C.3.1 Pretrained Models and Sources We study representative pretrained architectures, including MLPs, convolutional networks, and Transformers, to probe whether the observed phenomena are consistent across model families. Specifically, we utilize MLP-Mixer-B16 with pretrained weights from timm, ResNet18 with pretrained weights from torchvision, and TinyLlama-1.1B-Chat from HuggingFace Transformers. To ensure numerical stability and consistent linear-algebra behavior across libraries, we cast all extracted matrices to float32 before performing SVD computations. C.3.2 Weight-Matrix Extraction and Layer Filtering For each model, we iterate over the modules and extract two-dimensional internal weight tensors, converting each into a matrix W∈ℝm×nW∈R^m× n. For linear layers, we use the native weight matrix directly. For Conv2d layers, we flatten the convolutional kernel into a matrix with m representing the number of output channels and n denoting the product of the number of input channels, the kernel height, and the kernel width. To focus on internal transformation matrices in Transformer models, we exclude tensors with parameter names containing embed or lm_head, thereby omitting token embeddings and the output head, which may exhibit different statistical regularities. To control runtime while emphasizing the most informative tensors, we prioritize larger matrices by sorting all candidates according to d=min(m,n)d= (m,n) in descending order. We then limit the number of selected matrices per model based on the requirements of each panel: for Figure 2(a), we use up to 4040 matrices per model to obtain smooth average curves, whereas for Figure 2(b), we use up to 66 matrices per model because each layer contributes additional submatrix sampling and spectral computations. C.3.3 Sign–Magnitude Decomposition and the Sign Convention Given a weight matrix W∈ℝm×nW∈R^m× n, we perform a decomposition. S:=sign(W),A:=|W|,so thatW=S⊙A.S:=sign(W),~~A:=|W|,~~so that~~W=S A. Throughout the paper, we use the convention sign(0)=+1sign(0)=+1, ensuring that S∈±1m×nS∈\± 1\^m× n is always maintained. Even if sign(0)=+1sign(0)=+1, we have W=S⊙A=0W=S A=0 when A=0A=0, so this tie-break convention does not affect the reported statistics. C.3.4 Figure 2(a): SVD Compressibility Error vs. Rank Ratio For a matrix M∈W,S,AM∈\W,S,A\, let σ1SVD(M)≥⋯≥σdSVD(M)≥0σ^SVD_1(M)≥·s≥σ^SVD_d(M)≥ 0 denote its singular values (d=min(m,n)d= (m,n)). The optimal rank-r approximation error in the Frobenius norm can be expressed as Er(M):=‖M−Mr‖F‖M‖F=∑i>rσiSVD(M)2∑i≥1σiSVD(M)2.E_r(M):= \|M-M_r\|_F\|M\|_F= _i>rσ^SVD_i(M)^2 _i≥ 1σ^SVD_i(M)^2. We report Er(W)E_r(W), Er(S)E_r(S), and Er(A)E_r(A) as functions of the rank ratio q:=r/d∈(0,1]q:=r/d∈(0,1]. In our plots, we vary q over a log-spaced grid implementation. q∈12048,22048,42048,…,10242048,r=⌊qd⌉,r∈[1,d],q∈ \ 12048, 22048, 42048,…, 10242048 \,~~r= qd ,~~r∈[1,d], and average Er(⋅)E_r(·) over selected layers within each model. Exact vs. randomized SVD. When d≤2048d≤ 2048, we compute the singular values precisely. For larger matrices, we utilize a randomized low-rank SVD routine (e.g., torch.svd_lowrank) to estimate the tail energy necessary for evaluating Er(M)E_r(M) efficiently at the ranks in our sweep. C.3.5 Figure 2(b): Spectral Fit to a Rademacher Baseline (KS test) Figure 2(b) tests whether sign matrices are spectrally consistent with an i.i.d. Rademacher random matrix. For each selected sign matrix S∈±1m×nS∈\± 1\^m× n, we set d=min(m,n)d= (m,n) and choose s=min(256,d)s= (256,d). If s<32s<32 for a layer, we exclude it from Figure 2(b). Submatrix sampling. For each layer, we sample Nsub=20N_sub=20 contiguous submatrices Ssub∈ℝs×sS_sub∈R^s× s by drawing uniform offsets (i0,j0)(i_0,j_0) and taking the block slice S[i0:i0+s,j0:j0+s]S[i_0:i_0+s,\;j_0:j_0+s]. Normalization and pooling. For each sampled submatrix, we compute the singular values and normalize them by s s to eliminate trivial scaling: σ~iSVD:=σiSVD(Ssub)/s. σ^SVD_i:=σ^SVD_i(S_sub)/ s. We pool σ~iSVD σ^SVD_i across sampled submatrices and layers to form an empirical CDF FsignF_sign. We construct the baseline CDF FrandF_rand by repeating the same procedure on i.i.d. Rademacher matrices R∈±1s×sR∈\± 1\^s× s with equal probability. Two-sample KS statistic. We report the two-sample Kolmogorov–Smirnov statistic D:=supx|Fsign(x)−Frand(x)|,D:= _x |F_sign(x)-F_rand(x) |, and optionally the corresponding p-value from the KS test implementation. C.3.6 Figure 2(c): Transformer LM Init-Sign Drift on a Synthetic Language Task Figure 2(c) measures how much the sign pattern deviates from initialization during scratch training of a multi-layer Transformer language model (LM) on a simple next-token prediction task. Task (synthetic next-token prediction). We generate a synthetic token corpus with vocabulary size V=256V=256 and sequence length L=128L=128. Training uses teacher forcing: given input tokens (x1,…,xL)(x_1,…,x_L), the target is (x2,…,xL+1)(x_2,…,x_L+1). We use 40964096 sequences and mini-batches of size 6464. Model. We employ a causal Transformer language model with Nlayer=12N_layer=12 Transformer blocks, a model width of dmodel=256d_model=256, h=4h=4 attention heads, a feed-forward hidden size of dff=1024d_f=1024, and a dropout set to 0.00.0. Each block features explicit linear projections for q/k/v/oq/k/v/o and a two-layer MLP with GELU nonlinearity. Optimization. We train for 50005000 steps with Adam, learning rate 3×10−43× 10^-4, and cross-entropy loss. We log metrics every 5050 steps. Sign-flip (mismatch) ratio vs. initialization. For a tracked parameter tensor W(t)W(t) during training step t, define flip(t)≔1−1N∑i=1Nsign(wi(t))=sign(wi(0)),flip(t) 1- 1N _i=1^N1\sign(w_i(t))=sign(w_i(0))\, where N denotes the count of entries in the tensor. We report flip(t)flip(t) for the token embedding weights, for each Transformer block after averaging the linear weight tensors within that block, for the LM head weights, and for a pooled average over all tracked tensors; the corresponding curves are shown in Figure 2(c). C.4 Experimental Details for Language Task C.4.1 Dataset and Model This appendix summarizes the settings used for the CharLM sign lock-in experiments. Data. Tiny Shakespeare; contiguous 90/10 train/validation split; character vocabulary from the corpus. Task, batching, and evaluation. Next-character prediction with teacher forcing on random contiguous blocks. We use sequence length L=64L=64 and batch size B=64B=64. Validation loss is averaged over 20 randomly sampled validation mini-batches; PPL =exp(val_loss)= (val\_loss) (loss in nats). Model. TinyCharLM: causal Transformer with dmodel=128d_model=128, nlayers=2n_layers=2, nheads=4n_heads=4, dff=256d_f=256, maximum context length 256256, and no dropout. Optimization and regularization. AdamW; baseline initialization scale σinit=0.02 _init=0.02. When λ>0λ>0, add the log-barrier regularizer as in Section 4 (held constant for the first half of training, then linearly decayed to 0). Initial weights. • Baseline: GPT-like initialization. • Proposed method: apply the low-rank sign template (Section E.4.1, Eq. (21)) and gap initialization to all matrix-shaped parameters, with template rank r=2r=2 and a fixed global seed 1234. Figure 9: Estimated lock-in parameters over gap-init and regularization. Left: h h (initial-hit factor) as a function of the gap threshold ainita_init for different log-barrier weights λ. Right: g g (re-entry ratio) as a function of λ (log scale) for different ainita_init. Points show mean± over three seeds; see Appendix C.4 for the full hyperparameters (including T, ρ, and ϵε). Training horizons and learning rates. All results use three seeds (0/1/2). • Sign lock-in validation T=2000T=2000 steps. Tail plot sweeps learning rate in 10−4, 3⋅10−4, 5⋅10−4, 10−3\10^-4,\,3\!·\!10^-4,\,5\!·\!10^-4,\,10^-3\; the histogram uses 3⋅10−43\!·\!10^-4. • Sign lock-in enhancement: T=12000T=12000 steps, learning rate 3⋅10−43\!·\!10^-4, and ainit∈0.001, 0.005, 0.02, 0.03, 0.05a_init∈\0.001,\,0.005,\,0.02,\,0.03,\,0.05\, λ∈10−4, 10−3, 10−2, 0.05, 0.1, 0.3, 0.5λ∈\10^-4,\,10^-3,\,10^-2,\,0.05,\,0.1,\,0.3,\,0.5\. C.4.2 Analysis Tools Lock-in statistics. We track all matrix-shaped parameters (dim≥2≥ 2). For KTeff(ρ)K_T^eff(ρ), we use ρ=10−3ρ=10^-3 and ϵ=10−4ε=10^-4. Zero-inflated geometric fit. We model the empirical distribution of K:=KTeff(ρ)K:=K^eff_T(ρ) by a zero-inflated geometric distribution: ℙ[K=0] P[K=0] =1−h,ℙ[K=k]=h(1−g)gk−1(k≥1). =1-h,~~P[K=k]=h(1-g)g^k-1~~(k≥ 1). The maximum-likelihood estimates are h^=ℙ[K>0] h=P[K>0] and g^=1−1/[K∣K>0] g=1-1/E[K K>0] (with g^=0 g=0 when [K∣K>0]≤1E[K K>0]≤ 1). We interpret h h as the initial-hit factor and g g as the re-entry propensity. Step-wise flip-rate proxy flip_mean. We report the step-wise flip-rate estimation flip_mean: flip_mean:=1T∑t=0T−1(1N∑i=1Nsign(wi(t+1))≠sign(wi(t))). \_mean:= 1T _t=0^T-1 ( 1N _i=1^N1\sign(w_i(t+1)) (w_i(t))\ ). Here wi(t)w_i(t) is the i-th tracked scalar at step t, and N is the number of tracked scalars. SVD method. This setting is the same as Appendix C.3. Example of weight trajectory. To illustrate the underlying mechanism, Figure 10 shows representative 1D weight trajectories under baseline, gap-dominant, and strongly regularized regimes, together with the outer/boundary bands. The regularized regime keeps trajectories away from the boundary neighborhood, reducing repeated re-entry events. Figure 10: Representative one-dimensional weight trajectories (down sampled for printing). We plot a representative 1D weight trajectory (down-sampled) wtw_t under baseline (ainit=0,λ=0a_init=0,λ=0), a gap-dominant setting, and a strongly regularized setting, together with the outer/boundary bands used to define (σk,τk)( _k, _k) and KTeff(ρ)K^eff_T(ρ). Regularization keeps trajectories away from the boundary neighborhood, suppressing re-entry. C.4.3 Additional Figures for the Language Task Lock-in parameter control. We quantify how the lock-in parameters vary with enhancement: gap initialization (threshold ainita_init) and log-barrier outer-drift regularization (weight λ). For each (ainit,λ)(a_init,λ), we fit a zero-inflated geometric model to the empirical per-weight distribution of KTeff(ρ)K^eff_T(ρ) and extract (h^,g^)( h, g). Figure 9 shows the fitted (h^,g^)( h, g) over the sweep. As predicted by Proposition D.27, increasing the gap threshold ainita_init reduces h h, indicating fewer initial boundary hits. Moreover, increasing the log-barrier weight λ reduces g g (Proposition D.28), consistent with suppressed re-entry into the boundary neighborhood once weights have moved into the outer region. Together, these results support the interpretation that gap initialization controls hTh_T while outer-drift regularization controls gTg_T. C.5 Billion-Scale LLM Validation Table 1: Parameter-scale sweep. We report the configurations of Transformers Model Params (B) dmodeld_model Layers Heads 31M 0.032 512 10 8 64M 0.064 768 9 12 126M 0.126 1024 10 16 260M 0.260 1344 12 21 399M 0.399 1664 12 26 604M 0.604 2048 12 32 1.2B 1.209 2048 24 32 1.8B 1.813 2048 36 32 2.4B 2.417 2048 48 32 5.4B 5.437 3072 48 48 9.7B 9.665 4096 48 64 12.9B 12.887 4096 64 64 This appendix reports a parameter-scale sweep of the lock-in parameters (h^,g^)( h, g), estimated via the zero-inflated geometric fit described in Appendix C.4.2. Data. Tiny Shakespeare; contiguous 90/10 train/validation split; character vocabulary from the corpus. Task and optimizer. Next-character prediction with teacher forcing on random contiguous blocks. We use sequence length L=64L=64 and training micro-batch size B=1B=1. Each configuration is trained for T=1000T=1000 optimizer steps. Model. Char-level causal Transformer LM with learned token embeddings and learned absolute positional embeddings of length L=64L=64 (maximum context length 6464), and no dropout. Each Transformer block uses pre-LayerNorm, SDPA attention (PyTorch scaled_dot_product_attention with is_causal=True, dropout probability 0), and a 2-layer MLP with GELU. For each configuration we set: (i) dff=4dmodeld_f=4\,d_model, (i) the head dimension fixed to 6464 by using nheads=dmodel/64n_heads=d_model/64 (when divisible). To reduce activation memory, we enable per-block gradient checkpointing. Weights are stored in bf16; LayerNorm parameters and computations are kept in fp32 for stability. The model configurations are reported in Table 1. Optimization and regularization. We optimize the cross-entropy loss using AdamW with an 8-bit optimizer state (bitsandbytes AdamW8bit) to reduce memory. We use a constant learning rate η=3×10−4η=3× 10^-4 for T=1000T=1000 steps, weight decay 0.010.01, and global-norm gradient clipping at 1.01.0. We use bf16 autocast for the forward pass. Unless otherwise specified by the library defaults, AdamW uses β1=0.9 _1=0.9, β2=0.999 _2=0.999, and ϵ=10−8ε=10^-8. We do not use dropout or additional regularizers in this sweep. Initialization. All Linear and Embedding weights are initialized i.i.d. from (0,σinit2)N(0, _init^2) with σinit=0.02 _init=0.02; positional embeddings are initialized to zero. Lock-in statistics. We track matrix-shaped parameters (dim≥2≥ 2), using the tie-break convention sign(0)=+1sign(0)=+1. To keep memory bounded at large scale, for each tracked tensor we sample 20482048 scalar coordinates uniformly at random (fixed seed) and compute the effective flip count K:=Keff,T(ρ)K:=K_eff,T(ρ) on these coordinates with ρ=10−3ρ=10^-3 and ϵ=10−4ε=10^-4. We fit the zero-inflated geometric model and report the MLE h^=ℙ[K>0] h=P[K>0] and g^=1−1/[K∣K>0] g=1-1/E[K K>0] (with g^=0 g=0 when [K∣K>0]≤1E[K K>0]≤ 1). Evaluation. Figure 4 shows that both estimated initial-hit factor h h and re-entry ratio g g decrease monotonically with model scale. As the parameter count grows from tens of millions to ∼ 10B parameters, the estimated initial-hit factor h h drops by nearly an order of magnitude, indicating that an increasingly small fraction of weights ever approach the near-zero boundary. At the same time, the re-entry ratio g g also decreases and remains close to zero for the largest models, suggesting that once a weight exits the boundary neighborhood, repeated re-entries are exceedingly rare. These trends imply that sign lock-in strengthens with scale: larger models exhibit fewer initial boundary hits and substantially lower probability of repeated sign flips. This observation is consistent with the width-scaling predictions of the sign lock-in theory, and suggests that extreme-scale models naturally operate deeper in the lock-in regime. Appendix D Proofs and Extensions of Sign Lock-In Theory Figure 11: Sign lock-in trajectory and stopping times. We illustrate the one-dimensional weight trajectory (down-sampled) wtw_t together with the outer region and the sign-boundary neighborhood used in the sign lock-in theory. The outer region is defined as |wt|≥ρ\|w_t|≥ρ\ and the boundary neighborhood as |wt|≤ϵ\|w_t|≤ε\. Starting from an outer-entry time σ0 _0 (outer region, positive sign), the trajectory hits the boundary neighborhood at τ1 _1 and exits to the opposite outer side at σ1 _1 (negative sign), then returns to the boundary at τ2 _2 and re-enters the original outer side at σ2 _2 (positive sign). This outer-to-outer sign evolution is counted by the effective flip count KTeff(ρ)K^eff_T(ρ). This appendix provides a self-contained theoretical analysis underlying the sign lock-in phenomenon introduced in Section 3. For completeness, we restate the stopping-time construction and clarify how each part of this appendix connects to the main theoretical statements. Specifically: • Appendix D.1 provides a complete proof of Theorem 3.6 based solely on Assumptions 3.3 and 3.4. • Appendix D.2 details initial hit factor and sign flips results under the bounded update condition. • Appendix D.3 derives a schedule-aware SGD-based sufficient condition for Assumption 3.4 related to re-entry ratio, we note that this is a representative proof of SGD family optimizers. • Appendix D.4 analyzes implications for learning-rate schedules, batch size, and model width. • Appendix D.5 studies gap initialization and outer-drift regularization. • Appendix D.6 discusses a contrasting sign floating regime in the extreme experimental setting. Stopping times and regions (restated). We briefly restate the stopping-time construction used throughout this appendix, so that the arguments below can be read independently of the main text. Let (wt)t≥0(w_t)_t≥ 0 be a real-valued adapted process, and fix constants 0<ϵ<ρ0<ε<ρ and a finite horizon T. We define the outer region by |wt|≥ρ|w_t|≥ρ and the boundary neighborhood by |wt|≤ϵ|w_t|≤ε. Starting from the first time the trajectory enters the outer region, we define a sequence of stopping times recursively as follows. Let σ0:=inft≥0:|wt|≥ρ, _0:= \t≥ 0:|w_t|≥ρ\, and for k≥1k≥ 1, τk:=inft>σk−1:|wt|≤ϵ,σk:=inft>τk:|wt|≥ρ. _k:= \t> _k-1:|w_t|≤ε\, _k:= \t> _k:|w_t|≥ρ\. These stopping times decompose the trajectory into excursions between the outer region and the boundary neighborhood and form the basis of all proofs in Appendix D.1. The image of the region and stopping time setting is illustrated in Figure 11. Throughout the sign lock-in theory and its proofs, we assume the following bounded-update condition. Assumption 3.3 (restated). Fix T∈ℕT∈N. There exist deterministic constants Δ>0 >0 and δupd∈[0,1) _upd∈[0,1) such that, for every stopping time θ satisfying θ≤T−1θ≤ T-1, ℙ[ℰΔ(θ)|ℱθ]≥1−δupda.s.,P [E_ (θ)|F_θ ]≥ 1- _upd~~a.s., (1) where the good event ℰΔ(θ)E_ (θ) is ℰΔ(θ)≔maxθ≤t≤T−1|wt+1−wt|≤Δ.E_ (θ) \ _θ≤ t≤ T-1|w_t+1-w_t|≤ \. In particular, on ℰΔ≔ℰΔ(0)E_ E_ (0) we have |wt+1−wt|≤Δ|w_t+1-w_t|≤ for all 0≤t≤T−10≤ t≤ T-1, and the special case δupd=0 _upd=0 yields an almost-sure uniform increment bound. D.1 Proofs for Sign Lock-In Theory In the proof of Theorem 3.6, we assume the following Re-entry bound condition. We show in Appendix D.3 that this assumption is justified under standard SGD settings. Moreover, although we do not provide an explicit proof, similar arguments apply to SGD-family optimizers such as Adam. Therefore, this assumption is well justified under typical training settings. Roadmap of the proof. To keep Appendix D easy to follow, we first show in Lemma D.1 that an outer-to-outer sign change necessarily passes through the boundary neighborhood. Next, Lemma D.2 lifts this pathwise fact to an event-level statement: frequent effective flips imply frequent boundary hits. Finally, Theorem 3.6 combines Assumption 3.4 with this implication to obtain a geometric tail bound. Overall, the logic is flip ⇒ boundary hit ⇒ geometric decay. Assumption 3.4 (restated). There exists gT∈(0,1)g_T∈(0,1) such that for all k≥0k≥ 0, ℙ[τk+1≤T|ℱσk]≤gTa.s.onσk≤T.P[ _k+1≤ T|F_ _k]≤ g_T~~a.s.~on~\ _k≤ T\. Lemma D.1 (Outer-to-outer sign flip forces a boundary visit). On the good event ℰΔE_ from Assumption 3.3, on σk−1≤T,σk≤T\ _k-1≤ T,\ _k≤ T\, if sign(wσk)≠sign(wσk−1)sign(w_ _k) (w_ _k-1) then τk<σk _k< _k and in particular τk≤T _k≤ T. Proof. Work on the good event ℰΔE_ (Assumption 3.3). Assume sign(wσk)≠sign(wσk−1)sign(w_ _k) (w_ _k-1) on σk−1≤T,σk≤T\ _k-1≤ T,\ _k≤ T\. Then wσk−1w_ _k-1 and wσkw_ _k have opposite signs with |wσk−1|≥ρ|w_ _k-1|≥ρ and |wσk|≥ρ|w_ _k|≥ρ. Without loss of generality, take wσk−1≥ρ>0w_ _k-1≥ρ>0 and wσk≤−ρ<0w_ _k≤-ρ<0. Let t⋆:=inft>σk−1:wt≤0.t := \t> _k-1:w_t≤ 0\. Since wσk−1>0w_ _k-1>0 and wσk<0w_ _k<0, we have t⋆≤σk≤Tt ≤ _k≤ T. That is, t⋆t is the first post-σk−1 _k-1 contact with the sign boundary. Moreover, t⋆<σkt < _k: if t⋆=σkt = _k, then w(σk−1)>0w_( _k-1)>0 and wσk≤−ρw_ _k≤-ρ, so |wσk−w(σk−1)|≥ρ|w_ _k-w_( _k-1)|≥ρ. But on ℰΔE_ we have |wσk−w(σk−1)|≤Δ<ρ|w_ _k-w_( _k-1)|≤ <ρ (as ϵ=maxϵ0,Δ<ρε= \ _0, \<ρ), a contradiction. By definition, w(t⋆−1)>0w_(t -1)>0 and wt⋆≤0w_t ≤ 0, hence |wt⋆|=−wt⋆≤w(t⋆−1)−wt⋆=|wt⋆−w(t⋆−1)|≤Δ≤ϵ(on ℰΔ).|w_t |=-w_t ≤ w_(t -1)-w_t =|w_t -w_(t -1)|≤ ≤ε~~(on $ E_ $). Therefore |wt⋆|≤ϵ|w_t |≤ε, and by the definition of τk _k we get τk≤t⋆<σk _k≤ t < _k. In particular, τk≤T _k≤ T. ∎ Lemma D.2 (Many effective flips imply many boundary hits). For all k≥1k≥ 1, KTeff(ρ)≥k⊆τk≤T∪ℰΔc.\K^eff_T(ρ)≥ k\ \ _k≤ T\\ ∪\ E_ ^\,c. Consequently, ℙ[KTeff(ρ)≥k]≤ℙ[τk≤T]+δupd.P[K^eff_T(ρ)≥ k]≤P[ _k≤ T]+ _upd. Proof. On the good event ℰΔE_ , Lemma D.1 holds. If KTeff(ρ)≥kK^eff_T(ρ)≥ k, then there exist k indices 1≤j1<⋯<jk1≤ j_1<·s<j_k such that σjm≤T _j_m≤ T and sign(wσjm)≠sign(wσjm−1)sign(w_ _j_m) (w_ _j_m-1) for each m. In other words, we can explicitly list k distinct outer-to-outer sign-change episodes before time T. Fix such an index j=jmj=j_m. Since (σl)( _l) is nondecreasing, we have σj−1≤σj≤T _j-1≤ _j≤ T, so on ℰΔE_ Lemma D.1 applies and yields τj≤T _j≤ T. Because (τl)( _l) is nondecreasing and jk≥kj_k≥ k, we have τk≤τjk≤T _k≤ _j_k≤ T. Thus we have shown KTeff(ρ)≥k∩ℰΔ⊆τk≤T\K^eff_T(ρ)≥ k\∩E_ \ _k≤ T\, equivalently KTeff(ρ)≥k⊆τk≤T∪ℰΔc\K^eff_T(ρ)≥ k\ \ _k≤ T\∪E_ ^\,c. Taking probabilities and using ℙ(ℰΔc)≤δupdP(E_ ^\,c)≤ _upd gives Lemma D.2. ∎ Theorem 3.6 (restated). Under Assumptions 3.3–3.4. Define the effective outer-to-outer flip count up to time T by KTeff(ρ)≔∑k≥1[σk≤T,sign(wσk)≠sign(wσk−1)],K^eff_T(ρ) _k≥ 11 [ _k≤ T,\ sign(w_ _k) (w_ _k-1) ], (2) which is well-defined since |wσk|≥ρ>0|w_ _k|≥ρ>0 on σk≤T\ _k≤ T\. Let hT:=ℙ[τ1≤T]∈[0,1]h_T:=P[ _1≤ T]∈[0,1]. Then for all integers k≥1k≥ 1, ℙ[τk≤T]≤hTgTk−1,P[ _k≤ T]≤ h_T\,g_T^k-1, (3) and consequently, ℙ [KTeff(ρ)≥k]≤hTgTk−1+δupd.P [K^eff_T(ρ)≥ k]≤ h_Tg_T^k-1+ _upd. (4) Proof. We prove Eq. (3) by induction. For k=1k=1, Eq. (3) reduces to ℙ[τ1≤T]=hTP[ _1≤ T]=h_T. Assume ℙ[τk≤T]≤hTgTk−1P[ _k≤ T]≤ h_T\,g_T^k-1 for some k≥1k≥ 1. Then, by the tower property and Assumption 3.4, we first note that τk+1=inft>σk:|wt|≤ϵ _k+1= \t> _k:|w_t|≤ε\ implies τk+1≤T⊆σk≤T\ _k+1≤ T\ \ _k≤ T\. Therefore ℙ[τk+1≤T] P[ _k+1≤ T] =[ℙ[τk+1≤T∣ℱσk] 1σk≤T] =E\! [P[ _k+1≤ T F_ _k]\,1_\ _k≤ T\ ] ≤[gT 1σk≤T]=gTℙ[σk≤T]≤gTℙ[τk≤T] ≤E\! [g_T\,1_\ _k≤ T\ ]=g_T\,P[ _k≤ T]≤ g_T\,P[ _k≤ T] ≤gT⋅hTgTk−1=hTgTk. ≤ g_T· h_T\,g_T^k-1=h_T\,g_T^k. Here we used ℙ[σk≤T]≤ℙ[τk≤T]P[ _k≤ T]≤P[ _k≤ T], which follows from the stopping-time order τk<σk _k< _k whenever σk<∞ _k<∞. The bound for KTeff(ρ)K^eff_T(ρ) follows from Lemma D.2, which introduces the additional failure probability term δupd _upd under Assumption 3.3. ∎ D.2 Initial Hit Factor and Deterministic Foundations of Sign Flips This subsection isolates a deterministic prerequisite for sign changes that follows solely from the bounded-update event in Assumption 3.3. In particular, Proposition 3.8 shows that any deviation of the final sign from the initialization sign forces the trajectory to enter the Δ -band at least once, so sign drift can be upper bounded by a boundary-hit probability. We then lift this implication to the vector setting via Proposition D.3, which bounds the initialization-to-final mismatch rate pTinitp_T^init by the average of per-coordinate Δ -band hit probabilities, up to the failure term δupd _upd from Assumption 3.3. Finally, Proposition D.4 turns this boundary-hit viewpoint into an initialization-dominated control of the initial-hit factor hTh_T using the boundary radius in Definition 3.1. These deterministic foundations complement the excursion and re-entry analysis that controls repeated effective flips, and they will be used later when we design interventions that suppress boundary visits. Proposition 3.8 (restated). Assume the bounded-update condition. Let (wt)t=0T(w_t)_t=0^T be a real-valued process and define st:=sign(wt)s_t:=sign(w_t) with a fixed tie-breaking rule at zero. Define the first hitting time of the Δ -band by τΔ:=inft∈0,1,…,T:|wt|≤Δ, _ := \t∈\0,1,…,T\:|w_t|≤ \, with the convention inf∅=∞ =∞. Then the following deterministic implication holds: sT≠s0≤ 1τΔ≤T.1\s_T≠ s_0\\;≤\;1\ _ ≤ T\. Consequently, ℙ(sT≠s0)≤ℙ(τΔ≤T)+δupd.P(s_T≠ s_0)\;≤\;P( _ ≤ T)+ _upd. Proof. Work on the event ℰΔE_ and assume that sT≠s0s_T≠ s_0. Define the first instance where the sign differs from initialization: t⋆≔mint∈1,2,…,T:st≠s0.t \t∈\1,2,…,T\:s_t≠ s_0\. Then st⋆−1=s0s_t -1=s_0 and st⋆≠s0s_t ≠ s_0, hence st⋆≠st⋆−1s_t ≠ s_t -1. With a fixed tie-break at 0, st⋆≠st⋆−1s_t ≠ s_t -1 implies wt⋆wt⋆−1≤0w_t w_t -1≤ 0 (opposite signs or one is 0). Therefore, |wt⋆−wt⋆−1|=|wt⋆|+|wt⋆−1|.|w_t -w_t -1|=|w_t |+|w_t -1|. On ℰΔE_ , we also have |wt⋆−wt⋆−1|≤Δ|w_t -w_t -1|≤ , hence |wt⋆|+|wt⋆−1|≤Δ,|w_t |+|w_t -1|≤ , which implies min|wt⋆|,|wt⋆−1|≤Δ \|w_t |,|w_t -1|\≤ . Thus there exists t∈t⋆−1,t⋆t∈\t -1,t \ such that |wt|≤Δ|w_t|≤ , so τΔ≤t⋆≤T _ ≤ t ≤ T. This proves sT≠s0≤τΔ≤T1\s_T≠ s_0\ 1\ _ ≤ T\ on ℰΔE_ . Finally, ℙ[sT≠s0]≤ℙ[τΔ≤T]+ℙ[ℰΔc]≤ℙ[τΔ≤T]+δupd,P[s_T≠ s_0]≤P[ _ ≤ T]+P[E_ ^c]≤P[ _ ≤ T]+ _upd, where the last inequality employs Assumption 3.3 at θ=0θ=0. ∎ Proposition D.3 (Initialization-to-final sign drift bound). Assume Assumption 3.3. Let wt∈ℝNw_t∈R^N be the parameter vector and define st(i)≔sign(wt(i))∈±1s_t(i) (w_t(i))∈\± 1\ entrywise with a fixed tie-break at 0. Let pTinit≔1N∑i=1N[sT(i)≠s0(i)]∈[0,1].p_T^init 1N _i=1^N1 [s_T(i)≠ s_0(i) ]∈[0,1]. Define the per-coordinate first hit time of the Δ -band by τΔ,i≔inft∈0,1,…,T:|wt(i)|≤Δ∈0,1,…,T∪∞, _ ,i \\,t∈\0,1,…,T\:\ |w_t(i)|≤ \,\∈\0,1,…,T\∪\∞\, with inf∅=∞ =∞. Then: (A) On the event ℰΔ≔max0≤t≤T−1‖wt+1−wt‖∞≤ΔE_ \ _0≤ t≤ T-1\|w_t+1-w_t\|_∞≤ \, [sT(i)≠s0(i)]≤[τΔ,i≤T]for all i∈[N].1 [s_T(i)≠ s_0(i) ] 1 [ _ ,i≤ T ]~~for all i∈[N]. Consequently, pTinit≤1N∑i=1N[τΔ,i≤T]on ℰΔ.p_T^init≤ 1N _i=1^N1 [ _ ,i≤ T ]~~on E_ . (B) Let h¯T≔1N∑i=1Nℙ[τΔ,i≤T] h_T 1N _i=1^NP[ _ ,i≤ T]. Then [pTinit]≤h¯T+δupd.E[p_T^init]≤ h_T+ _upd. Proof. (A) Fix an index i∈[N]i∈[N] and work on the event ℰΔE_ . Assume that sT(i)≠s0(i)s_T(i)≠ s_0(i). Define the first time at which the i-th coordinate changes its sign relative to initialization: t⋆:=mint∈1,2,…,T:st(i)≠s0(i).t := \t∈\1,2,…,T\:\ s_t(i)≠ s_0(i)\. Then st⋆−1(i)=s0(i)s_t -1(i)=s_0(i) and st⋆(i)≠s0(i)s_t (i)≠ s_0(i), so st⋆(i)≠st⋆−1(i)s_t (i)≠ s_t -1(i). With the fixed tie-break at 0, this implies wt⋆(i)wt⋆−1(i)≤0w_t (i)\,w_t -1(i)≤ 0, and hence |wt⋆(i)−wt⋆−1(i)|=|wt⋆(i)|+|wt⋆−1(i)|.|w_t (i)-w_t -1(i)|=|w_t (i)|+|w_t -1(i)|. On the event ℰΔE_ , we have |wt⋆(i)−wt⋆−1(i)|≤Δ|w_t (i)-w_t -1(i)|≤ , which implies |wt⋆(i)|+|wt⋆−1(i)|≤Δ.|w_t (i)|+|w_t -1(i)|≤ . Therefore min|wt⋆(i)|,|wt⋆−1(i)|≤Δ \|w_t (i)|,\ |w_t -1(i)|\≤ , and hence there exists t∈t⋆−1,t⋆t∈\t -1,t \ such that |wt(i)|≤Δ|w_t(i)|≤ . By definition of τΔ,i _ ,i, this yields τΔ,i≤t⋆≤T _ ,i≤ t ≤ T. Thus, sT(i)≠s0(i)≤τΔ,i≤Ton EΔ.1\s_T(i)≠ s_0(i)\ 1\ _ ,i≤ T\ E_ . Averaging over i gives the stated bound for pTinitp_T^init on ℰΔE_ . (B) By part (A), pTinit≤1N∑i=1NτΔ,i≤Ton EΔ.p_T^init≤ 1N _i=1^N1\ _ ,i≤ T\ E_ . Taking expectations and using ℙ(EΔc)≤δupdP(E_ ^c)≤ _upd from Assumption 3.3, we obtain [pTinit]≤1N∑i=1Nℙ[τΔ,i≤T]+ℙ(EΔc)≤h¯T+δupd.E[p_T^init]≤ 1N _i=1^NP[ _ ,i≤ T]+P(E_ ^c)≤ h_T+ _upd. This completes the proof. ∎ Proposition D.4 (Initialization-dominated bound for the initial-hit factor). Assume Assumption 3.3. Let ϵε be the boundary radius in Definition 3.1 and define bT:=ϵ+TΔ.b_T:=ε+T . Then the initial-hit factor hT:=ℙ[τ1≤T]h_T:=P[ _1≤ T] satisfies hT≤ℙ[|w0|≤bT]+δupd.h_T≤P [|w_0|≤ b_T ]+ _upd. (5) Proof. Let ℰΔE_ be the bounded-update good event from Assumption 3.3. On ℰΔE_ , if τ1≤T _1≤ T, then there exists t≤Tt≤ T such that |wt|≤ϵ|w_t|≤ε. By the triangle inequality and bounded updates, |w0|≤|wt|+∑s=0t−1|ws+1−ws|≤ϵ+TΔ=bT.|w_0|≤|w_t|+ _s=0^t-1|w_s+1-w_s|≤ε+T =b_T. Hence τ1≤T∩ℰΔ⊆|w0|≤bT\ _1≤ T\∩E_ \|w_0|≤ b_T\, which implies Eq. (5). ∎ D.3 An SGD-based Sufficient Condition for Assumption 3.4 That Controls Re-entry Ratio This appendix provides a typical sufficient condition for Assumption 3.4 by combining: (i) bounded increments (Assumption 3.3), (i) an expected descent-lemma argument for scheduled SGD that yields cumulative inward drift control in expectation, and (i) a variance-aware martingale concentration argument (Freedman’s inequality). Importantly, the proof below does not rely on any SGD-specific property beyond bounded increments and a descent-type energy budget, and thus serves as a template argument for a broad class of SGD-family optimizers as discussed in Remark D.11. Proof sketch (roadmap for Proposition D.10). Proposition D.10 provides a schedule-dependent sufficient condition for the re-entry control in Assumption 3.4 under scheduled SGD (Assumption D.5) together with bounded increments (Assumption 3.3). The crucial point is that, under standard training settings, even if a local inward drift toward the sign boundary exists, it is budget-limited by the finite gradient-energy available for task optimization via telescoping expected descent over a bounded loss range. One can violate this condition by introducing an artificial and sufficiently strong inward drift, in which case the training dynamics undergo a qualitative transition into the sign floating mode D.6. The logical dependencies among the auxiliary lemmas are: Lemma D.6⇒Lemma D.7⇒(Lemma D.8&Lemma D.9)⇒Proposition D.10.Lemma~ lem:appF_expected_descent_step\; \;Lemma~ lem:appF_l2_grad\; \; (Lemma~ lem:appF_cum_drift\ \&\ Lemma~ lem:H4a )\; \;Proposition~ prop:appF_ass2_from_sgd. • Expected descent ⇒ gradient-energy budget. Lemma D.6 establishes the one-step expected descent inequality (Eq. (6)). Summing and telescoping this inequality over a (possibly random) time window yields Lemma D.7, which upper bounds the conditional gradient energy ∑tηt‖∇ℒ(vt)‖2 _t _t\|∇L(v_t)\|^2 by schedule-dependent cumulative quantities and the loss range. • Gradient-energy budget ⇒ drift budget. Lemma D.8 converts the gradient-energy control in Lemma D.7 into a bound on the cumulative inward predictable drift of the oriented coordinate process. In parallel, Lemma D.9 provides a closely related (and proof-wise parallel) bound for the drift-budget proxy U~ U, which is tailored for a subsequent Markov-inequality step. • Drift-good reduction on an excursion. In the proof of Proposition D.10, we condition on ℱσkF_ _k (Definition 3.2) and orient the excursion via zt=skwtz_t=s_kw_t. We then decompose the excursion dynamics into predictable drift dtd_t and martingale noise. We introduce a drift-good event drift G_drift ensuring that the total inward drift is at most BTSGDB_T^SGD (defined in Eq. (7)); Lemma D.9 combined with Markov’s inequality yields ℙ(driftc∣ℱσk)≤δP( G_drift^c F_ _k)≤δ. • Noise control via Freedman + union bound. On drift G_drift, reaching the boundary τk+1≤T\ _k+1≤ T\ can only happen if the martingale noise attains a negative excursion of size roughly (ρ−ϵ)−BTSGD(ρ-ε)-B_T^SGD. We enforce bounded increments through Assumption 3.3 by stopping at the first update-violation time, and we control the predictable quadratic variation using Eq. (9). Freedman’s inequality then yields the exponential term appearing in gTSGDg_T^SGD (Eq. (7)). Finally, a union bound over (i) update-violation, (i) drift-bad, and (i) martingale-deviation events completes the bound in Eq. (8). Finally, Remark D.11 explains how the two inputs—bounded updates (Assumption 3.3) and re-entry control (Assumption 3.4)—interface with momentum/Adam-type recursions via Eqs. (10)–(11). Setup. Recall the coordinate process wt:=ei⊤vtw_t:=e_i v_t and the stopping times σk _k (outer-entry) and τk+1 _k+1 (boundary-hit) from Definition 3.2. Fix k≥0k≥ 0 and condition on ℱσkF_ _k. Let sk:=sign(wσk)∈±1s_k:=sign(w_ _k)∈\± 1\ (well-defined on σk≤T\ _k≤ T\ since |wσk|≥ρ|w_ _k|≥ρ), and define the oriented coordinate zt:=skwt(t≥σk).z_t:=s_k\,w_t~~(t≥ _k). On the excursion interval σk≤t<τk+1\ _k≤ t< _k+1\ we have zt≥ϵz_t≥ε and zσk≥ρz_ _k≥ρ. Moreover, for the favorable event ℰΔ(σk)E_ ( _k) from Assumption 3.3, the increments satisfy |zt+1−zt|=|wt+1−wt|≤Δ∀t∈σk,…,T−1.|z_t+1-z_t|=|w_t+1-w_t|≤ ~~∀ t∈\ _k,…,T-1\. Remark. Assumption 3.3 can be enforced, for example, by deterministic gradient clipping: if ‖gt‖≤G\|g_t\|≤ G almost surely and the step-size schedule satisfies ηt≤ηmax _t≤ _ for all t, then one can take Δ=ηmaxG = _ G and δupd=0 _upd=0. More generally, if such a bound holds with high probability uniformly over the horizon, then δupd _upd captures the corresponding failure probability. An expected descent assumption for scheduled SGD. We use a standard smoothness-based expected descent inequality for the (population) objective ℒL as below. Assumption D.5 (Smooth objective and scheduled SGD expected descent). There exists a differentiable function ℒ:ℝd→ℝL:R^d→R and constants Lsm>0L_sm>0 and ℒ⋆∈ℝL_ ∈R such that: (i) (LsmL_sm-smoothness) ℒL has LsmL_sm-Lipschitz gradient. (i) (Lower bound) ℒ(v)≥ℒ⋆L(v)≥L_ for all v. (i) (Scheduled SGD step) vt+1=vt−ηtgtv_t+1=v_t- _tg_t where (gt)(g_t) is adapted to (ℱt)(F_t) and [gt∣ℱt]=∇ℒ(vt),[‖gt−∇ℒ(vt)‖2∣ℱt]≤ξ2for some ξ≥0.E[g_t F_t]=∇L(v_t),~~E[\|g_t-∇L(v_t)\|^2 F_t]≤ξ^2~~for some ξ≥ 0. (iv) (Step-size schedule) (ηt)t≥0( _t)_t≥ 0 is deterministic (or predictable) with 0≤ηt≤1/Lsm0≤ _t≤ 1/L_sm for all t. (v) (Bounded loss on the horizon) there exists ℒmax∈ℝL_ ∈R such that ℒ(vt)≤ℒmaxL(v_t)≤L_ for all t≤Tt≤ T. We also define Δℒmax:=ℒmax−ℒ⋆ L_ :=L_ -L_ . Lemma D.6 (One-step expected descent scheduled SGD). Assume Assumption D.5. Then for all t≥0t≥ 0, [ℒ(vt+1)∣ℱt]≤ℒ(vt)−ηt2‖∇ℒ(vt)‖2+Lsmηt22ξ2.E[L(v_t+1) F_t]≤L(v_t)- _t2\,\|∇L(v_t)\|^2+ L_sm _t^22\,ξ^2. (6) Proof. By LsmL_sm-smoothness, for any u we have ℒ(u)≤ℒ(vt)+⟨∇ℒ(vt),u−vt⟩+Lsm2‖u−vt‖2L(u)≤L(v_t)+ ∇L(v_t),u-v_t + L_sm2\|u-v_t\|^2. Plugging u=vt+1=vt−ηtgtu=v_t+1=v_t- _tg_t gives ℒ(vt+1)≤ℒ(vt)−ηt⟨∇ℒ(vt),gt⟩+Lsmηt22‖gt‖2.L(v_t+1)≤L(v_t)- _t ∇L(v_t),g_t + L_sm _t^22\,\|g_t\|^2. Taking [⋅∣ℱt]E[· F_t], using unbiasedness [gt∣ℱt]=∇ℒ(vt)E[g_t F_t]=∇L(v_t), and [‖gt‖2∣ℱt]≤‖∇ℒ(vt)‖2+ξ2E[\|g_t\|^2 F_t]≤\|∇L(v_t)\|^2+ξ^2, we obtain [ℒ(vt+1)∣ℱt]≤ℒ(vt)−ηt‖∇ℒ(vt)‖2+Lsmηt22(‖∇ℒ(vt)‖2+ξ2).E[L(v_t+1) F_t]≤L(v_t)- _t\|∇L(v_t)\|^2+ L_sm _t^22 (\|∇L(v_t)\|^2+ξ^2 ). Finally, since ηt≤1/Lsm _t≤ 1/L_sm we have 1−Lsmηt2≥121- L_sm _t2≥ 12, yielding Lemma D.6. ∎ Lemma D.7 (Conditional weighted squared-gradient bound). Assume Assumption D.5. Fix any stopping time θ≤Tθ≤ T and any integer n≥1n≥ 1 with θ+n≤Tθ+n≤ T. Then [∑t=θ+n−1ηt∥∇ℒ(vt)∥2|ℱθ]≤2Δℒmax+Lsmξ2∑t=θ+n−1ηt2.E\! [ _t=θ^θ+n-1 _t\,\|∇L(v_t)\|^2\, |\,F_θ ]≤ 2\, L_ +L_smξ^2 _t=θ^θ+n-1 _t^2. Proof. Apply Eq. (6) and take conditional expectations given ℱθF_θ. Summing from t=θt=θ to θ+n−1θ+n-1 and telescoping yields [ℒ(vθ+n)∣ℱθ]≤ℒ(vθ)−12[∑t=θ+n−1ηt∥∇ℒ(vt)∥2|ℱθ]+Lsmξ22∑t=θ+n−1ηt2.E[L(v_θ+n) F_θ]≤L(v_θ)- 12\,E\! [ _t=θ^θ+n-1 _t\,\|∇L(v_t)\|^2\, |\,F_θ ]+ L_smξ^22 _t=θ^θ+n-1 _t^2. Using ℒ(vθ+n)≥ℒ⋆L(v_θ+n)≥L_ and ℒ(vθ)≤ℒmaxL(v_θ)≤L_ (since θ≤Tθ≤ T) gives Lemma D.7. ∎ Lemma D.8 (Cumulative inward drift control in expectation). Assume Assumption D.5. Fix any k≥0k≥ 0 and define, for t≥σkt≥ _k, dt:=[zt+1−zt∣ℱt].d_t:=E\! [z_t+1-z_t F_t ]. Then for any integer n≥1n≥ 1 with σk+n≤T _k+n≤ T, [−∑t=σkσk+n−1dt|ℱσk]≤B¯n,E\! [- _t= _k _k+n-1d_t\, |\,F_ _k ]\;≤\; B_n, where (writing Λk,n:=∑t=σkσk+n−1ηt _k,n:= _t= _k _k+n-1 _t and Λk,n(2):=∑t=σkσk+n−1ηt2 ^(2)_k,n:= _t= _k _k+n-1 _t^2) B¯n:=Λk,nSk,n,Sk,n:=2Δℒmax+ξ2Λk,n+Lsmξ2Λk,n(2). B_n:= _k,n\,S_k,n,~~S_k,n:=2\, L_ +ξ^2 _k,n+L_smξ^2 ^(2)_k,n. In particular, letting ΛT:=∑t=0T−1ηt _T:= _t=0^T-1 _t and ΛT(2):=∑t=0T−1ηt2 _T^(2):= _t=0^T-1 _t^2, we have B¯n≤B¯T B_n≤ B_T for all n≤T−σkn≤ T- _k, where B¯T:=ΛT(2Δℒmax+ξ2ΛT+Lsmξ2ΛT(2)). B_T:= _T (2\, L_ +ξ^2 _T+L_smξ^2 _T^(2) ). Proof. By the scheduled SGD update, wt+1−wt=−ηtei⊤gtw_t+1-w_t=- _t\,e_i g_t, hence zt+1−zt=sk(wt+1−wt)=−ηtskei⊤gtz_t+1-z_t=s_k(w_t+1-w_t)=- _t\,s_k\,e_i g_t. Therefore −dt=ηt[skei⊤gt∣ℱt]≤ηt[‖gt‖∣ℱt]≤ηt[‖gt‖2∣ℱt],-d_t= _t\,E\! [s_k\,e_i g_t F_t ]≤ _t\,E[\|g_t\| F_t]≤ _t\, E[\|g_t\|^2 F_t], where we used |sk|=1|s_k|=1, |ei⊤x|≤‖x‖|e_i x|≤\|x\|, and Jensen. Summing from t=σkt= _k to σk+n−1 _k+n-1 and applying Cauchy–Schwarz conditionally on ℱσkF_ _k, [−∑t=σkσk+n−1dt|ℱσk]≤Λk,n[∑t=σkσk+n−1ηt[∥gt∥2∣ℱt]|ℱσk].E\! [- _t= _k _k+n-1d_t\, |\,F_ _k ]≤ _k,n\, E\! [ _t= _k _k+n-1 _t\,E[\|g_t\|^2 F_t]\, |\,F_ _k ]. Moreover, using [‖gt‖2∣ℱt]≤‖∇ℒ(vt)‖2+ξ2E[\|g_t\|^2 F_t]≤\|∇L(v_t)\|^2+ξ^2 and Lemma D.7 with θ=σkθ= _k yields [∑t=σkσk+n−1ηt[∥gt∥2∣ℱt]|ℱσk]≤(2Δℒmax+Lsmξ2Λk,n(2))+ξ2Λk,n=Sk,n.E\! [ _t= _k _k+n-1 _t\,E[\|g_t\|^2 F_t]\, |\,F_ _k ]≤ (2\, L_ +L_smξ^2 ^(2)_k,n )+ξ^2 _k,n=S_k,n. Combining proves Lemma D.8. The final inequality B¯n≤B¯T B_n≤ B_T follows from Λk,n≤ΛT _k,n≤ _T and Λk,n(2)≤ΛT(2) ^(2)_k,n≤ _T^(2). ∎ Lemma D.9 (Conditional bound for the drift-budget proxy U~ U). Assume Assumption D.5. Fix any stopping time θ≤Tθ≤ T and any integer n≥1n≥ 1 with θ+n≤Tθ+n≤ T. Define u~t:=ηt[‖gt‖∣ℱt],U~θ,n:=∑t=θ+n−1u~t. u_t:= _t\,E [\|g_t\| F_t ],~~ U_θ,n:= _t=θ^θ+n-1 u_t. Let Λθ,n:=∑t=θ+n−1ηt,Λθ,n(2):=∑t=θ+n−1ηt2,Δℒmax:=ℒmax−ℒ⋆. _θ,n:= _t=θ^θ+n-1 _t,~~ ^(2)_θ,n:= _t=θ^θ+n-1 _t^2,~~ L_ :=L_ -L_ . Then [U~θ,n∣ℱθ]≤B¯θ,n:=Λθ,nSθ,n,E [ U_θ,n F_θ ]\;≤\; B_θ,n:= _θ,n\,S_θ,n, where Sθ,n:=2Δℒmax+ξ2Λθ,n+Lsmξ2Λθ,n(2).S_θ,n:=2 L_ +ξ^2 _θ,n+L_ sm\,ξ^2\, ^(2)_θ,n. In particular, since ηt≥0 _t≥ 0 for all t (Assumption D.5(iv)), for any stopping time θ≤Tθ≤ T and any n≥1n≥ 1 with θ+n≤Tθ+n≤ T, the random partial sums satisfy almost surely Λθ,n≤ΛTandΛθ,n(2)≤ΛT(2), _θ,n≤ _T~~and~~ ^(2)_θ,n≤ _T^(2), and hence Sθ,n≤STS_θ,n≤ S_T and B¯θ,n≤B¯T B_θ,n≤ B_T almost surely, where ST:=2Δℒmax+ξ2ΛT+Lsmξ2ΛT(2),B¯T:=ΛTST.S_T:=2 L_ +ξ^2 _T+L_ smξ^2 _T^(2),~~ B_T:= _TS_T. Consequently, [U~θ,n∣ℱθ]≤B¯T.E [ U_θ,n F_θ ]≤ B_T. Proof. By the tower property, [U~θ,n∣ℱθ]=∑t=θ+n−1ηt[‖gt‖∣ℱθ].E [ U_θ,n F_θ ]= _t=θ^θ+n-1 _t\,E [\|g_t\| F_θ ]. Using Jensen’s inequality [‖gt‖∣ℱθ]≤[‖gt‖2∣ℱθ]E[\|g_t\| F_θ]≤ E[\|g_t\|^2 F_θ] and conditional Cauchy–Schwarz, [U~θ,n∣ℱθ]≤Λθ,n∑t=θ+n−1ηt[‖gt‖2∣ℱθ].E [ U_θ,n F_θ ]≤ _θ,n _t=θ^θ+n-1 _t\,E [\|g_t\|^2 F_θ ]. Moreover, [‖gt‖2∣ℱt]≤‖∇ℒ(vt)‖2+ξ2E[\|g_t\|^2 F_t]≤\|∇L(v_t)\|^2+ξ^2 by Assumption D.5(i), hence [∥gt∥2∣ℱθ]=[[∥gt∥2∣ℱt]|ℱθ]≤[∥∇ℒ(vt)∥2∣ℱθ]+ξ2.E [\|g_t\|^2 F_θ ]=E\! [E [\|g_t\|^2 F_t ] |F_θ ]≤E [\|∇L(v_t)\|^2 F_θ ]+ξ^2. Therefore, ∑t=θ+n−1ηt[∥gt∥2∣ℱθ]≤[∑t=θ+n−1ηt∥∇ℒ(vt)∥2|ℱθ]+ξ2Λθ,n. _t=θ^θ+n-1 _t\,E [\|g_t\|^2 F_θ ]≤E\! [ _t=θ^θ+n-1 _t\|∇L(v_t)\|^2 |F_θ ]+ξ^2 _θ,n. Applying Lemma D.7 gives [∑t=θ+n−1ηt∥∇ℒ(vt)∥2|ℱθ]≤2Δℒmax+Lsmξ2Λθ,n(2).E\! [ _t=θ^θ+n-1 _t\|∇L(v_t)\|^2 |F_θ ]≤ 2 L_ +L_ smξ^2 ^(2)_θ,n. Combining the last three displays yields [U~θ,n∣ℱθ]≤Λθ,n(2Δℒmax+ξ2Λθ,n+Lsmξ2Λθ,n(2))=B¯θ,n.E [ U_θ,n F_θ ]≤ _θ,n (2 L_ +ξ^2 _θ,n+L_ smξ^2 ^(2)_θ,n )= B_θ,n. It remains to justify the uniform bound by B¯T B_T. Since ηt≥0 _t≥ 0 for all t, for each outcome ω we have Λθ,n(ω)=∑t=0T−1ηt 1θ(ω)≤t≤θ(ω)+n−1≤∑t=0T−1ηt=ΛT, _θ,n(ω)= _t=0^T-1 _t\,1\θ(ω)≤ t≤θ(ω)+n-1\≤ _t=0^T-1 _t= _T, and similarly, Λθ,n(2)(ω)=∑t=0T−1ηt2 1θ(ω)≤t≤θ(ω)+n−1≤∑t=0T−1ηt2=ΛT(2). ^(2)_θ,n(ω)= _t=0^T-1 _t^2\,1\θ(ω)≤ t≤θ(ω)+n-1\≤ _t=0^T-1 _t^2= _T^(2). Therefore Sθ,n(ω)≤STS_θ,n(ω)≤ S_T and hence B¯θ,n(ω)=Λθ,n(ω)Sθ,n(ω)≤ΛTST=B¯Tfor all ω, B_θ,n(ω)= _θ,n(ω)\,S_θ,n(ω)≤ _TS_T= B_T all ω, which proves [U~θ,n∣ℱθ]≤B¯TE[ U_θ,n F_θ]≤ B_T. ∎ Proposition D.10 (Re-entry bound in SGD). Assume Assumptions 3.3 and D.5. Recall Δℒmax:=ℒmax−ℒ⋆ L_ :=L_ -L_ and define the cumulative schedule quantities ΛT:=∑t=0T−1ηt,ΛT(2):=∑t=0T−1ηt2. _T:= _t=0^T-1 _t,~~ _T^(2):= _t=0^T-1 _t^2. Fix δ∈(0,1)δ∈(0,1) and define B¯T B_T :=ΛT(2Δℒmax+ξ2ΛT+Lsmξ2ΛT(2)), = _T (2\, L_ +ξ^2 _T+L_smξ^2 _T^(2) ), (7) BTSGD B_T^SGD :=1δB¯T=1δΛT(2Δℒmax+ξ2ΛT+Lsmξ2ΛT(2)), = 1δ\, B_T= 1δ _T (2\, L_ +ξ^2 _T+L_smξ^2 _T^(2) ), VTSGD V_T^SGD :=ξ2∑t=0T−1ηt2=ξ2ΛT(2), =ξ^2 _t=0^T-1 _t^2=ξ^2 _T^(2), aT a_T :=((ρ−ϵ)−BTSGD)+, = ((ρ-ε)-B_T^SGD )_+, gTSGD g_T^SGD :=δupd+δ+exp(−aT22(VTSGD+2Δ3aT)). = _upd+δ+ \! (- a_T^22 (V_T^SGD+ 2 3a_T ) ). Then for every k≥0k≥ 0, ℙ[τk+1≤T∣ℱσk]≤gTSGDon σk≤T.P[ _k+1≤ T F_ _k]≤ g_T^SGD~~on \ _k≤ T\. (8) In particular, if gTSGD<1g_T^SGD<1 (e.g., δupd _upd and δ are small and (ρ−ϵ)>BTSGD(ρ-ε)>B_T^SGD), then Assumption 3.4 holds. Proof. Fix k and condition on ℱσkF_ _k. If σk=T _k=T, then τk+1>T _k+1>T by definition and the trivial result ℙ[τk+1≤T∣ℱσk]=0≤gTSGD,P[ _k+1≤ T F_ _k]=0≤ g_T^SGD, holds. Hence assume σk≤T−1 _k≤ T-1. Shifted filtration and shifted stopping times. Define the shifted filtration ℋm:=ℱσk+m(m≥0).H_m:=F_ _k+m~~(m≥ 0). We use ℋmH_m to avoid confusion with the global notation Gt:=ℱσk+tG_t:=F_ _k+t. Define shifted stopping times (w.r.t. (ℋm)m≥0(H_m)_m≥ 0) τ′:=τk+1−σk,ν′:=ν−σk,τ := _k+1- _k,~~ν :=ν- _k, where ν is defined below. Note that τ′∈1,2,…∪∞τ ∈\1,2,…\∪\∞\ and ν′∈0,1,2,…∪∞ν ∈\0,1,2,…\∪\∞\. Define the (random but ℱσkF_ _k-measurable) horizon T(k):=T−σk.T^(k):=T- _k. Under the conditioning on ℱσkF_ _k, T(k)T^(k) is deterministic. First update-violation time. Define the first update-violation time ν:=inft≥σk:|wt+1−wt|>Δ,ν:= \t≥ _k:\ |w_t+1-w_t|> \, with the convention inf∅=∞ =∞. Note that ν≤T−1=ℰΔ(σk)c\ν≤ T-1\=E_ ( _k)^c. By Assumption 3.3 applied at the stopping time θ=σkθ= _k, ℙ[ν≤T−1∣ℱσk]≤δupd.P[ν≤ T-1 F_ _k]≤ _upd. Martingale differences and conditional variance bound. Define the predictable drift increments dtd_t as in Lemma D.8 and the martingale differences Xt:=(zt+1−zt)−dt.X_t:=(z_t+1-z_t)-d_t. Using zt+1−zt=−ηtskei⊤gtz_t+1-z_t=- _ts_ke_i g_t and dt=[zt+1−zt∣ℱt]=−ηtskei⊤∇ℒ(vt)d_t=E[z_t+1-z_t F_t]=- _ts_ke_i ∇L(v_t), we have Xt=−ηtskei⊤(gt−∇ℒ(vt)),X_t=- _ts_ke_i (g_t-∇L(v_t) ), and therefore Var(Xt∣ℱt)≤[Xt2∣ℱt]≤ηt2[‖gt−∇ℒ(vt)‖2∣ℱt]≤ηt2ξ2.Var(X_t F_t)≤E[X_t^2 F_t]≤ _t^2\,E[\|g_t-∇L(v_t)\|^2 F_t]≤ _t^2ξ^2. (9) Drift-good event. Let n:=minT−σk,τk+1−σkn:= \T- _k, _k+1- _k\ and define u~t:=ηt[‖gt‖∣ℱt] u_t:= _tE[\|g_t\| F_t] and U~m:=∑t=σkσk+m−1u~t U_m:= _t= _k _k+m-1 u_t (with U~0:=0 U_0:=0). Consider the drift-good event Gdrift:=U~n≤BTSGD.G_ drift:=\ U_n≤ B_T SGD\. Since U~n≤U~T−σk U_n≤ U_T- _k and U~T−σk≥0 U_T- _k≥ 0, Markov’s inequality yields ℙ(Gdriftc∣ℱσk)=ℙ(U~n>BTSGD∣ℱσk)≤[U~T−σk∣ℱσk]BTSGD.P(G_ drift^c F_ _k)=P( U_n>B_T SGD F_ _k)≤ E[ U_T- _k F_ _k]B_T SGD. By Lemma D.9 (applied with θ=σkθ= _k and horizon T−σkT- _k), [U~T−σk∣ℱσk]≤B¯T.E[ U_T- _k F_ _k]≤ B_T. Therefore, using BTSGD=B¯T/δB_T SGD= B_T/δ, we conclude ℙ(Gdriftc∣ℱσk)≤δ.P(G_ drift^c F_ _k)≤δ. On drift G_drift, for any m≤nm≤ n we have ∑t=σkσk+m−1dt≥−U~m≥−U~n≥−BTSGD _t= _k _k+m-1d_t≥- U_m≥- U_n≥-B_T^SGD. Thus, zσk+m=zσk+∑t=σkσk+m−1dt+∑t=σkσk+m−1Xt≥ρ−BTSGD+∑t=σkσk+m−1Xt,z_ _k+m=z_ _k+ _t= _k _k+m-1d_t+ _t= _k _k+m-1X_t≥ρ-B_T^SGD+ _t= _k _k+m-1X_t, where we used zσk≥ρz_ _k≥ρ. Consequently, on min0≤m≤nzσk+m≤ϵ∩drift\ _0≤ m≤ nz_ _k+m≤ε\∩ G_drift we must have min0≤m≤n∑t=σkσk+m−1Xt≤−(ρ−ϵ)+BTSGD. _0≤ m≤ n\ _t= _k _k+m-1X_t≤-(ρ-ε)+B_T^SGD. Stopped martingale and Freedman under shifted filtration. Let x:=((ρ−ϵ)−BTSGD)+,x:= ((ρ-ε)-B_T^SGD )_+, so x=aTx=a_T in Proposition D.10. Define shifted increments Ym:=Xσk+mY_m:=X_ _k+m for m≥0m≥ 0. Then (Ym)m≥0(Y_m)_m≥ 0 is a martingale difference sequence w.r.t. (ℋm)m≥0(H_m)_m≥ 0. Define the stopped increments and stopped martingale Y¯m:=Ym 1m<ν′,M¯m:=∑j=0m−1Y¯j(m≥0),M¯0:=0. Y_m:=Y_m\,1\m<ν \,~~ M_m:= _j=0^m-1 Y_j\ \ (m≥ 0),~~ M_0:=0. Then (M¯m)m≥0( M_m)_m≥ 0 is a martingale w.r.t. (ℋm)(H_m). Increment bound (holds a.s. without conditioning on ν>T−1\ν>T-1\): Since ν is a stopping time, m<ν′∈ℋm\m<ν \∈H_m. On m<ν′\m<ν \, we have |wσk+m+1−wσk+m|≤Δ|w_ _k+m+1-w_ _k+m|≤ , hence |zσk+m+1−zσk+m|≤Δ|z_ _k+m+1-z_ _k+m|≤ and thus |dσk+m|=|[zσk+m+1−zσk+m∣ℋm]|≤[|zσk+m+1−zσk+m|∣ℋm]≤Δ.|d_ _k+m|= |E[z_ _k+m+1-z_ _k+m H_m] |≤E[\,|z_ _k+m+1-z_ _k+m|\, H_m]≤ . Therefore on m<ν′\m<ν \, |Ym|=|Xσk+m|≤|zσk+m+1−zσk+m|+|dσk+m|≤2Δ|Y_m|=|X_ _k+m|≤|z_ _k+m+1-z_ _k+m|+|d_ _k+m|≤ 2 . Since Y¯m=0 Y_m=0 on m≥ν′\m≥ν \, we obtain the uniform a.s. bound |Y¯m|≤2Δfor all m≥0a.s.| Y_m|≤ 2 ~~for all m≥ 0\ a.s. Conditional variance bound: Using Eq. (9) and Var(Y¯m∣ℋm)≤Var(Ym∣ℋm)Var( Y_m H_m) (Y_m H_m), we have Var(Y¯m∣ℋm)≤ξ2ησk+m2.Var( Y_m H_m)≤ξ^2 _ _k+m^2. Define the predictable quadratic variation up to m: V¯m:=∑j=0m−1Var(Y¯j∣ℋj). V_m:= _j=0^m-1Var( Y_j H_j). Then for all m≤T(k)m≤ T^(k), V¯m≤ξ2∑t=σkσk+m−1ηt2≤ξ2∑t=0T−1ηt2=VTSGD. V_m≤ξ^2 _t= _k _k+m-1 _t^2≤ξ^2 _t=0^T-1 _t^2=V_T^SGD. From the drift-good reduction above, on drift G_drift and τk+1≤T\ _k+1≤ T\ we have min0≤m≤n∑t=σkσk+m−1Xt≤−x. _0≤ m≤ n\ _t= _k _k+m-1X_t≤-x. If additionally ν>T−1\ν>T-1\ holds, then ν′>T(k)ν >T^(k) so m<ν′m<ν for all m≤T(k)m≤ T^(k), hence M¯m=∑j=0m−1Yj=∑t=σkσk+m−1Xt M_m= _j=0^m-1Y_j= _t= _k _k+m-1X_t for all m≤T(k)m≤ T^(k). Therefore, τk+1≤T∩ν>T−1∩drift⊆min0≤m≤T(k)M¯m≤−x.\ _k+1≤ T\∩\ν>T-1\∩ G_drift \ _0≤ m≤ T^(k) M_m≤-x \. Freedman’s inequality (maximal form) applied to (M¯m)( M_m): Equivalently, apply Freedman’s inequality to the martingale (−M¯m)(- M_m) to bound the lower-tail event. With increment bound 2Δ2 and variance bound VTSGDV_T^SGD, we obtain ℙ[min0≤m≤T(k)M¯m≤−x|ℱσk]≤exp(−x22(VTSGD+2Δ3x)).P\! [ _0≤ m≤ T^(k) M_m≤-x\, |\,F_ _k ]≤ \! (- x^22 (V_T^SGD+ 2 3x ) ). Consequently, ℙ[τk+1≤T,ν>T−1,drift∣ℱσk]≤exp(−x22(VTSGD+2Δ3x)).P[ _k+1≤ T,\ ν>T-1,\ G_drift F_ _k]≤ \! (- x^22 (V_T^SGD+ 2 3x ) ). Union bound completion. Finally, ℙ[τk+1≤T∣ℱσk] P[ _k+1≤ T F_ _k] ≤ℙ[ν≤T−1∣ℱσk]+ℙ[driftc∣ℱσk]+ℙ[τk+1≤T,ν>T−1,drift∣ℱσk] ≤P[ν≤ T-1 F_ _k]+P[ G_drift^c F_ _k]+P[ _k+1≤ T,\ ν>T-1,\ G_drift F_ _k] ≤δupd+δ+exp(−x22(VTSGD+2Δ3x)), ≤ _upd+δ+ \! (- x^22 (V_T^SGD+ 2 3x ) ), which is exactly Proposition D.10. ∎ Remark D.11 (How the proofs interface with momentum / AdamW). Theorem 3.6 is invoked through only two ingredients: (i) the bounded-update condition (Assumption 3.3), and (i) the Re-entry control (Assumption 3.4). In contrast, Proposition D.10 is merely a scheduled-SGD sufficient condition for verifying (i). Therefore, any optimizer for which one can verify (i)–(i) yields the same geometric-tail conclusion. As a generic template, consider a preconditioned momentum recursion pt+1=βtmompt−ηtDtgt,vt+1=vt+pt+1,p_t+1=β^mom_t\,p_t- _tD_tg_t,~~v_t+1=v_t+p_t+1, (10) with p0=0p_0=0 and a diagonal ℱtF_t-measurable preconditioner DtD_t. Since vt+1−vt=pt+1v_t+1-v_t=p_t+1, Assumption 3.3 holds on any good event on which ‖pt+1‖∞≤Δfor all t≤T−1,\|p_t+1\|_∞≤ ~~for all t≤ T-1, (11) (with failure probability given by the complement of the good event). A convenient sufficient condition for Eq. (11) is, for example, the conjunction of ‖ηtDtgt‖∞≤(1−β)Δ\| _tD_tg_t\|_∞≤(1-β) for all t≤T−1t≤ T-1 and supt≤T−1|βtmom|≤β<1 _t≤ T-1|β^mom_t|≤β<1 (so that Eq. (11) follows by a simple induction starting from p0=0p_0=0). For Adam/AdamW, one may instantiate Eq. (10) with Dt=diag((s^t+ϵadam)−1)D_t=diag\! (( s_t+ _adam)^-1 ) and gt=m^tg_t= m_t. Thus it suffices (along the trajectory / on a good event) to control ‖m^t‖∞\| m_t\|_∞, to ensure a uniform lower bound s^t+ϵadam≥c>0 s_t+ _adam≥ c>0, and to verify Eq. (11) for the resulting effective step pt+1p_t+1. For AdamW, the decoupled weight-decay term can be absorbed into the same bounded-update event (e.g., by including it in the definition of pt+1p_t+1 and bounding its l∞l_∞-magnitude). D.4 Practical Implication of Sign Lock-In Theory In this section, we translate our theoretical results into design principles for modern training pipelines. The key observation is that, by Theorem 3.6, the statistics of effective sign flips are characterized by the initial-hit factor hTh_T and the re-entry ratio gTg_T. For the initial-hit factor hTh_T, Appendix D.2 and Proposition D.4 provide an initialization-dominated upper bound, showing that lock-in becomes stronger as the probability of reaching the boundary band decreases. For the re-entry ratio gTg_T, satisfying Assumption 3.4 is essential, and Proposition D.10 justifies this assumption by giving a sufficient condition that upper bounds gTg_T in terms of SGD and schedule-dependent cumulative quantities. Below, through these two controlling factors, we systematically summarize how learning-rate schedules, batch size, width, and scale-invariant mechanisms shift lock-in in either direction. • Appendix D.4.1 compares learning-rate schedules by instantiating the gTg_T upper bound from Proposition D.10 with the corresponding schedule-dependent cumulative quantities, and summarizes which schedules tend to yield stronger lock-in. • Appendix D.4.2 explains how increasing the minibatch size and model size suppresses stochastic gradient noise, thereby reducing re-entry behavior and decreasing gTg_T, which strengthens lock-in. • Appendix D.4.3 summarizes how scale-invariant mechanisms stabilize effective updates and thereby support the conditions in Assumption 3.3 and Assumption 3.4, resulting in stronger lock-in. • Appendix D.4.4 presents auxiliary consequences of sign dynamics, including an initial-hit bound that does not require an outer start assumption and the resulting front-loaded nature of sign changes. • Appendix D.4.5 provides numerical validation of these practical insights through the learning-rate schedule, batch-size, and width sweeps, together with fitted trends (h^,g^)( h, g). D.4.1 Ordering of Learning-Rate Schedules Under Fixed Computing Resource We compare four learning-rate schedules under the same training horizon T and the same peak step size ηmax _ : a constant learning rate, warmup with cosine decay, warmup with exponential decay, and warmup with inverse decay. Our goal is to order the resulting re-entry bounds using the schedule-aware sufficient condition of Proposition D.10 (Appendix D.3), which verifies Assumption 3.4 and hence yields the geometric-tail conclusion of Theorem 3.6. Reminder (where gTg_T enters). Assumption 3.4 postulates the existence of a re-entry bound gTg_T such that ℙ[τk+1≤T∣ℱσk]≤gT.P[ _k+1≤ T F_ _k]≤ g_T. Theorem 3.6 then converts this bound into a geometric-tail estimate for both ℙ(τk≤T)P( _k≤ T) and ℙ(KTeff(ρ)≥k)P(K_T^eff(ρ)≥ k). Appendix D.3 provides a schedule-aware sufficient condition: Proposition D.10 upper-bounds the re-entry probability by an explicit quantity gTSGDg_T^SGD, which depends on the learning-rate schedule only through the cumulative quantities ΛT=∑t<Tηt,ΛT(2)=∑t<Tηt2. _T= _t<T _t,~~ _T^(2)= _t<T _t^2. Schedule-specific notation. To distinguish schedules, we write gT,const,gT,cos,gT,exp,gT,invg_T,const, g_T,cos, g_T,exp, g_T,inv for the re-entry bounds associated with a constant learning rate, warmup with cosine decay, warmup with exponential decay, and warmup with inverse decay, respectively. When referring to the sufficient bound in Proposition D.10, we analogously write gT,constSGDg_T,const^SGD, gT,cosSGDg_T,cos^SGD, gT,expSGDg_T,exp^SGD, and gT,invSGDg_T,inv^SGD. Lemma D.12 (Monotonicity of the SGD re-entry bound). Fix the constants ρ,ϵ,Δ,δ,δupd,Lsm,ξ,Δℒmaxρ,ε, ,δ, _upd,L_sm,ξ, L_ as in Proposition D.10. For a schedule ηtt=0T−1\ _t\_t=0^T-1, define ΛT(1):=∑t=0T−1ηt,ΛT(2):=∑t=0T−1ηt2. ^(1)_T:= _t=0^T-1 _t,~~ ^(2)_T:= _t=0^T-1 _t^2. If two schedules satisfy ΛT(1)≤Λ~T(1),ΛT(2)≤Λ~T(2), ^(1)_T≤ ^(1)_T,~~ ^(2)_T≤ ^(2)_T, then their schedule-specific bounds satisfy gTSGD≤g~TSGD.g_T^SGD≤ g_T^SGD. Moreover, if at least one inequality is strict and the effective margin aT=((ρ−ϵ)−B~TSGD)+a_T= ((ρ-ε)- B_T^SGD )_+ for the larger schedule is strictly positive, then the inequality is strict. Proof. By Proposition D.10, gTSGD=δupd+δ+exp(−aT22(VTSGD+2Δ3aT)),g_T^SGD= _upd+δ+ \! (- a_T^22 (V_T^SGD+ 2 3a_T ) ), where VTSGD=ξ2ΛT(2),BTSGD=1δΛT(1)(2Δℒmax+ξ2ΛT(1)+Lsmξ2ΛT(2)).V_T^SGD=ξ^2 ^(2)_T,~~B_T^SGD= 1δ ^(1)_T (2 L_ +ξ^2 ^(1)_T+L_smξ^2 ^(2)_T ). Both VTSGDV_T^SGD and BTSGDB_T^SGD are nondecreasing in (ΛT(1),ΛT(2)) ( ^(1)_T, ^(2)_T ), hence aT=((ρ−ϵ)−BTSGD)+a_T= ((ρ-ε)-B_T^SGD )_+ is nonincreasing in those quantities. Therefore increasing either ΛT(1) ^(1)_T or ΛT(2) ^(2)_T can only weaken the exponent (make it less negative), so gTSGDg_T^SGD is nondecreasing in (ΛT(1),ΛT(2)) ( ^(1)_T, ^(2)_T ). This proves the first claim. The strictness statement follows when at least one inequality is strict and the larger schedule has aT>0a_T>0, so the exponential term changes strictly. ∎ Proposition D.13 (Learning-rate dependence of the re-entry ratio). Assume the hypotheses of Proposition D.10 and fix the constants ρ,ϵ,Δ,δ,δupd,Lsm,ξ,Δℒmaxρ,ε, ,δ, _upd,L_sm,ξ, L_ as in Lemma D.12. Let ηtt=0T−1\ _t\_t=0^T-1 and η~tt=0T−1\ η_t\_t=0^T-1 be two learning-rate schedules, and define ΛT(1):=∑t=0T−1ηt,ΛT(2):=∑t=0T−1ηt2,Λ~T(1):=∑t=0T−1η~t,Λ~T(2):=∑t=0T−1η~t2. ^(1)_T:= _t=0^T-1 _t, ^(2)_T:= _t=0^T-1 _t^2, ^(1)_T:= _t=0^T-1 η_t, ^(2)_T:= _t=0^T-1 η_t^2. If ΛT(1)≤Λ~T(1),ΛT(2)≤Λ~T(2), ^(1)_T≤ ^(1)_T, ^(2)_T≤ ^(2)_T, then the corresponding schedule-aware re-entry bounds satisfy gTSGD≤g~TSGD.g_T^SGD≤ g_T^SGD. Moreover, if at least one of the two inequalities above is strict and the effective margin associated with the larger schedule is nontrivial, a~T:=((ρ−ϵ)−B~TSGD)+>0, a_T:= ((ρ-ε)- B_T^SGD )_+>0, then the inequality is strict, i.e., gTSGD<g~TSGD.g_T^SGD< g_T^SGD. Furthermore, suppose that the learning-rate schedules are related by a multiplicative rescaling η~t=cηt η_t=c\, _t with a constant c>1c>1. Then the cumulative quantities satisfy Λ~T(1)=cΛT(1) ^(1)_T=c\, ^(1)_T and Λ~T(2)=c2ΛT(2) ^(2)_T=c^2\, ^(2)_T, and consequently the corresponding re-entry bounds obey g~TSGD≥gTSGD. g_T^SGD≥ g_T^SGD. If, in addition, the effective margin for the rescaled schedule satisfies a~T>0 a_T>0, the inequality is strict. Proof. The claim follows directly from Lemma D.12. By Proposition D.10, the explicit bound gTSGDg_T^SGD depends on the learning-rate schedule only through the cumulative quantities ΛT(1) ^(1)_T and ΛT(2) ^(2)_T, and is nondecreasing in each of them. The strictness statement follows from the strict monotonicity part of Lemma D.12 whenever the effective margin is positive. ∎ Lemma D.14 (Constant learning rate is maximal). Fix T and ηmax>0 _ >0. Among all schedules satisfying 0≤ηt≤ηmax0≤ _t≤ _ for all t, the constant learning rate ηt≡ηmax _t≡ _ maximizes both ΛT(1) ^(1)_T and ΛT(2) ^(2)_T. Proof. For any schedule with 0≤ηt≤ηmax0≤ _t≤ _ , ΛT(1)=∑t=0T−1ηt≤∑t=0T−1ηmax=Tηmax,ΛT(2)=∑t=0T−1ηt2≤∑t=0T−1ηmax2=Tηmax2, ^(1)_T= _t=0^T-1 _t≤ _t=0^T-1 _ =T _ ,~~ ^(2)_T= _t=0^T-1 _t^2≤ _t=0^T-1 _ ^2=T _ ^2, with equality for the constant schedule. ∎ Lemma D.15 (Cosine decay yields linear squared-step accumulation (exact form)). Let TwuT_wu denote the warmup length and N:=T−TwuN:=T-T_wu. Assume N≥2N≥ 2. For warmup with cosine decay, ηTwu+k=ηmax1+cos(πk/N)2,k=0,…,N−1, _T_wu+k= _ 1+ (π k/N)2,~~k=0,…,N-1, we have the exact identity ∑k=0N−1ηTwu+k2=38ηmax2N+12ηmax2. _k=0^N-1 _T_wu+k^2= 38\, _ ^2\,N+ 12\, _ ^2. Proof. Let ϑk:=πk/N _k:=π k/N and write ηTwu+k2=ηmax2(1+cosϑk2)2. _T_wu+k^2= _ ^2 ( 1+ _k2 )^2. Using cos2x=1+cos(2x)2 ^2x= 1+ (2x)2, (1+cosx2)2=38+12cosx+18cos(2x). ( 1+ x2 )^2= 38+ 12 x+ 18 (2x). Hence ∑k=0N−1ηTwu+k2=ηmax2(38N+12∑k=0N−1cosϑk+18∑k=0N−1cos(2ϑk)). _k=0^N-1 _T_wu+k^2= _ ^2 ( 38N+ 12 _k=0^N-1 _k+ 18 _k=0^N-1 (2 _k) ). We evaluate the trigonometric sums exactly. First, (using N≥2N≥ 2 so that ei2π/N≠1e^i2π/N≠ 1), ∑k=0N−1cos(2πkN)=Re∑k=0N−1ei2πk/N=Re(1−(ei2π/N)N1−ei2π/N)=0. _k=0^N-1 \! ( 2π kN )= _k=0^N-1e^i2π k/N= ( 1-(e^i2π/N)^N1-e^i2π/N )=0. Second, using the standard closed form ∑k=0N−1cos(kθ)=sin(Nθ/2)cos((N−1)θ/2)sin(θ/2) _k=0^N-1 (kθ)= (Nθ/2) ((N-1)θ/2) (θ/2) with θ=π/Nθ=π/N, we obtain ∑k=0N−1cos(πkN)=sin(π/2)cos((N−1)π/(2N))sin(π/(2N))=cos(π/2−π/(2N))sin(π/(2N))=1. _k=0^N-1 \! ( π kN )= (π/2) ((N-1)π/(2N)) (π/(2N))= (π/2-π/(2N)) (π/(2N))=1. Substituting gives ∑k=0N−1ηTwu+k2=ηmax2(38N+12⋅1+18⋅0)=38ηmax2N+12ηmax2. _k=0^N-1 _T_wu+k^2= _ ^2 ( 38N+ 12· 1+ 18· 0 )= 38\, _ ^2\,N+ 12\, _ ^2. ∎ Lemma D.16 (Exponential versus cosine decay). With the same TwuT_wu and N:=T−TwuN:=T-T_wu, consider warmup with exponential decay ηTwu+k=ηmaxγk _T_wu+k= _ γ^k (0<γ<10<γ<1). If both N+12≤1−γN1−γand38N+12≤1−γ2N1−γ2, N+12≤ 1-γ^N1-γ~~and~~ 38N+ 12≤ 1-γ^2N1-γ^2, then gT,cosSGD≤gT,expSGD.g_T,cos^SGD≤ g_T,exp^SGD. Proof. Since the warmup phase is common, it suffices to compare the post-warmup tails. Define the tail cumulative sums Λcos(1):=∑k=0N−1ηmax1+cos(πk/N)2,Λcos(2):=∑k=0N−1(ηmax1+cos(πk/N)2)2, ^(1)_cos:= _k=0^N-1 _ 1+ (π k/N)2,~~ ^(2)_cos:= _k=0^N-1 ( _ 1+ (π k/N)2 )^2, and similarly for the exponential schedule, Λexp(1):=∑k=0N−1ηmaxγk,Λexp(2):=∑k=0N−1ηmax2γ2k. ^(1)_exp:= _k=0^N-1 _ γ^k,~~ ^(2)_exp:= _k=0^N-1 _ ^2γ^2k. We compute these exactly. For cosine, using ∑k=0N−1cos(πk/N)=1 _k=0^N-1 (π k/N)=1 (as in Lemma D.15), Λcos(1)=ηmax⋅12(N+∑k=0N−1cos(πk/N))=ηmax⋅N+12. ^(1)_cos= _ · 12 (N+ _k=0^N-1 (π k/N) )= _ · N+12. For cosine squared sum, Lemma D.15 gives Λcos(2)=38ηmax2N+12ηmax2. ^(2)_cos= 38\, _ ^2\,N+ 12\, _ ^2. For exponential, Λexp(1)=ηmax∑k=0N−1γk=ηmax1−γN1−γ,Λexp(2)=ηmax2∑k=0N−1γ2k=ηmax21−γ2N1−γ2. ^(1)_exp= _ _k=0^N-1γ^k= _ 1-γ^N1-γ,~~ ^(2)_exp= _ ^2 _k=0^N-1γ^2k= _ ^2 1-γ^2N1-γ^2. Thus the two stated conditions are exactly Λcos(1)≤Λexp(1) ^(1)_cos≤ ^(1)_exp and Λcos(2)≤Λexp(2) ^(2)_cos≤ ^(2)_exp. Adding the identical warmup contributions preserves these inequalities for ΛT(1) ^(1)_T and ΛT(2) ^(2)_T over the full horizon. Therefore Lemma D.12 implies gT,cosSGD≤gT,expSGDg_T,cos^SGD≤ g_T,exp^SGD. ∎ Lemma D.17 (Inverse decay is asymptotically minimal). With the same TwuT_wu and N, consider warmup with inverse decay ηTwu+k=ηmax(1+k)−p _T_wu+k= _ (1+k)^-p with p≥12p≥ 12. Then ∑k=0N−1ηTwu+k2=Θ(ηmax2logN)for p=12,O(ηmax2)for p>12. _k=0^N-1 _T_wu+k^2= ( _ ^2 N) p= 12,~~O( _ ^2) p> 12. In particular, for sufficiently large N, the strict inequality below holds whenever the corresponding bound is nontrivial, e.g., whenever the effective margin aT,cos:=((ρ−ϵ)−BT,cosSGD)+a_T,cos:= ((ρ-ε)-B_T,cos^SGD )_+ is strictly positive under the conditions of Proposition D.10, so that the strictness clause in Lemma D.12 applies, gT,invSGD<gT,cosSGD.g_T,inv^SGD<g_T,cos^SGD. Proof. We first prove the squared-step accumulation. Write ∑k=0N−1ηTwu+k2=ηmax2∑k=0N−1(1+k)−2p=ηmax2∑j=1Nj−2p. _k=0^N-1 _T_wu+k^2= _ ^2 _k=0^N-1(1+k)^-2p= _ ^2 _j=1^Nj^-2p. If p=12p= 12, then 2p=12p=1 and the harmonic sum yields ∑j=1Nj−1=Θ(logN) _j=1^Nj^-1= ( N). If p>12p> 12, then 2p>12p>1 so ∑j=1∞j−2p<∞ _j=1^∞j^-2p<∞ and hence ∑j=1Nj−2p=O(1) _j=1^Nj^-2p=O(1). This proves the first display. Next, we compare the inverse schedule to cosine to obtain a strict bound on gTSGDg_T^SGD for large N. As in Lemma D.16, warmup is common, so it suffices to compare tail sums. For cosine, Lemma D.15 gives Λcos(2)=38ηmax2N+12ηmax2=Θ(ηmax2N). ^(2)_cos= 38\, _ ^2\,N+ 12\, _ ^2= ( _ ^2N). For inverse, the first part shows Λinv(2)=ηmax2∑j=1Nj−2p=Θ(ηmax2logN)(p=12),O(ηmax2)(p>12), ^(2)_inv= _ ^2 _j=1^Nj^-2p= ( _ ^2 N)\ \ (p= 12), O( _ ^2)\ \ (p> 12), so Λinv(2)<Λcos(2) ^(2)_inv< ^(2)_cos for all sufficiently large N. We also compare the first-moment tail sums: Λcos(1)=ηmaxN+12=Θ(ηmaxN), ^(1)_cos= _ N+12= ( _ N), whereas for inverse decay, Λinv(1)=ηmax∑j=1Nj−p=Θ(ηmaxN),p=12,O(ηmaxN1−p),12<p<1,O(ηmaxlogN),p=1,O(ηmax),p>1, ^(1)_inv= _ _j=1^Nj^-p= cases ( _ N),&p= 12,\\ O( _ N^1-p),& 12<p<1,\\ O( _ N),&p=1,\\ O( _ ),&p>1, cases which is o(ηmaxN)o( _ N) for every p≥12p≥ 12. Hence Λinv(1)<Λcos(1) ^(1)_inv< ^(1)_cos for sufficiently large N. Adding the identical warmup parts preserves strict inequalities for the full-horizon ΛT(1) ^(1)_T and ΛT(2) ^(2)_T. Applying Lemma D.12 (with strictness, once aT>0a_T>0) yields gT,invSGD<gT,cosSGDg_T,inv^SGD<g_T,cos^SGD for sufficiently large N. ∎ Proposition D.18 (Ordering of schedule-aware re-entry guarantees). Assume the hypotheses of Proposition D.10. Fix T and ηmax _ and consider the four schedules (constant, warmup+cosine, warmup+exponential, warmup+inverse) with a common warmup length TwuT_wu. Then the schedule-specific re-entry bounds satisfy gT,invSGD<gT,cosSGD≲gT,expSGD<gT,constSGD,g_T,inv^SGD<g_T,cos^SGD\; \;g_T,exp^SGD<g_T,const^SGD, where the middle relation holds under the conditions of Lemma D.16. Moreover, each strict inequality “<<” above is to be understood under the strictness regime of Lemma D.12: it holds whenever the effective margin aT,sch:=((ρ−ϵ)−BT,schSGD)+a_T,sch:= ((ρ-ε)-B_T,sch^SGD )_+ for the larger schedule in the corresponding comparison (as defined via Proposition D.10) is strictly positive. Consequently, Assumption 3.4 holds with gT=gT,schSGDg_T=g_T,sch^SGD for each schedule sch∈const,cos,exp,invsch∈\const,cos,exp,inv\, and the geometric-tail conclusions of Theorem 3.6 apply with the corresponding bounds. Proof. We compare the bounds gT,schSGDg_T,sch^SGD through the cumulative quantities ΛT(1) _T^(1) and ΛT(2) _T^(2) that enter the explicit expression in Proposition D.10. The required monotonicity is provided by Lemma D.12. Constant schedule is maximal. By Lemma D.14, among all schedules satisfying the common peak constraint 0≤ηt≤ηmax0≤ _t≤ _ , the constant schedule maximizes both ΛT(1) _T^(1) and ΛT(2) _T^(2). Therefore, Lemma D.12 yields gT,schSGD≤gT,constSGDfor any schedule sch.g_T,sch^SGD≤ g_T,const^SGD~~for any schedule sch. Moreover, whenever at least one of the two cumulative inequalities is strict and the strictness condition in Lemma D.12 holds for the larger schedule (in particular, aT,const>0a_T,const>0), we obtain the strict inequality gT,schSGD<gT,constSGDg_T,sch^SGD<g_T,const^SGD. Inverse decay is asymptotically minimal relative to cosine decay. By Lemma D.17, for sufficiently large N:=T−TwuN:=T-T_wu, the inverse-decay schedule yields strictly smaller cumulative quantities ΛT(1) _T^(1) and ΛT(2) _T^(2) than the cosine-decay schedule. Hence, applying Lemma D.12 and its strictness clause (e.g., under aT,cos>0a_T,cos>0), we conclude gT,invSGD<gT,cosSGD.g_T,inv^SGD<g_T,cos^SGD. Cosine versus exponential decay. Under the conditions of Lemma D.16, we have ΛT,cos(1)≤ΛT,exp(1) _T,cos^(1)≤ _T,exp^(1) and ΛT,cos(2)≤ΛT,exp(2) _T,cos^(2)≤ _T,exp^(2). Therefore Lemma D.12 gives gT,cosSGD≤gT,expSGD,g_T,cos^SGD≤ g_T,exp^SGD, which is recorded in the statement via the conventional notation “≲ ”. Transfer to Assumption 3.4 and geometric tails. For each schedule schsch, Proposition D.10 provides an explicit bound gT,schSGDg_T,sch^SGD such that for all k≥0k≥ 0, ℙ(τk+1≤T|ℱσk)≤gT,schSGDalmost surely on σk≤T.P\! ( _k+1≤ T\, |\,F_ _k )≤ g_T,sch^SGD surely on \ _k≤ T\. Hence Assumption 3.4 holds with gT=gT,schSGDg_T=g_T,sch^SGD, and the geometric-tail conclusions follow by invoking Theorem 3.6. ∎ D.4.2 Batch Size and Model Size Dependency Proposition D.19 (Larger minibatch yields a smaller SGD re-entry bound). Assume the hypotheses of Proposition D.10. Suppose the conditional noise proxy satisfies ξ2=ξ2()ξ^2=ξ^2( B) for minibatch size B, where ξ2()ξ^2( B) is nonincreasing in B. Let gTSGD()g_T^SGD( B) denote the bound in Eq. (7) computed with ξ2=ξ2()ξ^2=ξ^2( B). Then for any ~≥ B≥ B, gTSGD(~)≤gTSGD().g_T^SGD( B)≤ g_T^SGD( B). Consequently, if Assumption 3.4 holds with gT=gTSGD()g_T=g_T^SGD( B), Theorem 3.6 implies ℙ(KTeff(ρ)≥k)≤hT(gTSGD())k−1+δupd(k≥1).P\! (K_T^eff(ρ)≥ k )≤ h_T\, (g_T^SGD( B) )^k-1+ _upd~~(k≥ 1). Proof. In Eq. (7), both VTSGD=ξ2ΛT(2)V_T^SGD=ξ^2 _T^(2) and BTSGD=δ−1B¯TB_T^SGD=δ^-1 B_T are nondecreasing in ξ2ξ^2, hence aT=((ρ−ϵ)−BTSGD)+a_T=((ρ-ε)-B_T^SGD)_+ is nonincreasing in ξ2ξ^2. Therefore the exponential term in Eq. (7) is nondecreasing in ξ2ξ^2, so gTSGDg_T^SGD is nondecreasing in ξ2ξ^2. Since ξ2()ξ^2( B) is nonincreasing in B, the claim follows. ∎ Proposition D.20 (Batch–schedule exchange condition for preserving the re-entry bound). Assume the hypotheses of Proposition D.10. Consider two training protocols indexed by j∈1,2j∈\1,2\ with (possibly different) minibatch sizes j B_j, noise proxies ξj2 _j^2, horizons TjT_j, and schedules ηj,tt=0Tj−1\ _j,t\_t=0^T_j-1. For each protocol, define ΛTj(j):=∑t=0Tj−1ηj,t,(ΛTj(2))(j):=∑t=0Tj−1ηj,t2, _T_j^(j):= _t=0^T_j-1 _j,t,~~( _T_j^(2))^(j):= _t=0^T_j-1 _j,t^2, and define B¯Tj(j) B_T_j^(j) and VTj(j)V_T_j^(j) by B¯Tj(j):=ΛTj(j)(2Δℒmax+ξj2ΛTj(j)+Lsmξj2(ΛTj(2))(j)),VTj(j):=ξj2(ΛTj(2))(j). B_T_j^(j):= _T_j^(j) (2 L_ + _j^2\, _T_j^(j)+L_sm _j^2\,( _T_j^(2))^(j) ),~~V_T_j^(j):= _j^2\,( _T_j^(2))^(j). Let gTjSGD,(j)g_T_j^SGD,(j) denote the Proposition D.10 bound computed from (B¯Tj(j),VTj(j))( B_T_j^(j),V_T_j^(j)) (equivalently from Eq. (7)). If B¯T2(2)≤B¯T1(1)andVT2(2)≤VT1(1), B_T_2^(2)≤ B_T_1^(1)~~and~~V_T_2^(2)≤ V_T_1^(1), then gT2SGD,(2)≤gT1SGD,(1).g_T_2^SGD,(2)≤ g_T_1^SGD,(1). In particular, when 2≥1 B_2≥ B_1 reduces the noise proxy (e.g., ξ22≤ξ12 _2^2≤ _1^2), one may trade a larger squared-step accumulation (ΛT2(2))(2)( _T_2^(2))^(2) against the smaller ξ22 _2^2 as long as the two budget inequalities above hold. Proof. From Eq. (7), BTSGD=δ−1B¯TB_T^SGD=δ^-1 B_T and VTSGD=VTV_T^SGD=V_T. If B¯ B decreases then BTSGDB_T^SGD decreases and hence aT=((ρ−ϵ)−BTSGD)+a_T=((ρ-ε)-B_T^SGD)_+ increases. If V decreases as well, the exponent −aT22(V+2Δ3aT)- a_T^22(V+ 2 3a_T) becomes no larger, so the exponential term decreases. Therefore gTSGD=δupd+δ+exp(⋯)g_T^SGD= _upd+δ+ (·s) decreases, proving the claim. ∎ Proposition D.21 (Similarity width expansion yields smaller initial hit). Fix a finite horizon T, an outer threshold ρ>0ρ>0, and a base boundary radius ϵ0∈(0,ρ) _0∈(0,ρ) as in Definition 3.1. Consider a family of scalar coordinate processes (wt())t≥0∈ℕ\(w_t^( m))_t≥ 0\_ m∈N indexed by a width parameter m. Assume that for every m, Assumption 3.3 holds for (wt())(w_t^( m)) with update bound Δ=Δ()>0 = ( m)>0 and the same failure probability δupd _upd. Define the boundary radius and the initial-hit radius by ϵ():=maxϵ0,Δ(),bT():=ϵ()+TΔ().ε( m):= \ _0, ( m)\,~~b_T( m):=ε( m)+T ( m). Assume that for any ~≥ m≥ m, Δ(~) ( m) ≤Δ(), ≤ ( m), (12) ℙ[|w0(~)|≤x] P\! [|w_0^( m)|≤ x ] ≤ℙ[|w0()|≤x]for all x≥0. ≤P\! [|w_0^( m)|≤ x ] all x≥ 0. (13) Let h¯T():=ℙ[|w0()|≤bT()]+δupd. h_T( m):=P\! [|w_0^( m)|≤ b_T( m) ]+ _upd. Then hT():=ℙ[τ1()≤T]≤h¯T()h_T^( m):=P[ _1^( m)≤ T]≤ h_T( m) (by Proposition D.4 applied to (wt())(w_t^( m))), and moreover h¯T(~)≤h¯T() h_T( m)≤ h_T( m). Proof. The upper bound hT()≤h¯T()h_T^( m)≤ h_T( m) was already stated above, so it remains to prove the monotonicity claim. Fix ~≥ m≥ m. From Eq. (12) and ϵ()=maxϵ0,Δ()ε( m)= \ _0, ( m)\, we have ϵ(~)≤ϵ()ε( m)≤ε( m) and hence bT(~)=ϵ(~)+TΔ(~)≤ϵ()+TΔ()=bT().b_T( m)=ε( m)+T ( m)≤ε( m)+T ( m)=b_T( m). Apply Proposition D.4 to the process (wt(~))(w_t^( m)) (with its constants ϵ(~),Δ(~),δupdε( m), ( m), _upd): hT(~)=ℙ(τ1(~)≤T)≤ℙ(|w0(~)|≤bT(~))+δupd=h¯T(~).h_T^( m)=P( _1^( m)≤ T)≤P\! (|w_0^( m)|≤ b_T( m) )+ _upd= h_T( m). To compare h¯T(~) h_T( m) and h¯T() h_T( m), use the monotonicity in the threshold and the stochastic ordering Eq. (13): ℙ[|w0(~)|≤bT(~)]≤ℙ[|w0(~)|≤bT()]≤ℙ[|w0()|≤bT()].P\! [|w_0^( m)|≤ b_T( m) ]≤P\! [|w_0^( m)|≤ b_T( m) ]≤P\! [|w_0^( m)|≤ b_T( m) ]. Adding δupd _upd to both sides yields h¯T(~)≤h¯T() h_T( m)≤ h_T( m). ∎ D.4.3 Scale-Invariance Enhancement of Sign Lock-In In architectures with BatchNorm and ReLU, the loss typically exhibits an (approximate) positive scale invariance along certain parameter blocks: rescaling a block by a positive factor can be compensated elsewhere without changing the network function, a consequence of BN-induced scale invariance together with the positive homogeneity of ReLU. In such settings, it is natural to analyze excursions in a fixed gauge, i.e., in a normalized coordinate that factors out the redundant scale. Concretely, let ut∈ℝmu_t∈R^m be a parameter block containing the tracked coordinate wt=ei⊤vtw_t=e_i v_t as one of its entries, and define its block scale rt:=‖ut‖2r_t:=\|u_t\|_2 as well as the normalized coordinate w~t:=wt/rt w_t:=w_t/r_t. Intuitively, if the block scale is bounded away from zero throughout an excursion, then the normalized coordinate has smaller effective one-step increments. This reduces the overshoot-driven part of the Freedman exponent in Proposition D.10 by replacing the raw increment bound Δ with a smaller normalized bound Δ~ . We formalize the required scale control as an additional setup event, and then show how the re-entry bound tightens in this scale-invariant case. Additional setup (scale control). Fix a stopping time θ≤Tθ≤ T (in our application, θ=σkθ= _k). We introduce a block-update good event ℰblk(θ)E_blk(θ) on which the block utu_t evolves in a controlled manner: there exist constants Δblk>0 _blk>0 and δblk∈[0,1) _blk∈[0,1) such that ‖ut+1−ut‖2≤Δblkfor all t∈θ,…,T−1,\|u_t+1-u_t\|_2≤ _blk~~for all t∈\θ,…,T-1\, and ℙ(ℰblk(θ)c∣ℱθ)≤δblkP(E_blk(θ)^c F_θ)≤ _blk. This setup can be enforced, for instance, by deterministic block-wise clipping of the update (or any mechanism yielding a uniform bound on ‖ut+1−ut‖2\|u_t+1-u_t\|_2). The next lemma shows that under this additional setup the block scale stays bounded away from zero over a finite horizon, and that the normalized coordinate w~t=wt/rt w_t=w_t/r_t admits a smaller increment bound. Lemma D.22 (Block-scale lower bound and normalized increment bound). Let ut∈ℝmu_t∈R^m be a parameter block that contains the tracked coordinate wt=ei⊤vtw_t=e_i v_t as one of its entries, and define the block scale rt:=‖ut‖2.r_t:=\|u_t\|_2. Fix a stopping time θ≤Tθ≤ T and assume that there exist constants Δblk>0 _blk>0 and δblk∈[0,1) _blk∈[0,1) such that on the good event ℰblk(θ)E_blk(θ) we have the uniform block update bound ‖ut+1−ut‖2≤Δblkfor all t∈θ,…,T−1,\|u_t+1-u_t\|_2≤ _blk~~for all t∈\θ,…,T-1\, and ℙ(ℰblk(θ)c∣ℱθ)≤δblkP(E_blk(θ)^c F_θ)≤ _blk. Then on ℰblk(θ)E_blk(θ), for all t∈θ,…,Tt∈\θ,…,T\, rθ−(t−θ)Δblk≤rt≤rθ+(t−θ)Δblk.r_θ-(t-θ) _blk\ ≤\ r_t\ ≤\ r_θ+(t-θ) _blk. In particular, if rθ>TΔblkr_θ>T _blk then rt≥rmin:=rθ−TΔblk>0r_t≥ r_ :=r_θ-T _blk>0 for all t≤Tt≤ T. Moreover, define the normalized coordinate w~t:=wt/rt w_t:=w_t/r_t. On ℰblk(θ)∩rt≥rmin∀t≤TE_blk(θ)∩\r_t≥ r_ \ ∀ t≤ T\, we have |w~t+1−w~t|≤2Δblkrminfor all t∈θ,…,T−1.| w_t+1- w_t|≤ 2 _blkr_ ~~for all t∈\θ,…,T-1\. Proof. On ℰblk(θ)E_blk(θ), the triangle inequality yields |rt+1−rt|=|‖ut+1‖2−‖ut‖2|≤‖ut+1−ut‖2≤Δblk|r_t+1-r_t|=|\|u_t+1\|_2-\|u_t\|_2|≤\|u_t+1-u_t\|_2≤ _blk, so telescoping gives the two-sided bound on (rt)(r_t). For the normalized increment, write w~t+1−w~t=wt+1rt+1−wtrt=wt+1−wtrt+1+wt(1rt+1−1rt). w_t+1- w_t= w_t+1r_t+1- w_tr_t= w_t+1-w_tr_t+1+w_t\! ( 1r_t+1- 1r_t ). On ℰblk(θ)E_blk(θ) we have |wt+1−wt|≤‖ut+1−ut‖2≤Δblk|w_t+1-w_t|≤\|u_t+1-u_t\|_2≤ _blk and |rt+1−rt|≤‖ut+1−ut‖2≤Δblk|r_t+1-r_t|≤\|u_t+1-u_t\|_2≤ _blk. Moreover, since |wt|≤‖ut‖2=rt|w_t|≤\|u_t\|_2=r_t, we have |w~t|=|wt|/rt≤1| w_t|=|w_t|/r_t≤ 1. Therefore, on ℰblk(θ)∩rs≥rmin∀s≤TE_blk(θ)∩\r_s≥ r_ \ ∀ s≤ T\ (so that rt+1≥rminr_t+1≥ r_ ), |w~t+1−w~t| | w_t+1- w_t| ≤|wt+1−wt|rt+1+|wt||1rt+1−1rt| ≤ |w_t+1-w_t|r_t+1+|w_t| | 1r_t+1- 1r_t | =|wt+1−wt|rt+1+|wtrt||rt+1−rt|rt+1 = |w_t+1-w_t|r_t+1+ | w_tr_t | |r_t+1-r_t|r_t+1 ≤Δblkrmin+1⋅Δblkrmin=2Δblkrmin. ≤ _blkr_ +1· _blkr_ = 2 _blkr_ . This proves the normalized increment bound with Δ~:=2Δblk/rmin :=2 _blk/r_ . ∎ Proposition D.23 (Scale control yields a tighter increment term). Assume the setting of Proposition D.10. In addition, suppose there exists a parameter block utu_t containing the tracked coordinate wtw_t such that (motivated by BN-induced scale invariance together with ReLU positive homogeneity) its scale stays bounded away from zero on the excursion with high probability: for θ=σkθ= _k, Lemma D.22 holds with constants Δblk _blk, δblk _blk, and rmin:=rσk−TΔblk>0r_ :=r_ _k-T _blk>0. Define the normalized coordinate w~t:=wt/rt w_t:=w_t/r_t and the corresponding oriented process z~t:=skw~t z_t:=s_k w_t. On the event ℰblk(σk)∩rt≥rmin∀t≤TE_blk( _k)∩\r_t≥ r_ \ ∀ t≤ T\, Lemma D.22 implies the normalized increment bound |w~t+1−w~t|≤Δ~| w_t+1- w_t|≤ with Δ~:=2Δblkrmin. := 2 _blkr_ . Consequently, the proof of Proposition D.10 yields the same form of re-entry bound Eq. (8) but with the increment term Δ replaced by Δ~ , at an additional failure cost δblk _blk: ℙ[τk+1≤T∣ℱσk]≤gTIvon σk≤T,P[ _k+1≤ T F_ _k]≤ g_T^Iv~~on \ _k≤ T\, where gTIv:=(δupd+δblk)+δ+exp(−aT22(VTSGD+2Δ~3aT)),g_T^Iv:=( _upd+ _blk)+δ+ \! (- a_T^22 (V_T^SGD+ 2 3a_T ) ), and aTa_T and VTSGDV_T^SGD are exactly as defined in Eq. (7). Proof. Repeat the proof of Proposition D.10, but intersect the “good” event ℰΔ(σk)E_ ( _k) with ℰblk(σk)∩rt≥rmin∀t≤TE_blk( _k)∩\r_t≥ r_ \ ∀ t≤ T\. By Lemma D.22, on this intersection we have the normalized increment bound |z~t+1−z~t|≤Δ~| z_t+1- z_t|≤ (and hence |Xt|≤2Δ~|X_t|≤ 2 in the Freedman step), while the drift and variance parts are handled exactly as in Proposition D.10. The additional failure probability contributes δblk _blk by a union bound. ∎ D.4.4 Sign dynamics This section records auxiliary consequences for sign dynamics implied by the sign lock-in theory. In particular, we characterize the temporal distribution of effective sign changes and show that sign flips are inherently front-loaded, occurring predominantly in the early phase of training. The resulting stabilization of signs explains why later optimization mainly affects magnitudes rather than signs. Lemma D.24 (Initial-hit bound without assuming an outer start). Assume the hypotheses of Proposition D.10. Let hT:=ℙ(τ1≤T)h_T:=P( _1≤ T) be the initial-hit factor, where σ0,τ1 _0, _1 are defined in Definition 3.2. For each learning-rate schedule sch∈const,cos,exp,invsch∈\const,cos,exp,inv\, let gT,schSGDg_T,sch^SGD denote the Proposition D.10 bound computed with that schedule. Then, without assuming σ0=0 _0=0, hT≤gT,schSGD.h_T\;≤\;g_T,sch^SGD. (14) Consequently, the schedule ordering in Proposition D.18 carries over to the initial-hit upper bounds. Proof. By Definition 3.2, τ1:=inft>σ0:|wt|≤ϵ _1:= \t> _0:\ |w_t|≤ε\, hence τ1≤T⊆σ0≤T\ _1≤ T\ \ _0≤ T\. Therefore, by the tower property, hT=ℙ[τ1≤T]=[ℙ[τ1≤T∣ℱσ0] 1σ0≤T].h_T=P[ _1≤ T]=E\! [P[ _1≤ T F_ _0]\,1\ _0≤ T\ ]. Applying Proposition D.10 with k=0k=0 yields ℙ[τ1≤T∣ℱσ0]≤gT,schSGDP[ _1≤ T F_ _0]≤ g_T,sch^SGD on σ0≤T\ _0≤ T\. Thus, hT≤[gT,schSGD 1σ0≤T]≤gT,schSGD,h_T≤E\! [g_T,sch^SGD\,1\ _0≤ T\ ]≤ g_T,sch^SGD, which proves Eq. (14). The final statement follows by combining with Proposition D.18. ∎ Lemma D.25 (Early initiation of sign flips). Fix a finite horizon T and consider the stopping times τk _k and σk _k defined in Section 3. Under Theorem 3.6’s hypotheses, any effective outer-to-outer sign flip up to time T must be preceded by an early boundary interaction: specifically, for any k≥1k≥ 1, σk≤T,sign(wσk)≠sign(wσk−1)∩ℰΔ⊆τ1≤τk≤T.\ _k≤ T,\ sign(w_ _k) (w_ _k-1)\∩E_ \ _1≤ _k≤ T\. Moreover, the probability that a new effective sign-flip sequence is initiated late in training is exponentially suppressed: for all k≥1k≥ 1, ℙ[τk≤T]≤hTgTk−1,P[ _k≤ T]≤ h_T\,g_T^\,k-1, where hT:=ℙ[τ1≤T]h_T:=P[ _1≤ T] is the initial-hit factor and gT∈(0,1)g_T∈(0,1) is the Re-entry bound from Assumption 3.4. Proof. We separate the claim into two parts. A sign change at σk _k implies an early boundary interaction. Fix k≥1k≥ 1 and consider the event σk≤T,sign(wσk)≠sign(wσk−1)∩ℰΔ.\ _k≤ T,\ sign(w_ _k) (w_ _k-1)\∩E_ . By Definition 3.2, an effective outer-to-outer sign flip is counted only when the trajectory (i) starts from an outer state at time σk−1 _k-1, (i) reaches the boundary band at time τk _k, and then (i) exits to the opposite outer side at time σk _k. Hence, on the above event, the boundary-hit time τk _k must exist and satisfy τk≤σk≤T. _k≤ _k≤ T. Also, since (τj)j( _j)_j is nondecreasing by construction of successive excursions, the first boundary-hit time satisfies τ1≤τk _1≤ _k. Combining these relations gives τ1≤τk≤T, _1≤ _k≤ T, which proves σk≤T,sign(wσk)≠sign(wσk−1)∩ℰΔ⊆τ1≤τk≤T.\ _k≤ T,\ sign(w_ _k) (w_ _k-1)\∩E_ \ _1≤ _k≤ T\. Geometric suppression of late initiations. Theorem 3.6 states that under the same hypotheses, for every k≥1k≥ 1, ℙ[τk≤T]≤hTgTk−1,P[ _k≤ T]≤ h_Tg_T^k-1, where hT=ℙ(τ1≤T)h_T=P( _1≤ T) and gT∈(0,1)g_T∈(0,1) is the re-entry factor. This is exactly the second displayed inequality in the lemma. Therefore both statements hold. ∎ Remark D.26 (Front-loaded structure of sign changes). Lemma D.25 shows that sign flips are front-loaded in the training trajectory. Importantly, this statement does not rely on specifying a concrete “early-time” window or on particular learning-rate schedules. Rather, it follows from the stopping-time structure and the geometric decay of boundary-hit probabilities: a parameter that flips its sign at any point during training must have already interacted with the boundary neighborhood at an earlier stage. Conversely, parameters that do not approach the boundary early are unlikely to exhibit sign flips later, regardless of the remaining training duration. D.4.5 Numerical validation of Practical Insights We follow the CharLM setup in Appendix C.4, and only change the learning-rate schedule. Data: Tiny Shakespeare with a contiguous 90/10 train/validation split; next-character prediction on random contiguous blocks with sequence length L=64L=64 and batch size B=64B=64. Model: TinyCharLM (causal Transformer) with dmodel=128d_model=128, nlayers=2n_layers=2, nheads=4n_heads=4, dff=256d_f=256, context length 256256, and no dropout. Optimization: AdamW with initialization scale σinit=0.02 _init=0.02; all runs use three seeds (0/1/2) and the same training horizon T and peak step size ηmax _ . Schedules: constant learning rate ηt=ηmax _t= _ ; and warmup of length TwuT_wu followed by (i) cosine decay, (i) exponential decay with factor γ, and (i) inverse decay with exponent p (as defined in this section). We compute K:=KTeff(ρ)K:=K^eff_T(ρ) from tracked weights, fit the zero-inflated geometric model to obtain h h and g g, and report mean± across seeds. We next validate the batchsize dependency. We fix the optimization setup above and only change the training minibatch size B∈1,2,4,8,16,32,64,128,256,512,1024B∈\1,2,4,8,16,32,64,128,256,512,1024\, keeping the same training horizon T and peak step size ηmax _ . Figure 14 shows that the estimated re-entry parameter g g decreases monotonically as B increases (mean± across seeds), supporting the batchsize dependency in Proposition D.19. Finally, we empirically validate the width dependence predicted by Corollary D.21. Using the CharLM setup described in Appendix C.4, we sweep the model width dmodel∈32,64,128,256,512d_model∈\32,64,128,256,512\ while keeping all other training hyperparameters fixed, including the initialization scale σinit _init. For each configuration, we track the effective flip count Keff,T(ρ)K_eff,T(ρ) and fit the zero-inflated geometric model (Appendix C.4.2) to estimate the lock-in parameters (h^,g^)( h, g). Figure 15 shows that both the initial-hit factor h h and the re-entry ratio g g decrease monotonically as the width increases. The reduction in h h indicates that fewer weights ever reach the near-zero boundary, consistent with Proposition D.21. Figure 12: Estimated g g (mean± ) across schedules. Figure 13: Estimated h h (mean± ) across schedules. Figure 14: Estimated g g (mean± across seeds) vs. train batch size B. Figure 15: Estimated initial-hit factor h h vs. model width dmodeld_model. D.5 Theory for Sign Lock-In Enhancement Having established the deterministic and stochastic foundations of sign lock-in, we now turn to theory-driven mechanisms that actively enhance this effect. The results in Appendix D.2 show that the initial-hit factor hTh_T is largely dominated by initialization under bounded updates, while Appendix D.3 demonstrates how stochastic optimization dynamics control the re-entry ratio gTg_T. This Appendix builds on these insights to formalize practical interventions that reduce boundary visits and re-entry, thereby strengthening sign lock-in beyond its naturally emerging regime. Proposition D.27 (Gap initialization suppresses the initial-hit factor). Assume the setting of Proposition D.4. Under the gap initialization described in Section 4, let Z∼(0,σinit2)Z N(0, _init^2) and set |w0|=|Z||w_0|=|Z| conditioned on |Z|≥ainit|Z|≥ a_init. Then hT≤ℙ[|Z|≤bT∣|Z|≥ainit]+δupd=ℙ[ainit≤|Z|≤bT]ℙ[|Z|≥ainit]+δupd.h_T≤P [|Z|≤ b_T |Z|≥ a_init ]+ _upd= P[a_init≤|Z|≤ b_T]P[|Z|≥ a_init]+ _upd. Here bT:=ϵ+TΔb_T:=ε+T . In particular, if bT<ainitb_T<a_init, then hT≤δupdh_T≤ _upd. Proof. By Proposition D.4, hT≤ℙ[|w0|≤bT]+δupdh_T≤P[|w_0|≤ b_T]+ _upd with bT=ϵ+TΔb_T=ε+T . Under gap initialization, |w0||w_0| is distributed as |Z||Z| conditioned on |Z|≥ainit|Z|≥ a_init. Therefore, hT=ℙ[τ1≤T]≤ℙ[|w0|≤bT]+δupd=ℙ[|Z|≤bT||Z|≥ainit]+δupd,h_T=P[ _1≤ T]≤P[|w_0|≤ b_T]+ _upd=P [|Z|≤ b_T\, |\,|Z|≥ a_init ]+ _upd, which proves Proposition D.27. If bT<ainitb_T<a_init, the conditional probability term vanishes, and hence hT≤δupdh_T≤ _upd. ∎ Proposition D.28 (Outer-drift implies an explicit Re-entry bound). Assume Assumption 3.3 and suppose additionally that there exists a constant μ>0μ>0 such that [|wt+1|−|wt||ℱt]≥μalmost surely on |wt|>ϵ.E [\,|w_t+1|-|w_t|\, |\,F_t ]≥μ~~almost surely on \|w_t|>ε\. (15) Then for every k≥0k≥ 0, on the event σk≤T\ _k≤ T\, ℙ[τk+1≤T|ℱσk]≤δupd+exp(−2μ(ρ−ϵ)Δ2).P [ _k+1≤ T\, |\,F_ _k ]≤ _upd+ \! (- 2μ(ρ-ε) ^2 ). (16) Consequently, Assumption 3.4 holds with the (time-uniform) choice gTOD:=δupd+exp(−2μ(ρ−ϵ)Δ2),g_T^OD:= _upd+ \! (- 2μ(ρ-ε) ^2 ), whenever gTOD<1g_T^OD<1. Interpretation for the log-barrier. In the presence of the log-barrier term in Section 4, a sufficient condition for Eq. (15) is that the regularizer-induced outward push dominates any (inward) bias from the task update near the boundary. One convenient way to express this is via a lower bound μ≈γλ/(ρ+ϵlb)−βμ≈γλ/(ρ+ _lb)-β (notation as in the main text), whenever the barrier is active throughout the band (ϵ,ρ](ε,ρ]. Proof. Fix k≥0k≥ 0 and work on σk≤T\ _k≤ T\. If σk=T _k=T then τk+1>T _k+1>T by definition, so Eq. (16) is trivial. Hence assume σk≤T−1 _k≤ T-1. Define the first update-violation time η:=inft≥σk:|wt+1−wt|>Δ,η:= \t≥ _k:\ |w_t+1-w_t|> \, with inf∅=∞ =∞. By Assumption 3.3 applied at θ=σkθ= _k, ℙ(η≤T−1∣ℱσk)≤δupdP(η≤ T-1 F_ _k)≤ _upd. Define the shifted filtration t:=ℱσk+tG_t:=F_ _k+t and the shifted process rt′:=|wσk+t|r _t:=|w_ _k+t| for t≥0t≥ 0. Let τ′:=inft≥1:rt′≤ϵ,η′:=inft≥0:σk+t≥η.τ := \t≥ 1:\ r _t≤ε\,~~η := \t≥ 0:\ _k+t≥η\. Then τk+1=σk+τ′ _k+1= _k+τ and η>T−1\η>T-1\ is equivalent to η′>T−σk−1\η >T- _k-1\. On the event t<η′\t<η \ we have |wσk+t+1−wσk+t|≤Δ|w_ _k+t+1-w_ _k+t|≤ and since x↦|x|x |x| is 11-Lipschitz, |rt+1′−rt′|≤|wσk+t+1−wσk+t|≤Δ.|r _t+1-r _t|≤|w_ _k+t+1-w_ _k+t|≤ . (17) Moreover, by Eq. (15), for all t<τ′t<τ we have [rt+1′−rt′∣t]≥μE[r _t+1-r _t G_t]≥μ. Let κ:=2μ/Δ2κ:=2μ/ ^2 and define Mt:=exp(−κrt∧τ′∧η′).M_t:= (-κ\,r _t τ η .). We claim that (Mt)t≥0(M_t)_t≥ 0 is a supermartingale with respect to (t)(G_t). Indeed, for t<τ′∧η′t<τ η , set Xt:=−(rt+1′−rt′)X_t:=-(r _t+1-r _t). Then Xt∈[−Δ,Δ]X_t∈[- , ] by Eq. (17) and [Xt∣t]≤−μE[X_t G_t]≤-μ. Hoeffding’s lemma gives [eκXt∣t]≤exp(κ[Xt∣t]+κ2Δ22)≤exp(−κμ+κ2Δ22)=1,E\! [e^κ X_t G_t ]≤ \! (κ\,E[X_t G_t]+ κ^2 ^22 )≤ \! (-κμ+ κ^2 ^22 )=1, where the last equality uses κ=2μ/Δ2κ=2μ/ ^2. Therefore [Mt+1∣t]≤MtE[M_t+1 G_t]≤ M_t for t<τ′∧η′t<τ η , and for t≥τ′∧η′t≥τ η we have Mt+1=MtM_t+1=M_t, so (Mt)(M_t) is a supermartingale. Fix an integer n≥1n≥ 1. By optional stopping applied to the bounded stopping time (τ′∧η′)∧n(τ η ) n, [M(τ′∧η′)∧n∣0]≤M0=exp(−κr0′).E[M_(τ η ) n G_0]≤ M_0= (-κ r _0). On the event τ′≤n,η′>n\τ ≤ n,\ η >n\ we have rτ′≤ϵr _τ ≤ε, hence M(τ′∧η′)∧n=Mτ′=exp(−κrτ′)≥exp(−κϵ)M_(τ η ) n=M_τ = (-κ r _τ )≥ (-κε). Thus, exp(−κϵ)ℙ(τ′≤n,η′>n∣0)≤[M(τ′∧η′)∧n∣0]≤exp(−κr0′), (-κε)\,P(τ ≤ n,\ η >n G_0)≤E[M_(τ η ) n G_0]≤ (-κ r _0), which gives ℙ(τ′≤n,η′>n∣ℱσk)≤exp(−κ(r0′−ϵ))≤exp(−κ(ρ−ϵ))=exp(−2μ(ρ−ϵ)Δ2).P(τ ≤ n,\ η >n F_ _k)≤ (-κ(r _0-ε) .)≤ (-κ(ρ-ε) .)= \! (- 2μ(ρ-ε) ^2 ). Finally, taking n=T−σkn=T- _k and using ℙ(τk+1≤T∣ℱσk)≤ℙ(η≤T−1∣ℱσk)+ℙ(τ′≤T−σk,η′>T−σk−1∣ℱσk)P( _k+1≤ T F_ _k)≤P(η≤ T-1 F_ _k)+P(τ ≤ T- _k,\ η >T- _k-1 F_ _k) yields the Proposition D.28. ∎ Proposition D.29 (Flip-histogram bound under Gap initialization and outer-drift). Assume the hypotheses of Proposition D.27 (Gap initialization) and Proposition D.28, and keep the definitions of KTeff(ρ)K^eff_T(ρ), τk _k, σk _k from Section 3. Then, for all k≥1k≥ 1, ℙ[KTeff(ρ)≥k]≤hTgap(gTOD)k−1+δupd,gTOD=δupd+exp(−2μ(ρ−ϵ)Δ2).P [K^eff_T(ρ)≥ k ]≤ h_T^gap\,(g_T^OD)^k-1+ _upd,~~g_T^OD= _upd+ \! (- 2μ(ρ-ε) ^2 ). (18) In particular, since KTeff(ρ)≤TK^eff_T(ρ)≤ T, the expected effective flip count admits the bound [KTeff(ρ)]=∑k=1Tℙ[KTeff(ρ)≥k]≤hTgap1−gTOD+Tδupd,E [K^eff_T(ρ) ]= _k=1^TP [K^eff_T(ρ)≥ k ]≤ h_T^gap1-g_T^OD+T\, _upd, whenever gTOD<1g_T^OD<1. Proof. Corollary D.28 verifies Assumption 3.4 with gT=gTODg_T=g_T^OD. Applying Theorem 3.6 then gives ℙ[τk≤T]≤hT(gTOD)k−1P[ _k≤ T]≤ h_T(g_T^OD)^k-1. Replacing hTh_T by the Gap-initialization probability hTgaph_T^gap (Corollary D.27) and using Eq. (4) in Theorem 3.6 yields Eq. (18). Finally, since KTeff(ρ)≤TK^eff_T(ρ)≤ T, we can sum the tail bound up to T: [KTeff(ρ)]=∑k=1Tℙ[KTeff(ρ)≥k]≤∑k=1T(hTgap(gTOD)k−1+δupd)≤hTgap1−gTOD+Tδupd.E[K^eff_T(ρ)]= _k=1^TP[K^eff_T(ρ)≥ k]≤ _k=1^T (h_T^gap(g_T^OD)^k-1+ _upd )≤ h_T^gap1-g_T^OD+T\, _upd. ∎ D.6 Sign Floating Mode Although sign lock-in is generally expected to occur, one can artificially establish conditions to break this lock-in. Here, we demonstrate an example of this approach. This sign floating mode is induced by an auxiliary regularizer with an extraordinarily large weight, and should be viewed as an artificial regime rather than the typical behavior of standard deep-network training. D.6.1 Theory for Sign Floating Mode We keep the same stopping-time framework as in Section 3: outer-entry times (σk)k≥0( _k)_k≥ 0, boundary-hit times (τk)k≥1( _k)_k≥ 1, and the effective outer-to-outer flip count KTeff(ρ)K^eff_T(ρ). In sign floating mode, boundary re-entries are frequent and, crucially, the sign upon exiting back to the outer region is well-mixed (approximately random). This yields a binomial law for the histogram of effective flips. Assumption D.30 (Floating regime: excursion-count concentration up to T). There exist an integer n¯T≥0 n_T≥ 0 and parameters ΔT≥0 _T≥ 0 and δT∈[0,1) _T∈[0,1) such that ℙ(|NT(ρ)−n¯T|≤ΔT)≥ 1−δT,P (|N_T(ρ)- n_T|≤ _T )\ ≥\ 1- _T, where the completed outer-entry count is NT(ρ):=maxk≥0:σk≤TN_T(ρ):= \k≥ 0: _k≤ T\. In the induced floating regime, n¯T n_T is typically moderate to large. Assumption D.31 (Sign mixing at outer re-entry). Define the outer-entry sign sequence Zk:=sign(wσk)∈±1,k≥0,Z_k:=sign(w_ _k)∈\± 1\,~~k≥ 0, (on σk≤T\ _k≤ T\ this is well-defined since |wσk|≥ρ>0|w_ _k|≥ρ>0). Assume that, for every k≥1k≥ 1, ℙ[Zk=+1∣ℱτk]=ℙ[Zk=−1∣ℱτk]=12,P[Z_k=+1 F_ _k]=P[Z_k=-1 F_ _k]= 12, and moreover, conditionally on the completed outer-entry count NT(ρ):=maxk≥0:σk≤T,N_T(ρ):= \k≥ 0: _k≤ T\, the random variables (Z1,Z2,…,ZNT(ρ))(Z_1,Z_2,…,Z_N_T(ρ)) are independent. (No assumption is imposed on Z0Z_0.) Remark D.32. Assumption D.30 posits that the completed outer-entry count NT(ρ)N_T(ρ) is effectively fixed (or tightly concentrated) up to time T, so boundary excursions are not governed by the “rare re-entry” regime behind Theorem 3.6. However, having many excursions alone does not determine the histogram shape of KTeff(ρ)K_T^eff(ρ). The binomial law in Theorem D.33 is driven by Assumption D.31, which enforces (approximately) symmetric sign re-randomization upon each outer re-entry; Assumption D.30 controls how the resulting conditional binomial laws mix through NT(ρ)N_T(ρ) (see the remark below). Theorem D.33 (Binomial histogram of effective flips in sign floating mode). Assume Assumption D.31. Conditionally on NT(ρ)=nN_T(ρ)=n, the effective flip count satisfies KTeff(ρ)|NT(ρ)=n∼δ0,n=0,Binomial(n,12),n≥1.K^eff_T(ρ)\ |\ \N_T(ρ)=n\\ \ cases _0,&n=0,\\ Binomial\! (n, 12 ),&n≥ 1. cases Proof. Fix n≥1n≥ 1 and condition on NT(ρ)=n\N_T(ρ)=n\. On this event, KTeff(ρ)=∑k=1nZk≠Zk−1.K^eff_T(ρ)= _k=1^n1\Z_k≠ Z_k-1\. Define flip indicators Ik:=Zk≠Zk−1∈0,1I_k:=1\Z_k≠ Z_k-1\∈\0,1\ for k=1,…,nk=1,…,n. Let Bk:=(1+Zk)/2∈0,1B_k:=(1+Z_k)/2∈\0,1\. Under Assumption D.31, (B1,…,Bn)(B_1,…,B_n) are i.i.d. Bernoulli(1/2)Bernoulli(1/2), and we impose no constraint on B0B_0. Condition on B0=b0∈0,1B_0=b_0∈\0,1\. Then (B1,…,Bn)(B_1,…,B_n) is uniform on 0,1n\0,1\^n. Moreover, Ik=Zk≠Zk−1=Bk⊕Bk−1,I_k=1\Z_k≠ Z_k-1\=B_k B_k-1, where ⊕ denotes XOR on 0,1\0,1\. Consider the mapping Φb0:(B1,…,Bn)↦(I1,…,In)with fixed B0=b0. _b_0:\ (B_1,…,B_n)\ \ (I_1,…,I_n)~~with fixed B_0=b_0. This map is a bijection: given (I1,…,In)(I_1,…,I_n), we recover recursively Bk=b0⊕I1⊕⋯⊕Ik.B_k=b_0 I_1 ·s I_k. Hence, conditional on B0=b0B_0=b_0, the vector (I1,…,In)(I_1,…,I_n) is uniform on 0,1n\0,1\^n, so I1,…,InI_1,…,I_n are i.i.d. Bernoulli(1/2)Bernoulli(1/2). Since the conditional law does not depend on b0b_0, the same holds given NT(ρ)=nN_T(ρ)=n. Therefore, KTeff(ρ)=∑k=1nIk∼Binomial(n,1/2)K^eff_T(ρ)= _k=1^nI_k (n,1/2). ∎ Remark D.34 (Unconditional law is a mixture). Unconditionally, KTeff(ρ)K^eff_T(ρ) is a mixture over the random excursion count: ℙ[KTeff(ρ)=k]=∑n≥kℙ[NT(ρ)=n](nk)(12)n.P\! [K^eff_T(ρ)=k ]= _n≥ kP[N_T(ρ)=n] nk ( 12 )^n. Thus, the empirical flip histogram is binomial-shaped once the layer-wise excursion count n is effectively fixed (or tightly concentrated). D.6.2 Induction of Sign Floating Inward Drift. To induce sign floating, we assume the opposite radial drift to Proposition D.28: the dynamics are biased toward the sign boundary on the band (ϵ,ρ](ε,ρ]. Concretely, we postulate the existence of μID>0 _ID>0 such that [|wt+1|−|wt||ℱt]≤−μIDalmost surely on ϵ<|wt|≤ρ.E [\,|w_t+1|-|w_t|\, |\,F_t ]≤- _ID~~almost surely on \ε<|w_t|≤ρ\. (19) One concrete enhancement method: local l1l_1 attraction. A simple way to realize Eq. (19) is to add a local l1l_1 penalty that is active only near the boundary: RID(W(l);ρf):=1mn∑i,jmin|Wij(l)|,ρf,l∈ℳfloat,R_ID(W^(l); _f):= 1mn _i,j \! \|W^(l)_ij|,\ _f \,~~l∈M_float, and optimize ℒfloat(θ)=ℒtask(θ)+λfloat∑l∈ℳfloatRID(W(l);ρf),λfloat>0.L_float(θ)=L_task(θ)+ _float _l∈M_floatR_ID(W^(l); _f),~~ _float>0. For a single coordinate update (e.g., SGD with step size γ), the added term contributes approximately −γλfloatsign(wt)-γ _floatsign(w_t) when 0<|wt|<ρf0<|w_t|< _f, which decreases |wt||w_t| in expectation and thus promotes repeated boundary visits, enabling sign floating. D.6.3 Experimental Validation (a) Histogram of effective flip counts Keff,T(ρ)K_eff,T(ρ) in a representative floating-mode run (λfloat=104 _float=10^4). (b) Sweep over λfloat _float: validation PPL and flip ratio at training end (mean± over 3 seeds). Figure 16: Empirical induction of sign floating mode via local l1l_1 attraction near the boundary (Appendix D.6). Setup. We empirically induce the sign floating mode by adding an inward-drift regularizer that attracts weights toward the sign boundary. Concretely, we train the same Tiny Shakespeare character-level Transformer (CharLM) setting as in Appendix C.4 (sequence length 64, batch size 64; AdamW optimizer used), but we optimize the following objective: ℒfloat(θ)=ℒtask(θ)+λfloat∑l∈ℳfloatRID(W(l);ρf),L_float(θ)=L_task(θ)+ _float _l∈M_floatR_ID\! (W^(l); _f ), where ℳfloatM_float denotes the set of matrix-shaped parameters (2D tensors), and the local l1l_1 attraction is RID(W;ρf):=1mn∑i,jmin|Wij|,ρf.R_ID\! (W; _f ):= 1mn _i,j \|W_ij|, _f \. In all runs, we set ρf=10−2 _f=10^-2 and train for T=2000T=2000 steps. We sweep λfloat∈10−2,10−1,1.0,10,102,103,104,105,106, _float∈\10^-2,10^-1,1.0,10,10^2,10^3,10^4,10^5,10^6\, and report mean± over 3 random seeds. Result. Figure 16 summarizes the behavior. As λfloat _float increases, the step-wise flip ratio increases sharply, indicating frequent sign changes consistent with sign floating. At the same time, the validation PPL degrades moderately, reflecting the cost of forcing weights to repeatedly approach the sign boundary. Moreover, the histogram of Keff,T(ρ)K_eff,T(ρ) becomes broad, rather than geometrically decaying as in the lock-in regime, qualitatively matching the binomial-shaped flip histogram predicted in Theorem D.33 when outer re-entries are frequent and the re-entry sign is well-mixed. We note that inducing sign floating required an extraordinarily large λfloat _float, such as 10410^4, suggesting that such strong attraction toward the sign boundary is hard to realize under standard training protocols. Appendix E Additional Experiments This appendix provides additional experiments and extended empirical results that complement the main text. Specifically: • Appendix E.1 summarizes additional empirical results on scale, ImageNet compression, AdamW update magnitudes, and CNN sign lock-in. • Appendix E.2 reports additional randomness tests and supplementary analyses. • Appendix E.3 provides additional vision-task experiments and sign lock-in validations. • Appendix E.4 documents the (approximately) zero-cost sign template method and the full bit-budget accounting for sub-bit compression. E.1 Additional Empirical Results This appendix collects additional empirical results on scale, expressivity, optimizer assumptions, and the practical role of sign-template compression. E.1.1 Template Expressivity Across Model Scales To test whether a fixed rank-22 sign template limits expressivity at larger scales, we trained Transformer language models from 0.30.3M to 512512M parameters. Table 2 reports the best validation perplexity. The gap between vanilla training and gap+regularization decreases with model size and becomes negligible. Table 2: Best validation PPL on Transformer next-token prediction. #Params Vanilla Gap+Reg Degradation PPL 0.3M 5.22 5.61 +0.39 1.9M 4.55 4.69 +0.14 4.8M 4.67 4.64 -0.03 10.8M 4.67 4.63 -0.04 25.4M 4.70 4.69 -0.01 59.3M 4.82 4.78 -0.04 99.5M 4.87 4.78 -0.09 512M 4.93 4.94 +0.01 Table 3 reports best-vs-final validation perplexity. At larger scales, vanilla training overfits strongly, while the low-rank sign-template training with gap+regularization substantially reduces this degradation. Table 3: Overfitting: best vs. final PPL on Transformer next-token prediction. #Params Vanilla Best Vanilla Final Vanilla Overfit Gap+Reg Best Gap+Reg Final Gap+Reg Overfit 0.3M 5.22 5.22 +0.00 5.61 5.56 -0.05 1.9M 4.55 4.69 +0.14 4.69 4.70 +0.01 4.8M 4.67 5.96 +1.29 4.64 4.67 +0.03 10.8M 4.67 9.42 +4.75 4.63 4.94 +0.31 25.4M 4.70 12.58 +7.88 4.69 5.76 +1.07 59.3M 4.82 15.36 +10.54 4.78 5.79 +1.01 99.5M 4.87 18.09 +13.22 4.78 5.79 +1.01 512M 4.93 7.92 +2.99 4.94 5.00 +0.06 E.1.2 Billion-Scale Sign Lock-In Evidence We computed the effective sign-flip histogram on a 1B-parameter Transformer trained on 300B tokens. Table 4 shows that the empirical distribution is well captured by a zero-inflated geometric fit. Table 4: Pythia-1B trained on 300B tokens: effective flip-count histogram and geometric-tail fit. KeffK_eff Count Empirical prob. Fitted prob. Abs. error 0 168032 0.840160 0.840160 0.000000 1 25405 0.127025 0.126960 0.000065 2 5170 0.025850 0.026116 0.000266 3 1120 0.005600 0.005372 0.000228 4 230 0.001150 0.001105 0.000045 5 37 0.000185 0.000227 0.000042 6 5 0.000025 0.000047 0.000022 7 1 0.000005 0.000010 0.000005 E.1.3 ImageNet-Scale End-to-End Compression To validate the compression consequence on a larger benchmark, we applied the sign-template method to ImageNet training with ResNet and evaluated SVD compression with NormalFloat factor quantization. Table 5 reports Top-5 accuracy. The sign-template method is substantially stronger near and below the one-bit regime. Table 5: ImageNet Top-5 accuracy (%) of ResNet under compression via SVD with NormalFloat factor quantization. bpw Sign Template (ours) Vanilla SVD (baseline) Δ 0.50 5.3 0.5 +4.8 0.75 23.1 0.5 +22.6 1.00 42.0 0.5 +41.5 1.50 65.6 12.1 +53.5 2.00 74.7 65.1 +9.6 2.50 78.1 76.3 +1.8 3.00 79.8 80.9 -1.1 4.00 81.3 83.8 -2.5 E.1.4 AdamW Update Magnitudes All main training experiments used AdamW. To empirically check the high-probability bounded-update condition, we measured elementwise update magnitudes over training phases using outer threshold ρ=10−3ρ=10^-3. Table 6 shows that the maximum observed update remains far below 2ρ=2×10−32ρ=2× 10^-3 in all phases, and no step exceeds this threshold. Table 6: Phase-wise empirical AdamW update magnitudes. Phase Statistic Value Early median update 9.999×10−59.999× 10^-5 Early 90th-percentile update 1.191×10−41.191× 10^-4 Early 99th-percentile update 1.540×10−41.540× 10^-4 Early 99.9th-percentile update 1.865×10−41.865× 10^-4 Early maximum update 2.346×10−42.346× 10^-4 Early # steps with max update >2ρ>2ρ 0 Intermediate median update 2.364×10−52.364× 10^-5 Intermediate 90th-percentile update 6.021×10−56.021× 10^-5 Intermediate 99th-percentile update 1.122×10−41.122× 10^-4 Intermediate 99.9th-percentile update 1.718×10−41.718× 10^-4 Intermediate maximum update 2.354×10−42.354× 10^-4 Intermediate # steps with max update >2ρ>2ρ 0 Late median update 1.452×10−51.452× 10^-5 Late 90th-percentile update 3.893×10−53.893× 10^-5 Late 99th-percentile update 6.902×10−56.902× 10^-5 Late 99.9th-percentile update 9.574×10−59.574× 10^-5 Late maximum update 1.943×10−41.943× 10^-4 Late # steps with max update >2ρ>2ρ 0 E.1.5 Component-Wise Low-Rank Error To clarify why direct low-rank approximation of W is difficult, Table 7 reports component-wise rank approximation error at r/d=0.03125r/d=0.03125. Across models, Er(W)E_r(W) is much closer to Er(sign(W))E_r(sign(W)) than to Er(|W|)E_r(|W|), supporting the claim that sign structure is the dominant obstacle. Table 7: Low-rank approximation error by component at r/d=0.03125r/d=0.03125. Model Er(sign(W))E_r(sign(W)) Er(|W|)E_r(|W|) Er(W)E_r(W) MLP 0.927 0.573 0.895 ResNet 0.943 0.593 0.921 TinyLlama 0.934 0.571 0.908 E.1.6 CNN Effective Flip Histogram Finally, Table 8 reports the effective sign-flip histogram in a CNN experiment, showing that sign lock-in is also observed outside Transformer language models. Table 8: Effective sign-flip histogram in the CNN experiment. Flip count Observed count Fitted count 0 39161 39161 1 2059 1929 2 703 961 3 574 479 4 263 239 5 142 119 6 57 59 7 26 30 8 15 15 9 6 7 E.2 Additional Randomness Test E.2.1 Algorithmic Compressibility of Sign Bits via General-Purpose Compressors Table 9: Compressibility of sign bits across compression algorithms (ratio = compressed / raw). model brotli(q11) brotli(q5) bz2(l9) gzip(l9) lzma(p6) raw_deflate(l9) snappy zlib(l9) zstd(l10) zstd(l3) Rademacher baseline (2048x2048) 1.000 1.000 1.005 1.000 1.000 1.000 1.000 1.000 1.000 1.000 2D Ising baseline (beta=0.6) 0.001 0.002 0.004 0.025 0.004 0.025 0.067 0.025 0.002 0.003 Low-rank sign baseline (rank=2) 0.047 0.054 0.062 0.184 0.043 0.184 0.417 0.184 0.053 0.083 MLP-Mixer-B16 0.978 0.983 0.998 0.985 0.984 0.985 1.000 0.985 0.983 0.983 ResNet18 0.957 0.958 0.984 0.964 0.966 0.964 1.000 0.964 0.960 0.959 TinyLlama-1.1B-Chat 0.979 0.984 0.999 0.989 0.973 0.989 1.000 0.989 0.991 0.992 In the sub-bit regime, the sign pattern becomes a first-class storage target: storing signs naively costs one bit per weight and can dominate the effective bit budget when magnitudes are aggressively compressed. While our earlier analyses examine randomness of sign matrices through spectral statistics, here we add a complementary check based on algorithmic compressibility. If the learned sign pattern contains exploitable regularities, then off-the-shelf lossless compressors should achieve nontrivial savings when applied directly to the sign bitstream. Protocol. For each selected weight matrix, we extract the elementwise sign pattern sign(W)sign(W) using the same convention as the rest of the paper. We bit-pack the binary signs into a byte stream and measure the compression ratio defined as compressed size divided by the raw packed size (smaller means more compressible). To avoid conclusions driven by a single coding scheme, we evaluate a diverse set of widely used lossless compressors. Concretely, we always include raw DEFLATE, zlib, gzip, bzip2, and LZMA; when available, we also report Zstandard, Brotli, LZ4, and Snappy. For the DEFLATE family, we report raw DEFLATE to reduce constant header/metadata effects (DEFLATE/zlib/gzip follow the corresponding RFC specifications) (rfc1951; rfc1950; rfc1952). Brotli and Zstandard follow their RFC specifications (rfc7932; rfc8878). bzip2 is a widely used implementation based on the Burrows–Wheeler transform (burrowswheeler1994; seward1996bzip2). LZMA is based on public SDK documentation (pavlov2008lzmasdk). LZ4 and Snappy follow the descriptions of their public reference implementations (colletlz4; googlesnappy). Baselines. To calibrate the results, we apply the same procedure to three reference sources and include them in the same table: (i) an i.i.d. Rademacher sign stream as a maximally unstructured reference; (i) a structured two-dimensional Ising sign field at a fixed inverse-temperature setting, which introduces short-range correlations (ising1925; onsager1944); and (i) a rank-2 low-rank sign template constructed by taking the sign of a low-rank factor product, which provides an explicit low-complexity sign pattern. Because (i) and (i) contain deliberate structure, they are expected to be more compressible than the i.i.d. reference under generic codecs. Results. Table 9 reports compression ratios for learned sign bitstreams across compressors. Across architectures, learned sign bits compress nearly as poorly as the i.i.d. Rademacher reference under multiple codecs, whereas the structured baselines (2D Ising and the rank-2 template) are consistently more compressible. This consistency across compression families supports the interpretation that the observed “random-like” behavior of learned sign patterns is not an artifact of a particular spectral statistic: from an algorithmic coding perspective, the learned sign bitstreams expose little redundancy that generic lossless compressors can exploit. Taken together with our spectral evidence, these results reinforce the picture that sign patterns can dominate bit-cost in the sub-bit regime while remaining difficult to compress by generic means. E.3 Additional Vision Task Experiments E.3.1 Experimental setup Dataset. We use MNIST in the standard Keras format, normalized by the usual mean and standard deviation (mean 0.13070.1307, std 0.30810.3081), and we add a single channel dimension. We form a deterministic train/validation split by shuffling the original 60k training examples with a fixed seed and taking 10k examples for validation (50k remain for training). The test set contains 10k examples. Model. We train a simple MLP with two hidden layers: Flatten → Linear(784→256784\!→\!256) → ReLU → Linear(256→256256\!→\!256) → ReLU → Linear(256→10256\!→\!10). Optimization. We optimize the cross-entropy loss using AdamW with batch size 128 (training) and 512 (evaluation), we train for T=2000T=2000 steps per run. We report validation error Errval=1−AccvalErr_val=1-Acc_val and also track sign-related statistics described below. Initialization and Gap+reg method. The baseline initialization samples each linear weight entry i.i.d. from (0,σinit2)N(0, _init^2) with σinit=0.02 _init=0.02, and sets biases to zero. For the Gap+reg runs, we additionally apply a near-zero rejection (gap) rule to matrix parameters: we resample entries until |w|≥a|w|≥ a (gap threshold a). We also add an outer-drift (log-barrier) regularizer with weight λ as in Appendix C.4. Sign tracking and effective flips. This setting is the same as Appendix C.4. Zero-inflated geometric model. This setting is the same as Appendix C.4. Mean flip rate. This setting is the same as Appendix C.4. E.3.2 Effective Flip Distribution and Geometric Tail Figure 17 shows a representative histogram of the effective flip count K under baseline training. A large fraction of parameters never experience an effective flip (K=0K=0), with rapidly diminishing probability mass for larger K. This heavy mass at zero is consistent with the sign lock-in picture: once a parameter commits to an outer-region sign, it rarely returns to the near-zero boundary band and re-emerges with the opposite sign. To test the geometric-tail prediction, Figure 18 plots the tail probability ℙ[K≥k]P[K≥ k] on a log scale for multiple learning rates. Across learning rates, the tail is approximately linear in k on the semi-log plot, indicating an exponential/geometric decay. The dashed lines show simple geometric fits of the form ℙ[K≥k]≈hgk−1P[K≥ k]≈ h\,g^k-1. We observe that increasing the learning rate increases both the fitted prefactor h and the ratio g (e.g., h and g are smallest for lr=10−4lr=10^-4 and largest for lr=10−3lr=10^-3), which corresponds to (i) more parameters reaching the boundary at least once and (i) a heavier tail once flips occur. This matches the intuition that larger updates make boundary re-entry events more common. Figure 17: Empirical distribution of the effective flip count K. Most weights exhibit k=0k=0, with a rapidly decaying tail for k≥1k≥ 1. Figure 18: Semi-log plot of tail probabilities ℙ[K≥k]P[K≥ k] for different learning rates. Dashed lines show geometric fits ℙ[K≥k]≈hgk−1P[K≥ k]≈ h\,g^k-1, indicating an approximately geometric tail whose heaviness increases with learning rate. E.3.3 Estimated lock-in parameters over gap-init and regularization on vision task. We sweep the gap threshold and regularizer weight over a∈0.001,0.005,0.02,0.03,0.05,λ∈10−4,10−3,10−2,0.05,0.1,0.3,0.5.a∈\0.001,0.005,0.02,0.03,0.05\, λ∈\10^-4,10^-3,10^-2,0.05,0.1,0.3,0.5\. We then fit (h^,g^)( h, g) from the sampled weights and report mean± over 3 seeds. Figure 19 supports the interpretation that the two mechanisms act in complementary ways: (i) Gap initialization primarily suppresses h h, because moving initial weights away from the boundary reduces the probability that a parameter ever visits the near-zero band during training; indeed, at λ≈0λ≈ 0, h h decreases sharply as a increases. (i) Outer-drift regularization strongly reduces g g, because the log-barrier discourages re-entry into the boundary band once a parameter is in the outer region; correspondingly, g g drops monotonically as λ grows. At sufficiently large λ, both h h and g g become small, implying a much lighter tail and fewer total effective sign flips. (a) Initial-hit factor reduction. (b) Re-entry ratio reduction. Figure 19: Estimated lock-in parameters over gap-init and regularization on vision task. Left: h h (initial-hit factor) as a function of the gap threshold ainita_init for different log-barrier weights λ. Right: g g (re-entry ratio) as a function of λ (log scale) for different ainita_init. Points show mean± over three seeds. E.3.4 Trade-Off Between Mean Flip Rate and Validation Error Figure 20 plots validation error against the mean per-step flip rate (log-x) for the baseline model and for the Gap+reg sweep. Each curve corresponds to a fixed gap threshold a and varying λ. We observe a clear sign-stability/performance trade-off for some settings (notably small a): aggressive regularization can reduce flip rates substantially but may increase validation error, suggesting that overly constraining early dynamics can slow or misdirect optimization. However, the sweep also reveals favorable regimes (e.g., moderate/large a with intermediate λ) where flip rates drop by orders of magnitude relative to the baseline while validation error remains comparable. Overall, these results indicate that sign stabilization can be tuned to reduce sign churn substantially without requiring a large accuracy penalty. The resulting curves are shown in Figure 21(a)–21(c). Consistent with the CharLM analysis in the main paper, the sign matrix is less compressible than the magnitude matrix in the baseline regime, while the sign becomes substantially more low-rank compressible under the enhanced lock-in settings. In particular, the regularized regime exhibits the strongest improvement for sign(W)sign(W) across the rank-ratio sweep, indicating that enforcing sign lock-in can mitigate the sign bottleneck in the low-rank compression view. Figure 20: Validation error vs. mean flip rate. Baseline is shown as a single point, and Gap+reg curves connect runs with the same a while varying λ. Error bars denote mean± over seeds. The plot highlights both the trade-off regime (very low flip rate can hurt validation error for small a) and regimes where flip rates are drastically reduced with little or no degradation in validation error. Curves connect points with the same gap threshold ainita_init and the log barrier weight decreases from left to right as 0.5,0.3,0.1,0.05,0.01,0.001,0.00010.5,0.3,0.1,0.05,0.01,0.001,0.0001. (a) Baseline. (b) Gap initialization. (c) Gap init. + regularization. Figure 21: Sign vs. magnitude low-rank compressibility. Relative Frobenius error Er(M)=‖M−Mr‖F/‖M‖FE_r(M)=\|M-M_r\|_F/\|M\|_F as a function of rank ratio q=r/dq=r/d (log scale), for the sign matrix S=sign(W)S=sign(W) and magnitude matrix A=|W|A=|W|, evaluated on the final trained weights under (a) baseline, (b) gap initialization only, and (c) gap initialization with outer-drift regularization. E.4 Approximately Zero-Cost Sign Template Method for Sub-Bit Compression We provide additional details on the template-based sub-bit compression method referenced in the main text. Let W∈ℝm×nW∈R^m× n be a weight matrix and decompose it as W=S⊙A,S∈±1m×n,A∈ℝ≥0m×n,W=S A,~~S∈\± 1\^m× n,~~A∈R_≥ 0^m× n, where S=sign(W)S=sign(W) and A=|W|A=|W|. In a conventional representation, storing S requires 11 bit per weight. In contrast, we restrict the sign matrix to a known sign template ∈±1m×nT∈\± 1\^m× n, which can be deterministically regenerated from minimal side information (e.g., a seed parameter). If we enforce sign(W)=sign(W)=T for the targeted matrices, the decoder can reconstruct T on the fly, and only the magnitudes A need to be stored. As a result, the cost per-weight sign becomes approximately zero: bitssign(W)≈0.bits_sign(W)≈ 0. (20) With respect to the memory capacity of the sign matrix in the sign template method, the storage cost of the model becomes zero when the matrix is generated using an identical seed and the same random number generator. Consequently, the storage cost associated with the sign matrix of the model is fully attributable to the storage cost of the program, except the seed integer. When deploying on GPUs, the constraints differ from those in CPU-based settings. In particular, constructing the coding (sign) matrices on-the-fly during inference via a pseudorandom number generator is impractical. Instead, we generate a single global coding matrix from a fixed seed and obtain the coding matrix for each weight matrix by slicing the corresponding submatrix. Even in this scheme, bitssign(W)bits_sign(W) is less than 1/1001/100 because the single global coding matrix is reduced by the low-rank matrix factorization in Section E.4.1 and the reuse effect, which has a large contribution in LLM. In this Appendix, the bit cost of magnitude is about 1/241/2^4 in lowest case and we ignore the cost of the sign template. E.4.1 Sign Template For each target layer l∈ℳl∈M with weight shape m×nm× n, we use a re-generable sign template (l)∈±1m×nT^(l)∈\± 1\^m× n and enforce the sign constraint. Low-rank sign template. We generate low-rank real factors and then take the sign: G∈ℝm×r,H∈ℝn×r,Gik∼i.i.d.Pinit,Hjk∼i.i.d.Pinit,(l)=sign(GH⊤),G∈R^m× r,\;H∈R^n× r,~~G_ik .i.d. P_init,\;H_jk .i.d. P_init,~~T^(l)=sign(GH ), (21) where r≪min(m,n)r (m,n) controls the intrinsic degrees of freedom. E.4.2 Additional Sign Lock-In Enhancement Mechanism In practice, we combine the template approach with the two interventions introduced in the main text, such as gap initialization and outer-drift regularization, to suppress early visits to the sign boundary. Template-aware gap initialization. Denoting the near-zero-rejection sample matrix by Z(l)Z^(l), we initialize W0(l)=(l)⊙|Z(l)|.W^(l)_0\;=\;T^(l) |Z^(l) |. Hard projection. After each optimizer update, we perform an element-wise hard projection W(l)←Πhard(l)(W(l)),Πhard(l)(W)ij=ij(l)⋅|Wij|,l∈ℳ,W^(l)\;←\; _hard^(l)(W^(l)),~~ _hard^(l)(W)_ij=T^(l)_ij·|W_ij|,~~l∈M, (22) which enforces sign(W(l))=(l)sign(W^(l))=T^(l) precisely while preserving magnitudes. This method eliminates remaining sign flips that could not be corrected by gap initialization and outer-drift regularization. Remark E.1 (Combining Gap init and outer-drift reduces hard-projection activations). Proposition D.29 shows that under Gap initialization and outer-drift regularization, the boundary-hit times satisfy a geometric tail bound: ℙ[τk≤T]≤hTgap(gTOD)k−1P[ _k≤ T]≤ h_T^gap(g_T^OD)^k-1. In our implementation, the hard projection (Eq. (22)) is nontrivial only when the raw update crosses the sign boundary, i.e., when an entry attempts to move to the opposite sign side. On the good event ℰΔE_ from Assumption 3.3 (hence |wt+1−wt|≤Δ|w_t+1-w_t|≤ up to time T) and the choice ϵ=maxϵ0,Δε= \ _0, \ (Definition 3.1), such a sign-crossing event can occur only after the trajectory approaches the boundary neighborhood |wt|≤ϵ\|w_t|≤ε\ closely (indeed, crossing 0 in one step forces |wt|≤Δ≤ϵ|w_t|≤ ≤ε). Therefore, reducing boundary visits (smaller hTgaph_T^gap) and suppressing re-entries (smaller gTODg_T^OD) also reduces the frequency of nontrivial hard-projection activations. Empirically, we observe that the combination of these three components—Gap initialization, outer-drift regularization, and hard projection—yields substantially fewer projection activations than using hard projection alone. E.4.3 Magnitude Quantization using SVD After fixing signs, we only store A(l)=|W(l)|A^(l)=|W^(l)| for each l∈ℳl∈M. We apply truncated SVD: A(l)≈Ar(l):=Ur(l)Σr(l)(Vr(l))⊤,A^(l)≈ A^(l)_r:=U^(l)_r\, ^(l)_r\,(V^(l)_r) , and quantize the factors with a b-bit symmetric uniform quantizer Qb(x;α):=α⋅clip(⌊x/α⌉,−2b−1, 2b−1−1).Q_b(x;α):=α·clip\! ( x/α ,\;-2^b-1,\;2^b-1-1 ). (23) Define A^(l):=QbU(Ur(l);αU(l))QbΣ(Σr(l);αΣ(l))QbV(Vr(l);αV(l))⊤, A^(l):=Q_b_U\! (U^(l)_r; _U^(l) )\;Q_b_ \! ( ^(l)_r; _ ^(l) )\;Q_b_V\! (V^(l)_r; _V^(l) ) , so the reconstructed weight is W^(l)=T(l)⊙A^(l). W^(l)=T^(l) A^(l). The amortized magnitude bit-cost per weight for an m×nm× n matrix is bitsamp(W(l))≈bUmr+bVnr+bΣrmn,bits_amp(W^(l))≈ b_U\,mr+b_V\,nr+b_ \,rmn, (24) while the sign bit-cost remains approximately zero for l∈ℳl∈M. E.4.4 Experimental Validation (a) charlm_kd (b) text8_char_kd (c) dbpedia14_kd Figure 22: Performance vs. effective bits per weight (bpweffbpw_eff) under knowledge distillation. Markers indicate the mean over three seeds and error bars show one standard deviation. Lower is better for perplexity (CharLM-KD, Text8-Char-KD), while higher is better for accuracy (DBPedia14-KD). These panels complement the main benchmark results in Figure 8. We next validate the practical effectiveness of the proposed zero-cost sign template approach. Recall that, when we constrain the sign of each targeted weight matrix to a deterministically re-generable template, the sign storage becomes zero in Eq. (20), and only the nonnegative magnitudes need to be stored and compressed. In our implementation, we compress magnitudes via truncated SVD and store quantized factors. We focus on the Transformer for language tasks due to emerging needs for the model compression. Figure 8 reports the main benchmark tasks, while Figure 22 reports additional knowledge-distillation variants. The details of this validation are reported in Section E.4.5. Protocol and metrics. Figures 8 and 22 report task performance as a function of the effective bits-per-weight bpweffbpw_eff on six benchmarks: CharLM and Text8-Char (test perplexity; lower is better), and DBPedia14 (test accuracy; higher is better), including their KD variants. Each marker indicates the mean over three random seeds and error bars show one standard deviation. All methods are compared at approximately matched bpweffbpw_eff under the accounting rules in Appendix E.4.5. Main result: magnitude-only SVD with zero-cost signs is consistently strong in the sub-bit regime. Across all benchmarks, our template-based method (SVD |Wlockin||W_lockin|) substantially improves over applying the same SVD budget directly to raw weights (SVD W baseline) applied to the baseline weight matrix in the extreme low-bit region bpweff<1bpw_eff<1. This behavior matches the empirical motivation in Section 1: sign patterns are random-like and difficult to compress, while magnitudes are more compressible and become the natural target once signs are made free. Concretely, on DBPedia14 at bpweff≈0.24bpw_eff≈ 0.24, SVD |Wlockin||W_lockin| achieves high accuracy while SVD W baseline collapses; similarly, on Text8-Char and CharLM at the same budget, SVD |Wlockin||W_lockin| yields clearly lower perplexity than SVD W baseline. The gains persist on KD tasks (Figure 22), indicating that the advantage is not specific to a single training objective. Effect of simple preconditioning (naive vs. z-score). We report two variants of magnitude SVD: (i) a naive pipeline that factorizes |W||W| as-is, and (i) a lightly preconditioned variant that normalizes magnitudes before SVD (z-score). The detail of this method is described in Appendix E.4.6. The z-score variant consistently dominates the naive variant across budgets and tasks, suggesting that even simple normalization improves the stability of low-rank factor quantization at very small bpweffbpw_eff. Comparison with existing extreme-compression baselines. We additionally include representative existing approaches: HashedNets (weight sharing), OneBit, and unstructured pruning baselines (magnitude pruning and WANDA). Overall, the template-based magnitude SVD is the most reliable performer in the sub-bit region. Notably, when pruning is evaluated with a realistic sparse-storage model (CSR; Appendix E.4.5), the indexing overhead dominates at very small densities, leading to weak performance at matched bpweffbpw_eff. HashedNets improves as the budget increases but remains unreliable at the smallest budgets. E.4.5 Experimental Details of Zero-Cost Sign Template Method This appendix describes the experimental setup, bit-budget accounting, and baseline implementations used to produce the results in Figures 8 and 22. Benchmarks. We evaluate on six tasks: • CharLM / Text8-Char: character-level language modeling; we report test perplexity (PPL). • DBPedia14: 14-way text classification; we report test accuracy. • KD variants (_kd): student models trained with knowledge distillation (KD) using a teacher trained on the corresponding base task. Across all tasks, we report mean± over three seeds (0/1/2). Models and Targeted Weights. Our zero-template method is applied to a fixed set of targeted weight matrices (linear layers) in each model; all other parameters are maintained in full precision. In the experiments presented in Figures 8 and 22, the number of targeted parameter tensors is as follows: • CharLM / Text8-Char: 14 targeted tensors. • DBPedia14: 28 targeted tensors. Sign template. For each targeted layer l with shape m×nm× n, we define a re-generable sign template (l)∈±1m×nT^(l)∈\± 1\^m× n. Unless stated otherwise, we use Pinit=Unif[−1,1]P_init=Unif[-1,1] to sample the i.i.d. entries of the low-rank factors (G,H)(G,H) in Eq. (21) with a fixed global seed. Gap initialization and regularization. We use a simple gap initialization that avoids near-zero magnitudes by enforcing an absolute minimum magnitude at initialization. In the reported runs, the gap threshold is chosen from ainit=0.0,0.01,0.02,0.03a_init=\0.0,0.01,0.02,0.03\ and the outer-drift (log-barrier) regularization weight is set from λ=0.0,10−4,3.0×10−4,10−3λ=\0.0,10^-4,3.0× 10^-4,10^-3\ within 0.02 PPL drop or 2 % accuracy drop. This small performance degradation is consistent across tasks, so we could choose parameters other than ainit=0a_init=0 and λ=0λ=0 for two proposed methods. The two proposed methods are evaluated using the resulting sign-fixed weights. For the baseline, we use the resulting weight of ainit=0a_init=0 and λ=0λ=0. Hard projection. During template-constrained training, after each optimizer update we apply the hard projection (Eq. (22)) to enforce sign(W(l))=(l)sign(W^(l))=T^(l) exactly for all targeted layers. This guarantees that the sign component incurs zero storage cost at compression time. E.4.6 Preconditioning Variants for Magnitude SVD. For SVD |Wlockin||W_lockin|-naive, we apply truncated SVD directly to the magnitude matrix A:=|Wlockin|∈ℝ≥0m×nA:=|W_lockin|∈R^m× n_≥ 0. For SVD |Wlockin||W_lockin|-zscore, we normalize A column-wise before SVD and invert the normalization after reconstruction. Let A:=|Wlockin|∈ℝ≥0m×nA:=|W_lockin|∈R^m× n_≥ 0. Define the per-column mean and (population) standard deviation by ζj:=1m∑i=1mAij,ωj:=1m∑i=1m(Aij−ζj)2,j=1,…,n. _j:= 1m _i=1^mA_ij,~~ _j:= 1m _i=1^m(A_ij- _j)^2,~~j=1,…,n. Let ζ:=(ζ1,…,ζn)⊤∈ℝnζ:=( _1,…, _n) ∈R^n, ω:=(ω1,…,ωn)⊤∈ℝ≥0nω:=( _1,…, _n) ∈R^n_≥ 0, and let m∈ℝm1_m∈R^m and n∈ℝn1_n∈R^n denote all-ones vectors. Introduce a small numerical stabilizer ϵzs>0 _zs>0 and define Dzs:=diag(ω+ϵzsn)∈ℝn×n.D_zs:=diag(ω+ _zs1_n)∈R^n× n. The column-wise z-score normalized matrix is Azs:=(A−mζ⊤)Dzs−1∈ℝm×n.A^zs:=(A-1_mζ )\,D_zs^-1∈R^m× n. We compute a rank-r truncated SVD Azs≈UrΣrVr⊤,A^zs≈ U_r _rV_r , fold the singular values into the left factor U~r:=UrΣr U_r:=U_r _r, apply the main-text symmetric uniform b-bit quantizer Qb(⋅)Q_b(·) to the factors, and reconstruct Abzs:=Qb(U~r)Qb(Vr⊤).A^zs_b:=Q_b( U_r)\,Q_b(V_r ). Finally, we invert the normalization and enforce nonnegativity (element-wise): Ab:=AbzsDzs+mζ⊤,Ab←max(Ab,0).A_b:=A^zs_b\,D_zs+1_mζ ,~~A_b← (A_b,0). We then form the sign-fixed weight using the fixed sign template T: Wb:=T⊙Ab.W_b:=T A_b. Bit-budget Grid. We evaluate a sweep of target effective budgets bpwtarget∈0.075, 0.125, 0.25, 0.5, 0.75, 1.0, 1.58, 2.0.bpw_target∈\0.075,\,0.125,\,0.25,\,0.5,\,0.75,\,1.0,\,1.58,\,2.0\. Figures 8 and 22 visualize the range up to 2.02.0 bpw to emphasize the sub-bit regime. Bit accounting of SVD-based storage for dense matrices (ours and SVDW baseline). Given a target matrix M∈ℝm×nM∈R^m× n and rank r, we store quantized SVD factors (Ur∈ℝm×rU_r∈R^m× r, Σr∈ℝr×r _r∈R^r× r as a diagonal vector, and Vr∈ℝn×rV_r∈R^n× r) using symmetric uniform b-bit quantizers (Eq. (23)). Following Eq. (24), the amortized bit cost per original weight is approximated by bpweff≈b⋅(mr+nr+r)mn,bpw_eff≈ b·(mr+nr+r)mn, with b=4b=4 in our experiments (i.e., 4-bit quantization for the stored factors). For the zero-template method, sign bits are not stored (Eq. (20)); for SVDW baseline, the same SVD method is applied directly to W. Bit accounting of CSR storage for unstructured sparsity (pruning and WANDA). For pruning-based methods, we assume CSR storage for each pruned matrix: bpweff≈nnz⋅bval⏟stored values+nnz⋅bidx+(m+1)⋅bptr⏟indices/row pointersmn,bpw_eff≈ nnz· b_val_stored values+ nnz· b_idx+(m+1)· b_ptr_indices/row pointersmn, where nnznnz is the number of nonzeros, bval=4b_val=4 bits for quantized nonzero values, and we use 32-bit indices/pointers (bidx=bptr=32b_idx=b_ptr=32). This accounting explains why pruning can be unfavorable at extreme densities: index overhead dominates. We also impose a practical start threshold and only run pruning methods when the implied keep ratio exceeds prune_start_keep_frac=0.005. prune\_start\_keep\_frac=0.005. Baselines. We summarize the baselines shown in Figures 8 and 22. All results shown in these figures are computed using three seeds (0, 1, 2) and reported as mean± . We log per-method bpweffbpw_eff along with its decomposition (e.g., sign/index overhead vs. value bits) to ensure bit-budget correctness for each storage model. • HashedNets: weight sharing through hashing into a budget-dependent number of buckets; bucket values are stored with the same value-bitwidth used elsewhere (4-bit in our implementation). No task-specific fine-tuning is employed. • OneBit: explicit 1-bit sign storage with a small overhead for scale information. • Pruning: magnitude-based unstructured pruning with CSR accounting; prune-only (no recovery fine-tuning applied). • WANDA: activation-aware pruning utilizing the standard WANDA score; prune-only; CSR accounting. • QAT (reference): we include 1-bit and ternary QAT as reference points. In our code, QAT employs a STE with a short fine-tuning schedule applied to the targeted tensors; KD tasks utilize the same teacher as the KD student training. @envEnd