Paper deep dive
Rethinking Activation Functions: Reducing Bias Can Improve Learning
Zirui Peng, Heyi Zheng, Ziming Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/14/2026, 10:04:59 AM
Summary
This paper investigates the impact of 'activation bias'—the nonzero mean of post-activation outputs in functions like ReLU, GELU, Swish, and Sigmoid—on learning dynamics. Through controlled teacher-student models and large-scale experiments on Vision Transformers, Diffusion Transformers, and GPT-2, the authors demonstrate that while activation bias accelerates initial loss reduction, it leads to non-smooth trajectories, hidden-state overshooting, and a steeper loss landscape. Conversely, reducing activation bias (via shifting) slows early learning but results in smoother optimization, flatter loss landscapes, and superior late-stage performance and generalization.
Entities (11)
Relation Signals (12)
Activation Bias → causes → Accelerated Initial Learning
confidence 95% · activation bias accelerates initial loss reduction
GeLU → exhibits → Activation Bias
confidence 95% · Many widely used activation functions, including... GELU... produce predominantly positive outputs
ReLU → exhibits → Activation Bias
confidence 95% · Many widely used activation functions, including ReLU... produce predominantly positive outputs, resulting in a nonzero post-activation mean
Swish → exhibits → Activation Bias
confidence 95% · Many widely used activation functions, including... Swish... produce predominantly positive outputs
Sigmoid → exhibits → Activation Bias
confidence 95% · Many widely used activation functions, including... Sigmoid... produce predominantly positive outputs
Reducing Activation Bias → leadsto → Lower Final Loss
confidence 93% · reducing activation bias slows initial learning but ultimately yields a lower final loss
Activation Bias → causes → Non-smooth Trajectories
confidence 92% · accompanied by overshooting, non-smooth trajectories
Activation Bias → causes → Steeper Loss Landscape
confidence 90% · activation bias is associated with a markedly steeper local loss landscape
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-activation mean that we term activation bias. Despite its ubiquity, the effect of activation bias on learning dynamics remains poorly understood. Using controlled models, we show that activation bias accelerates initial loss reduction but is accompanied by overshooting, non-smooth trajectories, and a steeper loss landscape. This early advantage also coincides with the model's use of task-relevant information encoded in the activation bias component. In contrast, reducing activation bias slows initial learning but ultimately yields a lower final loss. Experiments on larger vision and language models exhibit a similar pattern: smaller activation bias is associated with better late-stage performance. Thus, this seemingly minor property can substantially redirect learning and warrants explicit consideration in future activation function design.
Tags
Links
- Source: https://arxiv.org/abs/2610.14500v1
- Canonical: https://arxiv.org/abs/2610.14500v1
Trouble viewing inline? Open PDF directly →
Full Text
65,056 characters extracted from source content.
Expand or collapse full text
texgyretermes-regular.otf[ BoldFont=texgyretermes-bold.otf, ItalicFont=texgyretermes-italic.otf, BoldItalicFont=texgyretermes-bolditalic.otf] [table]skip=4pt [1] enhanced, colback=questionback, colframe=questionred, colbacktitle=questionred, coltitle=white, fonttitle=, title=#1, boxrule=1.1pt, arc=2m, left=3m, right=3m, top=2m, bottom=2m, before skip=0.7after skip=0.9 2026 Rethinking Activation Functions: Reducing Bias Can Improve Learning Zirui Peng Affiliation: Department of Physics, Peking University Heyi Zheng Affiliation: Department of Mathematics, Peking University Ziming Liu Affiliation: College of AI, Tsinghua University Affiliation: MetaCircle*Email: :fredpengphy@stu.pku.edu.cnfredpengphy@stu.pku.edu.cn metaabstract Many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-activation mean that we term activation bias. Despite its ubiquity, the effect of activation bias on learning dynamics remains poorly understood. Using controlled models, we show that activation bias accelerates initial loss reduction but is accompanied by overshooting, non-smooth trajectories, and a steeper loss landscape. This early advantage also coincides with the model’s use of task-relevant information encoded in the activation bias component. In contrast, reducing activation bias slows initial learning but ultimately yields a lower final loss. Experiments on larger vision and language models exhibit a similar pattern: smaller activation bias is associated with better late-stage performance. Thus, this seemingly minor property can substantially redirect learning and warrants explicit consideration in future activation function design. Figure 1: Overview of activation bias and its effects on learning. Blue denotes the unshifted activation, and red denotes the shifted activation. Reducing activation bias changes the post-activation distribution, model performance, hidden-state trajectory, and loss landscape. 1 Introduction Many common activation functions, including ReLU, GELU, Swish, and Sigmoid, favor positive outputs while suppressing or excluding negative ones. These asymmetries typically yield a nonzero post-activation mean. For a pre-activation z, we define the activation bias of an activation function ϕφ as the mean of its post-activation: bϕ≡[ϕ(z)].b_φ [φ(z)]. (1) These functions are widely used and perform well across diverse tasks. Their success, however, does not establish how the post-activation bias they induce affects learning dynamics. Prior work connects activation design to trainability through signal propagation, dynamical isometry, critical initialization, and normalization (Clevert et al., 2016; Brock et al., 2021; Qi et al., 2020; Lesser and Chowdhury, 2026; Björck et al., 2018; Santurkar et al., 2018). These studies primarily address propagation stability or optimization conditioning, leaving open how activation bias affects learning over time and whether the bias built into common activations is optimal for learning. We therefore ask: researchquestion Q: How does activation bias shape learning dynamics? (i) How does it alter training dynamics? (i) Does the activation bias component carry useful information, and how does the model use that information during learning? We first investigate the effect of activation bias on training dynamics in a controlled deep residual teacher–student model. For each activation function, we use a fixed post-activation shift to control the degree of activation bias. Both students learn from the same fixed teacher, with identical initial parameters, training data, and optimization settings; they differ only in whether the shift is applied. Reducing activation bias slows initial loss reduction, but the shifted student eventually overtakes its unshifted counterpart and attains a lower final loss. Theory and experiments associate this reversal with smoother optimization dynamics, distinct parameter-update directions, and a flatter loss landscape, reflected in substantially smaller Hessian eigenvalues. We then use a controlled image-reconstruction task to test whether the activation bias component carries information the model can exploit. Early in training, decoding this component alone recovers recognizable image content. Its output later departs from the full-representation reconstruction, suggesting that the model relies less on this information as training proceeds. This transient use reveals a non-smooth feature-learning trajectory and is largely suppressed when activation bias is reduced. Finally, we test whether these findings persist in larger models and across a wider range of tasks. These models exhibit analogous non-smooth dynamics under the original activations, while reducing activation bias yields comparable or better late-stage performance. Thus, this seemingly small activation property can produce pronounced changes in learning across architectures and tasks. researchquestion Our contributions are the following: [leftmargin=5m,labelsep=1.5m,itemsep=2pt,topsep=1pt, parsep=0pt,partopsep=0pt] • We use a controlled model to investigate activation bias and reveal its effect: it accelerates early learning but impedes later progress. • We characterize how activation bias shapes optimization dynamics and how the model uses information carried by the activation bias component during learning. • We observe similar learning dynamics and improved late-stage performance with reduced activation bias across larger models and a wider range of tasks. To our knowledge, this is the first study to examine how activation bias shapes learning dynamics throughout training. Our results show that a small change in activation bias can redirect optimization, yield smoother learning trajectories, and improve late-stage performance. This sensitivity calls for treating activation bias as an explicit property in activation-function design rather than an incidental by-product. 2 Analysis of Training Dynamics 2.1 Controlled Teacher–Student Setup Figure 2: Activation bias accelerates early learning, but this advantage coincides with a pronounced hidden-state overshoot and reverses later in training. Top: The shifted model learns more slowly initially but ultimately reaches a lower loss across activation functions. Bottom: The unshifted model’s hidden state overshoots and follows a non-smooth trajectory. To isolate the effect of activation bias from other factors, we adopt a simple controlled model. We omit token embeddings, attention, and the output head to isolate the activation function’s role while retaining the pre-normalized MLP blocks and residual-stream architecture of modern large language models. For layers ℓ=0,…,L−1 =0,…,L-1, the model evolves as (ℓ+1)=(ℓ)+2(ℓ)ϕ(1(ℓ)LN((ℓ))),x^( +1)=x^( )+W_2^( )φ\! (W_1^( )LN\! (x^( ) ) ), (2) where (ℓ)x^( ) denotes the residual-stream representation, 1(ℓ)W_1^( ) and 2(ℓ)W_2^( ) are the up- and down-projection matrices, respectively, LNLN denotes pre-normalization, and ϕφ is the nonlinear activation function. We control activation bias by subtracting the same fixed scalar b from the activation output in every layer: ϕshifted(z,b)=ϕ(z)−b. _shifted(z;b)=φ(z)-b. (3) To avoid confounding activation bias with complex data structure that is difficult to isolate and characterize, we use a teacher–student framework. The teacher and both students share the architecture in Equation 2. A fixed teacher using the unshifted activation ϕφ generates targets y=fT()y=f_T(x) for sampled inputs x. We refer to the student using ϕφ as the unshifted student and the student using ϕshifted _shifted as the shifted student. For each activation, we estimate b as its expected post-activation output at initialization. This shift keeps the post-activation mean near zero over an extended period of training, as detailed in Appendix A. Table 1: Activation shifts b used for each activation function. Activation functions ReLU GELU Swish Sigmoid b 0.2300.230 0.1150.115 0.0770.077 0.5000.500 2.2 Early Learning Advantage Coincides with Hidden-State Overshoot We first compare the two students through their loss curves. As shown in the top panel of Figure 2, the unshifted student learns faster initially, but the shifted student eventually overtakes it and converges to a lower loss. This reversal is notable because the unshifted student uses exactly the same activation as the teacher yet is ultimately outperformed by the shifted student. Motivated by this loss reversal, we first track the residual-stream hidden state over training. We average its elements for each input and then average across inputs, yielding a scalar trajectory that tracks the hidden state’s evolution during training. In the bottom panel of Figure 2, the unshifted student’s hidden state overshoots around step 100100, coinciding with its period of faster loss reduction. The shifted student’s hidden state instead evolves smoothly. These distinct hidden-state trajectories motivate examining whether the two students also follow different paths in parameter space. 2.3 Activation Bias Separates Parameter-Update Directions To compare parameter trajectories, let 0 θ_0 be the common initialization and let ug_u and sg_s be the students’ initial gradients, where the subscripts uu and ss denote the unshifted and shifted students, respectively. We construct the initial-gradient plane with orthonormal basis 1=u∥u∥,2=(−11⊤)s∥(−11⊤)s∥,e_1= g_u _u , _2= (I-e_1e_1 )g_s (I-e_1e_1 )g_s , (4) where 1e_1 follows the unshifted gradient and 2e_2 is the orthogonal component of the shifted gradient. To better understand the early optimization geometry, we visualize each student’s loss landscape on the initial-gradient plane. At each coordinate (α,β)(α,β), we reconstruct the parameters as 0+α1+β2 θ_0+ _1+ _2 and evaluate the corresponding student against the fixed teacher on the evaluation set. The resulting losses form a heat map of the projected landscape. Projecting training checkpoints onto the same plane overlays each student’s trajectory. As shown in the left and center panels of Figure 3, the two trajectories initially point in nearly orthogonal directions. The unshifted student initially follows the direction of strongest descent in its projected landscape and then turns sharply, revealing a distinctly non-smooth trajectory. The shifted student’s loss varies little on this plane, indicating that its early update develops largely outside these initial-gradient directions. To visualize the shifted student’s parameter movement outside this plane, we remove the parts of its parameter displacement along 1e_1 and 2e_2: t=(−11⊤−22⊤)((t)−0).r_t=(I-e_1e_1 -e_2e_2 )( θ(t)- θ_0). (5) Here, tr_t is the component of the shifted student’s parameter displacement orthogonal to the initial-gradient plane. To obtain two orthogonal directions from the shifted student’s parameter displacements outside the initial-gradient plane, we apply Gram–Schmidt orthogonalization to these displacements and use the first two resulting basis vectors 1n_1 and 2n_2 as its axes. We then visualize the loss and trajectories on this plane as above. The right panel of Figure 3 reveals clear loss variation along the shifted student’s smooth trajectory without a comparable turn. Together, these views show that activation bias induces qualitatively distinct parameter-update directions. The two students initially move in nearly orthogonal directions, and their subsequent updates remain largely separated: the unshifted trajectory turns within the initial-gradient plane, whereas the shifted trajectory develops primarily in complementary directions and evolves smoothly. Further theoretical analysis and experiments on update directions are provided in Appendix B. Figure 3: Activation bias redirects parameter updates, producing a distinct trajectory with a sharp, non-smooth turn. Left: the unshifted student on the initial-gradient plane. Center: the shifted student on the same plane. Right: the shifted student on the complementary plane. Colors closer to red indicate higher loss, while colors closer to blue indicate lower loss. 2.4 Activation Bias Steepens the Loss Landscape Although the projected landscapes suggest differences in curvature, each shows only a two-dimensional slice. We therefore quantify local curvature in the full parameter space using the leading eigenvalues of the Hessian =∇2ℒH=∇^2L, whose largest eigenvalue is commonly termed sharpness. Figure 4 shows that the shifted student has substantially smaller eigenvalues throughout training. Thus, in this controlled model, activation bias is associated with a markedly steeper local loss landscape, whereas reducing the bias yields a flatter one. Figure 4: Reducing activation bias substantially lowers Hessian eigenvalues throughout training, yielding a flatter local loss landscape. To examine how activation bias affects loss landscape curvature, we consider a simplified case with a post-activation vector (x)a(x) and a trainable down-projection 2W_2 with squared loss ℒb(2)=12x‖2((x)−b)−(x)‖2.L_b(W_2)= 12E_x \|W_2 (a(x)-b1 )-y(x) \|^2. (6) Suppose []=μE[a]= 1, and let Σ=Cov() =Cov(a). The Hessian H(b)H(b) with respect to 2W_2 then satisfies H(b)=[Σ+(μ−b)2⊤]⊗I⪰Σ⊗I,H(b)= [ +(μ-b)^211 ] I I, (7) where I is the identity matrix and ⊗ denotes the Kronecker product. Since (μ−b)2⊤⊗I(μ-b)^211 I is positive semidefinite, the largest eigenvalue satisfies λmax(H(b))≥λmax(H(μ)) _ (H(b))≥ _ (H(μ)). Thus, b=μb=μ minimizes Hessian sharpness in this setting. A full proof and an extension to a more general multi-layer model are provided in Appendix B. Taken together, these measurements show that activation bias gives the unshifted student a clear early learning advantage, but this advantage coincides with an overshooting hidden-state trajectory and a markedly steeper loss landscape. Reducing the bias slows initial learning, changes the parameter-update directions, and produces smoother dynamics with stronger long-term performance. Additional ablations are provided in Appendix C. 3 Analysis of Activation Bias Component Information Section 2 shows that activation bias yields a transient early advantage yet worse long-run optimization. We next ask whether the associated activation bias component carries task-relevant information that the model uses. Figure 5: The unshifted model recovers recognizable image content from the activation bias component early in training but relies less on it later; the shifted model recovers little content from this component. 3.1 Image Reconstruction Setup Image reconstruction makes information carried by the activation bias component directly observable in decoded images. We therefore choose this task and use a simplified VGG-style autoencoder, →encoder→decoder^,x\ \ encoder\ \ h\ \ decoder\ \ x, where x is an input image, h is its latent representation, and x is the reconstructed image. We use part of the VGG-19 architecture (Simonyan and Zisserman, 2015) as the encoder, followed by a convolutional decoder. The latent representation is the encoder’s final post-activation output, with 512 channels at each spatial location. The detailed architecture is provided in Appendix D. We compare otherwise identical models using the unshifted and shifted activations. Consistent with the controlled teacher–student model in Section 2, the unshifted model learns faster initially, but the shifted model slightly surpasses it later in training, as shown in Appendix D. Throughout training, both decoders receive the full representation h. For post-hoc analysis at selected checkpoints, let p∈ℝmh_p ^m denote the hidden state at spatial location p, where m=512m=512 is the number of channels. For every location p, we decompose it as p=(1m⊤p)⏟p,bias+(p−p,bias)⏟p,dev.h_p= ( 1m1 h_p )1_h_p,bias+ (h_p-h_p,bias )_h_p,dev. (8) 3.2 Visualization and Analysis of Activation-Bias-Component Information For each selected checkpoint, we pass the two components separately through the decoder. Figure 5 shows the resulting component-wise reconstructions. For the unshifted model, reconstruction from biash_bias alone yields a recognizable, though incomplete, image that retains coarse structure, color, and texture, implying that the model uses some information encoded in the activation bias component. By contrast, the corresponding reconstruction for the shifted model contains little recoverable structure, suggesting that this model makes less use of the component. The role of the unshifted model’s activation bias component evolves non-monotonically. Its isolated reconstruction becomes increasingly similar to the full reconstruction, reaching its closest agreement early in training. It subsequently diverges from the full reconstruction. This rise-and-fall pattern mirrors the early learning advantage and hidden-state overshoot in Section 2.2. Together, these visualizations show that activation bias can change how the model uses image information during training. The unshifted model’s transient use of the activation bias component provides a feature-learning counterpart to its early learning advantage, whereas the shifted model makes little use of this component. 4 Scale-Up Experiments We next examine whether our findings extend to larger, more complex architectures and diverse tasks. We consider supervised image classification, diffusion-based image generation, and language modeling. Unlike in the controlled model, subtracting the same fixed scalar in every layer is insufficient because activation bias differs across layers and evolves rapidly during training. We therefore use adaptive shifting as the bias-reduced condition. This layerwise intervention subtracts the current post-activation mean from every neuron in each layer: ϕ~(ℓ)((ℓ))=ϕ((ℓ))−1Nℓ∑i=1Nℓϕ(zi(ℓ)), φ^( )\! (z^( ) )=φ\! (z^( ) )- 1N_ _i=1^N_ φ\! (z_i^( ) )1, (9) where (ℓ)∈ℝNℓz^( ) ^N_ is the pre-activation vector in layer ℓ , NℓN_ is the number of neurons, and i indexes neurons. The shifted output has zero mean across neurons by construction. 4.1 Vision Transformer Image Classification We first consider Vision Transformers for supervised image classification. Following the ImageNet-1K training protocol of Zhu et al. (2025), we train ViT-Base (Dosovitskiy et al., 2021) for 300300 epochs. We hold all other training choices fixed across conditions, varying only whether adaptive shifting is applied. Representative ViT-Base results are shown in Figure 6. The model with adaptive shifting initially trails the unshifted baseline by a substantial margin, but the ordering reverses during training: by the end, it attains both lower test loss and higher top-1 classification accuracy, despite retaining a higher training loss. This result is consistent with our controlled-model observation that reduced activation bias is associated with lower Hessian sharpness and a flatter local landscape, which is often linked to better generalization and less overfitting. Figure 7(a) tracks the residual-stream hidden-state trajectory using the statistic defined in Section 2.2. The unshifted model exhibits a pronounced early overshoot during the period in which it holds a substantial performance advantage. Under this larger architecture and more complex task, both trajectories are noisy, but the adaptively shifted model exhibits a substantially smaller overshoot. This contrast echoes the pattern observed in the controlled model. Further examples of overshoot trajectories in scale-up language models are provided in Appendix E.2. Together, these results show that reducing activation bias remains beneficial for image classification at larger scale. (a) Training-loss residual (b) Test-loss residual (c) Top-1 accuracy residual Figure 6: For the ViT image-classification task, the shifted model retains a higher training loss but ultimately achieves lower test loss and higher test accuracy. The loss residuals are Lshifted−LunshiftedL_shifted-L_unshifted, and the accuracy residual is Accshifted−AccunshiftedAcc_shifted-Acc_unshifted. Figure 7: (a) Reducing activation bias suppresses hidden-state overshoot in ViT. (b) and (c): The shifted model performs better in both DiT image generation and GPT-2 language pretraining. The loss residual is Lshifted−LunshiftedL_shifted-L_unshifted. 4.2 Diffusion Transformer Image Generation We next consider image generation with Diffusion Transformers (DiTs) (Peebles and Xie, 2023). Following the official facebookresearch/DiT implementation, we train DiT-B/2 on ImageNet-256 for 400,000400,000 optimization steps. We hold all other training choices fixed and vary only whether adaptive shifting is applied. As shown in Figure 7(b), adaptive shifting achieves comparable or lower training loss throughout training. Additional results in Appendix E show that the shifted model achieves lower FID at every evaluated checkpoint. These results demonstrate that reducing activation bias can also benefit larger-scale image generation. 4.3 GPT-2 Pretraining We then move from vision to language modeling. Following Karpathy’s training setup (Karpathy, 2022), we train GPT-2 Small (Radford et al., 2019) from scratch on the full OpenWebText corpus (Gokaslan et al., 2020) for 300,000300,000 optimization steps. As above, we hold all other training choices fixed and vary only whether adaptive shifting is applied. Figure 7(c) reports the results. Adaptive shifting consistently improves the performance, stably lowering loss by about 0.01 in absolute terms and perplexity by 1% relative to the unshifted baseline. These gains show that the effect of reducing activation bias extends to language modeling. The larger-scale models and tasks considered here are widely adopted and have been extensively optimized, leaving relatively little headroom for improvement from a single change to the activation. Even so, adaptive shifting yields small but consistent gains. Together with the observed changes in learning dynamics, these gains indicate that activation bias has a non-negligible effect in practical models. Additional scale-up results are provided in Appendix E. 5 Discussions The actual effect of activation bias may vary in different settings. The extent of early acceleration, hidden-state overshoot, and final performance differences may depend on data and task structure, model scale, architectural components such as attention, and weight initialization. For example, during GPT-2 Small pretraining, the unshifted model does not exhibit the initial performance advantage seen in the controlled model, potentially due to the small initialization scale used by this architecture. Nevertheless, across the settings studied here, smaller activation bias consistently yields better late-stage performance. How these factors interact with activation bias merits further investigation. We use fixed or adaptive shifts to isolate a learning-dynamical consequence of activation bias. Although reducing the bias improves performance in our experiments, these results neither establish that zero post-activation bias is universally optimal nor guarantee that either intervention will benefit every architecture, task, or training regime. Performance can also depend on other desirable activation properties, including selectivity, signal scale, and smoothness. The methods used here to control activation bias therefore serve as diagnostic interventions instead of general prescriptions for activation design. Our results identify a set of connected signatures of activation bias: an early–late learning pattern, hidden-state overshoot and non-smooth dynamics, distinct parameter-update directions, a steeper loss landscape, and transient use of information in the activation bias component. We do not yet have a unified mathematical theory or a complete causal account of how these effects arise. Moreover, our larger-model experiments do not systematically test whether all the phenomena and theoretical results from the controlled setting extend to more complex architectures. Addressing these gaps could provide more principled guidance for the design of new activation functions. Activation functions have many properties beyond bias that we do not study here. Post-activation variance is one example: the variance most conducive to learning may depend on architecture, task, scale, and training regime, yet its effects on learning dynamics and final performance remain unclear. Future work could isolate variance and other properties of activation functions under controlled conditions, track their effects throughout training, and test whether the resulting patterns transfer across models and tasks. Such analyses would provide a broader foundation for activation-function design. 6 Related Work Activation-function design spans nonlinearities with markedly different input–output profiles. Early work used sigmoid and tanh (LeCun et al., 1998); later proposals introduced Softplus (Dugas et al., 2000), ReLU (Nair and Hinton, 2010), GELU (Hendrycks and Gimpel, 2016), and Swish (Ramachandran et al., 2018). Leaky ReLU (Maas et al., 2013), PReLU (He et al., 2015), and Mish (Misra, 2020) further broadened this design space. More recent work has explored gated activations such as SwiGLU (Shazeer, 2020), learnable activations such as ACON (Ma et al., 2021), and alternatives such as SMU (Biswas et al., 2022) and StarReLU (Yu et al., 2024). These activation functions have proved effective across tasks, yet many of them produce predominantly positive outputs and thus introduce activation bias. While prior work has focused on designing and evaluating the functions themselves, the effect of this bias on learning remains less understood. Our work complements these studies by isolating activation bias and examining how it shapes learning dynamics. Previous work on activation functions and their statistics has primarily sought to improve trainability. To stabilize intermediate signals, normalization methods center or rescale them (Ioffe and Szegedy, 2015; Ba et al., 2016; Arpit et al., 2016; Björck et al., 2018); mean-only batch normalization retains centering alone, whereas RMSNorm retains rescaling alone (Salimans and Kingma, 2016; Zhang and Sennrich, 2019). Santurkar et al. (2018) showed that batch normalization also smooths the loss landscape. To sustain signal and gradient flow through depth, mean-field and dynamical-isometry studies analyze activation choices (Schoenholz et al., 2017; Pennington et al., 2017; Hayou et al., 2019; Lesser and Chowdhury, 2026), and Qi et al. (2020) designed a shifted ReLU to improve isometry. To counter shifts associated with nonzero activation outputs, Raiko et al. (2012) centered neuron outputs and slopes to improve gradient conditioning, Eidnes and Nøkland (2018) introduced bipolar activations to facilitate deep training, and Kutsuna (2020) constrained weights to reduce shifts in downstream pre-activations. These studies alter normalization, activation shape, or weights, whereas we isolate activation bias as a single factor and investigate its effects on learning dynamics. Some studies have examined nonzero activation bias. Glorot and Bengio (2010) linked the positive mean of sigmoid activations to upper-layer saturation at initialization. Clevert et al. (2016) showed that ELUs can reduce bias shift, accelerate learning, and improve classification accuracy; Klambauer et al. (2017) designed SELUs to stabilize activation means and variances across depth; and Brock et al. (2021) controlled the growth of positive channel means in unnormalized ResNets. These studies establish consequences of activation bias in restricted settings. We examine its effects throughout training and show that reducing it can improve model performance across different architectures and tasks. 7 Conclusion We study how activation bias shapes training dynamics and the use of task-relevant information. In the controlled setting, activation bias confers an early learning advantage but is associated with hidden-state overshoot, non-smooth trajectories, a steeper loss landscape and worse late-stage performance. Our reconstruction analysis further shows that the learned model transiently uses information in the bias component. Across larger and more diverse settings, adaptive shifting retains a late-stage performance advantage. These results motivate a more complete causal account of activation bias and its explicit consideration in future activation-function design. AI Use Statement We used generative AI tools to brainstorm ideas, formulate candidate mathematical claims, draft code following explicit author instructions, and polish the writing. These tools were not used to create synthetic datasets, perform data analysis, or write mathematical proofs. The authors critically reviewed all AI-assisted ideas and text, independently inspected and verified generated code, and conducted the data analysis and proofs themselves. The authors take full responsibility for the final content of this paper. References Arpit et al. (2016) D. Arpit, Y. Zhou, B. Kota, and V. Govindaraju Normalization propagation: a parametric technique for removing internal covariate shift in deep networks. In Proceedings of the 33rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 48, p. 1168–1176. Cited by: §6. Ba et al. (2016) J. L. Ba, J. R. Kiros, and G. E. Hinton Layer normalization. In NIPS 2016 Deep Learning Symposium, Cited by: §6. Biderman et al. (2023) S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. Van Der Wal Pythia: a suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 2397–2430. Cited by: §E.2. Biswas et al. (2022) K. Biswas, S. Kumar, S. Banerjee, and A. K. Pandey Smooth maximum unit: smooth activation function for deep networks using smoothing maximum technique. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 794–803. Cited by: §6. Björck et al. (2018) N. Björck, C. P. Gomes, B. Selman, and K. Q. Weinberger Understanding batch normalization. In Advances in Neural Information Processing Systems, Vol. 31. Cited by: §1, §6. Brock et al. (2021) A. Brock, S. De, and S. L. Smith Characterizing signal propagation to close the performance gap in unnormalized ResNets. In International Conference on Learning Representations, Cited by: §1, §6. Clevert et al. (2016) D. Clevert, T. Unterthiner, and S. Hochreiter Fast and accurate deep network learning by exponential linear units (ELUs). In International Conference on Learning Representations, Cited by: §1, §6. Dosovitskiy et al. (2021) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, Cited by: §4.1. Dugas et al. (2000) C. Dugas, Y. Bengio, F. Bélisle, C. Nadeau, and R. Garcia Incorporating second-order functional knowledge for better option pricing. In Advances in Neural Information Processing Systems, Vol. 13. Cited by: §6. Eidnes and Nøkland (2018) L. H. Eidnes and A. Nøkland Shifting mean activation towards zero with bipolar activation functions. In International Conference on Learning Representations Workshop, Cited by: §6. Glorot and Bengio (2010) X. Glorot and Y. Bengio Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 9, p. 249–256. Cited by: §6. Gokaslan et al. (2020) A. Gokaslan, V. Cohen, E. Pavlick, and S. Tellex OpenWebText. Note: Zenodo datasetdoi: 10.5281/zenodo.3834942 Cited by: §4.3. Hayou et al. (2019) S. Hayou, A. Doucet, and J. Rousseau On the impact of the activation function on deep neural networks training. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 2672–2680. Cited by: §6. He et al. (2015) K. He, X. Zhang, S. Ren, and J. Sun Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE International Conference on Computer Vision, p. 1026–1034. Cited by: §6. Hendrycks and Gimpel (2016) D. Hendrycks and K. Gimpel Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415. Cited by: §6. Ioffe and Szegedy (2015) S. Ioffe and C. Szegedy Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 37, p. 448–456. Cited by: §6. Karpathy (2022) A. Karpathy nanoGPT. Note: GitHub repository External Links: Link Cited by: §4.3. Klambauer et al. (2017) G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter Self-normalizing neural networks. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §6. Kutsuna (2020) T. Kutsuna Linearly constrained weights: reducing activation shift for faster training of neural networks. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2019), Lecture Notes in Computer Science, Vol. 11907, p. 266–282. External Links: Document Cited by: §6. LeCun et al. (1998) Y. LeCun, L. Bottou, G. B. Orr, and K. Müller Efficient backprop. In Neural Networks: Tricks of the Trade, Lecture Notes in Computer Science, Vol. 1524, p. 9–50. External Links: Document Cited by: §6. Lesser and Chowdhury (2026) O. Lesser and D. Chowdhury Competing nonlinearities, criticality, and order-to-chaos transition in deep networks. arXiv preprint arXiv:2605.05294. Cited by: §1, §6. Liu et al. (2024) Z. Liu, A. Qiao, W. Neiswanger, H. Wang, B. Tan, T. Tao, J. Li, Y. Wang, S. Sun, O. Pangarkar, R. Fan, Y. Gu, V. Miller, Y. Zhuang, G. He, H. Li, F. Koto, L. Tang, N. Ranjan, Z. Shen, R. Iriondo, C. Mu, Z. Hu, M. Schulze, P. Nakov, T. Baldwin, and E. P. Xing LLM360: towards fully transparent open-source LLMs. In First Conference on Language Modeling, Cited by: §E.2. Ma et al. (2021) N. Ma, X. Zhang, M. Liu, and J. Sun Activate or not: learning customized activation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8032–8042. Cited by: §6. Maas et al. (2013) A. L. Maas, A. Y. Hannun, and A. Y. Ng Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing, Cited by: §6. Misra (2020) D. Misra Mish: a self regularized non-monotonic activation function. In Proceedings of the British Machine Vision Conference, External Links: Document Cited by: §6. Nair and Hinton (2010) V. Nair and G. E. Hinton Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, p. 807–814. Cited by: §6. Nguyen et al. (2023) E. Nguyen, M. Poli, M. Faizi, A. Thomas, M. Wornow, C. Birch-Sykes, S. Massaroli, A. Patel, C. Rabideau, Y. Bengio, S. Ermon, C. Ré, and S. Baccus HyenaDNA: long-range genomic sequence modeling at single nucleotide resolution. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Document Cited by: §E.1. OLMo Team et al. (2025) OLMo Team, P. Walsh, L. Soldaini, et al. 2 OLMo 2 furious (COLM’s version). In Second Conference on Language Modeling, Cited by: §E.1, §E.2. Peebles and Xie (2023) W. Peebles and S. Xie Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4195–4205. Cited by: §4.2. Pennington et al. (2017) J. Pennington, S. S. Schoenholz, and S. Ganguli Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §6. Qi et al. (2020) H. Qi, C. You, X. Wang, Y. Ma, and J. Malik Deep isometric learning for visual recognition. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, p. 7824–7835. Cited by: §1, §6. Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever Language models are unsupervised multitask learners. Technical report OpenAI. Cited by: §4.3. Raiko et al. (2012) T. Raiko, H. Valpola, and Y. LeCun Deep learning made easier by linear transformations in perceptrons. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 22, p. 924–932. Cited by: §6. Ramachandran et al. (2018) P. Ramachandran, B. Zoph, and Q. V. Le Searching for activation functions. In International Conference on Learning Representations Workshop, Cited by: §6. Salimans and Kingma (2016) T. Salimans and D. P. Kingma Weight normalization: a simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, Vol. 29. Cited by: §6. Santurkar et al. (2018) S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry How does batch normalization help optimization?. In Advances in Neural Information Processing Systems, Vol. 31. Cited by: §1, §6. Schoenholz et al. (2017) S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein Deep information propagation. In International Conference on Learning Representations, Cited by: §6. Shazeer (2020) N. Shazeer GLU variants improve transformer. arXiv preprint arXiv:2002.05202. Cited by: §6. Simonyan and Zisserman (2015) K. Simonyan and A. Zisserman Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, Cited by: §3.1. Yu et al. (2024) W. Yu, C. Si, P. Zhou, M. Luo, Y. Zhou, J. Feng, S. Yan, and X. Wang MetaFormer baselines for vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (2), p. 896–912. External Links: Document Cited by: §6. Zhang and Sennrich (2019) B. Zhang and R. Sennrich Root mean square layer normalization. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §6. Zhu et al. (2025) J. Zhu, X. Chen, K. He, Y. LeCun, and Z. Liu Transformers without normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14901–14911. Cited by: §4.1. Appendix A Estimating b for Different Activation Functions This appendix examines fixed, layer-shared activation shifts in the controlled model. We first derive the shift btheoryb_theory that brings the post-activation mean near zero at initialization and show that the empirical means remain close to zero during training. We then sweep b and measure the final loss. Across ReLU, GELU, Swish, Tanh, and sigmoid, the theory-predicted shifts achieve near-optimal final loss. A.1 Theory-predicted activation means For a pre-norm MLP block, let ∈ℝdx ^d be its input and =LN()u=LN(x). Let iw_i be the iith row of the up-projection linear layer. The pre-activation of neuron i is Zi=i⊤.Z_i=w_i u. (10) The weights are initialized independently of u as wij∼i.i.d.[−1d,1d].w_ij i.i.d. U\! [- 1 d, 1 d ]. (11) Using ‖22≈d\|u\|_2^2≈ d from LayerNorm gives i,[Zi]=0,i,[Zi2]=13d‖22≈13.E_w_i,x[Z_i]=0, _w_i,x[Z_i^2]= 13dE_x\|u\|_2^2≈ 13. (12) We approximate the pre-activations by Z∼(0,13).Z \! (0, 13 ). (13) The shifted activation is ϕshifted(Z,b)=ϕ(Z)−b. _shifted(Z;b)=φ(Z)-b. (14) Thus, btheory(ϕ):=[ϕ(Z)].b_theory(φ):=E[φ(Z)]. (15) ReLU. For ϕ(z)=max0,zφ(z)= \0,z\, btheory(ReLU) b_theory(ReLU) =∫−∞max0,z12π/3exp(−z22/3)z = _-∞^∞ \0,z\ 1 2π/3 \! (- z^22/3 )\,dz =32π∫0∞zexp(−3z22)z = 32π _0^∞z \! (- 3z^22 )\,dz =16π = 1 6π ≈0.230. ≈ 0.230. (16) GELU. We use the exact GELU, ϕ(z)=zΦ(z)φ(z)=z (z), where Φ and φ are the standard Gaussian cumulative distribution and density functions. btheory(GELU) b_theory(GELU) =∫−∞zΦ(z)12π/3exp(−z22/3)z = _-∞^∞z (z) 1 2π/3 \! (- z^22/3 )\,dz =16π∫−∞φ(z)exp(−3z22)z = 1 6π _-∞^∞ (z) \! (- 3z^22 )\,dz =124π≈0.115. = 1 24π≈ 0.115. (17) Swish. For ϕ(z)=z/(1+e−z)φ(z)=z/(1+e^-z), btheory(Swish) b_theory(Swish) =∫−∞z1+e−z12π/3exp(−z22/3)z = _-∞^∞ z1+e^-z 1 2π/3 \! (- z^22/3 )\,dz ≈0.077. ≈ 0.077. (18) Tanh. Since tanh is odd, btheory(tanh)=[tanh(Z)]=0.b_theory( )=E[ (Z)]=0. (19) Sigmoid. For ϕ(z)=1/(1+e−z)φ(z)=1/(1+e^-z), we have ϕ(z)+ϕ(−z)=1φ(z)+φ(-z)=1. Since Z and −Z-Z have the same distribution, btheory(sigmoid) b_theory(sigmoid) =[ϕ(Z)] =E[φ(Z)] =12[ϕ(Z)+ϕ(−Z)] = 12E[φ(Z)+φ(-Z)] =12=0.500. = 12=0.500. (20) Figure 8 shows that the predicted shifts keep the post-activation means near zero during training. Figure 8: The post-activation means of the shifted models remain near zero throughout training. A.2 Final loss as a function of the activation shift Using the same teacher–student training procedure, we sweep the shift b for each activation over multiple random seeds. Within each seed, all compared shifts use the same teacher, student initialization, and sequence of training batches. Figure 9 plots final loss against b. Across the tested activations, the theory-predicted shifts achieve near-optimal final loss. Figure 9: Across all five activations, our theory’s predicted shifts achieve near-optimal final loss. Thin curves show individual seeds, and thick curves show their mean. Appendix B Theoretical analysis of optimization landscape This appendix analyzes how activation bias affects loss landscape curvature and changes the direction of parameter updates early in training. B.1 Activation mean and landscape sharpness B.1.1 A single trainable down-projection We start with the simplest setting, where a(x)∈ℝma(x) ^m is fixed and only a down-projection layer B∈ℝdout×mB ^d_out× m is trained. Define zb(x):=a(x)−b,fB(x)=Bzb(x).z_b(x):=a(x)-b1, f_B(x)=Bz_b(x). (21) For target y(x)y(x), define ℒb(B) _b(B) =12x‖Bzb(x)−y(x)‖2, = 12E_x \|Bz_b(x)-y(x) \|^2, (22) Mb M_b :=x[zb(x)zb(x)⊤]. :=E_x [z_b(x)z_b(x) ]. (23) Proposition B.1 (Exact down-projection sharpness). The Hessian with respect to the vectorized matrix B is HB(b)=Mb⊗Idout,λB(b):=λmax(HB(b))=λmax(Mb).H_B(b)=M_b I_d_out, _B(b):= _ \! (H_B(b) )= _ (M_b). (24) Here ⊗ denotes the Kronecker product. Suppose [a]=μ,E[a]= 1, (25) so all neurons have the same mean activation μ. Then b=μb=μ globally minimizes λB(b) _B(b) over all scalar shifts. Proof. For any perturbation ΔB B, d2dt2ℒb(B+tΔB)|t=0=‖ΔBzb‖2=tr(ΔBMbΔB⊤), . d^2dt^2L_b(B+t B) |_t=0=E\| Bz_b\|^2=tr( BM_b B ), (26) which gives equation 24. Let Σ=Cov(a) =Cov(a). Then Mb=Σ+(μ−b)2⊤⪰Σ=Mμ.M_b= +(μ-b)^211 =M_μ. (27) λB(b)=λmax(Mb)≥λmax(Mμ)=λB(μ).∎ _B(b)= _ (M_b)≥ _ (M_μ)= _B(μ). When neuron-wise activation means differ, write [a]=μ¯+ε,μ¯:=1m⊤[a],ε⟂.E[a]= μ1+ , μ:= 1m1 E[a], 1. Since Mμ¯=Σ+εε⊤M_ μ= + and Mb⪰ΣM_b for every b, 0≤λmax(Mμ¯)−minbλmax(Mb)≤‖ε‖2.0≤ _ (M_ μ)- _b _ (M_b)≤\| \|^2. (28) Thus shifting by the average activation mean is within ‖ε‖2\| \|^2 of the optimal scalar shift in largest-eigenvalue curvature. The gap is small whenever ‖ε‖\| \| is small. B.1.2 Jointly training the up- and down-projections We then allow both the up-projection A and the down-projection B to be trained. At a fixed parameter state, let a(x)=ϕ(Ax),zb(x)=a(x)−b,fA,B(x)=Bzb(x),a(x)=φ(Ax), z_b(x)=a(x)-b1, f_A,B(x)=Bz_b(x), (29) and assume [a]=μE[a]= 1. Let Dx=diag(ϕ′(Ax))D_x=diag(φ (Ax)). To first order, Δf(x)=BDxΔAx+ΔBzb(x). f(x)=BD_x Ax+ Bz_b(x). (30) Let J=[JAJB]J=[J_A\;J_B] be the output Jacobian with respect to θ=(vec(A),vec(B))θ=(vec(A),vec(B)). Under column-wise vectorization, equation 30 gives JA(x)=x⊤⊗(BDx),JB(x)=zb(x)⊤⊗Idout.J_A(x)=x (BD_x), J_B(x)=z_b(x) I_d_out. (31) For squared loss, the Gauss–Newton (GN) matrix is Gb=[J(x)⊤J(x)]=(GAAGAB(b)GBA(b)GBB(b)),G_b=E[J(x) J(x)]= pmatrixG_A&G_AB(b)\\ G_BA(b)&G_B(b) pmatrix, (32) where GAA G_A =[(xx⊤)⊗(DxB⊤BDx)], =E\! [(x ) (D_xB BD_x) ], (33) GAB(b) G_AB(b) =[(xzb(x)⊤)⊗(DxB⊤)],GBA(b)=GAB(b)⊤, =E\! [(xz_b(x) ) (D_xB ) ], G_BA(b)=G_AB(b) , (34) GBB(b) G_B(b) =[zb(x)zb(x)⊤]⊗Idout=Mb⊗Idout. =E\! [z_b(x)z_b(x) ] I_d_out=M_b I_d_out. (35) Thus GAAG_A is independent of b, while the mixed blocks can depend on b. Using λB(b) _B(b) from Proposition B.1, define λA:=λmax(GAA),λG(b):=λmax(Gb). _A:= _ (G_A), _G(b):= _ (G_b). (36) Proposition B.2 (Full-GN bound and factor-two guarantee). For every shift b, maxλA,λB(b)≤λG(b)≤λA+λB(b). \ _A, _B(b)\≤ _G(b)≤ _A+ _B(b). (37) Let b∗∈argminbλG(b)b^*∈ _b _G(b). Consequently, λG(b∗)≤λG(μ)≤2λG(b∗). _G(b^*)≤ _G(μ)≤ 2 _G(b^*). (38) Proof. Restricting a parameter direction to the A or B block gives the lower bound. For any perturbation satisfying ‖ΔA‖F2+‖ΔB‖F2=1\| A\|_F^2+\| B\|_F^2=1, the triangle inequality and Cauchy–Schwarz give ‖JAvec(ΔA)+JBvec(ΔB)‖2 \|J_Avec( A)+J_Bvec( B)\|^2 ≤(λA‖ΔA‖F+λB(b)‖ΔB‖F)2 ≤ ( _A\| A\|_F+ _B(b)\| B\|_F )^2 (39) ≤λA+λB(b), ≤ _A+ _B(b), (40) which proves equation 37. Since b=μb=μ minimizes λB(b) _B(b), λG(b∗)≤λG(μ)≤λA+λB(μ)≤λA+λB(b∗)≤2λG(b∗). _G(b^*)≤ _G(μ)≤ _A+ _B(μ)≤ _A+ _B(b^*)≤ 2 _G(b^*). (41) ∎ We test these bounds in the one-block model across ten seeds. At each training checkpoint, we hold A and B fixed, compute the average activation mean μ¯=x,i[ai(x)] μ=E_x,i[a_i(x)], and sweep b to compare λG(b) _G(b) with the bounds in equation 37. The minimum over the tested shifts estimates λG(b∗) _G(b^*). In Figure 10, the bounds closely track λG(b) _G(b) across shifts (left), while λG(μ¯) _G( μ) remains close to the empirical minimum throughout training (right). Figure 10: Empirical evaluation of Proposition B.2. Left: The theoretical lower and upper bounds closely track the measured leading Gauss–Newton eigenvalue λG(b) _G(b). Right: The value at the average-mean shift, λG(μ¯) _G( μ), remains close to the empirical minimum, λG(b∗) _G(b^*), throughout training. Thus, removing the activation bias yields nearly optimal Gauss–Newton sharpness in this model, with a factor-two upper bound relative to the optimum. B.1.3 From Gauss–Newton to the true Hessian Let r(x)=fθ(x)−y(x)r(x)=f_θ(x)-y(x) and g(t)=fθ+tΔθ(x)g(t)=f_θ+t θ(x). For squared loss, Δθ⊤HΔθ=‖g′(0)‖2⏟Δθ⊤GΔθ+⟨r(x),g′(0)⟩. θ H θ= E\|g (0)\|^2_ θ G θ+E r(x),g (0) . (42) For ReLU, consider a local parameter region in which the activation gates Dx=diag(Ax>0)D_x=diag(1_Ax>0) remain fixed for each input x. Then fA,B(x)=B(DxAx−b),f_A,B(x)=B(D_xAx-b1), (43) and along a joint direction (ΔA,ΔB)( A, B), g′(0)=2ΔBDxΔAx.g (0)=2 BD_x Ax. (44) The diagonal blocks satisfy HAA=GAAH_A=G_A and HBB=GBBH_B=G_B. For a unit joint direction, |2⟨r,ΔBDxΔAx⟩|≤[‖r‖‖x‖]=:Rb, |2E r, BD_x Ax | [\|r\|\,\|x\|]=:R_b, (45) so |λmax(Hb)−λmax(Gb)|≤‖Hb−Gb‖2≤Rb. | _ (H_b)- _ (G_b) |≤\|H_b-G_b\|_2≤ R_b. (46) In particular, ℒb=0⟹Hb=Gb.L_b=0 H_b=G_b. Figure 11 shows that the largest eigenvalues of the Hessian and Gauss–Newton matrix at b=μ¯b= μ remain close for most of training. Figure 11: After 100 updates, the largest eigenvalues of the exact Hessian and its Gauss–Newton approximation remain close at b=μ¯b= μ. B.1.4 Multiple down-projections with fixed activations We finally consider multiple trainable down-projections with fixed activations: f(x)=∑ℓ=1LBℓ(aℓ(x)−bℓ),[aℓ]=μℓ.f(x)= _ =1^LB_ (a_ (x)-b_ 1 ), [a_ ]= _ 1. (47) Define Σstack:=Cov((a1(x)aL(x))),v:=((μ1−b1)(μL−bL)). _stack:=Cov\! ( pmatrixa_1(x)\\ \\ a_L(x) pmatrix ), v_b:= pmatrix( _1-b_1)1\\ \\ ( _L-b_L)1 pmatrix. (48) Then Hjoint()=Gjoint()=(Σstack+vv⊤)⊗Idout.H_joint(b)=G_joint(b)= ( _stack+v_bv_b ) I_d_out. (49) Thus, setting bℓ=μℓb_ = _ for every layer minimizes the joint sharpness. B.2 Activation mean and parameter gradients We next study the different update directions of unshifted and shifted models during the early stage of training. We analyze a simplified setting in which only a down-projection layer is trainable. Suppose [a(x)]=μ,Cov(a(x))=qI,y(x)=B⋆a(x).E[a(x)]= 1, (a(x))=qI, y(x)=B_ a(x). (50) The unshifted model uses a(x)a(x), while the shifted model uses zμ(x)=a(x)−μ.z_μ(x)=a(x)- 1. (51) Both models start from BinitB_init and use gradient descent with learning rate η. Let Bk−1(0)B_k-1^(0) and Bk−1(μ)B_k-1^(μ) denote their parameters before the kkth update, and define g0,k:=∇Bℒ0(Bk−1(0)),gμ,k:=∇Bℒμ(Bk−1(μ)).g_0,k:= _BL_0\! (B_k-1^(0) ), g_μ,k:= _BL_μ\! (B_k-1^(μ) ). (52) Let e=m.e= 1 m. (53) For the unshifted and shifted models, M0=qI+mμ2ee⊤,Mμ=qI.M_0=qI+mμ^2e , M_μ=qI. (54) Along e, the eigenvalue of M0M_0 is κ=(q+mμ2)/qκ=(q+mμ^2)/q times that of MμM_μ. The loss gradients satisfy ∇Bℒ0(B)=(B−B⋆)M0,∇Bℒμ(B)=(B−B⋆)Mμ. _BL_0(B)=(B-B_ )M_0, _BL_μ(B)=(B-B_ )M_μ. (55) Let Δ=Binit−B⋆=Δ∥+Δ⟂, =B_init-B_ = _ + _ , (56) where Δ∥=Δee⊤,Δ⟂=Δ(I−ee⊤). _ = e , _ = (I-e ). (57) The gradients used for the kkth update satisfy g0,k g_0,k =q(1−ηq)k−1(Δ⟂+tkΔ∥), =q(1-η q)^k-1 ( _ +t_k _ ), (58) gμ,k g_μ,k =q(1−ηq)k−1Δ, =q(1-η q)^k-1 , where tk=κ(1−ηκq1−ηq)k−1.t_k=κ ( 1-ηκ q1-η q )^k-1. (59) Suppose ‖Δ∥‖F2=1m‖Δ‖F2.\| _ \|_F^2= 1m\| \|_F^2. (60) Then the angle ψk _k between the gradients satisfies cosψk=tr(g0,kgμ,k⊤)‖g0,k‖F‖gμ,k‖F=m−1+tkm(m−1+tk2). _k= tr(g_0,kg_μ,k )\|g_0,k\|_F\|g_μ,k\|_F= m-1+t_k m (m-1+t_k^2 ). (61) Using the ReLU calculation in Appendix A, μ=16π,q=[ReLU(Z)2]−μ2=16−μ2=π−16π.μ= 1 6π, q=E[ReLU(Z)^2]-μ^2= 16-μ^2= π-16π. (62) Table 2 compares our one-layer theory prediction with full-model gradient angles measured during Adam training. Table 2: Angles between the unshifted and shifted parameter gradients. The theoretical and measured angles are close early in training. k Theory angle Full-model angle 1 77.45∘77.45 76.25∘76.25 5 77.44∘77.44 75.01∘75.01 20 77.40∘77.40 69.22∘69.22 100 77.22∘77.22 46.58∘46.58 Both the one-layer calculation and the full-model measurements show that the gradients of unshifted and shifted models are nearly orthogonal at the beginning of training. Appendix C Ablation studies This section tests whether the effect of activation bias is specific to the model configuration. We first examine a joint sweep over model size, then study how the optimal shifting changes with model size, and finally vary other configurations of the controlled model. C.1 Joint sweep over dimension, width, and depth We first compare shifted and unshifted ReLU models across a joint sweep of residual-stream dimension d, MLP hidden width m, and number of blocks L. Figure 12 shows the results for L=10L=10 and L=20L=20. The improvement from reducing activation bias is generally larger at greater depth and width. Figure 12: Shifting tends to yield greater improvements as model width and depth increase. For ReLU, colors show the relative improvement (ℒunshifted−ℒshifted)/ℒunshifted(L_unshifted-L_shifted)/L_unshifted, with red indicating improvement and blue indicating deterioration. C.2 Seed-level optimal shifts across model sizes We next compare three representative controlled ReLU models: a small model (d,m,L)=(5,20,2)(d,m,L)=(5,20,2), a medium model (20,80,10)(20,80,10), and a large model (50,200,20)(50,200,20). We use 100 paired seeds for each model. For each seed, the empirically optimal shift is the value of b with the lowest final loss. Figure 13 shows the distribution of these shifts across seeds for each model size. As model size increases, the optimal shifts concentrate near the theory-predicted value, suggesting that reducing post-activation bias becomes more consistently beneficial across seeds. Figure 13: As model size increases, the empirically optimal shifts across seeds become more concentrated near the theory-predicted shift. C.3 Robustness to architectural changes and cross-entropy loss We finally test whether reducing activation bias remains effective under three changes to the controlled model. First, we replace pre-norm blocks with post-norm blocks, moving LayerNorm from the block input to the residual-sum output. This tests whether a fixed shift remains effective when the distributions entering the activations can vary more across blocks. Second, we enable trainable biases in the teacher’s and students’ linear projections to test whether learned affine offsets can compensate for activation bias. Third, we replace mean-squared error with soft-target cross-entropy to test a different training objective. All three variants use ReLU, with all other settings unchanged. Figure 14 shows that the shifted model achieves lower loss than the unshifted model in all three variants. Thus, its advantage is not limited to pre-norm blocks, bias-free projections, or mean-squared error. Figure 14: The shifted model retains its loss advantage in more general settings. C.4 Residual-stream trajectories across all blocks Figure 15 shows the complete hidden-state mean trajectories across all 20 blocks for the ReLU experiment in Figure 2. The unshifted model exhibits a pronounced transient overshoot across most of the blocks, whereas shifting the activation substantially reduces this overshoot. Figure 15: Reducing activation bias suppresses hidden-state overshoot and yields smoother hidden-state mean trajectories. Appendix D Additional VGG-Style Autoencoder Experiments This appendix provides the architecture and additional experimental results for the simplified VGG-style autoencoder introduced in Section 3.1. Figure 16 shows the architecture. Figure 16: Architecture of the autoencoder. Numbers on arrows denote channel counts. The shifted model replaces each ReLU with ReLU(z)−bReLU(z)-b. We also track the reconstruction loss throughout training. Figure 17 shows that the shifted model initially has a higher loss than the unshifted model but achieves a lower loss later. Figure 17: The shifted model initially has a higher reconstruction loss but achieves a lower loss later in training. The plotted residual is Lshifted−LunshiftedL_shifted-L_unshifted. Figure 18 provides additional examples of the component-wise reconstructions analyzed in Section 3.2. Across these images, the unshifted model’s activation bias component retains recognizable image structure, whereas the corresponding reconstructions from the shifted model contain little discernible content. Figure 18: Across different images, reconstructions from the unshifted model’s activation bias component first approach and then diverge from the full reconstructions, whereas those from the shifted model contain little recognizable structure. Appendix E Additional Scale-Up Experiments This appendix provides additional scale-up results on performance, hidden-state dynamics, and activation distributions. E.1 Performance on Additional Scale-Up Models We test whether the gains reported in Section 4 extend across additional model sizes, tasks, and metrics. We evaluate ViT-L on ImageNet image classification, OLMo-2-1B (OLMo Team et al., 2025) on language pretraining, and HyenaDNA (Nguyen et al., 2023) on DNA base-sequence prediction. We also evaluate DiT-B/2 on image generation using FID. Figure 19 reports the results. With adaptive shifting, ViT-L achieves lower test loss and higher accuracy despite a higher training loss; DiT achieves lower FID at every evaluated checkpoint, while OLMo-2-1B and HyenaDNA achieve better performance. Thus, the benefit extends beyond the models and metrics highlighted in the main text. Figure 19: The performance improvements from shifting extend to additional scale-up models and tasks. Residuals are computed as ΔM=Mshifted−Munshifted M=M_shifted-M_unshifted, where M denotes loss, FID, or accuracy. E.2 Further Examples of Hidden-State Overshoot We examine whether the hidden-state overshoot observed in Section 4.1 also appears in other models. We evaluate publicly released training checkpoints from Pythia-410M (Biderman et al., 2023), OLMo-2-1B (OLMo Team et al., 2025), and Amber-7B (Liu et al., 2024) on a fixed set of inputs and compute hidden-state means as in Section 2.2. Figure 20 presents representative trajectories from these models. Across them, representative blocks exhibit transient overshoots that subsequently recede or reverse, although their timing and direction vary. Figure 20: Hidden-state mean trajectories exhibit overshoot across different scale-up models. E.3 Activation-Magnitude Distributions in OLMo-2-1B Our analysis so far has focused on activation bias, but the mean alone cannot show how shifting affects the broader distribution of activation magnitudes. We therefore compare the absolute-value distributions of post-activations and residual-stream states. Figure 21 shows that, with shifting, the post-activation distributions change relatively little over training. In the unshifted model, they develop an increasingly pronounced concentration near zero in deeper blocks. The residual-stream distributions, however, remain broadly similar across the two models and more closely resemble the shifted model’s post-activation distributions. This contrast suggests that the unshifted model must map a more sharply concentrated post-activation distribution to a comparatively smooth residual-stream distribution. Such a larger distributional adjustment may make learning less direct, offering a possible explanation for the improved dynamics observed when activation bias is reduced. Figure 21: In OLMo-2-1B, the shifted model maintains more stable post-activation magnitude distributions over training, with less accumulation of near-zero values. Moreover, its post-activation and residual-stream magnitude distributions closely resemble each other, whereas those of the unshifted model differ substantially.