Paper deep dive
Building a Neural Network from Scratch: Implementation, Evaluation, and Optimization
Yuanzhe Jia
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 7/21/2026, 3:52:02 AM
Summary
This paper presents a self-contained neural network framework implemented from scratch without relying on automatic differentiation or pre-built deep learning libraries. It details the implementation of core components including multi-layer architectures, Kaiming initialization, weight decay, batch normalization, dropout, label smoothing, various activation functions (ReLU, Tanh, GELU), and optimizers (Momentum SGD, Adam). The framework is evaluated on the CIFAR-10 dataset for multi-class classification, demonstrating robust performance, numerical stability, and generalization. The study serves as a pedagogical tool to bridge the gap between theoretical understanding and practical implementation of deep learning mechanics.
Entities (16)
Relation Signals (15)
Yuanzhe Jia → affiliatedwith → University of Sydney
confidence 99% · 11institutetext: University of Sydney, Australia 11email: yjia5612@uni.sydney.edu.au
Neural Network Framework → evaluatedon → CIFAR-10
confidence 98% · Experiments were organized... The dataset (CIFAR-10 [5]) comprises 50,000 training instances...
Yuanzhe Jia → authorof → Neural Network Framework
confidence 95% · Building a Neural Network from Scratch... Yuanzhe Jia... The paper presents a self-contained neural network framework
Neural Network Framework → implements → Dropout
confidence 95% · The implementation encompasses all essential components, including... dropout
Neural Network Framework → implements → Label Smoothing
confidence 95% · The implementation encompasses all essential components, including... label smoothing
Neural Network Framework → implements → ReLU
confidence 95% · The implementation encompasses all essential components, including... ReLU... activation functions
Neural Network Framework → implements → Tanh
confidence 95% · The implementation encompasses all essential components, including... Tanh... activation functions
Neural Network Framework → implements →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The widespread adoption of high-level deep learning libraries, while accelerating model development, has increasingly abstracted away the internal mechanics of neural networks, creating a gap between practical usage and fundamental understanding. To address this, the paper presents a self-contained neural network framework implemented entirely from scratch -- without relying on automatic differentiation or pre-built deep learning modules. The implementation encompasses all essential components, including multi-layer architectures, diverse activation functions, regularization techniques, and state-of-the-art optimizers. Beyond serving as a pedagogical instrument that demystifies forward/backward propagation, gradient dynamics, and optimization landscapes, the framework demonstrates robust performance when applied to a multi-class classification task, successfully validating its correctness, numerical stability, and generalization across varied configurations. The extensible design and clean modularity further position it as a reliable baseline for educational purposes and future research exploration.
Tags
Links
- Source: https://arxiv.org/abs/2607.16682v1
- Canonical: https://arxiv.org/abs/2607.16682v1
Trouble viewing inline? Open PDF directly →
Full Text
27,691 characters extracted from source content.
Expand or collapse full text
11institutetext: University of Sydney, Australia 11email: yjia5612@uni.sydney.edu.au Building a Neural Network from Scratch: Implementation, Evaluation, and Optimization Yuanzhe Jia Abstract The widespread adoption of high-level deep learning libraries, while accelerating model development, has increasingly abstracted away the internal mechanics of neural networks, creating a gap between practical usage and fundamental understanding. To address this, the paper presents a self-contained neural network framework implemented entirely from scratch—without relying on automatic differentiation or pre-built deep learning modules. The implementation encompasses all essential components, including multi-layer architectures, diverse activation functions, regularization techniques, and state-of-the-art optimizers. Beyond serving as a pedagogical instrument that demystifies forward/backward propagation, gradient dynamics, and optimization landscapes, the framework demonstrates robust performance when applied to a multi-class classification task, successfully validating its correctness, numerical stability, and generalization across varied configurations. The extensible design and clean modularity further position it as a reliable baseline for educational purposes and future research exploration. The relevant code is publicly available on GitHub111https://github.com/yuanzhe-jia/neural-network. 1 Introduction The primary objective of this study is to construct a neural network architecture from fundamental principles and apply it to a multi-class classification problem on a provided dataset. This research pursues two interconnected sub-objectives: (1) implementing a diverse set of neural network modules—including multiple hidden layers, Kaiming initialization [1], weight decay [6], batch normalization [3], dropout [7], label smoothing [8], ReLU, Tanh, and GELU [2] activation functions, softmax with cross-entropy loss, momentum-enhanced SGD, the Adam optimization [4], and mini-batch training—entirely without external deep learning frameworks or automatic gradient computation tools; and (2) systematically training and comparing multiple model configurations on the target dataset, subsequently identifying the optimal architecture through rigorous empirical analysis. This investigation holds substantial pedagogical and practical value. Contemporary deep learning frameworks such as TensorFlow and PyTorch enable researchers to implement complex models with minimal code, often at the expense of understanding the underlying computational processes. By reconstructing neural network modules from first principles, we gain granular insight into forward propagation, backpropagation, optimization dynamics, and regularization mechanics that remain opaque when relying on high-level APIs. Furthermore, this foundational understanding is indispensable for proposing novel architectures, diagnosing training failures, and performing targeted fine-tuning of existing models. This paper thus serves to bridge the gap between theoretical knowledge and practical implementation competence, addressing a critical need in the education of deep learning practitioners. 2 Methodology The implemented modules encompass the fundamental components of modern neural networks. Each module is described below with its mathematical formulation and functional rationale. 2.1 Multiple Hidden Layers y1=w1Tx+b1,a1=σ1(y1)y_1=w_1^Tx+b_1,a_1= _1(y_1) (1) y2=w2Ta1+b2,a2=σ2(y2)y_2=w_2^Ta_1+b_2,a_2= _2(y_2) (2) y^=w3Ta2+b3 y=w_3^Ta_2+b_3 (3) Multiple hidden layers are the basic structure in neural networks. The structure helps neural networks satisfy most of data transformation operations and fit various complex problems. Taking a simple neural network with two hidden layers as an example, w1w_1 and b1b_1 are the parameters of the first hidden layer, they scale and shift the original input x for many times to obtain y1y_1. The number of scaling and shifting depends on the output size (i.e., number of neurons) of this layer. For example, if the output size is 10, an input data will be transformed by 10 times at this layer. σ1 _1 is the activation function of the first hidden layer, after processing y1y_1 by σ1 _1, a1a_1 is obtained as the output of the first layer. Similar to the above, w2w_2 and b2b_2 are the parameters of the second hidden layer, σ2 _2 is the activation function of this layer, and a2a_2 is regarded as the output of the layer. w3w_3 and b3b_3 are the parameters of the output layer, they scale and shift the input data a2a_2 to obtain y y, which is the prediction results of the neural network. Basically, the multi-layer architecture of a neural network contains two main characteristics. The first is to perform feature extraction. The neurons in each layer are equivalent to feature extractors, implementing different transformations on the input data. The second is to introduce nonlinear properties through activation functions, thereby improving the generalization ability of neural networks to nonlinear problems. As is mentioned above, multiple hidden layers are the basic structure of neural networks, many modules are extended on this basis. Details about those modules will be presented below. 2.2 Kaiming Initialization wi∼(−6ni,6ni)w_i (- 6n_i, 6n_i ) (4) bi=0b_i=0 (5) The initial weights in neural networks should not be too large or too small, larger weights will lead to exploding gradient problem, while smaller weights will also bring vanishing gradient problem. The initial weights cannot even be zero, as this would cause all neurons to act exactly the same. Kaiming initialization is an effective way to solve these problems. In the above formulas, ReLU is considered as the default activation function, nin_i is the number of inputs in layer i, wiw_i and bib_i represent the weights and biases of layer i, respectively. By initializing the weights and biases of each layer according to the above formula, the mean of the activations will be zero, and the variance of the activations will be kept the same across layers. That is to say, the output of each layer follows the same distribution, which improves the learning efficiency of the neural network. If other activation functions are used, the upper and lower bounds of the uniform distribution should also be reset. 2.3 Weight Decay J^(θt)=J(θt)+λ2‖θt‖22 J( _t)=J( _t)+ λ2\| _t\|_2^2 (6) θt+1=θt−η∇θtJ^(θt) _t+1= _t-η _ _t J( _t) (7) Weight decay is one of effective ways to prevent overfitting. It can be regarded as the coefficient λ placed in front of the L2 regularization term, which is a penalty on the sum of squares of parameters. The L2 regularization term represents the model complexity, so the role of weight decay is to adjust the impact of model complexity on the loss function. If the weight decay is large, the loss of a complex model will also be large. As is well-known, the optimization purpose of neural networks is to minimize the loss function J(θt)J( _t). If weight decay λ is greater than zero, neural networks will minimize a new loss function J^(θt) J( _t), which is equivalent to iterate in the direction where parameters are not too complex, so as to mitigate the overfitting problem. In the above formulas, η is the learning rate, it is defined as the update step size of the optimizer. θt _t represents weights and biases at the time step t, it will be updated by the overall update vector η∇θtJ^(θt)η _ _t J( _t). Independent of the above theory, engineers found that an improved model would be obtained by directly decaying the updated parameters at each time step. In this paper, the engineer’s version of weight decay is adopted. 2.4 Batch Normalization μB←1m∑i=1mxi _B← 1m _i=1^mx_i (8) σB2←1m∑i=1m(xi−μB)2 _B^2← 1m _i=1^m(x_i- _B)^2 (9) x^i←xi−μBσB2+ϵ x_i← x_i- _B _B^2+ε (10) yi←γx^i+β≡BNγ,β(xi)y_i←γ x_i+β _γ,β(x_i) (11) Batch normalization is a normalization layer commonly used in neural networks, since it can reduce the covariance shift, as well as effects of exploding gradients and vanishing gradients. Batch normalization focuses on each feature over all the training data in the mini-batch. Specifically, it normalizes input data xix_i using the mean μB _B and variance σB2 _B^2 of the mini-batch and introduces two learnable parameters, γ and β, which scale and shift the normalised data x^i x_i respectively. The hyper-parameter ϵε is a very small value used to prevent the denominator from being zero. Finally, the processed data yiy_i is the output of the batch normalization layer. Batch normalization is sensitive to batch size m. When the batch size becomes smaller, the error increases rapidly, which is caused by inaccurate batch statistical estimation. In addition, since batch normalization is dependent on the batch size, it cannot be applied at test time the same way as the training time. Instead, during test time, batch normalization utilizes moving average and variance to perform inference. 2.5 Dropout rj(l)∼Bernoulli(p)r_j^(l) (p) (12) x^(l)=r(l)∗x(l) x^(l)=r^(l)*x^(l) (13) yi(l+1)=wi(l+1)x^(l)+bi(l+1)y_i^(l+1)=w_i^(l+1) x^(l)+b_i^(l+1) (14) xi(l+1)=f(yi(l+1))x_i^(l+1)=f(y_i^(l+1)) (15) Dropout is a simple but effective technique for mitigating the overfitting problem. When training neural networks, each neuron is retained with a certain probability of p, which is usually set to 0.5. And rj(l)r_j^(l) is the probability rate for the neuron j in the layer l, it will determine if the neuron should use the input x(l)x^(l). While in test phase, all neurons are involved in prediction. The intuition of this approach is similar to ensemble learning, which is a technique intended to deal with overfitting problem by combining predictions from many different models. However in real scenarios, the inverted dropout is usually applied. To be Specific, during training time, parameters in the neural network are amplified by a factor of 1/p1/p. While in test time, the network is used as a whole and no parameter scaling will be performed. 2.6 Label Smoothing ykLs=yk(1−α)+αKy_k^Ls=y_k(1-α)+ αK (16) Label smoothing is a technique that can prevent the overfitting problem and improve the generalization ability of the model. To be specific, label smoothing is to add noise on the basis of one-hot encoding. This method is simple but effective, and has achieved remarkable results in many image classification tasks. It is obvious that one-hot encoding will make the prediction probability of the correct classification get closer and closer to 1. In other words, one-hot encoding is not soft enough, resulting in the model being too confident in its prediction. By adding noise to the one-hot labels, the label smoothing technique can alleviate the problem of the model being too arbitrary, thereby enhancing the generalization ability of the model. In the above formula, K is the number of categories. yky_k is the one-hot label for a particular category k. α is a hyper-parameter, which is usually set to 0.1. And ykLsy_k^Ls is the label of the category k processed by the label smoothing technique. 2.7 ReLU Activation Function relu(x)=max(0,x)relu(x)= (0,x) (17) The full name of ReLU is Rectified Linear Unit. It performs a threshold operation on each input element x, where values less than zero are set to zero. ReLU activation function is by far the most widely used activation function in deep learning applications, since it has two main advantages. The one is the fast calculation, because it does not involve exponentiation and division, which improves the overall calculation speed. The second is that ReLU activation function introduces sparsity in the hidden units as it compresses some input values to zero. 2.8 Tanh Activation Function tanh(x)=ex−e−xex+e−x (x)= e^x-e^-xe^x+e^-x (18) The full name of Tanh is Hyperbolic Tangent Function. It squeezes each input element x between -1 and 1. Tanh activation function has been widely adopted in the recurrent neural networks (RNN) for natural language processing (NLP) tasks. The benefit of this function is to speed up the back-propagation process due to its zero-centered nature. However, this function cannot effectively solve the vanishing gradient problem. When the input elements are too large or too small, the gradients of the function will be close to zero. 2.9 GELU Activation Function gelu(x)=0.5x(1+tanh(2π(x+0.44715x3)))gelu(x)=0.5x (1+ ( 2π(x+0.44715x^3) ) ) (19) The full name of GELU is Gaussian Error Linear Unit. Similar to ReLU activation function, GELU activation function can also preserve the input element x, or compress it to zero. But their difference is that in GELU activation function, keeping the original value or compressing it to zero depends on probability that the current input is greater than the rest of the inputs. It can be seen that when the input element is larger, it is more likely to be retained, and the smaller the input is, the more likely it is to be reset to zero. In recent years, GELU activation function has been widely used in Transformer models including BERT, GPT2, etc. 2.10 Softmax and Cross-entropy Loss ai=softmax(zi)=ezi∑j=1Kezja_i=softmax(z_i)= e^z_i _j=1^Ke^z_j (20) J=−∑i=1Kyilog(ai)J=- _i=1^Ky_i (a_i) (21) The output values of a neural network are difficult to interpret, so they need to be converted into a probability distribution between 0 and 1 with the help of the softmax function. In the above formulas, K represents the number of categories, ziz_i represents the input value of the current category i, which will be normalized by the softmax function. The output value aia_i is between 0 and 1, and the sum of softmax outputs is equal to 1. Cross-entropy is used to evaluate the loss J between the probability distribution obtained by the neural network and the true distribution. It depicts the distance between the actual output probability and the expected output probability, that is, the smaller the cross-entropy, the closer the two probability distributions are. And yiy_i represents the expected output probability of the category i, which usually needs to be converted into one-hot format. 2.11 Momentum in SGD vt=γvt−1+η∇θtJ(θt)v_t=γ v_t-1+η _ _tJ( _t) (22) θt+1=θt−vt _t+1= _t-v_t (23) Momentum is a commonly used acceleration method for (Stochastic Gradient Descent) SGD optimizer. It adds a portion of the update vector from past time steps to the current update vector, resulting in the effect of speeding up optimization and suppressing oscillations. Specifically, if the current gradient ∇θtJ(θt) _ _tJ( _t) and the momentum term vt−1v_t-1 point in the same direction, the overall update vector vtv_t will increase, which is equivalent to speeding up the optimizer. If the current gradient and the momentum term point in the opposite direction, then the overall update vector is reduced, which is equivalent to reducing oscillations. In the above formulas, γ is a momentum factor, which is usually set to 0.9. η is the learning rate, it is defined as the update step size of the optimizer. And θt _t represents weights and biases at time step t, it will be updated by the overall update vector vtv_t. 2.12 Adam Optimizer gt=∇θtJ(θt)g_t= _ _tJ( _t) (24) mt=β1mt−1+(1−β1)gtm_t= _1m_t-1+(1- _1)g_t (25) vt=β2vt−1+(1−β2)gt2v_t= _2v_t-1+(1- _2)g_t^2 (26) m^t=mt1−β1t m_t= m_t1- _1^t (27) v^t=vt1−β2t v_t= v_t1- _2^t (28) θt+1=θt−ηv^t+ϵm^t _t+1= _t- η v_t+ε m_t (29) The full name of Adam is Adaptive Moment Estimation. It computes an adaptive learning rate for each parameter in the neural network. Similar to momentum in SGD mentioned above, Adam optimizer maintains an exponentially decaying average of past gradients mtm_t. In addition, Adam optimizer also applies the exponentially decaying average of past squared gradients vtv_t. mtm_t and vtv_t are estimates of the first and second moments of the gradients gtg_t, respectively. Since mtm_t and vtv_t are almost zero during the initial time steps, they are bias-corrected to m^t m_t and v^t v_t. It can be observed that both m^t m_t and v^t v_t gradually decrease as the time step t increases, which means the role of the first moment estimate starts to weaken, but the role of the second moment estimate starts to increase. This is because, as the time step goes on, the parameters in the neural network will be closer to the optimal, and the optimizer should slow down the update pace and focus more on the adjustment of the learning rate of different parameters. In the above formulas, β1 _1 is the first moment factor, which is usually set to 0.9. β2 _2 is the second moment factor, which is usually set to 0.999. η is the learning rate, it is defined as the update step size of the optimizer. ϵε is a very small value used to prevent the denominator from being zero. And θt _t represents weights and biases at the time step t, it will be updated by the overall update vector ηv^t+ϵm^t η v_t+ε m_t. 2.13 Mini-batch Training θt+1=θt−η∇θtJ(θt,x(i:i+m)) _t+1= _t-η _ _tJ( _t,x^(i:i+m)) (30) For a larger dataset, training all the data at the same time is extremely computationally expensive, and neural networks will converge quite slowly. Mini-batch training makes the training process more efficient. This method will first randomly shuffle the overall training data and then divides the entire dataset into several sub-data sets, and the size of the sub-data is controlled by a hyper-parameter called batch size m. In each time step, the neural network only learns one batch of training data x(i:i+m)x^(i:i+m), and back-propagates based on the loss generated by this batch of data. When the neural network learns the entire training dataset once, a.k.a. one epoch has been completed, the parameters in the neural network have been updated many times. This is why mini-batch training can help the neural network converge faster. In the above formula, η is the learning rate, it is defined as the update step size of the optimizer. And θt _t represents weights and biases at the time step t, it will be updated by the overall update vector η∇θtJ(θt,x(i:i+m))η _ _tJ( _t,x^(i:i+m)). 3 Experiments and Results 3.1 Data Pre-processing The dataset (CIFAR-10 [5]) comprises 50,000 training instances and 10,000 test instances, each containing 128 distinct features and a categorical label ranging from 0 to 9, corresponding to ten mutually exclusive classes. Pre-processing involved three sequential steps. First, labels were converted to one-hot encoded vectors to facilitate cross-entropy loss computation. For training labels specifically, label smoothing was applied to enhance model generalization. Second, the training data were randomly shuffled prior to mini-batch partitioning, ensuring that each batch receives a representative distribution of classes and mitigating potential ordering biases. Third, all feature values were standardized to maintain numerical stability during gradient-based optimization. 3.2 Experimental Setup Model performance was evaluated based on three metrics: test set prediction loss (cross-entropy), test set classification accuracy, and total training time. Experiments were organized into two groups. Group 1 (No. 1–9) employed a baseline architecture with 256-64-10 neurons and one-hot encoded targets. Group 2 (No. 10–16) utilized an expanded architecture with 256-128-10 neurons and label-smoothed targets. All models shared commonalities: Kaiming initialization, softmax with cross-entropy loss, and a batch size of 128. 3.3 Hyper-parameter Tuning Figure 1: Hyper-parameter tuning The experimental trajectory (see Figure 1) reveals key insights into and model optimization: Group 1: Initial attempts with ReLU, weight decay (λ=10−5λ=10^-5), and momentum suffered from overfitting and non-convergence (No. 1). Increasing weight decay to λ=10−3λ=10^-3 and reducing the learning rate to 10−410^-4 alleviated overfitting, achieving ∼ 51% accuracy (No. 2). Dropout alone proved insufficient, capping accuracy at ∼ 45% (No. 3). Introducing batch normalization dramatically improved performance, achieving ∼ 51.5% accuracy with stable convergence (No. 4). Replacing Tanh with GELU further improved accuracy to 51.8% (No. 6). Switching to the Adam optimizer with a higher learning rate (10−310^-3) elevated test accuracy to 53.15%, though overfitting emerged (No. 7). While weight decay mitigated overfitting, it slightly reduced peak accuracy (No. 8). Early stopping at 10 epochs on the No. 7 configuration yielded 53.86% accuracy without overfitting (No. 9). Group 2: Expanding the hidden layer to 128 units and applying label smoothing provided a favorable starting point, with accuracy exceeding 54% (No. 10). Incorporating weight decay (λ=10−3λ=10^-3) further improved generalization (No. 11). Replacing GELU with ReLU accelerated training and enhanced accuracy to 54.28% (No. 12). Early stopping at 10 epochs produced the best model with 55.42% accuracy (No. 13). Ablation studies confirmed the superiority of this configuration: replacing ReLU with Tanh reduced accuracy to 49.13% (No. 14), reducing the learning rate to 10−410^-4 with extended training degraded performance (No. 15), and substituting Adam with momentum yielded inferior results (No. 16). Through systematic experimentation, we identified a best-performing configuration (No. 13) comprising a three-layer fully-connected network with two hidden layers (256 and 128 units, respectively) followed by ReLU activation, and an output layer with 10 units corresponding to the classification categories. All weights were initialized using Kaiming uniform initialization. The loss function combines softmax with cross-entropy, computed against label-smoothed targets. 3.4 Comparative Performance Analysis Table 1: Experimental setups and results No. Activation Weight Decay Dropout Batch Norm Optimizer LR Epochs Time (s) Loss Acc (%) 1 ReLU 1e-5 × × Momentum 1e-3 20 54.76 1.9146 46.29 2 ReLU 1e-3 × × Momentum 1e-4 20 57.55 1.3851 51.38 3 ReLU × 0.5 × Momentum 1e-4 40 187.10 1.5445 45.58 4 ReLU × × ✓ Momentum 1e-4 20 89.33 1.3651 51.54 5 Tanh × × ✓ Momentum 1e-4 20 117.32 1.4847 47.99 6 GELU × × ✓ Momentum 1e-4 20 238.66 1.3551 51.82 7 GELU × × ✓ Adam 1e-3 20 263.82 1.6151 51.97 8 GELU 1e-3 × ✓ Adam 1e-3 20 275.13 1.3852 52.30 9 GELU × × ✓ Adam 1e-3 10 110.78 1.3551 53.86 10 GELU × × ✓ Adam 1e-3 20 286.44 1.3753 53.40 11 GELU 1e-3 × ✓ Adam 1e-3 20 226.18 1.3054 54.01 12 ReLU 1e-3 × ✓ Adam 1e-3 20 76.95 1.2954 54.28 13 ReLU 1e-3 × ✓ Adam 1e-3 10 38.67 1.2854 55.42 14 Tanh 1e-3 × ✓ Adam 1e-3 20 95.59 1.4449 49.13 15 ReLU 1e-3 × ✓ Adam 1e-4 40 152.32 1.4651 51.17 16 ReLU 1e-3 × ✓ Momentum 1e-4 40 135.50 1.3552 52.23 Table 1 presents comprehensive results for all 16 experimental configurations. The optimal model (No. 13) achieves the lowest test loss (1.2854), the highest test accuracy (55.42%), and the fastest training time (38.67 seconds), while exhibiting no overfitting. Notable observations include: (1) the batch normalization substantially improves stability and accuracy (cf. No. 2 vs. 4); (2) Dropout, while mitigating overfitting, constrains representational capacity (No. 3); (3) ReLU consistently outperforms Tanh and GELU in this architecture (cf. No. 11–13 vs. 14); and (4) the early stopping yields superior trade-offs between accuracy and computational cost (cf. No. 7 vs. 9, and 12 vs. 13). 4 Conclusion This paper presents a comprehensive implementation of a neural network framework entirely from scratch, without relying on high-level automatic differentiation libraries, thereby bridging the critical gap between abstract model application and fundamental algorithmic understanding. The framework incorporates a wide spectrum of essential modules—including diverse activation functions, advanced optimization strategies, and robust regularization techniques—all governed by manually derived gradient updates. Its practical validity is rigorously established through deployment on a multi-class classification task, where it consistently demonstrates numerical stability, reliable convergence, and strong generalization across varied configurations. Beyond its predictive utility, this work reaffirms the pedagogical imperative of low-level implementation, equipping practitioners with the granular intuition necessary to diagnose training pathologies, design novel components, and extend existing methodologies. The extensible and modular design further positions the framework as a dependable baseline for future research, underscoring that foundational proficiency remains indispensable even in an era dominated by automated deep learning pipelines. References [1] K. He, X. Zhang, S. Ren, and J. Sun (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, p. 1026–1034. Cited by: §1. [2] D. Hendrycks and K. Gimpel (2016) Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. Cited by: §1. [3] S. Ioffe and C. Szegedy (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, p. 448–456. Cited by: §1. [4] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §1. [5] A. Krizhevsky, V. Nair, and G. Hinton (2009) Learning multiple layers of features from tiny images. Technical report University of Toronto. Cited by: §3.1. [6] A. Krogh and J. Hertz (1991) A simple weight decay can improve generalization. Advances in neural information processing systems 4. Cited by: §1. [7] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov (2014) Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15 (1), p. 1929–1958. Cited by: §1. [8] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna (2016) Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2818–2826. Cited by: §1.