Paper deep dive
Don't stop me now: Rethinking Validation Criteria for Model Parameter Selection
Andrea Apicella, Francesco Isgrò, Andrea Pollastro, Roberto Prevete
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 11:31:23 AM
Summary
This study systematically evaluates how different validation criteria (accuracy vs. loss-based metrics like cross-entropy, C-Loss, and PolyLoss) affect model parameter selection in neural classifiers, particularly under early stopping. The authors find that early stopping based on validation accuracy performs worst, consistently selecting checkpoints with lower test accuracy compared to loss-based criteria or post-hoc selection. Loss-based validation criteria yield more stable and comparable test accuracy. However, across all methods, the selected model typically underperforms the test-optimal checkpoint found by evaluating all epochs.
Entities (9)
Relation Signals (7)
C-Loss → istypeof → Loss-based Criteria
confidence 95% · Models are trained with cross-entropy, C-Loss, or PolyLoss
PolyLoss → istypeof → Loss-based Criteria
confidence 95% · Models are trained with cross-entropy, C-Loss, or PolyLoss
Validation Accuracy → performsworsethan → Loss-based Criteria
confidence 95% · Early stopping based on validation accuracy performs worst, consistently selecting checkpoints with lower test accuracy than both loss-based early stopping and post-hoc selection.
Selected Model → underperforms → Test-Optimal Checkpoint
confidence 95% · Overall, the selected model typically achieves test-set performance statistically lower than the best performance across all epochs
Early Stopping → uses → Validation Accuracy
confidence 95% · Early stopping based on validation accuracy performs worst
Early Stopping → uses → Cross-Entropy
confidence 90% · Models are trained with cross-entropy... the model parameter selection on the validation set is made using accuracy or one of the three loss functions
Loss-based Criteria → yieldsstableperformance → test accuracy
confidence 90% · Loss-based validation criteria yield comparable and more stable test accuracy.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Despite the extensive literature on training loss functions, the evaluation of generalization on the validation set remains underexplored. In this work, we conduct a systematic empirical and statistical study of how the validation criterion used for model selection affects test performance in neural classifiers, with attention to early stopping. Using fully connected networks on standard benchmarks under $k$-fold evaluation, we compare: (i) early stopping with patience and (ii) post-hoc selection over all epochs (i.e. no early stopping). Models are trained with cross-entropy, C-Loss, or PolyLoss; the model parameter selection on the validation set is made using accuracy or one of the three loss functions, each considered independently. Three main findings emerge. (1) Early stopping based on validation accuracy performs worst, consistently selecting checkpoints with lower test accuracy than both loss-based early stopping and post-hoc selection. (2) Loss-based validation criteria yield comparable and more stable test accuracy. (3) Across datasets and folds, any single validation rule often underperforms the test-optimal checkpoint. Overall, the selected model typically achieves test-set performance statistically lower than the best performance across all epochs, regardless of the validation criterion. Our results suggest avoiding validation accuracy (in particular with early stopping) for parameter selection, favoring loss-based validation criteria.
Tags
Links
- Source: https://arxiv.org/abs/2602.22107v1
- Canonical: https://arxiv.org/abs/2602.22107v1
Trouble viewing inline? Open PDF directly →
Full Text
59,205 characters extracted from source content.
Expand or collapse full text
Don’t stop me now : Rethinking Validation Criteria for Model Parameter Selection Andrea Apicella 2† , Francesco Isgr`o 1† , Andrea Pollastro 1† , Roberto Prevete 1† 1 Department of Electrical Engineering and Information Technology, University of Naples Federico I, Via Claudio 21, Naples, 80125, Italy. 2 Department of Information Engineering, Electrical Engineering, and Applied Mathematics (DIEM), University of Salerno, Via Giovanni Paolo I, 132, Fisciano (Salerno), 84084, Italy. Contributing authors: andapicella@unisa.it; francesco.isgro@unina.it; andrea.pollastro@unina.it; roberto.prevete@unina.it; † These authors contributed equally to this work. Abstract Despite the extensive literature on training loss functions, the evaluation of gen- eralization on the validation set remains underexplored. In this work, we conduct a systematic empirical and statistical study of how the validation criterion used for model selection affects test performance in neural classifiers, with attention to early stopping. Using fully connected networks on standard benchmarks under k-fold evaluation, we compare: (i) early stopping with patience and (i) post-hoc selection over all epochs (i.e. no early stopping). Models are trained with cross- entropy, C-Loss, or PolyLoss; the model parameter selection on the validation set is made using accuracy or one of the three loss functions, each considered inde- pendently. Three main findings emerge. (1) Early stopping based on validation accuracy performs worst, consistently selecting checkpoints with lower test accu- racy than both loss-based early stopping and post-hoc selection. (2) Loss-based validation criteria yield comparable and more stable test accuracy. (3) Across datasets and folds, any single validation rule often underperforms the test-optimal checkpoint. Overall, the selected model typically achieves test-set performance statistically lower than the best performance across all epochs, regardless of the Corresponding author: Andrea Apicella, andapicella@unisa.it Preprint submitted to a journal for pubblication 1 arXiv:2602.22107v1 [cs.LG] 25 Feb 2026 validation criterion. Our results suggest avoiding validation accuracy (in partic- ular with early stopping) for parameter selection, favoring loss-based validation criteria. Keywords: Machine Learning, evaluation, data split, Deep Learning, AI 1 Introduction In neural network models, training and evaluation typically follow an iterative proce- dure — except in specific architectures such as radial basis function networks — in which each iteration corresponds to an epoch, up to a predefined maximum number of epochs. At each epoch, model parameters are updated on a designated training set through an update rule, usually driven by the gradient of a differentiable loss func- tion, such as cross-entropy in classification problems. Since each epoch yields a distinct parameter configuration, a separate validation set is commonly employed to estimate the model’s generalization capability and to identify the parameter setting expected to generalize best. After training and model parameter selection, the chosen model is finally evaluated on a fully unseen test set using a task-dependent metric, for example accuracy for balanced classification, F1 score under class imbalance, or AUROC for ranking tasks. This paradigm is standard in supervised learning and underlies most modern experimental protocols [1]. In practice, rather than running a fixed number of training epochs, it is common to adopt an early stopping procedure [2], whereby training is halted once performance on the validation set ceases to improve according to a predefined criterion. Early stopping can be viewed primarily as a computationally convenient trade-off between performance and training cost, as it avoids evaluating all possible intermediate models generated during training. From this perspective, overfitting is not prevented by pre- maturely interrupting optimization per se, but by selecting the model instance that maximizes an estimate of generalization. Importantly, the effectiveness of this selection process depends on the criterion used to assess generalization on the validation set. Despite the widespread use of validation-based selection, the criterion adopted to evaluate generalization is not uniquely specified. While the choice of the training loss is typically guided by optimization and statistical considerations, and the test metric is dictated by the deployment objective, the validation criterion occupies an intermediate role that is not clearly tied to either parameter optimization or final evaluation. As a result, different criteria are often adopted in practice, largely by convention. This ambiguity is particularly evident in classification problems. Model parameters are commonly optimized by minimizing the cross-entropy loss, which arises from maximum likelihood estimation and provides a principled surrogate for learning conditional class probabilities. Final performance, however, is often assessed using accuracy or other decision-based metrics that depend on an explicit prediction rule and directly reflect deployment-level objectives. Consequently, improvements in the training or validation loss do not necessarily translate into improvements in the evaluation metric of interest, giving rise to the well-known loss–metric mismatch [3]. 2 Motivated by this mismatch, a substantial body of work has explored alternative differentiable loss functions designed to better align optimization with accuracy- oriented objectives. Notable examples include the C-Loss proposed by [4], which targets classification error through a continuous surrogate, and PolyLoss [5], which generalizes cross-entropy by incorporating higher-order polynomial terms. Despite these developments, cross-entropy remains the dominant optimization objective in practice, and model parameter selection on the validation set is still most commonly performed using either validation loss or validation accuracy. As a consequence, different choices of performance metric on the validation set induce different orderings over the set of candidate models. In classification tasks, this raises a fundamental question: should generalization be estimated using a proba- bilistic criterion such as cross-entropy, which evaluates the quality of predicted class probabilities, or using a decision-based metric such as accuracy, which directly reflects classification performance? These criteria correspond to distinct notions of risk and need not agree in practice, nor coincide with the model that maximizes test-set accu- racy. This choice becomes even more consequential under early stopping: the monitored validation metric not only ranks checkpoints but also determines when training halts and which model parameters are actually chosen. As a consequence, misalignment between the monitored metric and the task objective can therefore terminate training prematurely around a suboptimal local minimum and lock in an inferior model. While it is well understood that generalization can be improved through explicit regularization techniques—such as weight decay, data augmentation, or dropout—as well as through implicit mechanisms including early stopping itself [6, 7], our analysis addresses a distinct but related question. Rather than modifying the learning process to induce better generalization, we examine how different validation criteria estimate generalization for the purpose of model parameter selection. Motivated by these considerations, this work investigates the practical and statis- tical implications of using validation cross-entropy versus validation accuracy, as well as alternative accuracy-aligned loss functions, as criteria for selecting models aimed at maximizing test-set accuracy. Through a systematic empirical analysis on standard supervised benchmarks under a k-fold cross-validation protocol, we assess the extent to which different validation criteria lead to statistically meaningful differences in test performance. We consider cross-entropy, C-Loss [4], and PolyLoss [5] as optimization objectives on the training set. On the validation set, generalization is evaluated using all corre- sponding losses as well as accuracy, and model parameter selection is performed both within a patience-based early-stopping scheme and by selecting the best-performing model across all training epochs. To explore different generalization regimes under con- trolled conditions, and following the theoretical insights of Advani et al. [8], we employ fully connected neural networks with a single hidden layer. This controlled architec- tural setting allows us to study model selection behavior while limiting confounding factors introduced by depth and complex optimization dynamics. Further details are provided in Section 4.2. Experiments were conducted on multiple benchmark datasets from the UCI Machine Learning Repository [9]. 3 From our experiments, the following main findings emerge: 1) When accuracy is used as the criterion to evaluate generalization on the validation set, we consistently observe the lowest test-set performance relative to the best achievable accuracy, regard- less of the loss function used during training. This effect is particularly pronounced under early stopping, where accuracy-based validation leads to a poorer alignment with test-optimal performance compared to loss-based criteria, highlighting the insta- bility of accuracy-based as a stopping criteria. 2) In contrast, when C-Loss, PolyLoss, or standard cross-entropy are used as validation criteria, the resulting test-set per- formance is comparable across methods, and remains largely independent of the loss function employed during training. 3) Overall, irrespective of the validation criterion, statistical testing indicates that the selected model achieves significantly lower test performance than the test-optimal model in the majority of cases. In summary, this work makes the following contributions: (i) a systematic and statistically grounded experimental analysis of how different validation criteria—including cross-entropy, C-Loss, PolyLoss, and accuracy—affect model selection and test-set generalization; (i) a quantitative assessment of the loss–metric mismatch in validation-based model parameter selection with and without early stopping; (i) practical implications for selecting validation criteria in accuracy-based classification tasks. The remainder of the paper is organized as follows. Section 3 introduces notation, the evaluation cri- teria and experimental protocol; Section 4 presents datasets, models, and evaluation procedures; Section 5 reports the results discussing implications and limitations; and Section 6 concludes the work with final remarks. 2 Related Work In supervised learning, models are trained on labeled data belonging to a given task, with the aim to achieve high values of a task performance measure [10]. However, in several tasks the target metric is often non-differentiable (e.g., accuracy, F1) or yields flat/unstable gradients for gradient-based optimization. Consequently, training relies on differentiable surrogate losses (e.g., cross-entropy) that act as proxies for the task metric. However, minimizing the training loss does not guarantee improvements in the deployment evaluation metric (loss–metric mismatch, [3]). Motivated by this gap, several works design losses that more closely reflect the task objectives: for example, the C-loss based on cross-correntropy as a surrogate to the 0-1 risk [11, 12], or prob- abilistic performance indices that jointly reward correctness, high probability for the true class, and low probability for the others [13], or score-oriented losses that target confusion-matrix summaries [14, 15]. Other works introduced task- and data-adaptive loss functions (e.g., PolyLoss [5]), defining parametric families in which standard objectives, such as cross-entropy, arise as special cases. Regardless of the training objective, generalization is ultimately assessed on held-out data (validation/test). While the test set is usually evaluated using the effective task-specific metric, the criterion used on the validation set can suffer from the same loss-metric mismatch: selecting checkpoints, i.e., the epoch corre- sponding to the model parameters to be selected, by a surrogate such as cross-entropy may fail to identify the model that maximizes the task-specic metric (e.g., accuracy). 4 This observation underlies methods that explicitly couple training objectives with val- idation feedback [3] and motivates a careful choice of validation criteria for model selection. Furthermore, when models are evaluated iteratively on a validation set, it is com- mon to use early stopping to truncate optimization before convergence [2]. It was shown that stopping early can yield solutions comparable to those of smaller, optimally sized models [16], and consistency results are available under specific assumptions [17]. The benefit of early stopping depends on the loss and the geometry of the opti- mization landscape: studies of loss surfaces and representation dynamics highlight plateaus, saddle points, overconfidence, and how regularization shapes hidden-layer encodings [18–21]. Classical analyses investigated overtraining dynamics for linear networks under quadratic loss and characterized validation-based stopping both geo- metrically and in time [22–24], while statistical views related early stopping to explicit regularization [6, 7]. It is interesting to notice that optimal–stopping effects appear beyond artificial neu- ral networks, notably in boosting methods [25–28], and SVMs [29], which helps explain the widespread use of early stopping. More in general, stopping rules can be applied either on training data or on a held-out validation set [30]. Training-monitored criteria include, for example, a log-sensitivity index for rare outcomes [31] and rules driven by training-loss trajectories [32]; protocol-centric choices around how the hold-out split is constructed have also been explored [33]. However, some early procedures relied solely on training-set criteria or repeatedly re-sampled “validation” from the train- ing pool [34, 35], practices that can bias selection and inflate performance estimates [36]. Validation-monitored rules span comparative studies and benchmarks of families and combinations [37–39], as well as practical heuristics such as fixed validation-error thresholds [40] or marginal-improvement criteria [41]. In particular, PACMAN [42] provides generalization bounds that explicitly account for the discrepancy between cross-entropy and accuracy, while other works address the loss–metric mismatch through adaptive loss design [3] or empirical analyses of generalization behavior [43]. However, these approaches do not directly examine the implications of this mismatch for validation-based model selection. In contrast, our work focuses on the statistical effectiveness of model selection procedures driven by validation criteria, explicitly comparing validation-selected models against the test- optimal model under controlled experimental settings. In summary, prior work highlights three themes that motivate our study: (i) models are trained with surrogate losses that may not align with task metrics; (i) validation criteria inherit this mismatch and thus critically determine which check- point is selected; and (i) early stopping is often used to avoid running all epochs, so the validation signal effectively chooses the model instance among the per-epoch checkpoints—making the choice of validation metric especially important. Our study addresses these themes by comparing validation accuracy with three loss-based val- idation criteria (cross-entropy, C-loss, and PolyLoss) within a unified experimental protocol. 5 3 Method 3.1 Notation In this work, we adopt the following notation. In supervised machine learning, a dataset D ∈ D consists of N input–label pairs D =(x (i) ,y (i) ) N i=1 , where x (i) denotes an input instance and y (i) the corresponding ground-truth value. The set D denotes the collection of all possible datasets for the task under consideration. We denote by Train, Val, Test ∈ D the training, validation, and test sets, respectively. We focus on a classification setting in which each input x (i) is assigned to one of K mutually exclusive classes 1, 2,...,K. For simplicity and without loss of generality, labels are treated as one-dimensional discrete values, i.e., y (i) ∈1, 2,...,K. Given a model M (θ) with parameters θ and a dataset D ∈ D, let ℓ : D×1,...,E→R and a : D×1,...,E→ [0, 1] denote the loss and accuracy functions of the model at iteration e of a training proce- dure composed of E ∈N epochs. That is, ℓ(D,e) and a(D,e) represent, respectively, the loss and the accuracy computed on dataset D at epoch e. For a given dataset D ∈ D, define the optimal loss and accuracy values as L ⋆ D = min 1≤e≤E ℓ(D,e), A ⋆ D = max 1≤e≤E a(D,e). We further define the corresponding optimal epochs as e ⋆ ℓ,D = arg min 1≤e≤E ℓ(D,e), e ⋆ a,D = arg max 1≤e≤E a(D,e), i.e., the epochs achieving the minimum loss and maximum accuracy, respectively (see Figure 1). 3.2 Post-hoc Checkpoint Selection versus Early Stopping We can distinguish between two validation-driven protocols that are often not clearly distinguished in the literature: (i) during training, a validation-based criterion is mon- itored and, once it fails, training is halted and the best checkpoint seen so far is retained. This is usually known as early stopping ; (i) training proceeds for a fixed number of epochs, after which the checkpoint with the best validation score among all saved models is selected. Here we refer to this as post-hoc checkpoint selection. Within 6 e ⋆ ℓ,D e ⋆ a,D E 0.5 1 1.5 2 (e ⋆ ℓ,D , L ⋆ D ) (e ⋆ a,D , A ⋆ D ) epoch e Score ℓ(D,e) a(D,e) Fig. 1: An example of loss ℓ(D,e) and accuracy a(D,e) across E epochs on a dataset D. Vertical dashed lines mark the epochs achieving the validation-loss minimum e ⋆ ℓ,D and the validation-accuracy maximum e ⋆ a,D ; horizontal dotted lines indicate the cor- responding values L ⋆ D = min e ℓ(D,e) (blue) and A ⋆ D = max e a(D,e) (orange). an early stopping protocol, training is terminated according to a predefined empirical criterion. A commonly adopted strategy is early stopping with patience T , whereby training halts at the first epoch such that no improvement in the validation loss has been observed for T consecutive epochs. Formally, this condition is expressed as ∃ˆe ℓ,V al :∀h∈1, 2,...,T, ℓ(V al, ˆe ℓ,V al + h)≥ ℓ(V al, ˆe ℓ,V al ). The selected model corresponds to the model with loss ˆ L V al = ℓ(V al, ˆe ℓ,V al ). Instead, in post-hoc checkpoint selection the training proceeds for all the fixed E epochs and the model is selected retrospectively as the one at iteration e ⋆ ℓ,V al = arg min 1≤e≤E ℓ(V al,e). Figure2 depicts both procedures, i.e., post-hoc checkpoint selection and early stopping—highlighting. Note that when T = E, i.e., when the patience parameter equals the total number of epochs, early stopping with patience T results in no early termination and is therefore equivalent to post-hoc checkpoint selection. 3.3 Statistical Comparison of Model Selection Criteria To assess the effect of different model selection criteria on generalization performance, we performed a systematic evaluation comparing the test accuracy A test of models selected in both early stopping and post-hoc protocols, against the best achievable test accuracy observed throughout training. Our empirical analysis focused on supervised 7 ˆe ℓ,V al ˆe ℓ,V al +T e ⋆ ℓ,V al E 0.4 0.6 0.8 early-stopping selection post-hoc checkpoint selection T epochs epoch e ℓ ( V al,e ) ℓ(V al,e) Fig. 2: An example comparing early stopping with patience T and post-hoc checkpoint selection on the validation loss ℓ(Val,e). The orange dashed line marks the performance value returned by early stopping (best-so-far at ˆe ℓ,Val , with training halted at ˆe ℓ,Val + T ), whereas the blue dashed line marks the best validation performance value e ⋆ ℓ,Val identified retrospectively. It is evident that the early-stopped checkpoint need not be the best-performing model: it corresponds to a local minimum reached before halting, whereas post-hoc checkpoint selection identifies the global minimum over all epochs. classification tasks, where models were trained using the CE loss and evaluated in terms of accuracy. Our goal was to quantify the extent to which accuracy obtained by validation-based selection, using either the minimum validation loss L ⋆ V al or the maximum validation accuracy A ⋆ V al , deviated from the test-optimal accuracy A ⋆ Test , defined as the model instance that attained the highest test accuracy A Test across all training epochs. In other words, we want to check how much A ⋆ Test differs from a(Test,e ⋆ ℓ,V al ) and a(Test,e ⋆ a,V al ), and similarly, a(Test, ˆe ℓ,V al ) and a(Test, ˆe a,V al ) (see Figure 3 for a visual summary). 4 Experimental assessment 4.1 Datasets Experiments were conducted on multiple benchmark datasets retrieved from the UCI Machine Learning Repository [9]. The list of the datasets involved in this work is shown in Table 1. All datasets were preprocessed using a unified and dataset-agnostic pipeline in order to ensure comparability across experiments. Specifically, categorical and binary features were transformed via one-hot encoding, while numerical features were kept 8 NameInstances N. classesNameInstances N. classes Pen-Based Recognition of Handwritten Digits840910Breast Cancer Coimbra892 Page Blocks Classification54735Maternal Health Risk7763 Molecular Biology (Splice-junction Gene Sequences)24403Spambase35192 Steel Plates Faults14842Bank Marketing59992 Blood Transfusion Service Center5722Raisin6882 Website Phishing10353Letter Recognition1530026 Taiwanese Bankruptcy Prediction52172Waveform Database Generator (Version 1)38253 Statlog (Image Segmentation)17677Haberman’s Survival2342 Vertebral Column2373Statlog (German Credit Data)7652 Optical Recognition of Handwritten Digits429910Breast Cancer2122 Drug Consumption (Quantified)14427Mammographic Mass6342 Yeast113510Credit Approval4992 Contraceptive Method Choice11273Hepatitis C Virus (HCV) for Egyptian patients10594 Japanese Credit Screening4992Chess (King-Rook vs. King-Pawn)24452 Student Performance on an Entrance Examination5104Predict Students’ Dropout and Academic Success33843 Heart Disease2275SPECT Heart2042 Room Occupancy Estimation77494Differentiated Thyroid Cancer Recurrence2932 ISOLET596526Statlog (Vehicle Silhouettes)6464 Musk (Version 2)50482National Poll on Healthy Aging (NPHA)5463 Breast Cancer Wisconsin (Diagnostic)4362Hayes-Roth1013 Congressional Voting Records1772Cardiotocography162610 Cirrhosis Patient Survival Prediction2113Autism Screening Adult4662 SPECTF Heart2042Statlog (Heart)2062 Image Segmentation1607ILPD (Indian Liver Patient Dataset)4432 NHANES 2013-2014 Age Prediction Subset17432Statlog (Australian Credit Approval)5272 Ionosphere2682Polish Companies Bankruptcy152752 Table 1: Summary of the datasets used in the experimental evaluation retrieved from the UCI Machine Learning Repository [9]. For each dataset, we report the total number of instances and the number of target classes. in their original form. No dataset-specific feature engineering or optimization was performed. We emphasize that the goal of this preprocessing was not to optimize per- formance on the individual datasets to reach new state-of-the-art results, but rather to provide a simple and reproducible input representation suitable for large-scale comparative analysis. Moreover, in the analysis of the results, we explicitly account for differences in dataset complexity by ordering datasets according to increasing linear separability between classes, as estimated by the generalized discrimination value (GDV) [44]. This allows us to assess how model selection behavior varies with dataset simplicity. 4.2 Models Following the theoretical insights of [8], we employed fully connected neural net- works with a single hidden layer and ReLU activation functions, in order to preserve architectural simplicity and experimental controllability while exploring different gen- eralization regimes. Indeed, as shown in [8], generalization behavior depends critically on the ratio between the number of trainable parameters and the number of training samples. Accordingly, we define a parameter-to-sample ratio r, where r = 1 corresponds to an equal number of model parameters and samples, while values below or above 1 indicate under- and over-parameterized regimes, respectively. Thus, the use of a shallow architecture allows us to systematically explore these regimes by varying the number of hidden units so as to control the total number of trainable parameters relative to the size of the training dataset. In our experiments, we consider the values r ∈0.3, 0.5, 0.7, 0.8, 1, 1.2, 5, 10, 50. 9 Notice that we deliberately focus on shallow neural networks with a single hid- den layer, as our goal is not to achieve state-of-the-art performance, but to isolate and analyze the effect of validation criteria on model selection. Deeper architectures introduce multiple additional factors–such as hierarchical representations, layer-wise implicit regularization, and complex optimization dynamics–that can confound the interpretation of validation-based selection mechanisms. By adopting a controlled shallow setting, we are able to systematically vary the parameter-to-sample ratio and explore different generalization regimes while keep- ing architectural and optimization-related effects to a minimum. This choice enables a clearer assessment of how different validation criteria influence model selection, independently of depth-related phenomena. 4.3 Adopted losses Cross-entropy: cross-entropy loss, widely used in classification tasks, emerges naturally from the principle of maximum likelihood estimation under the assumption that the model outputs a categorical distribution over the classes. It is defined as ℓ CE =− N X i=1 K X k=1 t (i) k log Ä m (i) k ä where t is the one-hot encoded target vector t (i) ∈ 0, 1 K of the actual label y (i) , i.e. t (i) k = 1 if k = y (i) , and t (i) k = 0 otherwise, and m (i) = (m (i) 1 ,...,m (i) K ) is the class output probability distribution of the model M on the input x (i) . Leng et al. [5] introduce PolyLoss, a polynomial reparameterization of cross-entropy obtained via its Taylor expansion around the correct-class confidence. Denoting by m y (i) the predicted probability for the true class of sample i, the loss takes the form ℓ PolyLoss = ∞ X j=1 α j 1− m y (i) j , with coefficientsα j j≥1 to be tuned. In its natural (infinite) form, PolyLoss is imprac- tical and does not consistently outperform standard cross-entropy. To address this, the authors propose a simplified, first-order truncation, ℓ Poly-1 = − logm y (i) + ε 1− m y (i) , controlled by a scalar hyperparameter ε. The C-Loss [4, 12] is a surrogate for the 0–1 loss built from the correntropy [11] between true labels and model scores. Unlike cross-entropy, the C-Loss can be more robust to outliers and label noise. In binary classification problems where y (i) ∈−1, 1 and single output m (i) = M (x (i) ), it is defined via a positive-definite kernel −k(·) (typically Gaussian): ℓ C (y (i) ,m (i) ) = β 1− k σ (y (i) − m (i) ) 10 with k σ (u) = exp − u 2 2σ 2 , β and σ parameters properly chosen. Multiclass variants can be built by applying the one-class-versus-the-rest strategy. 4.4 Training and Evaluation Protocol Models were trained for a maximum of E = 20, 000 epochs for each dataset using stochastic gradient descent with a batch size of 64 samples. The learning rate was set to 0.01 and fixed through all the training epochs. To obtain statistically reliable estimates, all results were computed under a 10-fold stratified cross-validation [45] scheme. For each fold, the 15 % of the training set was used for validation set V al using stratified sampling [45]. Prior to each training, all input features were then standardized using z-score nor- malization [46, 47]. The normalization parameters (mean and standard deviation) were computed exclusively on the training portion of each fold and subsequently applied to the corresponding validation and test sets, ensuring that no information from the held-out data leaked into the training process [36]. We emphasize that, as above discussed, the objective of this work is not to achieve state-of-the-art performance on these benchmarks, thus we intentionally adopt simple and uniform preprocessing rather than dataset-specific preprocessing prior to each training. For each dataset and each fold, the model was trained while monitoring validation loss ℓ(V al,e) and validation accuracy a(V al,e) at every epoch e. Model selection was performed based solely on validation criteria, but evaluation was always carried out on the corresponding Test fold. Specifically, for each Test fold we computed: 1. the test accuracy of the model corresponding to the epoch with the minimum validation loss, denoted as a(Test,e ⋆ ℓ,V al ); 2. the test accuracy of the model corresponding to the epoch with the maximum validation accuracy, denoted as a(Test,e ⋆ a,V al ); 3. the test-optimal accuracy, defined as the maximum test accuracy achieved across all training epochs, denoted as A ⋆ Test . These three quantities were collected for each fold, yielding paired samples of test accuracies for every dataset and every comparison. Analyses were performed through hypothesis testing. Formally, we tested: H 0 : μ a(Test,e ⋆ a,V al ) = μ A ⋆ T est vs. H 1 : μ a(Test,e ⋆ a,V al ) < μ A ⋆ T est , H 0 : μ a(Test,e ⋆ ℓ,V al ) = μ A ⋆ T est vs. H 1 : μ a(Test,e ⋆ ℓ,V al ) < μ A ⋆ T est , where μ a(Test,e ⋆ a,V al ) denotes the mean test accuracy obtained by selecting, for each fold, the model checkpoint corresponding to the epoch that maximizes the validation accuracy a, and μ a(Test,e ⋆ ℓ,V al ) denotes the mean test accuracy obtained by selecting the checkpoint corresponding to the epoch that minimizes the validation loss ℓ. Specifically, normality of the cross-validation results was first assessed using the Shapiro-Wilk test [46]. When normality was not rejected, a paired one-tailed t-test [46] was applied; otherwise, the one-tailed Wilcoxon signed-rank test [46] was used. The significance level was set to α = 0.05. 11 0.5 1 1.5 (e ⋆ ℓ,V al , L ⋆ V al (e ⋆ a,V al , A ⋆ V al ) Epoch e ℓ ( V al,e ) , a ( V al,e ) ℓ(V al,e) a(V al,e) e ⋆ a,V al e ⋆ ℓ,V al e ⋆ a,T est E 0.5 0.6 0.7 0.8 0.9 (e ⋆ ℓ,V al , a(Test,e ⋆ ℓ,V al ) e ⋆ a,V al , a(Test,e ⋆ a,V al ) (e ⋆ a,Test ,A ⋆ Test ) a ( Test,e ) a(test,e) Fig. 3: An example showing, in a single panel, the validation trajectories (loss ℓ(Val,e) in blue and accuracy a(Val,e) in orange, left axis) together with the test accuracy trajectory a(Test,e) (green, right axis). Vertical dashed lines indicate the validation- selected epochs e ⋆ a,Val and e ⋆ ℓ,Val , as well as the test–optimal epoch e ⋆ a,Test . Horizontal dotted lines mark L ⋆ Val and A ⋆ Val . The test accuracies achieved by the two valida- tion–driven selections, a(Test,e ⋆ ℓ,Val ) and a(Test,e ⋆ a,Val ), contrasted with the best achievableA ⋆ Test . 4.5 Validation Criteria and Loss-Metric Combinations Models were trained in separate runs, each using a single loss function, i.e. cross- entropy loss, C-Loss, or Poly-1, as the training objective. In particular, C-Loss was used with parameters σ = 0.5 and β = 1, while Poly-1 was configured with ε = 1. For each training run, the resulting sequence of model checkpoints was evaluated on the same validation set V al using the three loss functions ℓ CE , ℓ C , and ℓ Poly−1 and the accuracy a as validation criteria. This procedure was designed to disentangle the effect of the training objective from that of the model selection criterion; accordingly, we adopted a fully crossed experimental design. Precisely, at each training epoch we compute, on the validation set, the adopted losses and the classification accuracy, regardless of the loss used for optimization on the training data. Model selection is then performed independently for each validation criterion by identifying the epoch that optimizes the corresponding quantity. This procedure yields, for every training loss, multiple candidate models selected according to different quantity of validation performance. By evaluating all selected models on the same held-out test set, we can quantify how different validation criteria induce different orderings over the same set of candidate models, and how these orderings translate into test performance. Moreover, over all the epochs for each training run, early stopping is simulated independently for each validation loss. In the case of loss-based criteria, generalization 12 is considered to have improved whenever the validation loss decreases; for accuracy- based early stopping, improvement corresponds to an increase in validation accuracy We consider three configurations: post-hoc checkpoint selection (i.e. no early stopping), corresponding to selecting the best epoch across all training iterations (or until near- perfect fitting of the training data is achieved); early stopping with patience T = 10; and a more conservative patience of T = 50 epochs. For each configuration and each validation criterion, the model selected by early stopping is identified as the check- point corresponding to the best validation performance observed T epochs before the stopping condition is met. The test accuracy of the selected checkpoint is then com- pared against the test-optimal accuracy A ⋆ Test , defined as the maximum test accuracy attained over the entire training trajectory. This comparison allows us to quantify the extent to which standard early-stopping practices approximate or fail to recover the test-optimal model. 5 Results and discussion In the following, we report the experimental results. For each experimental setting, datasets are ordered by increasing linear separability, as measured by the generalized discrimination value (GDV), to highlight how model selection behavior varies with dataset complexity. Results obtained using cross-entropy as training objective and early stopping with T = 10 are shown in Figure 4. When cross-entropy is used as the validation criterion, the null hypothesis is not rejected in 5.98 % of the evaluated configurations, indicating scenarios in which the difference between the test accuracy achieved by validation- based model selection and the test-optimal accuracy is not statistically significant. In these cases, models selected based on the validation set exhibit test performance that is statistically indistinguishable from the test-optimal one. A similar behavior is observed when alternative loss functions are adopted as validation criteria. Specifically, when C-Loss and PolyLoss are used as validation criteria, the null hypothesis is not rejected in the 5.34 % and 5.98 % of the cases, respectively, leading to comparable conclusions. In contrast, a different behavior is observed when validation accuracy is used as the selection criterion. In this case, the null hypothesis is not rejected in the 0.43 % of the evaluated configurations, indicating that accuracy-based validation is substantially less likely to select models whose test performance is statistically indistinguishable from the test-optimal accuracy. This result suggests that, despite being the final evaluation metric, validation accuracy may constitute a less reliable criterion for model selection than loss-based alternatives. Figure 5 shows the same setup, but using early stopping with T = 50. When cross- entropy is used as the validation criterion, the null hypothesis is not rejected in 4.91 % of the evaluated configurations. Using C-Loss as validation criterion, this proportion increases to 6.20 %. With Poly-1, the null hypothesis is not rejected in 5.58 % of the configurations. When accuracy is used as the validation criterion, the null hypothesis is not rejected in 0.43 % of the evaluated configurations. These results confirm that, even with a larger early stopping patience, loss-based validation criteria provide a 13 Fig. 4: Graphical representation of the hypothesis testing results obtained using cross- entropy as the training objective and early stopping with patience T = 10. Each heatmap reports the p-values obtained from hypothesis tests comparing the test accu- racy of models selected using the validation set against the test-optimal accuracy A ⋆ Test across cross-validation folds. From left to right, panels correspond to validation based on cross-entropy loss, C-Loss, Poly-1, and validation accuracy, respectively. Rows rep- resent datasets and columns correspond to different parameter-to-sample ratios r. Datasets are ordered from top to bottom according to increasing linear separability, estimated using the generalized discrimination value (GDV). more reliable basis for model selection than validation accuracy to reach test-optimal accuracy. Figure 6 shows the results without the application of early stopping. When cross- entropy is adopted as the validation criterion, the null hypothesis is not rejected in 5.56 % of the evaluated configurations. A comparable behavior is observed also using the other loss functions as validation criterion: using C-Loss, this proportion increases to 6.84 %, while using the Poly-1, the null hypothesis is not rejected in 6.41 % of the cases. Also in this case, when accuracy is used as the validation criterion, the null hypothesis is not rejected with a lower proportion, i.e., 2.56 % of the evaluated configurations. Results obtained using C-Loss and Poly-1 as training objectives lead to similar conclusions; detailed statistical analyses and corresponding figures are reported in Appendix A. A summary of the percentages of null hypothesis acceptance across all training objectives, validation criteria, and early stopping configurations is reported in Table 2. Across all training objectives and early stopping settings, loss-based validation criteria consistently yield higher proportions of configurations in which validation-selected 14 Fig. 5: Graphical representation of the hypothesis testing results obtained using cross- entropy as the training objective and early stopping with patience T = 50. Each heatmap reports the p-values obtained from hypothesis tests comparing the test accu- racy of models selected using the validation set against the test-optimal accuracy A ⋆ Test across cross-validation folds. From left to right, panels correspond to validation based on cross-entropy loss, C-Loss, Poly-1, and validation accuracy, respectively. Rows rep- resent datasets and columns correspond to different parameter-to-sample ratios r. Datasets are ordered from top to bottom according to increasing linear separability, estimated using the generalized discrimination value (GDV). Table 2: Percentages of null hypothesis acceptance under different validation criteria and early stopping strategies, for each training objective. The acceptance of the null hypothe- sis corresponds to cases in which the validation-selected model achieves test performance statistically indistinguishable from the test-optimal model. Training Objective Early Stopping Cross-Entropy C-Loss PolyLoss Accuracy Cross-Entropy T = 105.98 %5.34 %5.98 %0.43 % T = 504.91 %6.20 %5.58 %0.43 % Disabled5.56 %6.84 %6.41 %2.56 % C-Loss T = 1017.74 %19.44 %18.38 %11.11 % T = 5017.95 %19.02 %18.38 %11.54 % Disabled19.23 %21.15 %18.59 %13.25 % Poly-1 T = 105.77 %6.62 %6.20 %0.64 % T = 505.56 %5.98 %5.77 %0.64 % Disabled5.56 %6.41 %5.98 %1.71 % 15 Fig. 6: Graphical representation of the hypothesis testing results obtained using cross- entropy as the training objective, without early stopping. Each heatmap reports the p-values obtained from hypothesis tests comparing the test accuracy of models selected using the validation set against the test-optimal accuracy A ⋆ Test across cross-validation folds. From left to right, panels correspond to validation based on cross-entropy loss, C-Loss, Poly-1, and validation accuracy, respectively. Rows represent datasets and columns correspond to different parameter-to-sample ratios r. Datasets are ordered from top to bottom according to increasing linear separability, estimated using the generalized discrimination value (GDV). models achieve test performance that is statistically indistinguishable from the test- optimal accuracy. In contrast, validation accuracy systematically exhibits the lowest acceptance rates in all considered scenarios. Across all training objectives, different loss-based validation criteria exhibit closely aligned acceptance rates, indicating that the benefit arises from loss-based model selection per se, rather than from a specific choice of loss function. As a practical con- sequence, this suggests that, among loss-based criteria, simpler and computationally less expensive losses, such as cross-entropy, may be preferred for validation without compromising model selection effectiveness. Consistently with this observation, despite being the final evaluation metric, validation accuracy proves to be a weaker signal for model selection compared to loss-based alternatives. The observed trends are consistent across all considered training objectives, indi- cating that the superiority of loss-based validation criteria does not rely on a specific alignment between training and validation losses. Finally, training with C-Loss is associated with higher acceptance rates across validation criteria, suggesting a potentially stronger alignment between validation- based selection and test-optimal performance. However, even in this case, the higher 16 acceptance rate remains largely independent of the validation loss used for model selection. From a practical perspective, these results suggest that monitoring validation loss, rather than validation accuracy, constitutes a more reliable strategy for model selection when the objective is to approach test-optimal accuracy. Fig. 7: Acceptance rate of the null hypothesis with α = 0.05 as a function of the parameter-to-sample ratio r, under different training objectives, early stopping strate- gies, and validation criteria. Rows correspond to the training objective (cross-entropy, C-Loss, and Poly-1), while columns report results obtained using early stopping with T = 10, T = 50, and with early stopping disabled. Bars represent different valida- tion criteria: cross-entropy, C-Loss, PolyLoss, and validation accuracy. The acceptance rate indicates the proportion of configurations in which the test accuracy of the model selected via validation is statistically indistinguishable from the test-optimal accuracy. Figure 7 reports the acceptance rate of the null hypothesis with respect to the parameter-to-sample ratio r, under different training objectives (rows), early stop- ping strategies (columns), and validation criteria (bars). Across all training objectives and early stopping configurations, the acceptance rates remain remarkably stable as r varies over several orders of magnitude, ranging from strongly under-parameterized to highly over-parameterized regimes. No systematic trend can be observed as a func- tion of r, suggesting that the ability of validation-based model selection to identify 17 models whose test accuracy is statistically indistinguishable from the test-optimal one is insensitive to the degree of model parameterization. This behavior is consistent across all validation criteria and early stopping strate- gies. In particular, the relative ordering between loss-based validation criteria and validation accuracy is preserved for all values of r, with loss-based criteria consis- tently achieving higher acceptance rates than accuracy-based validation. This suggests that the superiority of loss-based validation does not arise from a specific regime of parameterization, but rather reflects a more general property of the validation signal itself. Overall, these results indicate that the observed advantages of loss-based model selection are robust across under-parameterized, critically parameterized, and over- parameterized regimes. Consequently, the effectiveness of loss-based validation criteria in aligning validation-based model selection with test-optimal performance does not depend on fine-tuning the parameter-to-sample ratio, but persists across a wide range of model capacities. Figure 8 reports the acceptance rate of the null hypothesis with respect to the significance level α. Results are shown for different training objectives (rows), early stopping strategies (columns), and validation criteria (curves). Across all configura- tions, the acceptance rate exhibits a monotonic increase as α decreases, as expected from the behavior of hypothesis testing procedures. Interestingly, a consistent and pronounced separation emerges between loss-based validation criteria (dashed lines) and validation accuracy (solid). For all training objectives and early stopping settings, validation based on loss functions yields substantially higher acceptance rates than validation accuracy over the entire range of significance levels considered. This indi- cates that loss-based criteria are systematically more likely to select models whose test performance is statistically indistinguishable from the test-optimal one. The three loss-based validation criteria exhibit closely aligned trends, with only marginal quantitative differences across all values of α. This observation suggests that the advantage of loss-based validation does not stem from a particular choice of loss function, which is still coherent to what was observed before. In contrast, validation accuracy, despite being the final evaluation metric, confirms to provide a weaker and less reliable signal for selecting models that generalize optimally to the test set, as evidenced by its consistently lower acceptance rates. Finally, differences across training objectives are also observable. In particular, training with C-Loss is associated with higher acceptance rates across all validation criteria, suggesting a stronger alignment between validation-based model selection and test-optimal performance. Nonetheless, the relative advantage of loss-based validation over accuracy-based selection persists uniformly across all training objectives, reinforc- ing the conclusion that monitoring validation loss constitutes a more reliable strategy for model selection than validation accuracy when the goal is to approach test-optimal accuracy. 18 Fig. 8: Acceptance rate of the null hypothesis as a function of the significance level α, under different training objectives, early stopping strategies, and validation crite- ria. The acceptance rate indicates the proportion of configurations in which the test accuracy of the model selected via validation is statistically indistinguishable from the test-optimal accuracy. Rows correspond to the training objective (cross-entropy, C-Loss, and Poly-1), while columns report results obtained using early stopping with T = 10, T = 50, and with early stopping disabled. Curves represent different valida- tion criteria: loss-based criteria (cross-entropy, C-Loss, and Poly-1, dashed lines) and validation accuracy (solid lines). 6 Conclusions This study examined how different validation criteria lead model selection when the deployment objective is test accuracy. Across datasets, generalization regimes, and early-stopping settings, accuracy, despite being the main classification task metric, underperform as a selection criterion. Indeed, loss-based validation (cross-entropy, C- Loss, PolyLoss) selects checkpoints whose test accuracy is more often statistically close to the test-optimal model than those chosen by validation accuracy. The gap persists whether early stopping is used with moderate or large patience, or disabled in favor of post-hoc checkpoint selection. 19 This can be due to the fact that accuracy is a discrete, thresholded indicator with low sensitivity to incremental improvements. It changes only when predictions flip around the decision boundary, so it produces long plateaus and frequent ties across epochs—especially on small validation sets—making early-stopping triggers noisy and unstable. Moreover, accuracy ignores confidence: two checkpoints with equal accu- racy can differ substantially in margins. These finite-sample effects are amplified by patience-based rules, where small random oscillations can halt training early on a merely local optimum. In short, accuracy is excellent for final reporting, but it seems a poor compass to validate over an iterative training process. Conversely, using the adopted validation loss leads to higher acceptance rates in our hypothesis tests than validation accuracy, regardless of the training loss. The advantage is robust to stopping regime (early vs. post-hoc) and persists across datasets and model sizes. The specific loss used for validation matters less than being loss-based. In fact, cross-entropy, C-Loss, and PolyLoss used on the validation set deliver closely aligned acceptance rates. Practically, this means one can prefer the simpler, cheaper cross-entropy for validation without sacrificing selection quality. Finally, acceptance rates show no systematic dependence from under- to over-parameterized. It is also worth noting that, across validation criteria, the absolute proportion of accepted null hypothesis remains modest. Indeed, in most cases, validation-selected models do not achieve test performance that is statistically indistinguishable from the test-optimal checkpoint. This might suggests that validation-based selection alone may be insufficient and motivates further investigation into alternative analytical and methodological approaches. This study intentionally focused on accuracy-centred evaluation under standard supervised protocols. Extending the analysis to other endpoints (e.g., F1, MCC, PR- AUC), settings with pronounced class imbalance, or larger-scale regimes would clarify when accuracy-based validation narrows the gap. It would also be valuable to study validation-set size explicitly, and to assess whether combining a loss-based selector with lightweight post-selection threshold tuning further closes the distance to the test-optimal model. In conclusion, what we monitor matters. When model selection depends on a val- idation trajectory, especially under early stopping, loss-based criteria provide a more reliable estimate of generalization and, in turn, more dependable accuracy on unseen data. Acknowledgment This work was partially funded by the PNRR MUR project PE0000013-FAIR (CUP: E63C25000630006). Funding 20 References [1] Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, ??? (2016) [2] Bishop, C.M.: Regularization and complexity control in feed-forward networks (1995) [3] Huang, C., Zhai, S., Talbott, W., Martin, M.B., Sun, S.-Y., Guestrin, C., Susskind, J.: Addressing the loss-metric mismatch with adaptive loss alignment. In: International Conference on Machine Learning, p. 2891–2900 (2019). PMLR [4] Singh, A., Pokharel, R., Principe, J.: The c-loss function for pattern classification. Pattern Recognition 47(1), 441–453 (2014) [5] Leng, Z., Tan, M., Liu, C., Cubuk, E.D., Shi, J., Cheng, S., Anguelov, D.: Polyloss: A polynomial expansion perspective of classification loss functions. In: International Conference on Learning Representations (2022) [6] Hagiwara, K.: Regularization learning, early stopping and biased estimator. Neurocomputing 48(1-4), 937–955 (2002) [7] Evgeniou, T., Poggio, T., Pontil, M., Verri, A.: Regularization and statistical learning theory for data analysis. Computational Statistics & Data Analysis 38(4), 421–432 (2002) [8] Advani, M.S., Saxe, A.M., Sompolinsky, H.: High-dimensional dynamics of generalization error in neural networks. Neural Networks 132, 428–446 (2020) [9] Asuncion, A., Newman, D., et al.: UCI machine learning repository. Irvine, CA, USA (2007) [10] Terven, J., Cordova-Esparza, D.-M., Romero-Gonz ́alez, J.-A., Ram ́ırez-Pedraza, A., Ch ́avez-Urbiola, E.: A comprehensive survey of loss functions and metrics in deep learning. Artificial Intelligence Review 58(7), 195 (2025) [11] Santamar ́ıa, I., Pokharel, P.P., Principe, J.C.: Generalized correlation function: definition, properties, and application to blind equalization. IEEE Transactions on Signal Processing 54(6), 2187–2197 (2006) [12] Singh, A., Principe, J.C.: A loss function for classification based on a robust similarity metric. In: The 2010 International Joint Conference on Neural Networks (IJCNN), p. 1–6 (2010). IEEE [13] Wang, X.-N., Wei, J.-M., Jin, H., Yu, G., Zhang, H.-W.: Probabilistic confusion entropy for evaluating classifiers. Entropy 15(11), 4969–4992 (2013) [14] Marchetti, F., Guastavino, S., Piana, M., Campi, C.: Score-oriented loss (sol) functions. Pattern Recognition 132, 108913 (2022) 21 [15] Marchetti, F., Guastavino, S., Campi, C., Benvenuto, F., Piana, M.: A compre- hensive theoretical framework for the optimization of neural networks classifica- tion performance with respect to weighted metrics. Optimization Letters 19(1), 169–192 (2025) [16] Caruana, R., Lawrence, S., Giles, C.: Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Advances in neural information processing systems 13 (2000) [17] Ji, Z., Li, J., Telgarsky, M.: Early-stopped neural networks are consistent. Advances in Neural Information Processing Systems 34, 1805–1817 (2021) [18] Soudry, D., Carmon, Y.: No bad local minima: Data independent training error guarantees for multilayer neural networks. arXiv preprint arXiv:1605.08361 (2016) [19] Swirszcz, G., Czarnecki, W.M., Pascanu, R.: Local minima in training of neural networks. arXiv preprint arXiv:1611.06310 (2016) [20] Goodfellow, I.J., Vinyals, O., Saxe, A.M.: Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.6544 (2014) [21] Zhang, J., Ma, C., Liu, J., Shi, G.: Penetrating the influence of regularizations on neural network based on information bottleneck theory. Neurocomputing 393, 76–82 (2020) [22] Baldi, P., Chauvin, Y.: Temporal evolution of generalization during learning in linear networks. Neural Computation 3(4), 589–603 (1991) [23] Wang, C., Venkatesh, S., Judd, J.: Optimal stopping and effective machine complexity in learning. Advances in neural information processing systems 6 (1993) [24] Dodier, R.: Geometry of early stopping in linear networks. Advances in neural information processing systems 8 (1995) [25] B ̈uhlmann, P., Yu, B.: Boosting with the l 2 loss: regression and classification. Journal of the American Statistical Association 98(462), 324–339 (2003) [26] Barron, A.R., Cohen, A., Dahmen, W., DeVore, R.A.: Approximation and learning by greedy algorithms (2008) [27] Chen, H., Li, L., Pan, Z.: Learning rates of multi-kernel regression by orthogonal greedy algorithm. Journal of Statistical Planning and Inference 143(2), 276–282 (2013) [28] Wei, Y., Yang, F., Wainwright, M.J.: Early stopping for kernel boosting algo- rithms: A general analysis with localized complexities. Advances in Neural 22 Information Processing Systems 30 (2017) [29] Bandos, T.V., Camps-Valls, G., Soria-Olivas, E.: Statistical criteria for early- stopping of support vector machines. Neurocomputing 70(13-15), 2588–2592 (2007) [30] Ferro, M.V., Mosquera, Y.D., Pena, F.J.R., Bilbao, V.M.D.: Early stopping by correlating online indicators in neural networks. Neural Networks 159, 109–124 (2023) [31] Ennett, C.M., Frize, M., Scales, N.: Evaluation of the logarithmic-sensitivity index as a neural network stopping criterion for rare outcomes. In: 4th International IEEE EMBS Special Topic Conference on Information Technology Applications in Biomedicine, 2003., p. 207–210 (2003). IEEE [32] Lalis, J., Gerardo, B., Byun, Y.: An adaptive stopping criterion for backpropaga- tion learning in feedforward neural network. International Journal of Multimedia and Ubiquitous Engineering 9(8), 149–156 (2014) [33] Wu, X.-x., Liu, J.-g.: A new early stopping algorithm for improving neural network generalization. In: 2009 Second International Conference on Intelligent Computation Technology and Automation, vol. 1, p. 15–18 (2009). IEEE [34] Natarajan, S., Rhinehart, R.R.: Automated stopping criteria for neural network training. In: Proceedings of the 1997 American Control Conference (Cat. No. 97CH36041), vol. 4, p. 2409–2413 (1997). IEEE [35] Iyer, M.S., Rhinehart, R.R.: A novel method to stop neural network training. In: Proceedings of the 2000 American Control Conference. ACC (IEEE Cat. No. 00CH36334), vol. 2, p. 929–933 (2000). IEEE [36] Apicella, A., Isgr`o, F., Prevete, R.: Don’t push the button! exploring data leak- age risks in machine learning and transfer learning. Artificial Intelligence Review 58(11), 339 (2025) [37] Prechelt, L.: Early stopping-but when? In: Neural Networks: Tricks of the Trade, p. 55–69. Springer, ??? (2002) [38] Lodwich, A., Rangoni, Y., Breuel, T.: Evaluation of robustness and performance of early stopping rules with multi layer perceptrons. In: 2009 International Joint Conference on Neural Networks, p. 1877–1884 (2009). IEEE [39] Nguyen, M.H., Abbass, H.A., McKay, R.I.: Stopping criteria for ensemble of evolutionary artificial neural networks. Applied Soft Computing 6(1), 100–107 (2005) [40] Suliman, A., Omarov, B.: Early stopping criteria for levenberg-marquardt based 23 neural network training optimization. International Journal of Engineering and Technology (uae) 7(4.36), 1194–1198 (2018) [41] Shao, Y., Taff, G.N., Walsh, S.J.: Comparison of early stopping criteria for neural- network-based subpixel classification. IEEE Geoscience and Remote Sensing Letters 8(1), 113–117 (2010) [42] Vera, M., Rey Vega, L., Piantanida, P.: Pacman: Pac-style bounds accounting for the mismatch between accuracy and negative log-loss. Information and Inference: A Journal of the IMA 13(1), 002 (2024) [43] Liao, Q., Miranda, B., Banburski, A., Hidary, J., Poggio, T.: A surprising lin- ear relationship predicts test performance in deep networks. arXiv preprint arXiv:1807.09659 (2018) [44] Schilling, A., Maier, A., Gerum, R., Metzner, C., Krauss, P.: Quantifying the separability of data classes in neural networks. Neural Networks 139, 278–293 (2021) https://doi.org/10.1016/j.neunet.2021.03.035 [45] Bishop, C.M., Bishop, H.: Deep Learning: Foundations and Concepts. Springer, ??? (2023) [46] Hastie, T.: The elements of statistical learning: data mining, inference, and prediction. Springer (2009) [47] Apicella, A., Isgr`o, F., Pollastro, A., Prevete, R.: On the effects of data normal- ization for domain adaptation on eeg data. Engineering Applications of Artificial Intelligence 123, 106205 (2023) 24