Paper deep dive
On Hyperparameter Tuning on the Test Set
Matteo Fregonara, Tom Viering, Jan van Gemert
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:"Don't tune hyperparameters on the test set" is often stated in machine learning textbooks. Violating it is considered a cardinal sin that produces misleadingly optimistic results, corrupts benchmark integrity, and thus can even be interpreted as scientific fraud. Yet evidence suggests that test set hyperparameter tuning does occur in practice, making it all the more important to understand its actual consequences. So how bad is it, really? In this work we question this dogma and put it to an empirical test. We systematically study the magnitude of the performance inflation caused by tuning the hyperparameters on the test set for MNIST-1D, CIFAR-10, and three tasks from the GLUE benchmark. Our experiments show that while the effect is real and significant, it is frequently small relative to other sources of noise. In many cases, we find that tuning on the test set recovers exactly the same model as when tuning on the validation set. Most importantly, we find that the rankings of models remain essentially preserved after tuning on the test set and therefore that consistent test-set tuning may not invalidate benchmarks or model selection. Our results call for a more nuanced view of tuning hyperparameters on the test set, stimulating researchers to openly report test tuning.
Tags
Links
- Source: https://arxiv.org/abs/2610.05902v1
- Canonical: https://arxiv.org/abs/2610.05902v1
Trouble viewing inline? Open PDF directly →
Full Text
78,136 characters extracted from source content.
Expand or collapse full text
On Hyperparameter Tuning on the Test Set Matteo Fregonara Tom Viering Jan van Gemert Affiliation: Delft University of Technology Abstract ”Don’t tune hyperparameters on the test set” is often stated in machine learning textbooks. Violating it is considered a cardinal sin that produces misleadingly optimistic results, corrupts benchmark integrity, and thus can even be interpreted as scientific fraud. Yet evidence suggests that test set hyperparameter tuning does occur in practice, making it all the more important to understand its actual consequences. So how bad is it, really? In this work we question this dogma and put it to an empirical test. We systematically study the magnitude of the performance inflation caused by tuning the hyperparameters on the test set for MNIST-1D, CIFAR-10, and three tasks from the GLUE benchmark. Our experiments show that while the effect is real and significant, it is frequently small relative to other sources of noise. In many cases, we find that tuning on the test set recovers exactly the same model as when tuning on the validation set. Most importantly, we find that the rankings of models remain essentially preserved after tuning on the test set and therefore that consistent test-set tuning may not invalidate benchmarks or model selection. Our results call for a more nuanced view of tuning hyperparameters on the test set, stimulating researchers to openly report test tuning. 1 Introduction Overfitting is an inflated generalization estimate, possibly distorting model selection rankings, which may lead to deploying the wrong model. We investigate overfitting of hyperparameters: configurations that determine model training behavior. Specifically, we investigate overfitting by tuning hyperparameters on the test set, i.e., the same data set used for reporting generalization performance. Machine learning textbooks prohibit hyperparameter tuning on the test set. From a theoretical perspective, a hyperparameter is no different than any other model parameter that is set at training time. Thus, an extreme view can see tuning hyperparameters on the test set as academic fraud in the current benchmarking paradigm (Hardt, 2026). For example, tuning model parameter initialization on the test set (Picard, 2021) will give excellent accuracy; independent of other claims for empirical gain in machine learning scholarship (Lipton and Steinhardt, 2019; Sculley et al., 2018). In practice, however, several implementations already tune hyperparameters on the test set, both in computer vision (PyTorch Contributors, 2024; Wightman, 2019; Touvron et al., 2021a; Bao et al., 2022b; He et al., 2022b) and in natural language processing (Devlin et al., 2019; Google Research, 2018; Liu et al., 2019; Facebook AI Research, 2019). Accordingly, it can be argued that a hyperparameter is not like an individual model parameter, because hyperparameters are usually regularized by their global effect on a model, where e.g., the learning rate is typically shared over all model parameters. Moreover, in practice, not an infinite number of hyperparameter configurations are evaluated, but only a handful (Hastie et al., 2009). Thus, there is tension between theory and practice, and here we investigate how bad hyperparameter overfitting really is. In this paper, we give the first systematic empirical evaluation of overfitting by tuning hyperparameters on the test set. We carefully construct a controlled evaluation benchmark of repeated hyperparameter tuning on a validation set, compared to tuning on the test set. We evaluate on the controlled MNIST-1D (Greydanus and Kobak, 2024), image classifcation on CIFAR-10 (Krizhevsky, 2009) and three natural language understanding tasks from the GLUE benchmark (Wang et al., 2019) (RTE, MRPC and CoLA). We test multiple model families, and train 36,00036,000 model variants. We investigate how easily models overfit, how severe the overfitting is, and if the overfitting leads to differences in model selection. Our results show that overfitting by tuning on the test set is significant on every benchmark we evaluate. Yet, the same hyperparameter configuration is often best on both the validation set as on the test set and the accuracy gain by overfitting is mild when compared to other sources of randomness such as data splits. Model selection rankings barely change when all models are tuned on test data, and we did not observe that the best model is not selected. Our results call for a nuanced view on tuning hyperparameters on the test set. Overfitting is real, and comparing validation-tuned models vs test-tuned models benefits test-tuned models. However, model rankings rarely change when all models under comparison are tuned on the same data, making the effect of test-tuning relatively benign. Our paper gives empirical evidence to normalize hyperparameter tuning on the test set, and we encourage authors not to hide or be reluctant to admit test-tuning, but openly report it. 2 Related Work Adaptive overfitting and test set reuse. Adaptive data analysis is formalized (Dwork et al., 2015; Makarova et al., 2021) as a setting in which a holdout dataset is queried repeatedly by adaptively chosen hypotheses. Theory suggests that repeated queries may lead to overfitting of the hidden test set, even though this may be not intentional (e.g. through publication bias or human actions). This suggests that long-term reuse of benchmark test sets should lead to measurable overfitting. However, empirical evidence paints a more nuanced picture. New test sets for CIFAR-10 and ImageNet, constructed to match the original data collection process, show that accuracy drops of 3–15% on CIFAR-10 and 11–14% on ImageNet are largely explained by a distribution shift instead of adaptive overfitting (Recht et al., 2018; Recht et al., 2019). Importantly, model rankings are preserved, with an approximately linear relationship between original and replicated accuracy. Evidence at a larger scale points in the same direction (Liao et al., 2021; Roelofs et al., 2019). One explanation for this gap between theory and practice is that the effective number of distinct hypotheses explored during model development is smaller than it appears. Models trained on the same benchmark often produce highly similar predictions (Mania et al., 2019). This reduces the ability of adaptive procedures to exploit test-set noise, even when the test set is reused multiple times. Recent work has begun to formalize this risk for hyperparameter tuning (Schröder et al., 2025; Schneider et al., 2025; Makarova et al., 2021), but no study has isolated its effect when tuning is done directly on the test set of a widely used benchmark. We follow these works in a stronger setting: we give the tuning procedure direct access to the test set. Test set hyperparameter tuning is common practice. Despite this reassuring empirical picture, tuning and final reporting are usually performed on the same labels, because official test labels are often withheld: ImageNet uses the val split as a proxy test set (Russakovsky et al., 2015), and GLUE (Wang et al., 2019) the development split. The official PyTorch example (PyTorch Contributors, 2024) evaluates on val every epoch and keeps the best checkpoint, and others present hyperparameter search as an ablation on that same data (Wightman, 2019; Liu et al., 2022). The DeiT (Touvron et al., 2021a; Touvron et al., 2021b), BEiT (Bao et al., 2022b; Bao et al., 2022a) and MAE (He et al., 2022b; He et al., 2022a) implementations go further, calling val the “test” set in their own print statements and metric keys. On GLUE, BERT (Devlin et al., 2019; Google Research, 2018) performs an exhaustive search choosing “the model that performs best on the development set” and, for the smaller tasks, “several random restarts” selecting “the best model on the Dev set”, while RoBERTa (Liu et al., 2019; Facebook AI Research, 2019) keeps only the checkpoint that maximises validation accuracy and reports on that same split. Hyperparameter sensitivity. The extent to which hyperparameter tuning can cause adaptive overfitting depends on how strongly hyperparameters influence accuracy. Sensitivity analysis methods quantify these effects. Taylor et al. (2021) identify learning rate decay and batch size as consistently important. For image classification, a functional ANOVA study of 12 hyperparameters for Residual Neural Networks across 10 datasets (Sharma et al., 2019) shows that individual hyperparameters and their interactions have a substantial effect on accuracy. A complementary analysis over 10,000 CNN configurations (Wang et al., 2024) ranks the most influential factors and highlights the number of convolutional layers, learning rate, dropout rate, optimizer and number of epochs. Hyperparameter tuning leads to significant improvements. Empirical evidence shows that hyperparameter tuning is a major driver of model performance (Weerts et al., 2020). For example, tuning only learning rate, weight decay and batch size on a validation split results in a baseline that outperforms several specialised methods (Brigato et al., 2021), which suggests that reported gains may partly reflect more thorough hyperparameter optimisation rather than algorithmic innovation. The process is also heavily shaped by human skill: practitioner expertise and the trial-and-error budget significantly dictate final outcomes (Anand et al., 2020). The same property that makes tuning so effective, however, is what creates room for overfitting. Even adaptive search strategies such as Bayesian optimisation, which use prior evaluations to guide where to query next, can substantially overfit hyperparameters to a validation set (Makarova et al., 2021). Variance in benchmarks. Benchmark accuracy varies due to data sampling, model initialization, training dynamics and hyperparameter optimization. If this variability is not accounted for, comparisons between models can be biased because observed differences may reflect noise rather than systematic effects (Bouthillier et al., 2021). To address this, models can be compared using the probability of outperforming. Let A and B be the performances of two models to be compared. Then P(A>B)P(A>B) can be estimated over repeated runs, where all sources of randomness should be varied. This provides a more robust measure than single-run accuracy and allows uncertainty to be quantified explicitly. Empirical evidence shows that variability can be substantial. A sweep over up to 10410^4 random seeds on CIFAR-10 and ImageNet (Picard, 2021) finds that, despite low average variance, it is easy to obtain outlier seeds with significantly better or worse accuracy. Differences of performance in literature can therefore arise purely from randomness or adaptive overfitting. Here, we question if these benchmark rankings are still valid when hyperparameters are tuned on the test set. 3 Methodology We study adaptive overfitting induced by hyperparameter selection under repeated evaluation on a fixed dataset split. Formally, we define adaptive overfitting as the accuracy advantage obtained when hyperparameters are selected using test-set feedback rather than an independent validation set, following Dwork et al. (2015). Given a training pipeline, a hyperparameter search space and a dataset, we estimate adaptive overfitting and draw statistical conclusions. Problem Setup and Selection Strategies We focus on a classification setting and let DtrainD_train, DvalD_val and DtestD_test denote the training, validation and test datasets, respectively. The hyperparameter space for N hyperparameters is Λ , where a configuration λ∈Λλ∈ is an N-dimensional vector, and A(λ,D)A(λ,D) represents the accuracy of a model trained on DtrainD_train with hyperparameters λ and evaluated on dataset D. Hyperparameters are selected by sampling T candidate configurations ΛT=λ1,…,λT⊂Λ _T=\ _1,…, _T\⊂ via random sampling. We compare two hyperparameter selection strategies: validation-tuning selects λval∗=argmaxλ∈ΛTA(λ,Dval)λ^*_val= _λ∈ _TA(λ,D_val), while test-tuning selects λtest∗=argmaxλ∈ΛTA(λ,Dtest)λ^*_test= _λ∈ _TA(λ,D_test), where we carefully pair the validation-tuned and the test-tuned runs so that the only source of variation is in the evaluation dataset DvalD_val vs DtestD_test. Adaptive Overfitting Gap. We quantify adaptive overfitting through the adaptive overfitting gap, ΔAO=A(λtest∗,Dtest)−A(λval∗,Dtest), _AO\;=\;A(λ^*_test,D_test)\;-\;A(λ^*_val,D_test), (1) the excess test accuracy attributable to hyperparameter selection with direct knowledge of DtestD_test. Because λtest∗λ^*_test maximises A(λ,Dtest)A(λ,D_test) over the same pool ΛT _T from which λval∗λ^*_val is drawn, ΔAO≥0 _AO≥ 0 in every paired run, with equality when both rules pick the same configuration (or two configurations that score identically on DtestD_test). Following Bouthillier et al. (2021), we additionally report P^(ΔAO>0)≔P(ΔAO>0)+12P(ΔAO=0), P( _AO>0)\; \;P( _AO>0)+ 12\,P( _AO=0), (2) the empirical win probability of test-tuning, where ties contribute 0.50.5. We estimate it together with a 95%95\,\% percentile-bootstrap confidence interval over the K paired runs, and call adaptive overfitting significant when the lower bound of this Confidence Interval (CI) exceeds 0.50.5. Paired Run Design. Each of the K paired (val-tuned, test-tuned) training runs executes the following pipeline: 1. sample an independent bootstrap split (Dtrain,Dval,Dtest)(D_train,D_val,D_test) (Appendix I describes the splitting scheme), 2. draw T independent hyperparameter configurations ΛT _T via random sampling, 3. for each configuration λ∈ΛTλ∈ _T, train a model on DtrainD_train (from random initialisation for the image benchmarks, and by fine-tuning a pretrained checkpoint for the GLUE tasks) and record A(λ,Dval)A(λ,D_val) and A(λ,Dtest)A(λ,D_test), 4. select λval∗λ^*_val and λtest∗λ^*_test from the same pool of T trained models and compute ΔAO _AO. Because of the paired run design, both selection strategies operate on the same bootstrap split, so any difference between λval∗λ^*_val and λtest∗λ^*_test isolates the effect of the selection rule alone. We use K=30K=30 paired runs per model, the minimum needed for the Noether-bound paired-comparison guarantee (Bouthillier et al., 2021; Noether, 1987). Following Bouthillier et al. (2021), every experiment varies the major sources of randomness under our control (data splits, hyperparameter samples, initialisation seeds) so that the empirical distribution of ΔAO _AO reflects the full pipeline variance rather than a single favorable configuration. 4 Experiments We investigate overfitting by tuning hyperparameters directly on the test set, i.e., using the same dataset for hyperparameter selection and accuracy estimation. We first investigate model sensitivity to overfitting and how severe the overfitting could be. Then, we evaluate if tuning on the test set leads to different model selection compared to tuning on the validation set. Datasets. We evaluate on two image classification benchmarks and three natural language understanding tasks. The controlled benchmark is MNIST-1D (Greydanus and Kobak, 2024), a 1D-signal variant of MNIST designed for controlled experiments in machine learning. We use a 4,000-example source set bootstrapped per training run, and test a family of eight fully-connected networks spanning 1.3k1.3\,k to 2×1052×10^5 parameters (Appendix D), trained with Adam for 8,000 steps. We optimize learning rate, learning-rate decay rate (set to zero with probability 0.30.3), weight decay, and the random seed used to initialize the network weights. For CIFAR-10 we combine the training and test sets into a single 60,000-example pool, from which each training run draws an independent bootstrap split. We use 8 architectures from the CIFAR-10 leaderboard (Appendix D), covering ResNets (He et al., 2016), Shake-Shake (Gastaldi, 2017), VGG (Simonyan and Zisserman, 2015), MobileNetV2 (Sandler et al., 2018) and ResNet-9 (Page, 2019). We tune learning rate, weight decay, momentum, number of training steps and the random seed used to initialise the network weights, using Nesterov SGD (Sutskever et al., 2013) with cosine-annealed learning rate. We additionally evaluate three classification datasets from the GLUE benchmark (Wang et al., 2019): RTE (textual entailment), MRPC (paraphrase detection) and CoLA (grammatical acceptability). The official GLUE test splits are unlabeled and withheld on the leaderboard server, so we combine each dataset’s train and validation sets into their own single pool (RTE: 2,7672,767, MRPC: 4,0764,076 and CoLA: 9,5949,594 examples), from which each training run draws an independent bootstrap split. To save space we report MRPC in the main text and RTE and CoLA in Appendix A since all three behave alike. We fine-tune eight pretrained transformers spanning 11.811.8M to 141.9141.9M parameters (Appendix D) with AdamW and a linear warmup-decay schedule, tuning the fine-tuning hyperparameters: learning rate, weight decay, warmup ratio, number of epochs and the initialization seed. We score all three by accuracy, so that one metric applies across all five benchmarks. Experimental settings. We use K=30K=30 paired training runs per architecture, where we re-sample the train, val, and test sets differently for each of the K runs. The hyperparameter search budget per run is T=30T=30, which is in line with the number of configurations typically evaluated in deep learning architecture training (Hastie et al., 2009). We select the best accuracy over the T randomly sampled different hyperparameter configurations. All randomness derives from a single master seed. Hyperparameter search spaces are in Appendix C and additional implementation details in Appendix B. In total this yields 8×30×30=7,2008× 30× 30=7,200 trained models per benchmark, or 36,00036,000 across the five benchmarks. The full set of 36,00036,000 trained models and 72,00072,000 validation/test set evaluations used approximately 8,0008,000 GPU-hours. As a sanity check to validate if overfitting is possible, we compute an upper bound by training and tuning models directly on the test data. Every architecture nearly memorizes its own test fold, achieving exactly 100%100\% accuracy on both image benchmarks, and between 98.9%98.9\% and 100%100\% accuracy on the GLUE tasks, whose test folds are smaller (Appendix F). 4.1 Exp 1: Overfitting sensitivity and overfitting severity For tuning hyperparameters on the test set, we investigate overfitting sensitivity: how easy it is to overfit, and overfitting severity: how much a model overfits. Figure 1: Overfitting sensitivity: the tie-broken win probability of test-tuning, P^(ΔAO>0) P( _AO>0), as a function of the tuning budget T. Each line is one architecture, with its 95%95\,\% percentile-bootstrap CI shaded. A line whose CI lower bound clears the dashed threshold at 0.50.5 shows statistically significant adaptive overfitting at that budget. MNIST-1D and MRPC clear it at around T=3T=3, CIFAR-10 at around T=13T=13. RTE and CoLA are in Appendix A. Overfitting sensitivity. To quantify how easily a model overfits, we compare the hyperparameter search budget T against the empirical win probability P^(ΔAO>0) P( _AO>0) and its 95%95\,\% bootstrap CI. The rationale is that a model that overfits easily reaches significance, i.e. CIlower>0.5CI_lower>0.5, at a smaller budget T. Results in Figure 1 show that all models reach significance easily, needing only around T=3T=3 different hyperparameter configurations on MNIST-1D and MRPC, and around T=13T=13 on CIFAR-10. RTE and CoLA behave like MRPC (Appendix A). Note that P^(ΔAO>0) P( _AO>0) is not strictly increasing with T, because a larger budget also benefits validation-tuning, which can in turn lower P^(ΔAO>0) P( _AO>0). Benchmark ΔAO=0 _AO=0 Percentage Identical configuration MNIST-1D 99/24099/240 41.2%41.2\% 98.0%98.0\% CIFAR-10 116/240116/240 48.3%48.3\% 98.3%98.3\% MRPC 31/24031/240 12.9%12.9\% 90.3%90.3\% Table 1: Runs with ΔAO=0 _AO=0 pooled over the eight architectures of each benchmark (K=30K=30 runs each), as a count and as a percentage of all runs. The last column is the share of runs with ΔAO=0 _AO=0 in which validation-tuning and test-tuning selected the identical configuration. The remainder are distinct configurations that happen to score identically on the test split. RTE and CoLA are in Appendix A. A substantial fraction of runs have ΔAO=0 _AO=0: validation- and test-tuning select the same configuration, so test-tuning gains nothing. Table 1 shows this fraction is high on the image benchmarks (41.2%41.2\% on MNIST-1D, 48.3%48.3\% on CIFAR-10) and lower on MRPC (12.9%12.9\%), which is also where the gap is largest relative to run-to-run noise. On every benchmark this effect is overwhelmingly caused by both selection strategies picking the same hyperparameter configurations instead of accidental ties between distinct configurations. When one configuration wins on both splits, test-tuning has nothing to gain. Figure 2: Overfitting severity: per-run adaptive overfitting gap. Each translucent dot is one of the K=30K=30 paired runs of an architecture; the filled blue circle is the per-architecture mean ΔAO _AO. The red horizontal bar depicts one standard deviation of that architecture’s val-tuned test accuracy, representing run-to-run noise. On nearly every architecture the mean ΔAO _AO sits below the standard deviation, showing that the gap is small relative to natural run-to-run variability. RTE and CoLA are in Appendix A. Overfitting severity. To evaluate how much a model overfits, we compare the mean ΔAO _AO over the K=30K=30 paired runs against the natural run-to-run variability of the same architecture, measured as the standard deviation of the validation-tuned test accuracy across the same K=30K=30 runs. If ΔAO _AO is small relative to this experimental noise, the apparent overfitting is no larger than what re-running the pipeline under different data splits and initializations would already produce. Results in Figure 2 show that on 2020 of the 2424 architecture–benchmark pairs, the mean gap is smaller than the run-to-run standard deviation of the validation-tuned models. The margin is widest on the image benchmarks, where the gap is roughly half the standard deviation and never exceeds 0.740.74 of it, and narrowest on MRPC, where four of the eight architectures sit marginally above it, up to 1.131.13. We conclude that overfitting severity, in our setup, is comparable to the natural experimental variability in the experimental design. Robustness checks. Two additional experiments test whether these findings depend on the search budget, the split sizes, or the particular test fold. First, we reran MLP-Giant on MNIST-1D with T=100T=100 trials per run (a 3×3× larger budget) and larger fixed splits of 10,000/10,000/10,00010,000/10,000/10,000 drawn from a 70,00070,000-sample MNIST-1D set, denoted MLP-GiantT=100. Both findings hold at this budget: P^(ΔAO>0)=0.73 P( _AO>0)=0.73 at T=100T=100 with the CI lower bound well above 0.50.5, and 1616 of 3030 runs (53%53\%) still have ΔAO=0 _AO=0, essentially unchanged from the 17/3017/30 (57%57\%) in the main setup, with the two rules selecting the same configuration in all 1616. The large fraction of runs with ΔAO=0 _AO=0 is therefore structural rather than caused by the small search budget (Appendix G). Secondly, we re-evaluated the selected CIFAR-10 configurations on CIFAR-10.1 (Recht et al., 2019), a 2,0002,000-image replication of CIFAR-10 drawn from the same source distribution. On this independent set the val-tuned and test-tuned configurations score within run-to-run noise of each other, so the gap measured on the bootstrap fold does not carry over. Tuning directly on CIFAR-10.1 beats both by roughly 0.4%0.4\%, showing what a procedure with access to it could gain (Appendix H). Figure 3: Validation-tuned vs. test-tuned mean test accuracy. Markers are per-architecture means with 95%95\,\% bootstrap confidence intervals; dots are the individual paired runs. The dashed grey line is the identity y=xy=x and the orange line is the ordinary least squares fit through the eight architecture means, with its 95%95\,\% pairs-bootstrap confidence interval shaded. On every benchmark the means lie close to a line nearly parallel to the identity, so test-tuning acts as a near-uniform upward shift that preserves architecture rankings. RTE and CoLA are in Appendix A. 4.2 Exp 2: How much does overfitting influence model selection? Here, we investigate if overfitting leads to different model selection when compared to tuning hyperparameters on a validation set. We evaluate model selection by ranking models, as commonly done on a leaderboard. For each experiment, we fit test-tuned on validation-tuned test accuracy by ordinary least squares on the per-architecture means, and report Spearman and Kendall rank correlations on those same means. All confidence intervals are based on 100,000100,000 bootstrap resamples. For the linear fit we use a standard pairs bootstrap, resampling the eight architecture mean pairs with replacement. The rank correlations resample the K runs within each architecture. Figure 3 shows the ordinary least squares fit between validation-tuned and test-tuned test accuracies, with per-architecture rankings in Table 2. The slope is close to one (0.910.91–0.970.97) and the fit is tight (R2≥0.99R^2≥ 0.99). Test-tuning acts as a near-uniform upward shift and does not reorder the ranking. The architecture ranking is identical under both selection rules on MNIST-1D and CIFAR-10 (Spearman ρ=1.00ρ=1.00, Kendall τ=1.00τ=1.00). On MRPC a single adjacent pair exchanges places (ρ=0.976ρ=0.976, τ=0.929τ=0.929), between two architectures differing by 0.30%0.30\% against a run-to-run standard deviation of 1.34%1.34\%, so they are statistically indistinguishable under either rule. RTE and CoLA show the same pattern (Appendix A). For all benchmarks and all models, adaptive overfitting does better in the absolute accuracies, but not in the relative comparison: a leaderboard built from val-tuned numbers selects the same winner as a leaderboard built from test-tuned numbers. Val. Model Validation-tuned Test-tuned ΔAO _AO Val. Std. Δ Rank (%) (%) (%) MNIST-1D 1 MLP-Giant 64.59 65.56 +0.97+0.97 1.761.76 0 +0 2 MLP-Huge 64.25 64.96 +0.72+0.72 2.462.46 0 +0 3 MLP-Larger 63.58 64.75 +1.17+1.17 1.971.97 0 +0 4 MLP-Large 61.74 62.65 +0.91+0.91 1.961.96 0 +0 5 MLP-Base 60.08 61.33 +1.26+1.26 2.442.44 0 +0 6 MLP-Small 58.19 59.74 +1.56+1.56 2.412.41 0 +0 7 MLP-Mini 55.15 56.33 +1.19+1.19 2.542.54 0 +0 8 MLP-Tiny 53.99 55.28 +1.28+1.28 1.921.92 0 +0 CIFAR-10 1 shake_shake_32d 95.36 95.47 +0.11+0.11 0.210.21 0 +0 2 resnet9 94.68 94.79 +0.11+0.11 0.220.22 0 +0 3 mobilenetv2 94.43 94.51 +0.08+0.08 0.260.26 0 +0 4 resnet_basic_56 93.74 93.81 +0.08+0.08 0.240.24 0 +0 5 resnet_basic_44 93.61 93.73 +0.12+0.12 0.290.29 0 +0 6 vgg_15_BN_64 93.58 93.71 +0.13+0.13 0.260.26 0 +0 7 resnet_basic_32 93.10 93.25 +0.16+0.16 0.280.28 0 +0 8 resnet_basic_20 92.34 92.52 +0.18+0.18 0.250.25 0 +0 MRPC 1 deberta-v3-small 86.23 87.22 +0.99+0.99 1.331.33 0 +0 2 roberta-base 85.96 87.13 +1.18+1.18 1.361.36 0 +0 3 xlnet-base 84.78 85.83 +1.05+1.05 1.411.41 0 +0 4 albert-base 84.30 85.65 +1.35+1.35 1.211.21 0 +0 5 mobilebert 82.84 84.09 +1.25+1.25 1.221.22 −1-1 6 electra-small 82.54 84.12 +1.58+1.58 1.391.39 +1+1 7 bert-base 82.30 83.77 +1.47+1.47 1.371.37 0 +0 8 distilbert 81.33 82.71 +1.39+1.39 1.391.39 0 +0 Table 2: Model rankings ordered by validation-tuned accuracy, with the adaptive overfitting difference and the run-to-run standard deviation of validation-tuned accuracy. Several models within a benchmark are close together and have near-similar accuracy. On MNIST-1D and CIFAR-10 no model changes rank when tuned on the test set rather than the validation set. On MRPC exactly one adjacent pair exchanges places, between models separated by far less than the run-to-run standard deviation, so they are statistically indistinguishable under either rule. The top-ranked model is unchanged. RTE and CoLA are in Appendix A. 5 Discussion and Limitations The experiments tell a consistent story across five benchmarks, two modalities and three model families. It is easy to overfit by tuning on the test set, but the severity stays within the range of natural experimental variability such as random dataset splits and training randomness. The mean gap is roughly half the run-to-run standard deviation on the image benchmarks and between 0.70.7 and 1.01.0 times it on the GLUE tasks. Test-tuning yields a consistent absolute improvement without reordering models: rankings are identical under both selection rules on MNIST-1D, CIFAR-10 and RTE, and on MRPC and CoLA the only change is a single adjacent inversion between architectures separated by far less than run-to-run noise. The top-ranked model is unchanged everywhere. A key observation that explains why the magnitude stays small is the fraction of runs with an adaptive overfitting gap of 0, where validation and test tuning select exactly the same configuration and test-tuning gains nothing. This fraction is high on the image benchmarks (41.2%41.2\% on MNIST-1D, 48.3%48.3\% on CIFAR-10) and lower on the GLUE tasks (12.9%12.9\%–25.8%25.8\%), which are also where the gap is largest relative to run-to-run noise. The cause is structural: within a run, one configuration tends to win on both the validation and test split, and this dominance weakens on the small GLUE tasks, where the evaluation splits are small and fine-tuning is noisy, so the two rankings agree less often. We confirmed this by rerunning MLP-Giant on MNIST-1D with T=100T=100 trials and larger validation and test sets: the fraction of runs with ΔAO=0 _AO=0 stayed nearly unchanged at 53%53\%. This aligns with Mania et al. (2019), who attribute low adaptive overfitting to high prediction similarity across models. Our finding is specific to a single hyperparameter search but reflects the same: when one configuration dominates, adaptive selection has little to exploit. The results agree with the empirical literature. Recht et al. (2018); Recht et al. (2019) and Roelofs et al. (2019) both find that adaptive overfitting on popular benchmarks is small and does not change model rankings. Our work extends this by directly measuring adaptive overfitting for specific experiments and architectures in a controlled setting, compared to previous work which infers it from leaderboard differences. Our findings make a pattern clear: a substantial fraction of runs produce a zero gap, the rest a small positive one. Compared to the worst-case theoretical bounds from adaptive data analysis (Dwork et al., 2015), the observed effect is smaller, which is expected given that those bounds are conservative: they hold against an adversarial analyst that can adapt each query to prior test feedback, whereas random hyperparameter search draws queries independently and produces highly correlated trials, so the effective number of independent test set queries is much smaller than T. In practice, this means that reusing the test set for hyperparameter tuning under random search at this budget produces a measurable but very small accuracy inflation, on image classification and natural language understanding alike. 5.1 Limitations Random search is a weak adaptive querier. The methodology uses random search, which evaluates each hyperparameter configuration independently. In practice, methods such as Bayesian optimization, hyperband Li et al. (2018), or manual iterative tuning reuse results from earlier evaluations, and could produce larger adaptive overfitting gaps. Small hyperparameter budget. Each run uses T=30T=30 samples, which is in line with search budgets typically used in practice Hastie et al. (2009), but is still small in absolute terms. Additionally, the batch size is kept fixed and not tuned. A follow-up on MNIST-1D with T=100T=100 random-search trials showed no meaningful difference, but behavior at much larger compute budgets is not explored. Limited domain coverage. The larger GLUE tasks, other modalities, and regression are not tested. Liao et al. (2021) suggest that adaptive overfitting may be more severe for regression. Per-procedure rather than cumulative. Each ΔAO _AO estimate covers one hyperparameter search. In practice, benchmark inflation builds up across many groups reusing the same test set over time. Increasing T within our experiments only adds more queries from the same procedure and does not capture the adaptive selection across the deep learning community (architecture choices, publication filtering, etc.) that drives the community-level inflation studied by Recht et al. (2019); Roelofs et al. (2019). Wide confidence intervals. K=30K=30 paired runs is the minimum recommended by Bouthillier et al. (2021) via the Noether bound (Noether, 1987), so the bootstrap intervals around P^(ΔAO>0) P( _AO>0) are wide, around 0.100.10–0.170.17. The weakest architectures have lower bounds near 0.600.60, so their individual results are not robust, but the overall pattern of small positive gaps does not depend on them. Only visual and textual models evaluated. We only evaluate common deep learning architectures, limited to the image and text domain. We did not explore other machine learning algorithms. 6 Conclusion We presented a paired-run methodology for directly measuring adaptive overfitting from hyperparameter tuning, applied to a controlled MNIST-1D family of 88 MLPs, a realistic suite of 88 CIFAR-10 architectures, and three small GLUE tasks each fine-tuning 88 pretrained transformers, for 36,00036,000 trained models in total. We investigate how easy and severe test-tuning is and show statistically significant adaptive overfitting on all 4040 architecture–benchmark combinations. The effect was small in absolute terms, and within the range of the natural experimental variation. A large fraction of runs had no overfitting gap, where validation and test tuning selected the same configuration: 41.2%41.2\% of runs on MNIST-1D and 48.3%48.3\% on CIFAR-10, and a smaller but still substantial 12.9%12.9\%–25.8%25.8\% on the GLUE tasks. This zero-gap phenomenon persisted under a 3×3× larger search budget and larger splits, indicating that increasing the search budget does not substantially change the overlap between validation and test selections at this scale. We also investigate whether test-tuning changes which model would be selected, where it behaved as a near-uniform upward shift: the linear fit through the per-architecture means had slope close to one (0.910.91–1.021.02) and R2≥0.99R^2≥ 0.99 on every benchmark, and rank correlations were exactly 11 on MNIST-1D, CIFAR-10 and RTE. On MRPC and CoLA a single adjacent pair exchanged places, in both cases between architectures separated by far less than run-to-run noise, and the top-ranked model was unchanged everywhere. Taken together, the experiments suggest that for honest random-search hyperparameter tuning, the practical risk of test-set reuse is much smaller than worst-case theory predicts, and comparing accuracies under test-tuning does not change which model a leaderboard would select. These empirical results allow researchers to discuss, mention, and defend tuning hyperparameters on the test set, leading to a more open scientific discourse. AI use statement In this work, we used generative AI tools for coding support, such as refactoring and debugging. We have not used generative AI tools for generating research ideas, designing experiments, producing results, or creating data, and other required disclosure tasks are not applicable to this work. Additionally, we used generative AI tools to improve the clarity of the writing and to correct grammatical errors. We have reviewed all AI-assisted work. AI-assisted code was manually verified and tested for correctness, and all AI-suggested edits to the writing were checked against the original intent and meaning before acceptance. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI. Ethics statement This work studies a methodological issue in machine learning evaluation, namely how strongly tuning hyperparameters on the test set inflates reported accuracy. It does not introduce new models or datasets, and we do not anticipate direct negative societal consequences from its release. If anything, a clearer empirical picture of when test-set tuning matters and when it does not should help researchers and reviewers assess published results more accurately, reducing the risk that benchmark numbers are taken at face value without questioning the selection protocol behind them. All experiments are run on standard public datasets (MNIST-1D, CIFAR-10 and three GLUE tasks) on shared academic cluster hardware, and the released code and analysis scripts are intended to make the findings independently verifiable. Reproducibility statement All code, experiment configurations, and analysis scripts required to reproduce our results are available at https://anonymous.4open.science/r/hyperparameter-tuning-on-test-set/README.md. All randomness derives from a single master seed: NumPy’s SeedSequence splits it into independent sub-seeds, one per architecture, per run, and per source of randomness within a run (bootstrap split, hyperparameter sampling – including the per-trial initialisation seed – and other training stochasticity). No two sub-seeds coincide, so the full experiment is reproducible end-to-end from that single seed. The paired-run protocol, the two selection rules, and the definition of ΔAO _AO are given in Section 3, and the experimental settings, budgets, and compute environment in Section 4. Dataset construction and the out-of-bootstrap splitting scheme are detailed in Appendix I, the architectures and their reference implementations in Appendix D, the hyperparameter search spaces in Appendix C, and the training setup in Appendix B. Appendix E reports the exact bootstrap settings and full confidence intervals behind the ranking analysis, and Appendix J the empirical winner distributions used to verify that the search ranges are well-shaped. The supporting analyses are likewise fully specified: the memorisation upper bound in Appendix F, the larger-budget MLP-GiantT=100 rerun in Appendix G, and the CIFAR-10.1 transfer evaluation in Appendix H. References Anand et al. (2020) K. Anand, Z. Wang, M. Loog, and J. van Gemert Black magic in deep learning: how human skill impacts network training. In Proceedings of the British Machine Vision Conference (BMVC), External Links: Link Cited by: §2. Bao et al. (2022a) H. Bao, L. Dong, S. Piao, and F. Wei BEiT: BERT pre-training of image transformers. In International Conference on Learning Representations (ICLR 2022), External Links: Link Cited by: §2. Bao et al. (2022b) H. Bao, L. Dong, S. Piao, and F. Wei BEiT: BERT pre-training of image transformers. Note: https://github.com/microsoft/unilm/tree/master/beitAccessed: 2026-05-06 Cited by: §1, §2. Bouthillier et al. (2021) X. Bouthillier, P. Delaunay, M. Bronzi, A. Trofimov, B. Nichyporuk, J. Szeto, N. Mohammadi Sepahvand, E. Raff, K. Madan, V. Voleti, S. E. Kahou, V. Michalski, D. Serdyuk, T. Arbel, C. Pal, G. Varoquaux, and P. Vincent Accounting for variance in machine learning benchmarks. In Proceedings of Machine Learning and Systems 3 (MLSys 2021), External Links: Link Cited by: §2, §3, §3, §3, §5.1. Brigato et al. (2021) L. Brigato, B. Barz, L. Iocchi, and J. Denzler Tune it or don’t use it: benchmarking data-efficient image classification. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Vol. , p. 1071–1080. External Links: Document Cited by: §2. Clark et al. (2020) K. Clark, M. Luong, Q. V. Le, and C. D. Manning ELECTRA: pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR), Cited by: Table 10. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, p. 4171–4186. External Links: Link, Document Cited by: Table 10, §1, §2. Dodge et al. (2020) J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith Fine-tuning pretrained language models: weight initializations, data orders, and early stopping. External Links: 2002.06305, Link Cited by: Appendix B. Dwork et al. (2015) C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth Generalization in adaptive data analysis and holdout reuse. In Advances in Neural Information Processing Systems 28 (NIPS 2015), p. 2350–2358. External Links: Link Cited by: §2, §3, §5. Facebook AI Research (2019) Facebook AI Research Fairseq: fine-tuning RoBERTa on GLUE tasks. Note: https://github.com/facebookresearch/fairseq/blob/main/examples/roberta/README.glue.mdAccessed 2026 Cited by: §1, §2. Gastaldi (2017) X. Gastaldi Shake-shake regularization of 3-branch residual networks. In International Conference on Learning Representations Workshop (ICLR 2017), External Links: Link Cited by: Table 9, §4. Google Research (2018) Google Research BERT: TensorFlow code and pre-trained models. Note: https://github.com/google-research/bertAccessed 2026 Cited by: §1, §2. Greydanus and Kobak (2024) S. J. Greydanus and D. Kobak Scaling down deep learning with MNIST-1D. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 16404–16415. External Links: Link Cited by: §1, §4. Hardt (2026) M. Hardt The emerging science of machine learning benchmarks. Princeton University Press. External Links: Link Cited by: §1. Hastie et al. (2009) T. Hastie, R. Tibshirani, and J. Friedman The elements of statistical learning: data mining, inference, and prediction. 2 edition, Springer Series in Statistics, Springer, New York, NY. External Links: ISBN 978-0-387-84857-0, Document Cited by: §1, §4, §5.1. He et al. (2022a) K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 15979–15988. External Links: Document Cited by: §2. He et al. (2022b) K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick Masked autoencoders: a PyTorch implementation. Note: https://github.com/facebookresearch/maeAccessed: 2026-05-06 Cited by: §1, §2. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 770–778. External Links: Document Cited by: Table 9, Table 9, Table 9, Table 9, §4. He et al. (2023) P. He, J. Gao, and W. Chen DeBERTaV3: improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In International Conference on Learning Representations (ICLR), Cited by: Table 10. Krizhevsky (2009) A. Krizhevsky Learning multiple layers of features from tiny images. Technical report University of Toronto. External Links: Link Cited by: §1. Lan et al. (2020) Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut ALBERT: a lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR), Cited by: Table 10. Li et al. (2018) L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar Hyperband: a novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research 18 (185), p. 1–52. External Links: Link Cited by: §5.1. Liao et al. (2021) T. I. Liao, R. Taori, D. Raji, and L. Schmidt Are we learning yet? a meta review of evaluation failures across machine learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021), Round 2, External Links: Link Cited by: §2, §5.1. Lipton and Steinhardt (2019) Z. C. Lipton and J. Steinhardt Troubling trends in machine learning scholarship. Communications of the ACM 62 (6), p. 45–53. External Links: Document, Link Cited by: §1. Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov RoBERTa: a robustly optimized bert pretraining approach. External Links: 1907.11692, Link Cited by: Table 10, §1, §2. Liu et al. (2022) Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie ConvNeXt official PyTorch implementation. Note: https://github.com/facebookresearch/ConvNeXtAccessed: 2026-05-06 Cited by: §2. Makarova et al. (2021) A. Makarova, H. Shen, V. Perrone, A. Klein, J. B. Faddoul, A. Krause, M. Seeger, and C. Archambeau Overfitting in Bayesian optimization: an empirical study and early-stopping solution. In ICLR 2021 Workshop on Neural Architecture Search, External Links: Link Cited by: §2, §2, §2. Mania et al. (2019) H. Mania, J. Miller, L. Schmidt, M. Hardt, and B. Recht Model similarity mitigates test set overuse. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), p. 9993–10002. External Links: Link Cited by: §2, §5. Noether (1987) G. E. Noether Sample size determination for some common nonparametric tests. Journal of the American Statistical Association 82 (398), p. 645–647. External Links: Document, Link Cited by: §3, §5.1. Page (2019) D. Page How to train your ResNet 8: bag of tricks. Note: https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tricks/Blog post, Myrtle.ai Cited by: Table 9, §4. Picard (2021) D. Picard Torch.manual_seed(3407) is All You Need: On the Influence of Random Seeds in Deep Learning Architectures for Computer Vision. CoRR abs/2109.08203. External Links: Document, Link Cited by: §1, §2. PyTorch Contributors (2024) PyTorch Contributors PyTorch examples: ImageNet training in PyTorch. Note: https://github.com/pytorch/examples/tree/main/imagenetAccessed: 2026-04-21 Cited by: §1, §2. Recht et al. (2018) B. Recht, R. Roelofs, L. Schmidt, and V. Shankar Do CIFAR-10 classifiers generalize to cifar-10?. CoRR abs/1806.00451. External Links: Link, 1806.00451 Cited by: §2, §5. Recht et al. (2019) B. Recht, R. Roelofs, L. Schmidt, and V. Shankar Do imagenet classifiers generalize to imagenet?. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 5389–5400. External Links: Link Cited by: §2, §4.1, §5.1, §5. Roelofs et al. (2019) R. Roelofs, S. Fridovich-Keil, J. Miller, V. Shankar, M. Hardt, B. Recht, and L. Schmidt A meta-analysis of overfitting in machine learning. In Advances in Neural Information Processing Systems 32, Note: 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada External Links: Link Cited by: §2, §5.1, §5. Russakovsky et al. (2015) O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115 (3), p. 211–252. External Links: Document, Link Cited by: §2. Sandler et al. (2018) M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen MobileNetV2: inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vol. , p. 4510–4520. External Links: Document Cited by: Table 9, §4. Sanh et al. (2019) V. Sanh, L. Debut, J. Chaumond, and T. Wolf DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, NeurIPS, Cited by: Table 10. Schneider et al. (2025) L. Schneider, B. Bischl, and M. Feurer Overtuning in hyperparameter optimization. In Proceedings of the Fourth International Conference on Automated Machine Learning, Proceedings of Machine Learning Research, Vol. 293, p. 17/1–43. External Links: Link Cited by: §2. Schröder et al. (2025) S. Schröder, M. Baratchi, and J. N. van Rijn Overfitting in combined algorithm selection and hyperparameter optimization. In Advances in Intelligent Data Analysis XXIII (IDA 2025), Lecture Notes in Computer Science, Vol. 15669, Cham, p. 181–194. External Links: Document, Link Cited by: §2. Sculley et al. (2018) D. Sculley, J. Snoek, A. Wiltschko, and A. Rahimi Winner’s curse? On pace, progress, and empirical rigor. In ICLR 2018 Workshop on Reproducibility in Machine Learning, External Links: Link Cited by: §1. Sharma et al. (2019) A. Sharma, J. N. van Rijn, F. Hutter, and A. Müller Hyperparameter importance for image classification by residual neural networks. In Discovery Science (DS 2019), Lecture Notes in Computer Science, Vol. 11828, Cham, p. 112–126. External Links: Document, Link Cited by: §2. Simonyan and Zisserman (2015) K. Simonyan and A. Zisserman Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR 2015), External Links: Link Cited by: Table 9, §4. Sun et al. (2020) Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, p. 2158–2170. External Links: Link, Document Cited by: Table 10. Sutskever et al. (2013) I. Sutskever, J. Martens, G. Dahl, and G. Hinton On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning, S. Dasgupta and D. McAllester (Eds.), Proceedings of Machine Learning Research, Vol. 28, Atlanta, Georgia, USA, p. 1139–1147. External Links: Link Cited by: §4. Taylor et al. (2021) R. Taylor, V. Ojha, I. Martino, and G. Nicosia Sensitivity analysis for deep learning: ranking hyper-parameter influence. In 33rd IEEE International Conference on Tools with Artificial Intelligence (ICTAI), External Links: Document, Link Cited by: §2. Touvron et al. (2021a) H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou DeiT: data-efficient image transformers. Note: https://github.com/facebookresearch/deitAccessed: 2026-05-06 Cited by: §1, §2. Touvron et al. (2021b) H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 139, p. 10347–10357. External Links: Link Cited by: §2. Wang et al. (2019) A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman GLUE: a multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), Cited by: §1, §2, §4. Wang et al. (2024) R. Wang, I. Nabney, and M. Golbabaee Efficient hyperparameter importance assessment for CNNs. In Neural Information Processing (ICONIP 2024), Communications in Computer and Information Science, Vol. 2282, Singapore, p. 16–31. External Links: Document, Link Cited by: §2. Weerts et al. (2020) H. J. P. Weerts, A. C. Mueller, and J. Vanschoren Importance of tuning hyperparameters of machine learning algorithms. External Links: 2007.07588, Link Cited by: §2. Wightman (2019) R. Wightman Timm: PyTorch image models. Note: https://github.com/huggingface/pytorch-image-modelsAccessed: 2026-05-06 Cited by: §1, §2. Yang et al. (2019) Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le XLNet: generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, p. . External Links: Link Cited by: Table 10. Appendix A Additional GLUE Datasets: RTE and CoLA RTE and CoLA follow exactly the same protocol as MRPC in the main text. Overfitting sensitivity. Both datasets reach significance at around T=3T=3 (Figure 4), matching MNIST-1D and MRPC and well below CIFAR-10’s T=13T=13. Adaptive overfitting is significant for all 1616 architecture–dataset pairs. Runs with no gap. Table 3 gives the fraction of runs with ΔAO=0 _AO=0. As in the main text, the overwhelming majority of these happen because both selection rules pick the identical configuration rather than through accidental ties. Benchmark ΔAO=0 _AO=0 Percentage Identical configuration RTE 62/24062/240 25.8%25.8\% 95.2%95.2\% CoLA 48/24048/240 20.0%20.0\% 95.8%95.8\% Table 3: Runs with ΔAO=0 _AO=0 on RTE and CoLA, pooled over the eight architectures of each dataset (K=30K=30 runs each). Columns are as in Table 1. Overfitting severity. On 1515 of the 1616 architecture–dataset pairs the mean gap is smaller than the run-to-run standard deviation of the validation-tuned models (Figure 5). The single exception is distilbert on RTE, at 1.251.25 times the standard deviation, which is the largest ratio observed anywhere in the paper. On CoLA no architecture exceeds 0.890.89. Model selection. The linear fit is tight on both datasets (Figure 6): slope 0.9160.916 with R2=0.997R^2=0.997 on RTE, and slope 1.0201.020 with R2=0.992R^2=0.992 on CoLA. Rankings are identical under both selection rules on RTE (Spearman ρ=1.00ρ=1.00, Kendall τ=1.00τ=1.00). CoLA has a single adjacent inversion (ρ=0.976ρ=0.976, τ=0.929τ=0.929) between xlnet-base and distilbert, whose validation-tuned means differ by 0.14%0.14\% against a run-to-run standard deviation of 0.88%0.88\%, so they are statistically indistinguishable under either rule. The top-ranked model is unchanged on both datasets. Per-architecture results are in Table 4. Figure 4: Overfitting sensitivity on RTE and CoLA: the tie-broken win probability of test-tuning, P^(ΔAO>0) P( _AO>0), as a function of the tuning budget T. Each line is one architecture, with its 95%95\,\% percentile-bootstrap CI shaded. A line whose CI lower bound clears the dashed threshold at 0.50.5 shows statistically significant adaptive overfitting at that budget. Both datasets clear it at around T=3T=3. Compare Figure 1. Figure 5: Overfitting severity on RTE and CoLA: per-run adaptive overfitting gap. Each translucent dot is one of the K=30K=30 paired runs of an architecture; the filled blue circle is the per-architecture mean ΔAO _AO. The red horizontal bar depicts one standard deviation of that architecture’s val-tuned test accuracy, representing run-to-run noise. Only distilbert on RTE has a mean gap above its standard deviation. Compare Figure 2. Figure 6: Validation-tuned vs. test-tuned mean test accuracy on RTE and CoLA. Markers are per-architecture means with 95%95\,\% bootstrap confidence intervals; dots are the individual paired runs. The dashed grey line is the identity y=xy=x and the orange line is the ordinary least squares fit through the eight architecture means, with its 95%95\,\% pairs-bootstrap confidence interval shaded. On both datasets the means lie close to a line nearly parallel to the identity. Compare Figure 3. Val. Model Validation-tuned Test-tuned ΔAO _AO Val. Std. Δ Rank (%) (%) (%) RTE 1 roberta-base 73.62 74.88 +1.26+1.26 1.881.88 0 +0 2 deberta-v3-small 73.08 74.65 +1.57+1.57 1.751.75 0 +0 3 xlnet-base 68.53 70.45 +1.92+1.92 2.502.50 0 +0 4 albert-base 68.35 70.03 +1.68+1.68 1.921.92 0 +0 5 bert-base 67.85 69.60 +1.75+1.75 2.432.43 0 +0 6 mobilebert 66.24 68.12 +1.88+1.88 2.732.73 0 +0 7 electra-small 64.97 66.52 +1.55+1.55 2.612.61 0 +0 8 distilbert 59.06 61.87 +2.81+2.81 2.252.25 0 +0 CoLA 1 deberta-v3-small 86.22 86.78 +0.56+0.56 0.930.93 0 +0 2 roberta-base 84.97 85.54 +0.57+0.57 0.760.76 0 +0 3 electra-small 83.38 84.23 +0.85+0.85 0.960.96 0 +0 4 bert-base 82.90 83.61 +0.71+0.71 0.810.81 0 +0 5 mobilebert 82.38 83.02 +0.64+0.64 0.910.91 0 +0 6 albert-base 81.96 82.66 +0.70+0.70 0.970.97 0 +0 7 xlnet-base 80.36 80.55 +0.20+0.20 0.910.91 −1-1 8 distilbert 80.22 80.91 +0.69+0.69 0.810.81 +1+1 Table 4: Model rankings on RTE and CoLA, ordered by validation-tuned accuracy. Columns are as in Table 2. On RTE no model changes rank; on CoLA one adjacent pair exchanges places, between models separated by far less than the run-to-run standard deviation. The top-ranked model is unchanged on both. Appendix B Implementation Details Batch size. The two image experiments use a training batch size of 128128 and an evaluation batch size of 512512. The GLUE experiment uses a training batch size of 3232, following the published fine-tuning recipes of the model families we evaluate, and an evaluation batch size of 128128. Data augmentation (CIFAR-10). During training, each mini-batch is augmented with random cropping (32×3232× 32 with 44-pixel padding) and random horizontal flipping, followed by per-channel normalisation to zero mean and unit variance. At evaluation time, only the normalisation is applied. Mixed-precision training (CIFAR-10). All CIFAR-10 models are trained with automatic mixed precision (torch.amp) using a gradient scaler to maintain numerical stability. Training budget (CIFAR-10). The number of training steps is itself a tuned hyperparameter, sampled log-uniformly from [75,000, 141,000][75,000,\;141,000]. At batch size 128128, the bootstrapped 60,00060,000-example training split yields ≈ 470≈\,470 mini-batches per epoch, so this range corresponds to roughly 160160–300300 epochs. This sits within the typical training-length range reported for most of the architectures we use, providing enough budget for each configuration to converge while keeping experiments computationally manageable. Since our goal is to measure adaptive overfitting rather than to reproduce state-of-the-art accuracies, what matters is that the hyperparameter choices yield meaningful accuracy differences between configurations. Fine-tuning setup (GLUE). Each GLUE trial loads a fresh pretrained checkpoint with a randomly initialised classification head and fine-tunes all parameters; no layers are frozen. Inputs are tokenised with each model’s own tokenizer, truncated and padded to a maximum sequence length of 128128. Training uses AdamW with a linear warmup-then-decay schedule and gradient clipping at a maximum norm of 1.01.0. The number of epochs, the warmup ratio, the learning rate and the weight decay are all tuned (Table 7); the batch size is held fixed at 3232. The initialisation seed is sampled per trial and applied before the model is constructed, so the classification head’s initialisation and the data ordering are both part of the trial’s seed — fine-tuning on small GLUE tasks is known to be highly sensitive to this choice Dodge et al. (2020). Numerical precision (GLUE). All GLUE models are trained with automatic mixed precision and a gradient scaler, with one exception: MobileBERT’s activations overflow in fp16, so it is trained in full fp32. Checkpoints are loaded with an explicit fp32 master dtype, because some checkpoints (e.g. DeBERTa-v3-small) declare a half-precision torch_dtype and would otherwise load in fp16, which is incompatible with gradient scaling. With fp32 master weights, autocast still casts activations to fp16 in the forward pass while gradients remain fp32. Appendix C Hyperparameter Search Spaces Table 5: Hyperparameter search space for MNIST-1D. Hyperparameter Distribution Range Learning rate Log-Uniform [10−4,10−1][10^-4,10^-1] Learning-rate decay rate Log-Uniform† [10−5,10−2][10^-5,10^-2] Weight decay Log-Uniform [10−6,10−2][10^-6,10^-2] Initialization seed Uniform integer [0, 231−1][0,\;2^31-1] † Set to zero with probability 0.30.3 to explicitly sample the no-decay setting. Table 6: Hyperparameter search space for CIFAR-10. Hyperparameter Distribution Range Learning rate Log-Uniform [5×10−3, 0.5][5× 10^-3,\;0.5] Weight decay Log-Uniform [10−5, 10−2][10^-5,\;10^-2] SGD momentum Uniform [0.8, 0.99][0.8,\;0.99] Training steps Log-Uniform [75,000, 141,000][75,000,\;141,000] Initialization seed Uniform integer [0, 231−1][0,\;2^31-1] Table 7: Hyperparameter search space for the GLUE tasks, shared across all three tasks and all eight architectures. The ranges bracket the published fine-tuning grids of the model families in the registry: learning rates of 22–5×10−55× 10^-5 (BERT, XLNet), 11–3×10−53× 10^-5 (RoBERTa), 11–5×10−55× 10^-5 (ALBERT) and 10−410^-4 (ELECTRA-base); weight decays of 00 (HuggingFace run_glue default), 0.010.01 (BERT, DeBERTa) and 0.10.1 (RoBERTa); warmup ratios of 6%6\% (RoBERTa) and 10%10\% (BERT); and epoch counts of 33 (BERT) through 1010 (RoBERTa). Hyperparameter Distribution Range Learning rate Log-Uniform [5×10−6, 10−4][5× 10^-6,\;10^-4] Weight decay Log-Uniform [10−5, 10−1][10^-5,\;10^-1] Warmup ratio Uniform [0.0, 0.2][0.0,\;0.2] Number of epochs Uniform integer [2, 10][2,\;10] Initialization seed Uniform integer [0, 231−1][0,\;2^31-1] Appendix D Architectures D.1 MNIST-1D Table 8: MLP family evaluated on MNIST-1D. All networks use ReLU activations and residual connections between hidden layers. Architecture Hidden sizes Parameters MLP-Tiny [25][25] 1,2851,285 MLP-Mini [50][50] 2,5602,560 MLP-Small [100][100] 5,1105,110 MLP-Base [100,100][100,100] 15,21015,210 MLP-Large [100,100,100][100,100,100] 25,31025,310 MLP-Larger [200,200,200][200,200,200] 90,61090,610 MLP-Giant [200,200,200,200][200,200,200,200] 130,810130,810 MLP-Huge [300,300,300][300,300,300] 195,910195,910 D.2 CIFAR-10 Table 9: Architectures evaluated on CIFAR-10. Each model is ported from its reference implementation without architectural modifications. # Model Paper Repository 1 ResNet-20 He et al. (2016) github.com/hysts/pytorch_image_classification 2 ResNet-32 He et al. (2016) github.com/hysts/pytorch_image_classification 3 ResNet-44 He et al. (2016) github.com/hysts/pytorch_image_classification 4 ResNet-56 He et al. (2016) github.com/hysts/pytorch_image_classification 5 Shake-Shake (32d) Gastaldi (2017) github.com/hysts/pytorch_image_classification 6 VGG-15 BN Simonyan and Zisserman (2015) github.com/hysts/pytorch_image_classification 7 MobileNetV2 Sandler et al. (2018) github.com/hysts/pytorch_image_classification 8 ResNet-9 Page (2019) github.com/KellerJordan/cifar10-airbench D.3 GLUE Table 10: Pretrained transformers fine-tuned on each GLUE task. All checkpoints are public Hugging Face models, used without architectural modification; each trial loads the checkpoint afresh and attaches a randomly initialised classification head. The same eight models are evaluated on all three tasks. # Model Paper Parameters Hugging Face checkpoint 1 ALBERT-base-v2 Lan et al. (2020) 11.811.8M albert/albert-base-v2 2 ELECTRA-small Clark et al. (2020) 14.014.0M google/electra-small-discriminator 3 MobileBERT Sun et al. (2020) 25.325.3M google/mobilebert-uncased 4 DistilBERT-base Sanh et al. (2019) 66.466.4M distilbert/distilbert-base-uncased 5 BERT-base Devlin et al. (2019) 109.5109.5M google-bert/bert-base-uncased 6 XLNet-base Yang et al. (2019) 110.0110.0M xlnet/xlnet-base-cased 7 RoBERTa-base Liu et al. (2019) 124.6124.6M FacebookAI/roberta-base 8 DeBERTa-v3-small He et al. (2023) 141.9141.9M microsoft/deberta-v3-small Appendix E Ranking Preservation Details For each benchmark we fit test-tuned on validation-tuned test accuracy by ordinary least squares on the per-architecture means, and report Spearman and Kendall rank correlations on those same means. All confidence intervals are based on bootstrap resampling. For the linear fit we use a standard pairs bootstrap with 100,000100,000 resamples, drawing the eight architecture mean pairs with replacement and refitting the line at each iteration. The rank correlations use a cluster bootstrap with 20,00020,000 resamples, drawing the K runs within each architecture with replacement and recomputing the architecture means at each iteration. E.1 MNIST-1D Linear fit: • Slope =0.960=0.960, 95%95\,\% bootstrap CI: [0.893, 0.988][0.893,\;0.988] • Intercept =3.534=3.534, 95%95\,\% bootstrap CI: [1.885, 7.708][1.885,\;7.708] • R2=0.997R^2=0.997, 95%95\,\% bootstrap CI: [0.992, 1.000][0.992,\;1.000] • p-value =5.73×10−9=5.73× 10^-9 Rank correlations: • Spearman ρ=1.000ρ=1.000 (p≈0p≈ 0), 95%95\,\% bootstrap CI: [0.929, 1.000][0.929,\;1.000] • Kendall τ=1.000τ=1.000 (p=4.96×10−5p=4.96× 10^-5), 95%95\,\% bootstrap CI: [0.857, 1.000][0.857,\;1.000] E.2 CIFAR-10 Linear fit: • Slope =0.973=0.973, 95%95\,\% bootstrap CI: [0.942, 1.001][0.942,\;1.001] • Intercept =2.631=2.631, 95%95\,\% bootstrap CI: [0.043, 5.544][0.043,\;5.544] • R2=0.999R^2=0.999, 95%95\,\% bootstrap CI: [0.998, 1.000][0.998,\;1.000] • p-value =1.08×10−10=1.08× 10^-10 Rank correlations: • Spearman ρ=1.000ρ=1.000 (p≈0p≈ 0), 95%95\,\% bootstrap CI: [0.976, 1.000][0.976,\;1.000] • Kendall τ=1.000τ=1.000 (p=4.96×10−5p=4.96× 10^-5), 95%95\,\% bootstrap CI: [0.929, 1.000][0.929,\;1.000] E.3 RTE Linear fit: • Slope =0.916=0.916, 95%95\,\% bootstrap CI: [0.885, 1.007][0.885,\;1.007] • Intercept =7.523=7.523, 95%95\,\% bootstrap CI: [1.202, 9.531][1.202,\;9.531] • R2=0.997R^2=0.997, 95%95\,\% bootstrap CI: [0.992, 1.000][0.992,\;1.000] • p-value =1.06×10−8=1.06× 10^-8 Rank correlations: • Spearman ρ=1.000ρ=1.000 (p≈0p≈ 0), 95%95\,\% bootstrap CI: [0.905, 1.000][0.905,\;1.000] • Kendall τ=1.000τ=1.000 (p=4.96×10−5p=4.96× 10^-5), 95%95\,\% bootstrap CI: [0.786, 1.000][0.786,\;1.000] E.4 MRPC Linear fit: • Slope =0.910=0.910, 95%95\,\% bootstrap CI: [0.850, 0.960][0.850,\;0.960] • Intercept =8.787=8.787, 95%95\,\% bootstrap CI: [4.610, 13.889][4.610,\;13.889] • R2=0.994R^2=0.994, 95%95\,\% bootstrap CI: [0.982, 0.999][0.982,\;0.999] • p-value =6.58×10−8=6.58× 10^-8 Rank correlations: • Spearman ρ=0.976ρ=0.976 (p=3.31×10−5p=3.31× 10^-5), 95%95\,\% bootstrap CI: [0.905, 1.000][0.905,\;1.000] • Kendall τ=0.929τ=0.929 (p=3.97×10−4p=3.97× 10^-4), 95%95\,\% bootstrap CI: [0.786, 1.000][0.786,\;1.000] The single rank change is an exchange between mobilebert and electra-small, whose validation-tuned means differ by 0.30%0.30\% against a run-to-run standard deviation of 1.34%1.34\%. E.5 CoLA Linear fit: • Slope =1.020=1.020, 95%95\,\% bootstrap CI: [0.956, 1.157][0.956,\;1.157] • Intercept =−1.040=-1.040, 95%95\,\% bootstrap CI: [−12.260, 4.374][-12.260,\;4.374] • R2=0.992R^2=0.992, 95%95\,\% bootstrap CI: [0.985, 1.000][0.985,\;1.000] • p-value =1.47×10−7=1.47× 10^-7 Rank correlations: • Spearman ρ=0.976ρ=0.976 (p=3.31×10−5p=3.31× 10^-5), 95%95\,\% bootstrap CI: [0.952, 1.000][0.952,\;1.000] • Kendall τ=0.929τ=0.929 (p=3.97×10−4p=3.97× 10^-4), 95%95\,\% bootstrap CI: [0.857, 1.000][0.857,\;1.000] The single rank change is an exchange between xlnet-base and distilbert, whose validation-tuned means differ by 0.14%0.14\% against a run-to-run standard deviation of 0.88%0.88\%. Appendix F Upper-Bound Reference Figure 7: Upper-bound reference per architecture across the five benchmarks, ordered by mean validation-tuned test accuracy. Blue circles are the validation-tuned mean test accuracy and orange squares the test-tuned mean test accuracy (both with 95%95\,\% bootstrap CIs over the K=30K=30 runs). Orange triangles are the upper bound: the best test accuracy among T=30T=30 models trained directly on the same bootstrap test fold. The vertical distance between the blue marker and the orange triangle is the headroom available to any procedure that exploits the test set; the much smaller distance between the blue circle and orange square is the share that random search over T=30T=30 trials actually captures. Benchmark Upper bound (%) Headroom (%) Captured (%) MNIST-1D 100.00100.00 35.435.4–46.046.0 2.02.0–3.73.7 CIFAR-10 100.00100.00 4.64.6–7.77.7 1.21.2–2.42.4 RTE 99.8099.80–100.00100.00 26.426.4–40.940.9 4.44.4–6.96.9 MRPC 99.0699.06–100.00100.00 13.613.6–18.718.7 6.96.9–9.69.6 CoLA 98.8798.87–100.00100.00 13.813.8–19.719.7 1.01.0–5.55.5 Table 11: Ranges over the eight architectures of each benchmark. Upper bound is the best test accuracy among T=30T=30 models trained and tuned directly on the bootstrap test fold. Headroom is the distance from the validation-tuned mean to that bound, and captured is the share of it that test-tuning actually realises, (A(λtest∗)−A(λval∗))/headroom(A(λ^*_test)-A(λ^*_val))/headroom. Every architecture very nearly memorises its own test fold: exactly 100%100\% on both image benchmarks, and between 98.9%98.9\% and 100%100\% on the GLUE tasks, whose bootstrap test folds are smaller (∼509 509 examples on RTE to ∼1,763 1,763 on CoLA). Even MLP-Tiny has enough capacity to memorise the ∼736 736-sample MNIST-1D fold. Headroom above the validation-tuned mean varies widely across benchmarks, from 4.64.6–7.7%7.7\% on CIFAR-10, where validation-tuned accuracy is already in the low-90%90\%s, up to 3535–46%46\% on MNIST-1D. Yet in every case random search at T=30T=30 captures at most 9.6%9.6\% of it, so the observed gaps lie far below what any procedure exploiting the test set could achieve under this budget. Appendix G MLP-GiantT=100: Adaptive Overfitting Under Increased Tuning and Data Budget Figure 8: MLP-Giant rerun with T=100T=100 trials per run on larger 10,000/10,000/10,00010,000/10,000/10,000 MNIST-1D splits. Left: per-run ΔAO _AO across the K=30K=30 paired runs; the blue dot is the mean and the red horizontal bar marks the standard deviation of val-tuned accuracy. Right: P^(ΔAO>0) P( _AO>0) as a function of hyperparameter tuning budget T, with the 95%95\,\% percentile-bootstrap CI; the dashed grey reference at 0.50.5 is the significance threshold. The mean gap stays small relative to the val-tuned noise and P^(ΔAO>0) P( _AO>0) remains significant (CIlower>0.5CI_lower>0.5) across the full budget range. Appendix H Adaptive Overfitting Transfer to CIFAR-10.1 Model Val-tuned (%) Test-tuned (%) CIFAR-10.1-tuned (%) on CIFAR-10.1 on CIFAR-10.1 on CIFAR-10.1 shake_shake_32d 88.7988.79 88.8288.82 89.1189.11 resnet9 86.9586.95 86.9386.93 87.4187.41 mobilenetv2 87.1187.11 87.0687.06 87.2987.29 resnet_basic_56 86.0786.07 86.1186.11 86.3986.39 resnet_basic_44 85.9585.95 85.7885.78 86.2686.26 vgg_15_BN_64 85.4085.40 85.3485.34 85.8685.86 resnet_basic_32 85.2385.23 85.1185.11 85.6985.69 resnet_basic_20 83.8183.81 83.8183.81 84.3684.36 Pooled 86.1686.16 86.1286.12 86.5586.55 Table 12: Per-architecture mean CIFAR-10.1 accuracy under three selection rules. Val-tuned and Test-tuned pick the trial with the highest accuracy on the bootstrap validation or test fold, respectively, and the picked configuration is then re-evaluated on CIFAR-10.1; CIFAR-10.1-tuned picks the trial with the highest CIFAR-10.1 accuracy directly. “Pooled” is the mean across the eight architectures. The val-tuned and test-tuned columns are indistinguishable on CIFAR-10.1 (pooled gap −0.04%-0.04\%, within run-to-run noise), consistent with the small ΔAO≈0.12% _AO≈ 0.12\% already observed on the bootstrap test fold. CIFAR-10.1-tuned wins by +0.43%+0.43\% pooled — a small but consistent margin that the bootstrap-test-tuned configuration does not capture. Appendix I Data Splitting We use the same splitting scheme on all five benchmarks. We start from a single source pool: 4,0004,000 examples for MNIST-1D, the 60,00060,000-example pool that combines CIFAR-10’s training and test sets, and for the GLUE tasks the concatenation of the official train and validation splits (2,7672,767 for RTE, 4,0764,076 for MRPC and 9,5949,594 for CoLA). For each paired run we draw the train set by sampling from the pool with replacement, taking as many samples as the pool size. Examples that were never picked end up in the leftover set, which on average is about 36.8%36.8\% of the pool. We shuffle this leftover set and split it in half: one half is the validation set and the other is the test set. For MNIST-1D this gives roughly 4,0004,000 training, 736736 validation and 736736 test examples per run, and for CIFAR-10 roughly 60,00060,000 training, 11,04011,040 validation and 11,04011,040 test examples. For the GLUE tasks it gives roughly 509509 validation and 509509 test examples on RTE, 750750 and 750750 on MRPC, and 1,7631,763 and 1,7631,763 on CoLA. The validation and test sets never overlap, and they never share an example with the training set. The split is drawn once at the start of a run and stays the same across all T trials, so validation-tuning and test-tuning pick from the same pool of trained models. A new split is drawn for each of the K=30K=30 paired runs. We use the pooled train and validation splits for the GLUE tasks because the official GLUE test splits are unlabeled and withheld on the leaderboard server, so the validation split is the de facto test set in community practice. The MLP-Giant rerun in Appendix G uses a different scheme. It draws three disjoint sets of 10,00010,000 examples each (train, validation, test) from a larger 70,00070,000-example MNIST-1D set, without replacement. Appendix J Hyperparameter Range Validity The empirical winner distributions in Figures 9 and 10 suggest the search ranges are well-shaped: lr, weight_decay, and momentum have their modes inside the bounds with no edge clustering. The one exception is CIFAR-10’s max_steps, whose winners pile up at the upper bound of 141,000141,000; extending the range would benefit both selection rules equally and is unlikely to qualitatively change ΔAO _AO. For the GLUE tasks the ranges were instead set a priori to bracket the published fine-tuning grids of all eight model families (Table 7), with one deliberate exception: ELECTRA-small’s published 3×10−43× 10^-4 recipe lies above our 10−410^-4 learning-rate cap, which is kept below the divergence region of the base-size models. The winner distributions in Figures 11–13 confirm those ranges empirically: lr peaks around 33–5×10−55× 10^-5, where the published recipes sit, while weight_decay and warmup_ratio are close to flat, indicating that neither strongly determines which configuration wins. As with CIFAR-10’s max_steps, only num_epochs clusters at its upper bound of 1010, and extending it would again benefit both selection rules equally. Figure 9: MNIST-1D: empirical distribution of winning hyperparameter values for the validation-tuned and test-tuned selection rules. 30.8%30.8\% of trials had lr_decay=0 lr\_decay=0 and are excluded from the lr_decay panel. All three HPs have their modes well inside the bounds with no edge clustering. Figure 10: CIFAR-10: empirical distribution of winning hyperparameter values for the val-tuned and test-tuned selection rules. lr, weight_decay, and momentum have their modes well inside the bounds; only max_steps clusters at the upper bound of 141,000141,000, suggesting the optimum lies above this cap. Extending it would raise absolute accuracies but is unlikely to change ΔAO _AO, since both selection rules benefit equally from longer training. Figure 11: RTE: empirical distribution of winning hyperparameter values for the val-tuned and test-tuned selection rules, over the 240240 winners of each rule (88 architectures × K=30K=30 runs). num_epochs is binned on the integers 22–1010. Figure 12: MRPC: empirical distribution of winning hyperparameter values for the val-tuned and test-tuned selection rules, over the 240240 winners of each rule. Figure 13: CoLA: empirical distribution of winning hyperparameter values for the val-tuned and test-tuned selection rules, over the 240240 winners of each rule. As on the other two GLUE tasks, num_epochs rises towards the upper bound of the range.