Paper deep dive
Ctrl-A: Control-Driven Online Data Augmentation
Jesper B. Christensen, Ciaran Bench, Spencer A. Thomas, Hüsnü Aslan, David Balslev-Harder, Nadia A. S. Smith, Alessandra Manzin
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/26/2026, 2:34:14 AM
Summary
Ctrl-A (ControlAugment) is an automated data augmentation algorithm for image-vision tasks that utilizes control theory to dynamically adjust augmentation strength distributions during model training. By monitoring the relative operation response and maintaining a balance between training and validation loss, Ctrl-A eliminates the need for manual policy engineering and initialization, achieving competitive performance with state-of-the-art methods like TrivialAugment.
Entities (5)
Relation Signals (3)
Ctrl-A → evaluatedon → CIFAR-10
confidence 95% · Experiments on the CIFAR-10, CIFAR-100, and SVHN-core benchmark datasets... demonstrate that Ctrl-A is highly competitive
Ctrl-A → utilizes → control theory
confidence 95% · incorporates principles from control theory for online adjustment of augmentation strength distributions
Ctrl-A → improves → WideResNet-28-10
confidence 90% · demonstrate that Ctrl-A is highly competitive with existing state-of-the-art data augmentation strategies
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce ControlAugment (Ctrl-A), an automated data augmentation algorithm for image-vision tasks, which incorporates principles from control theory for online adjustment of augmentation strength distributions during model training. Ctrl-A eliminates the need for initialization of individual augmentation strengths. Instead, augmentation strength distributions are dynamically, and individually, adapted during training based on a control-loop architecture and what we define as relative operation response curves. Using an operation-dependent update procedure provides Ctrl-A with the potential to suppress augmentation styles that negatively impact model performance, alleviating the need for manually engineering augmentation policies for new image-vision tasks. Experiments on the CIFAR-10, CIFAR-100, and SVHN-core benchmark datasets using the common WideResNet-28-10 architecture demonstrate that Ctrl-A is highly competitive with existing state-of-the-art data augmentation strategies.
Tags
Links
- Source: https://arxiv.org/abs/2603.21819v1
- Canonical: https://arxiv.org/abs/2603.21819v1
Trouble viewing inline? Open PDF directly →
Full Text
77,874 characters extracted from source content.
Expand or collapse full text
Ctrl-A: Control-Driven Online Data Augmentation Jesper B. Christensen1, Ciaran Bench2, Spencer A. Thomas2, Hüsnü Aslan1, David Balslev-Harder1, Nadia A. S. Smith3,4, and Alessandra Manzin5 Abstract We introduce ControlAugment (Ctrl-A), an automated data augmentation algorithm for image-vision tasks, which incorporates principles from control theory for online adjustment of augmentation strength distributions during model training. Ctrl-A eliminates the need for initialization of individual augmentation strengths. Instead, augmentation strength distributions are dynamically, and individually, adapted during training based on a control-loop architecture and what we define as relative operation response curves. Using an operation-dependent update procedure provides Ctrl-A with the potential to suppress augmentation styles that negatively impact model performance, alleviating the need for manually engineering augmentation policies for new image-vision tasks. Experiments on the CIFAR-10, CIFAR-100, and SVHN-core benchmark datasets using the common WideResNet-28-10 architecture demonstrate that Ctrl-A is highly competitive with existing state-of-the-art data augmentation strategies. 1 Introduction Image-based computer vision (ICV) is one of the fields within computer science that has experienced a significant advancement with the rapid evolution of artificial intelligence (AI) [46]. AI-powered solutions, based on deep neural network (DNN) architectures, find applications within automated medical image processing (radiology, histopathology, radiotherapy planning etc.) [29, 32], in biometric facial recognition (for both personal security and criminology) [23], and for quality-assurance inspection systems within Industry 4.0 [31]. The DNN models typically consist of tens or hundreds of millions of parameters to effectively accomplish the complex image processing and analysis tasks. However, their effectiveness depends on access to large-scale training datasets, as insufficient data can lead to overfitting, where the model learns dataset-specific patterns and fails to generalize to new unseen inputs. A method that has proven to be highly efficient in mitigating overfitting effects and improving generalizability of DNN models in ICV tasks is that of data augmentation (DA) [33, 27, 42, 8, 6, 38]. DA employs various geometric transformations, color-based transformations, or adversarial changes to artificially increase the diversity of the training images presented to the model. Adversarial approaches focus on generating locally smooth perturbations to data points during training as a means of model regularization [8, 24]. This class of augmentations can generate data-agnostic perturbations that enforce invariant representations of the augmented data point(s), thereby improving model robustness [28]. The geometric and color-based transformations, on the other hand, are global transformations (such as rotations, translations, brightness, and contrast enhancements) [15]. The global nature of these transformations significantly enhances training data diversity, while assuming label preservation, and has been demonstrated to enhance model performance and serve as an efficient method for mitigating overfitting effects [27, 44]. Beyond ICV tasks, DA techniques have also shown effectiveness in non-vision tasks [37, 30, 39, 40], making it a widely applicable strategy for enhancing the practical utility of deep learning models. Yet, DA methods based on geometric and color-based transformations are often chosen based on prior studies that have demonstrated certain augmentation policies to be efficient, rather than a principled approach in which augmentations are tailored to the task at hand. Manually selecting the relevant transformations and their strengths is also a task that requires expert domain knowledge or a highly time-consuming and computationally expensive trial-and-error process. These challenges led to the development of automatic DA strategies, such as AutoAugment (A) [3], which demonstrated the possibility of learning augmentation sub-policies from a proxy AI task. Today, A is one of many automatic DA strategies that have been developed to enhance AI model performance and alleviate the need for manual selection of augmentation types. Still, most of the algorithms developed to date suffer from significant computational overhead related to determining desired augmentation policies and augmentation strengths. In this work, we take an alternative approach to the challenge of automated DA. Our algorithm, named ControlAugment (Ctrl-A), utilizes the regularizing effect of DA, and incorporates concepts from control theory, to individually update augmentation strengths for each augmentation type as model training progresses through different training phases. Ctrl-A provides only a marginal computational overhead, almost matching that of TrivialAugment [26], and no specific initialization of the augmentation strengths is required. In particular, this paper makes the following contributions to the field of automatic DA: • Drawing on concepts from control theory, we formulate an online DA strategy that serves as an adaptive model regularizer by dynamically controlling the training-to-validation loss ratio during learning. • We introduce parametrized augmentation-strength distributions and propose an online update procedure for their parameters, guided by a newly defined concept termed relative operation response, which quantifies a model’s response to individual operations. • We identify that standard WideResNet-28-10 training setups on CIFAR and SVHN-core are constrained by hyperparameter choices, which prevent efficient differentiation among DA methods. Consequently, we show that careful training setup selection is essential for enabling informative and fair comparisons between augmentation methods and avoiding misleading conclusions. 2 Related Work Advances in automated DA have significantly improved the generalization capacity of DNNs for ICV tasks. Before introducing Ctrl-A in detail, we review some of the key algorithms that have reduced the need for manual augmentation engineering and design, and therefore serve as foundation to our work. 2.1 Offline methods We first highlight automated DA approaches that fall into the category of offline method, which do not adapt augmentation policies dynamically during the training phase. Instead, these approaches rely on a preceding search procedure to determine fixed augmentation policies (or policy schedules) that are subsequently used during the actual model training. One of the most influential methods in automated DA is AutoAugment (A) [3], which formulates augmentation policy search as a discrete optimization problem, using reinforcement learning to discover effective transformation strategies. While novel, the computational overhead in finding paired augmentations (sub-policies) scales poorly due to the exponentially growing search space, defined by the number of transformations and augmentation strength levels. To reduce the computational overhead in learning relevant DA policies, several follow-up methods were developed, including Fast AutoAugment (Fast-A) [17], Faster AutoAugment (Faster-A) [9], Deep AutoAugment [47], and Population-based Augmentation (PBA) [10]. Both Fast- and Faster-A avoid the use of reinforcement learning and replace it with more efficient methods. Fast-A employs density matching and attempts to find augmentation policies that make the training data resemble a held-out validation set [17]. Faster-A, on the other hand, shares similarities with other differentiable methods [16, 19] by formulating augmentation search as a continuous gradient-based optimization problem [9]. PBA takes a different approach as it trains a population of models in parallel, allowing them to investigate different augmentation policies, and incorporates model-parameter sharing at specific training intervals to transfer beneficial augmentation strategies [10]. This enables PBA to apply different augmentation policies at different training stages, as one global policy is unlikely to be optimal. Like Fast-A and Faster-A, PBA improves the efficiency of automated DA learning while retaining the bi-operation sub-policy structure introduced in A [3]. In contrast to the algorithms mentioned above, which rely on pre-defined bi-augmentation sub-policies of fixed augmentation magnitudes for model training, RandAugment (RA) [4] and TrivialAugment (TA) [26] introduce a simpler approach to automated DA. Rather than solving an optimization problem or a proxy reinforcement learning task to determine efficient DA sub-policies, RA reduces the task to a simple two-dimensional search over the number of operations N and their (common) strength M [4]. With this approach, N operations are randomly selected, for each training sample, and applied with augmentation strength M, thereby emphasizing training data diversity. Thus, instead of employing predefined augmentation sub-policies, RA focuses on maximizing training-data diversity which forces the model to generalize and prevents overfitting to curated augmentation sub-policies. This concept of diversity-enhancement is also exploited in TA, which simplifies RA to only a single augmentation type, with a randomly sampled strength per training sample [26]. Despite this simplification, the simple TA method largely demonstrates similar performance enhancements to search-optimized RA in select experiments [26]. 2.2 Online methods The recent trend within automatic DA is a shift towards online DA learning, in which the employed augmentation policy is not fixed a priori but instead continuously updated throughout model training. Many different update criteria have already been formulated, including the algorithms Meta AutoAugment (Meta-A) [48], Online Hyperparameter Learning (OHL) [18], Universal Adaptive Data Augmentation (UADA) [43], OnlineAugment [35], RangeAugment [22], Diversity-based data Augmentation (DivAug) [20], and Invariance-Constrained Learning (ICL) [11]. Meta-A jointly optimizes model parameters and DA policies using implicit gradient-based optimization [48]; OHL formulates DA learning as a hyperparameter-learning task [18]; UADA maximizes training loss with an adversarial approach to updating DA parameters [43]; OnlineAugment employs augmentation models which learn efficient augmentations while being trained in alternation with the target model [35]; DivAug formulates a metric “variance diversity” which, based on the model state, is used to enhance augmentation diversity of the training data [20]; and ICL formulates DA as an invariance-constrained learning problem which is solved by Markov-Chain Monte Carlo sampling and avoids undesired augmentations [11]. In Ctrl-A, we combine the large training data diversity provided by RA and TA with aspects from control theory to realize adaptable online DA as a method for controllable model regularization. In our approach, the distribution of available augmentation strengths of each operation is updated online, and individually, as training progresses. To achieve this, we assign an additional role to the validation dataset, which plays a prominent role in our framework. 3 ControlAugment 3.1 Setting We consider the standard supervised image classification task, in which each image x∈x in the dataset ⊂ℝn×m×cX ^n× m× c is associated with a single class label y∈=1,…,Cy =\1,...,C\, with c denoting number of (color) input channels and C being the number of output labels. In the simplest possible case, the paired dataset is divided into a training set Train=(xTrain,yTrain)D_Train=(x_Train,y_Train) and a test set Test=(xTest,yTest)D_Test=(x_Test,y_Test). In this case, our goal is to learn model parameters θ for a chosen model fθ:→f_θ:X , which generalizes such that it approximately minimizes the expectation value of a loss function, ℒ(fθ(xTest),yTest)L(f_θ(x_Test),y_Test), while only allowing the model to be explicitly trained on dataset TrainD_Train. In addition to the training and test datasets, a validation dataset, Val=(xVal,yVal)D_Val=(x_Val,y_Val), is commonly used in practice during training to gauge model performance on data that the model is not explicitly trained on. As we describe shortly, the validation dataset plays an active role in the Ctrl-A framework. 3.2 Concept and framework Figure 1: Illustration of the ControlAugment framework. Model training proceeds in phases of npn_p epochs, separated by the ControlAugment algorithm, which regulates the informed augmentation pool by adjusting the augmentation strength parameters and α for the subsequent training phase. The adjustment is executed by the ControlAugment block which quantifies the response of each operation using augmented validation data and through an internal control parameter (ξ) which is updated by comparing the setpoint value (κsp _sp) with a relative training/validation performance metric (κ). Our automatic DA-based training procedure for supervised learning is illustrated in Fig. 1. Central to the procedure is the conception of an “informed augmentation pool” of size K, which provides augmented training samples for model training. The informed augmentation pool contains the same operations as the “full augmentation pool”, but with augmentation strengths that are periodically adapted by the ControlAugment block. The full augmentation pool may correspond to the standard one introduced for RA [4], the wide one used for TA [26], or the one introduced in this work (control) containing the set of operations O listed and parametrized in Appendix A, Table 5. In Ctrl-A, each operation i∈O_i is parametrized with a normalized augmentation strength γi∈[0,1] _i∈[0,1], for i∈1,2,…,Ki∈\1,2,...,K\. Moreover, an operation, applied to image x as i(x;γi)O_i(x; _i), is characterized by (1) yielding a progressively stronger perturbation to image x as γi _i increases from zero to unity, and (2) simplifying to the identity operator in the limit of vanishing augmentation strength, i.e., limγi→0i(x;γi)=x _ _i→ 0O_i(x; _i)=x. During model training using Ctrl-A, every single training sample is an augmented version, xAx_A, of original training data, x. A single image augmentation, x_A, is obtained by (1) randomly selecting N of the K operations (sampling without replacement) with integer indices i1,i2,…,iNi_1,i_2,...,i_N from O, (2) randomly drawing for each index, ini_n, the operations’ augmentation strength γin∼Uαin(0,Γin) _i_n U_ _i_n(0, _i_n), and finally (3) applying composite operations x=iN(…(i2(i1(x;γi1);γi2)…);γiN).x_A=O_i_N(...(O_i_2(O_i_1(x; _i_1); _i_2)...); _i_N). (1) We refer to the case of applying N operations as CtrlA(N), and note that different operations, in general, do not commute, which further enhances training-data diversity for N>1N>1. Figure 2: Examples of augmentation strength distributions with increasing distribution means from left to right. The probability distribution, Uα(0,Γ)U_α(0, ), from which individual augmentation strength are drawn, plays an important role in Ctrl-A, and we refer to it as the augmentation strength distribution (ASD). It is parametrized by two values, the distribution upper bound, Γ∈[0,1] ∈[0,1], and the skewness value, α∈[0,1]α∈[0,1]. For α=0α=0, U0(0,Γ)=U(0,Γ)U_0(0, )=U(0, ) is the uniform distribution with continuous support [0,Γ][0, ], and for α=1α=1, U1(0,Γ)U_1(0, ) is the triangular distribution with mode Γ . Examples of Uα(0,Γ)U_α(0, ) are visualized in Fig. 2, demonstrating the capability of the ASD to shift weight toward larger augmentation strengths as Γ and α are increased. The distribution mean is given by (1+α/3)Γ/2(1+α/3) /2 and is thereby a growing function of both Γ and α. As a result, the ensemble strength of DA, for each operation iO_i, can be controlled through the individual values for Γi _i and αi _i. Notably, in the special case of = = 1, = α=0, and N=1N=1, we retrieve the constant setting applied in TrivialAugment [26]. Finally, Ctrl-A partitions model training into training phases, indexed j, with each phase lasting npn_p epochs. A phase is characterized by constant vectors =(Γ1,Γ2,…,ΓK) =( _1, _2,..., _K) and =(α1,α2,…,αK) α=( _1, _2,..., _K), and is concluded by re-evaluating and α ahead of the following training phase. This re-evaluation enables the degree of DA to be dynamically updated during training, and is discussed next. 3.3 Updating the ASDs In succession to every training phase, we employ a procedure for computing (j+1) ^(j+1) and (j+1) α^(j+1), resulting in an updated set of ASDs, i.e. Uαi(0,Γi)U_ _i(0, _i) for i∈1,2,…,Ki∈\1,2,...,K\. The determination of (j+1) ^(j+1) and (j+1) α^(j+1) relies on the introduction of the relative operation response (ROR), Ri(γ)R_O_i(γ) defined as Ri(γi)=Accf(i(Val;γi))Accf(Val),R_O_i( _i)= Acc_f(O_i(D_Val; _i))Acc_f(D_Val), (2) in which the numerator, Accf(i(Val;γi))Acc_f(O_i(D_Val; _i)), is the augmented classification accuracy, obtained by computing the validation accuracy after the operation i(⋅;γi)O_i(\,·\,; _i) has been applied, with strength γ, to the entire dataset. Due to the parametrization of all operations, the ROR immediately satisfies Ri(0)=1R_O_i(0)=1 for all i∈O_i , and its deviation from unity with increasing γ quantifies the effect of each operation type. 3.3.1 Determining (j+1) ^(j+1) The ASD mean depends on Γ to first order, and so Γ is also considered the primary ASD parameter. For each i∈O_i , the upper bound parameter Γi(j+1) _i^(j+1) is set to the value γi _i that solves the implicit equation Ri(γi)−ξ(j+1)=0,R_O_i( _i)-ξ^(j+1)=0\,, (3) where ξ∈[0,1]ξ∈[0,1] is a control parameter (see Sec. 3.4). Equation 3 has a simple graphical explanation as demonstrated in Fig. 3, and its solution is estimated, using regression analysis, after evaluating the ROR for γi=0,Δγ,2Δγ,…,1 _i=\0, γ,2 γ,…,1\. With this formulation, the available augmentation strengths in the next training phase are individually tailored depending on how sensitive the model and data is to each individual operation. Cases in which min(Ri(γi))>ξ(j+1) (R_O_i( _i))>ξ^(j+1) are dealt with by setting Γi(j+1)=1 _i^(j+1)=1. Further details regarding the practical implementation are provided in Appendix B. 3.3.2 Determining (j+1) α^(j+1) The ASD mean does not have a leading-order dependence on α, so α can also be referred to as the secondary ASD parameter. Similarly to Γi(j+1) _i^(j+1), the determination of αi(j+1) _i^(j+1) depends on the control parameter ξ(j+1)ξ^(j+1) and is computed according to the heuristic rule αi(j+1)=Ri(γi=1)−ξ(j+1)1−ξ(j+1), _i^(j+1)= R_O_i( _i=1)-ξ^(j+1)1-ξ^(j+1)~, (4) and clipped to the unit interval [0,1][0,1] when necessary. With this definition, which is also visualized in Fig. 3, we choose to keep αi(j+1)=0 _i^(j+1)=0 unless Γi(j+1)=1 _i^(j+1)=1, i.e. adding a skew to the ASD only if Ri(1)>ξ(j+1)R_O_i(1)>ξ^(j+1). The maximum skew of αi=1 _i=1 is obtained if the model performance is unaffected (or improved) by operation iO_i, i.e. if Ri(1)≥Ri(0)=1R_O_i(1)≥ R_O_i(0)=1 Figure 3: Illustration of the Ctrl-A update procedure (Eqs. 3 and 4), which relies on relative operation response curves to determine new values for the ASD parameters Γ and α based on the value of the control parameter, ξ. Implicitly, Γ3=1 _3=1 and α1=α2=0 _1= _2=0. 3.4 Parameter control algorithm The parameter ξ(j)ξ^(j) controls the strength of DA in phase j through Eqs. 3 and 4. A low value for the parameter results in stronger perturbations of the training data, whereas a value close to unity leads to weak augmentations. The optimal region of ξ is expected to depend both on task, model architecture, number of operations, training state, and dataset representativeness and size, and it is therefore necessary to adjust the parameter as model training progresses. To achieve online adaptation, we make use of two well-known properties of DA, namely that (1) DA generally has a regularizing effect and helps prevent overfitting the model to training data [20], and (2) applying overly strong augmentations can homogenize data across classes, increasing inter-class overlap and potentially leading to implicit class mixing, which can degrade model performance [33]. In an attempt to find a compromise between these properties, we formulate the following minimization task minξ|ℒf(Trainaug)¯−κspℒf(Val)¯|, _ξ~ | L_f (D_Train^aug )- _sp L_f(D_Val) |, (5) where ℒL is a non-negative convex loss function used for model training, e.g. the cross entropy loss, the bar notation, ∙¯ , represents a training-phase average, and κsp _sp is a user-defined non-negative control setpoint that determines the desired imbalance between model training loss and model validation loss. Based on the value of κsp _sp, we refer to three different regimes of strong augmentation (κsp>1 _sp>1), balanced augmentation (κsp∼1 _sp 1), and weak augmentation (κsp<1 _sp<1). The latter case (weak) can lead to overfitting to the training data, whereas the strongly augmented case would potentially result in poor model performance due to undesired class mixing or model inadequacy. In carrying out the minimization in Eq. 5, we utilize that the magnitude of ℒf(Trainaug)L_f (D_Train^aug ) can broadly be controlled by tuning the strength of DA. At the end of each training phase, we therefore inform the augmentation strengths of the next phase by updating ξ according to ξ(j+1)=ξ(j)+Δξ=ξ(j)+Kg(κ(j)−κsp),ξ^(j+1)=ξ^(j)+ ξ=ξ^(j)+K_g (κ^(j)- _sp ), (6) where κ(j)=ℒf(Trainaug)¯/ℒf(Val)¯,κ^(j)= L_f (D_Train^aug )/ L_f(D_Val), (7) and Kg>0K_g>0 is the tuning rate, acting as the gain factor of the Ctrl-A algorithm. With Eq. 6 we obtain the following: if the phase-averaged training loss exceeds the phase-averaged validation loss (scaled by κsp _sp), Δξ ξ is positive (leading to weaker augmentations). Conversely, if the training loss is lower than the scaled validation loss, Δξ ξ is negative (leading to stronger augmentations). After the update of ξ (bound to the unity interval), the ASD parameters and α are computed for the next training phase. 3.5 An extra role for the validation dataset In the common AI setting, the validation dataset, ValD_Val, is used to monitor the generalization performance of the model fθf_θ during training. It provides a basis for hyperparameter tuning, model selection, and techniques such as early stopping, which help detect and mitigate overfitting to the training dataset. Within the Ctrl-A framework, the validation dataset is given a more prominent and active role. Firstly, it is used to gauge model sensitivity to the K augmenting operations, forming the basis for the update procedure for the ASD parameters and α. Notably, the training data is deliberately not used for this task, as the model is precisely trained using augmented training data. Secondly, the validation dataset enters in the minimization procedure through Eq. 5. Here, DA is used for tunable regularization to dynamically balance the ratio between training loss and validation loss to a predefined setpoint, κsp _sp. We emphasize that neither of these extra roles causes overfitting of the model fθf_θ to the validation data, which we demonstrate in both the experimental section and Appendix C. 3.6 ControlAugment in summary The Ctrl-A algorithm, in essence, functions as a control loop that continuously attempts to balance training and validation loss through DA regularization. The control loop follows a repetitive cycle consisting of the following three steps: Step (i): The model fθf_θ is trained using augmented versions of the training dataset TrainD_Train. Training proceeds for npn_p epochs, constituting a training phase. Step (ii): The control parameter ξ is updated according to Eq. 6. Step (iii): By evaluating the model fθf_θ on different degrees of augmented versions of the validation dataset ValD_Val, the ASD parameters, i _i and i α_i, to be used in the following training phase, are determined by solving Eqs. 3 and 4 for every operation iO_i. Steps (i)-(iii) are iterated until a predefined number of epochs has been reached. 4 Experimental design and results Framework and code. Experiments are conducted using the PyTorch deep learning framework, with DA implemented through a customized version of the torchvision transformation library to support implementation of Ctrl-A. Code and implementation details are available from our GitHub repository [2]. Experimental plan. The experiments are divided into two phases: the parameter initialization phase (Section 4.1) and the performance phase (Sections 4.2-4.5). In the initial phase, we use a shallow benchmarking model (the airbench-94, [12]) trained on CIFAR-10 to learn about the dynamics of the Ctrl-A algorithm and its dependence on N and κsp _sp. In the performance phase, the findings from the initial phase are directly employed to test the capacity of the Ctrl-A algorithm with the deeper Wide-ResNet-28-10 (WRN-28-10) model [45]. State-of-the-art benchmarks for CIFAR and SVHN-core are used for comparison to assess the performance of Ctrl-A. Training hyperparameters and scheduling. For model training we use the cross-entropy loss function, and a stochastic gradient descent (SGD) optimizer with weight decay and Nesterov momentum. Training lasts nmaxn_max epochs and uses a cosine learning-rate schedule of the form η(n)=η02(1+cos(π(n−1)nmax)),η(n)= _02 (1+~cos ( π(n-1)n_max ) ), (8) in which n is the epoch number and η0 _0 is the initial learning rate [21]. After training for nmaxn_max epochs, the model is evaluated on the test set resulting in per-run performance metrics. Figure 4: CIFAR-10 test accuracy (model: airbench-94) as a function of the control setpoint κsp _sp for (a) CtrlA(11), (b) CtrlA(22), and (c) CtrlA(33). Search-optimized RA is represented by the shaded region (95%95\,\% confidence interval). Model regularization. To emphasize the effect of Ctrl-A, we exclude dropout functionality [34], and we avoid the use of ℓ1 _1- and ℓ2 _2-norm regularization in the loss function [14, 36]. This ensures that DA is a primary source of regularization (alongside the effect of weight decay in the optimizer and in-built model regularization, e.g. batch normalization). Ctrl-A parameters. In terms of Ctrl-A parameters, we initialize Γi(1)=0 _i^(1)=0 and αi(1)=0 _i^(1)=0 for all i, and set the training phase length, npn_p, to 5 epochs. The threshold parameter ξ is initialized at 0.9, irrespective of model size, dataset, and use of auxiliary transforms in the augmentation pipeline. Notably, the initial value of ξ is not critically important as the control loop adjusts it in the early training phases. This is ensured by choosing a proper gain factor, KgK_g. For our experiments, we empirically observe that setting Kg=(1−ξ(j))/2K_g=(1-ξ^(j))/2 was effective, making the step size directly dependent on the current value of ξ. In addition, we constrain the magnitude of the step size to the interval |Δξ|∈[0.005,0.1]| ξ|∈[0.005,0.1], thereby ensuring that each update remains within a reasonable range and is neither excessively small nor excessively large. Validation datasets. The Ctrl-A algorithm relies on the availability of a validation dataset, which is, unfortunately, not explicitly included in standard CIFAR and SVHN-core data splits. To tackle this, our first approach is to use the first 1,000 images of the test datasets to form our validation datasets. This allows model training on the full training dataset, while the Ctrl-A algorithm is informed by a small subset of the test data. To ensure that this does not provide an unfair advantage (for the small part of the test set that also constitutes the validation set), we conduct an analysis to compare test- and validation performance (Sec. 4.3). In addition, in our second approach, we create the validation dataset from a training-validation split. We again use a validation set containing 1,000 samples (i.e., 1,000 samples are removed from the training dataset), which we find sufficient for guiding the control algorithm and for providing a robust basis for ROR curve fitting. 4.1 Initial experiments with Ctrl-A Setup. Airbench-94 models are trained for 500 epochs using an SGD optimizer with a Nesterov momentum of 0.9 and a weight decay of λ=0.00025λ=0.00025. The learning rate follows a cosine-annealing schedule of the form given in Eq. 8 with an initial value of η0=0.05 _0=0.05. For Ctrl-A, the ASD parameters are, for simplicity, initialized at zero, == = α=0 for the first training phase. Affine operations from the augmentation pools are performed with bilinear interpolation and are supplemented by alternating horizontal flips (pre), 4-pixel random pad-and-crop (pre), and standard CIFAR-10 normalization (post). RandAugment and TrivialAugment benchmark Using the same training pipeline and auxiliary transformations as for Ctrl-A, an RA benchmark was established by performing a coarse search, with full model training, on the (NRA,MRAN_RA,M_RA)-grid. This was feasible due to the relatively fast training time of the airbench-94 model. The best approximate value of MRAM_RA (augmentation strength) was found for each NRA=2,3,4N_RA=\2,3,4\ (number of operations per sample), which fed into a fine-tuning search in which MRAM_RA was incremented by ±1± 1 for the best initial configurations. In this way, search-optimized RA was finally obtained for (NRA=2N_RA=2, MRA=26M_RA=26), reaching a 25-run average of (95.29±0.06)%(95.29± 0.06)\,\%. Similarly, a TA benchmark of (95.14±0.07)%(95.14± 0.07)\,\% was obtained using the same training pipeline and employing the “Wide” augmentation space suggested in [26]. ControlAugment investigation. The first experiment with Ctrl-A, using the setup parameters described above, examines the performance dependence on the control parameter κsp _sp. Figure 4 visualizes the 10-run average results (error bars with 9595 %\% coverage) obtained for (a) CtrlA(11), (b) CtrlA(22), and (c) CtrlA(33). The investigated range of κsp _sp depends on the number of operations N due to an observed saturation of the ASD parameters for the largest value of κsp _sp. For example, for N=1N=1, we find that κsp≳0.9 _sp 0.9 saturates the ASD parameters in the model fine-tuning phases, such that investigating the region of κsp>1 _sp>1 provides little extra information in this case. The use of N=2N=2 and N=3N=3 operations provides more control authority, which allows us to increase κsp _sp to approximately 2.0 and 3.5, respectively, before saturation occurs in the model fine tuning phase. Notably, the obtained validation accuracy improves into the saturation region for both N=1N=1 and N=2N=2, and we only observe a degradation for N=3N=3 with κsp>3 _sp>3 due to excessively strong augmentations leading to data homogenisation. Evidently, performance is, in this case, not maximized in the case of balanced augmentation (κsp∼1 _sp 1), but rather in the strongly augmenting regime with κsp∼2 _sp 2. Table 1: Top-1 accuracy (model: WRN-28-10) on CIFAR-10, CIFAR-100, and SVHN-core. The first five columns are literature values [45, 3, 4, 47, 26], and the last five columns are from this work. Parenthesis represent shorthand notation for 95 %\% uncertainty coverage, i.e. 97.54(08)97.54(08) is 97.54±0.0897.54± 0.08. Lowercase font represents the used DA pool, s: Standard pool [4], w: Wide pool [26], and c: Control pool from this work (Appendix A). Dash indicates no available result. Standard setups Modified setups Base A RAs DeepAAs TAw CtrlAc(2) TAw CtrlAc(1) CtrlAc(2) CtrlAc(3) CIFAR-10 96.1 97.4 97.3 97.56(14) 97.46(06) 97.54(08) 97.96(08) 98.14(11) 98.19(06) 98.10(05) CIFAR-100 81.2 82.9 83.3 84.02(18) 84.33(17) 84.29(17) 84.43(24) 84.02(24) 84.80(13) 84.44(29) SVHN-c 96.9 98.1 98.3 - 98.11(03) 98.06(07) 98.14(07) 98.18(03) 98.25(03) 98.27(03) 4.2 Performance benchmarking of Ctrl-A To compare the Ctrl-A algorithm against established literature benchmarks for DA, we perform experiments using the WRN-28-10 model architecture and the datasets CIFAR-10, CIFAR-100, and SVHN-core. Standard training setup. Our first experiment uses the standard training setup for the WRN-28-10 model and the three datasets (see Appendix D). As performance benchmarks, we provide the original WRN-28-10 baseline [45], AutoAugment (A) [3], RandAugment (RA) [4], Deep AutoAugment (DeepAA) [47], and TrivialAugment (TA) [26]. Based on our findings from the airbench-94 experiment, we elect to use N=2N=2 operations for the Ctrl-A implementation. Additionally, given the enhanced model capacity of the WRN-28-10 architecture, we set κsp=1.5 _sp=1.5, corresponding to a regime of moderately strong augmentation across all three datasets. With the results shown in the left subdivision of Table 1, we find that CtrlA(2) performs at a state-of-the-art level, matching the previous benchmark accuracies for all three datasets. However, the differences observed between methods are strikingly small. This may indicate that model performance is limited by the training setup rather than the choice of DA method. To investigate this, we perform a second set of experiments in which we deviate from the conventional training pipeline and instead employ our own modified setup. Modified training setup. In our modified setups (detailed in Appendix D), we extend training to a larger number of epochs (500 for CIFAR and 300 for SVHN-c). For CIFAR-10, we reduce both the weight decay and the initial learning rate by a factor of two and completely exclude CutOut [5]. For CIFAR-100, we also reduce the initial learning rate by a factor of two, and for SVHN-c, we decrease the CutOut size to 10 pixels and introduce random pixel inversion. To provide new performance benchmarks with the modified training setups, we use TA with the wide augmentation pool due to its straightforward and parameter-free implementation. For Ctrl-A, the same three configurations are applied to the three datasets, namely CtrlA(1) with κsp=1.0 _sp=1.0, and CtrlA(2) and CtrlA(3) with κsp=1.5 _sp=1.5. Results obtained with the three modified training setups are shown in the right subdivision of Table 1. We find a significant statistical improvement for all three tasks for CtrlA(2) by transitioning to the modified setups. Most notable is the improvement observed for CIFAR-10, for which the error rate is reduced by nearly 30 %\%. The benchmark method, TA, also tends to perform better with the modified setups, and although the improvements are only marginal for CIFAR-100 and SVHN-c, the classification accuracy is improved by 20 %\% for CIFAR-10. Finally, we also observe the general behaviour that Ctrl-A with N=2N=2 outperforms the uni-augmenting case, N=1N=1, in all the investigated cases. Interestingly, the bi-augmenting case N=2N=2 also outmatches the case of N=3N=3 for the CIFAR datasets, while the latter performs slightly better on the number-based dataset SVHN-c, matching the benchmark result of RA, which has proved hard to reproduce [26]. 4.3 Validation versus test performance We now investigate the behavior of Ctrl-A for two different validation setups. In the first setup, used so far, the validation sets are small 1,000-image subsets drawn from the test datasets, i.e., Val⊂TestD_Val _Test. In the second setup, the training datasets are randomly partitioned into training and validation sets, with the validation sets containing 1,000 image-label pairs drawn from the training data (so that Val∩Test=∅D_Val _Test= ). In both cases, WRN-28-10 models are trained on the same three classification datasets using our modified training procedures. The results of the first part of the experiment, in which Val⊂TestD_Val _Test, are reported in Table 2. In none of the three cases does the validation accuracy exceed the test accuracy. Instead, for the SVHN-core dataset, the test accuracy is slightly higher than the validation accuracy, likely reflecting a bias toward a slightly more challenging validation dataset. These results strongly indicate that, although our algorithm explicitly makes use of the validation dataset, it does not induce overfitting to the validation data. This conclusion is further supported by the experiment presented in Appendix C. In addition to reporting average validation and test accuracies in Table 2, we also report test accuracies obtained using test-time augmentation (TTA) [13]. For the CIFAR datasets, we augment each image with its horizontally mirrored counterpart, while for SVHN-core, we include color-inverted versions. For each original–augmented image pair, the corresponding class prediction is computed by averaging the logit outputs. We observe a statistically significant improvement with TTA for the CIFAR datasets, but find no improvement in accuracy for SVHN-core, suggesting that color inversion does not provide much complementary information in this case. Table 2: Top-1 accuracy (%\%) (model: WRN-28-10) on CIFAR-10, CIFAR-100, and SVHN-core using CtrlA(2) with κsp=1.5 _sp=1.5. The validation dataset, containing 1000 image-label pairs, is a subset of the test dataset, i.e. Val⊂TestD_Val _Test. Notation x.x(y) is shorthand for x.x ± 0.y, representing a 95 %\% uncertainty coverage. Dataset Validation Test TestTTA CIFAR-10 98.06(23) 98.20(12) 98.35(11) CIFAR-100 84.60(40) 84.68(15) 85.15(26) SVHN-c 97.74(59) 98.27(06) 98.27(06) Table 3: Top-1 accuracy (%\%) (model: WRN-28-10) on CIFAR-10, CIFAR-100, and SVHN-core using CtrlA(2) with κsp=1.5 _sp=1.5. The validation dataset, containing 1,000 image-label pairs, is obtained from a random training-validation split, i.e. Val⊂Train(full),Train=Train(full)∖ValD_Val _Train^(full),~D_Train=D_Train^(full) _Val. Notation x.x(y) is shorthand for x.x ± 0.y, representing a 95 %\% uncertainty coverage. Dataset Validation Test TestTTA CIFAR-10 98.14(36) 98.11(14) 98.25(11) CIFAR-100 86.10(95) 84.57(36) 85.03(32) SVHN-c 96.52(18) 98.28(07) 98.31(07) Finally, Table 3 reports results obtained using a training-validation split, reducing the total number of training samples by 1,000 image-label pairs (2%2\% of the CIFAR training set). As may be expected, this marginally reduces the observed test accuracies for CIFAR in comparison with the results from Table 2, whereas the results for the slightly larger SVHN-core dataset remain largely unchanged. The validation datasets appear to be slightly biased (compared to the test dataset) in the cases of CIFAR-100 and SVHN-core. For SVHN-core, the observed discrepancy is consistent with previous observations that the training set is generally more challenging than the test set [41]. For CIFAR-100, we observe the opposite pattern, with the average validation accuracy exceeding the test accuracy. However, the discrepancy remains within two standard errors (≈2.3%≈ 2.3\,\%), calculated under the assumption of a binomial distribution of correct predictions, and may therefore be attributed to the particular choice of validation split rather than to a systematic effect. 4.4 Convergence testing Evaluation of DA methods on CIFAR and SVHN datasets is often performed using two different model types; WRN models and the ShakeShake model [7]. To ensure fairness when comparing methods, standard training pipelines have been established for each of the model types. These standard training pipelines are closely related to the setups used in the papers introducing the models and were used to acquire the results in the left part of Table 1. As a result, it has become standard to train ShakeShake-26-2x96d models for 1,600 epochs whereas WRN models are conventionally only trained for 200 epochs. Considering the compute cost of an epoch, the resources required to complete a training instance with the ShakeShake-26-2x96d model are approximately 7 times larger than those required with the WRN-28-10 architecture [26]. Figure 5: Convergence results for CIFAR-10 performance. Thus, although standard training pipelines ensure comparability among DA methods, the disparity in computational resources complicates a fair comparison between the two model types. Additionally, selecting hyperparameters based on a setup with minimal use of DA is likely to create a performance bottleneck in setups that rely more heavily on DA. To address this, we conducted a convergence study to investigate how performance scales with the number of training epochs nmaxn_max for the WRN-28-10 model. The results of the investigation, performed with both the standard and our modified CIFAR-10 setups using TA-Wide and CtrlA(22) are shown in Fig. 5. The results of the performed convergence study reveal three important points. Firstly, across the four training setups and augmentation methods, performance looks to converge after approximately 400-500 training epochs, i.e. a two-fold increase in training time compared to the standard setup. Secondly, the modified training setup provides a clear performance enhancement compared to the standard training setup. Finally, but most importantly, the standard training setup appears to diminish the effect of differences among chosen augmentation methods. This likely explains the minimal differences generally observed between the literature values for different DA methods applied to CIFAR-10 using the WRN-28-10 architecture. 4.5 Dependency on augmentation pool The augmentation pool is a vital part of any automated DA method. It is therefore also natural that the augmentation pool somewhat varies from study to study, and this work is no exception. However, the practical usability of any given DA method will be dependent on the algorithms (in)sensitivity to the exact choice of augmentation pool. As such, it is informative to carry out ablation studies with the augmentation pool as the changing parameter. Here we consider three different augmentation pools, namely (1) the standard one from RA [4], (2) the wide augmentation pool from TA [26], and (3) our own modified version. The first two augmentation pools are detailed in [26]. We carry out experiments in which WRN-28-10 models are trained on CIFAR-10 for 500 epochs with our modified training setup. As in the convergence study above, we perform parallel investigations for TA and CtrlA(22), using a control setpoint of κsp=1.5 _sp=1.5 for the latter. The results of this investigation, summarized in Table 4, show that our method outperforms the TA benchmark for two of the three augmentation pools. For the wide augmentation pool, originally developed alongside the TA method, we observe more comparable performance between the two approaches. Table 4: Top-1 test accuracy (%\%) (model: WRN-28-10) on CIFAR-10 for different augmentation pools. Notation x.x(y) is shorthand for x.x ± 0.y, representing a 95 %\% uncertainty coverage. Aug. pool TA CtrlA(2) Standard [4] 97.60(11) 97.76(09) Wide [26] 97.96(08) 97.93(10) Ours (App. A) 98.03(06) 98.19(06) 5 Discussion Our results demonstrate the capability of Ctrl-A to provide state-of-the-art performance across a range of benchmark tasks. Using concepts from control theory, our method dynamically adjusts the strength of DA by individual augmentation strengths to prevent overfitting to training data. In terms of computational overhead, the cost of Ctrl-A scales linearly with the number of operations in the augmentation pool, K, the number samples in the validation set, and the relative phase length of the update procedure, expressed as the ratio nmax/npn_max/n_p, and inversely with the augmentation strength step, Δγ γ, used to form the ROR curves. The experiments conducted with Ctrl-A in this study were found to have a single-run compute cost approximately 10%10\,\% higher than the tuning-free TA benchmark [26]. Notably, this cost could be reduced further limiting the re-computation of the ROR curves to a subset of training phases, such that only the control parameter ξ is updated every single phase. With respect to Ctrl-A hyperparameters, we find that N=2N=2 operations and a control setpoint of κsp∈[1,2] _sp∈[1,2] perform well across tasks. The optimal value of κsp _sp appears to be model- and task specific, indicating that more elaborate work is needed to better understand the dynamics of Ctrl-A. However, across all the conducted experiments, we find that operation in the strong augmentation regime (κsp>1 _sp>1) consistently outperforms operation in the weak augmentation regime (κsp<1 _sp<1). Secondly, our convergence study, performed for CIFAR-10, revealed an apparent sub-optimality for the standard WRN-28-10 training pipeline commonly used to benchmark DA methods. Evidently, the standard 200-epoch pipeline provides too few training iterations to achieve convergence given the high data variability introduced by strong DA. As noted by Hounie et al. [11], it is undesirable for the benchmarking setup, which in this case was developed for the original WRN implementation with minimal data augmentation, to become a performance bottleneck when the goal is to compare the performance of different DA algorithms. Intuitively, applying more excessive DA increases training data variability and in turn requires more training epochs to achieve efficient model convergence. In contrast, applying our modified setup for the original WRN-28-10 training pipeline leads to significant model overfitting with final accuracies in the 92-93 %\% range for CIFAR-10. With our modified and prolonged training setup, we demonstrate that WRN-28-10 achieves CIFAR-10 performance comparable to the best results obtained using the substantially more computationally expensive Shake-Shake-model training pipeline. In comparison with the TA-Wide benchmarking method, Ctrl-A demonstrated a significant advantage in terms of performance improvement (16%16\,\% versus 5%5\,\% relative decrease in error rate for CIFAR-10) by moving from the standard to the modified training setup. Our results thereby highlight the fact that sub-optimal training setup hyperparameters, such as weight decay, may complicate the direct inter-comparison of different DA methods if the hyperparameter setting becomes the performance bottleneck. A subset of the reported experiments employs a validation set that constitutes a small fraction of the official test set. We emphasize that this practice is not recommended in general and was adopted solely due to the absence of predefined validation splits in the benchmark datasets considered. Importantly, as shown in Table 3, the model accuracy on the validation subsets is slightly lower than the accuracy on the full test sets. Consequently, retaining these samples within the reported test-set evaluation yields conservative performance estimates. Excluding the validation subset from the test set would, in this case, have resulted in marginally higher test accuracies across all three datasets, thereby slightly favoring our method. Furthermore, despite the active use of the validation subset for augmentation parameter control, the results provide no indication of validation-specific overfitting. If the framework were implicitly adapting to characteristics of the validation data, one would expect elevated validation performance relative to the remaining test samples, which is not observed. Critically, this strongly suggests that the method does not extract dataset-specific information from the validation subset beyond its intended role in guiding the control mechanisms. Improvements to ControlAugment The Ctrl-A algorithm can be interpreted as a feedback control system in which the strength of data augmentation serves as the control input that is used to regulate the training and validation losses [1]. In this view, the ASD parameters and α play the role of the control loop actuators, and, as in any feedback system, actuator saturation may occur. In our framework, this arises if ξ→0ξ→ 0, resulting in Γi=1 _i=1 and αi=Ri(1) _i=R_O_i(1). Actuator saturation often results in insufficient control authority, such that the process variable (κ) fails to reach its setpoint (κsp _sp). Such behavior was observed in a subset of the results reported in Table 2, particularly for CIFAR-100 and configurations with N=1N=1. To counter cases like this, we envisage that an extra control layer may be incorporated into the algorithm, either by dynamically incrementing N, or by modifying the underlying augmentation pool to one that provides greater control authority. In this context, the wide augmentation pool offers the greatest control authority, the standard pool provides the least, and the proposed control pool lies between these two extremes. In our experiments we generally observe the tendency that increasing the number of operations N above 2 or 3 leads to either performance stagnation or degradation. Although the reason for this is not yet fully understood, we hypothesize that correlations between select pairs of transformations, e.g. translations, shears, and rotations, may lead to augmented samples that retain only a limited portion of the original image information. Correlations between operations could be investigated with a modified version of the ROR curves introduced in this paper. Taking into account these correlations between transformation types may enable a suppression of those image instances hypothesized to hinder efficient model training. A different path for advancing our algorithm could improve upon the default use of class-wide model sensitivities to each of the K operations in the augmentation pool. That is, our method updates the augmentation strengths for each individual operation in a manner that is independent of image labels. However, different classes in a given image classification task generally do not share the same sensitivity to image transformations. A well known example is that of integer classification, in which certain numbers are more sensitive to e.g. rotations than others [25]. Therefore, a natural extension of Ctrl-A is to reformulate and α as matrices of size K×CK× C rather than 1-dimensional arrays of length K. This expansion is also left for future work. All experiments reported in this work were conducted on a single-GPU workstation equipped with an NVIDIA RTX 4070 Ti SUPER 16 GB. The computational resources available in this setup precluded evaluation of Ctrl-A on large-scale datasets such as ImageNet and limited benchmarking of modified training configurations to the TA method (see Table 1). Access to additional computational resources would enable evaluation of Ctrl-A in larger-scale and more demanding settings in terms of parallel processing power and memory. Concluding remarks We have introduced ControlAugment (Ctrl-A), a control-driven algorithm that realizes online model regularization through adaptive data augmentation. To achieve this, we expand the purpose of the conventional validation dataset to encompass computations of what we define as relative operation response curves. These response curves form the basis for a control-based update procedure for each individual operation type in the chosen augmentation pool. With Ctrl-A, we demonstrate highly competitive performance in standard benchmarking tasks that use the Wide-ResNet model architecture. Notably, this is achieved without resource overhead from a separate search phase and while simply initializing all augmenting transformations to the identity operator. 6 Acknowledgement This work was supported by; Project 22HLT05 MAIBAI, which has received funding from the European Partnership on Metrology, co-financed from the European Union’s Horizon Europe Research and Innovation Program and by the Participating States; by Innovate UK under the Horizon Europe Guarantee Extension; and by the Danish Agency for Higher Education and Science. References [1] K. J. Åström and R. Murray (2021) Feedback systems: an introduction for scientists and engineers. Princeton university press. Cited by: §5. [2] J. B. Christensen (2026) ControlAugment (Ctrl-A). Note: https://github.com/jbcdfm/ControlAugmentGitHub repository Cited by: §4. [3] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le (2019) AutoAugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 113–123. Cited by: Appendix A, §1, §2.1, §2.1, §4.2, Table 1. [4] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le (2020) RandAugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, p. 702–703. Cited by: Appendix A, §2.1, §3.2, §4.2, §4.5, Table 1, Table 4. [5] T. DeVries and G. W. Taylor (2017) Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552. Cited by: §D.1, §4.2. [6] S. Du, Z. Yuan, P. Lai, and T. Ikenaga (2024) JoyPose: jointly learning evolutionary data augmentation and anatomy-aware global–local representation for 3d human pose estimation. Pattern Recognition 147, p. 110116. Cited by: §1. [7] X. Gastaldi (2017) Shake-shake regularization. arXiv preprint arXiv:1705.07485. Cited by: §4.4. [8] I. J. Goodfellow, J. Shlens, and C. Szegedy (2014) Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Cited by: §1. [9] R. Hataya, J. Zdenek, K. Yoshizoe, and H. Nakayama (2020) Faster Autoaugment: Learning augmentation strategies using backpropagation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16, p. 1–16. Cited by: §2.1. [10] D. Ho, E. Liang, X. Chen, I. Stoica, and P. Abbeel (2019) Population based augmentation: Efficient learning of augmentation policy schedules. In International conference on machine learning, p. 2731–2741. Cited by: §2.1. [11] I. Hounie, L. F. Chamon, and A. Ribeiro (2023) Automatic data augmentation via invariance-constrained learning. In International Conference on Machine Learning, p. 13410–13433. Cited by: §2.2, §5. [12] K. Jordan (2024) 94% on CIFAR-10 in 3.29 seconds on a single GPU. arXiv preprint arXiv:2404.00498. Cited by: §D.1, §D.1, §4. [13] M. Kimura (2021) Understanding test-time augmentation. In International Conference on Neural Information Processing, p. 558–569. Cited by: §4.3. [14] A. Krogh and J. Hertz (1991) A simple weight decay can improve generalization. Advances in neural information processing systems 4. Cited by: §4. [15] T. Kumar, R. Brennan, A. Mileo, and M. Bendechache (2024) Image data augmentation approaches: A comprehensive survey and future directions. Ieee Access 12, p. 187536–187571. Cited by: §1. [16] Y. Li, G. Hu, Y. Wang, T. Hospedales, N. M. Robertson, and Y. Yang (2020) Differentiable automatic data augmentation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16, p. 580–595. Cited by: §2.1. [17] S. Lim, I. Kim, T. Kim, C. Kim, and S. Kim (2019) Fast Autoaugment. Advances in neural information processing systems 32. Cited by: §2.1. [18] C. Lin, M. Guo, C. Li, X. Yuan, W. Wu, J. Yan, D. Lin, and W. Ouyang (2019) Online hyper-parameter learning for auto-augmentation strategy. In Proceedings of the IEEE/CVF international conference on computer vision, p. 6579–6588. Cited by: §2.2. [19] A. Liu, Z. Huang, Z. Huang, and N. Wang (2021) Direct differentiable augmentation search. In Proceedings of the IEEE/CVF international conference on computer vision, p. 12219–12228. Cited by: §2.1. [20] Z. Liu, H. Jin, T. Wang, K. Zhou, and X. Hu (2021) DivAug: Plug-in automated data augmentation with explicit diversity maximization. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4762–4770. Cited by: §2.2, §3.4. [21] I. Loshchilov and F. Hutter (2016) Sgdr: stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983. Cited by: §4. [22] S. Mehta, S. Naderiparizi, F. Faghri, M. Horton, L. Chen, A. Farhadi, O. Tuzel, and M. Rastegari (2022) RangeAugment: Efficient online augmentation with range learning. arXiv preprint arXiv:2212.10553. Cited by: §2.2. [23] S. Minaee, A. Abdolrashidi, H. Su, M. Bennamoun, and D. Zhang (2023) Biometrics recognition using deep learning: A survey. Artificial Intelligence Review 56 (8), p. 8647–8695. Cited by: §1. [24] T. Miyato, S. Maeda, M. Koyama, and S. Ishii (2018) Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence 41 (8), p. 1979–1993. Cited by: §1. [25] H. Mo and G. Zhao (2024) RIC-CNN: rotation-invariant coordinate convolutional neural network. Pattern Recognition 146, p. 109994. Cited by: §5. [26] S. G. Müller and F. Hutter (2021) TrivialAugment: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the IEEE/CVF international conference on computer vision, p. 774–782. Cited by: §1, §2.1, §3.2, §3.2, §4.1, §4.2, §4.2, §4.4, §4.5, Table 1, Table 4, §5. [27] A. Mumuni and F. Mumuni (2022) Data augmentation: A comprehensive survey of modern approaches. Array 16, p. 100258. Cited by: §1. [28] F. Ntelemis, Y. Jin, and S. A. Thomas (2023) A generic self-supervised framework of learning invariant discriminative features. IEEE Transactions on Neural Networks and Learning Systems 35 (9), p. 12938–12952. Cited by: §1. [29] A. S. Panayides, A. Amini, N. D. Filipovic, A. Sharma, S. A. Tsaftaris, A. Young, D. Foran, N. Do, S. Golemati, T. Kurc, et al. (2020) AI in medical imaging informatics: current challenges and future directions. IEEE journal of biomedical and health informatics 24 (7), p. 1837–1857. Cited by: §1. [30] D. S. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le (2019) SpecAugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779. Cited by: §1. [31] R. S. Peres, X. Jia, J. Lee, K. Sun, A. W. Colombo, and J. Barata (2020) Industrial artificial intelligence in industry 4.0-systematic review, challenges and outlook. IEEE access 8, p. 220121–220139. Cited by: §1. [32] P. Rajpurkar, E. Chen, O. Banerjee, and E. J. Topol (2022) AI in health and medicine. Nature medicine 28 (1), p. 31–38. Cited by: §1. [33] C. Shorten and T. M. Khoshgoftaar (2019) A survey on image data augmentation for deep learning. Journal of big data 6 (1), p. 1–48. Cited by: §1, §3.4. [34] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov (2014) Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15 (1), p. 1929–1958. Cited by: §4. [35] Z. Tang, Y. Gao, L. Karlinsky, P. Sattigeri, R. Feris, and D. Metaxas (2020) OnlineAugment: Online data augmentation with less domain knowledge. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16, p. 313–329. Cited by: §2.2. [36] R. Tibshirani (1996) Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58 (1), p. 267–288. Cited by: §4. [37] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel (2017) Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), p. 23–30. Cited by: §1. [38] J. Wang, D. Ruan, Y. Li, Z. Wang, Y. Wu, T. Tan, G. Yang, and M. Jiang (2025) Data augmentation strategies for semi-supervised medical image segmentation. Pattern Recognition 159, p. 111116. Cited by: §1. [39] J. Wei and K. Zou (2019) EDA: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196. Cited by: §1. [40] Q. Wen, L. Sun, F. Yang, X. Song, J. Gao, X. Wang, and H. Xu (2020) Time series data augmentation for deep learning: A survey. arXiv preprint arXiv:2002.12478. Cited by: §1. [41] T. Z. Xiao, J. Zenn, and R. Bamler (2023) The SVHN dataset is deceptive for probabilistic generative models due to a distribution mismatch. arXiv preprint arXiv:2312.02168. Cited by: §4.3. [42] M. Xu, S. Yoon, A. Fuentes, and D. S. Park (2023) A comprehensive survey of image augmentation techniques for deep learning. Pattern Recognition 137, p. 109347. Cited by: §1. [43] X. Xu and H. Zhao (2022) Universal adaptive data augmentation. arXiv preprint arXiv:2207.06658. Cited by: §2.2. [44] S. Yang, S. Guo, J. Zhao, and F. Shen (2024) Investigating the effectiveness of data augmentation from similarity and diversity: an empirical study. Pattern Recognition 148, p. 110204. Cited by: §1. [45] S. Zagoruyko and N. Komodakis (2016) Wide residual networks. arXiv preprint arXiv:1605.07146. Cited by: §4.2, Table 1, §4. [46] X. Zhao, L. Wang, Y. Zhang, X. Han, M. Deveci, and M. Parmar (2024) A review of convolutional neural networks in computer vision. Artificial Intelligence Review 57 (4), p. 99. Cited by: §1. [47] Y. Zheng, Z. Zhang, S. Yan, and M. Zhang (2022) Deep AutoAugment. arXiv preprint arXiv:2203.06172. Cited by: §2.1, §4.2, Table 1. [48] F. Zhou, J. Li, C. Xie, F. Chen, L. Hong, R. Sun, and Z. Li (2021) MetaAugment: Sample-aware data augmentation policy learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35, p. 11097–11105. Cited by: §2.2. Appendix A Augmentation pool in detail Table 5 contains the K=15K=15 transformations in our augmentation pool. The first six, TranslateX, TranslateY, ShearX, ShearY, Scale, and Rotation fall into the category of geometric (affine) transformations, whereas the last nine, Hue, Bright/dark, Sharpen/blur, Contrast, Saturation, Solarize, Posterize, AutoContrast, and Equalize are appearance-based (color) transformations. Table 5: List of the “control” augmentation pool and respective parametrizations that use normalized augmentation strength γi∈[0,1] _i∈[0,1]. All operations converge to the identity operator for γ→0γ→ 0, and the identity operator is therefore explicitly excluded from the augmentation pool. Operations appended by (s) indicates that the sign of γi _i is flipped, i.e. γi<0 _i<0, with a 50%50\,\% probability. Op. #\# Op. Name Parametrization 1. TranslateX (s) γ1/2× _1/2×Img. width 2. TranslateY (s) γ2/2× _2/2×Img. height 3. ShearX (s) 45∘γ345 _3 4. ShearY (s) 45∘γ445 _4 5. Scale (s) 1+γ5/21+ _5/2 6. Rotation (s) 60∘γ660 _6 7. Hue (s) γ7/2 _7/2 8. Bright/dark (s) 1+0.9γ81+0.9 _8 9. Sharpen/blur (s) 1+0.9γ91+0.9 _9 10. Contrast (s) 1+0.9γ101+0.9 _10 11. Saturation (s) 1+0.9γ111+0.9 _11 12. Solarize 255(1−γ12/2)255(1- _12/2) 13. Posterize 8(1−γ13/2)8(1- _13/2) 14. AutoContrast γ14 _14 15. Equalize γ15 _15 The parametrization of each operation (based on transformations from the torchvision.transforms python library), is provided in the right column of the table. As apparent from the parametrization of solarize and posterize, we assume 8-bit images. In addition most operation types, i.e. those appended with (s), are signed operations, meaning that γ1-11∈[−1,1] _1 -11∈[-1,1]. The sign of γ1-11 _1 -11 is determined by a probabilistic flip of probability 50 %\%. AutoContrast (14O_14) and Equalize (15O_15) are usually not treated as parametrized operations [3, 4], and so our implementation requires a short explanation. To construct parametrized versions, we use linear combinations 14,15(x;γ)=(1−γ)x+γ14,15′(x),O_14,15(x;γ)=(1-γ)x+γ\,O_14,15 (x), (9) in which 14′O _14 is the standard non-parametrized version of AutoContrast and 15′O _15 is the standard non-parametrized version of Equalize. In this form, operations 14O_14 and 15O_15 naturally adhere to the general rules for operations in ControlAugment. Figure 6 illustrates uni-transformed versions (γ=1γ=1) of a single CIFAR-10 image for each of the 15 operations in the control augmentation pool. These examples are not necessarily representable for augmented images, but they showcase the maximum perturbation an image can undergo for each operation iO_i if γi=1 _i=1. Figure 6: Visualization of uni-transformed images for each of the operations in the full augmentation pool with augmentation strength γ=+1γ=+1. The original image is shown in the lower right. Appendix B Augmentation strength updates This section details the augmentation strength update algorithm used to solve Eq. 3 which is repeated here for convenience: Accf(i(Val;γ))/Accf(Val)−ξ=0,Acc_f(O_i(D_Val;γ))/Acc_f(D_Val)-ξ=0, (10) where the first term is defined as the relative operation response (ROR). The evaluation uses different degrees of augmented versions of the validation data to determine, for each operator type iO_i, the augmentation strength that leads to a decrease in accuracy of ξ∈[0,1]ξ∈[0,1]. Our implementation uses a small fraction (1000 samples, corresponding to 10%10~\% for CIFAR-10) of either the test dataset or the training dataset. Our algorithm works as follows. For each operation iO_i, prepare the datasets i(Val;γ)O_i(D_Val;γ) for γ=0.1,0.2,…0.9,1γ=\0.1,0.2,… 0.9,1\ and compute, for each operation and each γ, the model accuracy Accf(i(Val;γ))Acc_f(O_i(D_Val;γ)). For each operation, this results in data arrays in which the augmentation strength γ is the independent variable, providing us with a basis for (approximately) solving the implicit equation above. To accomplish this, we make use of regression analysis as exemplified in Fig. 7 for different types of operations. The error function provides an adequate model fit to describe an increasing error due to an increasing amount of perturbation. Figure 7: Examples of relative operation response data (crosses) and fitting curves (dashed lines) used to determine Γi _i and αi _i. CIFAR-10 was used as example dataset. B.1 Linearly tilted distribution The original implementation of Ctrl-A simply drew augmentation strengths from a uniform distribution according to γi∼U(0,Γi) _i U(0, _i). However, it was quickly discovered, that this simple construction provided insufficient control capacity for larger models, such as the WRN-28-10 model variant, which ended up resulting in Γi=1 _i=1 for all i and the model overfitting to the training data in the fine tuning phases. To mitigate this, we introduce the possibility of a tilt αi _i to the distributions for which Γi=1 _i=1, such that the augmentation strengths in these cases are instead drawn from “tilted” distributions, i.e. γi∼Uαi(0,1) _i U_ _i(0,1), which are exemplified in Fig. 2 in the main text. This modification enables more control capacity and places more weight on stronger augmentation strengths for those transformation types that are efficiently learned by the model. In cases where the model fθf_θ is ξ-insensitive to operation iO_i such that Γi=1 _i=1, we compute the corresponding tilt parameter αi _i as αi=Accf(i(Val;γ=1))/Accf(Val)−ξ1−ξ. _i= Acc_f(O_i(D_Val;γ=1))/Acc_f(D_Val)-ξ1-ξ. (11) With this construction, αi=0 _i=0 if Eq. 3 is exactly solved for γ=1γ=1, and αi=1 _i=1 if the transformation type does not degrade model performance at all such that Accf(i(Val;γ=1))=Accf(Val)Acc_f(O_i(D_Val;γ=1))=Acc_f(D_Val). Appendix C Testing for overfitting Due to the Ctrl-A procedure invoking online feedforward based on augmented validation data to tune the available augmentation strength distributions, it should, for good measure, be ensured that the process does not lead to undesired overfitting of the model to the validation data. To this end, we create (randomized) 8000/2000 validation/test splits of the 10k CIFAR-10 test samples, and use the larger pool of 8000 validation images to inform the data augmentation. Our results, obtained for a 568-parameter LeNet model trained on CIFAR-10 using the CtrlA(2) policy, are visualized in Fig. 8 for three different values of κsp _sp. To statistically assess the data in Fig. 8 we use a one-sided Welch’s t-test with the null hypothesis, H0:H_0: [Accf(Val)]≤[Accf(Test)]E[Acc_f(D_ Val)] [Acc_f(D_ Test)] and the alternative hypothesis H1:H_1: [Accf(Val)]>[Accf(Test)]E[Acc_f(D_ Val)]>E[Acc_f(D_ Test)]. The observed p-values (0.690.69, 0.890.89, and 0.310.31) provide no significant evidence against H0H_0 which cannot be rejected at any conventional significance level. Thus, we find no statistical support for the validation accuracy being higher than the test accuracy, consistent with our expectation that the ControlAugment algorithm does not lead to the model overfitting to the validation data. Figure 8: Comparison study exploiting an 8000/2000 validation-test split to investigate whether the Ctrl-A algorithm leads to overfitting to the validation data used to update augmentation strengths. The results are obtained by training a (custom) LeNet model on CIFAR-10 employing CtrlA(2) augmentation. Appendix D Dataset details D.1 CIFAR CIFAR (Canadian Institute For Advanced Research) are benchmark image classification datasets consisting of downsampled (32-by-32) real-world RGB images containing centered objects (e.g. airplane, cat, ship, truck, etc.). The CIFAR dataset comes in two variants; the 10-class dataset CIFAR-10, and the 100-class dataset CIFAR-100. Each of the two datasets contain 50,000 training samples and 10,000 test samples. Standard setup For the CIFAR datasets, WideResNet models are trained in the 28-10 setting for 200 epochs using stochastic gradient descent with a Nesterov momentum of 0.9, an initial learning rate of 0.1, a batch size of 125, a weight decay of 5×10−45× 10^-4, and the cosine learning-rate schedule following Eq. 8. The data augmentation pipeline starts with a random horizontal flip (probability of 50%\%) followed by 4-pixel pad and crop. Then follows transformations from the RA/TA/Ctrl-A augmentation pools before applying standard data normalization using dataset-wide RGB means and standard deviations. Finally, 16-by-16 pixel Cutout is applied [5]. Modified setup, CIFAR-10 For the CIFAR-10 dataset, in the modified configuration, we train WideResNet models in the 28-10 setting for 500 epochs using stochastic gradient descent with a Nesterov momentum of 0.9, an initial learning rate of 0.05, a batch size of 125, a weight decay of 2.5×10−42.5× 10^-4, and the cosine learning rate schedule following Eq. 8. In terms of data transformations, the training dataset is horizontally flipped and appended (with maintained labels) to the original dataset, emulating alternating horizontal flips [12]. The augmentation pipeline begins with 4-pixel pad and crop, followed by transformations from the RA/TA/Ctrl-A augmentation pools, and is completed with standard data normalization using RGB means and standard deviations. No final Cutout is applied in this case. Modified setup, CIFAR-100 For the CIFAR-100 dataset, in the modified configuration, we train WideResNet models in the 28-10 setting for 500 epochs using stochastic gradient descent with a Nesterov momentum of 0.9, an initial learning rate of 0.05, a batch size of 125, a weight decay of 5×10−45× 10^-4, and the cosine learning rate schedule following Eq. 8. In terms of data transformations, the training dataset is horizontally flipped and appended (with maintained labels) to the original dataset, emulating alternating horizontal flips [12]. The augmentation pipeline begins with 4-pixel pad and crop, followed by transformations from the RA/TA/Ctrl-A augmentation pool. Finally, we apply standard data normalization using RGB means and standard deviations, and end with a 16-by-16 pixel Cutout. D.2 SVHN-core SVHN-core (Street View House Numbers - Core) is a real-world image dataset containing 32-by-32 pixel images of house number digits (0-9). The dataset contains 73,257 training samples and 26,032 test samples, and exhibits strong class imbalance as certain digits, especially “1”, naturally appears more frequently than others in house numbers. Standard setup For the SVHN-core dataset, WideResNet-28-10 models are trained for 200 epochs using stochastic gradient descent with a Nesterov momentum of 0.9, an initial learning rate of 0.005, a batch size of 125, a weight decay of 0.005, and the learning rate schedule following Eq. 8. Data augmentation involves operations from the RA/TA/Ctrl-A augmentation pools, standard SVHN-core data normalization, and 16-by-16 pixel Cutout. Modified setup For the SVHN-core dataset, we train WideResNet models in the 28-10 setting for 300 epochs using stochastic gradient descent with a Nesterov momentum of 0.9, an initial learning rate of 0.005, a batch size of 125, a weight decay of 0.005, and the learning rate schedule following Eq. 8. The augmentation pipeline begins with a probabilistic pixel inversion, which is otherwise not part of any of the three considered augmentation pools. The Invert operation is followed by transformations from the RA/TA/Ctrl-A augmentation pools. Data normalization, modified to take into account the randomly applied inversion operation, is applied as the final operation in the augmentation pipeline. Finally, a reduced size 10-by-10 pixel Cutout is applied in the end.