Paper deep dive
Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking
Afroditi Kolomvaki, Fangshuo Liao, Evan Dramko, Ziyun Guang, Anastasios Kyrillidis
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 11:53:57 PM
Summary
This paper provides a theoretical convergence analysis for training two-layer ReLU neural networks with Gaussian randomly masked inputs. Using Neural Tangent Kernel (NTK) analysis, the authors demonstrate that such training achieves linear convergence to an error region proportional to the mask's variance. The work resolves technical challenges regarding randomness within non-linear activations and establishes that the expected loss decomposes into a smoothed neural network loss plus an adaptive regularization term.
Entities (7)
Relation Signals (6)
Gaussian Input Masking → appliedto → Two-layer Neural Network
confidence 97% · convergence guarantee of two-layer neural network training with Gaussian randomly masked inputs
Two-layer Neural Network → analyzedby → Neural Tangent Kernel
confidence 96% · Using a Neural Tangent Kernel (NTK) analysis, we demonstrate that training a two-layer ReLU network...
Two-layer Neural Network → uses → ReLU Activation
confidence 95% · training a two-layer ReLU network with Gaussian randomly masked inputs
Two-layer Neural Network → exhibits → Linear Convergence
confidence 94% · achieves linear convergence up to an error region proportional to the mask's variance
Gaussian Input Masking → equivalentto → Gaussian Dropout
confidence 88% · This scenario corresponds to Gaussian dropout at the input level
Gaussian Input Masking → usedin → Federated Learning
confidence 85% · common in sensor networks, privacy-preserving training, and federated learning
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We investigate the convergence guarantee of two-layer neural network training with Gaussian randomly masked inputs. This scenario corresponds to Gaussian dropout at the input level, or noisy input training common in sensor networks, privacy-preserving training, and federated learning, where each user may have access to partial or corrupted features. Using a Neural Tangent Kernel (NTK) analysis, we demonstrate that training a two-layer ReLU network with Gaussian randomly masked inputs achieves linear convergence up to an error region proportional to the mask's variance. A key technical contribution is resolving the randomness within the non-linear activation, a problem of independent interest.
Tags
Links
- Source: https://arxiv.org/abs/2602.17423v1
- Canonical: https://arxiv.org/abs/2602.17423v1
Trouble viewing inline? Open PDF directly →
Full Text
342,434 characters extracted from source content.
Expand or collapse full text
Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking Afroditi Kolomvaki ak203@rice.edu Computer Science Dept., Rice University, Houston, TX, USA Fangshuo Liao fangshuo.liao@rice.edu Computer Science Dept., Rice University, Houston, TX, USA Evan Dramko ed55@rice.edu Computer Science Dept., Rice University, Houston, TX, USA Ziyun Guang cg105@rice.edu Computer Science Dept., Rice University, Houston, TX, USA Anastasios Kyrillidis anastasios@rice.edu Computer Science Dept., Rice University, Houston, TX, USA Abstract We investigate the convergence guarantee of two-layer neural network training with Gaussian randomly masked inputs. This scenario corresponds to Gaussian dropout at the input level, or noisy input training common in sensor networks, privacy-preserving training, and federated learning, where each user may have access to partial or corrupted features. Using a Neural Tangent Kernel (NTK) analysis, we demonstrate that training a two-layer ReLU network with Gaussian randomly masked inputs achieves linear convergence up to an error region proportional to the mask’s variance. A key technical contribution is resolving the randomness within the non-linear activation, a problem of independent interest. 1 Introduction Neural networks (NNs) have revolutionized AI applications, where their success largely stems from their ability to learn complex patterns when trained on well-curated datasets (Schuhmann et al., 2022; Li et al., 2023b; Gunasekar et al., 2023; Edwards, 2024). A component to the success of NNs is its ability to model a broad range of tasks and data distributions under various scenarios. Empirical evidence has suggested neural network’s ability to learn even under noisy input (Kariotakis et al., 2024), gradient noise (Ruder, 2017), as well as modifications to the internal representations during training (Srivastava et al., 2014; Yuan et al., 2022). Leveraging such ability of the neural networks, many real-world deployment adopts a modification to the data representations during training to achieve particular goals such as robustness, privacy, or efficiency. Among the methods, perturbing the representations with an additive noise has been studied by a number of prior works (Gao et al., 2019; Li et al., 2025; 2023a; Madry et al., 2018; Loo et al., 2022; Tsilivis and Kempe, 2022; Ilyas et al., 2019), showcasing both the benefit of such perturbation and the stable convergence of the training under this setting. Compared with additive noise, perturbing the representations by multiplying it with a mask has rarely been studied theoretically. Perturbing the representations with multiplicative noise appears in many real-world settings, either by design or unintentionally. For instance, in federated learning (FL) settings (McMahan et al., 2017; Kairouz et al., 2021), particularly vertical FL (Cheng et al., 2020; Liu et al., 2021; Romanini et al., 2021; He et al., 2020; Liu et al., 2022; 2024), different features of the input data may be available to different parties, effectively creating a form of sparsity-inducing multiplicative masking on the input space. Moreover, the drop-out family (Srivastava et al., 2014; Rey and Mnih, 2021) is a class of methods to prevent overfitting and improve generalization ability of neural networks during training. Lastly, training models under data-parallel protocol over a wireless channel incurs the channel effect that blurs the data passed to the workers through a multiplication Tse and Viswanath (2005). Theoretically analyzing the training dynamics of neural networks under these settings are difficult, especially when the introduced randomness are intertwined with the nonlinearity of the activation function. While there has been previous work that studies the convergence of neural network training under drop-out (Liao and Kyrillidis, 2022; Mianjy and Arora, 2020), they often assume that the drop-out happens after the nonlinear activations are applied. From a technical perspective, statistics of the neural network outputs are easier to handle as the randomness are not affected by the nonlinearity. In this paper, we take a step further into the understanding of multiplicative perturbations in neural network training by considering noise applied before the nonlinear activation. In particular, the setting we consider is the training of a two-layer MLP where the inputs bears a multiplicative Gaussian mask. This prototype provides a simplified scenario to study the noise-inside-activation difficulty, while generalizes various training scenarios ranging from input masking (Kariotakis et al., 2024) to Gaussian drop-out (Rey and Mnih, 2021), if one views the input in our setting as fixed embeddings from previous layers of a deep neural network. Under this setting, we aim to answer the following question: How do multiplicative perturbations at the input level propagate through the network and affect the training dynamics? Our Contributions. Analyzing the training dynamics under the Gaussian masks over the input means that we have to study the statistical properties of random variables inside a non-linear function. Our work takes a step towards resolving this technical difficulty. Moreover, we utilize an NTK-based analysis (Du et al., 2018; Song and Yang, 2020; Oymak and Soltanolkotabi, 2019; Liao and Kyrillidis, 2022) to study the training convergence of the two-layer MLP under sufficient overparameterization. To our knowledge, this work provides the first convergence analysis for neural network training under Gaussian multiplicative input masking. Specifically, for inputs x masked by ⊙x c where ∼(,κ2)c (1,κ^2I), we prove that: i)i) The expected loss decomposes into a smoothed neural network loss plus an adaptive regularization term; i)i) Training achieves linear convergence to an error ball of radius O(κ)O(κ). Empirical results showcase and support our theory. Our Contributions. To our knowledge, this work provides the first convergence analysis for neural network training under Gaussian multiplicative input masking. Our main contributions are summarized as follows: • Theoretical Analysis of Input Masking. We provide a rigorous characterization of the training dynamics for two-layer ReLU networks where noise is injected before the non-linear activation. We overcome the technical challenge of resolving the expectation of non-linear functions of random variables, proving that the expected loss decomposes into a smoothed objective plus an adaptive, data-dependent regularizer. • General Stochastic Training Framework. We develop a general convergence theorem for overparameterized neural networks trained with biased stochastic gradient estimators. This result, which establishes linear convergence to a noise-dependent error ball, is of independent interest beyond the specific setting of Gaussian masking. • Explicit Convergence Guarantees. We derive constructive bounds for the convergence rate and the final error radius. We show explicitly how these quantities depend on the mask variance κ2κ^2, the network width m, and the initialization scale, demonstrating that the training converges linearly up to a floor determined by the noise level. • Empirical Validation and Privacy Utility. We confirm our theoretical predictions regarding the expected gradient and loss landscape through simulations. Furthermore, we demonstrate the practical utility of this training regime as a defense against Membership Inference Attacks (MIA), highlighting a favorable trade-off between privacy and utility. 2 Related Work Neural Network Robustness. The study of neural network robustness has a rich history, with early work focusing primarily on additive perturbations. Results such as (Bartlett et al., 2017) and (Miyato et al., 2018) established generalization bounds for neural networks under adversarial perturbations, showing that the network’s Lipschitz constant plays a crucial role in determining robustness. Subsequent work by (Cohen et al., 2019) introduced randomized smoothing techniques for certified robustness against ℓ2 _2 perturbations, while (Wong et al., 2018) developed methods for training provably robust deep neural networks. Regularization techniques have emerged as powerful tools for enhancing network robustness. Dropout (Srivastava et al., 2014) pioneered the idea of randomly masking internal neurons during training, effectively creating an implicit ensemble of subnetworks (Yuan et al., 2022; Hu et al., 2023; Kariotakis et al., 2024; Wolfe et al., 2023; Liao and Kyrillidis, 2022; Dun et al., 2023; 2022). This connection between feature masking and regularization was further explored in (Ghorbani et al., 2021), who showed that dropout can be interpreted as a form of data-dependent regularization. Note that sparsity-inducing norms, based on Laplacian continuous distribution, have a long history in sparse recovery problems (Bach et al., 2011; Jenatton et al., 2011; Bach et al., 2012; Kyrillidis et al., 2015). Empirical studies on the effect of sparsity, represented by multiplicative Bernoulli distributions, can be found in (Kariotakis et al., 2024). Neural Tangent Kernel (NTK). Jacot et al. (2018) discovers that infinite-width neural network evolves as a Gaussian process with a stable kernel computed from the outer product of the tangent features of the neural network. Later works adopted finite-width correction and applied the framework to the analysis of neural network convergence Du et al. (2018; 2019b); Oymak and Soltanolkotabi (2019). The Neural Tangent Kernel framework is one of the few theoretical tools focused on theoretical understanding neural network training. Later works extended the proof to classification tasks, where the notion of tangent features is considered as a feature mapping onto a space where the training data are separable Ji and Telgarsky (2020). Although the NTK framework has been treated as "lazy training" that prevents useful featuers to be learned, it enables exact analysis of the neural network training dynamic under various scenarios for different architectures Nguyen (2021); Du et al. (2019a); Truong (2025); Wu et al. (2023). Based on the NTK framework, several paper studies the convergence of training shallow neural networks under random neuron masking (e.g. dropout Srivastava et al. (2014)) Liao and Kyrillidis (2022); Mianjy and Arora (2020). However, these works usually considers the masking applied after the nonlinearity is applied, which allows direct computation of the statistics of the output and gradient under randomness. 3 Problem Setup Given a dataset (i,yi)i=1n \ (x_i,y_i ) \_i=1^n, we are interested in training a neural network f(,⋅)f ( θ,· ) that maps each input i∈ℝdx_i ^d’s to an output f(,i)f ( θ,x_i ) that fits the labels yi∈ℝy_i . We consider f(,⋅)f ( θ,· ) as a two-layer ReLU activated Multi-Layer Perceptron (MLP) under the NTK scaling: f(,)=1m∑r=1marσ(r⊤),f ( θ,x )= 1 m _r=1^ma_rσ (w_r x ), where =(rr=1m,arr=1m) θ= ( \w_r \_r=1^m, \a_r \_r=1^m ) denotes the neural network parameters, and (⋅)=max0,⋅ σ (· )= \0,·\ denotes the ReLU activation function. We assume that the second-layer weights ar∈±1a_r∈\± 1\ are fixed, and only the first layer weights rw_r’s are trainable. Thus, we will be using f(,)≡f(,)f(W,x)≡ f( θ,x) where ∈ℝm×dW ^m× d, unless otherwise stated. This neural network set-up is studied widely in previous works (Du et al., 2018). We consider the training of the neural network by minimizing the MSE loss ℒ()L (W ) over the dataset (i,yi)i=1n \ (x_i,y_i ) \_i=1^n: ℒ()=12∑i=1n(f(,i)−yi)2.L (W )= 12 _i=1^n (f (W,x_i )-y_i )^2. With the influence of the ReLU activation, the loss is both non-convex and non-smooth. However, a line of previous works (Du et al., 2018; Song and Yang, 2020; Oymak and Soltanolkotabi, 2019) proves a linear convergence rate of the loss function under the assumption that the number of hidden neurons is sufficiently large by adopting an NTK-based analysis (Jacot et al., 2018). While there have been theoretical approaches and assumptions that go beyond the NTK assumption, our focus is on a generalized scenario where the input data may be corrupted in each iteration under a multiplicative Gaussian noise: Let ∼(d,κ2d)c ( 1_d,κ^2I_d ) be an isotropic Gaussian random vector centered at the all-one vector d 1_d, the neural network output is given by f(,⊙)f (W,x c ), where ⊙ denotes the Hadamard (element-wise) product between two vectors. Under the multiplicative noise, the neural network is trained with gradient descent where each gradient is computed based on the surrogate loss ℒ()L_C (W ) defined over the neural network with the masked input: ℒ()=12∑i=1n(f(,i⊙i)−yi)2.L_C (W )= 12 _i=1^n (f (W,x_i c_i )-y_i )^2. Here =ii=1nC=\c_i\_i=1^n denotes the collection of the masks for all input ix_i. We assume that ic_i’s are independent. In real-world applications, this scheme can be considered as training on an imprecise hardware, where each input data point is read-in with noise. Alternatively, one could view each ix_i as the output of a pre-trained large model, and our training scheme can be considered as fine-tuning the last two layers with the Gaussian drop-out (Wang and Manning, 2013; Kingma et al., 2015; Rey and Mnih, 2021) in the intermediate layer. Let kk=1K \W_k \_k=1^K be generated from the stochastic gradient descent given by: k+1=k−η∇ℒk(k),W_k+1=W_k-η _WL_C_k (W_k ), (1) where kC_k is sampled independently in every iteration of the gradient descent. Our goal is to study the convergence of the loss sequence ℒ(k)k=1∞ \L (W_k ) \_k=1^∞. Notice that the loss involved in the weight-update is the surrogate loss ℒk(k)L_C_k (W_k ), but the loss we aim to show convergence is the original loss ℒ()L (W ). Our set-up marks some differences from previous works. First, our set-up is distinct from unbiased estimators in current literature; our setup does not have such favorable property, since the randomness is applied at the input level of the neural network. Second, although there is a line of work that analyzes the convergence of vanilla drop-out tranining on two-layer neural networks (Liao and Kyrillidis, 2022; Mianjy and Arora, 2020), in their analysis the mask is applied to the hidden neurons after the activation function. On the contrary, our mask is applied directly to the input, which is contained in the non-linear function. Therefore, any analysis of the mask randomness must go through the ReLU function, which brings technical difficulty. We assume the following property for the training data. Assumption 3.1. The training dataset (i,yi)i=1n\ (x_i,y_i )\_i=1^n satisfies ‖i‖2≤1,|yi|≤O(1) \|x_i \|_2≤ 1, |y_i |≤ O (1 ), and for any pair i≠ji≠ j, there exists no real number q such that i=q⋅jx_i=q·x_j. This assumption guarantees the boundedness of the dataset, and that the input data are non-degenerate, which is a standard assumption in Du et al. (2018); Song and Yang (2020); Liao and Kyrillidis (2022). 4 Expectation of the Loss and Gradient under Gaussian Mask A formal mathematical characterization of the expected loss and gradient is essential not only in prior literature of neural network training convergence (Liao and Kyrillidis, 2022; Mianjy and Arora, 2020) but also in the classical analysis of SGD even in the convex domain (Shamir and Zhang, 2013; Garrigos and Gower, 2023; Tang et al., 2013). In this section, we focus on the derivation of the explicit form of the expected surrogate loss [ℒ()]E_C [L_C (W ) ] and the expected surrogate gradient [∇ℒ()]E_C [ _WL_C (W ) ]. Starting with gradient calculations, the surrogate gradient with respect to the r-th neuron can be written as: ∇rℒ() _w_rL_C (W ) =arm∑i=1n(f(,i⊙i)−yi)(i⊙i)r⊤(i⊙i)≥0 = a_r m _i=1^n (f (W,x_i c_i )-y_i ) (x_i c_i )I \w_r (x_i c_i )≥ 0 \ (2) Setting i=c_i= 1 for all i∈[n]i∈[n] gives the gradient of the original loss ∇rℒ() _w_rL (W ). Let 1(⋅) _1 (· ) denote the CDF of the standard (one-dimensional) Gaussian random variable, and let ϕ,ψ:ℝ→ℝφ,ψ:R be defined as: ϕ(x)=exp(−x2);ψ(x)=|x|⋅ϕ(x).φ (x )= (-x^2 ); ψ (x )=|x|·φ (x ). (3) Observe that ϕ(x)∈(0,1]φ (x )∈(0,1] and ψ(x)∈(0,1/2e]ψ (x )∈ (0, 1 2e ]. Before we state the results in this section, we need to define the following quantities. Definition 4.1. Fix a first-layer weight ∈ℝm×dW ^m× d and training data (i,yi)i=1n\ (x_i,y_i )\_i=1^n. We define the: • Data-related quantity: B=maxi∈[n]‖i‖∞,By:=maxi∈[n]|yi|B_x= _i∈[n] \|x_i \|_∞,B_y:= _i∈[n] |y_i |. • Weight-related quantity (row-wise): R=maxr∈[m]‖r‖2R_w= _r∈[m] \|w_r \|_2. • Mixed quantity: R:=maxr∈[m],i∈[n]‖r⊙i‖2R_u:= _r∈[m],i∈[n] \|w_r x_i \|_2 and ψmax=maxr∈[m],i∈[n]ψ(r⊤i2κ‖r⊙i‖2),ϕmax=maxr∈[m],i∈[n]ϕ(r⊤i2κ‖r⊙i‖2). _ = _r∈[m],i∈[n]ψ ( w_r x_i2κ \|w_r x_i \|_2 ), _ = _r∈[m],i∈[n]φ ( w_r x_i2κ \|w_r x_i \|_2 ). (a) (b) Figure 1: (a). Effect of the noise standard deviation κ on the shape of the smoothed activation function σ^(z;κ)=z⋅Φ1(z/(κ‖⊙‖2)) σ(z;κ)=z· _1(z/(κ\|w \|_2)), where z=⊤z=w x. For this visualization, ‖⊙‖2\|w \|_2 is held constant at 1.01.0. As κ increases, the activation becomes progressively smoother compared to the standard ReLU (dotted black line). For small κ (e.g., κ=0.01κ=0.01), σ σ closely approximates the standard ReLU. (b). Theoretical smoothed activation σ^(,) σ(w,x) versus its empirical estimate [σ(⊤(⊙))]E_c[σ(w (x ))] for a fixed pre-activation value ⊤≈0.77w x≈ 0.77 (actual value depends on fixed ,w,x) as the noise standard deviation κ varies. The close match across a range of—relatively small—κ values validates the theoretical model for σ σ. Note that this behavior consistently follows empirically for different ,w,x values. Expected Surrogate Loss. To start, we focus on the expected loss under the Gaussian input mask. For a fixed neural network f(,⋅)f (W,· ), we have the following result. Theorem 4.2. Let i,r=r⊙iu_i,r=w_r x_i. Define the smoothed activation and neural network as: σ^κ(,) σ_κ (w,x ) =⊤⋅1(⊤κ‖⊙‖2),f^(,)=1m∑r=1marσ^κ(r,). =w x· _1 ( w xκ \|w x \|_2 ),\;\; f (W,x )= 1 m _r=1^ma_r σ_κ (w_r,x ). Let ϕmax,ψmax,By,R _ , _ ,B_y,R_u, and R_w be defined in Definition 4.1. If By≤3mRB_y≤ 3 mR_w, then we have that: [ℒ()]=ℰ+12∑i=1n(f^(,i)−yi)22⏟1+κ22m∑i=1n‖∑r=1mari,r1(r⊤iκ‖i,r‖2)‖22⏟2, _C [L_C (W ) ]=E+ 12 _i=1^n ( f (W,x_i )-y_i )_2^2_T_1+ κ^22m _i=1^n \| _r=1^ma_ru_i,r _1 ( w_r x_iκ \|u_i,r \|_2 ) \|_2^2_T_2, with the magnitude of ℰE bounded by: |ℰ|≤mn(κ2R2ψmax2+(κ2R2+κR)ϕmax2). |E |≤ mn (κ^2R_u^2 _ ^2+ (κ^2R_u^2+κ R_w ) _ ^2 ). (4) Remark 4.3. The core of our analysis of the expected loss involves understanding how the ReLU activation behaves under the multiplicative Gaussian input mask. Lemma D.14 in the appendix provides the analytical form for the expectation of a truncated Gaussian random variable. This leads to the definition of a smoothed activation function, as presented in Theorem 4.2. Figure 1(b) demonstrates the correspondence between the theoretical and empirical values of this smoothed activation across a range of noise levels κ for a fixed input ⊤w x: being an approximation of ReLU, as κ increases, it is expected the two curves to deviate, yet for small enough κ values (here, κ⪅0.2κ 0.2) the two curves coincide. Figure 2(b) visually compares this theoretical smoothed activation σ σ with its empirical estimate [σ(⊤(⊙))]E_c[σ(w (x ))] for a fixed κ=0.2κ=0.2 as the input ⊤w x varies. The close agreement validates our analytical derivation of σ σ and illustrates its smoothing effect compared to the standard ReLU. Thus, the term 1(⊤κ‖⊙‖2) _1 ( w xκ \|w x \|_2 ) can be interpreted as a smoothed version of the indicator function ⊤≥0I\w x≥ 0\. To visualize the impact of the noise variance κ2κ^2 on the shape of this smoothed activation, see Figure 1(a) for various values ,w,x (For this illustration, we assume a fixed value for ‖⊙‖2=1.0\|w \|_2=1.0 to isolate the effect of z=⊤z=w x and κ). As κ increases, the transition of σ σ around the origin becomes progressively gentler compared to the sharp kink of the standard ReLU activation. Remark 4.4. Theorem 4.2 shows that the expected loss can be approximated by the combination of terms 1T_1 and 2T_2, with an additive error term defined by ℰE. Notice that the smoothed activation σ^κ(,) σ_κ (w,x ) satisfies (see (19)): σ^(,) σ(w,x) =⊤⋅Φ1(⊤κ‖⊙‖2)=∼(,κ2)[σ(⊤(⊙))]±O(κ‖⊙‖2ϕ(⊤κ‖⊙‖2)) =w x· _1 ( w xκ \|w x \|2 )=E_c (1,κ^2I)[σ(w (x c))]± O (κ \|w x \|_2φ ( w xκ \|w x \|_2 ) ) Therefore, here 1T_1 can be seen as loss defined on the smoothed neural network f^(,⋅) f (W,· ) with the same weights and dataset. (a) Exact expectation σ~(,)=zΦ(z/σ)+σφ(z/σ) σ(w,x)=z (z/σ)+σ (z/σ) (b) Proxy σ^(,)=zΦ(z/σ) σ(w,x)=z (z/σ) Figure 2: Smoothed ReLU under multiplicative Gaussian input masking for fixed κ=0.2κ=0.2, where z=⊤z=w x and σ=κ‖⊙‖2σ=κ\|w \|_2. (a) Exact closed-form expectation σ~(,)=[σ(⊤(⊙))]=zΦ(z/σ)+σφ(z/σ) σ(w,x)=E_c[σ(w (x ))]=z (z/σ)+σ (z/σ) (as shown in 19)) matches the Monte Carlo estimate. (b) Proxy smoothed activation σ^(,)=zΦ(z/σ) σ(w,x)=z (z/σ) (used in Theorem 4.2) differs mainly near z≈0z≈ 0 due to the missing σφ(z/σ)σ (z/σ) term. Remark 4.5. One may notice that the form of 2T_2 is similar to the ℓ2 _2 regularization in the ridge regression. To understand 2T_2, we first notice that: ∇rf^(,i)≈1m∑r=1mari1(r⊤iκ‖i,r‖2). _w_r f (W,x_i )≈ 1 m _r=1^ma_rx_i _1 ( w_r x_iκ \|u_i,r \|_2 ). Therefore, 2T_2 can approximately be written as: 2≈κ22∑i=1n‖∑r=1m∇rf^(,i)⊙r‖22=κ22∑i=1n∑j=1d(∇^jf(,i)⊤^j)2=vec()⊤^vec(). _2≈ κ^22 _i=1^n \| _r=1^m _w_r f (W,x_i ) w_r \|_2^2= κ^22 _i=1^n _j=1^d ( _ w_jf (W,x_i ) w_j )^2= vec (W ) H vec (W ). Here ^j w_j is the jjth row of the matrix =[1,…,m]∈ℝd×mW= [w_1,…,w_m ] ^d× m, vec()=concat(^1,…,^d) vec (W )=concat ( w_1,…, w_d ) is the concatenation of the ^j w_j’s, and ^∈ℝmd×md H ^md× md is the block-diagonal matrix whose jjth diagonal block is ^j:=∑i=1n∇^jf(,i)∇^jf(,i)⊤∈ℝm×m H_j:= _i=1^n _ w_jf (W,x_i ) _ w_jf (W,x_i ) ^m× m for j∈[d]j∈[d]. Intuitively, H can be seen as a matrix consisting of the tangent features’ (Baratin et al., 2021; LeJeune and Alemohammad, 2024) outer products. As a result, 2T_2 can be seen as the regularization term of W in terms of a norm defined by the tangent feature outer product matrix H. Remark 4.6. The magnitude of ℰE is given in (4). At a first glance, one could see that the term decreases monotonically as κ decrease, implying a smaller error when κ is small. As discussed in the beginning of this section, ϕmax _ and ψmax _ are upper-bounded by some constant. Therefore, in the worst case, we have |ℰ|≤O(mn(κ2R+κR)) |E |≤ O (mn (κ^2R_u+κ R_w ) ), which scales linearly with κ. Below, we sketch the proof of Theorem 4.2. The full proof of Theorem 4.2 is deferred to Appendix B.2. Proof sketch. Our proof starts with the decomposition of the expected loss as: [ℒ()] _C [L_C (W ) ] =12∑i=1n[(f(,i⊙i))2]+12∑i=1nyi2−∑i=1nyi[(f(,i⊙i))]. = 12 _i=1^nE_C [ (f (W,x_i c_i ) )^2 ]+ 12 _i=1^ny_i^2- _i=1^ny_iE_C [ (f (W,x_i c_i ) ) ]. It boils down to analyzing the terms [(f(,i⊙i))2]E_C [ (f (W,x_i c_i ) )^2 ] and [(f(,i⊙i))]E_C [ (f (W,x_i c_i ) ) ]. Plugging in f(,i⊙i)f (W,x_i c_i ), it suffices to analyze the following expectations: E1 E_1 =[σ(r⊤(⊙))σ(r′⊤(⊙))];E2=[σ(r⊤(⊙))]. =E_c [σ (w_r (x c ) )σ (w_r (x c ) ) ]; E_2=E_c [σ (w_r (x c ) ) ]. The trick of evaluating E1E_1 and E2E_2 is to notice that r⊤(⊙)=⊤(r⊙i)w_r (x c )=c (w_r x_i ). Since ∼(,κ2)c ( 1,κ^2I ), we must have that ⊤(r⊙)∼(r⊤,κ2‖r⊙‖22)c (w_r x ) (w_r x,κ^2 \|w_r x \|_2^2 ). Therefore, we can define z1=⊤(r⊙)z_1=c (w_r x ) and z2=⊤(r′⊙)z_2=c (w_r x ). Then the problem of evaluating E1E_1 and E2E_2 becomes computing: E1 E_1 =z1,z2[z1z2z1≥0;z2≥0];E2=z1[z1z1≥0]. =E_z_1,z_2 [z_1z_2I \z_1≥ 0;z_2≥ 0 \ ]; E_2=E_z_1 [z_1I \z_1≥ 0 \ ]. Here Cov(z1,z2)=(r⊙i)⊤(r′⊙i)Cov (z_1,z_2 )= (w_r x_i ) (w_r x_i ). To complete the proof, we prove the following two lemmas. Lemma 4.7. Let z1∼(μ1,κ12)z_1 ( _1, _1^2 ). Then, we have that: [z1z1≥0]=κ2πexp(−μ22κ2)+μ1(μκ).E [z_1I \z_1≥ 0 \ ]= κ 2π (- μ^22κ^2 )+μ _1 ( μκ ). Lemma 4.8. Let z1∼(μ1,κ12)z_1 ( _1, _1^2 ) and z2∼(μ2,κ22)z_2 ( _2, _2^2 ), with Cov(z1,z2)=κ1κ2ρCov (z_1,z_2 )= _1 _2ρ. Let 2(a,b,ρ) _2 (a,b,ρ ) denote the joint CDF of standard Gaussian random variables z^1,z^2 z_1, z_2 with covariance ρ at z1=a,z2=bz_1=a,z_2=b. Then, we have: [z1z2z1≥0;z2≥0] [z_1z_2I \z_1≥ 0;z_2≥ 0 \ ] =(μ1μ2+κ1κ2ρ)2(μ1κ1,μ2κ2,ρ)+12π(κ1μ2T1+κ2μ1T2) = ( _1 _2+ _1 _2ρ ) _2 ( _1 _1, _2 _2,ρ )+ 1 2π ( _1 _2T_1+ _2 _1T_2 ) +κ1κ22πexp(−12(1−ρ2)(μ12κ12−2ρμ1μ2κ1κ2+μ22κ22)) + _1 _22π (- 12 (1-ρ^2 ) ( _1^2 _1^2- 2ρ _1 _2 _1 _2+ _2^2 _2^2 ) ) Here, T1,T2T_1,T_2 are defined as: T1=exp(−μ122κ12)1(11−ρ2(μ2κ2−ρμ1κ1));T2=exp(−μ222κ22)1(11−ρ2(μ1κ1−ρμ2κ2)) T_1= (- _1^22 _1^2 ) _1 ( 1 1-ρ^2 ( _2 _2- ρ _1 _1 ) );\;\;T_2= (- _2^22 _2^2 ) _1 ( 1 1-ρ^2 ( _1 _1- ρ _2 _2 ) ) Plugging z1=⊤(r⊙)z_1=c (w_r x ) and z2=⊤(r′⊤)z_2=c (w_r x ) back into [(f(,i⊙i))2]E_C [ (f (W,x_i c_i ) )^2 ] and [(f(,i⊙i))]E_C [ (f (W,x_i c_i ) ) ] and bounding the emerging error terms would give the desired result. Details are deferred into the appendix. Expected Surrogate Gradient. In the following part, we study the expectation of the surrogate gradient. Theorem 4.9. Assume that i∼(,κ2)c_i ( 1,κ^2I ) for some κ≤1κ≤ 1. Let ϕmax,ψmax,By,R _ , _ ,B_y,R_u, and R_w be defined in Definition 4.1. Then, we have that: [∇rℒ()]=∇rℒ()+r+3κ2m∑r′=1mar′Diag(i)2r′⋅r⊤i≥0;r′⊤i≥0 _C [ _w_rL_C (W ) ]= _w_rL (W )+g_r+ [rgb]0,.5,.5 [named]pgfstrokecolorrgb0,.5,.5 3κ^2m _r =1^ma_r Diag (x_i )^2w_r · [rgb]0,.5,.5 [named]pgfstrokecolorrgb0,.5,.5I \w_r x_i≥ 0;w_r x_i≥ 0 \ (5) where the magnitude of rg_r can be bounded as: ‖r‖2≤(6nκ2B2R+5nκRd)ϕmax+σmax()mϕmaxℒ()12+6nκRψmax. \|g_r \|_2≤ (6nκ^2B_x^2R_w+5nκ R_u d ) _ + _ (X ) m _ L (W ) 12+6nκ R_u _ . (6) Remark 4.10. Theorem 4.9 shows that the expected gradient can be written as the summation of the vanilla loss gradient ∇rℒ() _w_rL (W ), a term 3T_3 above, and a gradient error rg_r. The magnitude of rg_r is controlled in (6). As discussed previously, when |r⊤i|>0 |w_r x_i |>0, both ϕmax _ and ψmax _ decreases exponentially as κ decreases. Note that, although the second term scales with ℒ()L (W ), when ℒ()L (W ) decreases during training, that term will also contribute less to the overall gradient error. Remark 4.11. One could observe that the third term on the right-hand side of (5) is the gradient of the function ℛ()R (W ) with respect to rw_r, where ℛ()R (W ) is given by: ℛ()=3κ2m‖∑r′=1mar′r′⊙ir′⊤i≥0‖2.R (W )= 3κ^2m \| _r =1^ma_r w_r x_iI \w_r x_i≥ 0 \ \|_2. Therefore, ℛ()R (W ) can be seen as a scaled version of the 2T_2-term in Theorem 4.2. This again verifies the regularization effect of the Gaussian random mask. The proof of Theorem 4.9 is provided in Appendix B.3, and we provide the proof sketch below. Proof sketch. By the form of the surrogate gradient in (2), we focus on the following two terms: 1=f(,i⊙i)(i⊙i)r⊤(i⊙i)≥0;2=yi(i⊙i)r⊤(i⊙i)≥0. _1=f (W,x_i c_i ) (x_i c_i )I \w_r (x_i c_i )≥ 0 \;T_2=y_i (x_i c_i )I \w_r (x_i c_i )≥ 0 \. Plug in the form of f(,i⊙i)f(W,x_i c_i), we can write 1T_1 as: 1 _1 =1m∑r′=1mar′σ(r′⊤(i⊙i))⋅(i⊙i)r⊤(i⊙i)≥0 = 1 m _r =1^ma_r σ(w_r (x_i c_i ))· (x_i c_i )I \w_r (x_i c_i )≥ 0 \ =1m∑r′=1mar′Diag(i)ii⊤i⊤(r⊙i)≥0⋅i⊤(r′⊙i≥0)(r′⊙i), = 1 m _r =1^ma_r Diag (x_i )c_ic_i I \c_i (w_r x_i )≥ 0 \·I \c_i (w_r x_i≥ 0 ) \ (w_r x_i ), while 2T_2 can be written as: 2=yii⊙(ii⊤(r⊙i)≥0).T_2=y_ix_i (c_iI \c_i (w_r x_i )≥ 0 \ ). This allows us to focus on the following quantities instead [⊤⊤≥0;⊤≥0];[⊤≥0] [cc I \c u≥ 0;c v≥ 0 \ ];E [cI \c u≥ 0 \ ] with multi-variate Gaussian random variable analysis. The rest of the proof then proceeds similarly as in the proof of Theorem 4.2. 5 Convergence Guarantee of Training with Gaussian Mask Here, we study the convergence property of a general framework of stochastic neural network training, which gives us a theoretical result that can be of independent interest. Recall also the setting in Section 3: Consider f(,⋅)f (W,· ) as a two-layer ReLU activated MLP, as described above. Let ξ denote the randomness in one step of (stochastic) gradient descent. Let ℒ^(,) L (W, ξ ) and ∇rℒ^(,) _w_r L (W, ξ ) denote the stochastic loss and the stochastic gradient induced by ξ, respectively. We consider the sequence kk=1K \W_k \_k=1^K generated by the following updates: k+1=k−η∇ℒ^(k,k).W_k+1=W_k-η _W L (W_k, ξ_k ). (7) Instead, the connection between ∇rℒ^(,) _w_r L (W, ξ ) and ℒ^(,) L (W, ξ ) with respect to rw_r, along with other requirements, are stated in the assumption below. Assumption 5.1. For all , ξ,W, we assume that the following properties hold: [ℒ^(,)]≤2ℒ()+ε1, _ ξ [ L (W, ξ ) ]≤ 2L (W )+ _1, (8) ‖[∇rℒ^(,)]−∇rℒ()‖2≤ε3ℒ()12+ε2, \|E_ ξ [ _w_r L (W, ξ ) ]- _w_rL (W ) \|_2≤ _3L (W ) 12+ _2, (9) ‖∇rℒ^(,)‖22≤γℒ^(,). \| _w_r L (W, ξ ) \|_2^2≤γ L (W, ξ ). (10) Here, (8) and (9) provide an upper bound on the expected loss and the error of the expected gradient. (10) can be seen as a relaxed form of the smoothness. Our analysis is based on the standard NTK-type argument as in (Du et al., 2018; Song and Yang, 2020; Liao and Kyrillidis, 2022), which considers the infinite-width NTK ∞H^∞ given by: ij∞=i⊤j∼(,)[⊤i≥0;⊤j≥0].H^∞_ij=x_i x_jE_w ( 0,I ) [I \w x_i≥ 0;w x_j≥ 0 \ ]. (11) It is shown in Du et al. (2018) that ∞H^∞ is positive definite. We define λ0:=λmin(∞)>0 _0:= _ (H^∞ )>0. Theorem 5.2. Assume that the first-layer weights of a neural network are initialized according to 0,r∼(,τ2)w_0,r ( 0,τ^2I ) for some τ>0τ>0, and the second-layer weights are initialized according to r∼Unif±1a_r Unif\± 1\. Let the number of hidden neurons satisfy m=Ω(n4K2λ04δ2τ2)m= ( n^4K^2 _0^4δ^2τ^2 ) and the step size satisfy η=O(λ0n2)η=O ( _0n^2 ). Assume that Assumptions 3.1, 5.1 hold for some γ=C1⋅nmγ=C_1· nm with some small enough ε1,ε2,ε3 _1, _2, _3 satisfying: ε1≤O(δmnK4),ε2≤O(δλ0nK2),ε3≤O(λ0mn). _1≤ O ( δ mnK^4 ), _2≤ O ( δ _0nK^2 ), _3≤ O ( _0 mn ). (12) Then, with probability at least 1−2δ−n2exp(−n3δ2τ2λ03)1-2δ-n^2 (- n^3δ^2τ^2 _0^3 ), for all k∈[K]k∈[K], the sequence kk=1K\W_k\_k=1^K generated by (7) satisfies: 0,…,k−1[ℒ(k)] _ ξ_0,…, ξ_k-1 [L (W_k ) ] ≤(1−ηλ02)kℒ(0)+O(mnλ02⋅ε22+ε1). ≤ (1- η _02 )^kL (W_0 )+O ( mn _0^2· _2^2+ _1 ). (13) Furthermore, we can guarantee that ‖k,r−0,r‖2≤O(τλ0n) \|w_k,r-w_0,r \|_2≤ O ( τ _0n ) for all r∈[m]r∈[m] and k∈[K]k∈[K]. In short, Theorem 5.2 shows that under a small enough ε1,ε2,ε3 _1, _2, _3 and γ, if the neural network is sufficiently overparameterized, then, with a small enough step size η, we can guarantee the convergence under the training given by (7) and that the change in each rw_r is bounded by O(τλ0n)O ( τ _0n ). As shown in (13), the expected loss converges linearly up to a ball around the global minimum with radius given by O(mnλ02⋅ε22+ε1)O ( mn _0^2· _2^2+ _1 ). This error region monotonically decreases as the error in the expected loss and gradient, namely ε1 _1 and ε2 _2, decreases. Training Convergence with Gaussian Input Mask. We apply the general result in Theorem 5.2 to the scenarios of Gaussian input masking, as given by (1). To apply Theorem 5.2, one need to make sure that the requirements in Assumption 5.1 are guaranteed. Here, we present two corollaries as extensions of Theorem 4.2 and Theorem 4.9, with the goal of showing (8) and (9). Corollary 5.3. Let By,ϕmax,ψmax,RB_y, _ , _ ,R_u, and R_w be defined in Definition 4.1. If By≤3mRB_y≤ 3 mR_w, then we have [ℒ()]≤2ℒ()+2mnκ2R2+mn(κ2R2+κR)ϕmax2+mnκ2(R2+1)ψmax2 _C [L_C (W ) ]≤ 2L (W )+2mnκ^2R_u^2+mn (κ^2R_u^2+κ R_w ) _ ^2+mnκ^2 (R_u^2+1 ) _ ^2 Corollary 5.3 follows simply from Theorem 4.2 by upper-bounding the difference between the smoothed neural network function f^(,⋅) f (W,· ) and the vanilla neural network function f(,⋅)f (W,· ), and by upper-bounding the regularization term. In particular, Corollary 5.3 implies the bound of the error ε1 _1 as 2mnκ2R+mn(κ2R2+κR)ϕmax2+mnκ2(R2+1)ψmax22mnκ^2R_u+mn (κ^2R_u^2+κ R_w ) _ ^2+mnκ^2 (R_u^2+1 ) _ ^2. Corollary 5.4. Let By,ϕmax,ψmax,RB_y, _ , _ ,R_u, and R_w be defined in Definition 4.1. If By≤3mRB_y≤ 3 mR_w, then we have ‖[∇rℒC()]−∇rℒ()‖2 \|E_C [ _w_rL_C (W ) ]- _w_rL (W ) \|_2 ≤O((nκ2B2R+nκRd)ϕmax)+O(σmax()ϕmaxm)ℒ()12 ≤ O ( (nκ^2B_x^2R_w+nκ R_u d ) _ )+O ( _ (X ) _ m )L (W ) 12 +O(nκRψmax+κ2mB2R) +O (nκ R_u _ +κ^2 mB_x^2R_w ) Similar to Corollary 5.3, Corollary 5.4 follows from upper bounding the regularization term in Theorem 4.9. By Corollary 5.4, we can write ε2 _2 and ε3 _3 in Assumption 5.1 as ε2=O((nκ2B2R+nκRd)ϕmax)+O(nκRψmax+κ2mB2R) _2=O ( (nκ^2B_x^2R_w+nκ R_u d ) _ )+O (nκ R_u _ +κ^2 mB_x^2R_w ) and likely ε3=O(σmax()ϕmaxm) _3=O ( _ (X ) _ m ). The proof of Corollary 5.3 and Corollary 5.4 are deferred to Appendix C.2. To complete the requirements in Assumption 5.1, we can show the following lemma for (10). Lemma 5.5. Assume that Assumption 3.1 holds. Then, we have: ‖∇rℒ()‖22≤C1nmℒ(). \| _w_rL_C (W ) \|_2^2≤ C_1 nmL_C (W ). The proof can be found in the appendix D.21 With the help of Corollary 5.3,5.4, and Lemma D.21, we can derive the convergence guarantee of training the two-layer ReLU neural network under Gaussian input mask. Theorem 5.6. Assume that the first-layer weights are initialized according to 0,r∼(,τ2)w_0,r ( 0,τ^2I ) for some τ>0τ>0, and the second-layer weights are initialized according to r∼Unif±1a_r Unif\± 1\. Let the number of hidden neurons satisfy m=Ω(n4K2λ04δ2τ2)m= ( n^4K^2 _0^4δ^2τ^2 ) and the step size satisfy η=O(λ0n2)η=O ( _0n^2 ). Assume that for all ∈kk=1KW∈ \W_k \_k=1^K, the following hold: κ=O(δλ0τ2K2(m14d+nd)(ϕ^max+ψ^max)) κ=O ( δ _0τ^2K^2 (m 14 d+nd ) ( φ_ + ψ_ ) ) (14) σmax()ϕ^max≤O(λ0n). _ (X ) φ_ ≤ O ( _0 n ). (15) Then, we have that, with probability at least 1−2δ−n2exp(−n3δ2τ2λ03)1-2δ-n^2 (- n^3δ^2τ^2 _0^3 ), for all k∈[K]k∈[K], the sequence kk=1K\W_k\_k=1^K generated by (7) satisfies: 0,…,k−1[ℒ(k)] _C_0,…,C_k-1 [L (W_k ) ] ≤(1−ηλ02)kℒ(0)+O(κτ2mn2d2(ϕ^max2+ψ^max2)) ≤ (1- η _02 )^kL (W_0 )+ [rgb].75,0,.25 [named]pgfstrokecolorrgb.75,0,.25O (κτ^2mn^2d^2 ( φ_ ^2+ ψ_ ^2 ) ) (16) +O(κ2τ2m2nd)+O(κτmndϕ^max2) + [rgb]0,.5,.5 [named]pgfstrokecolorrgb0,.5,.5O (κ^2τ^2m^2nd )+ [rgb]1,.5,0 [named]pgfstrokecolorrgb1,.5,0O (κτ mn d φ_ ^2 ) (17) where ϕ^max=maxk∈[K]ϕmax(k) φ_ = _k∈[K] _ (W_k ) and ψ^max=maxk∈[K]ψmax(k) ψ_ = _k∈[K] _ (W_k ). In short, Theorem 5.6 guarantees the convergence of training a two-layer ReLU neural network under Gaussian input mask in the form of (16) under the condition of sufficient overparameterization, proper step size, and the requirement in (14) and (15). In particular, (14) requires a sufficiently small Gaussian variance κ. The condition in (15) requires either a small maximum singular value of the input data matrix X, or a small ϕmax _ . Lastly, (16) shows a linear convergence of the expected loss up to some error region. Notice that the first part of the error region depends both on κ,τκ,τ and on ϕmax _ and ψmax _ , and the second part of the error region depends solely on κ and τ. This means that one can guarantee an arbitrarily small error region when the Gaussian noise κ and the initialization scale τ is sufficiently small. Remark 5.7. Both the requirement and the error region in Theorem 5.6 depend on the quantity ϕ^max φ_ and ψ^max ψ_ . Recall that: ϕ^max=maxk,r,iexp(−(r⊤i)4κ2‖r⊙i‖22);ψ^max=maxk,r,i|r⊤i|⋅exp(−(r⊤i)4κ2‖r⊙i‖22). φ_ = _k,r,i \ (- (w_r x_i )4κ^2 \|w_r x_i \|_2^2 ) \;\;\; ψ_ = _k,r,i \ |w_r x_i |· (- (w_r x_i )4κ^2 \|w_r x_i \|_2^2 ) \. Both quantities decay exponentially fast as κ decays, when r⊤i≠0w_r x_i≠ 0 for all k,r,ik,r,i. As the sequence kk=1K \W_k \_k=1^K is generated under the randomness of kC_k’s, intuitively it is almost never the case where k,r⊤i=0w_k,r x_i=0. Therefore, in most cases Theorem 5.6 should require only a log-dependency of κ on other parameters in order for (14) and (15) to be satisfied. However, it should be noticed that κ still need to decay in powerlaw if one want to sufficiently decrease the second part of the error region. 6 Experiments 6.1 Empirical Validation of Training Convergence with Gaussian Mask. Theorem 5.6 asserts that training a sufficiently overparameterized two-layer ReLU network with Gaussian multiplicative input noise results in linear convergence of the expected loss to an error ball. The radius of this error ball is proportional to the noise variance (controlled by κ) and other network and data-dependent terms. We empirically verify this convergence behavior. Simulation Setup. We train a two-layer ReLU MLP: As a toy example, the network has d=20d=20 input features and m=100m=100 hidden units. The training dataset comprised n=500n=500 synthetic samples, with input features ix_i normalized such that ‖i‖2≤1\|x_i\|_2≤ 1, and target values yiy_i generated from a non-linear function of ix_i with small added noise. First-layer weights W were initialized using Kaiming uniform initialization, and second-layer weights ar∈±1a_r∈\± 1\ were fixed. The network was trained for 20002000 iterations using full-batch gradient descent with a learning rate of 0.0050.005. We performed separate training runs for different noise levels: κ∈0.0,0.05,0.2,0.4,0.6,1.0,2.0κ∈\0.0,0.05,0.2,0.4,0.6,1.0,2.0\. For each run, we tracked the evolution of the clean training loss ℒ(k)L(W_k). Results and Discussion. The training trajectories, plotted in Figure 3(a), illustrate the theoretical predictions. For clean training (κ=0.0κ=0.0), the loss exhibits an initial linear convergence phase. When multiplicative Gaussian noise is introduced, the initial linear convergence trend is preserved. However, as training progresses, the loss converges not to the same minimal value but to a distinct error ball, plateauing at a value higher than the clean case. As expected, the size of this error ball, indicated by the final converged loss value, systematically increases with the noise level κ. This direct relationship between κ and the size of the error ball provides strong empirical support for the convergence guarantees outlined in Theorem 5.6. (a) (b) Figure 3: (a). Training loss ℒ(k)L(W_k) (log-scale) versus training iteration for a two-layer ReLU network (n=500,d=20,m=100n=500,d=20,m=100) trained with full-batch gradient descent under different levels of input multiplicative Gaussian noise standard deviation κ. (b). Distributed training with Gaussian mask for differen κ and number of local steps. 6.2 Impact of Multiplicative Gaussian Noise on Model Accuracy We trained a 1-hidden-layer MLP (4096 hidden units, GELU activation, dropout=0.2) on CIFAR-10 for 80 epochs using AdamW optimization with cosine annealing, label smoothing, and standard data augmentation. During training, we injected multiplicative Gaussian noise x←x⋅(1+κZ)where Z∼(0,1).x← x·(1+κ Z) Z (0,1). and evaluated the impact of noise strength κ on clean test accuracy. The results reveal that small amounts of MG noise κ≈0.2κ≈ 0.2 improve generalization, acting as an effective regularizer beyond the existing random cropping and flipping. This suggests that modest input perturbations help the model learn more robust features that transfer better to the test set. However, accuracy degrades monotonically beyond this point, dropping to 49.88% at κ=1.8.κ=1.8. (a) MLP (b) CNN Figure 4: Test accuracy versus multiplicative Gaussian noise strength κ for (a) a 1-hidden-layer MLP and (b) a CNN, trained on CIFAR-10. Small noise levels (κ≈0.2)κ≈ 0.2) can improve generalization for the MLP, likely due to regularization effects. In contrast, the CNN exhibits robustness by maintaining baseline accuracy. Beyond this point, accuracy degrades monotonically for both architectures as noise corrupts the training signal. Next, to assess the impact of multiplicative Gaussian noise on a well-regularized convolutional architecture, we trained a CNN on CIFAR-10 with varying noise strengths κ. The architecture consists of 44 convolutional layers (32→32→64→6432→ 32→ 64→ 64 filters with 3×33× 3 kernels), batch normalization after each convolutional layer, three dropout layers with rates 0.2,0.30.2,0.3 and 0.50.5 respectively, and a fully connected layer with ReLU activation. We trained the CNN for 80 epochs using the Adam optimizer with lr=10−3lr=10^-3, weight decay=10−5=10^-5 and batch size=128=128. During training, we applied Multiplicative Gaussian (MG) noise x←x⋅(1+κZ)where Z∼(0,1)x← x·(1+κ Z) Z (0,1), while all accuracy measurements were performed on the clean test set (noiseless). As shown in Figure 4(b), we observe that the model exhibits robustness to low-magnitude MG noise, maintaining its baseline accuracy (≈71%≈ 71\%) at κ=0.2κ=0.2. Beyond that point though, accuracy degrades monotonically and excessive noise corrupts the training signal. 6.3 Application: Distributed Training over Wireless Channels Communicating signals over wireless channels incurs fading phenomena to the signals (Tse and Viswanath, 2005). Specifically, a time-varying signal x(t)x(t) transmitted over channel given by h(t)h(t) and additive noise z(t)z(t) result in y(t)=x(t)h(t)+z(t)y(t)=x(t)h(t)+z(t). For data parallel distributed training over wireless channels, each input data is passed through the channel to the workers to perform local training. We consider using the Gaussian masked input training scheme studied in this paper as a simplified setup to model the channel fading in wireless communication. In particular, we let x be the signals transmitted x(t)x(t), and ∼(,κ2d)c ( 1,κ^2I_d ) be the channel effect h(t)h(t). For simplicity, we set the additive noise to 0. The time-depending behavior of x(t)x(t) and h(t)h(t) are transformed into the masking scheme that in each iteration, a new mask is applied to the sample. Under this setup, we train a two-layer MLP with 128 hidden neurons for the MNIST dataset using FedAvg. That is, we assume that the total training process is partitioned into multiple global iterations, where in each global iteration, the central server passes the updated model parameter together with the current copy of training data through a wireless channel. Each worker receives the training data with channel fading (in our case, modeled with Gaussian multiplicative noise), and updated its local copy of the model using gradient descent starting from the parameter shared by the server for some number of local steps. After the local update, the workers sends the updated parameter to the central server to perform an aggregation by averaing the worker’s weights. Under this setup, we train an aggregated model with 5 workers and the choice of 1, 20, 40 local steps with a batch size of 128. We also vary the variance of the Gaussian mask to study the relationship between the number of local steps and the noise scales. For each combination of local steps and Gaussian variance, we run 5 trials and record the mean and standard deviation of the resulting accuracy. We plot the result in Figure 3(b). In general, for all choices of the number of local steps, we observe a decay in the test accuracy as the input masking variance grows larger, indicating the negative influence of the noise to the overall training performance. In particular, we can also observe that, in the low noise regime (κ≈0κ≈ 0), the resulting final accuracy is not influenced much by the number of local iterations. However, as the noise scale grows larger (larger κ), the final accuracy decays drastically as we increase the number of local iterations. We hypothesis that this behavior is due to the fact more local iterations allows the workers to fit more to the noise instead of the original signals in the data. 6.4 Efficacy of Multiplicative Gaussian Noise Against Membership Inference Attacks In this section, we empirically evaluate the effectiveness of training with input multiplicative Gaussian (MG) noise as a defense against Membership Inference Attacks (MIAs). In simple words, the primary goal of a MIA is to determine if a specific data point x was part of the training set of a target model f. Threat Model and Attack Methodology. We adopt the black-box shadow model attack methodology (Adversary 1) from the ML-Leaks framework (Salem et al., 2019). In this setup, the adversary aims to determine whether a specific data point was part of a target model’s training set using only the model’s output posteriors. Because the adversary lacks the target’s training labels, they employ a shadow model trained on a proxy dataset to mimic the target’s behavior. By observing how the shadow model treats its own members versus non-members, the adversary generates labeled data to train an attack model. This binary classifier learns to identify the statistical "signatures" of membership—such as increased confidence or reduced entropy—enabling it to perform membership inference on the original target model. A detailed breakdown of the data partitioning and the five-stage attack pipeline is provided in Appendix A.2. Experimental Setup. We evaluate the effectiveness of Multiplicative Gaussian (MG) noise as a privacy defense using the CIFAR-10 dataset. The data is partitioned into four disjoint sets to train and audit both target and shadow models independently. Our evaluation covers two architectures—a fully-connected MLP and a multi-block CNN—to ensure the defense generalizes across different model complexities. We measure privacy leakage by training a Logistic Regression attack model against target models subjected to varying noise intensities (κ∈0.0,0.5,1.2,1.8κ∈\0.0,0.5,1.2,1.8\) and training durations (20–120 epochs). The defense is quantified via Precision, Recall, and AUC, where an AUC of 0.5 indicates perfect privacy (random guessing). Detailed hyperparameters, partitioning sizes, and architectural specifications are provided in Appendix A.2. 6.4.1 Results and Discussion Our experiments confirm that training with multiplicative Gaussian noise systematically enhances a model’s resilience to membership inference attacks. The results for the MLP and CNN model are presented in Figure 6 and 7 respectively, with the AUC values of our experiments illustrated in Figure 5. (a) MLP (b) CNN Figure 5: Attack AUC on the target model when it is (a) an MLP and (b) a CNN. Higher values indicate greater privacy leakage. Training with MG noise (larger κ) consistently reduces attack success. Figure 6: MLP target. Multiplicative Gaussian noise provides resilience against the attacks as κ increases. Figure 7: CNN target. We observe a similar trend as the MLP Figure 7 demonstrates the attack success for κ=0.0κ=0.0, in which case as the number of training epochs increases from 20 to 120, AUC rises from 0.5780.578 to a significant 0.7820.782. This validates that our attack implementation correctly captures privacy leakage. The central finding is the consistent defensive benefit of multiplicative Gaussian noise. As shown in Figures 6 for MLP, for any given number of training epochs, applying MG noise (increasing κ) retains (relatively) constant both the precision and recall of the attack. For example, after 120 epochs of training, the standard model is highly vulnerable (Attack AUC = 0.7820.782). In contrast, the model trained with κ=0.5κ=0.5 reduces this leakage (AUC = 0.6920.692), and models with stronger noise achieve even better privacy (AUC = 0.5850.585 for κ=1.2κ=1.2 and AUC = 0.5430.543 for κ=1.8κ=1.8). This demonstrates a clear dose-response relationship: greater noise variance leads to stronger privacy protection against MIAs. Similar trends were observed for the CNN architecture (see Figure 7). Figure 4 illustrates the classic privacy-utility trade-off. In essence, by sacrificing some model utility, training with multiplicative Gaussian noise effectively obfuscates the statistical signature of data membership, thereby mitigating privacy risks. 7 Conclusion This work investigates the fundamental question of how independent multiplicative Gaussian masking affects training dynamics. Focusing on a two-layer ReLU network in the NTK regime, we demonstrate that the masked objective admits a closed-form decomposition into a smoothed square loss plus an explicit, data-dependent regularizer. This structure allows training with the gradient from the masked objective to achieve linear convergence toward a noise-controlled error ball given small step size and large enough over-parameterization. Beyong our theory, We provided experimental result to validate the convergence to small ball, and presented applications in distributed training under channel fading and how the masking can defend against attacks. Limitations and Future Work. Our theoretical analysis is currently constrained to the small-noise regime and the extreme over-parameterization typical of NTK models. Furthermore, our proofs rely on the independence of masks across iterations and do not yet incorporate a formal privacy accounting pipeline, such as subsampling or composition. Despite these constraints, multiplicative Gaussian masking represents a provable and practically viable method for injecting input-level uncertainty. These results provide a principled foundation for future exploration of deep networks and more complex noise settings in feature-wise training. References Bach et al. [2011] Francis Bach, Rodolphe Jenatton, Julien Mairal, and Guillaume Obozinski. Convex optimization with sparsity-inducing norms. Optimization for Machine Learning, page 19, 2011. Bach et al. [2012] Francis Bach, Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, et al. Optimization with sparsity-inducing penalties. Foundations and Trends® in Machine Learning, 4(1):1–106, 2012. Baratin et al. [2021] Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In International Conference on Artificial Intelligence and Statistics, pages 2269–2277. PMLR, 2021. Bartlett et al. [2017] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in Neural Information Processing Systems, 30, 2017. Cheng et al. [2020] Yong Cheng, Yang Liu, Tianjian Chen, and Qiang Yang. Federated learning for privacy-preserving AI. Communications of the ACM, 63(12):33–36, 2020. Cohen et al. [2019] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pages 1310–1320. PMLR, 2019. Du et al. [2018] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054, 2018. Du et al. [2019a] Simon S. Du, Kangcheng Hou, Barnabás Póczos, Ruslan Salakhutdinov, Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernels, 2019a. URL https://arxiv.org/abs/1905.13192. Du et al. [2019b] Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks, 2019b. URL https://arxiv.org/abs/1811.03804. Dun et al. [2022] Chen Dun, Cameron R Wolfe, Christopher M Jermaine, and Anastasios Kyrillidis. ResIST: Layer-wise decomposition of resnets for distributed training. In Uncertainty in Artificial Intelligence, pages 610–620. PMLR, 2022. Dun et al. [2023] Chen Dun, Mirian Hipolito, Chris Jermaine, Dimitrios Dimitriadis, and Anastasios Kyrillidis. Efficient and light-weight federated learning via asynchronous distributed dropout. In International Conference on Artificial Intelligence and Statistics, pages 6630–6660. PMLR, 2023. Edwards [2024] Chris Edwards. Data quality may be all you need, 2024. Gao et al. [2019] Ruiqi Gao, Tianle Cai, Haochuan Li, Liwei Wang, Cho-Jui Hsieh, and Jason D. Lee. Convergence of adversarial training in overparametrized neural networks, 2019. URL https://arxiv.org/abs/1906.07916. Garrigos and Gower [2023] Guillaume Garrigos and Robert M. Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.11235, 2023. URL https://arxiv.org/abs/2301.11235. Ghorbani et al. [2021] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layer neural networks in high dimension. The Annals of Statistics, 49(2):1029–1054, 2021. Gunasekar et al. [2023] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. He et al. [2020] Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. FedML: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13518, 2020. Hu et al. [2023] Erdong Hu, Yuxin Tang, Anastasios Kyrillidis, and Chris Jermaine. Federated learning over images: Vertical decompositions and pre-trained backbones are difficult to beat. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19385–19396, 2023. Ilyas et al. [2019] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in Neural Information Processing Systems, 32, 2019. Jacot et al. [2018] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 31, 2018. Jenatton et al. [2011] Rodolphe Jenatton, Jean-Yves Audibert, and Francis Bach. Structured variable selection with sparsity-inducing norms. The Journal of Machine Learning Research, 12:2777–2824, 2011. Ji and Telgarsky [2020] Ziwei Ji and Matus Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks, 2020. URL https://arxiv.org/abs/1909.12292. Kairouz et al. [2021] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021. Kariotakis et al. [2024] Emmanouil Kariotakis, Grigorios Tsagkatakis, Panagiotis Tsakalides, and Anastasios Kyrillidis. Leveraging sparse input and sparse models: Efficient distributed learning in resource-constrained environments. In Conference on Parsimony and Learning, pages 554–569. PMLR, 2024. Kingma et al. [2015] Diederik P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems, volume 28, pages 2575–2583, 2015. URL https://papers.nips.c/paper/5666-variational-dropout-and-the-local-reparameterization-trick. Kyrillidis et al. [2015] Anastasios Kyrillidis, Luca Baldassarre, Marwa El Halabi, Quoc Tran-Dinh, and Volkan Cevher. Structured sparsity: Discrete and convex approaches. In Compressed Sensing and its Applications: MATHEON Workshop 2013, pages 341–387. Springer, 2015. LeJeune and Alemohammad [2024] Daniel LeJeune and Sina Alemohammad. An adaptive tangent feature perspective of neural networks. In Yuejie Chi, Gintare Karolina Dziugaite, Qing Qu, Atlas Wang, and Zhihui Zhu, editors, Conference on Parsimony and Learning, volume 234 of Proceedings of Machine Learning Research, pages 379–394. PMLR, 03–06 Jan 2024. URL https://proceedings.mlr.press/v234/lejeune24a.html. Li et al. [2023a] Guanlin Li, Han Qiu, Shangwei Guo, Jiwei Li, and Tianwei Zhang. Rethinking adversarial training with neural tangent kernel. arXiv preprint arXiv:2312.02236, 2023a. Li et al. [2025] Shuyao Li, Ilias Diakonikolas, and Jelena Diakonikolas. Distributionally robust optimization with adversarial data contamination, 2025. URL https://arxiv.org/abs/2507.10718. Li et al. [2023b] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need I: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023b. Liao and Kyrillidis [2022] Fangshuo Liao and Anastasios Kyrillidis. On the convergence of shallow neural network training with randomly masked neurons. Transactions on Machine Learning Research, 2022. URL https://openreview.net/forum?id=ebZ0gGRJwQx. Liu et al. [2021] Yang Liu, Tao Fan, Tianjian Chen, Qian Xu, and Qiang Yang. FATE: An industrial grade platform for collaborative learning with data protection. Journal of Machine Learning Research, 22(226):1–6, 2021. Liu et al. [2022] Yang Liu, Xinwei Zhang, Yan Kang, Liping Li, Tianjian Chen, Mingyi Hong, and Qiang Yang. FedBCD: A communication-efficient collaborative learning framework for distributed features. IEEE Transactions on Signal Processing, 70:4277–4290, 2022. Liu et al. [2024] Yang Liu, Yan Kang, Tianyuan Zou, Yanhong Pu, Yuanqin He, Xiaozhou Ye, Ye Ouyang, Ya-Qin Zhang, and Qiang Yang. Vertical federated learning: Concepts, advances, and challenges. IEEE Transactions on Knowledge and Data Engineering, 2024. Loo et al. [2022] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Evolution of neural tangent kernels under benign and adversarial training. Advances in Neural Information Processing Systems, 35:11642–11657, 2022. Madry et al. [2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. McMahan et al. [2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273–1282. PMLR, 2017. Mianjy and Arora [2020] Poorya Mianjy and Raman Arora. On convergence and generalization of dropout training. In Advances in Neural Information Processing Systems, volume 33, pages 14124–14134, 2020. URL https://proceedings.neurips.c/paper/2020/file/f1de5100906f31712a5166689bfdf4-Paper.pdf. Miyato et al. [2018] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):1979–1993, 2018. Nguyen [2021] Quynh Nguyen. On the proof of global convergence of gradient descent for deep relu networks with linear widths, 2021. URL https://arxiv.org/abs/2101.09612. Oymak and Soltanolkotabi [2019] Samet Oymak and Mahdi Soltanolkotabi. Towards moderate overparameterization: Global convergence guarantees for training shallow neural networks, 2019. URL https://arxiv.org/abs/1902.04674. Rey and Mnih [2021] Mélanie Rey and Andriy Mnih. Gaussian dropout as an information bottleneck layer. In Bayesian Deep Learning Workshop, NeurIPS, 2021. URL https://bayesiandeeplearning.org/2021/papers/40.pdf. Romanini et al. [2021] Daniele Romanini, Adam James Hall, Pavlos Papadopoulos, Tom Titcombe, Abbas Ismail, Tudor Cebere, Robert Sandmann, Robin Roehm, and Michael A Hoeh. PyVertical: A vertical federated learning framework for multi-headed SplitNN. arXiv preprint arXiv:2104.00489, 2021. Ruder [2017] Sebastian Ruder. An overview of gradient descent optimization algorithms, 2017. URL https://arxiv.org/abs/1609.04747. Salem et al. [2019] Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. In Proceedings of the 2019 Network and Distributed System Security Symposium (NDSS), 2019. ISBN 1-891562-55-X. URL https://w.ndss-symposium.org/ndss2019/papers/ndss2019_03A-1_Salem_paper.pdf. Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278–25294, 2022. Shamir and Zhang [2013] Ohad Shamir and Tong Zhang. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In Proceedings of the 30th International Conference on Machine Learning, pages 71–79, 2013. URL https://proceedings.mlr.press/v28/shamir13.html. Song and Yang [2020] Zhao Song and Xin Yang. Quadratic suffices for over-parametrization via matrix chernoff bound, 2020. URL https://arxiv.org/abs/1906.03593. Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014. Tang et al. [2013] Cheng Tang et al. Convergence analysis of stochastic gradient descent on strongly convex functions. In Proceedings of the 2013 European Signal Processing Conference, pages 1568–1572, 2013. URL https://w.esat.kuleuven.be/sista/ROKS2013/files/abstracts/ChengTang.pdf. Truong [2025] Lan V. Truong. Global convergence rate of deep equilibrium models with general activations, 2025. URL https://arxiv.org/abs/2302.05797. Tse and Viswanath [2005] David Tse and Pramod Viswanath. Fundamentals of Wireless Communication. Cambridge University Press, Cambridge, United Kingdom, 2005. ISBN 9780521845274. doi: 10.1017/CBO9780511807213. Tsilivis and Kempe [2022] Nikolaos Tsilivis and Julia Kempe. What can the neural tangent kernel tell us about adversarial robustness? Advances in Neural Information Processing Systems, 35:18116–18130, 2022. Wang and Manning [2013] Sida Wang and Christopher D Manning. Fast dropout training. In Proceedings of the 30th International Conference on Machine Learning, pages 118–126, 2013. URL https://proceedings.mlr.press/v28/wang13a.html. Wolfe et al. [2023] Cameron R Wolfe, Jingkang Yang, Fangshuo Liao, Arindam Chowdhury, Chen Dun, Artun Bayer, Santiago Segarra, and Anastasios Kyrillidis. GIST: Distributed training for large-scale graph convolutional networks. Journal of Applied and Computational Topology, pages 1–53, 2023. Wong et al. [2018] Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and J Zico Kolter. Scaling provable adversarial defenses. Advances in Neural Information Processing Systems, 31, 2018. Wu et al. [2023] Yongtao Wu, Fanghui Liu, Grigorios G Chrysos, and Volkan Cevher. On the convergence of encoder-only shallow transformers, 2023. URL https://arxiv.org/abs/2311.01575. Yousefpour et al. [2021] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in pytorch, 2021. Yuan et al. [2022] Binhang Yuan, Cameron R Wolfe, Chen Dun, Yuxin Tang, Anastasios Kyrillidis, and Chris Jermaine. Distributed learning of fully connected neural networks using independent subnet training. Proceedings of the VLDB Endowment, 2022. Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking (Supplementary Material) Appendix A Additional Experimental Results and Related Details. A.1 Empirical Validation of Expected Gradient Properties (Theorem 4.9). Theorem 4.9 characterizes the expected gradient under Gaussian input masking as [∇rℒ()]=∇rℒ()+3,r+rE_C [ _w_rL_C (W ) ]= _w_rL (W )+T_3,r+g_r. Here, ∇rℒ() _w_rL (W ) is the clean input gradient, 3,rT_3,r is a systematic deviation term proportional to κ2κ^2, and rg_r is a residual error bounded by Eq. (6). Simulation Setup. We used a two-layer ReLU MLP with d=20d=20 input features, m=100m=100 hidden units, on n=500n=500 synthetic samples (‖i‖2≤1\|x_i\|_2≤ 1, yi∼(0,0.52)y_i (0,0.5^2)). First-layer weights W are from (0,0.12)N(0,0.1^2); second-layer ar∈±1a_r∈\± 1\ are fixed. We analyze ∇rℒ() _w_rL_C(W) by averaging N=2000N=2000 Monte Carlo samples for κ∈[0.001,1.0]κ∈[0.001,1.0], for a representative neuron r. For this setup, the clean loss ℒ()≈71.52L(W)≈ 71.52 and ‖∇rℒ()‖2≈0.981\| _w_rL(W)\|_2≈ 0.981. Results and Discussion. Our simulations validate the decomposition in Theorem 4.9. Figure 8(a) displays the ℓ2 _2-norms of the gradient components versus κ. The clean gradient norm is constant. The 3,rT_3,r’s norm, ‖3,r‖2\|T_3,r\|_2, scales with κ (e.g., from ≈8.1×10−7≈ 8.1× 10^-7 at κ=0.001κ=0.001 to ≈0.81≈ 0.81 at κ=1.0κ=1.0), confirming its theoretical dependence. The norm of the empirically estimated expected masked gradient, ‖[∇rℒ()]‖2\|E_C[ _w_rL_C(W)]\|_2, follows the clean gradient for small κ and reflects the vector sum with the growing teal term for larger κ in Eq. 5. (a) ℓ2 _2-norms of key components of the expected gradient [∇rℒ()]E_C[ _w_rL_C(W)]:the clean gradient norm (‖∇rℒ()‖2\| _w_rL(W)\|_2), the 3,rT_3,r’s norm (‖3,r‖2\|T_3,r\|_2), and the total expected masked gradient norm. 3,rT_3,r scales with κ2κ^2. (b) Comparison of the ℓ2 _2-norm of the empirically estimated gradient error term, ‖r‖2\|g_r\|_2, against its theoretical upper bound from Eq. (6). The empirical error (solid line) remains below the derived bound (dashed line) across all tested κ. Log-log scale. Figure 8: ℓ2 _2-norms of the gradient components (left) and residual error bound check (right). Figure 8(b) examines the residual error term rg_r. It compares the ℓ2 _2-norm of the empirically estimated 0g_0 with its theoretical upper bound from Eq. (6). The estimated error norm, ‖r‖est\|g_r\|_est, increases with κ (from ≈1.65×10−2≈ 1.65× 10^-2 at κ=0.001κ=0.001 to ≈0.53≈ 0.53 at κ=1.0κ=1.0). Importantly, the theoretical bound on ‖r‖2\|g_r\|_2 consistently upper-bounds the empirical error across the entire range of κ. For instance, at κ=0.001κ=0.001, ‖r‖est≈0.0165\|g_r\|_est≈ 0.0165 while its bound is ≈7.85≈ 7.85. In summary, the simulations confirm that the expected gradient under Gaussian input masking, with sufficiently small κ values, is well-approximated by the sum of the clean gradient and the κ2κ^2-dependent term, with a residual error that is effectively bounded by our theoretical derivation. A.2 Experimental Details for Section 6.4 We adopt the black-box threat model and the shadow model attack methodology (Adversary 1) proposed in the ML-Leaks framework [Salem et al., 2019]. Threat Model. The adversary has black-box access to a trained target model f. This means the adversary can query the model with any input x and observe its output posterior probability vector =f()p=f(x) (i.e., the softmax output over the classes), but has no access to the model’s parameters, gradients or original training data. The adversary’s goal is to train an attack model A that, given the posterior f()p_f(x) from the target model for a point x, predicts whether x was a member of the target’s training set. Shadow Model Attack Pipeline. Since the attacker does not have access to the target model’s training set, they cannot directly generate labeled data (member vs. non-member posteriors) to train their attack model. The shadow model technique circumvents this by creating a proxy environment where the attacker controls data membership. The pipeline is as follows: 1. Data Partitioning: The attacker possesses a dataset DshadowD_shadow, disjoint from the target’s training set but drawn from the same data distribution. This set is split into DShadowTrainD^Train_Shadow and DShadowOutD^Out_Shadow. 2. Shadow Model Training: A shadow model S, which mimics the target model’s architecture and training process, is trained on DShadowTrainD^Train_Shadow. 3. Attack Dataset Generation: The trained shadow model S is queried on its own training data (members, DShadowTrainD^Train_Shadow) and its hold-out data (non-members, DShadowOutD^Out_Shadow). The resulting posterior vectors S()p_S(x) are collected. Following [Salem et al., 2019], the top-3 sorted probabilities of each posterior are used as features: ϕ(S())=(p(1),p(2),p(3))φ(p_S(x))=(p_(1),p_(2),p_(3)). These feature vectors are labeled “1” if ∈DShadowTrainx∈ D^Train_Shadow and “0” otherwise. 4. Attack Model Training: A binary classifier, the attack model A, is trained on this generated dataset of “(feature, label)” pairs. It learns to distinguish the statistical “signature” of a member’s posterior from a non-member’s. This signature often manifests as higher confidence (larger p(1)p_(1)) and lower entropy for members, a result of the target/shadow model overfitting to its training data. 5. Inference on Target Model: To attack the original target model f, the adversary queries it with a point of interest x, extracts the features ϕ(f())φ(p_f(x)), and feeds them to the trained attack model A to get a membership prediction. Dataset and Partitioning. We use the CIFAR-10 dataset, consisting of 60,000 images. The full pool is shuffled and divided into four disjoint sets of 10,520 images each: target_train (training MG-protected models), target_test (non-member audit data), shadow_train (training shadow models), and shadow_test (shadow non-member data). All data is normalized using the mean and standard deviation of their respective training sets. For protected models, inputs x are modified via elementwise multiplication with a random mask m, where mi∼(1,κ2)m_i (1,κ^2). Model Architectures. • MLP (“n”): A fully-connected network with one hidden layer of 100 neurons (Tanh activation). Input layer: 3,072 features. • CNN (“cnn”): Two Conv-ReLU-MaxPool blocks, followed by a Tanh-activated fully-connected layer with 100 hidden units. Training and Hyperparameters. Both models are trained using the Adam optimizer (Learning Rate: 10−310^-3, ℓ2 _2 Regularization: 10−710^-7) for intervals between 20 and 120 epochs. The attack model A is a LogisticRegression classifier (scikit-learn), trained on a balanced dataset of member and non-member posteriors. A.2.1 Multiplicative Gaussian noise vs Differential privacy We evaluate the privacy-utility tradeoff of Multiplicative Gaussian (MG) noise against Differential Privacy (DP-SGD) using the CIFAR-10 image classification dataset. Following the standard ML-Leaks evaluation protocol [Shokri et al., 2017], we utilize the dataset partitioning described in the previous section and employ a Standard CNN architecture, i.e. a shallow baseline consisting of two convolutional layers (5×55× 5 kernels, 32 filters) followed by max-pooling and a fully connected layer. For the multiplicative gaussian defense, we sweep the noise parameter κ∈0.0,0.2,0.4,0.6,0.8,1.0,1.2κ∈\0.0,0.2,0.4,0.6,0.8,1.0,1.2\ where κ=0.0κ=0.0 represents the undefended baseline. For each training batch, we apply element-wise multiplicative noise to input features x~=x⊙(1+κZ) x=x (1+κ Z ) ›where Z∼(0,I)Z (0,I) is the standard Gaussian noise. For the the Differential Privacy, (DP-SGD) part, we sweep the noise multiplier σ∈0.3,0.5,0.8,1.0,1.5,2.0,3.0σ∈\0.3,0.5,0.8,1.0,1.5,2.0,3.0\ using the opacus library (Yousefpour et al. [2021]). We set the per-sample gradient clipping norm C=1.0C=1.0 and target δ=10−5δ=10^-5. The empirical findings for this experimet are illustrated in Figure 9. The plots map the privacy leakage (Attack Precision and Recall) against the model’s utility (Target Accuracy) across the swept noise parameters. Figure 9: Privacy-utility tradeoff for the Standard CNN. The left panel shows Attack Precision vs. Accuracy, and the right panel shows Attack Recall vs. Accuracy. Evaluation on High-Capacity Architecture: We repeated the evaluation using an Improved CNN architecture. This model features a deeper convolutional structure (convolutional blocks with increasing filter sizes (32→64→12832→ 64→ 128) using 3×33× 3 kernels) and also incorporates Batch Normalization and Dropout to achieve higher baseline utility. Figure 10 below illustrates the results for the improved CNN architecture. We observe that the performance gap between the two methods narrows and the MG noise curve remains much closer to the near-random guess of the attacker for a longer stretch of the accuracy spectrum. Figure 10: Privacy-utility tradeoff for the Improved CNN. Unlike the Standard CNN, the MG noise curve stays much closer to the DP-SGD curve across the accuracy range Appendix B Proofs in Section 4 In this section, we first prove an exact form of the expected surrogate loss function. B.1 General Form of Expected Loss Lemma B.1. Let i,r=r⊙iu_i,r=w_r x_i, let σ^κ(,)=⊤⋅1(⊤κ‖⊙‖2) σ_κ (w,x )=w x· _1 ( w xκ \|w x \|_2 ), and let f^(,)=1m∑r=1marσ^κ(r,) f ( θ,x )= 1 m _r=1^ma_r σ_κ (w_r,x ). Then we have [ℒ()] _C [L_C ( θ ) ] =12∑i=1n(f^(,i)−yi)22+κ22m∑i=1n‖∑r=1mari,r(r⊤iκ‖i,r‖2)‖22 = 12 _i=1^n ( f ( θ,x_i )-y_i )_2^2+ κ^22m _i=1^n \| _r=1^ma_ru_i,r ( w_r x_iκ \|u_i,r \|_2 ) \|_2^2 +1m∑i=1n∑r,r′=1marar′(Ci,r,r′+κ22πEi,r,r′) + 1m _i=1^n _r,r =1^ma_ra_r (C_i,r,r + κ^22πE_i,r,r ) +2κ2πm∑i=1n∑r=1marGi,r(1m∑r′=1mar′Ti,r,r′−y) + 2κ 2π m _i=1^n _r=1^ma_rG_i,r ( 1 m _r =1^ma_r T_i,r,r -y ) where Ci,r,r′,Ei,r,r′,Ti,r,r′C_i,r,r ,E_i,r,r ,T_i,r,r and Gi,rG_i,r are defined as Ci,r,r′=((r⊤i)(r′⊤i)+κ2i,r⊤i,r′)(r⊤iκ‖i,r‖2,r′⊤iκ‖i,r′‖2,i,r⊤i,r′‖i,r‖2‖i,r′‖2) C_i,r,r = ( (w_r x_i ) (w_r x_i )+κ^2u_i,r u_i,r )C ( w_r x_iκ \|u_i,r \|_2, w_r x_iκ \|u_i,r \|_2, u_i,r u_i,r \|u_i,r \|_2 \|u_i,r \|_2 ) Ei,r,r′=‖i,r‖2‖i,r′‖2exp(−‖i,r′‖22(r⊤i)2−2(i,r⊤i,r′)(r⊤i)(r′⊤i)+‖i,r‖22(r′⊤i)22κ2(‖i,r‖22‖i,r′‖22−(i,r⊤i,r′)2)) E_i,r,r = \|u_i,r \|_2 \|u_i,r \|_2 (- \|u_i,r \|_2^2 (w_r x_i )^2-2 (u_i,r u_i,r ) (w_r x_i ) (w_r x_i )+ \|u_i,r \|_2^2 (w_r x_i )^22κ^2 ( \|u_i,r \|_2^2 \|u_i,r \|_2^2- (u_i,r u_i,r )^2 ) ) Ti,r,r′=r′⊤i⋅1(‖i,r‖2‖i,r′‖2⋅r′⊤−i,r⊤i,r′⋅r⊤iκ2‖i,r‖2‖i,r′‖2(‖i,r‖22‖i,r′‖22−(i,r⊤i,r′)2)) T_i,r,r =w_r x_i· _1 ( \|u_i,r \|_2 \|u_i,r \|_2·w_r x-u_i,r u_i,r ·w_r x_iκ^2 \|u_i,r \|_2 \|u_i,r \|_2 ( \|u_i,r \|_2^2 \|u_i,r \|_2^2- (u_i,r u_i,r )^2 ) ) Gi,r=‖i,r‖2exp(−(r⊤i)22κ2‖r‖22) G_i,r= \|u_i,r \|_2 (- (w_r x_i )^22κ^2 \|u_r \|_2^2 ) Proof. By definition, we have [ℒ()]=12∑i=1ni[(f(,i⊙i)−yi)2]E_C [L_C ( θ ) ]= 12 _i=1^nE_c_i [ (f ( θ,x_i c_i )-y_i )^2 ] For simplicity, we fix i∈[n]i∈[n], and study [(f(,⊙)−y)2]E_c [ (f ( θ,x c )-y )^2 ]. In the analysis below, we let r=r⊙u_r=w_r x. In particular, we have [(f(,⊙)−y)2] _c [ (f ( θ,x c )-y )^2 ] =[f(,⊙)2]−2y[f(,⊙)]+y2 =E_c [f ( θ,x c )^2 ]-2yE_c [f ( θ,x c ) ]+y^2 Here we shall evaluate the two expectations separately. To start, for the first-order term, we have [f(,⊙)]=1m∑r=1mar[σ(r⊤(⊙))]=1m∑r=1mar[σ(⊤(r⊙))]E_c [f ( θ,x c ) ]= 1 m _r=1^ma_rE [σ (w_r (c x ) ) ]= 1 m _r=1^ma_rE [σ (c (w_r x ) ) ] (18) Notice that since ∼(,κ2)c ( 1,κ^2I ). By Lemma D.13 we have that ⊤(r⊙)∼(r⊤,κ2‖r⊙‖22)c (w_r x ) (w_r x,κ^2 \|w_r x \|_2^2 ), since ⊤(r⊙)=r⊤ 1 (w_r x )=w_r x. Applying Lemma D.18 with z=⊤(r⊙)z=c (w_r x ), mean r⊤w_r x, and standard deviation κ‖r⊙‖κ \|w_r x \|, we have that [σ(⊤(r⊙))]=κ‖r⊙‖22πexp(−(r⊤)22κ2‖r⊙‖22)+r⊤1(r⊤κ‖r⊙‖2)E [σ (c (w_r x ) ) ]= κ \|w_r x \|_2 2π (- (w_r x )^22κ^2 \|w_r x \|_2^2 )+w_r x _1 ( w_r xκ \|w_r x \|_2 ) (19) Plugging in to the form of (18) gives [f(,⊙)]=1m∑r=1marσ^κ(r,)+κ2πm∑r=1mar‖r‖2exp(−(r⊤)22κ2‖r‖22)E_c [f ( θ,x c ) ]= 1 m _r=1^ma_r σ_κ (w_r,x )+ κ 2π m _r=1^ma_r \|u_r \|_2 (- (w_r x )^22κ^2 \|u_r \|_2^2 ) (20) Next, we focus on the second-order term. Since r⊤(⊙)=⊤(r⊙)w_r (c x )=c (w_r x ), we have [f(,⊙)2]=1m∑r,r′=1marar′[σ(⊤r)σ(⊤r′)]E_c [f ( θ,x c )^2 ]= 1m _r,r =1^ma_ra_r E [σ (c u_r )σ (c u_r ) ] Let z1=⊤rz_1=c u_r and z2=⊤r′z_2=c u_r , we have that z1∼(r⊤,κ2‖r‖22),z2∼(r′⊤,κ2‖r′‖22)z_1 (w_r x,κ^2 \|u_r \|_2^2 ),z_2 (w_r x,κ^2 \|u_r \|_2^2 ), and Cov(z1,z2)=κ2r⊤r′Cov (z_1,z_2 )=κ^2u_r u_r . Applying Lemma D.9 with a=b=0a=b=0 gives [σ(⊤r)σ(⊤r′)] [σ (c u_r )σ (c u_r ) ] =((r⊤)(r′⊤)+κ2r⊤r′)2(r⊤κ‖r‖2,r′⊤κ‖r′‖2,r⊤r′‖r‖2‖r′‖2) = ( (w_r x ) (w_r x )+κ^2u_r u_r ) _2 ( w_r xκ \|u_r \|_2, w_r xκ \|u_r \|_2, u_r u_r \|u_r \|_2 \|u_r \|_2 ) +κ22π‖r‖2‖r′‖2exp(−‖r′‖22(r⊤)2−2(r⊤r′)(r⊤)(r′⊤)+‖r‖22(r′⊤)22κ2(‖r‖22‖r′‖22−(r⊤r′)2)) + κ^22π \|u_r \|_2 \|u_r \|_2 (- \|u_r \|_2^2 (w_r x )^2-2 (u_r u_r ) (w_r x ) (w_r x )+ \|u_r \|_2^2 (w_r x )^22κ^2 ( \|u_r \|_2^2 \|u_r \|_2^2- (u_r u_r )^2 ) ) +κ2π(‖r‖2⋅r′⊤⋅T^1,r,r′+‖r′‖2⋅r⊤⋅T^2,r,r′) + κ 2π ( \|u_r \|_2·w_r x· T_1,r,r + \|u_r \|_2·w_r x· T_2,r,r ) where T^1,r,r′,T^2,r,r′ T_1,r,r , T_2,r,r are defined as T^1,r,r′=exp(−(r⊤)22κ2‖r‖22)1(‖r‖22⋅r′⊤−r⊤r′⋅r⊤κ‖r‖2(‖r‖22‖r′‖22−(r⊤r′)2)12) T_1,r,r = (- (w_r x )^22κ^2 \|u_r \|_2^2 ) _1 ( \|u_r \|_2^2·w_r x-u_r u_r ·w_r xκ \|u_r \|_2 ( \|u_r \|_2^2 \|u_r \|_2^2- (u_r u_r )^2 ) 12 ) T^2,r,r′=exp(−(r′⊤)22κ2‖r′‖22)1(‖r′‖22⋅r⊤−r⊤r′⋅r′⊤κ‖r′‖2(‖r‖22‖r′‖22−(r⊤r′)2)12) T_2,r,r = (- (w_r x )^22κ^2 \|u_r \|_2^2 ) _1 ( \|u_r \|_2^2·w_r x-u_r u_r ·w_r xκ \|u_r \|_2 ( \|u_r \|_2^2 \|u_r \|_2^2- (u_r u_r )^2 ) 12 ) For the simplicity of notations, we define T1,r,r′=‖r‖2⋅r′⊤⋅T^1,r,r′T_1,r,r = \|u_r \|_2·w_r x· T_1,r,r and T2,r,r′=‖r′‖2⋅r⊤⋅T^2,r,r′T_2,r,r = \|u_r \|_2·w_r x· T_2,r,r . Moreover, we define Er,r′=‖r‖2‖r′‖2exp(−‖r′‖22(r⊤)2−2(r⊤r′)(r⊤)(r′⊤)+‖r‖22(r′⊤)22κ2(‖r‖22‖r′‖22−(r⊤r′)2))E_r,r = \|u_r \|_2 \|u_r \|_2 (- \|u_r \|_2^2 (w_r x )^2-2 (u_r u_r ) (w_r x ) (w_r x )+ \|u_r \|_2^2 (w_r x )^22κ^2 ( \|u_r \|_2^2 \|u_r \|_2^2- (u_r u_r )^2 ) ) Lastly, we use the definition of the Gaussian Copula function (a,b,ρ)=2(a,b,ρ)−1(a)1(b)C (a,b,ρ )= _2 (a,b,ρ )- _1 (a ) _1 (b ) and define Cr,r′=((r⊤)(r′⊤)+κ2r⊤r′)(r⊤κ‖r‖2,r′⊤κ‖r′‖2,r⊤r′‖r‖2‖r′‖2)C_r,r = ( (w_r x ) (w_r x )+κ^2u_r u_r )C ( w_r xκ \|u_r \|_2, w_r xκ \|u_r \|_2, u_r u_r \|u_r \|_2 \|u_r \|_2 ) Under these definitions, we have that [σ(⊤r)σ(⊤r′)] [σ (c u_r )σ (c u_r ) ] =((r⊤)(r′⊤)+κ2r⊤r′)1(r⊤κ‖r‖2)1(r′⊤κ‖r′‖2) = ( (w_r x ) (w_r x )+κ^2u_r u_r ) _1 ( w_r xκ \|u_r \|_2 ) _1 ( w_r xκ \|u_r \|_2 ) +Cr,r′+κ22πEr,r′+κ2π(T1,r,r′+T2,r,r′) +C_r,r + κ^22πE_r,r + κ 2π (T_1,r,r +T_2,r,r ) =σ^κ(r,)σ^κ(r′,)+κ2r⊤r′1(r⊤κ‖r‖2)1(r′⊤κ‖r′‖2) = σ_κ (w_r,x ) σ_κ (w_r ,x )+κ^2u_r u_r _1 ( w_r xκ \|u_r \|_2 ) _1 ( w_r xκ \|u_r \|_2 ) +Cr,r′+κ22πEr,r′+κ2π(T1,r,r′+T2,r,r′) +C_r,r + κ^22πE_r,r + κ 2π (T_1,r,r +T_2,r,r ) Plugging back into the expression of [f(,⊙)2]E_c [f ( θ,x c )^2 ] gives [f(,⊙)2] _c [f ( θ,x c )^2 ] =1m∑r,r′=1marar′σ^κ(r,)σ^κ(r′,)+1m∑r,r′=1marar′κ2r⊤r′1(r⊤κ‖r‖2)1(r′⊤κ‖r′‖2) = 1m _r,r =1^ma_ra_r σ_κ (w_r,x ) σ_κ (w_r ,x )+ 1m _r,r =1^ma_ra_r κ^2u_r u_r _1 ( w_r xκ \|u_r \|_2 ) _1 ( w_r xκ \|u_r \|_2 ) +1m∑r,r′=1marar′(Cr,r′+κ22πEr,r′+κ2π(T1,r,r′+T2,r,r′)) + 1m _r,r =1^ma_ra_r (C_r,r + κ^22πE_r,r + κ 2π (T_1,r,r +T_2,r,r ) ) =(1m∑r=1marσ^κ(r,))2+κ2‖1m∑r=1marr(r⊤κ‖r‖2)‖22 = ( 1 m _r=1^ma_r σ_κ (w_r,x ) )^2+κ^2 \| 1 m _r=1^ma_ru_r ( w_r xκ \|u_r \|_2 ) \|_2^2 +1m∑r,r′=1marar′(Cr,r′+κ22πEr,r′+κ2π(T1,r,r′+T2,r,r′)) + 1m _r,r =1^ma_ra_r (C_r,r + κ^22πE_r,r + κ 2π (T_1,r,r +T_2,r,r ) ) Combining the expression of [f(,⊙)2]E_c [f ( θ,x c )^2 ] and [f(,⊙)]E_c [f ( θ,x c ) ], and noticing T1,r,r′=T2,r′,rT_1,r,r =T_2,r ,r, we have [(f(,⊙)−y)2] _c [ (f ( θ,x c )-y )^2 ] =(1m∑r=1marσ^κ(r,))2+κ2‖1m∑r=1marr(r⊤κ‖r‖2)‖22 = ( 1 m _r=1^ma_r σ_κ (w_r,x ) )^2+κ^2 \| 1 m _r=1^ma_ru_r ( w_r xκ \|u_r \|_2 ) \|_2^2 +1m∑r,r′=1marar′(Cr,r′+κ22πEr,r′+κ2π(T1,r,r′+T2,r,r′)) + 1m _r,r =1^ma_ra_r (C_r,r + κ^22πE_r,r + κ 2π (T_1,r,r +T_2,r,r ) ) −2ym∑r=1marσ^κ(r,)−2κy2πm∑r=1mar‖r‖2exp(−(r⊤)2‖r‖22)+y2 - 2y m _r=1^ma_r σ_κ (w_r,x )- 2κ y 2π m _r=1^ma_r \|u_r \|_2 (- (w_r x )^2 \|u_r \|_2^2 )+y^2 =(1m∑r=1marσ^κ(r,)−y)2+κ2m‖∑r=1marr(r⊤κ‖r‖2)‖22 = ( 1 m _r=1^ma_r σ_κ (w_r,x )-y )^2+ κ^2m \| _r=1^ma_ru_r ( w_r xκ \|u_r \|_2 ) \|_2^2 +1m∑r,r′=1marar′(Cr,r′+κ22πEr,r′+2κ2πT1,r,r′) + 1m _r,r =1^ma_ra_r (C_r,r + κ^22πE_r,r + 2κ 2πT_1,r,r ) −2κy2πm∑r=1mar‖r‖2exp(−(r⊤)22κ2‖r‖22) - 2κ y 2π m _r=1^ma_r \|u_r \|_2 (- (w_r x )^22κ^2 \|u_r \|_2^2 ) To extend to the case of i,yix_i,y_i, we need to re-define Ci,r,r′=((r⊤i)(r′⊤i)+κ2i,r⊤i,r′)(r⊤iκ‖i,r‖2,r′⊤iκ‖i,r′‖2,i,r⊤i,r′‖i,r‖2‖i,r′‖2) C_i,r,r = ( (w_r x_i ) (w_r x_i )+κ^2u_i,r u_i,r )C ( w_r x_iκ \|u_i,r \|_2, w_r x_iκ \|u_i,r \|_2, u_i,r u_i,r \|u_i,r \|_2 \|u_i,r \|_2 ) Ei,r,r′=‖i,r‖2‖i,r′‖2exp(−‖i,r′‖22(r⊤i)2−2(i,r⊤i,r′)(r⊤i)(r′⊤i)+‖i,r‖22(r′⊤i)22κ2(‖i,r‖22‖i,r′‖22−(i,r⊤i,r′)2)) E_i,r,r = \|u_i,r \|_2 \|u_i,r \|_2 (- \|u_i,r \|_2^2 (w_r x_i )^2-2 (u_i,r u_i,r ) (w_r x_i ) (w_r x_i )+ \|u_i,r \|_2^2 (w_r x_i )^22κ^2 ( \|u_i,r \|_2^2 \|u_i,r \|_2^2- (u_i,r u_i,r )^2 ) ) Ti,r,r′=r′⊤i⋅1(‖i,r‖22⋅r′⊤i−i,r⊤i,r′⋅r⊤iκ‖i,r‖2(‖i,r‖22‖i,r′‖22−(i,r⊤i,r′)2)12) T_i,r,r =w_r x_i· _1 ( \|u_i,r \|_2^2·w_r x_i-u_i,r u_i,r ·w_r x_iκ \|u_i,r \|_2 ( \|u_i,r \|_2^2 \|u_i,r \|_2^2- (u_i,r u_i,r )^2 ) 12 ) Gi,r=‖i,r‖2exp(−(r⊤i)22κ2‖r‖22) G_i,r= \|u_i,r \|_2 (- (w_r x_i )^22κ^2 \|u_r \|_2^2 ) Moreover, let f^(,)=1m∑r=1marσ^κ(r,) f ( θ,x )= 1 m _r=1^ma_r σ_κ (w_r,x ). Then we have that [ℒ()] _C [L_C ( θ ) ] =12∑i=1n(f^(,i)−yi)22+κ22m∑i=1n‖∑r=1mari,r(r⊤iκ‖i,r‖2)‖22 = 12 _i=1^n ( f ( θ,x_i )-y_i )_2^2+ κ^22m _i=1^n \| _r=1^ma_ru_i,r ( w_r x_iκ \|u_i,r \|_2 ) \|_2^2 +1m∑i=1n∑r,r′=1marar′(Ci,r,r′+κ22πEi,r,r′+κ2πTi,r,r′Gi,r) + 1m _i=1^n _r,r =1^ma_ra_r (C_i,r,r + κ^22πE_i,r,r +κ 2πT_i,r,r G_i,r ) −κy2πm∑i=1n∑r=1marGi,r -κ y 2π m _i=1^n _r=1^ma_rG_i,r =12∑i=1n(f^(,i)−yi)22+κ22m∑i=1n‖∑r=1mari,r(r⊤iκ‖i,r‖2)‖22 = 12 _i=1^n ( f ( θ,x_i )-y_i )_2^2+ κ^22m _i=1^n \| _r=1^ma_ru_i,r ( w_r x_iκ \|u_i,r \|_2 ) \|_2^2 +1m∑i=1n∑r,r′=1marar′(Ci,r,r′+κ22πEi,r,r′) + 1m _i=1^n _r,r =1^ma_ra_r (C_i,r,r + κ^22πE_i,r,r ) +2κ2πm∑i=1n∑r=1marGi,r(1m∑r′=1mar′Ti,r,r′−y) + 2κ 2π m _i=1^n _r=1^ma_rG_i,r ( 1 m _r =1^ma_r T_i,r,r -y ) ∎ B.2 Proof of Theorem 4.2 Proof. Let i,r=r⊙iu_i,r=w_r x_i. By Lemma B.1, we have that [ℒ()] _C [L_C ( θ ) ] =12∑i=1n(f^(,i)−yi)22+κ22m∑i=1n‖∑r=1mari,r(r⊤iκ‖i,r‖2)‖22 = 12 _i=1^n ( f ( θ,x_i )-y_i )_2^2+ κ^22m _i=1^n \| _r=1^ma_ru_i,r ( w_r x_iκ \|u_i,r \|_2 ) \|_2^2 +1m∑i=1n∑r,r′=1marar′(Ci,r,r′+κ22πEi,r,r′) + 1m _i=1^n _r,r =1^ma_ra_r (C_i,r,r + κ^22πE_i,r,r ) +2κ2πm∑i=1n∑r=1marGi,r(1m∑r′=1mar′Ti,r,r′−y) + 2κ 2π m _i=1^n _r=1^ma_rG_i,r ( 1 m _r =1^ma_r T_i,r,r -y ) where Ci,r,r′,Ei,r,r′,Ti,r,r′C_i,r,r ,E_i,r,r ,T_i,r,r and Gi,rG_i,r are defined as Ci,r,r′=((r⊤i)(r′⊤i)+κ2i,r⊤i,r′)(r⊤iκ‖i,r‖2,r′⊤iκ‖i,r′‖2,i,r⊤i,r′‖i,r‖2‖i,r′‖2) C_i,r,r = ( (w_r x_i ) (w_r x_i )+κ^2u_i,r u_i,r )C ( w_r x_iκ \|u_i,r \|_2, w_r x_iκ \|u_i,r \|_2, u_i,r u_i,r \|u_i,r \|_2 \|u_i,r \|_2 ) Ei,r,r′=‖i,r‖2‖i,r′‖2exp(−‖i,r′‖22(r⊤i)2−2(i,r⊤i,r′)(r⊤i)(r′⊤i)+‖i,r‖22(r′⊤i)22κ2(‖i,r‖22‖i,r′‖22−(i,r⊤i,r′)2)) E_i,r,r = \|u_i,r \|_2 \|u_i,r \|_2 (- \|u_i,r \|_2^2 (w_r x_i )^2-2 (u_i,r u_i,r ) (w_r x_i ) (w_r x_i )+ \|u_i,r \|_2^2 (w_r x_i )^22κ^2 ( \|u_i,r \|_2^2 \|u_i,r \|_2^2- (u_i,r u_i,r )^2 ) ) Ti,r,r′=r′⊤i⋅1(‖i,r‖22⋅r′⊤i−i,r⊤i,r′⋅r⊤iκ‖i,r‖2(‖i,r‖22‖i,r′‖22−(i,r⊤i,r′)2)12) T_i,r,r =w_r x_i· _1 ( \|u_i,r \|_2^2·w_r x_i-u_i,r u_i,r ·w_r x_iκ \|u_i,r \|_2 ( \|u_i,r \|_2^2 \|u_i,r \|_2^2- (u_i,r u_i,r )^2 ) 12 ) Gi,r=‖i,r‖2exp(−(r⊤i)22κ2‖r‖22) G_i,r= \|u_i,r \|_2 (- (w_r x_i )^22κ^2 \|u_r \|_2^2 ) Therefore, the proof of the theorem relies on the upper bound of Ci,r,r′,Ei,r,r′,Ti,r,r′C_i,r,r ,E_i,r,r ,T_i,r,r and Gi,rG_i,r. To upper-bound Ci,r,r′C_i,r,r , we utilize the result in that |C(a,b,ρ)|≤|arcsinρ|2πexp(−a2−2ρab+b22(1−ρ2))≤|ρ|4exp(−a2+b24) |C (a,b,ρ ) |≤ | ρ |2π (- a^2-2ρ ab+b^22 (1-ρ^2 ) )≤ |ρ|4 (- a^2+b^24 ) where we used Lemma D.19 that |arcsinx|≤π2⋅|x|| x|≤ π2·|x|. Plugging in a=r⊤iκ‖i,r‖2,b=r′⊤iκ‖i,r′‖2a= w_r x_iκ \|u_i,r \|_2,b= w_r x_iκ \|u_i,r \|_2 and ρ=i,r⊤i,r′‖i,r‖2‖i,r′‖2ρ= u_i,r u_i,r \|u_i,r \|_2 \|u_i,r \|_2 gives |Ci,r,r′| |C_i,r,r | ≤|(r⊤i)(r′⊤i)+κ2i,r⊤i,r′|⋅|i,r⊤i,r′|4‖i,r‖2‖i,r′‖2exp(−14κ2((r⊤i)2‖i,r‖22+(r′⊤i)2‖i,r′‖22)) ≤ | (w_r x_i ) (w_r x_i )+κ^2u_i,r u_i,r |· |u_i,r u_i,r |4 \|u_i,r \|_2 \|u_i,r \|_2 (- 14κ^2 ( (w_r x_i )^2 \|u_i,r \|_2^2+ (w_r x_i )^2 \|u_i,r \|_2^2 ) ) ≤14(|(r⊤i)(r′⊤i)|+κ2‖i,r‖2‖i,r′‖2)ϕ(r⊤i2κ‖i,r‖2)ϕ(r′⊤i2κ‖i,r′‖2) ≤ 14 ( | (w_r x_i ) (w_r x_i ) |+κ^2 \|u_i,r \|_2 \|u_i,r \|_2 )φ ( w_r x_i2κ \|u_i,r \|_2 )φ ( w_r x_i2κ \|u_i,r \|_2 ) =κ24‖i,r‖2‖i,r′‖2(|r⊤i|κ‖i,r‖2⋅|r′⊤i|κ‖i,r‖2+1)ϕ(r⊤i2κ‖i,r‖2)ϕ(r′⊤i2κ‖i,r′‖2) = κ^24 \|u_i,r \|_2 \|u_i,r \|_2 ( |w_r x_i |κ \|u_i,r \|_2· |w_r x_i |κ \|u_i,r \|_2+1 )φ ( w_r x_i2κ \|u_i,r \|_2 )φ ( w_r x_i2κ \|u_i,r \|_2 ) =κ24‖i,r‖2‖i,r′‖2(ψ(r⊤i2κ‖i,r‖2)ψ(r′⊤i2κ‖i,r′‖2)+ϕ(r⊤i2κ‖i,r‖2)ϕ(r′⊤i2κ‖i,r′‖2)) = κ^24 \|u_i,r \|_2 \|u_i,r \|_2 (ψ ( w_r x_i2κ \|u_i,r \|_2 )ψ ( w_r x_i2κ \|u_i,r \|_2 )+φ ( w_r x_i2κ \|u_i,r \|_2 )φ ( w_r x_i2κ \|u_i,r \|_2 ) ) where we use the definition Pi,r=|r⊤i|⋅exp(−(r⊤i)24κ2‖i,r‖22)P_i,r= |w_r x_i |· (- (w_r x_i )^24κ^2 \|u_i,r \|_2^2 ). For the term Ei,r,r′E_i,r,r , we notice that by letting a=r⊤iκ‖i,r‖2,b=r′⊤iκ‖i,r′‖2a= w_r x_iκ \|u_i,r \|_2,b= w_r x_iκ \|u_i,r \|_2 and ρ=i,r⊤i,r′‖i,r‖2‖i,r′‖ρ= u_i,r u_i,r \|u_i,r \|_2 \|u_i,r \|, we have exp(−‖i,r′‖22(r⊤i)2−2(i,r⊤i,r′)(r⊤i)(r′⊤i)+‖i,r‖22(r′⊤i)22κ2(‖i,r‖22‖i,r′‖22−(i,r⊤i,r′)2))=exp(−a2−2ρab+b22(1−ρ2)) (- \|u_i,r \|_2^2 (w_r x_i )^2-2 (u_i,r u_i,r ) (w_r x_i ) (w_r x_i )+ \|u_i,r \|_2^2 (w_r x_i )^22κ^2 ( \|u_i,r \|_2^2 \|u_i,r \|_2^2- (u_i,r u_i,r )^2 ) )= (- a^2-2ρ ab+b^22 (1-ρ^2 ) ) Using exp(−a2−2ρab+b22(1−ρ2))≤exp(−a2+b24) (- a^2-2ρ ab+b^22 (1-ρ^2 ) )≤ (- a^2+b^24 ), we have that |Ei,r,r′| |E_i,r,r | ≤‖i,r‖2‖i,r′‖2exp(−14κ2((r⊤i)2‖i,r‖22+(r′⊤i)2‖i,r′‖22)) ≤ \|u_i,r \|_2 \|u_i,r \|_2 (- 14κ^2 ( (w_r x_i )^2 \|u_i,r \|_2^2+ (w_r x_i )^2 \|u_i,r \|_2^2 ) ) =‖i,r‖2‖i,r′‖2ϕ(r⊤i2κ‖i,r‖2)ϕ(r′⊤i2κ‖i,r′‖2) = \|u_i,r \|_2 \|u_i,r \|_2φ ( w_r x_i2κ \|u_i,r \|_2 )φ ( w_r x_i2κ \|u_i,r \|_2 ) Therefore, we have |Ci,r,r′+κ22πEi,r,r′| |C_i,r,r + κ^22πE_i,r,r | ≤κ24‖i,r‖2‖i,r′‖2(ψ(r⊤i2κ‖i,r‖2)ψ(r′⊤i2κ‖i,r′‖2)+ϕ(r⊤i2κ‖i,r‖2)ϕ(r′⊤i2κ‖i,r′‖2)) ≤ κ^24 \|u_i,r \|_2 \|u_i,r \|_2 (ψ ( w_r x_i2κ \|u_i,r \|_2 )ψ ( w_r x_i2κ \|u_i,r \|_2 )+φ ( w_r x_i2κ \|u_i,r \|_2 )φ ( w_r x_i2κ \|u_i,r \|_2 ) ) This gives that |∑r,r′=1marar′(Ci,r,r′+κ22πEi,r,r′)| | _r,r =1^ma_ra_r (C_i,r,r + κ^22πE_i,r,r ) | (21) ≤κ24((∑r=1m‖i,r‖2ψ(r⊤i2κ‖i,r‖2))2+(∑r=1m‖i,r‖2ϕ(r⊤i2κ‖i,r‖2))2) ≤ κ^24 ( ( _r=1^m \|u_i,r \|_2ψ ( w_r x_i2κ \|u_i,r \|_2 ) )^2+ ( _r=1^m \|u_i,r \|_2φ ( w_r x_i2κ \|u_i,r \|_2 ) )^2 ) By definition, we have ‖i,r‖2≤R \|u_i,r \|_2≤ R_u. Therefore |∑r,r′=1marar′(Ci,r,r′+κ22πEi,r,r′)|≤14κ2R2((∑r=1mψ(r⊤i2κ‖i,r‖2))2+(∑r=1mϕ(r⊤i2κ‖i,r‖2))2) | _r,r =1^ma_ra_r (C_i,r,r + κ^22πE_i,r,r ) |≤ 14κ^2R_u^2 ( ( _r=1^mψ ( w_r x_i2κ \|u_i,r \|_2 ) )^2+ ( _r=1^mφ ( w_r x_i2κ \|u_i,r \|_2 ) )^2 ) Next, we focus on the term Ti,r,r′T_i,r,r and Gi,rG_i,r. By the property of CDF, we have that |Ti,r,r′|≤|r′⊤i|≤‖r‖2 |T_i,r,r |≤ |w_r x_i |≤ \|w_r \|_2. Therefore |1m∑r′=1mar′Ti,r,r′−yi|≤1m∑r′=1m‖r′‖2+|yi|≤mR+By≤2mR | 1 m _r =1^ma_r T_i,r,r -y_i |≤ 1 m _r =1^m \|w_r \|_2+ |y_i |≤ mR_w+B_y≤ 2 mR_w where we applied ‖r‖2≤R \|w_r \|_2≤ R_w and By≤3mRB_y≤ 3 mR_w. Thus |∑r=1marGi,r(1m∑r′=1mar′Ti,r,r′−yi)| | _r=1^ma_rG_i,r ( 1 m _r =1^ma_r T_i,r,r -y_i ) | ≤∑r=1mGi,r⋅2mR≤mR∑r=1mϕ(r⊤i2κ‖i,r‖2) ≤ _r=1^mG_i,r· 2 mR_w≤ mR_w _r=1^mφ ( w_r x_i2κ \|u_i,r \|_2 ) where we used ‖i,r‖2≤R \|u_i,r \|_2≤ R_u. Combining the inequality above and (21), we have |ℰ| |E | ≤nκ2R24m((∑r=1mψ(r⊤i2κ‖i,r‖2))2+(∑r=1mϕ(r⊤i2κ‖i,r‖2))2)+nκR2∑r=1mϕ(r⊤i2κ‖i,r‖2) ≤ nκ^2R_u^24m ( ( _r=1^mψ ( w_r x_i2κ \|u_i,r \|_2 ) )^2+ ( _r=1^mφ ( w_r x_i2κ \|u_i,r \|_2 ) )^2 )+ nκ R_w2 _r=1^mφ ( w_r x_i2κ \|u_i,r \|_2 ) Applying the definition of ψmax _ and ϕmax _ gives the desired results. ∎ B.3 Proof of Theorem 4.9 Proof. By the form of the gradient, we have k[∇rℒ()] _C_k [ _w_rL_C ( θ ) ] =arm∑i=1n[(f(,i⊙i)−yi)i⊙i⟨r,i⊙i⟩≥0] = a_r m _i=1^nE_C [ (f ( θ,x_i c_i )-y_i )x_i c_iI \ w_r,x_i c_i ≥ 0 \ ] (22) =arm∑i=1ni[f(,i⊙i)i⊙i⟨r,i⊙i⟩≥0]⏟1,i = a_r m _i=1^n E_c_i [f ( θ,x_i c_i )x_i c_iI \ w_r,x_i c_i ≥ 0 \ ]_T_1,i −arm∑i=1nyii[i⊙i⟨r,i⊙i⟩≥0]⏟2,i - a_r m _i=1^ny_i E_c_i [x_i c_iI \ w_r,x_i c_i ≥ 0 \ ]_T_2,i Let r,i=r⊙iu_r,i=w_r x_i. For 1,iT_1,i, we further have 1,i _1,i =1m∑r′=1mar′i[σ(r′⊤(i⊙i))i⊙ir⊤(i⊙i)≥0] = 1 m _r =1^ma_r E_c_i [σ (w_r (x_i c_i ) )x_i c_iI \w_r (x_i c_i )≥ 0 \ ] =1m∑r′=1mar′i[(i⊙i)(i⊙i)⊤r′r⊤(i⊙i)≥0;r′⊤(i⊙i)≥0] = 1 m _r =1^ma_r E_c_i [ (x_i c_i ) (x_i c_i ) w_r I \w_r (x_i c_i )≥ 0;w_r (x_i c_i )≥ 0 \ ] =1m∑r′=1mar′(i[ii⊤r⊤(i⊙i)≥0;r′⊤(i⊙i)≥0]⊙(ii⊤))r′ = 1 m _r =1^ma_r (E_c_i [c_ic_i I \w_r (x_i c_i )≥ 0;w_r (x_i c_i )≥ 0 \ ] (x_ix_i ) )w_r =1m∑r′=1mar′(i[ii⊤r,i⊤i≥0;r′,i⊤i≥0]⊙(ii⊤))r′ = 1 m _r =1^ma_r (E_c_i [c_ic_i I \u_r,i c_i≥ 0;u_r ,i c_i≥ 0 \ ] (x_ix_i ) )w_r =1m∑r=1mar′Diag()ii[ii⊤r,i⊤i≥0;r′,i⊤i≥0]r′,i = 1 m _r=1^ma_r Diag (x )_iE_c_i [c_ic_i I \u_r,i c_i≥ 0;u_r ,i c_i≥ 0 \ ]u_r ,i For 2,iT_2,i, we can easily obtain 2,i=i[ir,i⊤i≥0]⊙iT_2,i=E_c_i [c_iI \u_r,i c_i≥ 0 \ ] x_i Abstractly, we are thus interested in the following quantity: [⊤⊤≥0;⊤≥0];[⊤≥0]E_c [cc I \c u≥ 0;c v≥ 0 \ ]; _c [cI \c u≥ 0 \ ] where ∼(,κ2)c ( μ,κ^2I ), and ,u,v are fixed vectors. Let z1=⊤z_1=c u and z2=⊤z_2=c v. Then we have z1∼(⊤,κ2‖22);z2∼(⊤,κ2‖22)z_1 (c u,κ^2 \|u \|_2^2 ); z_2 (c v,κ^2 \|v \|_2^2 ) According to Lemma D.11 and Lemma D.12, and by defining = μ= 1, we have that k,r,i(1)∈ℝd ^(1)_k,r,i ^d and k,r,r′,i(2)∈ℝd×d ^(2)_k,r,r ,i ^d× d defined below k,r,i(1) ^(1)_k,r,i :=i[ii⊤r,i≥0]−⋅1(r⊤iκ‖r⊙i‖2) :=E_c_i [c_iI \c_i u_r,i≥ 0 \ ]- 1· _1 ( w_r x_iκ \|w_r x_i \|_2 ) k,r,r′,i _k,r,r ,i :=[⊤r,i⊤i≥0;r′,i⊤i≥0]r′,i :=E_c [cc I \u_r,i c_i≥ 0;u_r ,i c_i≥ 0 \ ]u_r ,i −(⊤r′,i+3κ2r′,i)1(r⊤iκ‖r⊙i‖2)1(r′⊤iκ‖r′⊙i‖2) - ( 1 1 u_r ,i+3κ^2u_r ,i ) _1 ( w_r x_iκ \|w_r x_i \|_2 ) _1 ( w_r x_iκ \|w_r x_i \|_2 ) satisfies ‖r,i(1)‖∞ \| ^(1)_r,i \|_∞ ≤κRϕmax ≤κ R_u _ ‖r,r′,i(2)‖∞ \| ^(2)_r,r ,i \|_∞ ≤4κ‖2(dϕmax+ψmax) ≤ 4κ \|v \|_2 ( d _ + _ ) Here we used ‖∞=1 \| μ \|_∞=1 and ⊤(r⊙i)=k,r⊤i μ (w_r x_i )=w_k,r x_i when =1 μ=1. Therefore, for 1,iT_1,i, we have 1,i _1,i =1m∑r′=1mar′Diag(i)((r′⊤i⋅)1(r′⊤iκ‖r′s⊙i‖2)1(r′⊤iκ‖r′⊙i‖2)+r,r′,i(2)) = 1 m _r =1^ma_r Diag (x_i ) ( (w_r x_i· 1 ) _1 ( w_r x_iκ \|w_r s x_i \|_2 ) _1 ( w_r x_iκ \|w_r x_i \|_2 )+ _r,r ,i^(2) ) +3κ2m∑r′=1mar′Diag(i)(r⊙i)1(r⊤iκ‖r⊙i‖2)1(r′⊤iκ‖r′⊙i‖2) + 3κ^2 m _r =1^ma_r Diag (x_i ) (w_r x_i ) _1 ( w_r x_iκ \|w_r x_i \|_2 ) _1 ( w_r x_iκ \|w_r x_i \|_2 ) =1m∑r′=1mar′r′⊤i⋅i1(r′⊤iκ‖r′s⊙i‖2)1(r′⊤iκ‖r′⊙i‖2)+1m∑r′=1mar′(i⊙r,r′,i(2)) = 1 m _r =1^ma_r w_r x_i·x_i _1 ( w_r x_iκ \|w_r s x_i \|_2 ) _1 ( w_r x_iκ \|w_r x_i \|_2 )+ 1 m _r =1^ma_r (x_i _r,r ,i^(2) ) +3κ2m∑r′=1mar′Diag(i)2r′1(r′⊤iκ‖r′s⊙i‖2)1(r′⊤iκ‖r′⊙i‖2) + 3κ^2 m _r =1^ma_r Diag (x_i )^2w_r _1 ( w_r x_iκ \|w_r s x_i \|_2 ) _1 ( w_r x_iκ \|w_r x_i \|_2 ) =f(,i)ir⊤i≥0+3κ2m∑r′=1mar′Diag(i)2r′r⊤i≥0;r′⊤i≥0 =f ( θ,x_i )x_iI \w_r x_i≥ 0 \+ 3κ^2 m _r =1^ma_r Diag (x_i )^2w_r I \w_r x_i≥ 0;w_r x_i≥ 0 \ +1m∑r′=1mar′(i⊙r,r′,i(2))+1,i+2,i + 1 m _r =1^ma_r (x_i _r,r ,i^(2) )+g_1,i+g_2,i where 1,i=1m∑r′=1mar′r′⊤i⋅i(1(r′⊤iκ‖r′s⊙i‖2)1(r′⊤iκ‖r′⊙i‖2)−r⊤i≥0;r′⊤i≥0) g_1,i= 1 m _r =1^ma_r w_r x_i·x_i ( _1 ( w_r x_iκ \|w_r s x_i \|_2 ) _1 ( w_r x_iκ \|w_r x_i \|_2 )-I \w_r x_i≥ 0;w_r x_i≥ 0 \ ) 2,i=3κ2m∑r′=1mar′Diag(i)2r′(1(r′⊤iκ‖r′s⊙i‖2)1(r′⊤iκ‖r′⊙i‖2)−r⊤i≥0;r′⊤i≥0) g_2,i= 3κ^2 m _r =1^ma_r Diag (x_i )^2w_r ( _1 ( w_r x_iκ \|w_r s x_i \|_2 ) _1 ( w_r x_iκ \|w_r x_i \|_2 )-I \w_r x_i≥ 0;w_r x_i≥ 0 \ ) Likely, for 2,iT_2,i we have 2,i _2,i =(1(r⊤iκ‖r⊙i‖2)+r,i(1))⊙i = ( 1 _1 ( w_r x_iκ \|w_r x_i \|_2 )+ ^(1)_r,i ) x_i =i1(r⊤iκ‖r⊙i‖2)+r,i(1)⊙i =x_i _1 ( w_r x_iκ \|w_r x_i \|_2 )+ ^(1)_r,i x_i =ir⊤i≥0+r,i(1)⊙i+3,i =x_iI \w_r x_i≥ 0 \+ ^(1)_r,i x_i+g_3,i where 3,i=i⋅(1(r⊤iκ‖r⊙i‖2)−r⊤i≥0)g_3,i=x_i· ( _1 ( w_r x_iκ \|w_r x_i \|_2 )-I \w_r x_i≥ 0 \ ). Therefore, the final gradient is given by [∇rℒ()] _C [ _w_rL_C ( θ ) ] =arm∑i=1n(f(,i)−yi)ir⊤i = a_r m _i=1^n (f ( θ,x_i )-y_i )x_iI \w_r x_i \ +arm∑i=1n(1m∑r′=1mar′(i⊙r,r′,i(2))+yir,i(1)⊙i) + a_r m _i=1^n ( 1 m _r =1^ma_r (x_i _r,r ,i^(2) )+y_i ^(1)_r,i x_i ) +3κ2m∑r′=1mar′Diag(i)2r′r⊤i≥0;r′⊤i≥0 + 3κ^2 m _r =1^ma_r Diag (x_i )^2w_r I \w_r x_i≥ 0;w_r x_i≥ 0 \ +arm∑i=1n(1,i+2,i−yi⋅3,i) + a_r m _i=1^n (g_1,i+g_2,i-y_i·g_3,i ) =∇rℒ()+3κ2m∑r′=1mar′Diag(i)2r′r⊤i≥0;r′⊤i≥0 = _w_rL ( θ )+ 3κ^2 m _r =1^ma_r Diag (x_i )^2w_r I \w_r x_i≥ 0;w_r x_i≥ 0 \ +arm∑i=1n(1m∑r′=1mar′(i⊙r,r′,i(2))+yir,i(1)⊙i)⏟4 + a_r m _i=1^n ( 1 m _r =1^ma_r (x_i _r,r ,i^(2) )+y_i ^(1)_r,i x_i )_g_4 +arm∑i=1n(1,i+2,i−yi⋅3,i) + a_r m _i=1^n (g_1,i+g_2,i-y_i·g_3,i ) Notice that we can re-write 1,ig_1,i as 1,i g_1,i =i(1(r⊤iκ‖r⊙i‖2)−r⊤i≥0)⋅1m∑r′=1mar′r′⊤i(r′⊤i≥0) =x_i ( _1 ( w_r x_iκ \|w_r x_i \|_2 )-I \w_r x_i≥ 0 \ )· 1 m _r =1^ma_r w_r x_iI \ (w_r x_i≥ 0 ) \ +i1(r⊤iκ‖r⊙i‖2)⋅1m∑r′=1mar′r′⊤i(1(r′⊤iκ‖r′⊙i‖2)−r′⊤i≥0) +x_i _1 ( w_r x_iκ \|w_r x_i \|_2 )· 1 m _r =1^ma_r w_r x_i ( _1 ( w_r x_iκ \|w_r x_i \|_2 )-I \w_r x_i≥ 0 \ ) =i1(r⊤iκ‖r⊙i‖2)⋅1m∑r′=1mar′r′⊤i(1(r′⊤iκ‖r′⊙i‖2)−r′⊤i≥0) =x_i _1 ( w_r x_iκ \|w_r x_i \|_2 )· 1 m _r =1^ma_r w_r x_i ( _1 ( w_r x_iκ \|w_r x_i \|_2 )-I \w_r x_i≥ 0 \ ) +i(1(r⊤iκ‖r⊙i‖2)−r⊤i≥0)⋅f(,i) +x_i ( _1 ( w_r x_iκ \|w_r x_i \|_2 )-I \w_r x_i≥ 0 \ )· f ( θ,x_i ) Then, by the definition of 3,ig_3,i, we have that 1,i−yi⋅3,i g_1,i-y_i·g_3,i =i1(r⊤iκ‖r⊙i‖2)⋅1m∑r′=1mar′r′⊤i(1(r′⊤iκ‖r′⊙i‖2)−r′⊤i≥0) =x_i _1 ( w_r x_iκ \|w_r x_i \|_2 )· 1 m _r =1^ma_r w_r x_i ( _1 ( w_r x_iκ \|w_r x_i \|_2 )-I \w_r x_i≥ 0 \ ) +(f(,i)−yi)i(1(r⊤iκ‖r⊙i‖2)−r⊤i≥0) + (f ( θ,x_i )-y_i )x_i ( _1 ( w_r x_iκ \|w_r x_i \|_2 )-I \w_r x_i≥ 0 \ ) Using Lemma D.4, we have that |1(a)−a≥0|≤exp(−a22)≤ϕ(a2) | _1 (a )-I \a≥ 0 \ |≤ (- a^22 )≤φ ( a2 ) Therefore, we have that ‖∑i=1n(1,i−yi⋅3,i)‖2 \| _i=1^n (g_1,i-y_i·g_3,i ) \|_2 ≤nm‖∑r′=1mar′r′⊤i(1(r′⊤iκ‖r′⊙i‖2)−r′⊤i≥0)‖2 ≤ n m \| _r =1^ma_r w_r x_i ( _1 ( w_r x_iκ \|w_r x_i \|_2 )-I \w_r x_i≥ 0 \ ) \|_2 +‖∑i=1n(f(,i)−yi)⋅i(1(r⊤iκ‖r⊙i‖2)−r⊤i≥0)‖2 + \| _i=1^n (f ( θ,x_i )-y_i )·x_i ( _1 ( w_r x_iκ \|w_r x_i \|_2 )-I \w_r x_i≥ 0 \ ) \|_2 ≤nm∑r′=1m|r′⊤i|ϕ(r′⊤i2κ‖r′⊙i‖2)+‖Diag()(f(−))‖2 ≤ n m _r =1^m |w_r x_i |φ ( w_r x_i2κ \|w_r x_i \|_2 )+ \|Diag ( )X (f ( θ-y ) ) \|_2 =κmRψmax+σmax()ϕmaxℒ()12 =κ mR_u _ + _ (X ) _ L ( θ ) 12 Moreover, we can bound 2,ig_2,i as ‖2,i‖ \|g_2,i \| ≤3κ2m∑r′=1m‖∞2‖r‖⋅2ϕmax≤6κ2mB2Rϕmax ≤ 3κ^2 m _r =1^m \|x \|_∞^2 \|w_r \|· 2 _ ≤ 6κ^2 mB_x^2R_w _ Lastly, we can bound 3g_3 as ‖3‖2 \|g_3 \|_2 ≤1m∑i=1n∑r′=1m‖i⊙r,r′,i(2)‖+1m∑i=1n|yi|‖r,i(1)⊙i‖2 ≤ 1m _i=1^n _r =1^m \|x_i _r,r ,i^(2) \|+ 1 m _i=1^n |y_i | \| ^(1)_r,i x_i \|_2 ≤1m∑i=1n∑r′=1m‖r,r′,i(2)‖∞‖i‖2+1m∑i=1n|yi|‖r,r′,i(1)‖∞‖i‖2 ≤ 1m _i=1^n _r =1^m \| _r,r ,i^(2) \|_∞ \|x_i \|_2+ 1 m _i=1^n |y_i | \| _r,r ,i^(1) \|_∞ \|x_i \|_2 ≤1m∑i=1n∑r′=1m‖r,r′,i(2)‖∞+Bym∑i=1n‖r,r′,i(1)‖∞ ≤ 1m _i=1^n _r =1^m \| _r,r ,i^(2) \|_∞+ B_y m _i=1^n \| _r,r ,i^(1) \|_∞ ≤4nκR(dϕmax+ψmax)+Bym⋅nκRϕmax ≤ 4nκ R_u ( d _ + _ )+ B_y m· nκ R_u _ ≤5nκR(dϕmax+ψmax) ≤ 5nκ R_u ( d _ + _ ) when By≤mdB_y≤ md. Therefore, we have that ‖[∇rℒ()]−(∇rℒ()+3κ2m∑r′=1mar′Diag(i)2r′r⊤i≥0;r′⊤i≥0)‖2 \|E_C [ _w_rL_C ( θ ) ]- ( _w_rL ( θ )+ 3κ^2 m _r =1^ma_r Diag (x_i )^2w_r I \w_r x_i≥ 0;w_r x_i≥ 0 \ ) \|_2 ≤nκRψmax+σmax()ϕmaxmℒ()12+6nκ2B2Rϕmax+5nκR(dϕmax+ψmax) ≤ nκ R_u _ + _ (X ) _ mL ( θ ) 12+6nκ^2B_x^2R_w _ +5nκ R_u ( d _ + _ ) ≤(σmax()mℒ()12+6nκ2B2R+5nκRd)ϕmax+6nκRψmax ≤ ( _ (X ) mL ( θ ) 12+6nκ^2B_x^2R_w+5nκ R_u d ) _ +6nκ R_u _ ∎ Appendix C Proofs in Section 5 C.1 Proof of Theorem 5.2 Proof. To start the proof, we define the following quantity in the standard NTK-based analysis of two-layer ReLU neural network. Let R=C1⋅τλ0nR=C_1· τ _0n for some C1>0C_1>0, we define event Ai,rA_i,r and set Si,Si⟂S_i,S_i as Ai,r=∃∈ℬ(0,r,R):0,r⊤i≥0≠⊤i≥0 A_i,r= \∃w (w_0,r,R ):I \w_0,r x_i≥ 0 \ \w x_i≥ 0 \ \ (23) Si=r∈[m]:¬Ai,r;Si⟂=[m]∖Si S_i= \r∈[m]: A_i,r \; S_i =[m] S_i (24) Lemma 16 from shows that with probability at least 1−nexp(−mRτ)1-n (- mRτ ), we have that |Si⟂|≤4mRτ |S_i |≤ 4mRτ. In the following of the proof, we assume that such event holds. Define K′=mink∈ℕ:∃r∈[m] s.t. ‖k,r−0,r‖2>RK = \k :∃ r∈[m] s.t. \|w_k,r-w_0,r \|_2>R \. Then for all k<K′k<K , we have that k,r∈ℬ(0,r,R)w_k,r (w_0,r,R ). Fix any k<K′−1k<K -1. Consider the expansion of ℒ(k+1)L ( θ_k+1 ) as the following ℒ(k+1) L ( θ_k+1 ) =12∑i=1n(f(k+1,i)−yi)2 = 12 _i=1^n (f ( θ_k+1,x_i )-y_i )^2 (25) =12∑i=1n((f(k+1,i)−f(k,i))+(f(k,i)−yi))2 = 12 _i=1^n ( (f ( θ_k+1,x_i )-f ( θ_k,x_i ) )+ (f ( θ_k,x_i )-y_i ) )^2 =12∑i=1n(f(k+1,i)−f(k,i))2+∑i=1n(f(k+1,i)−f(k,i))(f(k,i)−yi) = 12 _i=1^n (f ( θ_k+1,x_i )-f ( θ_k,x_i ) )^2+ _i=1^n (f ( θ_k+1,x_i )-f ( θ_k,x_i ) ) (f ( θ_k,x_i )-y_i ) +12∑i=1n(f(k,i)−yi)2 + 12 _i=1^n (f ( θ_k,x_i )-y_i )^2 We will analyze the three terms separately. To start, notice that 12∑i=1n(f(k,i)−yi)2=ℒ(k) 12 _i=1^n (f ( θ_k,x_i )-y_i )^2=L ( θ_k ) (26) For the first term, by the definition of f(,)f ( θ,x ), we have |f(k+1,i)−f(k,i)| |f ( θ_k+1,x_i )-f ( θ_k,x_i ) | =|1m∑r=1mar(σ(k+1,r⊤i)−σ(k,r⊤i))| = | 1 m _r=1^ma_r (σ (w_k+1,r x_i )-σ (w_k,r x_i ) ) | ≤1m∑r=1m|σ(k+1,r⊤i)−σ(k,r⊤i)| ≤ 1 m _r=1^m |σ (w_k+1,r x_i )-σ (w_k,r x_i ) | ≤1m∑r=1m|(k+1−k)⊤i| ≤ 1 m _r=1^m | (w_k+1-w_k ) x_i | ≤1m∑r=1m‖k+1−k‖ ≤ 1 m _r=1^m \|w_k+1-w_k \| =ηm∑r=1m‖∇rℒ^(k,k)‖2 = η m _r=1^m \| _w_r L ( θ_k, ξ_k ) \|_2 where in the first inequality we use the fact that a=±1a=± 1, and in the second inequality we use the 11-Lipschitzness of ReLU. Applying Assumption 5.1, we have that ∑i=1n(f(k+1,i)−f(k,i))2 _i=1^n (f ( θ_k+1,x_i )-f ( θ_k,x_i ) )^2 ≤η2m∑i=1n(∑r=1m‖∇rℒ^(k,k)‖2) ≤ η^2m _i=1^n ( _r=1^m \| _w_r L ( θ_k, ξ_k ) \|_2 ) (27) ≤η2nm⋅(m⋅γℒ^(k,k))2 ≤ η^2nm· (m· γ L ( θ_k, ξ_k ) )^2 =η2mnγℒ^(k,k) =η^2mnγ L ( θ_k, ξ_k ) Lastly, to analyze the second term, we use the following definition of Ii,kI_i,k and Ii,k⟂I_i,k Ii,k=1m∑r∈Siarσ(k,r⊤i);Ii,k⟂=1m∑r∈Si⟂arσ(k,r⊤i) I_i,k= 1 m _r∈ S_ia_rσ (w_k,r x_i ); I_i,k = 1 m _r∈ S_i a_rσ (w_k,r x_i ) Then we have that f(k,i)=Ii,k+Ii,k⟂f ( θ_k,x_i )=I_i,k+I_i,k . Therefore f(k+1,i)−f(k,i)=(Ii,k+1−Ii,k)+(Ii,k+1⟂−Ii,k⟂)f ( θ_k+1,x_i )-f ( θ_k,x_i )= (I_i,k+1-I_i,k )+ (I_i,k+1 -I_i,k ) By the 11-Lipschitzness of ReLU, we have that |Ii,k+1⟂−Ii,k⟂| |I_i,k+1 -I_i,k | =|1m∑r∈Si⟂ar(σ(k+1,r⊤i)−σ(k,r⊤i))| = | 1 m _r∈ S_i a_r (σ (w_k+1,r x_i )-σ (w_k,r x_i ) ) | ≤1m∑r∈Si⟂|σ(k+1,r⊤i)−σ(k,r⊤i)| ≤ 1 m _r∈ S_i |σ (w_k+1,r x_i )-σ (w_k,r x_i ) | ≤1m∑r∈Si⟂|(k+1,r−k,r)⊤i| ≤ 1 m _r∈ S_i | (w_k+1,r-w_k,r ) x_i | ≤ηm∑r∈Si⟂‖∇rℒ^(k,k)‖2 ≤ η m _r∈ S_i \| _w_r L ( θ_k, ξ_k ) \|_2 ≤ηγm|Si⟂|ℒ^(k,k)12 ≤ η γ m |S_i | L ( θ_k, ξ_k ) 12 Applying |Si⟂|≤4mRτ |S_i |≤ 4mRτ gives |Ii,k+1⟂−Ii,k⟂|≤4ηRτγmℒ^(k,k)12 |I_i,k+1 -I_i,k |≤ 4η Rτ γ m L ( θ_k, ξ_k ) 12.This gives that ∑i=1n(f(k+1,i)−f(k,i))(f(k,i)−yi) _i=1^n (f ( θ_k+1,x_i )-f ( θ_k,x_i ) ) (f ( θ_k,x_i )-y_i ) (28) =∑i=1n(Ii,k+1−Ii,k)(f(k,i)−yi)+∑i=1n(Ii,k+1⟂−Ii,k⟂)(f(k,i)−yi) = _i=1^n (I_i,k+1-I_i,k ) (f ( θ_k,x_i )-y_i )+ _i=1^n (I_i,k+1 -I_i,k ) (f ( θ_k,x_i )-y_i ) ≤∑i=1n(Ii,k+1−Ii,k)(f(k,i)−yi)+(∑i=1n(Ii,k+1⟂−Ii,k⟂)2)12(∑i=1n(f(k,i)−yi)2)12 ≤ _i=1^n (I_i,k+1-I_i,k ) (f ( θ_k,x_i )-y_i )+ ( _i=1^n (I_i,k+1 -I_i,k )^2 ) 12 ( _i=1^n (f ( θ_k,x_i )-y_i )^2 ) 12 ≤∑i=1n(Ii,k+1−Ii,k)(f(k,i)−yi)+4ηRτγmnℒ^(k,k)12ℒ(k)12 ≤ _i=1^n (I_i,k+1-I_i,k ) (f ( θ_k,x_i )-y_i )+ 4η Rτ γ mn L ( θ_k, ξ_k ) 12L ( θ_k ) 12 Plugging (26), (27), and (28) into (25) gives ℒ(k+1) L ( θ_k+1 ) ≤ℒ(k)+η2mnγℒ^(k,k)+4ηRτγmnℒ^(k,k)12ℒ(k)12 ≤L ( θ_k )+η^2mnγ L ( θ_k, ξ_k )+ 4η Rτ γ mn L ( θ_k, ξ_k ) 12L ( θ_k ) 12 +∑i=1n(Ii,k+1−Ii,k)(f(k,i)−yi) + _i=1^n (I_i,k+1-I_i,k ) (f ( θ_k,x_i )-y_i ) Under Jensen’s inequality, we have that k[ℒ^(k,k)12]≤k[ℒ^(k,k)]12E_ ξ_k [ L ( θ_k, ξ_k ) 12 ] _ ξ_k [ L ( θ_k, ξ_k ) ] 12. Using the property that k[ℒ^(k,k)]≤2ℒ(k)+ε1E_ ξ_k [ L ( θ_k, ξ_k ) ]≤ 2L ( θ_k )+ _1 from Assumption 5.1, we can also obtain that k[ℒ^(k,k)12]≤(2ℒ(k)+ε1)12E_ ξ_k [ L ( θ_k, ξ_k ) 12 ]≤ (2L ( θ_k )+ _1 ) 12 Therefore, taking the expectation of ℒ(k+1)L ( θ_k+1 ) gives k[ℒ(k+1)] _ ξ_k [L ( θ_k+1 ) ] ≤ℒ(k)+η2mnγ(2ℒ(k)+ε)+4ηRτγmn(2ℒ(k)+ε1)12ℒ(k)12 ≤L ( θ_k )+η^2mnγ (2L ( θ_k )+ )+ 4η Rτ γ mn (2L ( θ_k )+ _1 ) 12L ( θ_k ) 12 (29) +∑i=1nk[Ii,k+1−Ii,k](f(k,i)−yi) + _i=1^nE_ ξ_k [I_i,k+1-I_i,k ] (f ( θ_k,x_i )-y_i ) ≤ℒ(k)+η2mnγ(2ℒ(k)+ε1)+10ηRτγmnℒ(k)+4ηRτγmn⋅ε1 ≤L ( θ_k )+η^2mnγ (2L ( θ_k )+ _1 )+ 10η Rτ γ mnL ( θ_k )+ 4η Rτ γ mn· _1 +∑i=1nk[Ii,k+1−Ii,k](f(k,i)−yi) + _i=1^nE_ ξ_k [I_i,k+1-I_i,k ] (f ( θ_k,x_i )-y_i ) =(1+2η2mnγ+10Cηλ0γmn)ℒ(k)+(η2mnγ+4Cηλ0γmn)ε1 = (1+2η^2mnγ+10Cη _0 γ mn )L ( θ_k )+ (η^2mnγ+4Cη _0 γ mn ) _1 +∑i=1nk[Ii,k+1−Ii,k](f(k,i)−yi) + _i=1^nE_ ξ_k [I_i,k+1-I_i,k ] (f ( θ_k,x_i )-y_i ) where in the last inequality we use the property that a(a+b)≤54a+b a(a+b)≤ 54a+b. Recall that k+1,r,k,r∈ℬ(0,r,R)w_k+1,r,w_k,r (w_0,r,R ). Therefore, for r∈Sir∈ S_i, we must have that k+1,r⊤i≥0=0,r⊤i≥0=k,r⊤i≥0I \w_k+1,r x_i≥ 0 \=I \w_0,r x_i≥ 0 \=I \w_k,r x_i≥ 0 \. Thus, we have k[Ii,k+1−Ii,k] _ ξ_k [I_i,k+1-I_i,k ] =1m∑r∈Siark[k+1,r−k,r]⊤ik,r⊤i≥0 = 1 m _r∈ S_ia_rE_ ξ_k [w_k+1,r-w_k,r ] x_iI \w_k,r x_i≥ 0 \ (30) =−ηm∑r∈Siark[∇rℒ^(k,k)]⊤ik,r⊤i≥0 =- η m _r∈ S_ia_rE_ ξ_k [ _w_r L ( θ_k, ξ_k ) ] x_iI \w_k,r x_i≥ 0 \ Let k,r=k[∇rℒ^(k,k)]−∇rℒ(k)g_k,r=E_ ξ_k [ _w_r L ( θ_k, ξ_k ) ]- _w_rL ( θ_k ). Then by Assumption 5.1 we have that ‖k,r‖2≤ε3ℒ()12+ε2 \|g_k,r \|_2≤ _3L ( θ ) 12+ _2. Using k,rg_k,r, we can write (30) as k[Ii,k+1−Ii,k]=−η∑r∈Siarm∇rℒ(k)⊤ik,r⊤i≥0−ηm∑r∈Siark,r⊤ik,r⊤i≥0E_ ξ_k [I_i,k+1-I_i,k ]=-η _r∈ S_i a_r m _w_rL ( θ_k ) x_iI \w_k,r x_i≥ 0 \- η m _r∈ S_ia_rg_k,r x_iI \w_k,r x_i≥ 0 \ (31) By definition, we have ∇rℒ(k)=arm∑j=1n(f(k,i)−yi)jk,r⊤j≥0 _w_rL ( θ_k )= a_r m _j=1^n (f ( θ_k,x_i )-y_i )x_jI \w_k,r x_j≥ 0 \ Therefore, we have that arm∇rℒ(k)⊤ik,r⊤i≥0=1m∑j=1n(f(k,i)−yi)i⊤jk,r⊤i≥0k,r⊤j≥0 a_r m _w_rL ( θ_k ) x_iI \w_k,r x_i≥ 0 \= 1m _j=1^n (f ( θ_k,x_i )-y_i )x_i x_jI \w_k,r x_i≥ 0w_k,r x_j≥ 0 \ Combining with (31), we have that ∑i=1nk[Ii,k+1−Ii,k](f(k,i)−yi) _i=1^nE_ ξ_k [I_i,k+1-I_i,k ] (f ( θ_k,x_i )-y_i ) =−ηm∑i,j=1n∑r∈Si(f(k,i)−yi)(f(k,j)−yj)i⊤jk,r⊤i≥0;k,r⊤j≥0 =- ηm _i,j=1^n _r∈ S_i (f ( θ_k,x_i )-y_i ) (f ( θ_k,x_j )-y_j )x_i x_jI \w_k,r x_i≥ 0;w_k,r x_j≥ 0 \ −ηm∑i=1n∑r∈Si(f(k,i)−yi)ark,r⊤ik,r⊤i≥0 - η m _i=1^n _r∈ S_i (f ( θ_k,x_i )-y_i )a_rg_k,r x_iI \w_k,r x_i≥ 0 \ ≤−η∑i,j=1n(f(k,i)−yi)(i⊤jm∑r∈Sik,r⊤i≥0;k,r⊤j≥0)⏟k,ij(f(k,j)−yj) ≤-η _i,j=1^n (f ( θ_k,x_i )-y_i ) ( x_i x_jm _r∈ S_iI \w_k,r x_i≥ 0;w_k,r x_j≥ 0 \ )_H_k,ij (f ( θ_k,x_j )-y_j ) +ηm∑i=1m∑r=1n|f(k,i)−yi|‖k,r‖2 + η m _i=1^m _r=1^n |f ( θ_k,x_i )-y_i | \|g_k,r \|_2 ≤−ηλmin(k)∑i=1n(f(k,i)−yi)2+ηε2mn(∑r=1n(f(k,i)−yi)2)12+ηε3mnℒ(k) ≤-η _ (H_k ) _i=1^n (f ( θ_k,x_i )-y_i )^2+η _2 mn ( _r=1^n (f ( θ_k,x_i )-y_i )^2 ) 12+η _3 mnL ( θ_k ) =−(2ηλmin(k)+ηε3mn)ℒ(k)+2ηε2mnℒ(k)12 =- (2η _ (H_k )+η _3 mn )L ( θ_k )+2η _2 mnL ( θ_k ) 12 Using the property that ab≤a22+b22ab≤ a^22+ b^22, we have that for any C′>0C >0, 2ηε2mnℒ(k)12≤C′ηλ0γmn+ηε22nC′λ0mnγ2η _2 mnL ( θ_k ) 12≤ C η _0 γ mn+ η _2^2nC _0 mnγ Moreover, by Lemma C.1, we have that when m=Ω(n2λ02lognδ)m= ( n^2 _0^2 nδ ), with probability at least 1−δ−n2exp−mRτ1-δ-n^2 - mRτ, it holds that ‖k−∞‖F≤λ06+1m(∑i,j=1n|Si⟂|2)12+2nRτ \|H_k-H^∞ \|_F≤ _06+ 1m ( _i,j=1^n |S_i |^2 ) 12+ 2nRτ Plugging in |Si⟂|≤4mRτ |S_i |≤ 4mRτ and R≤C1⋅τλ0nR≤ C_1· τ _0n, we have that ‖k−∞‖F≤λ02 \|H_k-H^∞ \|_F≤ _02 for small enough C1C_1. Thus, we have that λmin(k)≥λ02 _ (H_k )≥ _02. Therefore, we have that ∑i=1nk[Ii,k+1−Ii,k](f(k,i)−yi)≤(C′ηλ0γmn+ηε3mn−ηλ0)ℒ(k)+ηε22nC′λ0mnγ _i=1^nE_ ξ_k [I_i,k+1-I_i,k ] (f ( θ_k,x_i )-y_i )≤ (C η _0 γ mn+η _3 mn-η _0 )L ( θ_k )+ η _2^2nC _0 mnγ Plugging this back into (31) gives k[ℒ(k+1)] _ ξ_k [L ( θ_k+1 ) ] ≤(1+2η2mnγ+10Cηλ0γmn)ℒ(k)+(η2mnγ+4Cηλ0γmn)ε1 ≤ (1+2η^2mnγ+10Cη _0 γ mn )L ( θ_k )+ (η^2mnγ+4Cη _0 γ mn ) _1 +(C′ηλ0γmn+ηε3mn−ηλ0)ℒ(k)+ηε22nC′λ0mnγ + (C η _0 γ mn+η _3 mn-η _0 )L ( θ_k )+ η _2^2nC _0 mnγ =(1−ηλ0+ηε3mn+2η2mnγ+(10C+C′)ηλ0γmn)ℒ(k) = (1-η _0+η _3 mn+2η^2mnγ+ (10C+C )η _0 γ mn )L ( θ_k ) +(η2mnγ+4Cηλ0γmn)ε1+ηε22nC′λ0mnγ + (η^2mnγ+4Cη _0 γ mn ) _1+ η _2^2nC _0 mnγ Apply ε3≤Cε⋅λ0mn,γ=C1⋅nm _3≤ C_ · _0 mn,γ=C_1· nm and η=C2⋅λ0n2η=C_2· _0n^2 gives k[ℒ(k+1)] _ ξ_k [L ( θ_k+1 ) ] ≤(1−(1−2C1C2−(10C+C′)C1)ηλ0)ℒ(k) ≤ (1- (1-2C_1C_2- (10C+C ) C_1 )η _0 )L ( θ_k ) +ηλ0(mnε22C′C1λ02+(C1C2+4CC1)ε1) +η _0 ( mn _2^2C C_1 _0^2+ (C_1C_2+4C C_1 ) _1 ) Choosing a small enough C1,C2,C,C′C_1,C_2,C,C gives k[ℒ(k+1)]≤(1−ηλ02)ℒ(k)+12C^ηλ0(mnλ02⋅ε22+ε1)E_ ξ_k [L ( θ_k+1 ) ]≤ (1- η _02 )L ( θ_k )+ 12 Cη _0 ( mn _0^2· _2^2+ _1 ) (32) for a large enough C C. Thus, unrolling the iterations gives 0,…,k−1[ℒ(k)]≤(1−ηλ02)kℒ(0)+C^(mnλ02⋅ε22+ε1)E_ ξ_0,…, ξ_k-1 [L (W_k ) ]≤ (1- η _02 )^kL (W_0 )+ C ( mn _0^2· _2^2+ _1 ) (33) for all k<K′k<K . Next, we shall lower bound K′K . For all k≤K′k≤ K , we have that ‖k,r−0,r‖2 \|w_k,r-w_0,r \|_2 ≤∑t=0k−1‖t+1,r−t,r‖2=η∑t=0k−1‖∇rℒ^(t,t)‖2≤ηγ∑t=0k−1ℒ^(t,t)12 ≤ _t=0^k-1 \|w_t+1,r-w_t,r \|_2=η _t=0^k-1 \| _w_r L (W_t, ξ_t ) \|_2≤η γ _t=0^k-1 L (W_t, ξ_t ) 12 By (33), we have 0,…,t−1[ℒ^(t,t)12] _ ξ_0,…, ξ_t-1 [ L (W_t, ξ_t ) 12 ] ≤(ℒ(t)+ε1)12 ≤ (L (W_t )+ _1 ) 12 ≤(2(1−ηλ02)tℒ(0)+(C^+1)(mnλ02⋅ε22+ε1))12 ≤ (2 (1- η _02 )^tL (W_0 )+ ( C+1 ) ( mn _0^2· _2^2+ _1 ) ) 12 ≤2(1−ηλ04)tℒ(0)12+C^+1(ε2λ0mn+ε1) ≤ 2 (1- η _04 )^tL (W_0 ) 12+ C+1 ( _2 _0 mn+ _1 ) Therefore, we have 0,…,k−1[‖k,r−0,r‖2] _ ξ_0,…, ξ_k-1 [ \|w_k,r-w_0,r \|_2 ] ≤ηγ∑t=0k−10,…,t−1[ℒ^(t,t)12] ≤η γ _t=0^k-1E_ ξ_0,…, ξ_t-1 [ L (W_t, ξ_t ) 12 ] ≤2ηγℒ(0)12∑t=0∞(1−ηλ04)t+k(C^+1)γ(ε2λ0mn+ε1) ≤ 2η γL (W_0 ) 12 _t=0^∞ (1- η _04 )^t+k ( C+1 )γ ( _2 _0 mn+ _1 ) =8γλ0ℒ(0)12+k(C^+1)γ(ε2λ0mn+ε1) = 8 γ _0L (W_0 ) 12+k ( C+1 )γ ( _2 _0 mn+ _1 ) By Lemma 26 in , we have that 0,[ℒ(0)2]=O(n)E_W_0,a [L (W_0 )^2 ]=O (n ). γ=C1⋅nmγ=C_1· nm, we have that 0,,0,…,k−1[‖k,r−0,r‖2]≤O(nλ0m)+O(k(ε2nλ0+ε1nm))E_W_0,a, ξ_0,…, ξ_k-1 [ \|w_k,r-w_0,r \|_2 ]≤ O ( n _0 m )+O (k ( _2n _0+ _1nm ) ) Thus, by Markov’s inequality, we have that with probability at least 1−δ3K1- δ3K, ‖k,r−0,r‖2≤O(nKλ0δm)⏟1+O(K2δ(ε2nλ0+ε1nm))⏟2 \|w_k,r-w_0,r \|_2≤ O ( nK _0δ m )_T_1+ O ( K^2δ ( _2n _0+ _1nm ) )_T_2 Setting m=Ω(n4λ04δ2τ2)m= ( n^4 _0^4δ^2τ^2 ) guarantees that T1≤C12⋅τλ0n=R2T_1≤ C_12· τ _0n= R2 and set ε2≤O(δλ0nK2),ε1≤O(δmK4n) _2≤ O ( δ _0nK^2 ), _1≤ O ( δ mK^4n ) gives that T2≤C12⋅τλ0n=R2T_2≤ C_12· τ _0n= R2. Combining the bound on 1T_1 and 2T_2 and taking a union bound gives that, with probability at least 1−δ31- δ3, it holds that ‖k,r−0,r‖2≤R;∀k∈[K] \|w_k,r-w_0,r \|_2≤ R; ∀ k∈[K] This shows that we must have K′>K >K, which completes the proof. ∎ Lemma C.1. Let ∞H^∞ be defined in (11), and let kH_k be defined as k,ij=i⊤jm∑r∈Sik,r⊤i≥0;k,r⊤j≥0H_k,ij= x_i x_jm _r∈ S_iI \w_k,r x_i≥ 0;w_k,r x_j≥ 0 \ Fix any R. Assume that k,r∈ℬ(0,r,R)w_k,r (w_0,r,R ) for all r∈[m]r∈[m]. If 0,r∼(,τ2)w_0,r ( 0,τ^2I ), and m=Ω()m= ( ), then we have that ‖k−∞‖F≤λ06+1m2∑i,j=1n|Si⟂|2+2nRτ \|H_k-H^∞ \|_F≤ _06+ 1m^2 _i,j=1^n |S_i |^2+ 2nRτ Proof. We define ^k H_k as follows ^k,ij=i⊤jm∑r=1mk,r⊤i≥0;k,r⊤j≥0 H_k,ij= x_i x_jm _r=1^mI \w_k,r x_i≥ 0;w_k,r x_j≥ 0 \ Then we have that ‖k−∞‖F≤‖k−^k‖F+‖^k−^0‖F+‖^0−∞‖F \|H_k-H^∞ \|_F≤ \|H_k- H_k \|_F+ \| H_k- H_0 \|_F+ \| H_0-H^∞ \|_F By Lemma 3.1 in Du et al. [2018], we have that with probability at least 1−δ1-δ, we have that ‖^0−∞‖F≤λ06 \| H_0-H^∞ \|_F≤ _06 when m=Ω(n2λ02lognδ)m= ( n^2 _0^2 nδ ). By Lemma 3.2 in Song and Yang [2020], we have that with probability at least 1−n2exp−mRτ1-n^2 - mRτ, it holds that ‖^k−^0‖≤2nRτ \| H_k- H_0 \|≤ 2nRτ. Lastly, for the first term, we have ‖k−^k‖F2 \|H_k- H_k \|_F^2 ≤∑i,j=1n(k,ij−^k,ij)2 ≤ _i,j=1^n (H_k,ij- H_k,ij )^2 =1m2∑i,j=1n(i⊤j∑r∈Si⟂k,r⊤i≥0;k,r⊤j≥0)2 = 1m^2 _i,j=1^n (x_i x_j _r∈ S_i I \w_k,r x_i≥ 0;w_k,r x_j≥ 0 \ )^2 ≤1m2∑i,j=1n|Si⟂|2 ≤ 1m^2 _i,j=1^n |S_i |^2 Combining the three bounds gives the desired result. ∎ C.2 Proof of Theorem 5.6 We view Gaussian input masking as a special case of the general stochastic training framework in Section 5. Recall that in that framework, the randomness at iteration k is denoted by k ξ_k, and the update rule is k+1=k−η∇ℒ^(k,k).W_k+1=W_k-η _W L(W_k, ξ_k). In the Gaussian-masked setting we take k≡k,ℒ^(,k)≡ℒk(), ξ_k≡C_k, L(W, ξ_k)≡L_C_k(W), where kC_k is the multiplicative Gaussian mask at iteration k and ℒkL_C_k is the masked loss. Thus ∇rℒ^(,k)≡∇rℒk(), _w_r L(W, ξ_k)≡ _w_rL_C_k(W), and the update (7) coincides with the masked gradient descent rule (1). Therefore, to apply Theorem 5.2 to training with Gaussian input masks, it suffices to verify that Assumption 5.1 holds with suitable ε1,ε2,ε3,γ _1, _2, _3,γ, and that these parameters satisfy the smallness conditions of Theorem 5.2 under the constraints (14)–(15). Throughout the proof we condition on the high-probability NTK event of Theorem 5.2, on which • the minimum eigenvalue of the empirical NTK satisfies λmin(k)≥λ0/2 _ (H_k)≥ _0/2 for all k∈[K]k∈[K], • all first-layer weights remain in a ball of radius R=C1τλ0/nR=C_1τ _0/n around their initialization, i.e., ‖k,r−0,r‖2≤R\|w_k,r-w_0,r\|_2≤ R for all k∈[K],r∈[m]k∈[K],r∈[m], • the data are bounded as in Assumption 3.1. The probability of this event is at least 1−2δ−n2exp(−n3/(δ2τ2λ03))1-2δ-n^2 (-n^3/(δ^2τ^2 _0^3)), as in Theorem 5.2. All inequalities below hold on this event. Comparing Corollary 5.3 with Assumption 5.1(8), we identify ε1()=2mnκ2R2+mn(κ2R2+κR)ϕmax()2+mnκ2(R2+1)ψmax()2. _1(W)=2mnκ^2R_u^2+mn (κ^2R_u^2+κ R_w ) _ (W)^2+mnκ^2 (R_u^2+1 ) _ (W)^2. (34) On the NTK event, the weights stay close to initialization, hence their norms are uniformly bounded; using Assumption 3.1 and the definition of R_w and R_u, we obtain: R(k):=maxr∈[m]‖k,r‖2≤CwτdR_w(W_k):= _r∈[m]\|w_k,r\|_2≤ C_wτ d (35) R(k):=maxr∈[m],i∈[n]‖k,r⊙i‖2≤Cuτd.R_u(W_k):= _r∈[m],i∈[n]\|w_k,r x_i\|_2≤ C_uτ d. (36) for constants Cw,Cu>0C_w,C_u>0. Lemma C.2 (Bound on R_w). On the NTK event of Theorem 5.2, there exists an absolute constant Cw>0C_w>0 such that for all iterations k≤Kk≤ K: R(k):=maxr∈[m]‖k,r‖2≤Cwτd.R_w(W_k):= _r∈[m]\|w_k,r\|_2≤ C_wτ d. (37) Proof. Recall that the first–layer weights are initialized as 0,r∼(,τ2d)w_0,r (0,τ^2I_d) for r=1,…,mr=1,…,m. During training, the NTK event of Theorem 5.2 ensures that each row stays in a small ball around its initialization: ‖k,r−0,r‖2≤R,R:=C1τλ0n,∀k≤K,r∈[m].\|w_k,r-w_0,r\|_2≤ R, R:=C_1τ _0n, ∀ k≤ K,r∈[m]. (38) We define r:=1τ0,rz_r:= 1τw_0,r. Each coordinate satisfies (zr)j∼(0,1)(z_r)_j (0,1), making rz_r a standard Gaussian vector (,d)N(0,I_d). Its squared norm follows a chi-square distribution: ‖r‖22∼χd2\|z_r\|_2^2 χ^2_d. Using the Laurent–Massart concentration inequality, for t=dt=d: Pr(‖r‖22≥5d)≤e−d. (\|z_r\|_2^2≥ 5d )≤ e^-d. Thus, with high probability, ‖r‖2≤5d\|z_r\|_2≤ 5d. Defining C0=5C_0= 5, we obtain the initialization bound: ‖0,r‖2=τ‖r‖2≤C0τd.\|w_0,r\|_2=τ\|z_r\|_2≤ C_0τ d. (39) By a union bound over r∈[m]r∈[m], this holds for all rows with probability at least 1−me−d1-me^-d. Combining the triangle inequality with (38) and (39), we find: ‖k,r‖2 \|w_k,r\|_2 ≤‖0,r‖2+‖k,r−0,r‖2 ≤\|w_0,r\|_2+\|w_k,r-w_0,r\|_2 ≤C0τd+C1τλ0n ≤ C_0τ d+C_1τ _0n =τd(C0+C1λ0nd). =τ d (C_0+ C_1 _0n d ). Since λ0/n _0/n is O(1)O(1) and d≥1d≥ 1, we define the absolute constant Cw:=C0+C1λ0ndC_w:=C_0+ C_1 _0n d. Taking the maximum over r∈[m]r∈[m] yields: Rw(k)=maxr∈[m]‖k,r‖2≤Cwτd.R_w(W_k)= _r∈[m]\|w_k,r\|_2≤ C_wτ d. (40) Intuitively, since the movement term C1λ0nd C_1 _0n d vanishes as d→∞d→∞, the weights remain on the same scale as their initialization throughout training. ∎ Lemma C.3 (Bound on RuR_u). Suppose Assumption 3.1 holds, such that the input data is bounded in ℓ∞ _∞-norm by Bx:=maxi∈[n]‖xi‖∞B_x:= _i∈[n]\|x_i\|_∞. On the NTK event of Theorem 5.2, there exists an absolute constant Cu>0C_u>0 such that, for all iterations k≤Kk≤ K: Ru(Wk):=maxr∈[m],i∈[n]‖wk,r⊙xi‖2≤Cuτd.R_u(W_k):= _r∈[m],i∈[n]\|w_k,r x_i\|_2≤ C_uτ d. (41) Proof. Consider any iteration k≤Kk≤ K, neuron r∈[m]r∈[m], and sample index i∈[n]i∈[n]. We analyze the squared ℓ2 _2-norm of the Hadamard product by pulling out the maximum coordinate of the input vector: ‖wk,r⊙xi‖22 \|w_k,r x_i\|_2^2 =∑j=1dwk,r,j2xi,j2 = _j=1^dw_k,r,j^2x_i,j^2 ≤(max1≤j≤dxi,j2)∑j=1dwk,r,j2 ≤ ( _1≤ j≤ dx_i,j^2 ) _j=1^dw_k,r,j^2 =‖xi‖∞2‖wk,r‖22. =\|x_i\|_∞^2\|w_k,r\|_2^2. Taking the square root of both sides, we obtain the inequality: ‖wk,r⊙xi‖2≤‖xi‖∞‖wk,r‖2≤Bx‖wk,r‖2.\|w_k,r x_i\|_2≤\|x_i\|_∞\|w_k,r\|_2≤ B_x\|w_k,r\|_2. Taking the maximum over all r∈[m]r∈[m] and i∈[n]i∈[n] yields: Ru(Wk)≤BxRw(Wk).R_u(W_k)≤ B_xR_w(W_k). (42) From the weight stability bound previously established (Proof of RwR_w), we know that on the NTK event, the weights are bounded by Rw(Wk)≤CwτdR_w(W_k)≤ C_wτ d, where CwC_w is an absolute constant. Substituting this into (42): Ru(Wk)≤Bx(Cwτd).R_u(W_k)≤ B_x(C_wτ d). Defining the absolute constant Cu:=BxCwC_u:=B_xC_w completes the proof. Note that since BxB_x is a fixed property of the dataset and CwC_w is independent of k, CuC_u is a valid absolute constant for the problem. ∎ We an iteration k∈[K]k∈[K] and denote, for brevity, R:=R(k),R:=R(k),ϕk:=ϕmax(k),ψk:=ψmax(k).R_w:=R_w(W_k), R_u:=R_u(W_k), _k:= _ (W_k), _k:= _ (W_k). Then (34) becomes ε1(k)=2mnκ2R2+mn(κ2R2+κR)ϕk2+mnκ2(R2+1)ψk2. _1(W_k)=2mnκ^2R_u^2+mn (κ^2R_u^2+κ R_w ) _k^2+mnκ^2 (R_u^2+1 ) _k^2. (43) We now bound each of the three terms on the right-hand side using (35),(36). (i) First term. Using R2≤Cu2τ2dR_u^2≤ C_u^2τ^2d from (36), we get 2mnκ2R2 2mnκ^2R_u^2 ≤2mnκ2⋅Cu2τ2d ≤ 2mnκ^2· C_u^2τ^2d =(2Cu2)κ2τ2mnd=O(κ2τ2mnd). =(2C_u^2)κ^2τ^2mnd=O (κ^2τ^2mnd ). (44) (i) Middle term. mn(κ2R2+κR)ϕk2=mnκ2R2ϕk2+mnκRϕk2.mn (κ^2R_u^2+κ R_w ) _k^2=mnκ^2R_u^2 _k^2+mnκ R_w _k^2. For the κ2R2ϕk2κ^2R_u^2 _k^2 component, we use R2≤Cu2τ2dR_u^2≤ C_u^2τ^2d from (36): mnκ2R2ϕk2 mnκ^2R_u^2 _k^2 ≤mnκ2(Cu2τ2d)ϕk2 ≤ mnκ^2(C_u^2τ^2d) _k^2 =Cu2κ2τ2mndϕk2=O(κ2τ2mndϕk2). =C_u^2κ^2τ^2mnd _k^2=O (κ^2τ^2mnd _k^2 ). (45) For the κRϕk2κ R_w _k^2 component, we use R≤CwτdR_w≤ C_wτ d from (35): mnκRϕk2 mnκ R_w _k^2 ≤mnκ(Cwτd)ϕk2 ≤ mnκ(C_wτ d) _k^2 =Cwκτmndϕk2=O(κτmndϕk2). =C_wκτ mn d _k^2=O (κτ mn d _k^2 ). (46) (i) Last term. For the last term, using (36), it is true that R2+1≤Cu2τ2d+1R_u^2+1≤ C_u^2τ^2d+1, so there exists a constant Cu′>0C_u >0 such that R2+1≤Cu′τ2dR_u^2+1≤ C_u τ^2d for big enough d. Hence, mnκ2(R2+1)ψk2 mnκ^2 (R_u^2+1 ) _k^2 ≤mnκ2(Cu′τ2d)ψk2 ≤ mnκ^2(C_u τ^2d) _k^2 =Cu′κ2τ2mndψk2=O(κ2τ2mndψk2). =C_u κ^2τ^2mnd _k^2=O (κ^2τ^2mnd _k^2 ). (47) Combining (43) with (44), (45), (46), and (47), we obtain ε1(k) _1(W_k) ≤O(κ2τ2mnd)+O(κ2τ2mndϕk2)+O(κτmndϕk2)+O(κ2τ2mndψk2) ≤ O (κ^2τ^2mnd )+O (κ^2τ^2mnd _k^2 )+O (κτ mn d _k^2 )+O (κ^2τ^2mnd _k^2 ) =O(κ2τ2mnd)+O(κ2τ2mnd(ϕk2+ψk2))+O(κτmndϕk2). =O (κ^2τ^2mnd )+O (κ^2τ^2mnd( _k^2+ _k^2) )+O (κτ mn d _k^2 ). (48) We denote ϕ^max:=maxk∈[K]ϕmax(k),ψ^max:=maxk∈[K]ψmax(k), φ_ := _k∈[K] _ (W_k), ψ_ := _k∈[K] _ (W_k), so that for each k, ϕk2≤ϕ^max2,ψk2≤ψ^max2. _k^2≤ φ_ ^2, _k^2≤ ψ_ ^2. Substituting these into (48) yields ε1(k) _1(W_k) ≤O(κ2τ2mnd)+O(κ2τ2mnd(ϕ^max2+ψ^max2))+O(κτmndϕ^max2),∀k∈[K]. ≤ O (κ^2τ^2mnd )+O (κ^2τ^2mnd( φ_ ^2+ ψ_ ^2) )+O (κτ mn d φ_ ^2 ), ∀ k∈[K]. (49) Taking the maximum over k∈[K]k∈[K] does not change the right-hand side, so ε1:=maxk∈[K]ε1(k)≤O(κ2τ2mnd)+O(κ2τ2mnd(ϕ^max2+ψ^max2))+O(κτmndϕ^max2). _1:= _k∈[K] _1(W_k)≤ O (κ^2τ^2mnd )+O (κ^2τ^2mnd( φ_ ^2+ ψ_ ^2) )+O (κτ mn d φ_ ^2 ). (50) or equivalently: ε1≤O(κ2τ2mnd(ϕ^max2+ψ^max2+1))+O(κτmndϕ^max2) _1≤ O (κ^2τ^2mnd( φ_ ^2+ ψ_ ^2+1) )+O (κτ mn d φ_ ^2 ) (51) Matching Corollary 5.4 with Assumption 5.1(9), we read off ε3()=O(σmax()ϕmax()m) _3(W)=O ( _ (X) _ (W) m ) and ε2() _2(W) =O((nκ2B2R+nκRd)ϕmax()) =O ((nκ^2B_x^2R_w+nκ R_u d) _ (W) ) +O(nκRψmax()+κ2mB2R). +O (nκ R_u _ (W)+κ^2 mB_x^2R_w ). (52) Using ‖i‖2≤1\|x_i\|_2≤ 1 from Assumption 3.1, we have B≤1B_x≤ 1. On the NTK event, for all k∈[K]k∈[K] we have the uniform bounds R(k)≤Cwτd,R(k)≤Cuτd,R_w(W_k)≤ C_wτ d, R_u(W_k)≤ C_uτ d, for some absolute constants Cw,Cu>0C_w,C_u>0 (cf. the bounds proved earlier for R_w and R_u). Substituting these into (52) yields: ε2(k) _2(W_k) ≤O((nκ2Cwτd+nκCuτdd)ϕmax(k)) ≤ O ( (nκ^2C_wτ d+nκ C_uτ d d ) _ (W_k) ) +O(nκCuτdψmax(k)+κ2mCwτd) +O (nκ C_uτ d _ (W_k)+κ^2 mC_wτ d ) =O(nκ2τdϕmax(k))+O(nκτdϕmax(k)) =O (nκ^2τ d _ (W_k) )+O (nκτ d _ (W_k) ) +O(nκτdψmax(k))+O(κ2τmd). +O (nκτ d _ (W_k) )+O (κ^2τ md ). (53) Since d≥1d≥ 1, we have d≤d d≤ d, and hence each term containing d d can be upper bounded by the corresponding expression with d in place of d d. Therefore, ε2(k) _2(W_k) ≤O(nκ2τdϕmax(k))+O(nκτdϕmax(k)) ≤ O (nκ^2τ d _ (W_k) )+O (nκτ d _ (W_k) ) +O(nκτdψmax(k))+O(κ2τmd) +O (nκτ d _ (W_k) )+O (κ^2τ md ) ≤O(κτnd(ϕmax(k)+ψmax(k))) ≤ O (κτ nd ( _ (W_k)+ _ (W_k) ) ) +O(κ2τd(nϕmax(k)+m)). +O (κ^2τ d (n _ (W_k)+ m ) ). (54) To obtain a uniform bound over the whole training trajectory, define ϕ^max:=maxk∈[K]ϕmax(k),ψ^max:=maxk∈[K]ψmax(k). φ_ := _k∈[K] _ (W_k), ψ_ := _k∈[K] _ (W_k). Taking the maximum over k in (54) yields ε2:=maxk∈[K]ε2(k)≤O(κτnd(ϕ^max+ψ^max))+O(κ2τd(nϕ^max+m)). _2:= _k∈[K] _2(W_k)≤ O (κτ nd( φ_ + ψ_ ) )+O (κ^2τ d (n φ_ + m ) ). We can simplify ε2 _2 further as: O(κτnd(ϕ^max+ψ^max))+O(κ2τd(nϕ^max+m)) O (κτ nd( φ_ + ψ_ ) )+O (κ^2τ d (n φ_ + m ) ) =O(κτnd(ϕ^max+ψ^max)+κ2τd(nϕ^max+m)) =O (κτ nd( φ_ + ψ_ )+κ^2τ d (n φ_ + m ) ) =O(κτnd(ϕ^max+ψ^max)+κ2τdnϕ^max+κ2τdm) =O (κτ nd( φ_ + ψ_ )+κ^2τ dn φ_ +κ^2τ d m ) ≤O(κτnd(ϕ^max+ψ^max)+κτdn(ϕ^max+ψ^max)+κ2τdm) ≤ O (κτ nd( φ_ + ψ_ )+κτ dn( φ_ + ψ_ )+κ^2τ d m ) =O(2κτnd(ϕ^max+ψ^max)+κ2τdm) =O (2κτ nd( φ_ + ψ_ )+κ^2τ d m ) Thus, ε2≤O(κτnd(ϕ^max+ψ^max))+O(κ2τdm) _2≤ O (κτ nd( φ_ + ψ_ ) )+O (κ^2τ d m ) (55) Theorem 5.2 requires ε2 _2 to satisfy, for some absolute constant c0>0c_0>0, ε2≤c0δλ0nK2 _2\;≤\;c_0 δ _0nK^2 (56) Combining (55) and (56), we get that the following inequality must hold: C1κτnd(ϕ^max+ψ^max)+C2κ2τdm≤c0δλ0nK2.C_1κτ nd ( φ_ + ψ_ )+C_2κ^2τ d m\;≤\;c_0 δ _0nK^2. (57) Let us denote a:=C1κτnd(ϕ^max+ψ^max),b:=C2κ2τdm,R:=c0δλ0nK2.a:=C_1κτ nd ( φ_ + ψ_ ), b:=C_2κ^2τ d m, R:=c_0 δ _0nK^2. Then (57) can be written as a+b≤Ra+b≤ R. A sufficient way to enforce this inequality is to require that each term a and b is at most R/2R/2: a≤R2,b≤R2a\;≤\; R2, b\;≤\; R2 (58) Indeed, if (58) holds, then a+b≤R2+R2=R,a+b\;≤\; R2+ R2=R, so (57) is automatically satisfied. Imposing a≤R/2a≤ R/2 yields: C1κτnd(ϕ^max+ψ^max)≤c02δλ0nK2.C_1κτ nd ( φ_ + ψ_ )\;≤\; c_02 δ _0nK^2. Thus, κ≤κlin2:=c02C1δλ0τn2dK2(ϕ^max+ψ^max).κ\;≤\; _lin_2:= c_02C_1 δ _0τ n^2dK^2 ( φ_ + ψ_ ). (59) Similarly, imposing b≤R/2b≤ R/2 gives: C2κ2τdm≤c02δλ0nK2⇒ C_2κ^2τ d m\;≤\; c_02 δ _0nK^2 5.0pt κ2≤c02C2δλ0τdmnK2 κ^2\;≤\; c_02C_2 δ _0τ d mnK^2 and therefore κ≤κquad2:=c02C2δλ0τdmnK2.κ\;≤\; _quad_2:= c_02C_2 δ _0τ d mnK^2. (60) To ensure (56) holds, it is sufficient that (59) and (60) both hold. Equivalently, κ≤minκlin2,κquad2.κ\;≤\; \ _lin_2, _quad_2\. In big-O notation we may write this as κ=O(δλ0τn2dK2(ϕ^max+ψ^max))andκ=O(δλ0τdmnK2).κ=O\! ( δ _0τ n^2dK^2( φ_ + ψ_ ) ) κ=O\! ( δ _0τ d mnK^2 ). (61) Similarly, for ε1 _1 the general stochastic convergence theorem requires that ε1≤c1δmnK4, _1\;≤\;c_1 δ mnK^4, for some absolute constant c1>0c_1>0. Combining (51) and (C.2) we get that the following must hold: O(κ2τ2mnd(ϕ^max2+ψ^max2+1))+O(κτmndϕ^max2)≤c1δmnK4⇒ O (κ^2τ^2mnd( φ_ ^2+ ψ_ ^2+1) )+O (κτ mn d φ_ ^2 )\;≤\;c_1 δ mnK^4 5.0pt Caκ2τ2nd(ϕ^max2+ψ^max2+1)+Cbκτndϕ^max2≤c1δnK4. C_aκ^2τ^2nd ( φ_ ^2+ ψ_ ^2+1 )+C_bκτ n d φ_ ^2\;≤\;c_1 δnK^4. Similarly, we impose Caκ2τ2nd(1+ϕ^max2+ψ^max2)≤c12δnK4⇒ C_aκ^2τ^2nd (1+ φ_ ^2+ ψ_ ^2 )\;≤\; c_12 δnK^4 5.0pt κ2≤c12Caδτ2n2dK4(ϕ^max2+ψ^max2+1)⇒ κ^2\;≤\; c_12C_a δτ^2n^2dK^4 ( φ_ ^2+ ψ_ ^2+1 ) κ≤κquad1:=c12Ca⋅δτndK2ϕ^max2+ψ^max2+1.κ\;≤\; _quad_1:= c_12C_a· δτ n dK^2 φ_ ^2+ ψ_ ^2+1. (61) and Cbκτndϕ^max2≤c12δnK4⇒ C_bκτ n d φ_ ^2\;≤\; c_12 δnK^4 5.0pt κ≤κlin1:=c12Cbδτn2dK4ϕ^max2.κ\;≤\; _lin_1:= c_12C_b δτ n^2 dK^4 φ_ ^2. (62) Thus, the ε1 _1 requirement (C.2) is guaranteed whenever κ≤minκquad1,κlin1.κ\;≤\; \ _quad_1,\; _lin_1 \. Combining all constraints from ε1 _1 and ε2 _2, we see that a sufficient set of conditions is κ≤minκlin1,κquad1,κlin2,κquad2.κ\;≤\; \ _lin_1, _quad_1, _lin_2, _quad_2 \. Equivalently, in big-O notation, κ=O(minδλ0τn2dK2(ϕ^max+ψ^max),δτn2dK4ϕ^max2,δλ0τdmnK2,δτ2n2dK4(ϕ^max2+ψ^max2+1)).κ=O\! ( \ δ _0τ n^2dK^2( φ_ + ψ_ ),\; δτ n^2 dK^4 φ_ ^2,\; δ _0τ d mnK^2,\; δτ^2n^2dK^4( φ_ ^2+ ψ_ ^2+1) \ ). (63) Instead of carrying this minimum in the theorem statement, we define a slightly more restrictive but cleaner condition that implies all of the above bounds: κ=O(δλ0τ2K2(m1/4d+nd)(ϕ^max+ψ^max)).κ\;=\;O\! ( δ _0τ^2K^2 (m^1/4 d+nd ) ( φ_ + ψ_ ) ). (64) For ε3 _3, we combine (9) and (5.4) which yields that: ε3=O(σmax()ϕmaxm) _3=O ( _ (X) _ m ) We assume that σmax()ϕ^max≤Cλ0/n _ (X) φ_ ≤ C _0/ n for some constant C>0C>0 (assumption (15)), and therefore ε3≤O(λ0mn), _3≤ O ( _0 mn ), (65) which is exactly the form required in Theorem 5.2. Theorem 5.2 includes the factor O(mnλ02⋅ε22+ε1)=(55),(50) O ( mn _0^2· _2^2+ _1 ) eq:eps2_scaling_correct, eq:eps1_before_simplify= So we study the quantity mnλ02⋅ε22 mn _0^2· _2^2. mnλ02⋅ε22 mn _0^2· _2^2 =mnλ02⋅(κ2τ2n2d2(ϕ^max+ψ^max)2+κ4τ2d2m) = mn _0^2· (κ^2τ^2n^2d^2( φ_ + ψ_ )^2+κ^4τ^2d^2m ) ≤2λ02κ2τ2n3d2m(ϕ^max2+ψ^max2)+1λ02κ4τ2d2m2n ≤ 2 _0^2κ^2τ^2n^3d^2m( φ_ ^2+ ψ_ ^2)+ 1 _0^2κ^4τ^2d^2m^2n =O(κτ2mn3d2(ϕ^max2+ψ^max2))+O(κ2τ2m2nd2) = [rgb].75,0,.25 [named]pgfstrokecolorrgb.75,0,.25O (κτ^2mn^3d^2 ( φ_ ^2+ ψ_ ^2 ) )+ [rgb]0,.5,.5 [named]pgfstrokecolorrgb0,.5,.5O (κ^2τ^2m^2nd^2 ) (66) because (a+b)2≤2(a2+b2)(a+b)^2≤ 2(a^2+b^2) and κ4≤κ2≤κ^4≤κ^2≤κ for κ≤1κ≤ 1 Furthermore, ε1 _1 =O(κ2τ2mnd)+O(κ2τ2mnd(ϕ^max2+ψ^max2))+O(κτmndϕ^max2) =O (κ^2τ^2mnd )+O (κ^2τ^2mnd( φ_ ^2+ ψ_ ^2) )+O (κτ mn d φ_ ^2 ) ≤O(κ2τ2m2nd2)+O(κτ2mn3d2(ϕ^max2+ψ^max2))+O(κτmndϕ^max2) ≤ [rgb]0,.5,.5 [named]pgfstrokecolorrgb0,.5,.5O (κ^2τ^2m^2nd^2 )+ [rgb].75,0,.25 [named]pgfstrokecolorrgb.75,0,.25O (κτ^2mn^3d^2( φ_ ^2+ ψ_ ^2) )+ [rgb]1,.5,0 [named]pgfstrokecolorrgb1,.5,0O (κτ mn d φ_ ^2 ) (67) Comment: Tried bounding O(κτmndϕ^max2)O (κτ mn d φ_ ^2 ) in the term O(κτ2mn3d2(ϕ^max2+ψ^max2))O (κτ^2mn^3d^2( φ_ ^2+ ψ_ ^2) ) as ϕ^max2≤ϕ^max2+ψ^max2 φ_ ^2≤ φ_ ^2+ ψ_ ^2 and d≤d2 d≤ d^2 (since in our case it definitely holds that d>1d>1) but it does not necessarily hold that τ≤τ2τ≤τ^2 Therefore, O(mnλ02⋅ε22+ε1)=(67),(66)O(κ2τ2m2nd2)+O(κτ2mn3d2(ϕ^max2+ψ^max2))+O(κτmndϕ^max2) O ( mn _0^2· _2^2+ _1 ) eq:eps1_bigO, eq:eps2_bigO= [rgb]0,.5,.5 [named]pgfstrokecolorrgb0,.5,.5O (κ^2τ^2m^2nd^2 )+ [rgb].75,0,.25 [named]pgfstrokecolorrgb.75,0,.25O (κτ^2mn^3d^2( φ_ ^2+ ψ_ ^2) )+ [rgb]1,.5,0 [named]pgfstrokecolorrgb1,.5,0O (κτ mn d φ_ ^2 ) Thus, the expected loss is bounded by: [ℒ(K)]≤(1−ηλ02)Kℒ(0)+O(κ2τ2m2nd2)+O(κτ2mn3d2(ϕ^max2+ψ^max2))+O(κτmndϕ^max2)E[L(W_K)]≤ (1- η _02 )^KL(W_0)+ [rgb]0,.5,.5 [named]pgfstrokecolorrgb0,.5,.5O (κ^2τ^2m^2nd^2 )+ [rgb].75,0,.25 [named]pgfstrokecolorrgb.75,0,.25O (κτ^2mn^3d^2( φ_ ^2+ ψ_ ^2) )+ [rgb]1,.5,0 [named]pgfstrokecolorrgb1,.5,0O (κτ mn d φ_ ^2 ) C.2.1 Proof of Corollary 5.3 Proof. By Theorem 4.2, we have that |[ℒ()]−ℒ()| |E_C [L_C ( θ ) ]-L ( θ ) | ≤|12∑i=1n((f^(,i)−yi)2−(f(,i)−yi)2)|⏟1 ≤ | 12 _i=1^n ( ( f ( θ,x_i )-y_i )^2- (f (W,x_i )-y_i )^2 ) |_T_1 +κ22m∑i=1n‖∑r=1mar(r⊙i)1(r⊤iκ‖r⊙i‖2)‖22⏟2 + κ^22m _i=1^n \| _r=1^ma_r (w_r x_i ) _1 ( w_r x_iκ \|w_r x_i \|_2 ) \|_2^2_T_2 +mn(κ2R2ψmax2+(κ2R2+κR)ϕmax2) +mn (κ^2R_u^2 _ ^2+ (κ^2R_u^2+κ R_w ) _ ^2 ) Now, we can bound 1T_1 and 2T_2 separately. For 1T_1, we have |12∑i=1n((f^(,i)−yi)2−(f(,i)−yi)2)| | 12 _i=1^n ( ( f ( θ,x_i )-y_i )^2- (f (W,x_i )-y_i )^2 ) | ≤12∑i=1n((f^(,i)−f(,i))2+2|(f^(,i)−f(,i))(f^(,i)−f(,i))|) ≤ 12 _i=1^n ( ( f ( θ,x_i )-f (W,x_i ) )^2+2 | ( f ( θ,x_i )-f (W,x_i ) ) ( f ( θ,x_i )-f (W,x_i ) ) | ) ≤∑i=1n(f^(,i)−f(,i))2+ℒ() ≤ _i=1^n ( f ( θ,x_i )-f (W,x_i ) )^2+L ( θ ) Here, we can bound f^(,i)−f(,i) f (W,x_i )-f (W,x_i ) as |f^(,i)−f(,i)| | f (W,x_i )-f (W,x_i ) | ≤1m∑r=1m|σ^(r,i)−σ(r⊤i)| ≤ 1 m _r=1^m | σ (w_r,x_i )-σ (w_r x_i ) | ≤1m∑r=1m|r⊤i|(r⊤i≥0−1(r⊤iκ‖r⊙i‖2)) ≤ 1 m _r=1^m |w_r x_i | (I \w_r x_i≥ 0 \- _1 ( w_r x_iκ \|w_r x_i \|_2 ) ) ≤1m∑r=1m|r⊤i|exp(−(r⊤i)22κ2‖r⊙i‖2) ≤ 1 m _r=1^m |w_r x_i | (- (w_r x_i )^22κ^2 \|w_r x_i \|_2 ) ≤1m∑r=1mψ(r⊤i2κ‖r⊤i‖2) ≤ 1 m _r=1^mψ ( w_r x_i2κ \|w_r x_i \|_2 ) ≤mψmax ≤ m _ where in the third inequality we used Lemma D.4. Therefore, 1T_1 can be bounded as 1≤ℒ()+mnψmaxT_1≤L ( θ )+mn _ For 2T_2, we an bound it as 2≤2κ2∑i=1n∑r=1m‖r⊙i‖≤2κ2mnR2 _2≤ 2κ^2 _i=1^n _r=1^m \|w_r x_i \|≤ 2κ^2mnR_u^2 Plugging in 1T_1 and 2T_2 gives the desired result. ∎ C.2.2 Proof of Corollary 5.4 Proof. By Theorem 4.9, we have that ‖[∇rℒ()]−∇rℒ()‖2 \|E_C [ _w_rL ( θ ) ]- _w_rL ( θ ) \|_2 =‖r+3κ2m∑r′=1mar′Diag(i)2r′r⊤i≥0;r′⊤i≥0‖2 = \|g_r+ 3κ^2 m _r =1^ma_r Diag (x_i )^2w_r I \w_r x_i≥ 0;w_r x_i≥ 0 \ \|_2 ≤‖r‖2+3κ2m∑r′=1m‖ar′Diag(i)2r′r⊤i≥0;r′⊤i≥0‖2 ≤ \|g_r \|_2+ 3κ^2 m _r =1^m \|a_r Diag (x_i )^2w_r I \w_r x_i≥ 0;w_r x_i≥ 0 \ \|_2 ≤‖r‖2+3κ2m∑r′=1m‖i‖∞2‖r‖2 ≤ \|g_r \|_2+ 3κ^2 m _r =1^m \|x_i \|_∞^2 \|w_r \|_2 ≤(σmax()mℒ()12+6nκ2B2R+5nκRd)ϕmax ≤ ( _ (X ) mL ( θ ) 12+6nκ^2B_x^2R_w+5nκ R_u d ) _ +6nκRψmax+3κ2mB2R +6nκ R_u _ +3κ^2 mB_x^2R_w Theorem 4.9 states that the norm ‖r‖2 \|g_r \|_2 satisfies ‖r‖2≤(6nκ2B2R+5nκRd)ϕmax+σmax()mϕmaxℒ()12+6nκRψmax \|g_r \|_2≤ (6nκ^2B_x^2R_w+5nκ R_u d ) _ + _ (X ) m _ L ( θ ) 12+6nκ R_u _ Moreover, by Assumption 3.1, we have that ‖∞≤B \|x \|_∞≤ B_x. Therefore, we obtain that ‖[∇rℒ()]−∇rℒ()‖2 \|E_C [ _w_rL ( θ ) ]- _w_rL ( θ ) \|_2 ≤(σmax()mℒ()12+6nκ2B2R+5nκRd)ϕmax ≤ ( _ (X ) mL ( θ ) 12+6nκ^2B_x^2R_w+5nκ R_u d ) _ +6nκRψmax+3κ2mB2R +6nκ R_u _ +3κ^2 mB_x^2R_w ∎ C.2.3 Proof of Lemma D.21 Proof. By the form of ∇rℒ() _w_rL_C (W ) in (2), we have: ‖∇rℒ()‖2 \| _w_rL_C (W ) \|_2 =‖arm∑i=1n(f(,i⊙i)−yi)(i⊙i)r⊤(i⊙i)≥0⏟≤1‖2 = \| a_r m _i=1^n (f (W,x_i c_i )-y_i ) (x_i c_i ) I \w_r (x_i c_i )≥ 0 \_≤ 1 \|_2 ≤|ar|m∑i=1n|f(,i⊙i)−yi|⋅‖i⊙i‖2 ≤ |a_r | m _i=1^n |f (W,x_i c_i )-y_i |· \|x_i c_i \|_2 ≤nm(∑i=1n(f(,i⊙i)−yi)2)1/2⋅‖i‖∞‖i‖2using D.20 ≤ n m ( _i=1^n(f (W,x_i c_i )-y_i)^2 )^1/2·\|c_i\|_∞\|x_i\|_2 5.0ptusing lem:CS ≤Cnm(2ℒ())1/2using‖i‖∞≤Cand‖i‖2≤1 ≤ C n m (2L_C (W ) )^1/2 5.0ptusing 5.0pt\|c_i\|_∞≤ C 5.0ptand 5.0pt \|x_i \|_2≤ 1 ≤C2nmℒ()1/2 ≤ C 2 n mL_C (W )^1/2 where, in the first inequality, we use the fact that the indicator function is upper-bounded by 1 and in the second inequality, we use the fact that ar=±1a_r=± 1. and so, ‖∇rℒ()‖22≤2C2⋅nmℒ() \| _w_rL_C (W ) \|^2_2≤ 2C^2· nmL_C (W ) ∎ Appendix D Auxiliary Results D.1 Gaussian Random Variables D.1.1 Conditional Expectation and Covariance Lemma D.1. Let ∼(,κ2)c ( μ,κ^2I ), and let z=⊤,z′=⊤z=c u,z =c v. Then we have that [∣z]=+‖22(z−⊤);[∣z,z′]=+1(z−⊤)+2(z′−⊤)E_c [c z ]= μ+ u \|u \|_2^2 (z- μ u ); _c [c z,z ]= μ+s_1 (z- μ u )+s_2 (z - μ v ) where the vectos 1,2s_1,s_2 are defined as 1=‖22−⊤⋅‖22‖22−(⊤)2;2=‖22−⊤⋅‖22‖22−(⊤)2s_1= \|v \|_2^2u-u v·v \|u \|_2^2 \|v \|_2^2- (u v )^2; s_2= \|u \|_2^2v-u v·u \|u \|_2^2 \|v \|_2^2- (u v )^2 Proof. By the formula of conditional expectation, we have [∣z]=[]+Cov(,z)Var(z)−1(z−[z])E_c [c z ]=E[c]+Cov (c,z )Var (z )^-1 (z-E[z] ) By the definition of c, we have []=E[c]= μ. Moreover, since z=⊤z=c u, by Lemma D.13, we have [z]=⊤E[z]= μ u and Var(z)=κ2‖22Var (z )=κ^2 \|u \|_2^2. Therefore, the covariance between c and z is given by Cov(,z)=[(−)(z−⟨,⟩)]=[z]−⟨,⟩Cov (c,z )=E[ (c- μ ) (z- μ,v )]=E [zc ]- μ μ,v where [z]j=∑j′=1d[cjcj′uj′]=∑j′=1d(μjμj′+j=j′κ2)uj′=μj⊤+κ2ujE[zc]_j= _j =1^dE[c_jc_j u_j ]= _j =1^d ( _j _j +I\j=j \κ^2 )u_j = _j μ u+κ^2u_j This gives [z]=⊤⋅+κ2E[zc]= μ u· μ+κ^2u Therefore [∣z]=+‖22(z−⊤)E_c [c z ]= μ+ u \|u \|_2^2 (z- μ u ) Similarly, since z′=⊤z =c v. Then [∣z,z′]=[]+Cov(,[z,z′])Cov(z,z′)−1([z,z′]⊤−[[z,z′]]⊤)E_c [c z,z ]=E[c]+Cov (c,[z,z ] )Cov (z,z )^-1 ([z,z ] -E[[z,z ]] ) where Cov(,[z,z′])=κ2[];Cov(z,z′)=κ2[‖22⊤⊤‖22]Cov (c,[z,z ] )=κ^2 bmatrixu\\ v bmatrix; (z,z )=κ^2 bmatrix \|u \|_2^2&u v\\ u v& \|v \|_2^2 bmatrix This gives [∣z,z′] _c [c z,z ] =+[][‖22⊤⊤‖22]−1[z−⊤z′−⊤] = μ+ bmatrixu\\ v bmatrix bmatrix \|u \|_2^2&u v\\ u v& \|v \|_2^2 bmatrix^-1 bmatrixz- μ u\\ z - μ v bmatrix =+1‖22‖22−(⊤)2[][‖22−⊤−⊤‖22]−1[z−⊤z′−⊤] = μ+ 1 \|u \|_2^2 \|v \|_2^2- (u v )^2 bmatrixu\\ v bmatrix bmatrix \|v \|_2^2&-u v\\ -u v& \|u \|_2^2 bmatrix^-1 bmatrixz- μ u\\ z - μ v bmatrix =+(‖22−⊤⋅)(z−⊤)+(‖22−⊤⋅)(z′−⊤)‖22‖22−(⊤)2 = μ+ ( \|v \|_2^2u-u v·v ) (z- μ u )+ ( \|u \|_2^2v-u v·u ) (z - μ v ) \|u \|_2^2 \|v \|_2^2- (u v )^2 =+1(z−⊤)+2(z′−⊤) = μ+s_1 (z- μ u )+s_2 (z - μ v ) where the vectos 1,2s_1,s_2 are defined as 1=‖22−⊤⋅‖22‖22−(⊤)2;2=‖22−⊤⋅‖22‖22−(⊤)2s_1= \|v \|_2^2u-u v·v \|u \|_2^2 \|v \|_2^2- (u v )^2; s_2= \|u \|_2^2v-u v·u \|u \|_2^2 \|v \|_2^2- (u v )^2 ∎ Lemma D.2. Let ∼(,κ2)c ( μ,κ^2I ), and let z=⊤,z′=⊤z=c u,z =c v. Then we have Cov(∣z,z′)=κ2−κ2(⊤−⊤)2‖22‖22−(⊤)2Cov (c z,z )=κ^2I- κ^2 (uv -vu )^2 \|u \|_2^2 \|v \|_2^2- (u v )^2 Proof. The conditional covariance of Gaussian random variables is given by Cov(∣z,z′)=Cov()−Cov(,[z,z′])Cov(z,z′)−1Cov(,[z,z′])⊤Cov (c z,z )=Cov (c )-Cov (c,[z,z ] )Cov (z,z )^-1Cov (c,[z,z ] ) Recall that in the previos lemma we have computed Cov(,[z,z′])=κ2[];Cov(z,z′)=κ2[‖22⊤⊤‖22] (c,[z,z ] )=κ^2 bmatrixu\\ v bmatrix; (z,z )=κ^2 bmatrix \|u \|_2^2&u v\\ u v& \|v \|_2^2 bmatrix Thus, we have Cov(∣z,z′) (c z,z ) =κ2−κ2[][‖22⊤⊤‖22]−1[] =κ^2I-κ^2 bmatrixu\\ v bmatrix bmatrix \|u \|_2^2&u v\\ u v& \|v \|_2^2 bmatrix^-1 bmatrixu&v bmatrix =κ2−κ2‖22‖22−(⊤)2[][‖22−⊤−⊤‖22][] =κ^2I- κ^2 \|u \|_2^2 \|v \|_2^2- (u v )^2 bmatrixu\\ v bmatrix bmatrix \|v \|_2^2&-u v\\ -u v& \|u \|_2^2 bmatrix bmatrixu&v bmatrix =κ2−κ2(⊤−⊤)2‖22‖22−(⊤)2 =κ^2I- κ^2 (uv -vu )^2 \|u \|_2^2 \|v \|_2^2- (u v )^2 ∎ D.1.2 Approximation of CDF In approximation of the Gaussian CDF, we use the following property Lemma D.3 (chu1955Bounds). Let x≥0x≥ 0 be given, then the following inequality holds 12(1−exp(−x22))12≤12π∫0xexp(−t22)t≤12(1−exp(−2x2π))12 12 (1- (- x^22 ) ) 12≤ 1 2π _0^x (- t^22 )dt≤ 12 (1- (- 2x^2π ) ) 12 To start, we analyze the CDF of a single variable Gaussian random variable Lemma D.4. Let 1 _1 be the CDF of a standard Gaussian random variable. Then we have |1(α)−α≥0|≤exp(−α22) | _1 (α )-I \α≥ 0 \ |≤ (- α^22 ) Proof. Let ν(x)=12π∫0xexp(−t22)tν(x)= 1 2π _0^x (- t^22 )dt for x≥0x≥ 0. We study the case of α≥0α≥ 0 and α<0α<0 separately. For α≥0α≥ 0, we have 1(α) _1 (α ) =12π∫−∞αexp(−t22)t = 1 2π _-∞^α (- t^22 )dt =12+ν(α) = 12+ν(α) ≥12+12(1−exp(−α22))12 ≥ 12+ 12 (1- (- α^22 ) ) 12 ≥1−exp(−α22) ≥ 1- (- α^22 ) =α≥0−exp(−α22) =I \α≥ 0 \- (- α^22 ) Since 1≤1=α≥0 _1≤ 1=I \α≥ 0 \ when α≥0α≥ 0, we must have that, for α≥0α≥ 0, |1(α)−α≥0|≤exp(−α22) | _1 (α )-I \α≥ 0 \ |≤ (- α^22 ). Similarly, for α<0α<0, we have 1(α) _1 (α ) =12π∫−∞αexp(−t22)t = 1 2π _-∞^α (- t^22 )dt =12−ν(α) = 12-ν(α) ≤12−12(1−exp(−α22))12 ≤ 12- 12 (1- (- α^22 ) ) 12 ≤exp(−α22) ≤ (- α^22 ) =α≥0+exp(−α22) =I \α≥ 0 \+ (- α^22 ) Since 1≥0=α≥0 _1≥ 0=I \α≥ 0 \ when α<0α<0, we must have that, for α<0α<0, |1(α)−α≥0|≤exp(−α22) | _1 (α )-I \α≥ 0 \ |≤ (- α^22 ). ∎ Lemma D.5. Let 2(α1,α2,ρ) _2 ( _1, _2,ρ ) be the joint CDF of two standard Gaussian random variables with covariance ρ at α1,α2 _1, _2. Then we have |2(α1,α2,ρ)−α1≥0;α2≥0|≤2exp(−minα1,α222) | _2 ( _1, _2,ρ )-I \ _1≥ 0; _2≥ 0 \ |≤ 2 (- \ _1, _2 \^22 ) Proof. Since z1,z2z_1,z_2 are standard Gaussian random variables with covariance ρ, we have z1∣z2=ζ∼(ρζ,1−ρ2)z_1 z_2=ζ (ρζ,1-ρ^2 ). According to the definition of CDF, 2(α1,α2,ρ) _2 ( _1, _2,ρ ) =∫−∞α2∫−∞α1fz1,z2(ζ1,ζ2,ρ)ζ1ζ2 = _-∞ _2 _-∞ _1f_z_1,z_2 ( _1, _2,ρ )d _1d _2 =∫−∞α2∫−∞α1fz1∣z2=ζ2(ζ1)ζ1fz2(ζ2)ζ2 = _-∞ _2 _-∞ _1f_z_1 z_2= _2 ( _1 )d _1f_z_2 ( _2 )d _2 Focusing on the inner integral, we substitute ζ′=ζ1−ρζ21−ρ2ζ = _1-ρ _2 1-ρ^2. Then we have dζ1=1−ρ2dζ′d _1= 1-ρ^2dζ . Thus ∫−∞α1fz1∣z2=ζ2(ζ1)ζ1 _-∞ _1f_z_1 z_2= _2 ( _1 )d _1 =12π(1−ρ2)∫−∞α1exp(−(ζ1−ρζ2)22(1−ρ2))ζ1 = 1 2π(1-ρ^2) _-∞ _1 (- ( _1-ρ _2 )^22(1-ρ^2) )d _1 =12π∫−∞α1−ρζ21−ρ2exp(−ζ′22)ζ′ = 1 2π _-∞ _1-ρ _2 1-ρ^2 (- ζ 22 )dζ =Φ1(α1−ρζ21−ρ2) = _1 ( _1-ρ _2 1-ρ^2 ) =α1−ρζ21−ρ2≥0+ε(α1−ρζ21−ρ2) =I \ _1-ρ _2 1-ρ^2≥ 0 \+ ( _1-ρ _2 1-ρ^2 ) =α1≥ρζ2+ε(α1−ρζ21−ρ2) =I \ _1≥ρ _2 \+ ( _1-ρ _2 1-ρ^2 ) Where |ε(α1−ρζ21−ρ2)|≤exp(−(α1−ρζ2)22(1−ρ2)) | ( _1-ρ _2 1-ρ^2 ) |≤ (- ( _1-ρ _2 )^22(1-ρ^2) ). Therefore 2(α1,α2,ρ) _2 ( _1, _2,ρ ) =∫−∞α2(α1≥ρζ2+ε(α1−ρζ21−ρ2))fz2(ζ2)ζ2 = _-∞ _2 (I \ _1≥ρ _2 \+ ( _1-ρ _2 1-ρ^2 ) )f_z_2 ( _2 )d _2 =∫−∞α2α1≥ρζ2fz2(ζ2)ζ2⏟ℐ1+∫−∞α2ε(α1−ρζ21−ρ2)fz2(ζ2)ζ2⏟ℐ2 = _-∞ _2I \ _1≥ρ _2 \f_z_2 ( _2 )d _2_I_1+ _-∞ _2 ( _1-ρ _2 1-ρ^2 )f_z_2 ( _2 )d _2_I_2 For ℐ1I_1, we have ℐ1 _1 =∫−∞α2α1≥ρζ2fz2(ζ2)ζ2 = _-∞ _2I \ _1≥ρ _2 \f_z_2 ( _2 )d _2 =∫−∞minα1ρ,α2fz2(ζ2)ζ2 = _-∞ \ _1ρ, _2 \f_z_2 ( _2 )d _2 =Φ1(minα1ρ,α2) = _1 ( \ _1ρ, _2 \ ) =minα1ρ,α2≥0+ε(minα1ρ,α2) =I \ \ _1ρ, _2 \≥ 0 \+ ( \ _1ρ, _2 \ ) Notice that, when α1≥0 _1≥ 0 and α2≥0 _2≥ 0, we must have minα1ρ,α2≥0 \ _1ρ, _2 \≥ 0. Conversely, when minα1ρ,α2≥0 \ _1ρ, _2 \≥ 0, we must have that α1≥0 _1≥ 0 and α2≥0 _2≥ 0. Therefore minα1ρ,α2≥0=α1≥0;α2≥0I \ \ _1ρ, _2 \≥ 0 \=I \ _1≥ 0; _2≥ 0 \. Thus, we have 2(α1,α2,ρ)=α1≥0;α2≥0+ε(minα1ρ,α2)+ℐ2 _2 ( _1, _2,ρ )=I \ _1≥ 0; _2≥ 0 \+ ( \ _1ρ, _2 \ )+I_2 For ℐ2I_2, we have |ℐ2| |I_2 | ≤∫−∞α2|ε(α1−ρζ21−ρ2)|fz2(ζ2)ζ2 ≤ _-∞ _2 | ( _1-ρ _2 1-ρ^2 ) |f_z_2 ( _2 )d _2 ≤∫−∞α2exp(−(α1−ρζ2)22(1−ρ2))fz2(ζ2)ζ2 ≤ _-∞ _2 (- ( _1-ρ _2 )^22(1-ρ^2) )f_z_2 ( _2 )d _2 =12π∫−∞α2exp(−(α1−ρζ2)22(1−ρ2)−ζ222)ζ2 = 1 2π _-∞ _2 (- ( _1-ρ _2 )^22(1-ρ^2)- _2^22 )d _2 =12πexp(−α122)∫−∞α2exp(−(ζ2−ρα1)22(1−ρ2))ζ2 = 1 2π (- _1^22 ) _-∞ _2 (- ( _2-ρ _1 )^22 (1-ρ^2 ) )d _2 ≤exp(−α122) ≤ (- _1^22 ) Moreover, since |ε(minα1ρ,α2)|≤exp(−12minα1ρ,α22) | ( \ _1ρ, _2 \ ) |≤ (- 12 \ _1ρ, _2 \^2 ), we have that |2(α1,α2,ρ)−α1≥0;α2≥0| | _2 ( _1, _2,ρ )-I \ _1≥ 0; _2≥ 0 \ | ≤exp(−12minα1ρ,α22)+exp(−α122) ≤ (- 12 \ _1ρ, _2 \^2 )+ (- _1^22 ) ≤exp(−minα1,α222)+exp(−α122) ≤ (- \ _1, _2 \^22 )+ (- _1^22 ) Exchanging α1 _1 and α2 _2, we have |2(α1,α2,ρ)−α1≥0;α2≥0| | _2 ( _1, _2,ρ )-I \ _1≥ 0; _2≥ 0 \ | ≤exp(−12minα1ρ,α22)+exp(−α122) ≤ (- 12 \ _1ρ, _2 \^2 )+ (- _1^22 ) ≤exp(−minα1,α222)+exp(−α222) ≤ (- \ _1, _2 \^22 )+ (- _2^22 ) Thus, we have |2(α1,α2,ρ)−α1≥0;α2≥0|≤2exp(−minα1,α222) | _2 ( _1, _2,ρ )-I \ _1≥ 0; _2≥ 0 \ |≤ 2 (- \ _1, _2 \^22 ) ∎ D.1.3 Uni-variate Coupled Expectation Lemma D.6. let z1,z2∼(0,1)z_1,z_2 (0,1 ) with Cov(z1,z2)=ρCov (z_1,z_2 )=ρ. Let a,b∈ℝa,b . Define T1=exp(−a22)1(ρa−b1−ρ2);T2=exp(−b22)1(ρb−a1−ρ2)T_1= (- a^22 ) _1 ( ρ a-b 1-ρ^2 ); T_2= (- b^22 ) _1 ( ρ b-a 1-ρ^2 ) Then we have that [z1z1≥a;z2≥b]=12π(T1+ρT2);[z2z1≥a;z2≥b]=12π(T2+ρT1)E [z_1I \z_1≥ a;z_2≥ b \ ]= 1 2π (T_1+ρ T_2 ); [z_2I \z_1≥ a;z_2≥ b \ ]= 1 2π (T_2+ρ T_1 ) Proof. Writing the expectation in integral form, we have that [z1z1≥a;z2≥b]=∫b∞∫a∞z1f(z1,z2)z1z2=∫b∞(∫a∞z1f(z1∣z2)z1)f(z2)z2E [z_1I \z_1≥ a;z_2≥ b \ ]= _b^∞ _a^∞z_1f (z_1,z_2 )dz_1dz_2= _b^∞ ( _a^∞z_1f (z_1 z_2 )dz_1 )f (z_2 )dz_2 Since z1,z2∼(0,1)z_1,z_2 (0,1 ) with covariance ρ, we have that z1∣z2∼(ρz2,1−ρ2)z_1 z_2 (ρ z_2,1-ρ^2 ). Therefore, let ρ′=1−ρ2ρ =1-ρ^2, we have f(z1∣z2)=12πρ′exp(−(z1−ρz2)22ρ′)f (z_1 z_2 )= 1 2πρ (- (z_1-ρ z_2 )^22ρ ) Thus, by Lemma D.14, we have that ∫a∞z1f(z1∣z2)z1 _a^∞z_1f (z_1 z_2 )dz_1 =12πρ′∫a∞z1exp(−(z1−ρz2)22ρ′)z1 = 1 2πρ _a^∞z_1 (- (z_1-ρ z_2 )^22ρ )dz_1 =12πρ′(ρ′exp(−(a−ρz2)22ρ′)+ρz22πρ′1(ρz2−aρ′)) = 1 2πρ (ρ (- (a-ρ z_2 )^22ρ )+ρ z_2 2πρ _1 ( ρ z_2-a ρ ) ) =ρ′2πexp(−(a−ρz2)22ρ′)+ρz21(ρz2−aρ′) = ρ 2π (- (a-ρ z_2 )^22ρ )+ρ z_2 _1 ( ρ z_2-a ρ ) where we set κ=ρ′κ= ρ and μ=ρz2μ=ρ z_2 in Lemma D.14. Plugging into the original integral gives [z1z1≥a;z2≥b] [z_1I \z_1≥ a;z_2≥ b \ ] =∫b∞(ρ′2πexp(−(a−ρz2)22ρ′)+ρz21(ρz2−aρ′))f(z2)z2 = _b^∞ ( ρ 2π (- (a-ρ z_2 )^22ρ )+ρ z_2 _1 ( ρ z_2-a ρ ) )f (z_2 )dz_2 =ρ′2π∫b∞exp(−(a−ρz2)22ρ′−z222)z2+ρ∫b∞z21(ρz2−aρ′)f(z2)z2 = ρ 2π _b^∞ (- (a-ρ z_2 )^22ρ - z_2^22 )dz_2+ρ _b^∞z_2 _1 ( ρ z_2-a ρ )f (z_2 )dz_2 Notice that exp(−(a−ρz2)22ρ′−z222)=exp(−a2−2ρaz2+z222ρ′)=exp(−(z2−ρa)22ρ′)⋅exp(−a22) (- (a-ρ z_2 )^22ρ - z_2^22 )= (- a^2-2ρ az_2+z_2^22ρ )= (- (z_2-ρ a )^22ρ )· (- a^22 ) From a previous result, we have an identity for the conditional probability, which expresses the CDF term as an integral: 1(ρz2−a1−ρ2)=∫a∞f(z1∣z2)z1 _1 ( ρ z_2-a 1-ρ^2 )= _a^∞f(z_1 z_2)dz_1 (68) and so: ρ∫b∞z21(ρz2−a1−ρ2)f(z2)z2=ρ∫b∞z2∫a∞f(z1∣z2)z1f(z2)z2 ρ _b^∞z_2 _1 ( ρ z_2-a 1-ρ^2 )f(z_2)dz_2=ρ _b^∞z_2 _a^∞f(z_1 z_2)dz_1f(z_2)dz_2 Moreover, by applying Lemma D.16, we have [z1z1≥a;z2≥b] [z_1I \z_1≥ a;z_2≥ b \ ] =ρ′2πexp(−a22)∫b∞exp(−(z2−ρa)22ρ′)z2+ρ∫b∞z2∫a∞f(z1∣z2)z1f(z2)z2 = ρ 2π (- a^22 ) _b^∞ (- (z_2-ρ a )^22ρ )dz_2+ρ _b^∞z_2 _a^∞f(z_1 z_2)dz_1f(z_2)dz_2 =ρ′2πexp(−a22)∫b∞f(z2∣z1=a)z2+ρ∫b∞∫a∞z2f(z1∣z2)f(z2)⏟f(z1,z2)z1z2 = ρ 2π (- a^22 ) _b^∞f (z_2 z_1=a )dz_2+ρ _b^∞ _a^∞z_2 f(z_1 z_2)f(z_2)_f(z_1,z_2)dz_1dz_2 =ρ′2πexp(−a22)1(ρa−b1−ρ2)+ρ[z2z1≥a;z2≥b] = ρ 2π (- a^22 ) _1 ( ρ a-b 1-ρ^2 )+ [z_2I \z_1≥ a;z_2≥ b \ ] Therefore, we can conclude that [z1z1≥a;z2≥b]−ρ[z2z1≥a;z2≥b]=ρ′2πexp(−a22)1(ρa−b1−ρ2)=ρ′2πT1E [z_1I \z_1≥ a;z_2≥ b \ ]- [z_2I \z_1≥ a;z_2≥ b \ ]= ρ 2π (- a^22 ) _1 ( ρ a-b 1-ρ^2 )= ρ 2πT_1 (69) Switching z1,z2z_1,z_2 and a,ba,b gives [z2z1≥a;z2≥b]−ρ[z1z1≥a;z2≥b]=ρ′2πexp(−b22)1(ρb−a1−ρ2)=ρ′2πT2E [z_2I \z_1≥ a;z_2≥ b \ ]- [z_1I \z_1≥ a;z_2≥ b \ ]= ρ 2π (- b^22 ) _1 ( ρ b-a 1-ρ^2 )= ρ 2πT_2 (70) Solving for [z1z1≥a;z2≥b]E [z_1I \z_1≥ a;z_2≥ b \ ] and [z2z1≥a;z2≥b]E [z_2I \z_1≥ a;z_2≥ b \ ] from (69) and (70) gives [z1z1≥a;z2≥b]=12π(T1+ρT2);[z2z1≥a;z2≥b]=12π(T2+ρT1)E [z_1I \z_1≥ a;z_2≥ b \ ]= 1 2π (T_1+ρ T_2 ); [z_2I \z_1≥ a;z_2≥ b \ ]= 1 2π (T_2+ρ T_1 ) ∎ Lemma D.7. Let z1,z2∼(0,1)z_1,z_2 (0,1 ) with Cov(z1,z2)=ρCov (z_1,z_2 )=ρ. Let a,b∈ℝa,b . Define T1=exp(−a22)1(ρa−b1−ρ2);T2=exp(−b22)1(ρb−a1−ρ2)T_1= (- a^22 ) _1 ( ρ a-b 1-ρ^2 ); T_2= (- b^22 ) _1 ( ρ b-a 1-ρ^2 ) Then we have that [z12z1≥a;z2≥b] [z_1^2I \z_1≥ a;z_2≥ b \ ] =ρ1−ρ22πexp(−a2−2ρab+b22(1−ρ2))+2(−a,−b,ρ)+12π(aT1+ρ2bT2) = ρ 1-ρ^22π (- a^2-2ρ ab+b^22 (1-ρ^2 ) )+ _2 (-a,-b,ρ )+ 1 2π (aT_1+ρ^2bT_2 ) [z22z1≥a;z2≥b] [z_2^2I \z_1≥ a;z_2≥ b \ ] =ρ1−ρ22πexp(−a2−2ρab+b22(1−ρ2))+2(−a,−b,ρ)+12π(bT2+ρ2aT1) = ρ 1-ρ^22π (- a^2-2ρ ab+b^22 (1-ρ^2 ) )+ _2 (-a,-b,ρ )+ 1 2π (bT_2+ρ^2aT_1 ) Proof. Again, we write the expectation in the integral form to get that [z12z1≥a;z2≥b]=∫b∞∫a∞z12f(z1,z2)z1z2=∫b∞(∫a∞z12f(z1∣z2)z1)f(z2)z2E [z_1^2I \z_1≥ a;z_2≥ b \ ]= _b^∞ _a^∞z_1^2f (z_1,z_2 )dz_1dz_2= _b^∞ ( _a^∞z_1^2f (z_1 z_2 )dz_1 )f (z_2 )dz_2 Since z1,z2∼(0,1)z_1,z_2 (0,1 ) with Cov(z1,z2)=ρCov (z_1,z_2 )=ρ, we have that z1∣z2∼(ρz2,1−ρ2)z_1 z_2 (ρ z_2,1-ρ^2 ). Define ρ′=1−ρ2ρ =1-ρ^2. By Lemma D.15, we have that ∫a∞z12f(z1∣z2)z1 _a^∞z_1^2f (z_1 z_2 )dz_1 =12πρ′∫a∞z12exp(−(z1−ρz2)22ρ′)z1 = 1 2πρ _a^∞z_1^2 (- (z_1-ρ z_2 )^22ρ )dz_1 =12πρ′(ρ′(a+ρz2)exp(−(a−ρz2)22ρ′)+2πρ′(ρ′+ρ2z22)1(ρz2−aρ′)) = 1 2πρ (ρ (a+ρ z_2 ) (- (a-ρ z_2 )^22ρ )+ 2πρ (ρ +ρ^2z_2^2 ) _1 ( ρ z_2-a ρ ) ) =ρ2z221(ρz2−aρ′)+ρz2ρ′2πexp(−(a−ρz2)22ρ′) =ρ^2z_2^2 _1 ( ρ z_2-a ρ )+ρ z_2 ρ 2π (- (a-ρ z_2 )^22ρ ) +aρ′2πexp(−(a−ρz2)22ρ′)+ρ′1(ρz2−aρ′) +a ρ 2π (- (a-ρ z_2 )^22ρ )+ρ _1 ( ρ z_2-a ρ ) where we set κ=ρ′κ= ρ and μ=ρz2μ=ρ z_2 in Lemma D.15. Therefore, we have [z12z1≥a;z2≥b] [z_1^2I \z_1≥ a;z_2≥ b \ ] =ρ2∫b∞z221(ρz2−aρ′)f(z2)z2 =ρ^2 _b^∞z_2^2 _1 ( ρ z_2-a ρ )f (z_2 )dz_2 +ρρ′2π∫b∞z2exp(−(a−ρz2)22ρ′)f(z2)z2 +ρ ρ 2π _b^∞z_2 (- (a-ρ z_2 )^22ρ )f (z_2 )dz_2 +aρ′2π∫b∞exp(−(a−ρz2)22ρ′)f(z2)z2 +a ρ 2π _b^∞ (- (a-ρ z_2 )^22ρ )f (z_2 )dz_2 +ρ′∫b∞1(ρz2−aρ′)f(z2)z2 +ρ _b^∞ _1 ( ρ z_2-a ρ )f (z_2 )dz_2 To start, by Lemma D.16, we have 1(ρz2−aρ′)f(z2)=∫a∞f(z1∣z2)f(z2)z1=∫a∞f(z1,z2)z1 _1 ( ρ z_2-a ρ )f (z_2 )= _a^∞f (z_1 z_2 )f (z_2 )dz_1= _a^∞f (z_1,z_2 )dz_1 Therefore, for the first term, we have ∫b∞z221(ρz2−aρ′)f(z2)z2=∫b∞∫a∞z22f(z1,z2)z2=[z22z1≥a;z2≥b] _b^∞z_2^2 _1 ( ρ z_2-a ρ )f (z_2 )dz_2= _b^∞ _a^∞z_2^2f (z_1,z_2 )dz_2=E [z_2^2I \z_1≥ a;z_2≥ b \ ] For the last term, we have ∫b∞1(ρz2−aρ′)f(z2)z2=∫b∞∫a∞f(z1,z2)z1z2=2(−a,−b,ρ) _b^∞ _1 ( ρ z_2-a ρ )f (z_2 )dz_2= _b^∞ _a^∞f (z_1,z_2 )dz_1dz_2= _2 (-a,-b,ρ ) Next, we notice that exp(−(a−ρz2)22ρ′)f(z2)=12πexp(−a2−2ρaz2+z222ρ′)=12πexp(−a22)exp(−(z2−ρa)22ρ′) (- (a-ρ z_2 )^22ρ )f (z_2 )= 1 2π (- a^2-2ρ az_2+z_2^22ρ )= 1 2π (- a^22 ) (- (z_2-ρ a )^22ρ ) Therefore, for the second term, we apply Lemma D.14 to get that ∫b∞z2exp(−(a−ρz2)22ρ′)f(z2)z2 _b^∞z_2 (- (a-ρ z_2 )^22ρ )f (z_2 )dz_2 =12πexp(−a22)∫b∞z2exp(−(z2−ρa)22ρ′)z2 = 1 2π (- a^22 ) _b^∞z_2 (- (z_2-ρ a )^22ρ )dz_2 =12πexp(−a22)(ρ′exp(−(ρa−b)22ρ′)+2πρ′ρa1(ρa−bρ′)) = 1 2π (- a^22 ) (ρ (- (ρ a-b )^22ρ )+ 2πρ ρ a _1 ( ρ a-b ρ ) ) =ρ′2πexp(−a2−2ρab+b22ρ′)+ρaρ′exp(−a22)1(ρa−bρ′) = ρ 2π (- a^2-2ρ ab+b^22ρ )+ρ a ρ (- a^22 ) _1 ( ρ a-b ρ ) where we set κ=ρ′κ= ρ and μ=ρaμ=ρ a in Lemma D.14. For the third term, we have ∫b∞exp(−(a−ρz2)22ρ′)f(z2)z2 _b^∞ (- (a-ρ z_2 )^22ρ )f (z_2 )dz_2 =12πexp(−a22)∫b∞exp(−(z2−ρa)22ρ′)z2 = 1 2π (- a^22 ) _b^∞ (- (z_2-ρ a )^22ρ )dz_2 =ρ′exp(−a22)∫b∞f(z2∣z1=a)z2 = ρ (- a^22 ) _b^∞f (z_2 z_1=a )dz_2 =ρ′exp(−a22)1(ρa−bρ′) = ρ (- a^22 ) _1 ( ρ a-b ρ ) Combining all four terms gives [z12z1≥a;z2≥b] [z_1^2I \z_1≥ a;z_2≥ b \ ] =ρρ′2π(ρ′2πexp(−a2−2ρab+b22ρ′)+ρaρ′exp(−a22)1(ρa−bρ′)) =ρ ρ 2π ( ρ 2π (- a^2-2ρ ab+b^22ρ )+ρ a ρ (- a^22 ) _1 ( ρ a-b ρ ) ) +aρ′2π⋅ρ′exp(−a22)1(ρa−bρ′) +a ρ 2π· ρ (- a^22 ) _1 ( ρ a-b ρ ) +ρ2[z22z1≥a;z2≥b]+ρ′2(−a,−b,ρ) +ρ^2E [z_2^2I \z_1≥ a;z_2≥ b \ ]+ρ _2 (-a,-b,ρ ) =ρ′32ρ2πexp(−a2−2ρab+b22ρ′)+ρ′a2π(ρ2+1)exp(−a22)1(ρa−bρ′) = ρ 32ρ2π (- a^2-2ρ ab+b^22ρ )+ ρ a 2π (ρ^2+1 ) (- a^22 ) _1 ( ρ a-b ρ ) +ρ2[z22z1≥a;z2≥b]+ρ′2(−a,−b,ρ) +ρ^2E [z_2^2I \z_1≥ a;z_2≥ b \ ]+ρ _2 (-a,-b,ρ ) Therefore, we can conclude that [z12z1≥a;z2≥b]−ρ2[z22z1≥a;z2≥b] [z_1^2I \z_1≥ a;z_2≥ b \ ]-ρ^2E [z_2^2I \z_1≥ a;z_2≥ b \ ] =ρ′32ρ2πexp(−a2−2ρab+b22ρ′)+ρ′a2π(ρ2+1)T1+ρ′2(−a,−b,ρ) = ρ 32ρ2π (- a^2-2ρ ab+b^22ρ )+ ρ a 2π (ρ^2+1 )T_1+ρ _2 (-a,-b,ρ ) [z22z1≥a;z2≥b]−ρ2[z12z1≥a;z2≥b] [z_2^2I \z_1≥ a;z_2≥ b \ ]-ρ^2E [z_1^2I \z_1≥ a;z_2≥ b \ ] =ρ′32ρ2πexp(−a2−2ρab+b22ρ′)+ρ′b2π(ρ2+1)T2+ρ′2(−a,−b,ρ) = ρ 32ρ2π (- a^2-2ρ ab+b^22ρ )+ ρ b 2π (ρ^2+1 )T_2+ρ _2 (-a,-b,ρ ) Solving for [z12z1≥a;z2≥b]E [z_1^2I \z_1≥ a;z_2≥ b \ ] and [z22z1≥a;z2≥b]E [z_2^2I \z_1≥ a;z_2≥ b \ ] gives [z12z1≥a;z2≥b] [z_1^2I \z_1≥ a;z_2≥ b \ ] =ρρ′2πexp(−a2−2ρab+b22ρ′)+2(−a,−b,ρ)+12π(aT1+ρ2bT2) = ρ ρ 2π (- a^2-2ρ ab+b^22ρ )+ _2 (-a,-b,ρ )+ 1 2π (aT_1+ρ^2bT_2 ) [z22z1≥a;z2≥b] [z_2^2I \z_1≥ a;z_2≥ b \ ] =ρρ′2πexp(−a2−2ρab+b22ρ′)+2(−a,−b,ρ)+12π(bT2+ρ2aT1) = ρ ρ 2π (- a^2-2ρ ab+b^22ρ )+ _2 (-a,-b,ρ )+ 1 2π (bT_2+ρ^2aT_1 ) Write ρ′=1−ρ2ρ =1-ρ^2 gives the desired result. ∎ Lemma D.8. Let z1,z2∼(0,1)z_1,z_2 (0,1 ) with Cov(z1,z2)=ρCov (z_1,z_2 )=ρ. Let a,b∈ℝa,b . Define T1=exp(−a22)1(ρa−b1−ρ2);T2=exp(−b22)1(ρb−a1−ρ2)T_1= (- a^22 ) _1 ( ρ a-b 1-ρ^2 ); T_2= (- b^22 ) _1 ( ρ b-a 1-ρ^2 ) Then we have that [z1z2z1≥a;z2≥b]=1−ρ22πexp(−a2−2ρab+b22(1−ρ2))+ρ2(−a,−b,ρ)+ρ2π(aT1+bT2)E [z_1z_2I \z_1≥ a;z_2≥ b \ ]= 1-ρ^22π (- a^2-2ρ ab+b^22 (1-ρ^2 ) )+ρ _2 (-a,-b,ρ )+ ρ 2π (aT_1+bT_2 ) Proof. To start, we write out the integral form of the expectation as [z1z2z1≥a;z2≥b]=∫b∞∫a∞z1z2f(z1,z2)z1z2=∫b∞(∫a∞z1f(z1∣z2)z1)z2f(z2)z2E [z_1z_2I \z_1≥ a;z_2≥ b \ ]= _b^∞ _a^∞z_1z_2f (z_1,z_2 )dz_1dz_2= _b^∞ ( _a^∞z_1f (z_1 z_2 )dz_1 )z_2f (z_2 )dz_2 Since z1,z2∼(0,1)z_1,z_2 (0,1 ) with covariance ρ, we have that z1∣z2∼(ρz2,1−ρ2)z_1 z_2 (ρ z_2,1-ρ^2 ). Therefore, let ρ′=1−ρ2ρ =1-ρ^2, we have f(z1∣z2)=12πρ′exp(−(z1−ρz2)22ρ′)f (z_1 z_2 )= 1 2πρ (- (z_1-ρ z_2 )^22ρ ) Thus, by Lemma D.14, we have that ∫a∞z1f(z1∣z2)z1 _a^∞z_1f (z_1 z_2 )dz_1 =12πρ′∫a∞z1exp(−(z1−ρz2)22ρ′)z1 = 1 2πρ _a^∞z_1 (- (z_1-ρ z_2 )^22ρ )dz_1 =12πρ′(ρ′exp(−(a−ρz2)22ρ′)+ρz22πρ′1(ρz2−aρ′)) = 1 2πρ (ρ (- (a-ρ z_2 )^22ρ )+ρ z_2 2πρ _1 ( ρ z_2-a ρ ) ) =ρ′2πexp(−(a−ρz2)22ρ′)+ρz21(ρz2−aρ′) = ρ 2π (- (a-ρ z_2 )^22ρ )+ρ z_2 _1 ( ρ z_2-a ρ ) where we set κ=ρ′κ= ρ and μ=ρz2μ=ρ z_2. Therefore [z1z2z1≥a;z2≥b] [z_1z_2I \z_1≥ a;z_2≥ b \ ] =∫b∞(ρ′2πexp(−(a−ρz2)22ρ′)+ρz21(ρz2−aρ′))z2f(z2)z2 = _b^∞ ( ρ 2π (- (a-ρ z_2 )^22ρ )+ρ z_2 _1 ( ρ z_2-a ρ ) )z_2f (z_2 )dz_2 =ρ′2π∫b∞z2exp(−(a−ρz2)22ρ′)f(z2)z2+ρ∫b∞z221(ρz2−aρ′)f(z2)z2 = ρ 2π _b^∞z_2 (- (a-ρ z_2 )^22ρ )f (z_2 )dz_2+ρ _b^∞z_2^2 _1 ( ρ z_2-a ρ )f (z_2 )dz_2 To start, by Lemma D.16, we have 1(ρz2−aρ′)f(z2)=∫a∞f(z1∣z2)z1f(z2)=∫a∞f(z1,z2)z1 _1 ( ρ z_2-a ρ )f (z_2 )= _a^∞f (z_1 z_2 )dz_1f (z_2 )= _a^∞f (z_1,z_2 )dz_1 Therefore, for the second term, we have ∫b∞z221(ρz2−aρ′)f(z2)z2=∫b∞∫a∞z22f(z1,z2)z1z2=[z22z1≥a;z2≥b] _b^∞z_2^2 _1 ( ρ z_2-a ρ )f (z_2 )dz_2= _b^∞ _a^∞z_2^2f (z_1,z_2 )dz_1dz_2=E [z_2^2I \z_1≥ a;z_2≥ b \ ] For the first term, we have exp(−(a−ρz2)22ρ′)f(z2)=12πexp(−z22−2ρaz2+a22ρ′)=12πexp(−a22)exp(−(z2−ρa)22ρ′) (- (a-ρ z_2 )^22ρ )f (z_2 )= 1 2π (- z_2^2-2ρ az_2+a^22ρ )= 1 2π (- a^22 ) (- (z_2-ρ a )^22ρ ) Therefore, by Lemma D.14, the first term can be written as ∫b∞z2exp(−(a−ρz2)22ρ′)f(z2)z2 _b^∞z_2 (- (a-ρ z_2 )^22ρ )f (z_2 )dz_2 =12πexp(−a22)∫b∞z2exp(−(z2−ρa)22ρ′)z2 = 1 2π (- a^22 ) _b^∞z_2 (- (z_2-ρ a )^22ρ )dz_2 =12πexp(−a22)(ρ′exp(−(ρa−b)22ρ′)+2πρ′ρa1(ρa−bρ′)) = 1 2π (- a^22 ) (ρ (- (ρ a-b )^22ρ )+ 2πρ ρ a _1 ( ρ a-b ρ ) ) =ρ′2πexp(−a2−2ρab+b22ρ′)+ρaρ′exp(−a22)1(ρa−bρ′) = ρ 2π (- a^2-2ρ ab+b^22ρ )+ρ a ρ (- a^22 ) _1 ( ρ a-b ρ ) where we set κ=ρ′κ= ρ and μ=ρaμ=ρ a in Lemma D.14. Combining the two terms, we have [z1z2z1≥a;z2≥b] [z_1z_2I \z_1≥ a;z_2≥ b \ ] =ρ′2π(ρ′2πexp(−a2−2ρab+b22ρ′)+ρaρ′exp(−a22)1(ρa−bρ′)) = ρ 2π ( ρ 2π (- a^2-2ρ ab+b^22ρ )+ρ a ρ (- a^22 ) _1 ( ρ a-b ρ ) ) +ρ[z22z1≥a;z2≥b] + [z_2^2I \z_1≥ a;z_2≥ b \ ] =ρ′322πexp(−a2−2ρab+b22ρ′)+aρρ′2πT1+ρ[z22z1≥a;z2≥b] = ρ 322π (- a^2-2ρ ab+b^22ρ )+ aρ 2πT_1+ [z_2^2I \z_1≥ a;z_2≥ b \ ] From Lemma D.7, we have that [z22z1≥a;z2≥b]=ρρ′2πexp(−a2−2ρab+b22ρ′)+2(−a,−b,ρ)+12π(bT2+ρ2aT1)E [z_2^2I \z_1≥ a;z_2≥ b \ ]= ρ ρ 2π (- a^2-2ρ ab+b^22ρ )+ _2 (-a,-b,ρ )+ 1 2π (bT_2+ρ^2aT_1 ) Therefore [z1z2z1≥a;z2≥b]=ρ′2πexp(−a2−2ρab+b22ρ′)+ρ2(−a,−b,ρ)+ρ2π(aT1+bT2)E [z_1z_2I \z_1≥ a;z_2≥ b \ ]= ρ 2π (- a^2-2ρ ab+b^22ρ )+ρ _2 (-a,-b,ρ )+ ρ 2π (aT_1+bT_2 ) Write ρ′=1−ρ2ρ =1-ρ^2 gives the desired result. ∎ Lemma D.9. Let z1∼(μ1,κ12)z_1 ( _1, _1^2 ) and z2∼(μ2,κ22)z_2 ( _2, _2^2 ), with Cov(z1,z2)=κ1κ2ρCov (z_1,z_2 )= _1 _2ρ. Let a,b∈ℝa,b . Then we have [z1z2z1≥a;z2≥b] [z_1z_2I \z_1≥ a;z_2≥ b \ ] =(μ1μ2+κ1κ2ρ)(μ1−aκ1,μ2−bκ2,ρ) = ( _1 _2+ _1 _2ρ ) ( _1-a _1, _2-b _2,ρ ) +κ1κ22πexp(−12(1−ρ2)((μ1−a)2κ12−2ρκ1κ2(μ1−a)(μ2−b)+(μ2−b)2κ22)) + _1 _22π (- 12 (1-ρ^2 ) ( ( _1-a )^2 _1^2- 2ρ _1 _2 ( _1-a ) ( _2-b )+ ( _2-b )^2 _2^2 ) ) +12π((κ2ρa+κ1μ2)T1+(κ1ρb+κ2μ1)T2) + 1 2π ( ( _2ρ a+ _1 _2 )T_1+ ( _1ρ b+ _2 _1 )T_2 ) Here T1,T2T_1,T_2 are defined as T1=exp(−(a−μ1)22κ12)1(11−ρ2(ρ(a−μ1)κ1−b−μ2κ2)) T_1= (- (a- _1 )^22 _1^2 ) _1 ( 1 1-ρ^2 ( ρ (a- _1 ) _1- b- _2 _2 ) ) T2=exp(−(b−μ2)22κ22)1(11−ρ2(ρ(b−μ2)κ2−a−μ1κ1)) T_2= (- (b- _2 )^22 _2^2 ) _1 ( 1 1-ρ^2 ( ρ (b- _2 ) _2- a- _1 _1 ) ) Proof. Let z^1=z1−μ1κ1 z_1= z_1- _1 _1 and z^1=z2−μ2κ2 z_1= z_2- _2 _2. Then we have z^1,z^2∼(0,1) z_1, z_2 (0,1 ). Moreover, Cov(z^1,z^2)=[z^1z^2]=1κ1κ2[(z1−μ1)(z2−μ2)]=ρCov ( z_1, z_2 )=E [ z_1 z_2 ]= 1 _1 _2E [ (z_1- _1 ) (z_2- _2 ) ]=ρ Since z1=κ1z^1+μ1z_1= _1 z_1+ _1 and z2=κ2z^2+μ2z_2= _2 z_2+ _2, we have [z1z2z1≥a;z2≥b] [z_1z_2I \z_1≥ a;z_2≥ b \ ] =[(κ1z^1+μ1)(κ2z^2+μ2)z^1≥a−μ1κ1;z^2≥b−μ2κ2] =E [ ( _1 z_1+ _1 ) ( _2 z_2+ _2 )I \ z_1≥ a- _1 _1; z_2≥ b- _2 _2 \ ] =κ1κ2[z^1z^2z^1≥a^;z^2≥b^]+μ1μ2[z^1≥a^;z^2≥b^] = _1 _2E [ z_1 z_2I \ z_1≥ a; z_2≥ b \ ]+ _1 _2E [I \ z_1≥ a; z_2≥ b \ ] +κ1μ2[z^1z^1≥a^;z^2≥b^]+κ2μ1[z^2z^1≥a^;z^2≥b^] + _1 _2E [ z_1I \ z_1≥ a; z_2≥ b \ ]+ _2 _1E [ z_2I \ z_1≥ a; z_2≥ b \ ] where we re-defined a^=a−μ1κ1 a= a- _1 _1 and b^=b−μ2κ2 b= b- _2 _2. By Lemma D.16, Lemma D.6, and Lemma D.8, we have [z^1z^1≥a^;z^2≥b^]=12π(T1+ρT2);[z^2z^1≥a^;z^2≥b^]=12π(T2+ρT1) [ z_1I \ z_1≥ a; z_2≥ b \ ]= 1 2π (T_1+ρ T_2 ); [ z_2I \ z_1≥ a; z_2≥ b \ ]= 1 2π (T_2+ρ T_1 ) [z^1z^2z^1≥a^;z^2≥b^]=1−ρ22πexp(−a^2−2ρa^b^+b^22(1−ρ2))+ρ2(−a^,−b^,ρ)+ρ2π(a^T1+b^T2) [ z_1 z_2I \ z_1≥ a; z_2≥ b \ ]= 1-ρ^22π (- a^2-2ρ a b+ b^22 (1-ρ^2 ) )+ρ _2 (- a,- b,ρ )+ ρ 2π ( aT_1+ bT_2 ) and [z^1≥a^;z^2≥b^]=2(−a^,−b^,ρ)E [I \ z_1≥ a; z_2≥ b \ ]= _2 (- a,- b,ρ ). Here, T1,T2T_1,T_2 are defined as T1=exp(−a^22)(ρa^−b^1−ρ2);T2=exp(−b^22)(ρb^−a^1−ρ2)T_1= (- a^22 ) ( ρ a- b 1-ρ^2 ); T_2= (- b^22 ) ( ρ b- a 1-ρ^2 ) Plugging in the value of a a and b b gives the desired result. ∎ D.1.4 Multi-variate Coupled Expectation Lemma D.10. Let ∼(,κ2)c ( μ,κ^2I ), and let ∈ℝdu ^d. Define z=⊤z=c u. Then we have [z≥0]=Φ1(−⊤κ‖2)+κ2πexp(−(⊤)22κ2‖22)⋅‖2E [cI \z≥ 0 \ ]= μ _1 (- μ uκ \|u \|_2 )+ κ 2π (- ( μ u )^22κ^2 \|u \|_2^2 )· u \|u \|_2 Proof. According to the law of total expectation, [⊤≥0]=z[[z≥0∣z]]=z[[∣z]z≥0]E [cI \c u≥ 0 \ ]=E_z [E_c [cI \z≥ 0 \ z ] ]=E_z [E_c [c z ]I \z≥ 0 \ ] By Lemma D.1, we have that [∣z]=+‖22(z−⊤)E_c [c z ]= μ+ u \|u \|_2^2 (z- μ u ). Therefore [⊤≥0]=‖22z[zz≥0]+(−⊤⋅‖22)z[z≥0]E [cI \c u≥ 0 \ ]= u \|u \|_2^2E_z [zI \z≥ 0 \ ]+ ( μ- μ u·u \|u \|_2^2 )E_z [I \z≥ 0 \ ] By definition, z[z≥0]=Pr(z≥0)=1−Pr(z≤0)E_z [I \z≥ 0 \ ]= (z≥ 0 )=1- (z≤ 0 ). Since, by Lemma D.13, z∼(⊤,κ2‖22)z ( μ u,κ^2 \|u \|_2^2 ), we have that Pr(z≤0)=Pr(z−⊤κ‖2≤−⊤‖2)=1(−⊤κ‖2) (z≤ 0 )= ( z- μ uκ \|u \|_2≤- μ u \|u \|_2 )= _1 (- μ uκ \|u \|_2 ) Moreover, let z^=z−⊤κ‖2 z= z- μ uκ \|u \|_2, then we have z[zz≥0] _z [zI \z≥ 0 \ ] =z^[(κ‖2z^+⊤)z^≥−⊤κ‖2] =E_ z [ (κ \|u \|_2 z+ μ u )I \ z≥- μ uκ \|u \|_2 \ ] =κ‖2z^[z^z^≥−⊤κ‖2]+⊤z[z≥0] =κ \|u \|_2E_ z [ zI \ z≥- μ uκ \|u \|_2 \ ]+ μ uE_z [z≥ 0 ] By the PDF of z z, we have z^[z^z≥0]=12π∫a∞zexp(−z22)z=−12πexp(−z22)|a∞=12πexp(−a22)E_ z [ zI \z≥ 0 \ ]= 1 2π _a^∞z (- z^22 )dz=- 1 2π (- z^22 )|_a^∞= 1 2π (- a^22 ) Therefore z[zz≥0]=κ‖22πexp(−(⊤)22κ2‖22)+⊤z[z≥0]E_z [zI \z≥ 0 \ ]= κ \|u \|_2 2π (- ( μ u )^22κ^2 \|u \|_2^2 )+ μ uE_z [z≥ 0 ] Plugging in gives [⊤≥0]=Φ1(−⊤κ‖2)+κ2πexp(−(⊤)22κ2‖22)⋅‖2E [cI \c u≥ 0 \ ]= μ _1 (- μ uκ \|u \|_2 )+ κ 2π (- ( μ u )^22κ^2 \|u \|_2^2 )· u \|u \|_2 ∎ Lemma D.11. Let ∼(,κ2) c ( μ,κ^2I ) with κ≤1κ≤ 1. Let ,u,v be given, and let z1=⊤,z2=⊤z_1=c u,z_2=c v. Then we have that ‖[z1≥0]−1(⊤κ‖2)‖∞≤κ‖2exp(−(⊤)22κ2‖2) \|E_c [cI \z_1≥ 0 \ ]- μ _1 ( μ uκ \|u \|_2 ) \|_∞≤κ \|u \|_2 (- ( μ u )^22κ^2 \|u \|_2 ) Proof. Given the form of the conditional expectation, we have [z1≥0]=∫0∞[∣z1]f1(z1)z1=∫0∞(+‖22(z1−⊤))f1(z1)z1E_c [cI \z_1≥ 0 \ ]= _0^∞E_c [c z_1 ]f_1(z_1)dz_1= _0^∞ ( μ+ u \|u \|_2^2 (z_1- μ u ) )f_1(z_1)dz_1 Since z1=⊤z_1=c u, we must have that z1∼(⊤,κ2‖22)z_1 ( μ u,κ^2 \|u \|_2^2 ). Define z′=z1−⊤κ‖2z = z_1- μ uκ \|u \|_2, then we have that z1=κ‖2z′+⊤z_1=κ \|u \|_2z + μ u [z1≥0] _c [cI \z_1≥ 0 \ ] =∫−⊤κ‖2∞(μ+κ⋅z′)f(z′)z′ = _ - μ uκ \|u \|_2^∞ (μ+κu· z )f (z )dz =μ⋅∫−∞⊤κ‖2f(z′)z′+κ⋅∫−∞⊤κ‖2z′f(z′)z′ =μ· _-∞ μ uκ \|u \|_2f (z )dz +κu· _-∞ μ uκ \|u \|_2z f (z )dz =1(⊤κ‖2)+κexp(−(⊤)22κ2‖2) = μ _1 ( μ uκ \|u \|_2 )+κu (- ( μ u )^22κ^2 \|u \|_2 ) where in the last equality we applied Lemma D.14 with a=0a=0 and κ=1κ=1 in the lemma. Therefore, we have that ‖[z1≥0]−1(⊤κ‖2)‖∞≤κ‖∞exp(−(⊤)22κ2‖2) \|E_c [cI \z_1≥ 0 \ ]- μ _1 ( μ uκ \|u \|_2 ) \|_∞≤κ \|u \|_∞ (- ( μ u )^22κ^2 \|u \|_2 ) ∎ Now, we shall dive into [⊤z1≥0;z2≥0]E_c [cc I \z_1≥ 0;z_2≥ 0 \ ]. Lemma D.12. Let ∼(,κ2) c ( μ,κ^2I ) with ‖2≥2 \| μ \|_2≥ 2 and κ≤1κ≤ 1. Let ,u,v be given, and let z1=⊤,z2=⊤z_1= μ u,z_2= μ v. Then we have that ‖[⊤z1≥0;z2≥0]−(⊤+3κ2)1(⊤2κ‖2)1(⊤2κ‖2)‖≤Δ \|E_c [cc I \z_1≥ 0;z_2≥ 0 \ ]v- ( μ μ v+3κ^2v ) _1 ( μ u2κ \|v \|_2 ) _1 ( μ v2κ \|v \|_2 ) \|≤ where Δ is given by Δ=2κ‖2(‖2(ϕ(⊤2κ‖2)+ϕ(⊤2κ‖2))+‖∞(ψ(⊤2κ‖2)+ψ(⊤2κ‖2))) =2κ \|v \|_2 ( \| μ \|_2 (φ ( μ u2κ \|u \|_2 )+φ ( μ u2κ \|v \|_2 ) )+ \| μ \|_∞ (ψ ( μ u2κ \|u \|_2 )+ψ ( μ u2κ \|u \|_2 ) ) ) Proof. We have [⊤z1≥0;z2≥0]=∫z1,z2≥0[⊤∣z1,z2]f(z1,z2)z1z2E_c [cc I \z_1≥ 0;z_2≥ 0 \ ]= _z_1,z_2≥ 0E_c [cc z_1,z_2 ]f (z_1,z_2 )dz_1dz_2 Notice that [⊤∣z1,z2]=Cov(∣z1,z2)+[∣z1,z2][∣z1,z2]⊤E_c [cc z_1,z_2 ]=Cov (c z_1,z_2 )+E[c z_1,z_2]E[c z_1,z_2] By the form of Cov(∣z1,z2)Cov (c z_1,z_2 ), we have Cov(∣z1,z2)=κ2−κ2(⊤−⊤)2‖22‖22−⟨,⟩2:=Cov (c z_1,z_2 )=κ^2I- κ^2 (uv -vu )^2 \|v \|_2^2 \|u \|_2^2- v,u ^2:=M By the form of [∣z1,z2]E[c z_1,z_2] we have [∣z1,z2]=z1⋅1+z2⋅2+3E[c z_1,z_2]=z_1·s_1+z_2·s_2+s_3 where 1=‖22−⟨,⟩‖22‖22−⟨,⟩2;2=‖22−⟨,⟩‖22‖22−⟨,⟩2 s_1= \|v \|_2^2u- u,v v \|v \|_2^2 \|u \|_2^2- v,u ^2;s_2= \|u \|_2^2v- u,v u \|v \|_2^2 \|u \|_2^2- v,u ^2 3=−(‖22−⟨,⟩)⟨,⟩+(‖22−⟨,⟩)⟨,⟩‖22‖22−⟨,⟩2 s_3= μ- ( \|v \|_2^2u- u,v v ) μ,u + ( \|u \|_2^2v- u,v u ) μ,v \|v \|_2^2 \|u \|_2^2- v,u ^2 Therefore [∣z1,z2][∣z1,z2]⊤=(z1⋅1+z2⋅2+3)(z1⋅1+z2⋅2+3)⊤E [c z_1,z_2 ]E [c z_1,z_2 ] = (z_1·s_1+z_2·s_2+s_3 ) (z_1·s_1+z_2·s_2+s_3 ) Recall that we are interested in [⊤z1≥0;z2≥0]=∫z1,z2≥0(Cov(∣z1,z2)+[∣z1,z2][∣z1,z2]⊤)f(z1,z2)z1z2E_c [cc I \z_1≥ 0;z_2≥ 0 \ ]= _z_1,z_2≥ 0 (Cov (c z_1,z_2 )+E[c z_1,z_2]E[c z_1,z_2] )f(z_1,z_2)dz_1dz_2 Let z^1=z1−⊤κ‖2;z^2=z1−⊤κ‖2 z_1= z_1- μ uκ \|u \|_2; z_2= z_1- μ vκ \|v \|_2 Then we have z^1,z^2∼(0,1);Cov(z^1,z^2)=⊤‖2‖2:=ρ z_1, z_2 (0,1); ( z_1, z_2 )= u v \|u \|_2 \|v \|_2:=ρ Thus, z^1∣z^2=γ2∼(ργ2,1−ρ2);z^2∣z^1=γ1∼(ργ1,1−ρ2) z_1 z_2= _2 (ρ _2,1-ρ^2 ); z_2 z_1= _1 (ρ _1,1-ρ^2 ) Moreover, [∣z1,z2] [c z_1,z_2] =z1⋅1+z2⋅2+3 =z_1·s_1+z_2·s_2+s_3 =(κ‖2z^1+⊤)1+(κ‖2z^2+⊤)2+3 = (κ \|u \|_2 z_1+ μ u )s_1+ (κ \|v \|_2 z_2+ μ v )s_2+s_3 Redefine ^1=κ‖21;^2=κ‖22;^3=⊤⋅1+⊤⋅2+3= s_1=κ \|u \|_2s_1; s_2=κ \|v \|_2s_2; s_3= μ u·s_1+ μ v·s_2+s_3= μ Then we have [∣z1,z2]=^1z^1+^2z^2+^3E[c z_1,z_2]= s_1 z_1+ s_2 z_2+ s_3 In this case, let f f be the joint PDF of z^1 z_1 and z^2 z_2, then we have f^(z^1,z^2) f ( z_1, z_2 ) =12π1−ρ2exp(−12(1−ρ2)(z12−2ρz1z2+z22)) = 12π 1-ρ^2 (- 12(1-ρ^2) (z_1^2-2ρ z_1z_2+z_2^2 ) ) =κ2‖2‖22πκ2‖2‖21−ρ2exp(−12(1−ρ2)(z12−2ρz1z2+z22)) = κ^2 \|u \|_2 \|v \|_22πκ^2 \|u \|_2 \|v \|_2 1-ρ^2 (- 12(1-ρ^2) (z_1^2-2ρ z_1z_2+z_2^2 ) ) =κ2‖2‖2f(z^1,z^2) =κ^2 \|u \|_2 \|v \|_2f ( z_1, z_2 ) Therefore, since dz1=κ‖2dz^1dz_1=κ \|u \|_2d z_1, and dz2=κ‖2dz^2dz_2=κ \|v \|_2d z_2, we have [⊤z1≥0;z2≥0] _c [cc I \z_1≥ 0;z_2≥ 0 \ ] =∫z1,z2≥0(Cov(∣z1,z2)+[∣z1,z2][∣z1,z2]⊤)f(z1,z2)z1z2 = _z_1,z_2≥ 0 (Cov (c z_1,z_2 )+E[c z_1,z_2]E[c z_1,z_2] )f(z_1,z_2)dz_1dz_2 =∫−⊤κ‖2∞∫−⊤κ‖2∞(Cov(∣z1,z2)+[∣z1,z2][∣z1,z2]⊤)f^(z^1,z^2)z^1z^2 = _- μ vκ \|v \|_2^∞ _- μ uκ \|u \|_2^∞ (Cov (c z_1,z_2 )+E[c z_1,z_2]E[c z_1,z_2] ) f( z_1, z_2)d z_1d z_2 =^1^1⊤∫−⊤κ‖2∞∫−⊤κ‖2∞z^12f^(z^1,z^2)z^1z^2⏟ℐ1 = s_1 s_1 _- μ vκ \|v \|_2^∞ _- μ uκ \|u \|_2^∞ z_1^2 f( z_1, z_2)d z_1d z_2_I_1 +^2^2⊤∫−⊤κ‖2∞∫−⊤κ‖2∞z^22f^(z^1,z^2)z^1z^2⏟ℐ2 + s_2 s_2 _- μ vκ \|v \|_2^∞ _- μ uκ \|u \|_2^∞ z_2^2 f( z_1, z_2)d z_1d z_2_I_2 +(^1^2⊤+^2^1⊤)∫−⊤κ‖2∞∫−⊤κ‖2∞z^1z^2f^(z^1,z^2)z^1z^2⏟ℐ3 + ( s_1 s_2 + s_2 s_1 ) _- μ vκ \|v \|_2^∞ _- μ uκ \|u \|_2^∞ z_1 z_2 f( z_1, z_2)d z_1d z_2_I_3 +(^1^3+^3^1⊤)∫−⊤κ‖2∞∫−⊤κ‖2∞z^1f^(z^1,z^2)z^1z^2⏟ℐ4 + ( s_1 s_3+ s_3 s_1 ) _- μ vκ \|v \|_2^∞ _- μ uκ \|u \|_2^∞ z_1 f( z_1, z_2)d z_1d z_2_I_4 +(^2^3⊤+^3^2⊤)∫−⊤κ‖2∞∫−⊤κ‖2∞z^2f^(z^1,z^2)z^1z^2⏟ℐ5 + ( s_2 s_3 + s_3 s_2 ) _- μ vκ \|v \|_2^∞ _- μ uκ \|u \|_2^∞ z_2 f( z_1, z_2)d z_1d z_2_I_5 +(^3^3⊤+)∫−⊤κ‖2∞∫−⊤κ‖2∞f^(z^1,z^2)z^1z^2⏟ℐ6 + ( s_3 s_3 +M ) _- μ vκ \|v \|_2^∞ _- μ uκ \|u \|_2^∞ f( z_1, z_2)d z_1d z_2_I_6 Since our goal is to study the term [⊤z1≥0;z2≥0]E_c [cc I \z_1≥ 0;z_2≥ 0 \ ]v, we need to understand the terms ℐ1I_1 to ℐ6I_6, as well as understanding the matrix-vector product in front of these terms. To start, under some standard computation, we have ^1⊤ s_1 u =κ‖2⋅‖22⊤−⊤⋅⊤‖22‖22−(⊤)2=κ‖2 =κ \|u \|_2· \|v \|_2^2u u-v u·v u \|v \|_2^2 \|u \|_2^2- (v u )^2=κ \|u \|_2 ^1⊤ s_1 v =κ‖2⋅‖22⊤−⊤⋅⊤‖22‖22−(⊤)2=0 =κ \|u \|_2· \|v \|_2^2u v-v u·v v \|v \|_2^2 \|u \|_2^2- (v u )^2=0 ^2⊤ s_2 u =κ‖2⋅‖22⊤−⊤⋅⊤‖22‖22−(⊤)2=0 =κ \|v \|_2· \|u \|_2^2v u-v u·u u \|v \|_2^2 \|u \|_2^2- (v u )^2=0 ^2⊤ s_2 v =κ‖2⋅‖22⊤−⊤⋅⊤‖22‖22−(⊤)2=κ‖2 =κ \|v \|_2· \|u \|_2^2v v-v u·u v \|v \|_2^2 \|u \|_2^2- (v u )^2=κ \|v \|_2 Therefore, the following must holds ^1^1⊤=;^2^2⊤=κ‖2^2;(^1^2⊤+^2^1⊤)=κ‖2^1; s_1 s_1 v= 0; s_2 s_2 v=κ \|v \|_2 s_2; ( s_1 s_2 + s_2 s_1 )v=κ \|v \|_2 s_1; (^1^3⊤+^3^1⊤)=⊤⋅^1;(^2^3⊤+^3^2⊤)=⊤⋅^2+κ‖2 ( s_1 s_3 + s_3 s_1 )v= μ v· s_1; ( s_2 s_3 + s_3 s_2 )v= μ v· s_2+κ \|v \|_2 μ Lastly, we have (^3^3⊤+) ( s_3 s_3 +M )v =⊤⋅+κ2−κ2‖22‖22−(⊤)2(⊤−⊤)2 = μ v· μ+κ^2v- κ^2 \|v \|_2^2 \|u \|_2^2- (v u )^2 (uv -vu )^2v =⊤⋅+κ2−κ2‖22‖22−(⊤)2(⊤−⊤)(⊤⋅−‖22) = μ v· μ+κ^2v- κ^2 \|v \|_2^2 \|u \|_2^2- (v u )^2 (uv -vu ) (u v·v- \|v \|_2^2u ) =⊤⋅+κ2 = μ v· μ+κ^2v −κ2‖22‖22−(⊤)2(⊤⋅‖22−(⊤)2−⊤⋅‖22+‖22‖22) - κ^2 \|v \|_2^2 \|u \|_2^2- (v u )^2 (u v· \|v \|_2^2u- (u v )^2v-u v· \|v \|_2^2u+ \|v \|_2^2 \|u \|_2^2v ) =⊤⋅+κ2+κ2 = μ v· μ+κ^2v+κ^2v =⊤⋅+2κ2 = μ v· μ+2κ^2v Therefore, we can write [⊤z1≥0;z2≥0]E_c [cc I \z_1≥ 0;z_2≥ 0 \ ]v as [⊤z1≥0;z2≥0] _c [cc I \z_1≥ 0;z_2≥ 0 \ ]v =κ‖2^2⋅ℐ2+κ‖2^1⋅ℐ3+⊤⋅^1⋅ℐ4 =κ \|v \|_2 s_2·I_2+κ \|v \|_2 s_1·I_3+ μ v· s_1·I_4 (71) +(⊤⋅^2+κ‖2)ℐ5+(⊤⋅+2κ2)ℐ6 + ( μ v· s_2+κ \|v \|_2 μ )I_5+ ( μ v· μ+2κ^2v )I_6 =κ‖2(ℐ2⋅^2+ℐ3⋅^1)+⊤(ℐ4⋅^1+ℐ5⋅^2) =κ \|v \|_2 (I_2· s_2+I_3· s_1 )+ μ v (I_4· s_1+I_5· s_2 ) +(κ‖2⋅ℐ5+⊤⋅ℐ6)+2κ2ℐ6 + (κ \|v \|_2·I_5+ μ v·I_6 ) μ+2κ^2I_6v By the definition of ℐ2I_2 to ℐ6I_6, we first notice that ℐ6 _6 =∫−⊤κ‖2∞∫−⊤κ‖2∞f(z^1,z^2)z^1z^2 = _- μ uκ \|u \|_2^∞ _- μ uκ \|u \|_2^∞f( z_1, z_2)d z_1d z_2 =ℙ(z^1≥−⊤κ‖2;z^2≥−⊤κ‖2) =P ( z_1≥- μ uκ \|u \|_2; z_2≥- μ vκ \|v \|_2 ) =ℙ(z^1≤⊤κ‖2;z^2≤⊤κ‖2) =P ( z_1≤ μ uκ \|u \|_2; z_2≤ μ vκ \|v \|_2 ) =Φ2(⊤κ‖2,⊤κ‖2,ρ) = _2 ( μ uκ \|u \|_2, μ vκ \|v \|_2,ρ ) Moreover, we can invoke Lemma D.6, Lemma D.7, and Lemma D.8 to get that ℐ4=12π(T1+ρT2);ℐ5=12π(T2+ρT1) _4= 1 2π (T_1+ρ T_2 ); _5= 1 2π (T_2+ρ T_1 ) ℐ2=ρ1−ρ22πexp(−a2−2ρab+b22(1−ρ2))+2(−a,−b,ρ)+12π(bT2+ρ2aT1) _2= ρ 1-ρ^22π (- a^2-2ρ ab+b^22 (1-ρ^2 ) )+ _2 (-a,-b,ρ )+ 1 2π (bT_2+ρ^2aT_1 ) ℐ3=1−ρ22πexp(−a2−2ρab+b22(1−ρ2))+ρ2(−a,−b,ρ)+ρ2π(aT1+bT2) _3= 1-ρ^22π (- a^2-2ρ ab+b^22 (1-ρ^2 ) )+ρ _2 (-a,-b,ρ )+ ρ 2π (aT_1+bT_2 ) where T1,T2T_1,T_2 and a,ba,b are defined as T1=exp(−a22)1(ρa−b1−ρ2);T2=exp(−b22)1(ρb−a1−ρ2) T_1= (- a^22 ) _1 ( ρ a-b 1-ρ^2 ); T_2= (- b^22 ) _1 ( ρ b-a 1-ρ^2 ) a=−⊤κ‖2;b=−⊤κ‖2;ρ=⊤‖2‖2 a=- μ uκ \|u \|_2; b=- μ vκ \|v \|_2; ρ= u v \|u \|_2 \|v \|_2 To ease our computation, we define E=1−ρ22πexp(−a2−2ρab+b22(1−ρ2));F=2(−a,−b,ρ)E= 1-ρ^22π (- a^2-2ρ ab+b^22 (1-ρ^2 ) );F= _2 (-a,-b,ρ ) Then the terms ℐ2I_2 to ℐ6I_6 can be written as ℐ2=ρE+F+12π(bT2+ρ2aT1);ℐ3=E+ρF+12π(bT2+aT1)ℐ4=12π(T1+ρT2);ℐ5=12π(T2+ρT1);ℐ6=F gatheredI_2=ρ E+F+ 1 2π (bT_2+ρ^2aT_1 ); _3=E+ρ F+ 1 2π (bT_2+aT_1 )\\ I_4= 1 2π (T_1+ρ T_2 ); _5= 1 2π (T_2+ρ T_1 ); _6=F gathered (72) Now, the trick of evaluating (71) is to re-write ^1 s_1 and ^2 s_2 as below ^1 s_1 =κ‖2‖22‖22−(⊤)2⋅(‖22−⊤⋅) = κ \|u \|_2 \|u \|_2^2 \|v \|_2^2- (u v )^2· ( \|v \|_2^2u-u v·v ) (73) =κ(‖22−⊤⋅)‖2‖22(1−ρ2) = κ ( \|v \|_2^2u-u v·v ) \|u \|_2 \|v \|_2^2 (1-ρ^2 ) =κ1−ρ2⋅‖2−κρ1−ρ2⋅‖2 = κ1-ρ^2· u \|u \|_2- κρ1-ρ^2· v \|v \|_2 =κ1−ρ2(‖2−ρ⋅‖2) = κ1-ρ^2 ( u \|u \|_2-ρ· v \|v \|_2 ) ^2 s_2 =κ‖2‖22‖22−(⊤)2⋅(‖22−⊤⋅) = κ \|v \|_2 \|u \|_2^2 \|v \|_2^2- (u v )^2· ( \|u \|_2^2v-u v·u ) =κ(‖22−⊤⋅)‖2‖22(1−ρ2) = κ ( \|u \|_2^2v-u v·u ) \|u \|_2 \|v \|_2^2 (1-ρ^2 ) =κ1−ρ2⋅‖2−κρ1−ρ2⋅‖2 = κ1-ρ^2· v \|v \|_2- κρ1-ρ^2· u \|u \|_2 =κ1−ρ2(‖2−ρ⋅‖2) = κ1-ρ^2 ( v \|v \|_2-ρ· u \|u \|_2 ) Now, we can simplify (71) with (72) and (73). To start, for the terms ℐ2⋅^2+ℐ3⋅^1I_2· s_2+I_3· s_1 we have ℐ2⋅^2+ℐ3⋅^1 _2· s_2+I_3· s_1 =κ1−ρ2(ρE+F+12π(bT2+ρ2aT1))(‖2−ρ⋅‖2) = κ1-ρ^2 (ρ E+F+ 1 2π (bT_2+ρ^2aT_1 ) ) ( v \|v \|_2-ρ· u \|u \|_2 ) +κ1−ρ2(E+ρF+ρ2π(bT2+aT1))(‖2−ρ⋅‖2) + κ1-ρ^2 (E+ρ F+ ρ 2π (bT_2+aT_1 ) ) ( u \|u \|_2-ρ· v \|v \|_2 ) =κE1−ρ2(ρ(‖2−ρ⋅‖2)+(‖2−ρ⋅‖2)) = κ E1-ρ^2 (ρ ( v \|v \|_2-ρ· u \|u \|_2 )+ ( u \|u \|_2-ρ· v \|v \|_2 ) ) +κF1−ρ2((‖2−ρ⋅‖2)+ρ(‖2−ρ⋅‖2)) + κ F1-ρ^2 ( ( v \|v \|_2-ρ· u \|u \|_2 )+ρ ( u \|u \|_2-ρ· v \|v \|_2 ) ) +κρaT12π(1−ρ2)(ρ(‖2−ρ⋅‖2)+(‖2−ρ⋅‖2)) + κρ aT_1 2π (1-ρ^2 ) (ρ ( v \|v \|_2-ρ· u \|u \|_2 )+ ( u \|u \|_2-ρ· v \|v \|_2 ) ) +κbT22π(1−ρ2)((‖2−ρ⋅‖2)+ρ(‖2−ρ⋅‖2)) + κ bT_2 2π (1-ρ^2 ) ( ( v \|v \|_2-ρ· u \|u \|_2 )+ρ ( u \|u \|_2-ρ· v \|v \|_2 ) ) =κE⋅‖2+κF⋅‖2+κρaT12π⋅‖2+κbT22π⋅‖2 =κ E· u \|u \|_2+κ F· v \|v \|_2+ κρ aT_1 2π· u \|u \|_2+ κ bT_2 2π· v \|v \|_2 =κ((E+ρaT12π)‖2+(F+bT22π)‖2) =κ ( (E+ ρ aT_1 2π ) u \|u \|_2+ (F+ bT_2 2π ) v \|v \|_2 ) Similarly, for the term ℐ4^1+ℐ4^2I_4 s_1+I_4 s_2, we have ℐ4^1+ℐ4^2 _4 s_1+I_4 s_2 =κ2π(1−ρ2)((T1+ρT2)(‖2−ρ⋅‖2)+(T2+ρT1)(‖2−ρ⋅‖2)) = κ 2π (1-ρ^2 ) ( (T_1+ρ T_2 ) ( v \|v \|_2-ρ· u \|u \|_2 )+ (T_2+ρ T_1 ) ( u \|u \|_2-ρ· v \|v \|_2 ) ) =κT12π(1−ρ2)((‖2−ρ⋅‖2)+ρ(‖2−ρ⋅‖2)) = κ T_1 2π (1-ρ^2 ) ( ( v \|v \|_2-ρ· u \|u \|_2 )+ρ ( u \|u \|_2-ρ· v \|v \|_2 ) ) +κT22π(1−ρ2)(ρ(‖2−ρ⋅‖2)+(‖2−ρ⋅‖2)) + κ T_2 2π (1-ρ^2 ) (ρ ( v \|v \|_2-ρ· u \|u \|_2 )+ ( u \|u \|_2-ρ· v \|v \|_2 ) ) =κT12π⋅‖2+κT22π⋅‖2 = κ T_1 2π· v \|v \|_2+ κ T_2 2π· u \|u \|_2 =κ2π(T1⋅‖2+T2⋅‖2) = κ 2π (T_1· v \|v \|_2+T_2· u \|u \|_2 ) Applying these evaluations, (71) becomes [⊤z1≥0;z2≥0] _c [cc I \z_1≥ 0;z_2≥ 0 \ ]v =κ2‖2((E+ρaT12π)‖2+(F+bT22π)‖2) =κ^2 \|v \|_2 ( (E+ ρ aT_1 2π ) u \|u \|_2+ (F+ bT_2 2π ) v \|v \|_2 ) (74) +κ⊤2π(T1⋅‖2+T2⋅‖2) + κ μ v 2π (T_1· v \|v \|_2+T_2· u \|u \|_2 ) +κ‖22π(T2+ρT1)+⊤⋅F⋅+2κ2F + κ \|v \|_2 2π (T_2+ρ T_1 ) μ+ μ v· F· μ+2κ^2Fv =κ2‖2(E⋅‖2+ρaT12π⋅‖2+bT22π⋅‖2)⏟1 = κ^2 \|v \|_2 (E· u \|u \|_2+ ρ aT_1 2π· u \|u \|_2+ bT_2 2π· v \|v \|_2 )_g_1 +κ2π(⊤(T1⋅‖2+T2⋅‖2)+‖2(T2+ρT1))⏟2 + κ 2π ( μ v (T_1· v \|v \|_2+T_2· u \|u \|_2 )+ \|v \|_2 (T_2+ρ T_1 ) μ )_g_2 +F(⊤+3κ2) +F ( μ μ v+3κ^2v ) Then we have that [⊤z1≥0;z2≥0]−(⊤+3κ2)1(⊤κ‖2)1(⊤κ‖2) _c [cc I \z_1≥ 0;z_2≥ 0 \ ]v- ( μ μ v+3κ^2v ) _1 ( μ uκ \|u \|_2 ) _1 ( μ vκ \|v \|_2 ) (75) =1+2+(⊤κ‖2,⊤κ‖2,⊤‖2‖2) =g_1+g_2+C ( μ uκ \|u \|_2, μ vκ \|v \|_2, u v \|u \|_2 \|v \|_2 ) The proof then proceed by estimating the magnitude of the three terms. To start, we need to bound T1T_1 and T2T_2. In particular, since 1 _1 is the CDF, its magnitude must be bounded by 11. Therefore 0≤T1≤exp(−a22);0≤T2≤exp(−b22)0≤ T_1≤ (- a^22 ); 0≤ T_2≤ (- b^22 ) Therefore, the ℓ∞ _∞ norm of 2g_2 is bounded by ‖2‖∞ \|g_2 \|_∞ ≤κ2π(⊤(exp(−a22)+exp(−b22)))+‖2‖2(exp(−a22)+ρexp(−b22)) ≤ κ 2π ( μ v ( (- a^22 )+ (- b^22 ) ) )+ \|v \|_2 \| μ \|_2 ( (- a^22 )+ρ (- b^22 ) ) (76) ≤2κ2π‖2‖2(exp(−a22)+exp(−b22)) ≤ 2κ 2π \|v \|_2 \| μ \|_2 ( (- a^22 )+ (- b^22 ) ) ≤κ‖2‖2(ϕ(a2)+ϕ(b2)) ≤κ \|v \|_2 \| μ \|_2 (φ ( a2 )+φ ( b2 ) ) Next, for E, we have E E =1−ρ22πexp(−a2−2ρab+b22(1−ρ2)) = 1-ρ^22π (- a^2-2ρ ab+b^22 (1-ρ^2 ) ) =1−ρ24π(exp(−a2−2ρab+ρ2b22(1−ρ2)−b22)+exp(−ρ2a2−2ρab+b22(1−ρ2)−a22)) = 1-ρ^24π ( (- a^2-2ρ ab+ρ^2b^22 (1-ρ^2 )- b^22 )+ (- ρ^2a^2-2ρ ab+b^22 (1-ρ^2 )- a^22 ) ) ≤14π(exp(−a22)+exp(−b22)) ≤ 14π ( (- a^22 )+ (- b^22 ) ) ≤14π(ϕ(a2)+ϕ(b2)) ≤ 14π (φ ( a2 )+φ ( b2 ) ) Therefore, the magnitude of 1g_1 can be bounded by ‖1‖2 \|g_1 \|_2 ≤κ2‖2(|E|+|a|T12π+|b|T22π) ≤κ^2 \|v \|_2 ( |E |+ |a |T_1 2π+ |b |T_2 2π ) (77) ≤κ2‖2(14π(ϕ(a2)+ϕ(b2))+|a|2πexp(−a22)+|b|2πexp(−b22)) ≤κ^2 \|v \|_2 ( 14π (φ ( a2 )+φ ( b2 ) )+ |a | 2π (- a^22 )+ |b | 2π (- b^22 ) ) ≤κ2‖2(14π(ϕ(a2)+ϕ(b2))+ψ(a2)+ψ(b2)) ≤κ^2 \|v \|_2 ( 14π (φ ( a2 )+φ ( b2 ) )+ψ ( a2 )+ψ ( b2 ) ) Moreover, by the bound of the Gaussian Copula function, we have that |(a,b,ρ)|≤14exp(−a2+b24) |C (a,b,ρ ) |≤ 14 (- a^2+b^24 ) Therefore, we have that (⊤κ‖2,⊤κ‖2,⊤‖2‖2)≤14exp(−a24)exp(−b24)=14ϕ(a2)ϕ(b2)C ( μ uκ \|u \|_2, μ vκ \|v \|_2, u v \|u \|_2 \|v \|_2 )≤ 14 (- a^24 ) (- b^24 )= 14φ ( a2 )φ ( b2 ) Combining the results gives ∥[⊤z1≥0;z2≥0]−(⊤+3κ2)1(⊤κ‖2)1(⊤κ‖2,)∥2 \|E_c [cc I \z_1≥ 0;z_2≥ 0 \ ]v- ( μ μ v+3κ^2v ) _1 ( μ uκ \|u \|_2 ) _1 ( μ vκ \|v \|_2, ) \|_2 ≤κ2‖2(14π(ϕ(a2)+ϕ(b2))+ψ(a2)+ψ(b2))+κ‖2‖2(ϕ(a2)+ϕ(b2)) ≤κ^2 \|v \|_2 ( 14π (φ ( a2 )+φ ( b2 ) )+ψ ( a2 )+ψ ( b2 ) )+κ \|v \|_2 \| μ \|_2 (φ ( a2 )+φ ( b2 ) ) +14(|⊤|‖∞+3κ2‖2)ϕ(a2)ϕ(b2) + 14 ( | μ v | \| μ \|_∞+3κ^2 \|v \|_2 )φ ( a2 )φ ( b2 ) =‖2((κ2+κ‖2)(ϕ(a2)+ϕ(b2))+(κ2+κ‖∞)(ψ(a2)+ψ(b2))) = \|v \|_2 ( (κ^2+κ \| μ \|_2 ) (φ ( a2 )+φ ( b2 ) )+ (κ^2+κ \| μ \|_∞ ) (ψ ( a2 )+ψ ( b2 ) ) ) ≤2κ‖2(‖2(ϕ(a2)+ϕ(b2))+‖∞(ψ(a2)+ψ(b2))) ≤ 2κ \|v \|_2 ( \| μ \|_2 (φ ( a2 )+φ ( b2 ) )+ \| μ \|_∞ (ψ ( a2 )+ψ ( b2 ) ) ) ∎ D.1.5 Other Results Lemma D.13. Let ∼(,κ2)c ( μ,κ^2I ), and let ∈ℝdu ^d be a vector. Define z=⊤z=c u. Then we have that z∼(⊤,κ2‖22)z ( μ u,κ^2 \|u \|_2^2 ). Proof. Since z=⊤z=c u where ∼(,κ2)c ( μ,κ^2I ). Then the moment generating function of z is given by Mz(t) M_z(t) =[exp(zt)] =E [ (zt ) ] =[∏j=1dexp(cjujt)] =E [ _j=1^d (c_ju_jt ) ] =∏j=1d[exp(cjujt)] = _j=1^dE [ (c_ju_jt ) ] =∏j=1dexp(ujμjt+12uj2κ2t2) = _j=1^d (u_j _jt+ 12u_j^2κ^2t^2 ) =exp((∑j=1dujμj)t+12(∑j=1duj2)κ2t2) = ( ( _j=1^du_j _j )t+ 12 ( _j=1^du_j^2 )κ^2t^2 ) =exp(⊤⋅t+12‖22κ2t2) = ( μ u· t+ 12 \|u \|_2^2κ^2t^2 ) Therefore, z∼(⊤,κ2‖22)z ( μ u,κ^2 \|u \|_2^2 ). ∎ Lemma D.14. Let κ,μ,a∈ℝκ,μ,a be given such that κ>0κ>0. Then we have that ∫a∞zexp(−(z−μ)22κ2)z=κ2exp(−(μ−a)22κ2)+κμ2π1(μ−aκ) _a^∞z (- (z-μ )^22κ^2 )dz=κ^2 (- (μ-a )^22κ^2 )+κμ 2π _1 ( μ-aκ ) Proof. We use a change of variable z′=z−μκz = z-μκ. Then we have that z=κz′+μz=κ z +μ, and dz=κdz′dz=κ dz . Therefore ∫a∞zexp(−(z−μ)22κ2)z _a^∞z (- (z-μ )^22κ^2 )dz =∫a−μκ∞(κz′+μ)exp(−z′22)κz′ = _ a-μκ^∞ (κ z +μ ) (- z 22 )κ dz =κ2∫a−μκ∞z′exp(−z′22)z′+κμ∫a−μκ∞exp(−z′22)z′ =κ^2 _ a-μκ^∞z (- z 22 )dz +κμ _ a-μκ^∞ (- z 22 )dz =κ2exp(−z′22)∣∞a−μκ+κμ2π(1−1(a−μκ)) =κ^2 (- z 22 ) _∞ a-μκ+κμ 2π (1- _1 ( a-μκ ) ) =κ2exp(−(μ−a)22κ2)+κμ2π1(μ−aκ) =κ^2 (- (μ-a )^22κ^2 )+κμ 2π _1 ( μ-aκ ) ∎ Lemma D.15. Let κ,μ,a∈ℝκ,μ,a be given such that κ>0κ>0. Then we have that ∫a∞z2exp(−(z−μ)22κ2)z=κ2(a+μ)exp(−(μ−a)22κ2)+2πκ(κ2+μ2)1(μ−aκ) _a^∞z^2 (- (z-μ )^22κ^2 )dz=κ^2 (a+μ ) (- (μ-a )^22κ^2 )+ 2πκ (κ^2+μ^2 ) _1 ( μ-aκ ) Proof. To start, let z′=z−μκz = z-μκ. Then we have that z=κz′+μz=κ z +μ, and dz=κdz′dz=κ dz . Therefore ∫a∞z2exp(−(z−μ)22κ2)z _a^∞z^2 (- (z-μ )^22κ^2 )dz =κ∫a−μκ∞(κz′+μ)2exp(−z′22)z′ =κ _ a-μκ^∞ (κ z +μ )^2 (- z 22 )dz =κ3∫a−μκ∞z′2exp(−z′22)z′+2κ2μ∫a−μκ∞z′exp(−z′22)z′ =κ^3 _ a-μκ^∞z 2 (- z 22 )dz +2κ^2μ _ a-μκ^∞z (- z 22 )dz +κμ2∫a−μκ∞exp(−z′22)z′ +κμ^2 _ a-μκ^∞ (- z 22 )dz Notice that for the third term, we have that ∫a−μκ∞exp(−z′22)z′=2π(1−1(a−μκ))=2π1(μ−aκ) _ a-μκ^∞ (- z 22 )dz = 2π (1- _1 ( a-μκ ) )= 2π _1 ( μ-aκ ) For the second term, we can directly apply Lemma D.14 with κ=1,μ=0κ=1,μ=0 to get that ∫a−μκ∞z′exp(−z′22)z′=exp(−(a−μ)22κ2) _ a-μκ^∞z (- z 22 )dz = (- (a-μ )^22κ^2 ) For the first term, we apply integration by parts with u(z′)=−z′u(z )=-z and v(z′)=exp(−z′22)v(z )= (- z 22 ). In particular, notice that v′(z′)=−z′exp(−z′22)v (z )=-z (- z 22 ) and u′(z′)=−1u (z )=-1. Therefore ∫a−μκ∞z′2exp(−z′22)z′ _ a-μκ^∞z 2 (- z 22 )dz =∫a−μκ∞u(z′)v(z′) = _ a-μκ^∞u(z )dv(z ) =u(z′)v(z′)∣a−μκ∞−∫a−μκ∞v(z′)u(z′) =u(z )v(z ) _ a-μκ^∞- _ a-μκ^∞v(z )du(z ) =−z′exp(−z′22)∣a−μκ∞+∫a−μκ∞exp(−z′22)z′ =-z (- z 22 ) _ a-μκ^∞+ _ a-μκ^∞ (- z 22 )dz =a−μκexp(−(μ−a)22κ2)+2π(1−1(a−μκ)) = a-μκ (- (μ-a )^22κ^2 )+ 2π (1- _1 ( a-μκ ) ) =a−μκexp(−(μ−a)22κ2)+2π1(μ−aκ) = a-μκ (- (μ-a )^22κ^2 )+ 2π _1 ( μ-aκ ) Putting things together, we have that ∫a∞z2exp(−(z−μ)22κ2)z _a^∞z^2 (- (z-μ )^22κ^2 )dz =κ3(a−μκexp(−(μ−a)22κ2)+2π1(μ−aκ)) =κ^3 ( a-μκ (- (μ-a )^22κ^2 )+ 2π _1 ( μ-aκ ) ) +2κ2μexp(−(a−μ)22κ2)+κμ22π1(μ−aκ) +2κ^2μ (- (a-μ )^22κ^2 )+κμ^2 2π _1 ( μ-aκ ) =(κ2(a−μ)+2κ2μ)exp(−(μ−a)22κ2)+2π(κ3+κμ2)1(μ−aκ) = (κ^2 (a-μ )+2κ^2μ ) (- (μ-a )^22κ^2 )+ 2π (κ^3+κμ^2 ) _1 ( μ-aκ ) =κ2(a+μ)exp(−(μ−a)22κ2)+2πκ(κ2+μ2)1(μ−aκ) =κ^2 (a+μ ) (- (μ-a )^22κ^2 )+ 2πκ (κ^2+μ^2 ) _1 ( μ-aκ ) ∎ Lemma D.16. Let z1,z2∼(0,1)z_1,z_2 (0,1 ) with Cov(z1,z2)=ρCov (z_1,z_2 )=ρ. Then we have that ∫a∞f(z1∣z2)z1=1(ρz2−a1−ρ2) _a^∞f (z_1 z_2 )dz_1= _1 ( ρ z_2-a 1-ρ^2 ) Proof. Since z1,z2∼(0,1)z_1,z_2 (0,1 ) with Cov(z1,z2)=ρCov (z_1,z_2 )=ρ, we have that z1∣z2∼(ρz2,1−ρ2)z_1 z_2 (ρ z_2,1-ρ^2 ). Therefore, using a change of variable z′=z1−ρz21−ρ2z = z_1-ρ z_2 1-ρ^2, we have ∫a∞f(z1∣z2)z1 _a^∞f (z_1 z_2 )dz_1 =12π(1−ρ2)∫a∞exp(−(z1−ρz2)22(1−ρ2))z1 = 1 2π (1-ρ^2 ) _a^∞ (- (z_1-ρ z_2 )^22 (1-ρ^2 ) )dz_1 =12π∫a−ρz21−ρ2∞exp(−z′22)z′ = 1 2π _ a-ρ z_2 1-ρ^2^∞ (- z 22 )dz =1−1(a−ρz21−ρ2) =1- _1 ( a-ρ z_2 1-ρ^2 ) =1(ρz2−a1−ρ2) = _1 ( ρ z_2-a 1-ρ^2 ) ∎ Lemma D.17. Let z1,z2∼(0,1)z_1,z_2 (0,1 ) with Cov(z1,z2)=ρCov (z_1,z_2 )=ρ. Then we have that 2(−a,−b,ρ)=∫a∞∫b∞f(z1,z2)z2z1=∫b∞1(ρz2−a1−ρ2)f(z2)z2 _2 (-a,-b,ρ )= _a^∞ _b^∞f (z_1,z_2 )dz_2dz_1= _b^∞ _1 ( ρ z_2-a 1-ρ^2 )f (z_2 )dz_2 Proof. Let z1′,z2′∼(0,1)z_1 ,z_2 (0,1 ) with Cov(z1,z2)=ρCov (z_1,z_2 )=ρ, and define z1=−z1′,z2=−z2′z_1=-z_1 ,z_2=-z_2 . Then we have that z1,z2∼(0,1)z_1,z_2 (0,1 ) with Cov(z1,z2)=ρCov (z_1,z_2 )=ρ. By symmetry, we have f(z1,z2)=f(−z1,−z2)=f(z1′,z2′)f (z_1,z_2 )=f (-z_1,-z_2 )=f (z_1 ,z_2 ). Moreover, dz2′dz1′=(−dz2)(−dz1)=dz2dz1dz_2 dz_1 = (-dz_2 ) (-dz_1 )=dz_2dz_1. Thus 2(−a,−b,ρ)=∫−∞−a∫−∞−bf(z1′,z2′)z2′z1′=∫a∞∫b∞f(z1,z2)z2z1 _2 (-a,-b,ρ )= _-∞^-a _-∞^-bf (z_1 ,z_2 )dz_2 dz_1 = _a^∞ _b^∞f (z_1,z_2 )dz_2dz_1 Recall that f(z1,z2)=f(z1∣z2)f(z2)f (z_1,z_2 )=f (z_1 z_2 )f (z_2 ). Then we can apply Lemma D.16 to get that ∫a∞∫b∞f(z1,z2)z2z1=∫b∞(∫a∞f(z1∣z2)z1)f(z2)z2=∫b∞1(ρz2−a1−ρ2)f(z2)z2 _a^∞ _b^∞f (z_1,z_2 )dz_2dz_1= _b^∞ ( _a^∞f (z_1 z_2 )dz_1 )f (z_2 )dz_2= _b^∞ _1 ( ρ z_2-a 1-ρ^2 )f (z_2 )dz_2 ∎ Lemma D.18. Let z∼(μ,κ2)z (μ,κ^2 ), and let a∈ℝa . Then we have [zz≥a]=κ2πexp(−(μ−a)22κ2)+μ1(μ−aκ)E [zI \z≥ a \ ]= κ 2π (- (μ-a )^22κ^2 )+μ _1 ( μ-aκ ) Proof. Define z^=z−μκ z= z-μκ. Then we have that z^∼(0,1) z (0,1 ). Since z=κz^+μz=κ z+μ, we have [zz≥0] [zI \z≥ 0 \ ] =[(κz^+μ)z^≥a−μκ] =E [ (κ z+μ )I \ z≥ a-μκ \ ] =κ[z^z^≥a−μκ]+μ[z^≥a−μκ] = [ zI \ z≥ a-μκ \ ]+ [I \ z≥ a-μκ \ ] Notice that [z^≥a−μκ]=Pr(z^≥a−μκ)=1(μ−aκ)E [I \ z≥ a-μκ \ ]= ( z≥ a-μκ )= _1 ( μ-aκ ). Moreover [z^z^≥−μκ]=12π∫a−μκ∞z^exp(−z22)z=12πexp(−(a−μ)22κ2) [ zI \ z≥- μκ \ ]= 1 2π _ a-μκ^∞ z (- z^22 )dz= 1 2π (- (a-μ )^22κ^2 ) Therefore [zz≥0]=κ2πexp(−(a−μ)22κ2)+μ1(μ−aκ)E [zI \z≥ 0 \ ]= κ 2π (- (a-μ )^22κ^2 )+μ _1 ( μ-aκ ) ∎ Lemma D.19. . Let x∈[−1,1]x∈[-1,1]. Then we have that |arcsinx|≤π2⋅|x|| x|≤ π2·|x|. Proof. To start, consider the case x>0x>0. Define f(x)=arcsinxxf(x)= xx. Then we have that f′(x)=x−2(x1−x2−arcsinx);f′(x)=x−3(3x3−2x(1−x2)32+2arcsinx)f (x)=x^-2 ( x 1-x^2- x ); f (x)=x^-3 ( 3x^3-2x (1-x^2 ) 32+2 x ) For all x∈(0,1]x∈(0,1], we have that 1−x2≥01-x^2≥ 0. Notice that by the Taylor expansion of arcsinx x, we have arcsinx≤x+x36 x≤ x+ x^36 when x∈(0,1]x∈(0,1]. Therefore 3x3−2x(1−x2)32+2arcsinx≥3x3−2x+2x(1−x2)32+x33(1−x2)32(1−x2)32≥3x3−2x(1−(1−x2)32)(1−x2)32 3x^3-2x (1-x^2 ) 32+2 x≥ 3x^3-2x+2x (1-x^2 ) 32+ x^33 (1-x^2 ) 32 (1-x^2 ) 32≥ 3x^3-2x (1- (1-x^2 ) 32 ) (1-x^2 ) 32 Since (1−x2)32≥(1−x2)3≥1−3x4+2x6 (1-x^2 ) 32≥ (1-x^2 )^3≥ 1-3x^4+2x^6, we must have that 3x3−2x(1−(1−x2)32)≤3x3−6x5+4x7=3x3(1−2x2+x4)+x7=3x3(1−x2)2+x7≥03x^3-2x (1- (1-x^2 ) 32 )≤ 3x^3-6x^5+4x^7=3x^3 (1-2x^2+x^4 )+x^7=3x^3 (1-x^2 )^2+x^7≥ 0 Thus, we must have that f′(x)≥0f (x)≥ 0. Therefore, for any ϵ∈(0,0.1]ε∈(0,0.1], we have that for x∈[ϵ,1]x∈[ε,1] f(x) f (x ) ≤f(((1+ϵ)x−ϵ)⋅1+(1−x)⋅ϵ) ≤ f ( ((1+ε)x-ε )· 1+(1-x)·ε ) ≤(1−x)⋅f(ϵ)+((1+ϵ)x−ϵ)⋅f(1) ≤(1-x)· f(ε)+ ((1+ε)x-ε )· f(1) =(1−x)⋅f(ϵ)+π2⋅x−π2⋅ϵ(1−x) =(1-x)· f (ε )+ π2· x- π2·ε (1-x ) ≤π2⋅x+1.002(1−x) ≤ π2· x+1.002 (1-x ) ≤π2 ≤ π2 This gives that f(x)≤π2f (x )≤ π2 for all x∈(0,1]x∈(0,1]. Since f(x)f (x ) is an even function, we have f(x)≤π2f (x )≤ π2 for all x∈[−1,0)x∈[-1,0). Therefore, |arcsinx|≤π2|x|| x|≤ π2|x| when x∈[−1,1]∖0x∈[-1,1] \0\. When x=0x=0, we have arcsinx=0 x=0. This completes the proof. ∎ Lemma D.20. Let :=(|e1|,…,|en|)∈ℝnu:=(|e_1|,...,|e_n|) ^n and :=(1,…,1)∈ℝn.1:=(1,...,1) ^n. Then ∑i=1n|ei|=u⊤. _i=1^n|e_i|=u 1. By the Cauchy–Schwarz inequality, u⊤≤‖u‖2‖2.u 1≤\|u\|_2\|1\|_2. Moreover, ‖u‖2=(∑i=1nui2)1/2=(∑i=1n|ei|2)1/2=(∑i=1nei2)1/2,‖2=(∑i=1n12)1/2=n.\|u\|_2= ( _i=1^nu_i^2 )^1/2= ( _i=1^n|e_i|^2 )^1/2= ( _i=1^ne_i^2 )^1/2, \|1\|_2= ( _i=1^n1^2 )^1/2= n. Combining the above gives ∑i=1n|ei|≤n(∑i=1nei2)1/2. _i=1^n|e_i|≤ n ( _i=1^ne_i^2 )^1/2. Lemma D.21. Assume that Assumption 3.1 holds. Then, we have: ‖∇rℒ()‖22≤nmℒ()12. \| _w_rL_C (W ) \|_2^2≤ n mL_C (W ) 12. Proof. By the form of ∇rℒ() _w_rL_C (W ) in (2), we have: ‖∇rℒ()‖2 \| _w_rL_C (W ) \|_2 =‖arm∑i=1n(f(,i⊙i)−yi)(i⊙i)r⊤(i⊙i)≥0⏟≤1‖2 = \| a_r m _i=1^n (f (W,x_i c_i )-y_i ) (x_i c_i ) I \w_r (x_i c_i )≥ 0 \_≤ 1 \|_2 ≤|ar|m∑i=1n|f(,i⊙i)−yi|⋅‖i⊙i‖2 ≤ |a_r | m _i=1^n |f (W,x_i c_i )-y_i |· \|x_i c_i \|_2 ≤nm(∑i=1n(f(,i⊙i)−yi)2)1/2⋅‖i‖∞‖i‖2using D.20 ≤ n m ( _i=1^n(f (W,x_i c_i )-y_i)^2 )^1/2·\|c_i\|_∞\|x_i\|_2 5.0ptusing lem:CS ≤Cnm(2ℒ())1/2using D.22 with ‖i‖∞≤ 1+κ2log(2dδ)=Cand‖i‖2≤1 ≤ C n m (2L_C (W ) )^1/2 5.0ptusing lem:c_infty_bound with \|c_i\|_∞\;≤\;1+κ 2 \! ( 2dδ )=C 5.0ptand 5.0pt \|x_i \|_2≤ 1 ≤C2nmℒ()1/2 ≤ C 2 n mL_C (W )^1/2 where, in the first inequality, we use the fact that the indicator function is upper-bounded by 1 and in the second inequality, we use the fact that ar=±1a_r=± 1. and so, ‖∇rℒ()‖22≤2C2⋅nmℒ() \| _w_rL_C (W ) \|^2_2≤ 2C^2· nmL_C (W ) ∎ Lemma D.22 (High-probability ℓ∞ _∞ bound for Gaussian masks). Fix δ∈(0,1)δ∈(0,1). Let i∈ℝdc_i ^d be a Gaussian mask with independent coordinates i∼(,κ2d),i.e.,ci,j=1+κgi,j,gi,j∼i.i.d.(0,1).c_i ( 1,κ^2I_d), .e., c_i,j=1+κ g_i,j,\;\;g_i,j .i.d. N(0,1). Then, with probability at least 1−δ1-δ, we have ‖i‖∞≤ 1+κ2log(2dδ).\|c_i\|_∞\;≤\;1+κ 2 \! ( 2dδ ). Proof. Since i∼(,κ2d)c_i ( 1,κ^2I_d) with independent coordinates, each coordinate can be written as ci,j=1+κgi,j,where gi,j∼(0,1)i.i.d.c_i,j=1+κ g_i,j, g_i,j (0,1) 5.0pti.i.d. Hence ‖i‖∞=maxj∈[d]|ci,j|=maxj∈[d]|1+κgi,j|.\|c_i\|_∞= _j∈[d]|c_i,j|= _j∈[d]|1+κ g_i,j|. We observe that |ci,j| |c_i,j| =|1+(ci,j−1)| =|1+(c_i,j-1)| ≤|1|+|ci,j−1|triangle inequality ≤|1|+|c_i,j-1| 5.0pttriangle inequality =1+|κgi,j| =1+|κ g_i,j| We will first bound maxj|ci,j−1|=maxj|κgi,j| _j|c_i,j-1|= _j|κ g_i,j|, and then convert this into a bound on ‖i‖∞\|c_i\|_∞. For brevity, we denote gi,jg_i,j as g∼(0,1)g (0,1) and we are going to show that Pr(|g|≥t)≤2e−t2/2 (|g|≥ t)≤ 2e^-t^2/2 (78) By symmetry of the standard normal distribution, Pr(|g|≥t)=Pr(g≥t)+Pr(g≤−t)=2Pr(g≥t). (|g|≥ t)= (g≥ t)+ (g≤-t)=2 (g≥ t). So it suffices to upper bound Pr(g≥t) (g≥ t). For any λ>0λ>0, since the exponential is monotone increasing we have: g≥t⇒λg≥λt⇒eλg≥eλtg≥ t λ g≥λ t e^λ g≥ e^λ t and so Pr(g≥t)=Pr(eλg≥eλt) (g≥ t)= (e^λ g≥ e^λ t) By Markov’s inequality, for any nonnegative random variable X and any a>0a>0, Pr(X≥a)≤[X]a (X≥ a)≤ E[X]a Applying this with X=eλgX=e^λ g and a=eλta=e^λ t gives Pr(g≥t)=Pr(eλg≥eλt)≤[eλg]eλt=[eλg]e−λt (g≥ t)= (e^λ g≥ e^λ t )≤ E[e^λ g]e^λ t=E[e^λ g]e^-λ t (79) Computation of the moment generating function [eλg]E[e^λ g]: The standard normal density is φ(x)=12πe−x2/2,x∈ℝ. (x)= 1 2πe^-x^2/2, x . Therefore, [eλg] [e^λ g] =∫−∞eλxφ(x)x=12π∫−∞exp(λx−x22)x. = _-∞^∞e^λ x (x)dx= 1 2π _-∞^∞ \! (λ x- x^22 )dx. (80) We now study the exponent: λx−x22=−12(x2−2λx)=−12((x−λ)2−λ2)=λ22−(x−λ)22.λ x- x^22=- 12 (x^2-2λ x )=- 12 ((x-λ)^2-λ^2 )= λ^22- (x-λ)^22. Plugging this into (80) yields [eλg] [e^λ g] =12π∫−∞exp(λ22−(x−λ)22)x = 1 2π _-∞^∞ \! ( λ^22- (x-λ)^22 )dx =eλ2/2⋅12π∫−∞exp(−(x−λ)22)x. =e^λ^2/2· 1 2π _-∞^∞ \! (- (x-λ)^22 )dx. We set u=x−λu=x-λ (change of variables with dx=dudx=du) ∫−∞exp(−(x−λ)22)x=∫−∞exp(−u22)u=2π. _-∞^∞ \! (- (x-λ)^22 )dx= _-∞^∞ \! (- u^22 )du= 2π. Hence [eλg]=eλ2/2.E[e^λ g]=e^λ^2/2. (81) Substituting (81) into (79) gives Pr(g≥t)≤exp(λ22−λt),∀λ>0. (g≥ t)≤ \! ( λ^22-λ t ), ∀λ>0. The right-hand side is a valid bound for every λ>0λ>0, so we choose λ to make it as small as possible. Define f(λ):=λ22−λt.f(λ):= λ^22-λ t. Then f′(λ)=λ−tf (λ)=λ-t, so the unique minimizer is λ=tλ=t (and f′(λ)=1>0f (λ)=1>0 confirms it is a minimum). Plugging λ=tλ=t gives Pr(g≥t)≤exp(t22−t2)=e−t2/2. (g≥ t)≤ \! ( t^22-t^2 )=e^-t^2/2. Using Pr(|g|≥t)=2Pr(g≥t) (|g|≥ t)=2 (g≥ t) proves (78). Now, we focus on one of the mask coordinates by fixing a coordinate j∈[d]j∈[d]. Since ci,j−1=κgi,jc_i,j-1=κ g_i,j with gi,j∼(0,1)g_i,j (0,1), for any u≥0u≥ 0 we have Pr(|ci,j−1|≥u)=Pr(|gi,j|≥u/κ)≤2exp(−u22κ2), (|c_i,j-1|≥ u )= (|g_i,j|≥ u/κ )≤ 2 \! (- u^22κ^2 ), where we applied (78) with t=u/κt=u/κ. Let’s define the event ℰi(u):=maxj∈[d]|ci,j−1|≤u.E_i(u):= \ _j∈[d]|c_i,j-1|≤ u \. Its complement is the event that at least one coordinate deviates by more than u: ℰi(u)c=∃j∈[d]s.t.|ci,j−1|>u.E_i(u)^c= \∃ j∈[d]\ s.t.\ |c_i,j-1|>u \. By the union bound, Pr(ℰi(u)c) (E_i(u)^c ) =Pr(⋃j=1d|ci,j−1|>u)≤∑j=1dPr(|ci,j−1|>u) = ( _j=1^d\|c_i,j-1|>u\ )≤ _j=1^d (|c_i,j-1|>u ) ≤∑j=1d2exp(−u22κ2)=2dexp(−u22κ2). ≤ _j=1^d2 \! (- u^22κ^2 )=2d \! (- u^22κ^2 ). Choose u so that the right-hand side is at most δ: 2dexp(−u22κ2)≤δ⟺−u22κ2≤log(δ2d)⟺u≥κ2log(2dδ).2d \! (- u^22κ^2 )≤δ - u^22κ^2≤ \! ( δ2d ) u≥κ 2 \! ( 2dδ ). Set u:=κ2log(2dδ).u:=κ 2 \! ( 2dδ ). Then Pr(ℰi(u)c)≤δ (E_i(u)^c)≤δ, i.e. Pr(ℰi(u))≥1−δ (E_i(u))≥ 1-δ, and on ℰi(u)E_i(u) we have |ci,j−1|≤u∀j∈[d].|c_i,j-1|≤ u ∀ j∈[d]. On ℰi(u)E_i(u), for each coordinate j, |ci,j|=|1+(ci,j−1)|≤|1|+|ci,j−1|≤1+u.|c_i,j|=|1+(c_i,j-1)|≤|1|+|c_i,j-1|≤ 1+u. Taking the maximum over j yields, with probability at least 1−δ1-δ, ‖i‖∞=maxj∈[d]|ci,j|≤1+u=1+κ2log(2dδ).\|c_i\|_∞= _j∈[d]|c_i,j|≤ 1+u=1+κ 2 \! ( 2dδ ). ∎