Paper deep dive
The Role of Gradient Modification in Heavy-Tailed Nonconvex Stochastic Min-Max Optimization
Tianxi Zhu, Yi Xu, Xiangyang Ji
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/9/2026, 5:38:16 AM
Summary
This paper investigates stochastic min-max optimization under heavy-tailed noise, where stochastic gradients have finite p-th moments (p in (1,2]) rather than bounded variance. The authors demonstrate that vanilla Stochastic Gradient Descent Ascent (SGDA) converges in both nonconvex-strongly-concave (NC-SC) and nonconvex-concave (NC-C) settings, challenging the belief that gradient clipping is necessary. Furthermore, they propose new clipping-free algorithms, Stoc-TRGDAM and Stoc-TRGDmax, for regularized min-max problems that achieve optimal complexity dependence on target accuracy without using gradient clipping.
Entities (8)
Relation Signals (8)
SGDA → convergesunder → Heavy-tailed noise
confidence 95% · vanilla SGDA, without any modification to its update rule, can converge under heavy-tailed noise
Stoc-TRGDAM → avoids → Gradient clipping
confidence 93% · developing new clipping-free algorithms, i.e., Stoc-TRGDAM
Stoc-TRGDmax → avoids → Gradient clipping
confidence 93% · developing new clipping-free algorithms, i.e., Stoc-TRGDmax
SGDA → appliesto → NC-C
confidence 92% · converge under heavy-tailed noise in both ... nonconvex-concave (NC-C) settings
SGDA → appliesto → NC-SC
confidence 92% · converge under heavy-tailed noise in both nonconvex-strongly-concave (NC-SC) ... settings
Stoc-TRGDmax → solves → Regularized Min-Max Optimization
confidence 90% · Stoc-TRGDmax ... handle regularized min-max problems
Stoc-TRGDAM → solves → Regularized Min-Max Optimization
confidence 90% · Stoc-TRGDAM ... handle regularized min-max problems
Gradient clipping → previouslybelievednecessaryfor → Convergence under Heavy-tailed noise
confidence 85% · gradient clipping or normalization is commonly believed to be necessary to guarantee convergence
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Stochastic min-max optimization has attracted increasing attention due to its applications in modern machine learning, while existing theoretical studies mainly rely on the bounded variance assumption for stochastic gradients. Under heavy-tailed noise, where stochastic gradients only possess a finite $p$-th moment for $p\in(1,2]$, gradient clipping or normalization is commonly believed to be necessary to guarantee convergence. In this work, we revisit stochastic min-max optimization under heavy-tailed noise and provide a comprehensive theoretical study of stochastic gradient descent ascent (SGDA). We first show that vanilla SGDA, without any modification to its update rule, can converge under heavy-tailed noise in both nonconvex-strongly-concave (NC-SC) and nonconvex-concave (NC-C) settings, establishing the first convergence guarantees for SGDA in these regimes. Beyond unregularized problems, we further investigate regularized stochastic min-max optimization, where directly incorporating gradient normalization into proximal updates is nontrivial due to the incompatibility between normalization and proximal structures. We overcome this difficulty by developing new clipping-free algorithms, i.e., Stoc-TRGDAM and Stoc-TRGDmax, and they both can achieve the optimal dependence on the target accuracy without using gradient clipping.
Tags
Links
- Source: https://arxiv.org/abs/2609.06064v1
- Canonical: https://arxiv.org/abs/2609.06064v1
Trouble viewing inline? Open PDF directly →
Full Text
296,759 characters extracted from source content.
Expand or collapse full text
The Role of Gradient Modification in Heavy-Tailed Nonconvex Stochastic Min-Max Optimization Tianxi Zhu11 1 School of Control Science and Engineering, Dalian University of Technology; email: zhutianxi3291@dlut.mail.edu.cn Yi Xu22 2 School of Control Science and Engineering, Dalian University of Technology; email: yxu@dlut.edu.cn Xiangyang Ji33 3 Department of Automation, Tsinghua University; email: xyji@tsinghua.edu.cn Abstract Stochastic min-max optimization has attracted increasing attention due to its applications in modern machine learning, while existing theoretical studies mainly rely on the bounded variance assumption for stochastic gradients. Under heavy-tailed noise, where stochastic gradients only possess a finite p-th moment for p∈(1,2]p∈(1,2], gradient clipping or normalization is commonly believed to be necessary to guarantee convergence. In this work, we revisit stochastic min-max optimization under heavy-tailed noise and provide a comprehensive theoretical study of stochastic gradient descent ascent (SGDA). We first show that vanilla SGDA, without any modification to its update rule, can converge under heavy-tailed noise in both nonconvex-strongly-concave (NC-SC) and nonconvex-concave (NC-C) settings, establishing the first convergence guarantees for SGDA in these regimes. Beyond unregularized problems, we further investigate regularized stochastic min-max optimization, where directly incorporating gradient normalization into proximal updates is nontrivial due to the incompatibility between normalization and proximal structures. We overcome this difficulty by developing new clipping-free algorithms, i.e., Stoc-TRGDAM and Stoc-TRGDmax, and they both can achieve the optimal dependence on the target accuracy without using gradient clipping. 1 Introduction This paper first focuses on the following min-max optimization problem: min∈dmax∈df(,), _x∈ d_x _y∈ d_yf(x,y), (1.1) where f:d×d↦f: d_x× d_y is a real-valued function that is differentiable w.r.t. 11 1 “w.r.t. ” is written as “with respect to ”. both x and y. The above minimax problem can be associated with many important areas in machine learning, such as generative adversarial networks (GANs) [16], multi-agent reinforcement learning (MARL) [51, 31], and AUC maximization [35, 56], among others. In most cases, it is difficult to guarantee that the objective function f is convex w.r.t. x. If f w.r.t. x can be nonconvex, this paper refers to Problem (1.1) as nonconvex min-max problems, which is also the problem setting considered in this paper. One of the simplest and most widely adopted optimization framework for solving Problem (1.1) is the stochastic gradient descent-ascent (SGDA) method. It extends the idea of stochastic gradient descent (SGD) method by performing one SGD step on x and one stochastic gradient ascent (SGA) step on y simultaneously in each iteration. t+1=t−η∇f(,,ξ)t+1=t+η∇f(,,ξ) casesx_t+1=x_t- _x _xf(x,y,ξ)\\ y_t+1=y_t+ _y _yf(x,y,ξ) cases (SGDA) Here, ∇f(,,ξ) _xf(x,y,ξ) and ∇f(,,ξ) _yf(x,y,ξ) are the stochastic estimators of ∇f(,) _xf(x,y) and ∇f(,) _xf(x,y), respectively, obtained by sampling the random variable ξ from an unknown distribution ξD_ξ. The stochastic estimators are usually unbiased, i.e., their expectations are true gradients. For nonconvex minimax problems, the convergence of SGDA and its variants has already been extensively studied [33, 52, 42, 25, 21, 54, 61, 8, 21, 15, 32]. Most previous work assumes that the gradient noises are variance-bounded, i.e., there exists a positive real number σ, if for any ∀(,)∈d×d∀(x,y)∈ d_x× d_y, we have [‖∇f(,,ξ)−∇f(,)‖2]≤σ2,[‖∇f(,,ξ)−∇f(,)‖2]≤σ2. [\| _xf(x,y,ξ)- _xf(x,y)\|^2 ]≤σ^2, [\| _yf(x,y,ξ)- _yf(x,y)\|^2 ]≤σ^2. The above assumption also means that the stochastic gradients are light-tailed. Nevertheless, some recent empirical studies, e.g., [57, 14, 1, 6], have observed that, during model training, the noise of stochastic gradients should be heavy-tailed. That is the following assumption: [‖∇f(,,ξ)−∇f(,)‖p]≤σp,[‖∇f(,,ξ)−∇f(,)‖p]≤σp, [\| _xf(x,y,ξ)- _xf(x,y)\|^p ]≤σ^p, [\| _yf(x,y,ξ)- _yf(x,y)\|^p ]≤σ^p, where p∈(1,2]p∈(1,2]. Note that the above noise assumption rigorously extends the typical bounded variance assumption because the variance may be infinite when p<2p<2. The popularity of the heavy-tailed noise assumption began with the study of minimization problems. In the smooth nonconvex setting, many previous works have show that the vanilla SGD with gradient clipping (Clip-SGD), e.g., [57, 43, 47], can achieve the lower bound Ω(ϵ−3p−2p−1) (ε^- 3p-2p-1) [57, 37] that is for general first-order algorithms under heavy-tailed noise. t+1=t−η∇f(t,ξt)⋅minλ‖∇f(t,ξt)‖,1,λ>0 _t+1=x_t-η∇ f(x_t, _t)· \ λ\|∇ f(x_t, _t)\|,1 \,\;λ>0 (Clip-SGD) The reason these works studied Clip-SGD rather than SGD itself is that they believed vanilla SGD could not converge under heavy-tailed noise. For example, consider a specific objective f=x2/2f=x^2/2 and a stochastic oracle f′(x,ξ)=x+ξf (x,ξ)=x+ξ, where ξ can be any heavy-tailed random variable with zero mean but unbounded variance. For the SGD with initial point x1=0x_1=0 and a deterministic step-size η, we have x2=x1−ηf′(x1,ξ)=−ηξx_2=x_1-η f (x_1,ξ)=-ηξ, and obtain [|f′(x2)|2]=η2[|ξ|2]=+∞E[|f (x_2)|^2]=η^2E[|ξ|^2]=+∞, which seems to imply that SGD cannot achieve convergence under heavy-tailed noise even for a simple quadratic function. However, we found that if we consider [|f′(x2)|p]E[|f (x_2)|^p], its value can still be finite. Until recently, [40, 13] proved that SGD can in fact converge under heavy-tailed noise with the complexity (ϵ−2p−1)O(ε^- 2pp-1) in the nonconvex smooth setting. To the best of our knowledge, for minimax problems, although some recent works, such as [22, 60, 36, 59, 5], have considered heavy-tailed noise, they all used gradient clipping or gradient normalization and no work gives an answer of whether SGDA itself can converge. Therefore, we pose the first question studied in this paper: Q1: Can vanilla SGDA converge under heavy-tailed noise? If it can converge, then compared with methods equipped with gradient clipping, is its complexity better or worse? Regarding the second part of Q1, that is, whether the complexity of SGDA is better or worse than that of the method using gradient clipping, readers may think the conclusion is obvious because, as we mentioned, there is indeed a gap between the complexity (ϵ−2p−1)O(ε -2pp-1) of SGD and the complexity (ϵ−3p−2p−1)O(ε^- 3p-2p-1) of Clip-SGD. However, if we consider a broader class of functions, such as nonconvex, nonsmooth, weakly convex functions, i.e., f(x)+ρ/2‖x‖2f(x)+ρ/2\|x\|^2 is convex, under bounded constraints, [62] has shown that both SGD and Clip-SGD can only achieve (ϵ−2p−1)O(ε -2pp-1), and the complexity is difficult to improve for Clip-SGD. This indicates that, in some cases, gradient clipping does not always improve complexity. It should be noted that minmaxf(,) _x _yf(x,y) can degenerate into minf() _xf(x), but we do not assume that x must lie within a bounded region. In fact, this paper studies SGDA under the following two common minimax problem settings: f(⋅,⋅) is L-smooth and f(,⋅) is μ-strongly concave,NC-SC settingf(,⋅) is concave.NC-C setting $f(·,·)$ is $L$-smooth and cases$f( x,·)$ is $μ$-strongly concave,& NC-SC setting\\ $f( x,·)$ is concave.& NC-C setting cases The formal statement of the NC-SC setting can be found in Assumption 2, and the formal statement of the NC-C setting can be found in Assumption 3. Here we emphasize that, compared with works that study minimax problems in the NC-SC or NC-C settings under light-tailed noise, we do not impose any nontrivial assumptions. Here, “nontrivial” refers to additional assumptions beyond those required for the general Danskin theorem (Lemmas 3.1, 3.2 and 3.5) to hold. Next, if the complexity of SGDA under heavy-tailed noise is indeed worse than that of methods equipped with gradient clipping under certain settings, we hope to develop new methods without using clipped gradients to achieve the same complexity as the methods using clipped gradients. The reason is that we do not want to introduce hyperparameter λ that require fine-tuning in practice. Moreover, we aim to address the following more general regularized minimax problem: min∈dmax∈dF(,),F(,)=Δf(,)−h()+r(), _x∈ d_x _y∈ d_yF(x,y), F(x,y) =f(x,y)-h(y)+r(x), (1.2) where the regularizers r:d↦(−∞,+∞]r: d_x (-∞,+∞] and h:d↦(−∞,+∞]h: d_y (-∞,+∞] are both proper, closed and convex on their domain. To illustrate our motivation, let us first consider the composite minimization problem, i.e., minf()+r() _x\f(x)+r(x)\. For such a problem, we can adopt the stochastic proximal gradient descent (SPGD) method, i.e., t+1=proxηr(t−η∇f(t,ξt))x_t+1=prox_η r (x_t-η∇ f(x_t, _t) ), where proxηr(⋅)=Δargminr()+(1/2η)∥−⋅∥2prox_η r (· ) = _x\r(x)+(1/2η)\|x-·\|^2\. For the noncomposite minimization problem under heavy-tailed noise, normalized SGD (NSGD), compared with Clip-SGD, can achieve the same complexity without introducing additional hyperparameters, and can also guarantee convergence even when p is unknown [37, 26]. Therefore, we hope to incorporate the gradient normalization module into the SPGD method. t+1=t−η∇f(t,ξt)‖∇f(t,ξt)‖ _t+1=x_t-η ∇ f(x_t, _t)\|∇ f(x_t, _t)\| (NSGD) We have two natural ways of incorporation: the first is the proximal normalized gradient, i.e., t+1=proxηr(t−η∇f(t,ξt)‖∇f(t,ξt)‖)x_t+1=prox_η r (x_t-η ∇ f(x_t, _t)\|∇ f(x_t, _t)\| ); the second is the normalized gradient mapping, i.e., t+1=t−η(t)/‖(t)‖x_t+1=x_t- (x_t)/\|G(x_t)\|, where (t)=Δηt−1(t−~t+1),~t+1=Δproxηtr(t−ηt∇f(t,ξt))G(x_t) = _t^-1(x_t- x_t+1), x_t+1 =prox_ _tr (x_t- _t∇ f(x_t, _t) ). However, through the discussion in Section 4.1, we realize that even in the deterministic setting, both of these ways are naive. Therefore, we believe that the barrier to incorporate exists even for composite minimization problems. Let us review the existing literature on composite optimization under heavy-tailed noise, although [17, 39, 55] have developed new methods that can achieve the same complexity as Clip-SGD for noncomposite optimization, these methods all used gradient clipping. [23] studied composite and compostional stochastic minimization under heavy-tailed noises, i.e., minξ[f(g(,ξ))]+r() _x\E_ξ[f(g(x,ξ))]+r(x)\, which includes composite stochastic minimization. Their method is t+1=ηt(t)⋅minρt/‖(t)‖,1x_t+1= _tG(x_t)· \ _t/\|G(x_t)\|,1\22 2 Note that they actually use μ to represent the step-szie. We replace μ with ηt _t here to avoid confusion with the strong concavity coefficient.. We point out that although they refer to such a update as “normalized stochastic proximal approximation,” we consider this equivalent to clipping (t)G(x_t) by a hyperparameter ρt _t. [9]’s method indeed reaches the lower bound for nonconvex smooth optimization without using gradient clipping, but the Frank-Wolfe method they used, i.e., t+1∈argmin∈⟨∇f(t,ξt),⟩+r()x_t+1∈ _x \ ∇ f(x_t, _t),x +r(x)\, requires the constraint region X to be compact; otherwise, the Frank-Wolfe update may not be well-defined33 3 Let r()=Δ1/r(x) =1/x and =Δ[1,+∞)X =[1,+∞), then for any k≤0k≤ 0, the optimal solution set of min∈1/+k _x \1/x+kx\ is empty.. For minimax problems, although [60, 59] have used gradient normalization to design their methods to avoid using gradient clipping, they only consider unregularized minimax problems, i.e., Problem (1.1). To the best of our knowledge, for regularized min-max problems, there is currently no known work that has developed methods without using gradient clipping that achieve the same complexity as the methods use gradient clipping. Therefore, we pose the second question studied in this paper: Q2: For the regularized min-max problem, i.e., Problem (1.2), can we develop provable (first-order) algorithms without using gradient clipping, while achieving the same complexity as those algorithms that use gradient clipping? 1.1 Contributions Around the two questions we raised for min-max stochastic optimization under heavy-tailed noise, let us summarize the main results we obtained: • In the NC-SC setting: – We prove the convergence of vanilla min-batch SGDA for Problem (1.1) under heavy-tailed noise. Theorem 3.3 shows that the complexity of min-batch SGDA is (κ3p2(p−1)σp−1ϵ−2p−1),O (κ 3p2(p-1)σ pp-1ε -2pp-1 ), with batch-size B=(max⌈κ4−p2(p−1)σp−1ϵ−2p−1⌉,1)B=O ( \ κ 4-p2(p-1)σ pp-1ε^- 2p-1 ,1 \ ). Here, κ is denoted as L/μL/μ. When p=2p=2, the complexity reduces to the best known result for SGDA under the bounded variance assumption. When p<2p<2, to the best of our knowledge, we show the fisrt upper bound result of vanilla SGDA under heavy-tailed noise in the NC-SC setting. – We realized that there is still a gap between the upper bound result we obtained for SGDA and the lower bound Ω(ϵ−3p−2p−1) (ε^- 3p-2p-1) in Theorem 3.4 for the NC-SC setting. Therefore, we propose two novel algorithms without using gradient clipping, Stoc-TRGDAM (Algorithm 2) and Stoc-TRGDAmax (Algorithm 3), which not only achieve the optimal complexity on ϵε, but also both handle regularized min-max problems, i.e., Problem (1.2). Specifically, as a single-loop method, Stoc-TRGDAM has a complexity of (σp−1κ3p−2p−1ϵ−3p−2p−1)O(σ pp-1κ 3p-2p-1ε^- 3p-2p-1), and the batch-size is (1)O(1). Stoc-TRGDAmax is a nested loop method and its total gradient complexity is (σp−1κ2p−1p−1ϵ−3p−2p−1)O (σ pp-1κ 2p-1p-1ε^- 3p-2p-1 ) with (max⌈σp−1ϵ−p−1⌉,1)O ( \ σ pp-1ε -pp-1 ,1 \ ) batch-size. Although Stoc-TRGDAmax involves nested loops and requires a mini-batch, it improves the dependence of the complexity on the condition number κ compared with Stoc-TRGDAM and requires an smaller batch-size than SGDA when p<2p<2. Through Table 2, we compare the proposed methods with some methods under similar settings from previous studies. • In the NC-C setting, we considered a slightly broader Problem (1.2’) than Problem (1.1), i.e., replacing r in Problem (1.2) with the indicator function δ:↦(−∞,+∞] _X:X (-∞,+∞] on a unbounded closed convex set X, while h remains a general closed convex function. For Problem (1.2’), Theorem 3.6 shows the complexity of SGDA under heavy-tailed noise is (σ3p−2p−1ϵ−6p−4p−1)O(σ 3p-2p-1ε^- 6p-4p-1), which is the first upper bound result of SGDA under heavy-tailed noise in NC-C setting. This complexity matches that of the method proposed in [22], which uses gradient clipping for the updates of tx_t and ty_t. Our results show that, for NC-C min-max problems, gradient clipping neither improves the complexity under heavy-tailed noise, but can instead converge under a stronger convergence measure (a more careful comparison with [22] can be found in Table 1). 1.2 Related Work Stochastic min-max optimization under heavy-tailed noise For min-max problems under light-tailed noise, the recent survey [58] reviews state-of-the-art upper and lower bound results under various settings. We focus on min-max problems under heavy-tailed noise. In the convex-concave (C-C) setting, [4] develops a single-loop zeroth-order method using extra-gradients and gradient clipping, whose complexity in terms of the duality gap, i.e., max′f(,′)−min′f(′,) _y f(x,y )- _x f(x ,y), can achieve (ϵ−p−1)O(ε^- pp-1). Under the NC-SC setting, [36] designs a nested-loop method that uses gradient clipping to update ty_t, and gradient normalization to update tx_t. The total gradient complexity of their method is (κ2p−1p−1ϵ−3p−2p−1)O(κ 2p-1p-1ε^- 3p-2p-1) to return a point x satisfying ‖∇Φ()‖≤ϵ\|∇ (x)\|≤ε, where κ is the condition number, and Φ()=Δmaxf(,) (x) = _yf(x,y). Note that their method requires a batch-size of (ϵ−p−1)O(ε^- pp-1). In the NC-PL setting, i.e., f(,⋅)f(x,·) is not necessarily μ-strongly concave w.r.t. y, but satisfies the Polyak–Łojasiewicz inequality, which relaxes strong concavity, [59] designs a single-loop method without gradient clipping. Their idea is to replace all gradients in SGDA with normalized momentum. They proves an upper bound (κ2p−1ϵ−2p−1)O(κ 2pp-1ε^- 2pp-1) for this method. Still in the NC-PL setting, [22] also proposes a single-loop method. Although their method uses clipped gradients to update tx_t and ty_t, they proved that the method achieves a tighter bound (κ3p−2p−1ϵ−3p−2p−1)O(κ 3p-2p-1ε^- 3p-2p-1), compared to the one presented in [59], and their method can be extended to the high-probability convergence as well as NC-C setting with the complexity (ϵ−6p−4p−1)O(ε^- 6p-4p-1). For Bilevel optimization problems which recover the min-max problems, [60] proposes new method to handle heavy-tailed noise, but they did not provide the upper bound result of their method for any specific min-max problem settings. We note also that [2] studies nonconvex stochastic variational inequalities (SVI), which include stochastic min-max problems, without assuming bounded variance. However, their assumptions on stochastic gradients are different from ours. Specifically, they assume the existence of B,σ>0B,σ>0 such that ξ[‖∇f(,ξ)−∇f()‖2]≤B2‖−0‖2+σ2E_ξ[ ∇ f(z,ξ)-∇ f(z) ^2]≤ B^2 -z_0 ^2+σ^2, where 0z_0 denotes the initial point of the algorithm. Overall, we find that existing studies on min-max problems under heavy-tailed noise reveal two limitations: i) they do not consider the convergence of classical gradient methods (such as SGDA) under heavy-tailed noise. i) they only study unregularized min-max problems, rather than Problem (1.2). Classic stochastic gradient methods under heavy-tailed noise We also review the existing work on vanilla gradient methods, e.g., SGD and stochastic mirror descent (SMD), for stochastic minimization problems, i.e., minf() _xf(x) under heavy-tailed noise. For convex optimization, [50, 50, 24, 13, 38, 41, 53, 40] have considered the convergence of classical SGD or SMD or their variants under heavy-tailed noise in certain settings, while this paper mainly focuses on progress in nonconvex optimization. For nonconvex and smooth optimization, [13] gave the first gradient complexity (ϵ−2p−1)O(ε -2pp-1) of mini-batch SGD for finding ϵε-stationary points of nonconvex and standard L-smooth functions, with batch-size (ϵ−2p−1)O(ε -2p-1) and stepsize η=1/2Lη=1/2L. In particular, they also considered Hölder smooth functions with curvature exponent ν∈(0,1]ν∈(0,1]. Through a more refined analysis, [40] showed that, for L-smooth functions, both SGD and SGD with momentumn (SGDM) can converge with the same complexity (ϵ−2p−1)O(ε -2pp-1), but with batch-size (1)O(1). Note that although [53] studied SGD and SGDM for nonconvex Hölder smooth functions under heavy-tailed noise, a key step in their analysis [53, Lemma 2.4] requires ν+1≤pν+1≤ p, which means that when ν=1ν=1, corresponding to standard L-smooth optimization, p must be 22, reducing to the bounded-variance assumption. For nonconvex and nonsmooth optimization, [41] studies nonconvex, nonsmooth, but G-Lipshchitz continuous objectives through the online-to-non-convex-conversation framework (O2NC), which was proposed by [11]. [62] shows that, for nonsmooth, nonconvex, and weakly convex optimization, SGD converges within a compact region with a complexity of (ϵ−2p−1)O(ε -2pp-1). Interestingly, they also shows that Clip-SGD converges over an unbounded region with the same complexity as SGD with bounded region. 2 Preliminaries Notations: We first introduce some standard notation in asymptotic analysis. Given two real numbers a and b, if there exists a positive numerical constant c such that a≤cba≤ cb or a≥cba≥ cb holds, we will express the inequality as a≲ba\; \;b or a≳ba\; \;b. If, in addtion, a≲ba\; \;b and a≳ba\; \;b are both hold, we say a≍ba b. If a satisfies a≲ba\; \;b, a≳ba\; \;b, or a≍ba b, we also use (b)O(b), Ω(b) (b), or Θ(b) (b) to represent a. Specifically, ~(b) O(b) represents (b)O(b) omitting some polynomial logarithmic factors related to b. In this paper, we discuss the standard inner product ⟨⋅,⋅⟩ ·,· in Euclidean space, and the norm ‖⋅‖ · represents the Euclidean norm, i.e., ‖⋅‖=⟨⋅,⋅⟩ · = ·,· . We denote (,η)B(z,η) as the closed ball ∣‖−‖≤η \x -z ≤η \. If we study a (Fréchet) differentiable function f(,)f(x,y), we denote ∇f(,)∇ f(x,y), ∇f(,) _xf(x,y), and ∇f(,) _yf(x,y) as the full gradient, the partial gradient w.r.t. x, and the partial gradient w.r.t. y, respectively. For a proper function r, we denote the (regular) subdifferential set of r at the point x as ∂r()∂ r(x). Given a closed convex set S, we denote Π(⋅) _S(·) as the projection operator onto S. Additionally, the following notations have been widely used in previous literature on min-max optimization: • We define the primal function Φ() (x) as maxf(,)−h() _y \f(x,y)-h(y) \ and Ψ() (x) as Φ()+r() (x)+r(x). • Given a x, if argminf(,)−h() _y \f(x,y)-h(y) \ is unqiue, we let ⋆()y (x) be the maxmizer of f(,)−h()f(x,y)-h(y). If tx_t is the t-th iteration point of a certatin algorithm, we let t⋆y_t be ⋆(t)y (x_t). • We let Δt=ΔΦ(t)−minΦ(t) _t = (x_t)- _x (x_t), Λt=ΔΨ(t)−minΨ(t) _t = (x_t)- _x (x_t) and δt=Δ‖t−t⋆‖ _t = _t-y_t . Through out this paper, we assume that the primal function Φ() (x) is lower bounded, i.e. infxΦ()>−∞ _x (x)>-∞. Moreover, we make following assumption Assumption 1. For function f:d×d↦f: d_x× d_y , we assume (a) f()f(z) is L-smooth. That is, for any 1,2∈d×dz_1,z_2∈ d_x× d_y, we have ‖∇f(1)−∇f(2)‖≤L‖1−2‖ ∇ f(z_1)-∇ f(z_2) ≤ L _1-z_2 , whcih indicates the following inequality |∇f(1)−∇f(2)−⟨∇f(2),1−2⟩|≤L2‖1−2‖2 |∇ f(z_1)-∇ f(z_2)- ∇ f(z_2),z_1-z_2 |≤ L2 _1-z_2 ^2 (2.1) (b) We assume that the stochastic oracle can access an unbiased partial gradient estimators, i.e., ξ[∇f(,,ξ)]=∇f(,)E_ξ[ _xf(x,y,ξ)]= _xf(x,y) and ξ[∇f(,,ξ)]=∇f(,)E_ξ[ _xf(x,y,ξ)]= _xf(x,y). (c) We assume that stochastic partial gradients have p-th bounded central moment (p-BCM) for p∈(1,2]p∈(1,2], which means ξ[‖∇f(,,ξ)−∇f(,)‖p],ξ[‖∇f(,,ξ)−∇f(,)‖p]≤σpE_ξ[ _xf(x,y,ξ)- _xf(x,y) ^p],E_ξ[ _yf(x,y,ξ)- _yf(x,y) ^p]≤σ^p, where 0<σ<+∞0<σ<+∞. 3 Convergence of SGDA under heavy-tailed noise 3.1 NC-SC min-max problems Algorithm 1 Stochastic Gradient Descent Ascent (SGDA) Input : Starting point (1,1)∈d×d(x_1,y_1)∈ d_x× d_y , step-sizes η,η>0 _x, _y>0 and batch-sizes B∈ℕ+B _+ . for t=1,2,…,Tt=1,2,\,…,T do % Case I: r=h≡0r=h≡ 0 Independently sample ξt,1,…,ξt,B∼ξ _t,1,…, _t,B _ξ to get the stochastic gradients ∇f(t,t,ξt,i)i=1B\∇ f(x_t,y_t, _t,i)\_i=1^B. Compute g,t=1B∑i=1B∇f(t,t,ξt,i)g_x,t= 1B _i=1^B _xf(x_t,y_t, _t,i) and g,t=1B∑i=1B∇f(t,t,ξt,i)g_y,t= 1B _i=1^B _yf(x_t,y_t, _t,i). Update t+1=t−ηg,t,t+1=t+ηg,t.x_t+1=x_t- _xg_x,t, _t+1=y_t+ _yg_y,t. % Case I: r(x)=δr(x)= _X and h()h(y) is a general proper, closed and convex function Sample ξt∼ξ _t _ξ to get the stochastic gradients ∇f(t,t,ξt)∇ f(x_t,y_t, _t). Update t+1=Π(t−η∇f(t,t,ξt)),t+1=proxηr(t+η∇f(t,t,ξt))x_t+1= _X (x_t- _x _xf(x_t,y_t, _t) ), _t+1=prox_ _yr (y_t+ _y _yf(x_t,y_t, _t) ). end for To analyze the convergence of SGDA (Algorithm 1) under heavy-tailed noise, we begin with the case: the non-regularized minimax problem, i.e. Problem 1.1, and we study such a problem in NC-SC setting. In NC-SC setting, we assume that f(,⋅)f(x,·) is μ-strongly concave w.r.t. y. This is the following assumption. Assumption 2. For any ∈dx∈ d_x, we assume that f(,⋅)f(x,·) is μ-strongly concave w.r.t. y, i.e., f(,1)−f(,2)≤⟨∇f(,2),1−2⟩−μ2‖1−2‖2,∀1,2∈d f(x,y_1)-f(x,y_2)≤ _yf(x,y_2),y_1-y_2 - μ2 _1-y_2 ^2,\; _1,y_2∈ d_y (3.1) Going further, if we also assume that f is L-smooth and μ-strongly concave, then we define κ=ΔL/μκ =L/μ as the condition number of f. Note that κ≥1κ≥ 1. If f(,⋅)f(x,·) is μ-strongly concave w.r.t. y, then for any ∈dx∈ d_x, the optimal solution set of maxf(,) _yf(x,y) must be a singleton. If, in addition, the sub-level set ∣Φ()≤α,α∈ℝ\x (x)≤α,α \ is compact, we have the following two key lemmas. Lemma 3.1. [10, Proposition 1.1] If Assumptions 1(a) and 2 hold, then for any 1,2∈dx_1,\,x_2∈ d_x, we have ‖⋆(1)−⋆(2)‖≤κ‖1−2‖ (x_1)-y (x_2) ≤κ _1-x_2 . Lemma 3.2. [10, Proposition 1.2] If Assumptions 1(a) and 2 hold, then Φ() (x) is differentiable over d d_x and ∇f(,⋆())=∇Φ() _xf(x,y (x))=∇ (x). Moreover, Φ() (x) is a 2κL2κ L-smooth function. Lemmas 3.2 means that we can use the gradient of primal function Φ() (x) to define ϵε-stationary point and the stochastic gradient oracle complexity of algorithms. Definition 3.1. For Problem (1.2), we assume that Assumptions 1(a) and 2 hold. Given ϵ>0ε>0 (usually required to be sufficiently small), we define #Grad(ϵ)\# Grad(ε) as the minimum number of the algorithm’s stochastic gradient oracle queries to achieve [(1/T)∑t=1T∥∇Φ(t)∥≤ϵ]E[(1/T) _t=1^T\|∇ (x_t)\|≤ε], where T is the number of iterations, and tx_t is the iteration points. Our theoretical results of SGDA under heavy tailed noise in NC-SC setting can be presented by the following theorem. Theorem 3.3. Let Assumption 1 (L-smoothness + heavy-tailed noise) and 2 (NC-SC Setting) hold. For SGDA (Algorithm 1) with Case I, if we set η≍1/L _y 1/L, η≍1/κ2L _x 1/κ^2L, and T≍maxκ2Δ1Lϵ2,(Lκδ1)2ϵ2,B≍max⌈κ4−p2(p−1)(Δ1L)2−p−1σp−1ϵ2p−1⌉,⌈κ4−p2(p−1)(Lδ1)2−p−1σp−1ϵ2p−1⌉,1 \ T \ κ^2 _1Lε^2, (Lκ _1)^2ε^2 \, \ B \ κ 4-p2(p-1)( _1L) 2-pp-1σ pp-1ε 2p-1 , κ 4-p2(p-1)(L _1) 2-pp-1σ pp-1ε 2p-1 ,1 \ with a given sufficiently small ϵ>0ε>0, then we have #Grad(ϵ)≲maxκ3p2(p−1)(σ⋅maxΔ1L,(Lδ1))p−1ϵ2p−1,κ2Δ1Lϵ2,(Lκδ1)2ϵ2. \# Grad(ε)\; \; \ κ 3p2(p-1)(σ· \ _1L,(L _1) \) pp-1ε 2pp-1, κ^2 _1Lε^2, (Lκ _1)^2ε^2 \. (3.2) The formal proof of Theorem 3.3 can be found in Appendix A.1. First, when p=2p=2, i.e., under the bounded variance assumption, the gradient complexity in Theorem 3.3 degenerates to (κ3(Lδ1σ)2Δ1ϵ−4)O(κ^3(L _1σ)^2 _1ε^-4), which matches the best known upper bound of SGDA type methods under the bounded variance assumption for NC-SC min-max optimization, e.g., [33, 52, 42, 25, 21]. Secondly, when p<2p<2, Theorem 3.3 show the first complexity result for SGDA under heavy-tailed noise and NC-SC setting. Theorem 3.3 shows that SGDA itself can converge under heavy-tailed noise without any modification. Specially, we found that the step size ratio η/η _y/ _x is independent of the noise parameter p and always keeps Θ(κ2) (κ^2) in p∈(1,2]p∈(1,2], which indicates that heavy-tailed noise does not actually affect the algorithm’s efficiency trade-off with respect to step sizes. For the lower bound of the first-order algorithms under heavy-taild noise in min-max optimization, let’s consider a specific objective f(,)=ΔF()−μ2‖2f(x,y) =F(x)- μ2\|y\|^2, where F()F(x) can be any standard smooth function with F(1)−minF()≤Δ1F(x_1)- _xF(x)≤ _1, and we have Φ()=maxF()−(μ/2)‖2=F() (x)= _y \F(x)-(μ/2)\|y\|^2 \=F(x). Then, for any stochastic oracle ∇f(,,ξ) _xf(x,y,ξ) satisfying the assumptions 1(b) and 1(c), we have ξ[∇f(,,ξ)]=∇f(,)=∇F()E_ξ[ _xf(x,y,ξ)]= _xf(x,y)=∇ F(x), which means that ∇f(,,ξ) _xf(x,y,ξ) is also an unbiased and heavy-tailed stochastic gradient estimator of ∇F()∇ F(x), and vice versa. Therefore, any first-order algorithm44 4 The first-order algorithms here refer to the first-order zero-respecting algorithms, see Definition 4 in [30]. for f can be seen as the first-order algorithm for F, regardless of the choices we make with η _y and ∇f(,,ξ) _yf(x,y,ξ). Note that f satisfies the NC-SC setting, which means that Ω(ϵ−3p−2p−1) (ε^- 3p-2p-1) in [37] as the lower bound of the first-order algorithm for solving non-convex smooth minimization problems can be used as a lower bound of the algorithm for solving the min-max problem under the NC-SC setting. However, due to the instance we used, we cannot determine the dependence of the lower bound of complexity on other parameters such as κ and σ. The following theorem extends the lower bound of variance-bounded noise provided by [30] to heavy-tailed noise. Theorem 3.4. Consider a min-max problem min∈max∈f(,) _x _y f(x,y), for any p∈(1,2]p∈(1,2], σ, Δ1 _1, κ, L and sufficiently small ϵ>0ε>0, there always exist X, Y, f(,)f(x,y) and a stochastic oracle ∇f(,,ξ)∇ f(x,y,ξ), where f(,)f(x,y) satisfies the Assumption 1(a), 2, and ∇f(,,ξ)∇ f(x,y,ξ) is unbiased and satisfies [‖∇f(,,ξ)−∇f(,)‖p]≤σpE[\|∇ f(x,y,ξ)-∇ f(x,y)\|^p]≤σ^p. Furthermore, Φ()−min∈Φ()≤Δ1 (0)- _x (x)≤ _1. For any first-order, zero-respecting algorithm A (see definition 4 in [30]) with initial point (,)(0,0), let tt≥1\x_t\_t≥ 1 be generated by A, then we have inftt∣[ℓΦ∥Π(t−ℓΦ−1∇Φ(t))−t∥]<ϵ≥Ω(Δ1Lκ13σp−1ε−3p−2p−1), _t \t [ _ \| _X(x_t- _ ^-1∇ (x_t))-x_t\| ]<ε \≥ ( _1Lκ 13σ pp-1 ^- 3p-2p-1 ), (3.3) where ℓΦ=cL _ =cL is the smoothness parameter of Φ , and ℓΦ _ differs from L by a numerical constant. Remark 3.1. We notice that the lower bound depends on the condition number κ is Ω(κ1/3) (κ^1/3), and its order is a fixed constant. This is because the proof in Appdenix A.3 follows the same framework based on zero-chain instance in [3, 30]. Under such a framework, the lower bound of the first-order (zero-respecting) algorithm is determined by “Effective length of the chain × Bound of p-th cenetral moment of the constructed oracle”. However, κ will only affect the effective length, rather than the bound of the moment, this causes the dependence on κ in lower bound we obtained cannot be unable to adapt to p. In fact, even for light-tailed noise, the currently best known dependence on κ is (κ)O(κ) for SAPD+ in [61], which is still far from the tightest known (κ1/3)O(κ^1/3) dependence provided by [30]. We expect to be able to close this gap in the future. Now, let’s have some discussion about the tightness of complexity (ϵ−2p/(p−1))O (ε^-2p/(p-1) ) in Theorem 3.3. We note that this complexity does not reach the lower bound Ω(ϵ−(3p−2)/(p−1)) (ε^-(3p-2)/(p-1) ) for general first-order algorithms when p<2p<2. However, this does not mean that the complexity (ϵ−2p/(p−1))O (ε^-2p/(p-1) ) is loose for SGDA. As we have mentioned, regardless of the choice of η _y and ∇f(,,ξ) _yf(x,y,ξ), the ϵε-complexity of SGDA with η _x for NC-SC min-max problems cannot be lower than the ϵε-complexity of SGD with η=η= _x for smooth minimization problems. Recently, we note that [13, 62, 40] have all studied the convergence of SGD under heavy-tailed noise for non-convex stochastic minimization problems. [13, 40] focus on nonconvex smooth fuctions while [62] focus on weakly-convex functions. Although these works consider different specific settings, they all show that the ϵε-complexity of SGD under heavy-tailed noise can only reach (ϵ−2p/(p−1))O(ε^-2p/(p-1)). Specifically, [13, Theorem 5.10] shows that the ϵε-complexity of SGD with η=1/2Lη=1/2L and B=(ϵ−2/(p−1))B=O(ε^-2/(p-1)) is upper bounded by (ϵ−2p/(p−1))O(ε^-2p/(p-1)) for L-smooth functions, and [13, Theorem 5.5] shows that the ϵε-complexity of SGD with η=1/t1/pη=1/t^1/p and B=1B=1 is lower bounded by Ω(ϵ−2p/(p−1)) (ε^-2p/(p-1)) for L-smooth functions55 5 Note that this lower bound does not apply to SGD with mini-batch, i.e., B is related to ϵε, because their oracle construction is designed for the single gradient. However, to the best of our knowledge, this lower bound is currently the only one specifically for SGD under heavy-tailed noise in nonconvex smooth optimization, rather than for general first-order algorithms.. Based on these previous works on SGD, we believe our ϵε-complexity (ϵ−2p/(p−1))O(ε^-2p/(p-1)) of SGDA in Theorem 3.3 is unimprovable. Remark 3.2. Here we point out that in the NC-SC setting, the need for mini-batch stochastic gradients in SGDA, is not a “preferential treatment” for heavy-tailed noise. In fact, even for light-tailed noise , mini-batch is commomly used for SGDA, or rather, achieving a complexity on the order of (ϵ−4)O(ε^-4) is required, e.g., [33, 52, 42, 25]. [33] can prove the convergence of SGDA with (1)O(1) batch-size in the NC-SC setting, but could only obtain a complexity of (ϵ−5)O(ε^-5). Although [54, 61, 8] developed the algorithms that can have the complexity of (ϵ−4)O(ε^-4) with (1)O(1) batch-size under light-tailed noise, Their algorithms differ from SGDA-type methods in that they do not compute partial gradients simultaneously, but instead use an alternating update scheme. In more detail, [8] and [54] used ∇f(t+1,t,ξt) _yf(x_t+1,y_t, _t) to update t+1y_t+1, while [61] used ∇f(t,t+1,ξt) _xf(x_t,y_t+1, _t) to update t+1x_t+1. To the best of our knowledge, no work has proven the complexity (ϵ−4)O(ε^-4) of SGDA with (1)O(1) batch-size and nonadaptive step-sizes even under light-tailed noise in the NC-SC setting; this problem has already been proposed by [33], and it still remains open to our knowledge. 3.2 NC-C min-max problems In this section, we study the following min-max problem min∈dmax∈df(,)−h()+δ(), _x∈ d_x _y∈ d_y \f(x,y)-h(y)+ _X(x) \, (1.2’) in which r()r(x) in Problem 1.2 is replaced by the indicator function δ:↦(−∞,+∞] _ X:X (-∞,+∞], where X is a nonempty, closed and convex set, while h()h(y) can still be a general proper, closed and convex function. We consider Problem (1.2’) under the NC-C setting, where f remains smooth w.r.t. x and y but is only required to be concave w.r.t. y, rather than strongly concave. Concretely, we let following Assumption 3 holds. Assumption 3. (a) We assume that dom(h)dom (h ) is bounded with diameter D_Y. (b) f(,⋅)f(x,·) is concave w.r.t. y for any ∈x , i.e., f(,1)≥f(,2)+⟨∇f(,2),1−2⟩f(x,y_1)≥ f(x,y_2)+ _yf(x,y_2),y_1-y_2 , where 1,2∈dom(h)y_1,y_2 (h ). (c) We assume that f(⋅,)f(·,y) is G-Lipschitz continious on X w.r.t. x for any ∈dom(h)y (h ). Remark 3.3. Assumption 3(a) guarantees that the optimal solution set of maxf(,)−h() _y \f(x,y)-h(y) \ is non-empty. otherwise. For example, let f()=Δ−1/+δ[1,+∞)()f(y) =-1/y+ _[1,+∞)(y), where δ[1,+∞)()=0 _[1,+∞)(y)=0 iff ∈[0,+∞)y∈[0,+∞). We have maxf()=0 _yf(y)=0, while the optimal solution set is empty. If we assume that f is L-smooth and sastisfies Assumption 3, we have following lemma: Lemma 3.5. [8, Proposition 3.1] Suppose that Assumptions 1(a) and 3 hold. Then, Φ() (x) is Fréchet subdifferentialble and ∇f(,⋆)∣⋆∈argmaxf(,)−h()⊂∂Φ() \ _xf(x,y ) ∈ _y \f(x,y)-h(y) \ \⊂∂ (x) for any ∈dx∈ d_x. Moreover, Φ(x) (x) is L-weakly convex on X, i.e., for any ∈x , Φ()+(L/2)‖2 (x)+(L/2)\|x\|^2 is convex, which is equivalent to saying that Φ(x1)−Φ(2)≥⟨gΦ(2),1−2⟩−(L/2)‖1−2‖2 (x_1)- (x_2)≥ g_ (x_2),x_1-x_2 -(L/2)\|x_1-x_2\|^2 holds for any 1,2∈x_1,x_2 and gΦ(2)∈∂Φ(2)g_ (x_2)∈∂ (x_2). For a weakly convex function, we can define its Moreau envelope as following: Φρ()=ΔΦ(+),where+=Δargmin′∈Φ(′)+(ρ/2)‖′−‖,∀ρ>L. _ρ(x) = (x^+),\;where\;x^+ = _x \ (x )+(ρ/2)\|x -x\| \,∀ρ>L. By [12, Section 1], Φρ() _ρ(x) is smooth with gradient ∇Φρ()=ρ(−+)∇ _ρ(x)=ρ(x-x^+). Moreover, let (+) N_X (x^+ ) is the normal cone of X defined on +x^+, ‖∇Φρ()‖≤ϵ\|∇ _ρ(x)\|≤ε implies that dist(0,(+)+∂Φ(+))≤ϵdist (0, N_X (x^+ )+∂ (x^+) )≤ε and ‖+−‖≤ϵ/ρ\|x^+-x\|≤ε/ρ, which means we can use ∇Φρ()∇ _ρ(x) to define convergence. Specifically, we have following definition: Definition 3.2. For Problem (1.2’), we assume that Assumptions 1(a) and 3 hold. Given ϵ>0ε>0 (usually required to be sufficiently small), we define #Grad(ϵ,ρ)\# Grad(ε,ρ) as the minimum number of the algorithm’s stochastic gradient oracle queries to achieve [(1/T)∑t=1T∥∇Φρ(t)∥≤ϵ]E[(1/T) _t=1^T\|∇ _ρ(x_t)\|≤ε], where T is the number of iterations, and tx_t is the iteration points. Our theoretical results of SGDA under heavy tailed noise in NC-C setting can be presented by the following theorem. Theorem 3.6. Let Assumptions 1 (L-smoothness + heavy-tailed noise) and 3 (NC-C Setting) hold. For SGDA (Algorithm 1) with Case I, if η≍min(Δ1/TL)2p−13p−2(Dσ)p3p−2(Gσ∨G2)p−13p−2,Δ1/TL(G2∨σ2∨Gσ),η≍minD3p−43p−2σ3p3p−2(Δ1(Gσ∨G2)TL)13p−2,12L, _x \ ( _1/TL) 2p-13p-2(D_Yσ) p3p-2(Gσ G^2) p-13p-2, _1/TL(G^2 σ^2 Gσ) \, _y \ D_Y 3p-43p-2σ 3p3p-2 ( _1(Gσ G^2)TL ) 13p-2, 12L \, where a∨ba b represents maxa,b \a,b\, then we have #Grad(ϵ,2L) \# Grad(ε,2L) ≲Δ1(Gσ∨σ2)(Dσ)p−1L2p−1p−1ϵ6p−4p−1+L7p−52p−1D5p−42p−1σp2p−1(Δ1(G2∨Gσ)ϵ12p−82p−1) \; \; _1(Gσ σ^2)(D_Yσ) pp-1L 2p-1p-1ε 6p-4p-1+ L 7p-52p-1D_Y 5p-42p-1σ p2p-1 ( _1(G^2 Gσ)ε 12p-82p-1 ) +(σϵ)p−1+Δ1L(G2∨σ2∨Gσ)ϵ4+Lδ^1ϵ2, + ( σε ) pp-1+ _1L(G^2 σ^2 Gσ)ε^4+ L δ_1ε^2, (3.4) where δ^1=ΔΦ(1)−f^(1,1) δ_1 = (x_1)- f(x_1,y_1). If, in addition, we have σ≥G∨1σ≥ G 1, L,D≥1L,D_Y≥ 1 and ϵ≤1ε≤ 1, we can further get #Grad(ϵ,2L)≲Δ1Dpp−1L2p−1p−1(σ3p−2p−1∨G2σp−1)ϵ6p−4p−1+(σϵ)p−1+Δ1L(G2∨σ2)ϵ4+Lδ^1ϵ2. \# Grad(ε,2L)\; \; _1D_Y pp-1L 2p-1p-1(σ 3p-2p-1 G^2σ pp-1)ε 6p-4p-1+ ( σε ) pp-1+ _1L(G^2 σ^2)ε^4+ L δ_1ε^2. (3.5) The proof of Theorem 3.6 can be found in Appdenix A.2. Here we point out that conditions σ≥G∨1σ≥ G 1, L,D≥1L,D_Y≥ 1 and ϵ≤1ε≤ 1 in Theorem 3.6 is mild and is used only to simplify the upper bound. When p=2p=2, the complexity (Δ1D2L3(σ4∨G2σ2)ϵ−8)O( _1D_Y^2L^3(σ^4 G^2σ^2)ε^-8) mathches [34]’s result for light-tailed noise; when p<2p<2, Theorem 3.6 provides the first complexity upper bound for SGDA under heavy-tailed noise in the NC-C setting. NC-C. Reg. Metric (E,ℙP) #Grad(ϵ,2L)\# Grad(ε,2L) SGDAClip_ Clip [22] (ℝdR^d,δ _Y) 1T∑t=1T‖∇Φ1/2L(t)‖2 1T _t=1^T\|∇ _1/2L(x_t)\|^2 ( ✓, ✓) ((Δ1+δ^1)L2p−1p−1Dpp−1(G2σp−1+σp2p−1)ϵ6p−4p−1)O ( ( _1+ δ_1)L 2p-1p-1D_Y pp-1(G^2σ pp-1+σ p^2p-1)ε 6p-4p-1 ) SGDA (Theorem 3.6) (δ _X,h) (1T∑t=1T‖∇Φ1/2L(t)‖2)p/2 ( 1T _t=1^T\|∇ _1/2L(x_t)\|^2 ) p2 ( ✓, ✗) (Δ1L2p−1p−1Dpp−1(G2σp−1∨σ3p−2p−1)ϵ6p−4p−1+Lδ^1ϵ2)O ( _1L 2p-1p-1D_Y pp-1(G^2σ pp-1 [rgb]0,0,1σ 3p-2p-1)ε 6p-4p-1+ [rgb]0,0,1 L δ_1ε^2 ) Table 1: Comparison of the results of vanilla SGDA in this paper and SGDAClip_ Clip in [22] using gradient clipping. “(a,b)(a,b)” in the “Reg.” column denotes the regularizer a for x and the regularizer b for y. “Metric” denotes the convergence metric, “E” denotes convergence in expectation, and “ℙP” denotes convergence with high probability. For the gradient complexity, We note that although the dependence on ϵε is the same for both methods, our dependence on σ and δ^1 δ_1 is tighter. Comparison with related work To the best of our knowledge, [22] is currently the only work that studies minimax optimization under heavy-tailed noise in the NC-C setting. The method they developed, SGDAClip_ Clip , can be viewed as Stoc-AGDA [54] equipped with gradient clipping. That is, t+1=t−∇^f(t,t,ξt),t+1=t−∇^f(t+1,t,ξt)x_t+1=x_t- ∇_xf(x_t,y_t, _t),y_t+1=y_t- ∇_yf(x_t+1,y_t, _t), where ∇^f(t,t,ξt) ∇_xf(x_t,y_t, _t) and ∇^f(t+1,t,ξt) ∇_yf(x_t+1,y_t, _t) are both the clipped gradients. Note that this is somewhat different from SGDA’s strategy of simultaneously updating partial gradients. They show that this method converges with an ϵε-complexity of (ϵ−6p−4p−1)O(ε^- 6p-4p-1). It is worth noting that this complexity is the same as the ϵε-complexity we obtain for the vanilla SGDA without using gradient clipping and gradient normalization. In particular, compared with [22], we do not introduce any additional assumptions. As can be seen from Table 1, for min-max optimization under the NC-C setting, gradient clipping does not improve the complexity but instead improves the convergence metric; that is, compared with SGDA, SGDAClip_ Clip can achieve high-probability convergence w.r.t. (1/T)∑t=1T‖∇Φ1/2L(t)‖2(1/T) _t=1^T\|∇ _1/2L(x_t)\|^2. Note that [22, Proposition 4.7] proved that even for SGDA under the NC-SC setting, the dependence of the complexity on the failure probability δ under heavy-tailed noise cannot be polylogarithmic. 3.3 Proof sketches of Theorem 3.3 and Theorem 3.6 Now let’s explain how we analyze the convergence of SGDA under heavy-tailed noise. Weaker convergence metrics Inspired by [40] on stochastic minimization problems, we replaced the convergence metric from (1/T)∑t=1T[‖∇Φ(t)‖2](1/T) _t=1^TE[\|∇ (x_t)\|^2] to a weaker one, [((1/T)∑t=1T‖∇Φ(t)‖2)p/2]E[((1/T) _t=1^T\|∇ (x_t)\|^2)^p/2] (In the NC-C setting, We will replace ‖∇Φ(t)‖\|∇ (x_t)\| with ‖∇Φ1/2L(t)‖\|∇ _1/2L(x_t)\|.). Note that, by the Jensen’s inequality, we have ((1/T)∑t=1T[‖∇Φ(t)‖])p≤(1/T)∑t=1T[‖∇Φ(t)‖p]≤[((1/T)∑t=1T‖∇Φ(t)‖2)p/2] ((1/T) _t=1^TE[\|∇ (x_t)\|] )^p≤(1/T) _t=1^TE[\|∇ (x_t)\|^p] [((1/T) _t=1^T\|∇ (x_t)\|^2)^p/2]. Therefore, if [((1/T)∑t=1T‖∇Φ(t)‖2)p/2]≤ϵpE[((1/T) _t=1^T\|∇ (x_t)\|^2)^p/2]≤ε^p holds, it implies that (1/T)∑t=1T[‖∇Φ(t)‖]≤ϵ(1/T) _t=1^TE[\|∇ (x_t)\|]≤ε also holds. Our analysis also relies on such a lemma: for a given martingale difference sequence Xkk=1N\X_k\_k=1^N w.r.t. filterations ℱtt≥0 \F_t \_t≥ 0, if [|Xi|p]<∞E [|X_i|^p ]<∞, we have [|∑k=1NXk|p]≲∑k=1N[|Xk|p]E [ | _k=1^NX_k |^p ]\; \; _k=1^NE [|X_k|^p ]. (A more formal statement can be found in Lemma C.4) We noticed that the vector version extended from this lemma, i.e., [‖∑i=1nXi‖p]≲∑i=1n[‖Xi‖p]E [ \| _i=1^nX_i \|^p ]\; \; _i=1^nE [\|X_i\|^p ], has been proposed by [27, 26, 37], primarily used to analyze the upper bound of p-th central moments of mini-batch stochastic gradients. Differently, in our analysis, we utilize Lemma C.4 to analyze the upper bound of the sum of martingale sequences. For convenience, we additionally introduce the following notation: • For SGDA with Case I, we denote ζ,t _x,t and ζ,t _y,t as g,t−∇f(t,t)g_x,t- _xf(x_t,y_t), g,t−∇f(t,t)g_y,t- _yf(x_t,y_t) respectively, and let p=Δmaxt∈1,2,…,T[‖ζ,t‖p],[‖ζ,t‖p] σ^p = _t∈\1,2,…,T\ \E [ _x,t ^p ],E [ _y,t ^p ] \, and ℱt=Δσ(ξ¯1,…,ξ¯t)F_t =σ( ξ_1,…, ξ_t), where ξ¯t=Δξt,1,…,ξt,B ξ_t = _t,1,…, _t,B. • For SGDA with Case I, we denote ζ,t _x,t and ζ,t _y,t as g∇f(t,t,ξt)−∇f(t,t)g _xf(x_t,y_t, _t)- _xf(x_t,y_t), ∇f(t,t,ξt)−∇f(t,t) _yf(x_t,y_t, _t)- _yf(x_t,y_t) respectively. Moreover, we let f^(,) f(x,y) be f(,)−h()f(x,y)-h(y) and δ^t δ_t be Φ(t)−f^(t,t) (x_t)- f(x_t,y_t). We let ℱt=Δσ(ξ1,…,ξt)F_t =σ( _1,…, _t). Here, we rule ℱ0=Δ∅F_0 =\ \. Proof sketch of Theorem 3.3 Since Φ is 2κL2κ L-smooth, starting from the qudratic upper bound of the smooth function, we can then obtain ∑t=1Tη‖∇Φ(t)‖2≲∑t=1Tη(2κLη−1)⟨∇Φ(t),ζ,t⟩+Δ1+∑t=1Tη2‖ζ,t‖2+∑t=1Tηδt2 _t=1^T _x\|∇ (x_t)\|^2\; \; _t=1^T _x(2κ L _x-1) ∇ (x_t), _x,t + _1+ _t=1^T _x^2\| _x,t\|^2+ _t=1^T _x _t^2. Here we ignore κ and L without affecting the convergence analysis. Note that at this point we cannot absorb ⟨∇Φ(t),ζ,t⟩ ∇ (x_t), _x,t by taking the expectation, because [‖ζ,t‖2]E[\| _x,t\|^2] could be infinite. We first raise both sides of the inequality to the power of p/2p/2, and then take the expectation to obtain [(∑t=1Tη‖∇Φ(t)‖2)p/2]≲[(|∑t=1Tη(2κLη−1)⟨∇Φ(t),ζ,t⟩|)p/2]⏟+Δ1p/2+ηpTp+[(∑t=1Tηδt2)p/2]. [( _t=1^T _x\|∇ (x_t)\|^2)^p/2]\; \; E[(| _t=1^T _x(2κ L _x-1) ∇ (x_t), _x,t |)^p/2]_ M+ _1^p/2+ _x^pT σ^p+E[( _t=1^T _x _t^2)^p/2]. For M, note that ⟨∇Φ(t),ζ,t⟩t≥0\ ∇ (x_t), _x,t \_t≥ 0 is indeed a martingale difference sequence w.r.t. ℱtt≥0\F_t\_t≥ 0, using Lemma C.4 and Hölder’s inequality, we can get ≲(1/4)[(∑t=1Tη‖∇Φ(t)‖2)p/2]+pT(2−p)/pηp/2 M\; \;(1/4)E[( _t=1^T _x\|∇ (x_t)\|^2)^p/2]+ σ^pT^(2-p)/p _x^p/2 We now determine the upper bound of [(∑t=1Tηδt2)p/2]E[( _t=1^T _x _t^2)^p/2] by analyzing a one-step stochastic gradient ascent on f(t,⋅)f(x_t,·). We point out that the techniques used to analyze the convergence of SGDA under light-tailed noise are difficult to extend to our setting. Specifically, they all first obtain [δt2]≲γ[δt−12]+η2[‖∇Φ(t−1)‖2]+η22+2E[δ^2_t]\; \; [δ^2_t-1]+ _x^2E[\|∇ (x_t-1)\|^2]+ _x^2 σ^2+ σ^2, where γ≤1γ≤ 1, and then obtain the upper bound of [δt2]E[ _t^2] by recursively applying the above inequality, thus obtaining the upper bound of its sum. However, we were unable to control [‖ζ,t‖2]E[\| _x,t\|^2] and [‖ζ,t‖2]E[\| _y,t\|^2] from the beginning of the analysis due to the heavy-tailed noise. In fact, we first obtained the inequality ηδt2≤βηδt−12+Bt−1+θXt+αYt−1, _x _t^2≤β _xδ^2_t-1+B_t-1+θ X_t+α Y_t-1, where Xt−1=Δ⟨∇Φ(t−1),ζ,t−1⟩,Yt−1=Δ⟨ζ,t−1,t−1−t−1⋆⟩X_t-1 = ∇ (x_t-1), _x,t-1 ,Y_t-1 = _y,t-1,y_t-1-y _t-1 and the definition of β,Bt−1,αβ,B_t-1,α and θ can be seen in (A.14). Then, We recursively apply the obtained inequality, and then raise both sides of the inequality to the power of p/2p/2, and take the expectation to get [(∑t=1Tηδt2)p/2] [( _t=1^T _x _t^2)^p/2] ≲(δ12η)p/2+[(∑t=1Tη‖∇Φ(t)‖2)p/2]+Tηp/2p \; \; ( _1^2 _y )^p/2+E[( _t=1^T _x\|∇ (x_t)\|^2)^p/2]+T _x^p/2 σ^p +[|∑t=1T−1(∑s=0T−1−tβs)θXt|p/2]+[|∑t=1T−1(∑s=0T−1−tβs)αYt|p/2]. +E[| _t=1^T-1( _s=0^T-1-tβ^s)θ X_t|^p/2]+E[| _t=1^T-1( _s=0^T-1-tβ^s)α Y_t|^p/2]. Next, since that Xtt≥1\X_t\_t≥ 1 and Ytt≥1\Y_t\_t≥ 1 are both martingale sequences w.r.t. ℱtt≥0\F_t\_t≥ 0, by Lemma C.4 again, we can get following inequality: [|∑t=1T−1(∑s=0T−1−tβs)θXt|p/2]+[|∑t=1T−1(∑s=0T−1−tβs)αYt|p/2] \ E[| _t=1^T-1( _s=0^T-1-tβ^s)θ X_t|^p/2]+E[| _t=1^T-1( _s=0^T-1-tβ^s)α Y_t|^p/2] ≲[(∑t=1Tη‖∇Φ(t)‖2)p/2]+[(∑t=1Tηδt2)p/2]+(θp+(α/L)p)pT1−p/2((1−β)η)p. \ \; \;E[( _t=1^T _x\|∇ (x_t)\|^2)^p/2]+E[( _t=1^T _x _t^2)^p/2]+ (θ^p+(α/L)^p) σ^pT^1-p/2((1-β) _x)^p. This leads to [(∑t=1Tηδt2)p/2]≲[(∑t=1Tη‖∇Φ(t)‖2)p/2]+(δ12η)p/2+ηp/2Tp+T1−p/2pηp/2. [( _t=1^T _x _t^2)^p/2]\; \;E[( _t=1^T _x\|∇ (x_t)\|^2)^p/2]+ ( _1^2 _y )^p/2+ _x^p/2T σ^p+T^1-p/2 σ^p _x^p/2. Putting these pieces together, we finally have [((1/T)∑t=1T‖∇Φ(t)‖2)p/2]≲1(Tη)p/2+ηp/2T1−p/2p+T1−p/2p+pTp−1. [((1/T) _t=1^T\|∇ (x_t)\|^2)^p/2]\; \; 1(T _x)^p/2+ _x^p/2T^1-p/2 σ^p+T^1-p/2 σ^p+ σ^pT^p-1. Note that if we set B=1B=1, i.e., =σ σ=σ, the variance term T1−p/2pT^1-p/2 σ^p cannot be controlled, which is why we use mini-batches because =σ/B(p−1)/p σ=σ/B^(p-1)/p. Proof sketch of Theorem 3.6 First, we attempt to derive a descent inequality for envelope Φ1/2L(t) _1/2L(x_t) of primal function Φ(t) (x_t) under heavy-tailed noise. This part of the analysis is similar to that of SGD under heavy-tailed noise for weakly convex (stochastic) optimization. We note that [62] obtained the first complexity upper bound for SGD in weakly convex optimization, but they considered optimization problems with constraints given by a compact convex set. In our setting, however, we only require the constraint region for x to be a general closed and convex set, without requiring it to be bounded. Therefore, [62]’s technique cannot be used to the analysis of Theorem 3.6. Similar to the analysis of Theorem 3.3, we first sum the descent inequality, then raise both sides to the power of p/2p/2, and finally take the expectation of the resulting inequality to obtain [(∑t=1Tη‖∇Φ1/2L(t)‖2)p/2] E [ ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ] ≲[(|∑t=1Tη⟨∇Φ1/2L(t),ζ,t⟩|)p/2]⏟′ \; \; E [ ( | _t=1^T _x ∇ _1/2L(x_t), _x,t | ) p2 ]_ M +(Lη)p/2[(∑t=1Tδ^t)p/2]+Δ1p/2++(TL)p/2ηp(Gp+σp), +(L _x) p2E [ ( _t=1^T δ_t ) p2 ]+ _1 p2++(TL) p2 _x^p(G^p+σ^p), where f^(,)=Δf(,)−h() f(x,y) =f(x,y)-h(y) and δ^t=ΔΦ(t)−f^(t,t) δ_t = (x_t)- f(x_t,y_t). For ′ M , note that η⟨∇Φ1/2L(t),ζ,t⟩t≥1\ _x ∇ _1/2L(x_t), _x,t \_t≥ 1 is a martingale difference sequence w.r.t. ℱtt≥0\F_t\_t≥ 0, where ℱt=Δσ(ξ1,…,ξt)F_t =σ( _1,…, _t). Therefore, by applying Lemma C.4 and Hölder’s inequality, we obtain: ′≲[(∑t=1Tη‖∇Φ1/2L(t)‖2)p/2]+σpT1−p/2ηp/2 M \; \;E [ ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ]+σ^pT^1-p/2 _x p2 For the function value gap [(∑t=1Tδ^t)p/2]E [( _t=1^T δ_t) p2 ], we first obtain the following inequality [δ^t]≲[Φ(t−1)−f^(t−1,)]+[‖t−1−‖2−‖t−‖2]η+[f^(t,t)−f^(t−1,t−1)] \ E [ δ_t ]\; \;E [ (x_t-1)- f(x_t-1,y) ]+ E [\|y_t-1-y\|^2-\|y_t-y\|^2 ] _y+E [ f(x_t,y_t)- f(x_t-1,y_t-1) ] +D2−pηp−1σp+G(σ+G)η \ +D_Y^2-p _y^p-1σ^p+G(σ+G) _x (3.6) for any ℱt−2F_t-2-measurable y. Note that if we let =t−1⋆y=y_t-1 , then we cannot bound ∑t=1T[‖t−1−t−1⋆‖2−‖t−t−1⋆‖2] _t=1^TE[\|y_t-1-y_t-1 \|^2-\|y_t-y_t-1 \|^2] by telescope summation. Inspired by [34], we partition the sequence δ^tt=1T\ δ_t\_t=1^T into N subsequences δ^nM+jj=1M\ δ_nM+j\_j=1^M, where n=0,…,N−1n=0,…,N-1. For the n+1n+1-th subsequence δ^nM+jj=1M\ δ_nM+j\_j=1^M, let =nM+1⋆y=y _nM+1, then obtain: ∑t=1T[δ^t]=∑n=1N∑j=1Mδ^nM+j≤δ^1+(M+1)TG(σ+G)η+TD2−pηp−1σp+TD22Mη. _t=1^TE [ δ_t ]= _n=1^N _j=1^M δ_nM+j≤ δ_1+(M+1)TG(σ+G) _x+TD_Y^2-p _y^p-1σ^p+ TD_Y^22M _y. Here we can set M=⌈D/2G(G+σ)ηη⌉M= D_Y 2G(G+σ) _x _y to further simplify the upper bound. Note that [(∑t=1Tδ^t)p/2]≤(∑t=1T[δ^t])p/2E[( _t=1^T δ_t) p2]≤ ( _t=1^TE[ δ_t] ) p2 for p∈(1,2]p∈(1,2]. Finally, by combining all the pieces we obtained and choosing the appropriate step sizes, we complete the proof of Theorem 3.6. 4 (Stoc-)TRGDA for NC-SC regularized min-max problems Since we have already shown that under heavy-tailed noise, SGDA can only achieve complexity (ϵ−2p−1)O(ε^- 2pp-1) in the NC-SC setting, there is clearly a gap compared with the lower bound Ω(ϵ−3p−2p−1) (ε^- 3p-2p-1). Therefore, the goal of this section is to design algorithms that can converge with complexity (ϵ−3p−2p−1)O(ε^- 3p-2p-1) under heavy-tailed noise. In particular, it can handle the regularized minimax problem, i.e. h and r in Problem 1.2 are both generally proper, closed and convex functions. Additionally, we do not wish to use modules such as gradient clipping that require introducing an additional hyperparameter. 4.1 How to incorporate normalized gradient into SPGD? To better understand how we design the algorithm, we first consider the composite minimization, i.e., minf()+r() _x\f(x)+r(x)\. Furthermore, we hope to incorporate gradient normalization technique into stochastic proximal gradient method (SPGD). We can do this in the following two ways: • Proximal normalized gradient. A natural way is to consider t+1=proxηtr(t−η∇f(t,ξt)‖∇f(t,ξt)‖)x_t+1=prox_ _tr (x_t- η∇ f(x_t, _t)\|∇ f(x_t, _t)\| ). However, this way is naive even for deterministic settings because it would violate the fixed-point property of the proximal operator. That is, let ⋆=argminf()+r()x_ = _x\f(x)+r(x)\, we have ⋆≠proxηtr(⋆−η∇f(⋆)‖∇f(⋆)‖)x_ _ _tr (x_ - η∇ f(x_ )\|∇ f(x_ )\| ), which precludes the sequence of iteration points from becoming stationary around ⋆x_ . • Normalized gradient mapping. Another way of incorporation is to consider the normalized gradient mapping, i.e., t+1=t−η(t)/‖(t)‖x_t+1=x_t- (x_t)/\|G(x_t)\|, where (t)=Δηt−1(t−~t+1),~t+1=Δproxηtr(t−ηt∇f(t,ξt))G(x_t) = _t^-1(x_t- x_t+1), x_t+1 =prox_ _tr (x_t- _t∇ f(x_t, _t) ). We still first consider the deterministic setting, i.e., ∇f(t,ξt)=∇f(t)∇ f(x_t, _t)=∇ f(x_t). Assuming that f is smooth and nonconvex, we then have f(t+1)+r(t+1) f(x_t+1)+r(x_t+1) ≤f(t)+r(t+1)−η⟨∇f(t),(t)‖(t)‖⟩+Lη22 ≤ f(x_t)+r(x_t+1)-η ∇ f(x_t), G(x_t)\|G(x_t)\| + Lη^22 =f(t)−η|(t)|+η⟨gr(~t+1),(t)‖(t)‖⟩+r(t+1)+Lη22 =f(x_t)-η\|G(x_t)\|+η g_r( x_t+1), G(x_t)\|G(x_t)\| +r(x_t+1)+ Lη^22 ≤f(t)+r(t)−η‖(t)‖−η⟨gr(t+1)−gr(~t+1),(t)‖(t)‖⟩+Lη22, ≤ f(x_t)+r(x_t)-η\|G(x_t)\| [rgb]0,0,1-η g_r(x_t+1)-g_r( x_t+1), G(x_t)\|G(x_t)\| + Lη^22, (4.1) where gr(t+1)∈∂r(t+1)g_r(x_t+1)∈∂ r(x_t+1) and gr(~t+1)∈∂r(~t+1)g_r( x_t+1)∈∂ r( x_t+1). The equality holds becasue (t)=∇f(t)+gr(~t+1)G(x_t)=∇ f(x_t)+g_r( x_t+1) by the optimality condition, and the last inequalty holds becasue r(t+1)−r(t)≤⟨gr(~t+1),t+1−t⟩+⟨gr(t+1)−gr(~t+1),t+1−t⟩r(x_t+1)-r(x_t)≤ g_r( x_t+1),x_t+1-x_t + g_r(x_t+1)-g_r( x_t+1),x_t+1-x_t by the convexity of r. We can regard −η‖(t)‖-η\|G(x_t)\| as the descent term, but for the blue term≤η‖gr(t+1)−gr(~t+1)‖≤η\|g_r(x_t+1)-g_r( x_t+1)\| it is difficult to upper-bound it when r is merely convex. Therefore, based on the above discussion, we did not adopt these two seemingly natural ways because they have analytical limitations even in deterministic settings. In fact, we are considering the following update: t+1∈argmin∈(t,ηt)r()+⟨∇f(t),⟩ _t+1∈ _x (x_t, _t)\r(x)+ ∇ f(x_t),x \ It is easy to observe that the above update is equivalent to replacing the quadratic term ηt‖−t‖2/2 _t\|x-x_t\|^2/2 in the proximal operator with a explicit constraint, i.e., ‖−t‖≤ηt\|x-x_t\|≤ _t, and it reduces to NGD when r is constant. We point out that the above update has actually been proposed by [28] for designing first-order methods for stochastic composite minimization problems, which they call trust-region gradient methods (TRGM). However, [28] did not focus on heavy-tailed noise; their main motivation was to understand orthogonal gradient methods serving matrix optimization, i.e., where tx_t is a 22D parameter66 6 Clearly, if we only consider composite minimization in a vector space under light-tailed noise, the stochastic proximal gradient method is already sufficient to solve the problem.. Consequently, they did not establish a connection between TRGM and NGD. For TRGM, its equivalence to NGD is obvious when r is constant. Actually, both proximal normalized gradient and normalized gradient mapping can reduce to NGD when r is constant. Here we emphasize that we are more interested in the case where r is a general nondifferentiable convex function. More importantly, they did not discuss the case where the optimal solution set for min∈(t,ηt)r()+⟨∇f(t),⟩ _x (x_t, _t)\r(x)+ ∇ f(x_t),x \ is a singleton. The following theorem answers these two questions. Theorem 4.1. Supposed that r:d↦(−∞,+∞]r: d (-∞,+∞] is a proper, closed and convex function. Given any ∈dom(r)z (r ), η>0η>0 and ∈dv∈ d, then ⋆†=Δargmin∈(,η)⟨,⟩+r()X _ = _x (z,η) \ ,x +r(x) \ must be a nonempty, closed and convex set. Furthermore, if ∉+∂r(†) 0 +∂ r(x ), then ‖†−‖=ηand∃gr(†)∈∂r(†),†=−η+gr(†)‖+gr(†)‖. \|x -z\|=η\;and\;∃ g_r(x )∈∂ r(x ),\;x =z-η v+g_r(x )\|v+g_r(x )\|. (4.2) If ∀†∈⋆† _ , we have ∉+∂r(†) 0 +∂ r(x ), then ⋆†X _ must be a singleton. The proof of Theorem 4.1 can be found in Appendix B.1. Let =∇f()v=∇ f(z), where f is L-smooth, if ∃†∈⋆† _ , such that ∈∇f()+∂r(†) 0∈∇ f(z)+∂ r(x ), then inequality dist(0,∇f(†)+∂r(†))≤‖∇f(†)−∇f()‖≤Lηdist (0,∇ f(x )+∂ r(x ) )≤\|∇ f(x )-∇ f(z)\|≤ Lη holds, which means that †x is a LηLη-stationary point of f+rf+r. Therefore, Theorem 4.1 tells us that TRGM can basically be seen as a computable approximation of t+1=t−ηt(∇f(t)+gr(t+1))/‖∇f(t)+gr(t+1)‖x_t+1=x_t- _t(∇ f(x_t)+g_r(x_t+1))\|∇ f(x_t)+g_r(x_t+1)\|. Inspired by TRGD, we can design a trust-region gradient descent ascent (TRGDA) method for regularized minimax optimization, i.e., t+1∈argmin∈(t,η)⟨∇f(t,t),⟩+r();t+1∈argmin∈(t,η)⟨−∇f(t,t),⟩+h(). casesx_t+1∈ _x (x_t, _x)\ _xf(x_t,y_t),x +r(x)\;\\ y_t+1∈ _y (y_t, _y)\ - _yf(x_t,y_t),y +h(y)\. cases If we replace the partial gradients with the stochastic partial gradients, then TRGDA becomes stochastic trust-region gradient descent ascent (Stoc-TRGDA). To the best of our knowledge, (Stoc-)TRGDA is novel. We present Stoc-TRGDA as a basic stochastic optimization framework, while our actual analysis focuses on its variants in the next section, such as replacing the stochastic gradient with Polyak momentum. Remark 4.1. We note some recent work with ideas similar to (Stoc-)TRGD. [18, 19, 20] developed Broximal Point Methods (BPM) for noncomposite optimization. BPM updates t+1∈argmin∈(t,ηt)f()x_t+1∈ _x (x_t, _t)f(x), and (Stoc-)TRGD can be viewed as its linear (stochastic) approximation when r is constant. [29] proposed a Trust-Region Proximal Point Method (TRPPM): t+1=argmin∈(t,ηt)f()+λt/2‖−t‖2x_t+1= _x (x_t, _t)\f(x)+ _t/2\|x-x_t\|^2\, which is equivalent to adding an extra quadratic term to BPM to guarantee the uniqueness of t+1x_t+1 when f is convex. It is worth noting that [19, Lemma 3.1] proves that when f is a differentiable convex function, BPM reduces to t+1=t−ηt∇f(t+1)/∥∇f(t+1)∥x_t+1=x_t- _t∇ f(x_t+1)/\|∇ f(x_t+1)\|77 7 Differently, TRGD reduces to t+1=t−ηt∇f(t)/∥∇f(t)∥x_t+1=x_t- _t∇ f(x_t)/\|∇ f(x_t)\| when r=0r=0. 4.2 Two variants of Stoc-TRGDA Algorithm 2 Stoc-TRGDA with Momentumn (Stoc-TRGDAM) Input : Starting point (1,1)∈dom(r)×dom(h)(x_1,y_1) (r )×dom (h ) , step-sizes η,η>0 _x, _y>0 and momentumn parameter β∈(0,1)β∈(0,1) for t=1,2,…,Tt=1,2,\,…,T do Sample ξt∼ξ _t _ξ to get the stochastic gradients ∇f(t,t,ξt)∇ f(x_t,y_t, _t). Compute [,t,t]=β[,t−1,t−1]+(1−β)[∇f(t,t,ξt)∇f(t,t,ξt)] bmatrixm_x,t\\ m_y,t bmatrix=β bmatrixm_x,t-1\\ m_y,t-1 bmatrix+(1-β) bmatrix _xf(x_t,y_t, _t)\\ _yf(x_t,y_t, _t) bmatrix (4.3) Choose any solution of min⟨,t,⟩+r()∣∈(t,η) _x \ _x,t,x +r(x) (x_t, _x) \ as t+1x_t+1. Choose any solution of min⟨−,t,⟩+h()∣∈(t,η) _y \ -m_y,t,y +h(y) (y_t, _y) \ as t+1y_t+1. end for Algorithm 3 Stochastic Trust-Region Gradient Descent and Ascent with Max-oracle (Stoc-TRGDAmax ) Input : Starting point (1,1)∈dom(r)×dom(h)(x_1,y_1) (r )×dom (h ) , step-sizes η,η>0 _x, _y>0 and batch size B∈ℕ+B _+ We set 0=1y_0=y_1 for t=1,2,…,Tt=1,2,\,…,T do t,1=t−1y_t,1=y_t-1. for k=1,2,…,Kk=1,2,\,…,K do Sample ξ^t,k∼ξ ξ_t,k _ξ to get the stochastic gradients ∇f(t,t,k,ξ^t,k) _yf(x_t,y_t,k, ξ_t,k). t,k+1=proxηh(t,k+η∇f(t,t,k,ξ^t,k))y_t,k+1=prox_ _yh (y_t,k+ _y _yf(x_t,y_t,k, ξ_t,k) ) end for t=t,Ky_t=y_t,K Independently sample ξt,1,…,ξt,B∼ξ _t,1,…, _t,B _ξ to get the stochastic gradients ∇f(t,t,ξt,i)i=1B\ _xf(x_t,y_t, _t,i)\_i=1^B. Compute g,t=1B∑i=1B∇f(t,t,ξt,i)g_x,t= 1B _i=1^B _xf(x_t,y_t, _t,i). Choose any solution of min⟨g,t,⟩+r()∣∈(t,η) _x \ g_x,t,x +r(x) (x_t, _x) \ as t+1x_t+1. end for In this section, we introduce our first algorithm: Algorithm 2, i.e., Stoc-TRGDAM. Compared to Stoc-TRGDA, Stoc-TRGDAM simply replaces the stochastic gradient with Polyak momentum. Note that when considering general closed convex regularizers r,hr,h. The conclusions in Lemma 3.1, 3.2 still hold, that is, ⋆()y (x) is a κ-smooth function w.r.t. x and the primal function Φ is a 2κL2κ L-smooth function w.r.t. x. Here, we redefine #Grad(ϵ)\# Grad(ε) as the minimum number of stochastic gradient queries required to reach [(1/T)∑t=1Tdist(0,∂r(t)+∇Φ(t))≤ϵ]E [(1/T) _t=1^Tdist (0,∂ r(x_t)+∇ (x_t) )≤ε ]. The following theorem demonstrates the upper-bound of gradient complexity of Stoc-TRGDAM . Theorem 4.2. Let Assumption 1 (L-smoothness + heavy-tailed noise) and 2 (NC-SC Setting) hold. We let ∂1 _1 be dist(0,∇Φ(1)+∂r(1))+5κdist(∇f(1,1),∂h(1))dist(0,∇ (x_1)+∂ r(x_1))+5 ( _yf(x_1,y_1),∂ h(y_1)). For Algorithm 2, we have following results: • If we konw Λ1 _1, L, σ and p, we let η5κ=η=Λ1(1−β)/LκTandβ=1−min1,max1Tp2p−1,(Λ1Lσ2T)p3p−2, _y5κ= _x= _1(1-β)/Lκ T\;and\;β=1- \1, \ 1T p2p-1, ( _1Lσ^2T ) p3p-2 \ \, then we have #Grad(ϵ)≲∂1ϵ+κ2Λ1Lϵ2+(κσ)2p−1p−1ϵ2p−1p−1+κ3p−2p−1Λ1Lσp−1ϵ3p−2p−1 \# Grad(ε)\; \; _1ε+ κ^2 _1Lε^2+ (κσ) 2p-1p-1ε 2p-1p-1+ κ 3p-2p-1 _1Lσ pp-1ε 3p-2p-1 (4.4) • If we don’t konw Λ1 _1, L, σ and p, we let η/5κ=η=1κT0.75 _y/5κ= _x= 1κ T^0.75 and β=1−1Tβ=1- 1 T, then we have #Grad(ϵ)≲∂1ϵ+(κΛ1+κL)4ϵ4+(κσϵ)2+(κσϵ)2p−1 \# Grad(ε)\; \; _1ε+ (κ _1+κ L)^4ε^4+ ( κσε )^2+ ( κσε ) 2pp-1 (4.5) The formal proof of Theorem 4.2 can be found in Appendix B.2. Knowing Λ1 _1, L, σ and p, the upper bound of the complexity of Stoc-TRGDAM is (ϵ−(3p−2)/(p−1))O(ε^-(3p-2)/(p-1)), which matches the lower bound Ω(ϵ−(3p−2)/(p−1)) (ε^-(3p-2)/(p-1)) for general first-order methods in Theorem 3.4. Not only that, the complexity is also compact with respect to Λ1 _1, L, and σ. To our knowledge, under smooth-strongly-concave setting setting, Stoc-TRGDAM is the first single-loop method with optimal complexity for stochastic regularized min-max optimization under heavy-tailed gradient noise in NC-SC setting. Specially, Stoc-TRGDAM does not require any techniques that rely on clipping gradients. Since that we have proven that SGDA can also converge under heavy-tailed noise, let’s compare the two. • Stoc-TRGDAM’s advantages over SGDA . First, from the perspective of ϵε-complexity, the advantage is obvious, since (ϵ−(3p−2)/(p−1))O(ε^-(3p-2)/(p-1)) is significantly smaller than (ϵ−2p/(p−1))O(ε^-2p/(p-1)) when 1<p<21<p<2, and SGDA requires a mini-batch to reach (ϵ−2p/(p−1))O(ε^-2p/(p-1)). Secondly, we find that when the parameters Λ1 _1, L, σ and p are unknown, which is common in practice, Stoc-TRGDAM can reach (ϵ−2p/(p−1))O(ε^-2p/(p-1)), consistent with SGDA . However, Theorem 3.3 shows that SGDA needs to adjust the batch size through p to achieve this complexity. • Stoc-TRGDAM’s disadvantages over SGDA . Although Stoc-TRGDAM has advantages over SGDA , it is not perfect. Regarding the dependence of the complexity on the condition number κ, (κ−(3p−2)/(p−1))O(κ^-(3p-2)/(p-1)) of Stoc-TRGDAM is smaller than (κ−3p/(2(p−1)))O(κ^-3p/(2(p-1))) of SGDA when p∈(1,4/3)p∈(1,4/3), whereas the opposite relationship holds for p∈[4/3,2]p∈[4/3,2]. We point out that this is because the trust region control on y prevents us from fully exploiting the strong concavity of f(,⋅)f(x,·) w.r.t. y in our analysis. The key challenge lies in establishing a contraction bound for the dual error ‖t−t⋆‖2\|y_t-y_t \|^2, analogous to that achieved in SGDA . Next, to improve the gradient complexity w.r.t. κ, we developed Algorithm 3 (Stoc-TRGDAmax ), inspired by the nested loop method SGDAmax. For updating the dual variable y, we use the stochastic proximal gradient method to return an approximate solution ty_t of t⋆y_t through K iterations like SGDAmax. Differently, we use trust-region gradient descent for updating the primal variable x. The following theorem demonstrates the upper bound of the gradient complexity of Stoc-TRGDAmax . Theorem 4.3. Let Assumption 1 (L-smoothness + heavy-tailed noise) and 2 (NC-SC Setting) hold. For Algorithm 3, we let η≍Δ1(T−1)κL _x _1(T-1)κ L. Moreover, given a ϵ>0ε>0, we let η≍minϵp−1μ(κσ)p−1,1μκp2(p−1) _y \ ε pp-1μ(κσ) pp-1, 1μκ p2(p-1) \ and B=max⌈(4σϵ)p−1⌉,1B= \ ( 4σε ) pp-1 ,1 \. • If ϵ<2Lδ1ε<2L _1, we let K≍ln(Lδ1ϵ)⋅maxσp−1κp−1ϵp−1,κp2(p−1),T≍maxΔ1κLϵ2,Lδ1ϵ, K ( L _1ε )· \ σ pp-1κ pp-1ε pp-1,κ p2(p-1) \, T \ _1κ Lε^2, L _1ε \, (4.6) then we have #Grad(ϵ)≲Lln(Lδ1ϵ)⋅maxδ1κp2p−2ϵ,Δ1κ3p−22p−2ϵ2,δ1(κσ)p−1ϵ2p−1p−1,Δ1κ2p−1p−1σp−1ϵ3p−2p−1; \# Grad(ε)\; \;L ( L _1ε )· \ _1κ p2p-2ε, _1κ 3p-22p-2ε^2, _1(κσ) pp-1ε 2p-1p-1, _1κ 2p-1p-1σ pp-1ε 3p-2p-1 \; (4.7) • If ϵ≥2Lδ1ε≥ 2L _1, we let K≍maxσp−1κp−1ϵp−1,κp2(p−1),T=Δ1κLϵ2, K \ σ pp-1κ pp-1ε pp-1,κ p2(p-1) \, T= _1κ Lε^2, (4.8) then we have #Grad(ϵ)≲maxΔ1Lκ3p−22p−2ϵ2,Δ1Lκ2p−1p−1σp−1ϵ3p−2p−1. \# Grad(ε)\; \; \ _1Lκ 3p-22p-2ε^2, _1Lκ 2p-1p-1σ pp-1ε 3p-2p-1 \. (4.9) The formal proof of Theorem 4.3 can be found in Appendix B.3. For p=2p=2 and ϵ<2Lδ1ε<2L _1, our result reduces to ~(κ3ϵ−4) O(κ^3ε^-4), which matches the light-tailed result [34] for SGDAmax. However, the difference is that [34] requires a batch size of (κσ2ϵ−2)O(κσ^2ε^-2) for updating tx_t and (kσ2ϵ−2)O(kσ^2ε^-2) for updating ty_t, whereas Stoc-TRGDAmax only requires a batch size of (σ2ϵ−2)O(σ^2ε^-2) for updating tx_t and (1)O(1) for updating ty_t. Note that (σ2ϵ−2)O(σ^2ε^-2) is smaller than (kσ2ϵ−2)O(kσ^2ε^-2) since κ≥1κ≥ 1. When p≤2p≤ 2, Stoc-TRGDAmax achieves a tighter dependence on κ compared to Stoc-TRGDAM . However, this improvement comes at the cost of increased space complexity. Specifically, a mini-batch is required for updating tx_t, and we can only obtain an upper bound without logarithmic factors when the initial point 1y_1 is sufficiently accurate such that ϵ≥2Lδ1ε≥ 2L _1. If we replace the update of tx_t in Stoc-TRGDAmax with the update of tx_t in Stoc-TRGDAM . we do not need mini-batch to guarantee convergence. However, this increases the number of outer iterations T to (ϵ3p−2p−1)O(ε 3p-2p-1), which is consistent with the iteration complexity of NSGD, while the number of inner iterations K remains unchanged, resulting in a worse complexity. Methods Settings Complexity Regularized Single loop No Clip SGDAClip_ Clip [22] NC-PL (κ3p−2p−1ϵ−3p−2p−1)O(κ 3p-2p-1ε^- 3p-2p-1) ✗ ✓ ( ✗, ✗) N2SBA [36] NC-SC ~(κ2p−1p−1ϵ−3p−2p−1) O(κ 2p-1p-1ε^- 3p-2p-1) ✗ ✗ ( ✓, ✗) Fed-NSGDA-M [59] NC-PL (κ2p−1ϵ−2p−1)O(κ 2pp-1ε^- 2pp-1) ✗ ✓ ( ✓, ✓) Stoc-TRGDAM (Ours) NC-SC (κ3p−2p−1ϵ−3p−2p−1)O(κ 3p-2p-1ε^- 3p-2p-1) ✓ ✓ ( ✓, ✓) Stoc-TRGDAmax (Ours) NC-SC ~(κ2p−1p−1ϵ−3p−2p−1) O(κ 2p-1p-1ε^- 3p-2p-1) ✓ ✗ ( ✓, ✓) Table 2: Summary of our method and previous methods on stochastic minimax optimization under heavy-tailed noise in NC-SC or NC-PL setting. The notation (⋅,⋅)(·,·) in the “No clip” column indicates whether the corresponding method employs gradient clipping for x and y, respectively. Comparison with related work We compare our method and its corresponding results with several recent methods for stochastic minimax problems under heavy-tailed noise, as summarized in Table 2. First, and importantly, all these previous works focus on nonregularized min-max problems. Second, for the single-loop methods SGDAClip_ Clip [22] and Fed-NSGDA-M [59], although they do not require the objective function to be strongly concave with respect to y and only assume the weaker PL condition, SGDAClip_ Clip still relies on gradient clipping for stochastic partial gradients. Moreover, the complexity (κ2p−1ϵ−2p−1)O(κ 2pp-1ε^- 2pp-1) of Fed-NSGDA-M is worse than the complexity (κ3p−2p−1ϵ−3p−2p−1)O(κ 3p-2p-1ε^- 3p-2p-1) of Stoc-TRGDAmax in terms of both κ and ϵε. For the nested-loop method N2SBA [36], its complexity ~(κ2p−1p−1ϵ−3p−2p−1) O(κ 2p-1p-1ε^- 3p-2p-1) matches that of Stoc-TRGDAmax . Moreover, N2SBA requires the same batch size (σp−1ϵ−p−1)O(σ pp-1ε^- pp-1) as Stoc-TRGDAmax . However, N2SBA still relies on clipping ∇f(,,ξ) _yf(x,y,ξ). Furthermore, we have shown that Stoc-TRGDAmax can achieve an upper complexity bound without the logarithmic factor when the initial point 1y_1 is sufficiently accurate. 4.3 Proof sketches of Theorems 4.2 and 4.3 Proof sketch of Theorem 4.2. To prove Theorem 4.2, we did not actually use the two-stage analysis used in proving Theorem 3.3, i.e., the convergence of SGDA under heavy-tailed noise, but instead utilized a potential function that we constructed as follows: Vt=Δ2Ψ(t)−F(t,t)+ηdist(0,∇Φ(t)+∂r(t))+ηdist(∇f(t,t),∂h(t)) _t =2 (x_t)-F(x_t,y_t)+ _xdist (0,∇ (x_t)+∂ r(x_t) )+ _ydist ( _yf(x_t,y_t),∂ h(y_t) ) (4.10) With such a potential function, we can have following descent inequality holds almost surely with η/η≍κ _y/ _x κ: Vt+1−Vt≲−ηdist(0,∇Φ(t+1)+∂r(t+1))+κη‖ζ,t‖+η‖ζ,t‖+κ2Lη2+κLη2, _t+1-V_t\; \;- _xdist (0,∇ (x_t+1)+∂ r(x_t+1) )+κ _x _y,t + _x _x,t +κ^2L _x^2+κ L _x^2, (4.11) where ζ,t=Δ,t−∇f(t,t) _x,t =m_x,t- _xf(x_t,y_t) and ζ,t=Δ,t−∇f(t,t) _y,t =m_y,t- _yf(x_t,y_t). Here we point out that the validity of (4.11) does not actually require the estimator to be unbiased; ζ,t _x,t and ζ,t _y,t simply defines the deviation between the true gradient and the gradient estimator (momentum in our case) we choose. Next, we sum the descent inequality and take the expectation. Then we only need to control the summations ∑t=1T[‖ζ,t‖] _t=1^TE[\| _x,t\|] and ∑t=1T[‖ζ,t‖] _t=1^TE[\| _y,t\|]. The analysis of this part is analogous to the analysis of NSGD in [37, 48, 26]. Proof skectch of Theorem 4.3. If Assumptions 1 and 2 hold, we can get following inequality: η[dist(0,∇Φ(t)+∂r(t))]≲ _xE [dist (0,∇ (x_t)+∂ r(x_t) ) ]\; \; [Φ(t−1)+r(t−1)]−[Φ(t)+r(t)] [ (x_t-1)+r(x_t-1) ]-E [ (x_t)+r(x_t) ] +Lη[‖t−1−t−1⋆‖]+ησB(p−1)/p+κLη2. +L _xE [ _t-1-y _t-1 ]+ _xσB^(p-1)/p+κ L _x^2. (4.12) Now, we only need to control ‖t−1−t−1⋆‖ _t-1-y _t-1 . If Assumptions 1 and 2 hold and η=min272p/(p−1)μ(ϵκσ)p−1,21p−1μκp2(p−1) _y= \ 272^p/(p-1)μ ( εκσ ) pp-1, 2 1p-1μκ p2(p-1) \, then we get following results: • If ϵ<2Lδ1ε<2L _1, η≲δ1κ _x\; \; _1κ, and K≍ln(Lδ1ϵ)⋅maxσp−1κp−1ϵp−1,κp2(p−1)K ( L _1ε )· \ σ pp-1κ pp-1ε pp-1,κ p2(p-1) \, then for any t=2,…,Tt=2,…,T, we have [‖t−t⋆‖]≲ϵ/LE[\|y_t-y_t \|]\; \;ε/L. • If δ1≤ϵ/2L _1≤ε/2L, η≲ϵκL _x\; \; εκ L, and K−1≍max(σκ)p−1ϵp−1,κp2(p−1)K-1 \ (σκ) pp-1ε pp-1,κ p2(p-1) \, then for any t=1,…,Tt=1,…,T, we have [‖t−t⋆‖]≲ϵ/LE[\|y_t-y_t \|]\; \;ε/L. The formal formulation of the above conclusion is Lemma B.7. 5 Conclusion and Future work In this work, we studied stochastic min-max optimization under heavy-tailed noise and characterized the convergence and complexity of gradient-based methods in both the nonconvex-strongly-concave (NC-SC) and nonconvex-concave (NC-C) settings. We established convergence guarantees for vanilla SGDA without modifying stochastic gradients and showed that, in the NC-C setting, its complexity matches that of clipping-based methods. In the NC-SC setting, however, vanilla SGDA is suboptimal, which motivates the development of new algorithms. For regularized min-max problems, our proposed methods achieve the optimal dependence on the target accuracy without introducing clipping thresholds, despite the nontrivial interaction between gradient normalization and proximal structures. Finally, the work represented in this paper does have limitations: although this paper does not primarily focus on lower-bound theory, because our proof technique follows the techniques previously used for the bounded-variance setting, the dependence on κ in the lower bound of Theorem 3.4 cannot match that of the upper bound, leaving us uncertain whether the resulting complexity is loose with respect to κ. Moreover, this paper indeed does not provide a single-loop method that does not require a batch size and can still achieve (κ2p−1p−1σp−1ϵ−3p−2p−1)O(κ 2p-1p-1σ pp-1ε^- 3p-2p-1) or sharper complexity under heavy-tailed and NC-SC setting, leaving room for the subsequent development of more efficient methods. For the future, an interesting direction is to investigate whether the techniques developed in this work can be extended to stochastic bilevel optimization under heavy-tailed noise since stochastic min-max optimization encompasses bilevel optimization as a special case. Another important direction is to incorporate adaptive step-size strategies commonly used in deep learning, such as those based on moving averages of stochastic gradients, and to understand their convergence and complexity under heavy-tailed noise. Finally, relaxing the structural assumptions considered in this work, such as moving from NC-(S)C problems to more general nonconvex-nonconcave settings, such as nonconvex-weakly-concave setting, may provide a broader understanding of gradient-based methods for heavy-tailed stochastic optimization. References [1] K. Ahn, X. Cheng, M. Song, C. Yun, A. Jadbabaie, and S. Sra (2024) Linear attention is (maybe) all you need (to understand transformer optimization). In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 16193–16205. Cited by: §1. [2] A. Alacaoglu and J. Kim (2026) Solving stochastic variational inequalities without the bounded variance assumption. arXiv preprint arXiv:2602.05531. Cited by: §1.2. [3] Y. Arjevani, Y. Carmon, J. C. Duchi, D. J. Foster, N. Srebro, and B. Woodworth (2022) Lower bounds for non-convex stochastic optimization. External Links: 1912.02365, Link Cited by: §A.3, §A.3, §A.3, Remark 3.1. [4] N. Bashirov, A. Gasnikov, and A. Lobanov (2026) Zeroth-order methods for non-smooth stochastic problems under heavy-tailed noise. Optimization Methods and Software, p. 1–26. Cited by: §1.2. [5] N. Bashirov, A. Gasnikov, and A. Lobanov (2026) Zeroth-order methods for non-smooth stochastic problems under heavy-tailed noise. Optimization Methods and Software, p. 1–26. Cited by: §1. [6] B. Battash, L. Wolf, and O. Lindenbaum (2024) Revisiting the noise model of stochastic gradient descent. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, S. Dasgupta, S. Mandt, and Y. Li (Eds.), Proceedings of Machine Learning Research, Vol. 238, p. 4780–4788. Cited by: §1. [7] D. Bertsekas (2009) Convex optimization theory. Athena Scientific optimization and computation series, Athena Scientific. External Links: ISBN 9781886529311, Link Cited by: §B.1. [8] R. I. Boţ and A. Böhm (2023) Alternating proximal-gradient steps for (stochastic) nonconvex-concave minimax problems. SIAM Journal on Optimization 33 (3), p. 1884–1913. Cited by: §1, Remark 3.2, Remark 3.2, Lemma 3.5. [9] E. M. Chayti (2026) Stochastic compositional optimization via hybrid momentum frank–wolfe. arXiv preprint arXiv:2605.15350. Cited by: §1. [10] Z. Chen, Y. Zhou, T. Xu, and Y. Liang (2021) Proximal gradient descent-ascent: variable convergence under kŁ geometry. External Links: 2102.04653, Link Cited by: Lemma 3.1, Lemma 3.2. [11] A. Cutkosky, H. Mehta, and F. Orabona (2025) Optimal stochastic non-smooth non-convex optimization through online-to-non-convex conversion. External Links: 2302.03775, Link Cited by: §1.2. [12] D. Davis and D. Drusvyatskiy (2018) Stochastic subgradient method converges at the rate O(k−1/4)O(k^-1/4) on weakly convex functions. External Links: 1802.02988, Link Cited by: §3.2. [13] I. Fatkhullin, F. Hübler, and G. Lan (2025) Can sgd handle heavy-tailed noise?. External Links: 2508.04860, Link Cited by: §1.2, §1, §3.1. [14] S. Garg, J. Zhanson, E. Parisotto, A. Prasad, Z. Kolter, Z. Lipton, S. Balakrishnan, R. Salakhutdinov, and P. Ravikumar (2021) On proximal policy optimization’s heavy-tailed gradients. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 3610–3619. Cited by: §1. [15] X. Gong, J. Hao, and M. Liu (2026) Adaptive algorithms with sharp convergence rates for stochastic hierarchical optimization. Advances in Neural Information Processing Systems 38, p. 57161–57210. Cited by: §1. [16] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2014) Generative adversarial networks. External Links: 1406.2661, Link Cited by: §1. [17] E. Gorbunov, A. Sadiev, M. Danilova, S. Horváth, G. Gidel, P. Dvurechensky, A. Gasnikov, and P. Richtárik (2023) High-probability convergence for composite and distributed stochastic minimization and variational inequalities with heavy-tailed noise. arXiv preprint arXiv:2310.01860. Cited by: §1. [18] K. Gruntkowska, H. Li, X. Qian, and P. Richtárik (2026) Broximal alignment for global non-convex optimization. arXiv preprint arXiv:2604.13483. Cited by: Remark 4.1. [19] K. Gruntkowska, H. Li, A. Rane, and P. Richtárik (2025) The ball-proximal (=” broximal”) point method: a new algorithm, convergence theory, and applications. arXiv preprint arXiv:2502.02002. Cited by: Lemma B.1, Remark 4.1, Remark 4.1. [20] K. Gruntkowska and P. Richtárik (2025) Non-euclidean broximal point method: a blueprint for geometry-aware optimization. arXiv preprint arXiv:2510.00823. Cited by: Remark 4.1. [21] Z. Guo, Y. Xu, W. Yin, R. Jin, and T. Yang (2025) Unified convergence analysis for adaptive optimization with moving average estimator. Machine Learning 114 (4), p. 86. Cited by: §1, §3.1. [22] J. Ha (2026) High probability convergence guarantees of stochastic gradient descent ascent in structured nonconvex min-max games. Proceedings of Machine Learning Research vol 336, p. 1–71. Cited by: 2nd item, §1.2, §1, §3.2, §3.2, Table 1, Table 1, Table 1, §4.2, Table 2. [23] C. Han, X. Wang, P. Xu, and J. Zhang (2026) Normalized stochastic proximal approximation methods for nonsmooth composite optimization under heavy-tailed noise. Cited by: §1. [24] C. He and Z. Lu (2025) Accelerated stochastic first-order method for convex optimization under heavy-tailed noise. arXiv preprint arXiv:2510.11676. Cited by: §1.2. [25] F. Huang, X. Wu, and H. Huang (2021) Efficient mirror descent ascent methods for nonsmooth minimax problems. Advances in Neural Information Processing Systems 34, p. 10431–10443. Cited by: §1, §3.1, Remark 3.2. [26] F. Hübler, I. Fatkhullin, and N. He (2024) From gradient clipping to normalization for heavy tailed SGD. In OPT 2024: Optimization for Machine Learning, External Links: Link Cited by: §B.2.1, Lemma C.5, §1, §3.3, §4.3. [27] N. Kornilov, O. Shamir, A. Lobanov, D. Dvinskikh, A. Gasnikov, I. Shibaev, E. Gorbunov, and S. Horváth (2023) Accelerated zeroth-order method for non-smooth stochastic convex optimization problem with infinite variance. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 64083–64102. External Links: Link Cited by: §3.3. [28] D. Kovalev (2025) Understanding gradient orthogonalization for deep learning via non-euclidean trust-region optimization. External Links: 2503.12645, Link Cited by: Remark B.2, Lemma B.4, §4.1. [29] H. Li, K. Gruntkowska, and P. Richtárik (2026) Stabilized proximal point method via trust region control. arXiv preprint arXiv:2604.02943. Cited by: Remark 4.1. [30] H. Li, Y. Tian, J. Zhang, and A. Jadbabaie (2021) Complexity lower bounds for nonconvex-strongly-concave min-max optimization. External Links: 2104.08708, Link Cited by: §A.3, §A.3, §A.3, §A.3, §A.3, §A.3, §3.1, Remark 3.1, Remark 3.1, Theorem 3.4, footnote 4. [31] S. Li, Y. Wu, X. Cui, H. Dong, F. Fang, and S. Russell (2019) Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. AAAI’19/IAAI’19/EAAI’19. External Links: ISBN 978-1-57735-809-1, Link, Document Cited by: §1. [32] X. Li, J. Yang, and N. He (2022) Tiada: a time-scale adaptive algorithm for nonconvex minimax optimization. arXiv preprint arXiv:2210.17478. Cited by: §1. [33] T. Lin, C. Jin, and M. I. Jordan (2025) Two-timescale gradient descent ascent algorithms for nonconvex minimax optimization. Journal of Machine Learning Research 26 (11), p. 1–45. Cited by: §1, §3.1, Remark 3.2, Remark 3.2. [34] T. Lin, C. Jin, and M. Jordan (2020) On gradient descent ascent for nonconvex-concave minimax problems. In International conference on machine learning, p. 6083–6093. Cited by: §3.2, §3.3, §4.2. [35] M. Liu, Z. Yuan, Y. Ying, and T. Yang (2020) Stochastic auc maximization with deep neural networks. In International Conference on Learning Representations, External Links: Link Cited by: §1. [36] Z. Liu and L. Luo (2025) Stochastic bilevel optimization with heavy-tailed noise. arXiv preprint arXiv:2509.14952. Cited by: §1.2, §1, §4.2, Table 2. [37] Z. Liu and Z. Zhou (2025) Nonconvex stochastic optimization under heavy-tailed noises: optimal convergence without gradient clipping. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §B.2.1, §1, §1, §3.1, §3.3, §4.3. [38] Z. Liu and Z. Zhou (2026) Revisiting the last-iterate convergence of stochastic gradient methods. External Links: 2312.08531, Link Cited by: §1.2. [39] Z. Liu (2026) Clipped gradient methods for nonsmooth convex optimization under heavy-tailed noise: a refined analysis. In International Conference on Learning Representations, Vol. 2026, p. 116560–116608. Cited by: §1. [40] Z. Liu (2026) In-expectation convergence of stochastic gradient methods under heavy-tailed noise. External Links: 2606.00520, Link Cited by: §1.2, §1, §3.1, §3.3. [41] Z. Liu (2026) Online convex optimization with heavy tails: old algorithms, new regrets, and applications. External Links: 2508.07473, Link Cited by: §1.2. [42] P. Mahdavinia, Y. Deng, H. Li, and M. Mahdavi (2022) Tight analysis of extra-gradient and optimistic gradient methods for nonconvex minimax problems. Advances in Neural Information Processing Systems 35, p. 31213–31225. Cited by: §1, §3.1, Remark 3.2. [43] T. D. Nguyen, T. H. Nguyen, A. Ene, and H. L. Nguyen (2023) High probability convergence of clipped-sgd under heavy-tailed noise. External Links: 2302.05437, Link Cited by: §1. [44] I. Pinelis (2015) Best possible bounds of the von Bahr–Esseen type. Annals of Functional Analysis 6 (4), p. 1 – 29. External Links: Document, Link Cited by: Lemma C.4. [45] R. T. Rockafellar (1997) Convex analysis. Vol. 28, Princeton university press. Cited by: §B.1, §B.1. [46] A. Rodomanov and Y. Nesterov (2020) Smoothness parameter of power of euclidean norm. Journal of Optimization Theory and Applications 185 (2), p. 303–326. External Links: ISSN 1573-2878, Link, Document Cited by: Appendix C. [47] A. Sadiev, M. Danilova, E. Gorbunov, S. Horváth, G. Gidel, P. Dvurechensky, A. Gasnikov, and P. Richtárik (2023) High-probability bounds for stochastic optimization and variational inequalities: the case of unbounded variance. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 29563–29648. External Links: Link Cited by: §1. [48] T. Sun, X. Liu, and K. Yuan (2025) Revisiting gradient normalization and clipping for nonconvex sgd under heavy-tailed noise: necessity, sufficiency, and acceleration. Journal of Machine Learning Research 26. External Links: Link Cited by: §B.2.1, §4.3. [49] B. von Bahr and C. Esseen (1965) Inequalities for the rth absolute moment of a sum of random variables, 1≤r≤21≤ r≤ 2. The Annals of Mathematical Statistics 36 (1), p. 299–303. External Links: ISSN 00034851, Link Cited by: Lemma C.4. [50] N. M. Vural, L. Yu, K. Balasubramanian, S. Volgushev, and M. A. Erdogdu (2022) Mirror descent strikes again: optimal stochastic convex optimization under infinite noise variance. External Links: 2202.11632, Link Cited by: §1.2. [51] Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang, and Q. Gu (2025) Self-play preference optimization for language model alignment. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1. [52] W. Xian, Z. Chen, and H. Huang (2024) Delving into the convergence of generalized smooth minimax optimization. In Forty-first International Conference on Machine Learning, Cited by: §1, §3.1, Remark 3.2. [53] R. Yamada, N. Sato, and H. Iiduka (2026) Vanilla sgd with momentum survives heavy-tailed noise: convergence analysis without gradient clipping or normalization. arXiv preprint arXiv:2607.08104. Cited by: §1.2. [54] J. Yang, A. Orvieto, A. Lucchi, and N. He (2022) Faster single-loop algorithms for minimax optimization without strong concavity. In International conference on artificial intelligence and statistics, p. 5485–5517. Cited by: §1, §3.2, Remark 3.2, Remark 3.2. [55] Z. Yang, P. Chen, Y. Zhao, and L. Chen (2025) Zeroth-order proximal clipped gradient method with shifts for distributed stochastic composite optimization problems with infinite variance: z.-p. yang et al.. Journal of Scientific Computing 105 (2), p. 36. Cited by: §1. [56] Z. Yuan, Y. Yan, M. Sonka, and T. Yang (2021) Large-scale robust deep auc maximization: a new surrogate loss and empirical studies on medical image classification. External Links: 2012.03173, Link Cited by: §1. [57] J. Zhang, S. P. Karimireddy, A. Veit, S. Kim, S. Reddi, S. Kumar, and S. Sra (2020) Why are adaptive methods good for attention models?. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 15383–15393. External Links: Link Cited by: §1, §1. [58] S. Zhang and Y. Hu (April 28, 2025) Avoid overclaims - summary of complexity bounds for algorithms in minimization and minimax optimization. In ICLR Blogposts 2025, Note: https://iclr-blogposts.github.io/2025/blog/opt-summary/ External Links: Link Cited by: §1.2. [59] X. Zhang and H. Gao (2025) Federated stochastic minimax optimization under heavy-tailed noises. arXiv preprint arXiv:2511.04456. Cited by: §1.2, §1, §1, §4.2, Table 2. [60] X. Zhang, Y. Zhang, and H. Gao (2025) Nonconvex decentralized stochastic bilevel optimization under heavy-tailed noises. arXiv preprint arXiv:2509.15543. Cited by: §1.2, §1, §1. [61] X. Zhang, N. S. Aybat, and M. Gurbuzbalaban (2022) Sapd+: an accelerated stochastic method for nonconvex-concave minimax problems. Advances in Neural Information Processing Systems 35, p. 21668–21681. Cited by: §1, Remark 3.1, Remark 3.2, Remark 3.2. [62] T. Zhu, Y. Xu, Q. Wang, and X. Ji (2026) Stochastic gradient methods under heavy-tailed noises in weakly convex optimization. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §1.2, §1, §3.1, §3.3, §3.3. Contents 1 Introduction 1.1 Contributions 1.2 Related Work 2 Preliminaries 3 Convergence of SGDA under heavy-tailed noise 3.1 NC-SC min-max problems 3.2 NC-C min-max problems 3.3 Proof sketches of and Theorem 4 (Stoc-)TRGDA for NC-SC regularized min-max problems 4.1 How to incorporate normalized gradient into SPGD? 4.2 Two variants of Stoc-TRGDA 4.3 Proof sketches of Theorems and 5 Conclusion and Future work References A Missing proofs in section A.1 Proof of A.2 Proof of Theorem A.3 Proof of Theorem B Missing proofs in section B.1 Proof of B.2 Proof of B.3 Proof of C Supplementary Lemmas Appendix A Missing proofs in section 3 A.1 Proof of Theorem 3.3 First, we prove the following key lemmas Lemma A.1. Let Assumptions 1 and 2 hold. For SGDA with Case I, if η≤1/(8κL) _x≤ 1/(8κ L), we have [(∑t=1Tη‖∇Φ(t)‖2)p2]≤ E [ ( _t=1^T _x ∇ (x_t) ^2 ) p2 ]≤ 4[(∑t=1TL2η‖t−t⋆‖2)p/2] 4E [ ( _t=1^TL^2 _x _t-y _t ^2 ) p2 ] +3Δ1p/2+8(κL)p/2ηpTp+6pT1−p2ηp2 +3 _1 p2+8(κ L) p2 _x^pT σ^p+6 σ^pT^1- p2 _x p2 (A.1) where Δ1=ΔΦ(1)−minΦ() _1 = (x_1)- _x (x). Proof. According to Lemma 3.2, we know that the dual function Φ()=Δmaxf(,) (x) = _yf(x,y) is a 2κL2κ L-smooth function. Therefore, the following inequality holds. Φ(t+1)−Φ(t)≤⟨∇Φ(t),t−1−t⟩+κ‖t+1−t‖2=−η⟨∇Φ(t),g,t⟩+κLη2‖g,t‖2 (x_t+1)- (x_t)≤ ∇ (x_t),x_t-1-x_t +κ _t+1-x_t ^2=- _x ∇ (x_t),g_x,t +κ L _x^2 g_x,t ^2 =−η⟨∇Φ(t),ζ,t⟩−η⟨∇Φ(t),∇f(t,t)−∇Φ(t)⟩−η‖∇Φ(t)‖2 =- _x ∇ (x_t), _x,t - _x ∇ (x_t), _xf(x_t,y_t)-∇ (x_t) - _x ∇ (x_t) ^2 +κLη2‖∇f(t,t)‖2+2κLη2⟨∇f(t,t),ζ,t⟩+κLη2‖ζ,t‖2 +κ L _x^2 _xf(x_t,y_t) ^2+2κ L _x^2 _xf(x_t,y_t), _x,t +κ L _x^2 _x,t ^2 =(−η+2κLη2)‖∇Φ(t)‖2+(−η+2κLη2)⟨∇Φ(t),ζ,t⟩+κLη2‖ζ,t‖2+κLη2‖∇f(t,t)−∇Φ(t)‖2 =(- _x+2κ L _x^2) ∇ (x_t) ^2+(- _x+2κ L _x^2) ∇ (x_t), _x,t +κ L _x^2 _x,t ^2+κ L _x^2 _xf(x_t,y_t)-∇ (x_t) ^2 +(−η+2κLη2)⟨∇Φ(t),∇f(t,t)−∇Φ(t)⟩+2κLη2⟨∇Φ(t),∇f(t,t)−∇Φ(t)⟩ +(- _x+2κ L _x^2) ∇ (x_t), _xf(x_t,y_t)-∇ (x_t) +2κ L _x^2 ∇ (x_t), _xf(x_t,y_t)-∇ (x_t) (A.2) Applying Cauchy-Schwarz inequality, the L-Lipschitz continuity of ∇f(t,t) _xf(x_t,y_t) and Young’s inequality , we can obtain κLη2‖∇f(t,t)−∇Φ(t)‖2+ κ L _x^2 _xf(x_t,y_t)-∇ (x_t) ^2+ (−η+2κLη2)⟨∇Φ(t),∇f(t,t)−∇Φ(t)⟩+2κLη2⟨∇Φ(t),∇f(t,t)−∇Φ(t)⟩ (- _x+2κ L _x^2) ∇ (x_t), _xf(x_t,y_t)-∇ (x_t) +2κ L _x^2 ∇ (x_t), _xf(x_t,y_t)-∇ (x_t) ≤Lη|2κLη−1|‖∇Φ(t)‖‖t−t⋆‖+2κη2L2‖t−t⋆‖‖ζ,t‖+κL3η2‖t−t⋆‖2 ≤ L _x |2κ L _x-1 | ∇ (x_t) _t-y _t +2κ _x^2L^2 _t-y _t _x,t +κ L^3 _x^2 _t-y _t ^2 ≤(a)η‖∇Φ(t)‖24+ηL2‖t−t⋆‖2+2κη2L2(L‖t−t⋆‖24+‖ζ,t‖2L)+κL3η2‖t−t⋆‖2 (a)≤ _x ∇ (x_t) ^24+ _xL^2 _t-y _t ^2+2κ _x^2L^2 ( L _t-y _t ^24+ _x,t ^2L )+κ L^3 _x^2 _t-y _t ^2 =η4‖∇Φ(t)‖2+(3κL3η22+L2η)‖t−t⋆‖2+2κLη2‖ζ,t‖2 = _x4 ∇ (x_t) ^2+ ( 3κ L^3 _x^22+L^2 _x ) _t-y _t ^2+2κ L _x^2 _x,t ^2 ≤(b)η4‖∇Φ(t)‖2+19L2η‖t−t⋆‖216+2κLη2‖ζ,t‖2, (b)≤ _x4 ∇ (x_t) ^2+ 19L^2 _x _t-y _t ^216+2κ L _x^2 _x,t ^2, (A.3) where (a)(a) holds beacause 2κLη≤12κ L _x≤ 1, and (b)(b) holds because 3κL3η2/2=3L2η/2⋅κLη≤3L2η/163κ L^3 _x^2/2=3L^2 _x/2·κ L _x≤ 3L^2 _x/16. Then, we can obtain Φ(t+1)−Φ(t)≤ (x_t+1)- (x_t)≤ η(2κLη−3/4)‖∇Φ(t)‖2+2η(2κLη−1)⟨∇Φ(t),ζ,t⟩+3κLη2‖ζ,t‖2 _x (2κ L _x-3/4 ) ∇ (x_t) ^2+2 _x(2κ L _x-1) ∇ (x_t), _x,t +3κ L _x^2 _x,t ^2 +3κLη2‖ζ,t‖2+19L2η‖t−t⋆‖216 +3κ L _x^2 _x,t ^2+ 19L^2 _x _t-y _t ^216 ≤ (c) - η x ‖ ∇Φ(x t ) ‖ 2 2 + η_x(2κ η_x-1)⟨∇Φ(x_t), ζ_x, t ⟩ + 3κ η_x^2 ‖ζ_x, t‖^2+ 19L 2 η x ‖ y t -y ⋆ t ‖ 2 16 , where (c)(c) holds due to the condition 2κLη≤1/42κ L _x≤ 1/4. Multiplying both sides of the above inequality by 22 and summing the result from t=1t=1 to T, we get ∑t=1Tη‖∇Φ(t)‖2≤ _t=1^T _x ∇ (x_t) ^2≤ |∑t=1T2η(2κLη−1)⟨∇Φ(t),ζ,t⟩| | _t=1^T2 _x(2κ L _x-1) ∇ (x_t), _x,t | +2Δ1++∑t=1T6κLη2‖ζ,t‖2+∑t=1T3L2η‖t−t⋆‖2, +2 _1++ _t=1^T6κ L _x^2 _x,t ^2+ _t=1^T3L^2 _x _t-y _t ^2, (A.4) where Δ1=ΔΦ(i)−minΦ() _1 = (x_i)- _x (x). Taking both sides of the above inequality to the power of p/2p/2 and then taking the expectation to get [(∑t=1Tη‖∇Φ(t)‖2)p/2]≤ E [ ( _t=1^T _x ∇ (x_t) ^2 ) p2 ]≤ [(|∑t=1T2η(2κLη−1)⟨∇Φ(t),ζ,t⟩|)p/2] E [ ( | _t=1^T2 _x(2κ L _x-1) ∇ (x_t), _x,t | ) p2 ] + + 2Δ1p/2+6(κL)p/2ηp∑t=1T[‖ζ,t‖p]+3[(∑t=1TL2η‖t−t⋆‖2)p/2] 2 _1 p2+6(κ L) p2 _x^p _t=1^TE [ _x,t ^p ]+3E [ ( _t=1^TL^2 _x _t-y _t ^2 ) p2 ] ≤ (d) E[ ( — ∑_t=1^T 2η_x(2κ η_x-1)⟨∇Φ(x_t), ζ_x, t ⟩ — ) ^p/2 p2] + + 2Δ1p/2+6(κL)p/2ηpTp+3[(∑t=1TL2η‖t−t⋆‖2)p/2], 2 _1 p2+6(κ L) p2 _x^pT σ^p+3E [ ( _t=1^TL^2 _x _t-y _t ^2 ) p2 ], (A.5) where (d)(d) holds due to [‖ζ,t‖p]=[t[‖ζ,t‖p]]≤pE[ _x,t ^p]=E[E_t[ _x,t ^p]]≤ σ^p. Furthermore, we have [(|∑t=1T2η(2κLη−1)⟨∇Φ(t),ζ,t⟩|)p/2] E [ ( | _t=1^T2 _x(2κ L _x-1) ∇ (x_t), _x,t | ) p2 ] ≤(c)[(|∑t=1T2η(2κLη−1)⟨∇Φ(t),ζ,t⟩|)p]=2pηp|2κLη−1|p[(|∑t=1T⟨∇Φ(t),ζ,t⟩|)p] (c)≤ E [ ( | _t=1^T2 _x(2κ L _x-1) ∇ (x_t), _x,t | )^p ]= 2^p _x^p |2κ L _x-1 |^pE [ ( | _t=1^T ∇ (x_t), _x,t | )^p ] ≤(d)2pηp[(|∑t=1T⟨∇Φ(t),ζ,t⟩|)p]≤Lemma C.44ηp∑t=1T[|⟨∇Φ(t),ζ,t⟩|p] (d)≤ 2^p _x^pE [ ( | _t=1^T ∇ (x_t), _x,t | )^p ] @crefcreftypecap~refnumlem:app/Supp_lems/p_moment_sum_martingales≤ 4 _x^p _t=1^TE [ | ∇ (x_t), _x,t |^p ] ≤(e)4ηp/2p[∑t=1Tηp/2‖∇Φ(t)‖p]≤(f)4p(∑t=1Tηp2−p)2−p2[(∑t=1Tη‖∇Φ(t)‖2)p2] (e)≤ 4 _x p2 σ^pE [ _t=1^T _x p2 ∇ (x_t) ^p ] (f)≤ 4 σ^p ( _t=1^T _x p2-p ) 2-p2E [ ( _t=1^T _x ∇ (x_t) ^2 ) p2 ] ≤(g)14[(∑t=1Tη‖∇Φ(t)‖2)p2]+4pT2−p2ηp2, (g)≤ 14E [ ( _t=1^T _x ∇ (x_t) ^2 ) p2 ]+4 σ^pT 2-p2 _x p2, (A.6) where (c)(c) holds due to the Jensen’s inequality, i.e., q1=q2=2q_1=q_2=2 and Y=ΔY = 1 in (C.2); (d)(d) holds due to the condition |2κLη−1|≤1|2κ L _x-1|≤ 1; (e)(e) holds true because we used Cauchy-Schwarz’s inequality and the law of total expectation, i.e., [‖∇Φ(t)‖‖ζ,t‖]=[‖∇Φ(t)‖t[‖ζ,t‖]]≤p[‖∇Φ(t)‖]E [\|∇ (x_t)\|\| _x,t\| ]=E [\|∇ (x_t)\|E_t [\| _x,t\| ] ]≤ σ^pE [\|∇ (x_t)\| ]; (f)(f) holds because we used Hölder’s inequality (Lemma C.1), i.e., let ata_t, btb_t, q1q_1 and q2q_2 in (C.1) be ‖∇Φ(t)‖p\|∇ (x_t)\|^p, ηp/2 _x p2, 2/p2/p and 2/(2−p)2/(2-p). Note that we can apply (C.11) in Lemma C.4 because ⟨∇Φ(t),ζ,t⟩t=1T\ ∇ (x_t), _x,t \_t=1^T is indeed a martingale difference sequence related to filteration ℱt−1=Δσ(1,1,…,t,t)F_t-1 =σ(x_1,y_1,…,x_t,y_t). Substituting (A.6) into (A.5), we get [(∑t=1Tη‖∇Φ(t)‖2)p2]≤ E [ ( _t=1^T _x ∇ (x_t) ^2 ) p2 ]≤ (8/3)Δ1p/2+8(κL)p/2ηpTp+(16/3)pT2−pηp2 (8/3) _1 p2+8(κ L) p2 _x^pT σ^p+(16/3) σ^pT 2-pp _x p2 +4[(∑t=1TL2η‖t−t⋆‖2)p/2] +4E [ ( _t=1^TL^2 _x _t-y _t ^2 ) p2 ] (A.7) By appropriately increasing the constants, we proved the lemma. ■ Lemma A.2. Suppose that Assumptions 1(a) and 2 hold. For SGDA with Case I, if η≤min1/(2κL),η/(12κ2) _x≤ \1/(2κ L), _y/(12κ^2)\ and η≤1/(2L) _y≤ 1/(2L), then after one iteration of SGDA , we have L2η‖t−t⋆‖2 L^2 _x _t-y _t ^2 ≤(1−μη2)L2η‖t−1−t−1⋆‖2+2κLη2‖∇Φ(t−1)‖2 ≤ (1- μ _y2 )L^2 _x _t-1-y _t-1 ^2+2κ L _x^2 ∇ (x_t-1) ^2 +3κLη2‖ζ,t−1‖22+4L2ηη2‖ζ,t−1‖2 + 3κ L _x^2 _x,t-1 ^22+4L^2 _x _y^2 _y,t-1 ^2 +2L2ηη(1+2κLη)⟨ζ,t−1,t−1−t−1⋆⟩ +2L^2 _x _y(1+2κ L _x) _y,t-1,y_t-1-y _t-1 +(2(κLη)2η+κLη2)⟨ζ,t−1,∇Φ(t−1)⟩ + (2(κ L _x)^2 _x+κ L _x^2 ) _x,t-1,∇ (x_t-1) (A.8) Proof. First, we have ‖t−t−1⋆‖2=‖t−t−1+t−1−t−1⋆‖2 \|y_t-y _t-1\|^2= _t-y_t-1+y_t-1-y _t-1 ^2 =‖t−t−1‖2+2⟨t−t−1,t−1−t−1⋆⟩+‖t−1−t−1⋆‖2 = _t-y_t-1 ^2+2 _t-y_t-1,y_t-1-y _t-1 + _t-1-y _t-1 ^2 =‖t−1−t−1⋆‖2+2η⟨g,t−1,t−1−t−1⋆⟩+η2‖g,t−1‖2 = _t-1-y _t-1 ^2+2 _y g_y,t-1,y_t-1-y _t-1 + _y^2 g_y,t-1 ^2 ≤‖t−1−t−1⋆‖2+2η⟨∇f(t−1,t−1),t−1−t−1⋆⟩+2η2‖∇f(t−1,t−1)‖2 ≤ _t-1-y _t-1 ^2+2 _y _yf(x_t-1,y_t-1),y_t-1-y _t-1 +2 _y^2 _yf(x_t-1,y_t-1) ^2 +2η⟨ζ,t−1,t−1−t−1⋆⟩+2η2‖ζ,t−1‖2 +2 _y _y,t-1,y_t-1-y _t-1 +2 _y^2 _y,t-1 ^2 ≤(a)(1−μη)‖t−1−t−1⋆‖2+2η⟨ζ,t−1,t−1−t−1⋆⟩+2η2‖ζ,t−1‖2 (a)≤(1-μ _y) _t-1-y _t-1 ^2+2 _y _y,t-1,y_t-1-y _t-1 +2 _y^2 _y,t-1 ^2 +2η(2Lη−1)(f(t−1,t−1⋆)−f(t−1,t−1)) +2 _y(2L _y-1) (f(x_t-1,y _t-1)-f(x_t-1,y_t-1) ) ≤(b)(1−μη)‖t−1−t−1⋆‖2+2η⟨ζ,t−1,t−1−t−1⋆⟩+2η2‖ζ,t−1‖2, (b)≤(1-μ _y) _t-1-y _t-1 ^2+2 _y _y,t-1,y_t-1-y _t-1 +2 _y^2 _y,t-1 ^2, (A.9) where (a)(a) holds due to the μ-strognly convavity of f(,⋅)f(x,·) with respect to y, i.e., f(t−1,t−1⋆)−f(t−1,t−1)≤⟨∇f(t−1,t−1),t−1⋆−t−1⟩−(μ/2)‖t−t⋆‖2f(x_t-1,y _t-1)-f(x_t-1,y_t-1)≤ _yf(x_t-1,y_t-1),y _t-1-y_t-1 -(μ/2)\|y_t-y _t\|^2, and Lemma C.3; (b)(b) holds due to condition η≤1/(2L) _y≤ 1/(2L). Second, for ‖t⋆−t−1⋆‖2\|y _t-y _t-1\|^2, we have ‖t⋆−t−1⋆‖2≤Lemma 3.1κ2‖t−t−1‖2=κ2η2‖g,t−1‖2 \|y _t-y _t-1\|^2 Lemma~ lem:prelims/smoothness_dual_optimizers≤κ^2 _t-x_t-1 ^2=κ^2 _x^2 g_x,t-1 ^2 =κ2η2‖∇f(t−1,t−1)‖2+2κ2η2⟨ζ,t−1,∇f(t−1,t−1)⟩+κ2η2‖ζ,t−1‖2 =κ^2 _x^2 _xf(x_t-1,y_t-1) ^2+2κ^2 _x^2 _x,t-1, _xf(x_t-1,y_t-1) +κ^2 _x^2 _x,t-1 ^2 ≤2κ2η2‖∇Φ(t−1)‖2+2κ2η2‖∇f(t−1,t−1)−∇Φ(t−1)‖2+κ2η2‖ζ,t−1‖2 ≤ 2κ^2 _x^2 ∇ (x_t-1) ^2+2κ^2 _x^2 _xf(x_t-1,y_t-1)-∇ (x_t-1) ^2+κ^2 _x^2 _x,t-1 ^2 +2κ2η2⟨ζ,t−1,∇Φ(t−1)⟩+2κ2η2⟨ζ,t−1,∇f(t−1,t−1)−∇Φ(t−1)⟩ +2κ^2 _x^2 _x,t-1,∇ (x_t-1) +2κ^2 _x^2 _x,t-1, _xf(x_t-1,y_t-1)-∇ (x_t-1) ≤(a)4(κLη)2‖t−1−t−1⋆‖2+2(κη)2(‖∇Φ(t−1)‖2+⟨ζ,t−1,∇Φ(t−1)⟩)+3(κη)2‖ζ,t−1‖22, (a)≤4(κ L _x)^2 _t-1-y _t-1 ^2+2(κ _x)^2 ( ∇ (x_t-1) ^2+ _x,t-1,∇ (x_t-1) )+ 3(κ _x)^2 _x,t-1 ^22, (A.10) where (a)(a) holds because ⟨ζ,t−1,∇f(t−1,t−1)−∇Φ(t−1)⟩≤ _x,t-1, _xf(x_t-1,y_t-1)-∇ (x_t-1) ≤ ‖ζ,t−1‖‖∇f(t−1,t−1)−∇Φ(t−1)‖ _x,t-1 _xf(x_t-1,y_t-1)-∇ (x_t-1) ≤ ≤ L‖ζ,t−1‖‖t−1−t−1⋆‖ L _x,t-1 _t-1-y _t-1 ≤ ≤ L2‖t−1−t−1⋆‖2+‖ζ,t−1‖24. L^2 _t-1-y _t-1 ^2+ _x,t-1 ^24. (A.11) Then, by the Young’s inequality, we can obtain ‖t−t⋆‖2≤(1+2κLη)‖t−t−1⋆‖2+(1+12κLη)‖t⋆−t−1⋆‖2 _t-y _t ^2≤(1+2κ L _x) _t-y _t-1 ^2+ (1+ 12κ L _x ) _t-y _t-1 ^2 ≤(A.9,A.10)[(1−μη)(1+2κLη)+4(κLη)2+2κLη]‖t−1−t−1⋆‖2+2η2(1+2κLη)‖ζ,t−1‖2 ( eq:app/SGDA/Proof_Lemma2/UB1, eq:app/SGDA/Proof_Lemma2/UB2)≤ [(1-μ _y)(1+2κ L _x)+4(κ L _x)^2+2κ L _x ] _t-1-y _t-1 ^2+2 _y^2(1+2κ L _x) _y,t-1 ^2 +(1+12κLη)(2(κη)2‖∇Φ(t−1)‖2+3(κη)2‖ζ,t−1‖22) + (1+ 12κ L _x ) (2(κ _x)^2 ∇ (x_t-1) ^2+ 3(κ _x)^2 _x,t-1 ^22 ) +2η(1+2κLη)⟨ζ,t−1,t−1−t−1⋆⟩+(2(κη)2+κηL)⟨ζ,t−1,∇Φ(t−1)⟩ +2 _y(1+2κ L _x) _y,t-1,y_t-1-y _t-1 + (2(κ _x)^2+ κ _xL ) _x,t-1,∇ (x_t-1) ≤(b)(1−μη2)‖t−1−t−1⋆‖2+2κη‖∇Φ(t−1)‖2L+3κη‖ζ,t−1‖22L+4η2‖ζ,t−1‖2 (b)≤ (1- μ _y2 ) _t-1-y _t-1 ^2+ 2κ _x ∇ (x_t-1) ^2L+ 3κ _x _x,t-1 ^22L+4 _y^2 _y,t-1 ^2 +2η(1+2κLη)⟨ζ,t−1,t−1−t−1⋆⟩+(2(κη)2+κηL)⟨ζ,t−1,∇Φ(t−1)⟩, \;+2 _y(1+2κ L _x) _y,t-1,y_t-1-y _t-1 + (2(κ _x)^2+ κ _xL ) _x,t-1,∇ (x_t-1) , (A.12) where (b)(b) holds because (1−μη)(1+2κLη)+4(κLη)2+2κLη≤ (1-μ _y)(1+2κ L _x)+4(κ L _x)^2+2κ L _x≤ 1−μη+4κLη+(2κLη)2 1-μ _y+4κ L _x+(2κ L _x)^2 ≤ ≤ 1−μη+6κLη 1-μ _y+6κ L _x (η≤12κL _x≤ 12κ L) ≤ ≤ 1−μη2, 1- μ _y2, (η≤η12κ2 _x≤ _y12κ^2) and 2(κη)2+κηL≤2κηL,3(κη)22+3κη4L≤3κη2L,2η2(1+2κLη)≤4η2. cases2(κ _x)^2+ κ _xL≤ 2κ _xL,\\ 3(κ _x)^22+ 3κ _x4L≤ 3κ _x2L,\\ 2 _y^2(1+2κ L _x)≤ 4 _y^2. cases (η≤12κL _x≤ 12κ L) We now finish the proof by multiplying both sides of (A.12) by L2ηL^2 _x. ■ Lemma A.3. Let Assumptions 1 and 2 hold. For SGDA with Case I, if η≤min1/(2κL),η/2(172/pκ2) _x≤ \1/(2κ L), _y/2(17 2pκ^2) \ and η≤1/(2L) _y≤ 1/(2L), we have 4[(∑t=1TκL2η‖t−t⋆‖2)p/2]≤ 4E [ ( _t=1^Tκ L^2 _x _t-y _t ^2 ) p2 ]≤ 12[(∑t=1Tη‖∇Φ(t)‖2)p/2] 12E [ ( _t=1^T _x ∇ (x_t) ^2 ) p2 ] +9(δ122η)p/2+18(κη)p/2Tp+2405T1−p/2(κ)pηp/2, +9 ( _1^22 _y ) p2+18(κ _x) p2T σ^p+2405T^1- p2(κ σ)^p _x p2, (A.13) where δ1=Δ‖1−1⋆‖ _1 = _1-y _1 . Proof. To simplify the calculation, we first introduce the following notation β=Δ1−μη/2,α=Δ2L2ηη(1+2κLη),θ=Δ2(κLη)2η+κLη2 β =1-μ _y/2,\;α =2L^2 _x _y(1+2κ L _x),\;θ =2(κ L _x)^2 _x+κ L _x^2 Dt=ΔL2η‖t−t⋆‖2,Bt−1=Δ2κLη2‖∇Φ(t−1)‖2+3κLη2‖ζ,t−1‖22+4L2ηη2‖ζ,t−1‖2 D_t =L^2 _x _t-y _t ^2,\;B_t-1 =2κ L _x^2 ∇ (x_t-1) ^2+ 3κ L _x^2 _x,t-1 ^22+4L^2 _x _y^2 _y,t-1 ^2 Xt−1=Δ⟨∇Φ(t−1),ζ,t−1⟩,Yt−1=Δ⟨ζ,t−1,t−1−t−1⋆⟩, X_t-1 = ∇ (x_t-1), _x,t-1 ,\;Y_t-1 = _y,t-1,y_t-1-y _t-1 , (A.14) then by Lemma A.2, we have Dt≤βDt−1+Bt−1+θXt−1+αYt−1. D_t≤β D_t-1+B_t-1+θ X_t-1+α Y_t-1. (A.15) Note that since we are asking for η≤1/2L _y≤ 1/2L, then we have 0<β<10<β<1 due to μη/2≤1/4κ<1μ _y/2≤ 1/4κ<1. Continuing recursively, we obtain the following: Dt≤ D_t≤ βt−1D1+∑s=1t−1βt−1−s(Bs+θXs+αYs) β^t-1D_1+ _s=1^t-1β^t-1-s(B_s+θ X_s+α Y_s) (A.16) We sum both sides of above inequality from t=1t=1 to T to obtain ∑t=1TDt= _t=1^TD_t= D11−β+∑t=1T−1∑s=1t−1βt−1−s(Bs+θXs+αYs) D_11-β+ _t=1^T-1 _s=1^t-1β^t-1-s(B_s+θ X_s+α Y_s) = = D11−β+∑t=1T−1(∑s=0T−1−tβs)(Bt+θXt+αYt) D_11-β+ _t=1^T-1 ( _s=0^T-1-tβ^s )(B_t+θ X_t+α Y_t) ≤ ≤ D11−β+∑t=1T(∑s=0T−1βs)Bt+|∑t=1T−1(∑s=0T−1−tβs)θXt|+|∑t=1T−1(∑s=0T−1−tβs)αYt| D_11-β+ _t=1^T ( _s=0^T-1β^s )B_t+ | _t=1^T-1 ( _s=0^T-1-tβ^s )θ X_t |+ | _t=1^T-1 ( _s=0^T-1-tβ^s )α Y_t | ≤ ≤ D1+∑t=1TBt1−β+|∑t=1T−1(∑s=0T−1−tβs)θXt|+|∑t=1T−1(∑s=0T−1−tβs)αYt|. D_1+ _t=1^TB_t1-β+ | _t=1^T-1 ( _s=0^T-1-tβ^s )θ X_t |+ | _t=1^T-1 ( _s=0^T-1-tβ^s )α Y_t |. (A.17) By raising both of the above inequalities to the power of p/2p/2 and then taking the expectation, we obtain [(∑t=1TDt)p/2] E [ ( _t=1^TD_t ) p2 ] ≤(2D1μη)p/2+[(∑t=1TBt1−β)p/2] ≤ ( 2D_1μ _y ) p2+E [ ( _t=1^TB_t1-β ) p2 ] +[|∑t=1T−1(∑s=0T−1−tβs)θXt|p/2]+[|∑t=1T−1(∑s=0T−1−tβs)αYt|p/2]⏟A + E [ | _t=1^T-1 ( _s=0^T-1-tβ^s )θ X_t | p2 ]+E [ | _t=1^T-1 ( _s=0^T-1-tβ^s )α Y_t | p2 ]_ A (A.18) Then for A, we have A≤ A≤ [|∑t=1T−1(∑s=0T−1−tβs)θXt|p]+[|∑t=1T−1(∑s=0T−1−tβs)αYt|p] E [ | _t=1^T-1 ( _s=0^T-1-tβ^s )θ X_t |^p ]+ E [ | _t=1^T-1 ( _s=0^T-1-tβ^s )α Y_t |^p ] ≤ Lemma C.4 2^2-p∑_t=1^T-1— ∑_s=0^T-1-tβ^s —^pθ^pE [ — X_t —^p ] + 2^2-p∑_t=1^T-1— ∑_s=0^T-1-tβ^s —^pα^pE [ — Y_t —^p ] ≤ ≤ 22−pθ^p∑t=1T[|Xt|p]+22−pα^p∑t=1T[|Yt|p] 2^2-p θ^p _t=1^TE [ |X_t |^p ]+ 2^2-p α^p _t=1^TE [ |Y_t |^p ] ≤ ≤ 22−pθ^p∑t=1T[‖∇Φ(xt)‖p‖ζ,t‖p]+22−pα^p∑t=1T[‖t−t⋆‖p‖ζ,t‖p] 2^2-p θ^p _t=1^TE [ ∇ (x_t) ^p _x,t ^p ]+ 2^2-p α^p _t=1^TE [ _t-y _t ^p _y,t ^p ] ≤ ≤ 22−ppηp/2[∑t=1T(θ^η)p‖∇Φ(xt)‖p]+22−pp(L2η)p/2[∑t=1T(α^Lη)p‖t−t⋆‖p] 2^2-p σ^p _x p2E [ _t=1^T ( θ _x )^p ∇ (x_t) ^p ]+ 2^2-p σ^p(L^2 _x) p2E [ _t=1^T ( αL _x )^p _t-y _t ^p ] ≤ Lemma C.1 2^2-pσ^p ( ∑_t=1^T ( θ η x ) 2p 2-p ) 2-p 2 E [ ( ∑_t=1^T η_x ‖ ∇Φ(x_t−1t-1) ‖ ^2 ) ^p/2 p2 ] +22−pp(∑t=1T(α^Lη)2p2−p)2−p2[(∑t=1TDt)p/2] + 2^2-p σ^p ( _t=1^T ( αL _x ) 2p2-p ) 2-p2E [ ( _t=1^TD_t ) p2 ] ≤ ≤ [(∑t=1Tη‖∇Φ(t)‖2)p/2]17+[(∑t=1TDt)p/2]17+17[(θ^/η)p+(α^/Lη)p]pT1−p/22 E [ ( _t=1^T _x ∇ (x_t) ^2 ) p2 ]17+ E [ ( _t=1^TD_t ) p2 ]17+ 17 [ ( θ _x )^p+ ( αL _x )^p ] σ^pT^1- p22 (A.19) where θ^=Δθ/(1−β) θ =θ/(1-β) and α^=Δα/(1−β) α =α/(1-β). We also can get [(∑t=1TBt1−β)p/2]≤ E [ ( _t=1^TB_t1-β ) p2 ]≤ [(∑t=1T2κLη2‖∇Φ(t)‖2μη)p/2]+[(3κLη2μη)p/2+(8L2ηημ)p/2]Tp E [ ( _t=1^T 2κ L _x^2 ∇ (x_t) ^2μ _y ) p2 ]+ [ ( 3κ L _x^2μ _y ) p2+ ( 8L^2 _x _yμ ) p2 ]T σ^p ≤ ≤ (2κ2η)p/2[(∑t=1Tη‖∇Φ(t)‖2)p/2]+[2(2κ2η)p/2ηp/2+(8κLηη)p/2]Tp ( 2κ^2 _x _y ) p2E [ ( _t=1^T _x ∇ (x_t) ^2 ) p2 ]+ [2 ( 2κ^2 _x _y ) p2 _x p2+ (8κ L _x _y ) p2 ]T σ^p ≤ (a) E [ ( ∑ t=1 T η x ‖ ∇Φ(x t ) ‖ 2 ) p/2 p2 ] 17 + ( 2η x p/2 p2 17 +4(κη_x)^p/2 p2 ) Tσ^p ≤ κ≥1 E [ ( ∑ t=1 T η x ‖ ∇Φ(x t ) ‖ 2 ) p/2 p2 ] 17 + ( 2 17 +4 ) T(κη_x)^p/2 p2σ^p, where (a)(a) holds due to (2κ2η/η)p/2≤1/17(2κ^2 _x/ _y) p2≤ 1/17 and η≤1/(2L) _y≤ 1/(2L). We now substitute (A.19) and (A.1) into (A.18), then multiply both sides of the inequaity we obtained by 17/417/4 to get 4[(∑t=1TDt)p/2]≤ 4E [ ( _t=1^TD_t ) p2 ]≤ 12[(∑t=1Tη‖∇Φ(t)‖2)p/2] 12E [ ( _t=1^T _x ∇ (x_t) ^2 ) p2 ] +9(D1μη)p/2+18(κη)p/2Tp+37T1−p/2p[(θ^η)p+(α^Lη)p] +9 ( D_1μ _y ) p2+18(κ _x) p2T σ^p+37T^1- p2 σ^p [ ( θ _x )^p+ ( αL _x )^p ] ≤ (b) 1 2 E[ ( ∑_t=1^T η_x‖∇Φ(x_t)‖^2 ) ^p/2 p2] +9(δ122η)p/2+18(κη)p/2Tp+2405T1−p/2(κ)pηp/2, +9 ( _1^22 _y ) p2+18(κ _x) p2T σ^p+2405T^1- p2(κ σ)^p _x p2, (A.20) where (b)(b) holds because θ^η=2κ2η(2Lη+1/2)η≤(1/κ+1/2)172/pη≤η θ _x= 2κ^2 _x _y(2L _x+1/2) _x≤ (1/κ+1/2)17 2p _x≤ _x ((2κ2η)p/2≤1/17,κLη≤12 ( 2κ^2 _x _y ) p2≤ 1/17,\;κ L _x≤ 12) α^Lη=4κ(1+2κLη)η≤8κη,D1μη=κLη‖1−1⋆‖2≤‖1−1⋆‖22η αL _x=4κ(1+2κ L _x) _x≤ 8κ _x, D_1μ _y= κ L _x _y _1-y _1 ^2≤ _1-y _1 ^22 _y (κLη≤12κ L _x≤ 12) Now, we finish the proof. ■ A.1.1 Prove the complexity Proof. If we let η=1/(1156κ2L) _x=1/(1156κ^2L) and η=1/(2L) _y=1/(2L), then we can easily verify that η≤min1/(8κL),η/2(172/pκ2) _x≤ \1/(8κ L), _y/2(17 2pκ^2) \ holds true. Therefore, we can substitute (A.13) into (A.1) to get 12[(∑t=1Tη‖∇Φ(t)‖2)p2] \ 12E [ ( _t=1^T _x ∇ (x_t) ^2 ) p2 ] ≤3Δ1p/2+8(κL)p/2(η)pT+6pT1−p/2ηp/2+9(δ122η)p/2+18(κη)p/2Tp+2405T1−p/2(κ)pηp/2 \ ≤ 3 _1 p2+8(κ L) p2( _x σ)^pT+6 σ^pT^1- p2 _x p2+9 ( _1^22 _y ) p2+18(κ _x) p2T σ^p+2405T^1- p2(κ σ)^p _x p2 ≤3Δ1p/2+9Lp/2δ1p+8(κL)p/2(η)pT+18(κη)p/2Tp+2411T1−p/2(κ)pηp/2 \ ≤ 3 _1 p2+9L p2 _1^p+8(κ L) p2( _x σ)^pT+18(κ _x) p2T σ^p+2411T^1- p2(κ σ)^p _x p2 (A.21) By multiplying both sides of the above equation by (ηT)−p/2( _xT)^- p2, we get 12[(1T∑t=1T‖∇Φ(t)‖2)p2] \ 12E [ ( 1T _t=1^T ∇ (x_t) ^2 ) p2 ] ≤3(Δ1Tη)p/2+9(Lδ12Tη)p/2+8(κL)p/2ηp/2pT1−p/2+18κp/2T1−p/2p+2411(κ)pTp−1 \ ≤ 3 ( _1T _x ) p2+9 ( L _1^2T _x ) p2+8(κ L) p2 _x p2 σ^pT^1- p2+18κ p2T^1- p2 σ^p+ 2411(κ σ)^pT^p-1 ≤3468(κΔ1LT)p+10404(Lκδ1T)p+19κp/2T1−p/2p+2411(κ)pTp−1 \ ≤ 3468 (κ _1LT )^p+10404 ( Lκ _1 T )^p+19κ p2T^1- p2 σ^p+ 2411(κ σ)^pT^p-1 ≤3468(κΔ1LT)p+10404(Lκδ1T)p+38κp/2σpT1−p/2Bp−1+4822(κσ)p(BT)p−1. \ ≤ 3468 (κ _1LT )^p+10404 ( Lκ _1 T )^p+ 38κ p2σ^pT^1- p2B^p-1+ 4822(κσ)^p(BT)^p-1. (A.22) Note that the last inequality holds because we have t[‖ζ,t‖p]= E_t [\| _x,t\|^p ]= t[‖∑s=1B∇f(t,t,ξt,s)−∇f(t,t)‖p]Bp E_t [ _s=1^B _xf(x_t,y_t, _t,s)- _xf(x_t,y_t) ^p ]B^p ≤ Lemma C.5 2B^-p∑_s=1^B E_t[∥∇_xf(x_t,y_t,ξ_t,s)-∇_xf(x_t,y_t)∥^p] ≤ 2σ p B p-1 From the law of total expectation, we get [‖ζ,t‖p]=[t[‖ζ,t‖p]]≤2σp/Bp−1E [\| _x,t\|^p ]=E [E_t [\| _x,t\|^p ] ]≤ 2σ^pB^p-1. The same analysis applies to [‖ζ,t‖p]E [\| _y,t\|^p ], and we can conclude that p=2σp/Bp−1 σ^p= 2σ^pB^p-1 holds. Now, if we let T≍maxκ2Δ1Lϵ2,(Lκδ1)2ϵ2, \ T \ κ^2 _1Lε^2, (Lκ _1)^2ε^2 \, B≍max⌈κp2(p−1)σp−1T2−p2(p−1)ϵp−1⌉,1≍max⌈κ4−p2(p−1)(Δ1L)2−p−1σp−1ϵ2p−1⌉,⌈κ4−p2(p−1)(Lδ1)2−p−1σp−1ϵ2p−1⌉,1, \ B \ κ p2(p-1)σ pp-1T 2-p2(p-1)ε pp-1 ,1 \ \ κ 4-p2(p-1)( _1L) 2-pp-1σ pp-1ε 2p-1 , κ 4-p2(p-1)(L _1) 2-pp-1σ pp-1ε 2p-1 ,1 \, (A.23) then we can conlude that (κΔ1LT)p+(Lκδ1T)p+κp/2σpT1−p/2Bp−1≲ϵp (κ _1LT )^p+ ( Lκ _1 T )^p+ κ p2σ^pT^1- p2B^p-1\; \;ε^p (A.24) If, in addition, ϵε is small enough that ϵ<κminΔ1L,Lδ1ε< κ \ _1L,L _1 \ holds true, then we can obtain (σκ)p(BT)p−1≳(σκ)p⋅(ϵ2Tκσ2)p/2≍ (σκ)^p(BT)^p-1\; \;(σκ)^p· ( ε^2Tκσ^2 ) p2 (σκ)p⋅min(ϵ4κ3Δ1Lσ2)p/2,(ϵ4κ3(Lδ1σ)2)p/2 (σκ)^p· \ ( ε^4κ^3 _1Lσ^2 ) p2, ( ε^4κ^3(L _1σ)^2 ) p2 \ =(ϵκminΔ1L,Lδ1)pϵp<ϵp = ( ε κ \ _1L,L _1 \ )^pε^p<ε^p (A.25) Hence, to achieve (T−1[∑t=1T‖∇Φ(t)‖])p≤[(T−1∑t=1T‖∇Φ(t)‖2)p2]≤ϵp (T^-1E [ _t=1^T ∇ (x_t) ] )^p≤E [ (T^-1 _t=1^T ∇ (x_t) ^2 ) p2 ]≤ε^p, the total gradient complexity is upper bounded by B⋅T≲ B· T\; \; maxκp2(p−1)σp−1Tp2(p−1)ϵp−1,T \ κ p2(p-1)σ pp-1T p2(p-1)ε pp-1,T \ ≲ \; \; maxκ3p2(p−1)(σ⋅maxΔ1L,(Lδ1))p−1ϵ2p−1,κ2Δ1Lϵ2,(Lκδ1)2ϵ2 \ κ 3p2(p-1)(σ· \ _1L,(L _1) \) pp-1ε 2pp-1, κ^2 _1Lε^2, (Lκ _1)^2ε^2 \ (A.26) ■ A.2 Proof of Theorem 3.6 First, we prove the following key lemmas Lemma A.4. Let Assumptions 1 (L-smoothness and heavy-tailed noise) and 3 (NC-C Setting) hold. For SGDA with Case I, we have 18[(∑t=1Tη‖∇Φ1/2L(t)‖2)p/2]≤ 18E [ ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ]≤ 2(Lη)p/2[(∑t=1T(Φ(t)−f^(t,t)))p/2] 2(L _x) p2E [ ( _t=1^T( (x_t)- f(x_t,y_t)) ) p2 ] +Δ1p/2+2(LT)p/2ηp(Gp+σp)+4σpT1−p/2ηp/2, + _1 p2+2(LT) p2 _x^p(G^p+σ^p)+4σ^pT^1- p2 _x p2, (A.27) where Δ1=ΔΦ(1)−minΦ() _1 = (x_1)- _x (x) and f^(,)=Δf(,)−h() f(x,y) =f(x,y)-h(y). Proof. By Lemma 3.5, we know Φ() (x) is a L-weakly convex function on X. Let t+=Δargmin∈Φ()+L‖−t‖2x_t^+ = _x \ (x)+L\|x-x_t\|^2 \, and we have Φ1/2L(t+1)=Φ(t+1+)+L‖t+1+−t+1‖2 \ _1/2L(x_t+1)= (x_t+1^+)+L _t+1^+-x_t+1 ^2 ≤Φ(t+)+L‖t+−t+1‖2 \ ≤ (x_t^+)+L _t^+-x_t+1 ^2 =Φ(t+)+L‖Π(t+)−Π(t−η∇f(t,t,ξt))‖2 \ = (x_t^+)+L _X(x_t^+)- _X(x_t- _x _xf(x_t,y_t, _t)) ^2 ≤Φ(t+)+L‖t+−t+η∇f(t,t,ξt)‖2 \ ≤ (x_t^+)+L _t^+-x_t+ _x _xf(x_t,y_t, _t) ^2 =Φ1/2L(t)+2Lη⟨t+−t,∇f(t,t)⟩−η⟨∇Φ1/2L(t),ζ,t⟩+Lη2‖∇f(t,t)+ζ,t‖2 \ = _1/2L(x_t)+2L _x _t^+-x_t, _xf(x_t,y_t) - _x ∇ _1/2L(x_t), _x,t +L _x^2 _xf(x_t,y_t)+ _x,t ^2 ≤Φ1/2L(t)+2Lη⟨t+−t,∇f(t,t)⟩−η⟨∇Φ1/2L(t),ζ,t⟩+2Lη2G2+2Lη2‖ζ,t‖2, \ ≤ _1/2L(x_t)+2L _x _t^+-x_t, _xf(x_t,y_t) - _x ∇ _1/2L(x_t), _x,t +2L _x^2G^2+2L _x^2 _x,t ^2, (A.28) where we let ζ,t=Δ∇f(t,t,ξt)−∇f(t,t) _x,t = _xf(x_t,y_t, _t)- _xf(x_t,y_t), and for the upper bound of 2Lη⟨t+−t,∇f(t,t)⟩2L _x _t^+-x_t, _xf(x_t,y_t) , by the L-smoothness of f(⋅,t)f(·,y_t) w.r.t. x, we can get ⟨t+−t,∇f(t,t)⟩≤ _t^+-x_t, _xf(x_t,y_t) ≤ f(t+,t)−f(t,t)+L2‖t+−t‖2 f(x_t^+,y_t)-f(x_t,y_t)+ L2\|x_t^+-x_t\|^2 = = f^(t+,t)−f^(t,t)+L2‖t+−t‖2 f(x_t^+,y_t)- f(x_t,y_t)+ L2\|x_t^+-x_t\|^2 ≤ ≤ Φ(t+)−f^(t,t)+L2‖t+−t‖2 (x_t^+)- f(x_t,y_t)+ L2\|x_t^+-x_t\|^2 = = Φ1/2L(t+)−f^(t,t)−‖∇Φ1/2L(t)‖28L _1/2L(x_t^+)- f(x_t,y_t)- \|∇ _1/2L(x_t)\|^28L ≤ ≤ Φ(t)−f^(t,t)−‖∇Φ1/2L(t)‖28L. (x_t)- f(x_t,y_t)- \|∇ _1/2L(x_t)\|^28L. (A.29) Substitute (A.29) into (A.28), then sum both sides of the resulting inequality from t=1t=1 to T to obtain ∑t=1Tη‖∇Φ1/2L(t)‖24 \ _t=1^T _x\|∇ _1/2L(x_t)\|^24 ≤Δ1+2Lη∑t=1T(Φ(t)−f^(t,t))+|∑t=1Tη⟨∇Φ1/2L(t),ζ,t⟩|+2TLη2(G2+‖ζ,t‖2). \ ≤ _1+2L _x _t=1^T( (x_t)- f(x_t,y_t))+ | _t=1^T _x ∇ _1/2L(x_t), _x,t |+2TL _x^2(G^2+\| _x,t\|^2). Note that Φ1/2L(1)−Φ1/2L(T+1)≤Φ(1)−minΦ()=ΔΔ1 _1/2L(x_1)- _1/2L(x_T+1)≤ (x_1)- _x (x) = _1. Next, raise both sides of the above inequality to the power of p/2p/2 to obtain 14(∑t=1Tη‖∇Φ1/2L(t)‖2)p/2 \ 14 ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ≤Δ1p/2+2(Lη)p/2(∑t=1T(Φ(t)−f^(t,t)))p/2+(|∑t=1Tη⟨∇Φ1/2L(t),ζ,t⟩|)p/2+2(TL)p/2ηp(Gp+‖ζ,t‖p), \ ≤ _1 p2+2(L _x) p2 ( _t=1^T( (x_t)- f(x_t,y_t)) ) p2+ ( | _t=1^T _x ∇ _1/2L(x_t), _x,t | ) p2+2(TL) p2 _x^p(G^p+\| _x,t\|^p), where we used the fact that 1/4p/2≥1/41/4 p2≥ 1/4 and (a+b)p/2≤ap/2+bp/2(a+b) p2≤ a p2+b p2 for a,b≥0a,b≥ 0. Then, taking the expectation on both sides of the above inequality yields 14[(∑t=1Tη‖∇Φ1/2L(t)‖2)p/2] \ 14E [ ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ] ≤Δ1p/2+2(Lη)p/2[(∑t=1T(Φ(t)−f^(t,t)))p/2]+[(|∑t=1Tη⟨∇Φ1/2L(t),ζ,t⟩|)p/2] \ ≤ _1 p2+2(L _x) p2E [ ( _t=1^T( (x_t)- f(x_t,y_t)) ) p2 ]+E [ ( | _t=1^T _x ∇ _1/2L(x_t), _x,t | ) p2 ] +2(TL)p/2ηp(Gp+σp) \ +2(TL) p2 _x^p(G^p+σ^p) (A.30) We can observe that ⟨η∇Φ1/2L(t),ζ,t⟩t≥1\ _x∇ _1/2L(x_t), _x,t \_t≥ 1 is a martingale difference sequence w.r.t. ℱtt≥0\F_t\_t≥ 0, where ℱt=Δσ(ξ1,…,ξt)F_t =σ( _1,…, _t) and ℱ0=Δ∅F_0 =\ \. Therefore, we can get [(|∑t=1Tη⟨∇Φ1/2L(t),ζ,t⟩|)p/2]≤ E [ ( | _t=1^T _x ∇ _1/2L(x_t), _x,t | ) p2 ]≤ [(|∑t=1Tη⟨∇Φ1/2L(t),ζ,t⟩|)p] E [ ( | _t=1^T _x ∇ _1/2L(x_t), _x,t | )^p ] ≤ Lemma C.4 2^2-p∑_t=1^T E [ — η_x ⟨ ∇Φ_1/2L(x_t), ζ_x, t ⟩ —^p ] ≤ ≤ 22−pσp∑t=1T[ηp‖∇Φ1/2L(t)‖p] 2^2-pσ^p _t=1^TE [ _x^p\|∇ _1/2L(x_t)\|^p ] = = 22−pσp[∑t=1Tηp/2⋅ηp/2‖∇Φ1/2L(t)‖p] 2^2-pσ^pE [ _t=1^T _x p2· _x p2\|∇ _1/2L(x_t)\|^p ] ≤ ≤ 22−pσp[(∑t=1Tηp2−p)2−p2⋅(∑t=1Tη‖∇Φ1/2L(t)‖2)p2] 2^2-pσ^pE [ ( _t=1^T _x p2-p ) 2-p2· ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ] = = 22−pσpT1−p/2ηp2[(∑t=1Tη‖∇Φ1/2L(t)‖2)p2] 2^2-pσ^pT^1- p2 _x p2E [ ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ] ≤ ≤ 18[(∑t=1Tη‖∇Φ1/2L(t)‖2)p2]+4σpT1−p/2ηp2 18E [ ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ]+4σ^pT^1- p2 _x p2 (A.31) After substituting (A.30) into (A.31), we finish the proof ■ Lemma A.5. Let Assumptions 1 (L-smoothness and heavy-tailed noise) and 3 (NC-C Setting) hold. For SGDA with Case I, we have ∑t=1T[Φ(t)−f^(t,t)]≤δ^1+T[D2G(G+σ)η+2G(σ+G)η+D2−pηp−1σp], _t=1^TE [ (x_t)- f(x_t,y_t) ]≤ δ_1+T [D_Y 2G(G+σ) _x _y+2G(σ+G) _x+D_Y^2-p _y^p-1σ^p ], (A.32) where δ^1=ΔΦ(1)−f^(1,1) δ_1 = (x_1)- f(x_1,y_1). Proof. On the one hand, since t=proxηh(t−1+η∇f(t−1,t−1,ξt−1))y_t=prox_ _yh (y_t-1+ _y _yf(x_t-1,y_t-1, _t-1) ), we have 0∈∂h(t)+η−1(t−t−1−η∇f(t−1,t−1,ξt−1))0∈∂ h(y_t)+ _y^-1(y_t-y_t-1- _y _yf(x_t-1,y_t-1, _t-1)) by the optimality condition; on the other hand, due to the convexity of h()h(y), for any ∈dom(h)y (h ), we have −h()≤−h(t)+⟨t−t−1−η∇f(t−1,t−1,ξt−1),−t⟩η \ -h(y)≤-h(y_t)+ _t-y_t-1- _y _yf(x_t-1,y_t-1, _t-1),y-y_t _y =−h(t)+⟨t−t−1,−t⟩η−⟨∇f(t−1,t−1,ξt−1),−t−1⟩ \ =-h(y_t)+ _t-y_t-1,y-y_t _y- _yf(x_t-1,y_t-1, _t-1),y-y_t-1 +⟨∇f(t−1,t−1),t−t−1⟩+⟨ζ,t,t−t−1⟩ \ + _yf(x_t-1,y_t-1),y_t-y_t-1 + _x,t,y_t-y_t-1 =−h(t)+‖t−1−‖2−‖t−‖22η−‖t−t−1‖22η−⟨∇f(t−1,t−1,ξt−1),−t−1⟩ \ =-h(y_t)+ \|y_t-1-y\|^2-\|y_t-y\|^22 _y- \|y_t-y_t-1\|^22 _y- _yf(x_t-1,y_t-1, _t-1),y-y_t-1 +⟨∇f(t−1,t−1),t−t−1⟩+⟨ζ,t,t−t−1⟩ \ + _yf(x_t-1,y_t-1),y_t-y_t-1 + _x,t,y_t-y_t-1 ≤−h(t)+‖t−1−‖2−‖t−‖22η+f(t−1,t)−f(t−1,t−1)+(L2−12η)‖t−t−1‖2 ≤-h(y_t)+ \|y_t-1-y\|^2-\|y_t-y\|^22 _y+f(x_t-1,y_t)-f(x_t-1,y_t-1)+ ( L2- 12 _y )\|y_t-y_t-1\|^2 −⟨∇f(t−1,t−1,ξt−1),−t−1⟩+⟨ζ,t,t−t−1⟩, \ - _yf(x_t-1,y_t-1, _t-1),y-y_t-1 + _x,t,y_t-y_t-1 , (A.33) where the last inequality holds becasue ⟨∇f(t−1,t−1),t−t−1⟩≤f(t−1,t)−f(t−1,t−1)+(L/2)‖t−t−1‖2 _yf(x_t-1,y_t-1),y_t-y_t-1 ≤ f(x_t-1,y_t)-f(x_t-1,y_t-1)+(L/2)\|y_t-y_t-1\|^2 due to the L-smoothness of f(t,⋅)f(x_t,·) w.r.t. y. In addtion, we notice that ⟨ζ,t−1,t−t−1⟩≤ _y,t-1,y_t-y_t-1 ≤ ‖ζ,t−1‖‖t−t−1‖ _y,t-1 _t-y_t-1 = = ‖ζ,t−1‖‖t−t−1‖1−2β‖t−t−1‖2β _y,t-1 _t-y_t-1 ^1- 2β _t-y_t-1 2β = = (4ηβ)1β‖ζ,t−1‖‖t−t−1‖1−2β(β‖t−t−1‖24η)1β ( 4 _yβ ) 1β _y,t-1 _t-y_t-1 ^1- 2β ( β _t-y_t-1 ^24 _y ) 1β = = [(4ηβ)p/β‖ζ,t−1‖p‖t−t−1‖p−2pβ]1p[(β‖t−t−1‖24η)pβ(p−1)]p−1p [ ( 4 _yβ )^p/β _y,t-1 ^p _t-y_t-1 ^p- 2pβ ] 1p [ ( β _t-y_t-1 ^24 _y ) pβ(p-1) ] p-1p ≤ Young’s Inequality 1 p ( 4η y β ) p β ‖ζ_y, t−1t-1‖^p‖y_t-y_t−1t-1‖^p- 2p β + p-1 p ( β ‖ y t -y t−1t-1 ‖ 2 4η y ) p β(p-1) We let β=(p−1)/pβ= (p-1)p to get ⟨ζ,t−1,t−t−1⟩≤ _y,t-1,y_t-y_t-1 ≤ (4p−4)p−1p‖t−t−1‖2−pηp−1‖ζ,t−1‖p+‖t−t−1‖24η (4p-4)^p-1p^p\|y_t-y_t-1\|^2-p _y^p-1\| _y,t-1\|^p+ _t-y_t-1 ^24 _y ≤ ≤ D2−pηp−1‖ζ,t−1‖p+‖t−t−1‖24η, D_Y^2-p _y^p-1\| _y,t-1\|^p+ _t-y_t-1 ^24 _y, (A.34) where the last inequality holds due to (4p−4)p−1/p≤1 (4p-4)^p-1p^p≤ 1 for p∈(1,2]p∈(1,2] and tt≥1\y_t\_t≥ 1 in the bounded dom(h)dom (h ). Substitute (A.34) into (A.33), then taking the expectation88 8 Suppose that y is σ(ξt−2,…,ξ1)σ( _t-2,…, _1)-measurable, we first take the condtional expectation w.r.t. σ(ξt−2,…,ξ1)σ( _t-2,…, _1), then we take the full expectation. on both sides of the resulting inequality, we can get [f(t−1,)−f(t−1,t−1)] \ E [f(x_t-1,y)-f(x_t-1,y_t-1) ] ≤[⟨∇f(t−1,t−1),−t−1⟩] \ ≤E [ _yf(x_t-1,y_t-1),y-y_t-1 ] ≤[h()−h(t)+‖t−1−‖2−‖t−‖22η+f(t−1,t)−f(t−1,t−1)] \ ≤E [h(y)-h(y_t)+ \|y_t-1-y\|^2-\|y_t-y\|^22 _y+f(x_t-1,y_t)-f(x_t-1,y_t-1) ] +[(L2−14η)‖t−t−1‖2]+D2−pηp−1σp +E [ ( L2- 14 _y )\|y_t-y_t-1\|^2 ]+D_Y^2-p _y^p-1σ^p ≤[h()−h(t)+‖t−1−‖2−‖t−‖22η+f(t−1,t)−f(t−1,t−1)]+D2−pηp−1σp, \ ≤E [h(y)-h(y_t)+ \|y_t-1-y\|^2-\|y_t-y\|^22 _y+f(x_t-1,y_t)-f(x_t-1,y_t-1) ]+D_Y^2-p _y^p-1σ^p, where the first inequality holds due to the concavity of f(t−1,⋅)f(x_t-1,·) w.r.t. y, and the last inequality holds because we let η≤1/2L _y≤ 1/2L. The above inequality is equivently to [f^(t−1,)−f^(t−1,t)]≤[‖t−1−‖2−‖t−‖22η]+D2−pηp−1σp. E [ f(x_t-1,y)- f(x_t-1,y_t) ]≤E [ \|y_t-1-y\|^2-\|y_t-y\|^22 _y ]+D_Y^2-p _y^p-1σ^p. (A.35) To upper bound [Φ(t−1)−f^f(t−1,t−1)]E [ (x_t-1)- ff(x_t-1,y_t-1) ], we consider folloiwng decomposition: Φ(t−1)−f^(t−1,t−1)= (x_t-1)- f(x_t-1,y_t-1)= Φ(t−1)−f^(t−1,)+f^(t−1,)−f^(t−1,t) (x_t-1)- f(x_t-1,y)+ f(x_t-1,y)- f(x_t-1,y_t) + + f^(t−1,t)−f^(t,t)+f^(t,t)−f^(t−1,t−1). f(x_t-1,y_t)- f(x_t,y_t)+ f(x_t,y_t)- f(x_t-1,y_t-1). By the G-Lipschitz continuity of f(⋅,t)f(·,y_t) w.r.t. x, we have f^(t−1,t)−f^(t,t)=f(t−1,t)−f(t,t)≤G‖t−1−t‖≤Gη(‖ζ,t−1‖+G). f(x_t-1,y_t)- f(x_t,y_t)=f(x_t-1,y_t)-f(x_t,y_t)≤ G\|x_t-1-x_t\|≤ G _x(\| _x,t-1\|+G). Therefore, we can obtatin [Φ(t−1)−f^(t−1,t−1)] \ E [ (x_t-1)- f(x_t-1,y_t-1) ] ≤[Φ(t−1)−f^(t−1,)]+[‖t−1−‖2−‖t−‖2]2η+[f^(t,t)−f^(t−1,t−1)] \ ≤E [ (x_t-1)- f(x_t-1,y) ]+ E [\|y_t-1-y\|^2-\|y_t-y\|^2 ]2 _y+E [ f(x_t,y_t)- f(x_t-1,y_t-1) ] +D2−pηp−1σp+G(σ+G)η \ +D_Y^2-p _y^p-1σ^p+G(σ+G) _x (A.36) In the subsequent analysis, we will denote Φ(t)−f^(t,t) (x_t)- f(x_t,y_t) as δ^t δ_t. Without loss of generality, we assume that T is large enough to be divided into N windows of width M, then ∑t=1Tδ^t=∑n=0N∑j=1Mδ^nM+j _t=1^T δ_t= _n=0^N _j=1^M δ_nM+j. For δ^t δ_t in the n+1n+1-th block, i.e., t=nM+j,j=1,…,Mt=nM+j,\,j=1,…,M, we substitute =nM+1⋆=Δargmaxf(nM+1,)−h()y=y _nM+1 = _y \f(x_nM+1,y)-h(y) \ into (A.36) to get [δ^nM+j]≤[Φ(nM+j)−f^(nM+j,nM+1⋆)]+[‖nM+j−nM+1⋆‖2−‖nM+j+1−nM+1⋆‖2]2η \ E [ δ_nM+j ]≤E [ (x_nM+j)- f(x_nM+j,y _nM+1) ]+ E [\|y_nM+j-y _nM+1\|^2-\|y_nM+j+1-y _nM+1\|^2 ]2 _y +[f^(nM+j+1,nM+j+1)−f^(nM+j,nM+j)]+D2−pηp−1σp+G(σ+G)η \ +E [ f(x_nM+j+1,y_nM+j+1)- f(x_nM+j,y_nM+j) ]+D_Y^2-p _y^p-1σ^p+G(σ+G) _x We sum both sides of abonve inequality from j=1j=1 to M to obtain ∑j=1M[δ^nM+j]≤∑j=1M[Φ(nM+j)−f^(nM+j,nM+1⋆)]+[δnM+12]2η \ _j=1^ME [ δ_nM+j ]≤ _j=1^ME [ (x_nM+j)- f(x_nM+j,y _nM+1) ]+ E [ _nM+1^2 ]2 _y +[f^((n+1)M+1,(n+1)M+1)−f^(nM+1,nM+1)]+M[D2−pηp−1σp+G(σ+G)η]. \ +E [ f(x_(n+1)M+1,y_(n+1)M+1)- f(x_nM+1,y_nM+1) ]+M [D_Y^2-p _y^p-1σ^p+G(σ+G) _x ]. (A.37) Note that [Φ(nM+j)−f^(nM+j,nM+1⋆)] \ E [ (x_nM+j)- f(x_nM+j,y _nM+1) ] =[f^(nM+j,nM+j⋆)−f^(nM+j,nM+1⋆)] \ =E [ f(x_nM+j,y _nM+j)- f(x_nM+j,y _nM+1) ] =[f(nM+j,nM+j⋆)−f(nM+1,nM+j⋆)+f^(nM+1,nM+j⋆)−f^(nM+j,nM+1⋆)] \ =E [f(x_nM+j,y _nM+j)-f(x_nM+1,y _nM+j)+ f(x_nM+1,y _nM+j)- f(x_nM+j,y _nM+1) ] ≤[f(nM+j,nM+j⋆)−f(nM+1,nM+j⋆)+f^(nM+1,nM+1⋆)−f^(nM+j,nM+1⋆)] \ ≤E [f(x_nM+j,y _nM+j)-f(x_nM+1,y _nM+j)+ f(x_nM+1,y _nM+1)- f(x_nM+j,y _nM+1) ] =[f(nM+j,nM+j⋆)−f(nM+1,nM+j⋆)+f(nM+1,nM+1⋆)−f(nM+j,nM+1⋆)] \ =E [f(x_nM+j,y _nM+j)-f(x_nM+1,y _nM+j)+f(x_nM+1,y _nM+1)-f(x_nM+j,y _nM+1) ] ≤2G[‖nM+j−nM+1‖]=η[‖∑s=2j∇f(nM+s−1,nM+s−1,ξnM+s−1)‖] \ ≤ 2GE [ _nM+j-x_nM+1 ]= _xE [ \| _s=2^j _xf(x_nM+s-1,y_nM+s-1, _nM+s-1) \| ] ≤2G(G+σ)(j−1)η, \ ≤ 2G(G+σ)(j-1) _x, which means ∑j=1M[Φ(nM+j)−f^(nM+j,nM+1⋆)]≤2G(G+σ)η∑j=1M(j−1)=M(M−1)G(G+σ) _j=1^ME [ (x_nM+j)- f(x_nM+j,y _nM+1) ]≤ 2G(G+σ) _x _j=1^M(j-1)=M(M-1)G(G+σ) (A.38) Substitute (A.38) into (A.37) and sum both sides of the resulting inequality from n=0n=0 to N to get ∑t=1T[δ^t]≤MTG(σ+G)η+TD2−pηp−1σp+TD22Mη+[f^(T+1,T+1)−f^(1,1)]. _t=1^TE [ δ_t ]≤ MTG(σ+G) _x+TD_Y^2-p _y^p-1σ^p+ TD_Y^22M _y+E [ f(x_T+1,y_T+1)- f(x_1,y_1) ]. (A.39) Note that δnM+1≤D _nM+1≤ D_Y because t⋆t≥1\y_t \_t≥ 1 also in the bounded dom(h)dom (h ). In addtion, we have [f^(T+1,T+1)−f^(1,1)]= E [ f(x_T+1,y_T+1)- f(x_1,y_1) ]= [f(T+1,T+1)−f(1,T+1)+f^(1,T+1)−f^(1,1)] E [f(x_T+1,y_T+1)-f(x_1,y_T+1)+ f(x_1,y_T+1)- f(x_1,y_1) ] ≤ ≤ G[∥T+1−1]+Φ(1)−f^(1,1)≤TG(G+σ)η+δ^1. GE [\|x_T+1-x_1 ]+ (x_1)- f(x_1,y_1)≤ TG(G+σ) _x+ δ_1. (A.40) Now, we substitute (A.40) into (A.39) to get ∑t=1T[δ^t]≤δ^1+(M+1)TG(σ+G)η+TD2−pηp−1σp+TD22Mη. _t=1^TE [ δ_t ]≤ δ_1+(M+1)TG(σ+G) _x+TD_Y^2-p _y^p-1σ^p+ TD_Y^22M _y. We let M=⌈D2G(G+σ)ηη⌉M= D_Y 2G(G+σ) _x _y , then we can have ∑t=1T[δ^t]≤ _t=1^TE [ δ_t ]≤ δ^1+(D2G(G+σ)ηη+2)TG(σ+G)η+TDG(G+σ)η2η+TD2−pηp−1σp δ_1+ ( D_Y 2G(G+σ) _x _y+2 )TG(σ+G) _x+TD_Y G(G+σ) _x2 _y+TD_Y^2-p _y^p-1σ^p = = δ^1+TD2G(G+σ)η+2TG(σ+G)η+TD2−pηp−1σp δ_1+TD_Y 2G(G+σ) _x _y+2TG(σ+G) _x+TD_Y^2-p _y^p-1σ^p ■ A.2.1 Prove the complexity Proof. We assume that that η≤1/2L _y≤ 1/2L, then by Lemma A.4 and Lemma A.5, we can get [(∑t=1Tη‖∇Φ1/2L(t)‖2)p/2] \ E [ ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ] ≲(Lη)p/2[(∑t=1T(Φ(t)−f^(t,t)))p/2]+Δ1p/2+(LT)p/2ηp(Gp+σp)+σpT1−p/2ηp/2 \ \; \;(L _x) p2E [ ( _t=1^T( (x_t)- f(x_t,y_t)) ) p2 ]+ _1 p2+(LT) p2 _x^p(G^p+σ^p)+σ^pT^1- p2 _x p2 ≲(Lη[∑t=1T(Φ(t)−f^(t,t))])p/2+Δ1p/2+ηp(LT(G2∨σ2))p/2+σpT1−p/2ηp/2 \ \; \; (L _xE [ _t=1^T( (x_t)- f(x_t,y_t)) ] ) p2+ _1 p2+ _x^p(LT(G^2 σ^2)) p2+σ^pT^1- p2 _x p2 ≲(Lηδ^1+TLη(DG(G∨σ)η+G(σ∨G)η+D2−pηp−1σp))p/2 \ \; \; (L _x δ_1+TL _x (D_Y G(G σ) _x _y+G(σ G) _x+D_Y^2-p _y^p-1σ^p ) ) p2 +Δ1p/2+ηp(LT(G2∨σ2))p/2+σpT1−p/2ηp/2 + _1 p2+ _x^p(LT(G^2 σ^2)) p2+σ^pT^1- p2 _x p2 Next, we multiply both sides of the above inequality by (Tη)−p/2(T _x)^-p/2 to obtain the right-hand side as L.H.S.≲ .H.S.\; \; (Δ1Tη)p/2+((G2∨σ2)Lη)p/2+(LG(σ∨G)η)p/2 ( _1T _x ) p2+ ((G^2 σ^2)L _x ) p2+ (LG(σ G) _x ) p2 + + (LDG(G∨σ)η+LD2−pηp−1σp)p/2+(Lδ^1T)p/2+σpTp−1 (LD_Y G(G σ) _x _y+LD_Y^2-p _y^p-1σ^p ) p2+ ( L δ_1T ) p2+ σ^pT^p-1 We denote maxa,b \a,b\ and mina,b \a,b\ as a∨ba b and a∧ba b, respectively. Let η=C,1T−2p−13p−2∧C,2T−1/2 _x=C_x,1T^- 2p-13p-2 C_x,2T^-1/2 and η=1/2L∧CT−13p−2 _y=1/2L C_yT^- 13p-2, where C,1,C,2C_x,1,\,C_x,2 and C_y are all positive constants, and substitute them into the above equation to obtain the right hand side as L.H.S. \ L.H.S. ≲(Δ1T(T2p−13p−2C,1∨TC,2))p/2+(C,2(G2∨σ2)LT)p/2+(C,2LG(σ∨G)T)p/2+(Lδ^1T)p/2+σpTp−1 \ \; \; ( _1T ( T 2p-13p-2C_x,1 TC_x,2 ) ) p2+ ( C_x,2(G^2 σ^2)L T ) p2+ ( C_x,2LG(σ G) T ) p2+ ( L δ_1T ) p2+ σ^pT^p-1 +(LDG(G∨σ)C,1T2p−13p−2(2L∨T13p−1C)+Cp−1LD2−pσpTp−13p−2)p/2 \ + (LD_Y G(G σ)C_x,1T 2p-13p-2 (2L T 13p-1C_y )+ C_y^p-1LD_Y^2-pσ^pT p-13p-2 ) p2 ≲(Δ1/C,1Tp−13p−2)p/2+(Δ1/C,2T)p/2+(LC,2(G2∨σ2∨Gσ)T)p/2+(Lδ^1T)p/2+σpTp−1 \ \; \; ( _1/C_x,1T p-13p-2 ) p2+ ( _1/C_x,2 T ) p2+ ( LC_x,2(G^2 σ^2 Gσ) T ) p2+ ( L δ_1T ) p2+ σ^pT^p-1 +(LDL(G2∨Gσ)C,1T2p−16p−4)p/2+(LD(G2∨Gσ)C,1/CTp−13p−2+Cp−1LD2−pσpTp−13p−2)p/2 \ + ( LD_Y L(G^2 Gσ)C_x,1T 2p-16p-4 ) p2+ ( LD_Y (G^2 Gσ)C_x,1/C_yT p-13p-2+ C_y^p-1LD_Y^2-pσ^pT p-13p-2 ) p2 (A.41) If we let C,1=(Δ1/L)2p−13p−2(Dσ)p3p−2(Gσ∨G2)p−13p−2,C,2=Δ1L(G2∨σ2∨Gσ), C_x,1= ( _1/L) 2p-13p-2(D_Yσ) p3p-2(Gσ G^2) p-13p-2, C_x,2= _1L(G^2 σ^2 Gσ), C=C=D3p−43p−2σ3p3p−2(Δ1(Gσ∨G2)L)13p−2. C_y=C_y= D_Y 3p-43p-2σ 3p3p-2 ( _1(Gσ G^2)L ) 13p-2. Then, we get Δ1/C,1=LD(Gσ∨G2)C,1/C=LD2−pCp−1σp=(Δ1(Gσ∨G2))p−13p−2(Dσ)p3p−2L2p−13p−2 _1/C_x,1=LD_Y (Gσ G^2)C_x,1/C_y=LD_Y^2-pC_y^p-1σ^p=( _1(Gσ G^2)) p-13p-2(D_Yσ) p3p-2L 2p-13p-2 Δ1/C,2=LC,2(G2∨σ2∨Gσ)=Δ1L(G2∨σ2∨Gσ) _1/C_x,2=LC_x,2(G^2 σ^2 Gσ)= _1L(G^2 σ^2 Gσ) LDL(G2∨Gσ)C,1=L7p−56p−4D5p−46p−4(Δ1(G2∨Gσ))2p−16p−4σp6p−4 LD_Y L(G^2 Gσ)C_x,1= L 7p-56p-4D_Y 5p-46p-4( _1(G^2 Gσ)) 2p-16p-4σ p6p-4 (A.42) Now, we substitute (A.42) into (A.41) to get [(∑t=1Tη‖∇Φ1/2L(t)‖2)p/2] \ E [ ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ] ≲((Δ1(Gσ∨G2))p−13p−2(Dσ)p3p−2L2p−13p−2Tp−13p−2)p/2+(L7p−56p−4D5p−46p−4σp6p−4(Δ1(G2∨Gσ)T)2p−16p−4)p/2 \ \; \; ( ( _1(Gσ G^2)) p-13p-2(D_Yσ) p3p-2L 2p-13p-2T p-13p-2 ) p2+ ( L 7p-56p-4D_Y 5p-46p-4σ p6p-4 ( _1(G^2 Gσ)T ) 2p-16p-4 ) p2 +(Δ1L(G2∨σ2∨Gσ)T)p/2+(Lδ^1T)p/2+σpTp−1. \ + ( _1L(G^2 σ^2 Gσ)T ) p2+ ( L δ_1T ) p2+ σ^pT^p-1. (A.43) If, we have σ≥Gσ≥ G99 9 Note that we used σ≥Gσ≥ G, which gives G2∨σG≤G2∨σ2G^2 σ G≤ G^2 σ^2. In fact, we retained the G2G^2 term, so this does not cause the degeneration of the dependence of the upper bound on G., then we can get [(∑t=1Tη‖∇Φ1/2L(t)‖2)p/2] \ E [ ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ] ≲(Δ1p−13p−2σDp3p−2L2p−13p−2Tp−13p−2∨(Δ1G2)p−13p−2(σD)p3p−2L2p−13p−2Tp−13p−2)p/2+(Δ12p−16p−4σL7p−56p−4D5p−46p−4T2p−16p−4)p/2 \ \; \; ( _1 p-13p-2σ D_Y p3p-2L 2p-13p-2T p-13p-2 ( _1G^2) p-13p-2(σ D_Y) p3p-2L 2p-13p-2T p-13p-2 ) p2+ ( _1 2p-16p-4 σL 7p-56p-4D_Y 5p-46p-4T 2p-16p-4 ) p2 +(Δ1L(G2∨σ2)T)p/2+(Lδ^1T)p/2+σpTp−1 \ + ( _1L(G^2 σ^2)T ) p2+ ( L δ_1T ) p2+ σ^pT^p-1 (A.44) To achieve [(∑t=1Tη‖∇Φ1/2L(t)‖2)p/2]≤ϵpE [ ( _t=1^T _x\|∇ _1/2L(x_t)\|^2 ) p2 ]≤ε^p, the total gradient complexity is upper bounded by (Δ1Dpp−1L2p−1p−1(σ3p−2p−1∨G2σp−1)ϵ6p−4p−1+Δ1σ3p−22p−1L7p−52p−1D5p−42p−1ϵ12p−82p−1+(σϵ)p−1+Δ1L(G2∨σ2)ϵ4+Lδ^1ϵ2) ( _1D_Y pp-1L 2p-1p-1(σ 3p-2p-1 G^2σ pp-1)ε 6p-4p-1+ _1σ 3p-22p-1L 7p-52p-1D_Y 5p-42p-1ε 12p-82p-1+ ( σε ) pp-1+ _1L(G^2 σ^2)ε^4+ L δ_1ε^2 ) If, in addition, we have σ,L,D≥1σ,L,D_Y≥ 1 and ϵ≤1ε≤ 1, then we can get σ3p−2p−1≥σ3p−22p−1σ 3p-2p-1≥σ 3p-22p-1, Dpp−1≥D5p−42p−1D_Y pp-1≥ D_Y 5p-42p-1, L2p−1p−1≥L7p−52p−1L 2p-1p-1≥ L 7p-52p-1 and ϵ−6p−4p−1≥ϵ−12p−82p−1ε^- 6p-4p-1≥ε^- 12p-82p-1, which means the total gradient complexity is upper bounded by (Δ1Dpp−1L2p−1p−1(σ3p−2p−1∨G2σp−1)ϵ6p−4p−1+(σϵ)p−1+Δ1L(G2∨σ2)ϵ4+Lδ^1ϵ2) ( _1D_Y pp-1L 2p-1p-1(σ 3p-2p-1 G^2σ pp-1)ε 6p-4p-1+ ( σε ) pp-1+ _1L(G^2 σ^2)ε^4+ L δ_1ε^2 ) (A.45) ■ A.3 Proof of Theorem 3.4 Additional notations For positive integers T,a,bT,a,b, [T][T] represents the set 1,…,T1,…,T, and [a:b][a:b] represents the set a,a+1,…,b\a,a+1,…,b\. Given a real vector v, we denote v[i]v[i] as the i-th coordinate of v, supp(v)=Δi∣v[i]≠0supp(v) =\i v[i]≠ 0\. ‖v‖2\|v\|_2 and ‖v‖∞\|v\|_∞ are the ℓ2 _2 and ℓ∞ _∞ norm of v, respectively. For a differentiable function f, ∇if() _if(v) represents ∇f()[i]∇ f(v)[i]. Given a matrix A, we let Ai,jA_i,j be the i,ji,j-th entry of A. In∈n×nI_n∈ n× n represents the identity matrix, and ie_i represents the standard unit vector with all zeros except for the i-th coordinate, which is 11. Necessary definitions We introduce some definitions needed to prove the lower bound. They basically come from [3] and [30] extend them to constrained optimization. A differentiable function f:⊂d↦f:V⊂ d is a (first-order) zero chain over V if for any ∈v and 1≤i≤d1≤ i≤ d, ∀j∈[i:d],[j]=0∀ j∈[i:d],v[j]=0 implies that ∀k∈[i+1:d]∀ k∈[i+1:d], ∇kf()=0 _kf(v)=0. Given a ojective f:↦f:X and its associated first-order stochastic oracle ∇f(,ξ)∇ f(x,ξ), if (t+1)(t+1)-th iterate t+1x_t+1 of a algorithm A with ∇f(,ξ)∇ f(x,ξ) awalys satisfies t+1∈Π()∣supp()⊂∪0≤i≤t(supp(i)∪supp(∇f(i,ξ)))x_t+1∈ \ _X(v) (v)⊂ _0≤ i≤ t (supp(x_i) (∇ f(x_i,ξ)) ) \, we say A is a stochasitc first-order (zero-respecting) algorithm. Moreover, if supp()⊂[1:i−1]supp(x)⊂[1:i-1] implies that ℙ(supp(∇f(,ξ))⊈[1:i−1])≤qP (supp(∇ f(x,ξ)) [1:i-1] )≤ q and ℙ(supp(∇f(,ξ))⊂[1:i])=1P (supp(∇ f(x,ξ))⊂[1:i] )=1 hold, we say f is a probability-q zero-chain with respect to the given oracle ∇f(,ξ)∇ f(x,ξ). The hard instance The hard instance for min-max problems comes from [30]. Specifically, we decompose the minimization variable into ∈Tx∈ T and ∈T−1z∈ T-1, and the maximization variable into ¯=Δ[(1),…,(T−1)] y =[y^(1),…,y^(T-1)], where (i)∈ny^(i)∈ n. The formal expression of the instance f^:T×T−1×n(T−1)↦ f: T× T-1× n(T-1) is following f^(,,¯)=Δ f(x,z, y) = −Ψ(1)Φ([1])+∑2T[Ψ(−[i])Φ(−[i])−Ψ([i])Φ([i])] - (1) (x[1])+ _2^T [ (-z[i]) (-x[i])- (z[i]) (x[i]) ] +∑i=1T−1h([i],[i+1],¯(i))+∑i=1T−1(c1[i]2+c2[i+1]2), + _i=1^T-1h(x[i],z[i+1], y^(i))+ _i=1^T-1 (c_1x[i]^2+c_2z[i+1]^2 ), (A.46) where Ψ(u)=Δ0,u≤1/2exp(1−1(2u−1)2),u>1/2,Φ(u)=Δe∫−∞uexp(−t22)t, \ (u) = cases0,&u≤ 1/2\\ (1- 1(2u-1)^2 ),&u>1/2 cases, (u) = e _-∞^u (- t^22 )dt, h(x,z,)=ΔCn[−12T(1n2In+A)+x,zT],x,zT=Δx−12z, \ h(x,z,y) = Cn [- 12y^T ( 1n^2I_n+A )y+b^T_x,zy ], ^T_x,z =xe_1- 12ze_n, A=ΔA1,1=An,n=1Ai,i=2,∀i∈[2:n]Ai,i+1=Ai+1,i=−1∀i∈[1:n−1]0,otherwise. \ A = casesA_1,1=A_n,n=1\\ A_i,i=2,&∀ i∈[2:n]\\ A_i,i+1=A_i+1,i=-1&∀ i∈[1:n-1]\\ 0,&otherwise cases. Here we stipulate [1]=1z[1]=1. We arrange x, z, ¯ y as follows =Δ([1],(1),[2],[2],(2),[3],…,[i],(i),[i+1],…,[T],[T]) u = (x[1],y^(1),z[2],x[2],y^(2),z[3],…,x[i],y^(i),z[i+1],…,z[T],x[T] ) We let ℬRd=Δx∈d∣‖∞≤RB^d_R = \x∈ d \|x\|_∞≤ R \. According to Lemma 11, 12 in [30], f f has following key properties for some numercial constants c1,c2,C,R1,R2c_1,c_2,C,R_1,R_2 and engough large n: 1. f f is a zero-chain with respect to u 2. f f is ℓ -smooth w.r.t. (,,¯)(x,z, y) and is 1/n31/n^3-strongly convcave w.r.t. ¯ y. Moreover, ∀(,,¯)∈ℬR1T×ℬR1T−1×ℬnR2n(T−1),‖∇f^(,,¯)‖∞≤G∀(x,z, y) ^T_R_1×B^T-1_R_1×B^n(T-1)_nR_2,\|∇ f(x,z, y)\|_∞≤ G. 3. Define Φ^(,)=Δmax¯∈ℬnR2n(T−1)f^(,,¯) (x,z) = _ y ^n(T-1)_nR_2 f(x,z, y), then Φ^(,) (x,z) is ℓΦ _ -smooth for any (,)∈ℬR1T×ℬR1T−1(x,z) ^T_R_1×B^T-1_R_1, and Φ^(,)−inf(,)∈ℬR1T×ℬR1T−1Φ^(,)≤12T ( 0, 0)- _(x,z) ^T_R_1×B^T-1_R_1 (x,z)≤ 12T. 4. If |[i]|<1|z[i]|<1 for some i∈[2:T]i∈[2:T], then ℓΦ^‖ΠℬR1T×ℬR1T−1((,)−ℓΦ^−1∇Φ(,))−(,)‖>1/3 _ _B^T_R_1×B^T-1_R_1((x,z)- _ ^-1∇ (x,z))-(x,z) >1/3. Here, we emphasize that ℓ , G, ℓΦ _ are all numerical constants. Proof. We state that the proof mostly follows the proof of Theorem 2 in [30]. The difference is that we need to modify the constructed stochastic oracle to make it match the heavy-tailed noise assumption. The instance f:ℬλR1T×ℬλR1T−1×ℬλnR2n(T−1)↦f:B^T_λ R_1×B^T-1_λ R_1×B^n(T-1)_λ nR_2 on which the lower bound is established is actually the rescaled f f, that is: f(,,¯)=ΔLλ2ℓf^(λ,λ,¯λ), f(x,z, y) = Lλ^2 f ( xλ, zλ, yλ ), (A.47) where λ is to be determined. By Property 1, f is still a zero-chain with respect to u after rescaling. Let Φ(,)=Δmax¯∈ℬλnR2n(T−1)f(,,¯) (x,z) = _ y ^n(T-1)_λ nR_2f(x,z, y), then the analysis in [30] tells that Φ(,)=(Lλ2/ℓ)Φ^(/λ,/λ) (x,z)=(Lλ^2/ ) (x/λ,z/λ), which means the smoothness parameter of Φ is ℓΦ=ℓΦ^L/ℓ _ = _ L/ by Property 3. Moreover, n=⌊(κ/ℓ)1/3⌋n= (κ/ )^1/3 and T=⌊LΔ1432ℓϵ2⌋T= L _1432 ε^2 can guarantee that f is L-smooth w.r.t. (,,¯)(x,z, y) and is μ-stongly concave w.r.t. ¯ y, and Φ(,)−inf(,)∈ℬλR1T×ℬλR1T−1Φ(,)≤Δ1 ( 0, 0)- _(x,z) ^T_λ R_1×B^T-1_λ R_1 (x,z)≤ _1. Therefore, f belongs to the objective functions that conforms to the NC-SC setting with given parameters L, μ and Δ1 _1. For convenience, we denote Gap(ℓ,f,,,) Gap( ,f,x,z,C) as ℓ‖Π((,)−ℓ−1∇f(,))−(,)‖2 \| _C((x,z)- ^-1∇ f(x,z))-(x,z) \|_2. If [T]=0z[T]=0 and λ=6ℓϵ/Lλ=6 ε/L, we can get Gap(ℓΦ,f,,,ℬλR1T×ℬλR1T−1)=Lλℓ⋅Gap(ℓΦ^,f^,/λ,/λ,ℬR1T×ℬR1T−1)>Lλ3ℓ=2ϵ, Gap( _ ,f,x,z,B^T_λ R_1×B^T-1_λ R_1)= Lλ · Gap( _ , f,x/λ,z/λ,B^T_R_1×B^T-1_R_1)> Lλ3 =2ε, (A.48) where the inequality holds due to the Property 4. The construction of the stochastic oracle is in the same way as [3] and [30]. Given q∈(0,1]q∈(0,1], the oracle ∇f(,ξ)∇ f( u,ξ) is defined as ∇f(,ξ)[i]=Δξq∇if(),i=infii∣∇if()=0∇if(),otherwise, ∇ f( u,ξ)[i] = cases ξq _if( u),&i= _i \i _if( u)=0 \\\ _if( u),&otherwise cases, (A.49) where ξ∼Bernoulli(q)ξ (q). Note that ‖∇f(,,¯)‖∞≤GLλ/ℓ\|∇ f(x,z, y)\|_∞≤ GLλ/ since we have already know that ‖∇f^(,,¯)‖∞≤G\|∇ f(x,z, y)\|_∞≤ G by Property 2. Then, the p-the central moment of ∇f(,ξ)∇ f( u,ξ) is bounded by ξ[‖∇f(,ξ)−∇f()‖p] _ξ [ ∇ f( u,ξ)-∇ f( u) ^p ] =ξ[|∇f(,ξ)[i⋆]−∇i⋆f()|p] =E_ξ [ |∇ f( u,ξ)[i ]- _i f( u) |^p ] =ξ[|ξ/q−1|p]⋅|∇i⋆f()|p =E_ξ [ |ξ/q-1 |^p ]· | _i f( u) |^p =(1−q+(1−q)pqp−1)(Lλℓ)|∇i⋆f^(/λ)|p = (1-q+ (1-q)^pq^p-1 ) ( Lλ ) | _i f ( u/λ ) |^p ≤(1−q+(1−q)pqp−1)(Lλℓ)(GLλℓ)p≤1qp(GLλℓ)p ≤ (1-q+ (1-q)^pq^p-1 ) ( Lλ ) ( GLλ )^p≤ 1q^p ( GLλ )^p Let q=min1,(GLλσℓ)p−1q= \1, ( GLλσ ) pp-1 \ to make the bound no greater than σpσ^p. Note that Lemma 3 in [3] implies that f()f( u) with ∇f(,ξ)∇ f( u,ξ) is a probability-q zero-chain, then by Lemma 1 in [3], we have ∀t≤w1nT−log2(1/δ)2q⇒ℙ(t[T]=0)≥1−δ⇒ℙ(Gap(ℓΦ,f,t,t,ℬλR1T×ℬλR1T−1)>2ϵ)≥1−δ ∀ t≤ w_1nT- _2(1/δ)2q (z_t[T]=0 )≥ 1-δ ( Gap( _ ,f,x_t,z_t,B^T_λ R_1×B^T-1_λ R_1)>2ε )≥ 1-δ where t,t,¯tt≥1\x_t,z_t, y_t\_t≥ 1 is generated by a certatin first-order (zero-respecting) algorithm A with (1,1,¯)=(,,)(x_1,z_1, y)=( 0, 0, 0), and w1w_1 is some constant. Furthermore, according to Markov’s inequality, we have [Gap(ℓΦ,f,t,t,ℬλR1T×ℬλR1T−1)]≥(1−δ)⋅2ϵ [ Gap( _ ,f,x_t,z_t,B^T_λ R_1×B^T-1_λ R_1) ]≥(1-δ)· 2ε Choosing δ=1/2δ=1/2, then for any t≤w2nTq=w2⋅⌊(κℓ)1/3⌋⋅⌊LΔ1ℓϵ2⌋⋅max1,(σℓGLλ)p−1≥w3⋅Δ1Lκ1/3σp−1ϵ3p−2p−1, t≤ w_2nTq=w_2· ( κ )^1/3 · L _1 ε^2 · \1, ( σ GLλ ) pp-1 \≥ w_3· _1Lκ^1/3σ pp-1ε 3p-2p-1, where w2,w3w_2,w_3 are some constants, t,tx_t,z_t cannot be a ϵε-stationary point of Φ over ℬλR1T×ℬλR1T−1B^T_λ R_1×B^T-1_λ R_1. Therefore, in order to find a ϵε-stationary point, the total number of iterations of A is at least Ω(Δ1Lκ1/3σp−1ϵ−3p−2p−1) ( _1Lκ^1/3σ pp-1ε^- 3p-2p-1) ■ Appendix B Missing proofs in section 4 B.1 Proof of Theorem 4.1 Additional notations ri()ri (S ) and cl()cl (S ) represent the relative interior and closure of a given subset S, respectively. For a function f:d↦(−∞,+∞]f: d (-∞,+∞], we let dom(f)=Δ∣f()<+∞dom (f ) = \x f(x)<+∞ \. If dom(f)≠∅dom (f )≠ , we say that f is proper. For the proper function f, we can define its nonempty epigraph epi(f)=Δ(,t)∣t∈,f()≤tepi(f) =\(x,t) t∈ ,f(x)≤ t\. The following lemmas in convex analysis are needed for our proof. Lemma B.1. [19, Lemma C.5] Given a closed ball (,η)=Δ∣‖−‖≤ηB(z,η) = \x \|x-z\|≤η \, then the normal cone (,η)()=Δv∣⟨v,−⟩≤0,∀∈(,η) N_B(z,η) (x ) = \v v,y-x ≤ 0,\; (z,η) \ is (,η)()=v∣λ(−),∀λ≥0‖−‖=η,‖−‖<η,∅‖−‖>η. N_B(z,η) (x )= cases \v λ(x-z),\;∀λ≥ 0 \& -z =η,\\ \ 0 \& -z <η,\\ & -z >η. cases (B.1) Lemma B.2. Let f:d↦(−∞,+∞]f: d (-∞,+∞] is proper and convex. Given a nonempty convex set X, where dom(f)∩X≠∅dom (f )∩ X≠ , then for ∈dom(f)∩Xx (f )∩ X, we have ri(X)∩ri(dom(f))≠∅and∂(f+δX)()≠∅⟹∂f()≠∅, (X ) (dom (f ) )≠ \;and\;∂(f+ _X)(x)≠ ∂ f(x)≠ , (B.2) where δX _X is the indicator function about X. Remark B.1. The relative interior condition ri(X)∩ri(dom(f))≠∅ri (X ) (dom (f ) )≠ is necessary. Assume that f(x)=−xf(x)=- x for x≥0x≥ 0 and f(x)≡+∞f(x)≡+∞ for x<0x<0, then f is proper and convex. If =0X=\0\, then ∂(f+δX)(0)=(−∞,+∞)∂(f+ _X)(0)=(-∞,+∞), while ∂f(0)=∅∂ f(0)= . Proof. First, we denote fδf_δ as f+δXf+ _X, then fδf_δ is also a proper and convex function, and epi(fδ)=epi(f)∩(X×)epi (f_δ )=epi (f )∩(X× ). Since ri(X)∩ri(dom(f))≠∅ri (X ) (dom (f ) )≠ , we know that ri(epi(f))∩ri(X×)≠∅ri (epi (f ) ) (X× )≠ also holds true. Then by the noraml cone intersection rule [45, Corollary 23.8.1], we get epi(fδ)((,f()))=epi(f)((,f()))+X×((,f())) N_epi (f_δ ) ((x,f(x)) )= N_epi (f ) ((x,f(x)) )+ N_X× ((x,f(x)) ). Since ∂(fδ)()≠∅∂(f_δ)(x)≠ , then for any u∈∂fδ()u∈∂ f_δ(x), we have (u,−1)∈epi(fδ)((,f()))(u,-1)∈ N_epi (f_δ ) ((x,f(x)) ), and there exists (v,α)∈epi(f)((,f()))(v,α)∈ N_epi (f ) ((x,f(x)) ) and (w,β)∈X×((,f()))(w,β)∈ N_X× ((x,f(x)) ) such that (u,−1)=(v+w,α+β)(u,-1)=(v+w,α+β). Let s>0s>0, then by the definition of normal cone, we know that ⟨(w,β),(,f())−(,f()+s)⟩≤0 (w,β),(x,f(x))-(x,f(x)+s) ≤ 0 and ⟨(w,β),(,f())−(,f()−s)⟩≤0 (w,β),(x,f(x))-(x,f(x)-s) ≤ 0 are both ture. Hence, β must be 00, which means α=−1α=-1 and v∈∂f()v∈∂ f(x). ■ Lemma B.3. Let f:d↦(−∞,+∞]f: d (-∞,+∞] is proper, closed and convex. Given any ∈dom(f)z (f ), then argmin∈(,η)f() _x (z,η)f(x) (η>0η>0) is nonempty, closed and convex. Moreover, for any †∈argmin∈(,η)f()x ∈ _x (z,η)f(x), ∂f(†)≠∅∂ f(x )≠ and ∂(f+δ(,η))(†)=∂f(†)+(,η)(†)∂(f+ _B(z,η))(x )=∂ f(x )+ N_B(z,η) (x ), where (,η)(†)=Δv∣⟨v,−†⟩≤0,∀∈(,η) N_B(z,η) (x ) = \v v,x-x ≤ 0,\; (z,η) \. Proof. Since f is a proper closed function, meaning f is lower semi-continuous, and because dom(f)∩(,η)≠∅dom (f ) (z,η)≠ , then according to the Weierstrass extreme value theorem, argmin∈(,η)f()≠∅ _x (z,η)f(x)≠ and argmin∈(,η)f()⊂dom(f)∩(,η) _x (z,η)f(x) (f ) (z,η). Furthermore, since f is closed, we can conclude that for any γ, the level set Vγ=Δ∣f()≤γV_γ = \x f(x)≤γ \ is also closed. Let γ⋆=min∈(,η)f() _ = _x (z,η)f(x), then argmin∈(,η)f()=Vγ⋆∩(,η) _x (z,η)f(x)=V_ _ (z,η), so argmin∈(,η)f() _x (z,η)f(x) is closed. let 1,2∈argmin∈(,η)f()x_1,x_2∈ _x (z,η)f(x), then for any λ∈(0,1)λ∈(0,1), (1−λ)1+λ2∈dom(f)∩(,η)(1-λ)x_1+ _2 (f ) (z,η) also minimizes f on (,η)B(z,η) due to the convexity of f. Therefore, argmin∈(,η)f() _x (z,η)f(x) must be nonempty, closed and convex. Now, we prepare to show ri(dom(f))∩ri((,η))≠∅ri (dom (f ) ) (B(z,η) )≠ . If ∈ri(dom(f))z (dom (f ) ), we surely have ri(dom(f))∩ri((,η))≠∅ri (dom (f ) ) (B(z,η) )≠ . If ∉ri(dom(f))z (dom (f ) ), we still can get ∈cl(dom(f))=cl(ri(dom(f)))z (dom (f ) )=cl (ri (dom (f ) ) ) by [7, Proposition 1.3.5], which means there exists a sequence tt∈ℕ+∈ri(dom(f)) \x_t \_t _+ (dom (f ) ) such that limt→+∞‖t−‖=0 _t→+∞\|x_t-z\|=0. Therefore, there always exists a sufficiently large t such that ‖t−‖<η\|x_t-z\|<η, and thus ri(dom(f))∩ri((,η))≠∅ri (dom (f ) ) (B(z,η) )≠ . Moreover, we know that ∈∂(f+δ(,η))(†) 0∈∂(f+ _B(z,η))(x ) for any †∈argmin∈(,η)f()x ∈ _x (z,η)f(x) by the optimality condtion, then we can conclude that ∂f(†)≠∅∂ f(x )≠ by Lemma B.2. Furthermore, by [45, Theorem 23.8], we have ∂(f+δ(,η))(†)=∂f(†)+∂δ(,η)(†)∂ (f+ _B(z,η) )(x )=∂ f(x )+∂ _B(z,η)(x ) because ri(dom(f))∩ri((,η))≠∅ri (dom (f ) ) (B(z,η) )≠ , and it is easy to verify that ∂δ(,η)(†)=(,η)(†)∂ _B(z,η)(x )= N_B(z,η) (x ) ■ Proof. First, since r()r(x) is a proper, closed and convex function, we know that r^()=Δ⟨,⟩+r() r(x) = ,x +r(x) is also a proper, closed and convex function for any ∈dv∈ d, and dom(r^)=dom(r)dom ( r )=dom (r ). Then by Lemma B.3, argminx∈(,η)r^() _x (z,η) r(x) is a nonempty, closed and convex set, and for any †∈argminx∈(,η)r^()x ∈ _x (z,η) r(x), we have ∂r^(†)≠∅∂ r(x )≠ , which also means ∂r(†)≠∅∂ r(x )≠ . Moreover, we get ∈∂(r^+δ(,η))(†)=+∂r(†)+(,η)(†) 0∈∂( r+ _B(z,η))(x )=v+∂ r(x )+ N_B(z,η) (x ) by the optimality condition and Lemma B.3. If ∉+∂r(†) 0 +∂ r(x ) By Lemma B.1, we know that ‖†−‖=η\|x -z\|=η and ∃λ>0,gr(†)∈∂r(†)∃λ>0,g_r(x )∈∂ r(x ) such that =+gr(†)+λ(†−) 0=v+g_r(x )+λ(x -z). That is to say ‖†−‖=λ−1‖+gr(†)‖=η\|x -z\|=λ^-1\|v+g_r(x )\|=η, which means †=−η+gr(†)‖+gr(†)‖x =z-η v+g_r(x )\|v+g_r(x )\|. We then prove that in the assumption that ∀†∈⋆†,∉+∂r(†) _ , 0 +∂ r(x ), †x must be unique. If 1†,2†∈⋆†x _1,x _2 _ and 1†≠2†x _1 _2, we define λ†=Δλ1†+(1−λ)2†x _λ = _1 +(1-λ)x_2 , where λ∈(0,1)λ∈(0,1). Note that ‖λ†−‖<η\|x _λ-z\|<η because ‖1†−‖=η\|x _1-z\|=η and ‖2†−‖=η\|x _2-z\|=η. Due to the convexity of ⟨∇f(),⟩+r() ∇ f(z),x +r(x), we get λ†∈⋆†x _λ _ . Consequently, we must have ∉+∂r(λ†) 0 +∂ r(x _λ). However, ∈∇f()+∂r(λ†)+(,η)(λ†)=∇f()+∂r(λ†) 0∈∇ f(z)+∂ r(x _λ)+ N_B(z,η) (x _λ )=∇ f(z)+∂ r(x _λ) by the optimality condition and Lemma B.1. Such a contradiction indicates that ⋆†X_ must be a singleton ■ B.2 Proof of Theorem 4.2 First, we prove the following lemmas Lemma B.4. [28, Lemma 3] Suppose that r:d↦(−∞,+∞]r: d (-∞,+∞] is a proper, closed and convex function. Given any ∈dom(r)z (r ), η>0η>0 and ∈dv∈ d, we let ⋆†X _ be the solution set of min∈(,η)⟨,⟩+r() _x (z,η) \ ,x +r(x) \. Then ∀†∈⋆† _ , there always exists gr(†)∈∂r(†)g_r(x )∈∂ r(x ), such that r()+⟨,−†⟩≥r(†)+η‖+gr(†)‖ r(z)+ ,z-x ≥ r(x )+η +g_r(x ) (B.3) holds true. Proof. First, we have r()+⟨,−†⟩≥r(†)+⟨gr(†)+,−†⟩r(z)+ ,z-x ≥ r(x )+ g_r(x )+v,z-x holds due to the convexity of r()r(x). If 0∈+∂r(†)0 +∂ r(x ), i.e., ∃gr(†)∈∂r(†),=−gr(†)∃ g_r(x )∈∂ r(x ),v=-g_r(x ), then (B.3) is obviously holds. If 0∉+∂r(†)0 +∂ r(x ), then accroding to the Theorem 4.1, we know that ∃gr(†)∈∂r(†),†=−η+gr(†)‖+gr(†)‖∃ g_r(x )∈∂ r(x ),\;x =z-η v+g_r(x )\|v+g_r(x )\|, then we immediately obtained (B.3). ■ Remark B.2. Although Lemma B.4 has been proposed by [28], their proof is more complex than ours. Once we understand through Theorem 4.1 that the trust-region gradient method is actually a generalization of gradient normalization to composite optimization problems, then the validity of (B.3) is intuitive. Lemma B.5. Suppose that Assumption 1(a) and Assumption 2 hold. After one iteration of TR-SGDAM, we have Ψ(t+1)−Ψ(t) \ (x_t+1)- (x_t) ≤−ηdist(0,∇Φ(t+1)+∂r(t+1))+2κηdist(∇f(t,t),∂h(t))+2η‖ζ,t‖+3κLη2, \ ≤- _xdist (0,∇ (x_t+1)+∂ r(x_t+1) )+2κ _xdist ( _yf(x_t,y_t),∂ h(y_t) )+2 _x _x,t +3κ L _x^2, (B.4) where Φ()=Δmaxf(,)−h() (x) = _yf(x,y)-h(y), Ψ()=ΔΦ()+r() (x) = (x)+r(x) and ζ,t=Δ,t−∇f(t,t) _x,t =m_x,t- _xf(x_t,y_t). Proof. Based on the 2κL2κ L-smoothness of Φ(⋅) (·), we have Φ(t+1)+r(t+1)≤ (x_t+1)+r(x_t+1)≤ Φ(t)+r(t+1)+⟨∇f(t,t⋆),t+1−t⟩+Lκη2 (x_t)+r(x_t+1)+ _xf(x_t,y _t),x_t+1-x_t +Lκ _x^2 = = Φ(t)+r(t+1)+⟨,t,t+1−t⟩+⟨∇f(t,t⋆)−∇f(t,t),t+1−t⟩ (x_t)+r(x_t+1)+ _x,t,x_t+1-x_t + _xf(x_t,y _t)- _xf(x_t,y_t),x_t+1-x_t +⟨∇f(t,t)−,t,t+1−t⟩+Lκη2 + _xf(x_t,y_t)-m_x,t,x_t+1-x_t +Lκ _x^2 ≤ ≤ Φ(t)+r(t+1)+⟨,t,t+1−t⟩+Lη‖t−t⋆‖+η‖ζ,t‖+Lκη2 (x_t)+r(x_t+1)+ _x,t,x_t+1-x_t +L _x _t-y_t + _x _x,t +Lκ _x^2 (B.5) holds for any t+1∈argmin∈(t,η)⟨,t,⟩+r()x_t+1∈ _x (x_t, _x) \ _x,t,x +r(x) \, then by Lemma B.4, there exists gr(t)∈∂r(t)g_r(x_t)∈∂ r(x_t), such that r(t)−r(t+1)+⟨,t,t−t+1⟩≥η‖,t+gr(t+1)‖ \ r(x_t)-r(x_t+1)+ _x,t,x_t-x_t+1 ≥ _x _x,t+g_r(x_t+1) =η‖∇Φ(t+1)+gr(t+1)+∇Φ(t)−∇Φ(t+1)+,t−∇f(t,t)+∇f(t,t)−∇Φ(t)‖ \ = _x ∇ (x_t+1)+g_r(x_t+1)+∇ (x_t)-∇ (x_t+1)+m_x,t- _xf(x_t,y_t)+ _xf(x_t,y_t)-∇ (x_t) ≥η(‖∇Φ(t+1)+gr(t+1)‖−‖∇Φ(t)−∇Φ(t+1)‖−‖ζ,t‖−‖∇f(t,t)−∇f(t,t⋆)‖) \ ≥ _x ( ∇ (x_t+1)+g_r(x_t+1) - ∇ (x_t)-∇ (x_t+1) - _x,t - _xf(x_t,y_t)- _xf(x_t,y _t) ) ≥η(dist(0,∇Φ(t+1)+∂r(t+1))−2κLη−‖ζ,t‖−L‖t−t⋆‖) \ ≥ _x (dist (0,∇ (x_t+1)+∂ r(x_t+1) )-2κ L _x- _x,t -L _t-y _t ) (B.6) Substituting (B.6) into (B.5), we get [Φ(t+1)+r(t+1)]−[Φ(t)+r(t)] \ [ (x_t+1)+r(x_t+1) ]- [ (x_t)+r(x_t) ] ≤−ηdist(0,∇Φ(t+1)+∂r(t+1))+2Lη‖t−t⋆‖+2η‖ζ,t‖+3κLη2. \ ≤- _xdist (0,∇ (x_t+1)+∂ r(x_t+1) )+2L _x _t-y _t +2 _x _x,t +3κ L _x^2. (B.7) By our assumption, f(t,⋅)f(x_t,·) is μ-strongly concave with respect to y, then f(t,⋅)−h(⋅)f(x_t,·)-h(·) is also μ-strongly concave since h(⋅)h(·) is convex. We also konw −∇f(t,)+∂h()=∂(−f(t,⋅)+h)()- _yf(x_t,y)+∂ h(y)=∂(-f(x_t,·)+h)(y) for any ∈dom(h)y (h ). Since t⋆=argmin−f(t,)+h()y _t= _y \-f(x_t,y)+h(y) \, we have 0∈−∇f(t,)+∂h()0∈- _yf(x_t,y)+∂ h(y), and −f(t,t)+h(t)+f(t,t⋆)−h(t⋆)≥μ2‖t−t⋆‖2. -f(x_t,y_t)+h(y_t)+f(x_t,y _t)-h(y _t)≥ μ2 _t-y _t ^2. (B.8) Note that ∂h(t)≠∅∂ h(y_t)≠ by Theorem 4.1. Hence, ∀gh(t)∈∂h(t)∀ g_h(y_t)∈∂ h(y_t), such that −f(t,t⋆)+h(t⋆)+f(t,t)−h(t)≥⟨gh(t)−∇f(t,t),t⋆−t⟩+μ2‖t⋆−t‖2, -f(x_t,y _t)+h(y _t)+f(x_t,y_t)-h(y_t)≥ g_h(y_t)- _yf(x_t,y_t),y _t-y_t + μ2 _t -y_t ^2, (B.9) then we can obtain ‖gh(t)−∇f(t,t)‖‖t⋆−t‖≥⟨gh(t)−∇f(t,t),t⋆−t⟩+μ‖t⋆−t‖2, g_h(y_t)- _yf(x_t,y_t) _t -y_t ≥ g_h(y_t)- _yf(x_t,y_t),y _t-y_t +μ _t -y_t ^2, (B.10) which implies that μ‖t⋆−t‖≤dist(∇f(t,t),∂h(t)). μ _t -y_t ( _yf(x_t,y_t),∂ h(y_t) ). (B.11) Now, by substituting (B.11) into (B.7), we have completed the proof. ■ Lemma B.6. Suppose that Assumption 1(a) and Assumption 2 hold. After one iteration of TR-SGDAM, we have F(t−1,t−1)−F(t,t)≤ F(x_t-1,y_t-1)-F(x_t,y_t)≤ ηdist(0,∇Φ(t−1)+∂r(t−1))+ηκdist(∇f(t−1,t−1),∂h(t−1)) _xdist (0,∇ (x_t-1)+∂ r(x_t-1) )+ _x ( _yf(x_t-1,y_t-1),∂ h(y_t-1) ) −ηdist(∇f(t,t),∂h(t))+2η‖ζ,t−1‖+3Lη2+2Lηη, - _ydist ( _yf(x_t,y_t),∂ h(y_t) )+2 _y _y,t-1 +3L _y^2+2L _x _y, (B.12) where F(t,t)=Δf(t,t)−h(t)+r(t)F(x_t,y_t) =f(x_t,y_t)-h(y_t)+r(x_t) and ζ,t−1=Δ,t−1−∇f(t−1,t−1) _y,t-1 =m_y,t-1- _yf(x_t-1,y_t-1). Proof. On the one hand, based on the L-smothness of f(t,⋅)f(x_t,·) with respect to y, we have f(t,t)−f(t,t−1)−⟨∇f(t,t−1),t−t−1⟩≥−L/2∥t−t−1∥f(x_t,y_t)-f(x_t,y_t-1)- _yf(x_t,y_t-1),y_t-y_t-1 ≥-L/2\|y_t-y_t-1\|, which means f(t,t−1)−h(t−1)≤f(t,t)−h(t−1)+⟨∇f(t,t−1),t−1−t⟩+Lη22 \ f(x_t,y_t-1)-h(y_t-1)≤ f(x_t,y_t)-h(y_t-1)+ _yf(x_t,y_t-1),y_t-1-y_t + L _y^22 =f(t,t)−h(t−1)+⟨,t−1,t−1−t⟩+⟨∇f(t,t−1)−∇f(t−1,t−1),t−1−t⟩ \ =f(x_t,y_t)-h(y_t-1)+ _y,t-1,y_t-1-y_t + _yf(x_t,y_t-1)- _yf(x_t-1,y_t-1),y_t-1-y_t +⟨∇f(t−1,t−1)−,t−1,t−1−t⟩+Lη22 \;+ _yf(x_t-1,y_t-1)-m_y,t-1,y_t-1-y_t + L _y^22 ≤f(t,t)−h(t−1)+⟨,t−1,t−1−t⟩+Lηη+η‖ζ,t−1‖+Lη22; \ ≤ f(x_t,y_t)-h(y_t-1)+ _y,t-1,y_t-1-y_t +L _x _y+ _y _y,t-1 + L _y^22; (B.13) on the other hand, by Lemma B.4, there exists gh(t)∈h(t)g_h(y_t)∈ h(y_t), such that h(t−1)−h(t)−⟨,t−1,t−1−t⟩≥η‖−,t−1+gh(t)‖ \ h(y_t-1)-h(y_t)- _y,t-1,y_t-1-y_t ≥ _y -m_y,t-1+g_h(y_t) =η‖∇f(t,t)−∇f(t−1,t−1)+∇f(t−1,t−1)−,t−1+gh(t)−∇f(t,t)‖ \ = _y _yf(x_t,y_t)- _yf(x_t-1,y_t-1)+ _yf(x_t-1,y_t-1)-m_y,t-1+g_h(y_t)- _yf(x_t,y_t) ≥η(‖gh(t)−∇f(t,t)‖−‖∇f(t,t)−∇f(t−1,t−1)‖−‖ζ,t−1‖) \ ≥ _y ( g_h(y_t)- _yf(x_t,y_t) - _yf(x_t,y_t)- _yf(x_t-1,y_t-1) - _y,t-1 ) ≥ηdist(∇f(t,t),∂h(t))−Lη(η+η)−η‖ζ,t−1‖. \ ≥ _ydist ( _yf(x_t,y_t),∂ h(y_t) )-L _y( _x+ _y)- _y _y,t-1 . (B.14) Therefore, we get [f(t,t−1)−h(t−1)]−[f(t,t)−h(t)] \ [f(x_t,y_t-1)-h(y_t-1) ]- [f(x_t,y_t)-h(y_t) ] ≤−ηdist(∇f(t,t),∂h(t))+2η‖ζ,t−1‖+2Lηη+3Lη22 \ ≤- _ydist ( _yf(x_t,y_t),∂ h(y_t) )+2 _y _y,t-1 +2L _x _y+ 3L _y^22 (B.15) Now, we consider the upper bound f(t−1,t−1)+r(t−1)−f(t,t−1)−r(t)f(x_t-1,y_t-1)+r(x_t-1)-f(x_t,y_t-1)-r(x_t). By the L-smoothness of f(⋅,t−1)f(·,y_t-1) with respect to x and the convexity of r, ∀gr(t−1)∈r(t−1)∀ g_r(x_t-1)∈ r(x_t-1) we have f(t−1,t−1)+r(t−1)−f(t,t−1)−r(t)≤⟨∇f(t,t−1)+gr(t−1),t−1−t⟩+Lη22 \ f(x_t-1,y_t-1)+r(x_t-1)-f(x_t,y_t-1)-r(x_t)≤ _xf(x_t,y_t-1)+g_r(x_t-1),x_t-1-x_t + L _x^22 =⟨∇Φ(t−1)+gr(t−1),t−1−t⟩+⟨∇f(t,t−1)−∇f(t−1,t−1),t−1−t⟩ \ = ∇ (x_t-1)+g_r(x_t-1),x_t-1-x_t + _xf(x_t,y_t-1)- _xf(x_t-1,y_t-1),x_t-1-x_t +⟨∇f(t−1,t−1)−∇f(t−1,t−1⋆),t−1−t⟩+Lη22 \ + _xf(x_t-1,y_t-1)- _xf(x_t-1,y_t-1 ),x_t-1-x_t + L _x^22 ≤η‖∇Φ(t−1)+gr(t−1)‖+Lη‖t−1−t−1⋆‖+3Lη22 \ ≤ _x ∇ (x_t-1)+g_r(x_t-1) +L _x _t-1-y_t-1 + 3L _x^22 ≤η‖∇Φ(t−1)+gr(t−1)‖+ηκdist(∇f(t−1,t−1),∂h(t−1))+3Lη22, ≤ _x ∇ (x_t-1)+g_r(x_t-1) + _x ( _yf(x_t-1,y_t-1),∂ h(y_t-1) )+ 3L _x^22, (B.16) where the last inequality holds because we have already obtained (B.11). We get the following by adding inequalities (B.15) and (B.16): F(t−1,t−1)−F(t,t)≤ F(x_t-1,y_t-1)-F(x_t,y_t)≤ ηdist(0,∇Φ(t−1)+∂r(t−1))+ηκdist(∇f(t−1,t−1),∂h(t−1)) _xdist (0,∇ (x_t-1)+∂ r(x_t-1) )+ _x ( _yf(x_t-1,y_t-1),∂ h(y_t-1) ) −ηdist(∇f(t,t),∂h(t))+2η‖ζ,t−1‖+3Lη2+2Lηη - _ydist ( _yf(x_t,y_t),∂ h(y_t) )+2 _y _y,t-1 +3L _y^2+2L _x _y (B.17) Note that we have replaced the term ‖∇Φ(t−1)+gr(t−1)‖ ∇ (x_t-1)+g_r(x_t-1) with dist(∇Φ(t−1)+∂r(t−1))dist (∇ (x_t-1)+∂ r(x_t-1) ) on R.H.S. of the above inequality, because (B.16) holds for any gr(t−1)∈∂r(t−1)g_r(x_t-1)∈∂ r(x_t-1). ■ B.2.1 Prove the complexity Proof. By Lemma B.5 and Lemma B.6, we can get the following inequality Ψ(t+1)−F(t+1,t+1)+ηdist(0,∇Φ(t+1)+∂r(t+1))+ηdist(∇f(t+1,t+1),∂h(t+1)) \ (x_t+1)-F(x_t+1,y_t+1)+ _xdist (0,∇ (x_t+1)+∂ r(x_t+1) )+ _ydist ( _yf(x_t+1,y_t+1),∂ h(y_t+1) ) −[Ψ(t)−F(t,t)+ηdist(0,∇Φ(t)+∂r(t))+ηdist(∇f(t,t),∂h(t))] \ - [ (x_t)-F(x_t,y_t)+ _xdist (0,∇ (x_t)+∂ r(x_t) )+ _ydist ( _yf(x_t,y_t),∂ h(y_t) ) ] ≤(3κη−η)dist(∇f(t,t),∂h(t))+2η‖ζ,t‖+2η‖ζ,t‖+3Lη2+3κLη2+2Lηη \ ≤(3κ _x- _y)dist ( _yf(x_t,y_t),∂ h(y_t) )+2 _y _y,t +2 _x _x,t +3L _y^2+3κ L _x^2+2L _x _y (B.18) Then, we add (B.18) to (B.4) to get Vt+1−Vt≤ _t+1-V_t≤ −ηdist(0,∇Φ(t+1)+∂r(t+1))+(5κη−η)dist(∇f(t,t),∂h(t)) - _xdist (0,∇ (x_t+1)+∂ r(x_t+1) )+(5κ _x- _y)dist ( _yf(x_t,y_t),∂ h(y_t) ) +2η‖ζ,t‖+4η‖ζ,t‖+3Lη2+6κLη2+2Lηη +2 _y _y,t +4 _x _x,t +3L _y^2+6κ L _x^2+2L _x _y = = −ηdist(0,∇Φ(t+1)+∂r(t+1))+10κη‖ζ,t‖+4η‖ζ,t‖+75κ2Lη2+16κLη2, - _xdist (0,∇ (x_t+1)+∂ r(x_t+1) )+10κ _x _y,t +4 _x _x,t +75κ^2L _x^2+16κ L _x^2, (B.19) where Vt=Δ2Ψ(t)−F(t,t)+ηdist(0,∇Φ(t)+∂r(t))+ηdist(∇f(t,t),∂h(t))V_t =2 (x_t)-F(x_t,y_t)+ _xdist (0,∇ (x_t)+∂ r(x_t) )+ _ydist ( _yf(x_t,y_t),∂ h(y_t) ), and the last equality holds beacuse we let 5κη=η5κ _x= _y. We rule V0=V1V_0=V_1, ζ,1=ζ,0 _y,1= _y,0 and ζ,1=ζ,0 _y,1= _y,0, then we sum both sides of the above inequality from t=0t=0 to T−1T-1 and multiply by (Tη)−1(T _x)^-1 to get 1T∑t=1Tdist(0,∇Φ(t)+∂r(t))≤V1−VTTη+1T∑t=1T(10κ‖ζ,t‖+4‖ζ,t‖)+75κ2Lη+16κLη 1T _t=1^Tdist (0,∇ (x_t)+∂ r(x_t) )≤ V_1-V_TT _x+ 1T _t=1^T (10κ _y,t +4 _x,t )+75κ^2L _x+16κ L _x (B.20) Note that V1−VT≤ _1-V_T≤ Ψ(1)−Ψ(T)+Ψ(1)−F(1,1)−[Ψ(T)−F(T,T)] (x_1)- (x_T)+ (x_1)-F(x_1,y_1)- [ (x_T)-F(x_T,y_T) ] +ηdist(0,∇Φ(1)+∂r(1))+ηdist(∇f(1,1),∂h(1)) + _xdist (0,∇ (x_1)+∂ r(x_1) )+ _ydist ( _yf(x_1,y_1),∂ h(y_1) ) ≤ ≤ Ψ(1)−minΨ()+Ψ(1)−F(1,1)⏟=ΔΛ1 (x_1)- _x (x)+ (x_1)-F(x_1,y_1)_ = _1 +η(dist(0,∇Φ(1)+∂r(1))+5κdist(∇f(1,1),∂h(1)))⏟=Δ∂1, + _x (dist (0,∇ (x_1)+∂ r(x_1) )+5 ( _yf(x_1,y_1),∂ h(y_1) ) )_ = _1, (B.21) where the last inequality holds because Ψ(T)−F(T,T)=Φ(T)−(f(T,T)−h(T))≥0 (x_T)-F(x_T,y_T)= (x_T)-(f(x_T,y_T)-h(x_T))≥ 0. Substituting (B.21) into (B.20) and taking expectation [⋅]E[·] on both sides of the inequality, we get 1T∑t=1Tdist(0,∇Φ(t)+∂r(t))≤∂1T+Λ1Tη+75κ2Lη+16κLη+1T∑t=1T(10κ[‖ζ,t‖]+4[‖ζ,t‖]) 1T _t=1^Tdist (0,∇ (x_t)+∂ r(x_t) )≤ _1T+ _1T _x+75κ^2L _x+16κ L _x+ 1T _t=1^T (10κE [ _y,t ]+4E [ _x,t ] ) (B.22) Now, we need to upper bound [‖ζ,t‖]E [ _y,t ] and [‖ζ,t‖]E [ _x,t ]. The analysis of this part is analogous to the analysis of [37, 48, 26] for NSGD with momentum. Concretetly, we first introduce following notations D,t=Δ∇f(t−1,t−1)−∇f(t,t),ε,t=Δ∇f(t,t,ξt)−∇f(t,t) D_x,t = _xf(x_t-1,y_t-1)- _xf(x_t,y_t), _x,t = _xf(x_t,y_t, _t)- _xf(x_t,y_t) (B.23) Then, we can decompose ζ,t _x,t into following ζ,t _x,t =,t−∇f(t,t) =m_x,t- _xf(x_t,y_t) =β,t−1+(1−β)∇f(t,t,ξt)−∇f(t,t)=βζ,t−1+βD,t+(1−β)ε,t = _x,t-1+(1-β) _xf(x_t,y_t, _t)- _xf(x_t,y_t)=β _x,t-1+β D_x,t+(1-β) _x,t (B.24) By recursion, we have ζ,t=βtζ,1+(1−β)∑s=1tβt−sε,s+∑s=1tβt−s+1D,s _x,t=β^t _x,1+(1-β) _s=1^tβ^t-s _x,s+ _s=1^tβ^t-s+1D_x,s (B.25) Thus, we have ‖ζ,t‖≤βt‖ζ,1‖+(1−β)‖∑s=1tβt−sε,s‖+∑s=1tβt−s+1‖D,s‖ _x,t ≤β^t _x,1 +(1-β) _s=1^tβ^t-s _x,s + _s=1^tβ^t-s+1 D_x,s (B.26) Note by the L-smoothness of f, we have ∑s=1tβt−s+1‖D,s‖ _s=1^tβ^t-s+1 D_x,s ≤∑s=1tβt−s+1L(‖s−1−s‖+‖s−1−s‖) ≤ _s=1^tβ^t-s+1L ( _s-1-x_s + _s-1-y_s ) ≤∑s=1tβt−s+1L(η+η)≤βL(η+η)1−β≤6κLηβ1−β ≤ _s=1^tβ^t-s+1L ( _x+ _y )≤ β L( _x+ _y)1-β≤ 6κ L _xβ1-β (B.27) By the filtration ℱt=Δσ(ξ1,…,ξt)F_t =σ ( _1,…, _t ) we defined. We know that βt−sε,sβ^t-s _x,s is ℱsF_s-measurable and [βt−sε,s|ℱs−1]=0E [β^t-s _x,s|F_s-1 ]=0 due to the unbiased estimator ∇f(s,s,ξs) _xf(x_s,y_s, _s). Moreover, by the Assumptions 1(c), we have [‖βt−sε,s‖]<∞E [ β^t-s _x,s ]<∞. Thus, we can get [‖∑s=1tβt−sε,s‖]= E [ _s=1^tβ^t-s _x,s ]= [(‖∑s=1tβt−sε,s‖p)1p]≤Jensen’s Ineq([‖∑s=1tβt−sε,s‖p])1p E [ ( _s=1^tβ^t-s _x,s ^p ) 1p ] Jensen's Ineq≤ (E [ _s=1^tβ^t-s _x,s ^p ] ) 1p ≤ Lemma C.5 ( 2∑_s=1^t β^p(t-s)E[‖ε_x,s‖^p] ) 1 p ≤ ( 2∑_s=1^t β^p(t-s)σ^p ) 1 p ≤2σ ( ∑_s=1^t β^p(t-s) ) 1 p Then, we get (1−β)[‖∑s=1tβt−sε,s‖]≤ (1-β)E [ _s=1^tβ^t-s _x,s ]≤ 2(1−β)σ(∑s=1tβp(t−s))1p=2(1−β)σ(∑s=0t−1βps)1p≤2(1−β)σ(1−βp)1p 2(1-β)σ ( _s=1^tβ^p(t-s) ) 1p=2(1-β)σ ( _s=0^t-1β^ps ) 1p≤ 2(1-β)σ (1-β^p ) 1p ≤ βp≤β^p≤β 2σ(1-β)^1- 1 p Finally, we get [‖ζ,t‖]≤(βt+2(1−β)1−1p)σ+6κLβη1−β E [ _x,t ]≤ (β^t+2(1-β)^1- 1p )σ+ 6κ Lβ _x1-β (B.28) Based on the exact same analysis, we also have [‖ζ,t‖]≤(βt+2(1−β)1−1p)σ+6κLβη1−βE [ _y,t ]≤ (β^t+2(1-β)^1- 1p )σ+ 6κ Lβ _x1-β holds true. Futhermore, we can get 1T∑t=1T[‖ζ,t‖],1T∑t=1T[‖ζ,t‖]≤σT(1−β)+2(1−β)1−1pσ+6κLβη1−β 1T _t=1^TE [ _x,t ], 1T _t=1^TE [ _y,t ]≤ σT(1-β)+2(1-β)^1- 1pσ+ 6κ Lβ _x1-β (B.29) Now, we substitute (B.29) into (B.22) to get 1T∑t=1Tdist(0,∇Φ(t)+∂r(t))≤ 1T _t=1^Tdist (0,∇ (x_t)+∂ r(x_t) )≤ ∂1T+Λ1Tη+75κ2Lη+16κLη+84κ2Lβη1−β _1T+ _1T _x+75κ^2L _x+16κ L _x+ 84κ^2Lβ _x1-β +14κσT(1−β)+28κ(1−β)1−1pσ + 14κσT(1-β)+28κ(1-β)^1- 1pσ ≤ ≤ ∂1T+Λ1Tη+91κ2Lη1−β+14κσT(1−β)+28κ(1−β)1−1pσ _1T+ _1T _x+ 91κ^2L _x1-β+ 14κσT(1-β)+28κ(1-β)^1- 1pσ (B.30) Case 1. If we can know Λ1 _1, L, σ and p Since we let η=Λ(1−β)/LκTandβ=1−min1,max1Tp2p−1,(Λ1Lσ2T)p3p−2, _x= (1-β)/Lκ T\;and\;β=1- \1, \ 1T p2p-1, ( _1Lσ^2T ) p3p-2 \ \, (B.31) we can obtain 1T∑t=1Tdist(0,∇Φ(t)+∂r(t))≤ 1T _t=1^Tdist (0,∇ (x_t)+∂ r(x_t) )≤ ∂1T+92κΛLT(1−β)+14κσT(1−β)+28κ(1−β)1−1pσ _1T+92κ LT(1-β)+ 14κσT(1-β)+28κ(1-β)^1- 1pσ ≤ ≤ ∂1T+92κ⋅maxΛ1LT,(Λ1LT)p−13p−2⋅σp3p−2+14κσTp−12p−1 _1T+92κ· \ _1LT, ( _1LT ) p-13p-2·σ p3p-2 \+ 14κσT p-12p-1 +28κmaxσTp−12p−1,(Λ1LT)p−13p−2⋅σp3p−2 +28κ \ σT p-12p-1, ( _1LT ) p-13p-2·σ p3p-2 \ ≤ ≤ ∂1T+92κΛ1LT+120κ(Λ1LT)p−13p−2⋅σp3p−2+42κσTp−12p−1 _1T+92κ _1LT+120κ ( _1LT ) p-13p-2·σ p3p-2+ 42κσT p-12p-1 (B.32) Hence, to achive (1/T)∑t=1Tdist(0,∇Φ(t)+∂r(t))≤ϵ(1/T) _t=1^Tdist (0,∇ (x_t)+∂ r(x_t) )≤ε, the total gradient complexity is upper bounded by (∂1ϵ+κ2Λ1Lϵ2+(κσ)2p−1p−1ϵ2p−1p−1+κ3p−2p−1Λ1Lσp−1ϵ3p−2p−1) ( _1ε+ κ^2 _1Lε^2+ (κσ) 2p-1p-1ε 2p-1p-1+ κ 3p-2p-1 _1Lσ pp-1ε 3p-2p-1 ) (B.33) Case 2. If we don’t konw Λ1 _1, L, σ and p We let η=1κT0.75 _x= 1κ T^0.75 and β=1−1Tβ=1- 1 T, then we obtain 1T∑t=1Tdist(0,∇Φ(t)+∂r(t))≤∂1T+κΛ1+91κLT1/4+14κσT+28κσTp−12p 1T _t=1^Tdist (0,∇ (x_t)+∂ r(x_t) )≤ _1T+ κ _1+91κ LT 14+ 14κσ T+ 28κσT p-12p (B.34) Hence, in such a case, to achive (1/T)∑t=1Tdist(0,∇Φ(t)+∂r(t))≤ϵ(1/T) _t=1^Tdist (0,∇ (x_t)+∂ r(x_t) )≤ε, the total gradient complexity is upper bounded by (∂1ϵ+(κΛ1+κL)4ϵ4+(κσϵ)2+(κσϵ)2p−1) ( _1ε+ (κ _1+κ L)^4ε^4+ ( κσε )^2+ ( κσε ) 2pp-1 ) (B.35) ■ B.3 Proof of Theorem 4.3 First, we prove the following lemma. Lemma B.7. Suppose that Assumption 1 and 2 hold. Let t,tt=1T\x_t,y_t\_t=1^T be the output of Stoc-TRGDAmax, t⋆y_t be defined as the maxmizer of f(t,)−h()f(x_t,y)-h(y) and δ1=Δ‖1−1⋆‖ _1 =\|y_1-y _1\|. Given any ϵ>0ε>0, we let η=min272p/(p−1)μ(ϵκσ)p−1,21p−1μκp2(p−1) _y= \ 272^p/(p-1)μ ( εκσ ) pp-1, 2 1p-1μκ p2(p-1) \. • If ϵ<2Lδ1ε<2L _1, η≤2δ13κ _x≤ 2 _13κ, and K−1=pln(6Lδ1ϵ)⋅max72p−1σp−1κp−1ϵp−1,κp2(p−1)22−p−1, K-1=p ( 6L _1ε )· \ 72 pp-1σ pp-1κ pp-1ε pp-1, κ p2(p-1)2 2-pp-1 \, (B.36) then for any t=2,…,Tt=2,…,T, we have [‖t−t⋆‖]≤ϵ/2LE[\|y_t-y_t \|]≤ε/2L. • If δ1≤ϵ/2L _1≤ε/2L, η≤ϵ3κL _x≤ ε3κ L, and K−1=(pln3)⋅max72p−1(σκ)p−1ϵp−1,κp2(p−1)22−p−1, K-1=(p 3)· \ 72 pp-1(σκ) pp-1ε pp-1, κ p2(p-1)2 2-pp-1 \, (B.37) then for any t=1,…,Tt=1,…,T, we have [‖t−t⋆‖]≤ϵ/2LE[\|y_t-y_t \|]≤ε/2L. Proof. Let’s temporarily ignore the outer loop, and for simplicity, we denote ~k=Δt,k y_k =y_t,k, ξ~k=Δξt,k ξ_k = _t,k and ζ~,k=Δ∇f(t,~k,ξ~k)−∇f(t,~k) ζ_y,k = _yf(x_t, y_k, ξ_k)- _yf(x_t, y_k). We still denote t⋆y_t as the maxmizer of f(t,)−h()f(x_t,y)-h(y). First, by the fixed-point property of the proximal operator, we have ‖~k+1−t⋆‖p= y_k+1-y_t ^p= ‖proxηh(~k+η∇f(t,~k,ξ~k))−proxηh(t⋆+η∇f(t,t⋆))‖p _ _yh ( y_k+ _y _yf(x_t, y_k, ξ_k) )-prox_ _yh (y_t + _y _yf(x_t,y_t ) ) ^p ≤ (a) ‖ ~ y _k-y_t^⋆+η_y ( ∇_yf(x_t, ~ y _k, ~ ξ _k)-∇_yf(x_t,y_t^⋆) ) ‖^p ≤ Lemma C.2 ‖ ~ y _k-y_t^⋆‖^p + pη y ‖ ~ y k -y t ⋆ ‖ 2-p ⟨ ~ y _k-y_t^⋆, ∇_yf(x_t, ~ y _k, ~ ξ _k)-∇_yf(x_t,y_t^⋆) ⟩ +22−pηp‖∇f(t,~k,ξ~k)−∇f(t,t⋆)‖p +2^2-p _y^p _yf(x_t, y_k, ξ_k)- _yf(x_t,y_t ) ^p ≤ ≤ ‖~k−t⋆‖p+pη‖~k−t⋆‖2−p⟨~k−t⋆,ζ~,k⟩+2ηp‖ζ~,k‖p y_k-y_t ^p+ p _y y_k-y_t ^2-p y_k-y_t , ζ_y,k +2 _y^p\| ζ_y,k\|^p +pη‖~k−t⋆‖2−p⟨~k−t⋆,∇f(t,~k)−∇f(t,t⋆)⟩ + p _y y_k-y_t ^2-p y_k-y_t , _yf(x_t, y_k)- _yf(x_t,y_t ) +2ηp‖∇f(t,~k)−∇f(t,t⋆)‖p +2 _y^p _yf(x_t, y_k)- _yf(x_t,y_t ) ^p ≤ (b) ( 1- μpη y 2 ) ‖ ~ y _k-y_t^⋆‖^p+ pη y ‖ ~ y k -y t ⋆ ‖ 2-p ⟨ ~ y _k-y_t^⋆, ~ ζ _y,k ⟩+ 2η_y^p ∥ ~ ζ _y,k∥^p +(2ηpLp/2−pημ2−p22)⟨t⋆−~k,∇f(t,~k)−∇f(t,t⋆)⟩p2 + (2 _y^pL^p/2- p _yμ 2-p22 ) _t - y_k, _yf(x_t, y_k)- _yf(x_t,y_t ) p2 ≤ (c) ( 1- μη y 2 ) ‖ ~ y _k-y_t^⋆‖^p+ 2η y ‖ ~ y k -y t ⋆ ‖ 2-p ⟨ ~ y _k-y_t^⋆, ~ ζ _y,k ⟩+ 2η_y^p ∥ ~ ζ _y,k∥^p, where (a)(a) holds due to the nonexpensiveness of proximal operator, and (c)(c) holds because we let η _y be smaller than21/(p−1)μκp/2(p−1) 2^1/(p-1)μκ^p/2(p-1). Here we explain why (b)(b) holds. First, because f(t,⋅)f(x_t,·) is L-smooth and μ-strongly concave with respct to y , we have following two well-known inequalities: μ‖~k−t‖2≤⟨t⋆−~k,∇f(t,~k)−∇f(t,t⋆)⟩ μ y_k-y_t ^2≤ _t - y_k, _yf(x_t, y_k)- _yf(x_t,y_t ) (Ineq.1) ⟨t⋆−~k,∇f(t,~k)−∇f(t,t⋆)⟩≥‖∇f(t,~k)−∇f(t,t⋆)‖2L _t - y_k, _yf(x_t, y_k)- _yf(x_t,y_t ) ≥ _yf(x_t, y_k)- _yf(x_t,y_t ) ^2L (Ineq.2) Then, we have 12‖~k−t⋆‖2−p⟨~k−t⋆,∇f(t,~k)−∇f(t,t⋆)⟩ \ 12 y_k-y_t ^2-p y_k-y_t , _yf(x_t, y_k)- _yf(x_t,y_t ) =1−2‖~k−t⋆‖2−p⟨t⋆−~k,∇f(t,~k)−∇f(t,t⋆)⟩ \ = 1-2 y_k-y_t ^2-p _t - y_k, _yf(x_t, y_k)- _yf(x_t,y_t ) ≤Ineq.1−(μ⟨t⋆−~k,∇f(t,~k)−∇f(t,t⋆)⟩)2−p2⋅⟨t⋆−~k,∇f(t,~k)−∇f(t,t⋆)⟩2 Ineq.1≤- ( μ _t - y_k, _yf(x_t, y_k)- _yf(x_t,y_t ) ) 2-p2· _t - y_k, _yf(x_t, y_k)- _yf(x_t,y_t ) 2 =−μ2−p22⟨t⋆−~k,∇f(t,~k)−∇f(t,t⋆)⟩p2, \ =- μ 2-p22 _t - y_k, _yf(x_t, y_k)- _yf(x_t,y_t ) p2, (B.38) and 12‖~k−t⋆‖2−p⟨~k−t⋆,∇f(t,~k)−∇f(t,t⋆)⟩≤Ineq.2−μ‖~k−t⋆‖22‖~k−t⋆‖2−p=−μ2‖~k−t⋆‖p. \ 12 y_k-y_t ^2-p y_k-y_t , _yf(x_t, y_k)- _yf(x_t,y_t ) Ineq.2≤- μ y_k-y_t ^22 y_k-y_t ^2-p=- μ2 y_k-y_t ^p. (B.39) Hence, (b)(b) holds true. Before we take the expectation on both sides of the inequality, note that [2η‖~k−t⋆‖2−p⟨~k−t⋆,ζ~,k⟩∣t,t−1,ξ^t,1,…,ξ^t,k−1]=0, E [ 2 _y y_k-y_t ^2-p y_k-y_t , ζ_y,k _t,y_t-1, ξ_t,1,…, ξ_t,k-1 ]=0, [∥ζ~,k∥p∣t,t−1,ξ^t,1,…,ξ^t,k−1]≤σp. E [\| ζ_y,k\|^p _t,y_t-1, ξ_t,1,…, ξ_t,k-1 ]≤σ^p. (B.40) Therefore, we can get [‖~k+1−t⋆‖p∣t,t−1,ξ^t,1,…,ξ^t,k−1]≤(1−μη2)1/p‖~k−t⋆‖+2ηpσ E [ y_k+1-y_t ^p _t,y_t-1, ξ_t,1,…, ξ_t,k-1 ]≤ (1- μ _y2 )^1/p y_k-y_t +2 _y^pσ (B.41) After taking the conditional expectation [⋅∣t,t−1]E [· _t,y_t-1 ] on both sides of above inequality, we get [‖~k+1−t⋆‖p∣t,t−1]≤(1−μη2)[‖~k−t⋆‖p∣t,t−1]+2ηpσp, E [ y_k+1-y_t ^p _t,y_t-1 ]≤ (1- μ _y2 )E [ y_k-y_t ^p _t,y_t-1 ]+2 _y^pσ^p, (B.42) which means [‖t−t⋆‖p∣t,t−1]≤ E [ _t-y_t ^p _t,y_t-1 ]≤ (1−μη2)K−1‖t−1−t⋆‖p+4ηp−1σpμ (1- μ _y2 )^K-1 _t-1-y_t ^p+ 4 _y^p-1σ^pμ ≤ ≤ ρ‖t−1−t⋆‖p+8σplnp−1(1/ρ)μp(K−1)p−1 ρ _t-1-y_t ^p+ 8σ^p ^p-1(1/ρ)μ^p(K-1)^p-1 (B.43) Here, we use μη2≤2(2−p)/(p−1)κp/2(p−1)≤1κp/2(p−1)≤1 μ _y2≤ 2^(2-p)/(p-1)κ^p/2(p-1)≤ 1κ^p/2(p-1)≤ 1 since κ≥1κ≥ 1. The last inequality holds because we let η=2ln(1/ρ)μ(K−1)⇔K−1=2ln(1/ρ)μη _y= 2 (1/ρ)μ(K-1) K-1= 2 (1/ρ)μ _y, where ρ is to be determined later and 0<ρ<10<ρ<1. [‖t−t⋆‖∣t,t−1]≤Jensen’s Ineq([‖t−t⋆‖p∣t,t−1])1p≤ρ1/p‖t−1−t⋆‖+8σlnp−1p(1/ρ)μ(K−1)p−1p E [ _t-y_t _t,y_t-1 ] Jensen's Ineq≤ (E [ _t-y_t ^p _t,y_t-1 ] ) 1p≤ρ^1/p _t-1-y_t + 8σ p-1p(1/ρ)μ(K-1) p-1p (B.44) Now, take full expectation on the both sides of above inequality, we can get [‖t−t⋆‖] E [ _t-y_t ] ≤ρ1/p[‖t−1−t⋆‖]+8σlnp−1p(1/ρ)μ(K−1)p−1p ≤ρ^1/pE [ _t-1-y_t ]+ 8σ p-1p(1/ρ)μ(K-1) p-1p ≤ρ1/p[‖t−1−t−1⋆‖]+ρ1/p[‖t⋆−t−1⋆‖]+8σlnp−1p(1/ρ)μ(K−1)p−1p ≤ρ^1/pE [ _t-1-y_t-1 ]+ρ^1/pE [ _t -y_t-1 ]+ 8σ p-1p(1/ρ)μ(K-1) p-1p ≤ρ1/p[‖t−1−t−1⋆‖]+ρ1/pκη+8σlnp−1p(1/ρ)μ(K−1)p−1p, ≤ρ^1/pE [ _t-1-y_t-1 ]+ρ^1/pκ _x+ 8σ p-1p(1/ρ)μ(K-1) p-1p, (B.45) where we used ‖t⋆−t−1⋆‖≤κ‖t−t−1‖≤κη _t -y_t-1 ≤κ _t-x_t-1 ≤κ _x. Still through recursion, we can obtain [‖t−t⋆‖]≤ρt−1p‖1−1⋆‖+ρ1/pκη1−ρ1/p+8σlnp−1p(1/ρ)(1−ρ1/p)μ(K−1)p−1p E [ _t-y_t ]≤ρ t-1p _1-y _1 + ρ^1/pκ _x1-ρ^1/p+ 8σ p-1p(1/ρ)(1-ρ^1/p)μ(K-1) p-1p (B.46) In the following discussion, we let r=Δϵ/2Lδ1r =ε/2L _1, where δ1=Δ‖1−1‖ _1 =\|y_1-y_1\| and ϵε can be any positive constant, representing the numerical precision we want ‖t−t⋆‖\|y_t-y_t \| to achieve. • If 0<r<10<r<1, we let ρ=(r/3)1/pρ=(r/3)^1/p. Then, we have 0<ρ<10<ρ<1 obviously, and ρ(t−1)/p≤ρ1/p=r/3ρ^(t-1)/p≤ρ^1/p=r/3, which means ρt−1p‖1−1⋆‖≤ϵ/6Lρ t-1p _1-y _1 ≤ε/6L holds true for any t=2,…,Tt=2,…,T. Moreover, we have ρ1/pκη1−ρ1/p=rκη3−r≤ϵ6L⇔η≤δ1(3−r)3κ⇐η≤2δ13κ, ρ^1/pκ _x1-ρ^1/p= rκ _x3-r≤ ε6L _x≤ _1(3-r)3κ _x≤ 2 _13κ, (B.47) and 8σlnp−1p(1/ρ)(1−ρ1/p)μ(K−1)p−1p=8σ(pln(3/r))p−1p(1−r/3)μ(K−1)p−1p≤12σ(pln(3/r))p−1pμ(K−1)p−1p≤ϵ6L \ 8σ p-1p(1/ρ)(1-ρ^1/p)μ(K-1) p-1p= 8σ (p (3/r) ) p-1p(1-r/3)μ(K-1) p-1p≤ 12σ (p (3/r) ) p-1pμ(K-1) p-1p≤ ε6L ⇔K−1≥72p−1σp−1pln(3/r)κp−1ϵp−1=72p−1σp−1pln(6Lδ1/ϵ)κp−1ϵp−1. \ K-1≥ 72 pp-1σ pp-1p (3/r)κ pp-1ε pp-1= 72 pp-1σ pp-1p (6L _1/ε)κ pp-1ε pp-1. (B.48) In addition, we let K−1≥2p−2p−1pln(6Lδ1/ϵ)κp2(p−1)=2p−2p−1ln(1/ρ)κp2(p−1)K-1≥ 2 p-2p-1p (6L _1/ε)κ p2(p-1)=2 p-2p-1 (1/ρ)κ p2(p-1) to make sure η=2ln(1/ρ)μ(K−1)≤21/(p−1)μκp2(p−1) _y= 2 (1/ρ)μ(K-1)≤ 2^1/(p-1)μκ p2(p-1). Therefore, we require K−1K-1 to be max72p−1σp−1pln(6Lδ1/ϵ)κp−1ϵp−1,pln(6Lδ1/ϵ)κp2(p−1)22−p−1, \ 72 pp-1σ pp-1p (6L _1/ε)κ pp-1ε pp-1, p (6L _1/ε)κ p2(p-1)2 2-pp-1 \, (B.49) which means η=2ln(1/ρ)μ(K−1)=min272p/(p−1)μ(ϵκσ)p−1,21p−1μκp2(p−1). _y= 2 (1/ρ)μ(K-1)= \ 272^p/(p-1)μ ( εκσ ) pp-1, 2 1p-1μκ p2(p-1) \. (B.50) Hence, we can conlude that for any t=2,…,Tt=2,…,T, we have [‖t−t⋆‖]≤ϵ/2LE[\|y_t-y_t \|]≤ε/2L if η≤2δ13κ,η=min272p/(p−1)μ(ϵκσ)p−1,21p−1μκp2(p−1),K−1=pln(6Lδ1ϵ)⋅max72p−1σp−1κp−1ϵp−1,κp2(p−1)22−p−1, cases _x≤ 2 _13κ,\\ _y= \ 272^p/(p-1)μ ( εκσ ) pp-1, 2 1p-1μκ p2(p-1) \,\\ K-1=p ( 6L _1ε )· \ 72 pp-1σ pp-1κ pp-1ε pp-1, κ p2(p-1)2 2-pp-1 \, cases (B.51) and ϵ<2Lδ1ε<2L _1. • If r≥1r≥ 1, which means ‖1−1⋆‖≤ϵ/2L\|y_1-y _1\|≤ε/2L. In such a case, we let ρ=1/3pρ=1/3^p, then ρ(t−1)/p‖1−1⋆‖≤ϵ/6Lρ^(t-1)/p\|y_1-y _1\|≤ε/6L. Moreover, we have ρ1/pκη1−ρ1/p=κη2≤ϵ6L⇔η≤ϵ3κL, ρ^1/pκ _x1-ρ^1/p= κ _x2≤ ε6L _x≤ ε3κ L, (B.52) and 8σlnp−1p(1/ρ)(1−ρ1/p)μ(K−1)p−1p=12σ(pln3)p−1pμ(K−1)p−1p≤ϵ6L⇔K−1≥72p−1pln3(σκ)p−1ϵp−1. \ 8σ p-1p(1/ρ)(1-ρ^1/p)μ(K-1) p-1p= 12σ(p 3) p-1pμ(K-1) p-1p≤ ε6L K-1≥ 72 pp-1p 3(σκ) pp-1ε pp-1. (B.53) In addition, we let K−1≥2p−2p−1(pln3)κp2(p−1)K-1≥ 2 p-2p-1(p 3)κ p2(p-1) to make sure η≤21/(p−1)μκp/2(p−1) _y≤ 2^1/(p-1)μκ^p/2(p-1). Therefore, we require K−1K-1 to be (pln3)⋅max72p−1(σκ)p−1ϵp−1,κp2(p−1)22−p−1, (p 3)· \ 72 pp-1(σκ) pp-1ε pp-1, κ p2(p-1)2 2-pp-1 \, (B.54) which means η=2ln(1/ρ)μ(K−1)=min272p/(p−1)μ(ϵκσ)p−1,21p−1μκp2(p−1) _y= 2 (1/ρ)μ(K-1)= \ 272^p/(p-1)μ ( εκσ ) pp-1, 2 1p-1μκ p2(p-1) \ (B.55) Hence, we can conlude that for any t=1,…,Tt=1,…,T, we have [‖t−t⋆‖]≤ϵ/2LE[\|y_t-y_t \|]≤ε/2L if η≤ϵ3κL,η=min272p/(p−1)μ(ϵκσ)p−1,21p−1μκp2(p−1),K−1=(pln3)⋅max72p−1(σκ)p−1ϵp−1,κp2(p−1)22−p−1, cases _x≤ ε3κ L,\\ _y= \ 272^p/(p-1)μ ( εκσ ) pp-1, 2 1p-1μκ p2(p-1) \,\\ K-1=(p 3)· \ 72 pp-1(σκ) pp-1ε pp-1, κ p2(p-1)2 2-pp-1 \, cases (B.56) and ϵ≥2Lδ1ε≥ 2L _1. ■ Lemma B.8. Suppose that Assumption 1 and 2 hold. Let t,tt=1T\x_t,y_t\_t=1^T be the output of TR-SGDAmax, and t⋆y_t be defined as the maxmizer of f(t,)−h()f(x_t,y)-h(y). Then, for any t=2,…,Tt=2,…,T, we have following inequality: η[dist(0,∇Φ(t)+∂r(t))] \ _xE [dist (0,∇ (x_t)+∂ r(x_t) ) ] ≤[Φ(t−1)+r(t−1)]−[Φ(t)+r(t)]+2Lη[‖t−1−t−1⋆‖]+4ησB(p−1)/p+3κLη2. \ [ (x_t-1)+r(x_t-1) ]-E [ (x_t)+r(x_t) ]+2L _xE [ _t-1-y _t-1 ]+ 4 _xσB^(p-1)/p+3κ L _x^2. (B.57) Proof. See proof of Lemma B.5, from (B.5) to (B.7). Note that the validity of (B.57) does not require any additional conditions for η _x. ■ B.3.1 Prove the complexity Proof. We start with (B.57) in Lemma B.8, summing both sides of (B.57) from t=2t=2 to T, then we get ∑t=2Tη[dist(0,∇Φ(t)+∂r(t))] \ _t=2^T _xE [dist (0,∇ (x_t)+∂ r(x_t) ) ] ≤Δ1+∑t=1T−12Lη[‖t−t⋆‖]+4(T−1)ησB(p−1)/p+3(T−1)κLη2 \ ≤ _1+ _t=1^T-12L _xE [ _t-y _t ]+ 4(T-1) _xσB^(p-1)/p+3(T-1)κ L _x^2 (B.58) Multiplying both sides of the above inequality by (η(T−1))−1( _x(T-1))^-1, we get 1T−1∑t=2T[dist(0,∇Φ(t)+∂r(t))] \ 1T-1 _t=2^TE [dist (0,∇ (x_t)+∂ r(x_t) ) ] ≤Δ1(T−1)η+1T−1∑t=1T−12L[‖t−t⋆‖]+4σB(p−1)/p+3κLη. \ ≤ _1(T-1) _x+ 1T-1 _t=1^T-12LE [ _t-y _t ]+ 4σB^(p-1)/p+3κ L _x. (B.59) We first let η=Δ13(T−1)κL _x= _13(T-1)κ L, T−1≥3Δ1κLϵ2T-1≥ 3 _1κ Lε^2 and B=max⌈(4σϵ)p−1⌉,1B= \ ( 4σε ) pp-1 ,1 \ to make sure that 1T−1∑t=2T[dist(0,∇Φ(t)+∂r(t))]≤1T−1∑t=1T−12L[‖t−t⋆‖]+3ϵ 1T-1 _t=2^TE [dist (0,∇ (x_t)+∂ r(x_t) ) ]≤ 1T-1 _t=1^T-12LE [ _t-y _t ]+3ε (B.60) • If ϵ<2Lδ1ε<2L _1, then T−1≥3Δ1κLϵ2≥3Δ1κ4Lδ12T-1≥ 3 _1κ Lε^2≥ 3 _1κ4L _1^2, which means η≤2δ1/3κ _x≤ 2 _1/3κ. By Lemma B.7, we set K−1=pln(6Lδ1ϵ)⋅max72p−1σp−1κp−1ϵp−1,κp2(p−1)22−p−1K-1=p ( 6L _1ε )· \ 72 pp-1σ pp-1κ pp-1ε pp-1, κ p2(p-1)2 2-pp-1 \ to make sure that for any t=2,…,Tt=2,…,T, we have [‖t−t⋆‖]≤ϵ/2LE[\|y_t-y_t \|]≤ε/2L, which means 1T−1∑t=2T[dist(0,∇Φ(t)+∂r(t))]≤2Lδ1T−1+(T−2)ϵT−1+3ϵ≤2Lδ1T−1+4ϵ. 1T-1 _t=2^TE [dist (0,∇ (x_t)+∂ r(x_t) ) ]≤ 2L _1T-1+ (T-2)εT-1+3ε≤ 2L _1T-1+4ε. (B.61) Therefore, If T−1=max3Δ1κLϵ2,2Lδ1ϵT-1= \ 3 _1κ Lε^2, 2L _1ε \, then we can have (1/(T−1))∑t=2T[dist(0,∇Φ(t)+∂r(t))]≤5ϵ(1/(T-1)) _t=2^TE [dist (0,∇ (x_t)+∂ r(x_t) ) ]≤ 5ε, and the total gradient complexity is upper bounded by (TB+TK)≲ (TB+TK)\; \; maxΔ1κLϵ2,Lδ1ϵ×[ln(Lδ1ϵ)⋅maxσp−1κp−1ϵp−1,κp2(p−1)+max(σϵ)p−1,1] \ _1κ Lε^2, L _1ε \× [ ( L _1ε )· \ σ pp-1κ pp-1ε pp-1,κ p2(p-1) \+ \ ( σε ) pp-1,1 \ ] ≲ \; \; Lln(Lδ1ϵ)⋅maxδ1κp2p−2ϵ,Δ1κ3p−22p−2ϵ2,δ1(κσ)p−1ϵ2p−1p−1,Δ1κ2p−1p−1σp−1ϵ3p−2p−1 L ( L _1ε )· \ _1κ p2p-2ε, _1κ 3p-22p-2ε^2, _1(κσ) pp-1ε 2p-1p-1, _1κ 2p-1p-1σ pp-1ε 3p-2p-1 \ (B.62) • If ϵ≥2Lδ1ε≥ 2L _1. We let T−1=3Δ1κLϵ2T-1= 3 _1κ Lε^2, then η=ϵ/3κL _x=ε/3κ L. By Lemma B.7, we set K−1=pln3⋅max72p−1σp−1κp−1ϵp−1,κp2(p−1)22−p−1K-1=p 3· \ 72 pp-1σ pp-1κ pp-1ε pp-1, κ p2(p-1)2 2-pp-1 \ to make sure that for any t=1,…,Tt=1,…,T, we have [‖t−t⋆‖]≤ϵ/2LE[\|y_t-y_t \|]≤ε/2L, which means 1T−1∑t=2T[dist(0,∇Φ(t)+∂r(t))]≤4ϵ 1T-1 _t=2^TE [dist (0,∇ (x_t)+∂ r(x_t) ) ]≤ 4ε holds. Therefore, the total gradient complexity is upper bounded by TK+TB≲ TK+TB\; \; Δ1κLϵ2×[maxσp−1κp−1ϵp−1,κp2(p−1)+max(σϵ)p−1,1] _1κ Lε^2× [ \ σ pp-1κ pp-1ε pp-1,κ p2(p-1) \+ \ ( σε ) pp-1,1 \ ] ≲ \; \; maxΔ1Lκ3p−22p−2ϵ2,Δ1Lκ2p−1p−1σp−1ϵ3p−2p−1 \ _1Lκ 3p-22p-2ε^2, _1Lκ 2p-1p-1σ pp-1ε 3p-2p-1 \ (B.63) ■ Appendix C Supplementary Lemmas Lemma C.1 (Hölder’s inequality). Let q1,q2>1q_1,q_2>1 and 1/q1+1/q2=11/q_1+1/q_2=1 (or q1=q2=1q_1=q_2=1), then for any sequences att=1T\a_t\_t=1^T and btt=1T\b_t\_t=1^T we have ∑t=1T|atbt|≤(∑t=1T|at|q1)1q1(∑t=1T|bt|q2)1q2 _t=1^T|a_tb_t|≤ ( _t=1^T|a_t|^q_1 ) 1q_1 ( _t=1^T|b_t|^q_2 ) 1q_2 (C.1) Moreover, for the probability space (Ω,ℱ,ℙ)( ,F,P), let X and Y are random variables defined on Ω . Hölder’s inequality also means [|XY|]≤([|X|]q1)1q1([|Y|]q2)1q2, E [|XY| ]≤ (E [|X| ]^q_1 ) 1q_1 (E [|Y| ]^q_2 ) 1q_2, (C.2) Lemma C.2. Given two real vectors v,w∈dv,w∈ d. Suppose that p∈[1,2]p∈[1,2] and v≠0v≠ 0, then we have ‖v+w‖p≤‖v‖p+p⟨v,w⟩‖v‖2−p+22−p‖w‖p v+w ^p≤ v ^p+ p v,w v ^2-p+2^2-p w ^p (C.3) Proof. The above lemma can be regarded as a corollary of [46, Theorem 6.3]. Concretely, inequalty (29) in the proof of [46, Theorem 6.3] tells us that for any x1,x2≠0x_1,x_2≠ 0, ν∈[0,1]ν∈[0,1], and any unit vector h∈d−1h ^d-1, |(1+ν)(⟨x2,h⟩/‖x2‖ν−1−⟨x1,h⟩/‖x1‖ν−1)|≤21−ν(1+ν)‖x1−x2‖ν|(1+ν)( x_2,h x_2 ^ν-1- x_1,h x_1 ^ν-1)|≤ 2^1-ν(1+ν)\|x_1-x_2\|^ν holds. Moreover, we have |(1+ν)⟨x2‖x2‖ν−1−x1‖x1‖ν−1,h⟩|≤21−ν(1+ν)‖x1−x2‖ν |(1+ν) x_2\|x_2\|^ν-1- x_1\|x_1\|^ν-1,h |≤ 2^1-ν(1+ν)\|x_1-x_2\|^ν ⇒ suph∈d−1(1+ν)⟨x2‖x2‖ν−1−x1‖x1‖ν−1,h⟩≤21−ν(1+ν)‖x1−x2‖ν _h ^d-1(1+ν) x_2\|x_2\|^ν-1- x_1\|x_1\|^ν-1,h ≤ 2^1-ν(1+ν)\|x_1-x_2\|^ν ⇔ (a) (1+ν)‖ x 2 ∥x 2 ∥ ν-1 - x 1 ∥x 1 ∥ ν-1 ‖ ≤2^1-ν(1+ν)∥x_1-x_2∥^ν ⇔ ‖∇(‖x2‖ν+1)−∇(‖x1‖ν+1)‖≤21−ν(1+ν)‖x1−x2‖ν, ∇(\|x_2\|^ν+1)-∇(\|x_1\|^ν+1) ≤ 2^1-ν(1+ν)\|x_1-x_2\|^ν, (C.4) Here, (a)(a) holds true because the dual norm of the ℓ2 _2 norm on d d is still the ℓ2 _2 norm. We let ‖x‖ν+1\|x\|^ν+1 be denoted as f(x)f(x). Assuming x≠0x≠ 0 and given any y∈dy∈ d, then there is at most one t such that x+t(y−x)=0x+t(y-x)=0. Let’s first assume that there exists a t′∈(0,1)t ∈(0,1) such that x+t′(y−x)=0x+t (y-x)=0. For sufficiently small ϵε, we have 0<t′−ϵ<t′+ϵ<10<t -ε<t +ε<1, and by the fundamental theorem of calculus f(y)−f(x)= f(y)-f(x)= ∫t′+ϵ1⟨∇f(x+t(y−x)),y−x⟩t+∫0t′−ϵ⟨∇f(x+t(y−x)),y−x⟩t _t +ε^1 ∇ f(x+t(y-x)),y-x dt+ _0^t -ε ∇ f(x+t(y-x)),y-x dt +f(x+(t′+ϵ)(y+x))−f(x+(t′−ϵ)(y−x)), +f(x+(t +ε)(y+x))-f(x+(t -ε)(y-x)), (C.5) then subtracting ⟨∇f(x),y−x⟩=([1−(t′+ϵ)]+(t′−ϵ)+2ϵ)⟨∇f(x),y−x⟩ ∇ f(x),y-x =([1-(t +ε)]+(t -ε)+2ε) ∇ f(x),y-x from both sides yields f(y)−f(x)−⟨∇f(x),y−x⟩= f(y)-f(x)- ∇ f(x),y-x = ∫t′+ϵ1⟨∇f(x+t(y−x))−∇f(x),y−x⟩t _t +ε^1 ∇ f(x+t(y-x))-∇ f(x),y-x dt +∫0t′−ϵ⟨∇f(x+t(y−x))−∇f(x),y−x⟩dt + _0^t -ε ∇ f(x+t(y-x))-∇ f(x),y-x dt −2ϵ⟨∇f(x),y−x⟩+f(x+(t′+ϵ)(y+x))−f(x+(t′−ϵ)(y−x)). -2ε ∇ f(x),y-x +f(x+(t +ε)(y+x))-f(x+(t -ε)(y-x)). (C.6) Meanwhile, we have ∫t′+ϵ1⟨∇f(x+t(y−x))−∇f(x),y−x⟩t+∫0t′−ϵ⟨∇f(x+t(y−x))−∇f(x),y−x⟩t _t +ε^1 ∇ f(x+t(y-x))-∇ f(x),y-x dt+ _0^t -ε ∇ f(x+t(y-x))-∇ f(x),y-x dt ≤‖y−x‖(∫t′+ϵ1‖∇f(x+t(y−x))−∇f(x)‖t+∫0t′−ϵ‖∇f(x+t(y−x))−∇f(x)‖t) ≤ y-x ( _t +ε^1 ∇ f(x+t(y-x))-∇ f(x) dt+ _0^t -ε ∇ f(x+t(y-x))-∇ f(x) dt ) ≤21−ν(1+ν)‖y−x‖ν+1(∫t′+ϵ1tνt+∫0t′−ϵtνt) ≤ 2^1-ν(1+ν) y-x ^ν+1 ( _t +ε^1t^νdt+ _0^t -εt^νdt ) =21−ν‖y−x‖ν+1(1−(t+ϵ)ν+1+(t−ϵ)ν+1). =2^1-ν y-x ^ν+1 (1-(t+ε)^ν+1+(t-ε)^ν+1 ). (C.7) Finally we can get f(y)−f(x)−⟨∇f(x),y−x⟩≤ f(y)-f(x)- ∇ f(x),y-x ≤ 21−ν‖y−x‖ν+1(1−(t+ϵ)ν+1+(t−ϵ)ν+1)−2ϵ⟨∇f(x),y−x⟩ 2^1-ν y-x ^ν+1 (1-(t+ε)^ν+1+(t-ε)^ν+1 )-2ε ∇ f(x),y-x +f(x+(t′+ϵ)(y+x))−f(x+(t′−ϵ)(y−x)). +f(x+(t +ε)(y+x))-f(x+(t -ε)(y-x)). (C.8) Taking limϵ→0 _ε→ 0 on both sides of the above inequality, we obtain inequality ‖y‖ν+1−‖x‖ν+1−⟨(ν+1)x‖x‖1−ν,y−x⟩≤21−ν‖y−x‖ν+1\|y\|^ν+1-\|x\|^ν+1- (ν+1)x\|x\|^1-ν,y-x ≤ 2^1-ν y-x ^ν+1. If t′=1t =1 such that x+t′(y−x)=0x+t (y-x)=0, i.e., y=0y=0. The inequality still holds. Let ν=p−1ν=p-1, x=vx=v (v≠0v≠ 0) and y=v+wy=v+w, then the lemma is proved. ■ Lemma C.3. Given a differentiable function f:d↦f: d , suppose f is L-smooth, and let f⋆=supf()<+∞f = _xf(x)<+∞, for any ∈dx∈ d, we have ‖∇f()‖≤2L(f⋆−f()) ∇ f(x) ≤ 2L(f -f(x)) (C.9) Proof. The above inequality is well known for L-smooth functions; here we provide a proof for completeness. Based on the quadratic lower bounds of L-smooth functions, we can obtain f⋆−f()≥f(+(1/L)∇f())−f()−‖∇f()‖2L≥−‖∇f()‖22L, f -f(x)≥ f(x+(1/L)∇ f(x))-f(x)- ∇ f(x) ^2L≥- ∇ f(x) ^22L, (C.10) then we immediately proved the lemma. ∎ Lemma C.4. ([44, Theorem 2], [49, Proposition 1.8]) Given a sequence of filterations ℱtt≥0 \F_t \_t≥ 0. Let p∈[1,2]p∈[1,2], and the random variables X1,X2,…,XnX_1,X_2,…,X_n be a martingale sequence with respect to ℱtt≥0 \F_t \_t≥ 0, i.e., XiX_i is ℱiF_i-measurable and [Xi|ℱi−1]=0E [X_i|F_i-1 ]=0 a.s. If [|Xi|p]<∞E [|X_i|^p ]<∞ for all i=1,…,ni=1,…,n, then we have [|∑i=1nXi|p]≤22−p∑i=1n[|Xi|p] E [ | _i=1^nX_i |^p ]≤ 2^2-p _i=1^nE [|X_i|^p ] (C.11) Lemma C.5. ([26, Lemma 7]) Given a sequence of filterations ℱtt≥1 \F_t \_t≥ 1. Let p∈[1,2]p∈[1,2], and the random vectors X1,X2,…,XnX_1,X_2,…,X_n be a martingale sequence with respect to ℱtt≥1 \F_t \_t≥ 1, i.e., XiX_i is ℱiF_i-measurable and [Xi|ℱi−1]=0E [X_i|F_i-1 ]=0 a.s. If [‖Xi‖p]<∞E [\|X_i\|^p ]<∞ for all i=1,…,ni=1,…,n, then we have [‖∑i=1nXi‖n]≤2∑i=1n[‖Xi‖p] E [ _i=1^nX_i ^n ]≤ 2 _i=1^nE [ X_i ^p ] (C.12)