Paper deep dive
Quasi-Quadratic Gradient: A New Direction for Accelerating the BFGS Method in Quasi-Newton Optimization
John Chiang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 6/21/2026, 7:36:57 AM
Summary
The paper introduces the Quasi-Quadratic Gradient (QQG), a novel search direction designed to accelerate the BFGS method within the quasi-Newton optimization framework. By utilizing the product of the inverse Hessian approximation and the current gradient, QQG leverages local second-order curvature to improve convergence speed and numerical stability compared to vanilla BFGS. The author demonstrates the integration of QQG into foundational optimization algorithms such as Nesterov's Accelerated Gradient (NAG), AdaGrad, and Adam, showing that it provides a robust solution for both convex and non-convex optimization landscapes.
Entities (8)
Relation Signals (5)
Quasi-Quadratic Gradient → accelerates → BFGS
confidence 100% · In this paper, we introduce the Quasi-Quadratic Gradient (QQG), a novel search direction designed to accelerate the BFGS method
Quasi-Quadratic Gradient → enhances → Adam
confidence 100% · In this work, we propose the enhanced Adam method, which is to apply the quadratic gradient to the Adam method.
Quasi-Quadratic Gradient → enhances → AdaGrad
confidence 100% · Specifically, this paper focuses on the enhancement of three foundational algorithms: NAG, AdaGrad, and Adam.
Quasi-Quadratic Gradient → enhances → Nesterov's Accelerated Gradient
confidence 100% · Specifically, this paper focuses on the enhancement of three foundational algorithms: NAG, AdaGrad, and Adam.
Quadratic Gradient → precursorto → Quasi-Quadratic Gradient
confidence 90% · The transition from the original Quadratic Gradient (QG) to the proposed Quasi-Quadratic Gradient (QQG) was driven by a series of empirical observations
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In this paper, we introduce the Quasi-Quadratic Gradient (QQG), a novel search direction designed to accelerate the BFGS method within the quasi-Newton framework. By defining the QQG as the product of the inverse Hessian approximation and the current gradient, we explicitly leverage local second-order curvature to rectify the search path. Theoretical analysis and empirical results demonstrate that our approach significantly outperforms vanilla BFGS in convergence speed while maintaining computational efficiency.
Tags
Links
- Source: https://arxiv.org/abs/2604.23922v1
- Canonical: https://arxiv.org/abs/2604.23922v1
Trouble viewing inline? Open PDF directly →
Full Text
41,612 characters extracted from source content.
Expand or collapse full text
Quasi-Quadratic Gradient: A New Direction for Accelerating the BFGS Method in Quasi-Newton Optimization John Chiang john.chiang.smith@gmail.com Abstract In this paper, we introduce the Quasi-Quadratic Gradient (QQG), a novel search direction designed to accelerate the BFGS method within the quasi-Newton framework. By defining the QQG as the product of the inverse Hessian approximation and the current gradient, we explicitly leverage local second-order curvature to rectify the search path. Theoretical analysis and empirical results demonstrate that our approach significantly outperforms vanilla BFGS in convergence speed while maintaining computational efficiency. 1 Introduction 1.1 Background Optimization is the cornerstone of modern machine learning and scientific computing. Among various techniques, Quasi-Newton methods, particularly the BFGS (Broyden–Fletcher–Goldfarb–Shanno) algorithm, are widely favored for their ability to achieve superlinear convergence without the heavy computational burden of calculating the exact Hessian matrix. 1.2 Related Work Classical Quasi-Newton: Works by Nocedal et al. established the foundations of BFGS and L-BFGS. Natural Gradients: Amari (1998) introduced the Natural Gradient, which utilizes the Fisher Information Matrix to find the steepest descent direction in the information manifold. Accelerated Methods: Various momentum-based and proximal gradient methods (Parikh &\& Boyd, 2014) have been proposed to stabilize and speed up first-order optimization. 1.3 Contributions Our primary contributions are as follows: We propose the Quasi-Quadratic Gradient (QQG), a new mathematical variant that bridges the gap between first-order gradients and second-order curvature. We integrate QQG into the BFGS framework to create an accelerated update rule. We provide extensive experimental evidence showing the superiority of QQG-BFGS over standard optimization benchmarks. 2 Preliminaries 2.1 Newton-Raphson Method The Newton-Raphson method updates parameters using: k+1=k−[∇2f(k)]−1∇f(k)x_k+1=x_k-[∇^2f(x_k)]^-1∇ f(x_k) (1) To reduce cost, the Simplified Fixed Hessian (SFH) method keeps the Hessian constant: k=fixedH_k=H_fixed. While efficient, SFH is sensitive to the initial matrix choice and often converges only linearly. 2.2 Simplified Fixed Hessian The SFH method reduces the cost of Newton’s iteration by keeping the Hessian H constant for several steps: k+1=k−fixed−1∇f(k)x_k+1=x_k-H_fixed^-1∇ f(x_k). While lowering per-iteration cost, it is often unstable in non-convex regions. For two symmetric matrices A and B, A≤BA≤ B is defined in the Loewner ordering iff their difference B−AB-A is positive semi-definite. 2.3 Chiang’s Quadratic Gradient In the following work to [2] in which a simplified diagonal matrix satisfying the fixed Hessian method [1] is constructed, Chiang [3] proposed a faster gradient variant called quadratic gradient. Original Quadratic Gradient Given [3] a differentiable scalar-valued function F()F(x) with its gradient g and Hessian matrix H. For the maximization problem, we need to find a good lower bound matrix H¯≤H H≤ H, where “≤” denotes the Loewner ordering. For the minimization problem, we try to find a good upper bound H¯ H such that H≤H¯H≤ H in the Loewner ordering. Note that the Hessian matrix H itself satisfies these two conditions and can substitute the good bound matrix H¯ H. We attempt to find a fixed good bound matrix of the Hessian matrix for efficiency but could just directly use the Hessian matrix itself. To build the quadratic gradient, we first construct a diagonal matrix B¯ B from the good bound matrix H¯ H as follows: B¯=[1ε+∑i=0d|h¯0i|0…001ε+∑i=0d|h¯1i|…0⋮⋱⋮00…1ε+∑i=0d|h¯di|], B= [ array[]c 1 + _i=0^d| h_0i|&0&…&0\\ 0& 1 + _i=0^d| h_1i|&…&0\\ & & & \\ 0&0&…& 1 + _i=0^d| h_di|\\ array ], where ε is a small positive number to avoid dividing by zero and h¯ji h_ji the elements of the matrix H¯ H . We can then defined the quadratic gradient for the function F()F(x) as G=B¯⋅gG= B· g. The multiplication between the diagonal matrix B¯ B and the gradient g, the quadratic gradient G, is of the same size as the gradient g. To use the quadratic gradient G, we can just use it the same way as the gradient but need a learning rate larger than 11. The well-studied first-order gradient descent methods can also be applied to develop enhanced methods via quadratic gradient. Simplified Quadratic Gradient While the original quadratic gradient utilizes the full information of each row in the bound matrix H¯ H, the computational overhead becomes prohibitive in high-dimensional settings such as deep learning. To address this, Chiang proposed the Simplified Quadratic Gradient [4], which constructs the diagonal scaling matrix B¯ B using only the diagonal elements of the bound matrix H¯ H (or the Hessian matrix H itself). Specifically, the diagonal matrix B¯ B is simplified as follows: B¯=diag(1ε+|h¯00|,1ε+|h¯11|,…,1ε+|h¯dd|), B=diag ( 1 +| h_00|, 1 +| h_11|,…, 1 +| h_d| ), where h¯jj h_j represents the j-th diagonal element of H¯ H. The simplified quadratic gradient is then defined as G=B¯⋅gG= B· g. By neglecting the off-diagonal terms, the construction of G is reduced to element-wise operations, which significantly lowers the memory footprint and accelerates the computation. Empirical evidence suggests that this simplified version maintains a convergence performance comparable to the original framework while being highly compatible with backpropagation and large-scale stochastic optimization in deep learning training. Quadratic Gradient Algorithms The versatility of the quadratic gradient (QG) framework—both in its original and simplified forms—allows it to be seamlessly integrated into prominent first-order optimization algorithms. By substituting the vanilla gradient with its quadratic counterpart, we can incorporate second-order curvature information to enhance convergence. Specifically, this paper focuses on the enhancement of three foundational algorithms: **NAG**, **AdaGrad**, and **Adam**. 1. Nesterov’s Accelerated Gradient (NAG): NAG is a sophisticated momentum-based method designed to anticipate the objective function’s landscape. The standard update rules for NAG are typically expressed as: Vt+1 V_t+1 =t+ηt∇J(t), = β_t+ _t∇ J( β_t), (2) t+1 β_t+1 =(1−γt)Vt+1+γtVt, =(1- _t)V_t+1+ _tV_t, (3) where γt∈(0,1) _t∈(0,1) is a smoothing parameter. The Enhanced NAG algorithm modifies the update rule by substituting the vanilla gradient with the quadratic gradient: Vt+1=t+NtGt,V_t+1= β_t+N_tG_t, (4) where NtN_t is the adaptive learning rate, typically set as Nt=1+ηtN_t=1+ _t. 2. AdaGrad: Well-suited for sparse data, AdaGrad adapts the learning rate for each parameter based on historical gradients. The transition from standard AdaGrad to the Enhanced Quadratic AdaGrad is formulated as follows for each parameter βi _i: • Standard: βi(t+1)=βi(t)−ηtε+∑k=1t(gi(k))2⋅gi(t) _i^(t+1)= _i^(t)- _t + _k=1^t(g_i^(k))^2· g_i^(t) • Enhanced: βi(t+1)=βi(t)−Ntε+∑k=1t(Gi(k))2⋅Gi(t) _i^(t+1)= _i^(t)- N_t + _k=1^t(G_i^(k))^2· G_i^(t) 3. Adam: Adam combines element-wise adaptation with momentum to handle noisy gradients. Following a similar logic, the Enhanced Adam replaces the vanilla gradient with GtG_t within its bias-corrected first and second moment estimates. Given that many contemporary first-order optimizers are derivatives or refinements of these three foundational algorithms, it is highly probable that the quadratic gradient paradigm can be generalized to a much broader class of optimization methods to further accelerate convergence. Building on diagonal scaling, Chiang introduced the Quadratic Gradient: q=∇f()g_q=D∇ f(x), where D acts as a proxy for the inverse Hessian. This paper extends this work to a generalized framework applicable to both convex and non-convex numerical optimization for quasi-Newton method. Building upon the principles of diagonal scaling, Chiang introduced the Quadratic Gradient (QG) framework: q=∇f()g_q=D∇ f(x), where the diagonal matrix D serves as a computationally efficient proxy for the inverse Hessian. This paper extends this methodology by developing a generalized framework that integrates the QG heuristic with the BFGS method within the quasi-Newton paradigm. Our approach transcends the limitations of static scaling, providing a robust solution for numerical optimization across both convex and non-convex regimes. In this work, we propose the enhanced Adam method, which is to apply the quadratic gradient to the Adam method. The naive Adam method and the enhanced Adam method are described in detail in Algorithms 1 and 2 respectively. See the [5] for the detailed description of the parameters in these two Algorithms. Algorithm 1 The Adam method 1:α: Stepsize; β1,β2∈[0,1) _1, _2∈[0,1): Exponential decay rates; f(θ)f(θ): Objective function with parameters θ θ0 _0: Initial parameter vector 2:θt _t: Resulting parameters 3:m0←0m_0← 0: Initialize 1st1^st moment vector 4:v0←0v_0← 0: Initialize 2nd2^nd moment vector 5:t←0t← 0: Initialize timestep 6:while θt _t not converged do 7: t←t+1t← t+1 8: gt←∇θft(θt−1)g_t← _θf_t( _t-1) 9: 10: mt←β1⋅mt−1+(1−β1)⋅gtm_t← _1· m_t-1+(1- _1)· g_t 11: vt←β2⋅vt−1+(1−β2)⋅gt2v_t← _2· v_t-1+(1- _2)· g_t^2 12: m^t←mt/(1−β1t) m_t← m_t/(1- _1^t) 13: v^t←vt/(1−β2t) v_t← v_t/(1- _2^t) 14: θt←θt−1−α⋅m^t/(v^t+ε) _t← _t-1-α· m_t/( v_t+ ) 15:end while 16:return θt _t ⊳ Resulting parameters Algorithm 2 Enhanced Adam method 1:η: Stepsize; β1,β2∈[0,1) _1, _2∈[0,1): Exponential decay rates; f(θ)f(θ): Objective function with parameters θ θ0 _0: Initial parameter vector 2:θt _t: Resulting parameters 3:m0←0m_0← 0: Initialize 1st1^st moment vector 4:v0←0v_0← 0: Initialize 2nd2^nd moment vector 5:t←0t← 0: Initialize timestep 6:while θt _t not converged do 7: t←t+1t← t+1 8: gt←∇θft(θt−1)g_t← _θf_t( _t-1) 9: Gt←B¯⋅gtG_t← B· g_t 10: mt←β1⋅mt−1+(1−β1)⋅m_t← _1· m_t-1+(1- _1)· GtG_t 11: vt←β2⋅vt−1+(1−β2)⋅v_t← _2· v_t-1+(1- _2)· Gt2G_t^2 12: m^t←mt/(1−β1t) m_t← m_t/(1- _1^t) 13: v^t←vt/(1−β2t) v_t← v_t/(1- _2^t) 14: θt←θt−1− _t← _t-1- η ⋅m^t/(v^t+ε)· m_t/( v_t+ ) 15:end while 16:return θt _t 3 Methodology 3.1 Quasi-Newton Optimization Quasi-Newton methods are designed to mitigate the high computational cost of the classical Newton’s method, which requires the evaluation and inversion of the Hessian matrix ∇2f()∇^2f(x) at each iteration (O(n3)O(n^3) complexity). Instead of calculating the exact Hessian, quasi-Newton methods construct a sequence of symmetric positive definite (SPD) approximations Bk≈∇2f(k)B_k≈∇^2f(x_k) using only first-order gradient information. This approach maintains a superlinear convergence rate while reducing the per-iteration complexity to O(n2)O(n^2), making it suitable for medium to large-scale problems. 3.1.1 BFGS Update Schemes The Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm is widely regarded as the most robust and effective quasi-Newton update. Given the step vector sk=xk+1−xks_k=x_k+1-x_k and the gradient displacement yk=gk+1−gky_k=g_k+1-g_k, the BFGS update for the Hessian approximation BkB_k is defined as: Bk+1=Bk+ykykTykTsk−BkskskTBkskTBkskB_k+1=B_k+ y_ky_k^Ty_k^Ts_k- B_ks_ks_k^TB_ks_k^TB_ks_k (5) A pivotal property of the BFGS update is that it preserves the positive definiteness of BkB_k as long as the initial B0B_0 is SPD and the curvature condition skTyk>0s_k^Ty_k>0 is satisfied. This property is crucial for ensuring that the search direction pk=−Bk−1gkp_k=-B_k^-1g_k is always a descent direction. 3.1.2 Line Search Techniques Line search is the strategy of finding an appropriate step length αk _k along the descent direction pkp_k. In the context of quasi-Newton methods, the line search must satisfy specific conditions to ensure both functional decrease and the stability of the Hessian update. 1. Exact Line Search Exact line search determines αk _k by minimizing the objective function along the search ray: αk=argminα>0f(xk+αpk) _k= _α>0f(x_k+α p_k). While theoretically optimal for quadratic functions, it is computationally prohibitive for general non-linear problems due to the requirement of multiple function evaluations. 2. Inexact Line Search and Armijo Condition In practice, finding an exact minimum is often unnecessary. Most algorithms utilize an inexact line search that guarantees a sufficient decrease, formalized by the Armijo condition: f(xk+αpk)≤f(xk)+c1α∇f(xk)Tpkf(x_k+α p_k)≤ f(x_k)+c_1α∇ f(x_k)^Tp_k (6) where c1∈(0,1)c_1∈(0,1) (typically 10−410^-4). This ensures that the reduction in f is proportional to the step size and the directional derivative. 3. Wolfe Conditions To ensure the convergence of quasi-Newton methods and the positive definiteness of the BFGS update, the Wolfe conditions are preferred. They combine the Armijo condition with the curvature condition: ∇f(xk+αpk)Tpk≥c2∇f(xk)Tpk∇ f(x_k+α p_k)^Tp_k≥ c_2∇ f(x_k)^Tp_k (7) where c1<c2<1c_1<c_2<1. The curvature condition forces αk _k into a region where the slope is greater than at the starting point, effectively ensuring skTyk>0s_k^Ty_k>0. For even stricter stability, the Strong Wolfe conditions are often employed by using the absolute value of the derivative: |∇f(xk+αpk)Tpk|≤c2|∇f(xk)Tpk||∇ f(x_k+α p_k)^Tp_k|≤ c_2|∇ f(x_k)^Tp_k|. 4. Backtracking Line Search Backtracking is a simple yet effective implementation of the Armijo-type search. It starts with a candidate step (usually α=1α=1 for Newton-type methods) and iteratively shrinks it by a factor ρ∈(0,1)ρ∈(0,1) until the sufficient decrease condition is met. This method is highly efficient as it avoids expensive interpolation and is the standard for most modern optimization software. 3.2 Quasi-Quadratic Gradient 3.2.1 Motivation The conceptual foundation of the Quadratic Gradient (QG) framework is primarily rooted in the fixed-Hessian variant of Newton’s method. By synthesizing gradient and second-order information, QG provides a robust mechanism for determining descent directions. However, a significant limitation of the current QG approach is its reliance on a static or pre-defined Hessian matrix H, which may fail to capture the dynamically evolving curvature of complex, non-convex loss landscapes. Parallel to the development of QG, the Quasi-Newton family—most notably the BFGS algorithm—has become a cornerstone of efficient optimization. BFGS effectively approximates the Hessian matrix by iteratively incorporating gradient information, satisfying the secant equation without the heavy computational cost of explicit Hessian evaluation. The primary motivation of this work is to bridge the gap between these two methodologies. We propose to extend the ideological core of QG to the Quasi-Newton framework, resulting in the Quasi-Quadratic Gradient (QQG). By embedding the QG heuristic into the BFGS update mechanism, we aim to leverage the adaptive curvature estimation of BFGS while maintaining the superior convergence and numerical stability inherent in the QG framework. This synergy is expected to accelerate convergence, especially in large-scale optimization tasks where local geometry changes rapidly. 3.2.2 Observation A fundamental property that distinguishes the BFGS update from other optimization schemes is its intrinsic ability to maintain the positive definiteness of the Hessian approximation. Throughout the iterative process, as long as the initial matrix B0B_0 is positive definite and the curvature condition skTyk>0s_k^Ty_k>0 is satisfied, the sequence of matrices Bk\B_k\ remains symmetric and positive definite (SPD). Importantly, this property is invariant to the optimization objective, holding true regardless of whether the task is a minimization or maximization problem. Within our proposed framework, this observation is critical; it ensures that the Quasi-Quadratic Gradient (QQG) consistently generates a valid descent direction. By leveraging this guaranteed positive definiteness, QQG achieves superior numerical stability, particularly when navigating the complex, highly non-convex landscapes often encountered in modern cryptographic and machine learning applications. Remark A pivotal observation in the integration of QG and BFGS is the guaranteed positive definiteness of the Hessian approximation. In the standard BFGS update, as long as the initial matrix B0B_0 is positive definite and the curvature condition skTyk>0s_k^Ty_k>0 is satisfied (typically ensured by Wolfe line search), the resulting sequence of matrices Bk\B_k\ remains symmetric and positive definite. Crucially, this property holds regardless of whether the optimization objective is a maximization or minimization task, provided the problem is formulated to seek a stationary point where the approximated curvature correctly reflects the local geometry. This intrinsic positive definiteness ensures that the QQG framework always yields a consistent descent direction, enhancing its numerical robustness in diverse application scenarios. 3.2.3 Evolution of the Proposed Approach The transition from the original Quadratic Gradient (QG) to the proposed Quasi-Quadratic Gradient (QQG) was driven by a series of empirical observations and a fundamental re-evaluation of curvature approximation strategies. Initial Attempts and Challenges In our early developments (mid-2022), we strictly adhered to the diagonal construction methodology of the original QG framework by applying the BFGS-derived curvature information to a diagonalized proxy. However, this hybrid approach yielded suboptimal performance and failed to achieve convergence. Retrospectively, this instability was primarily attributed to the learning rate scheduling; we initially employed a decaying learning rate starting above 1.01.0—consistent with the original QG’s mechanism—which proved incompatible with the dynamic nature of BFGS curvature updates. Rethinking the Diagonal Constraint Moving beyond conventional diagonal scaling, we explored the possibility of utilizing a full, positive-definite matrix that satisfies the convergence criteria of the Fixed Hessian (FH) Newton method. While theoretically appealing, this direction presented two significant bottlenecks: 1. Computational Overhead: The inversion of a full Hessian-like matrix at each iteration is computationally prohibitive for high-dimensional problems. 2. Bounding Efficiency: Systematically constructing a symmetric positive definite (SPD) matrix that provides a tighter Loewner bound than the Simplified Fixed Hessian (SFH) remains an open challenge in optimization theory. The Quasi-Quadratic Breakthrough By re-evaluating these constraints, we recognized that since the BFGS approximation BkB_k (and its inverse) is inherently maintained as an SPD matrix throughout the optimization process (for both maximization and minimization), it already satisfies the core requirement of a reliable curvature proxy. This realization led to the synthesis of the Quasi-Quadratic Gradient: rather than forcing a diagonal structure or searching for a static FH substitute, we directly utilize the product of the BFGS inverse Hessian and the gradient (Bk−1gkB_k^-1g_k) as the search vector. Subsequent experiments validated this intuition, demonstrating that this direct integration significantly outperforms previous diagonalized attempts and provides robust convergence across diverse optimization landscapes. Newton-Raphson Method↦ TtT [kim2018logistic] Fixed Hessian Method↦ TtT [kim2018logistic] Simplified Fixed Hessian Newton-Raphson Method TtT [cite][\@@bibrefNumberkim2018logistic] Fixed Hessian Method TtT [cite][\@@bibrefNumberkim2018logistic] Simplified Fixed Hessian ↦TtT [kim2018logistic] Original Quadratic Gradient↦ TtT [kim2018logistic] Simplified Quadratic Gradient TtT [cite][\@@bibrefNumberkim2018logistic] Original Quadratic Gradient TtT [cite][\@@bibrefNumberkim2018logistic] Simplified Quadratic Gradient Newton-Raphson Method↦ TtT [kim2018logistic] Fixed Hessian Method↦ TtT [kim2018logistic] Positive Definite Hessian Newton-Raphson Method TtT [cite][\@@bibrefNumberkim2018logistic] Fixed Hessian Method TtT [cite][\@@bibrefNumberkim2018logistic] Positive Definite Hessian ↦? Original Quadratic Gradient ? Original Quadratic Gradient Newton-Raphson Method↦ TtT [kim2018logistic] Quasi-Newton Optimization↦ TtT [kim2018logistic] Positive Definite Hessian Newton-Raphson Method TtT [cite][\@@bibrefNumberkim2018logistic] Quasi-Newton Optimization TtT [cite][\@@bibrefNumberkim2018logistic] Positive Definite Hessian ↦TtT [kim2018logistic] BFGS Hessian Matrix↦ TtT [kim2018logistic] Quasi-Quadratic Gradient TtT [cite][\@@bibrefNumberkim2018logistic] BFGS Hessian Matrix TtT [cite][\@@bibrefNumberkim2018logistic] Quasi-Quadratic Gradient The development of the Quasi-Quadratic Gradient (QQG) follows a systematic logical progression, moving from static optimization to dynamic curvature adaptation. Our research trajectory was guided by three progressive stages of reasoning: 1. Generalization of QG: We initially recognized that the original Quadratic Gradient framework acts as a bridge between first-order gradients and second-order Newton-type methods. However, the reliance on a fixed Hessian H posed a bottleneck for its generalization to diverse objective functions. 2. Integration of Secant Updates: To overcome the static nature of H, we turned to the Quasi-Newton branch. By observing that the BFGS algorithm effectively "learns" the local curvature through the secant equation, we identified the potential to replace the constant H in the QG framework with the dynamically evolving matrix BkB_k. 3. Synthesis of QQG: The final development phase involved reformulating the QG synthesis mechanism to accommodate the rank-two updates of BFGS. This resulted in the QQG paradigm, which not only inherits the fast convergence of BFGS but also preserves the structural robustness and numerical stability of the QG heuristic. Through this evolution, our approach transitions from a rigid second-order approximation to a self-adaptive optimization scheme, capable of accelerating convergence by continuously refining its internal representation of the loss landscape. 3.2.4 Definition Building upon the foundations of both the original Quadratic Gradient and the BFGS update mechanism, we define the Quasi-Quadratic Gradient (QQG) as a dynamic synthesis of first-order gradients and evolving curvature estimates. Unlike the original QG which relies on a fixed or pre-defined bound matrix H¯ H, the QQG directly leverages the inverse Hessian approximation Bk−1B_k^-1 generated by the BFGS algorithm. Definition (Quasi-Quadratic Gradient). Let BkB_k be the symmetric positive definite (SPD) matrix maintained by the BFGS update at iteration k. We define the Quasi-Quadratic Gradient GqqG_q as the product of the inverse Hessian approximation and the current gradient gkg_k: Gqq(k)=Bk−1gk.G_q^(k)=B_k^-1g_k. Since BkB_k is guaranteed to be positive definite through the BFGS update and appropriate line search conditions, GqqG_q inherently points in a reliable descent (or ascent) direction, adapted to the local geometric curvature. Update Rules The QQG is integrated into the parameter update process depending on the nature of the optimization task. Given the parameter vector β, the iterative update rules are formulated as follows: • Maximization Task: To find the local maximum of F()F(x), the update follows the direction of the QQG: βk+1=βk+ηkGqq(k), _k+1= _k+ _kG_q^(k), • Minimization Task: To find the local minimum of F()F(x), the update moves against the direction of the QQG: βk+1=βk−ηkGqq(k), _k+1= _k- _kG_q^(k), where ηk _k is the learning rate. By substituting the static bound matrix B¯ B from the original QG with the dynamic inverse Hessian Bk−1B_k^-1, the QQG effectively bridges the gap between first-order efficiency and second-order precision, allowing for accelerated convergence in complex numerical optimization landscapes. Compatibility with Line Search Techniques It is worth noting that the proposed Quasi-Quadratic Gradient framework is fully compatible and can coexist with existing line search methodologies. While the QQG provides a robust search direction GqqG_q, the determination of the optimal learning rate ηk _k (or step size) can be adaptively managed through various line search procedures. For instance, techniques such as backtracking line search or those satisfying the Wolfe conditions can be seamlessly integrated to dynamically adjust ηk _k at each iteration. This synergy ensures that the QQG update not only follows a curvature-aware direction but also maintains a step length that guarantees sufficient decrease in the objective function, further enhancing the global convergence and numerical stability of the algorithm. It is worth noting that the proposed Quasi-Quadratic Gradient framework is fully compatible and can coexist with existing line search methodologies, particularly when integrated into momentum-based first-order schemes such as Nesterov’s Accelerated Gradient (NAG). While the QQG provides a robust, curvature-aware search direction GqqG_q, the determination of the optimal learning rate ηk _k can be adaptively managed through various line search procedures. For instance, techniques such as backtracking line search or those satisfying the Wolfe conditions can be seamlessly integrated to dynamically adjust ηk _k at each iteration. This synergy ensures that the QQG update—which effectively synthesizes historical momentum with local second-order geometry—maintains a step length that guarantees a sufficient decrease in the objective function. Such integration further enhances the global convergence and numerical stability of the algorithm, particularly when navigating the ill-conditioned or highly non-convex landscapes prevalent in deep learning and complex optimization tasks. 3.2.5 Algorithms The integration of the Quasi-Quadratic Gradient (GqqG_q) into foundational first-order methods requires specific adaptations to leverage its dynamic curvature information. While the element-wise adaptive methods (AdaGrad and Adam) follow a straightforward substitution, the momentum-based NAG requires a more nuanced approach to learning rate scheduling and step-size determination. 1. Enhanced NAG with Adaptive Scaling (QQG-NAG): To harmonize the momentum term with the second-order information in GqqG_q, we modify the Nesterov update by introducing a dynamic scaling factor NtN_t. The updated iterative process is: Vt+1 V_t+1 =t+ηtGqq(t), = β_t+ _tG_q^(t), t+1 β_t+1 =(1−γt)Vt+1+γtVt, =(1- _t)V_t+1+ _tV_t, where the effective learning rate ηt _t is determined by one of two strategies: • Monotonic Warm-up: ηt _t is scheduled to increase gradually from a small positive value ηmin>0 _min>0 towards 11 (e.g., ηt=min(1,ηmin+Δt) _t= (1, _min+ t)), allowing the algorithm to stabilize the curvature estimate before taking full quasi-Newton steps. • Line Search Integration: ηt _t is determined via a backtracking line search satisfying the Wolfe conditions, ensuring that the step taken along the QQG-modified direction guarantees sufficient descent. 2. Enhanced AdaGrad (QQG-AdaGrad): The QQG-enhanced AdaGrad replaces the vanilla gradient with GqqG_q, utilizing the historical accumulation of the quasi-quadratic terms to scale each dimension: βi(t+1)=βi(t)−ηε+∑k=1t(Gqq,i(k))2⋅Gqq,i(t), _i^(t+1)= _i^(t)- η + _k=1^t(G_q,i^(k))^2· G_q,i^(t), where η is the global step size. This ensures that parameters with large accumulated curvature-adjusted gradients receive smaller updates. 3. Enhanced Adam (QQG-Adam): Following the same logic, QQG-Adam incorporates GqqG_q into the first and second moment estimates: mt m_t =β1mt−1+(1−β1)Gqq(t), = _1m_t-1+(1- _1)G_q^(t), vt v_t =β2vt−1+(1−β2)(Gqq(t))2. = _2v_t-1+(1- _2)(G_q^(t))^2. By substituting the vanilla gradient with GqqG_q, Adam can more effectively navigate non-convex landscapes by leveraging both momentum and adaptive second-order curvature. To evaluate the practical performance of the proposed Quasi-Quadratic Gradient (QQG) framework, we established specific hyperparameter configurations for the enhanced versions of AdaGrad and Adam. For QQG-AdaGrad, we recommend an augmented learning rate of Nt=0.1N_t=0.1, which is a ten-fold increase over the standard default of 0.010.01. Similarly, for QQG-Adam, the optimal configuration adopts an adjusted learning rate of α=0.01α=0.01 (compared to the baseline α=0.001α=0.001), while maintaining the standard momentum decay parameters at β1=0.9 _1=0.9 and β2=0.999 _2=0.999. The ability of QQG-enhanced optimizers to remain stable and converge faster with these larger step sizes suggests that the dynamic inverse Hessian information provides a more reliable descent direction than vanilla gradients, effectively mitigating the risk of divergence typically associated with high learning rates in complex optimization landscapes. 3.3 Example The efficiency of the Adam optimizer [5] is governed by three primary parameters, each controlling a distinct aspect of the stochastic gradient descent process: • Learning Rate (α): This parameter determines the step size of the parameter update. In Adam, α acts as an upper bound on the effective step size. It is the most critical hyper-parameter to tune, as it directly influences the convergence rate and the stability of the training process. • First Moment Decay (β1 _1): It controls the exponential moving average of the gradients (momentum). Physically, β1 _1 accounts for the “inertia” of the optimization trajectory. A higher β1 _1 (typically 0.90.9) helps the optimizer to smooth out high-frequency noise in the gradients and persist in the consistent direction of the descent. • Second Moment Decay (β2 _2): It governs the exponential moving average of the squared gradients. This term provides the “adaptive” nature of the algorithm by scaling the update inversely proportional to the square root of recent gradient magnitudes. This ensures that parameters with sparse or small gradients receive larger updates, while those with large, frequent gradients are tempered, preventing divergence. To demonstrate the plug-and-play capability of our proposed Quasi-Quadratic Gradient (QQG), we integrate it into the Adam framework, termed QQG-Adam. We observe that the fundamental roles of Adam’s hyper-parameters remain consistent when substituting the vanilla gradient with QQG: • Directional Smoothing (β1 _1): In QQG-Adam, β1 _1 governs the exponential moving average of the quasi-quadratic terms. It serves to filter out high-frequency noise inherent in stochastic approximations, ensuring that the optimization trajectory leverages the structural curvature information provided by QQG without succumbing to local instabilities. • Curvature-aware Adaptation (β2 _2): While β2 _2 traditionally tracks gradient variance, in our framework, it scales the update step based on the accumulated magnitude of QQG. This confirms that our QQG maintains the necessary statistical properties to allow for per-parameter learning rate adaptation, effectively balancing the update intensity across different layers. • Effective Step-size Control (α): Our experiments confirm that QQG-Adam exhibits a similar sensitivity to the learning rate α as vanilla Adam. This empirical alignment suggests that QQG does not introduce unexpected scale variances, allowing researchers to utilize standard hyper-parameter tuning strategies (e.g., 10−310^-3 as a baseline) for our algorithm. 4 Numerical Experiments 4.1 Convex Benchmarks These functions test the algorithm’s base convergence rate and handling of dimensional scaling: • Sphere Function: f()=∑i=1nxi2f(x)= _i=1^nx_i^2 • Sum of Different Powers: f()=∑i=1n|xi|i+1f(x)= _i=1^n|x_i|^i+1 4.2 Non-Convex Benchmarks These functions test the algorithm’s ability to navigate ill-conditioned valleys and escape local optima: • Rosenbrock Function: f()=∑i=1n−1[100(xi+1−xi2)2+(1−xi)2]f(x)= _i=1^n-1[100(x_i+1-x_i^2)^2+(1-x_i)^2] • Rastrigin Function: f()=10n+∑i=1n[xi2−10cos(2πxi)]f(x)=10n+ _i=1^n[x_i^2-10 (2π x_i)] 4.2.1 Saddle Points On the Monkey Saddle f(x,y)=x3−3xy2f(x,y)=x^3-3xy^2, SGD stagnates due to vanishing gradients. Our QG variant identifies near-zero curvature λi _i, yielding a large adaptive step ηi≈1/ε _i≈ 1/ , effectively "teleporting" the trajectory away from the saddle point. The theoretical derivation of the Quadratic Gradient suggests a superior ability to handle irregular curvature. We strategically select a suite of benchmark functions that simulate notorious challenges in numerical optimization. To evaluate the ability of the Quadratic Gradient variants to escape saddle points, we introduce functions with specific singular curvatures: • Monkey Saddle Function: A classic cubic surface with a single saddle point at the origin where the Hessian is indefinite. f(x,y)=x3−3xy2f(x,y)=x^3-3xy^2 (8) • Beale’s Function (Revisited): While previously mentioned, its flat regions near the boundary act as plateau-like saddle points that test gradient acceleration. • Himmelblau’s Function: Contains four local minima and one local maximum (saddle-like) in the center, testing the algorithm’s ability to navigate away from unstable equilibrium points. f(x,y)=(x2+y−11)2+(x+y2−7)2f(x,y)=(x^2+y-11)^2+(x+y^2-7)^2 (9) • Six-Hump Camel Function: Contains six local minima and several saddle points. f(x,y)=(4−2.1x2+x43)x2+xy+(−4+4y2)y2f(x,y)=(4-2.1x^2+ x^43)x^2+xy+(-4+4y^2)y^2 (10) Remark on Saddle Point Dynamics: In the neighborhood of a saddle point where ‖∇f‖≈0||∇ f||≈ 0, the update of first-order methods Δ=−α∇f =-α∇ f vanishes, leading to stagnation. However, the Quadratic Gradient utilizes the spectral information of the Hessian proxy. For directions where the eigenvalue λi _i vanishes, the adaptive scaling factor ηi=(ε+λi)−1 _i=( + _i)^-1 compensates for the diminishing gradient. This effectively reshapes the vector field near the saddle point, ensuring a non-zero, high-velocity traversal along the directions of minimal curvature. Empirical results show that the QG variant outperforms standard GD in terms of iteration count to convergence across most benchmarks. Notably, in the Monkey Saddle experiment, the QG variant successfully avoids the stagnation observed in SGD. By identifying minimal eigenvalues λi _i, the algorithm assigns a massive update rate to those dimensions, effectively "teleporting" the trajectory away from the saddle point. Performance Discussion. As illustrated in the experimental results LABEL:fig0,_fig1,_fig2, the two quadratic gradient variants for AdaGrad and Adam share a highly consistent parameter structure and exhibit remarkably similar convergence performance. Given that our Simplified Quadratic Gradient framework offers a significantly more streamlined construction process while maintaining comparable efficiency, we conclude that the SQG approach is particularly well-suited for integration with adaptive optimization methods like AdaGrad and Adam. The observed instability and divergence in the SQG-enhanced NAG variant are not unexpected. From a theoretical perspective, the convergence of the original quadratic gradient algorithm is anchored in the Fixed Hessian framework, which permits a learning rate lower bound as high as 11. However, our SQG framework does not strictly satisfy the stringent convergence conditions required by the Fixed Hessian method. Specifically, unlike the original method where the Hessian approximation provides a stable curvature estimate, the simplified nature of SQG necessitates a more conservative step-size policy for NAG-based updates. Consequently, while the original framework allows for a persistent learning rate, the SQG-enhanced NAG requires a learning rate that eventually decays below 11 and asymptotically approaches 0 to ensure numerical stability and global convergence. (a) The iDASH dataset (b) The Edinburgh datasetn (c) The lbw dataset (d) The nhanes3 dataset (e) The pcs dataset (f) The uis dataset (g) restructured MNIST dataset (h) The private financial dataset Figure 1: The training results of NAG + SQG vs. NAG + OQG vs. NAG in the clear domain. (a) The iDASH dataset (b) The Edinburgh datasetn (c) The lbw dataset (d) The nhanes3 dataset (e) The pcs dataset (f) The uis dataset (g) restructured MNIST dataset (h) The private financial dataset Figure 2: The training results of AdaGrad + SQG vs. AdaGrad + OQG vs. AdaGrad in the clear domain. (a) The iDASH dataset (b) The Edinburgh datasetn (c) The lbw dataset (d) The nhanes3 dataset (e) The pcs dataset (f) The uis dataset (g) restructured MNIST dataset (h) The private financial dataset Figure 3: The training results of Adam + SQG vs. Adam + OQG vs. Adam in the clear domain. 5 Conclusion In this work, we introduced the Quasi-Quadratic Gradient as a new direction for quasi-Newton optimization. By refining the search direction through the interaction of the inverse Hessian proxy and the gradient, we achieve faster convergence in complex optimization landscapes. Future work will explore the application of QQG in large-scale distributed training. All the python source code to implement the experiments in the paper is openly available at: https://github.com/petitioner/ML.QuasiQuadraticGradient . References [1] D. Böhning and B. G. Lindsay (1988) Monotonicity of quadratic-approximation algorithms. Annals of the Institute of Statistical Mathematics 40 (4), p. 641–663. Cited by: §2.3. [2] C. Bonte and F. Vercauteren (2018) Privacy-preserving logistic regression training. BMC medical genomics 11 (4), p. 13–21. Cited by: §2.3. [3] J. Chiang (2022) Privacy-preserving logistic regression training with a faster gradient variant. arXiv preprint arXiv:2201.10838. Cited by: §2.3, §2.3. [4] J. Chiang (2022) Quadratic gradient: a unified framework bridging gradient descent and newton-type methods by synthesizing hessians and gradients. arXiv preprint arXiv:2209.03282. Cited by: §2.3. [5] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §2.3, §3.3.