Paper deep dive
HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation
Qiyuan Chen, Xian Wu, Yi Wang, Xianhao Chen
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/22/2026, 5:14:21 AM
Summary
HO-SFL (Hybrid-Order Split Federated Learning) is a novel distributed learning framework that decouples the optimization landscape using a Lagrangian framework. It enables memory-efficient, backpropagation-free training on edge clients by utilizing zeroth-order optimization for client-side updates, while the server performs precise first-order backpropagation. The method achieves dimension-free model aggregation and convergence rates comparable to first-order methods, significantly reducing communication and memory overhead.
Entities (5)
Relation Signals (4)
HO-SFL → implements → Zeroth-order optimization
confidence 95% · whereas clients conduct memory-efficient zeroth-order optimization
HO-SFL → implements → Backpropagation
confidence 95% · The server performs precise first-order updates (i.e., BP)
HO-SFL → utilizes → Lagrangian framework
confidence 95% · By reformulating the split learning process within a Lagrangian framework, HO-SFL decouples the optimization landscape
HO-SFL → reduces → Communication costs
confidence 90% · enables dimension-free model aggregation, drastically lowering communication costs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Fine-tuning large models on edge devices is severely hindered by the memory-intensive backpropagation (BP) in standard frameworks like federated learning and split learning. While substituting BP with zeroth-order optimization can significantly reduce memory footprints, it typically suffers from prohibitively degraded convergence speed. To resolve this dilemma, we propose Hybrid-Order Split Federated Learning (HO-SFL). By reformulating the split learning process within a Lagrangian framework, HO-SFL decouples the optimization landscape: The server performs precise first-order updates (i.e., BP), whereas clients conduct memory-efficient zeroth-order optimization. This hybrid design not only eliminates the need for client-side BP but also enables dimension-free model aggregation, drastically lowering communication costs. Crucially, we provide a theoretical convergence analysis, demonstrating that HO-SFL mitigates the dimension-dependent convergence slowdown of zeroth-order optimization, achieving a convergence rate comparable to first-order methods. Extensive experiments on tasks across vision and language modalities validate that HO-SFL achieves convergence speeds comparable to first-order baselines while significantly reducing communication costs and client memory footprints.
Tags
Links
- Source: https://arxiv.org/abs/2603.14773v1
- Canonical: https://arxiv.org/abs/2603.14773v1
Trouble viewing inline? Open PDF directly →
Full Text
99,893 characters extracted from source content.
Expand or collapse full text
HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation Qiyuan Chen Xian Wu Yi Wang Xianhao Chen Abstract Fine-tuning large models on edge devices is severely hindered by the memory-intensive backpropagation (BP) in standard frameworks like federated learning and split learning. While substituting BP with zeroth-order optimization can significantly reduce memory footprints, it typically suffers from prohibitively degraded convergence speed. To resolve this dilemma, we propose Hybrid-Order Split Federated Learning (HO-SFL). By reformulating the split learning process within a Lagrangian framework, HO-SFL decouples the optimization landscape: The server performs precise first-order updates (i.e., BP), whereas clients conduct memory-efficient zeroth-order optimization. This hybrid design not only eliminates the need for client-side BP but also enables dimension-free model aggregation, drastically lowering communication costs. Crucially, we provide a theoretical convergence analysis, demonstrating that HO-SFL mitigates the dimension-dependent convergence slowdown of zeroth-order optimization, achieving a convergence rate comparable to first-order methods. Extensive experiments on tasks across vision and language modalities validate that HO-SFL achieves convergence speeds comparable to first-order baselines while significantly reducing communication costs and client memory footprints. Machine Learning, ICML 1 Introduction Driven by the imperative needs for task-specific-adaptation and data privacy, fine-tuning large models on edge devices—where data is generated—has emerged as a critical demand in the era of artificial intelligence (Malladi et al., 2023; Peng et al., 2024). Unlike centralized training, which requires aggregating user data, edge learning paradigms enable collaborative model adaptation without compromising user privacy (Lim et al., 2020; Sani et al., 2025). However, the resource-constrained nature of edge devices poses significant challenges to existing distributed learning frameworks (Lv et al., 2024; Wang et al., 2024). The most prominent framework, federated learning (FL) (McMahan et al., 2017), necessitates that clients perform complete backpropagation (BP) to compute gradients locally. To enable the training of larger models, split federated learning (SFL) (Thapa et al., 2022) was proposed to offload a substantial portion of the model computation to a resource-rich server. Nevertheless, standard SFL and most variants still require clients to execute BP on the client-side sub-model to compute gradients for updates (Oh et al., 2025; Nair et al., 2025). For modern large language models (LLMs) with billions or even trillions of parameters, the memory overhead of storing activations for BP far exceeds the capacity of typical edge hardware, even when the client-side sub-model consists of only a few layers (Lin et al., 2024; Dettmers et al., 2023). Consequently, the memory bottleneck remains a formidable barrier for LLM deployment on edge devices. As a memory-efficient alternative to BP, zeroth-order (ZO) optimization (Spall, 2002) has garnered increasing attention (Liu et al., 2020). By estimating gradients using only forward passes, ZO optimization eliminates the need to store intermediate activations, drastically reducing memory footprints. Several works have attempted to integrate ZO techniques into FL and SFL to achieve “BP-free” client training (Fang et al., 2022; Li et al., 2025; Liang et al., 2025). However, existing distributed ZO methods often suffer from slow convergence rates compared to first-order methods due to the high variance of gradient estimators and the lack of precise gradient information, particularly in high-dimensional parameter spaces (Shamir, 2017; Ma & Huang, 2025b). Therefore, a critical research question arises: Can we enable BP-free client training in distributed learning without inheriting the high-dimensional slowdown of zeroth-order optimization? We answer this question affirmatively by proposing Hybrid-Order Split Federated Learning (HO-SFL). By treating the consistency between client-side outputs and server-side inputs as an equality constraint, we decouple the SFL optimization landscape into two distinct local sub-problems via a Lagrangian framework. In this decoupled regime, the server performs BP to update its parameters and transmits the activation gradients to the clients. These gradients define a local proxy objective for the clients, enabling them to perform memory-efficient ZO updates that align with the global optimization direction. In this way, HO-SFL can leverage the server’s computational power for precise first-order updates to maintain high convergence speed, while enabling clients to perform memory-efficient ZO training without the burden of BP. Furthermore, by leveraging shared randomness, HO-SFL achieves dimension-free model aggregation (Li et al., 2025). Specifically, instead of transmitting high-dimensional model parameters, clients need only upload a few scalars to achieve client-side model aggregation, thereby significantly reducing communication overhead compared to other SFL methods. Our main contributions are summarized as follows: • We propose HO-SFL, a novel framework that enables BP-free training on clients while maintaining convergence speeds comparable to first-order methods. By reformulating the split learning process within a Lagrangian framework, the server performs first-order updates via BP, while clients leverage the back-transmitted gradients to execute ZO updates. Moreover, HO-SFL enables dimension-free model aggregation, drastically lowering communication costs. • We theoretically analyze HO-SFL and prove that it achieves a convergence rate of (dc/PT)O( d_c/PT), which effectively mitigates the dimension-dependent convergence slowdown inherent in ZO optimization. Our analysis establishes a theoretical characterization of the dynamics of hybrid-order optimization, providing a rigorous foundation for integrating first-order and zeroth-order optimization methods within a framework. • We conduct experiments across vision and language modalities to validate our claims. Our results demonstrate that HO-SFL achieves convergence speed comparable to first-order baselines, while reducing client memory usage to inference-only level and minimizing communication cost via dimension-free aggregation. 2 Related Work In this section, we review the literature relevant to our work across three primary dimensions: advances in SFL, the evolution of ZO optimization, and recent works integrating ZO into distributed frameworks. 2.1 Advances in Split Federated Learning Split Federated Learning combines the parallel training of FL with the model splitting of Split Learning (SL) (Gupta & Raskar, 2018; Vepakomma et al., 2018) to reduce client-side computational load. Recent works have focused on optimizing communication efficiency and training latency in SFL (Wu et al., 2023). SplitFC (Oh et al., 2025) introduces adaptive feature-wise compression, utilizing dropout and quantization strategies based on feature dispersion to reduce communication overhead. To address the high latency caused by the sequential server-side updates in SFL, FSL-SAGE (Nair et al., 2025) employs auxiliary models on clients to estimate server gradients, enabling parallel client training. Furthermore, GAS (Yang & Liu, 2025) introduces an asynchronous SFL framework with generative activation buffers to mitigate the impact of stragglers and biased updates caused by asynchronous transmissions. Unlike these works, which primarily focus on compression or asynchrony while retaining client-side BP, our work focuses on eliminating client-side BP entirely through a hybrid optimization approach. 2.2 Zeroth-Order Optimization ZO optimization estimates gradients using function value differences, avoiding the memory-intensive storage of activation graphs required by BP (Duchi et al., 2015; Nesterov & Spokoiny, 2017). A landmark advancement in this domain is MeZO (Malladi et al., 2023), which adapts ZO-SGD to fine-tune LLMs with memory footprints equivalent to inference. Theoretical advancements have also been made to improve ZO estimators. For instance, Hikima et al. (2025) leverages partial gradient information to shape the covariance matrix of perturbations for more effective update directions, while Ma & Huang (2025a) introduces a telescoping series framework to construct unbiased gradient estimators, effectively eliminating the approximation bias inherent in traditional finite-difference methods. While these methods demonstrate the efficacy of ZO in centralized settings, applying them directly to distributed environments without structural adaptation often leads to suboptimal convergence. 2.3 Zeroth-Order in Distributed Learning Recent research has begun to explore the intersection of ZO optimization with distributed learning frameworks (Yi et al., 2022). FedZO (Fang et al., 2022) integrates ZO optimization into FL, enabling clients to update models using stochastic gradient estimators without BP. DeComFL (Li et al., 2025) leverages ZO optimization to achieve dimension-free communication in FL by transmitting random seeds and scalars instead of high-dimensional model parameters. In the context of SFL, MU-SplitFed (Liang et al., 2025) combines ZO optimization with an unbalanced update strategy, where the server performs multiple update steps for every client step to mitigate straggler effects. However, these methods suffer from the high variance of ZO gradient estimators, resulting in a convergence rate significantly slower than first-order methods (Ajalloeian et al., 2019). In contrast, our HO-SFL framework not only enables BP-free training on clients to minimize memory costs but also achieves a convergence rate comparable to first-order methods, effectively bridging the performance gap. 3 Methodology In this section, we provide the theoretical motivation and the algorithmic design of the proposed Hybrid-Order Split Federated Learning (HO-SFL). We reformulate the standard SFL training process through the lens of constrained optimization, which naturally leads to a decoupled learning strategy where the server performs standard BP while the client employs ZO optimization. 3.1 System Model We consider an SFL system with one central server and M clients, indexed by m∈1,…,Mm∈\1,…,M\. The model is split into a client-side network fc(⋅;c)f_c(·; θ_c) and a server-side network fs(⋅;s)f_s(·; θ_s), with parameters :=[c;s]∈ℝdc+ds θ:=[ θ_c; θ_s] ^d_c+d_s. For clarity, we present the system model using a single-sample formulation. We define a data sample as ξm=(m,ym) _m=( x_m,y_m) drawn from client m’s local distribution mD_m. Client m computes the activation m z_m via the client-side model fcf_c: m=fc(m;c)∈ℝD. z_m=f_c( x_m; θ_c) ^D. (1) Upon receiving m z_m, the server completes the forward pass to obtain y^m=fs(m;s) y_m=f_s( z_m; θ_s) and evaluates the task loss ℓ(;ξm):=ℓ(y^m,ym). ( θ; _m):= ( y_m,y_m). (2) For each client m, we define the local objective function as ℒm():=ξm∼m[ℓ(;ξm)]L_m( θ):=E_ _m _m [ ( θ; _m) ] (3) and formulate the global optimization problem as ℒ():=1M∑m=1Mℒm().L( θ):= 1M _m=1^ML_m( θ). (4) It is worth noting that while we use uniform weights here for simplicity, this formulation can be trivially generalized to weighted averaging strategies, such as weighting based on the size of each client’s dataset. 3.2 Theoretical Motivation: A Lagrangian Perspective While the goal is to minimize the global loss ℒ()L( θ), standard SFL addresses this by directly targeting the nested composite function ℓ(fs(fc(;c);s),y) (f_s(f_c( x; θ_c); θ_s),y) as the immediate optimization objective for each data sample (,y)( x,y). Unfortunately, this specific per-step formulation creates a functional coupling between the client and server parameters. This dependency constrains the system to a unified optimization paradigm, enforcing that both clients and the server simultaneously employ either BP or ZO methods. To decouple the optimization landscapes of the server and client, we use the variable lifting (Carreira-Perpinan & Wang, 2014; Wang & Benning, 2023) technique. Specifically, we treat the activations z as an auxiliary variable and reformulate the objective as ℓ (fs(;s),y) (f_s( z; θ_s),y) (5) s.t. =fc(;c). z=f_c( x; θ_c). For this constrained problem, we construct the Lagrangian function ℒλL_λ by introducing a Lagrange multiplier λ ℒλ(s,c)=ℓ(fs(;s),y)+⊤(fc(;c)−).L_λ( θ_s, θ_c)= (f_s( z; θ_s),y)+ λ (f_c( x; θ_c)- z). (6) This formulation allows us to decompose the global objective into two sub-problems as follows. Server-Side Objective (ℒsL_s). Focusing on terms involving s θ_s, the server’s objective can be rearranged as ℒs(s)=ℓ(fs(;s),y).L_s( θ_s)= (f_s( z; θ_s),y). (7) Client-Side Objective (ℒcL_c). Focusing on terms involving c θ_c, the client’s objective is guided by the multiplier, which is given by ℒc(c)=⊤fc(;c).L_c( θ_c)= λ f_c( x; θ_c). (8) Derivation of State Variables. To solve this, we need to determine the values of z and λ at each communication round t. First, to enforce consistency, we anchor the auxiliary variable z to the current client forward output t=fc(;ct). z^t=f_c( x; θ_c^t). (9) Second, the optimal Lagrange multiplier t λ^t is determined by the stationarity condition ∇ℒλ= _ zL_λ=0 from (6) ∇ℓ(fs(;st),y)−t=. _ z (f_s( z; θ_s^t),y )- λ^t=0. (10) Rearranging the terms and substitute z with t z^t yields: t=∇ℓ(fs(t;st),y). λ^t= _ z (f_s( z^t; θ_s^t),y). (11) This confirms that the Lagrange multiplier t λ^t corresponds exactly to the standard gradient of the loss with respect to the input activations. Figure 1: Overview of the HO-SFL training loop. Each selected client m computes an activation mt=fc(m;ct) z_m^t=f_c( x_m; θ_c^t) and sends (mt,ym)( z_m^t,y_m) to the server. The server performs BP to update s θ_s and returns the activation-gradient feedback mt=∇mℓ λ_m^t= _ z_m . In parallel, the client runs P ZO perturbation forward passes ~m,pt=fc(m;ct+μpt) z_m,p^t=f_c( x_m; θ_c^t+μ u_p^t) and computes scalar projections vm,pt=mt⊤(~m,pt−mt)v_m,p^t= λ_m^t ( z_m,p^t- z_m^t). The server then aggregates these scalars into v¯pt=1K∑m∈tvm,pt v_p^t= 1K _m _tv_m,p^t, which are broadcast back for clients to construct ^ct=1Pμ∑p=1Pv¯ptpt g_c^t= 1Pμ _p=1^P v_p^t u_p^t and update c θ_c. Algorithm 1 Hybrid-Order Split Federated Learning 1: Initialize: Model parameters c0 θ_c^0 and s0 θ_s^0; Learning rate η; Perturbation count P; Smoothing parameter μ 2: Allocate: Timestamp map tm′m=1M←0\t _m\_m=1^M← 0; History buffers for seeds spτp=1Pτ\\s_p^τ\_p=1^P\_τ and scalars v¯pτp=1Pτ\\ v_p^τ\_p=1^P\_τ 3: for round t=0,…,T−1t=0,…,T-1 do 4: Server samples set tS_t; broadcast seeds sptp=1P\s_p^t\_p=1^P 5: // Phase 1: Synchronization & Forward 6: for each client m∈tm _t in parallel do 7: tm′←t _m← ClientSync(t,tm′)(t,t _m) 8: Sample ξm=(m,ym) _m=( x_m,y_m) 9: Compute activation mt←fc(m;ct) z_m^t← f_c( x_m; θ_c^t) 10: Client sends (mt,ym)( z_m^t,y_m) to Server 11: end for 12: // Phase 2: Server First-Order Update 13: for each client m∈tm _t do 14: Compute y^m←fs(mt;st) y_m← f_s( z_m^t; θ_s^t) and loss ℓ 15: Compute gradients s,mt←∇sℓ g_s,m^t← _ θ_s 16: Compute feedback mt←∇mtℓ λ_m^t← _ z_m^t 17: end for 18: Server update st+1←st−η1K∑m∈ts,mt θ_s^t+1← θ_s^t-η 1K _m _t g_s,m^t 19: Server send mt λ_m^t to client m for m∈tm _t 20: // Phase 3: Client Zeroth-Order Projection 21: for each client m∈tm _t in parallel do 22: vm,ptp=1P←\v_m,p^t\_p=1^P← ZOScalar(θct,λmt,sptp=1P)( θ_c^t, λ_m^t,\s_p^t\_p=1^P) 23: Send scalars vm,ptp=1P\v_m,p^t\_p=1^P to Server 24: end for 25: // Phase 4: Client Aggregation & Update 26: Server broadcasts v¯pt←1K∑m∈tvm,ptp=1P\ v_p^t← 1K _m _tv_m,p^t\_p=1^P 27: Clients compute ^ct←1Pμ∑p=1Pv¯ptpt g_c^t← 1Pμ _p=1^P v_p^t u_p^t 28: Clients update ct+1←ct−η^ct θ_c^t+1← θ_c^t-η g_c^t 29: end for 3.3 The Proposed HO-SFL Framework Based on the analysis above, we propose HO-SFL. The core training loop and data flow are illustrated in Figure 1, while the detailed procedural steps are summarized in Algorithm 1. Specifically, the server focuses on ℒsL_s, updating the server-side parameters s θ_s via BP while simultaneously deriving the Lagrange multiplier mt λ_m^t as consistency feedback. In parallel, each of the K sampled clients leverages the feedback to optimize the client-side objective ℒcL_c, through a memory-efficient ZO estimator. Furthermore, by leveraging shared random seeds, clients enable dimension-free aggregation by transmitting only scalars. In the following, we detail the specific steps of HO-SFL during each iteration. Client Sampling and Forward. At the beginning of communication round t, the server samples a subset t⊆1,…,MS_t \1,…,M\ with |t|=K|S_t|=K. Each sampled client m performs the forward pass on a data sample ξm=(m,ym) _m=( x_m,y_m) using its current parameters ct θ_c^t to generate mt z_m^t as defined in Eq. (1). The client transmits mt z_m^t and label ymy_m to the server. Algorithm 2 Function: ZO Scalar 1: Function ZOScalar(θc,λ,spp=1P)( θ_c, λ,\s_p\_p=1^P): 2: for p=1,…,Pp=1,…,P do 3: Generate perturbation p∼(,) u_p (0,I) from seed sps_p 4: Perturbed forward: ~p←fc(;c+μp) z_p← f_c( x; θ_c+μ u_p) 5: Compute projection: vp←⊤(~p−)v_p← λ ( z_p- z) ⊳ Eq. 17 6: end for 7: Return Scalars vpp=1P\v_p\_p=1^P Algorithm 3 Function: Client Synchronization 1: Function ClientSync(t,t′)(t,t ): 2: if t′<t <t then 3: Fetch history (spτ,v¯pτ)p=1P\(s_p^τ, v_p^τ)\_p=1^P for τ∈t′,…,t−1τ∈\t ,…,t-1\ 4: for missed round τ=t′,…,t−1τ=t ,…,t-1 do 5: Regenerate pτ←PRG(spτ) u_p^τ (s_p^τ) for p∈1,…,Pp∈\1,…,P\ 6: Reconstruct grad ^cτ←1Pμ∑p=1Pv¯pτpτ g_c^τ← 1Pμ _p=1^P v_p^τ u_p^τ 7: cτ+1←cτ−η^cτ θ_c^τ+1← θ_c^τ-η g_c^τ ⊳ Sequential Catch-up 8: end for 9: end if 10: Return t ⊳ Update timestamp Server-Side Backpropagation. Upon receiving the data (mt,ymt)m∈t\( z_m^t,y_m^t)\_m _t, the server completes the forward propagation and computes the loss using Eq. (2). Then, the server performs standard backpropagation to compute s,mt g_s,m^t =∇sℓ(fs(mt;st),ym),∀m∈t, = _ θ_s (f_s( z_m^t; θ_s^t),y_m), ∀ m _t, (12) mt λ_m^t =∇mtℓ(fs(mt;st),ymt),∀m∈t. = _ z_m^t (f_s( z_m^t; θ_s^t),y_m^t ), ∀ m _t. (13) Note that s,mt g_s,m^t and mt λ_m^t are obtained from a regular BP process, and no extra computing cost is incurred. The server updates its parameter by st+1=st−η(1K∑m∈ts,mt), θ_s^t+1= θ_s^t-η( 1K _m _t g_s,m^t), (14) and transmit gradient mt λ_m^t back to each client. Client-Side Zeroth-Order Projection. To update c θ_c without BP, we employ a ZO estimator to minimize the client-side objective ℒcL_c. First, using shared random seeds sptp=1P\s_p^t\_p=1^P, clients generate Gaussian perturbation vectors: pt∼(,dc),p∈1,…,P. u_p^t ( 0, I_d_c), p∈\1,…,P\. (15) Second, each client computes perturbed activations: ~m,pt=fc(mt;ct+μpt),p∈1,…,P, z_m,p^t=f_c( x_m^t; θ_c^t+μ u_p^t), p∈\1,…,P\, (16) where μ is a smoothing parameter. Third, each client computes a scalar projection vm,ptv_m,p^t representing the finite difference of the local objective function ℒcL_c. Recall that the local objective is defined as ℒc(c)=⊤fc(;c)L_c( θ_c)= λ f_c( x; θ_c). Setting the unperturbed activation mt z_m^t as the anchor variable z, the difference is calculated as: vm,pt=ℒc(ct+μpt)−ℒc(ct)=mt(~m,pt−mt)⊤. splitv_m,p^t&=L_c( θ_c^t+μ u_p^t)-L_c( θ_c^t)\\ &= λ_m^t ( z_m,p^t- z_m^t). split (17) This term estimates the directional derivative of the client’s output projected onto the loss gradient direction provided by the server. It is important to note that no uplink communication costs are incurred during the above process. Global Gradient Reconstruction. Each sampled client transmits vm,ptp=1Pv_m,p^t_p=1^P to the server, which aggregates them across clients: v¯pt=1K∑m∈tvm,pt,p∈1,…,P, v_p^t= 1K _m _tv_m,p^t, p∈\1,…,P\, (18) and broadcasts v¯ptp=1P\ v_p^t\_p=1^P. Each client constructs: ^ct=1Pμ∑p=1Pv¯ptpt, g_c^t= 1Pμ _p=1^P v_p^t u_p^t, (19) followed by the client-side update: ct+1=ct−η^ct. θ_c^t+1= θ_c^t-η g_c^t. (20) It is worth noting that clients only need to upload the scalars vm,ptp=1P\v_m,p^t\_p=1^P for aggregation. In contrast to standard SFL, which mandates the transmission of the full parameter vector scaling with dimension dcd_c, our approach achieves dimension-free uplink communication, thereby significantly reducing the bandwidth cost. Client Synchronization Strategy To accommodate stateless clients that do not participate in every round, HO-SFL incorporates a lightweight synchronization mechanism as outlined in Algorithm 3. Specifically, for a client with a stale model ct′ θ_c^t from round t′<t <t, it requests the server’s history of broadcast tuples (spτ,v¯pτ)p=1P\(s_p^τ, v_p^τ)\_p=1^P corresponding to the missed rounds τ∈t′,…,t−1τ∈\t ,…,t-1\. The client then iteratively catches up to the current global state by first regenerating the historical perturbation vectors from the random seeds: pτ=PRG(spτ). u_p^τ=PRG(s_p^τ). (21) It then combines them with the scalar projections to reconstruct the gradient estimates: ^cτ=1Pμ∑p=1Pv¯pτpτ. g_c^τ= 1Pμ _p=1^P v_p^τ u_p^τ. (22) These reconstructed gradients are then used to sequentially apply the historical updates: cτ+1←cτ−η^cτ, θ_c^τ+1← θ_c^τ-η g_c^τ, (23) aligning the local model with ct θ_c^t without the need to download high-dimensional model parameters. 3.4 Discussions: Efficiency of HO-SFL (a) Standard SFL Timeline (b) HO-SFL Timeline Figure 2: Comparison of system efficiency. (a) Standard SFL incurs significant idle time on the client side waiting for gradients. (b) HO-SFL effectively masks the computational cost of multiple client-side zeroth-order perturbations by overlapping them with the server’s backpropagation and communication processes. While the primary motivation of HO-SFL is to overcome the memory barrier of clients, we also highlight two extra advantages of HO-SFL concerning communication and latency efficiency (as illustrated in Figure 2). Dimension-Free Aggregation. Regarding client-side aggregation, HO-SFL significantly alleviates the communication bottleneck. In standard SFL, clients transmit high-dimensional model parameters whose size scales linearly with the client-side model dimension dcd_c, resulting in a communication complexity of (dc)O(d_c). In contrast, HO-SFL enables dimension-free aggregation by requiring each client to transmit only P scalars vm,pp=1P\v_m,p\_p=1^P per communication round, where P is independent of the model architecture and typically satisfies P≪dcP d_c. Thus, the communication cost for model aggregation is reduced to (P)O(P). Latency Hiding via Decoupling. In standard SFL, clients must remain idle while waiting for the activation gradient from the server. Benefiting from the decoupling strategy, HO-SFL can effectively overlap both computation and communication: the client-side perturbation can be overlapped with server-side forward/backward as well as the activation uplink and gradient downlink processes, thereby potentially shortening end-to-end latency. We conducted a simulation analysis based on the LLaMA-3.2-1B (Grattafiori et al., 2024) under realistic edge computing conditions and validated the feasibility of latency hiding in Appendix C.1. 4 Convergence Analysis In this section, we establish the theoretical guarantees for the proposed HO-SFL framework. Unlike standard SFL, HO-SFL involves a hybrid optimization landscape where the server performs first-order updates while clients perform ZO updates. We analyze the convergence properties of HO-SFL under non-convex settings and quantify the impact of the zeroth-order approximation error on the global convergence rate. Detailed proofs are provided in Appendix A. 4.1 Assumptions We adopt standard assumptions widely used in the analysis of non-convex federated optimization (Han et al., 2024). Assumption 4.1 (Smoothness). The global objective ℒ()L( θ) is β-smooth, i.e., for any 1,2 θ_1, θ_2, ‖∇ℒ(1)−∇ℒ(2)‖≤β‖1−2‖.\| ( θ_1)- ( θ_2)\|≤β\| θ_1- θ_2\|. Assumption 4.2 (Unbiased Stochastic Gradients, Bounded Variance, and Heterogeneity). Client m draws an independent data sample ξm∼m _m _m and forms a stochastic gradient m:=∇ℓ(;ξm) g_m:= _ θ ( θ; _m). We assume 1. Unbiasedness: ξm[m]=∇ℒm()E_ _m[ g_m]= _m( θ). 2. Bounded Variance: ξm[‖m−∇ℒm()‖2]≤σ2E_ _m\! [\| g_m- _m( θ)\|^2 ]≤σ^2. 3. Bounded Gradient Dissimilarity: ‖∇ℒm()−∇ℒ()‖2≤κ2\| _m( θ)- ( θ)\|^2≤κ^2. 4.2 Properties of the Hybrid Estimator A core challenge in analyzing HO-SFL is the bias and variance introduced by the zeroth-order gradient estimator on the client side. Based on the detailed derivations in Lemma A.2 and Lemma A.5, we characterize the behavior of the client-side estimator ^ct g_c^t. Proposition 4.3 (Bias and Variance Decomposition). Let Γ be the regularity bound on the gradient magnitudes and Hessian spectral norms. The client-side zeroth-order estimator ^ct g_c^t exhibits the following properties: 1. Bias Control (from Lemma A.6): The estimator is biased due to the curvature of the loss landscape. The expected deviation is bounded by the smoothing parameter μ and the client model dimension dcd_c: ‖t[^ct]−∇cℒ(t)‖2≤μ2Γ44(dc+3)3.\|E_t[ g_c^t]- _ θ_cL( θ^t)\|^2≤ μ^2 ^44(d_c+3)^3. (24) 2. Second Moment Bound (from Lemma A.4): The second moment of the aggregated client estimator ^ct g_c^t is bounded by the true gradient norm scaled by an expansion factor C1C_1, plus a composite variance term: t‖^ct‖2≤C1‖∇cℒ(t)‖2+Ωc,E_t\| g_c^t\|^2≤ C_1\| _ θ_cL( θ^t)\|^2+ _c, (25) where Ωc:=C1(σ2+κ2)+σZO2 _c:=C_1(σ^2+κ^2)+ _ZO^2 collects the combined client-side variance, including stochastic gradient noise (σ2σ^2), client heterogeneity (κ2κ^2), and zeroth-order estimation error (σZO2 _ZO^2). 4.3 Main Convergence Result Building on the properties of the hybrid estimator, we now establish the convergence guarantees of HO-SFL under the non-convex setting. Theorem 4.4 (Convergence Bound of HO-SFL). Suppose Assumptions 4.1 and 4.2 hold. Let the learning rate satisfy η≤12βC1η≤ 12β C_1. For any T≥1T≥ 1, the average squared gradient norm of HO-SFL is bounded by: 1T∑t=0T−1‖∇ℒ(t)‖2≤ 1T _t=0^T-1E \| ( θ^t) \|^2≤ 4ΔℒηT+2ηβ(Ωc+Ωs) 4 _Lη T+2ηβ ( _c+ _s ) (26) +μ2Γ42(dc+3)3, + μ^2 ^42(d_c+3)^3, where Δℒ=ℒ(0)−ℒ∗ _L=L( θ^0)-L^*, Ωs=σ2+κ2K _s= σ^2+κ^2K represents the server-side variance, and Ωc=C1(σ2+κ2)+σZO2 _c=C_1(σ^2+κ^2)+ _ZO^2 denotes the aggregated client-side variance. Theorem 4.4 provides a finite-time error bound on the gradient norm. The bound consists of the optimization error, the variance term and the bias term induced by the ZO approximation. To explicitly quantify the convergence rate with respect to the dimension dcd_c and the perturbation number P, we present the following corollary. Corollary 4.5 (Convergence Rate of HO-SFL). By choosing the learning rate η=Θ(PTdc)η= ( PTd_c ) and the smoothing parameter μ=((PT)−1/4dc−5/4)μ=O ((PT)^-1/4d_c^-5/4 ), the convergence rate of HO-SFL satisfies: min0≤t<T‖∇ℒ(t)‖2=(dcPT). _0≤ t<TE\| ( θ^t)\|^2=O ( d_cPT ). (27) Proof. Substituting the selected η into the bound in Eq. (26), the optimization error term (1ηT)O( 1η T) and the variance term (ηdcP)O(η d_cP) are balanced to scale as (dcPT)O( d_cPT). Simultaneously, the chosen scaling for μ ensures that the bias term (μ2dc3)O(μ^2d_c^3) does not exceed the order of the dominant term. Thus, the total convergence rate is dominated by (dcPT)O( d_cPT). ∎ Remark 4.6. Standard ZO optimization methods typically suffer from a linear dependence on the problem dimension, yielding a rate of (d/T)O( d/T) (where d is the full model dimension). In contrast, Corollary 4.5 highlights the critical advantage of dimensionality decoupling in HO-SFL. The dimension factor in the convergence rate depends only on the client-side dimension dcd_c, where dc≪d_c d in split learning settings. This effectively isolates the ZO optimization difficulty to a much smaller subspace, significantly mitigating the variance of the gradient estimator, which would otherwise be exacerbated by the massive server-side parameters. 5 Experiments In this section, we evaluate the performance of HO-SFL against state-of-the-art SFL baselines on vision and language tasks to demonstrate its fast convergence speed, communication efficiency, and substantial reductions in client memory usage. 5.1 Experimental Setup Datasets and Models. For vision tasks, we utilize CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2009). We evaluate under both IID and Non-IID settings, where Non-IID partitions are generated using a Dirichlet distribution with α=1α=1 (Hsu et al., 2019). The model is a pre-trained ResNet-18 (He et al., 2016) and split between the 2nd and 3rd residual blocks. For language tasks, we employ three pre-trained causal language models: OPT-125M (Zhang et al., 2022), Gemma-3-270M (Team et al., 2025), and LLaMA-3.2-1B (Grattafiori et al., 2024). OPT-125M, Gemma-3-270M, and LLaMA-3.2-1B contain 12, 20, and 18 transformer layers, respectively; unless otherwise specified, we place the first 3, 5, and 5 layers on the client side. These models are evaluated on the GLUE benchmark (Wang et al., 2018) (SST-2, RTE, WSC) using LoRA (Hu et al., 2022). We use a larger-scale partial-participation setup for vision tasks (100 total clients, 10 sampled per round) and a smaller-scale setup for LLM tasks (10 total clients, 3 sampled per round). Baselines and Fairness Criterion. We benchmark HO-SFL against multiple methods categorized by their optimization strategy. The first-order baselines include SFL (Thapa et al., 2022), SplitLoRA (Lin et al., 2024), and FSL-SAGE (Nair et al., 2025), while the zeroth-order baselines comprise ZO-SFL and MU-SplitFed (Liang et al., 2025). Unless otherwise specified, we set the number of perturbations P in HO-SFL to 5 for vision tasks and 2 for language tasks, and uniformly set μ to 1×10−31× 10^-3. Crucially, since HO-SFL and MU-SplitFed perform aggregation at every step, whereas other methods synchronize after multiple local updates, comparisons based on communication rounds are inherently misleading. To ensure a rigorous evaluation, we adopt proceeded samples as the unified metric, terminating training when the aggregate number of processed samples reaches 160k for vision tasks and 80k for language tasks. Detailed hyperparameter configurations, specific baseline introduction, and a comprehensive discussion on the fairness of the metric are provided in Appendix B. Figure 3: Validation accuracy convergence on CIFAR-10 under IID (left) and Non-IID (right) settings. Solid lines denote the mean performance, and shaded regions represent the standard deviation across 10 independent random seeds. 5.2 Performance Analysis Vision Tasks. Figure 3 visualizes the validation accuracy against processed samples on CIFAR-10. In the IID setting, HO-SFL exhibits only a marginal performance gap compared to SFL, demonstrating that HO-SFL effectively mitigates the convergence slowdown associated with ZO estimation. However, in Non-IID setting, HO-SFL demonstrates superior performance, even outperforming the first-order baseline (i.e., SFL). Unlike other methods that suffer from severe client drift due to infrequent aggregation of high-dimensional models, HO-SFL executes dimension-free aggregation at every step, effectively mimicking centralized training. Furthermore, pure zeroth-order approaches like MU-SplitFed and ZO-SFL struggle to converge within the limited sample budget, highlighting the efficiency of our proposed method. Due to space constraints, the CIFAR-100 convergence curves and the test accuracies with standard deviations are provided in Appendix 2 Language Tasks. Table 1 presents the fine-tuning results on GLUE tasks, demonstrating that HO-SFL maintains highly competitive performance against SplitLoRA (the first-order baseline) despite relying on zeroth-order client-side updates. Across varying model architectures, our method exhibits remarkable robustness; for instance, HO-SFL achieves a comparable 93.2% accuracy on SST2 and notably surpasses SplitLoRA on the RTE task when fine-tuning the LLaMA-3.2-1B. In contrast, the standard ZO-SFL baseline fails to converge effectively on most tasks, which underscores the necessity and effectiveness of our hybrid formulation in stabilizing convergence. Table 1: LLM Fine-tuning Accuracy (%) on GLUE tasks. Values in boldface indicate the highest accuracy. Model Task SplitLoRA ZO-SFL HO-SFL OPT (125M) SST2 87.5 52.8 87.6 WSC 64.4 36.5 60.6 RTE 57.8 52.0 59.2 Gemma-3 (270M) SST2 90.3 51.8 90.8 WSC 61.5 36.5 62.5 RTE 59.6 54.2 65.0 LLaMA-3.2 (1B) SST2 94.4 61.5 93.9 WSC 61.5 36.5 61.5 RTE 70.0 49.1 73.3 5.3 Communication and Memory Efficiency Communication Analysis. We analyze the communication cost breakdown in Figure 4(a) recorded in the CIFAR-10 IID experiment. Overall, HO-SFL achieves the lowest communication cost among the compared methods (2.44 GB), substantially lower than SFL (4.20 GB) and ZO-SFL (4.20 GB), and also lower than FSL-SAGE (3.71 GB). The savings mainly come from avoiding high-dimensional model transmission during aggregation, whereas MU-SplitFed incurs a massive overhead due to transmitting full model updates at every step. For readability, Figure 4(a) omits the scalar/seed components (e.g., HO-SFL seeds/scalars and the scalar traffic in ZO-SFL and Mu-SplitFed), each contributing less than 1 MB in total; see Appendix C.5 for the exact numbers. (a) Breakdown of communication traffic. (b) Client-side peak memory consumption Figure 4: Communication and memory profiling. Memory Consumption. We profiled the peak client-side memory usage for fine-tuning LLaMA-3.2-1B with LoRA on the SST-2 task, as illustrated in Figure 4(b). In this profiling setup, the client-side split contains 8 transformer layers. While full local training demands a prohibitive 20.81 GB that far exceeds the capacity of typical edge devices, even standard SFL imposes significant pressure by consuming 8.78 GB. In contrast, HO-SFL dramatically reduces this footprint to 4.14 GB. This is strikingly close to inference( 4.01GB), confirming that HO-SFL effectively eliminates the memory bottleneck of BP. Detailed information regarding the memory measurement methodology and device specifications is provided in Appendix C.6. 5.4 Ablation Studies Due to space constraints, we present detailed ablation studies in Appendix C.3 and Appendix C.4. Experiments on LLaMA-3.2-1B varying the client-side transformer layers from 2 to 8 confirm our theoretical analysis in Corollary 4.5. While increasing the client-side dimension dcd_c leads to a marginal reduction in convergence speed due to the dimension-dependent variance of the ZO estimator, HO-SFL exhibits strong robustness, effectively mitigating the severe slowdown typically observed in pure ZO methods. Results on ResNet-18 confirm that increasing the perturbation count P yields monotonic performance improvements. Meanwhile, the smoothing parameter μ requires a trade-off: values that are too large increase the approximation bias (Proposition 4.3), while values that are too small suffer from numerical precision issues (Jongeneel et al., 2024). 6 Conclusion In this paper, we propose Hybrid-Order Split Federated Learning, a novel framework that resolves the tension between memory constraints and convergence speed when fine-tuning large models on edge devices. By reformulating the split learning process through a Lagrangian perspective, HO-SFL decouples optimization into server-side first-order and client-side zeroth-order updates. Our theoretical analysis shows it mitigates the dimension-dependent convergence slowdown inherent to pure zeroth-order methods. Empirically, HO-SFL achieves convergence performance comparable to fully first-order baselines, while reducing client memory consumption to inference levels and minimizing communication overhead via dimension-free aggregation. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. References Ajalloeian et al. (2019) Ajalloeian, A., Jaggi, M., and Learning, M. Stochastic zeroth-order optimisation algorithms with variance reduction. Technical Report EPFL, 2019. Carreira-Perpinan & Wang (2014) Carreira-Perpinan, M. and Wang, W. Distributed optimization of deeply nested systems. In Artificial Intelligence and Statistics, p. 10–19. PMLR, 2014. Chen et al. (2016) Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016. Dettmers et al. (2023) Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115, 2023. Duchi et al. (2015) Duchi, J. C., Jordan, M. I., Wainwright, M. J., and Wibisono, A. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5):2788–2806, 2015. Fang et al. (2022) Fang, W., Yu, Z., Jiang, Y., Shi, Y., Jones, C. N., and Zhou, Y. Communication-efficient stochastic zeroth-order optimization for federated learning. IEEE Transactions on Signal Processing, 70:5058–5073, 2022. Grattafiori et al. (2024) Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Gupta & Raskar (2018) Gupta, O. and Raskar, R. Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications, 116:1–8, 2018. Han et al. (2024) Han, P., Huang, C., Tian, G., Tang, M., and Liu, X. Convergence analysis of split federated learning on heterogeneous data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=ud0RBkdBfE. He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 770–778, 2016. Hikima et al. (2025) Hikima, Y., Sawada, H., and Fujino, A. Guided zeroth-order methods for stochastic non-convex problems with decision-dependent distributions. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=cRmuEY7jhb. Hsu et al. (2019) Hsu, T.-M. H., Qi, H., and Brown, M. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019. Hu et al. (2022) Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. Jongeneel et al. (2024) Jongeneel, W., Yue, M.-C., and Kuhn, D. Small errors in random zeroth-order optimization are imaginary. SIAM Journal on Optimization, 34(3):2638–2670, 2024. Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009. Li et al. (2025) Li, Z., Ying, B., Liu, Z., Dong, C., and Yang, H. Achieving dimension-free communication in federated learning via zeroth-order optimization. In Yue, Y., Garg, A., Peng, N., Sha, F., and Yu, R. (eds.), International Conference on Learning Representations, volume 2025, p. 23769–23803, 2025. Liang et al. (2025) Liang, D., Zhang, J., Chen, E., Li, Z., Li, R., and Yang, H. Towards straggler-resilient split federated learning: An unbalanced update approach. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=MOAmJKj2Uc. Lim et al. (2020) Lim, W. Y. B., Luong, N. C., Hoang, D. T., Jiao, Y., Liang, Y.-C., Yang, Q., Niyato, D., and Miao, C. Federated learning in mobile edge networks: A comprehensive survey. IEEE communications surveys & tutorials, 22(3):2031–2063, 2020. Lin et al. (2024) Lin, Z., Hu, X., Zhang, Y., Chen, Z., Fang, Z., Chen, X., Li, A., Vepakomma, P., and Gao, Y. Splitlora: A split parameter-efficient fine-tuning framework for large language models. arXiv preprint arXiv:2407.00952, 2024. Liu et al. (2020) Liu, S., Chen, P.-Y., Kailkhura, B., Zhang, G., Hero I, A. O., and Varshney, P. K. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine, 37(5):43–54, 2020. Lv et al. (2024) Lv, K., Yang, Y., Liu, T., Guo, Q., and Qiu, X. Full parameter fine-tuning for large language models with limited resources. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8187–8198, 2024. Ma & Huang (2025a) Ma, S. and Huang, H. On the optimal construction of unbiased gradient estimators for zeroth-order optimization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a. URL https://openreview.net/forum?id=rVT1GK60Nt. Ma & Huang (2025b) Ma, S. and Huang, H. Revisiting zeroth-order optimization: Minimum-variance two-point estimators and directionally aligned perturbations. In The Thirteenth International Conference on Learning Representations, 2025b. Malladi et al. (2023) Malladi, S., Gao, T., Nichani, E., Damian, A., Lee, J. D., Chen, D., and Arora, S. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038–53075, 2023. McMahan et al. (2017) McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, p. 1273–1282. PMLR, 2017. Nair et al. (2025) Nair, S., Lin, M., Ju, P., Talebi, A., Bentley, E. S., and Liu, J. FSL-SAGE: Accelerating federated split learning via smashed activation gradient estimation. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=HnwcrtoDd4. Narayanan et al. (2020) Narayanan, A., Ramadan, E., Carpenter, J., Liu, Q., Liu, Y., Qian, F., and Zhang, Z.-L. A first look at commercial 5g performance on smartphones. In Proceedings of The Web Conference 2020, p. 894–905, 2020. Nesterov & Spokoiny (2017) Nesterov, Y. and Spokoiny, V. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527–566, 2017. Oh et al. (2025) Oh, Y., Lee, J., Brinton, C. G., and Jeon, Y.-S. Communication-efficient split learning via adaptive feature-wise compression. IEEE Transactions on Neural Networks and Learning Systems, 2025. Peng et al. (2024) Peng, D., Fu, Z., and Wang, J. Pocketllm: Enabling on-device fine-tuning for personalized llms. arXiv preprint arXiv:2407.01031, 2024. Sani et al. (2025) Sani, L., Iacob, A., Cao, Z., Lee, R., Marino, B., Gao, Y., Zhao, W., Cai, D., Li, Z., Qiu, X., et al. Photon: Federated llm pre-training. Proceedings of Machine Learning and Systems, 7, 2025. Shamir (2017) Shamir, O. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research, 18(52):1–11, 2017. Spall (2002) Spall, J. C. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37(3):332–341, 2002. Team et al. (2025) Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. Thapa et al. (2022) Thapa, C., Arachchige, P. C. M., Camtepe, S., and Sun, L. Splitfed: When federated learning meets split learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, p. 8485–8493, 2022. Vepakomma et al. (2018) Vepakomma, P., Gupta, O., Swedish, T., and Raskar, R. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564, 2018. Wang et al. (2018) Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, p. 353–355, 2018. Wang & Benning (2023) Wang, X. and Benning, M. Lifted bregman training of neural networks. Journal of Machine Learning Research, 24(232):1–51, 2023. Wang et al. (2024) Wang, Z., Shen, Z., He, Y., Sun, G., Wang, H., Lyu, L., and Li, A. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. Advances in Neural Information Processing Systems, 37:22513–22533, 2024. Wu et al. (2023) Wu, W., Li, M., Qu, K., Zhou, C., Shen, X., Zhuang, W., Li, X., and Shi, W. Split learning over wireless networks: Parallel design and resource management. IEEE Journal on Selected Areas in Communications, 41(4):1051–1066, 2023. Yang & Liu (2025) Yang, J. and Liu, Y. Gas: Generative activation-aided asynchronous split federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, p. 21956–21964, 2025. Yi et al. (2022) Yi, X., Zhang, S., Yang, T., and Johansson, K. H. Zeroth-order algorithms for stochastic distributed nonconvex optimization. Automatica, 142:110353, 2022. Zhang et al. (2022) Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. Appendix A Proofs A.1 Notions Let ℱtF_t be the filtration generated by all randomness up to the end of round t−1t-1, i.e., ℱt:=σ(0,(τ,τ,τ)τ=0t−1).F_t:=σ\! ( θ^0,\(S_τ, ξ^τ,U^τ)\_τ=0^t-1 ). In particular, t θ^t is ℱtF_t-measurable. We use t[⋅]:=[⋅∣ℱt]E_t[·]:=E[· _t] to denote conditional expectation given the history. At round t, the algorithm involves three mutually independent sources of randomness: the subset of sampled clients tS_t, the data samples t:=ξmtm∈t ξ^t:=\ _m^t\_m _t drawn by the selected clients, and the random perturbation vectors t:=ptp=1PU^t:=\ u_p^t\_p=1^P. To maintain notational conciseness, we define any expectation with a subscript of round t variables to implicitly condition on ℱtF_t. For instance, t[⋅]:=t[⋅∣ℱt]E_ ξ^t[·]:=E_ ξ^t[· _t] and t[⋅]:=t[⋅∣ℱt]E_S_t[·]:=E_S_t[· _t]. For each selected client m∈tm _t, the forward pass produces smashed data mt:=fc(xmt;ct)∈ℝD z_m^t:=f_c(x_m^t; θ_c^t) ^D and perturbed smashed data ~m,pt:=fc(xmt;ct+μpt) z_m,p^t:=f_c(x_m^t; θ_c^t+μ u_p^t). The server returns the activation gradient (dual variable) mt:=∇mtℓ(t;ξmt)∈ℝD λ_m^t:= _ z_m^t ( θ^t; _m^t) ^D. We also denote the local stochastic gradients at round t as c,mt:=∇cℓ(t;ξmt),s,mt:=∇sℓ(t;ξmt), g_c,m^t:= _ θ_c ( θ^t; _m^t), g_s,m^t:= _ θ_s ( θ^t; _m^t), (28) so mt λ_m^t, c,mt g_c,m^t, and s,mt g_s,m^t are measurable with respect to σ(ℱt,t,ξmt)σ(F_t,S_t, _m^t). We define their aggregated versions as ^ct:=1K∑m∈t^c,mt,st:=1K∑m∈ts,mt. g_c^t:= 1K _m _t g_c,m^t, g_s^t:= 1K _m _t g_s,m^t. (29) We denote the aggregated gradients at round t as st g_s^t (server-side) and ^ct g_c^t (client-side), such that the global update rule is t+1=t−η[^ct;st] θ^t+1= θ^t-η[ g_c^t; g_s^t]. For brevity, we denote the full parameter vector by :=[c;s] θ:=[ θ_c;\, θ_s]. The global objective function is given by: ℒ():=1M∑m=1Mℒm(),ℒm():=ξm∼m[ℓ(;ξm)],L( θ):= 1M _m=1^ML_m( θ), _m( θ):=E_ _m _m\! [ ( θ; _m) ], (30) where ℓ(;ξm) ( θ; _m) is the sample-wise loss for a single data sample ξm=(xm,ym) _m=(x_m,y_m) drawn from the local distribution mD_m. To facilitate the analysis of the zeroth-order estimator, we formally define the regularity bound Γ , which encapsulates the upper bounds on the dual variables and the derivatives of the client network. Definition A.1 (Regularity Bound Γ ). Let ℋm,pt:=∇c2fc(xmt;ct+τm,ptμpt)H_m,p^t:= _ θ_c^2f_c(x_m^t; θ_c^t+ _m,p^tμ u_p^t) denote the Hessian of the client model evaluated at the intermediate point determined by Taylor’s theorem (where τm,pt∈(0,1) _m,p^t∈(0,1)). We define the regularity bound Γt _t at round t as: Γt:=maxmaxm∈t‖mt‖,maxm∈t‖∇cfc(xmt;ct)‖op,maxm∈t,p∈1,…,P‖ℋm,pt‖op, _t~:=~ \ _m _t\| λ_m^t\|,~ _m _t\| _ θ_cf_c(x_m^t; θ_c^t)\|_op,~ _m _t,\,p∈\1,…,P\\|H_m,p^t\|_op \, (31) where ∥⋅∥op\|·\|_op denotes the operator norm. We further define the global bound Γ:=sup0≤t≤T−1Γt := _0≤ t≤ T-1 _t. A.2 Assumptions We adopt standard assumptions widely used in the analysis of non-convex federated optimization and zeroth-order methods. See 4.1 See 4.2 A.3 Proof of Lemma A.2: Bound on Local ZO Estimator Lemma A.2 (Conditional Bound on Local ZO Estimator). Condition on (ℱt,t,ξmt)(F_t,S_t, _m^t), so that ct θ_c^t, mt λ_m^t, and c,mt g_c,m^t are fixed. Let ^c,mt=1Pμ∑p=1Pvm,ptpt,pt∼(,dc) i.i.d. g_c,m^t= 1Pμ _p=1^Pv_m,p^t u_p^t, u_p^t ( 0, I_d_c) i.i.d. (32) where vm,pt:=(mt)⊤(~m,pt−mt),~m,pt:=fc(xmt;ct+μpt),mt:=fc(xmt;ct).v_m,p^t:=( λ_m^t) \! ( z_m,p^t- z_m^t ), z_m,p^t:=f_c(x_m^t; θ_c^t+μ u_p^t),~ z_m^t:=f_c(x_m^t; θ_c^t). (33) Let c,mt:=∇cℓ(;ξmt) g_c,m^t:= _ θ_c ( θ^t; _m^t). Using the regularity bound Γ from Definition A.1, we have: t[‖^c,mt‖2]≤C1‖c,mt‖2+σZO2,E_U^t\! [ \| g_c,m^t \|^2 ]≤ C_1 \| g_c,m^t \|^2+ _ZO^2, (34) with C1:=2(1+dc+1P),σZO2:=μ22dc(dc+2)(dc+4)Γ4.C_1:=2 (1+ d_c+1P ), _ZO^2:= μ^22\,d_c(d_c+2)(d_c+4)\, ^4. (35) Proof. By Taylor’s theorem with Lagrange remainder, there exists some τm,pt∈(0,1) _m,p^t∈(0,1) such that: Δm,pt z_m,p^t =~m,pt−mt = z_m,p^t- z_m^t =fc(xmt;ct+μpt)−fc(xmt;ct) =f_c(x_m^t; θ_c^t+μ u_p^t)-f_c(x_m^t; θ_c^t) =(a)μmtpt+μ22ℋm,pt[pt,pt]⏟=:m,pt. (a)=μ\,J_m^t\, u_p^t+ μ^22\,H_m,p^t[ u_p^t, u_p^t]_=: R_m,p^t. (36) Here mt:=∇cfc(xmt;ct)∈ℝD×dcJ_m^t:= _ θ_cf_c(x_m^t; θ_c^t) ^D× d_c is the Jacobian matrix, and ℋm,pt:=∇c2fc(xmt;ct+τm,ptμpt)H_m,p^t:= _ θ_c^2f_c(x_m^t; θ_c^t+ _m,p^tμ u_p^t) is the Hessian tensor. ℋm,pt[⋅,⋅]H_m,p^t[·,·] denotes its bilinear action on the perturbation vectors. Using (36) in the definition of vm,ptv_m,p^t gives vm,pt v_m,p^t =(mt)⊤Δm,pt =( λ_m^t) z_m,p^t =μ(mt)⊤mtpt+(mt)⊤m,pt =μ\,( λ_m^t) J_m^t u_p^t+( λ_m^t) R_m,p^t =(b)μ(pt)⊤c,mt+(mt)⊤m,pt, (b)=μ\,( u_p^t) g_c,m^t+( λ_m^t) R_m,p^t, (37) where (b)(b) follows from the chain rule: c,mt=∇cℓ(t;ξmt)=(∇cfc(xmt;ct))⊤∇mtℓ(t;ξmt)=(mt)⊤mt. g_c,m^t~=~ _ θ_c ( θ^t; _m^t)~=~( _ θ_cf_c(x_m^t; θ_c^t)) _ z_m^t ( θ^t; _m^t)~=~(J_m^t) λ_m^t. (38) Substituting (37) into the definition of the gradient estimator ^c,mt g_c,m^t yields two distinct terms: ^c,mt g_c,m^t =1Pμ∑p=1Pvm,ptpt = 1Pμ _p=1^Pv_m,p^t u_p^t =1P∑p=1Ppt(pt)⊤c,mt⏟=:+1Pμ∑p=1P((mt)⊤m,pt)pt⏟=:. = 1P _p=1^P u_p^t( u_p^t) g_c,m^t_=:~ A+ 1Pμ _p=1^P (( λ_m^t) R_m,p^t ) u_p^t_=:~ B. (39) Using the inequality ‖+‖2≤2‖2+2‖2\| A+ B\|^2≤ 2\| A\|^2+2\| B\|^2, we have: ‖^c,mt‖2≤2‖2+2‖2.\| g_c,m^t\|^2≤ 2\| A\|^2+2\| B\|^2. (40) Bounding Term A A: Based on the fourth moment of Gaussian vectors, we have: t‖2=(1+dc+1P)‖c,mt‖2.E_U^t\| A\|^2= (1+ d_c+1P )\| g_c,m^t\|^2. (41) Bounding Term B B: From (36) and the definition of Γ , the remainder term is bounded by: ‖m,pt‖=μ22‖ℋm,pt[pt,pt]‖≤μ22Γ‖pt‖2.\| R_m,p^t\|= μ^22\|H_m,p^t[ u_p^t, u_p^t]\|≤ μ^22\, \,\| u_p^t\|^2. (42) Consequently, the scalar projection is bounded by: |(mt)⊤m,pt|≤‖mt‖‖m,pt‖≤Γ⋅μ22Γ‖pt‖2=μ22Γ2‖pt‖2.|( λ_m^t) R_m,p^t|≤\| λ_m^t\|\| R_m,p^t\|≤ · μ^22 \| u_p^t\|^2= μ^22 ^2\| u_p^t\|^2. (43) Using Jensen’s inequality on the sum over p, we have: t‖2 _U^t\| B\|^2 ≤t[1Pμ2∑p=1P((mt)⊤m,pt)2‖pt‖2] _U^t [ 1Pμ^2 _p=1^P (( λ_m^t) R_m,p^t )^2\| u_p^t\|^2 ] ≤1μ2[(μ22Γ2‖2)2‖2] ≤ 1μ^2\,E_ u\! [ ( μ^22 ^2\| u\|^2 )^2\| u\|^2 ] =μ24Γ4[‖6], = μ^24 ^4\,E_ u[\| u\|^6], (44) where ∼(,dc) u (0, I_d_c) and the sixth moment of u is given by [‖6]=dc(dc+2)(dc+4)E[\| u\|^6]=d_c(d_c+2)(d_c+4). Combining (40), (41), and (44), we have: t‖^c,mt‖2 _U^t\| g_c,m^t\|^2 ≤2(1+dc+1P)‖c,mt‖2+2(μ24Γ4dc(dc+2)(dc+4)) ≤ 2 (1+ d_c+1P )\| g_c,m^t\|^2+2 ( μ^24 ^4d_c(d_c+2)(d_c+4) ) =C1‖c,mt‖2+μ22dc(dc+2)(dc+4)Γ4. =C_1\| g_c,m^t\|^2+ μ^22d_c(d_c+2)(d_c+4) ^4. (45) This completes the proof. ∎ A.4 Proof of Lemma A.3 Lemma A.3 (Server-side Second Moment Bound). Let st=1K∑m∈ts,mt g_s^t= 1K _m _t g_s,m^t be the aggregated server gradient. Under Assumption 4.2, the second moment of the server update is bounded by: t‖st‖2≤‖∇sℒ(t)‖2+Ωs,E_t\| g_s^t\|^2≤\| _ θ_sL( θ^t)\|^2+ _s, (46) where Ωs:=σ2+κ2K _s:= σ^2+κ^2K represents the server-side variance. Proof. We apply the bias–variance decomposition: t‖st‖2 _t\| g_s^t\|^2 =‖t[st]‖2+t‖st−t[st]‖2 =\|E_t[ g_s^t]\|^2+E_t\| g_s^t-E_t[ g_s^t]\|^2 =(a)‖∇sℒ(t)‖2+t‖st−t[st]‖2. (a)=\| _ θ_sL( θ^t)\|^2+E_t\| g_s^t-E_t[ g_s^t]\|^2. where (a)(a) holds because t[st]=∇sℒ(t)E_t[ g_s^t]= _ θ_sL( θ^t) under Assumption 4.2. We decompose the second term into two components: the intra-client data noise and the inter-client sampling noise: st−∇sℒ(t)=1K∑m∈t(s,mt−∇sℒm(t))⏟Data Sampling Noise Δ1t+(1K∑m∈t∇sℒm(t)−∇sℒ(t))⏟Client Sampling Noise Δ2t. g_s^t- _ θ_sL( θ^t)= 1K _m _t( g_s,m^t- _ θ_sL_m( θ^t))_Data Sampling Noise _1^t+ ( 1K _m _t _ θ_sL_m( θ^t)- _ θ_sL( θ^t) )_Client Sampling Noise _2^t. (47) Expanding the squared norm, the cross-term vanishes because the expectation of Δ1t _1^t is zero: t[⟨Δ1t,Δ2t⟩]=t[⟨t[Δ1t|t],Δ2t⟩]=0.E_t[ _1^t, _2^t ]=E_S_t [ _ ξ^t[ _1^t|S_t], _2^t ]=0. (48) Thus, the variance decomposes into the sum of the individual variances: t‖st−∇sℒ(t)‖2=t‖Δ1t‖2+t‖Δ2t‖2.E_t\| g_s^t- _ θ_sL( θ^t)\|^2=E_t\| _1^t\|^2+E_S_t\| _2^t\|^2. (49) By decoupling the expectations and expanding the squared norm, we obtain: t‖Δ1t‖2 _t\| _1^t\|^2 =t[t‖1K∑m∈t(s,mt−∇sℒm(t))‖2] =E_S_t [E_ ξ^t \| 1K _m _t( g_s,m^t- _ θ_sL_m( θ^t)) \|^2 ] =(b)t[1K2∑m∈t‖s,mt−∇sℒm(t)‖2] (b)=E_S_t [ 1K^2 _m _tE_ ξ^t\| g_s,m^t- _ θ_sL_m( θ^t)\|^2 ] ≤(c)t[1K2∑m∈tσ2] (c)≤E_S_t [ 1K^2 _m _tσ^2 ] =σ2K, = σ^2K, where (b)(b) holds because the data sampling is independent across clients and the local stochastic gradients are unbiased, causing the cross-terms to vanish; (c)(c) uses the Bounded Variance property in Assumption 4.2. Next, we bound the client sampling noise term t‖Δ2t‖2E_S_t\| _2^t\|^2 following a similar procedure: t‖Δ2t‖2 _S_t\| _2^t\|^2 =t‖1K∑m∈t(∇sℒm(t)−∇sℒ(t))‖2 =E_S_t \| 1K _m _t( _ θ_sL_m( θ^t)- _ θ_sL( θ^t)) \|^2 =(d)1K2∑m∈t‖∇sℒm(t)−∇sℒ(t)‖2 (d)= 1K^2 _m _tE_S_t\| _ θ_sL_m( θ^t)- _ θ_sL( θ^t)\|^2 ≤(e)1K2∑m∈tκ2 (e)≤ 1K^2 _m _tκ^2 =κ2K, = κ^2K, where (d)(d) is due to the independent and uniform sampling of clients, which ensures t[∇sℒm(t)]=∇sℒ(t)E_S_t[ _ θ_sL_m( θ^t)]= _ θ_sL( θ^t) and eliminates the cross-terms; (e)(e) applies the Bounded Gradient Dissimilarity property in Assumption 4.2. Combining the squared bias and the variances derived above yields the final result: t‖st‖2≤‖∇sℒ(t)‖2+σ2+κ2K.E_t\| g_s^t\|^2≤\| _ θ_sL( θ^t)\|^2+ σ^2+κ^2K. (50) Defining Ωs:=σ2+κ2K _s:= σ^2+κ^2K completes the proof. ∎ A.5 Proof of Lemma A.4 Lemma A.4 (Client-side Second Moment Bound). Let ^ct=1K∑m∈t^c,mt g_c^t= 1K _m _t g_c,m^t be the aggregated client ZO gradient. Under Assumptions 4.1 and 4.2, the second moment of the client update is bounded by: t‖^ct‖2≤C1‖∇cℒ(t)‖2+Ωc,E_t\| g_c^t\|^2≤ C_1\| _ θ_cL( θ^t)\|^2+ _c, (51) where Ωc:=C1(σ2+κ2)+σZO2 _c:=C_1(σ^2+κ^2)+ _ZO^2 collects the combined client-side variance, including stochastic gradient noise, client heterogeneity, and ZO estimation error. Proof. By Jensen’s inequality and the law of iterated expectations, we obtain: t‖^ct‖2 _t \| g_c^t \|^2 =t[t,t[∥1K∑m∈t^c,mt∥2|t]] =E_S_t [E_ ξ^t,U^t [ \| 1K _m _t g_c,m^t \|^2\, |\,S_t ] ] ≤t[1K∑m∈t,t[∥^c,mt∥2|t]]. _S_t [ 1K _m _tE_ ξ^t,U^t [\| g_c,m^t\|^2\, |\,S_t ] ]. (52) Applying Lemma A.2 and Assumption 4.2 yields: t,t[∥^c,mt∥2|t] _ ξ^t,U^t [\| g_c,m^t\|^2\, |\,S_t ] =ξmt[t[∥^c,mt∥2|ξmt,t]] =E_ _m^t [E_U^t [\| g_c,m^t\|^2\, |\, _m^t,S_t ] ] ≤ξmt[C1‖c,mt‖2+σZO2] _ _m^t [C_1\| g_c,m^t\|^2+ _ZO^2 ] ≤C1(‖∇cℒm(t)‖2+σ2)+σZO2. ≤ C_1 (\| _ θ_cL_m( θ^t)\|^2+σ^2 )+ _ZO^2. (53) Taking expectation over client sampling and using Bounded Gradient Dissimilarity from Assumption 4.2 yields: t‖^ct‖2 _t\| g_c^t\|^2 ≤1M∑m=1M(C1‖∇cℒm(t)‖2+C1σ2+σZO2) ≤ 1M _m=1^M (C_1\| _ θ_cL_m( θ^t)\|^2+C_1σ^2+ _ZO^2 ) =C1(1M∑m=1M‖∇cℒm(t)‖2)+C1σ2+σZO2 =C_1 ( 1M _m=1^M\| _ θ_cL_m( θ^t)\|^2 )+C_1σ^2+ _ZO^2 =C1(‖∇cℒ(t)‖2+1M∑m=1M‖∇cℒm(t)−∇cℒ(t)‖2)+C1σ2+σZO2 =C_1 (\| _ θ_cL( θ^t)\|^2+ 1M _m=1^M\| _ θ_cL_m( θ^t)- _ θ_cL( θ^t)\|^2 )+C_1σ^2+ _ZO^2 ≤C1(‖∇cℒ(t)‖2+κ2)+C1σ2+σZO2 ≤ C_1 (\| _ θ_cL( θ^t)\|^2+κ^2 )+C_1σ^2+ _ZO^2 =C1‖∇cℒ(t)‖2+C1(σ2+κ2)+σZO2. =C_1\| _ θ_cL( θ^t)\|^2+C_1(σ^2+κ^2)+ _ZO^2. (54) Substituting the definition of Ωc _c, we obtain the stated bound. ∎ A.6 Proof of Lemma A.5 Lemma A.5 (Local Bias of the ZO Gradient Estimator). Condition on (ℱt,t,ξmt)(F_t,S_t, _m^t), so that ct θ_c^t, mt λ_m^t, and c,mt g_c,m^t are fixed. Let ^c,mt g_c,m^t be the zero-order gradient estimate for client m. The bias of this estimator relative to the true local stochastic gradient c,mt=∇cℓ(;ξmt) g_c,m^t= _ θ_c ( θ^t; _m^t) satisfies: ‖t[^c,mt]−c,mt‖2≤μ2Γ44(dc+3)3.\|E_U^t[ g_c,m^t]- g_c,m^t\|^2≤ μ^2 ^44(d_c+3)^3. (55) Proof. We aim to bound the norm of the bias vector mt:=t[^c,mt]−c,mt b_m^t:=E_U^t[ g_c,m^t]- g_c,m^t. Recall from the proof of Lemma A.2 (Eq. 39) that the estimator decomposes into a primary gradient term and a residual term: ^c,mt=(1P∑p=1Ppt(pt)⊤)c,mt⏟Gradient Term +1Pμ∑p=1Pδptpt⏟Residual Term , g_c,m^t= ( 1P _p=1^P u_p^t( u_p^t) ) g_c,m^t_Gradient Term A+ 1Pμ _p=1^P _p^t u_p^t_Residual Term B, (56) where the scalar residual is δpt=(mt)⊤p,mt _p^t=( λ_m^t) R_p,m^t. Taking the expectation with respect to the perturbations t=ptp=1PU^t=\ u_p^t\_p=1^P: 1. For the Gradient Term A, since [pt(pt)⊤]=dcE[ u_p^t( u_p^t) ]= I_d_c, we have: t[]=dcc,mt=c,mt.E_U^t[ A]= I_d_c g_c,m^t= g_c,m^t. (57) 2. For the Residual Term B, due to the linearity of expectation and identical distribution of pt u_p^t: t[]=1μt[δ(t)t],E_U^t[ B]= 1μE_ u^t[δ( u^t) u^t], (58) where t∼(,dc) u^t ( 0, I_d_c) and δ(t)=(mt)⊤(t)δ( u^t)=( λ_m^t) R( u^t). Thus, the local bias vector is strictly mt=1μt[δ(t)t] b_m^t= 1μE_ u^t[δ( u^t) u^t]. Use Jensen’s inequality: ‖mt‖≤1μt[|δ(t)|‖t‖].\| b_m^t\|≤ 1μE_ u^t[|δ( u^t)|\| u^t\|]. (59) From the proof in Lemma A.2, we have ‖mt‖≤Γ\| λ_m^t\|≤ and ‖(t)‖≤μ22Γ‖t‖2\| R( u^t)\|≤ μ^22 \| u^t\|^2. Applying the Cauchy-Schwarz inequality: |δ(t)|=|(mt)⊤(t)|≤‖mt‖‖(t)‖≤Γ⋅μ22Γ‖t‖2=μ2Γ22‖t‖2.|δ( u^t)|=|( λ_m^t) R( u^t)|≤\| λ_m^t\|\| R( u^t)\|≤ · μ^22 \| u^t\|^2= μ^2 ^22\| u^t\|^2. (60) Substituting this back into the norm bound for mt b_m^t: ‖mt‖ \| b_m^t\| ≤1μt[(μ2Γ22‖t‖2)‖t‖] ≤ 1μE_ u^t [ ( μ^2 ^22\| u^t\|^2 )\| u^t\| ] (61) =μΓ22t[‖t‖3]. = μ ^22E_ u^t[\| u^t\|^3]. (62) For a Gaussian vector t∼(,dc) u^t ( 0, I_d_c), the k-th moment of its norm satisfies [‖t‖k]≤(dc+k)k/2E[\| u^t\|^k]≤(d_c+k)^k/2. For k=3k=3: t[‖t‖3]≤(dc+3)3/2.E_ u^t[\| u^t\|^3]≤(d_c+3)^3/2. (63) Therefore: ‖mt‖≤μΓ22(dc+3)3/2.\| b_m^t\|≤ μ ^22(d_c+3)^3/2. (64) Finally, squaring both sides yields the bound on the squared norm of the bias: ‖t[^c,mt]−c,mt‖2=‖mt‖2≤μ2Γ44(dc+3)3.\|E_U^t[ g_c,m^t]- g_c,m^t\|^2=\| b_m^t\|^2≤ μ^2 ^44(d_c+3)^3. (65) ∎ A.7 Proof of Lemma A.6 Lemma A.6 (Global Bias Bound). Under the same assumptions as Lemma A.5, the squared norm of the difference between the expected aggregated gradient estimator and the true global gradient is bounded by: ‖t[^ct]−∇cℒ(t)‖2≤μ2Γ44(dc+3)3.\|E_t[ g_c^t]- _ θ_cL( θ^t)\|^2≤ μ^2 ^44(d_c+3)^3. (66) Proof. By the tower property, t[^ct] _t[ g_c^t] =t[t,t[1K∑m∈t^c,mt|t]] =E_S_t [E_ ξ^t,U^t [ 1K _m _t g_c,m^t\, |\,S_t ] ] =t[1K∑m∈tξmt,t[^c,mt]], =E_S_t [ 1K _m _tE_ _m^t,U^t [ g_c,m^t ] ], (67) where the second equality holds because ^c,mt g_c,m^t depends only on (ξmt,t)( _m^t,U^t) when conditioned on (ℱt,t)(F_t,S_t). For a specific client m, we define mt:=t[^c,mt|ξmt]−c,mt. b_m^t:=E_U^t\! [ g_c,m^t\, |\, _m^t ]- g_c,m^t. (68) Then ξmt,t[^c,mt] _ _m^t,U^t [ g_c,m^t ] =ξmt[t[^c,mt|ξmt]] =E_ _m^t [E_U^t [ g_c,m^t\, |\, _m^t ] ] =ξmt[c,mt+mt] =E_ _m^t [ g_c,m^t+ b_m^t ] =∇cℒm(t)+ξmt[mt], = _ θ_cL_m( θ^t)+E_ _m^t[ b_m^t], (69) where the last line uses the unbiasedness part of Assumption 4.2. Substituting this identity back and using uniform client sampling, we obtain t[^ct]−∇cℒ(t)=t[1K∑m∈tξmt[mt]].E_t[ g_c^t]- _ θ_cL( θ^t)=E_S_t [ 1K _m _tE_ _m^t[ b_m^t] ]. (70) We now bound the squared norm of this difference: ‖t[^ct]−∇cℒ(t)‖2 \|E_t[ g_c^t]- _ θ_cL( θ^t) \|^2 ≤t[‖1K∑m∈tξmt[mt]‖2] _S_t [ \| 1K _m _tE_ _m^t[ b_m^t] \|^2 ] ≤t[1K∑m∈t‖ξmt[mt]‖2] _S_t [ 1K _m _t \|E_ _m^t[ b_m^t] \|^2 ] ≤t[1K∑m∈tξmt[‖mt‖2]] _S_t [ 1K _m _tE_ _m^t [\| b_m^t\|^2 ] ] ≤t[1K∑m∈tμ2Γ44(dc+3)3] _S_t [ 1K _m _t μ^2 ^44(d_c+3)^3 ] =μ2Γ44(dc+3)3. = μ^2 ^44(d_c+3)^3. (71) The first three inequalities are Jensen’s inequality, and the last one uses Lemma A.5. ∎ A.8 Proof of Theorem 4.4 See 4.4 Proof. By the β-smoothness of the objective function (Assumption 4.1), for the update t+1=t−η[^ct;st] θ^t+1= θ^t-η[ g_c^t; g_s^t], we have: ℒ(t+1)≤ℒ(t)−η⟨∇ℒ(t),[^ct;st]⟩+βη22‖[^ct;st]‖2.L( θ^t+1) ( θ^t)-η ( θ^t),[ g_c^t; g_s^t] + βη^22\|[ g_c^t; g_s^t]\|^2. (72) Taking the conditional expectation tE_t on both sides yields: t[ℒ(t+1)]≤ℒ(t) _t[L( θ^t+1)] ( θ^t) −η⟨∇cℒ(t),t[^ct]⟩−η⟨∇sℒ(t),t[st]⟩ -η _ θ_cL( θ^t),E_t[ g_c^t] -η _ θ_sL( θ^t),E_t[ g_s^t] +βη22(t‖^ct‖2+t‖st‖2). + βη^22 (E_t\| g_c^t\|^2+E_t\| g_s^t\|^2 ). (73) For the server side, we have t[st]=∇sℒ(t)E_t[ g_s^t]= _ θ_sL( θ^t), which implies: −η⟨∇sℒ,t[st]⟩=−η‖∇sℒ(t)‖2.-η _ θ_sL,E_t[ g_s^t] =-η\| _ θ_sL( θ^t)\|^2. (74) For the client side, applying Young’s inequality (−⟨,⟩≤12‖2+12‖2- a, b ≤ 12\| a\|^2+ 12\| b\|^2) yields: −η⟨∇cℒ,t[^ct]⟩ -η _ θ_cL,E_t[ g_c^t] =−η⟨∇cℒ,∇cℒ+(t[^ct]−∇cℒ)⟩ =-η _ θ_cL, _ θ_cL+(E_t[ g_c^t]- _ θ_cL) (75) =−η‖∇cℒ‖2−η⟨∇cℒ,t[^ct]−∇cℒ⟩ =-η\| _ θ_cL\|^2-η _ θ_cL,E_t[ g_c^t]- _ θ_cL (76) ≤−η2‖∇cℒ‖2+η2‖t[^ct]−∇cℒ‖2. ≤- η2\| _ θ_cL\|^2+ η2\|E_t[ g_c^t]- _ θ_cL\|^2. (77) Combining these, the total linear term is bounded by: −η⟨∇cℒ,t[^ct]⟩−η⟨∇sℒ,t[st]⟩≤−η‖∇sℒ‖2−η2‖∇cℒ‖2+η2‖t[^ct]−∇cℒ‖2-η _ θ_cL,E_t[ g_c^t] -η _ θ_sL,E_t[ g_s^t] ≤-η\| _ θ_sL\|^2- η2\| _ θ_cL\|^2+ η2\|E_t[ g_c^t]- _ θ_cL\|^2 (78) Next, we address the quadratic terms by substituting the second moment bounds from Lemmas A.3 and A.4. For the server gradients, we have t‖st‖2≤‖∇sℒ(t)‖2+ΩsE_t\| g_s^t\|^2≤\| _ θ_sL( θ^t)\|^2+ _s. For the client gradients, we have t‖^ct‖2≤C1‖∇cℒ(t)‖2+ΩcE_t\| g_c^t\|^2≤ C_1\| _ θ_cL( θ^t)\|^2+ _c. Substituting these bounds back into the descent inequality and applying the learning rate condition η≤12βC1η≤ 12β C_1, we have: t[ℒ(t+1)] _t[L( θ^t+1)] ≤ℒ(t)−(η−βη22)‖∇sℒ(t)‖2−(η2−βη2C12)‖∇cℒ(t)‖2 ( θ^t)- (η- βη^22 )\| _ θ_sL( θ^t)\|^2- ( η2- βη^2C_12 )\| _ θ_cL( θ^t)\|^2 +η2‖t[^ct]−∇cℒ(t)‖2+βη22(Ωc+Ωs) + η2\|E_t[ g_c^t]- _ θ_cL( θ^t)\|^2+ βη^22 ( _c+ _s ) ≤(a)ℒ(t)−η2‖∇sℒ(t)‖2−η4‖∇cℒ(t)‖2 (a)≤L( θ^t)- η2\| _ θ_sL( θ^t)\|^2- η4\| _ θ_cL( θ^t)\|^2 +η2‖t[^ct]−∇cℒ(t)‖2+βη22(Ωc+Ωs) + η2\|E_t[ g_c^t]- _ θ_cL( θ^t)\|^2+ βη^22 ( _c+ _s ) ≤(b)ℒ(t)−η4‖∇ℒ(t)‖2+η2‖t[^ct]−∇cℒ(t)‖2+βη22(Ωc+Ωs). (b)≤L( θ^t)- η4\| ( θ^t)\|^2+ η2\|E_t[ g_c^t]- _ θ_cL( θ^t)\|^2+ βη^22 ( _c+ _s ). (79) where (a)(a) holds because η≤12βC1η≤ 12β C_1 implies the server coefficient η−βη22≥η2η- βη^22≥ η2 and the client coefficient η2−βη2C12≥η4 η2- βη^2C_12≥ η4; and (b)(b) follows by relaxing −η2‖∇s‖2- η2\| _ θ_s\|^2 to −η4‖∇s‖2- η4\| _ θ_s\|^2 to combine with the client term, yielding the total gradient norm −η4‖∇ℒ‖2- η4\| \|^2. Simultaneously, we apply Lemma A.6 to bound the squared norm of the bias term: ‖t[^ct]−∇cℒ(t)‖2≤μ2Γ44(dc+3)3.\|E_t[ g_c^t]- _ θ_cL( θ^t)\|^2≤ μ^2 ^44(d_c+3)^3. (80) Plugging this bias bound and the variance terms (Ωc+Ωs _c+ _s) into the descent inequality yields: t[ℒ(t+1)]≤ℒ(t)−η4‖∇ℒ(t)‖2+η2[μ2Γ44(dc+3)3]+βη22(Ωc+Ωs).E_t[L( θ^t+1)] ( θ^t)- η4\| ( θ^t)\|^2+ η2 [ μ^2 ^44(d_c+3)^3 ]+ βη^22( _c+ _s). (81) Taking the total expectation E over the full history up to round T−1T-1, summing over t, and rearranging to isolate the gradient norm, we have: η4∑t=0T−1‖∇ℒ(t)‖2≤ℒ(0)−[ℒ(T)]+Tη2[μ2Γ44(dc+3)3]+Tβη22(Ωc+Ωs). η4 _t=0^T-1E\| ( θ^t)\|^2 ( θ^0)-E[L( θ^T)]+ Tη2 [ μ^2 ^44(d_c+3)^3 ]+ Tβη^22( _c+ _s). (82) Finally, using the fact that ℒ(T)≥ℒ∗L( θ^T) ^* and dividing both sides by Tη4 Tη4, we obtain the final convergence rate: 1T∑t=0T−1‖∇ℒ(t)‖2≤4(ℒ(0)−ℒ∗)ηT+2ηβ(Ωc+Ωs)+μ2Γ42(dc+3)3. 1T _t=0^T-1E\| ( θ^t)\|^2≤ 4(L( θ^0)-L^*)η T+2ηβ( _c+ _s)+ μ^2 ^42(d_c+3)^3. (83) This completes the proof. ∎ Appendix B Detailed Experimental Settings B.1 Dataset and Data Partitioning Vision Tasks. We evaluate on CIFAR-10 and CIFAR-100, two standard image classification benchmarks with 32×3232× 32 color images. CIFAR-10 contains 10 classes with 50,000 training images and 10,000 test images, while CIFAR-100 contains 100 classes with 50,000 training images and 10,000 test images. For both datasets, we employ standard data augmentation including random cropping, horizontal flipping and mean/std normalization. For Non-IID settings, we partition the training data among clients using a Dirichlet distribution with concentration parameter α=1α=1, which induces moderate label heterogeneity across clients. Language Tasks. We evaluate on three natural language understanding tasks from the GLUE benchmark. SST-2 is a binary sentiment classification task (67,349 training examples / 872 validation examples / 1,821 test examples). RTE is a binary textual entailment task (2,490 training / 277 validation / 3,000 test). WSC is a coreference resolution task (554 training / 104 validation / 146 test). All language tasks are conducted in an IID setting. B.2 Model Architecture and Split Configuration CV Model. We use a ResNet-18 model pre-trained on ImageNet. The model consists of 4 residual blocks. We split the model after the 2nd residual block. During training, all Batch Normalization (BN) layers are frozen to prevent statistics divergence in non-IID settings. LLM Models. We consider standard causal language models (decoder-only Transformers) consisting of an embedding layer followed by a stack of Transformer blocks and a language modeling head. In the following, “layer” refers to the Transformer blocks. Unless otherwise specified, we used the following split configuration in our experiments: • OPT-125M: Total 12 layers. Client holds the first 3 layers. • Gemma-3-270M: Total 20 layers. Client holds the first 5 layers. • LLaMA-3.2-1B: Total 18 layers. Client holds the first 5 layers. B.3 Baseline Introduction • SFL (Thapa et al., 2022): A hybrid framework amalgamating FL and SL to enable parallel training across distributed clients. We implement the SFLv1 variant, which performs parallel server-side aggregation, and set the local epoch to 1. • SplitLoRA (Lin et al., 2024): A first-order SFL framework for LLMs that combines standard split learning with LoRA, which utilizes full backpropagation on both client and server sides. In our experiments, we set the local epoch to 1. • FSL-SAGE (Nair et al., 2025): A communication-efficient framework that employs client-side auxiliary models to estimate server gradients, allowing clients to perform multiple local updates before communicating with the server. We implement this baseline with a local epoch of 1 and a server update interval of 5, while setting the auxiliary model alignment interval to 1. For the ResNet-18 experiments, the auxiliary model is constructed by cascading the first server-side ResNet block with the final fully-connected layer. • ZO-SFL: A memory-efficient baseline constructed by integrating zeroth-order optimization into the standard SFL architecture. Implementing the Simultaneous Perturbation Stochastic Approximation (SPSA) estimator (Malladi et al., 2023), ZO-SFL performs two forward passes per step with symmetric random perturbations ±μ± applied to the model parameters. The gradient is estimated via the central difference method: ^=ℒ(+μ)−ℒ(−μ)2μ g= L( θ+ )-L( θ- )2μz, enabling backpropagation-free training at the cost of dual activation transmissions. We set the perturbation magnitude μ=10−3μ=10^-3 and the local epoch to 1. • MU-SplitFed (Liang et al., 2025): A straggler-resilient ZO framework that utilizes an unbalanced update mechanism to decouple client and server computations. We configure the server to perform τ=2τ=2 local updates for every client interaction to maximize server-side computational utilization. B.4 Hyperparameters Batch Size and Client Participation. We use a batch size of 32 for both vision and language tasks. For vision experiments, we use 100 clients in total and sample 10 clients per round; for language experiments, we use 10 clients in total and sample 3 clients per round. Low-Rank Adaptation(LoRA). We apply LoRA for language tasks with rank r=8r=8 and scaling factor α=16α=16 to the query and value projection matrices. Optimizer. For HO-SFL, SFL, SplitLoRA, FSL-SAGE, and ZO-SFL, we use AdamW with learning rate 1×10−51× 10^-5, β1=0.9 _1=0.9, β2=0.999 _2=0.999, and weight decay 5×10−45× 10^-4. For MU-SplitFed, we follow the original paper and use standard SGD without momentum, with client learning rate 0.005, server learning rate 0.01, and global learning rate 0.3. Perturbation & Smoothing. Unless otherwise specified, we set P=5P=5 for vision tasks and P=2P=2 for language tasks, and use μ=10−3μ=10^-3. B.5 Fairness in Comparison. In federated settings, communication rounds are often used as a proxy for the communication budget. However, in our comparison, this metric is not directly comparable across methods because a “round” corresponds to different numbers of client updates. Specifically, HO-SFL and MU-SplitFed perform client-side aggregation at every local step, whereas other methods aggregate after E local epochs. Therefore, fixing the number of communication rounds would implicitly allow some methods to perform many more client updates than others, confounding optimization efficiency with update frequency. To make the comparison fair, we use proceeded samples as the unified horizontal axis and stopping criterion. Under our implementation, a batch size of processed samples corresponds to one client update step (one forward-backward or ZO update, depending on the method). Thus, matching proceeded samples ensures that all methods perform the same number of client update steps and consume the same amount of training data, regardless of how often they communicate. Appendix C Additional Results Figure 5: Feasibility analysis of latency hiding under constrained edge resources. C.1 Latency Hiding Feasibility Analysis To evaluate the feasibility of latency hiding in HO-SFL for resource-constrained devices, we simulated a split fine-tuning pipeline for the LLaMA-3.2-1B model. Our simulation adopts conservative parameters to reflect real-world 5G edge computing scenarios, where uplink bandwidth is often the primary bottleneck. The specific settings are detailed below: • Network Environment: Based on global 5G experience reports (Narayanan et al., 2020), we model a standard 5G connection with an uplink bandwidth of 30 Mbps and a downlink bandwidth of 200 Mbps. The round-trip time (RTT) is set to 30 ms. • Client: We simulate an edge AI accelerator with 2.0 TFLOPS (FP16) compute power. This is representative of current high-end embedded systems, such as the NVIDIA Jetson Orin Nano or flagship mobile NPUs. • Server: The server is modeled as a cloud-grade NVIDIA A100 GPU with 312 TFLOPS (FP16) performance. • Workload: We assume a batch size of 32 and a sequence length of 256. The simulation accounts for 10% random noise in network and computing speed to generate error bands. Figure 5 illustrates the maximum number of perturbation passes (P) that can be fully overlapped within the communication and server-side processing latency. As the client-side model deepens, the local forward pass time consumes more of the fixed latency budget. However, for typical split learning configurations where the client retains the first few layers (e.g., 3-5 layers) to preserve data privacy, the system exhibits high tolerance. Specifically, with 4 client layers, the idle window accommodates approximately 4 perturbation passes on average. This confirms that under typical 5G edge conditions, the communication bottleneck and server processing latency naturally mask the computational overhead of the multiple forward passes required by HO-SFL. C.2 More Results on Vision Tasks In this section, we provide additional vision results to complement the main text. Figure 6 reports the validation accuracy curves on CIFAR-100 under both IID and Non-IID settings, while Table 2 summarizes the final test accuracy with standard deviation across CIFAR-10/100 and different data partitions. Table 2: Performance comparison across different vision tasks. The notation Acc(Std)Acc_(Std) denotes the test accuracy (%) and its standard deviation. Bold indicates the best performance among comparative methods for each task. First Order Zeroth Order Hybrid Task (Dataset) SFL FSL SAGE SFL-ZO Mu-SplitFed HO-SFL (Ours) CIFAR-10 (IID) 77.5(0.5)77.5_(0.5) 62.1(1.5)62.1_(1.5) 12.3(1.0)12.3_(1.0) 14.0(0.9)14.0_(0.9) 75.0(0.3)75.0_(0.3) CIFAR-10 (Non-IID) 69.3(2.6)69.3_(2.6) 37.9(2.9)37.9_(2.9) 11.8(1.2)11.8_(1.2) 13.6(1.0)13.6_(1.0) 69.6(0.8)69.6_(0.8) CIFAR-100 (IID) 43.2(0.4)43.2_(0.4) 8.3(4.2)8.3_(4.2) 1.2(0.2)1.2_(0.2) 1.3(0.1)1.3_(0.1) 44.2(0.3)44.2_(0.3) CIFAR-100 (Non-IID) 39.5(0.6)39.5_(0.6) 4.0(3.0)4.0_(3.0) 1.2(0.1)1.2_(0.1) 1.3(0.1)1.3_(0.1) 42.5(0.9)42.5_(0.9) Figure 6: Validation accuracy convergence on CIFAR-100 under IID (left) and Non-IID (right) settings. C.3 Ablation Study on Client Model Depth To investigate the impact of the client-side model complexity on HO-SFL’s performance, we conducted an ablation study by varying the split point of the LLaMA-3.2-1B model on the SST-2 task. Specifically, we varied the number of transformer layers retained on the client side (denoted as “Cut Layer”) from 2 to 8, thereby directly altering the dimensionality of the client-side parameters dcd_c. As illustrated in Figure 7, while HO-SFL achieves convergence across all settings, we observe a consistent trend: as the number of client-side layers increases, the convergence speed slightly decelerates, and the final accuracy exhibits a marginal decline. For instance, the configuration with 2 client layers (Cut Layer 2) converges most rapidly to approx. 94.5%, whereas the 8-layer configuration (Cut Layer 8) saturates around 93.0%. This empirical observation aligns perfectly with our theoretical analysis in Section 4. Recall that Corollary 4.5 establishes the convergence rate of HO-SFL as (dc/PT)O( d_c/PT). Since the client employs zeroth-order optimization, the estimator’s variance is inherently sensitive to the problem dimension dcd_c. Increasing the cut layer count significantly expands dcd_c, thereby introducing higher variance into the gradient estimation and slightly hindering optimization efficiency. Nevertheless, the performance degradation remains minimal even with 8 layers, confirming that our hybrid-order design effectively mitigates the severe dimension-dependent slowdown typically observed in pure ZO methods. Figure 7: Impact of client-side model depth on convergence. We evaluate LLaMA-3.2-1B on the SST-2 task with varying numbers of transformer layers (2, 4, 6, 8) allocated to the client. C.4 Ablation Study on P and μ To analyze the impact of the number of perturbations P and the smoothing parameter μ, we conduct ablation studies using the ResNet-18 on the CIFAR-10 dataset. The results are illustrated in Figure 8. (a) Impact of P (b) Impact of μ Figure 8: Ablation studies on CIFAR-10 with ResNet-18, focusing on the perturbation number P and the smoothing parameter μ. Solid lines denote the mean performance, and shaded regions represent the standard deviation across 10 random seeds. Impact of Perturbation Number P. Figure 8(a) reports the test accuracy as P increases from 1 to 50. We fix the smoothing parameter to μ=10−4μ=10^-4 in this ablation. Consistent with our theoretical analysis in Corollary 4.5, increasing P yields a strictly monotonic improvement in model performance. Since HO-SFL only requires transmitting scalar projections, increasing P incurs negligible communication overhead. Sensitivity to Smoothing Parameter μ. Figure 8(b) illustrates the impact of μ, varying from 10−510^-5 to 10−210^-2. We fix the number of perturbations to P=10P=10 in this ablation. As μ increases beyond 10−310^-3, performance degrades. This aligns with Proposition 4.3, which states that the bias of the zeroth-order estimator scales with (μ2dc)O(μ^2d_c). Conversely, when μ becomes too small, we observe a drop in accuracy. While theory suggests that a smaller μ reduces bias, practical implementations are bounded by machine precision (Jongeneel et al., 2024), rendering the gradient estimate dominated by numerical noise rather than informative signals. C.5 Communication Breakdown Data Table 3 provides the raw data used for the communication efficiency analysis, recorded in the CIFAR-10 IID experiment. The values represent the total communication cost (GB) over the entire training process. Table 3: Breakdown of total communication cost (GB). Component HO-SFL SFL FSL-SAGE ZO-SFL MU-SplitFed Uplink (Act) 1.22 1.26 0.29 2.52 3.67 Uplink (Model) 0.00 0.84 0.84 0.84 12.97 Uplink (Scalar) 0.0002 0.0 0.0 0.0 0.0 Downlink (Grad) 1.22 1.26 0.00 0.00 0.00 Downlink (Model) 0.00 0.84 2.58 0.84 12.97 Downlink (Scalar/Seed) 0.004 0.0 0.0 0.00002 0.00002 C.6 Memory Profiling Methodology To rigorously evaluate the baseline hardware requirements, we explicitly disable advanced memory-saving techniques, such as gradient accumulation and activation checkpointing (Chen et al., 2016). The batch size is fixed at 32 across all experiments. Crucially, to accurately isolate and quantify the specific memory footprint of the client-side model, we implement a specialized Mocker module. This module simulates the server’s interaction by accepting uploaded activations and returning dummy gradients or loss scalars with matching dimensions, thereby decoupling the client’s memory usage from server-side overhead. We utilize Nvidia’s nvidia-smi command-line utility to monitor real-time GPU memory usage.