Paper deep dive
Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization
Pengxin Guo, Shuang Zeng, Zonggen Li, Weiying Zheng, Mengting Liu, Liangqiong Qu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/10/2026, 1:47:59 AM
Summary
The paper introduces Fed-GRPO, a federated learning framework for fine-tuning Large Language Models (LLMs) using Group Relative Policy Optimization (GRPO). It addresses privacy constraints by enabling collaborative training without sharing raw data. Fed-GRPO utilizes reward statistics (mean and standard deviation) generated during GRPO as zero-cost signals to drive three mechanisms: signal-weighted aggregation, global reward calibration, and adaptive sparse communication. This approach outperforms existing federated methods like FedAvg, approaches centralized training performance, and significantly reduces communication costs.
Entities (10)
Relation Signals (7)
Fed-GRPO → usesmechanism → Global Reward Calibration
confidence 95% · Fed-GRPO contains three reward-signal-driven mechanisms: (ii) global reward calibration
Fed-GRPO → usesmechanism → Adaptive Sparse Communication
confidence 95% · Fed-GRPO contains three reward-signal-driven mechanisms: (iii) adaptive sparse communication
Fed-GRPO → usesmechanism → Signal-Weighted Aggregation
confidence 95% · Fed-GRPO contains three reward-signal-driven mechanisms: (i) signal-weighted aggregation
Fed-GRPO → evaluatedon → MATH Dataset
confidence 90% · Extensive experiments on mathematical reasoning tasks... training Qwen2.5-3B-Instruct... on the MATH dataset
Fed-GRPO → outperforms → FedAvg
confidence 90% · Fed-GRPO... clearly outperforms FedAvg
Fed-GRPO → reducescommunicationby → 32x
confidence 90% · losslessly reducing communication by 32x
Signal-Weighted Aggregation → usesmetric → Reward Standard Deviation
confidence 90% · signal-weighted aggregation that weights clients by their reward standard deviation
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL), particularly through Group Relative Policy Optimization (GRPO). However, existing GRPO methods assume centralized access to training data, which may not hold in practice due to privacy or regulatory constraints. To this end, we propose Fed-GRPO, a federated GRPO training framework that addresses these privacy constraints by enabling collaborative reasoning training without sharing raw data, which leverages the reward statistics naturally produced during GRPO training as zero-cost signals to guide aggregation, local training, and communication. Fed-GRPO contains three reward-signal-driven mechanisms: (i) \emph{signal-weighted aggregation} that weights clients by their reward standard deviation, prioritizing clients with stronger learning signals; (ii) \emph{global reward calibration} that re-weights per-prompt objectives based on the local-global reward gap, steering each client toward its relative weaknesses; and (iii) \emph{adaptive sparse communication} that allocates bandwidth based on the informativeness of each client's update. Extensive experiments on mathematical reasoning tasks demonstrate that Fed-GRPO achieves the best performance among all federated methods, clearly outperforms FedAvg and approaches centralized training performance, while losslessly reducing communication by $32\times$ and supporting up to $621\times$ compression under tight bandwidth budgets with only graceful accuracy degradation. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.11502v1
- Canonical: https://arxiv.org/abs/2610.11502v1
Trouble viewing inline? Open PDF directly →
Full Text
81,136 characters extracted from source content.
Expand or collapse full text
Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization Pengxin Guo Shuang Zeng Zonggen Li Weiying Zheng Mengting Liu Liangqiong Qu †thanks: Corresponding author. Email: guopx,zengsh9,zonggen.li,weiyingzheng@connect.hku.hk, Email: liumt55@mail.sysu.edu.cn, liangqqu@hku.hk Affiliation: The University of Hong Kong Affiliation: Sun Yat-sen University Abstract Large Language Models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL), particularly through Group Relative Policy Optimization (GRPO). However, existing GRPO methods assume centralized access to training data, which may not hold in practice due to privacy or regulatory constraints. To this end, we propose Fed-GRPO, a federated GRPO training framework that addresses these privacy constraints by enabling collaborative reasoning training without sharing raw data, which leverages the reward statistics naturally produced during GRPO training as zero-cost signals to guide aggregation, local training, and communication. Fed-GRPO contains three reward-signal-driven mechanisms: (i) signal-weighted aggregation that weights clients by their reward standard deviation, prioritizing clients with stronger learning signals; (i) global reward calibration that re-weights per-prompt objectives based on the local-global reward gap, steering each client toward its relative weaknesses; and (i) adaptive sparse communication that allocates bandwidth based on the informativeness of each client’s update. Extensive experiments on mathematical reasoning tasks demonstrate that Fed-GRPO achieves the best performance among all federated methods, clearly outperforms FedAvg and approaches centralized training performance, while losslessly reducing communication by 32×32× and supporting up to 621×621× compression under tight bandwidth budgets with only graceful accuracy degradation. Our code is available at https://github.com/HKU-HealthAI/Fed-GRPO. 1 Introduction Large Language Models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL), in particular through Group Relative Policy Optimization (GRPO) Shao et al. (2024), which computes advantages from group-relative reward comparisons without a separate critic model. Building on this, DeepSeek-R1 Guo et al. (2025a) demonstrates that GRPO with simple rule-based rewards can produce strong reasoning behavior. Since then, a growing line of work has improved various aspects of GRPO-style training, including training stability Yu et al. (2025), advantage estimation granularity Liu et al. (2025); Sun et al. (2025); Yu et al. (2026), and multi-reward optimization Liu et al. (2026). However, these methods almost universally assume centralized access to training data. This assumption is increasingly unrealistic in high-value reasoning domains, where data is distributed across institutions, platforms, or jurisdictions. For example, hospitals, financial firms, educational platforms, and enterprise coding environments may each possess valuable reasoning data, but cannot pool it due to privacy, ownership, or regulatory constraints. Federated Learning (FL) McMahan et al. (2017); Zeng et al. (2024); Zheng et al. (2025); Guo et al. (2025b), which enables collaborative model training across institutions without sharing raw data, offers a natural solution. However, federated GRPO is not as simple as running GRPO inside a standard FL loop. The core difficulty is that, in GRPO, a client’s usefulness is governed by its reward signal. This makes federated GRPO fundamentally different from conventional supervised FL. In conventional supervised FL, client importance is usually determined by sample counts or controlled through parameter-, gradient-, or representation-space drift correction McMahan et al. (2017); Li et al. (2020a); Karimireddy et al. (2020); Li et al. (2021). In GRPO, however, a large client may be nearly saturated and provide little learning signal, whereas a smaller client with high reward dispersion may produce much more informative policy gradients. Thus, the server must answer a different question: which clients are currently useful for improving reasoning? Notably, a closer examination shows that this reward-dependent notion of usefulness extends far beyond simple client aggregation; it fundamentally alters the entire federated training loop (see Figure 1(a)). First, aggregation is reward-misaligned. FedAvg McMahan et al. (2017) and its follow-up work Li et al. (2020a); Karimireddy et al. (2020); Li et al. (2021) weight clients by dataset size or uniformly, at most mitigating drift in parameter-, gradient-, or representation-space. These methods do not capture the reward-space signal in GRPO, making the resulting global update misaligned with the clients that are most useful for RL training. Second, local objective is reward-miscalibrated. Each client performs GRPO using only local reward statistics, and therefore lacks a global view of prompt difficulty. This can cause clients to under-train globally lagging prompts and over-train prompts already mastered locally. Third, communication remains reward-agnostic. Each federated round requires transmitting billion-parameter models or updates, creating a severe bandwidth bottleneck. Existing compression methods Lin et al. (2018); Guastella et al. (2025); Sarfi et al. (2025) usually rely on parameter-space heuristics, which may over-compress reward-informative clients while wasting bandwidth on uninformative ones. Figure 1: Motivation of Fed-GRPO. Naïve FedAvg-GRPO suffers from three coupled issues: (1) reward-misaligned aggregation, (2) local–global objective miscalibration, and (3) reward-agnostic communication. We observe that the reward statistics (μk,σk)( _k, _k), computed by GRPO at zero extra cost, are precisely the signals needed to address all three: σk _k guides both signal-weighted aggregation and adaptive bandwidth allocation, while the local–global gap on μk _k calibrates per-prompt objectives. To investigate these challenges, we conduct a diagnostic experiment by training Qwen2.5-3B-Instruct Yang et al. (2025b) on the MATH dataset Hendrycks et al. (2021) under standard FedAvg-GRPO with a non-IID partition (Section 3). We observe three properties that connect directly back to the three challenges above: (1) per-client reward standard deviations σk _k vary substantially across clients within the same round, suggesting that learning signal strength—not dataset size—should drive aggregation; (2) per-client reward means μk _k diverges significantly from the global average, revealing a persistent local-global gap that biases local advantage estimation and misallocates training effort; and (3) GRPO updates are naturally sparse, with over 96.2%96.2\% of parameters unchanged after local training, and the update density correlates with σk _k, opening the door to adaptive sparse communication that gives more bandwidth to clients with stronger learning signals. These observations point to a simple but powerful principle: the reward statistics computed by GRPO contain precisely the information needed to guide federated aggregation, local objective calibration, and communication allocation. Importantly, these statistics are available at essentially zero additional training cost. Building on this insight, we propose Fed-GRPO, a federated GRPO training framework that turns these reward statistics into three coupled mechanisms: (i) signal-weighted aggregation weights clients by σk _k instead of dataset size, so that clients with stronger learning signals contribute more to the global update; (i) global reward calibration re-weights per-prompt objectives based on the local-global reward gap, steering each client toward prompts where it lags behind and away from those it has already mastered relative to the global average; and (i) adaptive sparse communication allocates bandwidth based on σk _k and compresses updates by exploiting both the natural sparsity of GRPO updates and module-level update efficiency. Our motivation is illustrated in Figure 1. Extensive experiments across several mathematical reasoning benchmarks demonstrate that Fed-GRPO outperforms all federated baselines by a clear margin and closely matches the centralized upper bound. We summarize our contributions as: i) We identify a fundamental mismatch between standard FL and GRPO, and reveal a powerful insight: the reward statistics (μk,σk _k, _k) inherently computed by GRPO provide zero-cost, perfect signals for federated coordination. i) We propose Fed-GRPO, seamlessly integrating signal-weighted aggregation, global reward calibration, and adaptive sparse communication to resolve the aggregation, optimization, and communication bottlenecks in federated GRPO training. i) We provide a convergence analysis showing that Fed-GRPO converges at rate (1/T)O(1/ T) to a neighborhood of a stationary point of the calibrated objective, with the neighborhood characterized by two interpretable terms—a drift floor from global reward calibration and a compression floor from adaptive sparse communication. iv) We empirically validate Fed-GRPO on five math reasoning benchmarks, where it outperforms all federated baselines and approaches centralized GRPO training, losslessly reduces communication by 32×32×, and supports up to 621×621× compression under tight bandwidth budgets with a graceful accuracy–bandwidth trade-off. 2 Preliminary 2.1 Group Relative Policy Optimization Group Relative Policy Optimization (GRPO) Shao et al. (2024) is a critic-free LLM post-training method that computes advantages via group-relative reward comparisons, eliminating the memory-intensive value network required by PPO Schulman et al. (2017). Given a question q, GRPO samples a group of rollouts o1,o2,…,oGrol\o_1,o_2,…,o_G_rol\ from the old policy πθold _ _old and optimizes the policy πθ _θ by maximizing the following objective: GRPO(θ)= _GRPO(θ)= q∼,oii=1Grol∼πθold(⋅|q) _q ,\o_i\_i=1^G_rol _ _old(·|q) (1) [1Grol∑i=1Grol1|oi|∑t=1|oi|(min(ρi,t(θ)A^i,clip(ρi,t(θ),1−ϵ,1+ϵ)A^i)−βDKL(πθ∥πref))], [ 1G_rol _i=1^G_rol 1|o_i| _t=1^|o_i| ( \! ( _i,t(θ) A_i,\;clip ( _i,t(θ),1-ε,1+ε ) A_i )-β\,D_KL\! ( _θ\| _ref ) ) ], where ρi,t(θ)=πθ(oi,t|q,oi,<t)πθold(oi,t|q,oi,<t) _i,t(θ)= _θ(o_i,t|q,o_i,<t) _ _old(o_i,t|q,o_i,<t), ϵε is the clipping parameter, and β controls the KL penalty strength. The advantage A^i A_i is shared across all tokens in rollout oio_i and is computed by normalizing rewards across the group: A^i=ri−μσ, A_i= r_i-μσ, (2) where rir_i is the reward for rollout oio_i, and μ and σ are the mean and standard deviation of rewards within the group. 2.2 Federated Learning Federated Learning (FL) McMahan et al. (2017); Qu et al. (2022); Yan et al. (2023); Pati et al. (2024); Zhang et al. (2024) enables collaborative model training across K clients without sharing raw data. In the standard FedAvg McMahan et al. (2017) framework, each communication round t consists of three steps: (1) the server broadcasts the global model θ(t)θ^(t) to all clients; (2) each client k performs local training on its private dataset kD_k to obtain updated parameters θk(t) _k^(t) and sends them back to the server; (3) the server aggregates client updates via weighted averaging: θ(t+1)=∑k=1Kwkθk(t),θ^(t+1)= _k=1^Kw_k\, _k^(t), (3) where wkw_k is the aggregation weight for client k, typically set to |k|∑j=1K|j| |D_k| _j=1^K|D_j| or 1K 1K for uniform weighting. This process repeats until convergence or until reaching the maximum number of rounds. 3 Analysis on Federated GRPO Training To motivate our design, we conduct a preliminary experiment by training Qwen2.5-3B-Instruct Yang et al. (2025b) on the MATH dataset Hendrycks et al. (2021) with a non-IID partition across 55 clients based on problem difficulty (one client per difficulty level). We run standard FedAvg with GRPO as the local training algorithm for 12 rounds, with 5 local steps per round. Throughout training, we track the reward mean μk _k and standard deviation σk _k, averaged over all prompts of client k, as well as the overall model update sparsity. Figure 2 summarizes the results, revealing three key observations that directly motivate our design. (a) (b) (c) (d) Figure 2: FedAvg-GRPO training dynamics on the MATH dataset with 55 clients split by difficulty. (a) Per-client reward standard deviation σk _k varies by up to 4×4× across clients within the same round, indicating large differences in learning signal strength. (b) Per-client reward mean μk _k diverges significantly across clients, revealing a persistent local-global gap. (c) Average model update sparsity across clients: over 96.2%96.2\% of parameters remain unchanged after local GRPO training. (d) Correlation between σk _k and model update sparsity: clients with stronger learning signals produce denser updates. Observation 1: Learning signals vary drastically across clients. Figure 2(a) shows that σk _k varies by up to 4×4× across clients within the same round. Since σk _k reflects the overall strength of a client’s learning signal in GRPO, such variation implies that actual gradient contributions also differ substantially. However, standard FedAvg ignores this and weights all clients equally by dataset size. This motivates signal-weighted aggregation, which weights clients by σk _k to prioritize those with stronger learning signals. Observation 2: Local reward means diverge from the global average. Figure 2(b) shows persistent gaps between per-client reward means μk _k and the global average throughout training. A client specializing in easy problems maintains a high μk _k but contributes little to global improvement, whereas a client on hard problems has a low μk _k yet holds the most room for improvement. Under standard GRPO, each client only sees its local reward statistics and cannot distinguish “locally hard but globally easy” prompts from “locally easy but globally hard” ones, resulting in misallocated training effort. This motivates global reward calibration, which uses the gap between μk _k and the global reward mean to re-weight per-prompt objectives so that each client focuses on its relative weaknesses. Observation 3: GRPO updates are naturally sparse and correlate with σk _k. Figure 2(c) shows that over 96.2%96.2\% of parameters remain unchanged after local GRPO training, and this high sparsity is remarkably stable across all training rounds. Moreover, Figure 2(d) reveals a strong negative correlation between σk _k and model update sparsity—clients with stronger learning signals leave fewer parameters unchanged and thus produce denser updates. This motivates adaptive sparse communication, which gives more bandwidth to high-σk _k clients whose updates carry more information, while compressing low-σk _k clients more aggressively, creating a natural alignment between update quality and communication cost. Taken together, these observations show that the reward statistics μk _k and σk _k, computed by GRPO at no extra cost, carry rich information about the aggregation, training, and communication needs of each client. In the next section, we describe how Fed-GRPO leverages these statistics to build a unified framework with three key mechanisms that jointly address the three challenges above. 4 Methodology Guided by the observations in Section 3, we design Fed-GRPO, a framework that leverages the reward statistics naturally produced during GRPO training to improve aggregation, local training, and communication for federated GRPO training. Fed-GRPO contains three key mechanisms: (i) signal-weighted aggregation weights clients by σk _k instead of dataset size, so that clients with stronger learning signals contribute more; (i) global reward calibration re-weights per-prompt objectives based on the local-global reward gap, steering each client toward prompts where it lags behind; and (i) adaptive sparse communication allocates bandwidth based on σk _k and compresses updates by exploiting sparsity and efficiency-based module selection. We detail each mechanism below. 4.1 Signal-Weighted Aggregation Standard FedAvg (Eq. (3)) weights clients by dataset size (or uniformly), which ignores the quality of each client’s update during GRPO training. In GRPO training, dataset size is a poor proxy for update quality while the reward standard deviation σk _k is a direct measure of the learning signal strength: a client with high σk _k produces more informative policy gradients and thus should have a larger influence on the global update. To this end, we propose to weight clients by their reward standard deviation σk _k, which reflects the diversity of rewards and hence the strength of the learning signal. Specifically, the aggregation weight for client k used in Eq. (3) is replaced by: wk=σk∑j=1Kσj,w_k= _k _j=1^K _j, (4) where σk _k is the reward standard deviation from client k’s local GRPO training in the current round. This weighting has two properties: (1) it is zero-cost, since σk _k is already computed during GRPO training for advantage normalization; and (2) it is adaptive, meaning the aggregation weights adjust dynamically throughout the training process as each client’s σk _k naturally evolves (see Figure 3). 4.2 Global Reward Calibration In non-IID federated settings, clients have varying data distributions and varying levels of mastery. Under standard GRPO, each client optimizes its policy based solely on local reward outcomes, lacking any awareness of the global training progress. As a result, a client cannot objectively evaluate its own performance—it does not know whether a locally low reward implies an inherently hard problem or a severe lag behind the global model. This lack of global perspective leads to inefficient local training: clients waste valuable computation over-training on prompts they have already mastered relative to the global average, while under-training on their relative weaknesses. To address this, we introduce global reward calibration, which uses the global reward statistics to re-weight per-prompt objectives during local training. After each aggregation round, the server computes global reward statistics from the clients’ reported metrics and broadcasts them back to all clients: μglobal=1K∑k=1Kμk,σglobal=1K−1∑k=1K(μk−μglobal)2. _global= 1K _k=1^K _k, _global= 1K-1 _k=1^K( _k- _global)^2. (5) Each client then applies a per-prompt weight based on the gap between its local reward mean and the global mean, encouraging it to focus on prompts where it falls behind and reducing effort on prompts it has already mastered relative to others. Formally, for a prompt q with local mean reward r¯q r_q (averaged over GrolG_rol rollouts), the calibration weight is: ωq=1+λ⋅tanh(μglobal−r¯qσglobal+εσ), _q=1+λ· \! ( _global- r_q _global+ _σ ), (6) where λ∈[0,1]λ∈[0,1] controls the calibration strength, and εσ _σ is a small constant to prevent division by zero. Then, the local GRPO training objective will be replaced by a weighted version that incorporates ωq _q: ℒk=∑qωq⋅ℓq∑qωq,L_k= _q _q· _q _q _q, (7) where k denotes the client index, and ℓq:=1Grol∑i=1Grol1|oi|∑t[min(ρi,tA^i,clip(ρi,t,1−ϵ,1+ϵ)A^i)−βDKL] _q:= 1G_rol _i=1^G_rol 1|o_i| _t [ ( _i,t A_i,clip( _i,t,1-ε,1+ε) A_i)-β D_KL ] is the standard GRPO objective for prompt q. The intuition behind this design is straightforward: when a client’s local performance on a prompt is below the global average, the weight ωq _q increases, amplifying the gradient contribution from that prompt and encouraging the client to focus on its weaknesses; conversely, when a client already exceeds the global average on a prompt, ωq _q decreases, reducing unnecessary optimization on that prompt. The tanh function ensures that weights are bounded within [1−λ,1+λ][1-λ,1+λ], providing a smooth and stable calibration effect while preventing extreme values. In the first round, since global statistics are unavailable, all weights default to 1 and the method reduces to standard GRPO. 4.3 Adaptive Sparse Communication The communication cost of transmitting full model parameters at each round is a major bottleneck in federated GRPO training, especially as model sizes grow. Fortunately, the natural sparsity of GRPO updates (over 96.2%96.2\% unchanged parameters, see Figure 2(c)) and the correlation between σk _k and update density (see Figure 2(d)) suggest an opportunity for adaptive sparse communication that allocates bandwidth based on the informativeness of each client’s update. Specifically, clients with higher σk _k—whose updates carry more learning signal and receive larger aggregation weights—are given more communication bandwidth, while low-σk _k clients (whose updates are mostly zero) are compressed more aggressively. This creates a natural alignment: low-signal clients are both compressed more and weighted less, minimizing the impact of compression-induced information loss. Formally, given a target average per-client communication budget ratio B∈(0,1]B∈(0,1] (defined as the fraction of changed parameters to retain), we determine the adaptive budget ratio BkB_k for client k based on its normalized signal strength σ^k=σk/maxjσj σ_k= _k/ _j _j as follows: Bk=clip(B⋅(1+α⋅(σ^k−σ^¯)),Bmin, 1),B_k=clip (B· (1+α·( σ_k- σ) ),\;B_ ,\;1 ), (8) where σ^¯=1K∑jσ^j σ= 1K _j σ_j is the mean normalized value across clients, BminB_ is a minimum budget to ensure that even low-signal clients can transmit some updates, and α controls the adaptation strength. To meet the target retention ratio BkB_k, we apply a two-stage compression strategy that exploits the structural properties of GRPO updates. First, we apply a lossless sparse encoding that extracts only the parameters that changed after local training, leveraging the fact that most parameters remain unchanged during GRPO training (see Figure 2(c)). This alone achieves significant compression (approximately 32×32×) without any loss in accuracy, keeping 100%100\% of the updated parameters (Bk=1.0B_k=1.0). When Bk<1.0B_k<1.0, we further apply lossy compression by selecting a subset of modules to transmit based on their update efficiency, ensuring that we preserve the most informative updates while dropping parameters until reaching the target fraction BkB_k. The update efficiency of a module is defined as the update energy per transmitted parameter: Em=‖Δm‖F2‖Δm‖0,E_m= \| _m\|_F^2\| _m\|_0, (9) where Δm _m is the update for module m and ‖Δm‖0\| _m\|_0 is the number of changed parameters in module m. We then sort modules by efficiency EmE_m in increasing order and greedily remove modules until the final transmitted parameter count roughly equals the fraction BkB_k of the changed parameters. This efficiency-based selection ensures that we drop only the least informative updates, maximizing the retained learning signal under tight communication constraints. Importantly, since BkB_k is adaptive to σk _k, clients with stronger learning signals are more likely to retain more of their updates, while clients with weaker signals (which contribute less to the global update) are compressed more aggressively, creating a natural synergy between update quality and communication cost. 4.4 Fed-GRPO Algorithm Fed-GRPO integrates the above three mechanisms into a unified training procedure. Specifically, each round begins with the server broadcasting the current global model and global reward statistics to all clients. Each client then performs local GRPO training with global reward calibration, computes its reward statistics, and applies adaptive sparse communication to transmit its compressed update back to the server. The server aggregates the received updates using signal-weighted aggregation and updates the global reward statistics for the next round. This iterative process continues until convergence or until reaching the maximum number of rounds. Algorithm 1 in Appendix A provides a detailed summary. 5 Convergence Analysis We make the following assumptions, some of which are commonly used in the FL literature Li et al. (2020b); Guo et al. (2025d); Karimireddy et al. (2019); Guo et al. (2025c), while Assumption 4 adapts a tool from non-stationary stochastic optimization Besbes et al. (2015); Jadbabaie et al. (2015). Assumption 1. (L-smoothness) Each local objective ℒkL_k is L-smooth: for any parameters θ,θ′θ,θ , ℒk(θ)≤ℒk(θ′)+⟨θ−θ′,∇ℒk(θ′)⟩F+L2‖θ−θ′‖F2,L_k(θ) _k(θ )+ θ-θ ,\, _k(θ ) _F+ L2\|θ-θ \|^2_F, where ⟨⋅,⋅⟩F ·,· _F and ∥⋅∥F\|·\|_F denote the Frobenius inner product and norm, respectively. Assumption 2. (Bounded stochastic gradient) Let ξk,s(t) _k,s^(t) be sampled from the k-th client’s local data uniformly at random at the s-th training step of round t. The expected squared norm of stochastic gradients is uniformly bounded: ‖∇ℒk(t)(θk,s(t),ξk,s(t))‖F2≤G2E\| _k^(t)( _k,s^(t); _k,s^(t))\|_F^2≤ G^2, for all k=1,⋯,Kk=1,·s,K, t=0,⋯,T−1t=0,·s,T-1, and s=1,⋯,Ss=1,·s,S. Assumption 3. (Bounded compression error) The adaptive sparse communication operator k(⋅)C_k(·) in Section 4.3 is contractive path-wise: ‖k(Δ)−Δ‖F2≤(1−Bk)‖Δ‖F2\|C_k( )- \|_F^2≤(1-B_k)\| \|_F^2 for any Δ , where BkB_k is the budget ratio assigned to client k. Assumption 4. (Bounded average calibration drift) The average round-to-round drift of the global objective along the iterate path is bounded in expectation: 1T∑t=0T−1[Δℒ(t)]≤R,Δℒ(t):=|ℒ(t+1)(θ(t+1))−ℒ(t)(θ(t+1))|. 1T _t=0^T-1E[ ^(t)_L]\;≤\;R, ^(t)_L\;:=\; |L^(t+1)(θ^(t+1))-L^(t)(θ^(t+1)) |. Here ℒ(T)L^(T) denotes any uniformly bounded extension of the round-statistics (e.g., reusing the round-(T−1)(T-1) weights and calibration); the specific choice does not affect the bound. Then, we present the convergence rate for Fed-GRPO, with the proof provided in Appendix B. Theorem 1. Let Assumptions 1–4 hold and L, G, R be defined therein. For a learning rate η, we have: 1T∑t=0T−1‖∇ℒ(t)(θ(t))‖F2≤ 2DMT+ 2RM+ 4K(1−B¯)G2, 1T _t=0^T-1E \| ^(t)(θ^(t)) \|_F^2\;≤\;2 D\,MT\;+\;2 R\,M\;+\;4 K(1- B)\,G^2, (10) where ℒ(t)(θ)=∑k=1Kwk(t)ℒk(t)(θ)L^(t)(θ)= _k=1^Kw_k^(t)L_k^(t)(θ) is the global objective at round t, D:=ℒ(0)(θ(0))−inf0≤t≤T,θℒ(t)(θ)D:=L^(0)(θ^(0))- _0≤ t≤ T,\,θL^(t)(θ), B¯=1KT∑t=0T−1∑k=1KBk(t) B= 1KT _t=0^T-1 _k=1^KB_k^(t) is the per-client budget ratio averaged over both clients and rounds (which equals the user-specified target B in Eq. (8) when clipping is inactive), and M is a constant depending on L,G,S,KL,G,S,K. According to Theorem 1, Fed-GRPO converges to a neighborhood of a stationary point of the time-varying calibrated objective at rate (1/T)O(1/ T), where the neighborhood size is determined by two irreducible terms: a drift floor of size (RM)O( RM) from the round-to-round change of the calibrated objective, and a compression floor of size (K(1−B¯)G2)O( K(1- B)\,G^2) from adaptive sparse communication. When the time-varying global objective is stationary across rounds (R=0R=0) and the encoding is lossless (B¯=1 B=1), both floors vanish and Fed-GRPO recovers the standard (1/T)O(1/ T) rate of FedAvg in the non-convex setting Li et al. (2020b). 6 Experiments 6.1 Experimental Setup For mathematical reasoning tasks tasks, we train on the MATH dataset Hendrycks et al. (2021), split across K=5K=5 clients by difficulty to form a naturally non-IID partition, and evaluate on five math reasoning benchmarks: GSM8K Cobbe et al. (2021), MATH-500 Hendrycks et al. (2021); Lightman et al. (2023), AIME 2024, AIME 2025, and AMC 2023. The base models are Qwen2.5-3B-Instruct Yang et al. (2025b) and Qwen3-4B-Instruct Yang et al. (2025a), optimized with AdamW Loshchilov and Hutter (2019) (lr 3×10−63×10^-6) under GRPO with a binary correctness reward, prompt batch size 6464, and 88 rollouts per prompt. For coding tasks, we use Qwen2.5-3B-Instruct as the base model, train on the MBPP dataset Austin et al. (2021) (”train” + ”validation”) and evaluate on the held-out MBPP ”test” split and the HumanEval dataset Chen et al. (2021). To construct a non-IID federation setting, we partition the MBPP dataset by the reference-solution length, bucket it into 5 quantile bins, and partition the data across clients with a Dirichlet distribution (α=0.5α=0.5) over these buckets. All federated methods run for T=12T=12 communication rounds with S=5S=5 local steps; centralized and local-only baselines are trained for an equivalent number of gradient updates. We compare Fed-GRPO against the base model, centralized GRPO (upper bound), local-only GRPO, sample- and uniform-weighted FedAvg McMahan et al. (2017) (FedAvg-SW / FedAvg-UW), two sparse-communication FL methods, SparsyFed Guastella et al. (2025) and SparseLoCo Sarfi et al. (2025), and one Federated GRPO method, FGRPO Chen et al. (2026). Following previous GRPO literature Yu et al. (2025); Guo et al. (2025a), we evaluate the best checkpoint of each method and report avg@3232 (32 samples per prompt at temperature 1.01.0, top-p=0.7p=0.7). More details are deferred to Appendix C. 6.2 Main Results Table 1: Main results on five math reasoning benchmarks using Qwen2.5-3B-Instruct as the base model. FedAvg-SW and FedAvg-UW denote sample-weighted and uniform-weighted FedAvg, respectively. Budget B is the target fraction of changed parameters retained for transmission; Comm. (MB) is the average per-round per-client uplink payload (BF16 values plus 4-byte indices for sparse encodings); Comp. (×) reports the compression ratio relative to dense full-model transmission as parameter ratio / byte ratio. Best result for federated methods in bold. †For SparsyFed and SparseLoCo, Budget denotes the kept fraction of all parameters. Method Budget Comm. (MB) Comp. (×) GSM8K MATH-500 AIME24 AIME25 AMC Avg. Qwen2.5-3B-Instruct - - - 59.47 46.54 2.50 1.46 27.11 27.42 Centralized - - - 84.75 63.42 7.50 3.44 41.95 40.21 Local-Only - - - 82.45 58.16 5.88 2.94 38.28 37.54 FedAvg-SW - 6,794.2 1×/ 1×1×\,/\,1× 79.55 56.84 6.56 3.02 39.53 37.10 FedAvg-UW - 6,794.2 1×/ 1×1×\,/\,1× 80.00 55.73 5.42 3.12 37.66 36.39 SparsyFed 5%†5\% 1,019.1 20×/ 7×20×\,/\,7× 79.10 56.61 6.56 2.92 39.53 36.94 SparseLoCo 3%†3\% 451.2 32×/ 15×32×\,/\,15× 78.69 56.05 6.04 2.81 39.30 36.58 FGRPO - 6,794.2 1×/ 1×1×\,/\,1× 83.11 59.79 6.83 3.02 39.73 38.49 Fed-GRPO (Ours) 100%100\% 645.5 32×/ 11×32×\,/\,11× 83.21 60.82 7.08 3.12 41.09 39.06 50%50\% 321.4 63×/ 21×63×\,/\,21× 83.11 60.84 7.19 2.71 41.25 39.02 20%20\% 126.9 161×/ 54×161×\,/\,54× 81.16 56.14 5.31 2.81 38.67 36.82 10%10\% 62.2 327×/ 109×327×\,/\,109× 79.69 55.21 5.42 3.12 36.25 35.94 5%5\% 32.8 621×/ 207×621×\,/\,207× 77.29 54.83 5.62 3.23 34.45 35.08 Table 1 reports the main results achieved by each method across the five benchmarks. From this table, we can draw several key conclusions. First, Fed-GRPO achieves the best performance among all federated methods, and approaches the centralized upper bound, demonstrating that it effectively mitigates the challenges of non-IID data and aggregation mismatch in federated GRPO training. Second, the fact that FedAvg-SW and FedAvg-UW perform worse than Local-Only highlights the pitfalls of naive aggregation strategies that ignore update quality, underscoring the importance of our signal-weighted aggregation mechanism. Third, Fed-GRPO offers a flexible communication–accuracy trade-off across a wide range of bandwidth budgets. At B=100%B=100\%, our adaptive sparse communication is mathematically lossless and already shrinks the per-round uplink from 6,794.26,794.2 MB to 645.5645.5 MB—a 32×32× parameter (11×11× byte) compression with zero accuracy loss. Halving the budget to B=50%B=50\% doubles the compression to 63×63× (21×21× in bytes) while preserving essentially the same average accuracy (39.0239.02 vs. 39.0639.06), and even at B=20%B=20\% (161×161×, 54×54× in bytes) Fed-GRPO still matches FedAvg’s accuracy at a small fraction of the bandwidth. Pushing further to B=10%B=10\% and B=5%B=5\% delivers extreme 327×327× and 621×621× parameter compression (109×109× and 207×207× in bytes; 32.832.8 MB per round at B=5%B=5\%), at the cost of a moderate accuracy drop of ∼ 3–4 points relative to lossless transmission. We emphasize that this 621×621× regime is intended for severely bandwidth-constrained deployments rather than as the default operating point: as the lower rows of Table 1 show, the degradation is gradual rather than abrupt, allowing practitioners to dial B along a smooth Pareto frontier and select an operating point—e.g., B=50%B=50\% as a near-lossless sweet spot, or B=5%B=5\% when bandwidth dominates—according to their bandwidth–accuracy preferences. 6.3 More Results on Other Base Models and Tasks Tables 3 and 3 report additional results on math reasoning benchmarks using Qwen3-4B-Instruct as the base model and on coding tasks using Qwen2.5-3B-Instruct as the base model, respectively. The results show that Fed-GRPO consistently outperforms all other federated methods across different base models and tasks, demonstrating its generality and effectiveness in various settings. More results on different data heterogeneity and analysis on learning dynamics are provided in Appendix D. Table 2: Results on five math reasoning benchmarks using Qwen3-4B-Instruct as the base model. Method GSM8K MATH-500 AIME24 AIME25 AMC Avg. Qwen3-4B-Instruct 77.56 84.55 32.7 29.79 77.5 60.42 Centralized 93.76 86.90 36.67 32.50 81.95 66.36 Local-Only 93.45 86.18 33.75 30.04 78.55 64.39 FedAvg-SW 93.66 86.76 35.62 29.79 82.50 65.66 FedAvg-UW 93.73 86.73 34.90 30.52 81.02 65.38 SparsyFed 93.48 86.53 35.02 31.22 81.13 65.47 SparseLoCo 93.28 86.32 34.93 31.03 81.07 65.33 FGRPO 93.77 86.75 35.23 31.78 81.62 65.83 Fed-GRPO (Ours) 93.93 86.85 35.94 32.29 82.73 66.35 Table 3: Results on coding tasks using Qwen2.5-3B-Instruct as the base model. Method GSM8K MATH-500 Qwen2.5-3B-Instruct 57.40 60.41 Centralized 62.80 62.85 Local-Only 59.60 60.80 FedAvg-SW 60.70 61.63 FedAvg-UW 60.82 61.82 SparsyFed 60.66 61.23 SparseLoCo 60.03 60.98 FGRPO 61.42 61.94 Fed-GRPO (Ours) 62.23 62.44 6.4 Ablation Studies To isolate the contribution of each mechanism in Fed-GRPO, we conduct ablation studies on the math tasks using Qwen2.5-3B-Instruct as the base model by starting from FedAvg-SW (i.e., Fed-GRPO with all three mechanisms disabled) and progressively adding each component. As shown in Table 4, each mechanism contributes complementarily. Starting from FedAvg-SW (37.10%), signal-weighted aggregation (SWA) lifts the average to 38.01% (+0.91+0.91) with consistent gains across all five benchmarks, confirming that σk _k better reflects update quality than dataset size. Adding global reward calibration (GRC) further improves the average to 39.07% (+1.06+1.06), with the largest gains on harder benchmarks (e.g., MATH-500: 58.73→61.2658.73→ 61.26), matching GRC’s design intent of steering clients toward globally lagging prompts. Finally, adaptive sparse communication (ASC) at B=1.0B=1.0 preserves accuracy (39.06% vs. 39.07%) while shrinking the per-client uplink from 6,794.26,794.2 MB to 645.5645.5 MB—a 11×11× reduction at zero accuracy cost. Overall, SWA and GRC drive the accuracy gains, while ASC contributes orthogonal communication savings. Table 4: Ablation studies on Fed-GRPO components. ++SWA: add signal-weighted aggregation. ++GRC: add global reward calibration. ++ASC: add adaptive sparse communication (at B=1.0B=1.0, lossless). Comm. (MB) refers to the average per-round per-client uplink payload. Configuration GSM8K MATH-500 AIME24 AIME25 AMC Avg. Comm. (MB) FedAvg-SW (no component) 79.55 56.84 6.56 3.02 39.53 37.10 6,794.2 ++ SWA (§4.1) 81.24 58.73 7.02 3.05 40.03 38.01 6,794.2 ++ SWA ++ GRC (§4.2) 82.75 61.26 7.26 3.13 40.94 39.07 6,794.2 ++ SWA ++ GRC ++ ASC (Fed-GRPO) 83.21 60.82 7.08 3.12 41.09 39.06 645.5 7 Related Work Several works also explore federated GRPO training but most from angles orthogonal to ours. FedGRPO Zhu et al. (2026) guides privacy-preserving server updates using GRPO’s ”Group Relative” rewards among experts based on clients’ domain knowledge. FedMOA Wang et al. (2026) tackles multi-objective RL via hypergradient-based online weighting and task-aware server aggregation. Concurrently, FGRPO Chen et al. (2026) introduces an adaptive aggregation mechanism that dynamically prioritizes effective learning trajectories based on each client’s performance gain relative to its historical baseline. Differently, Fed-GRPO targets the standard federated GRPO reasoning with a shared accuracy-style reward and non-IID prompt distribution. It is the first to identify that the reward statistics (μk,σk)( _k, _k) produced “for free” during GRPO advantage normalization carry rich information about each client’s aggregation, training, and communication needs—turning them into a unified, zero-cost signal that drives all three mechanisms. 8 Conclusion We present Fed-GRPO, a novel FL framework for training large language models with GRPO. Fed-GRPO introduces three key mechanisms that leverage GRPO’s unique reward statistics to address the challenges of naïve Fedavg GRPO training: signal-weighted aggregation that dynamically weights client updates based on their reward standard deviation, global reward calibration that re-weights local objectives based on local-global reward gaps, and adaptive sparse communication that allocates bandwidth based on update quality. Experiments on several mathematical reasoning tasks demonstrate that Fed-GRPO significantly outperforms federated baselines and approaches centralized performance while achieving up to 32×32× lossless communication compression. This work opens up new possibilities for collaborative reasoning training across distributed data sources without sharing raw data. References Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: Appendix C, §6.1. Besbes et al. (2015) O. Besbes, Y. Gur, and A. Zeevi Non-stationary stochastic optimization. Operations research 63 (5), p. 1227–1244. Cited by: §5. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating large language models trained on code. External Links: 2107.03374 Cited by: Appendix C, §6.1. Chen et al. (2026) P. Chen, S. Li, K. Wang, Y. Yuan, K. Han, J. Luo, and F. Li FGRPO: federated grpo with adaptive aggregation on non-iid data. arXiv preprint arXiv:2606.03094. Cited by: Appendix C, §6.1, §7. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: Appendix C, §6.1. Guastella et al. (2025) A. Guastella, L. Sani, A. Iacob, A. Mora, P. Bellavista, and N. D. Lane SparsyFed: sparse adaptive federated learning. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix C, §1, §6.1. Guo et al. (2025a) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: Appendix C, §1, §6.1. Guo et al. (2025b) P. Guo, Y. Wang, W. Li, M. Liu, M. Li, J. Zheng, and L. Qu Exploring federated pruning for large language models. arXiv preprint arXiv:2505.13547. Cited by: §1. Guo et al. (2025c) P. Guo, S. Zeng, W. Chen, X. Zhang, W. Ren, Y. Zhou, and L. Qu A new federated learning framework against gradient inversion attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 16969–16977. Cited by: §5. Guo et al. (2025d) P. Guo, S. Zeng, Y. Wang, H. Fan, F. Wang, and L. Qu Selective aggregation for low-rank adaptation in federated learning. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §5. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), External Links: Link Cited by: Appendix C, §1, §3, §6.1. Jadbabaie et al. (2015) A. Jadbabaie, A. Rakhlin, S. Shahrampour, and K. Sridharan Online optimization: competing with dynamic comparators. In Artificial Intelligence and Statistics, p. 398–406. Cited by: §5. Karimireddy et al. (2020) S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh Scaffold: stochastic controlled averaging for federated learning. In International conference on machine learning, p. 5132–5143. Cited by: §1, §1. Karimireddy et al. (2019) S. P. Karimireddy, Q. Rebjock, S. Stich, and M. Jaggi Error feedback fixes signsgd and other gradient compression schemes. In International conference on machine learning, p. 3252–3261. Cited by: §5. Li et al. (2021) Q. Li, B. He, and D. Song Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10713–10722. Cited by: §1, §1. Li et al. (2020a) T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2, p. 429–450. Cited by: §1, §1. Li et al. (2020b) X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, External Links: Link Cited by: §5, §5. Lightman et al. (2023) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In The twelfth international conference on learning representations, Cited by: Appendix C, §6.1. Lin et al. (2018) Y. Lin, S. Han, H. Mao, Y. Wang, and B. Dally Deep gradient compression: reducing the communication bandwidth for distributed training. In International Conference on Learning Representations, External Links: Link Cited by: §1. Liu et al. (2026) S. Liu, X. Dong, X. Lu, S. Diao, P. Belcak, M. Liu, M. Chen, H. Yin, Y. F. Wang, K. Cheng, et al. Gdpo: group reward-decoupled normalization policy optimization for multi-reward rl optimization. arXiv preprint arXiv:2601.05242. Cited by: §1. Liu et al. (2025) Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin Understanding r1-zero-like training: a critical perspective. In Second Conference on Language Modeling, External Links: Link Cited by: §1. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: Appendix C, §6.1. McMahan et al. (2017) B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, p. 1273–1282. Cited by: Appendix C, §1, §1, §2.2, §6.1. Pati et al. (2024) S. Pati, S. Kumar, A. Varma, B. Edwards, C. Lu, L. Qu, J. J. Wang, A. Lakshminarayanan, S. Wang, M. J. Sheller, et al. Privacy preservation for federated learning in health care. Patterns 5 (7). Cited by: §2.2. Qu et al. (2022) L. Qu, Y. Zhou, P. P. Liang, Y. Xia, F. Wang, E. Adeli, L. Fei-Fei, and D. Rubin Rethinking architecture design for tackling data heterogeneity in federated learning. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 10051–10061. Cited by: §2.2. Sarfi et al. (2025) A. Sarfi, B. Thérien, J. Lidin, and E. Belilovsky Communication efficient llm pre-training with sparseloco. arXiv preprint arXiv:2508.15706. Cited by: Appendix C, §1, §6.1. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2.1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §2.1. Sun et al. (2025) W. Sun, W. Yang, P. Jian, Q. Du, F. Cui, S. Ren, and J. Zhang KTAE: a model-free algorithm to key-tokens advantage estimation in mathematical reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1. von Werra et al. (2020) TRL: Transformers Reinforcement Learning External Links: Link Cited by: Appendix C. Wang et al. (2026) Z. Wang, D. Jung, Y. He, G. Sun, Z. Shen, M. Lee, and A. Li FedMOA: federated grpo for personalized reasoning llms under heterogeneous rewards. arXiv preprint arXiv:2602.00453. Cited by: §7. Yan et al. (2023) R. Yan, L. Qu, Q. Wei, S. Huang, L. Shen, D. L. Rubin, L. Xing, and Y. Zhou Label-efficient self-supervised federated learning for tackling data heterogeneity in medical imaging. IEEE Transactions on Medical Imaging 42 (7), p. 1932–1943. Cited by: §2.2. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: Appendix C, §6.1. Yang et al. (2025b) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: Appendix C, §1, §3, §6.1. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, YuYue, W. Dai, T. Fan, G. Liu, J. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, R. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, Y. Wu, and M. Wang DAPO: an open-source LLM reinforcement learning system at scale. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Appendix C, §1, §6.1. Yu et al. (2026) Z. Yu, Z. Chen, M. Liu, H. Zhang, and L. Qu Unveiling implicit advantage symmetry: why grpo struggles with exploration and difficulty adaptation. arXiv preprint arXiv:2602.05548. Cited by: §1. Zeng et al. (2024) S. Zeng, P. Guo, S. Wang, J. Wang, Y. Zhou, and L. Qu Tackling data heterogeneity in federated learning via loss decomposition. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 707–717. Cited by: §1. Zhang et al. (2024) J. Zhang, S. Zeng, M. Zhang, R. Wang, F. Wang, Y. Zhou, P. P. Liang, and L. Qu Flhetbench: benchmarking device and state heterogeneity in federated learning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 12098–12108. Cited by: §2.2. Zheng et al. (2025) W. Zheng, Z. Lin, P. Guo, Y. Zhou, F. Wang, and L. Qu Fedvlmbench: benchmarking federated fine-tuning of vision-language models. arXiv preprint arXiv:2506.09638. Cited by: §1. Zhu et al. (2026) G. Zhu, H. Gu, L. Fan, Q. Yang, and Y. Han FedGRPO: privately optimizing foundation models with group-relative rewards from domain clients. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 19487–19495. Cited by: §7. Appendix A Fed-GRPO Algorithm Algorithm 1 provides a detailed summary of the Fed-GRPO method. Algorithm 1 Fed-GRPO 0: Initial model θ(0)θ^(0), local datasets kk=1K\D_k\_k=1^K, communication rounds T, local training steps S, target communication budget ratio B 0: Trained global model θ(T)θ^(T) 1: Server initializes: μglobal←None _global , σglobal←None _global 2: for t=0,1,…,T−1t=0,1,…,T-1 do 3: Server broadcasts global model θ(t)θ^(t) and statistics (μglobal,σglobal)( _global, _global) to all clients 4: for each client k∈1,…,Kk∈\1,…,K\ in parallel do 5: θk←θ(t) _k←θ^(t) ⊳ Initialize local model 6: for s=1,…,Ss=1,…,S do 7: Sample a batch of prompts from kD_k; generate GrolG_rol rollouts per prompt 8: Compute rewards and advantages A^i\ A_i\ for the group (Eq. (2)) 9: if μglobal≠None _global then 10: Compute calibration weights ωq\ _q\ based on local–global reward gap (Eq. (6)) 11: else 12: ωq←1 _q← 1 for all prompts 13: end if 14: Update θk _k via the calibrated GRPO loss ℒkL_k (Eq. (7)) 15: end for 16: Compute local reward mean μk _k and standard deviation σk _k over the round 17: Compute local model update Δk←θk−θ(t) _k← _k-θ^(t) 18: Determine adaptive budget ratio BkB_k (Eq. (8)) ⊳ Allocate bandwidth 19: Let Δ~k _k be the lossless sparse encoding of Δk _k 20: if Bk<1.0B_k<1.0 then 21: Further compress Δ~k _k: retain top modules by efficiency EmE_m (Eq. (9)) until we only retain a fraction BkB_k of the changed parameters 22: end if 23: Client sends compressed update Δ~k _k and reward statistics (μk,σk)( _k, _k) to server 24: end for 25: Server computes signal-weighted aggregation weights wk\w_k\ (Eq. (4)) 26: Server decompresses each Δ~k _k back to a dense delta Δk _k 27: Server applies aggregated updates: θ(t+1)=θ(t)+∑k=1KwkΔkθ^(t+1)=θ^(t)+ _k=1^Kw_k\, _k 28: Server updates global statistics μglobal _global and σglobal _global (Eq. (5)) 29: end for 30: return θ(T)θ^(T) Appendix B Proof of Theorem 1 Proof. Let θ(t)θ^(t) be the global model at the beginning of round t, and let θk(t,s) _k^(t,s) be the local model of client k after s local steps within round t, with θk(t,0)=θ(t) _k^(t,0)=θ^(t). Within round t, the one-round update of the global model can be described as the following two sub-steps: θ(t)→local update of each clientθk(t,S)k=1K→server aggregationθ(t+1).θ^(t) [local update of each client] \ _k^(t,S) \_k=1^K [server aggregation]θ^(t+1). For convenience, we denote the intermediate quantities as follows: θk(t,s+1)=θk(t,s)−ηgk(t,s), _k^(t,s+1)= _k^(t,s)-η g_k^(t,s), s=0,1,…,S−1, s=0,1,…,S-1, Δk(t)=θk(t,S)−θ(t)=−η∑s=0S−1gk(t,s), _k^(t)= _k^(t,S)-θ^(t)=-η _s=0^S-1g_k^(t,s), U(t)=θ(t)+∑k=1Kwk(t)Δk(t), U^(t)=θ^(t)+ _k=1^Kw_k^(t) _k^(t), θ(t+1)=U(t)+∑k=1Kwk(t)(Δ~k(t)−Δk(t)), θ^(t+1)=U^(t)+ _k=1^Kw_k^(t) ( _k^(t)- _k^(t) ), where gk(t,s)g_k^(t,s) is the stochastic gradient of the round-t calibrated local objective ℒk(t)L_k^(t) on a mini-batch ξk,s(t) _k,s^(t), with [gk(t,s)]=∇ℒk(t)(θk(t,s))E [g_k^(t,s) ]= _k^(t) ( _k^(t,s) ) and ‖gk(t,s)‖F2≤G2E \|g_k^(t,s) \|_F^2≤ G^2 by Assumption 2; Δ~k(t)=k(Δk(t)) _k^(t)=C_k( _k^(t)) is the compressed update produced by the adaptive sparse communication operator in Section 4.3; and wk(t)=σk(t)/∑jσj(t)w_k^(t)= _k^(t)/ _j _j^(t) is the signal-weighted aggregation weight in Eq. (4). Here, U(t)U^(t) represents the immediate result obtained after each client transmits its uncompressed local update, and θ(t+1)θ^(t+1) represents the parameter obtained after the actual compressed-communication step. Since each ℒk(t)L_k^(t) is L-smooth (Assumption 1) and ∑kwk(t)=1 _kw_k^(t)=1, the global objective ℒ(t)=∑k=1Kwk(t)ℒk(t)(θ)L^(t)= _k=1^Kw_k^(t)L_k^(t)(θ) is also L-smooth. By the smoothness of ℒ(t)L^(t), ℒ(t)(U(t))≤ℒ(t)(θ(t))+⟨U(t)−θ(t),∇ℒ(t)(θ(t))⟩F+L2‖U(t)−θ(t)‖F2.L^(t)(U^(t)) ^(t)(θ^(t))+ U^(t)-θ^(t), ^(t)(θ^(t)) _F+ L2 \|U^(t)-θ^(t) \|_F^2. (11) For the squared-norm term in Eq. equation 11, since U(t)−θ(t)=−η∑kwk(t)∑sgk(t,s)U^(t)-θ^(t)=-η _kw_k^(t) _sg_k^(t,s), by Cauchy-Schwarz inequality, ‖U(t)−θ(t)‖F2= \|U^(t)-θ^(t) \|_F^2= η2‖∑kwk(t)∑sgk(t,s)‖F2≤η2(∑k(wk(t))2)(∑k‖∑s=0S−1gk(t,s)‖F2) η^2 \| _kw_k^(t) _sg_k^(t,s) \|_F^2≤η^2 ( _k(w_k^(t))^2 ) ( _k \| _s=0^S-1g_k^(t,s) \|_F^2 ) ≤ ≤ η2(∑kwk(t))(∑kS∑s=0S−1‖gk(t,s)‖F2)≤η2S∑k=1K∑s=0S−1‖gk(t,s)‖F2, η^2 ( _kw_k^(t) ) ( _kS _s=0^S-1\|g_k^(t,s)\|_F^2 )≤η^2S _k=1^K _s=0^S-1 \|g_k^(t,s) \|_F^2, where we use the fact that wk(t)≤1w_k^(t)≤ 1 for the last second step, and ∑kwk(t)=1 _kw_k^(t)=1 for the last step. Taking expectation, ‖U(t)−θ(t)‖F2≤η2S∑k=1K∑s=0S−1‖gk(t,s)‖F2≤Kη2S2G2.E \|U^(t)-θ^(t) \|_F^2≤η^2S _k=1^K _s=0^S-1E \|g_k^(t,s) \|_F^2≤ Kη^2S^2G^2. (12) For the inner-product term in Eq. equation 11, ⟨U(t)−θ(t),∇ℒ(t)(θ(t))⟩F=−η⟨∑kwk(t)∑sgk(t,s),∇ℒ(t)(θ(t))⟩F, U^(t)-θ^(t), ^(t)(θ^(t)) _F=-η _kw_k^(t) _sg_k^(t,s), ^(t)(θ^(t)) _F, taking expectation over the stochastic gradients in current round (Denote with tE_t), and using [XY]=[X][Y]+Cov(X,Y)E[XY]=E[X]E[Y]+Cov(X,Y), we get t[⟨∑kwk(t)∑sgk(t,s),∇ℒ(t)(θ(t))⟩F] _t [ _kw_k^(t) _sg_k^(t,s), ^(t)(θ^(t)) _F ] = = ⟨∑kt[wk(t)]∑st[gk(t,s)],∇ℒ(t)(θ(t))⟩F+∑k∑s⟨Covt(wk(t),gk(t,s)),∇ℒ(t)(θ(t))⟩ _kE_t[w_k^(t)] _sE_t[g_k^(t,s)], ^(t)(θ^(t)) _F+ _k _s _t (w_k^(t),g_k^(t,s) ), ^(t)(θ^(t)) = = ∑kt[wk(t)]∑s⟨∇ℒk(t)(θk(t,s)),∇ℒ(t)(θ(t))⟩F+∑k∑s⟨Covt(wk(t),gk(t,s)),∇ℒ(t)(θ(t))⟩ _kE_t[w_k^(t)] _s _k^(t)( _k^(t,s)), ^(t)(θ^(t)) _F+ _k _s _t (w_k^(t),g_k^(t,s) ), ^(t)(θ^(t)) If we assume ‖Covt(wk(t),gk(t,s))‖F≤δt \|Cov_t (w_k^(t),g_k^(t,s) ) \|_F≤ _t for any k,sk,s, then ∑k∑s⟨Covt(wk(t),gk(t,s)),∇ℒ(t)(θ(t))⟩ _k _s _t (w_k^(t),g_k^(t,s) ), ^(t)(θ^(t)) ≥ ≥ −∑k∑s∥Covt(wk(t),gk(t,s))∥F∥∇ℒ(t)(θ(t))∥F≥−KSδt∥∇ℒ(t)(θ(t))∥F - _k _s \|Cov_t (w_k^(t),g_k^(t,s) ) \|_F \| ^(t)(θ^(t)) \|_F≥-KS _t \| ^(t)(θ^(t)) \|_F Combing the above facts, [⟨U(t)−θ(t),∇ℒ(t)(θ(t))⟩F]≤−η[∑k=1Kwk(t)∑s=0S−1⟨∇ℒ(t)(θ(t)),∇ℒk(t)(θk(t,s))⟩F] [ U^(t)-θ^(t), ^(t)(θ^(t)) _F ]≤- [ _k=1^Kw_k^(t) _s=0^S-1 ^(t)(θ^(t)), _k^(t)( _k^(t,s)) _F ] +ηKSδt[‖∇ℒ(t)(θ(t))‖F]. +η KS _tE [ \| ^(t)(θ^(t)) \|_F ]. (13) We split the gradient as ∇ℒk(t)(θk(t,s))=∇ℒk(t)(θ(t))+(∇ℒk(t)(θk(t,s))−∇ℒk(t)(θ(t))) _k^(t)( _k^(t,s))= _k^(t)(θ^(t))+ ( _k^(t)( _k^(t,s))- _k^(t)(θ^(t)) ). For the first part, summing over k with weights wk(t)w_k^(t), ∑k=1Kwk(t)⟨∇ℒ(t)(θ(t)),∇ℒk(t)(θ(t))⟩F=‖∇ℒ(t)(θ(t))‖F2. _k=1^Kw_k^(t) ^(t)(θ^(t)), _k^(t)(θ^(t)) _F= \| ^(t)(θ^(t)) \|_F^2. (14) Summing s=0,…,S−1s=0,…,S-1, this contributes S‖∇ℒ(t)(θ(t))‖F2S\| ^(t)(θ^(t))\|_F^2. For the second (drift) part, by Young’s inequality with parameter ν=1ν=1 and the L-smoothness of ℒk(t)L_k^(t), ⟨∇ℒ(t)(θ(t)),∇ℒk(t)(θk(t,s))−∇ℒk(t)(θ(t))⟩F≥−12‖∇ℒ(t)(θ(t))‖F2−L22‖θk(t,s)−θ(t)‖F2. ^(t)(θ^(t)), _k^(t)( _k^(t,s))- _k^(t)(θ^(t)) _F≥- 12 \| ^(t)(θ^(t)) \|_F^2- L^22 \| _k^(t,s)-θ^(t) \|_F^2. (15) Since wk(t)≤1w_k^(t)≤ 1 and ∑k=1Kwk(t)=1 _k=1^Kw_k^(t)=1, we know ∑k=1Kwk(t)‖θk(t,s)−θ(t)‖F2≤∑k=1K‖θk(t,s)−θ(t)‖F2,∑k=1Kwk(t)‖∇ℒ(t)(θ(t))‖F2=‖∇ℒ(t)(θ(t))‖F2. _k=1^Kw_k^(t) \| _k^(t,s)-θ^(t) \|_F^2≤ _k=1^K \| _k^(t,s)-θ^(t) \|_F^2, _k=1^Kw_k^(t) \| ^(t)(θ^(t)) \|_F^2= \| ^(t)(θ^(t)) \|_F^2. Multiplying by wk(t)w_k^(t) and summing over k, then summing over s to both sides of Eq. equation 15, we get ∑k=1Kwk(t)∑s=0S−1⟨∇ℒ(t)(θ(t)),∇ℒk(t)(θk(t,s))−∇ℒk(t)(θ(t))⟩F≥−S2‖∇ℒ(t)(θ(t))‖F2−L22∑k=1K∑s=0S−1‖θk(t,s)−θ(t)‖F2. _k=1^Kw_k^(t)\! _s=0^S-1\! ^(t)(θ^(t)), _k^(t)( _k^(t,s))- _k^(t)(θ^(t)) _F\\ ≥- S2 \| ^(t)(θ^(t)) \|_F^2- L^22\! _k=1^K\! _s=0^S-1\! \| _k^(t,s)-θ^(t) \|_F^2. Taking expectation and using ‖θk(t,s)−θ(t)‖F2≤η2s2G2E\| _k^(t,s)-θ^(t)\|_F^2≤η^2s^2G^2 (per-client analogue of Eq. equation 12), together with ∑s=0S−1s2≤S3/3 _s=0^S-1s^2≤ S^3/3, we get [∑k=1K∑s=0S−1‖θk(t,s)−θ(t)‖F2]≤Kη2S3G23.E [ _k=1^K _s=0^S-1 \| _k^(t,s)-θ^(t) \|_F^2 ]≤ Kη^2S^3G^23. (16) Then, [∑k=1Kwk(t)∑s=0S−1⟨∇ℒ(t)(θ(t)),∇ℒk(t)(θk(t,s))−∇ℒk(t)(θ(t))⟩F]≥−S2[‖∇ℒ(t)(θ(t))‖F2]−Kη2L2S3G26.E [ _k=1^Kw_k^(t)\! _s=0^S-1\! ^(t)(θ^(t)), _k^(t)( _k^(t,s))- _k^(t)(θ^(t)) _F ]\\ ≥- S2E [ \| ^(t)(θ^(t)) \|_F^2 ]- Kη^2L^2S^3G^26. Combining with Eq. equation 14 and substituting back into Eq. equation 13, we get [⟨U(t)−θ(t),∇ℒ(t)(θ(t))⟩F] [ U^(t)-θ^(t), ^(t)(θ^(t)) _F ] ≤ ≤ −ηS2[‖∇ℒ(t)(θ(t))‖F2]+Kη3L2S3G26+ηKSδt[‖∇ℒ(t)(θ(t))‖F] - η S2E [ \| ^(t)(θ^(t)) \|_F^2 ]+ Kη^3L^2S^3G^26+η KS _tE [ \| ^(t)(θ^(t)) \|_F ] ≤ ≤ −ηS4[‖∇ℒ(t)(θ(t))‖F2]+Kη3L2S3G26, - η S4E [ \| ^(t)(θ^(t)) \|_F^2 ]+ Kη^3L^2S^3G^26, (17) where we assume δt≤[‖∇ℒ(t)(θ(t))‖F]/(4K) _t [ \| ^(t)(θ^(t)) \|_F ]/(4K) to derive the last step. Plugging Eqs. (12) and (17) into Eq. (11), [ℒ(t)(U(t))]≤[ℒ(t)(θ(t))]−ηS4‖∇ℒ(t)(θ(t))‖F2+Kη3L2S3G26+KLη2S2G22.E [L^(t)(U^(t)) ] [L^(t)(θ^(t)) ]- η S4E \| ^(t)(θ^(t)) \|_F^2+ Kη^3L^2S^3G^26+ KLη^2S^2G^22. (18) By the smoothness of ℒ(t)L^(t) at U(t)U^(t), ℒ(t)(θ(t+1))≤ℒ(t)(U(t))+⟨θ(t+1)−U(t),∇ℒ(t)(U(t))⟩F+L2‖θ(t+1)−U(t)‖F2.L^(t)(θ^(t+1)) ^(t)(U^(t))+ θ^(t+1)-U^(t), ^(t)(U^(t)) _F+ L2 \|θ^(t+1)-U^(t) \|_F^2. (19) To continue the proof, we first justify Assumption 3 for our efficiency-based module selection. By construction (Section 4.3), the operator kC_k drops modules in increasing order of efficiency Em=‖Δm‖F2/‖Δm‖0E_m=\| _m\|_F^2/\| _m\|_0 until only a fraction BkB_k of the changed parameters is retained. Let E¯:=‖Δ‖F2/‖Δ‖0=∑mEm‖Δm‖0/∑m‖Δm‖0 E:=\| \|_F^2/\| \|_0= _mE_m\| _m\|_0/ _m\| _m\|_0 denote the overall efficiency averaged over all modules with weights ‖Δm‖0\| _m\|_0. Since dropped modules carry the smallest EmE_m values, their weighted average EmE_m (with the same weights ‖Δm‖0\| _m\|_0) is no larger than E¯ E, so the total dropped energy satisfies ‖k(Δ)−Δ‖F2=∑m∈droppedEm⋅‖Δm‖0≤E¯⋅(1−Bk)‖Δ‖0=(1−Bk)‖Δ‖F2, \|C_k( )- \|_F^2= _m E_m·\| _m\|_0≤ E·(1-B_k)\| \|_0=(1-B_k)\| \|_F^2, (20) which validates Assumption 3. Now, since θ(t+1)−U(t)=∑kwk(t)(Δ~k(t)−Δk(t))θ^(t+1)-U^(t)= _kw_k^(t)( _k^(t)- _k^(t)), applying Jensen’s inequality together with wk(t)≤1w_k^(t)≤ 1, we get ‖θ(t+1)−U(t)‖F2= \|θ^(t+1)-U^(t) \|_F^2= [‖∑k=1Kwk(t)(Δ~k(t)−Δk(t))‖F2]≤[∑k=1Kwk(t)‖Δ~k(t)−Δk(t)‖F2] [ \| _k=1^Kw_k^(t)( _k^(t)- _k^(t)) \|_F^2 ] [ _k=1^Kw_k^(t)\| _k^(t)- _k^(t)\|_F^2 ] ≤ ≤ [∑k=1K‖Δ~k(t)−Δk(t)‖F2]≤[∑k=1K(1−Bk(t))‖Δk(t)‖F2] [ _k=1^K\| _k^(t)- _k^(t)\|_F^2 ] [ _k=1^K(1-B_k^(t))\| _k^(t)\|_F^2 ] where we use Eq. equation 20 in the last step. Applying [XY]=[X][Y]+Cov(X,Y)E[XY]=E[X]E[Y]+Cov(X,Y), this further imples ‖θ(t+1)−U(t)‖F2=∑k=1K[(1−Bk(t))][‖Δk(t)‖F2]−∑k=1KCov(Bk(t),‖Δk(t)‖F2)≤12K(1−B¯(t))η2S2G2,E \|θ^(t+1)-U^(t) \|_F^2= _k=1^KE[(1-B_k^(t))]E[\| _k^(t)\|_F^2]- _k=1^KCov(B_k^(t),\| _k^(t)\|_F^2)≤ 12K(1- B^(t))η^2S^2G^2, (21) where B¯(t):=1K∑k=1KBk(t) B^(t):= 1K _k=1^KB_k^(t) is the per-client average budget ratio at round t. Here we used ‖Δk(t)‖F2≤η2S2G2E \| _k^(t) \|_F^2≤η^2S^2G^2 (analogue to Eq. equation 12) together with the assumption that Cov(Bk(t),‖Δk(t)‖F2)≥−12(1−B¯(t))η2S2G2Cov(B_k^(t),\| _k^(t)\|_F^2)≥- 12(1- B^(t))η^2S^2G^2. For the inner-product term in Eq. (19), we first note ∥∇ℒk(t)(θ)∥F2=∥∇ℒk(t)(θ;ξk,s(t))∥F2≤∥∇ℒk(t)(θ;ξk,s(t))∥F2≤G2 \| _k^(t)(θ) \|_F^2= \|E _k^(t)(θ; _k,s^(t)) \|_F^2 \| _k^(t)(θ; _k,s^(t)) \|_F^2≤ G^2 (22) Hence by Jensen’s inequality, ∥∇ℒ(t)(U(t))∥F2=∥∑k=1Kwk(t)∇ℒk(t)(U(t))∥F2≤∑k=1Kwk(t)∥∇ℒk(t)(U(t))∥F2≤G2. \| ^(t)(U^(t)) \|_F^2= \| _k=1^Kw_k^(t) _k^(t)(U^(t)) \|_F^2≤ _k=1^Kw_k^(t) \| _k^(t)(U^(t)) \|_F^2≤ G^2. (23) Now, by Young’s inequality with parameter η0>0 _0>0 chosen below, [⟨θ(t+1)−U(t),∇ℒ(t)(U(t))⟩F] [ θ^(t+1)-U^(t), ^(t)(U^(t)) _F ] ≤ ≤\ η0G22+12η0‖θ(t+1)−U(t)‖F2 _0G^22+ 12 _0E \|θ^(t+1)-U^(t) \|_F^2 ≤ ≤\ η0G22+K(1−B¯(t))η2S2G22η0. _0G^22+ K(1- B^(t))η^2S^2G^22 _0. (24) We choose η0=K(1−B¯(t))ηS _0= K(1- B^(t))η S (when B¯(t)<1 B^(t)<1), which is the AM–GM-optimal value that balances the two terms, making both equal to K(1−B¯(t))ηSG2/2 K(1- B^(t))η SG^2/2. (When B¯(t)=1 B^(t)=1, we have θ(t+1)=U(t)θ^(t+1)=U^(t) a.s. and Eq. (24) trivially holds.) Plugging Eqs. (21) and (24) into Eq. (19), [ℒ(t)(θ(t+1))]≤[ℒ(t)(U(t))]+K(1−B¯(t))ηSG2+LK(1−B¯(t))η2S2G24.E [L^(t)(θ^(t+1)) ] [L^(t)(U^(t)) ]+ K(1- B^(t))η SG^2+ LK(1- B^(t))η^2S^2G^24. (25) Combining Eq. (18) and Eq. (25), and rearranging, ηS4‖∇ℒ(t)(θ(t))‖F2≤ η S4E \| ^(t)(θ^(t)) \|_F^2≤ [ℒ(t)(θ(t))]−[ℒ(t)(θ(t+1))] [L^(t)(θ^(t)) ]-E [L^(t)(θ^(t+1)) ] (26) +K(1−B¯(t))ηSG2⏟leading-order residual + K(1- B^(t))η SG^2_leading-order residual +Kη3L2S3G26+KLη2S2G2(2−B¯(t))4⏟=:Φ(η)(higher-order residuals). + Kη^3L^2S^3G^26+ KLη^2S^2G^2(2- B^(t))4_=: (η)(higher-order residuals). There exists a constant M=M(L,G,S,K,ηmax)>0M=M(L,G,S,K, _ )>0 such that Φ(η)≤(Mη)⋅(ηS/4)=Mη2S/4 (η)≤(Mη)·(η S/4)=Mη^2S/4 for all η∈(0,ηmax]η∈(0, _ ], so Φ(η)/(ηS/4)≤Mη (η)/(η S/4)≤ Mη. Concretely, since Φ(η)/(η2S/4)=2KηL2S2G23+KLSG2(2−B¯(t)) (η)/(η^2S/4)= 2Kη L^2S^2G^23+KLSG^2(2- B^(t)), we can take M=2KηmaxL2S2G23+2KLSG2M= 2K _ L^2S^2G^23+2KLSG^2. The descent inequality (26) involves ℒ(t)L^(t) on both sides. To telescope across rounds, we connect ℒ(t)(θ(t+1))L^(t)(θ^(t+1)) on the LHS of round t to ℒ(t+1)(θ(t+1))L^(t+1)(θ^(t+1)) which appears on the RHS of round t+1t+1. By the definition of Δℒ(t) ^(t)_L in Assumption 4, ℒ(t)(θ(t+1))≥ℒ(t+1)(θ(t+1))−Δℒ(t).L^(t)(θ^(t+1)) ^(t+1)(θ^(t+1))- ^(t)_L. (27) Substituting Eq. (27) into Eq. (26), ηS4‖∇ℒ(t)(θ(t))‖F2≤[ℒ(t)(θ(t))]−[ℒ(t+1)(θ(t+1))]+[Δℒ(t)]+K(1−B¯(t))ηSG2+Φ(η). η S4E \| ^(t)(θ^(t)) \|_F^2 [L^(t)(θ^(t)) ]-E [L^(t+1)(θ^(t+1)) ]+E[ ^(t)_L]+ K(1- B^(t))η SG^2+ (η). (28) Before telescoping, we fix the boundary term at t=Tt=T. Define ℒ(T)L^(T) using any uniformly bounded extension of the round-T statistics (e.g., reusing the round-(T−1)(T-1) weights and calibration) so that ℒ(T)(θ(T))≥inf0≤t≤T,θℒ(t)(θ)L^(T)(θ^(T))≥ _0≤ t≤ T,\,θL^(t)(θ); the specific choice of extension does not affect the bound, since the only round-T quantity entering the telescoped sum is ρ(T−1)ρ^(T-1), which is absorbed by Assumption 4. With this convention, D≥ℒ(0)(θ(0))−inf0≤t≤T,θℒ(t)(θ)≥[ℒ(0)(θ(0))]−[ℒ(T)(θ(T))]D ^(0)(θ^(0))- _0≤ t≤ T,\,θL^(t)(θ) [L^(0)(θ^(0))]-E[L^(T)(θ^(T))]. Summing Eq. (28) over t=0,1,…,T−1t=0,1,…,T-1, and using the bound on D above together with ∑t[Δℒ(t)]≤RT _tE[ ^(t)_L]≤ RT from Assumption 4 and Jensen’s inequality 1T∑t1−B¯(t)≤1−B¯ 1T _t 1- B^(t)≤ 1- B over rounds (where B¯=1T∑tB¯(t) B= 1T _t B^(t) is the budget averaged over both clients and rounds), ηS4∑t=0T−1‖∇ℒ(t)(θ(t))‖F2≤D+RT+T⋅[K(1−B¯)ηSG2+Φ(η)]. η S4 _t=0^T-1E \| ^(t)(θ^(t)) \|_F^2≤ D+RT+T· [ K(1- B)η SG^2+ (η) ]. (29) Dividing both sides by ηST/4η ST/4 (and using Φ(η)/(ηS/4)≤Mη (η)/(η S/4)≤ Mη), 1T∑t=0T−1‖∇ℒ(t)(θ(t))‖F2≤4DηST+4RηS+4K(1−B¯)G2+Mη. 1T _t=0^T-1E \| ^(t)(θ^(t)) \|_F^2≤ 4Dη ST+ 4Rη S+4 K(1- B)G^2+Mη. (30) Setting η=2(D/T+R)/(MS)η= 2(D/T+R)/(MS) to optimally balance the η-dependent terms (i.e., 2(D/T+R)ηS 2(D/T+R)η S against MηMη) via AM–GM, 1T∑t=0T−1‖∇ℒ(t)(θ(t))‖F2≤22M(D/T+R)S+4K(1−B¯)G2. 1T _t=0^T-1E \| ^(t)(θ^(t)) \|_F^2≤ 2 2M(D/T+R)S+4 K(1- B)G^2. (31) Rescaling the constant M to absorb the factor S/2S/2 (i.e., M′=2M/SM =2M/S, and renaming M′→M → M), and using a+b≤a+b a+b≤ a+ b for a,b≥0a,b≥ 0, we obtain the form stated in Eq. (10): 1T∑t=0T−1‖∇ℒ(t)(θ(t))‖F2≤2DMT+2RM+4K(1−B¯)G2. 1T _t=0^T-1E \| ^(t)(θ^(t)) \|_F^2≤ 2 DMT+2 RM+4 K(1- B)G^2. (32) ∎ Appendix C Experimental Details Datasets. For mathematical reasoning tasks, we use the MATH dataset Hendrycks et al. (2021), which contains approximately 7,500 competition-level math problems spanning 5 difficulty levels. To construct a naturally non-IID federated partition, we split the training set across K=5K=5 clients by difficulty level, so that each client specializes in problems of a single difficulty. For evaluation, we report accuracy on five widely-used math reasoning benchmarks: MATH-500 Hendrycks et al. (2021); Lightman et al. (2023), GSM8K Cobbe et al. (2021), AIME 2024, AIME 2025, and AMC 2023. For coding tasks, we train on the MBPP dataset Austin et al. (2021) (”train” + ”validation”) and evaluate on the held-out MBPP ”test” split and the HumanEval dataset Chen et al. (2021). To construct a non-IID federation setting, we partition the MBPP dataset by the reference-solution length, bucket it into 5 quantile bins, and partition the data across clients with a Dirichlet distribution (α=0.5α=0.5) over these buckets. Baselines. We compare Fed-GRPO against several baselines that represent different points in the centralized–federated spectrum: i) Base Model: base model without any training, serving as base performance; i) Centralized: standard GRPO training on the entire MATH dataset, serving as an upper bound for federated methods; i) Local-Only: each client independently runs GRPO on its local data without any communication or aggregation; iv) FedAvg-SW / FedAvg-UW: standard FedAvg McMahan et al. (2017) with GRPO as the local training algorithm. SW uses sample-weighted aggregation (wk=nk/Nw_k=n_k/N), while UW uses uniform weights (wk=1/Kw_k=1/K); v) SparsyFed Guastella et al. (2025) and SparseLoCo Sarfi et al. (2025): communication-effecient federated learning methods that apply fixed-ratio top-k sparsification to the model updates. vi) FGRPO Chen et al. (2026): incorporates an adaptive aggregation mechanism based on relative performance gain, characterizing each client’s improvement relative to its personalized historical baseline to dynamically prioritize effective learning trajectories regardless of local task difficulty. Training details. We use Qwen2.5-3B-Instruct Yang et al. (2025b) and Qwen3-4B-Instruct Yang et al. (2025a) as the base model for all methods. We use the AdamW Loshchilov and Hutter (2019) optimizer with learning rate 3×10−63× 10^-6, β1=0.9 _1=0.9, β2=0.999 _2=0.999, weight decay 00, a constant learning-rate schedule with warmup ratio 0.10.1, and gradient clipping at 1.01.0 for GRPO training. For rollout, the prompt batch size is 64 and we sample 8 responses for each prompt at sampling temperature 1.01.0 and top-p=1.0p=1.0. The clip ratio is set to ϵ=0.2ε=0.2, KL coefficient βKL=0 _KL=0, and we use BF16 mixed precision. The maximum prompt and completion lengths are capped at 10241024 and 20482048 tokens, respectively. The reward function is a binary accuracy reward that checks whether the extracted answer matches the ground truth or whether the code is right or not. All federated methods run for T=12T=12 communication rounds with S=5S=5 local steps per round. Centralized and local-only baselines are trained for up to 310310 steps (matching the total number of gradient updates in the federated setting). For Fed-GRPO, we set the global reward calibration strength λ=0.5λ=0.5; for adaptive sparse communication, we vary the budget B to sweep the compression–accuracy trade-off, with minimum budget Bmin=0.01B_ =0.01. All experiments are implemented on top of TRL von Werra et al. (2020) library and run on 88 NVIDIA H800 GPUs. Evaluation protocol. We save a model checkpoint at the end of each communication round for federated methods, and every 1010 training steps for centralized and local-only baselines, then evaluate each checkpoint on all five benchmarks. Following common practice in GRPO literature Yu et al. (2025); Guo et al. (2025a), we report the best accuracy achieved across checkpoints. For inference, we sample with temperature 1.01.0, top-p=0.7p=0.7, and a maximum generation length of 40964096 tokens; we generate 3232 samples per prompt and report avg@3232 (the mean accuracy over the 3232 samples) for results stability. Appendix D More Experimental Results D.1 Results on Different Data Heterogeneity We further evaluate the performance of Fed-GRPO under different data heterogeneity settings. Specifically, we partition the MATH dataset by ”subject” (Algebra, Geometry, Number Theory, Combinatorics) using a Dirichlet distribution with α=0.5α=0.5 to create a non-IID federation across K=20K=20 clients. The results are summarized in Table 5. We observe that Fed-GRPO consistently outperforms all baselines across all benchmarks, demonstrating its robustness and effectiveness in handling heterogeneous data distributions. Table 5: Results on five math reasoning benchmarks using Qwen2.5-3B-Instruct as the base model under dirichlet partitioning (α=0.5) by ”subject” with K=20 clients. Method GSM8K MATH-500 AIME24 AIME25 AMC Avg. Qwen2.5-3B-Instruct 59.47 46.54 2.50 1.46 27.11 27.42 Centralized 84.75 63.42 7.50 3.44 41.95 40.21 Local-Only 79.60 53.09 4.43 2.26 33.57 34.59 FedAvg-SW 81.32 56.04 5.04 2.72 35.12 36.05 FedAvg-UW 80.79 56.23 4.97 2.66 34.86 35.90 SparsyFed 80.96 56.02 5.02 2.53 35.12 35.93 SparseLoCo 81.02 55.96 4.89 2.55 34.97 35.22 FGRPO 81.43 56.77 5.48 2.76 38.10 36.91 Fed-GRPO (Ours) 82.02 58.77 6.18 3.02 39.11 38.02 D.2 Dynamics of Signal-Weighted Aggregation (a) (b) Figure 3: Dynamics of (a) per-client reward standard deviation σk _k and (b) aggregation weights wkw_k. The aggregation weights shift dynamically across rounds, concentrating on clients with stronger learning signals (higher σk _k) as training progresses. To understand how signal-weighted aggregation adapts during training, we track two quantities across 12 communication rounds on the MATH dataset using Qwen2.5-3B-Instruct as the base model: the per-client reward standard deviation σk _k (Figure 3(a)) and the resulting aggregation weights wkw_k (Figure 3(b)). As shown in Figure (3(a)), σk _k evolves differently for each client, reflecting their varying learning signals as training progresses. Some clients experience an increase in σk _k, indicating stronger learning signals, while others see a decrease in σk _k as they master their local data. Figure (3(b)) shows that the aggregation weights wkw_k adapt accordingly, shifting more weight toward clients with higher σk _k. This dynamic weighting allows Fed-GRPO to prioritize updates from clients that are currently contributing more to global improvement, effectively mitigating the aggregation mismatch issue in federated GRPO training. The fact that the weights change across rounds also suggests that the method is responsive to the evolving training dynamics, rather than relying on static heuristics. Overall, this analysis confirms that signal-weighted aggregation successfully captures and leverages the varying learning signals across clients to improve the quality of the global update.