Paper deep dive
Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation
Md Rafid Islam, Rafsan Jany, Zahid Hasan, Ratun Rahman
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/13/2026, 5:50:54 AM
Summary
The paper introduces EDPFRL-IM, an exploration-driven personalized federated reinforcement learning framework that uses intrinsic motivation (Random Network Distillation) to promote local exploration while preserving privacy. It coordinates clients via a server that aggregates minimal novelty summaries to broadcast global exploration priors, improving sample efficiency and policy personalization in sparse-reward environments compared to baselines like FedRL and FedAvg-RL.
Entities (10)
Relation Signals (9)
EDPFRL-IM → uses → RND
confidence 95% · clients add an intrinsic random network distillation (RND) signal to their extrinsic reward
RND → generates → Intrinsic Reward
confidence 92% · The intrinsic reward is the prediction error on the next state
EDPFRL-IM → outperforms → FedRL
confidence 92% · EDPFRL-IM outperformed all baselines (Local RL, FedRL, and FedRL+RND) consistently
Server → aggregates → Global Novelty Prior
confidence 90% · The server aggregates these statistics to form a global exploration profile
Server → broadcasts → Global Novelty Prior
confidence 90% · This prior is broadcast to all clients to guide their exploration
EDPFRL-IM → evaluatedon → MountainCar-v0
confidence 90% · We test the performance of EDPFRL-IM in standard reinforcement learning environments ... MountainCar-v0
EDPFRL-IM → evaluatedon → CartPole-sparse
confidence 90% · We test the performance of EDPFRL-IM in standard reinforcement learning environments ... CartPole-sparse
EDPFRL-IM → outperforms → pFedMe
confidence 90% · EDPFRL-IM outperformed all comparison methods, including ... pFedMe
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Personalized Federated Reinforcement Learning (PFRL) takes a decentralized approach to storing and accessing information based on past experiences while keeping each client's data private during the learning of each client's policy. Many current methods for PFRL rely heavily on exploiting existing reinforcement learning reward signals to derive an optimal policy for each client, thereby neglecting exploration in non-stationary or sparse-reward environments. In this work, we introduce a new exploration-driven framework, Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation (EDPFRL-IM), that leverages an inherent curiosity-driven exploration at each client to promote local exploration and protect client privacy. Furthermore, to facilitate policy discovery via exploration in previously unexplored state spaces, clients add an intrinsic random network distillation (RND) signal to their extrinsic reward. Additionally, the server does not have access to clients' raw experiences or local gradient estimates; instead, the server sends global exploration priors and collects minimal novelty summaries from each client to enable both diverse and coordinated exploration among clients. Experiments in benchmark environments show that our framework outperforms average PFRL benchmarks in policy personalization and sample efficiency, primarily in delayed and sparse reward systems. Overall, EDPFRL-IM enables the integration of a flexible exploratory learning structure into federated reinforcement learning systems while preserving client privacy.
Tags
Links
- Source: https://arxiv.org/abs/2608.10499v1
- Canonical: https://arxiv.org/abs/2608.10499v1
Trouble viewing inline? Open PDF directly →
Full Text
23,061 characters extracted from source content.
Expand or collapse full text
Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation Md Rafid Islam Rafsan Jany Zahid Hasan Ratun Rahman Abstract Personalized Federated Reinforcement Learning (PFRL) takes a decentralized approach to storing and accessing information based on past experiences while keeping each client’s data private during the learning of each client’s policy. Many current methods for PFRL rely heavily on exploiting existing reinforcement learning reward signals to derive an optimal policy for each client, thereby neglecting exploration in non-stationary or sparse-reward environments. In this work, we introduce a new exploration-driven framework, Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation (EDPFRL-IM), that leverages an inherent curiosity-driven exploration at each client to promote local exploration and protect client privacy. Furthermore, to facilitate policy discovery via exploration in previously unexplored state spaces, clients add an intrinsic random network distillation (RND) signal to their extrinsic reward. Additionally, the server does not have access to clients’ raw experiences or local gradient estimates; instead, the server sends global exploration priors and collects minimal novelty summaries from each client to enable both diverse and coordinated exploration among clients. Experiments in benchmark environments show that our framework outperforms average PFRL benchmarks in policy personalization and sample efficiency, primarily in delayed and sparse reward systems. Overall, EDPFRL-IM enables the integration of a flexible exploratory learning structure into federated reinforcement learning systems while preserving client privacy. I Introduction Reinforcement learning (RL) is a powerful framework for sequential decision-making that enables agents to learn optimal behaviors through interaction with their environment [9]. Federated reinforcement learning (FRL) enables multiple clients to cooperatively train reinforcement learning (RL) agents in decentralized settings without exchanging raw experience data. Personalized FRL (PFRL) expands on this by customizing policies to each client’s particular environment and objectives. This makes it suitable for real-world applications such as health monitoring and robotics [4, 8]. Although existing PFRL frameworks [4, 11] focus on policy optimization and aggregation, they often overlook the role of exploration under limited or delayed rewards. Standard RL strategies [9] like ϵε-greedy or entropy regularization become less effective in federated settings, where clients operate independently and face privacy constraints that limit coordination. As a result, clients may learn suboptimal policies, particularly in cold-start or non-stationary conditions. While approaches like FedRL [4], FedAvg-RL [11], and pFedMe [10] address personalization or aggregation, they assume homogeneous or supervised settings. Intrinsic motivation methods such as RND [7], ICM [3], and count-based exploration [6] show promise for centralized RL but remain underexplored in federated scenarios. Addressing this gap is key to unlocking more adaptive, intelligent on-device learning. We propose the EDPFRL-IM framework, which integrates intrinsic curiosity-driven motivation into personalized federated reinforcement learning. It not only enables efficient local exploration with sparse rewards but also preserves privacy and minimizes communication. We summarize our key contributions as follows. • We introduce EDPFRL-IM, a novel FRL framework that incorporates intrinsic motivation for personalized exploration in a decentralized setting. • We develop a communication-efficient protocol for compressing exploration statistics, enabling clients to align their curiosity-driven behavior while avoiding disclosure of raw data or policies. • We empirically evaluate our approach in sparse-reward benchmark environments and demonstrate significant improvements in personalization, exploration efficiency, and policy performance over existing federated RL baselines. I Methodology I-A Problem Formulation We consider a PFRL setting with N clients, each interacting with a local Markov decision process (MDP) [5] ℳi=(i,,i,Ri,γ)M_i=(S_i,A,P_i,R_i,γ), where iS_i and RiR_i are client-specific, while A and γ are shared. Due to heterogeneity in transition dynamics and rewards, each client learns a personalized policy πi(a|s) _i(a|s) to maximize its expected return: Ji(πi)=πi[∑t=0∞γtri,t].J_i( _i)=E_ _i [ _t=0^∞γ^tr_i,t ]. (1) Clients update their policy parameters via gradient ascent: θi←θi+η∇θiJi(πθi), _i← _i+η _ _iJ_i( _ _i), (2) with learning rate η. Unlike global FL or RL, PFRL optimizes individual client policies while enabling collaboration through lightweight, privacy-preserving exploration statistics. I-B Intrinsic Motivation for Local Exploration Effective exploration is crucial in PFRL due to sparse or delayed rewards. We adopt RND, a self-supervised method in which each client uses a fixed random target network ftgtf_tgt and a trainable predictor fpredf_pred. The intrinsic reward is the prediction error on the next state st+1s_t+1: rtint=‖ftgt(st+1)−fpred(st+1)‖22.r_t^int= \|f_tgt(s_t+1)-f_pred(s_t+1) \|_2^2. (3) This promotes the exploration of novel states. The total reward combines extrinsic and intrinsic terms as rttotal=rtext+αi⋅rtint,r_t^total=r_t^ext+ _i· r_t^int, (4) where αi _i controls the exploration-exploitation trade-off per client. This local RND setup enables adaptive exploration without external shaping or manual tuning. I-C Federated Coordination with Exploration Summaries Uncoordinated exploration can lead to redundant efforts across clients. To enable efficient collaboration, we propose a federated privacy-preserving coordination mechanism. Each client generates compact exploration statistics ℰiE_i, such as visitation counts, frequency histograms over state embeddings, or top-k novel state hashes based on intrinsic rewards. These summaries are low-dimensional and protect sensitive data. The central server aggregates these statistics to form a global exploration profile as =Aggregate(ℰii=1N),G=Aggregate(\E_i\_i=1^N), (5) where Aggregate(⋅)Aggregate(·) combines client contributions, e.g., summing visitation counts for the state cluster k as vglobal[k]=∑i=1Nvi[k].v_global[k]= _i=1^Nv_i[k]. (6) The server then computes a global novelty prior to identifying underexplored states: novel(s)=11+vglobal[cluster(s)],P_novel(s)= 11+v_global[cluster(s)], (7) where cluster(s)cluster(s) maps the state s to a cluster index. This prior is broadcast to all clients to guide their exploration. Each client incorporates the global novelty prior into its policy update as θi←θi+η∇θiπθi,pi(s)[∑t=0∞γt(ri,text+αiri,tint)], _i← _i+η _ _iE_ _ _i,p_i(s) [ _t=0^∞γ^t(r_i,t^ext+ _ir_i,t^int) ], (8) where the probability of sampling state s from the experience buffer of client i, iD_i, is expressed as pi(s)∝(1+β⋅novel(s))⋅piuniform(s),p_i(s) (1+β·P_novel(s) )· p^uniform_i(s), (9) with piuniform(s)=1/|i|p^uniform_i(s)=1/|D_i| as the uniform baseline probability and β≥0β≥ 0 controlling the influence of the global prior. This approach ensures that clients focus on globally novel states, promoting diverse exploration while maintaining privacy and low communication costs. Algorithm 1 describes the EDPFRL-IM method. Each client gathers experience and computes intrinsic rewards using RND (Lines 5–7), then adjusts its local policy (Line 8). Clients transmit compressed exploration summaries to the server (Line 10), which aggregates them into a global novelty prior (Line 12) and broadcasts it (Line 13). Clients use this prior for biased experience sampling (Line 15), which allows for coordinated exploration without compromising privacy. An overview of our framework is shown in Fig. 1. Algorithm 1 EDPFRL-IM: Exploration-Driven Personalized Federated Reinforcement Learning 0: Clients Cii=1N\C_i\_i=1^N, global rounds T, local epochs E, RND module (ftgt,fpred)(f_tgt,f_pred), exploration weight αi _i, novelty bias β 1: for each global round t=1t=1 to T do 2: for each client CiC_i in parallel do 3: Initialize local buffer iD_i 4: for local step e=1e=1 to E do 5: Interact with environment to collect (st,at,rtext,st+1)(s_t,a_t,r_t^ext,s_t+1) 6: Compute intrinsic reward: rtint=‖ftgt(st+1)−fpred(st+1)‖22r_t^int= \|f_tgt(s_t+1)-f_pred(s_t+1) \|_2^2 7: Store (st,at,rttotal,st+1)(s_t,a_t,r_t^total,s_t+1) in iD_i, where rttotal=rtext+αi⋅rtintr_t^total=r_t^ext+ _i· r_t^int 8: Update policy πi _i using local RL optimizer (e.g., PPO, SAC) 9: end for 10: Compute exploration summary (e.g., histogram, top-k novel states) 11: Send summary to server 12: end for 13: Server aggregates summaries into global novelty prior novelP_novel 14: Broadcast novelP_novel to all clients 15: for each client CiC_i do 16: Update sampling strategy: pi(s)∝(1+β⋅novel(s))⋅piuniform(s)p_i(s) (1+β·P_novel(s))· p^uniform_i(s) 17: end for 18: end for Figure 1: An overview of the proposed EDPFRL-IM framework. Based on local experience, each client uses intrinsic incentives to do reinforcement learning (via RND). Clients provide compressed exploration summaries to a central server regularly, which are then aggregated into a global novelty prior. This prior is transmitted back to guide tailored exploration, allowing clients to explore together while maintaining their private information. I Experiments I-A Simulation Setup Environments: We test the performance of EDPFRL-IM in standard reinforcement learning environments with sparse rewards and heterogeneous dynamics. We employ modified versions of MountainCar-v0 [2] and CartPole-sparse [1], which are chosen for their sensitivity to exploration quality and their significance in comparing sample efficiency. Federated Setup: We simulate a realistic FRL scenario by considering N=10N=10 clients, each running in a different local environment built from MountainCar-v0 and a modified CartPole environment with sparse rewards. We create heterogeneity among clients by adjusting the elements of the environment, including gravity, friction, and reward shaping. For instance, in MountainCar-v0, clients earn additional shaping incentives based on distance or velocity, and gravity varies from 0.00250.0025 to 0.0060.006. In CartPole-sparse, clients vary in pole mass and cart friction, and earn a reward only after balancing for a minimum duration. As an example, one client’s MountainCar-v0 variation employs gravity 0.00450.0045, friction 0.0050.005, and a shaped reward of rt=−1+0.1⋅|x−xgoal|r_t=-1+0.1·|x-x_goal| to promote goal-directed travel. These differences lead to different MDPs, indicating extremely non-IID conditions where effective policy learning requires personalization. Baseline Comparison: Training proceeds for T=100T=100 global communication rounds, with each client performing E=10E=10 local PPO updates per round. Policies are represented by fully connected two-layer neural networks with 64 hidden units and ReLU activations. Each client has its own Random Network Distillation (RND) module, with both the target and predictor networks consisting of a single hidden layer of size 128. The intrinsic reward coefficient αi _i is set to 0.1, while the novelty sampling bias β is set to 0.5, unless specified otherwise. Implementation Details: We compare our approach with the following baselines: (i) Local RL, in which every client trains on its own without any federated coordination; (i) FedRL, a common FRL method with global policy aggregation; and (i) FedRL+RND, which applies intrinsic rewards locally without exploration coordination. For the broader comparison in Table I, we additionally report single-agent intrinsic-motivation methods run in the non-federated setting (RND [7], ICM [3], and count-based exploration (CBE) [6]), as well as federated and personalized FL baselines without coordinated exploration (FedAvg-RL [11], pFedMe [10], and FedPer++ [12]). All baselines are trained under the same environment configurations, client counts, and total communication budget as EDPFRL-IM for a fair comparison. The Adam optimizer is used to train all models, with a discount factor of γ=0.99γ=0.99 and a learning rate of 3×10−43× 10^-4. I-B Simulation Results Main Performance Comparison: Fig. 2 shows the average return in MountainCar-v0 (left) versus CartPole-sparse (right) across communication rounds. EDPFRL-IM outperformed all baselines (Local RL, FedRL, and FedRL+RND) consistently throughout the initial rounds, with significantly higher returns due to the efficient use of available samples. In MountainCar-v0, EDPFRL-IM achieved quick convergence with a smooth learning curve despite the presence of sparse rewards and difficult state transition dynamics. In CartPole-sparse, EDPFRL-IM had a similar advantage over the other methods. Further, when comparing EDPFRL-IM to FedRL+RND, EDPFRL-IM produced greater average returns and exhibited a more consistent rate of convergence, supporting the claim that coordinated exploration through aggregated novelty priors contributed to superior performance across the range of tested conditions. Figure 2: Performance comparison of reinforcement learning methods across two environments over 100 communication rounds. The left subplot shows the average return for MountainCar-v0, and the right subplot shows the average return for CartPole-sparse with four baseline methods. Personalization Benefit: Table I presents the mean returns for each client from both MountainCar-v0 and CartPole-sparse, with EDPFRL-IM producing greater performance than the FedRL baseline across all clients. The consistent gains highlight the benefit of combining intrinsic motivation with personalization. Additionally, the lower performance variance resulting from EDPFRL-IM allows for greater consistency and personalization of policy learning, which is critical across a wide range of diverse conditions. TABLE I: Average return per client for EDPFRL-IM and FedRL across two environments. EDPFRL-IM consistently outperforms FedRL in all clients. Method C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 MountainCar-v0 EDPFRL-IM 0.72 0.68 0.70 0.66 0.69 0.74 0.71 0.73 0.75 0.70 FedRL 0.42 0.39 0.40 0.38 0.41 0.44 0.40 0.42 0.43 0.41 CartPole-sparse EDPFRL-IM 0.68 0.65 0.70 0.66 0.67 0.71 0.69 0.72 0.70 0.68 FedRL 0.36 0.34 0.35 0.33 0.37 0.38 0.36 0.35 0.37 0.36 Ablation: Role of Coordinated Exploration: Table I shows an ablation study that compares the impact of exploration coordination in our proposed strategy. When intrinsic motivation is completely removed, as with the FedRL implementation, agent performance is low due to the lack of exploratory behavior. Adding only the intrinsic reward component (FedRL+RND) provides some improvement but lacks clear directionality (i.e., no consensus on a path). Removing the global novelty prior from EDPFRL-IM (i.e., β=0β=0) improves performance slightly through individualized customization; however, performance is still substantially below the full method. EDPFRL-IM achieves the best returns in both environments when coordinated exploratory behavior is used (i.e., β=0.5β=0.5), demonstrating the value of leveraging cross-client novelty to guide exploration in federated reinforcement learning settings. TABLE I: Ablation study: average return over the final 10 communication rounds (rounds 91–100). Coordinated exploration (EDPFRL-IM) leads to the best performance across environments. Method Variant MountainCar-v0 CartPole-sparse FedRL (no RND) 0.40 0.35 FedRL+RND (no coordination) 0.60 0.54 EDPFRL-IM (β = 0) 0.63 0.58 EDPFRL-IM (ours) 0.80 0.76 Exploration Coverage Analysis: As illustrated in Fig. 3, clients’ exploration coverage (percentage of the state space that clients visit) in both environments is much greater when using EDPFRL-IM compared to using FedRL and FedRL+RND. This indicates that the coordinated use of intrinsic motivation between clients is crucial for increasing the diversity of exploration. Increased diversity in exploration provides for greater generalization and faster adaptation in heterogeneous environments. Figure 3: Exploration coverage across clients in two environments. EDPFRL-IM enables broader state-space exploration by coordinating curiosity across clients. Robustness to Cold Start Clients: We assessed the adaptability of the three methods to the introduction of a cold-start client during training (at round 5) by measuring how quickly and how well the cold-start client performed in both environments, as shown in Fig. 4. EDPFRL-IM uses globally aggregated exploration knowledge to facilitate rapid and smooth adaptation for cold-start clients in both environments, whereas FedRL and FedRL+RND do not use this form of globally aggregated knowledge, which negatively affects their ability to support early-stage learners, resulting in slower progress. Figure 4: Return trajectories of a cold-start client introduced at round 5 in (a) MountainCar-v0 and (b) CartPole-sparse. EDPFRL-IM quickly adapts by leveraging the global exploration prior, outperforming both FedRL and FedRL+RND. Comparison with Existing Approaches: Table I presents the final average return achieved by several federated and personalized reinforcement learning algorithms. EDPFRL-IM outperformed all comparison methods, including RND, ICM, CBE, FedAvg-RL, pFedMe, FedPer++, and FedRL+RND, in both environments. Although approaches such as pFedMe and FedPer++ support personalization of RL models, they provide limited exploration capabilities for discovering new behaviors. FedRL+RND incorporates intrinsic motivation through curiosity-driven exploration but lacks an efficient mechanism for personalized federated adaptation across clients. The proposed EDPFRL-IM framework combines structured curiosity-driven exploration with federated personalization, yielding significantly improved performance and learning efficiency compared with existing methods. TABLE I: Final Average Return Comparison Across Methods Method MountainCar-v0 CartPole-sparse RND [7] 0.31 0.32 ICM [3] 0.34 0.36 CBE [6] 0.39 0.49 FedAvg-RL [11] 0.38 0.42 pFedMe [10] 0.40 0.45 FedPer++ [12] 0.41 0.44 FedRL+RND [4, 7] 0.48 0.52 EDPFRL-IM (Ours) 0.76 0.74 IV Conclusion This paper presents EDPFRL-IM, an exploration-driven personalized federated reinforcement learning (PFRL) system that leverages intrinsic motivation to provide coordinated yet private client exploration. EDPFRL-IM addresses an important shortcoming of prior, less effective PFRL systems by incorporating a global novelty prior computed from a compressed set of client exploration statistics, enabling each client to learn while adapting to its local context and benefiting from the collective experience of the remaining clients. Comprehensive evaluation of EDPFRL-IM in environments with sparse reward structures and high variability demonstrates that it achieves significant improvements in sample efficiency and personalization, and adapts faster for cold-start clients than its peers. EDPFRL-IM consistently outperforms baseline systems using both standard and personalized FL algorithms, including various intrinsic-incentive-based FL algorithms, and does so with no compromise in terms of privacy. References [1] N. Bjorck, C. P. Gomes, and K. Q. Weinberger (2021) Towards deeper deep reinforcement learning with spectral normalization. Advances in neural information processing systems 34, p. 8242–8255. Cited by: §I-A. [2] S. T. Chavali, C. T. Kandavalli, T. Sugash, and J. Amudha (2022) Modelling a reinforcement learning agent for mountain car problem using q–learning with tabular discretization. In 2022 IEEE 2nd Mysore Sub Section International Conference (MysuruCon), p. 1–5. Cited by: §I-A. [3] C. Colas, P. Fournier, M. Chetouani, O. Sigaud, and P. Oudeyer (2019) Curious: intrinsically motivated modular multi-goal reinforcement learning. In International conference on machine learning, p. 1331–1340. Cited by: §I, §I-A, TABLE I. [4] X. Fan, Y. Ma, Z. Dai, W. Jing, C. Tan, and B. K. H. Low (2021) Fault-tolerant federated reinforcement learning with theoretical guarantee. Advances in neural information processing systems 34, p. 1007–1021. Cited by: §I, §I, TABLE I. [5] M. Lauri, D. Hsu, and J. Pajarinen (2022) Partially observable markov decision processes in robotics: a survey. IEEE Transactions on Robotics 39 (1), p. 21–40. Cited by: §I-A. [6] M. C. Machado, M. G. Bellemare, and M. Bowling (2020) Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, p. 5125–5133. Cited by: §I, §I-A, TABLE I. [7] A. Nikulin, V. Kurenkov, D. Tarasov, and S. Kolesnikov (2023) Anti-exploration by random network distillation. In International conference on machine learning, p. 26228–26244. Cited by: §I, §I-A, TABLE I, TABLE I. [8] R. Rahman and D. C. Nguyen (2024) Improved modulation recognition using personalized federated learning. IEEE Transactions on Vehicular Technology. Cited by: §I. [9] A. K. Shakya, G. Pillai, and S. Chakrabarty (2023) Reinforcement learning algorithms: a brief survey. Expert Systems with Applications 231, p. 120495. Cited by: §I, §I. [10] C. T Dinh, N. Tran, and J. Nguyen (2020) Personalized federated learning with moreau envelopes. Advances in neural information processing systems 33, p. 21394–21405. Cited by: §I, §I-A, TABLE I. [11] H. Wang, Z. Kaplan, D. Niu, and B. Li (2020) Optimizing federated learning on non-iid data with reinforcement learning. In IEEE INFOCOM 2020-IEEE conference on computer communications, p. 1698–1707. Cited by: §I, §I-A, TABLE I. [12] J. Xu, Y. Yan, and S. Huang (2022) FedPer++: toward improved personalized federated learning on heterogeneous and imbalanced data. In 2022 International Joint Conference on Neural Networks (IJCNN), p. 01–08. Cited by: §I-A, TABLE I.