Paper deep dive
Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation
Yuheng Huang, Yunpeng Qing, Yixiao Chi, Yilun Kong, Changqing Zou
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Offline-to-Online Reinforcement Learning (O2O RL) has emerged as a practical paradigm that pre-trains the policy using static offline datasets and subsequently adapts the policy through online interactions. Existing O2O methods primarily address the transition through value calibration, while generally treating the offline-trained policy as a given initialization. We instead study O2O adaptation from the perspective of network plasticity, asking whether the offline-trained policy remains sufficiently adaptable for online learning. Controlled experiments show that prolonged optimization on static offline data progressively reduces network plasticity even after offline performance has largely saturated, and that lower plasticity is associated with weaker subsequent online improvement. Motivated by these observations, we propose REstoring plasticity via Fresh Initialization and policy Transfer (REFIT), a lightweight model-level method for the O2O transition. Before online fine-tuning, REFIT distills the offline policy into a freshly initialized student while temporarily freezing a random subset of student units, transferring the learned offline behavior to a more plastic policy initialization. Extensive experiments on D4RL and OGBench demonstrate that REFIT consistently achieves higher aggregate performance than existing O2O plug-in methods across both Cal-QL and IQL backbones, while plasticity diagnostics and ablations provide further evidence of restored network plasticity.
Tags
Links
- Source: https://arxiv.org/abs/2609.33127v1
- Canonical: https://arxiv.org/abs/2609.33127v1
Trouble viewing inline? Open PDF directly →
Full Text
61,188 characters extracted from source content.
Expand or collapse full text
Preprint POLICY PLASTICITY MATTERS IN OFFLINE-TO-ONLINE REINFORCEMENT LEARNING: REFITTING OFFLINE POLICIES FOR ONLINE ADAPTATION Yuheng Huang ∗ Zhejiang University Yunpeng Qing ∗† Zhejiang University Yixiao Chi Carnegie Mellon University Yilun Kong Nanyang Technological University Changqing Zou ‡ Zhejiang University & Zhejiang Lab ABSTRACT Offline-to-Online Reinforcement Learning (O2O RL) has emerged as a practical paradigm that pre-trains the policy using static offline datasets and subsequently adapts the policy through online interactions. Existing O2O methods primarily ad- dress the transition through value calibration, while generally treating the offline- trained policy as a given initialization. We instead study O2O adaptation from the perspective of network plasticity, asking whether the offline-trained policy re- mains sufficiently adaptable for online learning. Controlled experiments show that prolonged optimization on static offline data progressively reduces network plas- ticity even after offline performance has largely saturated, and that lower plastic- ity is associated with weaker subsequent online improvement. Motivated by these observations, we propose REstoring plasticity via Fresh Initialization and policy Transfer (REFIT), a lightweight model-level method for the O2O transition. Be- fore online fine-tuning, REFIT distills the offline policy into a freshly initialized student while temporarily freezing a random subset of student units, transferring the learned offline behavior to a more plastic policy initialization. Extensive ex- periments on D4RL and OGBench demonstrate that REFIT consistently achieves higher aggregate performance than existing O2O plug-in methods across both Cal- QL and IQL backbones, while plasticity diagnostics and ablations provide further evidence of restored network plasticity. 1INTRODUCTION Deep Reinforcement Learning (DRL) has achieved strong performance across diverse real-world applications, such as robot control (Qing et al., 2024; 2025) and power grid control (Xu et al., 2024; Chen et al., 2025). To learn such effective policies, DRL typically relies on either collecting experience through online interaction or leveraging previously collected datasets. However, online interaction can be costly and unsafe (Lillicrap et al., 2015; Haarnoja et al., 2018), while learning solely from fixed datasets is limited by insufficient data distribution coverage and the challenge of out-of-distribution (OOD) actions (Fujimoto et al., 2019; Kostrikov et al., 2021; Kumar et al., 2020b). Offline-to-Online (O2O) RL bridges these two paradigms by pre-training on offline data and subsequently improving the policy through online interaction, thereby reducing the reliance on costly exploration and enabling adaptation beyond the coverage of the offline data (Zhou et al., 2025b; Nakamoto et al., 2023; Zhang et al., 2023). Despite these promising results, O2O RL often suffers from limited asymptotic performance during online fine-tuning (Zhang et al., 2024; Luo et al., 2023). To tackle this issue, existing methods ∗ These authors contributed equally to this work. † Project lead. ‡ Corresponding author. 1 arXiv:2609.33127v1 [cs.LG] 27 Sep 2026 Preprint Offline data Optimal Suboptimal IQL+REFIT key points IQL stays at suboptimal peak 0.00.20.40.60.81.0 reward (a) Environment setup 01M1M+10k1M+20k training iteration 0.0 0.1 0.2 0.3 0.4 0.5 average actor FAU IQL IQL+Expert IQL+REFIT (b) Actor FAU 01M1M+10k1M+20k training iteration 0.0 0.2 0.4 0.6 0.8 1.0 return IQL IQL+Expert IQL+REFIT (c) Learning curve Figure 1: Network plasticity and online adaptation in a controlled O2O setting. The dashed line marks the transition from offline pre-training to online fine-tuning. (a) The bimodal reward land- scape, offline data distribution, and policy trajectories. (b) Actor FAU throughout offline training and subsequent online fine-tuning. (c) Learning curve throughout training. generally take the offline-trained policy as a given initialization and focus on recalibrating value estimates under the offline-to-online shift, alleviating the excessive conservatism inherited from offline training (Nakamoto et al., 2023; Luo et al., 2024; Shin et al., 2025). These methods implicitly assume that the offline-trained policy remains sufficiently plastic to benefit from subsequent online updates. However, we argue that the network plasticity of the offline-trained policy can also be a critical bottleneck to subsequent online improvement. Specifically, the loss of plasticity during offline training can cause the network to become rigid, severely hindering its ability to move beyond the suboptimal behavior learned offline and adapt toward new behavior patterns that yield higher returns. To exemplify the above issue, we design a controlled 2D continuous bandit experiment and track the Fraction of Active Units (FAU) as a proxy for network plasticity. As shown in Figure 1a, the offline data cover only the suboptimal peak. During online fine-tuning, IQL remains near the suboptimal peak, while even with the guidance of expert data, the IQL+Expert variant can only improve slowly. Their low actor FAU (Figure 1b) suggests that insufficient network plasticity, rather than data coverage alone, can hinder the offline-trained policy from adapting toward better behaviors. Motivated by these observations, we propose REstoring plasticity via Fresh Initialization and policy Transfer (REFIT) for O2O RL. Unlike prior methods that focus on value correc- tion (Nakamoto et al., 2023; Luo et al., 2024; Shin et al., 2025) to improve online learning effi- ciency, REFIT restores the plasticity of the offline-trained policy before online fine-tuning, allowing it to better adapt to the online distribution while preserving the offline prior. Specifically, REFIT transfers the offline behavioral prior by distilling the teacher policy into a freshly initialized student network. Crucially, by temporarily freezing a random subset of units in the student network during this process, REFIT preserves a subset of unoptimized parameters that preserve high plasticity, fa- cilitating rapid adaptation during the subsequent online phase. As shown in Figure 1b and Figure 1c, even when the offline policy converges to a highly suboptimal solution, REFIT uses distillation to retain the offline prior while restoring network plasticity, moving beyond its offline behavior and continuing to improve during online fine-tuning. Our contributions are summarized as follows: • We identify an overlooked issue in O2O RL: Offline optimization can reduce the plasticity of the policy network, hindering its ability to continue adapting during online fine-tuning. • We propose REFIT, a simple plug-in method that distills the offline policy into a freshly initialized student policy while temporarily freezing a random subset of student units. This procedure transfers the learned offline behavioral prior to a more plastic network, enabling continued adaptation during online fine-tuning. • We demonstrate the effectiveness of REFIT through extensive experiments on D4RL and OGBench across two O2O RL backbones, showing that it consistently outperforms existing plug-in O2O RL methods and achieves stronger performance across various settings. 2 Preprint 2RELATED WORKS 2.1OFFLINE-TO-ONLINE REINFORCEMENT LEARNING O2O RL first pre-trains a policy on a static dataset, and then fine-tunes it through online environ- ment interactions (Nair et al., 2020; Nakamoto et al., 2023; Fujimoto et al., 2019; Fujimoto & Gu, 2021). It combines the strong initialization provided by offline RL with the continual improvement enabled by online interactions. Early O2O methods directly extend conservative offline RL algo- rithms to the online phase, including Implicit Q-Learning (Kostrikov et al., 2021) and Conservative Q-Learning (Kumar et al., 2020b). More recent methods have dived deep into the inner mechanisms of O2O RL such as recalibrating the value function to mitigate value underestimation (Nakamoto et al., 2023) and reducing distribution mismatch via a warm-up phase (Zhou et al., 2025b). However, existing O2O methods primarily focus on these algorithmic challenges, while offline-induced plas- ticity degradation has received comparatively limited attention. In contrast, our method explicitly restores network plasticity during O2O adaptation, leading to substantially improved performance. 2.2PLASTICITY IN REINFORCEMENT LEARNING Plasticity refers to the ability of a neural network to continually adapt to new data distributions (Klein et al., 2024; Dohare et al., 2021). Recent studies have shown that deep RL agents progressively suffer from degrading plasticity, limiting their ability to adapt to new distributions despite having sufficient model capacity (Abbas et al., 2023; Lyle et al., 2022; Wang et al., 2025). Existing work attributes this phenomenon to the non-stationarity of RL training and measures it through symptoms such as dormant neurons, representation collapse, and abnormal parameter or gradient dynamics (Sokar et al., 2023; Klein et al., 2024; Zhou et al., 2025a). To mitigate plasticity loss, prior methods have explored weight resets (Nikishin et al., 2022), normalization (Lyle et al., 2023) and regularization techniques (Kumar et al., 2024), as well as architectural designs that explicitly inject plasticity into the critic during training (Nikishin et al., 2023). Meanwhile, recent work (Kong et al., 2024) has realized that primacy bias can also be a key factor limiting subsequent O2O adaptation. However, di- rect assessment of policy plasticity during the O2O pipeline and corresponding solutions specifically targeting its restoration remain underexplored. To fill this gap, we first systematically characterize offline-induced plasticity loss and propose REFIT to restore network plasticity at the O2O transition by distilling the offline teacher policy to an initialized and partially frozen online student policy. 3PRELIMINARIES Offline-to-Online Reinforcement Learning Traditional RL can be viewed as a Markov Decision Process (MDP) (Puterman, 1990) paradigm, defined as a tuple M = (S,A,P,r,d 0 ,γ), where S is the state space, A is the action space, P (s ′ | s,a) represents the transition dynamics, r(s,a) is the reward function, d 0 (s) is the initial state distribution, and γ ∈ [0, 1) denotes the discount factor. The objective of RL is to learn a policy π(a | s) that maximizes the expected discounted return, π ∗ = arg max π E s 0 ∼d 0 , a t ∼π(·|s t ), s t+1 ∼P(·|s t ,a t ) [ P ∞ t=0 γ t r(s t ,a t )]. The policy is evaluated by the state-value function V π (s) = E π [ P ∞ t=0 γ t r(s t ,a t ) | s 0 = s] and the action-value function Q π (s,a) = E π [ P ∞ t=0 γ t r(s t ,a t ) | s 0 = s,a 0 = a]. As for O2O RL, the agent is given a dataset D off = (s t ,a t ,r t ,s t+1 ) N t=1 collected by a behavior policy π β . The agent leverages D off to pre- train the value functions Q offline , V offline and policy π offline , which then serve as the starting point for online interaction and optimization. Model Plasticity Network plasticity (Lyle et al., 2023; Dohare et al., 2021; Nikishin et al., 2022) refers to the capacity of a neural network to continually adapt to new data distributions, which can be quantitatively monitored through activation patterns, parameter scales, and gradient behaviors. In this work, we primarily focus on the Fraction of Active Units (FAU) (Ma et al., 2023) as a key metric for diagnosing network plasticity. Additional plasticity metrics are provided in Appendix F.1. Specifically, for a hidden layer l with width H l , we define the layer-wise FAU as well as the network FAU under a state distributionB as Φ l = E s∼B " 1 H l H l X n=1 I h (l) n (s) > 0 # ,Φ network = 1 L L X l=1 Φ l ,(1) 3 Preprint where h (l) n (s) denotes the pre-activation of unit n in layer l, L is the number of hidden layers, and I(·) is the indicator function. This statistic measures the fraction of hidden units with nonzero post-ReLU activation. 4UNDERSTANDING AND MITIGATING PLASTICITY LOSS IN O2O RL To understand what limits online adaptation in O2O RL, we examine the plasticity of the offline- trained policy.Specifically, we first provide controlled empirical evidence that offline over- optimization progressively reduces network plasticity and suppresses subsequent online adaptation. This observation motivates Plasticity Restoration as a key design principle for O2O RL. Following this principle, we propose REstoring plasticity via Fresh Initialization and policy Transfer, termed REFIT, a plug-in method orthogonal to existing O2O RL algorithms that transfers the behavioral prior of the offline teacher policy to a freshly initialized online student policy through randomized partial freezing and behavioral distillation. 4.1EMPIRICAL ANALYSIS: PLASTICITY LOSS IN O2O RL To investigate the relationship between offline training, network plasticity, and subsequent online adaptation, we conduct a controlled empirical study based on IQL (Kostrikov et al., 2021) on the D4RL antmaze-medium-play environment, as shown in Figure 2. To disentangle the effect of network plasticity from that of offline policy quality at the start of online fine-tuning, we pre-train identical policies with four offline training budgets: 0.25M, 0.5M, 1M, and 2M gradient steps. These checkpoints attain comparable near-saturated offline performance while exhibiting distinct levels of network plasticity, thereby allowing us to study how plasticity affects subsequent online adaptation. During the whole process, we track the normalized score to measure adaptation performance and FAU (Ma et al., 2023) as a proxy for network plasticity. A decline in FAU reflects a reduced fraction of active units, further indicating increased plasticity loss and diminished adaptability. All the above experiments are conducted over 20 random seeds. Given the results shown in Figure 2, we have the following observations. Offline 0.25MOffline 0.5MOffline 1MOffline 2M -2M-1.5M-1M-0.5M00.5M1M 0 20 40 60 80 100 Normalized Score Normalized Score vs Training Steps -2M-1.5M-1M-0.5M00.5M1M Training Steps 0.25 0.30 0.35 0.40 0.45 FAU (Overall) Fraction of Active Units (FAU) vs Training Steps (a) Performance and FAU of O2O IQL 0.240.260.280.300.320.340.360.380.40 FAU (Online Step 0) 10 5 0 5 10 15 20 Performance Gain Performance Gain vs FAU (R = 0.73) (b) Correlation between FAU and performance gain Figure 2: Offline over-optimization significantly degrades model plasticity, and the plasticity metric FAU is shown to be positively correlated with online performance improvements. (a) Overall perfor- mance and FAU curves across varying offline pre-training budgets. (b) Correlation between online initial policy FAU and performance improvement. 4 Preprint Observation 1: Offline Over-Optimization Leads to Plasticity Degradation. Continued optimization on static offline data progressively reduces FAU even after offline per- formance has largely saturated, indicating a gradual plasticity loss before online fine-tuning. As shown in Figure 2a, all of the settings achieve comparable offline performance, while FAU con- sistently decreases with continued offline optimization. This indicates that network plasticity can de- grade even after offline performance has largely saturated. The results suggest that over-optimization on a static offline dataset may progressively constrain the diversity of network activations, reduc- ing the fraction of active units and potentially limiting the capacity of the model to adapt to novel distributions encountered during online fine-tuning. Observation 2: Higher Plasticity Correlates with Stronger Online Adaptation. Policies with higher FAU exhibit larger online performance gains, whereas lower-FAU policies adapt more slowly and tend to plateau earlier. During online fine-tuning, the policy must adapt to online interaction data beyond the fixed offline distribution. As shown in Figure 2a, online improvement steadily diminishes as the offline training budget increases from 0.25M to 2M steps. In particular, the 1M- and 2M-step policies show little further improvement during online fine-tuning, consistent with weaker adaptability at lower plastic- ity. This relationship is further reflected in Figure 2b, where mean FAU is positively correlated with online performance gain, linking higher retained plasticity to stronger subsequent online adaptation. Design Implication: Plasticity Restoration for O2O RL These empirical observations suggest that offline optimization can progressively reduce policy plasticity and weaken subsequent online adaptation. This motivates treating plasticity restoration as an explicit design objective at the O2O transition. Guided by this insight, we design REFIT to transfer the behavioral prior from the offline teacher policy to a freshly initialized student policy, restoring the capacity of the agent for subsequent online adaptation without discarding the learned offline behavior. 4.2REFIT FOR PLASTICITY RESTORATION Figure 3: Schematic overview of REFIT. Motivated by the above analysis, we in- troduce REFIT, a lightweight model-level method for restoring policy plasticity at the O2O transition. Rather than directly con- tinuing online optimization from the offline- trained policy parameters, REFIT instanti- ates a freshly initialized student policy and transfers the behavior learned by the offline teacher through policy distillation. During distillation, a random subset of student units is temporarily frozen at their initial values, allow- ing the student to acquire the offline behavior while retaining part of its freshly initialized structure. Once distillation is complete, the frozen units are released, and the resulting online student policy undergoes online fine-tuning with all parameters trainable. Randomized Partial Freezing. Before online fine-tuning, REFIT instantiates a fresh online stu- dent policyπ student from the standard initialization distribution and randomly partitions its parameters θ student into two disjoint subsets: θ student = θ train ∪ θ freeze , θ train ∩ θ freeze =∅.(2) Specifically, REFIT randomly selects a fraction ρ of units in each dense layer and freezes their associated parameters at their initial values throughout distillation, forming θ freeze . The remaining parameters θ train are optimized to reproduce the behavior of the offline teacher policy. By leaving part of the freshly initialized student unoptimized during behavioral transfer, partial freezing allows the trainable parameters to acquire the offline behavioral prior through distillation while preserving a freshly initialized parameter subset for subsequent online adaptation. 5 Preprint Plasticity-Preserving Distillation. Given the partition above, REFIT transfers the behavior of the offline teacher policy π teacher to the student using the offline dataset D off . During distillation, only the trainable subset θ train is optimized, while θ freeze remains fixed at its initial values: L distill (θ train ) = E s∼D off h ∥π θ student (s)− π teacher (s)∥ 2 2 i .(3) This objective allows the trainable parameters to recover the behavior learned by the offline teacher, while the frozen subset remains unaffected by distillation and retains its fresh initialization. REFIT is applied only to the policy network; the critic and value networks are directly inherited from the offline phase. After distillation, the freeze mask is removed and all student parameters are set to trainable for subsequent online fine-tuning. In this way, REFIT combines fresh initialization, behavioral distillation, and temporary partial freezing to transfer the offline policy behavior while providing a more plastic initialization for online adaptation. Pseudocode of our algorithm is provided in Appendix A. 5EXPERIMENTS Our experiments aim to answer the following questions: whether REFIT (1) improves the online performance of original O2O RL algorithms over other plug-in methods (Section 5.1), (2) restores network plasticity during online fine-tuning (Section 5.2), (3) benefits from each design component (Section 5.3), (4) remains effective without manually tuning the offline training budget (Section 5.4), and (5) introduces only modest computational overhead (Appendix C). Table 1: Performance of our REFIT and baselines on AntMaze, MuJoCo, and OGBench, averaged over five random seeds. Bold and underlined values denote the best and second-best results within each backbone, respectively. Environment IQL (Kostrikov et al., 2021)Cal-QL (Nakamoto et al., 2023) BaselineOPTOurs (REFIT)BaselinePARSOurs (REFIT) antmaze-m-p76.2± 1.588.0± 3.086.6± 4.892.2± 2.192.4± 3.095.4± 1.7 antmaze-m-d71.6± 2.788.8± 1.289.2± 2.793.2± 0.889.6± 2.996.2± 0.8 antmaze-l-p46.0± 2.861.6± 5.163.8± 8.370.6± 7.578.6± 3.678.2± 5.0 antmaze-l-d 50.6± 5.763.6± 3.665.2± 4.053.6± 17.479.2± 12.480.2± 5.0 AntMaze total244.4302.0304.8309.6339.8350.0 halfcheetah-r13.8± 3.933.2± 11.846.3± 1.433.4± 3.827.0± 2.439.2± 3.5 hopper-r 8.0± 0.311.4± 0.515.7± 4.18.3± 1.715.0± 0.831.2± 0.9 walker2d-r 7.1± 1.010.0± 1.010.7± 1.111.5± 1.511.7± 0.413.5± 3.8 halfcheetah-m48.1± 0.154.3± 2.750.0± 0.365.3± 3.064.2± 1.151.1± 0.9 hopper-m63.9± 3.291.6± 12.179.8± 3.976.3± 6.171.7± 1.882.2± 2.2 walker2d-m71.9± 2.578.9± 3.087.9± 0.683.4± 2.288.8± 0.797.3± 2.9 halfcheetah-m-r43.9± 0.246.9± 2.847.1± 0.458.9± 0.854.4± 0.355.8± 0.7 hopper-m-r 77.5± 18.1104.8± 1.4102.7± 0.468.9± 7.885.7± 4.687.6± 7.5 walker2d-m-r74.8± 2.893.6± 1.196.9± 2.286.7± 8.7101.7± 1.397.9± 5.8 MuJoCo total409.0524.7537.1492.7520.2555.8 cube-single-play70.7± 9.773.0± 9.481.1± 5.315.8± 16.263.2± 1.979.6± 2.2 puzzle-3x3-play25.3± 4.627.7± 6.498.6± 1.81.4± 3.117.0± 9.79.0± 8.3 scene-play76.7± 5.777.0± 3.096.9± 2.411.8± 11.61.4± 1.16.0± 5.4 OGBench total172.7177.7276.629.081.694.6 All total826.11004.4(+21.6%)1118.5 (+35.4%)831.3941.6(+13.3%)1000.4 (+20.3%) Experiment Setting We conduct experiments on D4RL (Fu et al., 2020) MuJoCo locomotion and AntMaze tasks and three OGBench (Park et al., 2025) manipulation tasks. We select two represen- tative O2O RL algorithms as baselines: IQL (Kostrikov et al., 2021) and Cal-QL (Nakamoto et al., 2023). For each algorithm, we compare against a dedicated O2O plug-in method: OPT (Shin et al., 2025), which adds an online critic pre-training stage to IQL, and PARS (Kim et al., 2025), which regularizes OOD actions for Cal-QL. More experimental details are provided in Appendix B. 6 Preprint 5.1MAIN PERFORMANCE RESULTS Following prior work (Shin et al., 2025; Kim et al., 2025), we report performance after 300K on- line fine-tuning steps. Table 1 shows that REFIT improves the aggregate score over vanilla IQL and Cal-QL by 35.4% and 20.3% respectively, while also outperforming the corresponding plug-in methods. The training curves in Figure 4 further show pronounced gains on OGBench and sparse- reward AntMaze tasks, where the base algorithms and even their corresponding plug-in method often plateau during online fine-tuning. Overall, these results demonstrate that REFIT improves aggregate O2O performance across both backbones, especially in settings where the base algorithms struggle to improve further during online fine-tuning. Additional comparisons are provided in Appendix D. 00.5M1M1.15M1.3M 0 50 100 Success Rate (%) Scene-Play 00.5M1M1.15M1.3M 0 50 100 Success Rate (%) Puzzle-3x3 00.5M1M1.15M1.3M 0 20 40 Normalized Score HalfCheetah-Random 00.5M1M1.15M1.3M 0 20 40 Normalized Score HalfCheetah-Medium-Replay 00.5M1M1.15M1.3M Training Steps 10 20 Normalized Score Hopper-Random 00.5M1M1.15M1.3M Training Steps 0 50 100 Normalized Score Walker2d-Medium-Replay 00.5M1M1.15M1.3M Training Steps 0 50 Normalized Score AntMaze-Medium-Diverse 00.5M1M1.15M1.3M Training Steps 0 50 Normalized Score AntMaze-Medium-Play IQLIQL+OPTIQL+REFIT (Ours) Figure 4: O2O training curves on D4RL and OGBench. The dashed line marks the transition from offline pre-training to online fine-tuning, at which REFIT is applied. 5.2PLASTICITY DIAGNOSTICS Figure 5 tracks the FAU throughout offline and online training. The results show that FAU declines during offline pre-training for both methods and remains low under vanilla online fine-tuning. In contrast, REFIT sharply restores FAU at the O2O transition, consistent with its goal of restoring network plasticity for subsequent online adaptation. Meanwhile, Appendix F.2 further shows con- sistent trends across complementary plasticity metrics, including weight norm and SRank, providing additional support for the plasticity-restoring effect of REFIT. 00.5M1M1.15M1.3M 0.18 0.20 0.22 0.24 Actor FAU Scene-Play 00.5M1M1.15M1.3M 0.14 0.16 0.18 0.20 Puzzle-3x3 00.5M1M1.15M1.3M 0.22 0.24 0.26 0.28 0.30 HalfCheetah-Random 00.5M1M1.15M1.3M 0.30 0.35 0.40 HalfCheetah-Medium-Replay 00.5M1M1.15M1.3M Training Steps 0.15 0.20 0.25 Actor FAU Hopper-Random 00.5M1M1.15M1.3M Training Steps 0.30 0.35 0.40 Walker2d-Medium-Replay 00.5M1M1.15M1.3M Training Steps 0.25 0.30 0.35 0.40 AntMaze-Medium-Diverse 00.5M1M1.15M1.3M Training Steps 0.25 0.30 0.35 0.40 AntMaze-Medium-Play IQLIQL+REFIT Figure 5: Evolution of actor FAU across the offline-to-online transition. 5.3ABLATION ANALYSIS Ablation of REFIT Design Components We investigate the contributions of fresh initialization, distillation, and partial freezing. The Baseline directly deploys the classic IQL algorithm, while Re- 7 Preprint set reinitializes the policy network before online fine-tuning. Reset + Distillation further transfers the behavior learned by the offline teacher policy to the freshly initialized student through distilla- tion, without partial freezing. Finally, REFIT combines all three components. As shown in Table 2, none of these isolated variants can match the performance of the complete REFIT configuration, which achieves the highest overall score. These results suggest that fresh initialization, policy dis- tillation, and partial freezing provide complementary benefits for subsequent online adaptation. Table 2: Ablation of REFIT design components. All results are averaged over 5 random seeds. Methodhc-m hc-r wal-m-r hop-m-r ant-m-p ant-m-dScore Baseline48.1± 0.1 13.8± 3.9 74.8± 2.8 77.5± 18.1 76.2± 1.5 71.6± 2.7362.0 (-23.3%) Reset47.3± 0.4 38.6± 2.4 84.0± 10.6 86.5± 29.9 71.2± 6.8 74.0± 5.5401.6 (-14.9%) Reset + Distillation49.8± 0.245.7± 1.989.3± 11.297.9± 6.283.5± 1.879.2± 23.6445.4(-5.6%) REFIT50.0± 0.3 46.3± 1.4 96.9± 2.2 102.7± 0.4 86.6± 4.8 89.2± 2.7471.7 Ablation of Freeze Ratio We vary the freeze ratio over ρ ∈ 0.0, 0.25, 0.5, 0.75, 1.0, where ρ = 0.0 corresponds to distillation without freezing and ρ = 1.0 freezes the entire student network during distillation. As shown in Table 3, intermediate freeze ratios achieve comparable aggregate performance, whereas both extreme settings perform notably worse. When ρ = 0, all student pa- rameters are updated during distillation, which may reduce the fraction of active units and limit the plasticity gains from fresh initialization. Conversely, when ρ = 1, the student is entirely frozen and retains only the initialized parameters without inheriting the offline behavioral prior. The robustness of REFIT across the intermediate range suggests that the core mechanism driving performance is the combination of partial freezing and distillation itself, rather than any particular freeze ratio: as long as a sufficient subset of parameters is frozen to preserve plasticity while the remaining parameters absorb the teacher distribution, REFIT maintains comparable performance across these intermediate freeze ratios. Given its highest aggregate performance across the evaluated tasks, we adopt ρ = 0.5 as the default in the remaining experiments. Table 3: Performance of REFIT under various freeze ratio settings. The two lowest scores in each row are marked in red. Env0.00.250.50.751.0 halfcheetah-m49.8± 0.250.0± 0.250.0± 0.350.1± 0.447.3± 0.4 halfcheetah-r45.7± 1.946.2± 1.346.3± 1.446.5± 1.838.6± 2.4 antmaze-m-p83.5± 1.886.0± 3.786.6± 4.885.8± 4.171.2± 6.8 antmaze-m-d79.2± 23.686.4± 3.089.2± 2.784.4± 1.574.0± 5.5 Total258.2268.6272.1266.8231.1 00.5M1M1.15M1.3M Training Steps 18 24 30 36 42 48 Normalized Score HalfCheetah-Medium 00.5M1M1.15M1.3M Training Steps 10 20 30 40 Normalized Score HalfCheetah-Random 00.5M1M1.15M1.3M Training Steps 0 20 40 60 80 Normalized Score AntMaze-Medium-Play 00.5M1M1.15M1.3M Training Steps 0 20 40 60 80 Normalized Score AntMaze-Medium-Diverse Actor DistillCritic DistillFull Distill Figure 6: Learning curves of REFIT with different distillation targets. Ablation of Distillation Target We compare three distillation targets: actor-only, critic-only, and full actor-critic distillation. As shown in Figure 6, variants involving critic distillation exhibit a pro- nounced performance drop at the start of online fine-tuning, whereas actor-only distillation largely avoids this degradation and maintains more stable adaptation. This suggests that critic distillation 8 Preprint introduces additional perturbations to the learned value function, making the O2O transition less stable. Based on this observation, REFIT distills only the actor network while allowing the critic to adapt freely during online fine-tuning. HalfCheetah Medium HalfCheetah Random AntMaze Medium-Play AntMaze Medium-Diverse 25 40 55 70 85 100 Normalized Score 0K (From Scratch)10K50K100K200K500K Figure 7: Performance of REFIT under differ- ent distillation budget settings. Ablation of Distillation Budget We vary the policy-distillation budget from 0K, corresponding to no distillation, to 500K updates. As shown in Figure 7, all nonzero distillation budgets consis- tently outperform the 0K baseline, highlighting the importance of transferring the offline policy before online fine-tuning. Meanwhile, performance re- mains relatively stable across various settings, indi- cating limited sensitivity to the exact distillation du- ration. Together with the freeze ratio ablation, RE- FIT remains effective across a wide range of distil- lation configurations without requiring careful hy- perparameter tuning. Based on this robustness, we adopt 100K updates as the default setting, which provides strong performance without incurring unnecessary additional computation. 5.4ROBUSTNESS TO OFFLINE TRAINING BUDGET The motivation study in Section 4.1 shows that prolonged offline optimization can degrade network plasticity and hinder subsequent online adaptation. We therefore evaluate REFIT across offline train- ing budgets ranging from 0.25M to 2M steps under the same setting as in Section 4.1. As shown in Figure 8a and Figure 8b, REFIT consistently outperforms the corresponding baseline across all offline budget settings. Notably, as the offline training budget increases, the baseline exhibits in- creasingly pronounced plateaus during online fine-tuning, whereas REFIT continues to improve, suggesting that restoring network plasticity enables effective adaptation after prolonged offline opti- mization. Figure 8c further shows that REFIT maintains higher FAU throughout online fine-tuning across all offline budget settings, providing further evidence of improved network plasticity. To- gether, these results demonstrate the effectiveness of REFIT in mitigating the plasticity degradation induced by prolonged offline optimization and facilitating subsequent online adaptation. 0.25M0.5M1M2M 0 20 40 60 80 100 Normalized Score 86.4 87.4 86.6 82.9 85.5 86.2 73.2 75.4 w/ REFIT w/o REFIT (a) Online performance 0K200K400K600K800K1.0M Online Steps 50 60 70 80 90 100 Normalized Score Offline 0.25M (w/ REFIT) Offline 0.25M (w/o REFIT) Offline 0.5M (w/ REFIT) Offline 0.5M (w/o REFIT) Offline 1M (w/ REFIT) Offline 1M (w/o REFIT) Offline 2M (w/ REFIT) Offline 2M (w/o REFIT) (b) Online learning curves 200K400K600K800K1.0M Online Steps 0.25 0.30 0.35 0.40 FAU (Overall) (c) FAU Figure 8: Robustness of REFIT across different offline training budgets. (a) Normalized online performance of REFIT and the corresponding IQL baseline after online fine-tuning under different offline training budgets. (b) Online learning curves of REFIT and the baseline across different offline training budgets. (c) FAU during online fine-tuning. 6CONCLUSION In this work, we investigate O2O RL from the perspective of network plasticity. Our empirical anal- ysis shows that prolonged optimization on static offline data can progressively reduce network plas- ticity even after offline performance has largely saturated, and that policies retaining higher plasticity tend to exhibit stronger subsequent online adaptation. Motivated by these observations, we propose REFIT, a lightweight model-level method that distills the behavior of an offline teacher into a freshly 9 Preprint initialized student policy while temporarily freezing a random subset of student units. This proce- dure preserves the offline behavioral prior while restoring policy plasticity for subsequent online adaptation. Experiments across D4RL and OGBench demonstrate that REFIT improves aggregate performance for IQL and Cal-QL and achieves higher aggregate scores than the corresponding O2O plug-in methods. These results highlight policy plasticity as an important factor in O2O adaptation and demonstrate the effectiveness of explicitly restoring plasticity at the O2O transition. AI USE STATEMENT In this paper, we have used generative AI tools for implementing part of the code of our method. We have not used generative AI tools for helping develop theoretical models or conceptual frameworks, formulating mathematical claims, providing critical ingredients for proving mathematical claims and interpreting results. The rest of the required disclosure tasks are not applicable to this work. Additionally, we used generative AI tools to refine the expression of the paper. All LLM-generated content has been reviewed by the authors. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. ETHICS STATEMENT This work studies plasticity loss in O2O RL and proposes a model-level method for improving policy fine-tuning. All experiments are conducted in simulated continuous-control, navigation, and robotic- manipulation benchmarks, including D4RL and OGBench, and do not involve human subjects, per- sonal data or safety-critical systems. Our method does not introduce capabilities that directly target surveillance, identity inference, autonomous weapons, or other clearly harmful applications. While improved O2O RL could in principle accelerate the deployment of learned policies in downstream systems, we do not identify ethical concerns specific to this work beyond those common to general RL research. We therefore do not believe this work requires additional mitigation protocols. REPRODUCIBILITY STATEMENT We have made efforts to ensure the reproducibility of our work. To facilitate reproducibility, we provide our code, including hyperparameter configuration files in the supplementary material, with implementation details, hyperparameters, and hardware specifications documented in Appendix B. REFERENCES Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on lifelong learning agents, p. 620–636. PMLR, 2023. Kaixuan Chen, Wei Luo, Shunyu Liu, Yaoquan Wei, Yihe Zhou, Yunpeng Qing, Quan Zhang, Yong Wang, Jie Song, and Mingli Song. Powerformer: A section-adaptive transformer for power flow adjustment. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, p. 2204–2215, 2025. Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325, 2021. Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021. Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, p. 2052–2062. PMLR, 2019. Caglar Gulcehre, Srivatsan Srinivasan, Jakub Sygnowski, Georg Ostrovski, Mehrdad Farajtabar, Matt Hoffman, Razvan Pascanu, and Arnaud Doucet. An empirical study of implicit regulariza- tion in deep offline rl. arXiv preprint arXiv:2207.02099, 2022. 10 Preprint Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, August 2018. URL http://arxiv.org/abs/1801.01290. arXiv:1801.01290 [cs]. Jeonghye Kim, Yongjae Shin, Whiyoung Jung, Sunghoon Hong, Deunsol Yoon, Youngchul Sung, Kanghoon Lee, and Woohyung Lim. Penalizing infeasible actions and reward scaling in rein- forcement learning with offline data. arXiv preprint arXiv:2507.08761, 2025. Timo Klein, Christoph Luther, Manus McAuliffe, Lukas Miklautz, Claudia Plant, and Sebas- tian Tschiatschek. Plasticity loss in deep reinforcement learning: A survey. arXiv preprint arXiv:2411.04832, 2024. Rui Kong, Chenyang Wu, Chen-Xiao Gao, Zongzhang Zhang, and Ming Li. Efficient and stable offline-to-online reinforcement learning via continual policy revitalization. In IJCAI, p. 4317– 4325, 2024. Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06169, 2021. Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010.14498, 2020a. Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for of- fline reinforcement learning. Advances in neural information processing systems, 33:1179–1191, 2020b. Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learn- ing via regenerative regularization, 2024. URL https://arxiv.org/abs/2308.11958. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. Qin-Wen Luo, Ming-Kun Xie, Ye-Wen Wang, and Sheng-Jun Huang. Optimistic critic reconstruc- tion and constrained fine-tuning for general offline-to-online rl. Advances in Neural Information Processing Systems, 37:108167–108207, 2024. Yicheng Luo, Jackie Kay, Edward Grefenstette, and Marc Peter Deisenroth. Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions, 2023. URL https:// arxiv.org/abs/2303.17396. Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in rein- forcement learning. arXiv preprint arXiv:2204.09560, 2022. Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, p. 23190–23211. PMLR, 2023. Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado Van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762, 2024. Guozheng Ma, Lu Li, Sen Zhang, Zixuan Liu, Zhen Wang, Yixin Chen, Li Shen, Xueqian Wang, and Dacheng Tao. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. arXiv preprint arXiv:2310.07418, 2023. Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020. Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 36:62244–62269, 2023. 11 Preprint Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, p. 16828–16847. PMLR, 2022. Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Infor- mation Processing Systems, 36:37142–37159, 2023. Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations, volume 2025, p. 94937–94982, 2025. Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331–434, 1990. Yunpeng Qing, Shunyu Liu, Jingyuan Cong, Kaixuan Chen, Yihe Zhou, and Mingli Song. A2po: Towards effective offline reinforcement learning from an advantage-aware perspective. Advances in Neural Information Processing Systems, 37:29064–29090, 2024. Yunpeng Qing, Yixiao Chi, Shuo Chen, Shunyu Liu, Kexuan Zhou, Sixu Lin, Litao Liu, and Changqing Zou. Bitrajdiff: Bidirectional trajectory generation with diffusion models for offline reinforcement learning. arXiv preprint arXiv:2506.05762, 2025. Yongjae Shin, Jeonghye Kim, Whiyoung Jung, Sunghoon Hong, Deunsol Yoon, Youngsoo Jang, Geonhyeong Kim, Jongseong Chae, Youngchul Sung, Kanghoon Lee, et al. Online pre-training for offline-to-online reinforcement learning. arXiv preprint arXiv:2507.08387, 2025. Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning, p. 32145–32168. PMLR, 2023. Haoyu Wang, Wei Dai, Jiawei Zhang, Jialun Ma, Mingyi Huang, and Yuguo Yu. The dual nature of plasticity loss in deep continual learning: Dissection and mitigation. Advances in Neural Information Processing Systems, 38:69412–69444, 2025. Feiyang Xu, Shunyu Liu, Yunpeng Qing, Yihe Zhou, Yuwen Wang, and Mingli Song. Tempo- ral prototype-aware learning for active voltage control on power distribution networks. In Pro- ceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 3598–3609, 2024. Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforce- ment learning. arXiv preprint arXiv:2302.00935, 2023. Yinmin Zhang, Jie Liu, Chuming Li, Yazhe Niu, Yaodong Yang, Yu Liu, and Wanli Ouyang. A perspective of q-value estimation on offline-to-online reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 38, p. 16908–16916, 2024. Huaicheng Zhou, Zifeng Zhuang, and Donglin Wang. Stay hungry, keep learning: Sustainable plasticity for deep reinforcement learning. In Forty-second International Conference on Machine Learning, 2025a. Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforce- ment learning fine-tuning need not retain offline data. In International Conference on Learning Representations, volume 2025, p. 32343–32368, 2025b. 12 Preprint APSEUDOCODE Algorithm 1 REFIT Training Procedure Require: Offline datasetD off ; environment E; O2O algorithm; freeze ratio ρ. Require: Budgets N off = 10 6 , N distill = 10 5 , and N on = 3× 10 5 . 1: Phase 1: Offline Pre-training 2: for t = 1 to N off do 3:Sample batch (s,a,r,s ′ )∼D off . 4:Update policy π θ and backbone value functions using the offline RL objective. 5: end for 6: Phase 2: Plasticity-Preserving Policy Distillation 7: Set π teacher ← π θ and initialize a fresh student policy π student . 8: Randomly partition θ student into frozen parametersθ freeze and trainable parametersθ train according to the freeze ratio ρ. 9: for k = 1 to N distill do 10:Sample s∼D off and update unfrozen student units usingL distill in Eq. (3) 11: end for 12: Replace π θ ← π student ; retain the offline-trained value functions. 13: Phase 3: Online Fine-tuning 14: Unfreeze θ freeze , making all student policy parameters trainable. 15: for t = 1 to N on do 16:Interact with E, add the transition to the replay buffer, update with online RL objective. 17: end for BEXPERIMENT DETAILS B.1ENVIRONMENT DETAILS We evaluate D4RL MuJoCo and AntMaze (Fu et al., 2020) and three OGBench manipulation tasks (Park et al., 2025). Table 4 defines the compositional abbreviations used in our result ta- bles. To fit space-constrained tables, we additionally shorten the environment prefixes to hc, hop, wal, and ant; for example, halfcheetah-m, walker2d-m-r, and antmaze-m-p become hc-m, wal-m-r, and ant-m-p, respectively. OGBench task names are not abbreviated. Table 4: Environment naming convention. Tokens are composed from left to right, as illustrated in the examples. SuiteTokenMeaningExample Environment hc halfcheetah hc-m hop hopper hop-m-r wal walker2d wal-r ant antmaze ant-l-d MuJoCo Postfix r random hc-r m medium hop-m m-r medium-replay wal-m-r AntMaze Postfix m-d medium-diverse ant-m-d m-p medium-play ant-m-p l-d large-diverse ant-l-d l-p large-play ant-l-p B.2REFIT IMPLEMENTATION All experiments use NVIDIA A100 GPUs with CUDA 12.4. 13 Preprint Three-phase training. We first train the selected backbone on D off for 10 6 steps. At the O2O transition, we distill the offline teacher into a freshly initialized student policy for 10 5 steps while applying a fixed mask that freezes the kernel columns and biases of ⌊ρd out ⌋ randomly selected output channels in each actor Dense layer, including the policy output head; the critic and value networks are inherited from the offline phase. We then remove the mask and fine-tune all networks for 3× 10 5 online steps. Algorithm 1 summarizes the full procedure. Hyperparameters. Across all environments and both backbones, we use 10 6 offline training steps, 10 5 policy-distillation updates, and 3 × 10 5 online fine-tuning steps. The default freeze ratio is ρ = 0.5; for Cal-QL on MuJoCo, we use ρ = 0.1 for HalfCheetah and Hopper and ρ = 0.5 for Walker2d. Other settings, such as network architecture, policy learning rate, IQL expectile and temperature, and Cal-QL conservative weight, follow the corresponding backbone implementations. Complete environment-specific configurations, including batch size and normalization, are provided in the configuration files of the supplementary code. B.3BACKBONE IMPLEMENTATION We evaluate REFIT on IQL (Kostrikov et al., 2021) and Cal-QL (Nakamoto et al., 2023), comparing against OPT (Shin et al., 2025) and PARS (Kim et al., 2025), respectively. Our implementations build on the following public repositories: • IQL: https://github.com/ikostrikov/implicit_q_learning • Cal-QL: https://github.com/nakamotoo/Cal-QL • OPT: https://github.com/LGAI-Research/opt • PARS: https://github.com/LGAI-Research/pars CCOMPUTATIONAL COST In this section, we evaluate the computational overhead of REFIT by comparing the wall-clock time of IQL+REFIT with that of the IQL baseline on halfcheetah-random-v2. All runs are conducted on a single NVIDIA A100 GPU and averaged over five random seeds. 00.5M1.0M1.1M1.3M1.4M Training steps 0 500 1,000 1,500 2,163.4 2,298.9 Cumulative wall-clock time (s) REFIT distillation IQLIQL + REFIT Figure 9: Wall-clock time of IQL+REFIT and the IQL baseline, averaged over five random seeds. As shown in Figure 9, the complete pipeline requires 2163.4 seconds for IQL and 2298.9 seconds for IQL+REFIT, corresponding to a 6.26% increase in wall-clock time. The small offset near the end of distillation arises from one-time JAX just-in-time compilation when the distillation update and student policy are initialized. After compilation, the two curves progress approximately in parallel, indicating that REFIT introduces limited sustained overhead beyond the additional distillation phase. DADDITIONAL COMPARISON WITH CPR CPR (Kong et al., 2024) links primacy bias to limited O2O adaptation, but does not directly quantify offline-induced policy plasticity loss with network-level diagnostics. Its periodic policy revitaliza- 14 Preprint tion retains earlier policies for action selection. REFIT instead diagnoses this loss and applies a single model-level intervention at the O2O transition, then fine-tunes one actor online. Table 5 com- pares both methods on IQL and Cal-QL across six D4RL tasks. We adapt CPR from their released code (https://github.com/LAMDA-RL/CPR), following its method-specific hyperparame- ters where applicable. Table 5: Normalized scores on six representative tasks. Method hc-m hc-r wal-m-r hop-m-r ant-m-p ant-m-dTotal IQL48.1± 0.1 13.8± 3.9 74.8± 2.8 77.5± 18.1 76.2± 1.571.6± 2.7362.0 IQL+CPR51.7± 0.5 26.0± 4.0 86.0± 11.7 101.8± 3.9 66.0± 3.371.0± 6.8402.5 IQL+REFIT50.0± 0.3 46.3± 1.4 96.9± 2.2 102.7± 0.4 86.6± 4.889.2± 2.7471.7 Cal-QL65.3± 3.0 33.4± 3.8 86.7± 8.768.9± 7.892.2± 2.193.2± 0.8439.7 Cal-QL+CPR43.3± 0.6 2.2± 0.117.6± 2.830.3± 7.9 27.0± 10.3 26.0± 13.9146.3 Cal-QL+REFIT 51.1± 0.9 39.2± 3.5 97.9± 5.887.6± 7.595.4± 1.796.2± 0.8467.4 These results demonstrate that REFIT outperforms CPR in most cases by directly restoring policy plasticity at the model level. ESUPPLEMENTARY ABLATION ANALYSIS This section provides supplementary ablations that complement the analysis in Section 5.3. We ex- amine the necessity of policy distillation and network reinitialization, the sensitivity to the distilla- tion loss function, the effect of standard-deviation transfer on exploration, and the role of unfreezing during online adaptation. E.1ABLATION OF POLICY DISTILLATION To evaluate the role of distillation in mitigating policy degradation during the O2O transition, we compare REFIT against a direct reset baseline (referred to as From-Scratch), which randomly reini- tializes the network parameters at the start of online fine-tuning without any behavioral prior from the offline teacher. 00.5M1M1.15M1.3M Training Steps 0 20 40 60 80 Normalized Score AntMaze-Medium-Play 00.5M1M1.15M1.3M Training Steps 0 20 40 60 80 Normalized Score AntMaze-Medium-Diverse 00.5M1M1.15M1.3M Training Steps 20 30 40 50 Normalized Score HalfCheetah-Medium 00.5M1M1.15M1.3M Training Steps 10 20 30 40 Normalized Score HalfCheetah-Random IQLIQL+REFITIQL From Scratch Figure 10: Performance comparison between IQL, IQL From Scratch and IQL+REFIT As shown in Figure 10, although the direct reinitialization baseline outperforms standard fine-tuning on aggregate by restoring network plasticity, it consistently falls short of the full REFIT configura- tion. This result confirms that combining policy distillation with structured reinitialization effec- tively preserves the offline behavioral prior, whereas naive parameter resetting discards the behav- ioral prior acquired during offline training. 15 Preprint E.2ABLATION OF POLICY REINITIALIZATION hc-mhc-rant-m-pant-m-d 0 20 40 60 80 100 Normalized Score 50.0 46.3 86.6 89.2 47.9 14.7 74.0 72.2 REFIT vs Offline Freeze IQL + REFIT Offline Freeze Figure 11: Impact of network reset during distillation. A natural question arises regarding the distillation phase: is it essential to freshly initialize the online student policy, or could one simply freeze parts of the network during offline pre-training and unfreeze them for online fine-tuning? To iso- late the effect of network reinitialization, we compare the de- fault REFIT method against this unfreezing baseline across four representative environments. As shown in Figure 11, di- rectly unfreezing offline pre-trained parameters leads to sub- stantial performance degradation, particularly on tasks re- quiring significant adaptation (e.g., halfcheetah-r drops from 46.3 to 14.7). These results indicate that fresh initializa- tion is essential for overcoming the optimization bottlenecks accumulated during offline training and enabling effective on- line policy improvement. E.3ABLATION OF DISTILLATION LOSS FUNCTION Table 6: Ablation of distillation loss. Results report mean± standard deviation. EnvironmentKL fwd KL rev MSE antmaze-m-d86.9± 6.787.0± 5.289.2± 2.7 antmaze-m-p87.8± 2.784.8± 5.286.6± 4.8 halfcheetah-m 49.9± 0.250.0± 0.650.0± 0.3 halfcheetah-r46.3± 1.547.5± 1.746.3± 1.4 Total Score270.9± 2.9 269.2± 2.9 272.1± 2.8 REFIT adopts mean squared error (MSE) as its default distillation objective due to computational simplicity: MSE directly re- gresses the student policy toward the mean action of the frozen teacher policy, without requiring action sampling or full distribu- tion matching. To verify that the method is not tied to a particular choice of loss function, we compare MSE against forward KL divergence and reverse KL divergence, keeping all other components unchanged. As shown in Table 6, all three objectives yield comparable aggregate performance across the eval- uated environments, with per-task differences largely within the reported standard deviations. This result confirms that the effectiveness of REFIT stems from its structural design—fresh initialization, partial freezing, and knowledge transfer—rather than from the specific form of the distillation loss. We therefore adopt MSE as the default for its simplicity. E.4ABLATION OF STANDARD-DEVIATION TRANSFER Because REFIT reinitializes the student network, the log-standard-deviation parameters of the policy are also reset to their default initial values, which may differ from those learned during offline training. A natural concern is whether the performance gain of REFIT is primarily attributable to the increased policy standard deviation induced by this reinitialization, rather than to the restoration of network plasticity. Table 7: Ablation of standard-deviation transfer. “Default” denotes the standard REFIT configura- tion where the log-standard-deviation parameters are reset along with the rest of the student network; “Copy Std” denotes the variant that inherits the standard deviation from the offline teacher. Results are reported as mean± standard deviation across random seeds. Methodantmaze-m-d antmaze-m-p halfcheetah-m halfcheetah-r Total Score Default89.2± 2.786.6± 4.850.0± 0.346.3± 1.4272.1± 2.8 Copy Std86.4± 4.584.3± 4.050.1± 0.346.9± 2.9267.7± 2.3 To disentangle these two factors, we evaluate a Copy Std variant that directly inherits the standard- deviation parameters from the offline teacher after distillation, so that the student begins online fine-tuning with the same exploration profile as the teacher. As shown in Table 7, the Copy Std variant achieves performance comparable to the default REFIT configuration across all evaluated environments, with aggregate scores differing by only a small margin. This result indicates that 16 Preprint the performance gain of REFIT does not rely on increased exploration induced by resetting the standard deviation, and is instead consistently attributable to the plasticity restoration provided by the structured reinitialization and distillation procedure. E.5ABLATION OF ONLINE UNFREEZING 10%30%50%70%90% 0.0 0.5 1.0 1.5 2.0 2.5 Performance Drop ( ) HalfCheetah-Medium 10%30%50%70%90% 0 2 4 6 8 Performance Drop ( ) HalfCheetah-Random 10%30%50%70%90% 0.0 2.5 5.0 7.5 10.0 12.5 Performance Drop ( ) AntMaze-Medium-Play 10%30%50%70%90% 0.0 2.5 5.0 7.5 10.0 12.5 Performance Drop ( ) AntMaze-Medium-Diverse 10% 30% 50% 70% 90% Figure 12: Online performance with and without unfreezing the parameter subset after distillation. A final design question is whether the parameter subset frozen during distillation should remain frozen during online fine-tuning. The online-freeze variant retains the same mask throughout the online phase, whereas the default REFIT configuration removes the mask and updates the entire policy. As shown in Figure 12, retaining the mask consistently limits online improvement and yields lower asymptotic performance. Unfreezing is therefore necessary to recover the full optimization capacity of the student policy during online adaptation. FSUPPLEMENTARY PLASTICITY DIAGNOSTICS F.1DEFINITIONS OF ADDITIONAL PLASTICITY METRICS Here we introduce the two other metrics we used in the experiment: 1. Weight Norm: The magnitude of network parameters provides a complementary diagnostic of network plasticity (Lyle et al., 2024). We track the L2 norm of all actor parameters: ∥θ∥ weight = v u u t P X i=1 θ 2 i ,(4) whereθ ∈ R P contains all actor parameters, including weights, biases, and policy standard- deviation parameters. We interpret changes in this norm alongside FAU and SRank, rather than as direct evidence of plasticity loss or restoration. 2. SRank: SRank measures the effective dimensionality of learned representations (Kumar et al., 2020a). For an actor hidden layer, let F ∈ R d×m denote the matrix of post-ReLU activations for d sampled states and hidden width m, and let σ 1 (F ) ≥ · ≥ σ q (F ) be its singular values, where q = min(d,m). We define SRank as the minimum number of leading singular values required to retain a fraction 1− ε of the total singular-value mass: SRank ε (F ) = min ( k : P k j=1 σ j (F ) P q j=1 σ j (F ) ≥ 1− ε ) , ε = 0.01.(5) Unlike an absolute singular-value threshold, this relative criterion is invariant to a global rescaling of F and therefore separates changes in spectral shape from changes in feature magnitude. We compute SRank separately for each actor hidden layer using 10 mini-batches of d = 256 states sampled uniformly from the replay buffer, average the resulting values across batches and layers, and record the metric every 5000 policy updates. Because d = 256, the reported SRank is upper bounded by 256. A higher value indicates that the actor representation distributes its spectral mass across more directions, providing a complementary measure of representational capacity rather than a direct measure of performance. 17 Preprint F.2ADDITIONAL DIAGNOSTIC RESULTS 00.5M1M1.15M1.3M 75 100 125 150 175 Weight Norm Scene-Play 00.5M1M1.15M1.3M 75 100 125 150 Puzzle-3x3 00.5M1M1.15M1.3M 40 50 60 70 HalfCheetah-Random 00.5M1M1.15M1.3M 40 50 60 70 HalfCheetah-Medium-Replay 00.5M1M1.15M1.3M Training Steps 40 60 80 Weight Norm Hopper-Random 00.5M1M1.15M1.3M Training Steps 40 50 60 70 Walker2d-Medium-Replay 00.5M1M1.15M1.3M Training Steps 40 50 60 70 AntMaze-Medium-Diverse 00.5M1M1.15M1.3M Training Steps 40 50 60 70 AntMaze-Medium-Play IQLIQL+REFIT Figure 13: Weight Norm of REFIT and baseline throughout the training process. Weight Norm Figure 13 depicts the evolution of the actor weight norm throughout training. The baseline maintains a relatively high weight norm across the O2O transition, whereas REFIT pro- duces a sharp reduction at the start of online fine-tuning, consistent with transferring the offline pol- icy behavior into a freshly initialized student. These results show that REFIT changes the parameter scale at the transition and, together with the FAU and SRank diagnostics, provide complementary evidence for its plasticity-restoring effect. 00.5M1M1.15M1.3M 226 228 230 232 SRank Scene-Play 00.5M1M1.15M1.3M 215 220 225 Puzzle-3x3 00.5M1M1.15M1.3M 160 180 200 HalfCheetah-Random 00.5M1M1.15M1.3M 200 202 204 206 HalfCheetah-Medium-Replay 00.5M1M1.15M1.3M Training Steps 140 160 SRank Hopper-Random 00.5M1M1.15M1.3M Training Steps 195.0 197.5 200.0 202.5 205.0 Walker2d-Medium-Replay 00.5M1M1.15M1.3M Training Steps 200 205 210 215 AntMaze-Medium-Diverse 00.5M1M1.15M1.3M Training Steps 205 210 215 AntMaze-Medium-Play IQLIQL+REFIT Figure 14: SRank of REFIT and baseline throughout the training process. SRank Finally, we evaluate the representational richness using SRank, where a higher value sig- nifies a more diverse and robust feature space (Gulcehre et al., 2022; Lyle et al., 2023). As shown in Figure 14, transferring directly from O2O learning often causes the baseline representations to suffer from representational stagnation, characterized by a stabilized yet severely bounded SRank that fails to expand when encountering new online environments. REFIT increases SRank sharply on several tasks and is comparable on others; the relationship is task-dependent. This preservation of higher-rank representations suggests that the network retains sufficient plasticity and adaptive capacity to capture the novel dynamics encountered during online exploration. 18 Preprint GLIMITATIONS AND DISCUSSION While REFIT consistently improves aggregate performance and outperforms the baselines and other plug-in methods on the majority of environments, its primary contribution lies in addressing O2O adaptation from the perspective of network plasticity. This perspective is largely orthogonal to ex- isting methods that target other bottlenecks in O2O RL, such as inaccurate value estimation (Shin et al., 2025). A more systematic understanding of how these factors interact, as well as how com- plementary mechanisms can be jointly incorporated into O2O RL, remains an interesting direction for future work. 19