Paper deep dive
Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
Yongjae Shin, Jongseong Chae, Jongeui Park, Youngchul Sung
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 10:39:09 PM
Summary
The paper introduces FINO (Flow Matching with Injected Noise for Offline-to-Online RL), a novel method that enhances sample efficiency in offline-to-online reinforcement learning. FINO addresses the limited exploration of standard flow matching policies by injecting noise during offline pre-training to broaden the action space coverage. During online fine-tuning, it employs an entropy-guided sampling mechanism to dynamically balance exploration and exploitation, achieving superior performance on challenging tasks with limited interaction budgets.
Entities (7)
Relation Signals (6)
FINO → addresses → Offline-to-Online RL
confidence 95% · FINO is a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL.
FINO → uses → Flow Matching
confidence 95% · FINO leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL.
FINO → employs → Entropy-Guided Sampling
confidence 92% · In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation.
FINO → evaluatedon → OGBench
confidence 90% · We experiment on 45 diverse and challenging tasks from OGBench... and D4RL
FINO → evaluatedon → D4RL
confidence 90% · We experiment on 45 diverse and challenging tasks from OGBench... and D4RL
FINO → outperforms → FQL
confidence 85% · The results show that FINO achieves consistently strong performance across tasks... demonstrating FINO as an effective and reliable approach... compared to FQL.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.
Tags
Links
- Source: https://arxiv.org/abs/2602.18117v1
- Canonical: https://arxiv.org/abs/2602.18117v1
Trouble viewing inline? Open PDF directly →
Full Text
94,499 characters extracted from source content.
Expand or collapse full text
Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning Yongjae Shin Jongseong Chae Jongeui Park Youngchul Sung KAIST yongjae.shin, ycsung@kaist.ac.kr Corresponding author. Our code is available at https://github.com/CTID282/FINO. Abstract Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets. 1 Introduction Generative models have recently demonstrated substantial success across diverse domains, producing high-quality outputs in areas such as text and image (Brown et al., 2020; Rombach et al., 2022). By leveraging their expressive capacity, these models can capture complex or multimodal distributions present in the underlying datasets, beyond the reach of conventional parametric models. This opens up new opportunities in reinforcement learning (RL), particularly for policy design. Since a policy in RL can be regarded as a generative model conditioned on states, there has been increasing interest in applying generative modeling to policy design, such as denoising diffusion (Sohl-Dickstein et al., 2015; Ho et al., 2020) and flow matching (Lipman et al., 2023; Albergo and Vanden-Eijnden, 2023). While Gaussian policies have been the conventional choice, they often struggle to represent multimodal or high-dimensional action distributions (Park et al., 2024). By contrast, generative policies provide the expressivity required to handle complex RL tasks and have demonstrated superior performance, particularly in offline RL where the target distribution is explicitly defined (Wang et al., 2023b; Hansen-Estruch et al., 2023; Kang et al., 2023; Zhang et al., 2025; Fang et al., 2025; Park et al., 2025b). Despite such expressivity, offline RL inherently suffers from a fundamental limitation in that the performance of the policy is constrained by the quality of the offline dataset. Accordingly, offline-to-online RL has been proposed to address this issue, enabling a pre-trained policy to further improve its performance through short-term direct interaction with the environment (Lee et al., 2022; Zhang et al., 2023; Nakamoto et al., 2023; Zhang et al., 2024; Zhou et al., 2024). While some studies (Hansen-Estruch et al., 2023; Park et al., 2025b) have incorporated additional online fine-tuning of generative policies, they merely treat it as a continuation of offline pre-training rather than designing approaches specialized to the online fine-tuning. As offline-to-online RL encompasses both offline and online stages, it naturally introduces challenges beyond those faced in a purely offline RL framework. Unlike offline RL, which relies solely on pre-collected datasets, offline-to-online RL incorporates an online fine-tuning phase, making it beneficial to design the offline pre-training with this subsequent stage in mind from the beginning. At the same time, the framework raises the practical question of how to best exploit the pre-trained policy during online fine-tuning. Thus, framing offline-to-online RL merely as an extension of offline RL can limit the extent to which its potential is realized. In this work, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel policy learning approach for the offline-to-online RL framework. Motivated by recent findings that maintaining diversity facilitates more effective fine-tuning (Fan et al., 2025; Li et al., 2025; Zhai et al., 2025; Sorokin et al., 2025), we introduce a training strategy that injects noise into the flow matching to explicitly promote diversity in the policy from the beginning of offline pre-training. This injection encourages the policy to learn a broader range of action space than that present in the offline dataset, thereby establishing a strong foundation for exploration during online fine-tuning. To effectively leverage this during online fine-tuning, we introduce an entropy-guided sampling mechanism that exploits the acquired diversity for exploration while balancing exploration and exploitation by adapting to the evolving behavior of the policy. We experiment on 45 diverse and challenging tasks from OGBench (Park et al., 2025a) and D4RL (Fu et al., 2020) under a limited online fine-tuning budget. The results show that FINO achieves consistently strong performance across tasks, even in complex environments, thereby demonstrating FINO as an effective and reliable approach for offline-to-online RL. 2 Preliminaries Offline-to-Online Reinforcement Learning. In this paper, we consider a Markov Decision Process (MDP) (Sutton et al., 1998) ℳ=(,,r,,γ)M=(S,A,r,P,γ), where S denotes the state space, A the action space, r the reward function, P the transition probability distribution, and γ the discount factor. The objective of RL is to train a policy that maximizes the expected cumulative return π[∑iγir(si,ai)]E_π[ _iγ^ir(s_i,a_i)]. Offline-to-online RL is a two-stage learning framework consisting of offline pre-training and online fine-tuning (Lee et al., 2022; Zhang et al., 2023; Nakamoto et al., 2023; Zhang et al., 2024; Zhou et al., 2024). This framework is designed to combine the strengths of offline and online RL: the stability gained from pre-collected datasets without interactions and the adaptability that comes from environment interaction. In the offline pre-training, a policy is trained on a static dataset D=(s,a,r,s′)D=\(s,a,r,s )\, providing a reliable initialization. Subsequently, during the online fine-tuning, the pre-trained policy directly interacts with the environment, allowing it to refine its behavior and correct limitations inherited from the offline dataset. Flow Matching. Flow matching (Lipman et al., 2023; Albergo and Vanden-Eijnden, 2023) is a generative modeling framework that constructs a transformation between two probability distributions via ordinary differential equations (ODEs). Unlike diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020), which rely on stochastic differential equations (SDEs), flow matching is based on deterministic ODEs. This design not only simplifies training but also enables faster inference. The central component of flow matching is a time-dependent vector field vθ(t,x)v_θ(t,x) that defines a flow ϕt _t, mapping a base distribution p0p_0 into a target data distribution p1p_1: dtϕt(x)=vθ(t,ϕt(x)),ϕ0(x)=x. ddt _t(x)=v_θ(t, _t(x)), _0(x)=x. (1) A widely used formulation of flow matching is based on Optimal Transport (OT) (Lipman et al., 2023), where transformations are constructed by linearly interpolating between samples from the base and target distributions (x0∼p0x_0 p_0 and x1∼p1x_1 p_1), with the interpolation time t sampled uniformly: xt=(1−t)x0+tx1,t∼Unif([0,1]).x_t=(1-t)x_0+tx_1, t ([0,1]). (2) The vector field is trained to align its prediction with the direction of this linear path: minθx0∼p0,x1∼p1,t∼Unif([0,1])[‖vθ(t,xt)−(x1−(1−σmin)x0)‖22]. _θE_ subarraycx_0 p_0,x_1 p_1,\\ t ([0,1]) subarray [||v_θ(t,x_t)- (x_1-(1- _ )x_0 )||^2_2 ]. (3) where σmin _min is a sufficiently small constant. Once the vector field vθv_θ is trained, generation is performed by sampling x0∼p0x_0 p_0 and solving the learned ODE until t=1t=1 to obtain ϕ1(x0)∼p1 _1(x_0) p_1. In this work, we use the Euler method to solve the ODE for sample generation. Flow Q-Learning. Flow Q-Learning (FQL) (Park et al., 2025b) applies flow matching to policy design for offline RL. It formulates the policy as a state-conditioned flow model and trains it by adapting flow matching to behavior cloning: ℒπ(θ)=x0∼(0,I),s,a=x1∼D,t∼Unif([0,1])[‖vθ(t,s,xt)−(x1−x0)‖22].L_π(θ)=E_ subarraycx_0 (0,I),\\ s,a=x_1 D,\\ t ([0,1]) subarray [||v_θ(t,s,x_t)-(x_1-x_0)||^2_2 ]. (4) Integrating the trained vector field vθv_θ induces a mapping aθ(s,z)a_θ(s,z) from state s and noise z to action, which defines a policy βθ _θ, linking the flow formulation to a policy representation. To enable efficient training, FQL further introduces a one-step policy πω _ω, which is jointly optimized by distillation from the flow policy and action-value maximization: ℒπ(ω)=s∼D,z∼(0,I),aω(s,z)∼πω[−Qϕ(s,aω(s,z))+α‖aω(s,z)−aθ(s,z)‖22],L_π(ω)=E_ subarraycs D,\\ z (0,I),\\ a_ω(s,z) _ω subarray [\;-\;Q_φ(s,a_ω(s,z))\;+\;α\,\|a_ω(s,z)-a_θ(s,z)\|_2^2 ], (5) where α is a hyperparameter. In practice, the one-step policy provides a direct mapping from noise to actions without sequential ODE integration, enabling efficient action selection while inheriting the expressiveness of the flow model. 3 Motivation Figure 1: Comparison of FQL and FINO (ours) in terms of performance and exploration patterns on the environment antmaze-giant-navigate. The green circle and red star indicate the initial and goal states, respectively. Our motivation lies in better leveraging the expressivity of generative policy, flow policy in particular, to address the challenges of offline-to-online RL. There exist prior studies in offline RL (Hansen-Estruch et al., 2023; Park et al., 2025b) employing generative policies and extending them to online fine-tuning. To examine their behavior during online fine-tuning, we conducted an experiment with the challenging task antmaze-giant-navigate with FQL (Park et al., 2025b). The second plot of Figure 1 illustrates the maze, where the gray region marks the feasible paths, while the third plot shows the visitation frequency of the FQL agent during the first 100k interaction steps. It is seen that the agent stays mostly near the starting point and reaches the goal only via the upper path, ignoring other possible routes, yielding degraded performance as shown in the first plot in Figure 1. This behavior reflects an offline pre-trained policy that is overly confined to the dataset, which mainly contains the upper success route, resulting in limited exploration during online fine-tuning. In a strictly offline setting, such confinement to the data distribution is a primary design objective to ensure stability. But, considering the subsequent online learning, such confinement may not be the best strategy. One could address this limitation by constructing a larger dataset with diverse data, but this incurs additional cost and time. Then, how can one induce diverse behavior from a given dataset without increasing the dataset size? To answer this question, we propose perturbed cloning during the offline pre-training phase, especially suited to flow-based policy learning. In our training scheme, we inject noise into flow matching, thereby driving the flow behavior model to extend its support beyond the coverage of the dataset to some reasonable extent. The policy is then distilled from this perturbed behavior model, which allows it to acquire behaviors spanning a broader action space. So learned policies can leverage this broader coverage during subsequent online fine-tuning, facilitating effective exploration to yield better performance, as shown in the rightmost plot of Figure 1. Details of the proposed method follow in the next section. 4 Method In this section, we present Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that utilizes flow policies within the offline-to-online RL framework. Our approach consists of two main components: (1) from the beginning of offline pre-training, we inject controlled noise into flow matching, encouraging the policy to explore a broader range of actions beyond those in the dataset; (2) in the online fine-tuning, we leverage this expanded action space for exploration, while introducing an entropy-guided sampling mechanism that dynamically balances exploration and exploitation according to the behavior of the policy. 4.1 Noise Injection for Flow Matching Since our method builds directly on the flow matching formulation, we begin by presenting its conditional probability path. The rationale behind training flow matching as in Equation 3 lies in the design of its conditional probability path (Lipman et al., 2023): ptFM(x|x1)=(x|tx1,(1−(1−σmin)t)2I)p^FM_t(x|x_1)=N(x|tx_1,(1-(1- _ )t)^2I) (6) where σmin _min is a sufficiently small constant ensuring that p1FM(x|x1)p^FM_1(x|x_1) concentrates around the given data point. In FQL (Park et al., 2025b), the variance is set to σmin=0 _ =0 in Equation 6, which reduces the training objective to Equation 4. With σmin=0 _ =0, the distribution collapses onto individual data points, leaving little coverage beyond the dataset itself. This narrow formulation shows clear limitations as shown in the previous section, as it restricts effective exploration during online fine-tuning. To overcome this limitation, we propose a noise-injected training scheme that retains the core objective of flow matching while enabling the model to learn a wider action space than point-wise matching: ℒFINO(θ)=s,a=x1∼D,x0∼(0,I),t∼Unif([0,1])[‖vθ(t,s,xt+ϵt)−(x1−(1−η)x0)‖22],ϵt∼(0,αt2I)L_FINO(θ)=E_ subarraycs,a=x_1 D,\\ x_0 (0,I),\\ t ([0,1]) subarray [||v_θ(t,s,x_t+ _t)-(x_1-(1-η)x_0)||^2_2 ], _t (0, _t^2I) (7) where αt2=(η2−2η)t2+(2η)t _t^2= (η^2-2η )t^2+ (2η )t is the scheduled variance for some η∈[0,1]η∈[0,1], and t is the interpolation time. Equation 7 reduces to the standard flow matching (Equation 4) when η=0η=0. The variance of the injected noise is non-negative for all η≥0η≥ 0 and t∈[0,1]t∈[0,1]. Note that at t=0t=0, α02=0 _0^2=0, and at t=1t=1, α12=η2>0 _1^2=η^2>0. Proposition 1. For notational simplicity, we denote (si,xi1)(s_i,x_i^1) as xix_i. Given a dataset =xii=1ND=\x_i\_i=1^N, the proposed time-dependent noise injection ϵt∼(0,αt2I) _t (0, _t^2I) induces the following conditional probability paths of flow ϕt _t: ptFINO(x|xi)=(x|μt(xi)=txi,Σt(xi)=(1−(1−η)t)2I), p^FINO_t(x|x_i)=N (x\; |\; _t(x_i)=tx_i,\, _t(x_i)= (1-(1-η)t )^2I ), in which the mean txitx_i is equal to the mean induced from flow matching, and the variance (1−(1−η)t)2(1-(1-η)t)^2 is greater than or equal to the variance induced from flow matching. Theorem 1. Given a data xix_i from a dataset distribution and a noise x0x_0 from the base distribution, the conditional probability paths in Proposition 1 induce the unique conditional vector field that has the following form: vt(x|xi)=xi−(1−η)x0. v_t(x|x_i)=x_i-(1-η)x_0. Then, for any dataset distribution, the marginal vector field vt(x)v_t(x) generates the marginal probability path pt(x)p_t(x), in other words, both vt(x)v_t(x) and pt(x)p_t(x) satisfy the continuity equation. Theorem 1 shows that FINO (Equation 7) yields a valid continuous normalizing flow, which means that the flow model trained by Equation 7 can generate samples close to those obtained by the behavior policy of the training dataset. Theorem 2. Suppose the cardinality of the dataset is finite, =x1,x2,…,xND=\x_1,x_2,…,x_N\ for some N, and data are independently and identically distributed (i.i.d.) sampled. The variance of the marginal probability path induced by FINO (Equation 7) is greater than or equal to that of the marginal probability path induced by the flow matching (FM) objective (Equation 3). For any time t∈[0,1]t∈[0,1], Var(XtFINO)≥Var(XtFM),XtFINO∼ptFINO(x),XtFM∼ptFM(x). (X_t^FINO ) (X_t^FM ), X_t^FINO p_t^FINO(x),\;X_t^FM p_t^FM(x). Theorem 2 states that at time t=1t=1, the marginal probability path induced by FINO (p1FINO(x)p_1^FINO(x)) exhibits larger variance than flow matching (p1FM(x)p_1^FM(x)). This means that the model trained by Equation 7 represents wider action regions than the flow matching model (Equation 3), making it more suitable for exploration. The proofs of Proposition 1 and Theorems 1, 2 are provided in Appendix C. To illustrate the effect of our design, we conduct a simple toy experiment. We consider a setting with a fixed state and a two-dimensional action space, where the dataset is generated by sampling points Figure 2: Toy example: blue contours represent the log-density of model samples; red circles denote the dataset. inside four circular regions. Both flow matching and our proposed method are trained on the same dataset. As shown in Figure 2, flow matching predominantly focuses on the data points themselves, leading the trained actions to remain almost entirely within the dataset distribution. In contrast, our method with noise injection learns to cover a wider region of the action space. Notably, this expansion occurs in a reliable manner, remaining centered around the dataset and thereby providing a broader yet plausible coverage of the action space. The expanded flow model then guides the training of the one-step policy that interacts with the environment. As the one-step policy is trained under Equation 5, the expanded flow model enables action-value maximization over a broader region of the action space, which is then utilized for exploration during online fine-tuning. We provide a detailed explanation in Appendix D. Algorithm 1 FINO: Flow Matching with Injected Noise for Offline-to-Online RL 1: Inputs: flow matching policy βθ _θ, one-step policy πω _ω, value function QϕQ_φ, candidate action samples NsampleN_sample, entropy update steps NξN_ξ 2: while in offline pre-training do 3: Update ω,θω,θ based on Equation 5, 7 and update ϕφ via TD loss 4: end while 5: while in online fine-tuning do 6: Sample NsampleN_sample candidate actions a1,a2,⋯,aNsample∼πω(s)\a_1,a_2,·s,a_N_sample\ _ω(s) 7: Compute sampling probability p(i)p(i) using Equation 8 8: Select a from categorical distribution p 9: Update ω,θω,θ based on Equation 5, 7 and update ϕφ via TD loss 10: if step mod NξN_ξ == 0 then 11: Estimate the entropy of policy ℋH 12: Update ξ using Equation 9 13: end if 14: end while 4.2 Entropy-Guided Sampling After offline pre-training, the next step is to leverage the policy effectively during online fine-tuning, where the agent continues improving through direct interaction with the environment. With noise injection, the policy is trained to yield more diverse actions, each reflecting slightly different behaviors. To exploit this action diversity, the agent first samples multiple candidate actions for a given state using multiple base noises for the flow model. Now, we do not simply choose the action that maximizes action-value, which corresponds to exploitation. Instead, we construct a sampling distribution over the candidate actions based on their action-values as psampling(i)=exp(ξ⋅Qϕ(s,ai))∑jexp(ξ⋅Qϕ(s,aj)),∀i∈[1,⋯,Nsample]p_sampling(i)= (ξ· Q_φ(s,a_i)) _j (ξ· Q_φ(s,a_j)), ∀ i∈[1,·s,N_sample] (8) where ξ is a temperature parameter. An actual action is drawn from psamplingp_sampling, so that even lower-value actions can be sampled for exploration purposes. A smaller ξ produces a flatter distribution that promotes more uniform sampling and encourages exploration, whereas a larger ξ sharpens the distribution and prioritizes greedy actions for exploitation. Since sample-efficient learning under a limited interaction budget is the primary objective of online fine-tuning, maintaining an appropriate balance between exploration and exploitation remains a critical challenge. However, relying on a fixed value of ξ cannot adequately address the dynamics of the learning process. So, we adapt the sampling strategy to the behavior of the policy, using entropy of the policy (ℋH) as an indicator and adjusting ξ accordingly: ξnew=ξ−αξ[ℋ−ℋ¯], _new=ξ- _ξ[H- H], (9) where ℋ¯ H is the target entropy, and αξ _ξ denotes the learning rate. By adapting its behavior according to the policy entropy, it properly controls the balance between exploration and exploitation throughout online fine-tuning. At inference time, the agent deterministically selects the action with the highest action-value, ensuring stable performance. The overall training pipeline is summarized in Algorithm 1. 4.3 Practical Implementation We use FQL (Park et al., 2025b) as the backbone model, and accordingly the one-step policy, trained with Equation 5 is employed for environment interaction. Since this policy is obtained through distillation and action-value maximization, its distribution is intractable, making direct entropy computation infeasible. To address this, we follow prior work (Wang et al., 2024) and estimate entropy by sampling multiple actions from the same state and fitting them with a Gaussian Mixture Model (GMM). A detailed description of the computation procedure is provided in Appendix E.1. Regarding hyperparameters, FINO involves two key parameters. For η, which determines the variance of the injected noise, we set its value based on the action range. Since all experimental environments use actions bounded within [−1,1][-1,1], we fix η=0.1η=0.1. For NsampleN_sample, as the volume of the action space to explore grows significantly with the dimension, more samples are required to obtain a sufficiently diverse set of candidates for effective exploration. Therefore, we set the number of sampled actions to half of the action dimension. In implementation, we adopt a smooth shifted exponential schedule for αt _t that satisfies the same boundary conditions, i.e., α02≈0 _0^2≈ 0 and α12=η2>0 _1^2=η^2>0, and we simply use the target vector x1−x0x_1-x_0, as we empirically observed no difference in performance. We note that these core hyperparameters remain fixed throughout the training process. Further implementation details and additional hyperparameter settings are provided in Appendix E and G. 5 Experiments In this section, we empirically demonstrate the effectiveness of FINO. To this end, we evaluate the proposed method across a range of challenging environments, comparing its performance against several baselines. Table 1: Performance of FINO and baselines across OGBench and D4RL tasks. Results show scores after offline pre-training and after online fine-tuning, averaged over 10 seeds with mean and 95% confidence intervals. D4RL antmaze and adroit aggregate six and four tasks, respectively, while OGBench reports results over five tasks (task names abbreviated by omitting the singletask suffix). Full results are presented in Table 4. Task ReBRAC Cal-QL RLPD IFQL FQL FINO OGBench humanoidmaze-medium-navigate 21± 5 → 3± 3 0± 0 → 0± 0 0± 0 → 1± 1 59± 7 → 70± 5 53± 6 → 61± 2 50± 7 → 97± 1 OGBench humanoidmaze-large-navigate 2± 1 → 1± 0 0± 0 → 0± 0 0± 0 → 0± 0 11± 3 → 10± 2 5± 1 → 10± 3 6± 2 → 33± 7 OGBench antmaze-large-navigate 85± 3 → 99± 0 12± 9 → 12± 8 0± 0 → 80± 7 32± 4 → 72± 6 81± 2 → 92± 1 81± 2 → 99± 0 OGBench antmaze-giant-navigate 35± 7 → 96± 4 2± 3 → 0± 0 0± 0 → 47± 9 1± 1 → 0± 0 16± 6 → 71± 5 14± 6 → 79± 0 OGBench antsoccer-arena-navigate 0± 0 → 0± 0 0± 0 → 0± 0 0± 0 → 2± 1 33± 4 → 35± 5 61± 2 → 74± 4 57± 3 → 77± 5 OGBench cube-double-play 9± 2 → 28± 2 2± 3 → 0± 0 0± 0 → 2± 3 14± 1 → 40± 2 31± 3 → 73± 2 34± 3 → 79± 2 OGBench puzzle-4x4-play 14± 1 → 29± 5 2± 3 → 20± 8 0± 0 → 58± 11 26± 2 → 42± 4 15± 2 → 45± 8 19± 3 → 56± 5 D4RL antmaze 80± 5 → 89± 5 50± 3 → 89± 2 0± 0 → 91± 2 66± 5 → 79± 5 80± 4 → 95± 1 79± 4 → 96± 1 D4RL adroit 21± 2 → 83± 2 -0± 0 → -0± 0 0± 0 → 73± 5 18± 1 → 42± 3 14± 3 → 100± 6 13± 2 → 112± 1 Environments. We primarily evaluate the performance of FINO on OGBench (Park et al., 2025a), a recently proposed benchmark that extends beyond the commonly used D4RL (Fu et al., 2020) by incorporating tasks with greater diversity and complexity. Although OGBench is originally introduced for benchmarking offline goal-conditioned RL, we adapt it to our setting by employing its single-task variant, where each goal is treated as an independent task. We also include results on the widely adopted D4RL benchmark, which remains a common benchmark in offline-to-online RL. Baselines. We consider the following baselines for comparison: (1) ReBRAC (Tarasov et al., 2023) is a Gaussian policy-based method, has demonstrated strong performance across offline RL and the offline-to-online RL setting. (2) Cal-QL (Nakamoto et al., 2023) is a representative offline-to-online RL algorithm that extends CQL (Kumar et al., 2020) to the offline-to-online setting. (3) RLPD (Ball et al., 2023) is an online RL algorithm initialized with an offline dataset, which achieves superior performance despite relying solely on online training. Since prior work has provided limited investigation of flow matching and denoising diffusion in the offline-to-online RL setting, we additionally construct flow matching variants of existing algorithms to provide a meaningful point of comparison. (4) IFQL, introduced in the FQL paper (Park et al., 2025b), is an adaptation of IDQL (Hansen-Estruch et al., 2023) to the flow matching setting. Similar to our approach, it samples multiple actions from a single state and selects one for execution. (5) FQL (Park et al., 2025b), described in Section 2, serves as the backbone algorithm upon which our method is built. Evaluation. For all baselines, we report results using the same experimental protocol, consisting of 1M offline pre-training steps followed by 500K online fine-tuning steps. To assess the performance gain during online fine-tuning, we present both the results immediately after offline pre-training and those obtained at the end of online fine-tuning. All experiments are averaged over 10 random seeds, and we report the mean and 95% confidence intervals. The best-performing results are highlighted in bold when they fall within 95% of the best performance. Figure 3: Aggregate performance across two benchmark domains. Each figure reports the averaged learning curves over the common environments within the respective domain. Full results are presented in Figures 9 and 10. Results. Table 1 summarizes the results across a total of 45 tasks, aggregated by task category. Overall, FINO consistently achieves strong performance across a diverse range of environments. Crucially, this is achieved without degrading offline performance, as our method learns the model that preserves the mean of the probability path while increasing variance from the offline dataset, supported by Theorem 2. When compared to ReBRAC (Tarasov et al., 2023), we observe that although ReBRAC exhibits strong performance on environments such as antmaze, it struggles to effectively learn in more complex and challenging humanoidmaze environments due to the inherent limitations of its conventional policy. The comparison with IFQL underscores that action candidate sampling alone is insufficient to explain the performance improvement. In addition, when compared to the backbone algorithm FQL (Park et al., 2025b), the results highlight the effectiveness of our method during online fine-tuning, where FINO demonstrates both efficient exploration and a balanced trade-off between exploration and exploitation. The improvements observed in the navigate environments further suggest that FINO is well suited to environments where effective exploration is critical. In addition to the tabular summary, we provide aggregate learning curves by benchmark in Figure 3 to visualize the progression of performance over training steps. Consistent with the results in Table 1, the figure shows that FINO consistently outperforms the baselines throughout training. In particular, the experiments on OGBench demonstrate that, despite starting from the same performance as the backbone algorithm, our method achieves stronger improvements, underscoring its high sample efficiency. 6 Discussion 6.1 Impact of Noise Injection Point Figure 4: Comparison between FINO and the direct action noise injection baseline. Each plot shows results aggregated over five tasks and averaged across 10 seeds, with shaded regions indicating 95% confidence intervals. One of the key components of the proposed method is the injection of noise into the flow matching objective, which expands the action space and enables more efficient exploration during online fine-tuning. To evaluate this design choice, we compare our approach with a simpler alternative that promotes exploration by injecting Gaussian noise into the actions generated by the policy rather than into the flow matching objective (denoted as Action Noise). For a fair comparison, we retain other components such as the action-candidate mechanism and entropy guidance in this baseline as well. The results in Figure 4 demonstrate that the noise injection strategy of the proposed method yields a notable performance gain. Consistent improvement is observed across both navigation and manipulation environments, indicating that the proposed noise injection scheme remains effective across task categories. This difference arises because, as mentioned in Section 4.1, the proposed method enables the one-step policy to maximize action-value over a broader action space, rather than simply adding noise to the action. Notably, the results on door-cloned show that, when compared with the backbone algorithm FQL (whose performance is approximately 100), simply adding noise to the action fails to facilitate exploration and can even degrade performance. These findings highlight that the proposed noise injection method serves as an effective approach for promoting exploration during online fine-tuning. Additional analyses of various noise injection strategies are provided in Appendix D. 6.2 Comparison with Entropy-Regulated Noise Scaling Figure 5: Comparison between FINO and the entropy-regulated noise scaling baseline. Full results are presented in Table 5. In Section 4.2, we introduce an entropy-based guidance method for action sampling. This approach enables the policy to achieve a balanced trade-off between exploration and exploitation during online fine-tuning, which is critical under a limited online budget. Since previous studies (Haarnoja et al., 2018; Wang et al., 2024) have also employed entropy to regulate this balance, we compare our method with an alternative entropy-driven strategy (denoted as ER-Noise) to evaluate its effectiveness. Specifically, instead of using entropy to guide the action sampling, the baseline replaces it with a simpler approach that scales the Gaussian noise based on the entropy. The noise is then directly added to the action, allowing the action to be adjusted according to the entropy of the policy. Figure 5 shows that the proposed method significantly outperforms entropy-based noise scaling by effectively balancing exploration and exploitation through sampling aligned with the entropy of the policy. In particular, despite the inherent difficulty of finding relevant actions in high-dimensional action spaces such as humanoidmaze, the proposed method successfully identifies appropriate actions in such settings. However, the performance of ER-Noise indicates that mere noise scaling guided by entropy is insufficient for selecting actions consistent with the policy. These findings thus confirm that the proposed method effectively leverages entropy to enable sample-efficient learning during online fine-tuning. 6.3 Analysis of Noise Injection and Entropy-Guided Sampling Figure 6: Comparison of performance and computational efficiency. The left figure shows the learning curve on the humanoidmaze-medium-navigate task, averaged over five tasks with 10 random seeds, with shaded regions denoting 95% confidence intervals. The middle and right figures report the training and inference time per step of each baseline. Full results are presented in Table 5. In our proposed method, we incorporate two key components, namely noise injection during offline pre-training and entropy-guided sampling during online fine-tuning. To clarify the contribution of each element, we design controlled experiments that reflect its intended role. In particular, we first examine the case without injected noise (w/o Noise), where the procedure reduces to the same formulation as Equation 4 while still retaining entropy-guided sampling. We then consider the case without entropy guidance (w/o Guidance), in which the sampling process still produces action candidates, but the selection is restricted to the one with the highest action-value, thereby excluding the entropy-based balancing mechanism. The left plot of Figure 6 demonstrates that both components are indispensable to the effectiveness of our method. Noise injection, in particular, proves to be especially critical. This is because, without it, the offline pre-training is limited to actions contained in the offline dataset. Even when action candidates are generated, this restriction makes them lack diversity, which in turn leads to insufficient exploration and thereby reduces overall performance. In the absence of entropy guidance, the performance deteriorates in later stages, as the training process fails to maintain an appropriate balance between exploration and exploitation. These observations suggest that both noise injection and entropy-guided sampling play an important role in enabling sample-efficient learning in the offline-to-online RL setting. 6.4 Training and Inference Efficiency Since computational cost is also an important factor in methods employing generative models, we compare our algorithm with the baselines on humanoidmaze-medium, where it achieves the largest performance improvement. The middle and right plots of Figure 6 present the training time and inference time, respectively. The results show that although additional components such as entropy estimation and action candidate sampling slightly increase training time relative to the backbone algorithm, this increase is negligible when compared to algorithms such as Cal-QL, leaving overall training efficiency largely unaffected. Regarding inference time, our method requires fewer samples than baselines such as IFQL, demonstrating that the additional computation does not impose a significant overhead. 6.5 Effect of Action Sample Size (NsampleN_sample) Figure 7: Comparison of performance with varying NsampleN_sample. The proposed method injects noise into the flow matching objective and samples action from a set of action candidates to utilize the expanded policy. Since the size of the exploration space increases with the action dimension of the task, we set the hyperparameter NsampleN_sample, which determines the number of action candidates, to half of the action dimension. To examine the impact of this hyperparameter, we evaluate performance across 6 environments with action dimensions greater than 10 while varying NsampleN_sample. As shown in Figure 7, performance generally improves as the number of action candidates increases, but the marginal gains diminish beyond a certain point. Since NsampleN_sample directly affects inference cost, we adopt the choice of setting it to half of the action dimension, achieving a practical balance between performance and computational overhead. 7 Related Work Offline-to-Online Reinforcement Learning. Offline-to-online RL is a framework that learns through two stages: offline pre-training and online fine-tuning (Lee et al., 2022; Zhang et al., 2023; Wang et al., 2023a; Nakamoto et al., 2023; Zhou et al., 2024). The goal is to improve an agent, initially trained on offline data, by allowing it to further refine through environment interaction. The simplest way to train within this framework is to employ the same offline RL algorithm for both offline pre-training and online fine-tuning (Lyu et al., 2022; Wu et al., 2022; Tarasov et al., 2023). However, this strategy inherits the conservative nature of offline RL methods, which restricts exploration during online fine-tuning (Yu and Zhang, 2023; Luo et al., 2024; Zhang et al., 2024; Kim et al., 2025). Several prior studies have attempted to ease this conservatism and support more effective online fine-tuning (Wang et al., 2023a; Nakamoto et al., 2023). Still, because these approaches remain conservative, they fail to provide sample efficiency through effective exploration (Shin et al., 2025). In contrast, our method leverages the expressivity of generative models to encourage exploration and regulates it with entropy-guided sampling, thereby achieving high sample efficiency. Reinforcement Learning with Generative Models. Recent advances in generative models such as denoising diffusion (Sohl-Dickstein et al., 2015; Ho et al., 2020) and flow matching (Lipman et al., 2023; Albergo and Vanden-Eijnden, 2023) have spurred growing interest in applying these techniques to RL. Among these efforts, research that employs generative models as policies has shown strong results in both offline (Wang et al., 2023b; Hansen-Estruch et al., 2023; Kang et al., 2023; Zhang et al., 2025; Kim et al., 2024b; a; Fang et al., 2025; Park et al., 2025b; Chae et al., 2026) and online (Wang et al., 2024; Psenka et al., 2024; Ding et al., 2024) settings by framing the policy as a state-conditioned generative model. Within the offline-to-online RL framework, there have been studies that exploit the expressive capacity of diffusion models for data augmentation (Liu et al., 2024; Huang et al., 2025). However, no prior work has leveraged such expressivity directly as a policy in this setting. In contrast, our work harnesses the generative model for exploration, demonstrating a method that achieves strong sample efficiency in offline-to-online RL. 8 Conclusion In this work, we propose FINO, a novel approach that leverages the expressivity of flow matching through noise injection and enhances online fine-tuning via entropy-guided sampling. Noise injection, applied to the offline pre-training, broadens the action space and yields a stronger initialization for exploration, while entropy-guided sampling adapts to the policy’s evolving behavior to maintain a workable exploration–exploitation balance. FINO achieves sample-efficient learning across diverse and challenging benchmarks while maintaining modest computational cost. Beyond empirical gains, our study highlights how flow matching can be effectively utilized to address the challenges of offline-to-online RL, and we believe this line of work opens new directions for harnessing generative policies in advancing the broader offline-to-online RL paradigm. Acknowledgements This work was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-I220469, Development of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones, 50%) and in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2025-00557589, Generative Model Based Efficient Reinforcement Learning Algorithms for Multi-modal Expansion in Generalized Environments, 50%). We would like to thank Woohyeon Byeon for providing valuable insights into Theorem 2. References M. Albergo and E. Vanden-Eijnden (2023) Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, Cited by: §1, §2, §7. P. J. Ball, L. Smith, I. Kostrikov, and S. Levine (2023) Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, p. 1577–1594. Cited by: §5. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: §1. J. Chae, J. Park, Y. Shin, G. Kim, S. Han, and Y. Sung (2026) Flow actor-critic for offline reinforcement learning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §7. S. Ding, K. Hu, Z. Zhang, K. Ren, W. Zhang, J. Yu, J. Wang, and Y. Shi (2024) Diffusion-based reinforcement learning via q-weighted variational policy optimization. Advances in Neural Information Processing Systems 37, p. 53945–53968. Cited by: §7. J. Fan, S. Shen, C. Cheng, Y. Chen, C. Liang, and G. Liu (2025) Online reward-weighted fine-tuning of flow matching with wasserstein regularization. In The Thirteenth International Conference on Learning Representations, Cited by: §1. L. Fang, R. Liu, J. Zhang, W. Wang, and B. Jing (2025) Diffusion actor-critic: formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §7. J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine (2020) D4rl: datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. Cited by: §F.1, §1, §5. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, p. 1861–1870. Cited by: §6.2. P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine (2023) Idql: implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573. Cited by: §1, §1, §3, §5, §7. D. Hendrycks and K. Gimpel (2016) Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. Cited by: Table 2. J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §1, §2, §7. X. Huang, X. Liu, E. Zhang, T. Yu, and S. Li (2025) Offline-to-online reinforcement learning with classifier-free diffusion generation. arXiv preprint arXiv:2508.06806. Cited by: §7. M. F. Huber, T. Bailey, H. Durrant-Whyte, and U. D. Hanebeck (2008) On entropy approximation for gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, p. 181–188. Cited by: Appendix A, §E.1. B. Kang, X. Ma, C. Du, T. Pang, and S. Yan (2023) Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems 36, p. 67195–67212. Cited by: §1, §7. J. Kim, S. Lee, W. Kim, and Y. Sung (2024a) Adaptive Q-aid for conditional supervised learning in offline reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §7. J. Kim, S. Lee, W. Kim, and Y. Sung (2024b) Decision convformer: local filtering in metaformer is sufficient for decision making. In International Conference on Learning Representations, Cited by: §7. J. Kim, Y. Shin, W. Jung, S. Hong, D. Yoon, Y. Sung, K. Lee, and W. Lim (2025) Penalizing infeasible actions and reward scaling in reinforcement learning with offline data. In Forty-second International Conference on Machine Learning, Cited by: §7. D. P. Kingma and J. Ba (2015) Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Cited by: Table 2. A. Kumar, A. Zhou, G. Tucker, and S. Levine (2020) Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems 33, p. 1179–1191. Cited by: §5. S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin (2022) Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, p. 1702–1712. Cited by: §1, §2, §7. Z. Li, C. Chen, T. Xu, Z. Qin, J. Xiao, Z. Luo, and R. Sun (2025) Preserving diversity in supervised fine-tuning of large language models. In The Thirteenth International Conference on Learning Representations, Cited by: §1. Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2023) Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, Cited by: Appendix C, Appendix C, Appendix C, Appendix C, Appendix C, §1, §2, §2, §4.1, §7. X. Liu, T. Liu, S. Jiang, R. Chen, Z. Zhang, X. Chen, and Y. Yu (2024) Energy-guided diffusion sampling for offline-to-online reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, p. 31541–31565. Cited by: §7. Q. Luo, M. Xie, Y. Wang, and S. Huang (2024) Optimistic critic reconstruction and constrained fine-tuning for general offline-to-online rl. Advances in Neural Information Processing Systems 37, p. 108167–108207. Cited by: §7. J. Lyu, X. Ma, X. Li, and Z. Lu (2022) Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems 35, p. 1711–1724. Cited by: §7. M. Nakamoto, S. Zhai, A. Singh, M. Sobol Mark, Y. Ma, C. Finn, A. Kumar, and S. Levine (2023) Cal-ql: calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems 36, p. 62244–62269. Cited by: §1, §2, §5, §7. J. Park, M. Cho, and Y. Sung (2024) EMPO: a clustering-based on-policy algorithm for offline reinforcement learing. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, Cited by: §1. S. Park, K. Frans, B. Eysenbach, and S. Levine (2025a) OGBench: benchmarking offline goal-conditioned rl. In The Thirteenth International Conference on Learning Representations, Cited by: §F.1, §1, §5. S. Park, Q. Li, and S. Levine (2025b) Flow q-learning. In International Conference on Machine Learning (ICML), Cited by: §G.1, §G.2, §1, §1, §2, §3, §4.1, §4.3, §5, §5, §7. M. Psenka, A. Escontrela, P. Abbeel, and Y. Ma (2024) Learning a diffusion model policy from rewards via q-score matching. In Proceedings of the 41st International Conference on Machine Learning, p. 41163–41182. Cited by: §7. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10684–10695. Cited by: §1. Y. Shin, J. Kim, W. Jung, S. Hong, D. Yoon, Y. Jang, G. Kim, J. Chae, Y. Sung, K. Lee, and W. Lim (2025) Online pre-training for offline-to-online reinforcement learning. In Forty-second International Conference on Machine Learning, Cited by: §7. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, p. 2256–2265. Cited by: §1, §2, §7. D. Sorokin, M. Nakhodnov, A. Kuznetsov, and A. Alanov (2025) ImageReFL: balancing quality and diversity in human-aligned diffusion models. arXiv preprint arXiv:2505.22569. Cited by: §1. R. S. Sutton, A. G. Barto, et al. (1998) Reinforcement learning: an introduction. Vol. 1, MIT press Cambridge. Cited by: §2. D. Tarasov, V. Kurenkov, A. Nikulin, and S. Kolesnikov (2023) Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems 36, p. 11592–11620. Cited by: §G.2, §5, §5, §7. C. Villani et al. (2008) Optimal transport: old and new. Vol. 338, Springer. Cited by: Appendix C. S. Wang, Q. Yang, J. Gao, M. Lin, H. Chen, L. Wu, N. Jia, S. Song, and G. Huang (2023a) Train once, get a family: state-adaptive balances for offline-to-online reinforcement learning. Advances in Neural Information Processing Systems 36, p. 47081–47104. Cited by: §7. Y. Wang, L. Wang, Y. Jiang, W. Zou, T. Liu, X. Song, W. Wang, L. Xiao, J. Wu, J. Duan, et al. (2024) Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems 37, p. 54183–54204. Cited by: Appendix A, §E.1, §E.1, §4.3, §6.2, §7. Z. Wang, J. J. Hunt, and M. Zhou (2023b) Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, Cited by: §1, §7. J. Wu, H. Wu, Z. Qiu, J. Wang, and M. Long (2022) Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems 35, p. 31278–31291. Cited by: §7. Z. Yu and X. Zhang (2023) Actor-critic alignment for offline-to-online reinforcement learning. In International Conference on Machine Learning, p. 40452–40474. Cited by: §7. S. Zhai, R. ZHANG, P. Nakkiran, D. Berthelot, J. Gu, H. Zheng, T. Chen, M. Á. Bautista, N. Jaitly, and J. M. Susskind (2025) Normalizing flows are capable generative models. In Forty-second International Conference on Machine Learning, Cited by: §1. H. Zhang, W. Xu, and H. Yu (2023) Policy expansion for bridging offline-to-online reinforcement learning. In The Eleventh International Conference on Learning Representations, Cited by: §1, §2, §7. S. Zhang, W. Zhang, and Q. Gu (2025) Energy-weighted flow matching for offline reinforcement learning. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §7. Y. Zhang, J. Liu, C. Li, Y. Niu, Y. Yang, Y. Liu, and W. Ouyang (2024) A perspective of q-value estimation on offline-to-online reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 16908–16916. Cited by: §1, §2, §7. Z. Zhou, A. Peng, Q. Li, S. Levine, and A. Kumar (2024) Efficient online reinforcement learning fine-tuning need not retain offline data. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §2, §7. Appendix A Limitations The entropy of the policy is approximated using a Gaussian Mixture Model (GMM) (Huber et al., 2008), which incurs computational overhead and remains an approximation rather than an exact calculation (Wang et al., 2024). Future work could focus on developing entropy estimation methods that are both computationally efficient and more precise, or on exploring alternative metrics that capture policy behavior beyond entropy. In addition, while our method directly addresses the challenges of exploration and the exploration–exploitation trade-off in offline-to-online RL, the issue of distribution shift remains. Addressing this challenge constitutes another promising direction for future research. Appendix B The Use of Large Language Models (LLMs) Large Language Models are used solely to aid and polish the writing. They are not involved in research ideation, methodological design, or experimental analysis. Appendix C Theoretical Proofs See 1 Proof. We consider time-dependent conditional probability paths of flow ϕt(x) _t(x) as follows: pt(x|xi)=(x|μt(xi),σt2(xi)I) p_t(x|x_i)=N (x| _t(x_i), _t^2(x_i)I ) (10) Following the flow matching (Lipman et al., 2023), we set the time-dependent mean and variance as μt(xi)=txi,σt(xi)=1−(1−σmin)t, _t(x_i)=tx_i, _t(x_i)=1-(1- _min)t, where σmin _min is a negligible small constant. The conditional probability paths provide the following canonical transformation for Gaussian distribution, i.e., the flow conditioned on xix_i: ϕt(x) _t(x) =σt(xi)x+μt(xi)=(1−(1−σmin)t)x+txi = _t(x_i)x+ _t(x_i)=(1-(1- _min)t)x+tx_i ≈(1−t)x+txi ≈(1-t)x+tx_i By injecting the introduced time-dependent noise ϵt _t, we obtain the perturbed flow ϕ~t(x) φ_t(x) =txi+(1−t)x+ϵt, =tx_i+(1-t)x+ _t, where x is distributed as normal distribution (0,I)N(0,I). Since the injected noise is designed as a Gaussian distribution (0,αt2I)N(0, _t^2I) and is independent over distribution 10, the perturbed flow can be expressed as the sum of two Gaussian distributions. This leads to the following conditional probability paths: p~t(x|xi) p_t(x|x_i) =(x|txi,((1−t)2+αt2)I) =N (x|tx_i,((1-t)^2+ _t^2)I ) =(x|txi,((1−t)2+(η2−2η)t2+2ηt)I) =N (x|tx_i,((1-t)^2+(η^2-2η)t^2+2η t )I) =(x|txi,(1−(1−η)t)2I) =N (x|tx_i,(1-(1-η)t )^2I) as desired. ∎ See 1 Proof. The conditional probability path pt(xt|xi)=pt(ϕt(x0)|xi)p_t(x_t|x_i)=p_t( _t(x_0)|x_i) provides the canonical transformation of Gaussian distribution as the perturbed flow ϕt(x) _t(x) conditioned on xix_i: xt=ϕt(x0)=txi+(1−(1−η)t)x0, x_t= _t(x_0)=tx_i+(1-(1-η)t)x_0, its derivative is the vector field that generates the flow ϕt _t by the definition of vector fields of continuous normalizing flow (Lipman et al., 2023). dtϕt(x0) ddt _t(x_0) =dt(txi+(1−(1−η)t)x0)=xi−(1−η)x0, = ddt (tx_i+(1-(1-η)t)x_0 )=x_i-(1-η)x_0, which is the same result as one from Theorem 3 of (Lipman et al., 2023): vt(xt|xi) v_t(x_t|x_i) =σt′(xi)σt(xi)(xt−μt(xi))+μt′(xi), = _t (x_i) _t(x_i)(x_t- _t(x_i))+ _t (x_i), =−(1−η)1−(1−η)t(xt−txi)+xi = -(1-η)1-(1-η)t(x_t-tx_i)+x_i =−(1−η)xt+(1−η)txi+(1−(1−η)t)xi1−(1−η)t = -(1-η)x_t+(1-η)tx_i+(1-(1-η)t)x_i1-(1-η)t =−(1−η)(txi+(1−(1−η)t)x0)+xi1−(1−η)t = -(1-η)(tx_i+(1-(1-η)t)x_0)+x_i1-(1-η)t =(1−(1−η)t)xi−(1−η)(1−(1−η)t)x01−(1−η)t = (1-(1-η)t)x_i-(1-η)(1-(1-η)t)x_01-(1-η)t =xi−(1−η)x0 =x_i-(1-η)x_0 where f′f denotes the derivative w.r.t interpolation time t, and σt(xi)I=Σt(xi) _t(x_i)I= _t(x_i) and μt(xi) _t(x_i) are the covariance and the mean of pt(xt|xi)p_t(x_t|x_i), respectively. This means that the Gaussian probability path pt(xt|xi)p_t(x_t|x_i) induces the unique conditional vector field vt(xt|xi)=xi−(1−η)x0v_t(x_t|x_i)=x_i-(1-η)x_0. Since the conditional probability path pt(xt|xi)p_t(x_t|x_i) has the form of Gaussian probability distribution, and the unique vector field vt(xt|xi)v_t(x_t|x_i) generates the perturbed flow ϕt(x) _t(x) conditioned on xix_i, we can use the definition of the marginal probability paths pt(xt)p_t(x_t) and the marginal vector field vt(xt)v_t(x_t) in flow matching (Lipman et al., 2023): pt(x)=∫pt(xt|xi)q(xi)xi,p1(xi)≈q(xi) p_t(x)= p_t(x_t|x_i)q(x_i)dx_i, p_1(x_i)≈ q(x_i) vt(x)=∫vt(x|xi)pt(x|xi)q(xi)pt(xi)xi, v_t(x)= v_t(x|x_i) p_t(x|x_i)q(x_i)p_t(x_i)dx_i, where q is a dataset distribution, both marginal probability paths pt(xt)p_t(x_t) and vector field vt(xt)v_t(x_t) satisfy the continuity equation (Villani and others, 2008) (refer to Theorem 1 of Lipman et al. (2023)). ∎ See 2 Proof. For notational simplicity, given a data xix_i, let ptFINO(x)p_t^FINO(x) and ptFINO(x|xi)p_t^FINO(x|x_i) be the marginal and conditional probability paths induced by equation 7, and ptFM(x)p_t^FM(x) and ptFM(x|xi)p_t^FM(x|x_i) be them induced by equation 3. Since data are i.i.d. sampled, we can write the marginal probability paths as follows: ptFINO(x)=∫ptFINO(x|xi)q(xi)xi=1N∑i=1NptFINO(x|xi) p_t^FINO(x)= p_t^FINO(x|x_i)q(x_i)dx_i= 1N _i=1^Np_t^FINO(x|x_i) ptFM(x)=∫ptFM(x|xi)q(xi)xi=1N∑i=1NptFM(x|xi) p_t^FM(x)= p_t^FM(x|x_i)q(x_i)dx_i= 1N _i=1^Np_t^FM(x|x_i) To simplify notation, we denote random variables as XtX_t and XtiX_t^i that follow a marginal distribution pt(x)p_t(x) and pt(x|xi)p_t(x|x_i), respectively, given a data xix_i. Assume the distributions pt(x)p_t(x) and pt(x|xi)p_t(x|x_i) have identity covariances, then, for fixed t, the random variable XtX_t from the marginal probability path pt(x)p_t(x) has the variance as follows (the following equation can apply to both ptFINO(x)p_t^FINO(x) and ptFM(x)p_t^FM(x) since their conditional probability paths are isotropic Gaussian distributions): Var(Xt) (X_t ) =pt[Xt2]−pt[Xt]2=∫pt(x)x2x−(∫pt(x)xx)2 =E_p_t[X_t^2]-E_p_t[X_t]^2= p_t(x)x^2dx- ( p_t(x)xdx )^2 =∫(1N∑i=1Npti(x))x2x−(∫(1N∑i=1Npti(x))xx)2 = ( 1N _i=1^Np_t^i(x) )x^2dx- ( ( 1N _i=1^Np_t^i(x) )xdx )^2 =1N∑i∫pti(x)x2x−(1N∑i∫pti(x)xx)2 = 1N _i p_t^i(x)x^2dx- ( 1N _i p_t^i(x)xdx )^2 =1N∑ipti[Xt2]−(1N∑ipti[Xt])2 = 1N _iE_p_t^i[X_t^2]- ( 1N _iE_p_t^i[X_t] )^2 =1N2(∑iNpti[Xt2]−∑i=1N∑j=1Npti[Xt]ptj[Xt]) = 1N^2 ( _iNE_p_t^i[X_t^2]- _i=1^N _j=1^NE_p_t^i[X_t]E_p_t^j[X_t] ) =1N2(∑iNpti[Xt2]−∑ipti[Xt]2−∑i∑j:j≠ipti[Xt]ptj[Xt]) = 1N^2 ( _iNE_p_t^i[X_t^2]- _iE_p_t^i[X_t]^2- _i _j:j≠ iE_p_t^i[X_t]E_p_t^j[X_t] ) =1N2(∑iNpti[Xt2]−∑iNpti[Xt]2+∑i(N−1)pti[Xt]2−∑i∑j:j≠ipti[Xt]ptj[Xt]) = 1N^2 ( _iNE_p_t^i[X_t^2]- _iNE_p_t^i[X_t]^2+ _i(N-1)E_p_t^i[X_t]^2- _i _j:j≠ iE_p_t^i[X_t]E_p_t^j[X_t] ) =1N2(∑iNVar(Xti)+∑i(N−1)pti[Xt]2−∑i∑j:j≠ipti[Xt]ptj[Xt]) = 1N^2 ( _iNVar(X_t^i)+ _i(N-1)E_p_t^i[X_t]^2- _i _j:j≠ iE_p_t^i[X_t]E_p_t^j[X_t] ) Using the equation above, the variance of the marginal probability path ptFINOp_t^FINO can be rewritten as Var(XtFINO)=1N2(∑i=1Nσti,FINOd+∑i(N−1)μti,FINO−∑i∑j:j≠iμti,FINOμtj,FINO), (X_t^FINO)= 1N^2 ( _i=1^NN _t^i,FINOd+ _i(N-1) _t^i,FINO- _i _j:j≠ i _t^i,FINO _t^j,FINO ), (11) where d is the dimension of data xix_i, given data xix_i, σti,FINO _t^i,FINO is the variance of the conditional probability path ptFINO(x|xi)p_t^FINO(x|x_i), and μti,FINO _t^i,FINO is the mean of the path. By the same argument, the variance of the marginal probability path of FM ptFMp_t^FM Var(XtFM)=1N2(∑i=1Nσti,FMd+∑i(N−1)μti,FM−∑i∑j:j≠iμti,FMμtj,FM), (X_t^FM)= 1N^2 ( _i=1^NN _t^i,FMd+ _i(N-1) _t^i,FM- _i _j:j≠ i _t^i,FM _t^j,FM ), (12) where d is the dimension of data xix_i, given data xix_i, σti,FM _t^i,FM is the variance of the conditional probability path ptFM(x|xi)p_t^FM(x|x_i), and μti,FM _t^i,FM is the mean of the path. From Proposition 1, we already have μti,FINO=μti,FM _t^i,FINO= _t^i,FM and σti,FINO≥σti,FM _t^i,FINO≥ _t^i,FM, by subtracting equation 11 from equation 12, then we obtain Var(XtFINO)−Var(XtFM)=1N2∑i=1Nd(σti,FINO−σti,FM)≥0 (X_t^FINO)-Var(X_t^FM)= 1N^2 _i=1^NNd ( _t^i,FINO- _t^i,FM )≥ 0 ∎ Appendix D Analysis of Noise Injection In Section 4.1, we describe a training approach that injects noise into the flow matching objective, enabling the model to learn over a broader action space. In this section, we discuss alternative noise injection strategies that can be applied to the flow matching objective and illustrate their effects through a toy example. Throughout this section, we assume the use of zero-mean noise ϵ∼pnoise ε p_noise (e.g., Gaussian Noise). D.1 Case 1: Injecting Noise to Velocity Target Adding noise to the target velocity in Equation 4 is the simplest form of noise injection. The flow matching objective with the added noise can be written as follows: ℒπ(θ) _π(θ) =x0∼(0,I),s,a=x1∼D,t∼Unif([0,1]),ϵ∼pnoise [‖vθ(t,s,xt)−(x1−x0)−ϵ‖22] =E_ subarraycx_0 (0,I),\\ s,a=x_1 D,\\ t ([0,1]),\\ ε p_noise subarray [||v_θ(t,s,x_t)-(x_1-x_0)-ε||^2_2 ] =[‖vθ(t,s,xt)−(x1−x0)‖22−2(vθ(t,s,xt)−(x1−x0))⊤ϵ+‖ϵ‖22]. =E [\|v_θ(t,s,x_t)-(x_1-x_0)\|_2^2-2\,(v_θ(t,s,x_t)-(x_1-x_0)) ε+\|ε\|_2^2 ]. Since the noise has zero mean (ϵ∼psample[ϵ]=0E_ε p_sample[ε]=0), the second term becomes zero in expectation. Furthermore, because ϵε is independent of θ, the last term ‖ϵ‖22\|ε\|_2^2 is a constant with respect to θ, so it does not contribute to the gradient during optimization. As a result, the total gradient of the objective is identical to that of the original flow matching objective in Equation 4, which means that training proceeds in exactly the same way in expectation. D.2 Case 2: Injecting Noise to Policy Action A straightforward way to encourage exploration is to inject noise directly to the actions generated by the policy. However, as shown in Section 6.1, this approach leads to limited improvement during online fine-tuning. This difference stems from the training process of the one-step policy, as described in Section 4.1, which is the component that directly interacts with the environment. The one-step policy is trained with the following objective: ℒπ(ω)=s∼D,z∼(0,I),aω(s,z)∼πω[−Qϕ(s,aω(s,z))+α‖aω(s,z)−aθ(s,z)‖22].L_π(ω)=E_ subarraycs D,\\ z (0,I),\\ a_ω(s,z) _ω subarray [\;-\;Q_φ(s,a_ω(s,z))\;+\;α\,\|a_ω(s,z)-a_θ(s,z)\|_2^2 ]. It distills the flow model trained from the offline dataset while simultaneously maximizing action-value through the value function. Since our algorithm employs the flow model trained with the expanded action space from Equation 7, the resulting one-step policy learns to explore regions that are more informative for improving action-values. In contrast, simply adding noise to the action ignores action-value information, making it an inherently less efficient exploration strategy. To illustrate this difference, we conduct a comparative experiment in a toy example. In this setting, the state is fixed, and the action is two-dimensional, corresponding to the x- and y- axes in the figure. The dataset is sampled from a Gaussian distribution centered at the origin, and the reward increases monotonically toward the right. We train two flow models using Equations 4 and 7, respectively, and each flow model is then used to train a separate one-step policy via Equation 5. The baseline that uses the flow model trained with Equation 4 and injects Gaussian noise directly into the action output is referred to as the Action Noise. The approach based on the flow model trained with Equation 7 corresponds to our proposed method, which does not apply any modification to the action output. Figure 8: Action samples of two one-step policies: blue contours indicate the log-density of the sampled actions, and red × marks denote the centers of the dataset. Figure 8 shows that, since the reward increases toward the right, the action samples generated by both policies shift rightward relative to the dataset. While adding noise directly to the actions causes the samples to spread in random directions, FINO guides the samples toward regions with higher action-values. This is because the method leverages both the expanded action space and the action-value maximization objective. As a result, the one-step policy is guided toward a more informed learning direction, leading to more effective exploration during online fine-tuning and ultimately explaining the superior performance of our approach. Appendix E Implementation Details We implement our proposed method, FINO, based on the official implementation of FQL111https://github.com/seohongpark/fql. In FINO, the entropy estimation module is adapted from the official DACER implementation222https://github.com/happy-yan/DACER-Diffusion-with-Online-RL. For all baselines except Cal-QL, we adopt the components provided in the FQL implementation, while for Cal-QL333https://github.com/nakamotoo/Cal-QL we rely on its official implementation. E.1 Entropy Estimation In our proposed algorithm, entropy-guided sampling requires an estimate of policy entropy. However, since the one-step policy training objective combines behavior cloning from flow policy with action-value maximization, the entropy cannot be derived in closed form. To address this, we adopt an estimation strategy introduced in prior work (Wang et al., 2024). To compute the policy entropy, we employ a Gaussian Mixture Model (GMM) as an approximation of the action distribution. A GMM represents complex data distributions by combining multiple Gaussian components. Formally, the likelihood of a sample under a GMM is defined as a mixture of Gaussian densities: p(a)=∑k=1Kπk(a|μk,Σk)p(a)= _k=1^K _kN(a| _k, _k) (13) where K denotes the number of Gaussian components and πk∈[0,1] _k∈[0,1] is the mixing coefficient that specifies the probability of selecting the k-th Gaussian, satisfying ∑k=1Kπk=1 _k=1^K _k=1. Training a GMM corresponds to estimating its parameters such that the likelihood of the given data is maximized. To approximate the action distribution of the policy using a GMM, we first sample multiple actions (a1,a2,⋯,aNa^1,a^2,·s,a^N) from the policy for each state. We then fit the GMM to these samples using the Expectation-Maximization (EM) algorithm. The EM algorithm alternates between two iterative phases, namely the expectation step and the maximization step. In the expectation step, the latent probabilities required to compute the likelihood are estimated: γ(zkn)=πk(an|μk,Σk)∑i=1Kπi(an|μi,Σi)γ(z_k^n)= _kN(a^n| _k, _k) _i=1^K _iN(a^n| _i, _i) (14) where γ(zkn)γ(z_k^n) denotes that under the current parameter estimates, the observed data ana^n come from the k-th component of the probability. In the maximization step, the GMM parameters are updated based on these probabilities: πk=1N∑n=1Nγ(zkn),μk=∑n=1Nγ(zkn)⋅an∑n=1Nγ(zkn),Σk=∑n=1Nγ(zkn)(an−μk)(an−μk)T∑n=1Nγ(zkn) _k= 1N _n=1^Nγ(z_k^n), _k= _n=1^Nγ(z_k^n)· a^n _n=1^Nγ(z_k^n), _k= _n=1^Nγ(z_k^n)(a^n- _k)(a^n- _k)^T _n=1^Nγ(z_k^n) (15) By repeating these two steps until convergence, we obtain a GMM that approximates the action distribution of the policy. The entropy of the fitted GMM is then computed following the approach established in prior work (Huber et al., 2008): ℋ≈∑k=1Kπk⋅(−logπk+12log((2πe)d|Σk|))H≈ _k=1^K _k·(- _k+ 12 ((2π e)^d| _k|)) (16) where d denotes the dimensionality of the action space. The entropy estimate of the policy is obtained by averaging this quantity across a batch of sampled states. In practice, we set the number of mixture components to K=3K=3, which we found sufficient across all tasks. The number of states used for entropy estimation is determined by the batch size, and the number of actions sampled per state is fixed at 200, following prior work (Wang et al., 2024). Appendix F Experimental Details F.1 Benchmarks We conduct experiments on 35 tasks from OGBench (Park et al., 2025a) and 10 tasks from D4RL (Fu et al., 2020). For OGBench, we adopt single-task variants (singletask) provided in the benchmark and configure them to fit the offline-to-online RL framework. Each environment consists of five distinct tasks, each defined by a different goal. In our experiments, we use the following 7 environments and datasets: • humanoidmaze-medium-navigate-v0 • humanoidmaze-large-navigate-v0 • antmaze-large-navigate-v0 • antmaze-giant-navigate-v0 • antsoccer-arena-navigate-v0 • cube-double-play-v0 • puzzle-4x4-play-v0 In humanoidmaze, the objective is to control a humanoid robot with a 21-dimensional action space to reach the designated goal. In antmaze and antsoccer, the agent controls a quadrupedal robot with an 8-dimensional action space to navigate the goal; in antsoccer, the robot is required to move a ball to the goal. For cube and puzzle, the agent manipulates a robotic arm with a 5-dimensional action space. The cube task requires pick-and-place, while the puzzle task involves pressing the buttons to solve the puzzle. For D4RL, we evaluate on the following environments and datasets: • antmaze-umaze-v2 • antmaze-umaze-diverse-v2 • antmaze-medium-play-v2 • antmaze-medium-diverse-v2 • antmaze-large-play-v2 • antmaze-large-diverse-v2 • pen-cloned-v1 • door-cloned-v1 • hammer-cloned-v1 • relocate-cloned-v1 The antamze tasks share the same 8-dimensional robot as in OGBench but differ in their environment and dataset settings. The adroit suite (pen, door, hammer, relocate) involves high-dimensional dexterous manipulation tasks, with action spaces exceeding 24 dimensions. F.2 Evaluation We evaluate all baselines during online fine-tuning by reporting the average return over 50 episodes every 50,000 environment steps. For OGBench and the antamze tasks in D4RL, we follow the original evaluation protocol and use the success rate as the performance metric, while for the adroit suite in D4RL, we adopt the normalized score. All experiments are conducted with 10 random seeds. Appendix G Hyperparameter Settings G.1 FINO Since our method builds on FQL (Park et al., 2025b) as the backbone algorithm, we retain all hyperparameters from FQL without modification. The hyperparameters introduced in our method, namely αt _t for noise injection and NsampleN_sample and ℋ¯ H for entropy-guided sampling, are configured depending on the environment settings. The entropy update interval (NξN_ξ) is aligned with the evaluation frequency and set to 50,000 steps. A complete list of hyperparameters is provided in Table 2. Table 2: Hyperparameters Hyperparameter Value Noise constant η 0.05 ⋅||·|A| Candidate action samples NsampleN_sample 0.5 ⋅dim()· (A) Target entropy ℋ¯ H −dim()- (A) Entropy update steps NξN_ξ 50,000 Standard deviation of noise αt _t η⋅exp(5(t−1))η· (5(t-1)) Hyperparameter (from FQL) Value Learning rate 0.0003 Optimizer Adam (Kingma and Ba, 2015) Minibatch size 256 MLP dimensions [512, 512, 512, 512] Nonlinearity GELU (Hendrycks and Gimpel, 2016) Target network smoothing coefficient 0.005 Discount factor γ 0.99 (default), 0.995 (antmaze-giant, humanoidmaze, antsoccer) Flow steps 10 Flow time sampling distribution Unif([0, 1]) Clipped double Q-learning False (default), True (adroit, antmaze-large, giant-navigate) BC coefficient α Table 3 G.2 Other Baselines For the other baselines, we retain the hyperparameters used in FQL (Park et al., 2025b). For ReBRAC (Tarasov et al., 2023), we treat the actor bc coefficient (α1 _1) and the critic bc coefficient (α2 _2) as tunable hyperparameters, while keeping all other settings at their default values. For Cal-QL, the cql regularizer coefficient (α) and the target action gap (β) are used as hyperparameters. Regarding the network size, we set it to [256, 256, 256, 256] for manipulation tasks and [512, 512, 512, 512] for locomotion tasks of OGBench, with all other parameters kept at their default values. For RLPD, we use a re-implementation of RLPD from the codebase of FQL and adopt the same configuration as FQL, including setting the update-to-data ratio to 1 and using two value functions. For IFQL, the only hyperparameter is the number of action samples (N). For FQL, the behavior cloning coefficient (α) is the sole hyperparameter. The task-specific hyperparameters are summarized in Table 3. Table 3: Task-specific hyperparameters for each baseline. Task ReBRAC (α1,α2)( _1, _2) Cal-QL (α,β)(α,β) IFQL (N) FQL (α) humanoidmaze-medium-navigate-v0 (0.01, 0.01) (5, 0.8) 32 100 humanoidmaze-large-navigate-v0 (0.01, 0.01) (5, 0.8) 32 30 antmaze-large-navigate-v0 (0.003, 0.01) (5, 0.8) 32 30 antmaze-giant-navigate-v0 (0.003, 0.01) (5, 0.8) 32 10 antsoccer-arena-navigate-v0 (0.01, 0.01) (5, 0.2) 64 30 cube-double-play-v0 (0.1, 0) (0.01, 1) 32 300 puzzle-4x4-play-v0 (0.3, 0.01) (0.003, 1) 32 1000 antmaze-umaze-v2 (0.003, 0.002) (5, 0.8) 32 10 antmaze-umaze-diverse-v2 (0.003, 0.001) (5, 0.8) 32 10 antmaze-medium-play-v2 (0.001, 0.0005) (5, 0.8) 32 10 antmaze-medium-diverse-v2 (0.001, 0) (5, 0.8) 32 10 antmaze-large-play-v2 (0.002, 0.001) (5, 0.8) 32 3 antmaze-large-diverse-v2 (0.002, 0.002) (5, 0.8) 32 3 pen-cloned-v1 (0.05, 0.5) (1, 0.8) 128 1000 door-cloned-v1 (0.01, 0.1) (1, 0.8) 128 1000 hammer-cloned-v1 (0.1, 0.5) (1, 0.8) 128 1000 relocate-cloned-v1 (0.1, 0.01) (1, 0.8) 128 10000 Table 4: Full results for main experiments (corresponding to Table 1 and Fig. 3). Scores show offline pre-training → online fine-tuning, averaged over 10 seeds (mean ± 95% CI). For OGBench, the singletask suffix is omitted. Environment ReBRAC Cal-QL RLPD IFQL FQL FINO OGBench humanoidmaze-medium-navigate-task1 14± 7 → 1± 1 0± 0 → 0± 0 0± 0 → 0± 0 65± 18 → 45± 11 11± 5 → 14± 4 13± 3 → 91± 3 OGBench humanoidmaze-medium-navigate-task2 18± 9 → 1± 1 0± 0 → 0± 0 0± 0 → 0± 0 92± 4 → 83± 7 89± 13 → 90± 3 77± 25 → 99± 1 OGBench humanoidmaze-medium-navigate-task3 30± 14 → 1± 1 0± 0 → 0± 0 0± 0 → 1± 1 38± 30 → 74± 18 56± 23 → 89± 2 52± 24 → 99± 1 OGBench humanoidmaze-medium-navigate-task4 17± 11 → 1± 1 0± 0 → 0± 0 0± 0 → 0± 0 0± 0 → 56± 9 9± 12 → 18± 6 11± 4 → 94± 3 OGBench humanoidmaze-medium-navigate-task5 28± 14 → 10± 16 0± 0 → 0± 0 0± 0 → 4± 3 99± 1 → 92± 3 100± 0 → 94± 3 99± 1 → 100± 1 OGBench humanoidmaze-large-navigate-task1 1± 1 → 0± 0 0± 0 → 0± 0 0± 0 → 0± 0 1± 1 → 0± 0 4± 3 → 1± 1 5± 4 → 5± 9 OGBench humanoidmaze-large-navigate-task2 0± 0 → 0± 0 0± 0 → 0± 0 0± 0 → 0± 0 0± 0 → 0± 0 0± 0 → 0± 0 0± 0 → 6± 6 OGBench humanoidmaze-large-navigate-task3 9± 6 → 2± 2 0± 0 → 0± 0 0± 0 → 0± 0 45± 10 → 41± 7 17± 6 → 36± 11 22± 10 → 99± 1 OGBench humanoidmaze-large-navigate-task4 1± 1 → 1± 1 0± 0 → 0± 0 0± 0 → 0± 0 0± 0 → 5± 3 2± 2 → 8± 8 0± 0 → 48± 29 OGBench humanoidmaze-large-navigate-task5 1± 1 → 0± 1 0± 0 → 0± 0 0± 0 → 0± 0 8± 10 → 5± 3 1± 1 → 5± 7 0± 0 → 8± 16 OGBench antmaze-large-navigate-task1 94± 4 → 100± 0 20± 26 → 30± 30 0± 0 → 93± 8 32± 10 → 66± 15 82± 5 → 98± 1 82± 6 → 98± 2 OGBench antmaze-large-navigate-task2 88± 2 → 98± 1 0± 0 → 0± 0 0± 0 → 54± 25 17± 7 → 70± 4 63± 5 → 71± 4 62± 6 → 97± 1 OGBench antmaze-large-navigate-task3 65± 14 → 100± 0 20± 26 → 10± 20 0± 0 → 99± 1 57± 9 → 88± 4 94± 2 → 100± 1 92± 3 → 100± 0 OGBench antmaze-large-navigate-task4 86± 5 → 99± 1 0± 0 → 0± 0 0± 0 → 86± 11 13± 5 → 75± 8 80± 5 → 96± 1 83± 4 → 99± 1 OGBench antmaze-large-navigate-task5 90± 4 → 100± 1 20± 26 → 20± 26 0± 0 → 69± 24 39± 10 → 59± 23 85± 4 → 95± 2 85± 5 → 99± 1 OGBench antmaze-giant-navigate-task1 53± 16 → 97± 1 0± 0 → 0± 0 0± 0 → 7± 12 0± 0 → 0± 0 8± 7 → 65± 24 3± 4 → 96± 1 OGBench antmaze-giant-navigate-task2 25± 17 → 98± 1 0± 0 → 0± 0 0± 0 → 48± 24 0± 0 → 0± 0 17± 11 → 96± 1 0± 1 → 99± 1 OGBench antmaze-giant-navigate-task3 34± 20 → 86± 19 0± 0 → 0± 0 0± 0 → 44± 18 0± 0 → 0± 0 0± 1 → 2± 2 0± 0 → 0± 0 OGBench antmaze-giant-navigate-task4 0± 0 → 98± 1 0± 0 → 0± 0 0± 0 → 59± 26 0± 0 → 0± 0 10± 13 → 96± 2 29± 23 → 99± 1 OGBench antmaze-giant-navigate-task5 61± 12 → 99± 1 10± 20 → 0± 0 0± 0 → 80± 12 4± 4 → 0± 0 43± 21 → 99± 1 36± 15 → 99± 1 OGBench antsoccer-arena-navigate-task1 1± 1 → 0± 0 0± 0 → 0± 0 0± 0 → 6± 4 69± 15 → 64± 10 82± 4 → 91± 2 77± 6 → 93± 2 OGBench antsoccer-arena-navigate-task2 0± 1 → 0± 0 0± 0 → 0± 0 0± 0 → 5± 4 70± 7 → 66± 21 88± 4 → 97± 3 84± 5 → 98± 1 OGBench antsoccer-arena-navigate-task3 0± 0 → 0± 0 0± 0 → 0± 0 0± 0 → 1± 1 6± 6 → 26± 9 60± 4 → 88± 4 56± 5 → 91± 2 OGBench antsoccer-arena-navigate-task4 1± 1 → 0± 0 0± 0 → 0± 0 0± 0 → 0± 0 20± 9 → 17± 6 32± 4 → 70± 5 34± 6 → 70± 6 OGBench antsoccer-arena-navigate-task5 0± 0 → 0± 0 0± 0 → 0± 0 0± 0 → 0± 0 1± 2 → 4± 3 43± 9 → 22± 18 32± 7 → 33± 25 OGBench cube-double-play-task1 27± 8 → 100± 0 10± 20 → 0± 0 0± 0 → 11± 15 31± 5 → 89± 3 59± 9 → 97± 2 62± 5 → 98± 1 OGBench cube-double-play-task2 8± 3 → 20± 6 0± 0 → 0± 0 0± 0 → 0± 0 13± 3 → 29± 4 42± 9 → 86± 7 40± 7 → 90± 3 OGBench cube-double-play-task3 3± 2 → 16± 7 0± 0 → 0± 0 0± 0 → 0± 0 6± 2 → 31± 7 29± 5 → 89± 9 35± 8 → 90± 4 OGBench cube-double-play-task4 1± 1 → 0± 1 0± 0 → 0± 0 0± 0 → 0± 0 1± 1 → 0± 0 5± 3 → 4± 2 11± 4 → 21± 8 OGBench cube-double-play-task5 3± 2 → 3± 2 0± 0 → 0± 0 0± 0 → 0± 0 17± 3 → 51± 6 20± 6 → 88± 3 23± 10 → 96± 3 OGBench puzzle-4x4-play-task1 25± 4 → 100± 0 10± 20 → 70± 30 0± 0 → 100± 0 38± 6 → 100± 1 32± 6 → 100± 0 33± 8 → 100± 0 OGBench puzzle-4x4-play-task2 10± 4 → 0± 0 0± 0 → 0± 0 0± 0 → 40± 32 16± 5 → 1± 1 13± 4 → 0± 0 11± 4 → 0± 0 OGBench puzzle-4x4-play-task3 16± 4 → 42± 26 0± 0 → 30± 30 0± 0 → 89± 19 49± 8 → 91± 12 18± 5 → 70± 28 22± 9 → 100± 0 OGBench puzzle-4x4-play-task4 8± 2 → 3± 3 0± 0 → 0± 0 0± 0 → 40± 32 20± 4 → 19± 17 7± 4 → 53± 30 20± 5 → 80± 24 OGBench puzzle-4x4-play-task5 9± 2 → 0± 0 0± 0 → 0± 0 0± 0 → 20± 26 8± 3 → 0± 0 5± 2 → 0± 0 8± 2 → 0± 0 D4RL antmaze-umaze-v2 90± 4 → 100± 1 81± 3 → 99± 1 0± 0 → 100± 1 94± 2 → 96± 2 98± 1 → 99± 1 98± 2 → 100± 1 D4RL antmaze-umaze-diverse-v2 75± 12 → 100± 1 36± 12 → 94± 4 0± 0 → 99± 1 71± 14 → 53± 22 85± 7 → 99± 1 78± 5 → 99± 1 D4RL antmaze-medium-play-v2 8± 8 → 91± 11 60± 12 → 91± 11 0± 0 → 97± 1 54± 14 → 79± 18 78± 5 → 94± 2 79± 5 → 97± 1 D4RL antmaze-medium-diverse-v2 11± 8 → 98± 2 61± 5 → 95± 2 0± 0 → 98± 1 41± 20 → 86± 3 66± 9 → 95± 2 60± 11 → 97± 1 D4RL antmaze-large-play-v2 42± 21 → 51± 28 35± 5 → 74± 6 0± 0 → 79± 11 64± 5 → 78± 3 73± 18 → 95± 1 75± 17 → 95± 1 D4RL antmaze-large-diverse-v2 77± 5 → 93± 2 29± 8 → 80± 4 0± 0 → 83± 8 73± 6 → 84± 3 81± 11 → 91± 2 82± 4 → 97± 1 D4RL pen-cloned-v1 77± 6 → 129± 4 -1± 1 → -1± 1 3± 2 → 93± 8 73± 3 → 97± 7 53± 12 → 141± 4 51± 8 → 140± 3 D4RL door-cloned-v1 0± 0 → 83± 6 0± 0 → 0± 0 0± 0 → 101± 4 1± 1 → 23± 5 0± 0 → 101± 2 0± 0 → 104± 1 D4RL hammer-cloned-v1 5± 3 → 119± 2 0± 0 → 0± 0 0± 0 → 99± 19 1± 1 → 44± 8 0± 0 → 110± 25 0± 0 → 134± 2 D4RL relocate-cloned-v1 0± 0 → 0± 0 0± 0 → 0± 0 0± 0 → 0± 0 0± 0 → 2± 1 1± 0 → 47± 2 1± 0 → 72± 2 Table 5: Full results for ablation studies (Fig. 5, Fig. 6). Scores show offline pre-training → online fine-tuning, averaged over 10 seeds (mean ± 95% CI). For OGBench, the singletask suffix is omitted. Environment FINO Direct Noise w/o Noise w/o Guidance OGBench humanoidmaze-medium-navigate-task1 13± 3 → 91± 3 14± 5 → 0± 1 18± 5 → 0± 0 16± 3 → 50± 16 OGBench humanoidmaze-medium-navigate-task2 77± 25 → 99± 1 88± 16 → 96± 6 80± 19 → 99± 2 71± 25 → 84± 20 OGBench humanoidmaze-medium-navigate-task3 52± 24 → 99± 1 54± 22 → 60± 32 45± 24 → 49± 32 48± 23 → 79± 25 OGBench humanoidmaze-medium-navigate-task4 11± 4 → 94± 3 7± 10 → 31± 24 2± 4 → 20± 26 0± 0 → 74± 16 OGBench humanoidmaze-medium-navigate-task5 99± 1 → 100± 1 99± 1 → 100± 1 99± 1 → 99± 1 98± 1 → 99± 1 OGBench humanoidmaze-large-navigate-task1 5± 4 → 5± 9 4± 4 → 0± 0 3± 4 → 8± 14 4± 4 → 0± 1 OGBench humanoidmaze-large-navigate-task2 0± 0 → 6± 6 0± 0 → 0± 0 0± 0 → 2± 3 0± 1 → 3± 5 OGBench humanoidmaze-large-navigate-task3 22± 10 → 99± 1 19± 6 → 41± 5 18± 7 → 95± 4 25± 10 → 78± 26 OGBench humanoidmaze-large-navigate-task4 0± 0 → 48± 29 2± 2 → 8± 7 0± 0 → 5± 9 0± 0 → 18± 22 OGBench humanoidmaze-large-navigate-task5 0± 0 → 8± 16 0± 0 → 4± 4 1± 2 → 0± 0 0± 0 → 13± 18 OGBench antmaze-large-navigate-task1 82± 6 → 98± 2 82± 5 → 98± 1 68± 15 → 98± 1 78± 5 → 35± 27 OGBench antmaze-large-navigate-task2 62± 6 → 97± 1 59± 5 → 69± 8 62± 5 → 94± 3 75± 6 → 91± 3 OGBench antmaze-large-navigate-task3 92± 3 → 100± 0 96± 2 → 99± 1 95± 2 → 100± 0 95± 3 → 99± 1 OGBench antmaze-large-navigate-task4 83± 4 → 99± 1 72± 16 → 96± 2 69± 15 → 98± 1 81± 4 → 98± 1 OGBench antmaze-large-navigate-task5 85± 5 → 99± 1 83± 3 → 95± 2 83± 3 → 98± 2 82± 5 → 83± 21 OGBench antmaze-giant-navigate-task1 3± 4 → 96± 1 9± 7 → 64± 22 9± 10 → 85± 13 4± 5 → 79± 26 OGBench antmaze-giant-navigate-task2 0± 1 → 99± 1 18± 11 → 96± 1 1± 2 → 98± 2 1± 2 → 99± 1 OGBench antmaze-giant-navigate-task3 0± 0 → 0± 0 0± 1 → 1± 2 0± 0 → 0± 0 0± 0 → 7± 15 OGBench antmaze-giant-navigate-task4 29± 23 → 99± 1 10± 12 → 85± 15 5± 7 → 98± 1 28± 23 → 97± 3 OGBench antmaze-giant-navigate-task5 36± 15 → 99± 1 43± 20 → 98± 1 22± 13 → 99± 1 31± 17 → 99± 1 OGBench antsoccer-arena-navigate-task1 77± 6 → 93± 2 81± 4 → 93± 4 69± 7 → 94± 2 68± 5 → 73± 16 OGBench antsoccer-arena-navigate-task2 84± 5 → 98± 1 90± 3 → 97± 2 83± 6 → 97± 2 83± 4 → 93± 2 OGBench antsoccer-arena-navigate-task3 56± 5 → 91± 2 58± 4 → 87± 4 54± 4 → 81± 4 57± 4 → 78± 4 OGBench antsoccer-arena-navigate-task4 34± 6 → 70± 6 33± 4 → 71± 8 43± 5 → 62± 7 43± 4 → 50± 11 OGBench antsoccer-arena-navigate-task5 32± 7 → 33± 25 43± 7 → 14± 17 19± 5 → 10± 19 33± 8 → 42± 23 OGBench cube-double-play-task1 62± 5 → 98± 1 64± 9 → 97± 3 73± 6 → 95± 3 74± 8 → 96± 3 OGBench cube-double-play-task2 40± 7 → 90± 3 40± 5 → 86± 3 60± 7 → 80± 9 61± 9 → 89± 5 OGBench cube-double-play-task3 35± 8 → 90± 4 26± 5 → 88± 6 57± 5 → 88± 5 52± 7 → 96± 2 OGBench cube-double-play-task4 11± 4 → 21± 8 5± 2 → 3± 2 14± 1 → 2± 1 8± 3 → 6± 4 OGBench cube-double-play-task5 23± 10 → 96± 3 21± 6 → 88± 5 43± 15 → 86± 6 26± 9 → 91± 4 OGBench puzzle-4x4-play-task1 33± 8 → 100± 0 31± 5 → 100± 0 56± 7 → 100± 0 61± 7 → 100± 0 OGBench puzzle-4x4-play-task2 11± 4 → 0± 0 12± 2 → 0± 0 15± 5 → 0± 0 10± 3 → 0± 0 OGBench puzzle-4x4-play-task3 22± 9 → 100± 0 20± 2 → 80± 26 53± 11 → 88± 15 59± 6 → 97± 6 OGBench puzzle-4x4-play-task4 20± 5 → 80± 24 9± 3 → 61± 31 18± 4 → 33± 29 19± 6 → 10± 20 OGBench puzzle-4x4-play-task5 8± 2 → 0± 0 7± 3 → 0± 0 6± 3 → 0± 0 9± 4 → 0± 0 D4RL antmaze-umaze-v2 98± 2 → 100± 1 96± 2 → 99± 1 98± 1 → 99± 1 97± 2 → 99± 1 D4RL antmaze-umaze-diverse-v2 78± 5 → 99± 1 85± 5 → 97± 1 85± 7 → 99± 1 82± 6 → 100± 0 D4RL antmaze-medium-play-v2 79± 5 → 97± 1 74± 4 → 93± 3 80± 5 → 96± 2 79± 4 → 96± 2 D4RL antmaze-medium-diverse-v2 60± 11 → 97± 1 62± 8 → 95± 1 62± 10 → 95± 6 61± 11 → 97± 1 D4RL antmaze-large-play-v2 75± 17 → 95± 1 72± 19 → 90± 5 67± 22 → 92± 2 73± 16 → 94± 3 D4RL antmaze-large-diverse-v2 82± 4 → 97± 1 83± 9 → 90± 4 72± 16 → 94± 3 81± 3 → 94± 3 D4RL pen-cloned-v1 51± 8 → 140± 3 57± 9 → 137± 4 60± 8 → 135± 4 57± 7 → 136± 4 D4RL door-cloned-v1 0± 0 → 104± 1 0± 0 → 102± 2 0± 0 → 102± 2 0± 0 → 101± 4 D4RL hammer-cloned-v1 0± 0 → 134± 2 0± 0 → 116± 26 0± 0 → 120± 12 0± 0 → 112± 14 D4RL relocate-cloned-v1 1± 0 → 72± 2 1± 0 → 59± 3 1± 0 → 61± 5 1± 1 → 62± 7 Figure 9: Full results on OGBench environments. Each row corresponds to one environment, with five single-task variants shown side by side. Shaded areas denote 95% confidence intervals over 10 seeds. Figure 10: Full results on D4RL environments. For AntMaze tasks, the prefix “antmaze-” is omitted for clarity. Shaded areas denote 95% confidence intervals over 10 seeds.