Paper deep dive
Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning
Daoyi Li, Yixian Zhang, Wenbo Ding, Yu Wang, Chao Yu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/14/2026, 4:05:58 AM
Summary
The paper introduces Critic-Free Pretraining (CFP), a paradigm for offline-to-online (O2O) reinforcement learning that abandons offline critic training to prevent value estimate misalignment during online fine-tuning. By retaining a pretrained behavior-cloning actor and initializing a fresh critic with a short warm-up phase, CFP improves sample efficiency and performance across various benchmarks compared to conventional O2O methods.
Entities (8)
Relation Signals (6)
Critic-Free Pretraining → evaluatedon → OGBench
confidence 95% · We conduct our experiments on 8 sparse reward domains, including 5 domains from OGBench
Critic-Free Pretraining → evaluatedon → RoboMimic
confidence 95% · and 3 tasks in Robomimic: Lift, Can and Square.
Critic-Free Pretraining → solvesproblem → Offline-to-Online Reinforcement Learning
confidence 95% · CFP is introduced to address the problem of value estimate misalignment in O2O RL.
Critic-Free Pretraining → usescomponent → Actor
confidence 95% · CFP retains the pretrained actor.
Critic-Free Pretraining → usescomponent → Critic
confidence 95% · CFP introduces a fresh critic initialized from scratch.
Flow Matching → usedby → Critic-Free Pretraining
confidence 90% · Flow-based policy train the network via Flow Matching... CFP... utilize the dataset to train a behavior-cloning (BC) actor.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce Critic-Free Pretraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.
Tags
Links
- Source: https://arxiv.org/abs/2608.10473v2
- Canonical: https://arxiv.org/abs/2608.10473v2
Trouble viewing inline? Open PDF directly →
Full Text
69,174 characters extracted from source content.
Expand or collapse full text
Critic-free Pretraining for Efficient Online Reinforcement Learning Fine-tuning Daoyi Li Affiliation: [2m] Equal contribution. Yixian Zhang Affiliation: [2m] Equal contribution. Yu Wang Wenbo Ding Chao YuTsinghua University Affiliation: Corresponding authors. Abstract Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce Critic-Free Pretraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks. Figure 1: Overview of conventional offline-to-online fine-tuning and CFP. Conventional O2O directly reuses the offline-trained critic, which can become misaligned with the rapidly changing online policy. CFP instead retains the pretrained actor, restores a fresh critic, and calibrates it using a short warm-up before fine-tuning. 1 Introduction Reinforcement learning (RL) provides a general framework for learning decision-making policies through interactions with an environment (34; 11; 12; 13; 21; 1). However, learning from scratch remains highly sample-inefficient in complex environments. At the beginning of training, an online agent rarely visits task-relevant regions. The agent must therefore improve its policy using low-quality experience, while the quality of future experience itself depends on the current policy. This coupling between policy learning and data collection creates a fundamental problem: without a meaningful policy, the agent cannot efficiently collect useful data, and without useful data, it cannot learn a meaningful policy. The introduction of an offline phase significantly alleviates the pressure of exploration (17). Offline training on a dataset greatly enhances the agent’s training efficiency during the online phase (7). However, the O2O paradigm faces a critical issue: the transition from the dataset to the real-world environment often encounters significant discrepancies in data distribution, whereas the inherited critic reflects the offline behavior and data support. Its value estimates may consequently become misaligned with the evolving online policy, leading to inefficient policy updates, and even degradation of the useful behavior acquired during offline pretraining. This represents a major challenge for O2O approaches (16; 28; 15). In this work, we revisit the role of offline stage in O2O RL. Our insight is that the goal of offline training is not to produce a generalizable policy or value network, but rather to encourage the agent to operate in regions where it is more likely to collect effective trajectories, thereby improving sampling efficiency. This suggests that policy pretraining and critic pretraining, which are conventionally performed together, should be decoupled. Based on this insight, we propose Critic-Free Pretraining (CFP). Drawing inspiration from the concept of imitation learning, we utilize the dataset to train a behavior-cloning (BC) actor. Before online interaction begins, CFP retains the pretrained actor and introduces a fresh critic. A short warm-up phase then calibrates the critic. The pretrained policy provides efficient data collection, while the fresh critic avoids inheriting value bias tied to the offline distribution. Our main contributions are listed below: • A critic-free offline training paradigm. We demonstrate that training the critic during the offline phase can actually impair subsequent online fine-tuning performance. By completely eliminating offline critic training, we substantially reduce computational costs and memory requirements, improving training efficiency while providing novel insights for scaling large-scale reinforcement learning systems. • An effective yet simple implementation of CFP. We conduct extensive experiments on the implementation of CFP and identify an execution approach that balances computational efficiency with experimental performance. Building on this, we designed a toy example to simulate the training workflows of O2O and CFP, intuitively demonstrating the effectiveness of CFP through visualization. This work also offers a new methodology for future research into efficient reinforcement learning. • Strong and broadly applicable online fine-tuning performance. CFP achieves comparable and outstanding performance across multiple O2O algorithms and diverse benchmarks. The superior performance across different algorithms and task domains validates the effectiveness and generality of our approach. 2 Preliminaries 2.1 Reinforcement Learning We consider reinforcement learning in a Markov Decision Process (MDP) defined by the tuple ℳ=(,,P,r,γ,ρ)M=(S,A,P,r,γ,ρ). Here, S and A denote the state and action spaces; P(s′|s,a)P(s |s,a) represents the transition dynamics; r(s,a)r(s,a) is the reward function; γ∈[0,1)γ∈[0,1) is the discount factor; and ρ is the initial state distribution. RL aims to find an policy π∗(a|s)π^*(a|s) that maximizes the expected discounted reward: J(π)=s0∼ρ,at∼π(⋅|st),st+1∼P(⋅|st,at)[∑t=0∞γtr(st,at)].J(π)=E_s_0 ρ,a_t π(·|s_t),s_t+1 P(·|s_t,a_t) [ _t=0^∞γ^tr(s_t,a_t) ]. (1) 2.2 Flow matching Flow Matching is a simple framework to train continuous normalizing flows (CNFs) (22; 23; 24; 26; 39). In contrast to denoising diffusion models (37; 31; 4; 14; 36), which aim at solving Stochastic Differential Equations (SDEs), Flow Matching model learns a time-dependent vector field vθ(t,x):[0,1]×ℝd→ℝdv_θ(t,x):[0,1]×R^d ^d, where d is the distribution dimension, that characterizes the velocity of points moving from a noise distribution to the data distribution (30). Generation is then performed by solving the ODE: dxtdt=vθ(t,xt),x0∼ρ0. dx_tdt=v_θ(t,x_t), x_0 _0. (2) 2.3 Flow-based policies Flow-based policy train the network via Flow Matching. The objective is to minimize the loss function: LFlow(θ)=x0∼(0,Id),(x1=a,s)∼,t∼Unif([0,1])[‖vθ(t,s,xt)−(x1−x0)‖22]L_Flow(θ)=E_ subarraycx_0 (0,I_d),\\ (x_1=a,s) ,\\ t ([0,1]) subarray[\|v_θ(t,s,x_t)-(x_1-x_0)\|_2^2] (3) where D denotes the static dataset. In inference, policies generate the action depend on the learned velocity field by computing the flow: xti+1=xti+(ti+1−ti)vθ(ti,s,xti),xt0∼(0,I).x_t_i+1=x_t_i+(t_i+1-t_i)v_θ(t_i,s,x_t_i), x_t_0 (0,I). (4) In actual implementation, generating each action requires K-times computations of the vector field, which can lead to vanishing or exploding gradient issues during neural network training. Consequently, single-step policies μw(s,z) _w(s,z) have emerged as a crucial solution. Rather than learning the velocity field directly, a single-step network learns to map to the final actions produced by K-step network. This allows actions that previously required multiple computations to be obtained in a single pass, significantly enhancing efficiency and enhancing training stability (30; 42; 43; 19; 18; 20; 5). The overall loss function of flow-based policies is as below: LTotal(θ,ω)=LFlow(θ)+Lπ(ω)L_Total(θ,ω)=L_Flow(θ)+L_π(ω) (5) where Lπ(ω)=αs∼,z∼(0,Id)[∥μω(s,z)−μθ(s,z)∥22]+s∼,a∼πω(⋅|s)[−Qϕ(s,a)]L_π(ω)= _s ,z (0,I^d)[\| _ω(s,z)- _θ(s,z)\|_2^2]+E_s ,a _ω(·|s)[-Q_φ(s,a)] (6) 3 Efficient online fine-tuning via CFP We present our main methodology in this section. Our method entirely omits critic training during the offline phase, resulting in a critic-free policy-pretraining strategy. At the beginning of online fine-tuning, we initialize a fresh critic from scratch and perform a short warm-up stage on the offline dataset. 3.1 Critic-Free Pretraining Unlike previous O2O methods that update both the Actor and Critic simultaneously in the offline phase, we forgo training the latter in this phase. This also means that the actor’s update in the offline phase does not receive gradients provided by the critic; that is, the actor’s update formula is: Lπ(ω)=αs∼,z∼(0,Id)[‖μω(s,z)−μθ(s,z)‖22]L_π(ω)= _s ,z (0,I^d)[\| _ω(s,z)- _θ(s,z)\|_2^2] (7) we reach a BC actor loss function in flow policies. BC is a widely-used algorithm that reaches outstanding performance in O2O situations and robot manipulation (3; 35; 6; 32). 3.2 Why a Fresh Critic facilitates online fine-tuning The ability of the critic to accurately describe the true Q-values is considered key to the algorithm’s success. We offer a deeper insight: since an offline critic can hinder an agent’s online fine-tuning capabilities, we can achieve our objective by simply initializing the critic, rather than expending effort on calibrating it. The rationale behind this idea is that a freshly initialized critic is initially inaccurate, but it does not inherit the systematic value bias induced by prolonged fitting to the offline distribution. However, such a fresh critic may fail to provide proper guidance to the actor and could even disrupt the pretrained offline policy, leading to instability during online training. To address this, we introduce a brief warm-up phase in which the critic is trained on offline data to acquire a preliminary ability to estimate Q-values. We present the comparison between O2O and CFP in Alg.1 and Alg.2. Algorithm 1 Traditional O2O 1: Init: Critic QϕQ_φ, Policy πθ _θ, Buffer ℬ←B 2: ⊳ Offline Stage 3: for ℓ=1…Loff =1… L_off do 4: Sample batch (s,a,r,s′)∼ℬ\(s,a,r,s )\ 5: Update πθ _θ via offline actor loss 6: Update QϕQ_φ via TD loss 7: end for 8: ⊳ Online Stage 9: for ℓ=1…Lon =1… L_on do 10: Collect (s,a,r,s′)∼πθ(s,a,r,s ) _θ, append to ℬB 11: Sample batch (s,a,r,s′)∼ℬ\(s,a,r,s )\ 12: Update πθ _θ via online actor loss 13: Update QϕQ_φ via TD loss 14: end for Algorithm 2 CFP (Ours) 1: Init: Policy πθ _θ, Buffer ℬ←B 2: ⊳ Offline Stage (Critic-Free) 3: for ℓ=1…Loff =1… L_off do 4: Sample batch (s,a,r,s′)∼ℬ\(s,a,r,s )\ 5: Update πθ _θ via BC Flow loss (Eq. 7) 6: end for 7: ⊳ Warm-up Stage Init fresh critic QϕQ_φ from scratch for ℓ=1…Lwarm-up =1… L_warm-up do a Sample batch (s,a,r,s′)∼ℬ\(s,a,r,s )\ a Update πθ _θ via online actor loss a Update QϕQ_φ via TD loss 8: ⊳ Online Stage 9: for ℓ=1…Lon =1… L_on do 10: Collect (s,a,r,s′)∼πθ(s,a,r,s ) _θ, append to ℬB 11: Sample batch (s,a,r,s′)∼ℬ\(s,a,r,s )\ 12: Update πθ _θ via online actor loss 13: Update QϕQ_φ via TD loss 14: end for Even though flow algorithms do not contain explicit penalties in loss functions, we assume that the inherent limitations of offline training are the implicit cause of the critic’s pessimism. Since: Qβ(s,a)≤Q∗(s,a)Q_β(s,a)≤ Q^*(s,a) (8) where Q∗(s,a)Q^*(s,a) is the optimal Q-value in the real environment. However, the problem extends far beyond this natural bound. In an offline dataset generated by behavior policies, the learned value function QβQ_β is theoretically bounded by the sub-optimality of the dataset. Modern offline datasets provide complex and sparse-reward tasks to test the effectiveness of algorithms. These datasets are overwhelmingly dominated by suboptimal, random, or failed trajectories, with high-quality successful demonstrations being exceptionally rare (29). Due to the bootstrapping nature of Temporal Difference learning, the low returns from these abundant suboptimal trajectories propagate backward throughout the state-action space, results in accumulated pessimism of the critic. Figure 2: Critic Q-mean estimates and success rates on Cube Triple task 4. Curves and shaded regions show the mean and 95% confidence interval across five seeds, respectively. As illustrated in Figure 2, we compared the Q-values calculated by the critic for state-action pairs in a batch under O2O and CFP. We find that during the online phase of CFP, the critic’s Q-value improves rapidly as fine-tuning progressed; in contrast, the Q-value in the O2O consistently remains at a lower level. Furthermore, we identified a potential mismatch between the critic’s Q evaluations and the true Q values; this discrepancy prevents the critic from effectively guiding the actor toward superior actions. Given the complexity of the benchmark environments, we could not demonstrate this deviation through visualization; instead, we designed a toy example to illustrate our insight. As shown in Box 3.2, we construct a tabular MDP with 16 states arranged on a 4×44× 4 grid. The environment contains a negative terminal state, a positive terminal state, and several hazardous states. We deliberately induce a mismatch between the dataset Q distribution and the ground-truth Q distribution. We then separately simulate the training processes of O2O and CFP and evaluate the training performance of their respective critics. Toy Example: Critic Mismatch under Distribution Shift Schematic diagrams illustrating the MDP and Q-mismatch. The left panel represents the underlying transition dynamics within the MDP process. In the right panel, colors closer to blue indicate a greater degree of mismatch. We evaluate each critic using the root mean squared error (RMSE) between the learned Q-values and the exact online Q-function Qπ^onQ π_on over all valid state transitions. As shown in Figure 3, the CFP critic exhibits a Q-value structure that more closely resembles Qπ^onQ π_on than the O2O critic. Moreover, CFP maintains a lower RMSE throughout online fine-tuning than O2O. These results suggest that reusing the offline critic retains considerable bias, whereas initializing critic enables more accurate estimation of the online-policy value function. For more details, see Appendix D. Figure 3: Critic calibration in the toy MDP. CFP produces Q value estimates closer to the ground truth QπonQ _on and achieves substantially lower RMSE than O2O during online fine-tuning. 4 Related Work 4.1 Offline-to-Online RL The primary objective of the O2O RL is to leverage offline pre-training on static datasets to enable more efficient fine-tuning during the subsequent online stage. This approach mitigates the fundamental challenges where training online RL from-scratch often fails to acquire effective policies in complex environments, thereby significantly reducing both computational overhead and time costs (17; 38; 10; 40; 41; 43). However, O2O RL faces the problem of the distribution shift between the dataset and the environment, which causes catastrophic performance drop in online stage. To confront this problem, a few solutions are introduced. In this section, we list some of them that are related to our work. Pessimism and Conservatism Early algorithms mainly focus on eliminating exploration errors. By augmenting the Bellman objective with a conservative regularizer, CQL (16) lowers the estimated values of actions that are poorly supported by the offline dataset, thereby mitigating overestimation caused by distribution shift. Cal-QL (28) constrains the learned conservative values to remain above the value of a reference policy, thereby placing the Q-values on a reasonable scale for subsequent online fine-tuning. Some algorithms reach the same goal by explicitly restrict the learned policy from deviating too far from dataset distribution. For instance, BCQ (8) uses a conditional generative model and a perturbation model to restrict action selection to actions that are likely under, or close to, the dataset distribution. TD3+BC (7) explicitly augments the actor objective with a behavior-cloning term, whereas AWAC (27) performs an advantage-weighted maximum-likelihood policy update, implicitly constraining the learned policy toward actions supported by the replay distribution. Replay Design and Critic Calibration Beyond conservatism, recent offline-to-online RL methods improve fine-tuning through replay design and explicit value calibration. For example, RLPD (2) trains an off-policy online RL agent with minibatches drawn jointly from fixed offline data and newly collected online experience, together with high update-to-data ratios and critic regularization for stable and sample-efficient learning. WSRL (44) introduces the warm-up stage in which only online data are used to calibrate the pretrained critic network. OPT (33) introduces a newly initialized critic and an additional online pre-training phase, during which the new critic is trained with both the offline dataset and a small set of online transitions. During fine-tuning, policy improvement is guided by a scheduled weighted combination of the offline-pretrained critic and the online-pretrained critic. 5 Experiments To verify the effectiveness of CFP, we conduct extensive experiments on manipulation tasks under different environments. We further conducted ablation experiments, providing solid evidence for the implementation of the warm-up stage. 5.1 Settings We conduct our experiments on 8 sparse reward domains, including 5 domains from OGBench (29): Cube-Double/Triple/Quadruple, Puzzle-4×4, Scene, and 3 tasks in Robomimic (25): Lift, Can and Square. We use default play dataset for each OGBench domain. In particular, for Cube Quadruple, we use 100M-size dataset released by the OGBench authors (29). In addition, we use the sparse-reward formulation for Scene. We use default Multi-Human (MH) dataset. For more details about the benchmark and dataset, see Appendix A. 5.2 Baselines We select 4 representative flow-based algorithms as our baselines, including (1) FQL (30), (2) QC (9; 20), (3) QCFQL (20), (4) QCFQL-nstep (20; 43). We further implement CFP paradigm on all of them and compare the online performance among them. For more details, see Appendix B. 5.3 Main Result Figure 4 reports the offline-to-online performance across five OGBench manipulation domains, with each curve aggregating results over five tasks. Overall, incorporating CFP consistently improves or preserves the performance of the underlying offline RL algorithms. The advantage of CFP is especially pronounced on Cube Triple, where all the CFP variants appear to outperform their corresponding O2O baselines. In particular, QCFQL-CFP and QCFQL-nstep-CFP exhibit remarkably similar learning dynamics across the five domains. Both variants rapidly improve after online fine-tuning begins and achieve nearly identical final performance on Cube Double, Cube Triple, Cube Quadruple, and Scene. Figure 4: Performance across five tasks in each OGBench environment. Curves and shaded regions represent the mean and 95% confidence interval across five random seeds, respectively. Figure 5: Performance on each Robomimic environment. Curves and shaded regions represent the mean and 95% confidence interval across five random seeds, respectively. We also observe that the benefit of CFP depends on the underlying base algorithm. FQL-CFP achieves the largest improvement on Puzzle 4x4, reaching a final success rate of approximately 0.70.7, compared with less than 0.40.4 for FQL-O2O. Notably, FQL-CFP outperforms all QCFQL- and QC-based variants on this domain, despite being less competitive on Cube Triple and Cube Quadruple. This indicates that CFP can complement the inductive bias of FQL particularly well in environments requiring combinatorial manipulation, rather than producing a uniform improvement across all domains. QC-CFP, by contrast, yields more moderate but consistent gains, with its clearest improvements appearing on Cube Triple and Puzzle 4x4, while remaining comparable to QC-O2O on Cube Double, Cube Quadruple, and Scene. Taken together, these results suggest that CFP is broadly compatible with different offline RL backbones, while the magnitude of its benefit is jointly determined by the base algorithm and the structural characteristics of the downstream environment. Figure 5 reports the offline-to-online performance on three Robomimic manipulation domains. Overall, the CFP variants achieve performance comparable to their corresponding O2O baselines across most algorithms and environments. The main exception is QCFQL-nstep-CFP on Square, which exhibits a substantial improvement over QCFQL-nstep-O2O and achieves a higher final success rate. For full results, see Appendix C.2. 5.4 Ablation study In our ablation studies, we aim to answer the following questions: • How should the warm-up stage be designed? • How sensitive is CFP to the number of warm-up steps? 5.4.1 Should we allow the fresh critic to guide actor? Since the fresh critic in CFP is initially untrained, a natural concern is that the gradients propagated through the Q-loss, LQ=s∼,a∼πω(⋅|s)[−Qϕ(s,a)]L_Q=E_s ,a _ω(·|s)[-Q_φ(s,a)], may be uninformative or even harmful to the pretrained actor. This consideration suggests that, during the warm-up stage, the actor should continue to be optimized solely with the behavior cloning objective. Our experiments, however, lead to the opposite conclusion. We find that, despite the short duration of the warm-up stage—only 0.5%0.5\% of the total training steps---the critic converges rapidly and already exerts a substantial influence on the actor. To investigate this effect, we compare online fine-tuning performance with and without incorporating the Q-function loss into the actor update during warm-up11 1 Among the algorithms considered in this paper, only QC does not apply backpropagation; therefore, the discussion in this section is restricted to the other three algorithms.. Figure 6: Ablation of the Q-loss during warm-up on OGBench. Curves and shaded regions represent the mean and 95% confidence interval across five random seeds, respectively. Figure 6 suggests that the Q-loss backpropagating through flow networks during warm-up generally improves the subsequent online fine-tuning performance, with the clearest gains appearing on the more challenging Cube Triple tasks and Cube Double task 4. 5.4.2 Does the number of warm-up steps matter? We view the number of steps during warm-up stage as a hyperparameter. We sweep this hyperparameter from 0 to 0.1M to see whether CFP is sensitive to it. Figure 7: Ablation of the number of steps during warm-up on OGBench. The applied algorithm is QCFQL-nstep-CFP. Curves and shaded regions represent the mean and 95% confidence interval across five random seeds, respectively. Figure 7 suggests that CFP is relatively insensitive to the length of warm-up. Warm-up-free CFP may experience unstable training during online fine-tuning. Hence, to balance stability and computation efficiency, we choose 10K as our main hyperparameter. We process our training on NVIDIA A800 GPU. In the absence of other concurrent workloads, a 10K-step warm-up requires only approximately 30 seconds, making CFP computationally efficient. For more results of ablation study, see Appendix C.3. 6 Conclusion and Discussion We re-examined the training process of the Actor-Critic framework within the O2O paradigm. Our research reveals that the asymmetric training of the two components in this framework is key to achieving efficient online fine-tuning. Specifically, while the offline phase provides the actor with a sampling prior—thereby enhancing sampling efficiency during the online phase—it simultaneously introduces potential pessimism and distribution-shift-induced mismatches for the critic. We compared the mean critic outputs between O2O and CFP and designed an MDP to validate our inferences. These findings offer new insights into the role of datasets in RL training process. However, CFP does not consistently outperform conventional O2O training on Robomimic. This limitation suggests that discarding the offline critic is most beneficial when inherited value estimates exhibit substantial distribution-induced mismatch. Developing diagnostics that predict when such mismatch occurs, and designing more effective critic-initialization and warm-up strategies, are promising directions for future work. References Arulkumaran et al. (2017) K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath Deep reinforcement learning: a brief survey. IEEE signal processing magazine 34 (6), p. 26–38. Cited by: §1. Ball et al. (2023) P. J. Ball, L. Smith, I. Kostrikov, and S. Levine Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, p. 1577–1594. Cited by: §4.1. Bhatt et al. (2026) D. Bhatt, S. Chou, and N. Atanasov Rainbow-demorl: combining improvements in demonstration-augmented reinforcement learning. arXiv preprint arXiv:2603.27400. Cited by: §3.1. Chi et al. (2025) C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song Diffusion policy: visuomotor policy learning via action diffusion. The International Journal of Robotics Research 44 (10-11), p. 1684–1704. Cited by: §2.2. Doo et al. (2026) J. Doo, B. Jeon, S. Ye, K. Lee, and M. Seo Q-flow: stable and expressive reinforcement learning with flow-based policy. In International Conference on Machine Learning, Cited by: §2.3. Foster et al. (2024) D. J. Foster, A. Block, and D. Misra Is behavior cloning all you need? understanding horizon in imitation learning. Advances in Neural Information Processing Systems 37, p. 120602–120666. Cited by: §3.1. Fujimoto and Gu (2021) S. Fujimoto and S. S. Gu A minimalist approach to offline reinforcement learning. Advances in neural information processing systems 34, p. 20132–20145. Cited by: §1, §4.1. Fujimoto et al. (2019) S. Fujimoto, D. Meger, and D. Precup Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, p. 2052–2062. Cited by: §4.1. Ghasemipour et al. (2021) S. K. S. Ghasemipour, D. Schuurmans, and S. S. Gu Emaq: expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, p. 3682–3691. Cited by: §5.2. Guo et al. (2023) S. Guo, L. Zou, H. Chen, B. Qu, H. Chi, P. S. Yu, and Y. Chang Sample efficient offline-to-online reinforcement learning. IEEE Transactions on Knowledge and Data Engineering 36 (3), p. 1299–1310. Cited by: §4.1. Haarnoja et al. (2018a) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, p. 1861–1870. Cited by: §1. Haarnoja et al. (2018b) T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine Soft actor-critic algorithms and applications. CoRR abs/1812.05905. External Links: Link, 1812.05905 Cited by: §1. Kaelbling et al. (1996) L. P. Kaelbling, M. L. Littman, and A. W. Moore Reinforcement learning: a survey. Journal of artificial intelligence research 4, p. 237–285. Cited by: §1. Kang et al. (2023) B. Kang, X. Ma, C. Du, T. Pang, and S. Yan Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems 36, p. 67195–67212. Cited by: §2.2. Kostrikov et al. (2022) I. Kostrikov, A. Nair, and S. Levine Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations(ICLR), Cited by: §1. Kumar et al. (2020) A. Kumar, A. Zhou, G. Tucker, and S. Levine Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems 33, p. 1179–1191. Cited by: §1, §4.1. Levine et al. (2020) S. Levine, A. Kumar, G. Tucker, and J. Fu Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643. Cited by: §1, §4.1. Li and Levine (2026) Q. Li and S. Levine Q-learning with adjoint matching. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2.3. Li et al. (2025) Q. Li, S. Park, and S. Levine Decoupled q-chunking. External Links: 2512.10926, Link Cited by: §2.3. Li et al. (2026) Q. Li, Z. P. Zhou, and S. Levine Reinforcement learning with action chunking. Advances in Neural Information Processing Systems 38, p. 55518–55553. Cited by: §2.3, §5.2. Li (2017) Y. Li Deep reinforcement learning: an overview. arXiv preprint arXiv:1701.07274. Cited by: §1. Lipman et al. (2023) Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. In Conference on Learning Representations(ICLR), Cited by: §2.2. Lipman et al. (2024) Y. Lipman, M. Havasi, P. Holderrieth, N. Shaul, M. Le, B. Karrer, R. T. Chen, D. Lopez-Paz, H. Ben-Hamu, and I. Gat Flow matching guide and code. arXiv preprint arXiv:2412.06264. Cited by: §2.2. Liu et al. (2023) X. Liu, C. Gong, and Q. Liu Flow straight and fast: learning to generate and transfer data with rectified flow. In Conference on Learning Representations(ICLR), Cited by: §2.2. Mandlekar et al. (2021) A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.03298, Cited by: §5.1. McAllister et al. (2026) D. McAllister, S. Ge, B. Yi, C. M. Kim, E. Weber, H. Choi, H. Feng, and A. Kanazawa Flow matching policy gradients. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2.2. Nair et al. (2020) A. Nair, A. Gupta, M. Dalal, and S. Levine Awac: accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359. Cited by: §4.1. Nakamoto et al. (2023) M. Nakamoto, S. Zhai, A. Singh, M. Sobol Mark, Y. Ma, C. Finn, A. Kumar, and S. Levine Cal-ql: calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems 36, p. 62244–62269. Cited by: §1, §4.1. Park et al. (2025a) S. Park, K. Frans, B. Eysenbach, and S. Levine OGBench: benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), Cited by: §3.2, §5.1. Park et al. (2025b) S. Park, Q. Li, and S. Levine Flow q-learning. In International Conference on Machine Learning(ICML), Cited by: §2.2, §2.3, §5.2. Ren et al. (2025) A. Ren, J. Lidard, L. Ankile, A. Simeonov, P. Agrawal, A. Majumdar, B. Burchfiel, H. Dai, and M. Simchowitz Diffusion policy policy optimization. In International Conference on Learning Representations, Vol. 2025, p. 77288–77329. Cited by: §2.2. Sasaki and Yamashina (2021) F. Sasaki and R. Yamashina Behavioral cloning from noisy demonstrations. In International Conference on Learning Representations, External Links: Link Cited by: §3.1. Shin et al. (2025) Y. Shin, J. Kim, W. Jung, S. Hong, D. Yoon, Y. Jang, G. Kim, J. Chae, Y. Sung, K. Lee, et al. Online pre-training for offline-to-online reinforcement learning. In International Conference on Machine Learning (ICML) 2025, Cited by: §4.1. Sutton and Barto (2018) R. S. Sutton and A. G. Barto Reinforcement learning: an introduction. Second edition edition, Adaptive computation and machine learning series, The MIT Press, Cambridge, Massachusetts (en). External Links: ISBN 978-0-262-03924-6 Cited by: §1. Torabi et al. (2018) F. Torabi, G. Warnell, and P. Stone Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, p. 4950–4957. Cited by: §3.1. Wagenmaker et al. (2025) A. Wagenmaker, M. Nakamoto, Y. Zhang, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, and S. Levine Steering your diffusion policy with latent space reinforcement learning. Conference on Robot Learning. Cited by: §2.2. Wang et al. (2023) Z. Wang, J. J. Hunt, and M. Zhou Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2.2. Xie et al. (2021) T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai Policy finetuning: bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems 34, p. 27395–27407. Cited by: §4.1. Yang et al. (2026) S. Yang, B. Liu, and H. Chen PolicyFlow: policy optimization with continuous normalizing flow in reinforcement learning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2.2. Yu and Zhang (2023) Z. Yu and X. Zhang Actor-critic alignment for offline-to-online reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 40452–40474. External Links: Link Cited by: §4.1. Zhang et al. (2023) H. Zhang, W. Xu, and H. Yu Policy expansion for bridging offline-to-online reinforcement learning. In The Eleventh International Conference on Learning Representations, Cited by: §4.1. Zhang et al. (2026a) T. Zhang, C. Yu, S. Su, and Y. Wang ReinFlow: fine-tuning flow matching policy with online reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2.3. Zhang et al. (2026b) Y. Zhang, S. Yu, T. Zhang, M. Guang, H. Hui, K. Long, Y. Wang, C. Yu, and W. Ding SAC flow: sample-efficient reinforcement learning of flow-based policies via velocity-reparameterized sequential modeling. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2.3, §4.1, §5.2. Zhou et al. (2025) Z. Zhou, A. Peng, Q. Li, S. Levine, and A. Kumar Efficient online reinforcement learning fine-tuning need not retain offline data. In International Conference on Learning Representations(ICLR), External Links: Link Cited by: §4.1. Appendix A Environment In this section, we present the domain we use in our experiments in detail. Table 1 summarizes some information of each domain, including Reward Type, Dataset Size, Episode Length and Action Dimension. (a) Cube Double (b) Cube Triple (c) Cube Quadruple (d) Puzzle 4×44× 4 (e) Scene (f) Lift (g) Can (h) Square Figure 8: Visualizations of the environments. Subfigures (a-e) show the OGBench domains. Subfigures (f-h) depict the manipulation tasks from Robomimic. A.1 OGBench We choose 5 OGBench manipulation domains and use the single-task version of it. We use the standard play version of the dataset, in which trajectories were collected using a non-Markovian expert policy. For Cube Double/Triple/Quadruple, the agents need to manipulate an arm and learn a pick-and-place skill to place colored cube blocks to the right locations. Particularly, Cube Quadruple is an extremely challenging domain in which the agent must process four cubes in sequence. It is almost impossible to learn an effective policy under standard dataset. Hence, we use the 100M-size dataset provided officially. Puzzle 4×4 requires solving the ”Lights Out” puzzle with a robot arm. For Scene-sparse, we use the sparse-reward mode of the reward function such that the agent receives a reward of 0 only upon completing the task, and -1 in all other cases. Table 1: Specifications of the environments used in our experiments. Environment Reward Type Dataset Size Episode Length Action Dimension OGBench cube-double Sparse 1M 500 5 cube-triple Sparse 3M 1000 5 cube-quadruple Sparse 100M 1000 5 puzzle-4x4 Sparse 1M 500 5 scene Sparse 1M 750 5 Robomimic lift Sparse 31,127 500 7 can Sparse 62,756 500 7 square Sparse 80,731 500 7 A.2 Robomimic We select 3 challenging domain from Robomimic, including Lift, Can and Square. We conduct our experiments on Multi-Human datasets. These datasets were collected by 6 operators, including 300 successful trajectories. Lift requires the robot arm to pick up a cube. Can requires the robot arm to pick up a can and place it into a container. Square requires the robot arm to pick up a square sleeve and fit it onto a square post. Appendix B Implementation In this section, we present the details of our implementation of CFP on all the baselines. Flow Q-Learning. FQL is a strong baseline for offline reinforcement learning. Its main innovation is to exploit the expressive power of a multi-step flow network to approximate the behavior distribution in the offline dataset, while using a Q-loss to propagate gradients from the critic to the actor. Directly backpropagating through the multi-step flow generation process, however, may lead to unstable gradients. To avoid this issue, FQL introduces a one-step noise-conditioned policy that directly distills the actions generated by the multi-step behavior flow. Although the one-step policy may sacrifice some of the expressive capacity of the original multi-step flow, it enables more stable policy optimization. More concretely, FQL consists of the following three networks: 1. a behavior flow policy μθ(s,z) _θ(s,z), which generates an action from a state s and Gaussian noise z∼(0,Id)z (0,I^d) through multi-step flow integration; 2. a one-step noise-conditioned policy μω(s,z) _ω(s,z), which distills the output of the behavior flow policy; and 3. a critic Qϕ(s,a)Q_φ(s,a), which estimates the expected return of a state–action pair. The one-step policy is optimized by combining flow-policy distillation with Q-value maximization: Lπ(ω)=αs∼,z∼(0,Id)[‖μω(s,z)−μθ(s,z)‖22]+s∼,a∼πω(⋅∣s)[−Qϕ(s,a)],L_π(ω)= _ subarraycs ,\\ z (0,I^d) subarray [ \| _ω(s,z)- _θ(s,z) \|_2^2 ]+E_ subarraycs ,\\ a _ω(· s) subarray [-Q_φ(s,a) ], (9) where α controls the strength of behavior regularization. The first term distills the output of the multi-step behavior flow into the one-step policy, whereas the second term propagates the critic gradient to the one-step actor. The critic is trained with the temporal-difference objective LQ(ϕ)=(s,a,r,s′)∼[(Qϕ(s,a)−[r+γQϕ¯(s′,a′)])2],L_Q(φ)=E_(s,a,r,s ) [ (Q_φ(s,a)- [r+γ Q_ φ(s ,a ) ] )^2 ], (10) where a′∼πω(⋅∣s′)a _ω(· s ) and Qϕ¯Q_ φ denotes the target critic. QCFQL. QCFQL extends FQL to temporally extended action spaces by jointly generating and evaluating an action chunk t=(at,…,at+h−1)a_t=(a_t,…,a_t+h-1). The QCFQL actor objective is Lπ(ω)=αst∼,z∼(0,Ihd)[‖μω(st,z)−μθ(st,z)‖22]+st∼,t∼πω(⋅∣st)[−Qϕ(st,t)].L_π(ω)= _ subarraycs_t ,\\ z (0,I^hd) subarray [ \| _ω(s_t,z)- _θ(s_t,z) \|_2^2 ]+E_ subarraycs_t ,\\ a_t _ω(· s_t) subarray [-Q_φ(s_t,a_t) ]. (11) The QCFQL critic is trained using an h-step temporal-difference target: LQ(ϕ)=[(Qϕ(st,t)−[rt(h)+γhQϕ¯(st+h,t+h)])2],L_Q(φ)=E [ (Q_φ(s_t,a_t)- [r_t^(h)+γ^hQ_ φ(s_t+h,a_t+h) ] )^2 ], (12) where rt(h)=∑j=0h−1γjrt+j,t+h∼πω(⋅∣st+h).r_t^(h)= _j=0^h-1γ^jr_t+j, _t+h _ω(· s_t+h). (13) QCFQL-nstep. QCFQL-nstep is a similar implementation of QCFQL. Instead of distilling the multi-step network with single-step network, we use direct backpropagation. The QCFQL-nstep actor objective is Lπ(θ)=α(st,t)∼,z∼(0,Ihd)[‖μθ(st,z)−t‖22]+st∼,t∼πθ(⋅∣st)[−Qϕ(st,t)].L_π(θ)= _ subarrayc(s_t,a_t) ,\\ z (0,I^hd) subarray [ \| _θ(s_t,z)-a_t \|_2^2 ]+E_ subarraycs_t ,\\ a_t _θ(· s_t) subarray [-Q_φ(s_t,a_t) ]. (14) The remaining components of QCFQL-nstep are identical to those of QCFQL. QC. QC uses the same flow-matching behavior policy μθ _θ as FQL, but does not introduce an additional one-step noise-conditioned actor. Instead, QC implicitly defines its policy through best-of-N candidate selection. Specifically, for a given state s, we first sample N independent Gaussian noise vectors, z1,z2,…,zN∼i.i.d.(0,Ihd),z^1,z^2,…,z^N i.i.d. N(0,I^hd), (15) and generate N candidate actions using the multi-step behavior flow: ti=μθ(st,zi),i∈1,…,N.a_t^i= _θ(s_t,z^i), i∈\1,…,N\. (16) The policy then selects the candidate with the largest estimated Q-value: i∗=argmaxi∈1,…,NQϕ(st,i),πQC(s)=i∗.i^*= _i∈\1,…,N\Q_φ(s_t,a^i), _QC(s)=a^i^*. (17) Unlike FQL and QCFQL, QC does not backpropagate the critic gradient. The selected best-of-N action is directly used to construct the temporal-difference target. Accordingly, the critic is optimized by LQ(ϕ)=(s,t,r,s′)∼[(Qϕ(st,)−[rt(h)+γhQϕ¯(st+h,t+hi∗)])2],L_Q(φ)=E_(s,a_t,r,s ) [ (Q_φ(s_t,a_t)- [r_t^(h)+γ^hQ_ φ (s_t+h,a_t+h^\,i^* ) ] )^2 ], (18) where +∗∼πQC(⋅|st+h)a^\,i^*_t+h _QC(·|s_t+h). Next, we demonstrate how we incorporate CFP into the original algorithm. FQL-CFP. Since the implementations of all the algorithms are largely similar, we primarily describe how FQL-CFP is implemented. The objective of FQL-CFP during the offline stage is: Lπ(ω)=αs∼,z∼(0,Id)[‖μω(s,z)−μθ(s,z)‖22].L_π(ω)= _ subarraycs ,\\ z (0,I^d) subarray [ \| _ω(s,z)- _θ(s,z) \|_2^2 ]. (19) CFP does not present critic updates during the offline stage. During the warm-up phase, the actor and the initialized critic undergo brief updates and calibration using the offline dataset, with the following update objective: Lπ(ω)=αs∼,z∼(0,Id)[‖μω(s,z)−μθ(s,z)‖22]−s∼,a∼πω(⋅|s)[Qϕ(s,a)].L_π(ω)= _ subarraycs ,\\ z (0,I^d) subarray [ \| _ω(s,z)- _θ(s,z) \|_2^2 ]-E_s ,\,a _ω(·|s)[Q_φ(s,a)]. (20) The critic is trained with the temporal-difference objective LQ(ϕ)=(s,a,r,s′)∼[(Qϕ(s,a)−[r+γQϕ¯(s′,a′)])2],L_Q(φ)=E_(s,a,r,s ) [ (Q_φ(s,a)- [r+γ Q_ φ(s ,a ) ] )^2 ], (21) where a′∼πω(⋅∣s′)a _ω(· s ) and terminal transitions are masked in the bootstrap target. During the online phase, the agent samples trajectories in the real environment and stores them in the batch. The objectives are: Lπ(ω)=αs∼ℬ,z∼(0,Id)[‖μω(s,z)−μθ(s,z)‖22]−s∼,a∼πω(⋅|s)[Qϕ(s,a)],L_π(ω)= _ subarraycs ,\\ z (0,I^d) subarray [ \| _ω(s,z)- _θ(s,z) \|_2^2 ]-E_s ,\,a _ω(·|s)[Q_φ(s,a)], (22) LQ(ϕ)=(s,a,r,s′)∼ℬ[(Qϕ(s,a)−[r+γQϕ¯(s′,a′)])2].L_Q(φ)=E_(s,a,r,s ) [ (Q_φ(s,a)- [r+γ Q_ φ(s ,a ) ] )^2 ]. (23) Since the objectives for the warm-up and online phases are identical—differing only in the source of the updates (the warm-up phase relies entirely on the dataset, whereas the online phase utilizes batches containing online data)—no further distinction will be made between the two. QCFQL-CFP. The update method is very similar to that of FQL-CFP; QCFQL simply incorporates an action chunk. Its update objective during the offline stage is Lπ(ω)=αst∼,z∼(0,Ihd)[‖μω(st,z)−μθ(st,z)‖22]L_π(ω)= _ subarraycs_t ,\\ z (0,I^hd) subarray [ \| _ω(s_t,z)- _θ(s_t,z) \|_2^2 ] (24) For both warm-up and online stage, the targets are: Lπ(ω)=αst∼,z∼(0,Ihd)[‖μω(st,z)−μθ(st,z)‖22]+st∼,t∼πω(⋅∣st)[−Qϕ(st,t)],L_π(ω)= _ subarraycs_t ,\\ z (0,I^hd) subarray [ \| _ω(s_t,z)- _θ(s_t,z) \|_2^2 ]+E_ subarraycs_t ,\\ a_t _ω(· s_t) subarray [-Q_φ(s_t,a_t) ], (25) LQ(ϕ)=[(Qϕ(st,t)−[rt(h)+γhQϕ¯(st+h,t+h)])2].L_Q(φ)=E [ (Q_φ(s_t,a_t)- [r_t^(h)+γ^hQ_ φ(s_t+h,a_t+h) ] )^2 ]. (26) QCFQL-nstep-CFP. QCFQL-nstep-CFP has a similar implementation. The QCFQL-nstep actor objective during offline phase is: Lπ(θ)=α(st,t)∼,z∼(0,Ihd)[‖μθ(st,z)−t‖22]L_π(θ)= _ subarrayc(s_t,a_t) ,\\ z (0,I^hd) subarray [ \| _θ(s_t,z)-a_t \|_2^2 ] (27) For warm-up and online updates, the objectives are: Lπ(θ)=α(st,t)∼,z∼(0,Ihd)[‖μθ(st,z)−t‖22]+st∼,t∼πθ(⋅∣st)[−Qϕ(st,t)],L_π(θ)= _ subarrayc(s_t,a_t) ,\\ z (0,I^hd) subarray [ \| _θ(s_t,z)-a_t \|_2^2 ]+E_ subarraycs_t ,\\ a_t _θ(· s_t) subarray [-Q_φ(s_t,a_t) ], (28) LQ(ϕ)=[(Qϕ(st,t)−[rt(h)+γhQϕ¯(st+h,t+h)])2].L_Q(φ)=E [ (Q_φ(s_t,a_t)- [r_t^(h)+γ^hQ_ φ(s_t+h,a_t+h) ] )^2 ]. (29) QC-CFP. QC naturally shares the same training approach as CFP—specifically, using only the actor network to approximate the velocity field. So its training objective is: Lπ(θ)=x0∼(0,Ihd),(x1=a,s)∼,t∼Unif([0,1])[‖vθ(t,s,xt)−(x1−x0)‖22].L_π(θ)=E_ subarraycx_0 (0,I_hd),\\ (x_1=a,s) ,\\ t ([0,1]) subarray[\|v_θ(t,s,x_t)-(x_1-x_0)\|_2^2]. (30) For warm-up and online training, the objectives are: Lπ(θ)=x0∼(0,Ihd),(x1=a,s)∼,t∼Unif([0,1])[‖vθ(t,s,xt)−(x1−x0)‖22],L_π(θ)=E_ subarraycx_0 (0,I_hd),\\ (x_1=a,s) ,\\ t ([0,1]) subarray[\|v_θ(t,s,x_t)-(x_1-x_0)\|_2^2], (31) LQ(ϕ)=(s,t,r,s′)∼[(Qϕ(st,)−[rt(h)+γhQϕ¯(st+h,t+hi∗)])2],L_Q(φ)=E_(s,a_t,r,s ) [ (Q_φ(s_t,a_t)- [r_t^(h)+γ^hQ_ φ (s_t+h,a_t+h^\,i^* ) ] )^2 ], (32) where +∗a^\,i^*_t+h follows the Best-of-N sampling. Appendix C Experiment C.1 Hyperparameters Unless otherwise specified, we use the same hyperparameters across all environments and algorithms. The common hyperparameters are summarized in Table 2. Table 2: Common hyperparameters used in our experiments. Parameter Value Batch size (M) 256256 Discount factor (γ) 0.990.99 Optimizer Adam Learning rate 3×10−43× 10^-4 Target network update rate (τ) 5×10−35× 10^-3 Critic ensemble size 22 Number of flow integration steps (T) 44 for QCFQL-nstep; 1010 otherwise Action chunk length (h) 55 Number of offline training steps 1×1061× 10^6 Number of online environment steps 1×1061× 10^6 Network width 512512 Network depth 44 hidden layers Number of best-of-N candidates (QC) 3232 For specific hyperparameters of α value for backpropagation algorithms and N for Best-of-N algorithms, see Table 3. For hyperparameters of warm-up steps, see Table 4. Table 3: Task-specific hyperparameters for all O2O and CFP variants. Environment Task QC (N) FQL (α) QCFQL-nstep (α) QCFQL (α) O2O CFP O2O CFP O2O CFP O2O CFP Cube Double Task 1 32 32 300 300 100 100 100 100 Task 2 32 32 300 300 100 100 100 100 Task 3 32 32 300 300 100 100 100 100 Task 4 32 32 300 300 100 100 100 100 Task 5 32 32 300 300 100 100 100 100 Cube Triple Task 1 32 32 300 300 100 100 100 100 Task 2 32 32 300 300 100 100 100 100 Task 3 32 32 300 300 100 100 100 100 Task 4 32 32 300 300 100 100 100 100 Task 5 32 32 300 300 100 100 100 100 Cube Quadruple Task 1 32 32 300 300 100 100 100 100 Task 2 32 32 300 300 100 100 100 100 Task 3 32 32 300 300 100 100 100 100 Task 4 32 32 300 300 100 100 100 100 Task 5 32 32 300 300 100 100 100 100 Puzzle 4×44× 4 Task 1 64 64 1000 1000 1000 1000 1000 1000 Task 2 64 64 1000 1000 1000 1000 1000 1000 Task 3 64 64 1000 1000 1000 100 1000 100 Task 4 64 64 1000 1000 1000 1000 1000 1000 Task 5 64 64 1000 1000 1000 1000 1000 1000 Scene Task 1 32 32 300 300 300 300 300 300 Task 2 32 32 300 300 300 300 300 300 Task 3 32 32 300 300 300 300 300 300 Task 4 32 32 300 300 300 300 300 300 Task 5 32 32 300 300 300 300 300 300 Lift – 16 16 10000 10000 10000 10000 10000 10000 Can – 16 16 10000 10000 10000 10000 10000 10000 Square – 16 16 10000 10000 10000 10000 10000 10000 Table 4: Task-specific critic warm-up steps for the CFP variants. Environment Task QC-CFP FQL-CFP QCFQL-nstep-CFP QCFQL-CFP Cube Double Task 1 10000 10000 10000 10000 Task 2 10000 10000 10000 10000 Task 3 10000 10000 10000 10000 Task 4 10000 10000 10000 10000 Task 5 10000 10000 10000 10000 Cube Triple Task 1 10000 10000 10000 10000 Task 2 10000 10000 10000 10000 Task 3 10000 10000 10000 10000 Task 4 10000 10000 10000 10000 Task 5 10000 10000 0.1M 10000 Cube Quadruple Task 1 0.2M 0.2M 0.2M 0.2M Task 2 0.2M 0.2M 0.2M 0.2M Task 3 0.2M 0.2M 0.2M 0.2M Task 4 0.2M 0.2M 0.2M 0.2M Task 5 0.2M 0.2M 0.2M 0.2M Puzzle 4×44× 4 Task 1 10000 10000 10000 10000 Task 2 10000 10000 10000 10000 Task 3 50000 50000 50000 50000 Task 4 10000 50000 10000 10000 Task 5 10000 50000 10000 10000 Scene Task 1 10000 30000 10000 10000 Task 2 10000 30000 10000 10000 Task 3 10000 30000 10000 10000 Task 4 10000 30000 10000 10000 Task 5 10000 30000 10000 10000 Lift Task 1 10000 5000 10000 10000 Can Task 1 10000 5000 10000 10000 Square Task 1 10000 5000 10000 5000 C.2 Full Results OGBench. Figure 9 shows the full results on OGBench. Figure 9: Full OGBench results by tasks. Curves and shaded regions represent the mean and 95% confidence interval across five random seeds, respectively. Robomimic. Figure 5 shows the full results on Robomimic. Table 5 shows the detailed performance comparison between O2O and CFP. Almost all CFP variants reach performance comparable to or even better than those of the O2O variants. Table 5: Detailed mean performance comparison (%) by task. All highlighted cells correspond to CFP. Blue-shaded cells indicate that CFP significantly outperforms O2O; light-blue-shaded cells indicate minor variations or no significant difference; and gray-shaded cells indicate that O2O significantly outperforms CFP. Environment Task QC FQL QCFQL-nstep QCFQL O2O CFP O2O CFP O2O CFP O2O CFP Cube Double Task 1 84.8→ 100 99.2→ 100 48.8→ 100 12.8→ 100 48.8→ 100 20.0→ 100 53.2→ 100 9.2→ 100 Task 2 81.2→ 100 86.4→ 100 30.4→ 97.2 0→ 100 36.0→ 100 17.2→ 100 16.8→ 100 1.6→ 100 Task 3 73.2→ 100 89.2→ 100 25.6→ 98.8 0→ 100 26.0→ 100 4.8→ 100 12.8→ 100 0.4→ 100 Task 4 25.2→ 90.0 29.2→ 96.4 5.6→ 9.2 0→ 89.2 2.4→ 94.4 0.4→ 99.6 1.2→ 100 0→ 100 Task 5 80.4→ 100 78.8→ 100 20.0→ 97.6 0→ 100 34.8→ 100 13.6→ 100 12.0→ 100 0.8→ 100 Cube Triple Task 1 25.2→ 100 34.0→ 100 5.2→ 78.8 1.2→ 99.2 33.2→ 100 37.6→ 100 24.4→ 100 4.8→ 100 Task 2 0.8→ 91.2 0→ 96.0 0→ 1.2 0→ 1.2 5.2→ 97.6 0.4→ 100 0→ 92.0 0→ 98.4 Task 3 2.0→ 80.4 2.8→ 84.0 0.4→ 8.4 0→ 13.2 14.0→ 87.6 0→ 95.6 0→ 68.8 0→ 99.6 Task 4 0→ 38.0 0→ 68.4 0→ 0.4 0→ 0.4 3.2→ 75.6 0→ 91.6 0→ 30.8 0→ 98.4 Task 5 0→ 2.4 0→ 1.6 0→ 0 0→ 0 1.2→ 2.4 0.4→ 25.6 0→ 0 0→ 4.0 Cube Quadruple Task 1 21.2→ 100 16.0→ 100 2.4→ 48.8 0→ 68.4 16.8→ 100 1.6→ 100 8.4→ 100 0→ 100 Task 2 0→ 99.6 0.4→ 98.8 0→ 0.4 0→ 2.8 2.4→ 100 0→ 100 0→ 99.6 0→ 100 Task 3 1.6→ 76.8 1.2→ 86.4 0→ 2.0 0→ 10.8 0→ 100 0.4→ 100 0→ 80.0 0→ 100 Task 4 0→ 89.6 0→ 71.6 0→ 0 0→ 0 0→ 99.6 0→ 100 0→ 86.8 0→ 100 Task 5 0→ 0 0→ 2.4 0→ 0 0→ 0 0→ 0 0→ 0 0→ 0 0→ 0 Puzzle 4x4 Task 1 65.2→ 100 20.8→ 100 32.0→ 100 6.4→ 100 34.4→ 100 38.4→ 100 50.4→ 100 0.8→ 100 Task 2 0.8→ 10.0 3.2→ 22.8 12.8→ 12.8 2.8→ 62.0 0.8→ 0.8 1.6→ 1.6 6.4→ 6.8 5.2→ 5.2 Task 3 62.0→ 90.0 35.2→ 100 20.4→ 84.8 13.2→ 100 34.4→ 100 12.0→ 100 44.8→ 100 21.6→ 82.0 Task 4 1.2→ 1.2 14.8→ 85.2 9.6→ 64.4 9.2→ 100 4.0→ 42.0 2.0→ 40.4 12.0→ 29.2 2.0→ 80.4 Task 5 0.4→ 0.4 2.0→ 22.4 8.0→ 8.0 7.2→ 7.2 0.4→ 0.4 1.2→ 1.2 0.4→ 0.4 1.2→ 1.2 Scene Task 1 100→ 100 98.4→ 100 70.8→ 100 20.0→ 100 49.6→ 100 4.8→ 100 100→ 100 16.4→ 100 Task 2 98.0→ 100 99.6→ 100 48.4→ 100 3.2→ 100 39.6→ 100 0→ 100 88.8→ 100 3.2→ 100 Task 3 86.0→ 100 22.0→ 100 62.4→ 100 4.4→ 100 85.6→ 100 0.4→ 100 98.8→ 100 5.2→ 100 Task 4 94.4→ 100 99.6→ 100 72.4→ 100 6.4→ 100 83.6→ 100 12.0→ 100 89.2→ 100 15.2→ 100 Task 5 27.2→ 99.6 16.0→ 99.6 15.6→ 96.0 1.6→ 85.2 27.2→ 100 0.4→ 100 45.6→ 100 2.0→ 100 Lift – 92.8→ 100 91.6→ 100 79.2→ 98.8 81.6→ 98.4 96.4→ 100 96.0→ 100 98.0→ 100 95.6→ 100 Can – 80.8→ 97.2 80.8→ 96.8 26.4→ 77.6 35.6→ 76.4 82.4→ 99.6 81.6→ 99.6 83.6→ 98.8 90.0→ 97.6 Square – 38.0→ 94.4 37.2→ 94.0 6.0→ 23.2 4.4→ 12.4 30.8→ 73.2 30.4→ 82.4 32.4→ 82.8 27.2→ 80.4 C.3 Ablation Study We present more results of the ablation study here. Figure 10 further confirms our conclusion that CFP is insensitive to the steps of warm-up stage. Figure 10: Ablation of the number of steps during warm-up on OGBench. The applied algorithm is QCFQL-CFP. Curves and shaded regions represent the mean and 95% confidence interval across five random seeds, respectively. We also investigate CFP’s sensitivity to warm-up steps in the absence of Q-loss during the warm-up phase. Figure 11 shows that CFP without Q-loss during warm-up stage is more sensitive to the length of warm-up steps. Overall, the longer the warm-up period, the worse the performance of online fine-tuning of CFP. This result is significantly different from the conclusion we make in section 5.4.2. We examined the corresponding mean Q-values and find that the Q mean assigned by the critic decreased as the warm-up length increased. We attribute this phenomenon to the interplay within the actor-critic architecture. We have demonstrated that a fresh critic can provide guidance even during the warm-up phase, enabling the actor to output higher-value actions; conversely, since the target action in the critic’s update formula is provided by the actor, higher-quality actions from the actor result in higher action ratings from the critic. Without the inclusion of Q-loss, the actor remains confined to the low-quality actions present in the dataset, producing progressively lower critic estimates of state-action pairs. Our research further confirms that offline datasets—characterized by low environmental coverage and limited trajectory success rates—exacerbate the critic’s pessimism, thereby hindering generalization during the online phase. Figure 11: Ablation study of the sensitivity of CFP (without Q-loss) to warm-up steps. Appendix D Toy Example In this section, we introduce the toy example we implement in Section 3.2 in detail. We designed an MDP table and simulated the complete O2O phase as well as the CFP implementation during the actual training process. In our experiments, we rigorously controlled the variables associated with O2O and CFP, ensuring that the sole difference between the two was whether the critic was initialized during the online phase. D.1 Environment The environment is a finite MDP with 1616 states. For visualization, the states are placed on a 4×44× 4 canvas, 0123456789101112131415, matrix0&1&2&3\\ 4&5&6&7\\ 8&9&10&11\\ 12&13&14&15, matrix (33) but these coordinates do not define the transition dynamics. The dynamics are instead specified by an irregular directed graph. An action at state s selects one of its valid destination states s′∈+(s)s ^+(s); therefore the action-value function is represented as a matrix Q(s,s′)Q(s,s ). The graph includes both local edges and nonlocal directed shortcuts, and the existence of s→s′s→ s does not imply the existence of s′→s → s. Table 6 provides specific information. The figure in toy example 3.2 also demonstrates the same dynamics in a visual way. Table 6: Possible transitions of the sixteen-state directed MDP. s +(s)N^+(s) s +(s)N^+(s) 0 ∅ 8 1,4,9,12,14\1,4,9,12,14\ 1 0,2,5,9\0,2,5,9\ 9 2,5,8,10,15\2,5,8,10,15\ 2 1,6,8,11\1,6,8,11\ 10 3,6,9,11,14\3,6,9,11,14\ 3 2,7,10,14\2,7,10,14\ 11 4,7,10,15\4,7,10,15\ 4 0,5,8,11\0,5,8,11\ 12 0,5,8,13\0,5,8,13\ 5 1,4,6,10,13\1,4,6,10,13\ 13 4,7,9,12,14\4,7,9,12,14\ 6 2,5,7,9,14\2,5,7,9,14\ 14 3,6,10,13,15\3,6,10,13,15\ 7 3,6,11,13\3,6,11,13\ 15 ∅ State 0 is a bad terminal with reward −8-8, whereas state 15 is a good terminal with reward +12+12. Every other transition has a base reward of −0.05-0.05. Entering one of the hazard states ℋ=5,10,12H=\5,10,12\ incurs an additional penalty of −1.25-1.25. Several directed shortcuts also have edge-specific costs, listed in Table 7. Thus, for a nonterminal destination, r(s,s′)=−0.05−1.25[s′∈ℋ]+c(s,s′),r(s,s )=-0.05-1.25\,I[s ]+c(s,s ), (34) while transitions into states 0 and 15 receive their terminal rewards. The discount factor is γ=0.97γ=0.97. Table 7: Edge-specific costs c(s,s′)c(s,s ). Edge Cost Edge Cost Edge Cost 1→91→ 9 −0.70-0.70 2→112→ 11 +0.15+0.15 3→143→ 14 −0.40-0.40 4→114→ 11 −0.20-0.20 5→135→ 13 −0.60-0.60 6→146→ 14 +0.10+0.10 7→137→ 13 −0.30-0.30 8→18→ 1 −0.50-0.50 9→29→ 2 −0.40-0.40 10→310→ 3 −0.20-0.20 11→411→ 4 −0.60-0.60 12→512→ 5 −0.25-0.25 13→413→ 4 −0.35-0.35 14→314→ 3 −0.45-0.45 To simulate the inability of the offline dataset to provide optimal or suboptimal action coverage in complex environments, we designed the offline dataset to tend to reach the bad terminal. Specifically, for every nonterminal state, we compute the shortest directed distance from each valid successor to state 0. A probability mass of 0.9850.985 is distributed among the successors with minimum distance to state 0, while the remaining mass is distributed uniformly over all valid successors. This policy generates an offline dataset whose dominant trajectories terminate at state 0. Furthermore, to simulate a policy that more closely resembles the one operating in the real world environment—rather than the dataset policy—after the actor has undergone fine-tuning during the online phase, we implemented an online sampling strategy. Specifically, the prescribed online behavior is an ϵε-greedy policy with respect to the exact optimal Q-function of the finite MDP, with ϵ=0.05ε=0.05. It therefore predominantly follows high-value routes toward the positive terminal state 15. This policy is used only to generate state-action samples that model the rapid actor-distribution shift observed after a small number of online Q-loss updates. For clarity, the implementation uses three distinct datasets: 1. Offline dataset offD_off: generated by the policy attracted to terminal state 0; used to fit the offline flow and the offline critic. 2. Post-shift actor dataset actorD_actor: generated by the prescribed near-optimal online behavior. 3. Online critic replay replayD_replay: generated by πreplay=0.70πnear-optimal+0.30πuniform, _replay=0.70\, _near -optimal+0.30\, _uniform, (35) where πuniform _uniform is uniform over valid outgoing edges. By decoupling the online actor and the online critic (regardless of whether offline pre-training is employed), we ensure that the actor remains high-performing and reliable during the online phase; this prevents low-quality offline data from impairing the actor’s exploration capabilities—and, consequently, the training of the online critic. Similarly, by incorporating a uniform policy into the online critic’s training data, we simulate the presence of low-quality data that might otherwise affect the critic’s training. Each dataset contains 35,000 episodes. An episode begins from a uniformly sampled nonterminal state and ends at a terminal state or after 80 transitions. Truncated transitions are treated as terminal in the TD mask. D.2 Actor Each discrete destination state is embedded using its normalized two-dimensional canvas coordinate in [−1,1]2[-1,1]^2. Given a state s, Gaussian noise x0x_0, and flow time t, the flow network predicts a velocity field. For a dataset action x1x_1, Flow Matching minimizes xt x_t =(1−t)x0+tx1, =(1-t)x_0+tx_1, (36) LFlow L_Flow =[‖vθ(s,xt,t)−(x1−x0)‖22]. =E\! [ \|v_θ(s,x_t,t)-(x_1-x_0) \|_2^2 ]. (37) The flow network contains two hidden layers of width 128. At inference time, we use 10 Euler integration steps, clip the continuous action to [−1,1]2[-1,1]^2, and project it to the closest valid destination of the current state. We first train an offline flow π^off π_off on offD_off. Starting from this fitted flow, we then perform 500 additional flow updates on actorD_actor and freeze the resulting post-shift policy π^on π_on. D.3 Critic The critic concatenates a one-hot state vector with the two-dimensional destination embedding, applies a single hidden layer of width 48 with a tanh activation, and outputs one scalar Q-value. The critic minimizes LQ(ϕ)=(s,a,r,s′)∼,a′∼π^(⋅∣s′)[(Qϕ(s,a)−[r+γQϕ¯(s′,a′)])2]L_Q(φ)=E_(s,a,r,s ) ,a π(· s ) [ (Q_φ(s,a)- [r+γ Q_ φ(s ,a ) ] )^2 ] (38) The target parameters are updated with τ=0.02τ=0.02. D.4 Implementation The complete training pipeline is as follows. 1. Initialize one flow network and fit it to offD_off, obtaining π^off π_off. 2. Initialize a small critic once and save its random parameters ϕinit _init. 3. Train that critic for 500 TD updates on offD_off using π^off π_off in the TD target, obtaining ϕoff _off. 4. Adapt the flow policy on actorD_actor and freeze the resulting post-shift policy π^on π_on. 5. Draw one common sequence of online-replay minibatches and one common sequence of Gaussian policy-noise keys. 6. Run O2O from ϕoff _off and CFP from the saved ϕinit _init. Then perform 500 TD updates on the same replay using the same frozen π^on π_on, minibatches, and policy noises. D.5 Hyperparameters Table 8 summarizes the hyperparameters used in our toy example. Table 8: Hyperparameters of the toy example. Hyperparameter Value Hyperparameter Value Batch size 256 Discount factor γ 0.97 Offline episodes 35,000 Online actor episodes 35,000 Online replay episodes 35,000 Maximum episode length 80 Offline-policy greed 0.985 Online-policy ϵε 0.05 Uniform replay mixture 0.30 Flow steps 10 Actor hidden widths [128,128128,128] Actor learning rate 3×10−43× 10^-4 Critic hidden width 48 Critic learning rate 10−310^-3 Offline updates 500 Online updates 500 Target update τ 0.02 RMSE recording interval 10 D.6 Results and Interpretation We calculated the target Q-values for the offline and online phases based on offline and online strategies, respectively. We also evaluate the Q-values produced by the O2O and CFP critics. The left side of the figure 12 displays the theoretical target Q-values for the offline and online settings, as well as the Q-values output by the O2O and CFP critics, respectively. The right side shows heatmaps representing the absolute differences between pairs of the plots on the left; areas appearing more gray indicate greater similarity, while darker blue areas indicate larger differences. The O2O critic output is significantly closer to the target value of the offline dataset than the CFP critic output. Conversely, the CFP critic output is closer to the online target Q-value. Figure 12: Results of toy example.