Paper deep dive

Higher-Order Action Supervision Makes A Strong Policy Class

Peng Cheng, Yunxian Hou, Zhi Zhou, Qian Zhang, Chang Huang, Xianyuan Zhan

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 155

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 10/12/2026, 5:56:19 AM

Summary

This paper introduces a novel higher-order action supervision method to improve the robustness and stability of reinforcement learning (RL) and imitation learning (IL) policies. The authors argue that standard methods fail due to a lack of temporal consistency in action outputs. They propose a lightweight, plug-and-play loss scheme that supervises both zeroth-order (action values) and first-order (action time derivatives) actions. This is achieved by leveraging the policy's time derivative property ($\dot{a} = \nabla_s \pi(s) \cdot \dot{s}$) without modifying the underlying network architecture. The method is applicable to deterministic, stochastic, and flow-matching policies and integrates seamlessly with offline RL frameworks. Evaluations on OGBench and D4RL demonstrate significant improvements in performance, robustness, and out-of-distribution (OOD) generalization.

Entities (11)

Higher-Order Action Supervision · method · 98%First-Order Action · concept · 95%Zeroth-Order Action · concept · 95%D4RL · datasetbenchmark · 92%OGBench · datasetbenchmark · 92%Control Instability · problem · 90%Imitation Learning · field · 90%Reinforcement Learning · field · 90%Tsinghua University · organization · 90%Behavior Cloning · algorithm · 85%Flow Matching · algorithm · 85%

Relation Signals (9)

Higher-Order Action Supervision → improves → Control Robustness

confidence 95% · simultaneously supervising both zeroth- and first-order actions can dramatically enhance policies' performance and control robustness.

Higher-Order Action Supervision → supervises → First-Order Action

confidence 95% · we introduce a novel and elegant loss scheme... with the capability for higher-order action supervision

Higher-Order Action Supervision → supervises → Zeroth-Order Action

confidence 95% · simultaneously supervising both zeroth- and first-order actions

Higher-Order Action Supervision → isevaluatedon → OGBench

confidence 92% · Extensive evaluations on OGBench and D4RL demonstrate that our approach yields substantial performance and robustness improvements

Higher-Order Action Supervision → isevaluatedon → D4RL

confidence 92% · Extensive evaluations on OGBench and D4RL demonstrate that our approach yields substantial performance and robustness improvements

Higher-Order Action Supervision → enhances → Out-of-Distribution Generalization

confidence 90% · our method can also enhance policies’ out-of-distribution (OOD) generalization capability in the challenging low-data regime

Higher-Order Action Supervision → isappliedto → Reinforcement Learning

confidence 90% · seamlessly integrates with a broad spectrum of existing offline RL frameworks.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Modern data-driven decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), have achieved great success in solving many complex tasks. However, these methods often suffer from serious control instability and robustness issues when applied in real-world applications such as robotics and autonomous driving, posing notable challenges for their practical deployment. We argue that this instability issue stems largely from their limitations in solely supervising and optimizing zeroth-order actions (i.e., the action labels), failing to account for higher-order action dynamics and temporal consistency. In this paper, we show that simultaneously supervising both zeroth- and first-order actions can dramatically enhance policies' performance and control robustness. To achieve this, we introduce a novel and elegant loss scheme supported by formal theoretical guarantees that can equip any off-the-shelf policy model (e.g., deterministic, stochastic, or flow policies) with the capability for higher-order action supervision, without requiring any structural modifications. Moreover, our proposed method can serve as a lightweight plug-and-play module that seamlessly integrates with a broad spectrum of existing offline RL frameworks. Extensive evaluations on OGBench and D4RL demonstrate that our approach yields substantial performance and robustness improvements across a wide range of continuous control environments. Notably, our method can also enhance policies' out-of-distribution (OOD) generalization capability in the challenging low-data regime, making it an ideal tool in tackling many real-world control problems.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

152,115 characters extracted from source content.

Expand or collapse full text

Higher-Order Action Supervision Makes A Strong Policy Class Peng Cheng †thanks: Equal contribution. Affiliation: Institute for AI Industry Research (AIR), Tsinghua University Affiliation: Zenfex AI Lab Yunxian Hou11footnotemark: 1 Affiliation: University of Electronic Science and Technology of China Zhi Zhou Affiliation: Institute for AI Industry Research (AIR), Tsinghua University Affiliation: Zenfex AI Lab Qian Zhang Affiliation: Horizon Robotics Chang Huang Affiliation: Horizon Robotics Xianyuan Zhan †thanks: Corresponding author. Affiliation: Institute for AI Industry Research (AIR), Tsinghua University Abstract Modern data-driven decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), have achieved great success in solving many complex tasks. However, these methods often suffer from serious control instability and robustness issues when applied in real-world applications such as robotics and autonomous driving, posing notable challenges for their practical deployment. We argue that this instability issue stems largely from their limitations in solely supervising and optimizing zeroth-order actions (i.e., the action labels), failing to account for higher-order action dynamics and temporal consistency. In this paper, we show that simultaneously supervising both zeroth- and first-order actions can dramatically enhance policies’ performance and control robustness. To achieve this, we introduce a novel and elegant loss scheme supported by formal theoretical guarantees that can equip any off-the-shelf policy model (e.g., deterministic, stochastic, or flow policies) with the capability for higher-order action supervision, without requiring any structural modifications. Moreover, our proposed method can serve as a lightweight plug-and-play module that seamlessly integrates with a broad spectrum of existing offline RL frameworks. Extensive evaluations on OGBench and D4RL demonstrate that our approach yields substantial performance and robustness improvements across a wide range of continuous control environments. Notably, our method can also enhance policies’ out-of-distribution (OOD) generalization capability in the challenging low-data regime, making it an ideal tool in tackling many real-world control problems. 1 Introduction Learning-based decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), have achieved notable success in many complex tasks, such as games (Mnih et al., 2015; Silver et al., 2017), robotics (Kalashnikov et al., 2018; Chi et al., 2023), and autonomous driving (Kendall et al., 2019; Zheng et al., 2025b). However, when these learning-based approaches are deployed in real-world control problems, they are frequently criticized for their control instability and poor robustness. Examples of such issues include lack of smoothness in action outputs, severe jittering across consecutive steps, and poor generalization in offline learning under data sparsity (Dulac-Arnold et al., 2021; Song et al., 2023; Cheng et al., 2023; Cheng et al., 2026). In contrast to conventional control theories (Kirk, 2004) that place heavy emphasis on control stability and higher-order behavior modeling, contemporary learning-based decision-making methods in the AI community predominantly only focus on supervising and optimizing zeroth-order actions (i.e., the actual action labels). In realistic control scenarios, the neglect of higher-order temporal consistency in actions frequently leads to control instability and non-smooth action execution, which constitutes a critical bottleneck for real-world deployment (Dulac-Arnold et al., 2021). Under physical hardware constraints, such high-frequency action fluctuations can cause excessive mechanical wear and energy consumption, drastically shortening the lifespan of both the system and its constituent parts (Song et al., 2023; Song et al., 2025; Mysore et al., 2021; de La Presilla et al., 2023; Zhan et al., 2022; Zhan et al., 2025). These vulnerabilities pose severe safety risks and degrade operational reliability across various domains, including drone control (Mysore et al., 2021; Shi et al., 2019), robotic manipulation (Yu et al., 2021; Zhang et al., 2024), and autonomous driving (Cai et al., 2020; Chen et al., 2021; Wasala et al., 2020; Lee et al., 2024). Fundamentally, this issue arises because the policy becomes excessively sensitive to minor state-action perturbations, yielding disproportionately large action variations (Song et al., 2023; Chen et al., 2021). We empirically illustrate this vulnerability using a maze navigation task in Figure 1. As shown in Figure 1(a), the policy learned via standard behavior cloning (BC), which lacks higher-order action supervision, produces large and highly oscillatory first-order actions, eventually producing a suboptimal trajectory. By contrast, jointly supervising both zeroth- and first-order actions enforces much stronger behavioral regularization, resulting in significantly smoother and optimal execution (Figure 1(b)). Figure 1: Trajectories and zeroth-/first-order actions of policies learned on the maze2d task with 20 successful trajectories and 500k training steps. Green arrows denote the zeroth-order action a (joint torque), while red arrows denote the first-order action a˙ a (time derivative of joint torque, computed as a′−a -a). (a) Standard behavior cloning (BC) without a˙ a supervision, which yields a suboptimal trajectory with highly noisy and temporally inconsistent first-order actions. (b) Our method with higher-order action supervision, which yields an optimal navigation trajectory with much smoother and more stable zeroth- and first-order actions. However, incorporating higher-order action information in policy modeling is not an easy task. As the 0th0^th-order action a and the 1st1^st-order action a˙ a live in two completely different action spaces, naïvely predicting these two sets of actions not only doubles the action space, causing learning difficulties, but also faces potential conflicting gradient issues that are commonly encountered in multi-task learning (Yu et al., 2020). Therefore, existing works on action smoothness improvement primarily focus on introducing action fluctuation penalty regularizations (Mysore et al., 2021; Kobayashi, 2022; Shen et al., 2020; Zhao et al., 2022), or designing special neural network architectures to constrain the network’s Lipschitz constant, e.g., via spectral normalization or specialized modules like LipsNet (Song et al., 2023; Song et al., 2025). However, both streams of work suffer from a number of limitations. Adding smoothness penalties relies on task-specific ad-hoc design and cumbersome tuning efforts, whereas using specialized, heavily regularized neural network architectures loses generality and restricts the model’s expressivity to capture complex data distributions. In this work, we introduce a novel and elegant learning scheme that can achieve simultaneous 0th0^th- and 1st1^st-order action supervision. The best part is, we can easily transform any given policy model into a more powerful policy class without making any structural modification. The secret recipe is to leverage the intrinsic property of the policy model’s time derivative for supervision, i.e., π˙(s)=∇sπ(s)⋅s˙ π(s)= _sπ(s)· s. We provide practical instantiations of loss schemes for deterministic, stochastic, and generative flow matching policy classes, supported by formal theoretical guarantees for the IL setting. Moreover, we show that our enhanced policy class is naturally compatible with the continuous-time RL paradigm, like the Hamilton-Jacobi-Bellman (HJB) framework. This leads to a plug-and-play enhancement module that can be seamlessly integrated with a wide spectrum of existing offline RL algorithms with minimal code modifications. We evaluate our method through extensive experiments on diverse tasks from the large-scale OGBench (Park et al., 2025a) and widely used D4RL (Fu et al., 2020) benchmarks. Despite its simplicity, our approach achieves consistent and significant performance improvements when added to different offline policy optimization paradigms. Notably, in challenging small-dataset settings, our proposed modification excels by providing robust OOD generalization support, enabling the agent to maintain stable and high performance even on tasks where the base algorithms fail or severely degrade. This makes our method an ideal tool for tackling many real-world control problems. 2 Related work Smoothness and robustness enhancement for policy learning. Mitigating action fluctuation and trajectory drift remains a critical challenge for deploying policies in physical systems (Dulac-Arnold et al., 2021; Song et al., 2023). To enforce action smoothness and robustness, prior studies in continuous control have primarily explored two directions: penalty-based regularizations (Mysore et al., 2021; Kobayashi, 2022; Shen et al., 2020; Zhao et al., 2022) and architectural enhancements (Song et al., 2023; Song et al., 2025). Penalty-based regularization approaches, such as CAPS (Mysore et al., 2021), L2C2 (Kobayashi, 2022), and SR2L (Shen et al., 2020; Zhao et al., 2022), introduce action fluctuation penalty regularizations to their policy optimization objectives. However, they inherently rely on querying or sampling synthetic states to enforce local action consistency. In the offline setting, where interaction with the environment is not permitted, such state-dependent sampling may introduce significant OOD extrapolation errors, and often suffer from significant performance variability across environments (Christmann et al., 2024). In contrast, architectural enhancement methods explicitly constrain the network’s Lipschitz constant, e.g., via spectral normalization or specialized modules like LipsNet (Song et al., 2023; Song et al., 2025). However, imposing extra structural priors can also restrict the model’s expressivity. This introduces a fundamental trade-off: while the policy becomes smoother, the restricted capacity commonly leads to underfitting, degraded control performance, and additional computational overhead during inference. Concurrently, there are also a few attempts to incorporate higher-order action derivatives to enhance control smoothness, such as TaSIL (Pfrommer et al., 2022), which minimizes the gap between the Taylor series expansion of the learned and expert policies. Although promising, it remains more theoretical rather than practical, due to the complex evaluation procedure involved. Offline RL and continuous-time RL. In this study, we primarily focus on offline policy learning, such as offline IL (e.g., behavior cloning, BC) and offline RL, which directly imitate or optimize over a fixed offline dataset without environment interaction. Compared with IL, offline RL is more complex, which optimizes the cumulative return while being regularized towards the behavior data to avoid extrapolation error accumulation (Levine et al., 2020; Kumar et al., 2019). Depending on how the learned policy is parameterized, existing offline RL algorithms can be classified into three categories. The first category employs deterministic policies and explicitly constrains the learned actions toward the dataset using distance-based behavior cloning penalties (Fujimoto and Gu, 2021; Li et al., 2023b; Cheng et al., 2023). The second and also the more popular category adopts stochastic policies (e.g., Gaussian policies) (Wu et al., 2019; Kumar et al., 2019; Kumar et al., 2020; Kostrikov et al., 2022; Lyu et al., 2022; Xu et al., 2023; Wang et al., 2023; Xu et al., 2022a; Xu et al., 2022b; Niu et al., 2022; Li et al., 2023a; Mao et al., 2024b; Xu et al., 2026), which maximizes the likelihood of high-return actions under some distribution-based data regularizations. Recently, the third category of offline RL methods has emerged, which leverages expressive generative models, such as diffusion or flow matching policies, to capture the complex, multimodal action distributions in the datasets (Wang et al., 2022; Janner et al., 2022; Hansen-Estruch et al., 2023; Li et al., 2024; Mao et al., 2024a; Zheng et al., 2024; Park et al., 2025b; Zheng et al., 2025a; Liang et al., 2026; Zheng et al., 2026). However, all of the above methods are restricted to learning policies in the 0th0^th-order action spaces, which suffer from notable control stability and robustness issues. Incorporating higher-order action supervision in offline RL remains entirely unexplored. Another research avenue closely related to our work is continuous-time RL, which transcends the structural limitations of discrete-time Markov decision processes (MDPs) by modeling state and action evolution continuously over time (Doya, 2000; Wang et al., 2020; Munos, 2006). The Hamilton-Jacobi-Bellman (HJB) equation serves as the core theoretical foundation for this domain (Lee and Sutton, 2021; Vamvoudakis and Lewis, 2010; Bhasin et al., 2013; Vamvoudakis et al., 2014). Recent advancements have successfully utilized HJB-based frameworks to demonstrate how continuous-time structural priors can directly guide stable policy evaluation and control (Kim et al., 2021; Jia and Zhou, 2022; Lien et al., 2024). Although the HJB-based framework naturally requires 1st1^st-order action a˙ a information to capture Q-value’s variation in continuous time, due to the lack of a proper strategy to evaluate a˙ a from policy π, existing methods often resort to a crude approximation or replace it with a fixed bounding value. This drastically degrades the value estimation quality under the HJB framework and leads to suboptimal performance. 3 Extending Standard Policy Classes for Higher-Order Action Supervision In this paper, unlike prior methods that mitigate action prediction instability by using specialized network architectures or introducing task-specific ad-hoc designs, we propose an elegant, lightweight, and plug-and-play learning scheme to transform a standard policy model (regardless of deterministic, stochastic, or generative flow policies) into a more powerful policy class. Our proposed method enables simultaneous 0th0^th and 1st1^st-order action supervision for a policy model, maximally utilizing every bit of information in the offline data. 3.1 Notations We consider the sequential decision-making problem modeled as an MDP, defined by the tuple ℳ=,,,r,ρ0,γM=\S,A,P,r, _0,γ\. S represents the state space, ⊂ℝdA ^d is the d-dimensional continuous action space, ⁡(s′|s,a)P(s |s,a) denotes the stochastic transition dynamics, r:×→ℝr:S×A is the reward function, ρ0(s) _0(s) specifies the initial state distribution, and γ∈[0,1)γ∈[0,1) is the discount factor. In the offline setting, the agent cannot interact with the environment and is instead provided with a static dataset D consisting of offline trajectories τ=s0,a0,r0,s1,a1,…τ=\s_0,a_0,r_0,s_1,a_1,…\. In the offline IL setting, we aim to learn an imitative policy π to clone the expert behaviors specified in the data trajectories; whereas in offline RL, we aim to learn a policy π exclusively from the dataset D that maximizes the expected cumulative discounted return: J⁡(π)=τ∼π[∑t=0∞γtr(st,at)]J(π)=E_τ π [ _t=0^∞γ^tr(s_t,a_t) ]. In this paper, we use π⁡(s)π(s) to denote the deterministic policy and π(⋅|s)π(·|s) to denote the stochastic policy. Unless told otherwise, we use a to denote the 0th0^th-order action and a˙=dadt a= dadt to denote the 1st1^st-order action. In the discrete-time MDP setting, we approximate the time derivatives of the state and actions using finite differences, i.e., s˙≈(s′−s)/Δt s≈(s -s)/ t and a˙≈(a′−a)/Δt a≈(a -a)/ t, with the time interval set to Δt=1 t=1 by default. 3.2 Supervising Both Zeroth and First-Order Action for Imitation Learning The core difficulty to achieve simultaneous 0th0^th and 1st1^st-order action supervision lies in how to properly inject a˙ a information within the original model structure of the policy, since conventional policy models, regardless of deterministic policy π⁡(s)π(s) or stochastic policy π(⋅|s)π(·|s) are designed for the straightforward mapping between state s and the 0th0^th-order action a. However, if we are allowed to take the time derivative of policy π, then we will notice the following property (using the simplest deterministic-policy case as an example): a˙=π˙(s)=dπ(s)dt=dπ(s)ds⋅dsdt=dπ(s)dss˙=∇sπ(s)s˙. a= π(s)= dπ(s)dt= dπ(s)ds· dsdt= dπ(s)ds s= _sπ(s) s. (1) The above equation suggests that we can express the time derivative of policy π as the product of the gradient of policy ∇sπ(s) _sπ(s) and 1st1^st-order state s˙ s. As s˙ s is directly obtainable from the offline dataset during training, this offers us a new perspective to supervise 1st1^st-order action signal directly on the original policy model. In the following, we describe the detailed higher-order action supervision strategies for deterministic, stochastic, and generative flow-matching policies. Deterministic policy. With Eq. (1), incorporating 1st1^st-order action supervision for deterministic policies under the IL setting is straightforward. We can simply add an extra 1st1^st-order loss term ℒa˙detL_ a^det to the original BC loss: minπ⁡(s,a,s′,a′)∼[‖π⁡(s)−a‖2+‖∇sπ(s)s˙−a˙‖2⏟ℒa˙det]. _πE_(s,a,s ,a ) [ \|π(s)-a \|^2+ \| _sπ(s) s- a \|^2_L_ a^det ]. (2) where s˙=s′−s s=s -s and a˙=a′−a a=a -a are directly calculated from dataset samples. The second term ℒa˙detL_ a^det regresses π˙(s) π(s) to the ground truth a˙ a, thereby achieving first-order temporal consistency regularization without the need to make any changes to the original policy network. In practice, we can directly evaluate ∇sπ(s)s˙ _sπ(s) s using Jacobian-vector product (JVP), which is much cheaper to compute than calculating the Jacobian matrix ∇sπ(s) _sπ(s), and is natively supported in modern deep learning frameworks like JAX and PyTorch 2.0. A keen reader might notice that we do not introduce any weighting parameter to the two loss terms in Eq. (2), as we find empirically that keeping the same supervision strength of 0th0^th and 1st1^st-order actions normally provides the best performance. Stochastic policy. Stochastic policies are often modelled as a Gaussian policy π⁡(a|s)=⁡(a,μ⁡(s),diag(σ2(s)))π(a|s)=N(a;μ(s),diag(σ^2(s))) in practice, with μ⁡(s)μ(s), σ2(s)σ^2(s) being implemented as parameterized neural networks outputting mean and dimension-independent variances. They have been widely used in many offline IL and RL algorithms due to their flexibility in capturing dataset distributions. To enable 1st1^st-order action supervision on a stochastic policy, one naïve approach is to supervise the mean policy μ⁡(s)μ(s) with an additional loss term ℒa˙sto-mean=‖∇sμ(s)s˙−a˙‖L_ a^sto-mean= \| _sμ(s) s- a \| as in Eq. (2). We refer to this version as "mean supervision". Below, we also provide a more rigorous version consistent with the stochastic process framework. Specifically, if we model the stochastic component of the Gaussian policy using an Ornstein–Uhlenbeck (OU) (Uhlenbeck and Ornstein, 1930) process, dXt=−λXtdt+2λdWt,X0∼⁡(0,I)dX_t=-λ X_t\,dt+ 2λ\,dW_t,X_0 (0,I), where λ controls the temporal correlation of the process, and WtW_t denotes a d-dimensional standard Wiener process. By the stationarity of the OU process, for any t, we have Xt∼⁡(0,I)X_t (0,I). The action can thus be written as a=μ⁡(s)+σ⁡(s)⊙Xta=μ(s)+σ(s) X_t, where ⊙ denotes element-wise multiplication. By the multi-dimensional Itô formula (Oksendal, 2013), we obtain the stochastic differential equation (SDE) of the action process, da=[μ˙(s)+(σ˙(s)−λσ(s))⊙((a−μ⁡(s))⊘σ⁡(s))]dt+2λσ(s)⊙dWt,da= [ μ(s)+ ( σ(s)-λσ(s) ) ( (a-μ(s) ) σ(s) ) ]dt+ 2λ\,σ(s) _t, (3) where ⊘ denotes element-wise division. The resulting SDE induces a first-order policy distribution over the time derivative of actions (see Appendix C.1 for detailed derivation), π˙(a˙|s,a,s˙)=⁡(a˙,μ˙(s)+(σ˙(s)−λσ(s))⊙((a−μ⁡(s))⊘σ⁡(s)), 2λdiag⁡(σ2(s))), π( a|s,a, s)=N\! ( a;\, μ(s)+ ( σ(s)-λσ(s) ) ( (a-μ(s) ) σ(s) ),\,2 (σ^2(s)) ), (4) where μ˙(s)=∇sμ(s)⋅s˙ μ(s)= _sμ(s)· s and σ˙(s)=∇sσ(s)⋅s˙ σ(s)= _sσ(s)· s can be evaluated in a similar way as in Eq. (1). The above distribution specifies the likelihood of the time derivative of stochastic actions conditioned on the current state, action, and state derivative. We can thus jointly maximize the likelihood of 0th0^th and 1st1^st-order action for stochastic policies under the IL setting as follows: minπ⁡(s,a,s′,a′)∼[−log⁡π⁡(a|s)−log⁡π˙(a˙|s,a,s˙)⏟ℒa˙sto]. _πE_(s,a,s ,a ) [- π(a|s)- π( a|s,a, s)_L_ a^sto ]. (5) Beyond conventional action-level maximum likelihood in the first term, the derivative likelihood ℒa˙stoL_ a^sto in Eq.(5) serves as a probabilistic first-order regularizer that aligns the local temporal evolution of sampled actions with the expert trajectory. We provide the full derivation in Appendix C.1. Flow policy. Flow matching provides an effective and powerful framework for generative modeling (Lipman et al., 2023; Liu et al., 2023), and has gained increasing popularity in recent RL and embodied AI models. The central idea is to learn a time-dependent velocity field v:[0,1]×ℝd→ℝdv:[0,1]×R^d ^d (note this "time" is different from the physical time, which we denote as k) that induces a continuous flow ψ:[0,1]×ℝd→ℝdψ:[0,1]×R^d ^d between a base distribution and the target data distribution, where dkψ(k,x)=v⁡(k,ψ⁡(k,x)),ψ⁡(0,x)=x ddkψ(k,x)=v (k,ψ(k,x) ),ψ(0,x)=x. In this paper, we consider the standard linear interpolation path (Lipman et al., 2023; Liu et al., 2023), which defines an interpolation process for k∈[0,1]k∈[0,1] as xk=(1−k)x0+kx1,v=x1−x0x^k=(1-k)x^0+kx^1,v=x^1-x^0 between a noise sample x0x^0 and a data sample x1x^1. For flow policy (Park et al., 2025b), let a0∼⁡(0,I)a^0 (0,I) denote a noise sample and a1∼a^1 denote an action. Given the input state s, the generated action is obtained by integrating the learned velocity field along the interpolation path. Note that under the formalism of flow matching, if we differentiate the generated action a1=a0+∫01v⁡(k,s,ak)ka^1=a^0+ _0^1v(k,s,a^k)\,dk with respect to physical time t, we have a˙1=d∫01v⁡(k,s,ak)kdt=∫01dv(k,s,ak)dtk=∫01∂v⁡(k,s,ak)∂ss˙+∂v⁡(k,s,ak)∂aka˙kk. a^1= d _0^1v(k,s,a^k)\,dkdt= _0^1 dv(k,s,a^k)dt\,dk= _0^1 ∂ v(k,s,a^k)∂ s s\ + ∂ v(k,s,a^k)∂ a^k a^k\ dk. (6) We omit a˙0 a^0 as the noise sample a0a^0 is drawn independently of the physical trajectory, hence calculating the time derivative on noise is meaningless. Equation (6) shows that first-order action evolution can be expressed by integrating v˙(k,s,ak) v(k,s,a^k) over flow time, analogous to the original action generation process. This shared structure motivates supervising v˙(k,s,ak) v(k,s,a^k) in a flow-matching manner by regressing it toward the pointwise target a˙1=a′−a1 a^1=a -a^1. In practice, we approximate v˙(k,s,ak) v(k,s,a^k) with ∂v⁡(k,s,ak)∂ss˙ ∂ v(k,s,a^k)∂ s s. We omit the action-dependent term in Eq. (6) because evaluating it increased computational cost without improving performance, as discussed in Appendix B.2. Therefore, the new objective for flow policies with higher-order supervision can be given as: minπ⁡a0∼⁡(0,I),(s,a1,s′,a′)∼,k∈[0,1][‖v⁡(k,s,ak)−(a1−a0)‖+‖∂v⁡(k,s,ak)∂ss˙−a˙1‖⏟ℒa˙flow]. _πE_a^0 (0,I),\,(s,a^1,s ,a ) ,\,k∈[0,1] [ \|v(k,s,a^k)-(a^1-a^0) \|+ \| ∂ v(k,s,a^k)∂ s s- a^1 \|_L_ a^flow ]. (7) Standard flow matching mainly supervises transformation along the probability path at time k, while leaving the physical-time response of the conditional velocity field largely unconstrained. In Eq. (7), the additional term ℒa˙flowL_ a^flow completes this missing piece by aligning the state-directional change of the velocity field with the observed action evolution. We provide further discussion in Appendix C.2. 3.3 Theoretical Analysis With the additional first-order supervision, we can show in the following theorems that this scheme can lead to a tighter error bound for imitation learning under a set of commonly used assumptions (see Appendix D.2). Specifically, consider the learned policy π and expert policy π∗π^* start at the same state s⁡(0)s(0) (we denote s⁡(t)s(t) as the state in continuous time t), and define the policy error as e⁡(s)=π⁡(s)−π∗(s)e(s)=π(s)-π^*(s). We can show that the policy error can be better bounded with stronger 1st1^st-order supervision (see Appendix D.4 for the proof): Theorem 1. Under Assumption 1 in Appendix D.2, the policy error e(s∗(t))e(s^*(t)) satisfies: ‖e⁡(s∗(t))‖≤‖e⁡(s∗(0))‖+tmax0≤τ≤t‖∇sπ(s∗(τ))s˙∗(τ)−π˙∗(s∗(τ))‖.\|e(s^*(t))\|≤\|e(s^*(0))\|+t\, _0≤τ≤ t \| _sπ(s^*(τ)) s^*(τ)- π^*(s^*(τ)) \|. Theorem 1 shows that the growth of policy error e⁡(s∗)e(s^*) on expert trajectory s∗s^* is governed by its initial value and the accumulated 1st1^st-order discrepancy between the learned and expert policies. Note that the standard BC objective does not supervise the second term, hence could lead to substantially increased errors. As Theorem 1 only provides analysis on the per-state policy error, next, we provide a stronger conclusion to bound the state deviation δ⁡(t)=s⁡(t)−s∗(t)δ(t)=s(t)-s^*(t) between states from policy rollout s and expert trajectories s∗s^*, which captures the closed-loop long-term error accumulation. Theorem 2. Under Assumptions 1–4 in Appendix D.2, for δ⁡(0)=0δ(0)=0, the state deviation δ⁡(t)δ(t) satisfies, ‖δ⁡(t)‖≤Laβ(eβt−1)max0≤τ≤t‖e⁡(s∗(τ))‖.\|δ(t)\|≤ L_aβ (e^β t-1 ) _0≤τ≤ t\|e(s^*(τ))\|. The detailed proof and exact forms of constants LaL_a and β are provided in Appendix D.5. Theorem 2 decomposes closed-loop policy rollout state deviation into two parts: the policy mismatch error e⁡(s∗)e(s^*) measured on the expert trajectory s∗s^*, and the amplification gain Laβ(eβt−1) L_aβ(e^β t-1) contributed by the environment dynamics, capturing the error amplification due to closed-loop policy-environment interaction. Again, we can see that the long-term state deviation can also be better controlled under higher-order supervision, due to the tighter error bound ‖e(s∗(t))‖\|e(s^*(t))\| obtained in Theorem 1. 4 Higher-Order Policy Optimization in Offline RL The extended policy in Section 3 not only enables higher-order action supervision, but also provides a practical way to evaluate action derivatives π˙θ(s) π_θ(s), a quantity that existing offline RL methods never explicitly use. We now show that it has a natural place in critic learning through the Hamilton-Jacobi-Bellman based continuous-time RL framework, where value learning depends not only on scalar Bellman consistency, but also on how the value changes along the local state-action flow. 4.1 HJB Equation and Value Learning with First-Order Supervision In the HJB-based continuous-time RL framework, the Q-function is defined by (Kim et al., 2021): QHJB(s,a):=max⁡∫0∞a∈⁡e−γtr(s⁡(t),a⁡(t))t.Q^HJB(s,a):= _a _0^∞e^-γ tr(s(t),a(t))dt. (8) The dynamic programming equation of the above Q-function is expressed as an HJB equation, which is derived by using the Taylor expansion: γQHJB(s,a)=maxa∈⁡[r⁡(s,a)+∇sQHJB(s,a)s˙+∇aQHJB(s,a)a˙]γ Q^HJB(s,a)= _a [r(s,a)+ _sQ^HJB(s,a) s+ _aQ^HJB(s,a) a ] (9) Unlike the typical discrete-time Bellman equation, which compresses value learning into a scalar one-step regression, Eq. (9) reveals what is missing in existing offline RL methods. The value function may match 0th0^th-order Bellman targets, yet its gradient is never required to align with the variation direction of states and actions in time. This makes the learned value landscape numerically plausible but structurally under-specified, especially for later policy improvement. However, with the extended policy class introduced in Section 3, we can easily obtain the time derivative of actions through the learned policy, i.e., a˙=π˙θ(s)=∇sπθ(s)⋅s˙ a= π_θ(s)= _s _θ(s)· s. This allows us to jointly optimize the value function with policy’s higher-order information during Bellman update, which leads to the following objective: Q=argminQ(s,a,r,s′,a′)∼[ Q= _QE_(s,a,r,s ,a ) [ (πQ^(s,a)−Q⁡(s,a))2 (T_π Q(s,a)-Q(s,a) )^2 (10) −λQ(∇sQ(s,a)s˙‖∇sQ(s,a)‖‖s˙‖+∇aQ(s,a)π˙θ(s)‖∇aQ(s,a)‖‖π˙θ(s)‖)]. [rgb]0,0,1 - _Q ( _sQ(s,a) s\| _sQ(s,a)\|\| s\|+ _aQ(s,a) π_θ(s)\| _aQ(s,a)\|\| π_θ(s)\| ) ]. As s˙=s′−s s=s -s can be directly calculated from the dataset during offline learning, we can actually seamlessly transform the original value learning scheme of an existing offline RL algorithm into a higher-order supervision version by adding the above blue terms, with πT_π denoting the specific Bellman operator used in the offline RL algorithm. Moreover, to improve numerical stability and to prevent overly large gradient magnitudes of the first-order terms, we normalize the two HJB directional terms in Eq. (10), similar to existing HJB-based methods Kim et al. (2021); Lien et al. (2024). 4.2 Higher-Order Policy Learning With the Q-function learned with 1st1^st-order supervision, we can also transform the policy optimization procedure of an existing offline RL algorithm through a plug-and-play modification. This can be achieved by simply adding an additional 1st1^st-order imitation loss ℒa˙L_ a as defined in Section 3, π=arg⁡max(s,a,s′,a′)∼,a^∼πQ(s,a^)−ℛπ(s)−λa˙ℒa˙.π= π \;E_(s,a,s ,a ) , a πQ(s, a)-R_π(s) [rgb]0,0,1- _ aL_ a. (11) where we use Q(s,a^)−ℛπ(s)Q(s, a)-R_π(s) to represent the general policy optimization objective of existing offline RL algorithms, which typically optimize a policy by maximizing the Q-function with some behavior regularization term ℛπ(s)R_π(s). ℛπ(s)R_π(s) can either be an explicit policy constraint, such as the BC term ℛπ(s)=(s,a)∼‖π⁡(s)−a‖2R_π(s)=E_(s,a) \|π(s)-a\|^2 in TD3+BC (Fujimoto and Gu, 2021), or an implicit behavior regularization scheme as adopted in many algorithms Kostrikov et al. (2022); Xu et al. (2023); Wang et al. (2024); Zheng et al. (2024); Xu et al. (2026); Liang et al. (2026), which results in an advantage-weighted regression type of learning objective. Depending on the actual policy class used in the original offline RL algorithm, one can flexibly choose ℒa˙L_ a to be either ℒa˙detL_ a^det, ℒa˙stoL_ a^sto, or ℒa˙flowL_ a^flow in Section 3. We introduce a hyperparameter λa˙ _ a to balance the regularization strength of the original policy optimization objective and the first-order supervision term. Both the first-order value supervision term and the ℒa˙L_ a term in policy optimization introduce minimal modifications to the original offline RL algorithm, but as we show in the experiments, they can lead to substantial performance gains, as the policy is taught not only what to do, but how to evolve reasonably over time. 5 Experiments In this section, we empirically evaluate our proposed policy class. We first integrate it into a range of representative offline RL algorithms and benchmark the resulting variants against their original counterparts. We then conduct ablation studies to isolate the contributions of the first-order policy constraint and the first-order value consistency regularizer. 5.1 Experimental Setup Benchmarks. To comprehensively evaluate our proposed framework, we conduct experiments on two widely adopted offline RL benchmarks. We first evaluate on OGBench (Park et al., 2025a), a recent benchmark featuring high-dimensional state spaces, multimodal behaviors, and sparse rewards. We adopt its reward-based "-singletask" variants so that standard reward-maximizing offline RL algorithms can be directly applied. We further conduct experiments on D4RL (Fu et al., 2020) under a more challenging low-data setting with only 10k transitions per task, which better exposes performance gaps and OOD generalization since recent offline RL methods have largely saturated the D4RL benchmark under the full-dataset setting. The benchmark details and task descriptions are presented in Appendix E.1. Comparison methods. We integrate our proposed policy class into representative offline RL algorithms spanning three policy families (deterministic, stochastic, and flow-matching), and compare each base algorithm with its first-order-supervised variant. For deterministic policies, we adopt TD3+BC (Fujimoto and Gu, 2021) as the base algorithm. For stochastic policies, we instantiate the two strategies introduced in Section 3.2: a simplified mean supervision variant, which applies first-order supervision only to the Gaussian mean and is built upon ReBRAC (Tarasov et al., 2023) (denoted “mean sup.” in Table 2), and a standard variant based on the likelihood objective in Eq. 5 and built upon IQL (Kostrikov et al., 2022) (denoted “standard”). For flow-matching policies, we adopt IFQL, an algorithmic variant of FQL (Park et al., 2025b) that enables our first-order supervision to be directly applied to the flow policy without any additional policy-distillation step. Experimental details are provided in Appendix E.2. 5.2 Evaluation Results Table 1: Average normalized scores on 10 OGBench task categories, each aggregating 5 sub-tasks over 5 random seeds. Values after ± denote standard deviations. The ↑ [rgb]0.0195,0.6523,0.082 marks indicate improvements greater than 1.01.0, whereas ↑ [rgb]0.5,0.5,0.5 denotes minor improvements. Full per-task results are presented in Table 4. Det. Policy Sto. Policy (mean sup.) Sto. Policy (standard) Flow Policy Task Category (Avg. of 5 tasks) TD3+BC TD3+BC+ours ReBRAC ReBRAC+ours IQL IQL+ours IFQL IFQL+ours antmaze-large-navigate 83±483 ± 4 98±3(↑15)98 ± 3( [rgb]0.0195,0.6523,0.082 \!15) 81±581 ± 5 98±2(↑17)98 ± 2( [rgb]0.0195,0.6523,0.082 \!17) 53±353 ± 3 68±5(↑15)68 ± 5( [rgb]0.0195,0.6523,0.082 \!15) 28±1328 ± 13 36±14(↑8)36 ± 14( [rgb]0.0195,0.6523,0.082 \!8) antmaze-giant-navigate 42±842 ± 8 65±10(↑23)65 ± 10( [rgb]0.0195,0.6523,0.082 \!23) 26±826 ± 8 61±12(↑35)61 ± 12( [rgb]0.0195,0.6523,0.082 \!35) 4±14 ± 1 9±1(↑5)9 ± 1( [rgb]0.0195,0.6523,0.082 \!5) 3±23 ± 2 4±3(↑1)4 ± 3( [rgb]0.0195,0.6523,0.082 \!1) humanoidmaze-medium-navigate 34±834 ± 8 42±5(↑8)42 ± 5( [rgb]0.0195,0.6523,0.082 \!8) 22±822 ± 8 55±10(↑33)55 ± 10( [rgb]0.0195,0.6523,0.082 \!33) 33±233 ± 2 36±3(↑3)36 ± 3( [rgb]0.0195,0.6523,0.082 \!3) 60±1660 ± 16 64±13(↑4)64 ± 13( [rgb]0.0195,0.6523,0.082 \!4) humanoidmaze-large-navigate 2±12 ± 1 4±1(↑2)4 ± 1( [rgb]0.0195,0.6523,0.082 \!2) 2±12 ± 1 4±2(↑2)4 ± 2( [rgb]0.0195,0.6523,0.082 \!2) 2±12 ± 1 3±1(↑1)3 ± 1( [rgb]0.0195,0.6523,0.082 \!1) 11±211 ± 2 12±3(↑1)12 ± 3( [rgb]0.0195,0.6523,0.082 \!1) antsoccer-arena-navigate 13±313 ± 3 16±3(↑3)16 ± 3( [rgb]0.0195,0.6523,0.082 \!3) 0±00 ± 0 16±4(↑16)16 ± 4( [rgb]0.0195,0.6523,0.082 \!16) 8±28 ± 2 23±3(↑15)23 ± 3( [rgb]0.0195,0.6523,0.082 \!15) 33±633 ± 6 40±13(↑7)40 ± 13( [rgb]0.0195,0.6523,0.082 \!7) cube-single-play 88±588 ± 5 91±4(↑3)91 ± 4( [rgb]0.0195,0.6523,0.082 \!3) 91±291 ± 2 93±2(↑2)93 ± 2( [rgb]0.0195,0.6523,0.082 \!2) 83±383 ± 3 92±3(↑9)92 ± 3( [rgb]0.0195,0.6523,0.082 \!9) 79±279 ± 2 95±7(↑16)95 ± 7( [rgb]0.0195,0.6523,0.082 \!16) cube-double-play 7±17 ± 1 12±1(↑5)12 ± 1( [rgb]0.0195,0.6523,0.082 \!5) 12±112 ± 1 24±3(↑12)24 ± 3( [rgb]0.0195,0.6523,0.082 \!12) 7±17 ± 1 8±2(↑1)8 ± 2( [rgb]0.0195,0.6523,0.082 \!1) 14±314 ± 3 42±15(↑28)42 ± 15( [rgb]0.0195,0.6523,0.082 \!28) scene-play 44±344 ± 3 47±2(↑3)47 ± 2( [rgb]0.0195,0.6523,0.082 \!3) 41±341 ± 3 47±4(↑6)47 ± 4( [rgb]0.0195,0.6523,0.082 \!6) 28±128 ± 1 33±3(↑5)33 ± 3( [rgb]0.0195,0.6523,0.082 \!5) 30±330 ± 3 56±7(↑26)56 ± 7( [rgb]0.0195,0.6523,0.082 \!26) puzzle-3x3-play 27±327 ± 3 29±1(↑2)29 ± 1( [rgb]0.0195,0.6523,0.082 \!2) 21±121 ± 1 24±2(↑3)24 ± 2( [rgb]0.0195,0.6523,0.082 \!3) 9±19 ± 1 12±4(↑3)12 ± 4( [rgb]0.0195,0.6523,0.082 \!3) 19±119 ± 1 43±15(↑24)43 ± 15( [rgb]0.0195,0.6523,0.082 \!24) puzzle-4x4-play 6±16 ± 1 9±1(↑3)9 ± 1( [rgb]0.0195,0.6523,0.082 \!3) 14±114 ± 1 17±3(↑3)17 ± 3( [rgb]0.0195,0.6523,0.082 \!3) 7±17 ± 1 13±2(↑6)13 ± 2( [rgb]0.0195,0.6523,0.082 \!6) 25±525 ± 5 40±12(↑15)40 ± 12( [rgb]0.0195,0.6523,0.082 \!15) Average Score 35 41(↑6)( [rgb]0.0195,0.6523,0.082 \!6) 31 44(↑13)( [rgb]0.0195,0.6523,0.082 \!13) 23 30(↑7)( [rgb]0.0195,0.6523,0.082 \!7) 30 43(↑13)( [rgb]0.0195,0.6523,0.082 \!13) Table 2: Average normalized scores on D4RL MuJoCo locomotion tasks with reduced-size (10k) datasets. Standard deviations are reported after the ± sign. The ↑ [rgb]0.0195,0.6523,0.082 symbol indicates a performance increase greater than 1.01.0, whereas ↑ [rgb]0.5,0.5,0.5 denotes minor improvements. All evaluation results are averaged over 5 random seeds. Det. Policy Sto. Policy (mean sup.) Sto. Policy (standard) Flow Policy Task Category TD3+BC TD3+BC+ours ReBRAC ReBRAC+ours IQL IQL+ours IFQL IFQL+ours Hopper-m 40.1±18.640.1 ± 18.6 44.1±12.7(↑4.0)44.1 ± 12.7( [rgb]0.0195,0.6523,0.082 \!4.0) 9.8±3.19.8 ± 3.1 32.4±11.0(↑22.6)32.4 ± 11.0( [rgb]0.0195,0.6523,0.082 \!22.6) 47.7±1.047.7 ± 1.0 48.5±2.5(↑0.8)48.5 ± 2.5( [rgb]0.5,0.5,0.5 \!0.8) 50.8±2.350.8 ± 2.3 51.3±2.9(↑0.5)51.3 ± 2.9( [rgb]0.5,0.5,0.5 \!0.5) Hopper-mr 7.3±6.17.3 ± 6.1 22.3±3.3(↑15.0)22.3 ± 3.3( [rgb]0.0195,0.6523,0.082 \!15.0) 19.0±1.019.0 ± 1.0 24.9±2.8(↑5.9)24.9 ± 2.8( [rgb]0.0195,0.6523,0.082 \!5.9) 19.3±3.019.3 ± 3.0 19.9±2.0(↑0.6)19.9 ± 2.0( [rgb]0.5,0.5,0.5 \!0.6) 23.1±1.423.1 ± 1.4 28.2±0.4(↑5.1)28.2 ± 0.4( [rgb]0.0195,0.6523,0.082 \!5.1) Hopper-me 17.8±7.917.8 ± 7.9 42.3±10.8(↑24.5)42.3 ± 10.8( [rgb]0.0195,0.6523,0.082 \!24.5) 36.2±3.136.2 ± 3.1 43.0±3.3(↑6.8)43.0 ± 3.3( [rgb]0.0195,0.6523,0.082 \!6.8) 34.3±8.734.3 ± 8.7 42.3±3.3(↑8.0)42.3 ± 3.3( [rgb]0.0195,0.6523,0.082 \!8.0) 45.7±1.945.7 ± 1.9 47.9±1.7(↑2.2)47.9 ± 1.7( [rgb]0.0195,0.6523,0.082 \!2.2) Hopper-e 23.2±18.223.2 ± 18.2 93.5±13.6(↑70.3)93.5 ± 13.6( [rgb]0.0195,0.6523,0.082 \!70.3) 59.2±3.559.2 ± 3.5 97.2±5.9(↑38.0)97.2 ± 5.9( [rgb]0.0195,0.6523,0.082 \!38.0) 55.7±22.555.7 ± 22.5 99.5±3.2(↑43.8)99.5 ± 3.2( [rgb]0.0195,0.6523,0.082 \!43.8) 78.5±13.678.5 ± 13.6 96.9±4.9(↑18.4)96.9 ± 4.9( [rgb]0.0195,0.6523,0.082 \!18.4) Halfcheetah-m 16.4±10.216.4 ± 10.2 31.5±2.4(↑15.1)31.5 ± 2.4( [rgb]0.0195,0.6523,0.082 \!15.1) 37.1±1.337.1 ± 1.3 38.9±0.5(↑1.8)38.9 ± 0.5( [rgb]0.0195,0.6523,0.082 \!1.8) 38.2±1.438.2 ± 1.4 39.1±0.1(↑0.9)39.1 ± 0.1( [rgb]0.5,0.5,0.5 \!0.9) 38.9±1.438.9 ± 1.4 39.5±1.5(↑0.6)39.5 ± 1.5( [rgb]0.5,0.5,0.5 \!0.6) Halfcheetah-mr 17.9±9.517.9 ± 9.5 21.2±7.1(↑3.3)21.2 ± 7.1( [rgb]0.0195,0.6523,0.082 \!3.3) 13.0±5.713.0 ± 5.7 15.1±4.1(↑2.1)15.1 ± 4.1( [rgb]0.0195,0.6523,0.082 \!2.1) 28.2±1.328.2 ± 1.3 29.5±0.6(↑1.3)29.5 ± 0.6( [rgb]0.0195,0.6523,0.082 \!1.3) 29.6±1.029.6 ± 1.0 31.1±1.2(↑1.5)31.1 ± 1.2( [rgb]0.0195,0.6523,0.082 \!1.5) Halfcheetah-me 15.4±10.715.4 ± 10.7 35.6±2.4(↑20.2)35.6 ± 2.4( [rgb]0.0195,0.6523,0.082 \!20.2) 12.2±6.712.2 ± 6.7 19.7±1.7(↑7.5)19.7 ± 1.7( [rgb]0.0195,0.6523,0.082 \!7.5) 34.4±2.234.4 ± 2.2 36.8±1.5(↑2.4)36.8 ± 1.5( [rgb]0.0195,0.6523,0.082 \!2.4) 40.2±0.740.2 ± 0.7 40.4±1.3(↑0.2)40.4 ± 1.3( [rgb]0.5,0.5,0.5 \!0.2) Halfcheetah-e 1.7±3.31.7 ± 3.3 3.9±2.1(↑2.2)3.9 ± 2.1( [rgb]0.0195,0.6523,0.082 \!2.2) 2.3±0.92.3 ± 0.9 4.0±2.3(↑1.7)4.0 ± 2.3( [rgb]0.0195,0.6523,0.082 \!1.7) 1.2±1.01.2 ± 1.0 6.7±2.6(↑5.5)6.7 ± 2.6( [rgb]0.0195,0.6523,0.082 \!5.5) 18.8±7.818.8 ± 7.8 35.9±6.2(↑17.1)35.9 ± 6.2( [rgb]0.0195,0.6523,0.082 \!17.1) Walker2d-m 7.4±13.17.4 ± 13.1 41.1±10.8(↑33.7)41.1 ± 10.8( [rgb]0.0195,0.6523,0.082 \!33.7) 21.2±4.821.2 ± 4.8 56.9±4.3(↑35.7)56.9 ± 4.3( [rgb]0.0195,0.6523,0.082 \!35.7) 34.6±9.534.6 ± 9.5 45.0±4.8(↑10.4)45.0 ± 4.8( [rgb]0.0195,0.6523,0.082 \!10.4) 50.6±6.850.6 ± 6.8 56.7±4.7(↑6.1)56.7 ± 4.7( [rgb]0.0195,0.6523,0.082 \!6.1) Walker2d-mr 5.7±5.85.7 ± 5.8 23.2±5.7(↑17.5)23.2 ± 5.7( [rgb]0.0195,0.6523,0.082 \!17.5) 10.0±2.210.0 ± 2.2 22.8±5.5(↑12.8)22.8 ± 5.5( [rgb]0.0195,0.6523,0.082 \!12.8) 10.7±11.910.7 ± 11.9 17.6±4.0(↑6.9)17.6 ± 4.0( [rgb]0.0195,0.6523,0.082 \!6.9) 17.5±5.417.5 ± 5.4 29.0±2.8(↑11.5)29.0 ± 2.8( [rgb]0.0195,0.6523,0.082 \!11.5) Walker2d-me 7.9±9.17.9 ± 9.1 37.8±12.3(↑29.9)37.8 ± 12.3( [rgb]0.0195,0.6523,0.082 \!29.9) 17.6±8.617.6 ± 8.6 58.9±11.8(↑41.3)58.9 ± 11.8( [rgb]0.0195,0.6523,0.082 \!41.3) 29.1±16.929.1 ± 16.9 41.2±9.2(↑12.1)41.2 ± 9.2( [rgb]0.0195,0.6523,0.082 \!12.1) 38.8±3.738.8 ± 3.7 57.6±10.4(↑18.8)57.6 ± 10.4( [rgb]0.0195,0.6523,0.082 \!18.8) Walker2d-e 2.7±3.12.7 ± 3.1 82.3±10.8(↑79.5)82.3 ± 10.8( [rgb]0.0195,0.6523,0.082 \!79.5) 23.6±9.423.6 ± 9.4 83.9±6.8(↑60.3)83.9 ± 6.8( [rgb]0.0195,0.6523,0.082 \!60.3) 43.1±6.043.1 ± 6.0 84.8±13.9(↑41.7)84.8 ± 13.9( [rgb]0.0195,0.6523,0.082 \!41.7) 95.2±11.895.2 ± 11.8 104.9±3.7(↑9.7)104.9 ± 3.7( [rgb]0.0195,0.6523,0.082 \!9.7) Average Score 13.613.6 39.9(↑26.3)39.9( [rgb]0.0195,0.6523,0.082 \!26.3) 21.821.8 41.5(↑19.7)41.5( [rgb]0.0195,0.6523,0.082 \!19.7) 31.431.4 42.6(↑10.2)42.6( [rgb]0.0195,0.6523,0.082 \!10.2) 44.044.0 51.6(↑7.6)51.6( [rgb]0.0195,0.6523,0.082 \!7.6) Evaluation results on OGBench tasks. Table 2 reports the average normalized scores on 50 diverse OGBench tasks, with complete per-task results provided in Appendix B.1. Our first-order action supervision yields consistent performance gains across all three policy paradigms. On control-sensitive navigation tasks, these gains underline the importance of temporally consistent actions, since small action fluctuations can accumulate over time and easily lead to collisions or failed navigation. For instance, applying our scheme to ReBRAC raises the score from 26 to 61 on antmaze-giant-navigate and from 22 to 55 on humanoidmaze-medium-navigate, both revealing over 100% relative improvement. The performance gains are even more pronounced beyond navigation. On object-centric play tasks, applying our proposed scheme to IFQL improves cube-double-play from 14 to 42 (a 200% relative improvement) and scene-play from 30 to 56 (an 87% relative improvement). In particular, on several challenging sub-tasks where the IFQL baseline completely fails with a score of exactly 0, equipping it with our scheme lifts the score to clearly non-trivial levels, e.g., from 0 to 88 on scene-play-singletask-task2-v0, expanding the set of tasks on which offline policy learning is feasible at all (see Table 4 in Appendix B.1 for full per-task results). Improved OOD generalization under small-dataset settings. Our first-order action supervision is particularly beneficial in small-dataset regimes, where standard offline RL baselines often struggle to recover the expert’s robust control behavior from sparse state-action samples, even though such samples still encode rich local variations along expert trajectories. Table 2 reports the D4RL locomotion results on 10k datasets. By supervising the policy’s first-order action derivative, our scheme enables the local dynamics and control structure of the expert behavior to be effectively recovered from an extremely limited dataset, leading to substantial performance gains. For instance, on hopper-expert-v2, IQL improves from 55.755.7 to 99.599.5, nearly recovering its full-dataset performance. More notably, on walker2d-expert-v2, TD3+BC collapses to a score of only 2.72.7, indicating severe extrapolation failure under such a tiny-data regime; equipping it with our scheme recovers the score to 82.382.3, an absolute gain of 79.579.5 points. In addition to the score improvements, our scheme also yields noticeably smaller standard deviations across most tasks, indicating more stable and reproducible policy learning under small-dataset settings. Additional limited-data imitation learning results, as well as full-data D4RL results, are provided in Appendix B.1. Benefits of utilizing first-order supervision. Figure 2 provides a mechanistic view of how our first-order action supervision improves performance. With only 10k transitions, deterministic BC produces a dispersed distribution of ‖a˙‖|| a||, reflecting erratic action changes during execution. In contrast, our scheme yields a compact distribution concentrated around small ‖a˙‖|| a||, indicating smoother and temporally consistent action evolution. This pattern is even more pronounced for TD3+BC in Figure 3, where its plain MSE penalty leaves the policy vulnerable to action jittering, whereas our scheme directly aligns action changes with the local expert dynamics. Overall, by constraining how actions evolve along trajectories, first-order supervision yields noticeably smoother and more robust policies. We conduct additional analysis results in Appendix B.1. Figure 2: Comparison of vanilla deterministic BC and our 1st1^st-order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation ‖a˙‖=‖a′−a‖|| a||=||a -a|| (a is joint torque) are aggregated from 10 trajectories. Figure 3: Comparison of TD3+BC and our 1st1^st-order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation ‖a˙‖=‖a′−a‖|| a||=||a -a|| (a is joint torque) are aggregated from 10 trajectories. 5.3 Ablation experiments Table 3: Comparison with action-smoothness approaches and explicit first-order action prediction on D4RL MuJoCo locomotion tasks with reduced-size (10k) datasets. All methods use TD3+BC as the shared backbone. Arrows show score differences relative to TD3+BC on the same task: ↑ [rgb]0.0195,0.6523,0.082 indicates a gain of at least 1.01.0, ↑ [rgb]0.5,0.5,0.5 a smaller positive gain, and ↓ [rgb]1,0,0 a decrease. No arrow indicates no change. Task TD3+BC +CAPS +L2C2 +LipsNet +Extra a˙ a head TD3+BC+Ours Hopper-m 40.1±18.640.1 ± 18.6 42.6±5.1(↑2.5)42.6 ± 5.1( [rgb]0.0195,0.6523,0.082 \!2.5) 42.7±7.5(↑2.6)42.7 ± 7.5( [rgb]0.0195,0.6523,0.082 \!2.6) 43.2±3.6(↑3.1)43.2 ± 3.6( [rgb]0.0195,0.6523,0.082 \!3.1) 41.5±7.8(↑1.4)41.5 ± 7.8( [rgb]0.0195,0.6523,0.082 \!1.4) 44.1±12.7(↑4.0)44.1 ± 12.7( [rgb]0.0195,0.6523,0.082 \!4.0) Hopper-me 17.8±7.917.8 ± 7.9 39.6±8.2(↑21.8)39.6 ± 8.2( [rgb]0.0195,0.6523,0.082 \!21.8) 39.8±8.3(↑22.0)39.8 ± 8.3( [rgb]0.0195,0.6523,0.082 \!22.0) 32.8±18.5(↑15.0)32.8 ± 18.5( [rgb]0.0195,0.6523,0.082 \!15.0) 26.6±8.6(↑8.8)26.6 ± 8.6( [rgb]0.0195,0.6523,0.082 \!8.8) 42.3±10.8(↑24.5)42.3 ± 10.8( [rgb]0.0195,0.6523,0.082 \!24.5) Hopper-e 23.2±18.223.2 ± 18.2 57.1±29.7(↑33.9)57.1 ± 29.7( [rgb]0.0195,0.6523,0.082 \!33.9) 55.2±34.5(↑32.0)55.2 ± 34.5( [rgb]0.0195,0.6523,0.082 \!32.0) 16.9±2.8(↓6.3)16.9 ± 2.8( [rgb]1,0,0 \!6.3) 30.6±15.1(↑7.4)30.6 ± 15.1( [rgb]0.0195,0.6523,0.082 \!7.4) 93.5±13.6(↑70.3)93.5 ± 13.6( [rgb]0.0195,0.6523,0.082 \!70.3) HalfCheetah-m 16.4±10.216.4 ± 10.2 29.6±5.6(↑13.2)29.6 ± 5.6( [rgb]0.0195,0.6523,0.082 \!13.2) 30.3±7.1(↑13.9)30.3 ± 7.1( [rgb]0.0195,0.6523,0.082 \!13.9) 6.6±3.0(↓9.8)6.6 ± 3.0( [rgb]1,0,0 \!9.8) 28.1±4.5(↑11.7)28.1 ± 4.5( [rgb]0.0195,0.6523,0.082 \!11.7) 31.5±2.4(↑15.1)31.5 ± 2.4( [rgb]0.0195,0.6523,0.082 \!15.1) HalfCheetah-me 15.4±10.715.4 ± 10.7 19.8±3.4(↑4.4)19.8 ± 3.4( [rgb]0.0195,0.6523,0.082 \!4.4) 28.6±3.8(↑13.2)28.6 ± 3.8( [rgb]0.0195,0.6523,0.082 \!13.2) 3.3±2.2(↓12.1)3.3 ± 2.2( [rgb]1,0,0 \!12.1) 20.6±3.5(↑5.2)20.6 ± 3.5( [rgb]0.0195,0.6523,0.082 \!5.2) 35.6±2.4(↑20.2)35.6 ± 2.4( [rgb]0.0195,0.6523,0.082 \!20.2) HalfCheetah-e 1.7±3.31.7 ± 3.3 1.7±1.61.7 ± 1.6 2.4±1.9(↑0.7)2.4 ± 1.9( [rgb]0.5,0.5,0.5 \!0.7) 0.8±0.6(↓0.9)0.8 ± 0.6( [rgb]1,0,0 \!0.9) 0.2±1.8(↓1.5)0.2 ± 1.8( [rgb]1,0,0 \!1.5) 3.9±2.1(↑2.2)3.9 ± 2.1( [rgb]0.0195,0.6523,0.082 \!2.2) Walker2d-m 7.4±13.17.4 ± 13.1 24.5±3.6(↑17.1)24.5 ± 3.6( [rgb]0.0195,0.6523,0.082 \!17.1) 20.4±4.1(↑13.0)20.4 ± 4.1( [rgb]0.0195,0.6523,0.082 \!13.0) 8.9±2.8(↑1.5)8.9 ± 2.8( [rgb]0.0195,0.6523,0.082 \!1.5) 8.6±6.5(↑1.2)8.6 ± 6.5( [rgb]0.0195,0.6523,0.082 \!1.2) 41.1±10.8(↑33.7)41.1 ± 10.8( [rgb]0.0195,0.6523,0.082 \!33.7) Walker2d-me 7.9±9.17.9 ± 9.1 17.8±10.1(↑9.9)17.8 ± 10.1( [rgb]0.0195,0.6523,0.082 \!9.9) 23.3±9.8(↑15.4)23.3 ± 9.8( [rgb]0.0195,0.6523,0.082 \!15.4) 5.9±3.6(↓2.0)5.9 ± 3.6( [rgb]1,0,0 \!2.0) 8.3±6.1(↑0.4)8.3 ± 6.1( [rgb]0.5,0.5,0.5 \!0.4) 37.8±12.3(↑29.9)37.8 ± 12.3( [rgb]0.0195,0.6523,0.082 \!29.9) Walker2d-e 2.7±3.12.7 ± 3.1 3.9±2.1(↑1.2)3.9 ± 2.1( [rgb]0.0195,0.6523,0.082 \!1.2) 7.4±3.6(↑4.7)7.4 ± 3.6( [rgb]0.0195,0.6523,0.082 \!4.7) 13.8±6.8(↑11.1)13.8 ± 6.8( [rgb]0.0195,0.6523,0.082 \!11.1) 3.9±1.6(↑1.2)3.9 ± 1.6( [rgb]0.0195,0.6523,0.082 \!1.2) 82.3±10.8(↑79.5)82.3 ± 10.8( [rgb]0.0195,0.6523,0.082 \!79.5) Average Score 14.714.7 26.3(↑11.6)26.3( [rgb]0.0195,0.6523,0.082 \!11.6) 27.8(↑13.1)27.8( [rgb]0.0195,0.6523,0.082 \!13.1) 14.714.7 18.7(↑4.0)18.7( [rgb]0.0195,0.6523,0.082 \!4.0) 45.8(↑31.1)45.8( [rgb]0.0195,0.6523,0.082 \!31.1) Comparing with alternative action-consistency approaches. To further benchmark our method, we compare against two categories of representative alternatives, using TD3+BC as the shared backbone. The first category enforces action smoothness, including the penalty-based methods CAPS (Mysore et al., 2021) and L2C2 (Kobayashi, 2022), which encourage local action consistency on perturbed or interpolated states, and the architectural method LipsNet (Song et al., 2023), which explicitly bounds the network’s Lipschitz constant. The second category explicitly predicts first-order actions, for which we construct an auxiliary-head variant of TD3+BC, denoted as +Extra a˙ a head, where the actor is equipped with a dedicated prediction head, i.e., learning a policy π⁡(s)=(a,a˙)π(s)=(a, a), and the predicted a˙ a is also utilized during inference. All baselines follow their original configurations or exactly the same training configurations as our method. For action-smoothness approaches, as reported in Table 3 (+CAPS/+L2C2/+LipsNet columns), our method achieves consistent improvements across all tasks. Penalty-based methods were originally designed for online settings; in offline RL, enforcing smoothness on synthetic, unobserved states risks OOD extrapolation errors, and this degradation is particularly pronounced on high-quality (-e) datasets, whose narrow and near-deterministic trajectories conflict with isotropic perturbations or uniform interpolations. Meanwhile, strictly constraining the network architecture (LipsNet) limits model expressivity, making it difficult to fit offline data from limited samples. For explicit first-order action prediction, the auxiliary-head variant brings only marginal improvements over the backbone and substantially underperforms our method (+Extra a˙ a head column). We attribute this gap to two factors. First, predicting two distinct action outputs converts policy optimization into a multi-task learning problem, which introduces conflicting gradient directions between the zeroth-order and first-order action objectives and requires extra hyperparameter tuning to balance their trade-off. Second, the dedicated head doubles the output dimension and requires structural modification of the base network, losing compatibility with off-the-shelf policy architectures. In contrast, our method requires zero architectural changes, acting as a lightweight, plug-and-play module that preserves temporal consistency without sacrificing network capacity. Figure 4: Ablation study on value regularization weight λQ _Q and 1st1^st-order action supervision weight λa˙ _ a of IFQL. Effectiveness of value and policy constraints. Figure 4 isolates the respective contributions of the two components in our IFQL instantiation on OGBench by sweeping one weight while strictly setting the other to zero. The weight λQ _Q controls the first-order value-consistency regularizer; sweeping it over 0,0.5,1\0,0.5,1\ reveals steady performance gains, indicating that aligning the value landscape with the local state-action flow provides a highly beneficial training signal. Similarly, increasing the first-order action supervision weight λa˙ _ a from the same grid 0,0.5,1\0,0.5,1\ consistently improves performance over standard IFQL. This confirms that first-order action supervision offers crucial guidance beyond mere zeroth-order action fitting, even when the value-side regularizer is completely disabled. Additional ablations are deferred to Appendix B.2. 6 Conclusion In this paper, we identified the neglect of higher-order action dynamics as a fundamental source of control instability and poor robustness in contemporary offline RL and IL methods. To bridge this gap, we proposed an elegant, plug-and-play learning scheme that enables simultaneous zeroth- and first-order action supervision for any off-the-shelf policy model, including deterministic, stochastic, and generative flow policies, without requiring any structural modifications. Supported by formal theoretical guarantees, our enhanced policy class also integrates seamlessly into the continuous-time HJB framework, providing additional first-order regularization on both value estimation and policy optimization. Extensive empirical evaluations confirm that our scheme yields substantial improvements in control smoothness, robustness, and out-of-distribution generalization, particularly under challenging small-dataset settings, making it a promising tool for real-world control problems. Acknowledgment This work was supported by the Wuxi Research Institute of Applied Technologies at Tsinghua University under the Grant 20242001120, Xiongan AI Institute, and Horizon Robotics. References Bhasin et al. (2013) S. Bhasin, R. Kamalapurkar, M. Johnson, K. G. Vamvoudakis, F. L. Lewis, and W. E. Dixon A novel actor–critic–identifier architecture for approximate optimal control of uncertain nonlinear systems. Automatica 49 (1), p. 82–92. Cited by: §2. Cai et al. (2020) P. Cai, X. Mei, L. Tai, Y. Sun, and M. Liu High-speed autonomous drifting with deep reinforcement learning. IEEE Robotics and Automation Letters 5 (2), p. 1247–1254. Cited by: §1. Chen et al. (2021) C. Chen, H. Tang, J. Hao, W. Liu, and Z. Meng Addressing action oscillations through learning policy inertia. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, p. 7020–7027. Cited by: §1. Cheng et al. (2026) P. Cheng, Z. Wu, J. Li, Z. He, H. Xu, W. Sun, Y. Lin, Y. Liu, and X. Zhan Sample efficient offline rl via t-symmetry enforced latent state-stitching. In International Conference on Learning Representations, Vol. 2026, p. 139752–139781. Cited by: §1. Cheng et al. (2023) P. Cheng, X. Zhan, Z. Wu, W. Zhang, Y. Lin, H. Wang, L. Jiang, et al. Look beneath the surface: exploiting fundamental symmetry for sample-efficient offline rl. Advances in Neural Information Processing Systems 36. Cited by: §1, §2. Chi et al. (2023) C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song Diffusion policy: visuomotor policy learning via action diffusion. In Robotics: Science and Systems, Cited by: §1. Christmann et al. (2024) G. Christmann, Y. Luo, H. Mandala, and W. Chen Benchmarking smoothness and reducing high-frequency oscillations in continuous control policies. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 627–634. Cited by: §2. de Boor (2001) C. de Boor A practical guide to splines. Applied Mathematical Sciences, Springer, New York, NY. Cited by: Appendix A. de La Presilla et al. (2023) R. de La Presilla, S. Wandel, M. Stammler, M. Grebe, G. Poll, and S. Glavatskih Oscillating rolling element bearings: a review of tribotesting and analysis approaches. Tribology International 188, p. 108805. Cited by: §1. Doya (2000) K. Doya Reinforcement learning in continuous time and space. Neural Computation 12 (1), p. 219–245. Cited by: §2. Dulac-Arnold et al. (2021) G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning 110 (9), p. 2419–2468. Cited by: §1, §1, §2. Emmrich (1999) E. Emmrich Discrete versions of gronwall’s lemma and their application to the numerical analysis of parabolic problems. Preprint Technische Universität Berlin, Fachbereich Mathematik. Cited by: Lemma 2, Lemma 2. Fu et al. (2020) J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine D4rl: datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. Cited by: Appendix E, §1, §5.1. Fujimoto and Gu (2021) S. Fujimoto and S. S. Gu A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems 34. Cited by: §B.1, §E.2, §E.2, §2, §4.2, §5.1. Haarnoja et al. (2018) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, p. 1861–1870. Cited by: §B.1. Hafner et al. (2020) D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi Dream to control: learning behaviors by latent imagination. In International Conference on Learning Representations, Cited by: Appendix A. Hansen-Estruch et al. (2023) P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine Idql: implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573. Cited by: §2. Hendrycks and Gimpel (2016) D. Hendrycks and K. Gimpel Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. Cited by: Table 11. Janner et al. (2022) M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991. Cited by: §2. Jia and Zhou (2022) Y. Jia and X. Y. Zhou Policy evaluation and temporal-difference learning in continuous time and space: a martingale approach. Journal of Machine Learning Research 23 (154), p. 1–55. Cited by: §2. Kalashnikov et al. (2018) D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, p. 651–673. Cited by: §1. Kendall et al. (2019) A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J. Allen, V. Lam, A. Bewley, and A. Shah Learning to drive in a day. In 2019 international conference on robotics and automation (ICRA), p. 8248–8254. Cited by: §1. Kim et al. (2021) J. Kim, J. Shin, and I. Yang Hamilton–jacobi deep q-learning for deterministic continuous-time systems with lipschitz continuous controls. Journal of Machine Learning Research 22 (206), p. 1–34. Cited by: §2, §4.1, §4.1. Kingma and Ba (2014) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: Table 11. Kirk (2004) D. E. Kirk Optimal control theory: an introduction. Courier Corporation. Cited by: §1. Kobayashi (2022) T. Kobayashi L2c2: locally lipschitz continuous constraint towards stable and smooth reinforcement learning. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 4032–4039. Cited by: §1, §2, §5.3. Kostrikov et al. (2022) I. Kostrikov, A. Nair, and S. Levine Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, Cited by: §E.2, §E.2, §2, §4.2, §5.1. Kumar et al. (2019) A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, p. 11761–11771. Cited by: §2. Kumar et al. (2020) A. Kumar, A. Zhou, G. Tucker, and S. Levine Conservative q-learning for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), Cited by: §2. Lee et al. (2024) I. Lee, H. Cao, C. Dao, Y. Chen, and I. Wu Gradient-based regularization for action smoothness in robotic control with reinforcement learning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 603–610. Cited by: §1. Lee and Sutton (2021) J. Lee and R. S. Sutton Policy iterations for reinforcement learning problems in continuous time and space—fundamental theory and methods. Automatica 126, p. 109421. Cited by: §2. Levine et al. (2020) S. Levine, A. Kumar, G. Tucker, and J. Fu Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643. Cited by: §2. Li et al. (2024) G. Li, Y. Shan, Z. Zhu, T. Long, and W. Zhang Diffstitch: boosting offline reinforcement learning with diffusion-based trajectory stitching. In Forty-first International Conference on Machine Learning, p. 28597 – 28609. Cited by: §2. Li et al. (2023a) J. Li, X. Hu, H. Xu, J. Liu, X. Zhan, Q. Jia, and Y. Zhang Mind the gap: offline policy optimization for imperfect rewards. In The Eleventh International Conference on Learning Representations, Cited by: §2. Li et al. (2023b) J. Li, X. Zhan, H. Xu, X. Zhu, J. Liu, and Y. Zhang When data geometry meets deep function: generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, Cited by: §2. Liang et al. (2026) R. Liang, Y. Zheng, K. Zheng, T. Tan, J. Li, L. Mao, Z. Wang, G. Chen, H. Ye, J. Liu, J. Wang, and X. Zhan Dichotomous diffusion policy optimization. In International Conference on Learning Representations, Cited by: §2, §4.2. Lien et al. (2024) Y. Lien, P. Hsieh, T. Li, and Y. Wang Enhancing value function estimation through first-order state-action dynamics in offline reinforcement learning. In Forty-first International Conference on Machine Learning, Cited by: §2, §4.1. Lipman et al. (2023) Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. In International Conference on Learning Representations, Cited by: §3.2, §3.2. Liu et al. (2023) X. Liu, C. Gong, and Q. Liu Flow straight and fast: learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, Cited by: §3.2, §3.2. Lyu et al. (2022) J. Lyu, X. Ma, X. Li, and Z. Lu Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems 35, p. 1711–1724. Cited by: §2. Mao et al. (2024a) L. Mao, H. Xu, X. Zhan, W. Zhang, and A. Zhang Diffusion-dice: in-sample diffusion guidance for offline reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §2. Mao et al. (2024b) L. Mao, H. Xu, W. Zhang, and X. Zhan ODICE: revealing the mystery of distribution correction estimation via orthogonal-gradient update. In The Twelfth International Conference on Learning Representations, Cited by: §2. Mnih et al. (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature 518 (7540), p. 529–533. Cited by: §1. Munos (2006) R. Munos Policy gradient in continuous time. Journal of Machine Learning Research 7, p. 771–791. Cited by: §2. Mysore et al. (2021) S. Mysore, B. Mabsout, R. Mancuso, and K. Saenko Regularizing action policies for smooth control with reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), p. 1810–1816. Cited by: §1, §1, §2, §5.3. Niu et al. (2022) H. Niu, S. Sharma, Y. Qiu, M. Li, G. Zhou, J. HU, and X. Zhan When to trust your simulator: dynamics-aware hybrid offline-and-online reinforcement learning. In Advances in Neural Information Processing Systems, Cited by: §2. Oksendal (2013) B. Oksendal Stochastic differential equations: an introduction with applications. Springer Science & Business Media. Cited by: §C.1, §3.2. Park et al. (2025a) S. Park, K. Frans, B. Eysenbach, and S. Levine OGBench: benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), Cited by: Appendix E, §1, §5.1. Park et al. (2025b) S. Park, Q. Li, and S. Levine Flow q-learning. In Forty-second International Conference on Machine Learning, Cited by: §B.1, §E.2, §E.2, §E.2, §2, §3.2, §5.1. Pfrommer et al. (2022) D. Pfrommer, T. Zhang, S. Tu, and N. Matni Tasil: taylor series imitation learning. Advances in Neural Information Processing Systems 35, p. 20162–20174. Cited by: §2. Savitzky and Golay (1964) A. Savitzky and M. J. E. Golay Smoothing and differentiation of data by simplified least squares procedures. Analytical Chemistry 36 (8), p. 1627–1639. Cited by: Appendix A. Shen et al. (2020) Q. Shen, Y. Li, H. Jiang, Z. Wang, and T. Zhao Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, p. 8707–8718. Cited by: §1, §2. Shi et al. (2019) G. Shi, X. Shi, M. O’Connell, R. Yu, K. Azizzadenesheli, A. Anandkumar, Y. Yue, and S. Chung Neural lander: stable drone landing control using learned dynamics. In 2019 international conference on robotics and automation (icra), p. 9784–9790. Cited by: §1. Silver et al. (2017) D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. nature 550 (7676), p. 354–359. Cited by: §1. Song et al. (2025) X. Song, L. Chen, T. Liu, W. Wang, Y. Wang, S. Qin, Y. Ma, J. Duan, and S. E. Li Lipsnet++: unifying filter and controller into a policy network. In Forty-second International Conference on Machine Learning, Cited by: §1, §1, §2. Song et al. (2023) X. Song, J. Duan, W. Wang, S. E. Li, C. Chen, B. Cheng, B. Zhang, J. Wei, and X. S. Wang LipsNet: a smooth and robust neural network with adaptive lipschitz constant for high accuracy optimal control. In International Conference on Machine Learning, p. 32253–32272. Cited by: §1, §1, §1, §2, §5.3. Tarasov et al. (2023) D. Tarasov, V. Kurenkov, A. Nikulin, and S. Kolesnikov Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems 36, p. 11592–11620. Cited by: §E.2, §E.2, §5.1. Uhlenbeck and Ornstein (1930) G. E. Uhlenbeck and L. S. Ornstein On the theory of the brownian motion. Physical review 36 (5), p. 823. Cited by: §C.1, §3.2. Vamvoudakis and Lewis (2010) K. G. Vamvoudakis and F. L. Lewis Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46 (5), p. 878–888. Cited by: §2. Vamvoudakis et al. (2014) K. G. Vamvoudakis, D. Vrabie, and F. L. Lewis Online adaptive algorithm for optimal control with integral reinforcement learning. International Journal of Robust and Nonlinear Control 24 (17), p. 2686–2710. Cited by: §2. Wang et al. (2020) H. Wang, T. Zariphopoulou, and X. Y. Zhou Reinforcement learning in continuous time and space: a stochastic control approach. Journal of Machine Learning Research 21 (198), p. 1–34. Cited by: §2. Wang et al. (2023) X. Wang, H. Xu, Y. Zheng, and X. Zhan Offline multi-agent reinforcement learning with implicit global-to-local value regularization. In Advances in Neural Information Processing Systems, Cited by: §2. Wang et al. (2024) X. Wang, H. Xu, Y. Zheng, and X. Zhan Offline multi-agent reinforcement learning with implicit global-to-local value regularization. Advances in Neural Information Processing Systems 36. Cited by: §4.2. Wang et al. (2022) Z. Wang, J. J. Hunt, and M. Zhou Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193. Cited by: §2. Wasala et al. (2020) A. Wasala, D. Byrne, P. Miesbauer, J. O’Hanlon, P. Heraty, and P. Barry Trajectory based lateral control: a reinforcement learning case study. Engineering Applications of Artificial Intelligence 94, p. 103799. Cited by: §1. Wu et al. (2019) Y. Wu, G. Tucker, and O. Nachum Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361. Cited by: §2. Xu et al. (2023) H. Xu, L. Jiang, J. Li, Z. Yang, Z. Wang, V. W. K. Chan, and X. Zhan Offline rl with no ood actions: in-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, Cited by: §2, §4.2. Xu et al. (2022a) H. Xu, L. Jiang, J. Li, and X. Zhan A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, Cited by: §2. Xu et al. (2026) H. Xu, L. Mao, H. Jin, W. Zhang, X. Zhan, and A. Zhang Uni-rl: unifying online and offline rl via implicit value regularization. Advances in Neural Information Processing Systems 38, p. 133162–133187. Cited by: §2, §4.2. Xu et al. (2022b) H. Xu, X. Zhan, and X. Zhu Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: §2. Yu et al. (2021) H. Yu, W. Xu, and H. Zhang Taac: temporally abstract actor-critic for continuous control. Advances in neural information processing systems 34, p. 29021–29033. Cited by: §1. Yu et al. (2020) T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn Gradient surgery for multi-task learning. Advances in neural information processing systems 33, p. 5824–5836. Cited by: §1. Zhan et al. (2022) X. Zhan, H. Xu, Y. Zhang, X. Zhu, H. Yin, and Y. Zheng Deepthermal: combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, p. 4680–4688. Cited by: §1. Zhan et al. (2025) X. Zhan, X. Zhu, P. Cheng, X. Hu, Z. He, H. Geng, J. Leng, H. Zheng, C. Liu, T. Hong, et al. Data center cooling system optimization using offline reinforcement learning. In International Conference on Learning Representations, Vol. 2025, p. 47735–47757. Cited by: §1. Zhang et al. (2024) Y. Zhang, B. Nie, and Y. Gao Robust locomotion policy with adaptive lipschitz constraint for legged robots. IEEE Robotics and Automation Letters 10 (1), p. 272–279. Cited by: §1. Zhao et al. (2022) Z. Zhao, S. Zuo, T. Zhao, and Y. Zhao Adversarially regularized policy learning guided by trajectory optimization. In Learning for Dynamics and Control Conference, p. 844–857. Cited by: §1, §2. Zheng et al. (2025a) K. Zheng, L. Teyssier, Y. Zheng, Y. Luo, and X. Zhan Towards robust zero-shot reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §2. Zheng et al. (2026) K. Zheng, X. Zhan, and X. Yan Horizon adaptive offline policy learning via value stitching. In The Fortieth Annual Conference on Neural Information Processing Systems, Cited by: §2. Zheng et al. (2024) Y. Zheng, J. Li, D. Yu, Y. Yang, S. E. Li, X. Zhan, and J. Liu Safe offline reinforcement learning with feasibility-guided diffusion model. In The Twelfth International Conference on Learning Representations, Cited by: §2, §4.2. Zheng et al. (2025b) Y. Zheng, R. Liang, K. ZHENG, J. Zheng, L. Mao, J. Li, W. Gu, R. Ai, S. E. Li, X. Zhan, and J. Liu Diffusion-based planning for autonomous driving with flexible guidance. In The Thirteenth International Conference on Learning Representations, Cited by: §1. Appendix A Limitations In this section, we summarize the limitations of our proposed method and outline potential directions for future research. First, applying first-order supervision directly in raw pixel space can be computationally expensive and semantically noisy. The issue is not that higher-order supervision is incompatible with visual inputs, but that pixels are a poor coordinate system for measuring dynamics and control evolution. A practical remedy is to apply the same supervision after mapping observations into a compact state representation [16], where derivatives reflect task-relevant changes rather than appearance variation. Pixel-level higher-order supervision therefore requires a good representation. Second, our higher-order supervision relies on finite-difference estimates of local derivatives, i.e., s˙=s′−s s=s -s. With coarse sampling, these estimated tangents may provide poor approximations of the true local dynamics. This issue can be alleviated by reconstructing smoother trajectories via interpolation [8] or filtering [51] before applying supervision. Appendix B Additional Results In this section, we first present the detailed evaluation results on OGBench in Table 4, followed by the performance scores on the D4RL benchmark under the full-dataset setting in Table 5. Additionally, we report the imitation learning results in Table 6. Finally, Section B.2 provides more ablation studies. B.1 Additional Evaluation Results Full Evaluation results on OGBench. We present the comprehensive per-task results on OGBench in Table 4. Note that the baseline performance of TD3+BC [14] is evaluated using the hyperparameters provided in Appendix E.2, while the baseline results for ReBRAC, IQL, and IFQL are directly sourced from FQL [49]. Table 4: Average normalized scores on 50 OGBench tasks over 5 random seeds. Values after ± denote standard deviations. The ↑ [rgb]0.0195,0.6523,0.082 marks indicate improvements greater than 1.01.0, whereas ↑ [rgb]0.5,0.5,0.5 denotes minor improvements. Det. Policy Sto. Policy (mean sup.) Sto. Policy (standard) Flow Policy Task Category TD3+BC TD3+BC+ours ReBRAC ReBRAC+ours IQL IQL+ours IFQL IFQL+ours antmaze-large-navigate-singletask-task1-v0 (∗ ) 92±892 ± 8 99±2(↑7)99 ± 2\,( [rgb]0.0195,0.6523,0.082 \!7) 91±1091 ± 10 100±1(↑9)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!9) 48±948 ± 9 80±10(↑32)80 ± 10\,( [rgb]0.0195,0.6523,0.082 \!32) 24±1724 ± 17 56±14(↑32)56 ± 14\,( [rgb]0.0195,0.6523,0.082 \!32) antmaze-large-navigate-singletask-task2-v0 85±385 ± 3 91±4(↑6)91 ± 4\,( [rgb]0.0195,0.6523,0.082 \!6) 88±488 ± 4 92±4(↑4)92 ± 4\,( [rgb]0.0195,0.6523,0.082 \!4) 42±642 ± 6 42±1542 ± 15 8±38 ± 3 16±11(↑8)16 ± 11\,( [rgb]0.0195,0.6523,0.082 \!8) antmaze-large-navigate-singletask-task3-v0 72±1072 ± 10 100±0(↑28)100 ± 0\,( [rgb]0.0195,0.6523,0.082 \!28) 51±1851 ± 18 100±1(↑49)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!49) 72±772 ± 7 90±6(↑18)90 ± 6\,( [rgb]0.0195,0.6523,0.082 \!18) 52±1752 ± 17 72±10(↑20)72 ± 10\,( [rgb]0.0195,0.6523,0.082 \!20) antmaze-large-navigate-singletask-task4-v0 85±585 ± 5 99±1(↑14)99 ± 1\,( [rgb]0.0195,0.6523,0.082 \!14) 84±784 ± 7 100±1(↑16)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!16) 51±951 ± 9 52±22(↑1)52 ± 22\,( [rgb]0.5,0.5,0.5 \!1) 18±818 ± 8 40±17(↑22)40 ± 17\,( [rgb]0.0195,0.6523,0.082 \!22) antmaze-large-navigate-singletask-task5-v0 82±1082 ± 10 99±0(↑17)99 ± 0\,( [rgb]0.0195,0.6523,0.082 \!17) 90±290 ± 2 99±1(↑9)99 ± 1\,( [rgb]0.0195,0.6523,0.082 \!9) 54±2254 ± 22 78±9(↑24)78 ± 9\,( [rgb]0.0195,0.6523,0.082 \!24) 38±1838 ± 18 48±19(↑10)48 ± 19\,( [rgb]0.0195,0.6523,0.082 \!10) antmaze-giant-navigate-singletask-task1-v0 (∗ ) 52±2052 ± 20 57±19(↑5)57 ± 19\,( [rgb]0.0195,0.6523,0.082 \!5) 27±2227 ± 22 39±21(↑12)39 ± 21\,( [rgb]0.0195,0.6523,0.082 \!12) 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 antmaze-giant-navigate-singletask-task2-v0 47±3047 ± 30 71±36(↑24)71 ± 36\,( [rgb]0.0195,0.6523,0.082 \!24) 16±1716 ± 17 70±26(↑54)70 ± 26\,( [rgb]0.0195,0.6523,0.082 \!54) 1±11 ± 1 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 0±00 ± 0 0±00 ± 0 antmaze-giant-navigate-singletask-task3-v0 44±1344 ± 13 44±1244 ± 12 34±2234 ± 22 40±15(↑6)40 ± 15\,( [rgb]0.0195,0.6523,0.082 \!6) 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 antmaze-giant-navigate-singletask-task4-v0 3±43 ± 4 63±31(↑60)63 ± 31\,( [rgb]0.0195,0.6523,0.082 \!60) 5±125 ± 12 67±33(↑62)67 ± 33\,( [rgb]0.0195,0.6523,0.082 \!62) 0±00 ± 0 1±1(↑1)1 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 0±00 ± 0 0±00 ± 0 antmaze-giant-navigate-singletask-task5-v0 65±765 ± 7 92±11(↑27)92 ± 11\,( [rgb]0.0195,0.6523,0.082 \!27) 49±2249 ± 22 96±3(↑47)96 ± 3\,( [rgb]0.0195,0.6523,0.082 \!47) 19±719 ± 7 41±6(↑22)41 ± 6\,( [rgb]0.0195,0.6523,0.082 \!22) 13±913 ± 9 21±17(↑8)21 ± 17\,( [rgb]0.0195,0.6523,0.082 \!8) humanoidmaze-medium-navigate-singletask-task1-v0 (∗ ) 24±1124 ± 11 26±4(↑2)26 ± 4\,( [rgb]0.0195,0.6523,0.082 \!2) 16±916 ± 9 36±19(↑20)36 ± 19\,( [rgb]0.0195,0.6523,0.082 \!20) 32±732 ± 7 34±5(↑2)34 ± 5\,( [rgb]0.0195,0.6523,0.082 \!2) 69±1969 ± 19 71±16(↑2)71 ± 16\,( [rgb]0.0195,0.6523,0.082 \!2) humanoidmaze-medium-navigate-singletask-task2-v0 45±1345 ± 13 49±9(↑4)49 ± 9\,( [rgb]0.0195,0.6523,0.082 \!4) 18±1618 ± 16 68±24(↑50)68 ± 24\,( [rgb]0.0195,0.6523,0.082 \!50) 41±941 ± 9 51±5(↑10)51 ± 5\,( [rgb]0.0195,0.6523,0.082 \!10) 85±1185 ± 11 91±17(↑6)91 ± 17\,( [rgb]0.0195,0.6523,0.082 \!6) humanoidmaze-medium-navigate-singletask-task3-v0 40±2640 ± 26 60±19(↑20)60 ± 19\,( [rgb]0.0195,0.6523,0.082 \!20) 36±1336 ± 13 66±22(↑30)66 ± 22\,( [rgb]0.0195,0.6523,0.082 \!30) 25±525 ± 5 27±6(↑2)27 ± 6\,( [rgb]0.0195,0.6523,0.082 \!2) 49±4949 ± 49 56±32(↑7)56 ± 32\,( [rgb]0.0195,0.6523,0.082 \!7) humanoidmaze-medium-navigate-singletask-task4-v0 30±2030 ± 20 44±9(↑14)44 ± 9\,( [rgb]0.0195,0.6523,0.082 \!14) 15±1615 ± 16 30±21(↑15)30 ± 21\,( [rgb]0.0195,0.6523,0.082 \!15) 0±10 ± 1 0±00 ± 0 1±11 ± 1 1±11 ± 1 humanoidmaze-medium-navigate-singletask-task5-v0 31±1331 ± 13 31±731 ± 7 24±2024 ± 20 78±16(↑54)78 ± 16\,( [rgb]0.0195,0.6523,0.082 \!54) 66±466 ± 4 66±466 ± 4 98±298 ± 2 100±1(↑2)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!2) humanoidmaze-large-navigate-singletask-task1-v0 (∗ ) 3±33 ± 3 3±23 ± 2 2±12 ± 1 7±7(↑5)7 ± 7\,( [rgb]0.0195,0.6523,0.082 \!5) 3±13 ± 1 4±1(↑1)4 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 6±26 ± 2 8±4(↑2)8 ± 4\,( [rgb]0.0195,0.6523,0.082 \!2) humanoidmaze-large-navigate-singletask-task2-v0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 humanoidmaze-large-navigate-singletask-task3-v0 8±48 ± 4 17±6(↑9)17 ± 6\,( [rgb]0.0195,0.6523,0.082 \!9) 8±48 ± 4 11±6(↑3)11 ± 6\,( [rgb]0.0195,0.6523,0.082 \!3) 7±37 ± 3 7±17 ± 1 48±1048 ± 10 52±16(↑4)52 ± 16\,( [rgb]0.0195,0.6523,0.082 \!4) humanoidmaze-large-navigate-singletask-task4-v0 1±11 ± 1 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 1±11 ± 1 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 1±01 ± 0 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 1±11 ± 1 1±11 ± 1 humanoidmaze-large-navigate-singletask-task5-v0 0±00 ± 0 1±1(↑1)1 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 2±22 ± 2 2±12 ± 1 1±11 ± 1 3±1(↑2)3 ± 1\,( [rgb]0.0195,0.6523,0.082 \!2) 0±00 ± 0 0±00 ± 0 antsoccer-arena-navigate-singletask-task1-v0 9±39 ± 3 11±5(↑2)11 ± 5\,( [rgb]0.0195,0.6523,0.082 \!2) 0±00 ± 0 13±6(↑13)13 ± 6\,( [rgb]0.0195,0.6523,0.082 \!13) 14±514 ± 5 36±10(↑22)36 ± 10\,( [rgb]0.0195,0.6523,0.082 \!22) 61±2661 ± 26 68±21(↑7)68 ± 21\,( [rgb]0.0195,0.6523,0.082 \!7) antsoccer-arena-navigate-singletask-task2-v0 19±819 ± 8 19±219 ± 2 0±10 ± 1 13±4(↑13)13 ± 4\,( [rgb]0.0195,0.6523,0.082 \!13) 17±717 ± 7 37±5(↑20)37 ± 5\,( [rgb]0.0195,0.6523,0.082 \!20) 75±375 ± 3 83±12(↑8)83 ± 12\,( [rgb]0.0195,0.6523,0.082 \!8) antsoccer-arena-navigate-singletask-task3-v0 25±825 ± 8 27±10(↑2)27 ± 10\,( [rgb]0.0195,0.6523,0.082 \!2) 0±00 ± 0 32±11(↑32)32 ± 11\,( [rgb]0.0195,0.6523,0.082 \!32) 6±46 ± 4 19±7(↑13)19 ± 7\,( [rgb]0.0195,0.6523,0.082 \!13) 14±2214 ± 22 22±18(↑8)22 ± 18\,( [rgb]0.0195,0.6523,0.082 \!8) antsoccer-arena-navigate-singletask-task4-v0 (∗ ) 1±11 ± 1 1±11 ± 1 0±00 ± 0 20±4(↑20)20 ± 4\,( [rgb]0.0195,0.6523,0.082 \!20) 3±23 ± 2 9±3(↑6)9 ± 3\,( [rgb]0.0195,0.6523,0.082 \!6) 16±916 ± 9 16±816 ± 8 antsoccer-arena-navigate-singletask-task5-v0 12±712 ± 7 20±9(↑8)20 ± 9\,( [rgb]0.0195,0.6523,0.082 \!8) 0±00 ± 0 6±5(↑6)6 ± 5\,( [rgb]0.0195,0.6523,0.082 \!6) 2±22 ± 2 11±7(↑9)11 ± 7\,( [rgb]0.0195,0.6523,0.082 \!9) 0±10 ± 1 8±7(↑8)8 ± 7\,( [rgb]0.0195,0.6523,0.082 \!8) cube-single-play-singletask-task1-v0 89±389 ± 3 94±2(↑5)94 ± 2\,( [rgb]0.0195,0.6523,0.082 \!5) 89±589 ± 5 94±2(↑5)94 ± 2\,( [rgb]0.0195,0.6523,0.082 \!5) 88±388 ± 3 90±5(↑2)90 ± 5\,( [rgb]0.0195,0.6523,0.082 \!2) 79±479 ± 4 96±9(↑17)96 ± 9\,( [rgb]0.0195,0.6523,0.082 \!17) cube-single-play-singletask-task2-v0 (∗ ) 87±587 ± 5 88±2(↑1)88 ± 2\,( [rgb]0.5,0.5,0.5 \!1) 92±492 ± 4 93±4(↑1)93 ± 4\,( [rgb]0.5,0.5,0.5 \!1) 85±885 ± 8 93±3(↑8)93 ± 3\,( [rgb]0.0195,0.6523,0.082 \!8) 73±373 ± 3 100±1(↑27)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!27) cube-single-play-singletask-task3-v0 93±393 ± 3 96±3(↑3)96 ± 3\,( [rgb]0.0195,0.6523,0.082 \!3) 93±393 ± 3 94±2(↑1)94 ± 2\,( [rgb]0.5,0.5,0.5 \!1) 91±591 ± 5 95±5(↑4)95 ± 5\,( [rgb]0.0195,0.6523,0.082 \!4) 88±488 ± 4 100±1(↑12)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!12) cube-single-play-singletask-task4-v0 87±487 ± 4 89±3(↑2)89 ± 3\,( [rgb]0.0195,0.6523,0.082 \!2) 92±392 ± 3 92±492 ± 4 73±673 ± 6 90±5(↑17)90 ± 5\,( [rgb]0.0195,0.6523,0.082 \!17) 79±679 ± 6 88±16(↑9)88 ± 16\,( [rgb]0.0195,0.6523,0.082 \!9) cube-single-play-singletask-task5-v0 85±585 ± 5 85±185 ± 1 87±887 ± 8 90±4(↑3)90 ± 4\,( [rgb]0.0195,0.6523,0.082 \!3) 78±978 ± 9 90±6(↑12)90 ± 6\,( [rgb]0.0195,0.6523,0.082 \!12) 77±777 ± 7 92±10(↑15)92 ± 10\,( [rgb]0.0195,0.6523,0.082 \!15) cube-double-play-singletask-task1-v0 30±430 ± 4 46±6(↑16)46 ± 6\,( [rgb]0.0195,0.6523,0.082 \!16) 45±645 ± 6 45±645 ± 6 27±527 ± 5 33±7(↑6)33 ± 7\,( [rgb]0.0195,0.6523,0.082 \!6) 35±935 ± 9 76±8(↑41)76 ± 8\,( [rgb]0.0195,0.6523,0.082 \!41) cube-double-play-singletask-task2-v0 (∗ ) 6±16 ± 1 9±1(↑3)9 ± 1\,( [rgb]0.0195,0.6523,0.082 \!3) 7±37 ± 3 32±7(↑25)32 ± 7\,( [rgb]0.0195,0.6523,0.082 \!25) 1±11 ± 1 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 9±59 ± 5 40±22(↑31)40 ± 22\,( [rgb]0.0195,0.6523,0.082 \!31) cube-double-play-singletask-task3-v0 1±11 ± 1 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 4±14 ± 1 17±5(↑13)17 ± 5\,( [rgb]0.0195,0.6523,0.082 \!13) 0±00 ± 0 1±1(↑1)1 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 8±58 ± 5 25±13(↑17)25 ± 13\,( [rgb]0.0195,0.6523,0.082 \!17) cube-double-play-singletask-task4-v0 0±10 ± 1 1±1(↑1)1 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 1±11 ± 1 3±1(↑2)3 ± 1\,( [rgb]0.0195,0.6523,0.082 \!2) 0±00 ± 0 0±00 ± 0 1±11 ± 1 12±10(↑11)12 ± 10\,( [rgb]0.0195,0.6523,0.082 \!11) cube-double-play-singletask-task5-v0 1±11 ± 1 1±11 ± 1 4±24 ± 2 22±7(↑18)22 ± 7\,( [rgb]0.0195,0.6523,0.082 \!18) 4±34 ± 3 4±24 ± 2 17±617 ± 6 55±21(↑38)55 ± 21\,( [rgb]0.0195,0.6523,0.082 \!38) scene-play-singletask-task1-v0 95±395 ± 3 96±1(↑1)96 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 95±295 ± 2 95±395 ± 3 94±394 ± 3 95±2(↑1)95 ± 2\,( [rgb]0.5,0.5,0.5 \!1) 98±398 ± 3 99±2(↑1)99 ± 2\,( [rgb]0.5,0.5,0.5 \!1) scene-play-singletask-task2-v0 (∗ ) 54±1054 ± 10 58±4(↑4)58 ± 4\,( [rgb]0.0195,0.6523,0.082 \!4) 50±1350 ± 13 60±13(↑10)60 ± 13\,( [rgb]0.0195,0.6523,0.082 \!10) 12±312 ± 3 17±6(↑5)17 ± 6\,( [rgb]0.0195,0.6523,0.082 \!5) 0±00 ± 0 88±17(↑88)88 ± 17\,( [rgb]0.0195,0.6523,0.082 \!88) scene-play-singletask-task3-v0 68±1068 ± 10 78±7(↑10)78 ± 7\,( [rgb]0.0195,0.6523,0.082 \!10) 55±1655 ± 16 75±10(↑20)75 ± 10\,( [rgb]0.0195,0.6523,0.082 \!20) 32±732 ± 7 52±9(↑20)52 ± 9\,( [rgb]0.0195,0.6523,0.082 \!20) 54±1954 ± 19 92±17(↑38)92 ± 17\,( [rgb]0.0195,0.6523,0.082 \!38) scene-play-singletask-task4-v0 4±34 ± 3 4±24 ± 2 3±33 ± 3 5±3(↑2)5 ± 3\,( [rgb]0.0195,0.6523,0.082 \!2) 0±10 ± 1 1±1(↑1)1 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 0±00 ± 0 0±00 ± 0 scene-play-singletask-task5-v0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 puzzle-3x3-play-singletask-task1-v0 96±396 ± 3 98±3(↑2)98 ± 3\,( [rgb]0.0195,0.6523,0.082 \!2) 97±497 ± 4 98±3(↑1)98 ± 3\,( [rgb]0.5,0.5,0.5 \!1) 33±633 ± 6 39±17(↑6)39 ± 17\,( [rgb]0.0195,0.6523,0.082 \!6) 94±394 ± 3 100±1(↑6)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!6) puzzle-3x3-play-singletask-task2-v0 13±913 ± 9 14±4(↑1)14 ± 4\,( [rgb]0.5,0.5,0.5 \!1) 1±11 ± 1 4±2(↑3)4 ± 2\,( [rgb]0.0195,0.6523,0.082 \!3) 4±34 ± 3 9±6(↑5)9 ± 6\,( [rgb]0.0195,0.6523,0.082 \!5) 1±21 ± 2 60±32(↑59)60 ± 32\,( [rgb]0.0195,0.6523,0.082 \!59) puzzle-3x3-play-singletask-task3-v0 5±65 ± 6 6±1(↑1)6 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 3±13 ± 1 4±4(↑1)4 ± 4\,( [rgb]0.5,0.5,0.5 \!1) 3±23 ± 2 3±13 ± 1 0±00 ± 0 28±17(↑28)28 ± 17\,( [rgb]0.0195,0.6523,0.082 \!28) puzzle-3x3-play-singletask-task4-v0 (∗ ) 7±47 ± 4 11±4(↑4)11 ± 4\,( [rgb]0.0195,0.6523,0.082 \!4) 2±12 ± 1 4±2(↑2)4 ± 2\,( [rgb]0.0195,0.6523,0.082 \!2) 2±12 ± 1 2±12 ± 1 0±00 ± 0 16±12(↑16)16 ± 12\,( [rgb]0.0195,0.6523,0.082 \!16) puzzle-3x3-play-singletask-task5-v0 14±714 ± 7 15±2(↑1)15 ± 2\,( [rgb]0.5,0.5,0.5 \!1) 5±35 ± 3 11±6(↑6)11 ± 6\,( [rgb]0.0195,0.6523,0.082 \!6) 3±23 ± 2 4±2(↑1)4 ± 2\,( [rgb]0.5,0.5,0.5 \!1) 0±00 ± 0 12±11(↑12)12 ± 11\,( [rgb]0.0195,0.6523,0.082 \!12) puzzle-4x4-play-singletask-task1-v0 1±21 ± 2 14±2(↑13)14 ± 2\,( [rgb]0.0195,0.6523,0.082 \!13) 26±426 ± 4 33±9(↑7)33 ± 9\,( [rgb]0.0195,0.6523,0.082 \!7) 12±212 ± 2 20±6(↑8)20 ± 6\,( [rgb]0.0195,0.6523,0.082 \!8) 49±949 ± 9 53±6(↑4)53 ± 6\,( [rgb]0.0195,0.6523,0.082 \!4) puzzle-4x4-play-singletask-task2-v0 7±37 ± 3 9±2(↑2)9 ± 2\,( [rgb]0.0195,0.6523,0.082 \!2) 12±412 ± 4 13±7(↑1)13 ± 7\,( [rgb]0.5,0.5,0.5 \!1) 7±47 ± 4 10±3(↑3)10 ± 3\,( [rgb]0.0195,0.6523,0.082 \!3) 4±44 ± 4 12±9(↑8)12 ± 9\,( [rgb]0.0195,0.6523,0.082 \!8) puzzle-4x4-play-singletask-task3-v0 3±23 ± 2 8±2(↑5)8 ± 2\,( [rgb]0.0195,0.6523,0.082 \!5) 15±315 ± 3 19±6(↑4)19 ± 6\,( [rgb]0.0195,0.6523,0.082 \!4) 9±39 ± 3 14±6(↑5)14 ± 6\,( [rgb]0.0195,0.6523,0.082 \!5) 50±1450 ± 14 84±16(↑34)84 ± 16\,( [rgb]0.0195,0.6523,0.082 \!34) puzzle-4x4-play-singletask-task4-v0 (∗ ) 5±25 ± 2 7±2(↑2)7 ± 2\,( [rgb]0.0195,0.6523,0.082 \!2) 10±310 ± 3 10±410 ± 4 5±25 ± 2 10±2(↑5)10 ± 2\,( [rgb]0.0195,0.6523,0.082 \!5) 21±1121 ± 11 30±17(↑9)30 ± 17\,( [rgb]0.0195,0.6523,0.082 \!9) puzzle-4x4-play-singletask-task5-v0 6±16 ± 1 6±16 ± 1 7±37 ± 3 8±4(↑1)8 ± 4\,( [rgb]0.5,0.5,0.5 \!1) 4±14 ± 1 8±5(↑4)8 ± 5\,( [rgb]0.0195,0.6523,0.082 \!4) 2±22 ± 2 21±14(↑19)21 ± 14\,( [rgb]0.0195,0.6523,0.082 \!19) The empirical results show that integrating our first-order action supervision yields consistent performance improvements across a diverse range of offline policy optimization paradigms. Notably, our scheme can improve policy learning in cases where baseline algorithms fundamentally fail. On several complex, sparse-reward tasks, baselines collapse and obtain a score of exactly 00, indicating that they cannot extract any meaningful learning signal from the offline dataset. Equipping the same algorithm with our first-order supervision lifts these scores from 00 to clearly non-trivial levels. For example, on scene-play-singletask-task2-v0, IFQL completely fails (00), whereas IFQL with our first-order supervision reaches 8888, a near-expert level. The same breakthrough is observed on the puzzle-3x3-play domain (e.g., task3-v0 and task4-v0), where the baselines obtain no useful learning signal (scoring 00), while the augmented policies acquire the temporal consistency required to achieve non-trivial success rates. Beyond raising scores on already-learnable tasks, our scheme thus expands the set of tasks on which offline policy learning is feasible at all. The improvements are also consistent across all three policy paradigms. On high-dimensional dexterous manipulation tasks such as cube-double-play-singletask-task2-v0, our scheme raises the score of IFQL from 99 to 4040. On locomotion and navigation tasks such as antmaze-giant-navigate-singletask-task5-v0, integrating our scheme into ReBRAC further raises the score from 4949 to 9696, approaching expert-level proficiency with noticeably reduced seed-level variance. Together, these results highlight the broad applicability of our scheme and indicate that supervising higher-order action dynamics is crucial for stable offline policy learning and reliable closed-loop execution. Table 5: Average normalized scores on D4RL MuJoCo locomotion tasks under full-datasize. Standard deviations are reported after the ± sign. The ↑ [rgb]0.0195,0.6523,0.082 symbol indicates a performance increase greater than 1.01.0, whereas ↑ [rgb]0.5,0.5,0.5 denotes minor improvements. All results and standard deviations (±) are averaged across 5 random seeds. Det. Policy Sto. Policy (mean sup.) Sto. Policy (standard) Flow Policy Task Category TD3+BC TD3+BC+ours ReBRAC ReBRAC+ours IQL IQL+ours IFQL IFQL+ours Hopper-m 55.5±7.755.5 ± 7.7 58.9±7.4(↑3.4)58.9 ± 7.4( [rgb]0.0195,0.6523,0.082 \!3.4) 84.9±2.684.9 ± 2.6 85.7±1.3(↑0.8)85.7 ± 1.3( [rgb]0.5,0.5,0.5 \!0.8) 67.3±5.767.3 ± 5.7 68.8±5.5(↑1.5)68.8 ± 5.5( [rgb]0.0195,0.6523,0.082 \!1.5) 89.5±5.789.5 ± 5.7 94.2±7.2(↑4.7)94.2 ± 7.2( [rgb]0.0195,0.6523,0.082 \!4.7) Hopper-mr 57.9±17.657.9 ± 17.6 58.4±9.2(↑0.5)58.4 ± 9.2( [rgb]0.5,0.5,0.5 \!0.5) 81.2±25.181.2 ± 25.1 94.3±10.1(↑13.1)94.3 ± 10.1( [rgb]0.0195,0.6523,0.082 \!13.1) 91.7±8.691.7 ± 8.6 94.2±5.5(↑2.5)94.2 ± 5.5( [rgb]0.0195,0.6523,0.082 \!2.5) 96.2±4.796.2 ± 4.7 96.3±2.3(↑0.1)96.3 ± 2.3( [rgb]0.5,0.5,0.5 \!0.1) Hopper-me 89.8±15.289.8 ± 15.2 89.9±8.4(↑0.1)89.9 ± 8.4( [rgb]0.5,0.5,0.5 \!0.1) 106.4±10.7106.4 ± 10.7 106.6±5.4(↑0.2)106.6 ± 5.4( [rgb]0.5,0.5,0.5 \!0.2) 91.5±14.391.5 ± 14.3 109.2±9.8(↑17.7)109.2 ± 9.8( [rgb]0.0195,0.6523,0.082 \!17.7) 77.1±25.577.1 ± 25.5 89.8±8.7(↑12.7)89.8 ± 8.7( [rgb]0.0195,0.6523,0.082 \!12.7) Hopper-e 108.3±6.7108.3 ± 6.7 110.5±0.2(↑2.2)110.5 ± 0.2( [rgb]0.0195,0.6523,0.082 \!2.2) 108.5±4.1108.5 ± 4.1 108.6±3.8(↑0.1)108.6 ± 3.8( [rgb]0.5,0.5,0.5 \!0.1) 99.3±9.299.3 ± 9.2 106.5±5.1(↑7.2)106.5 ± 5.1( [rgb]0.0195,0.6523,0.082 \!7.2) 96.8±12.296.8 ± 12.2 106.2±8.6(↑9.4)106.2 ± 8.6( [rgb]0.0195,0.6523,0.082 \!9.4) Halfcheetah-m 47.8±0.447.8 ± 0.4 47.9±0.4(↑0.1)47.9 ± 0.4( [rgb]0.5,0.5,0.5 \!0.1) 55.4±1.055.4 ± 1.0 56.9±1.1(↑1.5)56.9 ± 1.1( [rgb]0.0195,0.6523,0.082 \!1.5) 47.4±0.447.4 ± 0.4 48.6±0.2(↑1.2)48.6 ± 0.2( [rgb]0.0195,0.6523,0.082 \!1.2) 51.1±0.651.1 ± 0.6 52.8±0.3(↑1.7)52.8 ± 0.3( [rgb]0.0195,0.6523,0.082 \!1.7) Halfcheetah-mr 41.6±0.341.6 ± 0.3 42.3±0.2(↑0.7)42.3 ± 0.2( [rgb]0.5,0.5,0.5 \!0.7) 46.6±0.346.6 ± 0.3 46.7±0.4(↑0.1)46.7 ± 0.4( [rgb]0.5,0.5,0.5 \!0.1) 41.3±2.341.3 ± 2.3 41.4±2.0(↑0.1)41.4 ± 2.0( [rgb]0.5,0.5,0.5 \!0.1) 42.5±0.442.5 ± 0.4 43.6±1.3(↑1.1)43.6 ± 1.3( [rgb]0.0195,0.6523,0.082 \!1.1) Halfcheetah-me 88.1±5.188.1 ± 5.1 89.5±4.7(↑1.4)89.5 ± 4.7( [rgb]0.0195,0.6523,0.082 \!1.4) 96.2±1.696.2 ± 1.6 97.4±1.4(↑1.2)97.4 ± 1.4( [rgb]0.0195,0.6523,0.082 \!1.2) 86.7±3.486.7 ± 3.4 87.4±2.6(↑0.7)87.4 ± 2.6( [rgb]0.5,0.5,0.5 \!0.7) 88.4±2.288.4 ± 2.2 89.5±1.6(↑1.1)89.5 ± 1.6( [rgb]0.0195,0.6523,0.082 \!1.1) Halfcheetah-e 90.1±0.390.1 ± 0.3 93.4±0.4(↑3.3)93.4 ± 0.4( [rgb]0.0195,0.6523,0.082 \!3.3) 91.3±1.091.3 ± 1.0 91.7±0.3(↑0.4)91.7 ± 0.3( [rgb]0.5,0.5,0.5 \!0.4) 89.6±7.889.6 ± 7.8 91.4±1.2(↑1.8)91.4 ± 1.2( [rgb]0.0195,0.6523,0.082 \!1.8) 91.2±1.591.2 ± 1.5 91.6±1.7(↑0.4)91.6 ± 1.7( [rgb]0.5,0.5,0.5 \!0.4) Walker2d-m 79.7±4.179.7 ± 4.1 80.1±1.6(↑0.4)80.1 ± 1.6( [rgb]0.5,0.5,0.5 \!0.4) 76.7±9.976.7 ± 9.9 82.6±2.2(↑5.9)82.6 ± 2.2( [rgb]0.0195,0.6523,0.082 \!5.9) 71.5±10.871.5 ± 10.8 78.8±5.0(↑7.3)78.8 ± 5.0( [rgb]0.0195,0.6523,0.082 \!7.3) 79.5±4.179.5 ± 4.1 82.6±4.4(↑3.1)82.6 ± 4.4( [rgb]0.0195,0.6523,0.082 \!3.1) Walker2d-mr 67.2±16.467.2 ± 16.4 73.3±4.1(↑6.1)73.3 ± 4.1( [rgb]0.0195,0.6523,0.082 \!6.1) 70.2±11.570.2 ± 11.5 83.6±7.1(↑13.4)83.6 ± 7.1( [rgb]0.0195,0.6523,0.082 \!13.4) 71.5±8.271.5 ± 8.2 73.9±5.5(↑2.4)73.9 ± 5.5( [rgb]0.0195,0.6523,0.082 \!2.4) 72.5±19.172.5 ± 19.1 83.1±7.1(↑10.6)83.1 ± 7.1( [rgb]0.0195,0.6523,0.082 \!10.6) Walker2d-me 108.8±0.5108.8 ± 0.5 108.9±0.3(↑0.1)108.9 ± 0.3( [rgb]0.5,0.5,0.5 \!0.1) 110.3±1.4110.3 ± 1.4 110.4±0.1(↑0.1)110.4 ± 0.1( [rgb]0.5,0.5,0.5 \!0.1) 108.4±0.7108.4 ± 0.7 108.5±0.5(↑0.1)108.5 ± 0.5( [rgb]0.5,0.5,0.5 \!0.1) 106.1±1.0106.1 ± 1.0 107.5±0.7(↑1.4)107.5 ± 0.7( [rgb]0.0195,0.6523,0.082 \!1.4) Walker2d-e 108.6±0.2108.6 ± 0.2 109.3±0.1(↑0.7)109.3 ± 0.1( [rgb]0.5,0.5,0.5 \!0.7) 102.2±1.5102.2 ± 1.5 109.7±0.5(↑7.5)109.7 ± 0.5( [rgb]0.0195,0.6523,0.082 \!7.5) 110.3±1.7110.3 ± 1.7 110.4±0.5(↑0.1)110.4 ± 0.5( [rgb]0.5,0.5,0.5 \!0.1) 102.7±11.5102.7 ± 11.5 106.4±0.5(↑3.7)106.4 ± 0.5( [rgb]0.0195,0.6523,0.082 \!3.7) Table 6: Imitation learning performance on D4RL MuJoCo locomotion tasks with 10k datasets. Average normalized scores across deterministic, stochastic, and flow-based policy paradigms. The ↑ [rgb]0.0195,0.6523,0.082 indicates a performance increase >1.0>1.0, whereas ↑ [rgb]0.5,0.5,0.5 represents marginal gains. All results and standard deviations (±) are averaged across 5 random seeds. Det. Policy Sto. Policy (standard) Flow Policy Task Category base base + ℒa˙detL_ a^det base base + ℒa˙stoL_ a^sto base base + ℒa˙flowL_ a^flow Hopper-e 21.5±13.521.5 ± 13.5 67.3±14.3(↑45.8)67.3 ± 14.3( [rgb]0.0195,0.6523,0.082 \!45.8) 57.5±15.257.5 ± 15.2 93.1±10.7(↑35.6)93.1 ± 10.7( [rgb]0.0195,0.6523,0.082 \!35.6) 72.2±9.272.2 ± 9.2 97.2±12.3(↑25.0)97.2 ± 12.3( [rgb]0.0195,0.6523,0.082 \!25.0) Halfcheetah-e 2.4±1.22.4 ± 1.2 5.7±1.5(↑3.3)5.7 ± 1.5( [rgb]0.0195,0.6523,0.082 \!3.3) 2.8±1.62.8 ± 1.6 16.2±10.1(↑13.4)16.2 ± 10.1( [rgb]0.0195,0.6523,0.082 \!13.4) 11.4±4.511.4 ± 4.5 17.9±6.9(↑6.5)17.9 ± 6.9( [rgb]0.0195,0.6523,0.082 \!6.5) Walker2d-e 26.7±12.226.7 ± 12.2 91.1±14.7(↑64.4)91.1 ± 14.7( [rgb]0.0195,0.6523,0.082 \!64.4) 46.4±15.246.4 ± 15.2 91.6±9.5(↑45.2)91.6 ± 9.5( [rgb]0.0195,0.6523,0.082 \!45.2) 89.8±14.689.8 ± 14.6 91.5±6.3(↑1.7)91.5 ± 6.3( [rgb]0.0195,0.6523,0.082 \!1.7) Evaluation results on D4RL under the full-dataset setting. To further evaluate the effectiveness of our scheme, we additionally conduct experiments on the D4RL MuJoCo locomotion tasks under the standard full-size dataset setting. As reported in Table 5, integrating our first-order action supervision yields consistent performance improvements across all four base algorithms, which further confirms the broad applicability of our scheme, showing it reliably stabilizes and improves offline policy learning across varying dataset sizes. Evaluation results of imitation learning methods on the D4RL 10k setting. Within each policy paradigm, “base” denotes the standard behavior cloning baseline natively implemented under the corresponding framework, while “base + ℒa˙(⋅)L_ a^(·)” denotes the same baseline equipped with our first-order action supervision loss. As shown in Table 6, our first-order supervision consistently improves all three behavior cloning baselines, demonstrating its standalone effectiveness even without any reward signal or RL-based optimization objective. In particular, on the deterministic Walker2d-e task, adding ℒa˙detL_ a^det raises the standard BC score from 26.726.7 to 91.191.1, an absolute improvement of 64.464.4. Such a substantial gain reveals that, under small-dataset settings, standard BC is highly susceptible to compounding errors along the rollout trajectory. By directly supervising higher-order action dynamics, our scheme provides the missing temporal regularization that aligns rollout actions with expert dynamics and effectively recovers the expert behavior. More detailed ablation studies that further validate the mechanism of first-order supervision are provided in the following subsections. Learning curves and policy rollout distributions of ‖a˙‖|| a||. We additionally present learning curves and rollout distributions of ‖a˙‖|| a|| for several baseline policies and their first-order action supervised variants, covering both imitation learning and offline RL settings. These results extend the analysis in Figure 2 and Figure 3 to additional algorithms and tasks, and they consistently show that adding our first-order supervision yields smoother learning curves and more concentrated ‖a˙‖|| a|| distributions, confirming that our scheme produces temporally consistent action evolution beyond the cases highlighted in the main text. Figure 5: Comparison of vanilla stochastic BC and our 1st1^st-order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation ‖a˙‖=‖a′−a‖|| a||=||a -a|| (a is joint torque) are aggregated from 10 trajectories. Figure 6: Comparison of vanilla IQL and our 1st1^st-order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation ‖a˙‖=‖a′−a‖|| a||=||a -a|| (a is joint torque) are aggregated from 10 trajectories. Training time comparison. Table 7 compares the training time of each base algorithm against its variant equipped with our first-order action supervision on hopper-medium-v2 (D4RL) and cube-double (OGBench). Across all four policy parameterizations, integrating our scheme only moderately increases the training time, indicating that our scheme is computationally efficient and practical for real-world use. All algorithms are trained on a workstation equipped with an Intel Xeon Platinum 8336C CPU @ 2.30 GHz and NVIDIA A100-SXM4-80GB GPUs, running Ubuntu 22.04.5 LTS 64-bit. Table 7: Training time comparison between each base algorithm and its variant equipped with our first-order action supervision. All runs use the same number of training steps and are conducted on a single GPU. Det. Policy Sto. Policy (mean sup.) Sto. Policy (standard) Flow Policy Task Category TD3+BC TD3+BC+ours ReBRAC ReBRAC+ours IQL IQL+ours IFQL IFQL+ours Training time (D4RL) 28 min 30 s 33 min 6 s 25 min 8 s 30 min 34 s 34 min 44 s 39 min 56 s 30 min 49 s 35 min 52 s Training time (OGBench) 28 min 23 s 38 min 56 s 25 min 11 s 31 min 54 s 35 min 7 s 42 min 5 s 32 min 0 s 36 min 51 s Generalizing to online RL. To evaluate the versatility of our approach beyond the offline setting, we integrate both the first-order action supervision and the continuous-time HJB value regularization into the standard SAC [15] algorithm, and evaluate on three MuJoCo benchmarks over 1M training steps. We keep SAC’s original hyperparameters intact, and set λa˙=λQ=0.1 _ a= _Q=0.1 for Hopper-v2 and Walker2d-v2, and λa˙=0.01,λQ=0.1 _ a=0.01, _Q=0.1 for HalfCheetah-v2. As reported in Table 8, incorporating our first-order supervision leads to noticeably faster convergence, which is particularly evident at 300k steps, alongside reduced performance variance across seeds. These results confirm that our lightweight, plug-and-play module generalizes effectively to the online RL paradigm, improving sample efficiency and training stability without modifying the base algorithm. Table 8: Performance snapshots of SAC and SAC+Ours on online MuJoCo benchmarks over 1M training steps. Results are averaged over 5 random seeds, with standard deviations reported after the ± sign. Bold indicates the better result at each snapshot. Environment Method 100k 300k 500k 700k 1M Hopper-v2 SAC 1767.1±1005.91767.1 ± 1005.9 2399.1±1391.82399.1 ± 1391.8 3076.8±405.93076.8 ± 405.9 3389.6±162.2 3389.6 ± 162.2 3530.4±32.63530.4 ± 32.6 SAC + Ours 2375.8±1036.4 2375.8 ± 1036.4 3273.9±111.6 3273.9 ± 111.6 3226.4±101.1 3226.4 ± 101.1 3359.2±116.43359.2 ± 116.4 3531.8±33.2 3531.8 ± 33.2 HalfCheetah-v2 SAC 5659.2±676.45659.2 ± 676.4 8564.9±1699.78564.9 ± 1699.7 9750.7±1228.69750.7 ± 1228.6 10805.1±804.710805.1 ± 804.7 11557.4±493.411557.4 ± 493.4 SAC + Ours 5790.9±553.5 5790.9 ± 553.5 8614.4±1423.9 8614.4 ± 1423.9 9954.8±1126.3 9954.8 ± 1126.3 10893.5±612.1 10893.5 ± 612.1 11858.2±414.1 11858.2 ± 414.1 Walker2d-v2 SAC 702.9±343.5702.9 ± 343.5 2413.7±1150.32413.7 ± 1150.3 2989.8±1820.92989.8 ± 1820.9 4291.2±950.54291.2 ± 950.5 4768.9±393.14768.9 ± 393.1 SAC + Ours 869.9±310.3 869.9 ± 310.3 3300.3±879.6 3300.3 ± 879.6 4141.4±572.9 4141.4 ± 572.9 4805.8±184.5 4805.8 ± 184.5 4991.1±264.5 4991.1 ± 264.5 B.2 Additional Ablation Results To verify that the benefits of our scheme are not specific to a single algorithmic instantiation, we conduct ablation studies on TD3+BC (deterministic policy) and IQL (stochastic policy), evaluated on the D4RL hopper-expert-v2 and walker2d-expert-v2 tasks under the 10k dataset setting. We sweep one of the two weights over 0,0.5,1\0,0.5,1\ while strictly setting the other to zero, so that the contribution of each component can be attributed independently. All results are averaged over 5 random seeds. Figure 7: Ablation study on value regularization weight λQ _Q and 1st1^st-order action supervision weight λa˙ _ a of TD3+BC. Figure 8: Ablation study on value regularization weight λQ _Q and 1st1^st-order action supervision weight λa˙ _ a of IQL Table 9: Performance on OGBench environments. Results are reported as mean ± standard deviation. Bold indicates the highest mean in each environment. Environment (OGBENCH) IFQL +Ours +AK (λa=0.5 _a=0.5) +AK (λa=0.1 _a=0.1) +AK (λa=1 _a=1) humanoidmaze-medium-navigate-singletask-task1-v0 69±1969 ± 19 71±1671 ± 16 73.2±7.0 73.2 ± 7.0 51.2±32.451.2 ± 32.4 67.6±7.167.6 ± 7.1 humanoidmaze-medium-navigate-singletask-task4-v0 ± 1 ± 1 ± 1 ± 1 0.0±0.00.0 ± 0.0 0.4±0.80.4 ± 0.8 0.0±0.00.0 ± 0.0 antsoccer-arena-navigate-singletask-task2-v0 75±375 ± 3 ± 83 ± 12 66.8±15.366.8 ± 15.3 40.4±33.640.4 ± 33.6 35.2±20.435.2 ± 20.4 antsoccer-arena-navigate-singletask-task4-v0 16±916 ± 9 16±816 ± 8 13.6±9.613.6 ± 9.6 20.4±20.8 20.4 ± 20.8 6.4±7.36.4 ± 7.3 cube-single-play-singletask-task2-v0 73±373 ± 3 ± 100 ± 1 90.0±7.090.0 ± 7.0 27.2±33.527.2 ± 33.5 94.0±2.894.0 ± 2.8 cube-single-play-singletask-task4-v0 79±679 ± 6 88±1688 ± 16 93.6±2.3 93.6 ± 2.3 26.8±34.226.8 ± 34.2 92.0±4.092.0 ± 4.0 cube-double-play-singletask-task1-v0 35±935 ± 9 ± 76 ± 8 23.6±7.523.6 ± 7.5 29.6±5.629.6 ± 5.6 25.6±10.825.6 ± 10.8 cube-double-play-singletask-task4-v0 1±11 ± 1 ± 12 ± 10 1.2±1.01.2 ± 1.0 1.2±1.01.2 ± 1.0 1.2±1.01.2 ± 1.0 puzzle-4x4-play-singletask-task2-v0 4±44 ± 4 ± 12 ± 9 2.8±5.62.8 ± 5.6 0.0±0.00.0 ± 0.0 8.4±4.38.4 ± 4.3 puzzle-4x4-play-singletask-task3-v0 50±1450 ± 14 ± 84 ± 16 5.6±10.25.6 ± 10.2 0.0±0.00.0 ± 0.0 37.2±7.837.2 ± 7.8 Table 10: Computational cost and GPU memory usage of Ours and AK. Additional overhead is measured relative to Ours. Metric Ours AK Additional overhead Computational cost (per update) 2.498 GFLOPs 4.978 GFLOPs +2.480 GFLOPs (+99.3%) GPU memory 1.141 GiB 1.266 GiB +0.125 GiB (+11.0%) As shown in Figures 8 and 8, both algorithms exhibit the same trend: increasing either the first-order action supervision weight λa˙ _ a or the value regularization weight λQ _Q from 0 to 1 yields steady performance gains. This confirms that the policy-side and value-side first-order regularizers are robustly beneficial across both deterministic and stochastic policy classes. To evaluate the action-dependent term omitted from Eq. (7), we compare Ours with an ablation denoted as AK. AK adds ∂v⁡(k,s,ak)∂aka˙k ∂ v(k,s,a^k)∂ a^k a^k from Eq. (6) to the state-directional term, uses the same objective as Eq. (7), replacing ℒa˙flowL_ a^flow with ℒa˙AK=‖∂v⁡(k,s,ak)∂ss˙+∂v⁡(k,s,ak)∂aka˙k−a˙1‖.L_ a^AK= \| ∂ v(k,s,a^k)∂ s s+ ∂ v(k,s,a^k)∂ a^k a^k- a^1 \|. (12) Here a˙k=ka˙1 a^k=k a^1 on the linear interpolation path. Overall, Table 9 shows that Ours performs more consistently across OGBench environments. It matches or improves upon IFQL on every task and generally outperforms AK across the tested values of λa _a. Although AK benefits a few tasks, it produces substantial drops on others, particularly cube-double and puzzle-4x4. Thus, including the action-dependent term does not yield a reliable performance benefit. Table 10 further shows that AK nearly doubles computation per update (+99.3%) and increases GPU memory usage by 11.0% relative to Ours. Together, these results support our empirical choice of the state-directional approximation. Appendix C Derivation of the Higher-Order Supervision This section provides the derivations of the higher-order policy as demonstrated in section 3. Specifically, Appendix C.1 derives the action differential for stochastic Gaussian policies, while Appendix C.2 derives the time derivative of the generated action for flow-based policies. C.1 First-Order Stochastic Gaussian Policies Derivation in Eq. (3) Consider the Gaussian policy with diagonal covariance π⁡(a|s)=⁡(a,μ⁡(s),diag⁡(σ2(s))),π(a|s)=N\! (a;μ(s),diag(σ^2(s)) ), (13) where a,μ⁡(s),σ⁡(s)∈ℝda,μ(s),σ(s) ^d, and σ2(s)σ^2(s) denotes the element-wise variance. We use σ⁡(s)σ(s) to denote the element-wise standard deviation, i.e., the element-wise square root of the variance output. We represent the stochastic component of the sampled action by a=μ⁡(s)+σ⁡(s)⊙Xt,a=μ(s)+σ(s) X_t, (14) where ⊙ denotes element-wise multiplication, and Xt∈ℝdX_t ^d is assumed to follow the stationary vector OU process [58] dXt=−λXtdt+2λdWt,X0∼⁡(0,I).dX_t=-λ X_t\,dt+ 2λ\,dW_t, X_0 (0,I). (15) Here WtW_t is a d-dimensional standard Wiener process. By stationarity, Xt∼⁡(0,I)X_t (0,I) for all t. Under the diagonal covariance assumption, the stochastic perturbation is applied independently across action dimensions. Therefore, a=μ(s)+σ(s)⊙Xt,Xt∼(0,I)⟹a∼(μ(s),diag(σ2(s))).a=μ(s)+σ(s) X_t, X_t (0,I) a \! (μ(s),diag(σ^2(s)) ). We condition on (s,s˙)(s, s) and treat s=s⁡(t)s=s(t) i.e., ds=s˙dtds= s\,dt as a differentiable trajectory. For notational simplicity, we omit the explicit dependence of s on t. Define F⁡(t,Xt)=μ⁡(s)+σ⁡(s)⊙Xt,a=F⁡(t,Xt).F(t,X_t)=μ(s)+σ(s) X_t, a=F(t,X_t). For each action dimension i, define ai=Fi(t,Xt)=μi(s)+σi(s)Xt,i.a_i=F_i(t,X_t)= _i(s)+ _i(s)X_t,i. (16) Since XtX_t is a vector Itô process, applying the multi-dimensional Itô formula [47] to Fi(t,Xt)F_i(t,X_t) and substituting the OU dynamics of XtX_t gives, dai _i =[∂Fi∂t+∇XFi(t,Xt)⊤(−λXt)+12Tr⁡((2λI)⊤HXFi(t,Xt)(2λI))]dt = [ ∂ F_i∂ t+ _XF_i(t,X_t) (-λ X_t )+ 12Tr (( 2λI) H_XF_i(t,X_t)( 2λI) ) ]dt +∇XFi(t,Xt)⊤2λIdWt + _XF_i(t,X_t) 2λI\,dW_t =[∂Fi∂t−λ∇XFi(t,Xt)⊤Xt+λTr⁡(HXFi(t,Xt))]dt = [ ∂ F_i∂ t-λ _XF_i(t,X_t) X_t+ (H_XF_i(t,X_t) ) ]dt +2λ∇XFi(t,Xt)⊤dWt. + 2λ\, _XF_i(t,X_t) dW_t. (17) From Eq. (16), FiF_i is linear with respect to XtX_t. HXFi(t,Xt)=0.H_XF_i(t,X_t)=0. (18) Moreover, ∂Fi∂t=μ˙i(s)+σ˙i(s)Xt,i,∇XFi(t,Xt)=σi(s)ei, ∂ F_i∂ t= μ_i(s)+ σ_i(s)X_t,i, _XF_i(t,X_t)= _i(s)e_i, (19) where eie_i is the i-th standard basis vector. Substituting these identities into the previous expression gives dai=[μ˙i(s)+(σ˙i(s)−λσi(s))Xt,i]dt+2λσi(s)dWt,i.da_i= [ μ_i(s)+ ( σ_i(s)-λ _i(s) )X_t,i ]dt+ 2λ\, _i(s)dW_t,i. (20) Stacking all action dimensions yields da=[μ˙(s)+(σ˙(s)−λσ(s))⊙Xt]dt+2λσ(s)⊙dWt.da= [ μ(s)+ ( σ(s)-λσ(s) ) X_t ]dt+ 2λ\,σ(s) _t. (21) Using Eq. (14), we obtain Xt=(a−μ⁡(s))⊘σ⁡(s),X_t= (a-μ(s) ) σ(s), (22) where ⊘ denotes element-wise division. Hence da=[μ˙(s)+(σ˙(s)−λσ(s))⊙((a−μ⁡(s))⊘σ⁡(s))]dt+2λσ(s)⊙dWt.da= [ μ(s)+ ( σ(s)-λσ(s) ) ( (a-μ(s) ) σ(s) ) ]dt+ 2λ\,σ(s) _t. (23) Since the infinitesimal Wiener increment satisfies dWt∼⁡(0,Idt)dW_t (0,Idt), Eq. (23) implies that the infinitesimal action increment conditioned on (s,a,s˙)(s,a, s) follows da∼⁡([μ˙(s)+(σ˙(s)−λσ(s))⊙((a−μ⁡(s))⊘σ⁡(s))]dt, 2λdiag⁡(σ2(s))dt).da \! ( [ μ(s)+ ( σ(s)-λσ(s) ) ( (a-μ(s) ) σ(s) ) ]dt,\,2 (σ^2(s))dt ). (24) As mentioned in Section 3.1, in our discrete-time implementation, we use the finite-difference approximation a˙≈(a′−a)/Δt a≈(a -a)/ t with Δt=1 t=1. Under this unit-step finite-difference approximation, the induced first-order policy distribution over the action derivative is π˙(a˙|s,a,s˙)=⁡(a˙,μ˙(s)+(σ˙(s)−λσ(s))⊙((a−μ⁡(s))⊘σ⁡(s)), 2λdiag⁡(σ2(s))). π( a|s,a, s)=N\! ( a;\, μ(s)+ ( σ(s)-λσ(s) ) ( (a-μ(s) ) σ(s) ),\,2 (σ^2(s)) ). (25) C.2 First-Order Flow Policies Derivation in Eq. (6) Given the state s and initial noise sample a0a^0, the generated action is obtained by solving dakdk=v⁡(k,s,ak),ak=0=a0, da^kdk=v(k,s,a^k), a^k=0=a^0, (26) and taking a1=π⁡(s)a^1=π(s), a1=a0+∫01v⁡(k,s,ak)k.a^1=a^0+ _0^1v(k,s,a^k)\,dk. (27) Assuming sufficient smoothness, differentiating a1a^1 with respect to physical time t and applying the Leibniz integral rule gives a˙1 a^1 =a˙0+dt∫01v⁡(k,s,ak)k = a^0+ ddt _0^1v(k,s,a^k)\,dk =a˙0+∫01dtv(k,s,ak)k. = a^0+ _0^1 ddtv(k,s,a^k)\,dk. (28) Since v⁡(k,s,ak)v(k,s,a^k) depends on t through both s and aka^k, the chain rule gives dtv(k,s,ak)=∂v⁡(k,s,ak)∂ss˙+∂v⁡(k,s,ak)∂aka˙k. ddtv(k,s,a^k)= ∂ v(k,s,a^k)∂ s s+ ∂ v(k,s,a^k)∂ a^k a^k. (29) Conditioning on the sampled noise a0a^0 when taking the physical-time derivative gives a˙0=0 a^0=0. Combining Eqs. (28) and (29) yields a˙1=∫01v˙(k,s,ak)k=∫01(∂v⁡(k,s,ak)∂ss˙+∂v⁡(k,s,ak)∂aka˙k)k. a^1= _0^1 v(k,s,a^k)\,dk= _0^1 ( ∂ v(k,s,a^k)∂ s s+ ∂ v(k,s,a^k)∂ a^k a^k )dk. (30) We consider the linear interpolation path used in flow matching, ak=(1−k)a0+ka1,k∈[0,1].a^k=(1-k)a^0+ka^1, k∈[0,1]. (31) Taking its physical-time derivative while holding a0a^0 fixed gives a˙k=(1−k)a˙0+ka˙1=ka˙1. a^k=(1-k) a^0+k a^1=k a^1. (32) Equation (30) therefore takes the following form along this path, a˙1=∫01(∂v⁡(k,s,ak)∂ss˙+k∂v⁡(k,s,ak)∂aka˙1)k. a^1= _0^1 ( ∂ v(k,s,a^k)∂ s s+k ∂ v(k,s,a^k)∂ a^k a^1 )dk. (33) As in the zeroth-order action generation process in Eq. (27), the first-order action is obtained by integrating a vector field over flow time k. For the linear path, the sample-wise flow-matching target is dakdk=a1−a0 da^kdk=a^1-a^0. Taking its physical-time derivative while holding a0a^0 fixed gives dt(a1−a0)=a˙1 ddt(a^1-a^0)= a^1. Thus, a˙1=a′−a1 a^1=a -a^1 provides a pointwise target for v˙(k,s,ak) v(k,s,a^k). The shared integral structure motivates supervising this first-order field in a flow-matching manner. Based on the computational cost and performance results in Appendix B.2, we approximate v˙(k,s,ak) v(k,s,a^k) by ∂v⁡(k,s,ak)∂ss˙ ∂ v(k,s,a^k)∂ s s. Accordingly, Eq. (7) supervises this approximation at sampled values of k; the integral describes action generation rather than the training loss. Appendix D Theoretical Analysis We first introduce the formal notation, assumptions, and auxiliary lemmas used in the analysis. Then we provide the proofs of Theorems 1 and 2. D.1 Notation We briefly recap the formal notation used in Section 3.3, with the time dependence made explicit for the analysis below. Let π and π∗π^* denote the learned and expert policies, and let s⁡(t)s(t) and s∗(t)s^*(t) denote their corresponding trajectories. The closed-loop dynamics are s˙(t)=f⁡(s⁡(t),π⁡(s⁡(t))) s(t)=f(s(t),π(s(t))) and s˙∗(t)=f⁡(s∗(t),π∗(s∗(t))) s^*(t)=f(s^*(t),π^*(s^*(t))). The policy error and state deviation are denoted by e⁡(s)=π⁡(s)−π∗(s)e(s)=π(s)-π^*(s) and δ⁡(t)=s⁡(t)−s∗(t)δ(t)=s(t)-s^*(t), respectively. For each time t, define T⁡(t)=s∗(t)+τ⁡(s⁡(t)−s∗(t)):τ∈[0,1]T(t)=\s^*(t)+τ(s(t)-s^*(t)):τ∈[0,1]\. D.2 Assumptions Assumption 1 (Domain and regularity). There exists a convex compact set Ω⊂ℝd ^d such that s⁡(t),s∗(t)∈Ωs(t),s^*(t)∈ for all t under consideration. The policies π and π∗π^* are continuously differentiable on Ω , and the policy error e admits the Jacobian ∇se _se on Ω . Assumption 2 (Lipschitz dynamics). For any s1,s2∈Ωs_1,s_2∈ and any a∈ℝma ^m, the dynamics f is Lipschitz continuous in the state variable with constant LsL_s, i.e., ‖f⁡(s1,a)−f⁡(s2,a)‖≤Ls‖s1−s2‖.\|f(s_1,a)-f(s_2,a)\|≤ L_s\|s_1-s_2\|. For any s∈Ωs∈ and any a1,a2∈ℝma_1,a_2 ^m, the dynamics f is Lipschitz continuous in the action variable with constant LaL_a, i.e., ‖f⁡(s,a1)−f⁡(s,a2)‖≤La‖a1−a2‖.\|f(s,a_1)-f(s,a_2)\|≤ L_a\|a_1-a_2\|. Assumption 3 (Lipschitz expert policy). The expert policy π∗π^* is Lipschitz continuous with constant Lπ∗L_π^* on Ω , i.e., for any s1,s2∈Ωs_1,s_2∈ , ‖π∗(s1)−π∗(s2)‖≤Lπ∗‖s1−s2‖.\|π^*(s_1)-π^*(s_2)\|≤ L_π^*\|s_1-s_2\|. Assumption 4 (Bounded ∇se _se). For all t under consideration, the Jacobian of the policy error is bounded on T⁡(t)T(t) by LeL_e. That is, for any z∈T⁡(t)z∈ T(t), ‖∇se(z)‖≤Le. \| _se(z) \|≤ L_e. D.3 Auxiliary Lemmas Lemma 1. Under Assumption 1, for any s1,s2∈Ωs_1,s_2∈ , letting Δ=s1−s2 =s_1-s_2 and γ⁡(τ)=s2+τΔγ(τ)=s_2+τ for τ∈[0,1]τ∈[0,1], we have e⁡(s1)−e⁡(s2)=∫01∇se(γ⁡(τ))Δτ.e(s_1)-e(s_2)= _0^1 _se(γ(τ))\, \,dτ. Consequently, for any time t, along the segment T⁡(t)T(t), we have ‖e⁡(s⁡(t))‖≤‖e⁡(s∗(t))‖+supz∈T⁡(t)‖∇se(z)‖‖s⁡(t)−s∗(t)‖.\|e(s(t))\|≤\|e(s^*(t))\|+ _z∈ T(t) \| _se(z) \|\,\|s(t)-s^*(t)\|. Proof. Fix s1,s2∈Ωs_1,s_2∈ , and define Δ=s1−s2 =s_1-s_2 and γ⁡(τ)=s2+τΔγ(τ)=s_2+τ . By Assumption 1, γ⁡(τ)∈Ωγ(τ)∈ for all τ∈[0,1]τ∈[0,1]. Let g⁡(τ)=e⁡(γ⁡(τ))g(τ)=e(γ(τ)). Since e is continuously differentiable by Assumption 1, the chain rule gives g′(τ)=∇se(γ⁡(τ))Δ.g (τ)= _se(γ(τ))\, . By the fundamental theorem of calculus, e⁡(s1)−e⁡(s2)=g⁡(1)−g⁡(0)=∫01g′(τ)τ=∫01∇se(γ⁡(τ))Δτ.e(s_1)-e(s_2)=g(1)-g(0)= _0^1g (τ)\,dτ= _0^1 _se(γ(τ))\, \,dτ. Now take s1=s⁡(t)s_1=s(t) and s2=s∗(t)s_2=s^*(t). By the definitions of δ⁡(t)δ(t) and T⁡(t)T(t), we have Δ=δ⁡(t) =δ(t) and γ⁡(τ)∈T⁡(t)γ(τ)∈ T(t). Therefore, e⁡(s⁡(t))−e⁡(s∗(t))=∫01∇se(γ⁡(τ))δ(t)τ.e(s(t))-e(s^*(t))= _0^1 _se(γ(τ))\,δ(t)\,dτ. Taking norms yields ‖e⁡(s⁡(t))‖ \|e(s(t))\| ≤‖e⁡(s∗(t))‖+∫01‖∇se(γ⁡(τ))δ(t)‖τ ≤\|e(s^*(t))\|+ _0^1 \| _se(γ(τ))\,δ(t) \|\,dτ ≤‖e⁡(s∗(t))‖+∫01‖∇se(γ⁡(τ))‖‖δ⁡(t)‖τ ≤\|e(s^*(t))\|+ _0^1 \| _se(γ(τ)) \|\,\|δ(t)\|\,dτ ≤‖e⁡(s∗(t))‖+supz∈T⁡(t)‖∇se(z)‖‖δ⁡(t)‖. ≤\|e(s^*(t))\|+ _z∈ T(t) \| _se(z) \|\,\|δ(t)\|. Using δ⁡(t)=s⁡(t)−s∗(t)δ(t)=s(t)-s^*(t), we obtain ‖e⁡(s⁡(t))‖≤‖e⁡(s∗(t))‖+supz∈T⁡(t)‖∇se(z)‖‖s⁡(t)−s∗(t)‖.\|e(s(t))\|≤\|e(s^*(t))\|+ _z∈ T(t) \| _se(z) \|\,\|s(t)-s^*(t)\|. This completes the proof. ∎ Lemma 2 (Gronwall’s lemma). [12] Let T∈(0,∞)T∈(0,∞). Assume u is absolutely continuous on [0,T)[0,T) and satisfies the differential inequality u˙(t)≤a⁡(t)u(t)+b⁡(t),for a.e. t∈(0,T), u(t)≤ a(t)\,u(t)+b(t), a.e. t∈(0,T), for some a,b∈L1(0,T)a,b∈ L^1(0,T). Then u satisfies the pointwise estimate u⁡(t)≤eA⁡(t)u(0)+∫0tb⁡(s)eA⁡(t)−A⁡(s)s,∀t∈[0,T),u(t)≤ e^A(t)u(0)+ _0^tb(s)\,e^A(t)-A(s)\,ds, ∀\,t∈[0,T), where we define A⁡(t)=∫0ta⁡(s)s.A(t)= _0^ta(s)\,ds. This is the differential form of Gronwall’s inequality, see Proposition 2.2 of [12] for a proof. D.4 Proof of Theorem 1 Theorem 1. Under Assumption 1, the expert-trajectory policy error e(s∗(t))e(s^*(t)) satisfies: ‖e⁡(s∗(t))‖≤‖e⁡(s∗(0))‖+tmax0≤τ≤t‖∇sπ(s∗(τ))s˙∗(τ)−π˙∗(s∗(τ))‖.\|e(s^*(t))\|≤\|e(s^*(0))\|+t\, _0≤τ≤ t \| _sπ(s^*(τ)) s^*(τ)- π^*(s^*(τ)) \|. Proof. Recall the definition of the expert-trajectory policy error: e⁡(s∗(t))=π⁡(s∗(t))−π∗(s∗(t)).e(s^*(t))=π(s^*(t))-π^*(s^*(t)). Define the expert action derivative along the expert trajectory as π˙∗(s∗(t))=dtπ∗(s∗(t))=∇sπ∗(s∗(t))s˙∗(t). π^*(s^*(t))= ddtπ^*(s^*(t))= _sπ^*(s^*(t))\, s^*(t). By Assumption 1 and the chain rule, the time derivative of the expert-trajectory policy error is dte(s∗(t))=∇sπ(s∗(t))s˙∗(t)−π˙∗(s∗(t)). ddte(s^*(t))= _sπ(s^*(t))\, s^*(t)- π^*(s^*(t)). Integrating over time yields: e⁡(s∗(t))=e⁡(s∗(0))+∫0t(∇sπ(s∗(τ))s˙∗(τ)−π˙∗(s∗(τ)))τ.e(s^*(t))=e(s^*(0))+ _0^t ( _sπ(s^*(τ))\, s^*(τ)- π^*(s^*(τ)) )\,dτ. Taking norms: ‖e⁡(s∗(t))‖≤‖e⁡(s∗(0))‖+∫0t‖∇sπ(s∗(τ))s˙∗(τ)−π˙∗(s∗(τ))‖τ.\|e(s^*(t))\|≤\|e(s^*(0))\|+ _0^t \| _sπ(s^*(τ))\, s^*(τ)- π^*(s^*(τ)) \|\,dτ. Therefore, ‖e⁡(s∗(t))‖≤‖e⁡(s∗(0))‖+tmax0≤τ≤t‖∇sπ(s∗(τ))s˙∗(τ)−π˙∗(s∗(τ))‖.\|e(s^*(t))\|≤\|e(s^*(0))\|+t\, _0≤τ≤ t \| _sπ(s^*(τ))\, s^*(τ)- π^*(s^*(τ)) \|. This completes the proof. ∎ D.5 Proof of Theorem 2 Theorem 2. Under Assumptions 1–4, for δ⁡(0)=0δ(0)=0, the state deviation δ⁡(t)=s⁡(t)−s∗(t)δ(t)=s(t)-s^*(t) satisfies ‖δ⁡(t)‖≤Laβ(eβt−1)max0≤τ≤t‖e⁡(s∗(τ))‖,whereβ=Ls+LaLπ∗+LaLe.\|δ(t)\|≤ L_aβ (e^β t-1 ) _0≤τ≤ t\|e(s^*(τ))\|, β=L_s+L_aL_π^*+L_aL_e. Proof. From Lemma 1 and Assumption 4, we have: ‖e⁡(s⁡(t))‖≤‖e⁡(s∗(t))‖+Le‖s⁡(t)−s∗(t)‖.\|e(s(t))\|≤\|e(s^*(t))\|+L_e\|s(t)-s^*(t)\|. (34) The time derivative of the state deviation is: δ˙(t)=f⁡(s⁡(t),π⁡(s⁡(t)))−f⁡(s∗(t),π∗(s∗(t))). δ(t)=f(s(t),π(s(t)))-f(s^*(t),π^*(s^*(t))). By Assumptions 2 and 3, we have ‖δ˙(t)‖ \| δ(t)\| =‖f⁡(s⁡(t),π⁡(s⁡(t)))−f⁡(s∗(t),π∗(s∗(t)))‖ =\|f(s(t),π(s(t)))-f(s^*(t),π^*(s^*(t)))\| ≤|f⁡(s⁡(t),π⁡(s⁡(t)))−f⁡(s⁡(t),π∗(s⁡(t)))|+‖f⁡(s⁡(t),π∗(s⁡(t)))−f⁡(s∗(t),π∗(s∗(t)))‖ ≤\|f(s(t),π(s(t)))-f(s(t),π^*(s(t)))\|+\|f(s(t),π^*(s(t)))-f(s^*(t),π^*(s^*(t)))\| ≤La|π⁡(s⁡(t))−π∗(s⁡(t))|+‖f⁡(s⁡(t),π∗(s⁡(t)))−f⁡(s∗(t),π∗(s⁡(t)))‖ ≤ L_a\|π(s(t))-π^*(s(t))\|+\|f(s(t),π^*(s(t)))-f(s^*(t),π^*(s(t)))\| +‖f⁡(s∗(t),π∗(s⁡(t)))−f⁡(s∗(t),π∗(s∗(t)))‖ +\|f(s^*(t),π^*(s(t)))-f(s^*(t),π^*(s^*(t)))\| ≤La‖e⁡(s⁡(t))‖+Ls|s⁡(t)−s∗(t)|+La‖π∗(s⁡(t))−π∗(s∗(t))‖ ≤ L_a\|e(s(t))\|+L_s\|s(t)-s^*(t)\|+L_a\|π^*(s(t))-π^*(s^*(t))\| ≤La|e⁡(s⁡(t))‖+Ls‖δ(t)|+LaLπ∗‖δ⁡(t)‖ ≤ L_a\|e(s(t))\|+L_s\|δ(t)\|+L_aL_π^*\|δ(t)\| =La|e⁡(s⁡(t))|+(Ls+LaLπ∗)‖δ⁡(t)‖. =L_a\|e(s(t))\|+ (L_s+L_aL_π^* )\|δ(t)\|. Using the bound in Eq. (34), we get: ‖δ˙(t)‖≤La‖e⁡(s∗(t))‖+(Ls+LaLπ∗+LaLe)‖δ⁡(t)‖.\| δ(t)\|≤ L_a\|e(s^*(t))\|+ (L_s+L_aL_π^*+L_aL_e )\|δ(t)\|. Let β=Ls+LaLπ∗+LaLeβ=L_s+L_aL_π^*+L_aL_e. Then: ‖δ˙(t)‖≤β‖δ⁡(t)‖+La‖e⁡(s∗(t))‖.\| δ(t)\|≤β\|δ(t)\|+L_a\|e(s^*(t))\|. Let u⁡(t)=‖δ⁡(t)‖u(t)=\|δ(t)\|. Under Assumptions 1 and 2, the trajectories s⁡(t)s(t) and s∗(t)s^*(t) are absolutely continuous. Hence, δ⁡(t)=s⁡(t)−s∗(t)δ(t)=s(t)-s^*(t) is absolutely continuous. Since the norm is Lipschitz continuous, u⁡(t)u(t) is also absolutely continuous. Moreover, for almost all t∈(0,T)t∈(0,T), we have u˙(t)≤‖δ˙(t)‖. u(t)≤\| δ(t)\|. Therefore, u˙(t)≤βu(t)+La‖e⁡(s∗(t))‖,for a.e. t∈(0,T). u(t)≤β u(t)+L_a\|e(s^*(t))\|, a.e. t∈(0,T). (35) For Eq. (35), applying Lemma 2 with a⁡(t)=βa(t)=β and b⁡(t)=La‖e⁡(s∗(t))‖b(t)=L_a\|e(s^*(t))\| yields ‖δ⁡(t)‖≤eβt‖δ⁡(0)‖+La∫0teβ⁡(t−τ)‖e⁡(s∗(τ))‖τ.\|δ(t)\|≤ e^β t\|δ(0)\|+L_a _0^te^β(t-τ)\|e(s^*(τ))\|\,dτ. This can be further bounded as: ‖δ⁡(t)‖≤eβt|δ⁡(0)|+Laβ(eβt−1)max0≤τ≤t‖e⁡(s∗(τ))‖.\|δ(t)\|≤ e^β t\|δ(0)\|+ L_aβ (e^β t-1 ) _0≤τ≤ t\|e(s^*(τ))\|. Assuming δ⁡(0)=0δ(0)=0, we get: ‖δ⁡(t)‖≤Laβ(eβt−1)max0≤τ≤t‖e⁡(s∗(τ))‖.\|δ(t)\|≤ L_aβ (e^β t-1 ) _0≤τ≤ t\|e(s^*(τ))\|. This completes the proof. ∎ Appendix E Experimental Details We evaluate our method on two widely used offline RL benchmarks: OGBench [48] and D4RL [13]. OGBench provides high-dimensional, long-horizon, and structurally diverse tasks that go well beyond standard locomotion, while D4RL offers a well-established testbed of MuJoCo locomotion tasks with datasets of varying behavioral quality. Together, they cover a broad spectrum of state and action dimensionalities, task structures, and dataset qualities, providing a comprehensive evaluation of our scheme. All methods are trained on the offline datasets only, with no online interaction. E.1 Benchmarks and Tasks OGBench. Since OGBench was originally designed for goal-conditioned RL, we adopt its reward-based single-task variants (denoted by the -singletask suffix) to benchmark standard reward-maximizing offline RL methods. In total, we evaluate on 5050 state-based tasks drawn from 1010 environments (55 tasks per environment), covering long-horizon locomotion (antmaze, humanoidmaze), object interaction (antsoccer), and robotic manipulation (cube, scene, puzzle), where scene additionally requires long-horizon multi-object manipulation and puzzle involves more combinatorial reasoning. These tasks pose substantial challenges in high-dimensional continuous control, with state dimensions ranging from 2828 to 8383 and action dimensions from 55 to 2121. For instance, humanoidmaze has a 6969-dimensional state space and a 2121-dimensional action space, while cube-double, scene, and puzzle-4x4 have 3737-, 4040-, and 8383-dimensional state spaces, respectively. We use the standard reward-based datasets provided by OGBench, which are highly suboptimal as they aggregate trajectories from policies performing diverse random tasks (e.g., reaching different goals or manipulating different objects). This makes it particularly difficult for offline RL methods to extract task-relevant behaviors from such heterogeneous data. D4RL. We further evaluate our method on the MuJoCo locomotion tasks from D4RL, including halfcheetah, hopper, and walker2d, with the standard medium, medium-replay, medium-expert, and expert dataset types collected from behavior policies of varying quality. The full-size datasets contain about 11M transitions for the medium and expert settings, 22M transitions for medium-expert, and around 200200k–400400k transitions for medium-replay. Beyond the standard full-data setting, we additionally consider a small-dataset setting to study the data efficiency of our scheme. Specifically, for each task we first split the original dataset into individual trajectories, and then sample a small set of complete trajectories until the accumulated transitions reach 1010k, rather than randomly sampling individual transitions. This trajectory-level sampling preserves the temporal continuity and dynamical consistency of the original demonstrations, which is essential for our first-order action supervision since it relies on consecutive transitions (s,s′,a,a′)(s,s ,a,a ) to compute the local state and action derivatives (s˙,a˙)( s, a). To ensure a fair comparison, all methods use the same subsampled dataset under the same random seed. E.2 Methods and Hyperparameters We instantiate our first-order action supervision on four representative offline RL algorithms covering deterministic, stochastic, and flow-based policies: TD3+BC [14], ReBRAC [57], IQL [27], and IFQL [49]. For a fair comparison, all algorithms are implemented on top of the FQL [49] codebase and share the same network architectures and default hyperparameters unless otherwise specified. For OGBench tasks, we follow the hyperparameter settings reported in FQL [49] for ReBRAC, IQL, and IFQL. After integrating our scheme, we keep all baseline-specific hyperparameters unchanged, and only specify the two additional weights λQ _Q (the value-side regularization weight) and λa˙ _ a (the policy-side first-order supervision weight) introduced by our scheme. The basic shared hyperparameters are summarized in Table 11. TD3+BC. We re-implement TD3+BC on top of the FQL [49] codebase. On OGBench, we sweep the Q-value scaling coefficient α over 1,3,10,30,100,300\1,3,10,30,100,300\ and select the best value per environment. Intuitively, a smaller α is preferable in complex environments where the Q-function is hard to learn reliably, while a larger α works better on simpler ones where the learned Q-function offers stronger policy improvement signals. We also observe Q-learning instability on several challenging OGBench tasks, and apply layer normalization to the Q-network to mitigate this issue. On D4RL, we follow the hyperparameter settings of the original TD3+BC paper [14], and use the default values for all other hyperparameters (e.g., noise standard deviation, noise clipping threshold). For our scheme, we set λQ=0.1 _Q=0.1 on all OGBench tasks, λa˙=1.0 _ a=1.0 on antmaze and λa˙=0.1 _ a=0.1 on all other OGBench tasks; on D4RL, we set λQ=0.1 _Q=0.1 and λa˙=1.0 _ a=1.0. ReBRAC. We follow the hyperparameter settings reported in the original ReBRAC paper [57] on D4RL, including the actor and critic BC coefficients (αa,αc)( _a, _c). For our scheme, we set λQ=0.1 _Q=0.1 on all OGBench tasks, λa˙=1.0αa _ a=1.0\, _a on antmaze, humanoidmaze, and antsoccer, and λa˙=0.1αa _ a=0.1\, _a on cube, scene, and puzzle; on D4RL, we set λQ=0.1 _Q=0.1 and λa˙=1.0αa _ a=1.0\, _a. IQL. We follow the hyperparameter settings reported in the original IQL paper [27] on D4RL, including the expectile value and the AWR inverse temperature. For our scheme on OGBench, we set λQ=0.5 _Q=0.5 on scene and λQ=0.1 _Q=0.1 on all other tasks, and λa˙=1.0 _ a=1.0 on antmaze and scene and λa˙=0.1 _ a=0.1 on all other tasks; on D4RL, we set λQ=0.1 _Q=0.1 and λa˙=1.0 _ a=1.0. IFQL. For IFQL, we use N=32N=32 action samples on D4RL tasks, and follow all other settings from FQL [49]. For our scheme, we set λQ=1.0 _Q=1.0 and λa˙=1.0 _ a=1.0 on all OGBench and D4RL tasks. Table 11: Shared hyperparameters used across all four offline RL algorithms. Hyperparameter Value Learning rate 3×10−43× 10^-4 Optimizer Adam [24] Gradient steps 1,000,0001,000,000 Minibatch size 256256 MLP dimensions [512,512,512,512][512,512,512,512] (default); [256,256,256,256][256,256,256,256] (IQL) Nonlinearity GELU [18] Target network smoothing coefficient 0.0050.005 Discount factor γ 0.990.99 (default); 0.9950.995 (antmaze-giant, humanoidmaze, antsoccer) Temporal correlation coefficient λ 1.01.0 Appendix F Broader Impact Modern learning-based decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), often suffer from control instability and poor robustness when deployed in real-world systems, due to their sole reliance on zeroth-order action supervision. To address this, we propose a lightweight and plug-and-play learning scheme that simultaneously supervises zeroth- and first-order actions, transforming any standard policy model into a more powerful policy class without structural modifications. Our method substantially improves action smoothness, control stability, and out-of-distribution generalization, especially in challenging low-data regimes, making it well-suited for safety-critical domains such as robotics, drone control, and autonomous driving, where high-frequency action fluctuations can cause mechanical wear, energy waste, and safety risks. By bridging classical control principles with modern policy learning, our work offers a promising pathway toward more reliable and deployable learning-based controllers. A potential limitation is that, like other offline methods, the learned policy may still inherit unsafe or suboptimal behaviors present in the historical dataset.

Instant research discovery