Paper deep dive
Higher-Order Action Supervision Makes A Strong Policy Class
Peng Cheng, Yunxian Hou, Zhi Zhou, Qian Zhang, Chang Huang, Xianyuan Zhan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/9/2026, 8:27:05 AM
Summary
The paper introduces a novel higher-order action supervision method for reinforcement learning and imitation learning policies. It addresses control instability and robustness issues in real-world applications by simultaneously supervising zeroth-order actions (action labels) and first-order actions (time derivatives of actions). The proposed method uses a theoretical framework to compute first-order action supervision via the policy's time derivative (gradient of policy w.r.t state multiplied by state derivative), applicable to deterministic, stochastic, and flow policies without structural modifications. It serves as a lightweight plug-and-play module for offline RL frameworks, demonstrating significant performance and robustness improvements on OGBench and D4RL benchmarks, particularly in low-data regimes and out-of-distribution generalization.
Entities (14)
Relation Signals (10)
Higher-Order Action Supervision → evaluatedon → OGBench
confidence 95% · Extensive evaluations on OGBench and D4RL demonstrate that our approach yields substantial performance and robustness improvements...
Higher-Order Action Supervision → evaluatedon → D4RL
confidence 95% · Extensive evaluations on OGBench and D4RL demonstrate that our approach yields substantial performance and robustness improvements...
Higher-Order Action Supervision → solves → Control Instability
confidence 95% · We argue that this instability issue stems largely from their limitations in solely supervising and optimizing zeroth-order actions... In this paper, we show that simultaneously supervising both zeroth- and first-order actions can dramatically enhance policies' performance and control robustness.
Higher-Order Action Supervision → appliesto → Imitation Learning
confidence 90% · supported by formal theoretical guarantees for the IL setting.
Higher-Order Action Supervision → appliesto → Reinforcement Learning
confidence 90% · seamlessly integrates with a broad spectrum of existing offline RL frameworks.
Higher-Order Action Supervision → supports → Deterministic Policy
confidence 90% · We provide practical instantiations of loss schemes for deterministic, stochastic, and generative flow matching policy classes...
Higher-Order Action Supervision → supports → Stochastic Policy
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Modern data-driven decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), have achieved great success in solving many complex tasks. However, these methods often suffer from serious control instability and robustness issues when applied in real-world applications such as robotics and autonomous driving, posing notable challenges for their practical deployment. We argue that this instability issue stems largely from their limitations in solely supervising and optimizing zeroth-order actions (i.e., the action labels), failing to account for higher-order action dynamics and temporal consistency. In this paper, we show that simultaneously supervising both zeroth- and first-order actions can dramatically enhance policies' performance and control robustness. To achieve this, we introduce a novel and elegant loss scheme supported by formal theoretical guarantees that can equip any off-the-shelf policy model (e.g., deterministic, stochastic, or flow policies) with the capability for higher-order action supervision, without requiring any structural modifications. Moreover, our proposed method can serve as a lightweight plug-and-play module that seamlessly integrates with a broad spectrum of existing offline RL frameworks. Extensive evaluations on OGBench and D4RL demonstrate that our approach yields substantial performance and robustness improvements across a wide range of continuous control environments. Notably, our method can also enhance policies' out-of-distribution (OOD) generalization capability in the challenging low-data regime, making it an ideal tool in tackling many real-world control problems.
Tags
Links
- Source: https://arxiv.org/abs/2610.11175v1
- Canonical: https://arxiv.org/abs/2610.11175v1
Trouble viewing inline? Open PDF directly →
Full Text
172,327 characters extracted from source content.
Expand or collapse full text
Higher-Order Action Supervision Makes A Strong Policy Class Peng Cheng †thanks: Equal contribution. Affiliation: Institute for AI Industry Research (AIR), Tsinghua University Affiliation: Zenfex AI Lab Yunxian Hou11footnotemark: 1 Affiliation: University of Electronic Science and Technology of China Zhi Zhou Affiliation: Institute for AI Industry Research (AIR), Tsinghua University Affiliation: Zenfex AI Lab Qian Zhang Affiliation: Horizon Robotics Chang Huang Affiliation: Horizon Robotics Xianyuan Zhan †thanks: Corresponding author. Affiliation: Institute for AI Industry Research (AIR), Tsinghua University Abstract Modern data-driven decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), have achieved great success in solving many complex tasks. However, these methods often suffer from serious control instability and robustness issues when applied in real-world applications such as robotics and autonomous driving, posing notable challenges for their practical deployment. We argue that this instability issue stems largely from their limitations on solely supervising and optimizing zeroth-order actions (i.e., the action labels), failing to account for higher-order action dynamics and temporal consistency. In this paper, we show that simultaneously supervising both zeroth- and first-order actions can dramatically enhance policies’ performance and control robustness. To achieve this, we introduce a novel and elegant loss scheme supported by formal theoretical guarantees that can equip any off-the-shelf policy model (e.g., deterministic, stochastic, or flow policies) with the capability for higher-order action supervision, without requiring any structural modifications. Moreover, our proposed method can serve as a lightweight plug-and-play module that seamlessly integrates with a broad spectrum of existing offline RL frameworks. Extensive evaluations on OGBench and D4RL demonstrate that our approach yields substantial performance and robustness improvements across a wide range of continuous control environments. Notably, our method can also enhance policies’ out-of-distribution (OOD) generalization capability in the challenging low-data regime, making it an ideal tool in tackling many real-world control problems. 1 Introduction Learning-based decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), have achieved notable success in many complex tasks, such as games (Mnih et al., 2015; Silver et al., 2017), robotics (Kalashnikov et al., 2018; Chi et al., 2023), and autonomous driving (Kendall et al., 2019; Zheng et al., 2025b). However, when these learning-based approaches are deployed in real-world control problems, they are frequently criticized for their control instability and poor robustness. Examples of such issues include lack of smoothness in action outputs, severe jittering across consecutive steps, and poor generalization in offline learning under data sparsity (Dulac-Arnold et al., 2021; Song et al., 2023; Cheng et al., 2023; Cheng et al., 2026). In contrast to conventional control theories (Kirk, 2004) that place heavy emphasis on control stability and higher-order behavior modeling, contemporary learning-based decision-making methods in the AI community predominantly only focus on supervising and optimizing zeroth-order actions (i.e., the actual action labels). In realistic control scenarios, the neglect of higher-order temporal consistency in actions frequently leads to control instability and non-smooth action execution, which constitutes a critical bottleneck for real-world deployment (Dulac-Arnold et al., 2021). Under physical hardware constraints, such high-frequency action fluctuations can cause excessive mechanical wear and energy consumption, drastically shortening the lifespan of both the system and its constituent parts (Song et al., 2023; Song et al., 2025; Mysore et al., 2021; de La Presilla et al., 2023; Zhan et al., 2022; Zhan et al., 2025). These vulnerabilities pose severe safety risks and degrade operational reliability across various domains, including drone control (Mysore et al., 2021; Shi et al., 2019), robotic manipulation (Yu et al., 2021; Zhang et al., 2024), and autonomous driving (Cai et al., 2020; Chen et al., 2021; Wasala et al., 2020; Lee et al., 2024). Fundamentally, this issue arises because the policy becomes excessively sensitive to minor state-action perturbations, yielding disproportionately large action variations (Song et al., 2023; Chen et al., 2021). We empirically illustrate this vulnerability using a maze navigation task in Figure 1. As shown in Figure 1(a), the policy learned via standard behavior cloning (BC), which lacks higher-order action supervision, produces large and highly oscillatory first-order actions, eventually producing a suboptimal trajectory. By contrast, jointly supervising both zeroth- and first-order actions enforces much stronger behavioral regularization, resulting in significantly smoother and optimal execution (Figure 1(b)). Figure 1: Trajectories and zeroth-/first-order actions of policies learned on the maze2d task with 20 successful trajectories and 500k training steps. Green arrows denote the zeroth-order action a (joint torque), while red arrows denote the first-order action a˙ a (time derivative of joint torque, computed as a′−a -a). (a) Standard behavior cloning (BC) without a˙ a supervision, which yields a suboptimal trajectory with highly noisy and temporally inconsistent first-order actions. (b) Our method with higher-order action supervision, which yields an optimal navigation trajectory with much smoother and more stable zeroth- and first-order actions. However, incorporating higher-order action information in policy modeling is not an easy task. As the 0th0^th-order action a and the 1st1^st-order action a˙ a live in two completely different action spaces, naïvely predicting these two sets of actions not only doubles the action space, causing learning difficulties, but also faces potential conflicting gradient issues that are commonly encountered in multi-task learning (Yu et al., 2020). Therefore, existing works on action smoothness improvement primarily focus on introducing action fluctuation penalty regularizations (Mysore et al., 2021; Kobayashi, 2022; Shen et al., 2020; Zhao et al., 2022), or designing special neural network architectures to constrain the network’s Lipschitz constant, e.g., via spectral normalization or specialized modules like LipsNet (Song et al., 2023; Song et al., 2025). However, both streams of work suffer from a number of limitations. Adding smoothness penalties relies on task-specific ad-hoc design and cumbersome tuning efforts, whereas using specialized, heavily regularized neural network architectures loses generality and restricts the model’s expressivity to capture complex data distributions. In this work, we introduce a novel and elegant learning scheme that can achieve simultaneous 0th0^th- and 1st1^st-order action supervision. The best part is, we can easily transform any given policy model into a more powerful policy class without making any structural modification. The secret recipe is to leverage the intrinsic property of the policy model’s time derivative for supervision, i.e., π˙(s)=∇sπ(s)⋅s˙ π(s)= _sπ(s)· s. We provide practical instantiations of loss schemes for deterministic, stochastic, and generative flow matching policy classes, supported by formal theoretical guarantees for the IL setting. Moreover, we show that our enhanced policy class is naturally compatible with the continuous-time RL paradigm, like the Hamilton-Jacobi-Bellman (HJB) framework. This leads to a plug-and-play enhancement module that can be seamlessly integrated with a wide spectrum of existing offline RL algorithms with minimal code modifications. We evaluate our method through extensive experiments on diverse tasks from the large-scale OGBench (Park et al., 2025a) and widely used D4RL (Fu et al., 2020) benchmarks. Despite its simplicity, our approach achieves consistent and significant performance improvements when added to different offline policy optimization paradigms. Notably, in challenging small-dataset settings, our proposed modification excels by providing robust OOD generalization support, enabling the agent to maintain stable and high performance even on tasks where the base algorithms fail or severely degrade. This makes our method an ideal tool for tackling many real-world control problems. 2 Related work Smoothness and robustness enhancement for policy learning. Mitigating action fluctuation and trajectory drift remains a critical challenge for deploying policies in physical systems (Dulac-Arnold et al., 2021; Song et al., 2023). To enforce action smoothness and robustness, prior studies in continuous control have primarily explored two directions: penalty-based regularizations (Mysore et al., 2021; Kobayashi, 2022; Shen et al., 2020; Zhao et al., 2022) and architectural enhancements (Song et al., 2023; Song et al., 2025). Penalty-based regularization approaches, such as CAPS (Mysore et al., 2021), L2C2 (Kobayashi, 2022), and SR2L (Shen et al., 2020; Zhao et al., 2022), introduce action fluctuation penalty regularizations to their policy optimization objectives. However, they inherently rely on querying or sampling synthetic states to enforce local action consistency. In the offline setting, where interaction with the environment is not permitted, such state-dependent sampling may introduce significant OOD extrapolation errors, and often suffer from significant performance variability across environments (Christmann et al., 2024). In contrast, architectural enhancement methods explicitly constrain the network’s Lipschitz constant, e.g., via spectral normalization or specialized modules like LipsNet (Song et al., 2023; Song et al., 2025). However, imposing extra structural priors can also restrict the model’s expressivity. This introduces a fundamental trade-off: while the policy becomes smoother, the restricted capacity commonly leads to underfitting, degraded control performance, and additional computational overhead during inference. Concurrently, there are also a few attempts to incorporate higher-order action derivatives to enhance control smoothness, such as TaSIL (Pfrommer et al., 2022), which minimizes the gap between the Taylor series expansion of the learned and expert policies. Although promising, it remains more theoretical rather than practical, due to the complex evaluation procedure involved. Offline RL and continuous-time RL. In this study, we primarily focus on offline policy learning, such as offline IL (e.g., behavior cloning, BC) and offline RL, which directly imitate or optimize over a fixed offline dataset without environment interaction. Compared with IL, offline RL is more complex, which optimizes the cumulative return while being regularized towards the behavior data to avoid extrapolation error accumulation (Levine et al., 2020; Kumar et al., 2019). Depending on how the learned policy is parameterized, existing offline RL algorithms can be classified into three categories. The first category employs deterministic policies and explicitly constrains the learned actions toward the dataset using distance-based behavior cloning penalties (Fujimoto and Gu, 2021; Li et al., 2023b; Cheng et al., 2023). The second and also the more popular category adopts stochastic policies (e.g., Gaussian policies) (Wu et al., 2019; Kumar et al., 2019; Kumar et al., 2020; Kostrikov et al., 2022; Lyu et al., 2022; Xu et al., 2023; Wang et al., 2023; Xu et al., 2022a; Xu et al., 2022b; Niu et al., 2022; Li et al., 2023a; Mao et al., 2024b; Xu et al., 2026), which maximizes the likelihood of high-return actions under some distribution-based data regularizations. Recently, the third category of offline RL methods has emerged, which leverages expressive generative models, such as diffusion or flow matching policies, to capture the complex, multimodal action distributions in the datasets (Wang et al., 2022; Janner et al., 2022; Hansen-Estruch et al., 2023; Li et al., 2024; Mao et al., 2024a; Zheng et al., 2024; Park et al., 2025b; Zheng et al., 2025a; Liang et al., 2026; Zheng et al., 2026). However, all of the above methods are restricted to learning policies in the 0th0^th-order action spaces, which suffer from notable control stability and robustness issues. Incorporating higher-order action supervision in offline RL remains entirely unexplored. Another research avenue closely related to our work is continuous-time RL, which transcends the structural limitations of discrete-time Markov decision processes (MDPs) by modeling state and action evolution continuously over time (Doya, 2000; Wang et al., 2020; Munos, 2006). The Hamilton-Jacobi-Bellman (HJB) equation serves as the core theoretical foundation for this domain (Lee and Sutton, 2021; Vamvoudakis and Lewis, 2010; Bhasin et al., 2013; Vamvoudakis et al., 2014). Recent advancements have successfully utilized HJB-based frameworks to demonstrate how continuous-time structural priors can directly guide stable policy evaluation and control (Kim et al., 2021; Jia and Zhou, 2022; Lien et al., 2024). Although the HJB-based framework naturally requires 1st1^st-order action a˙ a information to capture Q-value’s variation in continuous time, due to the lack of a proper strategy to evaluate a˙ a from policy π, existing methods often resort to a crude approximation or replace it with a fixed bounding value. This drastically degrades the value estimation quality under the HJB framework and leads to suboptimal performance. 3 Extending Standard Policy Classes for Higher-Order Action Supervision In this paper, unlike prior methods that mitigate action prediction instability by using specialized network architectures or introducing task-specific ad-hoc designs, we propose an elegant, lightweight, and plug-and-play learning scheme to transform a standard policy model (regardless of deterministic, stochastic, or generative flow policies) into a more powerful policy class. Our proposed method enables simultaneous 0th0^th and 1st1^st-order action supervision for a policy model, maximally utilizing every bit of information in the offline data. 3.1 Notations We consider the sequential decision-making problem modeled as an MDP, defined by the tuple ℳ=,,,r,ρ0,γM=\S,A,P,r, _0,γ\. S represents the state space, ⊂ℝdA ^d is the d-dimensional continuous action space, (s′|s,a)P(s |s,a) denotes the stochastic transition dynamics, r:×→ℝr:S×A is the reward function, ρ0(s) _0(s) specifies the initial state distribution, and γ∈[0,1)γ∈[0,1) is the discount factor. In the offline setting, the agent cannot interact with the environment and is instead provided with a static dataset D consisting of offline trajectories τ=s0,a0,r0,s1,a1,…τ=\s_0,a_0,r_0,s_1,a_1,…\. In the offline IL setting, we aim to learn an imitative policy π to clone the expert behaviors specified in the data trajectories; whereas in offline RL, we aim to learn a policy π exclusively from the dataset D that maximizes the expected cumulative discounted return: J(π)=τ∼π[∑t=0∞γtr(st,at)]J(π)=E_τ π [ _t=0^∞γ^tr(s_t,a_t) ]. In this paper, we use π(s)π(s) to denote the deterministic policy and π(⋅|s)π(·|s) to denote the stochastic policy. Unless told otherwise, we use a to denote the 0th0^th-order action and a˙=dadt a= dadt to denote the 1st1^st-order action. In the discrete-time MDP setting, we approximate the time derivatives of the state and actions using finite differences, i.e., s˙≈(s′−s)/Δt s≈(s -s)/ t and a˙≈(a′−a)/Δt a≈(a -a)/ t, with the time interval set to Δt=1 t=1 by default. 3.2 Supervising Both Zeroth and First-Order Action for Imitation Learning The core difficulty to achieve simultaneous 0th0^th and 1st1^st-order action supervision lies in how to properly inject a˙ a information within the original model structure of the policy, since conventional policy models, regardless of deterministic policy π(s)π(s) or stochastic policy π(⋅|s)π(·|s) are designed for the straightforward mapping between state s and the 0th0^th-order action a. However, if we are allowed to take the time derivative of policy π, then we will notice the following property (using the simplest deterministic-policy case as an example): a˙=π˙(s)=dπ(s)dt=dπ(s)ds⋅dsdt=dπ(s)dss˙=∇sπ(s)s˙. a= π(s)= dπ(s)dt= dπ(s)ds· dsdt= dπ(s)ds s= _sπ(s) s. (1) The above equation suggests that we can express the time derivative of policy π as the product of the gradient of policy ∇sπ(s) _sπ(s) and 1st1^st-order state s˙ s. As s˙ s is directly obtainable from the offline dataset during training, this offers us a new perspective to supervise 1st1^st-order action signal directly on the original policy model. In the following, we describe the detailed higher-order action supervision strategies for deterministic, stochastic, and generative flow-matching policies. Deterministic policy. With Eq. (1), incorporating 1st1^st-order action supervision for deterministic policies under the IL setting is straightforward. We can simply add an extra 1st1^st-order loss term ℒa˙detL_ a^det to the original BC loss: minπ(s,a,s′,a′)∼[‖π(s)−a‖2+‖∇sπ(s)s˙−a˙‖2⏟ℒa˙det]. _πE_(s,a,s ,a ) [ \|π(s)-a \|^2+ \| _sπ(s) s- a \|^2_L_ a^det ]. (2) where s˙=s′−s s=s -s and a˙=a′−a a=a -a are directly calculated from dataset samples. The second term ℒa˙detL_ a^det regresses π˙(s) π(s) to the ground truth a˙ a, thereby achieving first-order temporal consistency regularization without the need to make any changes to the original policy network. In practice, we can directly evaluate ∇sπ(s)s˙ _sπ(s) s using Jacobian-vector product (JVP), which is much cheaper to compute than calculating the Jacobian matrix ∇sπ(s) _sπ(s), and is natively supported in modern deep learning frameworks like JAX and PyTorch 2.0. A keen reader might notice that we do not introduce any weighting parameter to the two loss terms in Eq. (2), as we find empirically that keeping the same supervision strength of 0th0^th and 1st1^st-order actions normally provides the best performance. Stochastic policy. Stochastic policies are often modelled as a Gaussian policy π(a|s)=(a,μ(s),diag(σ2(s)))π(a|s)=N(a;μ(s),diag(σ^2(s))) in practice, with μ(s)μ(s), σ2(s)σ^2(s) being implemented as parameterized neural networks outputting mean and dimension-independent variances. They have been widely used in many offline IL and RL algorithms due to their flexibility in capturing dataset distributions. To enable 1st1^st-order action supervision on a stochastic policy, one naïve approach is to supervise the mean policy μ(s)μ(s) with an additional loss term ℒa˙sto-mean=‖∇sμ(s)s˙−a˙‖L_ a^sto-mean= \| _sμ(s) s- a \| as in Eq. (2). We refer to this version as "mean supervision". Below, we also provide a more rigorous version consistent with the stochastic process framework. Specifically, if we model the stochastic component of the Gaussian policy using an Ornstein–Uhlenbeck (OU) (Uhlenbeck and Ornstein, 1930) process, dXt=−λXtdt+2λdWt,X0∼(0,I)dX_t=-λ X_t\,dt+ 2λ\,dW_t,X_0 (0,I), where λ controls the temporal correlation of the process, and WtW_t denotes a d-dimensional standard Wiener process. By the stationarity of the OU process, for any t, we have Xt∼(0,I)X_t (0,I). The action can thus be written as a=μ(s)+σ(s)⊙Xta=μ(s)+σ(s) X_t, where ⊙ denotes element-wise multiplication. By the multi-dimensional Itô formula (Oksendal, 2013), we obtain the stochastic differential equation (SDE) of the action process, da=[μ˙(s)+(σ˙(s)−λσ(s))⊙((a−μ(s))⊘σ(s))]dt+2λσ(s)⊙dWt,da= [ μ(s)+ ( σ(s)-λσ(s) ) ( (a-μ(s) ) σ(s) ) ]dt+ 2λ\,σ(s) _t, (3) where ⊘ denotes element-wise division. The resulting SDE induces a first-order policy distribution over the time derivative of actions (see Appendix C.1 for detailed derivation), π˙(a˙|s,a,s˙)=(a˙,μ˙(s)+(σ˙(s)−λσ(s))⊙((a−μ(s))⊘σ(s)), 2λdiag(σ2(s))), π( a|s,a, s)=N\! ( a;\, μ(s)+ ( σ(s)-λσ(s) ) ( (a-μ(s) ) σ(s) ),\,2 (σ^2(s)) ), (4) where μ˙(s)=∇sμ(s)⋅s˙ μ(s)= _sμ(s)· s and σ˙(s)=∇sσ(s)⋅s˙ σ(s)= _sσ(s)· s can be evaluated in a similar way as in Eq. (1). The above distribution specifies the likelihood of the time derivative of stochastic actions conditioned on the current state, action, and state derivative. We can thus jointly maximize the likelihood of 0th0^th and 1st1^st-order action for stochastic policies under the IL setting as follows: minπ(s,a,s′,a′)∼[−logπ(a|s)−logπ˙(a˙|s,a,s˙)⏟ℒa˙sto]. _πE_(s,a,s ,a ) [- π(a|s)- π( a|s,a, s)_L_ a^sto ]. (5) Beyond conventional action-level maximum likelihood in the first term, the derivative likelihood ℒa˙stoL_ a^sto in Eq.(5) serves as a probabilistic first-order regularizer that aligns the local temporal evolution of sampled actions with the expert trajectory. We provide the full derivation in Appendix C.1. Flow policy. Flow matching provides an effective and powerful framework for generative modeling (Lipman et al., 2023; Liu et al., 2023), and has gained increasing popularity in recent RL and embodied AI models. The central idea is to learn a time-dependent velocity field v:[0,1]×ℝd→ℝdv:[0,1]×R^d ^d (note this "time" is different from the physical time, which we denote as k) that induces a continuous flow ψ:[0,1]×ℝd→ℝdψ:[0,1]×R^d ^d between a base distribution and the target data distribution, where dkψ(k,x)=v(k,ψ(k,x)),ψ(0,x)=x ddkψ(k,x)=v (k,ψ(k,x) ),ψ(0,x)=x. In this paper, we consider the standard linear interpolation path (Lipman et al., 2023; Liu et al., 2023), which defines an interpolation process for k∈[0,1]k∈[0,1] as xk=(1−k)x0+kx1,v=x1−x0x^k=(1-k)x^0+kx^1,v=x^1-x^0 between a noise sample x0x^0 and a data sample x1x^1. For flow policy (Park et al., 2025b), let a0∼(0,I)a^0 (0,I) denote a noise sample and a1∼a^1 denote an action. Given the input state s, the generated action is obtained by integrating the learned velocity field along the interpolation path. Note that under the formalism of flow matching, if we differentiate the generated action a1=a0+∫01v(k,s,ak)ka^1=a^0+ _0^1v(k,s,a^k)\,dk with respect to physical time t, we have a˙1=d∫01v(k,s,ak)kdt=∫01dv(k,s,ak)dtk=∫01∂v(k,s,ak)∂ss˙+∂v(k,s,ak)∂aka˙kk. a^1= d _0^1v(k,s,a^k)\,dkdt= _0^1 dv(k,s,a^k)dt\,dk= _0^1 ∂ v(k,s,a^k)∂ s s\ + ∂ v(k,s,a^k)∂ a^k a^k\ dk. (6) We omit a˙0 a^0 as the noise sample a0a^0 is drawn independently of the physical trajectory, hence calculating the time derivative on noise is meaningless. Equation (6) shows that first-order action evolution can be expressed by integrating v˙(k,s,ak) v(k,s,a^k) over flow time, analogous to the original action generation process. This shared structure motivates supervising v˙(k,s,ak) v(k,s,a^k) in a flow-matching manner by regressing it toward the pointwise target a˙1=a′−a1 a^1=a -a^1. In practice, we approximate v˙(k,s,ak) v(k,s,a^k) with ∂v(k,s,ak)∂ss˙ ∂ v(k,s,a^k)∂ s s. We omit the action-dependent term in Eq. (6) because evaluating it increased computational cost without improving performance, as discussed in Appendix B.2. Therefore, the new objective for flow policies with higher-order supervision can be given as: minπa0∼(0,I),(s,a1,s′,a′)∼,k∈[0,1][‖v(k,s,ak)−(a1−a0)‖+‖∂v(k,s,ak)∂ss˙−a˙1‖⏟ℒa˙flow]. _πE_a^0 (0,I),\,(s,a^1,s ,a ) ,\,k∈[0,1] [ \|v(k,s,a^k)-(a^1-a^0) \|+ \| ∂ v(k,s,a^k)∂ s s- a^1 \|_L_ a^flow ]. (7) Standard flow matching mainly supervises transformation along the probability path at time k, while leaving the physical-time response of the conditional velocity field largely unconstrained. In Eq. (7), the additional term ℒa˙flowL_ a^flow completes this missing piece by aligning the state-directional change of the velocity field with the observed action evolution. We provide further discussion in Appendix C.2. 3.3 Theoretical Analysis With the additional first-order supervision, we can show in the following theorems that this scheme can lead to a tighter error bound for imitation learning under a set of commonly used assumptions (see Appendix D.2). Specifically, consider the learned policy π and expert policy π∗π^* start at the same state s(0)s(0) (we denote s(t)s(t) as the state in continuous time t), and define the policy error as e(s)=π(s)−π∗(s)e(s)=π(s)-π^*(s). We can show that the policy error can be better bounded with stronger 1st1^st-order supervision (see Appendix D.4 for the proof): Theorem 1. Under Assumption 1 in Appendix D.2, the policy error e(s∗(t))e(s^*(t)) satisfies: ‖e(s∗(t))‖≤‖e(s∗(0))‖+tmax0≤τ≤t‖∇sπ(s∗(τ))s˙∗(τ)−π˙∗(s∗(τ))‖.\|e(s^*(t))\|≤\|e(s^*(0))\|+t\, _0≤τ≤ t \| _sπ(s^*(τ)) s^*(τ)- π^*(s^*(τ)) \|. Theorem 1 shows that the growth of policy error e(s∗)e(s^*) on expert trajectory s∗s^* is governed by its initial value and the accumulated 1st1^st-order discrepancy between the learned and expert policies. Note that the standard BC objective does not supervise the second term, hence could lead to substantially increased errors. As Theorem 1 only provides analysis on the per-state policy error, next, we provide a stronger conclusion to bound the state deviation δ(t)=s(t)−s∗(t)δ(t)=s(t)-s^*(t) between states from policy rollout s and expert trajectories s∗s^*, which captures the closed-loop long-term error accumulation. Theorem 2. Under Assumptions 1–4 in Appendix D.2, for δ(0)=0δ(0)=0, the state deviation δ(t)δ(t) satisfies, ‖δ(t)‖≤Laβ(eβt−1)max0≤τ≤t‖e(s∗(τ))‖.\|δ(t)\|≤ L_aβ (e^β t-1 ) _0≤τ≤ t\|e(s^*(τ))\|. The detailed proof and exact forms of constants LaL_a and β are provided in Appendix D.5. Theorem 2 decomposes closed-loop policy rollout state deviation into two parts: the policy mismatch error e(s∗)e(s^*) measured on the expert trajectory s∗s^*, and the amplification gain Laβ(eβt−1) L_aβ(e^β t-1) contributed by the environment dynamics, capturing the error amplification due to closed-loop policy-environment interaction. Again, we can see that the long-term state deviation can also be better controlled under higher-order supervision, due to the tighter error bound ‖e(s∗(t))‖\|e(s^*(t))\| obtained in Theorem 1. 4 Higher-Order Policy Optimization in Offline RL The extended policy in Section 3 not only enables higher-order action supervision, but also provides a practical way to evaluate action derivatives π˙θ(s) π_θ(s), a quantity that existing offline RL methods never explicitly use. We now show that it has a natural place in critic learning through the Hamilton-Jacobi-Bellman based continuous-time RL framework, where value learning depends not only on scalar Bellman consistency, but also on how the value changes along the local state-action flow. 4.1 HJB Equation and Value Learning with First-Order Supervision In the HJB-based continuous-time RL framework, the Q-function is defined by (Kim et al., 2021): QHJB(s,a):=max∫0∞a∈e−γtr(s(t),a(t))t.Q^HJB(s,a):= _a _0^∞e^-γ tr(s(t),a(t))dt. (8) The dynamic programming equation of the above Q-function is expressed as an HJB equation, which is derived by using the Taylor expansion: γQHJB(s,a)=maxa∈[r(s,a)+∇sQHJB(s,a)s˙+∇aQHJB(s,a)a˙]γ Q^HJB(s,a)= _a [r(s,a)+ _sQ^HJB(s,a) s+ _aQ^HJB(s,a) a ] (9) Unlike the typical discrete-time Bellman equation, which compresses value learning into a scalar one-step regression, Eq. (9) reveals what is missing in existing offline RL methods. The value function may match 0th0^th-order Bellman targets, yet its gradient is never required to align with the variation direction of states and actions in time. This makes the learned value landscape numerically plausible but structurally under-specified, especially for later policy improvement. However, with the extended policy class introduced in Section 3, we can easily obtain the time derivative of actions through the learned policy, i.e., a˙=π˙θ(s)=∇sπθ(s)⋅s˙ a= π_θ(s)= _s _θ(s)· s. This allows us to jointly optimize the value function with policy’s higher-order information during Bellman update, which leads to the following objective: Q=argminQ(s,a,r,s′,a′)∼[ Q= _QE_(s,a,r,s ,a ) [ (πQ^(s,a)−Q(s,a))2 (T_π Q(s,a)-Q(s,a) )^2 (10) −λQ(∇sQ(s,a)s˙‖∇sQ(s,a)‖‖s˙‖+∇aQ(s,a)π˙θ(s)‖∇aQ(s,a)‖‖π˙θ(s)‖)]. [rgb]0,0,1 - _Q ( _sQ(s,a) s\| _sQ(s,a)\|\| s\|+ _aQ(s,a) π_θ(s)\| _aQ(s,a)\|\| π_θ(s)\| ) ]. As s˙=s′−s s=s -s can be directly calculated from the dataset during offline learning, we can actually seamlessly transform the original value learning scheme of an existing offline RL algorithm into a higher-order supervision version by adding the above blue terms, with πT_π denoting the specific Bellman operator used in the offline RL algorithm. Moreover, to improve numerical stability and to prevent overly large gradient magnitudes of the first-order terms, we normalize the two HJB directional terms in Eq. (10), similar to existing HJB-based methods Kim et al. (2021); Lien et al. (2024). 4.2 Higher-Order Policy Learning With the Q-function learned with 1st1^st-order supervision, we can also transform the policy optimization procedure of an existing offline RL algorithm through a plug-and-play modification. This can be achieved by simply adding an additional 1st1^st-order imitation loss ℒa˙L_ a as defined in Section 3, π=argmax(s,a,s′,a′)∼,a^∼πQ(s,a^)−ℛπ(s)−λa˙ℒa˙.π= π \;E_(s,a,s ,a ) , a πQ(s, a)-R_π(s) [rgb]0,0,1- _ aL_ a. (11) where we use Q(s,a^)−ℛπ(s)Q(s, a)-R_π(s) to represent the general policy optimization objective of existing offline RL algorithms, which typically optimize a policy by maximizing the Q-function with some behavior regularization term ℛπ(s)R_π(s). ℛπ(s)R_π(s) can either be an explicit policy constraint, such as the BC term ℛπ(s)=(s,a)∼‖π(s)−a‖2R_π(s)=E_(s,a) \|π(s)-a\|^2 in TD3+BC (Fujimoto and Gu, 2021), or an implicit behavior regularization scheme as adopted in many algorithms Kostrikov et al. (2022); Xu et al. (2023); Wang et al. (2024); Zheng et al. (2024); Xu et al. (2026); Liang et al. (2026), which results in an advantage-weighted regression type of learning objective. Depending on the actual policy class used in the original offline RL algorithm, one can flexibly choose ℒa˙L_ a to be either ℒa˙detL_ a^det, ℒa˙stoL_ a^sto, or ℒa˙flowL_ a^flow in Section 3. We introduce a hyperparameter λa˙ _ a to balance the regularization strength of the original policy optimization objective and the first-order supervision term. Both the first-order value supervision term and the ℒa˙L_ a term in policy optimization introduce minimal modifications to the original offline RL algorithm, but as we show in the experiments, they can lead to substantial performance gains, as the policy is taught not only what to do, but how to evolve reasonably over time. 5 Experiments In this section, we empirically evaluate our proposed policy class. We first integrate it into a range of representative offline RL algorithms and benchmark the resulting variants against their original counterparts. We then conduct ablation studies to isolate the contributions of the first-order policy constraint and the first-order value consistency regularizer. 5.1 Experimental Setup Benchmarks. To comprehensively evaluate our proposed framework, we conduct experiments on two widely adopted offline RL benchmarks. We first evaluate on OGBench (Park et al., 2025a), a recent benchmark featuring high-dimensional state spaces, multimodal behaviors, and sparse rewards. We adopt its reward-based "-singletask" variants so that standard reward-maximizing offline RL algorithms can be directly applied. We further conduct experiments on D4RL (Fu et al., 2020) under a more challenging low-data setting with only 10k transitions per task, which better exposes performance gaps and OOD generalization since recent offline RL methods have largely saturated the D4RL benchmark under the full-dataset setting. The benchmark details and task descriptions are presented in Appendix E.1. Comparison methods. We integrate our proposed policy class into representative offline RL algorithms spanning three policy families (deterministic, stochastic, and flow-matching), and compare each base algorithm with its first-order-supervised variant. For deterministic policies, we adopt TD3+BC (Fujimoto and Gu, 2021) as the base algorithm. For stochastic policies, we instantiate the two strategies introduced in Section 3.2: a simplified mean supervision variant, which applies first-order supervision only to the Gaussian mean and is built upon ReBRAC (Tarasov et al., 2023) (denoted “mean sup.” in Table 2), and a standard variant based on the likelihood objective in Eq. 5 and built upon IQL (Kostrikov et al., 2022) (denoted “standard”). For flow-matching policies, we adopt IFQL, an algorithmic variant of FQL (Park et al., 2025b) that enables our first-order supervision to be directly applied to the flow policy without any additional policy-distillation step. Experimental details are provided in Appendix E.2. 5.2 Evaluation Results Table 1: Average normalized scores on 10 OGBench task categories, each aggregating 5 sub-tasks over 5 random seeds. Values after ± denote standard deviations. The ↑ [rgb]0.0195,0.6523,0.082 marks indicate improvements greater than 1.01.0, whereas ↑ [rgb]0.5,0.5,0.5 denotes minor improvements. Full per-task results are presented in Table 4. Det. Policy Sto. Policy (mean sup.) Sto. Policy (standard) Flow Policy Task Category (Avg. of 5 tasks) TD3+BC TD3+BC+ours ReBRAC ReBRAC+ours IQL IQL+ours IFQL IFQL+ours antmaze-large-navigate 83±483 ± 4 98±3(↑15)98 ± 3( [rgb]0.0195,0.6523,0.082 \!15) 81±581 ± 5 98±2(↑17)98 ± 2( [rgb]0.0195,0.6523,0.082 \!17) 53±353 ± 3 68±5(↑15)68 ± 5( [rgb]0.0195,0.6523,0.082 \!15) 28±1328 ± 13 36±14(↑8)36 ± 14( [rgb]0.0195,0.6523,0.082 \!8) antmaze-giant-navigate 42±842 ± 8 65±10(↑23)65 ± 10( [rgb]0.0195,0.6523,0.082 \!23) 26±826 ± 8 61±12(↑35)61 ± 12( [rgb]0.0195,0.6523,0.082 \!35) 4±14 ± 1 9±1(↑5)9 ± 1( [rgb]0.0195,0.6523,0.082 \!5) 3±23 ± 2 4±3(↑1)4 ± 3( [rgb]0.0195,0.6523,0.082 \!1) humanoidmaze-medium-navigate 34±834 ± 8 42±5(↑8)42 ± 5( [rgb]0.0195,0.6523,0.082 \!8) 22±822 ± 8 55±10(↑33)55 ± 10( [rgb]0.0195,0.6523,0.082 \!33) 33±233 ± 2 36±3(↑3)36 ± 3( [rgb]0.0195,0.6523,0.082 \!3) 60±1660 ± 16 64±13(↑4)64 ± 13( [rgb]0.0195,0.6523,0.082 \!4) humanoidmaze-large-navigate 2±12 ± 1 4±1(↑2)4 ± 1( [rgb]0.0195,0.6523,0.082 \!2) 2±12 ± 1 4±2(↑2)4 ± 2( [rgb]0.0195,0.6523,0.082 \!2) 2±12 ± 1 3±1(↑1)3 ± 1( [rgb]0.0195,0.6523,0.082 \!1) 11±211 ± 2 12±3(↑1)12 ± 3( [rgb]0.0195,0.6523,0.082 \!1) antsoccer-arena-navigate 13±313 ± 3 16±3(↑3)16 ± 3( [rgb]0.0195,0.6523,0.082 \!3) 0±00 ± 0 16±4(↑16)16 ± 4( [rgb]0.0195,0.6523,0.082 \!16) 8±28 ± 2 23±3(↑15)23 ± 3( [rgb]0.0195,0.6523,0.082 \!15) 33±633 ± 6 40±13(↑7)40 ± 13( [rgb]0.0195,0.6523,0.082 \!7) cube-single-play 88±588 ± 5 91±4(↑3)91 ± 4( [rgb]0.0195,0.6523,0.082 \!3) 91±291 ± 2 93±2(↑2)93 ± 2( [rgb]0.0195,0.6523,0.082 \!2) 83±383 ± 3 92±3(↑9)92 ± 3( [rgb]0.0195,0.6523,0.082 \!9) 79±279 ± 2 95±7(↑16)95 ± 7( [rgb]0.0195,0.6523,0.082 \!16) cube-double-play 7±17 ± 1 12±1(↑5)12 ± 1( [rgb]0.0195,0.6523,0.082 \!5) 12±112 ± 1 24±3(↑12)24 ± 3( [rgb]0.0195,0.6523,0.082 \!12) 7±17 ± 1 8±2(↑1)8 ± 2( [rgb]0.0195,0.6523,0.082 \!1) 14±314 ± 3 42±15(↑28)42 ± 15( [rgb]0.0195,0.6523,0.082 \!28) scene-play 44±344 ± 3 47±2(↑3)47 ± 2( [rgb]0.0195,0.6523,0.082 \!3) 41±341 ± 3 47±4(↑6)47 ± 4( [rgb]0.0195,0.6523,0.082 \!6) 28±128 ± 1 33±3(↑5)33 ± 3( [rgb]0.0195,0.6523,0.082 \!5) 30±330 ± 3 56±7(↑26)56 ± 7( [rgb]0.0195,0.6523,0.082 \!26) puzzle-3x3-play 27±327 ± 3 29±1(↑2)29 ± 1( [rgb]0.0195,0.6523,0.082 \!2) 21±121 ± 1 24±2(↑3)24 ± 2( [rgb]0.0195,0.6523,0.082 \!3) 9±19 ± 1 12±4(↑3)12 ± 4( [rgb]0.0195,0.6523,0.082 \!3) 19±119 ± 1 43±15(↑24)43 ± 15( [rgb]0.0195,0.6523,0.082 \!24) puzzle-4x4-play 6±16 ± 1 9±1(↑3)9 ± 1( [rgb]0.0195,0.6523,0.082 \!3) 14±114 ± 1 17±3(↑3)17 ± 3( [rgb]0.0195,0.6523,0.082 \!3) 7±17 ± 1 13±2(↑6)13 ± 2( [rgb]0.0195,0.6523,0.082 \!6) 25±525 ± 5 40±12(↑15)40 ± 12( [rgb]0.0195,0.6523,0.082 \!15) Average Score 35 41(↑6)( [rgb]0.0195,0.6523,0.082 \!6) 31 44(↑13)( [rgb]0.0195,0.6523,0.082 \!13) 23 30(↑7)( [rgb]0.0195,0.6523,0.082 \!7) 30 43(↑13)( [rgb]0.0195,0.6523,0.082 \!13) Table 2: Average normalized scores on D4RL MuJoCo locomotion tasks with reduced-size (10k) datasets. Standard deviations are reported after the ± sign. The ↑ [rgb]0.0195,0.6523,0.082 symbol indicates a performance increase greater than 1.01.0, whereas ↑ [rgb]0.5,0.5,0.5 denotes minor improvements. All evaluation results are averaged over 5 random seeds. Det. Policy Sto. Policy (mean sup.) Sto. Policy (standard) Flow Policy Task Category TD3+BC TD3+BC+ours ReBRAC ReBRAC+ours IQL IQL+ours IFQL IFQL+ours Hopper-m 40.1±18.640.1 ± 18.6 44.1±12.7(↑4.0)44.1 ± 12.7( [rgb]0.0195,0.6523,0.082 \!4.0) 9.8±3.19.8 ± 3.1 32.4±11.0(↑22.6)32.4 ± 11.0( [rgb]0.0195,0.6523,0.082 \!22.6) 47.7±1.047.7 ± 1.0 48.5±2.5(↑0.8)48.5 ± 2.5( [rgb]0.5,0.5,0.5 \!0.8) 50.8±2.350.8 ± 2.3 51.3±2.9(↑0.5)51.3 ± 2.9( [rgb]0.5,0.5,0.5 \!0.5) Hopper-mr 7.3±6.17.3 ± 6.1 22.3±3.3(↑15.0)22.3 ± 3.3( [rgb]0.0195,0.6523,0.082 \!15.0) 19.0±1.019.0 ± 1.0 24.9±2.8(↑5.9)24.9 ± 2.8( [rgb]0.0195,0.6523,0.082 \!5.9) 19.3±3.019.3 ± 3.0 19.9±2.0(↑0.6)19.9 ± 2.0( [rgb]0.5,0.5,0.5 \!0.6) 23.1±1.423.1 ± 1.4 28.2±0.4(↑5.1)28.2 ± 0.4( [rgb]0.0195,0.6523,0.082 \!5.1) Hopper-me 17.8±7.917.8 ± 7.9 42.3±10.8(↑24.5)42.3 ± 10.8( [rgb]0.0195,0.6523,0.082 \!24.5) 36.2±3.136.2 ± 3.1 43.0±3.3(↑6.8)43.0 ± 3.3( [rgb]0.0195,0.6523,0.082 \!6.8) 34.3±8.734.3 ± 8.7 42.3±3.3(↑8.0)42.3 ± 3.3( [rgb]0.0195,0.6523,0.082 \!8.0) 45.7±1.945.7 ± 1.9 47.9±1.7(↑2.2)47.9 ± 1.7( [rgb]0.0195,0.6523,0.082 \!2.2) Hopper-e 23.2±18.223.2 ± 18.2 93.5±13.6(↑70.3)93.5 ± 13.6( [rgb]0.0195,0.6523,0.082 \!70.3) 59.2±3.559.2 ± 3.5 97.2±5.9(↑38.0)97.2 ± 5.9( [rgb]0.0195,0.6523,0.082 \!38.0) 55.7±22.555.7 ± 22.5 99.5±3.2(↑43.8)99.5 ± 3.2( [rgb]0.0195,0.6523,0.082 \!43.8) 78.5±13.678.5 ± 13.6 96.9±4.9(↑18.4)96.9 ± 4.9( [rgb]0.0195,0.6523,0.082 \!18.4) Halfcheetah-m 16.4±10.216.4 ± 10.2 31.5±2.4(↑15.1)31.5 ± 2.4( [rgb]0.0195,0.6523,0.082 \!15.1) 37.1±1.337.1 ± 1.3 38.9±0.5(↑1.8)38.9 ± 0.5( [rgb]0.0195,0.6523,0.082 \!1.8) 38.2±1.438.2 ± 1.4 39.1±0.1(↑0.9)39.1 ± 0.1( [rgb]0.5,0.5,0.5 \!0.9) 38.9±1.438.9 ± 1.4 39.5±1.5(↑0.6)39.5 ± 1.5( [rgb]0.5,0.5,0.5 \!0.6) Halfcheetah-mr 17.9±9.517.9 ± 9.5 21.2±7.1(↑3.3)21.2 ± 7.1( [rgb]0.0195,0.6523,0.082 \!3.3) 13.0±5.713.0 ± 5.7 15.1±4.1(↑2.1)15.1 ± 4.1( [rgb]0.0195,0.6523,0.082 \!2.1) 28.2±1.328.2 ± 1.3 29.5±0.6(↑1.3)29.5 ± 0.6( [rgb]0.0195,0.6523,0.082 \!1.3) 29.6±1.029.6 ± 1.0 31.1±1.2(↑1.5)31.1 ± 1.2( [rgb]0.0195,0.6523,0.082 \!1.5) Halfcheetah-me 15.4±10.715.4 ± 10.7 35.6±2.4(↑20.2)35.6 ± 2.4( [rgb]0.0195,0.6523,0.082 \!20.2) 12.2±6.712.2 ± 6.7 19.7±1.7(↑7.5)19.7 ± 1.7( [rgb]0.0195,0.6523,0.082 \!7.5) 34.4±2.234.4 ± 2.2 36.8±1.5(↑2.4)36.8 ± 1.5( [rgb]0.0195,0.6523,0.082 \!2.4) 40.2±0.740.2 ± 0.7 40.4±1.3(↑0.2)40.4 ± 1.3( [rgb]0.5,0.5,0.5 \!0.2) Halfcheetah-e 1.7±3.31.7 ± 3.3 3.9±2.1(↑2.2)3.9 ± 2.1( [rgb]0.0195,0.6523,0.082 \!2.2) 2.3±0.92.3 ± 0.9 4.0±2.3(↑1.7)4.0 ± 2.3( [rgb]0.0195,0.6523,0.082 \!1.7) 1.2±1.01.2 ± 1.0 6.7±2.6(↑5.5)6.7 ± 2.6( [rgb]0.0195,0.6523,0.082 \!5.5) 18.8±7.818.8 ± 7.8 35.9±6.2(↑17.1)35.9 ± 6.2( [rgb]0.0195,0.6523,0.082 \!17.1) Walker2d-m 7.4±13.17.4 ± 13.1 41.1±10.8(↑33.7)41.1 ± 10.8( [rgb]0.0195,0.6523,0.082 \!33.7) 21.2±4.821.2 ± 4.8 56.9±4.3(↑35.7)56.9 ± 4.3( [rgb]0.0195,0.6523,0.082 \!35.7) 34.6±9.534.6 ± 9.5 45.0±4.8(↑10.4)45.0 ± 4.8( [rgb]0.0195,0.6523,0.082 \!10.4) 50.6±6.850.6 ± 6.8 56.7±4.7(↑6.1)56.7 ± 4.7( [rgb]0.0195,0.6523,0.082 \!6.1) Walker2d-mr 5.7±5.85.7 ± 5.8 23.2±5.7(↑17.5)23.2 ± 5.7( [rgb]0.0195,0.6523,0.082 \!17.5) 10.0±2.210.0 ± 2.2 22.8±5.5(↑12.8)22.8 ± 5.5( [rgb]0.0195,0.6523,0.082 \!12.8) 10.7±11.910.7 ± 11.9 17.6±4.0(↑6.9)17.6 ± 4.0( [rgb]0.0195,0.6523,0.082 \!6.9) 17.5±5.417.5 ± 5.4 29.0±2.8(↑11.5)29.0 ± 2.8( [rgb]0.0195,0.6523,0.082 \!11.5) Walker2d-me 7.9±9.17.9 ± 9.1 37.8±12.3(↑29.9)37.8 ± 12.3( [rgb]0.0195,0.6523,0.082 \!29.9) 17.6±8.617.6 ± 8.6 58.9±11.8(↑41.3)58.9 ± 11.8( [rgb]0.0195,0.6523,0.082 \!41.3) 29.1±16.929.1 ± 16.9 41.2±9.2(↑12.1)41.2 ± 9.2( [rgb]0.0195,0.6523,0.082 \!12.1) 38.8±3.738.8 ± 3.7 57.6±10.4(↑18.8)57.6 ± 10.4( [rgb]0.0195,0.6523,0.082 \!18.8) Walker2d-e 2.7±3.12.7 ± 3.1 82.3±10.8(↑79.5)82.3 ± 10.8( [rgb]0.0195,0.6523,0.082 \!79.5) 23.6±9.423.6 ± 9.4 83.9±6.8(↑60.3)83.9 ± 6.8( [rgb]0.0195,0.6523,0.082 \!60.3) 43.1±6.043.1 ± 6.0 84.8±13.9(↑41.7)84.8 ± 13.9( [rgb]0.0195,0.6523,0.082 \!41.7) 95.2±11.895.2 ± 11.8 104.9±3.7(↑9.7)104.9 ± 3.7( [rgb]0.0195,0.6523,0.082 \!9.7) Average Score 13.613.6 39.9(↑26.3)39.9( [rgb]0.0195,0.6523,0.082 \!26.3) 21.821.8 41.5(↑19.7)41.5( [rgb]0.0195,0.6523,0.082 \!19.7) 31.431.4 42.6(↑10.2)42.6( [rgb]0.0195,0.6523,0.082 \!10.2) 44.044.0 51.6(↑7.6)51.6( [rgb]0.0195,0.6523,0.082 \!7.6) Evaluation results on OGBench tasks. Table 2 reports the average normalized scores on 50 diverse OGBench tasks, with complete per-task results provided in Appendix B.1. Our first-order action supervision yields consistent performance gains across all three policy paradigms. On control-sensitive navigation tasks, these gains underline the importance of temporally consistent actions, since small action fluctuations can accumulate over time and easily lead to collisions or failed navigation. For instance, applying our scheme to ReBRAC raises the score from 26 to 61 on antmaze-giant-navigate and from 22 to 55 on humanoidmaze-medium-navigate, both revealing over 100% relative improvement. The performance gains are even more pronounced beyond navigation. On object-centric play tasks, applying our proposed scheme to IFQL improves cube-double-play from 14 to 42 (a 200% relative improvement) and scene-play from 30 to 56 (an 87% relative improvement). In particular, on several challenging sub-tasks where the IFQL baseline completely fails with a score of exactly 0, equipping it with our scheme lifts the score to clearly non-trivial levels, e.g., from 0 to 88 on scene-play-singletask-task2-v0, expanding the set of tasks on which offline policy learning is feasible at all (see Table 4 in Appendix B.1 for full per-task results). Improved OOD generalization under small-dataset settings. Our first-order action supervision is particularly beneficial in small-dataset regimes, where standard offline RL baselines often struggle to recover the expert’s robust control behavior from sparse state-action samples, even though such samples still encode rich local variations along expert trajectories. Table 2 reports the D4RL locomotion results on 10k datasets. By supervising the policy’s first-order action derivative, our scheme enables the local dynamics and control structure of the expert behavior to be effectively recovered from an extremely limited dataset, leading to substantial performance gains. For instance, on hopper-expert-v2, IQL improves from 55.755.7 to 99.599.5, nearly recovering its full-dataset performance. More notably, on walker2d-expert-v2, TD3+BC collapses to a score of only 2.72.7, indicating severe extrapolation failure under such a tiny-data regime; equipping it with our scheme recovers the score to 82.382.3, an absolute gain of 79.579.5 points. In addition to the score improvements, our scheme also yields noticeably smaller standard deviations across most tasks, indicating more stable and reproducible policy learning under small-dataset settings. Additional limited-data imitation learning results, as well as full-data D4RL results, are provided in Appendix B.1. Benefits of utilizing first-order supervision. Figure 2 provides a mechanistic view of how our first-order action supervision improves performance. With only 10k transitions, deterministic BC produces a dispersed distribution of ‖a˙‖|| a||, reflecting erratic action changes during execution. In contrast, our scheme yields a compact distribution concentrated around small ‖a˙‖|| a||, indicating smoother and temporally consistent action evolution. This pattern is even more pronounced for TD3+BC in Figure 3, where its plain MSE penalty leaves the policy vulnerable to action jittering, whereas our scheme directly aligns action changes with the local expert dynamics. Overall, by constraining how actions evolve along trajectories, first-order supervision yields noticeably smoother and more robust policies. We conduct additional analysis results in Appendix B.1. Figure 2: Comparison of vanilla deterministic BC and our 1st1^st-order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation ‖a˙‖=‖a′−a‖|| a||=||a -a|| (a is joint torque) are aggregated from 10 trajectories. Figure 3: Comparison of TD3+BC and our 1st1^st-order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation ‖a˙‖=‖a′−a‖|| a||=||a -a|| (a is joint torque) are aggregated from 10 trajectories. 5.3 Ablation experiments Table 3: Comparison with action-smoothness approaches and explicit first-order action prediction on D4RL MuJoCo locomotion tasks with reduced-size (10k) datasets. All methods use TD3+BC as the shared backbone. Arrows show score differences relative to TD3+BC on the same task: ↑ [rgb]0.0195,0.6523,0.082 indicates a gain of at least 1.01.0, ↑ [rgb]0.5,0.5,0.5 a smaller positive gain, and ↓ [rgb]1,0,0 a decrease. No arrow indicates no change. Task TD3+BC +CAPS +L2C2 +LipsNet +Extra a˙ a head TD3+BC+Ours Hopper-m 40.1±18.640.1 ± 18.6 42.6±5.1(↑2.5)42.6 ± 5.1( [rgb]0.0195,0.6523,0.082 \!2.5) 42.7±7.5(↑2.6)42.7 ± 7.5( [rgb]0.0195,0.6523,0.082 \!2.6) 43.2±3.6(↑3.1)43.2 ± 3.6( [rgb]0.0195,0.6523,0.082 \!3.1) 41.5±7.8(↑1.4)41.5 ± 7.8( [rgb]0.0195,0.6523,0.082 \!1.4) 44.1±12.7(↑4.0)44.1 ± 12.7( [rgb]0.0195,0.6523,0.082 \!4.0) Hopper-me 17.8±7.917.8 ± 7.9 39.6±8.2(↑21.8)39.6 ± 8.2( [rgb]0.0195,0.6523,0.082 \!21.8) 39.8±8.3(↑22.0)39.8 ± 8.3( [rgb]0.0195,0.6523,0.082 \!22.0) 32.8±18.5(↑15.0)32.8 ± 18.5( [rgb]0.0195,0.6523,0.082 \!15.0) 26.6±8.6(↑8.8)26.6 ± 8.6( [rgb]0.0195,0.6523,0.082 \!8.8) 42.3±10.8(↑24.5)42.3 ± 10.8( [rgb]0.0195,0.6523,0.082 \!24.5) Hopper-e 23.2±18.223.2 ± 18.2 57.1±29.7(↑33.9)57.1 ± 29.7( [rgb]0.0195,0.6523,0.082 \!33.9) 55.2±34.5(↑32.0)55.2 ± 34.5( [rgb]0.0195,0.6523,0.082 \!32.0) 16.9±2.8(↓6.3)16.9 ± 2.8( [rgb]1,0,0 \!6.3) 30.6±15.1(↑7.4)30.6 ± 15.1( [rgb]0.0195,0.6523,0.082 \!7.4) 93.5±13.6(↑70.3)93.5 ± 13.6( [rgb]0.0195,0.6523,0.082 \!70.3) HalfCheetah-m 16.4±10.216.4 ± 10.2 29.6±5.6(↑13.2)29.6 ± 5.6( [rgb]0.0195,0.6523,0.082 \!13.2) 30.3±7.1(↑13.9)30.3 ± 7.1( [rgb]0.0195,0.6523,0.082 \!13.9) 6.6±3.0(↓9.8)6.6 ± 3.0( [rgb]1,0,0 \!9.8) 28.1±4.5(↑11.7)28.1 ± 4.5( [rgb]0.0195,0.6523,0.082 \!11.7) 31.5±2.4(↑15.1)31.5 ± 2.4( [rgb]0.0195,0.6523,0.082 \!15.1) HalfCheetah-me 15.4±10.715.4 ± 10.7 19.8±3.4(↑4.4)19.8 ± 3.4( [rgb]0.0195,0.6523,0.082 \!4.4) 28.6±3.8(↑13.2)28.6 ± 3.8( [rgb]0.0195,0.6523,0.082 \!13.2) 3.3±2.2(↓12.1)3.3 ± 2.2( [rgb]1,0,0 \!12.1) 20.6±3.5(↑5.2)20.6 ± 3.5( [rgb]0.0195,0.6523,0.082 \!5.2) 35.6±2.4(↑20.2)35.6 ± 2.4( [rgb]0.0195,0.6523,0.082 \!20.2) HalfCheetah-e 1.7±3.31.7 ± 3.3 1.7±1.61.7 ± 1.6 2.4±1.9(↑0.7)2.4 ± 1.9( [rgb]0.5,0.5,0.5 \!0.7) 0.8±0.6(↓0.9)0.8 ± 0.6( [rgb]1,0,0 \!0.9) 0.2±1.8(↓1.5)0.2 ± 1.8( [rgb]1,0,0 \!1.5) 3.9±2.1(↑2.2)3.9 ± 2.1( [rgb]0.0195,0.6523,0.082 \!2.2) Walker2d-m 7.4±13.17.4 ± 13.1 24.5±3.6(↑17.1)24.5 ± 3.6( [rgb]0.0195,0.6523,0.082 \!17.1) 20.4±4.1(↑13.0)20.4 ± 4.1( [rgb]0.0195,0.6523,0.082 \!13.0) 8.9±2.8(↑1.5)8.9 ± 2.8( [rgb]0.0195,0.6523,0.082 \!1.5) 8.6±6.5(↑1.2)8.6 ± 6.5( [rgb]0.0195,0.6523,0.082 \!1.2) 41.1±10.8(↑33.7)41.1 ± 10.8( [rgb]0.0195,0.6523,0.082 \!33.7) Walker2d-me 7.9±9.17.9 ± 9.1 17.8±10.1(↑9.9)17.8 ± 10.1( [rgb]0.0195,0.6523,0.082 \!9.9) 23.3±9.8(↑15.4)23.3 ± 9.8( [rgb]0.0195,0.6523,0.082 \!15.4) 5.9±3.6(↓2.0)5.9 ± 3.6( [rgb]1,0,0 \!2.0) 8.3±6.1(↑0.4)8.3 ± 6.1( [rgb]0.5,0.5,0.5 \!0.4) 37.8±12.3(↑29.9)37.8 ± 12.3( [rgb]0.0195,0.6523,0.082 \!29.9) Walker2d-e 2.7±3.12.7 ± 3.1 3.9±2.1(↑1.2)3.9 ± 2.1( [rgb]0.0195,0.6523,0.082 \!1.2) 7.4±3.6(↑4.7)7.4 ± 3.6( [rgb]0.0195,0.6523,0.082 \!4.7) 13.8±6.8(↑11.1)13.8 ± 6.8( [rgb]0.0195,0.6523,0.082 \!11.1) 3.9±1.6(↑1.2)3.9 ± 1.6( [rgb]0.0195,0.6523,0.082 \!1.2) 82.3±10.8(↑79.5)82.3 ± 10.8( [rgb]0.0195,0.6523,0.082 \!79.5) Average Score 14.714.7 26.3(↑11.6)26.3( [rgb]0.0195,0.6523,0.082 \!11.6) 27.8(↑13.1)27.8( [rgb]0.0195,0.6523,0.082 \!13.1) 14.714.7 18.7(↑4.0)18.7( [rgb]0.0195,0.6523,0.082 \!4.0) 45.8(↑31.1)45.8( [rgb]0.0195,0.6523,0.082 \!31.1) Comparing with alternative action-consistency approaches. To further benchmark our method, we compare against two categories of representative alternatives, using TD3+BC as the shared backbone. The first category enforces action smoothness, including the penalty-based methods CAPS (Mysore et al., 2021) and L2C2 (Kobayashi, 2022), which encourage local action consistency on perturbed or interpolated states, and the architectural method LipsNet (Song et al., 2023), which explicitly bounds the network’s Lipschitz constant. The second category explicitly predicts first-order actions, for which we construct an auxiliary-head variant of TD3+BC, denoted as +Extra a˙ a head, where the actor is equipped with a dedicated prediction head, i.e., learning a policy π(s)=(a,a˙)π(s)=(a, a), and the predicted a˙ a is also utilized during inference. All baselines follow their original configurations or exactly the same training configurations as our method. For action-smoothness approaches, as reported in Table 3 (+CAPS/+L2C2/+LipsNet columns), our method achieves consistent improvements across all tasks. Penalty-based methods were originally designed for online settings; in offline RL, enforcing smoothness on synthetic, unobserved states risks OOD extrapolation errors, and this degradation is particularly pronounced on high-quality (-e) datasets, whose narrow and near-deterministic trajectories conflict with isotropic perturbations or uniform interpolations. Meanwhile, strictly constraining the network architecture (LipsNet) limits model expressivity, making it difficult to fit offline data from limited samples. For explicit first-order action prediction, the auxiliary-head variant brings only marginal improvements over the backbone and substantially underperforms our method (+Extra a˙ a head column). We attribute this gap to two factors. First, predicting two distinct action outputs converts policy optimization into a multi-task learning problem, which introduces conflicting gradient directions between the zeroth-order and first-order action objectives and requires extra hyperparameter tuning to balance their trade-off. Second, the dedicated head doubles the output dimension and requires structural modification of the base network, losing compatibility with off-the-shelf policy architectures. In contrast, our method requires zero architectural changes, acting as a lightweight, plug-and-play module that preserves temporal consistency without sacrificing network capacity. Figure 4: Ablation study on value regularization weight λQ _Q and 1st1^st-order action supervision weight λa˙ _ a of IFQL. Effectiveness of value and policy constraints. Figure 4 isolates the respective contributions of the two components in our IFQL instantiation on OGBench by sweeping one weight while strictly setting the other to zero. The weight λQ _Q controls the first-order value-consistency regularizer; sweeping it over 0,0.5,1\0,0.5,1\ reveals steady performance gains, indicating that aligning the value landscape with the local state-action flow provides a highly beneficial training signal. Similarly, increasing the first-order action supervision weight λa˙ _ a from the same grid 0,0.5,1\0,0.5,1\ consistently improves performance over standard IFQL. This confirms that first-order action supervision offers crucial guidance beyond mere zeroth-order action fitting, even when the value-side regularizer is completely disabled. Additional ablations are deferred to Appendix B.2. 6 Conclusion In this paper, we identified the neglect of higher-order action dynamics as a fundamental source of control instability and poor robustness in contemporary offline RL and IL methods. To bridge this gap, we proposed an elegant, plug-and-play learning scheme that enables simultaneous zeroth- and first-order action supervision for any off-the-shelf policy model, including deterministic, stochastic, and generative flow policies, without requiring any structural modifications. Supported by formal theoretical guarantees, our enhanced policy class also integrates seamlessly into the continuous-time HJB framework, providing additional first-order regularization on both value estimation and policy optimization. Extensive empirical evaluations confirm that our scheme yields substantial improvements in control smoothness, robustness, and out-of-distribution generalization, particularly under challenging small-dataset settings, making it a promising tool for real-world control problems. Acknowledgment This work was supported by the Wuxi Research Institute of Applied Technologies at Tsinghua University under the Grant 20242001120, Xiongan AI Institute, and Horizon Robotics. References Bhasin et al. (2013) S. Bhasin, R. Kamalapurkar, M. Johnson, K. G. Vamvoudakis, F. L. Lewis, and W. E. Dixon A novel actor–critic–identifier architecture for approximate optimal control of uncertain nonlinear systems. Automatica 49 (1), p. 82–92. Cited by: §2. Cai et al. (2020) P. Cai, X. Mei, L. Tai, Y. Sun, and M. Liu High-speed autonomous drifting with deep reinforcement learning. IEEE Robotics and Automation Letters 5 (2), p. 1247–1254. Cited by: §1. Chen et al. (2021) C. Chen, H. Tang, J. Hao, W. Liu, and Z. Meng Addressing action oscillations through learning policy inertia. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, p. 7020–7027. Cited by: §1. Cheng et al. (2026) P. Cheng, Z. Wu, J. Li, Z. He, H. Xu, W. Sun, Y. Lin, Y. Liu, and X. Zhan Sample efficient offline rl via t-symmetry enforced latent state-stitching. In International Conference on Learning Representations, Vol. 2026, p. 139752–139781. Cited by: §1. Cheng et al. (2023) P. Cheng, X. Zhan, Z. Wu, W. Zhang, Y. Lin, H. Wang, L. Jiang, et al. Look beneath the surface: exploiting fundamental symmetry for sample-efficient offline rl. Advances in Neural Information Processing Systems 36. Cited by: §1, §2. Chi et al. (2023) C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song Diffusion policy: visuomotor policy learning via action diffusion. In Robotics: Science and Systems, Cited by: §1. Christmann et al. (2024) G. Christmann, Y. Luo, H. Mandala, and W. Chen Benchmarking smoothness and reducing high-frequency oscillations in continuous control policies. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 627–634. Cited by: §2. de Boor (2001) C. de Boor A practical guide to splines. Applied Mathematical Sciences, Springer, New York, NY. Cited by: Appendix A. de La Presilla et al. (2023) R. de La Presilla, S. Wandel, M. Stammler, M. Grebe, G. Poll, and S. Glavatskih Oscillating rolling element bearings: a review of tribotesting and analysis approaches. Tribology International 188, p. 108805. Cited by: §1. Doya (2000) K. Doya Reinforcement learning in continuous time and space. Neural Computation 12 (1), p. 219–245. Cited by: §2. Dulac-Arnold et al. (2021) G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning 110 (9), p. 2419–2468. Cited by: §1, §1, §2. Emmrich (1999) E. Emmrich Discrete versions of gronwall’s lemma and their application to the numerical analysis of parabolic problems. Preprint Technische Universität Berlin, Fachbereich Mathematik. Cited by: Lemma 2, Lemma 2. Fu et al. (2020) J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine D4rl: datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. Cited by: Appendix E, §1, §5.1. Fujimoto and Gu (2021) S. Fujimoto and S. S. Gu A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems 34. Cited by: §B.1, §E.2, §E.2, §2, §4.2, §5.1. Haarnoja et al. (2018) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, p. 1861–1870. Cited by: §B.1. Hafner et al. (2020) D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi Dream to control: learning behaviors by latent imagination. In International Conference on Learning Representations, Cited by: Appendix A. Hansen-Estruch et al. (2023) P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine Idql: implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573. Cited by: §2. Hendrycks and Gimpel (2016) D. Hendrycks and K. Gimpel Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. Cited by: Table 11. Janner et al. (2022) M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991. Cited by: §2. Jia and Zhou (2022) Y. Jia and X. Y. Zhou Policy evaluation and temporal-difference learning in continuous time and space: a martingale approach. Journal of Machine Learning Research 23 (154), p. 1–55. Cited by: §2. Kalashnikov et al. (2018) D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, p. 651–673. Cited by: §1. Kendall et al. (2019) A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J. Allen, V. Lam, A. Bewley, and A. Shah Learning to drive in a day. In 2019 international conference on robotics and automation (ICRA), p. 8248–8254. Cited by: §1. Kim et al. (2021) J. Kim, J. Shin, and I. Yang Hamilton–jacobi deep q-learning for deterministic continuous-time systems with lipschitz continuous controls. Journal of Machine Learning Research 22 (206), p. 1–34. Cited by: §2, §4.1, §4.1. Kingma and Ba (2014) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: Table 11. Kirk (2004) D. E. Kirk Optimal control theory: an introduction. Courier Corporation. Cited by: §1. Kobayashi (2022) T. Kobayashi L2c2: locally lipschitz continuous constraint towards stable and smooth reinforcement learning. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 4032–4039. Cited by: §1, §2, §5.3. Kostrikov et al. (2022) I. Kostrikov, A. Nair, and S. Levine Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, Cited by: §E.2, §E.2, §2, §4.2, §5.1. Kumar et al. (2019) A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, p. 11761–11771. Cited by: §2. Kumar et al. (2020) A. Kumar, A. Zhou, G. Tucker, and S. Levine Conservative q-learning for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), Cited by: §2. Lee et al. (2024) I. Lee, H. Cao, C. Dao, Y. Chen, and I. Wu Gradient-based regularization for action smoothness in robotic control with reinforcement learning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 603–610. Cited by: §1. Lee and Sutton (2021) J. Lee and R. S. Sutton Policy iterations for reinforcement learning problems in continuous time and space—fundamental theory and methods. Automatica 126, p. 109421. Cited by: §2. Levine et al. (2020) S. Levine, A. Kumar, G. Tucker, and J. Fu Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643. Cited by: §2. Li et al. (2024) G. Li, Y. Shan, Z. Zhu, T. Long, and W. Zhang Diffstitch: boosting offline reinforcement learning with diffusion-based trajectory stitching. In Forty-first International Conference on Machine Learning, p. 28597 – 28609. Cited by: §2. Li et al. (2023a) J. Li, X. Hu, H. Xu, J. Liu, X. Zhan, Q. Jia, and Y. Zhang Mind the gap: offline policy optimization for imperfect rewards. In The Eleventh International Conference on Learning Representations, Cited by: §2. Li et al. (2023b) J. Li, X. Zhan, H. Xu, X. Zhu, J. Liu, and Y. Zhang When data geometry meets deep function: generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, Cited by: §2. Liang et al. (2026) R. Liang, Y. Zheng, K. Zheng, T. Tan, J. Li, L. Mao, Z. Wang, G. Chen, H. Ye, J. Liu, J. Wang, and X. Zhan Dichotomous diffusion policy optimization. In International Conference on Learning Representations, Cited by: §2, §4.2. Lien et al. (2024) Y. Lien, P. Hsieh, T. Li, and Y. Wang Enhancing value function estimation through first-order state-action dynamics in offline reinforcement learning. In Forty-first International Conference on Machine Learning, Cited by: §2, §4.1. Lipman et al. (2023) Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. In International Conference on Learning Representations, Cited by: §3.2, §3.2. Liu et al. (2023) X. Liu, C. Gong, and Q. Liu Flow straight and fast: learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, Cited by: §3.2, §3.2. Lyu et al. (2022) J. Lyu, X. Ma, X. Li, and Z. Lu Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems 35, p. 1711–1724. Cited by: §2. Mao et al. (2024a) L. Mao, H. Xu, X. Zhan, W. Zhang, and A. Zhang Diffusion-dice: in-sample diffusion guidance for offline reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §2. Mao et al. (2024b) L. Mao, H. Xu, W. Zhang, and X. Zhan ODICE: revealing the mystery of distribution correction estimation via orthogonal-gradient update. In The Twelfth International Conference on Learning Representations, Cited by: §2. Mnih et al. (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature 518 (7540), p. 529–533. Cited by: §1. Munos (2006) R. Munos Policy gradient in continuous time. Journal of Machine Learning Research 7, p. 771–791. Cited by: §2. Mysore et al. (2021) S. Mysore, B. Mabsout, R. Mancuso, and K. Saenko Regularizing action policies for smooth control with reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), p. 1810–1816. Cited by: §1, §1, §2, §5.3. Niu et al. (2022) H. Niu, S. Sharma, Y. Qiu, M. Li, G. Zhou, J. HU, and X. Zhan When to trust your simulator: dynamics-aware hybrid offline-and-online reinforcement learning. In Advances in Neural Information Processing Systems, Cited by: §2. Oksendal (2013) B. Oksendal Stochastic differential equations: an introduction with applications. Springer Science & Business Media. Cited by: §C.1, §3.2. Park et al. (2025a) S. Park, K. Frans, B. Eysenbach, and S. Levine OGBench: benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), Cited by: Appendix E, §1, §5.1. Park et al. (2025b) S. Park, Q. Li, and S. Levine Flow q-learning. In Forty-second International Conference on Machine Learning, Cited by: §B.1, §E.2, §E.2, §E.2, §2, §3.2, §5.1. Pfrommer et al. (2022) D. Pfrommer, T. Zhang, S. Tu, and N. Matni Tasil: taylor series imitation learning. Advances in Neural Information Processing Systems 35, p. 20162–20174. Cited by: §2. Savitzky and Golay (1964) A. Savitzky and M. J. E. Golay Smoothing and differentiation of data by simplified least squares procedures. Analytical Chemistry 36 (8), p. 1627–1639. Cited by: Appendix A. Shen et al. (2020) Q. Shen, Y. Li, H. Jiang, Z. Wang, and T. Zhao Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, p. 8707–8718. Cited by: §1, §2. Shi et al. (2019) G. Shi, X. Shi, M. O’Connell, R. Yu, K. Azizzadenesheli, A. Anandkumar, Y. Yue, and S. Chung Neural lander: stable drone landing control using learned dynamics. In 2019 international conference on robotics and automation (icra), p. 9784–9790. Cited by: §1. Silver et al. (2017) D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. nature 550 (7676), p. 354–359. Cited by: §1. Song et al. (2025) X. Song, L. Chen, T. Liu, W. Wang, Y. Wang, S. Qin, Y. Ma, J. Duan, and S. E. Li Lipsnet++: unifying filter and controller into a policy network. In Forty-second International Conference on Machine Learning, Cited by: §1, §1, §2. Song et al. (2023) X. Song, J. Duan, W. Wang, S. E. Li, C. Chen, B. Cheng, B. Zhang, J. Wei, and X. S. Wang LipsNet: a smooth and robust neural network with adaptive lipschitz constant for high accuracy optimal control. In International Conference on Machine Learning, p. 32253–32272. Cited by: §1, §1, §1, §2, §5.3. Tarasov et al. (2023) D. Tarasov, V. Kurenkov, A. Nikulin, and S. Kolesnikov Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems 36, p. 11592–11620. Cited by: §E.2, §E.2, §5.1. Uhlenbeck and Ornstein (1930) G. E. Uhlenbeck and L. S. Ornstein On the theory of the brownian motion. Physical review 36 (5), p. 823. Cited by: §C.1, §3.2. Vamvoudakis and Lewis (2010) K. G. Vamvoudakis and F. L. Lewis Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46 (5), p. 878–888. Cited by: §2. Vamvoudakis et al. (2014) K. G. Vamvoudakis, D. Vrabie, and F. L. Lewis Online adaptive algorithm for optimal control with integral reinforcement learning. International Journal of Robust and Nonlinear Control 24 (17), p. 2686–2710. Cited by: §2. Wang et al. (2020) H. Wang, T. Zariphopoulou, and X. Y. Zhou Reinforcement learning in continuous time and space: a stochastic control approach. Journal of Machine Learning Research 21 (198), p. 1–34. Cited by: §2. Wang et al. (2023) X. Wang, H. Xu, Y. Zheng, and X. Zhan Offline multi-agent reinforcement learning with implicit global-to-local value regularization. In Advances in Neural Information Processing Systems, Cited by: §2. Wang et al. (2024) X. Wang, H. Xu, Y. Zheng, and X. Zhan Offline multi-agent reinforcement learning with implicit global-to-local value regularization. Advances in Neural Information Processing Systems 36. Cited by: §4.2. Wang et al. (2022) Z. Wang, J. J. Hunt, and M. Zhou Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193. Cited by: §2. Wasala et al. (2020) A. Wasala, D. Byrne, P. Miesbauer, J. O’Hanlon, P. Heraty, and P. Barry Trajectory based lateral control: a reinforcement learning case study. Engineering Applications of Artificial Intelligence 94, p. 103799. Cited by: §1. Wu et al. (2019) Y. Wu, G. Tucker, and O. Nachum Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361. Cited by: §2. Xu et al. (2023) H. Xu, L. Jiang, J. Li, Z. Yang, Z. Wang, V. W. K. Chan, and X. Zhan Offline rl with no ood actions: in-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, Cited by: §2, §4.2. Xu et al. (2022a) H. Xu, L. Jiang, J. Li, and X. Zhan A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, Cited by: §2. Xu et al. (2026) H. Xu, L. Mao, H. Jin, W. Zhang, X. Zhan, and A. Zhang Uni-rl: unifying online and offline rl via implicit value regularization. Advances in Neural Information Processing Systems 38, p. 133162–133187. Cited by: §2, §4.2. Xu et al. (2022b) H. Xu, X. Zhan, and X. Zhu Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: §2. Yu et al. (2021) H. Yu, W. Xu, and H. Zhang Taac: temporally abstract actor-critic for continuous control. Advances in neural information processing systems 34, p. 29021–29033. Cited by: §1. Yu et al. (2020) T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn Gradient surgery for multi-task learning. Advances in neural information processing systems 33, p. 5824–5836. Cited by: §1. Zhan et al. (2022) X. Zhan, H. Xu, Y. Zhang, X. Zhu, H. Yin, and Y. Zheng Deepthermal: combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, p. 4680–4688. Cited by: §1. Zhan et al. (2025) X. Zhan, X. Zhu, P. Cheng, X. Hu, Z. He, H. Geng, J. Leng, H. Zheng, C. Liu, T. Hong, et al. Data center cooling system optimization using offline reinforcement learning. In International Conference on Learning Representations, Vol. 2025, p. 47735–47757. Cited by: §1. Zhang et al. (2024) Y. Zhang, B. Nie, and Y. Gao Robust locomotion policy with adaptive lipschitz constraint for legged robots. IEEE Robotics and Automation Letters 10 (1), p. 272–279. Cited by: §1. Zhao et al. (2022) Z. Zhao, S. Zuo, T. Zhao, and Y. Zhao Adversarially regularized policy learning guided by trajectory optimization. In Learning for Dynamics and Control Conference, p. 844–857. Cited by: §1, §2. Zheng et al. (2025a) K. Zheng, L. Teyssier, Y. Zheng, Y. Luo, and X. Zhan Towards robust zero-shot reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §2. Zheng et al. (2026) K. Zheng, X. Zhan, and X. Yan Horizon adaptive offline policy learning via value stitching. In The Fortieth Annual Conference on Neural Information Processing Systems, Cited by: §2. Zheng et al. (2024) Y. Zheng, J. Li, D. Yu, Y. Yang, S. E. Li, X. Zhan, and J. Liu Safe offline reinforcement learning with feasibility-guided diffusion model. In The Twelfth International Conference on Learning Representations, Cited by: §2, §4.2. Zheng et al. (2025b) Y. Zheng, R. Liang, K. ZHENG, J. Zheng, L. Mao, J. Li, W. Gu, R. Ai, S. E. Li, X. Zhan, and J. Liu Diffusion-based planning for autonomous driving with flexible guidance. In The Thirteenth International Conference on Learning Representations, Cited by: §1. Appendix A Limitations In this section, we summarize the limitations of our proposed method and outline potential directions for future research. First, applying first-order supervision directly in raw pixel space can be computationally expensive and semantically noisy. The issue is not that higher-order supervision is incompatible with visual inputs, but that pixels are a poor coordinate system for measuring dynamics and control evolution. A practical remedy is to apply the same supervision after mapping observations into a compact state representation [16], where derivatives reflect task-relevant changes rather than appearance variation. Pixel-level higher-order supervision therefore requires a good representation. Second, our higher-order supervision relies on finite-difference estimates of local derivatives, i.e., s˙=s′−s s=s -s. With coarse sampling, these estimated tangents may provide poor approximations of the true local dynamics. This issue can be alleviated by reconstructing smoother trajectories via interpolation [8] or filtering [51] before applying supervision. Appendix B Additional Results In this section, we first present the detailed evaluation results on OGBench in Table 4, followed by the performance scores on the D4RL benchmark under the full-dataset setting in Table 5. Additionally, we report the imitation learning results in Table 6. Finally, Section B.2 provides more ablation studies. B.1 Additional Evaluation Results Full Evaluation results on OGBench. We present the comprehensive per-task results on OGBench in Table 4. Note that the baseline performance of TD3+BC [14] is evaluated using the hyperparameters provided in Appendix E.2, while the baseline results for ReBRAC, IQL, and IFQL are directly sourced from FQL [49]. Table 4: Average normalized scores on 50 OGBench tasks over 5 random seeds. Values after ± denote standard deviations. The ↑ [rgb]0.0195,0.6523,0.082 marks indicate improvements greater than 1.01.0, whereas ↑ [rgb]0.5,0.5,0.5 denotes minor improvements. Det. Policy Sto. Policy (mean sup.) Sto. Policy (standard) Flow Policy Task Category TD3+BC TD3+BC+ours ReBRAC ReBRAC+ours IQL IQL+ours IFQL IFQL+ours antmaze-large-navigate-singletask-task1-v0 (∗ ) 92±892 ± 8 99±2(↑7)99 ± 2\,( [rgb]0.0195,0.6523,0.082 \!7) 91±1091 ± 10 100±1(↑9)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!9) 48±948 ± 9 80±10(↑32)80 ± 10\,( [rgb]0.0195,0.6523,0.082 \!32) 24±1724 ± 17 56±14(↑32)56 ± 14\,( [rgb]0.0195,0.6523,0.082 \!32) antmaze-large-navigate-singletask-task2-v0 85±385 ± 3 91±4(↑6)91 ± 4\,( [rgb]0.0195,0.6523,0.082 \!6) 88±488 ± 4 92±4(↑4)92 ± 4\,( [rgb]0.0195,0.6523,0.082 \!4) 42±642 ± 6 42±1542 ± 15 8±38 ± 3 16±11(↑8)16 ± 11\,( [rgb]0.0195,0.6523,0.082 \!8) antmaze-large-navigate-singletask-task3-v0 72±1072 ± 10 100±0(↑28)100 ± 0\,( [rgb]0.0195,0.6523,0.082 \!28) 51±1851 ± 18 100±1(↑49)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!49) 72±772 ± 7 90±6(↑18)90 ± 6\,( [rgb]0.0195,0.6523,0.082 \!18) 52±1752 ± 17 72±10(↑20)72 ± 10\,( [rgb]0.0195,0.6523,0.082 \!20) antmaze-large-navigate-singletask-task4-v0 85±585 ± 5 99±1(↑14)99 ± 1\,( [rgb]0.0195,0.6523,0.082 \!14) 84±784 ± 7 100±1(↑16)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!16) 51±951 ± 9 52±22(↑1)52 ± 22\,( [rgb]0.5,0.5,0.5 \!1) 18±818 ± 8 40±17(↑22)40 ± 17\,( [rgb]0.0195,0.6523,0.082 \!22) antmaze-large-navigate-singletask-task5-v0 82±1082 ± 10 99±0(↑17)99 ± 0\,( [rgb]0.0195,0.6523,0.082 \!17) 90±290 ± 2 99±1(↑9)99 ± 1\,( [rgb]0.0195,0.6523,0.082 \!9) 54±2254 ± 22 78±9(↑24)78 ± 9\,( [rgb]0.0195,0.6523,0.082 \!24) 38±1838 ± 18 48±19(↑10)48 ± 19\,( [rgb]0.0195,0.6523,0.082 \!10) antmaze-giant-navigate-singletask-task1-v0 (∗ ) 52±2052 ± 20 57±19(↑5)57 ± 19\,( [rgb]0.0195,0.6523,0.082 \!5) 27±2227 ± 22 39±21(↑12)39 ± 21\,( [rgb]0.0195,0.6523,0.082 \!12) 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 antmaze-giant-navigate-singletask-task2-v0 47±3047 ± 30 71±36(↑24)71 ± 36\,( [rgb]0.0195,0.6523,0.082 \!24) 16±1716 ± 17 70±26(↑54)70 ± 26\,( [rgb]0.0195,0.6523,0.082 \!54) 1±11 ± 1 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 0±00 ± 0 0±00 ± 0 antmaze-giant-navigate-singletask-task3-v0 44±1344 ± 13 44±1244 ± 12 34±2234 ± 22 40±15(↑6)40 ± 15\,( [rgb]0.0195,0.6523,0.082 \!6) 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 antmaze-giant-navigate-singletask-task4-v0 3±43 ± 4 63±31(↑60)63 ± 31\,( [rgb]0.0195,0.6523,0.082 \!60) 5±125 ± 12 67±33(↑62)67 ± 33\,( [rgb]0.0195,0.6523,0.082 \!62) 0±00 ± 0 1±1(↑1)1 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 0±00 ± 0 0±00 ± 0 antmaze-giant-navigate-singletask-task5-v0 65±765 ± 7 92±11(↑27)92 ± 11\,( [rgb]0.0195,0.6523,0.082 \!27) 49±2249 ± 22 96±3(↑47)96 ± 3\,( [rgb]0.0195,0.6523,0.082 \!47) 19±719 ± 7 41±6(↑22)41 ± 6\,( [rgb]0.0195,0.6523,0.082 \!22) 13±913 ± 9 21±17(↑8)21 ± 17\,( [rgb]0.0195,0.6523,0.082 \!8) humanoidmaze-medium-navigate-singletask-task1-v0 (∗ ) 24±1124 ± 11 26±4(↑2)26 ± 4\,( [rgb]0.0195,0.6523,0.082 \!2) 16±916 ± 9 36±19(↑20)36 ± 19\,( [rgb]0.0195,0.6523,0.082 \!20) 32±732 ± 7 34±5(↑2)34 ± 5\,( [rgb]0.0195,0.6523,0.082 \!2) 69±1969 ± 19 71±16(↑2)71 ± 16\,( [rgb]0.0195,0.6523,0.082 \!2) humanoidmaze-medium-navigate-singletask-task2-v0 45±1345 ± 13 49±9(↑4)49 ± 9\,( [rgb]0.0195,0.6523,0.082 \!4) 18±1618 ± 16 68±24(↑50)68 ± 24\,( [rgb]0.0195,0.6523,0.082 \!50) 41±941 ± 9 51±5(↑10)51 ± 5\,( [rgb]0.0195,0.6523,0.082 \!10) 85±1185 ± 11 91±17(↑6)91 ± 17\,( [rgb]0.0195,0.6523,0.082 \!6) humanoidmaze-medium-navigate-singletask-task3-v0 40±2640 ± 26 60±19(↑20)60 ± 19\,( [rgb]0.0195,0.6523,0.082 \!20) 36±1336 ± 13 66±22(↑30)66 ± 22\,( [rgb]0.0195,0.6523,0.082 \!30) 25±525 ± 5 27±6(↑2)27 ± 6\,( [rgb]0.0195,0.6523,0.082 \!2) 49±4949 ± 49 56±32(↑7)56 ± 32\,( [rgb]0.0195,0.6523,0.082 \!7) humanoidmaze-medium-navigate-singletask-task4-v0 30±2030 ± 20 44±9(↑14)44 ± 9\,( [rgb]0.0195,0.6523,0.082 \!14) 15±1615 ± 16 30±21(↑15)30 ± 21\,( [rgb]0.0195,0.6523,0.082 \!15) 0±10 ± 1 0±00 ± 0 1±11 ± 1 1±11 ± 1 humanoidmaze-medium-navigate-singletask-task5-v0 31±1331 ± 13 31±731 ± 7 24±2024 ± 20 78±16(↑54)78 ± 16\,( [rgb]0.0195,0.6523,0.082 \!54) 66±466 ± 4 66±466 ± 4 98±298 ± 2 100±1(↑2)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!2) humanoidmaze-large-navigate-singletask-task1-v0 (∗ ) 3±33 ± 3 3±23 ± 2 2±12 ± 1 7±7(↑5)7 ± 7\,( [rgb]0.0195,0.6523,0.082 \!5) 3±13 ± 1 4±1(↑1)4 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 6±26 ± 2 8±4(↑2)8 ± 4\,( [rgb]0.0195,0.6523,0.082 \!2) humanoidmaze-large-navigate-singletask-task2-v0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 humanoidmaze-large-navigate-singletask-task3-v0 8±48 ± 4 17±6(↑9)17 ± 6\,( [rgb]0.0195,0.6523,0.082 \!9) 8±48 ± 4 11±6(↑3)11 ± 6\,( [rgb]0.0195,0.6523,0.082 \!3) 7±37 ± 3 7±17 ± 1 48±1048 ± 10 52±16(↑4)52 ± 16\,( [rgb]0.0195,0.6523,0.082 \!4) humanoidmaze-large-navigate-singletask-task4-v0 1±11 ± 1 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 1±11 ± 1 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 1±01 ± 0 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 1±11 ± 1 1±11 ± 1 humanoidmaze-large-navigate-singletask-task5-v0 0±00 ± 0 1±1(↑1)1 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 2±22 ± 2 2±12 ± 1 1±11 ± 1 3±1(↑2)3 ± 1\,( [rgb]0.0195,0.6523,0.082 \!2) 0±00 ± 0 0±00 ± 0 antsoccer-arena-navigate-singletask-task1-v0 9±39 ± 3 11±5(↑2)11 ± 5\,( [rgb]0.0195,0.6523,0.082 \!2) 0±00 ± 0 13±6(↑13)13 ± 6\,( [rgb]0.0195,0.6523,0.082 \!13) 14±514 ± 5 36±10(↑22)36 ± 10\,( [rgb]0.0195,0.6523,0.082 \!22) 61±2661 ± 26 68±21(↑7)68 ± 21\,( [rgb]0.0195,0.6523,0.082 \!7) antsoccer-arena-navigate-singletask-task2-v0 19±819 ± 8 19±219 ± 2 0±10 ± 1 13±4(↑13)13 ± 4\,( [rgb]0.0195,0.6523,0.082 \!13) 17±717 ± 7 37±5(↑20)37 ± 5\,( [rgb]0.0195,0.6523,0.082 \!20) 75±375 ± 3 83±12(↑8)83 ± 12\,( [rgb]0.0195,0.6523,0.082 \!8) antsoccer-arena-navigate-singletask-task3-v0 25±825 ± 8 27±10(↑2)27 ± 10\,( [rgb]0.0195,0.6523,0.082 \!2) 0±00 ± 0 32±11(↑32)32 ± 11\,( [rgb]0.0195,0.6523,0.082 \!32) 6±46 ± 4 19±7(↑13)19 ± 7\,( [rgb]0.0195,0.6523,0.082 \!13) 14±2214 ± 22 22±18(↑8)22 ± 18\,( [rgb]0.0195,0.6523,0.082 \!8) antsoccer-arena-navigate-singletask-task4-v0 (∗ ) 1±11 ± 1 1±11 ± 1 0±00 ± 0 20±4(↑20)20 ± 4\,( [rgb]0.0195,0.6523,0.082 \!20) 3±23 ± 2 9±3(↑6)9 ± 3\,( [rgb]0.0195,0.6523,0.082 \!6) 16±916 ± 9 16±816 ± 8 antsoccer-arena-navigate-singletask-task5-v0 12±712 ± 7 20±9(↑8)20 ± 9\,( [rgb]0.0195,0.6523,0.082 \!8) 0±00 ± 0 6±5(↑6)6 ± 5\,( [rgb]0.0195,0.6523,0.082 \!6) 2±22 ± 2 11±7(↑9)11 ± 7\,( [rgb]0.0195,0.6523,0.082 \!9) 0±10 ± 1 8±7(↑8)8 ± 7\,( [rgb]0.0195,0.6523,0.082 \!8) cube-single-play-singletask-task1-v0 89±389 ± 3 94±2(↑5)94 ± 2\,( [rgb]0.0195,0.6523,0.082 \!5) 89±589 ± 5 94±2(↑5)94 ± 2\,( [rgb]0.0195,0.6523,0.082 \!5) 88±388 ± 3 90±5(↑2)90 ± 5\,( [rgb]0.0195,0.6523,0.082 \!2) 79±479 ± 4 96±9(↑17)96 ± 9\,( [rgb]0.0195,0.6523,0.082 \!17) cube-single-play-singletask-task2-v0 (∗ ) 87±587 ± 5 88±2(↑1)88 ± 2\,( [rgb]0.5,0.5,0.5 \!1) 92±492 ± 4 93±4(↑1)93 ± 4\,( [rgb]0.5,0.5,0.5 \!1) 85±885 ± 8 93±3(↑8)93 ± 3\,( [rgb]0.0195,0.6523,0.082 \!8) 73±373 ± 3 100±1(↑27)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!27) cube-single-play-singletask-task3-v0 93±393 ± 3 96±3(↑3)96 ± 3\,( [rgb]0.0195,0.6523,0.082 \!3) 93±393 ± 3 94±2(↑1)94 ± 2\,( [rgb]0.5,0.5,0.5 \!1) 91±591 ± 5 95±5(↑4)95 ± 5\,( [rgb]0.0195,0.6523,0.082 \!4) 88±488 ± 4 100±1(↑12)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!12) cube-single-play-singletask-task4-v0 87±487 ± 4 89±3(↑2)89 ± 3\,( [rgb]0.0195,0.6523,0.082 \!2) 92±392 ± 3 92±492 ± 4 73±673 ± 6 90±5(↑17)90 ± 5\,( [rgb]0.0195,0.6523,0.082 \!17) 79±679 ± 6 88±16(↑9)88 ± 16\,( [rgb]0.0195,0.6523,0.082 \!9) cube-single-play-singletask-task5-v0 85±585 ± 5 85±185 ± 1 87±887 ± 8 90±4(↑3)90 ± 4\,( [rgb]0.0195,0.6523,0.082 \!3) 78±978 ± 9 90±6(↑12)90 ± 6\,( [rgb]0.0195,0.6523,0.082 \!12) 77±777 ± 7 92±10(↑15)92 ± 10\,( [rgb]0.0195,0.6523,0.082 \!15) cube-double-play-singletask-task1-v0 30±430 ± 4 46±6(↑16)46 ± 6\,( [rgb]0.0195,0.6523,0.082 \!16) 45±645 ± 6 45±645 ± 6 27±527 ± 5 33±7(↑6)33 ± 7\,( [rgb]0.0195,0.6523,0.082 \!6) 35±935 ± 9 76±8(↑41)76 ± 8\,( [rgb]0.0195,0.6523,0.082 \!41) cube-double-play-singletask-task2-v0 (∗ ) 6±16 ± 1 9±1(↑3)9 ± 1\,( [rgb]0.0195,0.6523,0.082 \!3) 7±37 ± 3 32±7(↑25)32 ± 7\,( [rgb]0.0195,0.6523,0.082 \!25) 1±11 ± 1 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 9±59 ± 5 40±22(↑31)40 ± 22\,( [rgb]0.0195,0.6523,0.082 \!31) cube-double-play-singletask-task3-v0 1±11 ± 1 2±1(↑1)2 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 4±14 ± 1 17±5(↑13)17 ± 5\,( [rgb]0.0195,0.6523,0.082 \!13) 0±00 ± 0 1±1(↑1)1 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 8±58 ± 5 25±13(↑17)25 ± 13\,( [rgb]0.0195,0.6523,0.082 \!17) cube-double-play-singletask-task4-v0 0±10 ± 1 1±1(↑1)1 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 1±11 ± 1 3±1(↑2)3 ± 1\,( [rgb]0.0195,0.6523,0.082 \!2) 0±00 ± 0 0±00 ± 0 1±11 ± 1 12±10(↑11)12 ± 10\,( [rgb]0.0195,0.6523,0.082 \!11) cube-double-play-singletask-task5-v0 1±11 ± 1 1±11 ± 1 4±24 ± 2 22±7(↑18)22 ± 7\,( [rgb]0.0195,0.6523,0.082 \!18) 4±34 ± 3 4±24 ± 2 17±617 ± 6 55±21(↑38)55 ± 21\,( [rgb]0.0195,0.6523,0.082 \!38) scene-play-singletask-task1-v0 95±395 ± 3 96±1(↑1)96 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 95±295 ± 2 95±395 ± 3 94±394 ± 3 95±2(↑1)95 ± 2\,( [rgb]0.5,0.5,0.5 \!1) 98±398 ± 3 99±2(↑1)99 ± 2\,( [rgb]0.5,0.5,0.5 \!1) scene-play-singletask-task2-v0 (∗ ) 54±1054 ± 10 58±4(↑4)58 ± 4\,( [rgb]0.0195,0.6523,0.082 \!4) 50±1350 ± 13 60±13(↑10)60 ± 13\,( [rgb]0.0195,0.6523,0.082 \!10) 12±312 ± 3 17±6(↑5)17 ± 6\,( [rgb]0.0195,0.6523,0.082 \!5) 0±00 ± 0 88±17(↑88)88 ± 17\,( [rgb]0.0195,0.6523,0.082 \!88) scene-play-singletask-task3-v0 68±1068 ± 10 78±7(↑10)78 ± 7\,( [rgb]0.0195,0.6523,0.082 \!10) 55±1655 ± 16 75±10(↑20)75 ± 10\,( [rgb]0.0195,0.6523,0.082 \!20) 32±732 ± 7 52±9(↑20)52 ± 9\,( [rgb]0.0195,0.6523,0.082 \!20) 54±1954 ± 19 92±17(↑38)92 ± 17\,( [rgb]0.0195,0.6523,0.082 \!38) scene-play-singletask-task4-v0 4±34 ± 3 4±24 ± 2 3±33 ± 3 5±3(↑2)5 ± 3\,( [rgb]0.0195,0.6523,0.082 \!2) 0±10 ± 1 1±1(↑1)1 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 0±00 ± 0 0±00 ± 0 scene-play-singletask-task5-v0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 0±00 ± 0 puzzle-3x3-play-singletask-task1-v0 96±396 ± 3 98±3(↑2)98 ± 3\,( [rgb]0.0195,0.6523,0.082 \!2) 97±497 ± 4 98±3(↑1)98 ± 3\,( [rgb]0.5,0.5,0.5 \!1) 33±633 ± 6 39±17(↑6)39 ± 17\,( [rgb]0.0195,0.6523,0.082 \!6) 94±394 ± 3 100±1(↑6)100 ± 1\,( [rgb]0.0195,0.6523,0.082 \!6) puzzle-3x3-play-singletask-task2-v0 13±913 ± 9 14±4(↑1)14 ± 4\,( [rgb]0.5,0.5,0.5 \!1) 1±11 ± 1 4±2(↑3)4 ± 2\,( [rgb]0.0195,0.6523,0.082 \!3) 4±34 ± 3 9±6(↑5)9 ± 6\,( [rgb]0.0195,0.6523,0.082 \!5) 1±21 ± 2 60±32(↑59)60 ± 32\,( [rgb]0.0195,0.6523,0.082 \!59) puzzle-3x3-play-singletask-task3-v0 5±65 ± 6 6±1(↑1)6 ± 1\,( [rgb]0.5,0.5,0.5 \!1) 3±13 ± 1 4±4(↑1)4 ± 4\,( [rgb]0.5,0.5,0.5 \!1) 3±23 ± 2 3±13 ± 1 0±00 ± 0 28±17(↑28)28 ± 17\,( [rgb]0.0195,0.6523,0.082 \!28) puzzle-3x3-play-singletask-task4-v0 (∗ ) 7±47 ± 4 11±4(↑4)11 ± 4\,( [rgb]0.0195,0.6523,0.082 \!4) 2±12 ± 1 4±2(↑2)4 ± 2\,( [rgb]0.0195,0.6523,0.082 \!2) 2±12 ± 1 2±12 ± 1 0±00 ± 0 16±12(↑16)16 ± 12\,( [rgb]0.0195,0.6523,0.082 \!16) puzzle-3x3-play-singletask-task5-v0 14±714 ± 7 15±2(↑1)15 ± 2\,( [rgb]0.5,0.5,0.5 \!1) 5±35 ± 3 11±6(↑6)11 ± 6\,( [rgb]0.0195,0.6523,0.082 \!6) 3±23 ± 2 4±2(↑1)4 ± 2\,( [rgb]0.5,0.5,0.5 \!1) 0±00 ± 0 12±11(↑12)12 ± 11\,( [rgb]0.0195,0.6523,0.082 \!12) puzzle-4x4-play-singletask-task1-v0 1±21 ± 2 14±2(↑13)14 ± 2\,( [rgb]0.0195,0.6523,0.082 \!13) 26±426 ± 4 33±9(↑7)33 ± 9\,( [rgb]0.0195,0.6523,0.082 \!7) 12±212 ± 2 20±6(↑8)20 ± 6\,( [rgb]0.0195,0.6523,0.082 \!8) 49±949 ± 9 53±6(↑4)53 ± 6\,( [rgb]0.0195,0.6523,0.082 \!4) puzzle-4x4-play-singletask-task2-v0 7±37 ± 3 9±2(↑2)9 ± 2\,( [rgb]0.0195,0.6523,0.082 \!2) 12±412 ± 4 13±7(↑1)13 ± 7\,( [rgb]0.5,0.5,0.5 \!1) 7±47 ± 4 10±3(↑3)10 ± 3\,( [rgb]0.0195,0.6523,0.082 \!3) 4±44 ± 4 12±9(↑8)12 ± 9\,( [rgb]0.0195,0.6523,0.082 \!8) puzzle-4x4-play-singletask-task3-v0 3±23 ± 2 8±2(↑5)8 ± 2\,( [rgb]0.0195,0.6523,0.082 \!5) 15±315 ± 3 19±6(↑4)19 ± 6\,( [rgb]0.0195,0.6523,0.082 \!4) 9±39 ± 3 14±6(↑5)14 ± 6\,( [rgb]0.0195,0.6523,0.082 \!5) 50±1450 ± 14 84±16(↑34)84 ± 16\,( [rgb]0.0195,0.6523,0.082 \!34) puzzle-4x4-play-singletask-task4-v0 (∗ ) 5±25 ± 2 7±2(↑2)7 ± 2\,( [rgb]0.0195,0.6523,0.082 \!2) 10±310 ± 3 10±410 ± 4 5±25 ± 2 10±2(↑5)10 ± 2\,( [rgb]0.0195,0.6523,0.082 \!5) 21±1121 ± 11 30±17(↑9)30 ± 17\,( [rgb]0.0195,0.6523,0.082 \!9) puzzle-4x4-play-singletask-task5-v0 6±16 ± 1 6±16 ± 1 7±37 ± 3 8±4(↑1)8 ± 4\,( [rgb]0.5,0.5,0.5 \!1) 4±14 ± 1 8±5(↑4)8 ± 5\,( [rgb]0.0195,0.6523,0.082 \!4) 2±22 ± 2 21±14(↑19)21 ± 14\,( [rgb]0.0195,0.6523,0.082 \!19) The empirical results show that integrating our first-order action supervision yields consistent performance improvements across a diverse range of offline policy optimization paradigms. Notably, our scheme can improve policy learning in cases where baseline algorithms fundamentally fail. On several complex, sparse-reward tasks, baselines collapse and obtain a score of exactly 00, indicating that they cannot extract any meaningful learning signal from the offline dataset. Equipping the same algorithm with our first-order supervision lifts these scores from 00 to clearly non-trivial levels. For example, on scene-play-singletask-task2-v0, IFQL completely fails (00), whereas IFQL with our first-order supervision reaches 8888, a near-expert level. The same breakthrough is observed on the puzzle-3x3-play domain (e.g., task3-v0 and task4-v0), where the baselines obtain no useful learning signal (scoring 00), while the augmented policies acquire the temporal consistency required to achieve non-trivial success rates. Beyond raising scores on already-learnable tasks, our scheme thus expands the set of tasks on which offline policy learning is feasible at all. The improvements are also consistent across all three policy paradigms. On high-dimensional dexterous manipulation tasks such as cube-double-play-singletask-task2-v0, our scheme raises the score of IFQL from 99 to 4040. On locomotion and navigation tasks such as antmaze-giant-navigate-singletask-task5-v0, integrating our scheme into ReBRAC further raises the score from 4949 to 9696, approaching expert-level proficiency with noticeably reduced seed-level variance. Together, these results highlight the broad applicability of our scheme and indicate that supervising higher-order action dynamics is crucial for stable offline policy learning and reliable closed-loop execution. Table 5: Average normalized scores on D4RL MuJoCo locomotion tasks under full-datasize. Standard deviations are reported after the ± sign. The ↑ [rgb]0.0195,0.6523,0.082 symbol indicates a performance increase greater than 1.01.0, whereas ↑ [rgb]0.5,0.5,0.5 denotes minor improvements. All results and standard deviations (±) are averaged across 5 random seeds. Det. Policy Sto. Policy (mean sup.) Sto. Policy (standard) Flow Policy Task Category TD3+BC TD3+BC+ours ReBRAC ReBRAC+ours IQL IQL+ours IFQL IFQL+ours Hopper-m 55.5±7.755.5 ± 7.7 58.9±7.4(↑3.4)58.9 ± 7.4( [rgb]0.0195,0.6523,0.082 \!3.4) 84.9±2.684.9 ± 2.6 85.7±1.3(↑0.8)85.7 ± 1.3( [rgb]0.5,0.5,0.5 \!0.8) 67.3±5.767.3 ± 5.7 68.8±5.5(↑1.5)68.8 ± 5.5( [rgb]0.0195,0.6523,0.082 \!1.5) 89.5±5.789.5 ± 5.7 94.2±7.2(↑4.7)94.2 ± 7.2( [rgb]0.0195,0.6523,0.082 \!4.7) Hopper-mr 57.9±17.657.9 ± 17.6 58.4±9.2(↑0.5)58.4 ± 9.2( [rgb]0.5,0.5,0.5 \!0.5) 81.2±25.181.2 ± 25.1 94.3±10.1(↑13.1)94.3 ± 10.1( [rgb]0.0195,0.6523,0.082 \!13.1) 91.7±8.691.7 ± 8.6 94.2±5.5(↑2.5)94.2 ± 5.5( [rgb]0.0195,0.6523,0.082 \!2.5) 96.2±4.796.2 ± 4.7 96.3±2.3(↑0.1)96.3 ± 2.3( [rgb]0.5,0.5,0.5 \!0.1) Hopper-me 89.8±15.289.8 ± 15.2 89.9±8.4(↑0.1)89.9 ± 8.4( [rgb]0.5,0.5,0.5 \!0.1) 106.4±10.7106.4 ± 10.7 106.6±5.4(↑0.2)106.6 ± 5.4( [rgb]0.5,0.5,0.5 \!0.2) 91.5±14.391.5 ± 14.3 109.2±9.8(↑17.7)109.2 ± 9.8( [rgb]0.0195,0.6523,0.082 \!17.7) 77.1±25.577.1 ± 25.5 89.8±8.7(↑12.7)89.8 ± 8.7( [rgb]0.0195,0.6523,0.082 \!12.7) Hopper-e 108.3±6.7108.3 ± 6.7 110.5±0.2(↑2.2)110.5 ± 0.2( [rgb]0.0195,0.6523,0.082 \!2.2) 108.5±4.1108.5 ± 4.1 108.6±3.8(↑0.1)108.6 ± 3.8( [rgb]0.5,0.5,0.5 \!0.1) 99.3±9.299.3 ± 9.2 106.5±5.1(↑7.2)106.5 ± 5.1( [rgb]0.0195,0.6523,0.082 \!7.2) 96.8±12.296.8 ± 12.2 106.2±8.6(↑9.4)106.2 ± 8.6( [rgb]0.0195,0.6523,0.082 \!9.4) Halfcheetah-m 47.8±0.447.8 ± 0.4 47.9±0.4(↑0.1)47.9 ± 0.4( [rgb]0.5,0.5,0.5 \!0.1) 55.4±1.055.4 ± 1.0 56.9±1.1(↑1.5)56.9 ± 1.1( [rgb]0.0195,0.6523,0.082 \!1.5) 47.4±0.447.4 ± 0.4 48.6±0.2(↑1.2)48.6 ± 0.2( [rgb]0.0195,0.6523,0.082 \!1.2) 51.1±0.651.1 ± 0.6 52.8±0.3(↑1.7)52.8 ± 0.3( [rgb]0.0195,0.6523,0.082 \!1.7) Halfcheetah-mr 41.6±0.341.6 ± 0.3 42.3±0.2(↑0.7)42.3 ± 0.2( [rgb]0.5,0.5,0.5 \!0.7) 46.6±0.346.6 ± 0.3 46.7±0.4(↑0.1)46.7 ± 0.4( [rgb]0.5,0.5,0.5 \!0.1) 41.3±2.341.3 ± 2.3 41.4±2.0(↑0.1)41.4 ± 2.0( [rgb]0.5,0.5,0.5 \!0.1) 42.5±0.442.5 ± 0.4 43.6±1.3(↑1.1)43.6 ± 1.3( [rgb]0.0195,0.6523,0.082 \!1.1) Halfcheetah-me 88.1±5.188.1 ± 5.1 89.5±4.7(↑1.4)89.5 ± 4.7( [rgb]0.0195,0.6523,0.082 \!1.4) 96.2±1.696.2 ± 1.6 97.4±1.4(↑1.2)97.4 ± 1.4( [rgb]0.0195,0.6523,0.082 \!1.2) 86.7±3.486.7 ± 3.4 87.4±2.6(↑0.7)87.4 ± 2.6( [rgb]0.5,0.5,0.5 \!0.7) 88.4±2.288.4 ± 2.2 89.5±1.6(↑1.1)89.5 ± 1.6( [rgb]0.0195,0.6523,0.082 \!1.1) Halfcheetah-e 90.1±0.390.1 ± 0.3 93.4±0.4(↑3.3)93.4 ± 0.4( [rgb]0.0195,0.6523,0.082 \!3.3) 91.3±1.091.3 ± 1.0 91.7±0.3(↑0.4)91.7 ± 0.3( [rgb]0.5,0.5,0.5 \!0.4) 89.6±7.889.6 ± 7.8 91.4±1.2(↑1.8)91.4 ± 1.2( [rgb]0.0195,0.6523,0.082 \!1.8) 91.2±1.591.2 ± 1.5 91.6±1.7(↑0.4)91.6 ± 1.7( [rgb]0.5,0.5,0.5 \!0.4) Walker2d-m 79.7±4.179.7 ± 4.1 80.1±1.6(↑0.4)80.1 ± 1.6( [rgb]0.5,0.5,0.5 \!0.4) 76.7±9.976.7 ± 9.9 82.6±2.2(↑5.9)82.6 ± 2.2( [rgb]0.0195,0.6523,0.082 \!5.9) 71.5±10.871.5 ± 10.8 78.8±5.0(↑7.3)78.8 ± 5.0( [rgb]0.0195,0.6523,0.082 \!7.3) 79.5±4.179.5 ± 4.1 82.6±4.4(↑3.1)82.6 ± 4.4( [rgb]0.0195,0.6523,0.082 \!3.1) Walker2d-mr 67.2±16.467.2 ± 16.4 73.3±4.1(↑6.1)73.3 ± 4.1( [rgb]0.0195,0.6523,0.082 \!6.1) 70.2±11.570.2 ± 11.5 83.6±7.1(↑13.4)83.6 ± 7.1( [rgb]0.0195,0.6523,0.082 \!13.4) 71.5±8.271.5 ± 8.2 73.9±5.5(↑2.4)73.9 ± 5.5( [rgb]0.0195,0.6523,0.082 \!2.4) 72.5±19.172.5 ± 19.1 83.1±7.1(↑10.6)83.1 ± 7.1( [rgb]0.0195,0.6523,0.082 \!10.6) Walker2d-me 108.8±0.5108.8 ± 0.5 108.9±0.3(↑0.1)108.9 ± 0.3( [rgb]0.5,0.5,0.5 \!0.1) 110.3±1.4110.3 ± 1.4 110.4±0.1(↑0.1)110.4 ± 0.1( [rgb]0.5,0.5,0.5 \!0.1) 108.4±0.7108.4 ± 0.7 108.5±0.5(↑0.1)108.5 ± 0.5( [rgb]0.5,0.5,0.5 \!0.1) 106.1±1.0106.1 ± 1.0 107.5±0.7(↑1.4)107.5 ± 0.7( [rgb]0.0195,0.6523,0.082 \!1.4) Walker2d-e 108.6±0.2108.6 ± 0.2 109.3±0.1(↑0.7)109.3 ± 0.1( [rgb]0.5,0.5,0.5 \!0.7) 102.2±1.5102.2 ± 1.5 109.7±0.5(↑7.5)109.7 ± 0.5( [rgb]0.0195,0.6523,0.082 \!7.5) 110.3±1.7110.3 ± 1.7 110.4±0.5(↑0.1)110.4 ± 0.5( [rgb]0.5,0.5,0.5 \!0.1) 102.7±11.5102.7 ± 11.5 106.4±0.5(↑3.7)106.4 ± 0.5( [rgb]0.0195,0.6523,0.082 \!3.7) Table 6: Imitation learning performance on D4RL MuJoCo locomotion tasks with 10k datasets. Average normalized scores across deterministic, stochastic, and flow-based policy paradigms. The ↑ [rgb]0.0195,0.6523,0.082 indicates a performance increase >1.0>1.0, whereas ↑ [rgb]0.5,0.5,0.5 represents marginal gains. All results and standard deviations (±) are averaged across 5 random seeds. Det. Policy Sto. Policy (standard) Flow Policy Task Category base base + ℒa˙detL_ a^det base base + ℒa˙stoL_ a^sto base base + ℒa˙flowL_ a^flow Hopper-e 21.5±13.521.5 ± 13.5 67.3±14.3(↑45.8)67.3 ± 14.3( [rgb]0.0195,0.6523,0.082 \!45.8) 57.5±15.257.5 ± 15.2 93.1±10.7(↑35.6)93.1 ± 10.7( [rgb]0.0195,0.6523,0.082 \!35.6) 72.2±9.272.2 ± 9.2 97.2±12.3(↑25.0)97.2 ± 12.3( [rgb]0.0195,0.6523,0.082 \!25.0) Halfcheetah-e 2.4±1.22.4 ± 1.2 5.7±1.5(↑3.3)5.7 ± 1.5( [rgb]0.0195,0.6523,0.082 \!3.3) 2.8±1.62.8 ± 1.6 16.2±10.1(↑13.4)16.2 ± 10.1( [rgb]0.0195,0.6523,0.082 \!13.4) 11.4±4.511.4 ± 4.5 17.9±6.9(↑6.5)17.9 ± 6.9( [rgb]0.0195,0.6523,0.082 \!6.5) Walker2d-e 26.7±12.226.7 ± 12.2 91.1±14.7(↑64.4)91.1 ± 14.7( [rgb]0.0195,0.6523,0.082 \!64.4) 46.4±15.246.4 ± 15.2 91.6±9.5(↑45.2)91.6 ± 9.5( [rgb]0.0195,0.6523,0.082 \!45.2) 89.8±14.689.8 ± 14.6 91.5±6.3(↑1.7)91.5 ± 6.3( [rgb]0.0195,0.6523,0.082 \!1.7) Evaluation results on D4RL under the full-dataset setting. To further evaluate the effectiveness of our scheme, we additionally conduct experiments on the D4RL MuJoCo locomotion tasks under the standard full-size dataset setting. As reported in Table 5, integrating our first-order action supervision yields consistent performance improvements across all four base algorithms, which further confirms the broad applicability of our scheme, showing it reliably stabilizes and improves offline policy learning across varying dataset sizes. Evaluation results of imitation learning methods on the D4RL 10k setting. Within each policy paradigm, “base” denotes the standard behavior cloning baseline natively implemented under the corresponding framework, while “base + ℒa˙(⋅)L_ a^(·)” denotes the same baseline equipped with our first-order action supervision loss. As shown in Table 6, our first-order supervision consistently improves all three behavior cloning baselines, demonstrating its standalone effectiveness even without any reward signal or RL-based optimization objective. In particular, on the deterministic Walker2d-e task, adding ℒa˙detL_ a^det raises the standard BC score from 26.726.7 to 91.191.1, an absolute improvement of 64.464.4. Such a substantial gain reveals that, under small-dataset settings, standard BC is highly susceptible to compounding errors along the rollout trajectory. By directly supervising higher-order action dynamics, our scheme provides the missing temporal regularization that aligns rollout actions with expert dynamics and effectively recovers the expert behavior. More detailed ablation studies that further validate the mechanism of first-order supervision are provided in the following subsections. Learning curves and policy rollout distributions of ‖a˙‖|| a||. We additionally present learning curves and rollout distributions of ‖a˙‖|| a|| for several baseline policies and their first-order action supervised variants, covering both imitation learning and offline RL settings. These results extend the analysis in Figure 2 and Figure 3 to additional algorithms and tasks, and they consistently show that adding our first-order supervision yields smoother learning curves and more concentrated ‖a˙‖|| a|| distributions, confirming that our scheme produces temporally consistent action evolution beyond the cases highlighted in the main text. Figure 5: Comparison of vanilla stochastic BC and our 1st1^st-order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation ‖a˙‖=‖a′−a‖|| a||=||a -a|| (a is joint torque) are aggregated from 10 trajectories. Figure 6: Comparison of vanilla IQL and our 1st1^st-order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation ‖a˙‖=‖a′−a‖|| a||=||a -a|| (a is joint torque) are aggregated from 10 trajectories. Training time comparison. Table 7 compares the training time of each base algorithm against its variant equipped with our first-order action supervision on hopper-medium-v2 (D4RL) and cube-double (OGBench). Across all four policy parameterizations, integrating our scheme only moderately increases the training time, indicating that our scheme is computationally efficient and practical for real-world use. All algorithms are trained on a workstation equipped with an Intel Xeon Platinum 8336C CPU @ 2.30 GHz and NVIDIA A100-SXM4-80GB GPUs, running Ubuntu 22.04.5 LTS 64-bit. Table 7: Training time comparison between each base algorithm and its variant equipped with our first-order action supervision. All runs use the same number of training steps and are conducted on a single GPU. Det. Policy Sto. Policy (mean sup.) Sto. Policy (standard) Flow Policy Task Category TD3+BC TD3+BC+ours ReBRAC ReBRAC+ours IQL IQL+ours IFQL IFQL+ours Training time (D4RL) 28 min 30 s 33 min 6 s 25 min 8 s 30 min 34 s 34 min 44 s 39 min 56 s 30 min 49 s 35 min 52 s Training time (OGBench) 28 min 23 s 38 min 56 s 25 min 11 s 31 min 54 s 35 min 7 s 42 min 5 s 32 min 0 s 36 min 51 s Generalizing to online RL. To evaluate the versatility of our approach beyond the offline setting, we integrate both the first-order action supervision and the continuous-time HJB value regularization into the standard SAC [15] algorithm, and evaluate on three MuJoCo benchmarks over 1M training steps. We keep SAC’s original hyperparameters intact, and set λa˙=λQ=0.1 _ a= _Q=0.1 for Hopper-v2 and Walker2d-v2, and λa˙=0.01,λQ=0.1 _ a=0.01, _Q=0.1 for HalfCheetah-v2. As reported in Table 8, incorporating our first-order supervision leads to noticeably faster convergence, which is particularly evident at 300k steps, alongside reduced performance variance across seeds. These results confirm that our lightweight, plug-and-play module generalizes effectively to the online RL paradigm, improving sample efficiency and training stability without modifying the base algorithm. Table 8: Performance snapshots of SAC and SAC+Ours on online MuJoCo benchmarks over 1M training steps. Results are averaged over 5 random seeds, with standard deviations reported after the ± sign. Bold indicates the better result at each snapshot. Environment Method 100k 300k 500k 700k 1M Hopper-v2 SAC 1767.1±1005.91767.1 ± 1005.9 2399.1±1391.82399.1 ± 1391.8 3076.8±405.93076.8 ± 405.9 3389.6±162.2 3389.6 ± 162.2 3530.4±32.63530.4 ± 32.6 SAC + Ours 2375.8±1036.4 2375.8 ± 1036.4 3273.9±111.6 3273.9 ± 111.6 3226.4±101.1 3226.4 ± 101.1 3359.2±116.43359.2 ± 116.4 3531.8±33.2 3531.8 ± 33.2 HalfCheetah-v2 SAC 5659.2±676.45659.2 ± 676.4 8564.9±1699.78564.9 ± 1699.7 9750.7±1228.69750.7 ± 1228.6 10805.1±804.710805.1 ± 804.7 11557.4±493.411557.4 ± 493.4 SAC + Ours 5790.9±553.5 5790.9 ± 553.5 8614.4±1423.9 8614.4 ± 1423.9 9954.8±1126.3 9954.8 ± 1126.3 10893.5±612.1 10893.5 ± 612.1 11858.2±414.1 11858.2 ± 414.1 Walker2d-v2 SAC 702.9±343.5702.9 ± 343.5 2413.7±1150.32413.7 ± 1150.3 2989.8±1820.92989.8 ± 1820.9 4291.2±950.54291.2 ± 950.5 4768.9±393.14768.9 ± 393.1 SAC + Ours 869.9±310.3 869.9 ± 310.3 3300.3±879.6 3300.3 ± 879.6 4141.4±572.9 4141.4 ± 572.9 4805.8±184.5 4805.8 ± 184.5 4991.1±264.5 4991.1 ± 264.5 B.2 Additional Ablation Results To verify that the benefits of our scheme are not specific to a single algorithmic instantiation, we conduct ablation studies on TD3+BC (deterministic policy) and IQL (stochastic policy), evaluated on the D4RL hopper-expert-v2 and walker2d-expert-v2 tasks under the 10k dataset setting. We sweep one of the two weights over 0,0.5,1\0,0.5,1\ while strictly setting the other to zero, so that the contribution of each component can be attributed independently. All results are averaged over 5 random seeds. Figure 7: Ablation study on value regularization weight λQ _Q and 1st1^st-order action supervision weight λa˙ _ a of TD3+BC. Figure 8: Ablation study on value regularization weight λQ _Q and 1st1^st-order action supervision weight λa˙ _ a of IQL Table 9: Performance on OGBench environments. Results are reported as mean ± standard deviation. Bold indicates the highest mean in each environment. Environment (OGBENCH) IFQL +Ours +AK (λa=0.5 _a=0.5) +AK (λa=0.1 _a=0.1) +AK (λa=1 _a=1) humanoidmaze-medium-navigate-singletask-task1-v0 69±1969 ± 19 71±1671 ± 16 73.2±7.0 73.2 ± 7.0 51.2±32.451.2 ± 32.4 67.6±7.167.6 ± 7.1 humanoidmaze-medium-navigate-singletask-task4-v0 ± 1 ± 1 ± 1 ± 1 0.0±0.00.0 ± 0.0 0.4±0.80.4 ± 0.8 0.0±0.00.0 ± 0.0 antsoccer-arena-navigate-singletask-task2-v0 75±375 ± 3 ± 83 ± 12 66.8±15.366.8 ± 15.3 40.4±33.640.4 ± 33.6 35.2±20.435.2 ± 20.4 antsoccer-arena-navigate-singletask-task4-v0 16±916 ± 9 16±816 ± 8 13.6±9.613.6 ± 9.6 20.4±20.8 20.4 ± 20.8 6.4±7.36.4 ± 7.3 cube-single-play-singletask-task2-v0 73±373 ± 3 ± 100 ± 1 90.0±7.090.0 ± 7.0 27.2±33.527.2 ± 33.5 94.0±2.894.0 ± 2.8 cube-single-play-singletask-task4-v0 79±679 ± 6 88±1688 ± 16 93.6±2.3 93.6 ± 2.3 26.8±34.226.8 ± 34.2 92.0±4.092.0 ± 4.0 cube-double-play-singletask-task1-v0 35±935 ± 9 ± 76 ± 8 23.6±7.523.6 ± 7.5 29.6±5.629.6 ± 5.6 25.6±10.825.6 ± 10.8 cube-double-play-singletask-task4-v0 1±11 ± 1 ± 12 ± 10 1.2±1.01.2 ± 1.0 1.2±1.01.2 ± 1.0 1.2±1.01.2 ± 1.0 puzzle-4x4-play-singletask-task2-v0 4±44 ± 4 ± 12 ± 9 2.8±5.62.8 ± 5.6 0.0±0.00.0 ± 0.0 8.4±4.38.4 ± 4.3 puzzle-4x4-play-singletask-task3-v0 50±1450 ± 14 ± 84 ± 16 5.6±10.25.6 ± 10.2 0.0±0.00.0 ± 0.0 37.2±7.837.2 ± 7.8 Table 10: Computational cost and GPU memory usage of Ours and AK. Additional overhead is measured relative to Ours. Metric Ours AK Additional overhead Computational cost (per update) 2.498 GFLOPs 4.978 GFLOPs +2.480 GFLOPs (+99.3%) GPU memory 1.141 GiB 1.266 GiB +0.125 GiB (+11.0%) As shown in Figures 8 and 8, both algorithms exhibit the same trend: increasing either the first-order action supervision weight λa˙ _ a or the value regularization weight λQ _Q from 0 to 1 yields steady performance gains. This confirms that the policy-side and value-side first-order regularizers are robustly beneficial across both deterministic and stochastic policy classes. To evaluate the action-dependent term omitted from Eq. (7), we compare Ours with an ablation denoted as AK. AK adds ∂v(k,s,ak)∂aka˙k ∂ v(k,s,a^k)∂ a^k a^k from Eq. (6) to the state-directional term, uses the same objective as Eq. (7), replacing ℒa˙flowL_ a^flow with ℒa˙AK=‖∂v(k,s,ak)∂ss˙+∂v(k,s,ak)∂aka˙k−a˙1‖.L_ a^AK= \| ∂ v(k,s,a^k)∂ s s+ ∂ v(k,s,a^k)∂ a^k a^k- a^1 \|. (12) Here a˙k=ka˙1 a^k=k a^1 on the linear interpolation path. Overall, Table 9 shows that Ours performs more consistently across OGBench environments. It matches or improves upon IFQL on every task and generally outperforms AK across the tested values of λa _a. Although AK benefits a few tasks, it produces substantial drops on others, particularly cube-double and puzzle-4x4. Thus, including the action-dependent term does not yield a reliable performance benefit. Table 10 further shows that AK nearly doubles computation per update (+99.3%) and increases GPU memory usage by 11.0% relative to Ours. Together, these results support our empirical choice of the state-directional approximation. Appendix C Derivation of the Higher-Order Supervision This section provides the derivations of the higher-order policy as demonstrated in section 3. Specifically, Appendix C.1 derives the action differential for stochastic Gaussian policies, while Appendix C.2 derives the time derivative of the generated action for flow-based policies. C.1 First-Order Stochastic Gaussian Policies Derivation in Eq. (3) Consider the Gaussian policy with diagonal covariance π(a|s)=(a,μ(s),diag(σ2(s))),π(a|s)=N\! (a;μ(s),diag(σ^2(s)) ), (13) where a,μ(s),σ(s)∈ℝda,μ(s),σ(s) ^d, and σ2(s)σ^2(s) denotes the element-wise variance. We use σ(s)σ(s) to denote the element-wise standard deviation, i.e., the element-wise square root of the variance output. We represent the stochastic component of the sampled action by a=μ(s)+σ(s)⊙Xt,a=μ(s)+σ(s) X_t, (14) where ⊙ denotes element-wise multiplication, and Xt∈ℝdX_t ^d is assumed to follow the stationary vector OU process [58] dXt=−λXtdt+2λdWt,X0∼(0,I).dX_t=-λ X_t\,dt+ 2λ\,dW_t, X_0 (0,I). (15) Here WtW_t is a d-dimensional standard Wiener process. By stationarity, Xt∼(0,I)X_t (0,I) for all t. Under the diagonal covariance assumption, the stochastic perturbation is applied independently across action dimensions. Therefore, a=μ(s)+σ(s)⊙Xt,Xt∼(0,I)⟹a∼(μ(s),diag(σ2(s))).a=μ(s)+σ(s) X_t, X_t (0,I) a \! (μ(s),diag(σ^2(s)) ). We condition on (s,s˙)(s, s) and treat s=s(t)s=s(t) i.e., ds=s˙dtds= s\,dt as a differentiable trajectory. For notational simplicity, we omit the explicit dependence of s on t. Define F(t,Xt)=μ(s)+σ(s)⊙Xt,a=F(t,Xt).F(t,X_t)=μ(s)+σ(s) X_t, a=F(t,X_t). For each action dimension i, define ai=Fi(t,Xt)=μi(s)+σi(s)Xt,i.a_i=F_i(t,X_t)= _i(s)+ _i(s)X_t,i. (16) Since XtX_t is a vector Itô process, applying the multi-dimensional Itô formula [47] to Fi(t,Xt)F_i(t,X_t) and substituting the OU dynamics of XtX_t gives, dai _i =[∂Fi∂t+∇XFi(t,Xt)⊤(−λXt)+12Tr((2λI)⊤HXFi(t,Xt)(2λI))]dt = [ ∂ F_i∂ t+ _XF_i(t,X_t) (-λ X_t )+ 12Tr (( 2λI) H_XF_i(t,X_t)( 2λI) ) ]dt +∇XFi(t,Xt)⊤2λIdWt + _XF_i(t,X_t) 2λI\,dW_t =[∂Fi∂t−λ∇XFi(t,Xt)⊤Xt+λTr(HXFi(t,Xt))]dt = [ ∂ F_i∂ t-λ _XF_i(t,X_t) X_t+ (H_XF_i(t,X_t) ) ]dt +2λ∇XFi(t,Xt)⊤dWt. + 2λ\, _XF_i(t,X_t) dW_t. (17) From Eq. (16), FiF_i is linear with respect to XtX_t. HXFi(t,Xt)=0.H_XF_i(t,X_t)=0. (18) Moreover, ∂Fi∂t=μ˙i(s)+σ˙i(s)Xt,i,∇XFi(t,Xt)=σi(s)ei, ∂ F_i∂ t= μ_i(s)+ σ_i(s)X_t,i, _XF_i(t,X_t)= _i(s)e_i, (19) where eie_i is the i-th standard basis vector. Substituting these identities into the previous expression gives dai=[μ˙i(s)+(σ˙i(s)−λσi(s))Xt,i]dt+2λσi(s)dWt,i.da_i= [ μ_i(s)+ ( σ_i(s)-λ _i(s) )X_t,i ]dt+ 2λ\, _i(s)dW_t,i. (20) Stacking all action dimensions yields da=[μ˙(s)+(σ˙(s)−λσ(s))⊙Xt]dt+2λσ(s)⊙dWt.da= [ μ(s)+ ( σ(s)-λσ(s) ) X_t ]dt+ 2λ\,σ(s) _t. (21) Using Eq. (14), we obtain Xt=(a−μ(s))⊘σ(s),X_t= (a-μ(s) ) σ(s), (22) where ⊘ denotes element-wise division. Hence da=[μ˙(s)+(σ˙(s)−λσ(s))⊙((a−μ(s))⊘σ(s))]dt+2λσ(s)⊙dWt.da= [ μ(s)+ ( σ(s)-λσ(s) ) ( (a-μ(s) ) σ(s) ) ]dt+ 2λ\,σ(s) _t. (23) Since the infinitesimal Wiener increment satisfies dWt∼(0,Idt)dW_t (0,Idt), Eq. (23) implies that the infinitesimal action increment conditioned on (s,a,s˙)(s,a, s) follows da∼([μ˙(s)+(σ˙(s)−λσ(s))⊙((a−μ(s))⊘σ(s))]dt, 2λdiag(σ2(s))dt).da \! ( [ μ(s)+ ( σ(s)-λσ(s) ) ( (a-μ(s) ) σ(s) ) ]dt,\,2 (σ^2(s))dt ). (24) As mentioned in Section 3.1, in our discrete-time implementation, we use the finite-difference approximation a˙≈(a′−a)/Δt a≈(a -a)/ t with Δt=1 t=1. Under this unit-step finite-difference approximation, the induced first-order policy distribution over the action derivative is π˙(a˙|s,a,s˙)=(a˙,μ˙(s)+(σ˙(s)−λσ(s))⊙((a−μ(s))⊘σ(s)), 2λdiag(σ2(s))). π( a|s,a, s)=N\! ( a;\, μ(s)+ ( σ(s)-λσ(s) ) ( (a-μ(s) ) σ(s) ),\,2 (σ^2(s)) ). (25) C.2 First-Order Flow Policies Derivation in Eq. (6) Given the state s and initial noise sample a0a^0, the generated action is obtained by solving dakdk=v(k,s,ak),ak=0=a0, da^kdk=v(k,s,a^k), a^k=0=a^0, (26) and taking a1=π(s)a^1=π(s), a1=a0+∫01v(k,s,ak)k.a^1=a^0+ _0^1v(k,s,a^k)\,dk. (27) Assuming sufficient smoothness, differentiating a1a^1 with respect to physical time t and applying the Leibniz integral rule gives a˙1 a^1 =a˙0+dt∫01v(k,s,ak)k = a^0+ ddt _0^1v(k,s,a^k)\,dk =a˙0+∫01dtv(k,s,ak)k. = a^0+ _0^1 ddtv(k,s,a^k)\,dk. (28) Since v(k,s,ak)v(k,s,a^k) depends on t through both s and aka^k, the chain rule gives dtv(k,s,ak)=∂v(k,s,ak)∂ss˙+∂v(k,s,ak)∂aka˙k. ddtv(k,s,a^k)= ∂ v(k,s,a^k)∂ s s+ ∂ v(k,s,a^k)∂ a^k a^k. (29) Conditioning on the sampled noise a0a^0 when taking the physical-time derivative gives a˙0=0 a^0=0. Combining Eqs. (28) and (29) yields a˙1=∫01v˙(k,s,ak)k=∫01(∂v(k,s,ak)∂ss˙+∂v(k,s,ak)∂aka˙k)k. a^1= _0^1 v(k,s,a^k)\,dk= _0^1 ( ∂ v(k,s,a^k)∂ s s+ ∂ v(k,s,a^k)∂ a^k a^k )dk. (30) We consider the linear interpolation path used in flow matching, ak=(1−k)a0+ka1,k∈[0,1].a^k=(1-k)a^0+ka^1, k∈[0,1]. (31) Taking its physical-time derivative while holding a0a^0 fixed gives a˙k=(1−k)a˙0+ka˙1=ka˙1. a^k=(1-k) a^0+k a^1=k a^1. (32) Equation (30) therefore takes the following form along this path, a˙1=∫01(∂v(k,s,ak)∂ss˙+k∂v(k,s,ak)∂aka˙1)k. a^1= _0^1 ( ∂ v(k,s,a^k)∂ s s+k ∂ v(k,s,a^k)∂ a^k a^1 )dk. (33) As in the zeroth-order action generation process in Eq. (27), the first-order action is obtained by integrating a vector field over flow time k. For the linear path, the sample-wise flow-matching target is dakdk=a1−a0 da^kdk=a^1-a^0. Taking its physical-time derivative while holding a0a^0 fixed gives dt(a1−a0)=a˙1 ddt(a^1-a^0)= a^1. Thus, a˙1=a′−a1 a^1=a -a^1 provides a pointwise target for v˙(k,s,ak) v(k,s,a^k). The shared integral structure motivates supervising this first-order field in a flow-matching manner. Based on the computational cost and performance results in Appendix B.2, we approximate v˙(k,s,ak) v(k,s,a^k) by ∂v(k,s,ak)∂ss˙ ∂ v(k,s,a^k)∂ s s. Accordingly, Eq. (7) supervises this approximation at sampled values of k; the integral describes action generation rather than the training loss. Appendix D Theoretical Analysis We first introduce the formal notation, assumptions, and auxiliary lemmas used in the analysis. Then we provide the proofs of Theorems 1 and 2. D.1 Notation We briefly recap the formal notation used in Section 3.3, with the time dependence made explicit for the analysis below. Let π and π∗π^* denote the learned and expert policies, and let s(t)s(t) and s∗(t)s^*(t) denote their corresponding trajectories. The closed-loop dynamics are s˙(t)=f(s(t),π(s(t))) s(t)=f(s(t),π(s(t))) and s˙∗(t)=f(s∗(t),π∗(s∗(t))) s^*(t)=f(s^*(t),π^*(s^*(t))). The policy error and state deviation are denoted by e(s)=π(s)−π∗(s)e(s)=π(s)-π^*(s) and δ(t)=s(t)−s∗(t)δ(t)=s(t)-s^*(t), respectively. For each time t, define T(t)=s∗(t)+τ(s(t)−s∗(t)):τ∈[0,1]T(t)=\s^*(t)+τ(s(t)-s^*(t)):τ∈[0,1]\. D.2 Assumptions Assumption 1 (Domain and regularity). There exists a convex compact set Ω⊂ℝd ^d such that s(t),s∗(t)∈Ωs(t),s^*(t)∈ for all t under consideration. The policies π and π∗π^* are continuously differentiable on Ω , and the policy error e admits the Jacobian ∇se _se on Ω . Assumption 2 (Lipschitz dynamics). For any s1,s2∈Ωs_1,s_2∈ and any a∈ℝma ^m, the dynamics f is Lipschitz continuous in the state variable with constant LsL_s, i.e., ‖f(s1,a)−f(s2,a)‖≤Ls‖s1−s2‖.\|f(s_1,a)-f(s_2,a)\|≤ L_s\|s_1-s_2\|. For any s∈Ωs∈ and any a1,a2∈ℝma_1,a_2 ^m, the dynamics f is Lipschitz continuous in the action variable with constant LaL_a, i.e., ‖f(s,a1)−f(s,a2)‖≤La‖a1−a2‖.\|f(s,a_1)-f(s,a_2)\|≤ L_a\|a_1-a_2\|. Assumption 3 (Lipschitz expert policy). The expert policy π∗π^* is Lipschitz continuous with constant Lπ∗L_π^* on Ω , i.e., for any s1,s2∈Ωs_1,s_2∈ , ‖π∗(s1)−π∗(s2)‖≤Lπ∗‖s1−s2‖.\|π^*(s_1)-π^*(s_2)\|≤ L_π^*\|s_1-s_2\|. Assumption 4 (Bounded ∇se _se). For all t under consideration, the Jacobian of the policy error is bounded on T(t)T(t) by LeL_e. That is, for any z∈T(t)z∈ T(t), ‖∇se(z)‖≤Le. \| _se(z) \|≤ L_e. D.3 Auxiliary Lemmas Lemma 1. Under Assumption 1, for any s1,s2∈Ωs_1,s_2∈ , letting Δ=s1−s2 =s_1-s_2 and γ(τ)=s2+τΔγ(τ)=s_2+τ for τ∈[0,1]τ∈[0,1], we have e(s1)−e(s2)=∫01∇se(γ(τ))Δτ.e(s_1)-e(s_2)= _0^1 _se(γ(τ))\, \,dτ. Consequently, for any time t, along the segment T(t)T(t), we have ‖e(s(t))‖≤‖e(s∗(t))‖+supz∈T(t)‖∇se(z)‖‖s(t)−s∗(t)‖.\|e(s(t))\|≤\|e(s^*(t))\|+ _z∈ T(t) \| _se(z) \|\,\|s(t)-s^*(t)\|. Proof. Fix s1,s2∈Ωs_1,s_2∈ , and define Δ=s1−s2 =s_1-s_2 and γ(τ)=s2+τΔγ(τ)=s_2+τ . By Assumption 1, γ(τ)∈Ωγ(τ)∈ for all τ∈[0,1]τ∈[0,1]. Let g(τ)=e(γ(τ))g(τ)=e(γ(τ)). Since e is continuously differentiable by Assumption 1, the chain rule gives g′(τ)=∇se(γ(τ))Δ.g (τ)= _se(γ(τ))\, . By the fundamental theorem of calculus, e(s1)−e(s2)=g(1)−g(0)=∫01g′(τ)τ=∫01∇se(γ(τ))Δτ.e(s_1)-e(s_2)=g(1)-g(0)= _0^1g (τ)\,dτ= _0^1 _se(γ(τ))\, \,dτ. Now take s1=s(t)s_1=s(t) and s2=s∗(t)s_2=s^*(t). By the definitions of δ(t)δ(t) and T(t)T(t), we have Δ=δ(t) =δ(t) and γ(τ)∈T(t)γ(τ)∈ T(t). Therefore, e(s(t))−e(s∗(t))=∫01∇se(γ(τ))δ(t)τ.e(s(t))-e(s^*(t))= _0^1 _se(γ(τ))\,δ(t)\,dτ. Taking norms yields ‖e(s(t))‖ \|e(s(t))\| ≤‖e(s∗(t))‖+∫01‖∇se(γ(τ))δ(t)‖τ ≤\|e(s^*(t))\|+ _0^1 \| _se(γ(τ))\,δ(t) \|\,dτ ≤‖e(s∗(t))‖+∫01‖∇se(γ(τ))‖‖δ(t)‖τ ≤\|e(s^*(t))\|+ _0^1 \| _se(γ(τ)) \|\,\|δ(t)\|\,dτ ≤‖e(s∗(t))‖+supz∈T(t)‖∇se(z)‖‖δ(t)‖. ≤\|e(s^*(t))\|+ _z∈ T(t) \| _se(z) \|\,\|δ(t)\|. Using δ(t)=s(t)−s∗(t)δ(t)=s(t)-s^*(t), we obtain ‖e(s(t))‖≤‖e(s∗(t))‖+supz∈T(t)‖∇se(z)‖‖s(t)−s∗(t)‖.\|e(s(t))\|≤\|e(s^*(t))\|+ _z∈ T(t) \| _se(z) \|\,\|s(t)-s^*(t)\|. This completes the proof. ∎ Lemma 2 (Gronwall’s lemma). [12] Let T∈(0,∞)T∈(0,∞). Assume u is absolutely continuous on [0,T)[0,T) and satisfies the differential inequality u˙(t)≤a(t)u(t)+b(t),for a.e. t∈(0,T), u(t)≤ a(t)\,u(t)+b(t), a.e. t∈(0,T), for some a,b∈L1(0,T)a,b∈ L^1(0,T). Then u satisfies the pointwise estimate u(t)≤eA(t)u(0)+∫0tb(s)eA(t)−A(s)s,∀t∈[0,T),u(t)≤ e^A(t)u(0)+ _0^tb(s)\,e^A(t)-A(s)\,ds, ∀\,t∈[0,T), where we define A(t)=∫0ta(s)s.A(t)= _0^ta(s)\,ds. This is the differential form of Gronwall’s inequality, see Proposition 2.2 of [12] for a proof. D.4 Proof of Theorem 1 Theorem 1. Under Assumption 1, the expert-trajectory policy error e(s∗(t))e(s^*(t)) satisfies: ‖e(s∗(t))‖≤‖e(s∗(0))‖+tmax0≤τ≤t‖∇sπ(s∗(τ))s˙∗(τ)−π˙∗(s∗(τ))‖.\|e(s^*(t))\|≤\|e(s^*(0))\|+t\, _0≤τ≤ t \| _sπ(s^*(τ)) s^*(τ)- π^*(s^*(τ)) \|. Proof. Recall the definition of the expert-trajectory policy error: e(s∗(t))=π(s∗(t))−π∗(s∗(t)).e(s^*(t))=π(s^*(t))-π^*(s^*(t)). Define the expert action derivative along the expert trajectory as π˙∗(s∗(t))=dtπ∗(s∗(t))=∇sπ∗(s∗(t))s˙∗(t). π^*(s^*(t))= ddtπ^*(s^*(t))= _sπ^*(s^*(t))\, s^*(t). By Assumption 1 and the chain rule, the time derivative of the expert-trajectory policy error is dte(s∗(t))=∇sπ(s∗(t))s˙∗(t)−π˙∗(s∗(t)). ddte(s^*(t))= _sπ(s^*(t))\, s^*(t)- π^*(s^*(t)). Integrating over time yields: e(s∗(t))=e(s∗(0))+∫0t(∇sπ(s∗(τ))s˙∗(τ)−π˙∗(s∗(τ)))τ.e(s^*(t))=e(s^*(0))+ _0^t ( _sπ(s^*(τ))\, s^*(τ)- π^*(s^*(τ)) )\,dτ. Taking norms: ‖e(s∗(t))‖≤‖e(s∗(0))‖+∫0t‖∇sπ(s∗(τ))s˙∗(τ)−π˙∗(s∗(τ))‖τ.\|e(s^*(t))\|≤\|e(s^*(0))\|+ _0^t \| _sπ(s^*(τ))\, s^*(τ)- π^*(s^*(τ)) \|\,dτ. Therefore, ‖e(s∗(t))‖≤‖e(s∗(0))‖+tmax0≤τ≤t‖∇sπ(s∗(τ))s˙∗(τ)−π˙∗(s∗(τ))‖.\|e(s^*(t))\|≤\|e(s^*(0))\|+t\, _0≤τ≤ t \| _sπ(s^*(τ))\, s^*(τ)- π^*(s^*(τ)) \|. This completes the proof. ∎ D.5 Proof of Theorem 2 Theorem 2. Under Assumptions 1–4, for δ(0)=0δ(0)=0, the state deviation δ(t)=s(t)−s∗(t)δ(t)=s(t)-s^*(t) satisfies ‖δ(t)‖≤Laβ(eβt−1)max0≤τ≤t‖e(s∗(τ))‖,whereβ=Ls+LaLπ∗+LaLe.\|δ(t)\|≤ L_aβ (e^β t-1 ) _0≤τ≤ t\|e(s^*(τ))\|, β=L_s+L_aL_π^*+L_aL_e. Proof. From Lemma 1 and Assumption 4, we have: ‖e(s(t))‖≤‖e(s∗(t))‖+Le‖s(t)−s∗(t)‖.\|e(s(t))\|≤\|e(s^*(t))\|+L_e\|s(t)-s^*(t)\|. (34) The time derivative of the state deviation is: δ˙(t)=f(s(t),π(s(t)))−f(s∗(t),π∗(s∗(t))). δ(t)=f(s(t),π(s(t)))-f(s^*(t),π^*(s^*(t))). By Assumptions 2 and 3, we have ‖δ˙(t)‖ \| δ(t)\| =‖f(s(t),π(s(t)))−f(s∗(t),π∗(s∗(t)))‖ =\|f(s(t),π(s(t)))-f(s^*(t),π^*(s^*(t)))\| ≤|f(s(t),π(s(t)))−f(s(t),π∗(s(t)))|+‖f(s(t),π∗(s(t)))−f(s∗(t),π∗(s∗(t)))‖ ≤\|f(s(t),π(s(t)))-f(s(t),π^*(s(t)))\|+\|f(s(t),π^*(s(t)))-f(s^*(t),π^*(s^*(t)))\| ≤La|π(s(t))−π∗(s(t))|+‖f(s(t),π∗(s(t)))−f(s∗(t),π∗(s(t)))‖ ≤ L_a\|π(s(t))-π^*(s(t))\|+\|f(s(t),π^*(s(t)))-f(s^*(t),π^*(s(t)))\| +‖f(s∗(t),π∗(s(t)))−f(s∗(t),π∗(s∗(t)))‖ +\|f(s^*(t),π^*(s(t)))-f(s^*(t),π^*(s^*(t)))\| ≤La‖e(s(t))‖+Ls|s(t)−s∗(t)|+La‖π∗(s(t))−π∗(s∗(t))‖ ≤ L_a\|e(s(t))\|+L_s\|s(t)-s^*(t)\|+L_a\|π^*(s(t))-π^*(s^*(t))\| ≤La|e(s(t))‖+Ls‖δ(t)|+LaLπ∗‖δ(t)‖ ≤ L_a\|e(s(t))\|+L_s\|δ(t)\|+L_aL_π^*\|δ(t)\| =La|e(s(t))|+(Ls+LaLπ∗)‖δ(t)‖. =L_a\|e(s(t))\|+ (L_s+L_aL_π^* )\|δ(t)\|. Using the bound in Eq. (34), we get: ‖δ˙(t)‖≤La‖e(s∗(t))‖+(Ls+LaLπ∗+LaLe)‖δ(t)‖.\| δ(t)\|≤ L_a\|e(s^*(t))\|+ (L_s+L_aL_π^*+L_aL_e )\|δ(t)\|. Let β=Ls+LaLπ∗+LaLeβ=L_s+L_aL_π^*+L_aL_e. Then: ‖δ˙(t)‖≤β‖δ(t)‖+La‖e(s∗(t))‖.\| δ(t)\|≤β\|δ(t)\|+L_a\|e(s^*(t))\|. Let u(t)=‖δ(t)‖u(t)=\|δ(t)\|. Under Assumptions 1 and 2, the trajectories s(t)s(t) and s∗(t)s^*(t) are absolutely continuous. Hence, δ(t)=s(t)−s∗(t)δ(t)=s(t)-s^*(t) is absolutely continuous. Since the norm is Lipschitz continuous, u(t)u(t) is also absolutely continuous. Moreover, for almost all t∈(0,T)t∈(0,T), we have u˙(t)≤‖δ˙(t)‖. u(t)≤\| δ(t)\|. Therefore, u˙(t)≤βu(t)+La‖e(s∗(t))‖,for a.e. t∈(0,T). u(t)≤β u(t)+L_a\|e(s^*(t))\|, a.e. t∈(0,T). (35) For Eq. (35), applying Lemma 2 with a(t)=βa(t)=β and b(t)=La‖e(s∗(t))‖b(t)=L_a\|e(s^*(t))\| yields ‖δ(t)‖≤eβt‖δ(0)‖+La∫0teβ(t−τ)‖e(s∗(τ))‖τ.\|δ(t)\|≤ e^β t\|δ(0)\|+L_a _0^te^β(t-τ)\|e(s^*(τ))\|\,dτ. This can be further bounded as: ‖δ(t)‖≤eβt|δ(0)|+Laβ(eβt−1)max0≤τ≤t‖e(s∗(τ))‖.\|δ(t)\|≤ e^β t\|δ(0)\|+ L_aβ (e^β t-1 ) _0≤τ≤ t\|e(s^*(τ))\|. Assuming δ(0)=0δ(0)=0, we get: ‖δ(t)‖≤Laβ(eβt−1)max0≤τ≤t‖e(s∗(τ))‖.\|δ(t)\|≤ L_aβ (e^β t-1 ) _0≤τ≤ t\|e(s^*(τ))\|. This completes the proof. ∎ Appendix E Experimental Details We evaluate our method on two widely used offline RL benchmarks: OGBench [48] and D4RL [13]. OGBench provides high-dimensional, long-horizon, and structurally diverse tasks that go well beyond standard locomotion, while D4RL offers a well-established testbed of MuJoCo locomotion tasks with datasets of varying behavioral quality. Together, they cover a broad spectrum of state and action dimensionalities, task structures, and dataset qualities, providing a comprehensive evaluation of our scheme. All methods are trained on the offline datasets only, with no online interaction. E.1 Benchmarks and Tasks OGBench. Since OGBench was originally designed for goal-conditioned RL, we adopt its reward-based single-task variants (denoted by the -singletask suffix) to benchmark standard reward-maximizing offline RL methods. In total, we evaluate on 5050 state-based tasks drawn from 1010 environments (55 tasks per environment), covering long-horizon locomotion (antmaze, humanoidmaze), object interaction (antsoccer), and robotic manipulation (cube, scene, puzzle), where scene additionally requires long-horizon multi-object manipulation and puzzle involves more combinatorial reasoning. These tasks pose substantial challenges in high-dimensional continuous control, with state dimensions ranging from 2828 to 8383 and action dimensions from 55 to 2121. For instance, humanoidmaze has a 6969-dimensional state space and a 2121-dimensional action space, while cube-double, scene, and puzzle-4x4 have 3737-, 4040-, and 8383-dimensional state spaces, respectively. We use the standard reward-based datasets provided by OGBench, which are highly suboptimal as they aggregate trajectories from policies performing diverse random tasks (e.g., reaching different goals or manipulating different objects). This makes it particularly difficult for offline RL methods to extract task-relevant behaviors from such heterogeneous data. D4RL. We further evaluate our method on the MuJoCo locomotion tasks from D4RL, including halfcheetah, hopper, and walker2d, with the standard medium, medium-replay, medium-expert, and expert dataset types collected from behavior policies of varying quality. The full-size datasets contain about 11M transitions for the medium and expert settings, 22M transitions for medium-expert, and around 200200k–400400k transitions for medium-replay. Beyond the standard full-data setting, we additionally consider a small-dataset setting to study the data efficiency of our scheme. Specifically, for each task we first split the original dataset into individual trajectories, and then sample a small set of complete trajectories until the accumulated transitions reach 1010k, rather than randomly sampling individual transitions. This trajectory-level sampling preserves the temporal continuity and dynamical consistency of the original demonstrations, which is essential for our first-order action supervision since it relies on consecutive transitions (s,s′,a,a′)(s,s ,a,a ) to compute the local state and action derivatives (s˙,a˙)( s, a). To ensure a fair comparison, all methods use the same subsampled dataset under the same random seed. E.2 Methods and Hyperparameters We instantiate our first-order action supervision on four representative offline RL algorithms covering deterministic, stochastic, and flow-based policies: TD3+BC [14], ReBRAC [57], IQL [27], and IFQL [49]. For a fair comparison, all algorithms are implemented on top of the FQL [49] codebase and share the same network architectures and default hyperparameters unless otherwise specified. For OGBench tasks, we follow the hyperparameter settings reported in FQL [49] for ReBRAC, IQL, and IFQL. After integrating our scheme, we keep all baseline-specific hyperparameters unchanged, and only specify the two additional weights λQ _Q (the value-side regularization weight) and λa˙ _ a (the policy-side first-order supervision weight) introduced by our scheme. The basic shared hyperparameters are summarized in Table 11. TD3+BC. We re-implement TD3+BC on top of the FQL [49] codebase. On OGBench, we sweep the Q-value scaling coefficient α over 1,3,10,30,100,300\1,3,10,30,100,300\ and select the best value per environment. Intuitively, a smaller α is preferable in complex environments where the Q-function is hard to learn reliably, while a larger α works better on simpler ones where the learned Q-function offers stronger policy improvement signals. We also observe Q-learning instability on several challenging OGBench tasks, and apply layer normalization to the Q-network to mitigate this issue. On D4RL, we follow the hyperparameter settings of the original TD3+BC paper [14], and use the default values for all other hyperparameters (e.g., noise standard deviation, noise clipping threshold). For our scheme, we set λQ=0.1 _Q=0.1 on all OGBench tasks, λa˙=1.0 _ a=1.0 on antmaze and λa˙=0.1 _ a=0.1 on all other OGBench tasks; on D4RL, we set λQ=0.1 _Q=0.1 and λa˙=1.0 _ a=1.0. ReBRAC. We follow the hyperparameter settings reported in the original ReBRAC paper [57] on D4RL, including the actor and critic BC coefficients (αa,αc)( _a, _c). For our scheme, we set λQ=0.1 _Q=0.1 on all OGBench tasks, λa˙=1.0αa _ a=1.0\, _a on antmaze, humanoidmaze, and antsoccer, and λa˙=0.1αa _ a=0.1\, _a on cube, scene, and puzzle; on D4RL, we set λQ=0.1 _Q=0.1 and λa˙=1.0αa _ a=1.0\, _a. IQL. We follow the hyperparameter settings reported in the original IQL paper [27] on D4RL, including the expectile value and the AWR inverse temperature. For our scheme on OGBench, we set λQ=0.5 _Q=0.5 on scene and λQ=0.1 _Q=0.1 on all other tasks, and λa˙=1.0 _ a=1.0 on antmaze and scene and λa˙=0.1 _ a=0.1 on all other tasks; on D4RL, we set λQ=0.1 _Q=0.1 and λa˙=1.0 _ a=1.0. IFQL. For IFQL, we use N=32N=32 action samples on D4RL tasks, and follow all other settings from FQL [49]. For our scheme, we set λQ=1.0 _Q=1.0 and λa˙=1.0 _ a=1.0 on all OGBench and D4RL tasks. Table 11: Shared hyperparameters used across all four offline RL algorithms. Hyperparameter Value Learning rate 3×10−43× 10^-4 Optimizer Adam [24] Gradient steps 1,000,0001,000,000 Minibatch size 256256 MLP dimensions [512,512,512,512][512,512,512,512] (default); [256,256,256,256][256,256,256,256] (IQL) Nonlinearity GELU [18] Target network smoothing coefficient 0.0050.005 Discount factor γ 0.990.99 (default); 0.9950.995 (antmaze-giant, humanoidmaze, antsoccer) Temporal correlation coefficient λ 1.01.0 Appendix F Broader Impact Modern learning-based decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), often suffer from control instability and poor robustness when deployed in real-world systems, due to their sole reliance on zeroth-order action supervision. To address this, we propose a lightweight and plug-and-play learning scheme that simultaneously supervises zeroth- and first-order actions, transforming any standard policy model into a more powerful policy class without structural modifications. Our method substantially improves action smoothness, control stability, and out-of-distribution generalization, especially in challenging low-data regimes, making it well-suited for safety-critical domains such as robotics, drone control, and autonomous driving, where high-frequency action fluctuations can cause mechanical wear, energy waste, and safety risks. By bridging classical control principles with modern policy learning, our work offers a promising pathway toward more reliable and deployable learning-based controllers. A potential limitation is that, like other offline methods, the learned policy may still inherit unsafe or suboptimal behaviors present in the historical dataset. NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We introduce the contributions and scope in the abstract and introduction. Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discussed the limitations of our work in Appendix A. Guidelines: • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We presented the utilized assumptions in Section 3 and in Appendix D.2, and provided complete proof for the theory in Appendix D. Guidelines: • The answer [N/A] means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We explained our problem settings in Section 5 and implementation details in Appendix E.2. Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: For anonymity reasons, we have not made our code public. Upon acceptance, we will release our code on GitHub. Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: All the training and model details are specified in Appendix E.2 and the experiment setup details are presented in Appendix E. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We have provided the necessary information in the table caption to clarify how the results and standard deviations were calculated. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: In Table 7, we report the computational training time and the hardware resources we used to obtain the experimental results. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification: We read and followed the NeurIPS Code of Ethics. Guidelines: • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No] , they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss societal impacts of the work in Appendix F. Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. • If the authors answer [N/A] or [No] , they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: N/A Guidelines: • The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [N/A] Justification: [N/A] Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., C-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [N/A] Justification: N/A Guidelines: • The answer [N/A] means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: N/A Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: N/A Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [N/A] Justification: N/A Guidelines: • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.