Paper deep dive
Pessimistic Auxiliary Policy for Offline Reinforcement Learning
Fan Zhang, Baoru Huang, Xin Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 7:09:29 AM
Summary
This paper proposes a Pessimistic Auxiliary Policy for Offline Reinforcement Learning to mitigate error accumulation and overestimation caused by out-of-distribution actions. By maximizing the lower confidence bound of the Q-function using epistemic uncertainty estimates, the method samples reliable actions with low uncertainty, thereby improving the efficacy of existing offline RL algorithms like TD3BC and Diffusion-QL on D4RL benchmarks.
Entities (10)
Relation Signals (8)
Pessimistic Auxiliary Policy → mitigates → Overestimation
confidence 95% · avoiding the learned policy sampling high-value actions with potentially high errors ... alleviation of error accumulation.
Pessimistic Auxiliary Policy → mitigates → Error Accumulation
confidence 95% · Less approximation error introduced by sampled action from pessimistic auxiliary strategy leads to the alleviation of error accumulation.
Offline Reinforcement Learning → suffersfrom → Error Accumulation
confidence 95% · Offline RL faces serious overestimation issues ... error accumulation arises when inferring the value of unseen actions.
Pessimistic Auxiliary Policy → uses → Lower Confidence Bound
confidence 92% · we develop a pessimistic auxiliary strategy by maximizing the lower confidence bound of the Q-function.
Lower Confidence Bound → derivedfrom → Epistemic Uncertainty
confidence 90% · We utilize epistemic uncertainty estimate on the Q-function to construct a lower confidence bound
Pessimistic Auxiliary Policy → evaluatedon → D4RL
confidence 90% · Extensive experiments conducted on D4RL benchmark demonstrate pessimistic auxiliary policy can effectively improve
Pessimistic Auxiliary Policy → improves → Diffusion-QL
confidence 88% · apply pessimistic auxiliary policy to the ... Diffusion-QL
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning process introduces approximation errors, causing the error accumulation and considerable overestimation. In this paper, we construct a new pessimistic auxiliary policy for sampling reliable actions. Specifically, we develop a pessimistic auxiliary strategy by maximizing the lower confidence bound of the Q-function. The pessimistic auxiliary strategy exhibits a relatively high value and low uncertainty in the vicinity of the learned policy, avoiding the learned policy sampling high-value actions with potentially high errors during the learning process. Less approximation error introduced by sampled action from pessimistic auxiliary strategy leads to the alleviation of error accumulation. Extensive experiments on offline reinforcement learning benchmarks reveal that utilizing the pessimistic auxiliary strategy can effectively improve the efficacy of other offline RL approaches.
Tags
Links
- Source: https://arxiv.org/abs/2602.23974v2
- Canonical: https://arxiv.org/abs/2602.23974v2
PDF not stored locally. Use the link above to view on the source site.
Full Text
43,462 characters extracted from source content.
Expand or collapse full text
Pessimistic Auxiliary Policy for Offline Reinforcement Learning Fan Zhang Baoru Huang, Xin Zhang Corresponding author: Xin Zhang (xinzhang_cn@163.com)Fan Zhang are with Department of Mathematics, The Hong Kong University of Science and Technology, Hong Kong, China. Baoru Huang is with University of Liverpool, UKXin Zhang is with Department of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology, Hong Kong, China. Abstract Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-of-distribution actions during the learning process introduces approximation errors, causing the error accumulation and considerable overestimation. In this paper, we construct a new pessimistic auxiliary policy for sampling reliable actions. Specifically, we develop a pessimistic auxiliary strategy by maximizing the lower confidence bound of the Q-function. The pessimistic auxiliary strategy exhibits a relatively high value and low uncertainty in the vicinity of the learned policy, avoiding the learned policy sampling high-value actions with potentially high errors during the learning process. Less approximation error introduced by sampled action from pessimistic auxiliary strategy leads to the alleviation of error accumulation. Extensive experiments on offline reinforcement learning benchmarks reveal that utilizing the pessimistic auxiliary strategy can effectively improve the efficacy of other offline RL approaches. Index Terms: Offline Reinforcement Learning, Error Exacerbation, Overestimation, Pessimistic I Introduction Reinforcement Learning (DRL) utilizing high-capacity neural network approximators [17, 10, 26] has demonstrated exceptional performance across various decision-making domains, such as robotics [38, 32, 35], strategy games [2, 1, 44, 19], and more. However, unsafe and inefficient interactions in the learning process impede the deployment of DRL. Currently, there exist two methodologies for resolving this issue: 1) constructing a world model to simulate the real world [20, 27], requiring complex knowledge of physics and so on, difficult to achieve; 2) learning the agent merely on pre-collected dataset, that is Offline RL [8, 5]. However, offline RL faces serious overestimation issues compared to online RL. Pre-collected dataset is impossible to cover all state-actions [16]. Consequently, in temporal difference (TD) update [r+γQ(s′,a′)−Q(s,a)]2[r+γ Q(s ,a )-Q(s,a)]^2 , out-of-distribution (OOD) actions a′a might be encountered, introducing considerable approximation errors [16]. These inflated approximations of OOD actions a′a lead to overestimation of other state-actions (s,a)(s,a), eventually engendering policy degeneration [10, 7, 17, 42]. In order to inhibit error accumulation of OOD actions, many previous offline RL algorithms [18, 6, 33, 24] prohibit the agent from extrapolating on OOD actions. Previous algorithms could be roughly divided into two categories: 1) value regularization approaches [33, 16, 11], which directly regularize the value of actions sampled by the learned policy; 2) policy constraint approaches regularize the learned policy to be close to the behavior policy, utilizing explicit divergence penalties [8, 15], or implicit divergence penalties [25, 31]. In contrast to previous methods impose constrain on learned policy, we propose to construct a new pessimistic auxiliary strategy to sample reliable actions. Specifically, new policy impedes exploration by applying the principle of pessimism in the face of uncertainty [3]. We utilize epistemic uncertainty estimate on the Q-function [23] to construct a lower confidence bound to the critic. By first-order Taylor expansion of the lower confidence bound, an appropriate pessimistic auxiliary policy within the neighborhood of current policy can be found. Intuitively, we achieve actions with small approximation error through end-to-end learning, without requiring additional data sparsity-based computations [42]. It is worth noting that the proposed pessimistic auxiliary policy can be adapted to the majority of offline RL methods. We theoretically provide the convergence guarantee of TD update with pessimistic auxiliary policy. To empirically verify the proposed method, we apply pessimistic auxiliary policy to the TD3BC [6] and Diffusion-QL [29]. Extensive experiments conducted on D4RL benchmark demonstrate pessimistic auxiliary policy can effectively improve the efficacy of TD3BC and Diffusion-QL. I Related Work Offline reinforcement learning (RL) aims to learn effective policies from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, data-driven offline RL also faces challenges: error accumulation [8] arises when inferring the value of unseen actions. To overcome the overestimation problem, previous offline RL algorithms intend to prevent agents from querying unseen samples. Previous algorithms related to our method could be divided into two categories: 1) Policy constraint approaches regularize the learned policy to be close to the behavior policy, utilizing explicit divergence penalties [8, 15, 33], or implicit divergence penalties [25, 31, 22]. Policy constraint approaches impose strict constrains and fail to learn optimal policies [5]. Moreover, in real-world applications, behavior policy might be difficult to be obtained. Recently, some related works have tried to relax the full-coverage assumption [37, 36], which is merely in theoretical analysis; 2) Value regularization approaches directly regularize the value function. BRAC-v [33] introduces the divergence penalty to temporal difference (TD) update. CQL [16] directly minimizes the Q-values of OOD actions and maximizes the Q-values of in-distribution actions. Besides, there are uncertainty-based methods [34, 11] regularize the value function uncertainty estimated from value function or world model. There are many other attempts in offline RL domains, such as Model-based offline RL [13, 40], Transformer-based methods [28]. Despite effectiveness, these methods deviate from the paradigm of reinforcement learning. In contrast to prior works to constrain the learned policy, we propose a new pessimistic auxiliary strategy to sample more reliable actions. I Preliminaries We consider a Markov Decision Process formulated environment which is defined by a tuple (S,A,P,R,γ)(S,A,P,R,γ) w.r.t. state space, action space, transition distribution, reward function and discount factor respectively. The agent takes action a∈Aa∈ A in state s∈Ss∈ S, and acquires a new state s′∈Ss ∈ S and immediate reward r(s,a)r(s,a), in the light of the transition distribution P(s′|s,a)P(s |s,a). The goal of reinforcement learning is to find an optimal policy π(a|s)π(a|s) that maximizes the cumulative discounted reward π,P∑t=0∞γtr(st,at) E_π,P _t=0^∞γ^tr(s_t,a_t). Offline RL. Other from online learning paradigms learned with the experience from the environment, offline RL relies solely on a prior collected dataset =(s,a,r,s′)iD=\(s,a,r,s )_i\. The end goal is still to obtain a policy from a value function, which consists of the following two steps. 1) Policy evaluation. The value function is learned by minimizing the bellman residual: Q(θ)=(s,a,r,s′)∼[Qθ(s,a)−Qθ(s,a)]2,J_Q(θ)= E_(s,a,r,s ) [TQ_θ(s,a)-Q_θ(s,a)]^2, (1) where θ is the parameters of Q-net, and T is the Bellman optimality operator: Qθ(s,a)=r(s,a)+γa′∼π(⋅|s′)Qθ−(s′,a′),TQ_θ(s,a)=r(s,a)+γ E_a π(·|s )Q_θ^-(s ,a ), (2) θ is the parameters of Q-net, and θ−θ^- is the parameters of a separate target Q-net without gradient propagation [21]. 2) Policy extraction. The optimal policy under the value function is extracted as follows: π(ϕ)=−s∼,a∼π(a|s)Qθ(s,a),J_π(φ)=- E_s ,a π(a|s)Q_θ(s,a), (3) where ϕφ is the parameters of policy. Uncertainty Quantification. Epistemic uncertainty is adopted to assess the reliability of the predictive model, which is widely applied in many domains [41]. Intuitively, ensemble regression functions yield similar estimations on the area with rich data. We introduce the uncertainty quantification δ with respect to the regression functions f on data x δ(x)=1K∑i=1K(fi−f¯)2(s,a),δ(x)= 1K _i=1^K(f^i- f)^2(s,a), (4) where f¯ f is the mean of fif^i. IV Analysis of Overestimation Overestimation in the field of offline reinforcement learning stems from immense approximation error which is caused by the distribution shift[18]. The distribution shift refers to the difference between the behavior policy and the current learned policy. The learned policy inevitably queries a strange action a′a on state s′s . Value function infers the estimation on (s′,a′)(s ,a ) with the generation capability of the neural network. Approximation error upwardly updates Q-values by standard temporal difference (TD) update[7, 18], inducing the consistent overestimation: δ(s,a)=Q∗(s,a)−Qπ(s,a) δ(s,a)=Q (s,a)-Q^π_D(s,a) (5) =s′[r(s,a)+γa′Q∗(s′,a′)]−Qπ(s,a) =E_s [r(s,a)+ _a Q (s ,a )]-Q^π_D(s,a) =∑s′pℳ[r(s,a)+γ∑a′π∗(a′|s′)Q∗(s′,a′)]−Qπ(s,a) = _s p_M[r(s,a)+γ _a π (a |s )Q (s ,a )]-Q^π_D(s,a) =∑s′pℳ[r(s,a)+γ∑a′π∗(a′|s′)Qπ(s′,a′)+δ(s′,a′)] = _s p_M[r(s,a)+γ _a π (a |s )Q^π_D(s ,a )+δ(s ,a )] −∑s′p[r(s,a)+γ∑a′π(a′|s′)Qπ(s′,a′)] - _s p_D[r(s,a)+γ _a π(a |s )Q^π_D(s ,a )] =∑s′(pℳ−p)r(s,a)+γ∑s′∑a′[pℳπ∗(a′|s′) = _s (p_M-p_D)r(s,a)+γ _s _a [p_Mπ (a |s ) −pπ(a′|s′)]Qπ(s′,a′)+γ∑s′∑a′pℳπ∗(a′|s′)δ(s′,a′). -p_Dπ(a |s )]Q^π_D(s ,a )+γ _s _a p_Mπ (a |s )δ(s ,a ). Unfortunately, the paradigm of offline reinforcement learning makes it difficult to solve the problem of overestimation. Estimation error will accumulate through the TD update. Due to the agent cannot interact with the environment, there are no new state-action pair (s′,a′)(s ,a ) and corresponding reward r to correct the estimation error. Furthermore, we conduct simulation study to delve into the causes of estimation error. Apart from the bootstrapping associated with temporal difference (TD) updates, the neural network of the agent engages in linear regression. Hence, we depict the linear regression based on different number of observations in Fig. 1. We can observe that as the number of observations increases, the estimates produced by linear regression (red line) converge more closely to the true underlying distribution (blue line). That’s why few-shot learning or data distribution is widely studied in many domains [43, 39].Meanwhile, the confidence level of the predictions also increases, corresponding to a reduction in the red shaded area. Return to the offline reinforcement learning scenario, the agent is prone to yield a larger epistemic uncertainty (lower confidence level) of estimation on the strange action a′a . As consequence, these actions are more likely to introduce larger estimation errors, exacerbating the error accumulation and overestimation of offline RL methods. The proof and experiments inspire us that we can sample actions with lower uncertainty for TD update. More accurate value estimation will contribute to the learning of more effective policies. Figure 1: Simulation study shows the causes of estimation error of linear regression. V Pessimistic Auxiliary Policy As shown in Eq. 5, encountering out-of-distribution actions a′a would introduce high approximation error. In this section, we construct a pessimistic auxiliary policy to sample actions with lower uncertainty, mitigating the error accumulation through TD update. Inspired by [12], when the agent reaches a new state, we maximize the lower confidence bound of the Q function which is constructed with epistemic uncertainty. Pessimistic auxiliary policy πp _p are used for computing Q(s′,πp(⋅|s′))Q(s , _p(·|s )) in TD updates and for policy learning s∼,a∼πp(⋅|s)Qθ(s,a) E_s ,a _p(·|s)Q_θ(s,a). The pessimistic auxiliary policy πp=δ(μp) _p=δ( _p) can be formalized as follows: μp=μ+2σ‖[▽aQLB(s,a)]a=μ‖[▽aQLB(s,a)]a=μ, _p=μ+ 2σ||[ _aQ_LB(s,a)]_a=μ||[ _aQ_LB(s,a)]_a=μ, (6) where π=δ(μ)π=δ(μ) is the learned policy under Dirac distribution δ, σ is the largest distance between pessimistic auxiliary policy πp _p and the learned π. Concretely, starting from epistemic uncertainty, we obtain a low confidence bound of Q function. By constraining the difference between the pessimistic auxiliary policy and the current learned policy, a pessimistic auxiliary policy is obtained to suppress error accumulation. Epistemic uncertainty. For efficient computation, we model the Epistemic uncertainty using Gaussian distribution, where the Q function is used to fit the mean and standard deviation. Due to common RL methods consist of two Q functions, the mean confidence can be defined as μQ(s,μ)=12(Q1(s,μ)+Q2(s,μ)) _Q(s,μ)= 12(Q^1(s,μ)+Q^2(s,μ)), the standard deviation can be obtained by δQ(s,μ) _Q(s,μ) =∑i∈1,212(Qi(s,μ)−μQ(s,μ))2 = _i∈1,2 12(Q^i(s,μ)- _Q(s,μ))^2 (7) =(12Q1(s,μ)−12Q2(s,μ))2 = ( 12Q^1(s,μ)- 12Q^2(s,μ))^2 =12(Q1(s,μ)−Q2(s,μ))2 = 12 (Q^1(s,μ)-Q^2(s,μ))^2 =12|Q1(s,μ)−Q2(s,μ)|. = 12|Q^1(s,μ)-Q^2(s,μ)|. Note that, since existing offline reinforcement learning algorithms retain two Q functions, the computation of μQ _Q and δQ _Q does not introduce excessive computational cost, nor does it introduce unnecessary network parameters. Lower confidence bound. Through uncertainty estimation, a lower bound QLB(s,μ)=μQ(s,μ)−βδQ(s,μ)Q_LB(s,μ)= _Q(s,μ)-β _Q(s,μ) can be obtained. The parameter β is used to control the level of pessimism. Since the pessimistic policy replaces the agent’s learning policy in action sampling, excessive distribution deviation can result in instability in agent learning. Thus, the pessimistic policy will be in proximity to the learning policy. Naturally, we employ a first-order Taylor expansion of QLBQ_LB, and the lower confidence bound of the Q function within the neighborhood of the current policy is obtained: Q¯LB(s,a;μ) Q_LB(s,a;μ) =(a−μ)T[▽aQLB(s,a)]a=μ+Q(s,μ) =(a-μ)^T[ _aQ_LB(s,a)]_a=μ+Q(s,μ) (8) =aT[▽aQLB(s,a)]a=μ+const =a^T[ _aQ_LB(s,a)]_a=μ+const According to Taylor’s theorem, within a small enough neighborhood of μ, QLB(s,a;μ)Q_LB(s,a;μ) can provide an accurate linear approximation of Q(s,a)Q(s,a). Therefore, the choice of neighborhood size is important. Pessimistic auxiliary policy. The pessimistic auxiliary policy balances two criteria: 1) maximizing QLB(s,a)Q_LB(s,a) to ensure the selection of relatively optimal actions in pessimistic scenarios; 2) constraining the distance between the agent’s learned policy and the pessimistic auxiliary policy. This constraint limits the difference between the pessimistic auxiliary policy and the agent’s learned policy, preventing catastrophic actions during action sampling and promoting optimization stability. Additionally, when the gap between the pessimistic policy and the agent’s learning policy is sufficiently small, the approximate QLB(s,a)Q_LB(s,a) is accurate. In conclusion, the pessimistic auxiliary policy addresses the following optimization problem, μp=argmaxW(δ(μp),δ(μ))≤σQ¯LB(s,μp), _p= argmax_W(δ( _p),δ(μ))≤σ Q_LB(s, _p), (9) here, we utilize wasserstein distance to calculate the distance of two deterministic policy πp _p and π. Proposition 1. Pessimistic auxiliary policy can be defined as πp=δ(μp) _p=δ( _p), where μp=μ+2σ‖[▽aQLB(s,a)]a=μ‖[▽aQLB(s,a)]a=μ. _p=μ+ 2σ||[ _aQ_LB(s,a)]_a=μ||[ _aQ_LB(s,a)]_a=μ. (10) Proof. Consider the wasserstein distance two Dirac distribution: W(δ(μp),δ(μ))=12‖μp−μ‖2.W(δ( _p),δ(μ))= 12|| _p-μ||^2. (11) Plug Eq. 12 into Eq. 10, we can obtain: μp=argmaxμpQ¯LB(s,μp) _p= argmax_ _p Q_LB(s, _p) (12) subjectto12(μp−μ)T(μp−μ)≤σ. subject~to~ 12( _p-μ)^T( _p-μ)≤σ. To solve the Eq. 12, we construct the Lagrangian function: L=Q¯LB(s,μp)−λ(12(μp−μ)T(μp−μ)−σ).L= Q_LB(s, _p)-λ( 12( _p-μ)^T( _p-μ)-σ). (13) Differentiate both sides with respect to μp _p: ▽μpL=[▽aQLB(s,a)]a=μ−λ(μp−μ). _ _pL=[ _aQ_LB(s,a)]_a=μ-λ( _p-μ). (14) Let ▽μpL _ _pL equal to zero, we can get μp=1λ[▽aQLB(s,a)]a=μ+μ. _p= 1λ[ _aQ_LB(s,a)]_a=μ+μ. (15) By KKT condition, we can know λ>0λ>0 and (μp−μ)T(μp−μ)=2δ.( _p-μ)^T( _p-μ)=2δ. (16) Plug Eq. into Eq. 15, we can obtain λ λ =[▽aQLB(s,a)]a=μT[▽aQLB(s,a)]a=μ2σ = [ _aQ_LB(s,a)]_a=μ^T[ _aQ_LB(s,a)]_a=μ2σ (17) =||▽aQLB(s,a)]a=μ||2σ, = || _aQ_LB(s,a)]_a=μ|| 2σ, Plug λ into Eq. 15, we can derive the final conclusion: μp=μ+2σ‖[▽aQLB(s,a)]a=μ‖[▽aQLB(s,a)]a=μ. _p=μ+ 2σ||[ _aQ_LB(s,a)]_a=μ||[ _aQ_LB(s,a)]_a=μ. (18) ∎ Figure 2: Depiction of lower confidence bound of Q function and pessimistic auxiliary policy. Summary. We construct a pessimistic auxiliary policy by maximizing the lower confidence bound of Q function. Intuitively speaking, we find a new direction to construct the policy for sampling actions which is shown in Fig. 2. Due to the lower bound of Q function inversely proportional to the standard deviation δQ _Q, the second term on the right-hand side is actually the new sample direction which points to regions of low uncertainty. As a consequence, pessimistic auxiliary policy can sample reliable actions with low uncertainty and low approximation error, compared to the π. Hence, pessimistic auxiliary policy can impede introduction of approximation error, mitigating the error accumulation. VI Offline RL with Pessimistic Auxiliary Policy Offline reinforcement learning algorithms with pessimistic auxiliary policy are divided into three stages: computing the pessimistic auxiliary policy, evaluating the value of state-action pairs with Q networks (policy evaluation), and learning the agent’s policy (policy extraction). 1) pessimistic auxiliary policy: according to the current learned policy μ, we can get μp=μ+2σ‖[▽aQLB(s,a)]a=μ‖[▽aQLB(s,a)]a=μ. _p=μ+ 2σ||[ _aQ_LB(s,a)]_a=μ||[ _aQ_LB(s,a)]_a=μ. (19) 2) policy evaluation: we utilize the pessimistic auxiliary policy to sample actions on state s′s , Q(θ)=(s′|s,a)∼[pQ−Qθ(s,a)]2, _Q(θ)= E_(s |s,a) [T_pQ-Q_θ(s,a)]^2, (20) where pQ=r(s,a)+γQθ−(s′,πp(s′))T_pQ=r(s,a)+γ Q_θ^-(s , _p(s )) is the new bellman operator with pessimistic auxiliary policy. 3) policy extraction: we leverage Q function to guide the policy extraction. In addition, like most offline reinforcement learning algorithms, the bias between the learning policy and the data collection policy is incorporated into the update process as a constraint, π(ϕ) _π(φ) =(s,a)∼−Qθ(s,π(s))−λlogπβ(π(s)|s), = E_(s,a) -Q_θ(s,π(s))-λ log _β(π(s)|s), (21) To summarize, the procedure of offline RL with pessimistic auxiliary policy is outlined in Algorithm 1. Due to this algorithm merely replace the oral policy with pessimistic auxiliary policy to sample actions, we can adapt this learning procedure to the majority offline RL methods. For reproducibility, we disclose the hyper-parameters used in the algorithm, i.e. β and δ. β controls the level of pessimism, we set β=1β=1 for all test environments. δ controls the distance between pessimistic auxiliary policy and learned policy. As the training steps increase, the learning strategy becomes more effective. The training process does not require more pessimism. Hence, we adopt cosine annealing to scale the δ appropriately, ensuring a smooth transition from the initial value (δinit=1 _init=1) to zero. Algorithm 1 Offline RL with Pessimistic Auxiliary Policy 1:θ, ψ, ϕφ⊳ Initial parameters 2:D,λ, τ ⊳ Initialize Dataset, learning rate and target smoothing coefficient 3:θ−←θ^-←θ⊳ Initialize target Q-net weights 4:for each training step do 5: μp _p⊳ obtain pessimistic auxiliary policy via Eq.19 6: θ←θ−λθ∇θQ(θ)θ←θ- _θ _θJ_Q(θ)⊳ train Q-net via Eq. 20 7: ϕ←ϕ−λϕ∇ϕπ(ϕ)φ←φ- _φ _φJ_π(φ)⊳ train policy net via Eq. 21 8: θ−←τθ+(1−τ)θ−θ^-←τθ+(1-τ)θ^-⊳ Soft update target V-net 9:end for 10:θ, ψ, ϕφ VII Convergence Analysis Due to the new bellman operator in Eq. 20, we further provide the convergence analysis in this section. Proposition 2. Boundedness. Suppose ∀s,a∀ s,a, R(s,a)<RmaxR(s,a)<R_max, and initial Q0∈[−Rmax,Rmax]Q_0∈[-R_max,R_max], then for new bellman operator pT_p, after k times update, Qk(s,a)∈[−Rmax1−γ,Rmax1−γ]Q^k(s,a)∈[- R_max1-γ, R_max1-γ]. Proof. we use mathematical induction to prove Proposition 2,Qk(s,a)≤∑i=0kγkRmax≤Rmax1−γQ^k(s,a)≤ _i=0^kγ^kR_max≤ R_max1-γ. 1) when k=1k=1, Q1(s,a) Q^1(s,a) =pQ0(s,a)≤Q0(s,a) =T_pQ^0(s,a) ^0(s,a) (22) ≤Rmax+γRmax≤∑i=01γ1Rmax; ≤ R_max+γ R_max≤ _i=0^1γ^1R_max; 2) suppose k=jk=j,Qj(s,a)≤∑i=0jγjRmaxQ^j(s,a)≤ _i=0^jγ^jR_max holds; 3) when k=j+1k=j+1, Qj+1(s,a)=pQj(s,a)≤Qj(s,a)≤∑i=0j+1γiRmax; Q^j+1(s,a)=T_pQ^j(s,a) ^j(s,a)≤ _i=0^j+1γ^iR_max; (23) Hence, Qk(s,a)≤∑i=0kγiRmax<Rmax1−γQ^k(s,a)≤ _i=0^kγ^iR_max< R_max1-γ holds. Meanwhile,Qk(s,a)≥−∑i=0kγiRmax>−Rmax1−γQ^k(s,a)≥- _i=0^kγ^iR_max>- R_max1-γ can be proved. ∎ Proposition 3. γ concentration. For two Q function Q1Q^1 and Q2Q^2, the new bellman operator pT_p has the following property, ‖pQ1−pQ2‖≤γ‖Q1−Q2‖,||T_pQ^1-T_pQ^2||≤γ||Q^1-Q^2||, (24) Proof. Firstly, ‖pQ1−pQ2‖ ||T_pQ^1-T_pQ^2|| (25) =‖R+γQ1(s′,μp(s′))−R−γQ2(s′,μp(s′))‖ =||R+γ Q^1(s , _p(s ))-R-γ Q^2(s , _p(s ))|| =γ‖Q1(s′,μp(s′))−Q2(s′,μp(s′))‖. =γ||Q^1(s , _p(s ))-Q^2(s , _p(s ))||. Due to the uncertainty of μp _p is lower than μ, hence, ‖Q1(s′,μp(s′))−Q2(s′,μp(s′))‖≤‖Q1(s′,μ(s′))−Q2(s′,μ(s′))‖||Q^1(s , _p(s ))-Q^2(s , _p(s ))||≤||Q^1(s ,μ(s ))-Q^2(s ,μ(s ))||. Plug the above equation into Eq. 25, we can obtain: ‖pQ1−pQ2‖=γ‖Q1(s′,μp(s′))−Q2(s′,μp(s′))‖ ||T_pQ^1-T_pQ^2||=γ||Q^1(s , _p(s ))-Q^2(s , _p(s ))|| (26) ≤γ‖Q1(s′,μ(s′))−Q2(s′,μ(s′))‖=γ‖Q1−Q2‖. ≤γ||Q^1(s ,μ(s ))-Q^2(s ,μ(s ))||=γ||Q^1-Q^2||. ∎ Due to the bellman operator is a contractor under Banach space, Boundedness (Proposition 2) and γ concentration guarantee the convergence of pT_p. To sum up, introducing pessimistic auxiliary policy to offline RL does not change the convergence. VIII Experiments TABLE I: Comparison of TD3PA, DQLPA and baseline methods in terms of normalized average score on D4RL benchmarks over the final 5 evaluations. The experiments are run on MuJoCo-v2, Adroit-v0 and AntMaze-v0 dataset over 10 random seeds. Gym Tasks CQL TD3BC IQL DT DQL TD3PA DQLPA halfcheetah-m 47.0 48.3 47.4 42.6 51.1 46.4 54.3 halcheetah-m-e 90.7 90.7 86.7 86.8 96.8 92.1 99.0 halfcheetah-m-r 44.6 44.6 44.2 36.6 47.8 44.7 52.3 hopper-m 53.0 59.3 66.2 67.6 90.5 65.8 96.3 hopper-m-e 98.0 98.0 91.5 107.6 111.1 104.3 110.4 hopper-m-r 60.9 60.9 94.7 82.7 101.3 74.2 102.3 walker2d-m 73.3 83.7 78.3 74.0 87.0 82.7 91.1 walker2d-m-e 110.1 109.6 110.2 108.1 110.1 110.3 109.1 walker2d-m-r 81.8 81.9 73.8 66.6 95.5 82.8 96.4 Gym total 659.4 677.5 692.4 672.6 791.2 703.3 811.2 Adroit Tasks CQL TD3BC IQL DT DQL TD3PA DQLPA pen-human 37.5 0.0 71.5 57.8 72.8 62.0 74.2 pen-cloned 39.2 9.9 37.3 23.2 57.3 62.6 65.1 Adroit total 76.7 9.9 108.8 81.0 130.1 124.6 139.3 AntMaze Tasks CQL TD3BC IQL DT DQL TD3PA DQLPA antmaze-u 74.0 78.6 87.5 59.2 93.4 95.0 97.1 antmaze-u-d 39.2 71.4 62.2 53.0 66.2 78.4 80.2 antmaze-m-p 61.2 10.6 71.2 0.0 76.6 87.4 89.1 antmaze-m-d 53.7 3.0 70.0 0.0 78.6 94.0 88.1 antmaze-l-p 15.8 0.2 39.6 0.0 46.4 65.2 63.6 antmaze-l-d 14.9 0.0 47.5 0.0 56.6 23.5 60.1 Antmaze total 258.8 163.8 378.0 112.2 417.7 425.6 478.2 All total 994.9 851.2 1179.2 865.8 1339.0 1253.5 1428.7 VIII-A Experiment Settings In this section, we conduct experiments to answer the following research questions: RQ1 Does our proposed pessimistic auxiliary policy improve the offline RL methods? RQ2 Why does pessimistic auxiliary policy improve the offline RL methods? TABLE I: Comparison of IEEDS and baseline methods on NeoRL-2 benchmarks over three random seeds. Task Name CQL TD3-BC TD3PA Pipeline 81.08 81.95 82.31 Simglucose 10.99 74.21 74.42 RocketRecovery 74.32 79.74 76.21 RandomFrictionHopper 32.96 29.51 34.23 DMSD 70.24 60.01 71.21 Fusion 55.86 54.64 55.63 SafetyHalfCheetah 71.18 68.58 71.63 Total 396.63 448.64 465.64 Figure 3: The distances between the action executed by policy and the action in the HalfCheetah dataset. Figure 4: The distances between the action executed by policy and the action in the AntMaze dataset. Datasets. We appraise our method against the previous offline RL algorithms on the D4RL benchmarks, including following three domains: 1) Gym domain: includes three environments (HalfCheetah, Hopper, and Walker2d) with three dataset types: 1) medium amasses 1M samples from a policy trained to approximately 1/3 the performance of the expert; 2) medium-expert uses 50-50 split of medium and expert data; 3) m-r utilizes the replay buffer of a policy trained up to the performance of the medium agent; 2) Adroit domain: involves controlling a 24-DoF robotic hand, which are more challenging than the Gym domain in task complexity. The Adroit domain we used includes one environment (Pen) with two dataset types: 1) human uses the 25 human demonstrations ; 2) cloned mixes demonstration data and trajectories sampled from a behavioral cloned policy; 3) AntMaze domain: involves controlling an 8-DoF Ant quadruped robot to navigate to a particular goal state. Three maze layouts are used: umaze, medium and large. The ant is required to reach a specific goal from a fixed start location; reach a random goal from a random start location (diverse dataset); reach a specific hand-picked locations from a different set of hand-picked start locations(play dataset). Additionally, we evaluate our method on NeoRL-2 [9], which focuses on the nature of datasets from real-world scenarios. It extends NeoRL by providing many challenges in the real world such as time delay, external factors and so on. Baselines. Three types of model-free offline RL methods are selected : a) value regularization method, including CQL [16], IQL [14]; b) policy constraint method, e.g. AWAC [22], TD3BC [6], DQL [30]; c) other methods, such as trajectory optimization (DT [4]). TABLE I: Approximation error of different algorithms. Task Name TD3BC CQL TD3PA halfcheetah-m 132.85 36.24 17.45 halfcheetah-m-e 341.42 32.76 16.43 halfcheetah-m-r 149.80 31.17 15.76 antmaze-umaze 86.42 54.59 54.23 antmaze-medium-play 110.08 86.08 79.42 antmaze-large-play 110.89 65.79 64.32 VIII-B Overall Performance (RQ1). Implement Details. We apply pessimistic auxiliary policy to TD3BC and DQL, called TD3PA and DQLPA. 1) Our implement of TD3PA is based on the official TD3 implementations 111TD3BC: https://github.com/sfujim/TD3_BC. 2) Our implement of DQLPA is based on the official DQL implementations 222DQL: https://github.com/Zhendong-Wang/Diffusion-Policies-for-Offline-RL. From Table I, we can observe that: 1) TD3PA has improved by 3.8%, 14.5% and 159.5% compared to TD3BC on Gym, Adroit and AntMaze tasks respectively; 2) DQLPA has improved by 2.5%, 7.1% and 14.5% compared to DQL on Gym, Adroit and AntMaze tasks respectively; 3) Pessimistic auxiliary policy improves the efficacy of offline RL methods on almost all tasks; 4) DQLPA achieves substantial gains over baselines. We also evaluate our method on more complex, real-world scenarios. From Table I, we can observe that TD3PA achieves 3.79% improvement in terms of normalized score on NeoRL-2. Besides. TD3PA outperforms the baseline in overall performance across all 7 environments, demonstrating the adaptability and generalizability for real world applications. VIII-C Performance Analysis (RQ2). Pessimistic auxiliary policy impedes the emergence of OOD actions in the learning process. Therefore, the use of the pessimistic auxiliary policy brings about the following two results: 1) Less approximation error can be brought in the TD update. Hence, the Q function is more accurate. 2) The reduction of overestimation leads to the avoidance of strange action. Hence, the agent will select more similar actions to the pre-collected dataset. Consequently, we investigate why pessimistic auxiliary policy achieves improvement from the following two perspectives. Due to TD3PA achieves larger improvement than DQLPA, we select TD3BC and TD3PA as examples to conduct experiments. 1) Value Estimation. We assess the difference between the Q estimation and actual discounted return of TD3BC, CQL and TD3PA, which is widely adopted to investigate the over-estimation [7]. We rollout episodes from 1000 random selected states. We calculate the average discounted returns of 1000 episodes as actual discounted returns. From Table I, we can notice that: TD3PA reduces overestimation across different tasks. Especially compared to TD3BC, the Q estimation error of TD3PA is decreased by 86.8%, 95.2%,89.5% on HalfCheetah tasks and 37.2%, 27.9%, 42.0% on AntMaze tasks. 2) Policy Analysis. We conduct the experiment to compute the distance of sampled actions and the dataset actions. Specifically, we calculate (s,a)∼,a~∼π(⋅|s)‖a~−a‖22 E_(s,a) , a π(·|s)|| a-a||_2^2 for TD3BC, CQL and TD3PA. Besides, we devise a random policy for clearer comparison. Overall, TD3PA chooses more closely to the action in the dataset for complex tasks. IX Conclusion In this paper, we construct a new pessimistic auxiliary policy for sampling reliable actions, this policy can be applied in other offline RL methods. Pessimistic auxiliary strategy allows for the selection of actions with decreased uncertainty and lower approximation error. Less approximation error introduced by sampled action leads to the alleviation of error accumulation, improving the efficacy of offline RL. Extensive experiments on offline reinforcement learning benchmarks reveal that utilizing the pessimistic auxiliary strategy can effectively improves the efficacy of other offline RL approaches. References [1] R. Akrour, D. Tateo, and J. Peters (2022) Continuous action reinforcement learning from a mixture of interpretable experts. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (10), p. 6795–6806. External Links: Document Cited by: §I. [2] C. Bai, L. Wang, Y. Wang, Z. Wang, R. Zhao, C. Bai, and P. Liu (2023) Addressing hindsight bias in multigoal reinforcement learning. IEEE Transactions on Cybernetics 53 (1), p. 392–405. External Links: Document Cited by: §I. [3] R. I. Brafman and M. Tennenholtz (2002) R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research 3 (Oct), p. 213–231. Cited by: §I. [4] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch (2021) Decision transformer: reinforcement learning via sequence modeling. Advances in neural information processing systems 34, p. 15084–15097. Cited by: §VIII-A. [5] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine (2020) D4rl: datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. Cited by: §I, §I. [6] S. Fujimoto and S. S. Gu (2021) A minimalist approach to offline reinforcement learning. Advances in neural information processing systems 34, p. 20132–20145. Cited by: §I, §I, §VIII-A. [7] S. Fujimoto, H. Hoof, and D. Meger (2018) Addressing function approximation error in actor-critic methods. In International conference on machine learning, p. 1587–1596. Cited by: §I, §IV, §VIII-C. [8] S. Fujimoto, D. Meger, and D. Precup (2019) Off-policy deep reinforcement learning without exploration. In International conference on machine learning, p. 2052–2062. Cited by: §I, §I, §I, §I. [9] S. Gao, Z. Tu, R. Qin, Y. Sun, X. Chen, and Y. Yu (2025) Neorl-2: near real-world benchmarks for offline reinforcement learning with extended realistic scenarios. arXiv preprint arXiv:2503.19267. Cited by: §VIII-A. [10] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, p. 1861–1870. Cited by: §I, §I. [11] M. Janner, J. Fu, M. Zhang, and S. Levine (2019) When to trust your model: model-based policy optimization. Advances in Neural Information Processing Systems 32. Cited by: §I, §I. [12] C. Jin, Z. Allen-Zhu, S. Bubeck, and M. I. Jordan (2018) Is q-learning provably efficient?. Advances in neural information processing systems 31. Cited by: §V. [13] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims (2020) Morel: model-based offline reinforcement learning. Advances in neural information processing systems 33, p. 21810–21823. Cited by: §I. [14] I. Kostrikov, A. Nair, and S. Levine (2021) Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, Cited by: §VIII-A. [15] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine (2019) Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems 32. Cited by: §I, §I. [16] A. Kumar, A. Zhou, G. Tucker, and S. Levine (2020) Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems 33, p. 1179–1191. Cited by: §I, §I, §I, §VIII-A. [17] Q. Lan, Y. Pan, A. Fyshe, and M. White (2019) Maxmin q-learning: controlling the estimation bias of q-learning. In International Conference on Learning Representations, Cited by: §I, §I. [18] S. Levine, A. Kumar, G. Tucker, and J. Fu (2020) Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643. Cited by: §I, §IV. [19] Y. Li, Y. Wang, and X. Tan (2023) Self-imitation guided goal-conditioned reinforcement learning. Pattern Recognition 144, p. 109845. External Links: Document, ISSN 0031-3203, Link Cited by: §I. [20] H. Ma, J. Wu, N. Feng, C. Xiao, D. Li, H. Jianye, J. Wang, and M. Long (2024) HarmonyDream: task harmonization inside world models. In Forty-first International Conference on Machine Learning, Cited by: §I. [21] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. (2015) Human-level control through deep reinforcement learning. nature 518 (7540), p. 529–533. Cited by: item 1). [22] A. Nair, A. Gupta, M. Dalal, and S. Levine (2020) Awac: accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359. Cited by: §I, §VIII-A. [23] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy (2016) Deep exploration via bootstrapped dqn. Advances in neural information processing systems 29. Cited by: §I. [24] T. Pang, G. Wu, Y. Zhang, B. Wang, and Y. Yin (2025) QFAE: q-function guided action exploration for offline deep reinforcement learning. Pattern Recognition 158, p. 111032. External Links: Document, ISSN 0031-3203, Link Cited by: §I. [25] X. B. Peng, A. Kumar, G. Zhang, and S. Levine (2019) Advantage-weighted regression: simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177. Cited by: §I, §I. [26] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. (2016) Mastering the game of go with deep neural networks and tree search. nature 529 (7587), p. 484–489. Cited by: §I. [27] R. Sun, H. Zang, X. Li, and R. Islam (2024) Learning latent dynamic robust representations for world models. arXiv preprint arXiv:2405.06263. Cited by: §I. [28] K. Wang, H. Zhao, X. Luo, K. Ren, W. Zhang, and D. Li (2022) Bootstrapped transformer for offline reinforcement learning. Advances in Neural Information Processing Systems 35, p. 34748–34761. Cited by: §I. [29] Z. Wang, J. J. Hunt, and M. Zhou (2022) Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193. Cited by: §I. [30] Z. Wang, J. J. Hunt, and M. Zhou (2023) Diffusion policies as an expressive policy class for offline reinforcement learning. External Links: 2208.06193, Link Cited by: §VIII-A. [31] Z. Wang, A. Novikov, K. Zolna, J. S. Merel, J. T. Springenberg, S. E. Reed, B. Shahriari, N. Siegel, C. Gulcehre, N. Heess, et al. (2020) Critic regularized regression. Advances in Neural Information Processing Systems 33, p. 7768–7778. Cited by: §I, §I. [32] Y. Wen, J. Si, A. Brandt, X. Gao, and H. H. Huang (2020) Online reinforcement learning control for the personalization of a robotic knee prosthesis. IEEE Transactions on Cybernetics 50 (6), p. 2346–2356. External Links: Document Cited by: §I. [33] Y. Wu, G. Tucker, and O. Nachum (2019) Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361. Cited by: §I, §I, §I. [34] Y. Wu, S. Zhai, N. Srivastava, J. Susskind, J. Zhang, R. Salakhutdinov, and H. Goh (2021) Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140. Cited by: §I. [35] G. Xiang and J. Su (2021) Task-oriented deep reinforcement learning for robotic skill acquisition and control. IEEE Transactions on Cybernetics 51 (2), p. 1056–1069. External Links: Document Cited by: §I. [36] T. Xie, C. Cheng, N. Jiang, P. Mineiro, and A. Agarwal (2021) Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems 34, p. 6683–6694. Cited by: §I. [37] T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai (2021) Policy finetuning: bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems 34, p. 27395–27407. Cited by: §I. [38] X. Yang, Z. Ji, J. Wu, Y. Lai, C. Wei, G. Liu, and R. Setchi (2022) Hierarchical reinforcement learning with universal policies for multistep robotic manipulation. IEEE Transactions on Neural Networks and Learning Systems 33 (9), p. 4727–4741. External Links: Document Cited by: §I. [39] P. You, L. Wang, A. Nguyen, X. Zhang, and B. Huang (2025) Channel-adaptive generative reconstruction and fusion for multi-sensor graph features in few-shot fault diagnosis. Information Fusion, p. 103742. Cited by: §IV. [40] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma (2020) Mopo: model-based offline policy optimization. Advances in Neural Information Processing Systems 33, p. 14129–14142. Cited by: §I. [41] F. Zhang, H. Kung, F. Zhang, C. Yang, and J. Gan (2025) AI-powered spatiotemporal imputation and prediction of chlorophyll-a concentration in coastal ecosystems. Nature Communications 16 (1), p. 7656. Cited by: §I. [42] F. Zhang, M. Zhang, W. Chen, S. Wang, X. Zhang, J. Li, and Y. Yang (2025) Inhibiting error exacerbation in offline reinforcement learning with data sparsity. IEEE Transactions on Neural Networks and Learning Systems (), p. 1–15. External Links: Document Cited by: §I, §I. [43] X. Zhang, J. Liu, X. Zhang, and Y. Lu (2024) Multiscale channel attention-driven graph dynamic fusion learning method for robust fault diagnosis. IEEE Transactions on Industrial Informatics 20 (9), p. 11002–11013. Cited by: §IV. [44] Z. Zhang, Y. Ong, D. Wang, and B. Xue (2021) A collaborative multiagent reinforcement learning method based on policy gradient potential. IEEE Transactions on Cybernetics 51 (2), p. 1015–1027. External Links: Document Cited by: §I.