Paper deep dive
Chunk-Guided Q-Learning
Gwanwoo Song, Kwanyoung Park, Youngwoon Lee
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/22/2026, 5:03:49 AM
Summary
Chunk-Guided Q-Learning (CGQ) is an offline reinforcement learning algorithm that addresses the trade-off between single-step TD learning (which suffers from error accumulation) and action-chunked TD learning (which can be suboptimal due to open-loop constraints). CGQ regularizes a fine-grained single-step critic toward a chunk-based critic, achieving tighter optimality bounds and superior performance on long-horizon OGBench tasks.
Entities (4)
Relation Signals (3)
Chunk-Guided Q-Learning → evaluatedon → OGBench
confidence 95% · Empirically, CGQ achieves strong performance on challenging long-horizon OGBench tasks
Chunk-Guided Q-Learning → improvesupon → Temporal Difference Learning
confidence 95% · CGQ attains tighter critic optimality bounds than either single-step or action-chunked TD learning alone.
Chunk-Guided Q-Learning → utilizes → Flow Q-Learning
confidence 90% · In this paper, we use FQL for single-step policy extraction.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In offline reinforcement learning (RL), single-step temporal-difference (TD) learning can suffer from bootstrapping error accumulation over long horizons. Action-chunked TD methods mitigate this by backing up over multiple steps, but can introduce suboptimality by restricting the policy class to open-loop action sequences. To resolve this trade-off, we present Chunk-Guided Q-Learning (CGQ), a single-step TD algorithm that guides a fine-grained single-step critic by regularizing it toward a chunk-based critic trained using temporally extended backups. This reduces compounding error while preserving fine-grained value propagation. We theoretically show that CGQ attains tighter critic optimality bounds than either single-step or action-chunked TD learning alone. Empirically, CGQ achieves strong performance on challenging long-horizon OGBench tasks, often outperforming both single-step and action-chunked methods.
Tags
Links
- Source: https://arxiv.org/abs/2603.13971v1
- Canonical: https://arxiv.org/abs/2603.13971v1
Trouble viewing inline? Open PDF directly →
Full Text
73,923 characters extracted from source content.
Expand or collapse full text
Chunk-Guided Q-Learning Gwanwoo Song 1 Kwanyoung Park 2 Youngwoon Lee 1 Abstract In offline reinforcement learning (RL), single- step temporal-difference (TD) learning can suf- fer from bootstrapping error accumulation over long horizons. Action-chunked TD methods mit- igate this by backing up over multiple steps, but can introduce suboptimality by restricting the pol- icy class to open-loop action sequences. To re- solve this trade-off, we present Chunk-Guided Q-Learning (CGQ), a single-step TD algorithm that guides a fine-grained single-step critic by reg- ularizing it toward a chunk-based critic trained using temporally extended backups. This reduces compounding error while preserving fine-grained value propagation. We theoretically show that CGQ attains tighter critic optimality bounds than either single-step or action-chunked TD learn- ing alone. Empirically, CGQ achieves strong performance on challenging long-horizon OG- Bench tasks, often outperforming both single- step and action-chunked methods. Project page: https://gwanwoosong.github.io/cgq 1. Introduction Q-learning is a central approach in offline reinforcement learning (RL), yet it often struggles to learn accurate value functions on long-horizon tasks with sparse rewards (Park et al., 2025c; 2024; Park & Lee, 2025). A core issue lies in single-step temporal difference (TD) learning. Because the regression target depends on the critic’s own value es- timates, small errors compound across successive Bellman backups (Figure 1, left) (Sutton et al., 1998; Van Hasselt et al., 2018). In offline RL, this effect is amplified by limited data coverage, so target errors cannot be corrected by new interaction, and the accumulated error can overwhelm the learning signal and severely degrade performance as the horizon grows (Park et al., 2025c). 1 Department of Artificial Intelligence, Yonsei Univer- sity 2 UC Berkeley.Correspondence to:Gwanwoo Song <sgw1213@yonsei.ac.kr>. Preprint. March 17, 2026. Motivated by the observation that horizon reduction can mitigate bootstrapping error accumulation (Park et al., 2024; 2025c; Li et al., 2025b), we propose a simple yet effective way to leverage chunked backups by guiding a single-step critic with a chunk-based critic, gaining long-range stability while retaining fine-grained single-step compositionality. Horizon reduction approaches reduce the effective backup depth using action chunking (Li et al., 2025b;a),n-step re- turns (Park et al., 2025c), or hierarchical RL (Park et al., 2024; 2025c), and have been shown effective in long- horizon, sparse-reward tasks. Notably, action chunking has been widely adopted in offline RL due to its simplicity and strong empirical performance (Li et al., 2025b;a; Park et al., 2025a; Kim et al., 2025). However, this benefit comes at a cost in offline settings. In particular,n-step returns rely on multi-step off-policy rollouts, which can introduce bias when behavior and target policies differ (Hernandez-Garcia & Sutton, 2019), leading to suboptimal value estimates in long-horizon tasks (Park et al., 2025c). Action chunking exhibits a related but distinct limitation: it assumes open-loop execution over the entire chunk and therefore cannot properly value reactive decisions within the chunk (Li et al., 2025a), hindering fine-grained credit assignment and trajectory stitching (Figure 1, right). These limitations raise a natural question: “Can we leverage the stability benefits of horizon reduction without sacrificing the optimality and compositionality of single-step TD learning?” To address this challenge, we propose Chunk-Guided Q- Learning (CGQ), which augments standard single-step TD learning with a regularizer that guides the critic toward an action-chunked critic trained via chunk-based TD up- dates. This combines the strengths of both views: the action- chunked critic provides a stable long-range learning signal, while the single-step critic preserves fine-grained value prop- agation and trajectory stitching. Our contributions are threefold: • We introduce CGQ, an offline RL method that effec- tively mitigates TD error accumulation by regularizing a single-step critic toward a chunk-based critic. •We provide theoretical results showing CGQ yields 1 arXiv:2603.13971v1 [cs.LG] 14 Mar 2026 Chunk-Guided Q-Learning x CGQ (Ours)Action-chunked TDCGQ (Ours)Single-step TD E Error TD backups Error TD backups S S E Single-stepTD backupsChunk-guidedTD backups Figure 1. Advantage of Chunk-Guided Q-Learning (CGQ) over single-step and action-chunked TD learning. (Left) Single-step TD learning can suffer from compounding error because it bootstraps from its own value estimates; CGQ mitigates this by guiding the critic toward an action-chunked critic trained with temporally extended backups. (Right) Action-chunked TD learning can be suboptimal (in red arrows) because chunked TD backups assume open-loop action sequences, limiting fine-grained stitching across trajectories (gray arrows denote dataset trajectories). In contrast, CGQ can recover the optimal reactive policy (in blue arrows) by retaining single-step TD learning. tighter critic optimality bounds than either single-step or action-chunked TD learning alone, and give an intu- itive explanation of when CGQ is beneficial. • We demonstrate that CGQ often outperforms both action-chunked and single-step methods on long- horizon OGBench (Park et al., 2025b) tasks, surpassing prior action-chunking approaches. 2. Preliminaries Problem setting.We consider a Markov Decision Process (MDP) defined asM = (S,A,p,r,μ,γ), whereSis the state space,Ais the action space,p(s ′ | s,a) : S ×A→ ∆(S)is the transition dynamics,r(s,a) : S ×A → R is the reward function,μ(s) ∈ ∆(S)is the initial state distribution, and γ ∈ (0, 1) is the discount factor. 1 We study the offline RL setting, where the goal is to learn a policyπthat maximizesE [ P ∞ t=0 γ t r(s t ,a t )] given a fixed dataset of transitionsD =(s,a,r,s ′ )collected by a be- havior policyπ D , without further environment interactions. TD learning. We consider a parameterized policyπ θ (a| s) that will be optimized from the offline dataset and used for control. We learn a criticQ φ (s,a)that estimates the discounted return,E s 0 =s, a 0 =a [ P ∞ t=0 γ t r(s t ,a t )] by minimizing the one-step temporal difference (TD) loss (Sutton, 1988): L TD (φ) = E (s,a,r,s ′ )∼D, a ′ ∼π θ (·|s ′ ) h Q φ (s,a)− r− γQ ̄ φ (s ′ ,a ′ ) 2 i , (1) whereQ ̄ φ is a target network updated as an exponential moving average of Q φ (Mnih et al., 2013). TD learning with action chunks. Instead of training a critic that evaluates a single actiona t , we also con- sider an action-chunked criticQ φ c (s t , a t )overh-step ac- 1 ∆(X ) denotes the set of probability distributions on spaceX. tion sequencesa t = (a t ,a t+1 ,· ,a t+h−1 ). Letr t = P h−1 i=0 γ i r(s t+i ,a t+i ) denote theh-step reward. We train Q φ c by minimizing the chunked version of Eq. (1): L TD c (φ c ) =E (s t ,a t ,s t+h )∼D, a t+h ∼π θ c (·|s t+h ) h Q φ c (s t , a t ) − r t − γ h Q ̄ φ c (s t+h , a t+h ) 2 i , (2) whereQ ̄ φ c is a target network forQ φ c , andπ θ c (a| s) is an action-chunked policy. SinceQ φ c (s t , a t )evaluates open- loop action sequences, greedy chunk selection may fail to recover the optimal reactive policy in the original MDP. Flow Q-Learning (FQL).A common challenge in offline RL is that policy improvement can select out-of-distribution actions, leading to unreliable value estimates. FQL (Park et al., 2025d) addresses this by learning an expressive flow- based behavior policy and constraining policy improvement toward the behavior distribution. Concretely, FQL first fits a flow-based behavior policy π θ (s,z) : S × A → Aby learning a velocity fieldv θ with a flow matching loss (Lipman et al., 2023; Albergo & Vanden-Eijnden, 2023; Liu et al., 2023a) L flow (θ) = E (s t ,a t )∼D, u∼U([0,1]), z∼N (0,I A ) ∥v θ (u,s t ,a z )− (a t − z)∥ 2 2 , (3) wherea z = (1− u)z + ua t . FQL then learns a constrained policyπ ω (s,z)by trading off Q-value maximization and distillation toward the flow-based behavior policy: L FQL (ω) = E s∼D,z∼N (0,I A ) −Q φ (s,π ω (s,z)) + α∥π θ (s,z)− π ω (s,z)∥ 2 2 . (4) In this paper, we use FQL for single-step policy extraction. FQL with action chunks (QC-FQL). FQL naturally ex- tends to action-chunked Q-learning by operating on action 2 Chunk-Guided Q-Learning Figure 2. CGQ improves value estimation over single-step and action-chunked TD learning. We compare single-step TD, action- chunked TD, and CGQ in a simple gridworld using a fixed offline dataset. To simulate function approximation error, we add noise to TD targets. The upper-right panel shows the optimal value function. The left panels visualize the value prediction error after k ∈1, 3, 10, 100TD updates and the middle panels show the learned value functions. Single-step TD exhibits large errors in states far from the goal due to bootstrapping error accumulation over long backup chains, while action-chunked TD propagates values quickly but can converge to a suboptimal critic due to open-loop chunks, leaving persistent errors at intermediate states. In contrast, CGQ achieves lower error (see the lower-right plot) by combining rapid chunked value propagation with fine-grained single-step value backups. sequences instead of single actions. QC-FQL (Li et al., 2025b) learns an action-chunked policy and critic with chunk lengthh. Analogous to FQL, QC-FQL first fits a flow-based behavior policyπ θ c (s, z) : S ×A h → A h by learning a velocity field v θ c : L flow c (θ c ) = E (s t ,a t )∼D, z∼N (0,I A h ), u∼U([0,1]) ∥v θ c (u,s t , a z )− (a t − z)∥ 2 2 , (5) wherea z = (1− u)z + ua t . QC-FQL then learns a con- strained action-chunked policy π ω c (s, z) by optimizing L FQL c (ω c ) = E s∼D,z∼N (0,I A ) −Q φ c (s,π ω c (s, z)) + α∥π θ c (s, z)− π ω c (s, z)∥ 2 2 . (6) In this paper, we use QC-FQL for action-chunked policy extraction. 3. Motivation: Trade-offs in Action-Chunked TD Learning Before introducing our method, we examine the comple- mentary strengths and weaknesses of single-step and action- chunked TD learning, which our approach is designed to balance. Figure 2 illustrates this trade-off in a simple grid- world. Single-step TD learning eventually converges to the correct value function, but requires many updates to prop- agate reward signal, making it vulnerable to bootstrapping error accumulation in long-horizon tasks. Action-chunked TD learning propagates values rapidly in early iterations, but its open-loop backups can fail to stitch across action chunks, leading to a suboptimal value function even after convergence. 3.1. Action-Chunked TD Learning Reduces Bootstrapping Error Accumulation Single-step TD learning is vulnerable to bootstrapping er- ror accumulation since the TD target,r t + γQ(s t+1 ,a t+1 ), depends on the critic’s own predictions. Errors in value estimates at future states can therefore propagate backward through successive Bellman updates. This issue is exac- erbated as the horizon increases and reward information must be propagated through many consecutive bootstrap- ping steps. Action-chunked TD learning partially mitigates this issue by reducing the effective horizon length (backup depth). For chunk sizeh, lettinga t = (a t ,a t+1 ,· ,a t+h−1 ), the chunked TD target takes the form: h−1 X k=0 γ k r t+k + γ h Q(s t+h , a t+h ),(7) which requires fewer bootstrapping steps to propagate re- ward information across long horizons. Although the target still depends on the critic, the error accumulation term scales with 1 1−γ h rather than 1 1−γ , which can yield up to anh-fold improvement whenγ ≈ 1(see Theorem 4.1). This reduc- tion helps explain the fast initial value propagation observed for chunk-based TD learning in Figure 2. 3.2. Action-Chunked Critics Can Be Suboptimal Despite its improved stability, action chunking can be struc- turally suboptimal relative to single-step TD learning be- cause it restricts the policy class to open-loop action se- quences. LetQ ∗ (s t ,a t )denote the optimal single-step Q- function and letQ ∗ c (s t , a t )as the optimal Q-function for an action-chunked policy. Since action chunking restricts the 3 Chunk-Guided Q-Learning policy class to open-loop action sequences, in general Q ∗ c (s t , a t )≤ Q ∗ (s t ,a t ).(8) In stochastic environments where optimal behavior is re- active, this restriction can prevent precise action stitching across states, leading to persistent value inaccuracies at in- termediate states. This effect is visible in Figure 2: despite reduced early propagation, chunk-based backups can con- verge to a value function that remains suboptimal in parts of the state space. Implication. Taken together, these observations suggest a fundamental trade-off. Action-chunked critics provide more stable long-horizon value propagation by reducing backup depth, but can sacrifice optimality due to open-loop execution. Single-step TD learning supports fine-grained value updates and trajectory stitching, but is more suscepti- ble to bootstrapping error accumulation over long horizons. This complementary structure motivates combining the two: leverage chunked learning signals for stability while retain- ing single-step structure for compositionality. 4. Chunk-Guided Q-Learning We introduce Chunk-Guided Q-Learning (CGQ), a sim- ple offline RL method that combines the long-horizon sta- bility of action-chunked TD learning with the fine-grained optimality of single-step TD learning. CGQ regularizes the single-step critic toward a chunked critic trained with temporally extended backups, reducing compounding error while preserving step-wise updates. We emphasize that our goal is to train a single-step policy and critic; the action-chunked policy and critic are trained only as auxiliary components to guide learning of the single- step critic. Throughout this paper, we useφto denote critic parameters andθ,ωto denote policy parameters. We use the subscript (·) c to indicate quantities associated with action-chunked critics, policies, or losses. 4.1. Practical Implementation Action-chunked TD learning. We define an action- chunked criticQ φ c (s t , a t )that evaluates anh-step action sequencea t = (a t ,...,a t+h−1 ). Given an action-chunked policyπ ω c (a t | s t ), we train the action-chunked criticQ φ c with the action-chunked TD loss: L TD c (φ c ) = E (s t ,a t )∼D h Q φ c (s t , a t )− r t − γ h Q ̄ φ c (s t+h , a t+h ) 2 i , (9) where r t = P h−1 k=0 γ k r t+k and a t+h ∼ π ω c (·| s t+h ). While the action-chunked critic can be suboptimal, it typ- ically reduces bootstrapping error accumulation. We opti- mizeπ ω c using QC-FQL (Li et al., 2025b) for its simplicity and robustness across tasks. Single-step TD learning. We train a single-step critic Q φ (s,a) with the standard TD objective: L TD (φ) = E (s,a,r ′ ,s ′ )∼D, a ′ ∼π ω (·|s ′ ) h Q φ (s,a)− r− γQ ̄ φ (s ′ ,a ′ ) 2 i , (10) where ̄ φis a target network updated as an exponential mov- ing average ofφ(Mnih et al., 2013). Single-step TD enables fine-grained value propagation across states, but suffers from bootstrapping error accumulation over long horizons. Critic regularization via action-chunked critic.To com- bine the complementary strengths of both critics, CGQ reg- ularizes the single-step criticQ φ (s t ,a t )toward the action- chunked criticQ φ c (s t , a t )using an upper-expectile distilla- tion loss: L reg (φ) = E (s t ,a t )∼D h ℓ τ Q φ c (s t , a t )− Q φ (s t ,a t ) i . (11) whereℓ τ (u) = |τ − I(u < 0)|u 2 , with0.5 ≤ τ < 1 (Kostrikov et al., 2022). This asymmetric alignment biases the single-step critic toward higher-value estimates from the chunked critic, allowing the single-step critic to be guided by the action-chunked critic, while being less affected from the suboptimality of the action-chunked critic. Overall objective. We train the single-step criticQ φ by minimizing L CGQ (φ) =L TD (φ) + βL reg (φ),(12) whereβcontrols the strength of guidance from the action- chunked critic. We then extract the single-step policy π ω fromQ φ using the FQL policy extraction (Park et al., 2025d). The overall algorithm is described in Algorithm 1. 4.2. Theoretical Analysis We now provide a theoretical analysis of CGQ to formalize the trade-off introduced by chunk guidance. In particular, we study how regularizing single-step TD learning toward an action-chunked critic affects the optimality and error propagation of the learned value function. Minimizing the one-step Bellman residual∥T Q−Q∥should recover the optimal action-value functionQ ∗ , sinceQ ∗ is the unique fixed point of the Bellman operatorT Q(s,a) = E[r(s,a) + γ max a ′ Q(s ′ ,a ′ )]. In practice, however, the Bellman evaluation is noisy due to function approximation, finite data coverage, and other sources of error. To model 4 Chunk-Guided Q-Learning Algorithm 1 Chunk-Guided Q-Learning (CGQ) Require: Offline datasetD, action chunk size h 1: Initialize single-step criticQ φ , behavioral flow policy π θ , policy π ω 2: Initialize action-chunked criticQ φ c , behavioral flow policy π θ c , policy π ω c 3: while not converged do 4:Sample(s t , a t ,r t:t+h ,s t+1 ,s t+h )∼D 5: ▷ Train action-chunked policy π ω c and critic Q φ c 6:Update chunked critic Q φ c to minimizeL TD c (φ c ) 7:Update chunked policiesπ θ c ,π ω c with Eqs. (5) and (6) 8: ▷ Train single-step policy π ω and critic Q φ 9:Update critic Q φ to minimizeL TD (φ) + βL reg (φ) 10:Update policies π θ ,π ω with Eqs. (3) and (4) 11: end while 12: return single-step policy π ω this behavior, we consider a stochastic iterative process defined as b T Q k =T Q k + ε k ,(13) whereε k is a zero-mean random perturbation satisfying E ∥ε k ∥ 2 ≤ σ 2 for some σ > 0. LetQ ∗ denote the fixed point ofT. Under a contraction as- sumption of operatorT, we obtain the following asymptotic error bound. Theorem 4.1 (Error accumulation of one-step TD learn- ing). LetTbe aγ-Lipschitz linear operator inL 2 norm, and b Tbe a stochastic iterative process defined as above. Then, the asymptotic expected squared error satisfies: lim sup k→∞ E ∥Q k − Q ∗ ∥ 2 ≤ σ 2 1− γ 2 (14) Theorem 4.1 indicates that the error in each Bellman update accumulates geometrically, and therefore the overall error scales withO( p 1/(1− γ 2 )) . In long-horizon tasks, where γ is close to 1, this error accumulation exacerbates. Here, we prove how CGQ can improve this bound by adding a regularization towards the suboptimal but stable action- chunked critic. We note that for the following derivations, the regularization target does not need to be an action- chunked critic in general; however, we keep the notation of regularization target asQ c , as we focus on the action- chunked critic in our method. To control interactions be- tween stochastic noise and regularization bias, we addition- ally assume that the curvature ofT is bounded. Theorem 4.2 (Improved bound via CGQ regularization, informal). Let b T β be the regularized stochastic iterative process: b T β Q k = 1 1+β b T Q k + β 1+β Q c . Then, the asymptotic expected squared error satisfies: lim sup k→∞ E ∥Q k − Q ∗ ∥ 2 ≤ σ 2 (1 + β) 2 − γ 2 |z Error accumulation + β 2 ∥Q ∗ − Q c ∥ 2 (1 + β− γ) 2 |z Suboptimality , + L(1 + β)β∥Q ∗ − Q c ∥σ 2 ((1 + β) 2 − γ 2 )(1 + β− γ) 2 |z Cross-term (15) where L denotes the maximum curvature ofT . This bound is minimized at someβ ∗ < ∞, being strictly smaller than∥Q ∗ − Q c ∥ 2 . Moreover, if (but not necessarily only if)L∥Q c − Q ∗ ∥ < 2(1−γ) 1+γ , this bound is minimized at some0 < β ∗ <∞, and is strictly smaller than both σ 2 1−γ 2 and∥Q ∗ − Q c ∥ 2 . Theorem 4.2 makes explicit the bias–variance trade-off in- duced by chunk regularization. The first term captures error accumulation from noisy single-step TD updates, the sec- ond reflects bias from regularization toward a suboptimal critic, and the third term quantifies their interaction. The theorem shows that CGQ always gives tighter bound than action-chunked TD learning. Importantly, whether an interior optimum exists (i.e., whether CGQ outperforms pure single-step TD learning) depends on the magnitude of the cross-term, which is de- termined by the curvatureLof the Bellman operator. We emphasize that this curvature condition is sufficient but not necessary; it can still have a nonzero finite minimizerβ ∗ with strictly smaller bound, even if the condition is not sat- isfied. Please see Section E for the formal statement and the proof of Theorem 4.2. 5. Experiments In this section, we evaluate CGQ on challenging long- horizon tasks and compare against strong single-step and action-chunked offline RL methods. We further analyze when chunk guidance helps via ablations on the regulariza- tion strength and chunk length. 5.1. Experimental Setup We evaluate CGQ on the OGBench benchmark (Park et al., 2025b), which contains challenging long-horizon tasks.For robotic manipulation, we consider six tasks:cube-double,cube-triple,cube-quadruple, puzzle-3x3,puzzle-4x4, andscene, and train on theplaydataset.For navigation, we evaluate three tasks:antmaze-large,antmaze-giant, and humanoidmaze-medium, and train on thenavigate dataset. Additional training details are provided in Ap- pendix, Section A. 5 Chunk-Guided Q-Learning Table 1. Results in OGBench manipulation and navigation benchmarks. Entries report average success rate over 150 evaluation episodes, aggregated across 4 seeds. We report mean and standard deviation within each task category when available; entries without “±” are copied from prior work (Park et al., 2025d; Li et al., 2025b; Kim et al., 2025). We highlight results within95%of the best performance. We note that DQC is evaluated under smaller, reward-based datasets in our setting (rather than the large goal-conditioned datasets used in the original paper), which may lead to lower performance than originally reported. Task Category FQL NFQL QC-FQL DEAS DQC CGQ scene-sparse (5 tasks) 57 18 84 93±2 81±2 86±3 cube-double (5 tasks) 29 11 39 48 31±2 69±3 puzzle-3x3-sparse (5 tasks) 100 98 63 99±1 100±0 99±3 puzzle-4x4 (5 tasks) 17 23±5 26±2 39±3 8±2 28±2 cube-triple (5 tasks) 10 23 83 82 2±1 94±1 cube-quadruple (5 tasks) 17 36 45 64 56±22 82±2 Average (Manipulation) 38 35 57 71 46±3 76±2 antmaze-large (5 tasks) 79 47±5 20±3 67±3 71±6 67±3 antmaze-giant (5 tasks) 9 2±1 0±0 8±1 10±1 4±2 humanoidmaze-medium (5 tasks) 58 23±3 4±1 37±3 93±2 34±5 Average (Navigation) 49 24±3 8±2 37±1 58±3 35±4 Methods.We compare against both single-step and action- chunked offline RL methods. For single-step methods, we consider FQL (Park et al., 2025d), which learns a behavior flow policy and regularizes the one-step policy with the be- havior policy, and its variant withn-step return, denoted as NFQL. For action-chunked methods, we consider QC (Li et al., 2025b), DEAS (Kim et al., 2025), and DQC (Li et al., 2025a). QC adapts action-chunked critic and policy to exist- ing single-step frameworks (e.g., FQL). DEAS decouples policy and value learning using expectile regression, follow- ing IQL (Kostrikov et al., 2022). DQC decouples chunk lengths for the critic and policy by distilling a large-chunk critic into a smaller-chunk critic for the policy. 5.2. Results We report the performance of CGQ and baselines in Table 1 on OGBench manipulation and navigation tasks. Manipulation.On manipulation tasks, CGQ achieves the best performance across all task categories, outperforming single-step,n-step, and action-chunked offline RL methods. Notably, FQL performs well on relatively shorter-horizon tasks such aspuzzle-3x3-sparseandscene-sparse, but degrades significantly on longer-horizon tasks includ- ingpuzzle-4x4,cube-triple, andcube-quadruple. In contrast, action-chunked methods struggle on shorter- horizon tasks (e.g.,puzzle-3x3-sparse) due to the re- stricted open-loop policy class. CGQ substantially outper- forms in both regimes while using a single-step policy, indi- cating that chunking guidance successfully balances accu- rate step-wise value propagation with reduced bootstrapping error. Navigation (negative results). On navigation tasks, CGQ outperforms or matches methods that use action-chunked policies, but underperforms FQL, which does not rely on an action-chunked critic. This is consistent with prior observa- tions that action-chunked methods can be less effective in locomotion domains (Park et al., 2025a; Li et al., 2025a), where highly reactive control and fine-grained value esti- mation are crucial. We believe this is primarily due to the difficulty of action-chunked policy learning in locomotion, which in turn yields an unreliable chunked critic that cor- rupts the single-step critic via regularization. While DEAS employs IQL-style critic learning, it also suffers from this issue. DQC partially avoids this by decoupling the chunk sizes for the critic and the policy; notably, our empirical tuning selects a policy chunk size of 1 for all navigation tasks, thereby reducing the burden of chunked policy learn- ing. Improving the chunked critic and developing adaptive weighting schemes for the regularizer are promising direc- tions for future work. 5.3. Q&As Q: How is CGQ different from n-step methods, which also use single-step policy and critic? While CGQ andn-step methods both learn a single-step policy and critic, they differ fundamentally in how they incorporate horizon reduction. Inn-step methods, horizon reduction is achieved by modifying the one-step TD target with multi-step returns. However,n-step targets typically still bootstrap from the same single-step critic at the tail, so long-horizon information remains entangled with the critic’s own estimates. 6 Chunk-Guided Q-Learning Although CGQ may appear superficially related to TD(λ)(Sutton, 1988), in that both blend single-step learn- ing with longer-horizon information, the key distinction is thatn-step TD continues to rely on the single-step critic for bootstrapping, and can therefore inherit the same error accumulation issue. In contrast, CGQ introduces horizon re- duction via an auxiliary action-chunked critic that is trained separately, avoiding additional error propagation from the TD target. Consistent with this distinction, NFQL, ann- step variant of FQL, performs substantially worse than CGQ (Table 1). Q: Is CGQ the best way to blend action-chunked RL with single-step RL? To investigate this question, we evaluate several alternative designs for combining action-chunked and single-step value learning, and compare them against CGQ. CGQ-DISTILL. This variant removes single-step TD learning and trains the critic using only the distillation loss, i.e.,L(φ) = L reg , relying entirely on chunk-based value backups. It is analogous to DQC (Li et al., 2025a) with h a = 1, and tests whether chunk guidance alone is suffi- cient for accurate value estimation. CGQ-MAX. This variant constructs an optimistic target by taking the maximum of the single-step and chunk-based value estimates: L max (φ) = E s,a,r,s ′ ,a∼D,a ′ ∼π ω h Q φ (s,a)− max r + γ ̄ Q φ (s ′ ,a ′ ),Q φ c (s, a) 2 i . (16) This tests whether aggressively following the larger of the single-step and chunk-based estimates improves per- formance. CGQ-OPPOSITE. This variant regularizes the action- chunked critic toward a single-step critic. While superfi- cially symmetric, it takes the worst side of both; the chunked 0.00.51.0 Training steps (M) 0 50 Success rate (%) TD Update CGQ (Ours) Max Distill Opposite Figure 3. Performance of various TD update designs for blend- ing single-step and multi-step TD learning. CGQ’s regulariza- tion yields the most effective critic. critic remains constrained by open-loop execution, and guid- ance from the single-step critic reintroduces sensitivity to bootstrapping errors. As shown in Figure 3, none of these alternatives matches CGQ. While CGQ is not necessarily the optimal or unique way to combine action-chunked and single-step RL, these results suggest that CGQ’s simple regularization scheme provides an effective balance between reduced error accu- mulation and fine-grained single-step value propagation. Q: How does action-chunk size affect CGQ? We analyze the effect of the chunk sizehfor the action- chunked TD learning by evaluating CGQ withh ∈ 5, 10, 15, 20incube-double. In CGQ, the chunk-based critic primarily serves to provide a more stable value backup with reduced bootstrapping error accumulation. Conse- quently, we expect larger chunk sizes to yield better reg- ularization and improved performance. Consistent with this intuition, we observe that performance generally improves as the chunk size increases, as shown in Figure 4. 0.00.51.0 Training steps (M) 0 50 Success rate (%) Chunk size 5 10 15 20 Figure 4. Performance of CGQ across different action-chunk sizes h. In general, larger chunks improve performance. Q: How important is the distillation coefficient β? We investigate the importance of the distillation coefficient β, which controls the strength of regularization towards the action-chunked critic. Specifically, we evaluate CGQ withβ ∈0.001, 0.01, 0.1, 1incube-double, and report the result in Figure 5. We observe that the choice ofβis crucial to the performance: small values provide insuffi- cient stabilization against bootstrapping error accumulation, while overly large values bias the critic too much toward the suboptimal chunk-based estimates. 0.00.51.0 Training steps (M) 0 50 Success rate (%) Weight(β) 0.001 0.01 0.1 1 Figure 5. Performance of CGQ under varyingβ. Choice ofβis important for the performance. 7 Chunk-Guided Q-Learning 0.00.51.0 Training steps (M) 0 50 Success rate (%) Expectile 0.95 0.9 0.8 0.7 0.6 0.5 Figure 6. Performance of CGQ under varying expectile coeffi- cient τ . CGQ is robust to choice of τ . Q: Is expectile coefficient τ important? We study the effect of the expectile parameterτused in the distillation loss by running CGQ incube-doubleforτ ∈ 0.5, 0.6, 0.7, 0.8, 0.9, 0.95 . As shown in Figure 6, CGQ is robust to the choice ofτ: the performance is similar even when usingτ = 0.5(i.e., without optimistic weighting), in contrast to prior observations in DQC (Li et al., 2025a). We attribute this robustness CGQ using the action-chunked critic as a stabilizing regularizer, rather than relying on pure distillation, as in DQC. 6. Related Work Offline RL. Offline RL aims to learn a policy from a fixed dataset without further environment interaction. The main challenge is distributional shift (Levine et al., 2020), maximizing return often pushes the learned policy away from the data distribution, making value estimation unreli- able. Prior work has explored several strategies have been explored to mitigate this issue, including behavior regular- ization (Wu et al., 2019; Peng et al., 2019; Fujimoto & Gu, 2021; Tarasov et al., 2023; Park et al., 2025d), conservative value estimation (Kumar et al., 2020), in-sample maximiza- tion (Kostrikov et al., 2022; Xiao et al., 2023), rejection sampling (Chen et al., 2023; Hansen-Estruch et al., 2023), and other approaches such as sequence-modeling (Chen et al., 2021; Janner et al., 2021; Jiang et al., 2023) or model- based methods (Kidambi et al., 2020; Yu et al., 2020; 2021; Liu et al., 2023b), and more (Eysenbach et al., 2022; Wang et al., 2023; An et al., 2021; Brandfonbrener et al., 2021; Myers et al., 2025; Park et al., 2025e). Despite these advances, offline RL remains challenging in long-horizon tasks due to error accumulation: small value estimation errors can compound across successive Bellman backups when regression targets depend on the model’s own predictions (Park et al., 2025c; Sutton et al., 1998). Recent work has shown that horizon-reduction techniques can effec- tively mitigate bootstrapping error accumulation by short- ening the effective backup depth, including action chunk- ing (Li et al., 2025b;a),n-step returns (Park et al., 2025c), and hierarchical RL (Park et al., 2024; 2025c). Our method follows this line of work but introduces a novel approach that balances single-step TD learning with horizon-reduced TD learning, combining their complementary strengths. Action chunking for offline RL. Originally popularized in imitation learning (Zhao et al., 2023; Chi et al., 2023; Kim et al., 2024), action chunking has recently emerged as a practical horizon-reduction strategy for offline RL (Seo & Abbeel, 2025; Li et al., 2025b; Kim et al., 2025; Li et al., 2025a). In this framework, the policy predicts a sequence of actionsa t = (a t ,...,a t+h ), and the critic evaluates the value of an entire action chunk,Q(s t , a t ), rather than a single action. By reducing the number of TD backups (by a factor ofh), these methods improve robustness to the bias introduced by self-bootstrapping during value propagation. However, action chunking imposes several challenges in of- fline RL. For example, by enlarging the action space, action chunking complicates policy optimization and exacerbates value overestimation. Prior work mitigates these issues through different mechanisms: QC (Li et al., 2025b) con- strains policies to limit distributional shift via FQL-based extraction (Park et al., 2025d) or rejection sampling (Chen et al., 2023); DEAS (Kim et al., 2025) decouples policy and value learning using expectile regression, following IQL (Kostrikov et al., 2022); and DQC (Li et al., 2025a) high- lights the difficulty of predicting long action sequences and instead performs implicit value learning over large chunks, and distill into a critic with a smaller chunk. However, action-chunked critics have a fundamental limitation: they fail to account for reactive policies, leading to suboptimal value estimation even if perfectly optimized. Addressing this limitation is the focus of this paper. 7. Closing Remarks We presented Chunk-Guided Q-Learning (CGQ), a single- step offline RL method that stabilizes value estimation by regularizing TD learning toward an auxiliary chunk- based critic. CGQ is designed to combine the complemen- tary strengths of single-step and chunk-based TD learning: chunk-based value backups provide stable long-horizon learning signals with reduced compounding error, while single-step TD learning preserves critic optimality and sup- ports reactive policy learning. Theoretically, we formalize the trade-off between the error accumulation of single-step TD learning and the potential suboptimality of chunk-based TD learning, and show that CGQ can find a sweet spot that can improve over either extreme. Empirically, CGQ achieves strong performance on challenging long-horizon OGBench tasks, often outperforming both single-step and action-chunked methods. CGQ also has limitations. When the action-chunked critic is inaccurate, the guidance can be not helpful and purely single-step methods may perform better. Moreover, CGQ 8 Chunk-Guided Q-Learning depends on the regularization weightβand selectingβcan be challenging when transferring to new tasks or multi- task settings. Promising directions for future work include improving the chunk-based guidance signal and developing adaptive or principled strategies for tuning β. Acknowledgements This research was supported by the National Research Foun- dation of Korea (NRF) grant (RS-2024-00333634), the In- stitute of Information & Communications Technology Plan- ning & Evaluation (IITP) grants (RS-2020-I201361, Arti- ficial Intelligence Graduate School Program (Yonsei Uni- versity); and RS-2024-00509279, Global AI Frontier Lab) funded by the Korean government (MSIT). Impact Statement This work aims to improve long-horizon offline learning for autonomous agents, e.g., robots and self-driving cars. While the methods could be used in safety-critical settings, we do not anticipate immediate negative societal impacts from this paper. References Albergo, M. S. and Vanden-Eijnden, E. Building normaliz- ing flows with stochastic interpolants. In International Conference on Learning Representations, 2023. An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertainty- based offline reinforcement learning with diversified q- ensemble. Advances in neural information processing systems, 34:7436–7447, 2021. Ba, J., Kiros, J. R., and Hinton, G. E. Layer normalization. ArXiv, abs/1607.06450, 2016. Brandfonbrener, D., Whitney, W., Ranganath, R., and Bruna, J. Offline rl without off-policy evaluation. In Neural Information Processing Systems, 2021. Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J. Offline reinforcement learning via high-fidelity generative behav- ior modeling. In International Conference on Learning Representations, 2023. Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems, 2021. Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023. Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. R. Contrastive learning as goal-conditioned reinforce- ment learning. In Neural Information Processing Systems, 2022. Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. In Neural Information Processing Systems, volume 34, p. 20132–20145, 2021. Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor- critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023. Hernandez-Garcia, J. F. and Sutton, R. S. Understanding multi-step deep reinforcement learning: A systematic study of the dqn target. arXiv preprint arXiv:1901.07510, 2019. Janner, M., Li, Q., and Levine, S. Offline reinforcement learning as one big sequence modeling problem. In Neu- ral Information Processing Systems, volume 34, p. 1273– 1286, 2021. Jiang, Z., Zhang, T., Janner, M., Li, Y., Rockt ̈ aschel, T., Grefenstette, E., and Tian, Y. Efficient planning in a compact latent action space. In International Conference on Learning Representations, 2023. Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: Model-based offline reinforcement learning. In Neural Information Processing Systems, volume 33, p. 21810–21823, 2020. Kim, C., Lee, H., Seo, Y., Lee, K., and Zhu, Y. Deas: Detached value learning with action sequence for scalable offline rl. arXiv preprint arXiv:2510.07730, 2025. Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., et al. Openvla: An open-source vision-language- action model. In Conference on Robot Learning, 2024. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. In International Conference on Learning Representations, 2022. Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conser- vative q-learning for offline reinforcement learning. In Neural Information Processing Systems, volume 33, p. 1179–1191, 2020. Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. 9 Chunk-Guided Q-Learning Li, Q., Park, S., and Levine, S. Decoupled q-chunking. arXiv preprint arXiv:2512.10926, 2025a. Li, Q., Zhou, Z., and Levine, S. Reinforcement learning with action chunking. In Neural Information Processing Systems, 2025b. Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In Interna- tional Conference on Learning Representations, 2023. Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, 2023a. Liu, Z., Li, S., Lee, W. S., Shuicheng, Y., and Xu, Z. Effi- cient offline policy optimization with a learned model. In International Conference on Learning Representations, 2023b. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. Myers, V., Zheng, B. C., Eysenbach, B., and Levine, S. Of- fline goal-conditioned reinforcement learning with quasi- metric representations. In Neural Information Processing Systems, 2025. Park, K. and Lee, Y. Model-based offline reinforcement learning with lower expectile q-learning. In International Conference on Learning Representations, 2025. Park, K., Park, S., Lee, Y., and Levine, S. Scalable of- fline model-based rl with action chunks. arXiv preprint arXiv:2512.08108, 2025a. Park, S., Ghosh, D., Eysenbach, B., and Levine, S. Hiql: Offline goal-conditioned rl with latent states as actions. In Neural Information Processing Systems, volume 36, 2024. Park, S., Frans, K., Eysenbach, B., and Levine, S. Ogbench: Benchmarking offline goal-conditioned rl. In Interna- tional Conference on Learning Representations, 2025b. Park, S., Frans, K., Mann, D., Eysenbach, B., Kumar, A., and Levine, S. Horizon reduction makes rl scalable. In Neural Information Processing Systems, 2025c. Park, S., Li, Q., and Levine, S. Flow q-learning. In Interna- tional Conference on Machine Learning, 2025d. Park, S., Oberai, A., Atreya, P., and Levine, S. Transitive rl: Value learning via divide and conquer. arXiv preprint arXiv:2510.22512, 2025e. Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019. Seo, Y. and Abbeel, P. Coarse-to-fine q-network with action sequence for data-efficient robot learning. In Neural Information Processing Systems, 2025. Sutton, R. S. Learning to predict by the methods of temporal differences. Machine learning, 3:9–44, 1988. Sutton, R. S., Barto, A. G., et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. Tarasov, D., Kurenkov, V., Nikulin, A., and Kolesnikov, S. Revisiting the minimalist approach to offline reinforce- ment learning. In Neural Information Processing Systems, 2023. Van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648, 2018. Wang, T., Torralba, A., Isola, P., and Zhang, A. Opti- mal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learn- ing, 2023. Wu, Y., Tucker, G., and Nachum, O.Behavior regu- larized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019. Xiao, C., Wang, H., Pan, Y., White, A., and White, M. The in-sample softmax for offline reinforcement learning. In International Conference on Learning Representations, 2023. Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. In Neural Information Processing Systems, volume 33, p. 14129–14142, 2020. Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. Combo: Conservative offline model-based policy optimization. In Neural Information Processing Systems, volume 34, p. 28954–28967, 2021. Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learn- ing fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems, 2023. 10 Chunk-Guided Q-Learning A. Training Details We implement CGQ on top of the codebase of (Li et al., 2025b). Each experiment tasks around 3 hours and approximately 18 hours for cube-triple, cube-quadruple tasks on a single RTX3090 GPU. A.1. Benchmark and Datasets In this work, we consider 3 navigation tasksets and 6 manipulation tasksets from OGBench (Park et al., 2025b) and use the singletaskversions in our experiments (45 tasks in total). Reward setup in the navigation domain is sparse: a reward of -1 is given unless the task is complete, otherwise 0. Forsceneandpuzzle-3x3domain in manipulation, we sparsify the rewards following(Li et al., 2025b). Forcube-double,cube-triple,cube-quadruple,puzzle-4x4, reward is calculated according to the number of completed subtasks (e.g., the number of correctly placed cubes or the number of correctly matched buttons) (Park et al., 2025b). Table 2. Metadata of each task categories used in OGBench experiments. We report the dataset size, episode length, and action dimensionality for all 9 task sets used in our experiments. For the harder tasks, such ascube-tripleandcube-quadruple, we use larger 10M/100M-sized dataset. Task CategoryDataset SizeEpisode LengthAction Dimension) antmaze-large1M10008 antmaze-giant1M20008 humanoidmaze-medium2M (default)200021 scene-sparse1M7505 puzzle-3x3-sparse1M5005 cube-double1M5005 puzzle-4x41M5005 cube-triple10M10005 cube-quadruple100M10005 A.2. Training and Evaluation Protocol We train our method and baselines for 1M gradient steps when we use 1M or 2M datasets, and 2.5M gradient steps when we use 10M and 100M datasets. The 10M dataset is constructed from the first 10 files of the 100M dataset, following Kim et al. (2025). We report the average success rates over the final three evaluation epochs (800K/900K/1M for 1M gradient steps and 2.3M/2.4M/2.5M for 2.5M gradient steps), following (Park et al., 2025c). Results from prior works.For tasks that are already evaluated in prior work, we use the reported results (i.e., numbers with- out±in Table 1). Specifically, for manipulation tasks, results ofcube-double,scene-sparse, andpuzzle-3x3-sparse are taken from (Li et al., 2025b), except for DEAS oncube-double, where results are from (Kim et al., 2025). Results for cube-tripleandcube-quadrupleare also obtained from (Kim et al., 2025). For navigation tasks, FQL results are taken from (Park et al., 2025d). Otherwise, we implement the baselines in our codebase and evaluated according to our evaluation protocol. A.3. Implementation Details Network architecture. Following FQL (Park et al., 2025d), we use [512, 512, 512, 512] 4-layer MLPs with layer normalization (Ba et al., 2016) for critic and policy networks. Forcube-tripleandcube-quadrupletasks, which we use larger datasets, we increased the model size to [1024, 1024, 1024, 1024] for all neural networks. Value learning.For both 1-step critic and chunked critic, we train two Q functions and take the mean of the two Q values for the Q value in the actor objective, calculating the target for the TD learning and the regularization. to improve stability. A.4. Baselines. Here, we briefly introduce the baselines used in this paper. 11 Chunk-Guided Q-Learning FQL (Park et al., 2025d) is a behavior regularization-based offline RL method that learns a constrained policy from an expressive flow-based behavior policy. Despite its simplicity, FQL serves as a strong baseline for various offline RL benchmarks (e.g., OGBench). In this paper, we also use FQL as a policy extraction method for the single-step policy. QC-FQL (Li et al., 2025b) extends FQL to action-chunked Q-learning by operating on action sequences instead of single actions. QC-FQL improves FQL’s performance in manipulation tasks, especially in challenging tasks such ascube-triple or cube-quadruple. In this paper, we also use QC-FQL as a policy extraction method for the action-chunked policy. DEAS (Kim et al., 2025) extends action-chunked offline RL by incorporating detached value learning (Kostrikov et al., 2022) which decouples critic training from the policy to prevent value overestimation in the expanded action space. DEAS further combines distributional RL objectives and dual discount factors to mitigate the high variance introduced by the n-step returns. N-FQL is ann-step return variant of FQL that incorporates horizon reduction via multi-step returns, while retaining a single-step critic and policy. DQC. decouples the chunk sizes used by the critic and the policy by distilling an action-chunked critic into a critic with a smaller chunk size used by the policy. While DQC employs a binary cross-entropy loss for value learning under a goal-conditioned setting, we utilize mean-squared error loss instead, as rewards generally lie in the range[−N, 0]in our experimental setup. We use the official implementation only with this necessary change. A.5. Gridworld Experimental Details We provide the details for the gridworld experiment used in Figure 2. Environment and dataset generation.The environment consists of an18× 18discrete grid. The agent’s objective is to reach a fixed goal state located at coordinates (6, 8). • Dynamics: The agent can move in four cardinal directions (up, down, left, right). Transitions that would lead out of the grid boundaries makes the agent stay in its current state. • Reward Function: A sparse reward of1is given only when the agent enters the goal state. For all other transitions, the reward is 0. The episode is terminated when the agent reaches the goal. We use a discount factor of γ = 0.9. •Offline Dataset: We collect a datasetDconsisting of60trajectories, each with a maximum length of15steps. To simulate a suboptimal, exploratory data distribution, trajectories are generated using anε-greedy sampling (ε = 0.9) from the optimal policy. Value Backup Definitions We compare three distinct backup operators to analyze the efficiency of value propagation over iteration k. 1. Single-step Value Backup The standard 1-step backup updates the value function using individual transitions(s,s ′ ) available in the dataset: V k+1 (s)← max s ′ ∈D(s) [R(s,s ′ ) + γV k (s ′ )](17) 2. Chunk-based Value Backup (h-step) The chunk-based operator utilizes a sequence of states(s t ,s t+1 ,...,s t+h )of length h. The backup is defined as: V k+1 (s t )← max chunk∈D R + γ h V k (s t+h ) (18) whereRis the discounted cumulative return P h k=0 γ k r t+k within the chunk. If the goal state is reached at stepi≤ H, the return is truncated, and no bootstrapping from V k is performed. In our experiments, we set the chunk horizon h = 4. 3. Chunk Guided Q-Learning (CGQ) The CGQ backup performs a convex combination of the 1-step and chunk-based updates, weighted by a hyperparameter w = 0.7: V k+1 ← w· V single + (1− w)· V chunk (19) 12 Chunk-Guided Q-Learning Evaluation. Gridworld is a tabular environment, where we do not have a source of bias such as function approximation error, as in deep RL. To simulate the accumulation of errors in single-step TD learning, we added Gaussian noise withσ = 0.05to each value update. We calculate the accuracy of value function with mean squared error E s∈D [((V (s)− V ∗ (s)) 2 ]. A.6. Q&As We usecube-doubleenvironment for Q&A and all ablation studies. For the critic design experiments, we use the same α,α step ,β, andτin CGQ across all variants. For ablation studies on horizon length,β, andτ, except for the target hyperparameter, all other hyperparameter values follow those of the cube-double setting reported in Table 9. 13 Chunk-Guided Q-Learning B. Hyperparameters Shared hyperparameters. Here, we report the shared hyperparameters across all methods in Table 3. Table 3. Shared hyperparameters across all methods. HyperparameterValue Learning rate0.0003 OptimizerAdam (Kingma & Ba, 2015) Gradient steps1000000 (default), 2500000 (cube-triple, cube-quadruple) Batch size256 (default), 1024 (cube-triple, Cube-quadruple) Network architectures.For the network size, we consider[512, 512, 512, 512]- and[1024, 1024, 1024, 1024]-sized MLPs for all the baselines except DEAS, using a larger architecture forcube-tripleandcube-quadrupletasks and the smaller one otherwise. For DEAS, we follow the original setup and use[512, 512, 512, 512]for the policy,[256, 256, 256, 256]for the critic, and [128, 128, 128, 128] for the value function. Hyperparameters of CGQOur main hyperparameters are BC constraint for the chunked criticα, one for the single step criticα step , distillation coefficientβand distillation expectileτ. We report those hyperparameters tuning range in Table 5 and hyperparameters for all the tasks in Table 9. Table 4. Hyperparameters of CGQ. HyperparameterValue Learning rate0.0003 OptimizerAdam (Kingma & Ba, 2015) Gradient steps1000000 (default), 2500000 (cube-triple, cube-quadruple) Batch size256 (default), 1024 (cube-triple, Cube-quadruple) Discount factor γ0.99 Critic Ensemble Size2 Clipped Q-LearningFalse Flow-steps10 Flow time sampling distributionUnif([0, 1]) Chunk size for Q φ c 10 BC coefficient α,α step Table 9 Distillation coefficient βTable 9 Distillation expectile τ0.95 (default) Table 5. Hyperparameter tuning range for CGQ. Environment BC Constraint for π θ c BC Constraint for π θ Distillation CoefficientDistillation ExpectileChunk size (α)(α step )(β)(τ )(h) Manipulation 100, 300100, 3001, 0.1, 0.010.5, 0.8, 0.95 10, 20 Navigation30, 10010, 301, 0.5, 0.1, 0.01 0.5, 0.6, 0.7, 0.8, 0.95 10 BC Constraint (α).For QC-FQL and NFQL, we adopt the default hyperparameterαfrom FQL for each domain, and tune all methods on the default task (singletask-v0) using three choices ofα∈α default /3,α default , 3α default , following (Park et al., 2025d). For DEAS, we sweep α over0.03, 0.1, 0.3, 1.0, 3.0, 10.0 following the tuning range. Chunk Sizeh.Here, we clarify the chunk sizes used to obtain the results reported in the tables. We follow the chunk size settings used in the original papers for each method. For some baselines (e.g., NFQL and QC-FQL), results are drawn from multiple prior works, which use different chunk sizes across tasks. Accordingly, we report the task-specific chunk size used for each method, including those adopted from prior work. See Table 8. 14 Chunk-Guided Q-Learning Table 6. Hyperparameter tuning range for DEAS. BC ConstraintChunk sizeIntra-chunk discountHL-Gaussian support (α)(h)(γ 1 )(s) 0.03, 0.1, 0.3, 1, 3, 10 4, 8 0.8, 0.9, 0.99, 0.999 data-centric, universal Table 7. Hyperparameter tuning range for DQC. Environment Backup QuantileDistillation ExpectileBackup HorizonPolicy Chunk Size (κ b )(κ d )(h) or (n)(h a ) cube-∗ 0.5, 0.7, 0.9, 0.93, 0.95, 0.97, 0.99 0.5, 0.85, 10, 251, 5, 25 Others0.5, 0.7, 0.90.5, 0.85, 10, 251, 5, 25 Table 8. Task-specific chunk size configurations. This table summarizes the chunk sizes used by each agent for each task, including both our experimental settings and those reported in prior work. Gray entries indicate chunk sizes corresponding to results reported in prior work. EnvironmentNFQLQC-FQLDEASDQCCGQ antmaze-large5542510 antmaze-giant5542510 humanoidmaze-medium5541010 puzzle-3x3-sparse5541020 scene-sparse554520 cube-double554520 cube-triple4442520 puzzle-4x4558510 cube-quadruple4441020 Hyperparameter search for QC-FQL. Following the tuning process of the original paper (Li et al., 2025b), we set the optimalαof FQL (Park et al., 2025d) asα default and sweep overα default /3,α default , 3α default . For the puzzle-4x4 and navigation tasks, we use the chunk size 5. See Table 9 for the value of α. Hyperparameter search for DQC.We perform an exhaustive hyperparameter sweep for DQC, evaluating 98 configura- tions forcube-∗environments and 42 configurations for the others. In particular, we tune the backup quantileκ b , distillation expectileκ d , backup horizonhand policy chunk sizeh a . We use clipped Q-learning and use 32 as the number of Best-of-N samples following the original paper. See Table 7 for the full hyperparameter sweep range. Hyperparameter search for DEAS.We perform an extensive hyperparameter sweep for DEAS, evaluating 96 configura- tions per environment. In particular, we tune the BC constraintα, chunk sizeh, and intra-chunk discountγ 1 . For selecting v min andv max in the distributional RL component, we evaluate both approaches proposed in the original paper, data-centric and universal, and select the better-performing protocol. We denote this selection protocol assin the table, wheredandu correspond to data-centric and universal, respectively. We additionally apply Q-loss normalization and clipped Q-learning following the original setting. See Table 6 for the full hyperparameter sweep range. Hyperparameter search for NFQL.We usehto 5 following Li et al. (2025b). Otherwise, we use the same strategy as in QC-FQL. 15 Chunk-Guided Q-Learning Table 9. Task-specific hyperparameters for all methods. Environment CGQDQCDEASNFQLQC-FQL (α, α step , β, τ , h)(h, h a , κ b , κ d )(α, h, γ 1 , γ 2 , s)(α, h)(α, h) antmaze-large(100, 10, 0.01, 0.7, 10)(25, 1, 0.5, 0.8)(1, 4, 0.99, 0.999,d)(30, 5)(30, 5) antmaze-giant(300, 10, 0.01, 0.8, 10)(25, 1, 0.9, 0.5)(0.3, 4, 0.99, 0.999,u)(30, 5)(10, 5) humanoidmaze-medium(100, 10, 0.5, 0.8, 10)(10, 1, 0.9, 0.5)(1, 4, 0.8, 0.999,d)(30, 5)(30, 5) puzzle-3x3-sparse(100, 100, 0.01, 0.8, 20)(10, 5, 0.7, 0.8)(3, 8, 0.9, 0.99)-- scene-sparse(300, 100, 0.1, 0.8, 20)(5, 1, 0.9, 0.5)(3, 4, 0.99, 0.999,u)-- cube-double(100, 100, 1, 0.8, 20)(5, 1, 0.99, 0.8)--- cube-triple(300, 300, 0.01, 0.8, 20)(25, 5, 0.7, 0.8)--- puzzle-4x4(300, 300, 0.1, 0.5, 10)(5, 1, 0.7, 0.5)(1, 4, 0.999, 0.999,u)(1000, 5)(3000, 5) cube-quadruple(300, 300, 0.01, 0.95, 20)(10, 5, 0.9, 0.8)--- 16 Chunk-Guided Q-Learning C. Complete Numerical Results For completeness, we provide the complete per-task results for OGBench experiments in Table 10 (corresponding to Table 1). The results are averaged over4seeds and we report the standard deviations for each tasks. We highlight the numbers that are above or equal to 95% of the best performance. Table 10. Complete results for OGBench experiments. We present the full results on 45 tasks. The results are averaged over 4 seeds. Task Type Task Category FQL NFQL QC-FQL DEAS DQC CGQ Manipulation scene-task1 69 22 99 99±0 94±4 90±2 scene-task2 51 4 88 98±3 93±3 79±3 scene-task3 68 1 98 84±3 62±7 85±4 scene-task4 75 50 92 99±1 84±4 80±6 scene-task5 25 14 41 87±5 73±7 25±5 cube-double-task1 63 33 66 76 51±7 81±9 cube-double-task2 36 8 43 51 28±6 77±3 cube-double-task3 22 1 38 47 22±3 78±4 cube-double-task4 9 0 11 8 4±1 9±1 cube-double-task5 12 13 37 57 48±5 74±7 puzzle-3x3-task1 99 100 100 100±0 100±0 99±1 puzzle-3x3-task2 99 100 86 100±0 100±0 100±0 puzzle-3x3-task3 100 100 50 100±1 100±0 97±2 puzzle-3x3-task4 100 100 75 97±5 100±0 98±1 puzzle-3x3-task5 100 91 4 100±0 100±0 98±2 puzzle-4x4-task1 34 35±5 48±3 87±6 13±6 65±3 puzzle-4x4-task2 16 21±6 18±2 9±10 11±4 5±1 puzzle-4x4-task3 18 26±3 39±4 60±7 6±1 51±11 puzzle-4x4-task4 11 20±4 18±1 30±6 4±3 18±5 puzzle-4x4-task5 7 13±1 16±3 10±2 3±1 2±0 cube-triple-task1 31 17 100 98 8±3 100±0 cube-triple-task2 9 91 92 95 0±0 100±0 cube-triple-task3 12 0 92 88 0±0 96±0 cube-triple-task4 0 0 59 45 0±0 80±6 cube-triple-task5 2 0 74 87 0±0 85±9 cube-quadruple-task1 79 70 79 92 90±14 100±0 cube-quadruple-task2 0 97 63 100 62±29 100±0 cube-quadruple-task3 6 1 33 62 53±31 89±9 cube-quadruple-task4 0 13 38 31 34±17 72±3 cube-quadruple-task5 0 0 12 35 40±32 24±12 Navigation antmaze-large-task1 80 38±15 20±9 74±3 72±6 88±2 antmaze-large-task2 57 39±2 0±0 46±9 61±7 74±6 antmaze-large-task3 93 81±5 49±8 80±4 74±4 92±3 antmaze-large-task4 80 38±11 6±6 54±9 73±16 26±20 antmaze-large-task5 83 38±22 23±15 82±3 74±6 55±33 antmaze-giant-task1 4 0±0 0±0 0±0 0±1 0±0 antmaze-giant-task2 9 1±1 0±0 28±3 41±4 0±1 antmaze-giant-task3 0 0±0 0±0 1±1 1±1 0±0 antmaze-giant-task4 14 0±0 0±0 8±4 5±2 0±0 antmaze-giant-task5 16 8±3 0±0 2±3 2±1 17±8 humanoidmaze-medium-task1 19 3±3 0±0 22±4 87±4 0±0 humanoidmaze-medium-task2 94 26±11 1±1 41±6 94±3 80±3 humanoidmaze-medium-task3 74 49±8 0±0 59±7 94±2 26±48 humanoidmaze-medium-task4 3 4±1 0±0 7±2 90±3 0±1 humanoidmaze-medium-task5 97 32±3 17±5 56±6 98±0 62±26 17 Chunk-Guided Q-Learning D. Training curves We provide the training curve of CGQ for experiments in OGBench manipulation and locomotion environments in Figure 7 and Figure 8 (corresponding to Table 1). We plot the mean and the standard deviation (across 4 seeds) by covering [mean - std, mean + std] area with a lighter color. 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 Success rate (%) cube-double-task1 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 cube-double-task2 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 cube-double-task3 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 cube-double-task4 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 cube-double-task5 0.00.51.01.52.02.5 Training steps(×10 6 ) 0 20 40 60 80 100 Success rate (%) cube-quadruple-task1 0.00.51.01.52.02.5 Training steps(×10 6 ) 0 20 40 60 80 100 cube-quadruple-task2 0.00.51.01.52.02.5 Training steps(×10 6 ) 0 20 40 60 80 100 cube-quadruple-task3 0.00.51.01.52.02.5 Training steps(×10 6 ) 0 20 40 60 80 100 cube-quadruple-task4 0.00.51.01.52.02.5 Training steps(×10 6 ) 0 20 40 60 80 100 cube-quadruple-task5 0.00.51.01.52.02.5 Training steps(×10 6 ) 0 20 40 60 80 100 Success rate (%) cube-triple-task1 0.00.51.01.52.02.5 Training steps(×10 6 ) 0 20 40 60 80 100 cube-triple-task2 0.00.51.01.52.02.5 Training steps(×10 6 ) 0 20 40 60 80 100 cube-triple-task3 0.00.51.01.52.02.5 Training steps(×10 6 ) 0 20 40 60 80 100 cube-triple-task4 0.00.51.01.52.02.5 Training steps(×10 6 ) 0 20 40 60 80 100 cube-triple-task5 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 Success rate (%) puzzle-3x3-task1 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 puzzle-3x3-task2 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 puzzle-3x3-task3 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 puzzle-3x3-task4 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 puzzle-3x3-task5 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 Success rate (%) puzzle-4x4-task1 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 puzzle-4x4-task2 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 puzzle-4x4-task3 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 puzzle-4x4-task4 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 puzzle-4x4-task5 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 Success rate (%) scene-task1 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 scene-task2 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 scene-task3 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 scene-task4 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 scene-task5 Figure 7. Training curve of CGQ in OGBench manipulation environments. We report the success rate for 50 evaluation episodes across 4 seeds (total 200 episodes). Shaded region represents the [mean - std, mean + std]. 18 Chunk-Guided Q-Learning 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 Success rate (%) ant-giant-task1 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 ant-giant-task2 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 ant-giant-task3 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 ant-giant-task4 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 ant-giant-task5 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 Success rate (%) ant-large-task1 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 ant-large-task2 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 ant-large-task3 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 ant-large-task4 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 ant-large-task5 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 Success rate (%) humanoid-medium-task1 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 humanoid-medium-task2 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 humanoid-medium-task3 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 humanoid-medium-task4 0.00.51.0 Training steps(×10 6 ) 0 20 40 60 80 100 humanoid-medium-task5 Figure 8. Training curve of CGQ in OGBench locomotion environments. We report the success rate for 50 evaluation episodes across 4 seeds (total 200 episodes). Shaded region represents the [mean - std, mean + std]. 19 Chunk-Guided Q-Learning E. Proofs Here, we prove the main theorems of the paper. Throughout the proof, we consider the Hilbert spaceQ = L 2 (S×A,∥·∥) of state-action value functions, equipped with the weighted norm∥Q∥ = (E (s,a)∼μ [Q(s,a) 2 ]) 1/2 , whereμdenotes the state-action distribution of an offline datasetD. E.1. Proof of Theorem 4.1 Theorem E.1 (Bias accumulation of one-step TD learning). Let the operatorTbeγ-Lipschitz operator, i.e.,∥T Q 1 − T Q 2 ∥≤ γ∥Q 1 − Q 2 ∥, with γ < 1. Consider a stochastic iterative process Q k+1 = b T Q k , where b T Q k =T Q k + ε k ,(20) whichε k is a sequence of independent, zero-mean noise vectors withE ∥ε k ∥ 2 ≤ σ 2 . LetQ ∗ be the unique fixed point ofT (i.e., Q ∗ =T Q ∗ ). Then, the asymptotic expected squared error satisfies: lim sup k→∞ E ∥Q k − Q ∗ ∥ 2 ≤ σ 2 1− γ 2 (21) Proof. By substituting the update rule and using the definition of Q ∗ (Q ∗ =T Q ∗ ), Q k+1 − Q ∗ = (T Q k + ε k )−T Q ∗ = (T Q k −T Q ∗ ) + ε k .(22) Taking the expectation of squared L 2 norms: E[∥Q k+1 − Q ∗ ∥ 2 ] = E ∥(T Q k −T Q ∗ ) + ε k ∥ 2 ,(23) which can be expanded as E[∥Q k+1 − Q ∗ ∥ 2 ] = E[∥T Q k −T Q ∗ ∥ 2 ] + E[∥ε k ∥ 2 ] + 2E [⟨T Q k −T Q ∗ ,ε k ⟩].(24) Since ε k have zero mean and indepedent from Q k , we can remove the cross-term: E[∥Q k+1 − Q ∗ ∥ 2 ] = E[∥T Q k −T Q ∗ ∥ 2 ] + E[∥ε k ∥ 2 ].(25) Since E ∥ε∥ 2 ≤ σ 2 , E[∥Q k+1 − Q ∗ ∥ 2 ]≤ E[∥T Q k −T Q ∗ ∥ 2 ] + σ 2 .(26) and by the Lipschitz property ofT , E[∥Q k+1 − Q ∗ ∥ 2 ]≤ γ 2 E[∥Q k − Q ∗ ∥ 2 ] + σ 2 .(27) By rolling out the inequality, we have E[∥Q k+1 − Q ∗ ∥ 2 ]≤ γ 2k+2 E[∥Q 0 − Q ∗ ∥ 2 ] + k X i=0 γ 2i σ 2 .(28) Applying the limit k →∞ to each side of the equation concludes the proof: lim k→∞ E ∥Q k − Q ∗ ∥ 2 ≤ σ 2 1− γ 2 (29) 20 Chunk-Guided Q-Learning On top of that, with the assumption that the curvature ofTis bounded, we can derive the bound for the difference between the expectation of the result of the stochastic process and Q ∗ . Lemma E.2. Consider a stochastic iterative process Q k+1 = b T Q k , where b T Q k =T Q k + ε k ,(30) and Q ∗ be the fixed point ofT . Suppose thatT has a bounded curvature; i.e., ∥T (Q 1 )−T (Q 2 )− DT (Q 2 )(Q 1 − Q 2 )∥≤ L 2 ∥Q 1 − Q 2 ∥ 2 ,(31) whereDTis a Fr ́ echet derivative ofT. Then, the difference between the expectation of the process and theQ ∗ is bounded as: lim sup k→∞ ∥E [Q k ]− Q ∗ ∥≤ L 2 · σ 2 (1− γ 2 )(1− γ) ,(32) Proof. Since ε k has zero mean, E [Q k+1 ]− Q ∗ = E [T Q k ]− Q ∗ .(33) Since Q ∗ is not a random variable, E [Q k+1 ]− Q ∗ = E [T Q k − Q ∗ ].(34) By the first-order taylor expansion ofT in Q ∗ : T (Q) =T (Q ∗ ) + DT (Q ∗ )(Q− Q ∗ ) + R(Q− Q ∗ ), E [Q k+1 ]− Q ∗ = E [DT (Q ∗ )(Q k − Q ∗ ) + R(Q k − Q ∗ )].(35) Split the expectation and taking the norm, ∥E [Q k+1 ]− Q ∗ ∥ =∥E [DT (Q ∗ )(Q k − Q ∗ )] + E [R(Q k − Q ∗ )]∥.(36) By triangle inequality, ∥E [Q k+1 ]− Q ∗ ∥≤∥E [DT (Q ∗ )(Q k − Q ∗ )]∥ +∥E [R(Q k − Q ∗ )]∥.(37) Since DT (Q ∗ ) is a linear operator, ∥E [Q k+1 ]− Q ∗ ∥≤∥DT (Q ∗ )(E [Q k ]− Q ∗ )∥ +∥E [R(Q k − Q ∗ )]∥.(38) SinceT is a γ-Lipschitz operator,∥DT (Q ∗ )∥≤ γ, thus ∥E [Q k+1 ]− Q ∗ ∥≤ γ∥E [Q k ]− Q ∗ ∥ +∥E [R(Q k − Q ∗ )]∥.(39) AsT has a curvature bounded by L,∥R(Q k − Q ∗ )∥≤ L 2 ∥Q k − Q ∗ ∥ 2 . Thus, ∥E [Q k+1 ]− Q ∗ ∥≤ γ∥E [Q k ]− Q ∗ ∥ + L 2 E ∥Q k − Q ∗ ∥ 2 .(40) Taking the limit: lim sup k→∞ ∥E [Q k+1 ]− Q ∗ ∥≤ lim sup k→∞ γ∥E [Q k ]− Q ∗ ∥ + lim sup k→∞ L 2 E ∥Q k − Q ∗ ∥ 2 .(41) By Theorem E.1, we have lim sup k→∞ ∥E [Q k+1 ]− Q ∗ ∥≤ lim sup k→∞ γ∥E [Q k ]− Q ∗ ∥ + L 2 · σ 2 1− γ 2 .(42) 21 Chunk-Guided Q-Learning Since k →∞, we can have lim sup k→∞ (1− γ)∥E [Q k ]− Q ∗ ∥≤ L 2 · σ 2 1− γ 2 ,(43) which concludes the proof: lim sup k→∞ ∥E [Q k ]− Q ∗ ∥≤ L 2 · σ 2 (1− γ 2 )(1− γ) ,(44) E.2. Proof of Theorem 4.2 Lemma E.3. LetFbe a normed vector space and letT : F → Fbeγ-Lipschitz operator in theL 2 norm, i.e., ∥T Q 1 −T Q 2 ∥≤ γ∥Q 1 − Q 2 ∥, withγ < 1. Let its fixed point ofTbex ∗ (T x ∗ = x ∗ ). Then for anyx∈F, we have the following inequality: ∥x− x ∗ ∥≤ 1 1− γ ∥T x− x∥.(45) Proof. We write x− x ∗ = x−T x +T x−T x ∗ .(46) Taking norms and using the triangle inequality and the linearlity ofT , ∥x− x ∗ ∥≤∥x−T x∥ +∥T x−T x ∗ ∥.(47) Since T is a contraction with factor γ,∥T x−T x ∗ ∥≤ γ∥x− x ∗ ∥ holds, thus ∥x− x ∗ ∥≤∥x−T x∥ + γ∥x− x ∗ ∥.(48) By rearranginging the inequality, (1− γ)∥x− x ∗ ∥≤∥x−T x∥,(49) which proves the theorem: ∥x− x ∗ ∥≤ 1 1− γ ∥T x− x∥.(50) Lemma E.4. LetT β be the regularized operator for someQ c ∈Q:T β Q = 1 1+β T Q + β 1+β Q c . ThenT β is Lipschitz with factor of γ/(1 + β), i.e., ∥T β Q 1 −T β Q 2 ∥≤ γ 1 + β ∥Q 1 − Q 2 ∥.(51) Proof. By definition, ∥T β Q 1 −T β Q 2 ∥ = 1 1 + β T Q 1 − 1 1 + β T Q 2 .(52) SinceT is a contraction operator with factor γ, ∥T β Q 1 −T β Q 2 ∥≤ γ 1 + β ∥Q 1 − Q 2 ∥,(53) which concludes the proof. 22 Chunk-Guided Q-Learning Lemma E.5. Let Q ∗ β be the fixed point ofT β . Then ∥Q ∗ − Q ∗ β ∥≤ β 1− γ + β ∥Q ∗ − Q c ∥.(54) Proof. From Lemma E.3 and Lemma E.4, we have ∥Q ∗ − Q ∗ β ∥≤ 1 1− γ/(1 + β) ∥T β Q ∗ − Q ∗ ∥.(55) By definition ofT β , we have: ∥Q ∗ − Q ∗ β ∥≤ 1 1− γ/(1 + β) ∥Q ∗ − 1 β + 1 T Q ∗ − β β + 1 Q c )∥.(56) SinceT Q ∗ = Q ∗ , ∥Q ∗ − Q ∗ β ∥≤ 1 1− γ/(1 + β) ∥ β β + 1 (Q ∗ − Q c )∥,(57) concluding the proof: ∥Q ∗ − Q ∗ β ∥≤ β 1 + β− γ ∥Q ∗ − Q c ∥.(58) Lemma E.6. Consider a stochastic iterative process b Q k+1 β = b T β b Q k β , where b T β b Q k β = 1 1 + β b T b Q k β + β 1 + β Q c ,(59) Suppose thatThas a curvature bounded byL. Then, the difference between the asymptotic expectation of the process and the Q ∗ β satisfies: lim sup k→∞ ∥E h b Q k β i − Q ∗ β ∥≤ L 2 · (1 + β)σ 2 ((1 + β) 2 − γ 2 )(1 + β− γ) ,(60) Proof. We can repeat the proof of Lemma E.2 with contraction factor ofγ/(1 + β)and noise level ofσ/(1 + β), leading to the bound: lim sup k→∞ ∥E h b Q k β i − Q ∗ β ∥≤ L 2 · (1 + β)σ 2 ((1 + β) 2 − γ 2 )(1 + β− γ) .(61) By combining two results, we show the main theorem of the paper: Theorem E.7 (Improved bound via CGQ regularization). LetQ ∗ be the fixed point ofT, and b T β be a stochastic iterative process b T β Q = 1 1+β b T Q + β 1+β Q c . Then, the asymptotic expected squared error satisfies: lim sup k→∞ E h ∥ b Q k β − Q ∗ ∥ 2 i ≤ σ 2 (1 + β) 2 − γ 2 + β 2 ∥Q ∗ − Q c ∥ 2 (1 + β− γ) 2 + L(1 + β)β∥Q ∗ − Q c ∥σ 2 ((1 + β) 2 − γ 2 )(1 + β− γ) 2 .(62) Proof. We can write lim sup k→∞ E h ∥ b Q k β − Q ∗ ∥ 2 i = lim sup k→∞ E h ∥( b Q k β − Q ∗ β ) + (Q ∗ β − Q ∗ )∥ 2 i .(63) 23 Chunk-Guided Q-Learning Expanding the formula gives lim sup k→∞ E h ∥ b Q k β − Q ∗ ∥ 2 i = lim sup k→∞ E h ∥ b Q k β − Q ∗ β ∥ 2 +∥Q ∗ β − Q ∗ ∥ 2 + 2⟨ b Q k β − Q ∗ β ,Q ∗ β − Q ∗ ⟩ i .(64) Since∥Q ∗ β − Q ∗ ∥ 2 is not a random variable, lim sup k→∞ E h ∥ b Q k β − Q ∗ ∥ 2 i =∥Q ∗ β − Q ∗ ∥ 2 + lim sup k→∞ E h ∥ b Q k β − Q ∗ β ∥ 2 i + 2⟨E h b Q k β − Q ∗ β i ,Q ∗ β − Q ∗ ⟩.(65) By Cauchy-Schwarz inequality, lim sup k→∞ E h ∥ b Q k β − Q ∗ ∥ 2 i ≤∥Q ∗ β − Q ∗ ∥ 2 + lim sup k→∞ E h ∥ b Q k β − Q ∗ β ∥ 2 i + 2∥E h b Q k β i − Q ∗ β ∥Q ∗ β − Q ∗ ∥.(66) From Lemma E.5, Lemma E.6, Lemma E.2, we have lim sup k→∞ E h ∥ b Q k β − Q ∗ ∥ 2 i ≤ σ 2 (1 + β) 2 − γ 2 + β 2 ∥Q ∗ − Q c ∥ 2 (1 + β− γ) 2 + L(1 + β)β∥Q ∗ − Q c ∥σ 2 ((1 + β) 2 − γ 2 )(1 + β− γ) 2 (67) concluding the proof. Here, we are interested in when this bound can be strictly better than pure single-step TD learning (β = 0) or action-chunked TD learning (β =∞). We can derive the condition for this with simple algebra. Lemma E.8. Let f (β) be the bound derived from Theorem E.7, f (β) = σ 2 (1 + β) 2 − γ 2 + β 2 ∥Q ∗ − Q c ∥ 2 (1 + β− γ) 2 + L(1 + β)β∥Q ∗ − Q c ∥σ 2 ((1 + β) 2 − γ 2 )(1 + β− γ) 2 .(68) Then lim β→∞ f ′ (β) = 0 + . Proof. We analyze the dominating term for each terms separately. Term 1: f 1 (β) = σ 2 (1 + β) 2 − γ 2 .(69) The derivative is f ′ 1 (β) =− 2σ 2 (1 + β) ((1 + β) 2 − γ 2 ) 2 .(70) thus f ′ 1 (β)≃− 2σ 2 β 3 .(71) Term 2: f 2 (β) = β 2 ∥Q ∗ − Q c ∥ 2 (1 + β− γ) 2 .(72) Its derivative is f ′ 2 (β) = 2β∥Q ∗ − Q c ∥ 2 (1− γ) (1 + β− γ) 3 ,(73) thus f ′ 2 (β)≃ 2(1− γ)∥Q ∗ − Q c ∥ 2 β 2 .(74) 24 Chunk-Guided Q-Learning Term 3: f 3 (β) = L(1 + β)β∥Q ∗ − Q c ∥σ 2 ((1 + β) 2 − γ 2 )(1 + β− γ) 2 .(75) Atβ = 0, the numerator is zero, and its derivative is(1 + 2β)∥Q ∗ − Q c ∥σ 2 , and the denominator evaluates asymptotically to β 4 . Therefore by the quotient rule, f ′ 3 (β)≃ L∥Q ∗ − Q c ∥σ 2 (2β· β 4 − 4β 3 · β 2 ) (β 4 ) 2 = L∥Q ∗ − Q c ∥σ 2 2β 3 .(76) Thus, the limit is dominated by the 1/β 2 term, proving lim β→∞ f ′ (β) = 0 + .(77) Corollary E.9. Let f (β) be the bound derived from Theorem E.7, f (β) = σ 2 (1 + β) 2 − γ 2 + β 2 ∥Q ∗ − Q c ∥ 2 (1 + β− γ) 2 + L(1 + β)β∥Q ∗ − Q c ∥σ 2 ((1 + β) 2 − γ 2 )(1 + β− γ) 2 .(78) WhenL∥Q c −Q ∗ ∥ < 2 1−γ 1+γ ,f (β)minimized at some0 < β ∗ <∞, and is strictly smaller than both σ 2 1−γ 2 and∥Q ∗ −Q c ∥ 2 . Proof. We show that f ′ (0) < 0. We analyze each term separately. Term 1: f 1 (β) = σ 2 (1 + β) 2 − γ 2 .(79) The derivative is f ′ 1 (β) =− 2σ 2 (1 + β) ((1 + β) 2 − γ 2 ) 2 .(80) thus f ′ 1 (0) =− 2σ 2 (1− γ 2 ) 2 .(81) Term 2: f 2 (β) = β 2 ∥Q ∗ − Q c ∥ 2 (1 + β− γ) 2 .(82) Its derivative is f ′ 2 (β) = 2β∥Q ∗ − Q c ∥ 2 (1− γ) (1 + β− γ) 3 ,(83) thus f ′ 2 (0) = 0.(84) Term 3: f 3 (β) = L(1 + β)β∥Q ∗ − Q c ∥σ 2 ((1 + β) 2 − γ 2 )(1 + β− γ) 2 .(85) Atβ = 0, the numerator is zero, and its derivative isL(1 + 2β)∥Q ∗ − Q c ∥σ 2 = L∥Q ∗ − Q c ∥σ 2 , and the denominator evaluates to (1− γ 2 )(1− γ) 2 . Therefore by the quotient rule, 25 Chunk-Guided Q-Learning f ′ 3 (0) = L∥Q ∗ − Q c ∥σ 2 (1− γ 2 )(1− γ) 2 .(86) Thus, f ′ (0) = f ′ 1 (0) + f ′ 2 (0) + f ′ 3 (0) =− 2σ 2 (1− γ 2 ) 2 + 0 + L∥Q ∗ − Q c ∥σ 2 (1− γ) 2 (1− γ 2 ) .(87) If L∥Q ∗ − Q c ∥ < 2(1−γ) (1+γ) , then f ′ (0) <− 2σ 2 (1− γ 2 ) 2 + 2(1− γ)/(1 + γ)σ 2 (1− γ) 2 (1− γ) 2 = 0.(88) Sincef ′ is a continuous function andlim β→∞ f ′ (β) = 0 + andf ′ (0) < 0, we havef ′ (β ∗ ) = 0in0 < β ∗ <∞. Moreover, among those β ∗ , we have at least one β ∗ that is strictly smaller than f ′ (β ∗ ), as f ′ (0) < 0 and f ′ (∞) = 0 + . 26