Paper deep dive
Mixing Regularization for Average-Reward $Q$-Learning and $Q$-Planning
Julien Roger, Sam Bodmer, Jian Qian, Benjamin Riviere
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The average-reward setting for reinforcement learning is challenging because its Bellman operator is known to contract in span only when the underlying Markov chain is mixing. Whereas existing work typically assumes a contraction property, e.g., through environment mixing, we consider the setting where we are given a reward-agnostic $m$-step mixing policy, and we control the contraction and exploration properties of the learned policy by regularizing toward the mixing policy. We provide finite-time analysis of $Q$-learning algorithms that can be instantiated with multiple choices of regularization, with guarantees measured with respect to the regularized fixed point. Our framework exposes novel regularizer-dependent objects that control contraction, exploration, and operator bias. In the one-step setting, our rates match the best known rates for $Q$-iteration and synchronous $Q$-learning, and for $m$-step mixing we derive new rates. We also study $Q$-planning variants, and show that planning acts as time-localized learning, where the planning horizon $H$ trades off the bias from the boundary value against the sample complexity. Finally, we present numerical studies validating the analysis: (i) knowledge of a strong mixing policy improves convergence and exploration, and its advantage increases with bottleneck severity and, (ii) our theory predicts the convergence and bias rankings of regularizers from the $\alpha$-divergence family.
Tags
Links
- Source: https://arxiv.org/abs/2610.14988v1
- Canonical: https://arxiv.org/abs/2610.14988v1
Trouble viewing inline? Open PDF directly →
Full Text
348,296 characters extracted from source content.
Expand or collapse full text
Mixing Regularization for Average-Reward Q-Learning and Q-Planning Julien Roger Affiliation: École polytechnique Email: julien.roger@polytechnique.org Sam Bodmer Affiliation: New York University Email: sgb329@nyu.edu Jian Qian Affiliation: The University of Hong Kong Email: jianqian@hku.hk Benjamin Riviere Affiliation: New York University Email: bpr3966@nyu.edu Abstract The average-reward setting for reinforcement learning is challenging because its Bellman operator is known to contract in span only when the underlying Markov chain is mixing. Whereas existing work typically assumes a contraction property, e.g., through environment mixing, we consider the setting where we are given a reward-agnostic m-step mixing policy, and we control the contraction and exploration properties of the learned policy by regularizing toward the mixing policy. We provide finite-time analysis of Q-learning algorithms that can be instantiated with multiple choices of regularization, with guarantees measured with respect to the regularized fixed point. Our framework exposes novel regularizer-dependent objects that control contraction, exploration, and operator bias. In the one-step setting, our rates match the best known rates for Q-iteration and synchronous Q-learning, and for m-step mixing we derive new rates. We also study Q-planning variants, and show that planning acts as time-localized learning, where the planning horizon H trades off the bias from the boundary value against the sample complexity. Finally, we present numerical studies validating the analysis: (i) knowledge of a strong mixing policy improves convergence and exploration, and its advantage increases with bottleneck severity and, (i) our theory predicts the convergence and bias rankings of regularizers from the α-divergence family. 1 Introduction The average-reward setting of reinforcement learning is challenging because, without a discount factor, the contraction of the Bellman operator is solely determined by the closed-loop dynamics, formed by the transitions and the policy. The classical requirement for learning algorithms is the existence of a policy that mixes the induced Markov chain, which holds when the Markov decision process (MDP) is communicating and its transition graph is aperiodic. In that case, every full-support policy mixes, albeit at potentially very different rates. However, the design of algorithms that exploit knowledge of a strongly mixing policy to improve the constants of their own convergence and exploration remains relatively understudied. This setting is natural in domains like robotics and fine-tuning, where new tasks are learned starting from an existing policy. Although regularization toward the mixing policy is a natural approach, its effects on average-reward learning are not understood, raising the question: How does regularization toward a mixing policy control contraction, exploration, bias and, ultimately, sample complexity of average-reward Q-learning? How does this analysis inform regularization design? We give the first finite-time analysis of average-reward Q-learning in which regularization toward a mixing policy provides both the contraction of the Bellman operator and the exploration needed to learn. Specifically, our main contributions are: • A framework for mixing regularization. Given a reward-agnostic policy μ that mixes in m steps with parameter βm _m, we identify regularizer-dependent quantities that drive contraction, exploration, and bias: the domination profile γΩ _ controls contraction, at rate β~=βmγ¯m−1 β= _m γ^m-1, and exploration, guaranteeing coverage with probability βmγ¯mηmin _m γ^m _ every m steps where ηmin _ is the smallest mass of the minorizing measure. The regularization modulus controls the operator bias with a term scaling with m/β~m/ β, thereby exposing a trade-off between faster contraction and larger bias. Unlike existing methods, which assume contraction conditions like uniform mixing or reachability, our assumption exploits knowledge from a single user-chosen policy. • Sample complexity of regularized Q-learning. We analyze algorithms that isolate, in turn, contraction, statistical error, exploration and localization. Regularized Q-iteration converges linearly, with O(||2||mβ~−1log(1/ε))O(|S|^2|A|\,m\, β^-1 (1/ )) model queries; synchronous regularized Q-learning reaches ε -accuracy with high probability after O~(||||m4w¯2β~−3ε−2) O(|S||A|\,m^4 w^2 β^-3 ^-2) samples where w¯ w bounds the noise; epoch regularized Q-learning, which learns from a single trajectory of its own greedy policies, needs O~(m4w¯2β~−4(γ¯ηmin)−1ε−2) O(m^4 w^2 β^-4( γ _ )^-1 ^-2) transitions; regularized Q-planning, which restricts the learning process to the state-action pairs within H steps of the reset distribution, needs O~((h1+m/β~)4w¯2q−1ε−2) O ((h_1+m/ β)^4 w^2q^-1 ^-2 ) transitions where h1h_1 is the depth after which the states reachable from the reset distribution no longer change and q is the minimum visitation probability across depth. In the last three cases, we match the optimal ε−2 ^-2 dependence. Although direct comparison to state-of-the-art analysis is challenging because there is not a standard setting, we achieve comparable rates, summarized in Table 2. • Empirical validation. In our first experiment, we empirically demonstrate that knowledge of a strong mixing policy can improve convergence and exploration properties, and its advantage increases with bottleneck severity. In our second experiment, we empirically demonstrate that our theory predicts the convergence and bias rankings of regularizers from the α-divergence family. 2 Background on Regularized Average-Reward MDPs 2.1 Average-Reward MDPs We consider an infinite-horizon average-reward MDP ℳ:=(,,p,r)M:=(S,A,p,r), where S is the state space, A the action space, p:×→[0,1]p:S×A×S→[0,1] the transition probability and r:×→ℝr:S×A the deterministic reward function. We assume that both S and A are finite. An action-value function is any Q∈ℝ×Q ^S×A. A policy π maps each state s to an action distribution: π:→Δ()π:S→ (A), inducing the state-action kernel Kπ(s′,a′|s,a):=p(s′|s,a)π(a′|s′)K_π(s ,a |s,a):=p(s |s,a)π(a |s ) and the m-step state-action kernel Kπm(s′,a′|s,a):=∑s~,a~p(s~|s,a)π(a~|s~)Kπm−1(s′,a′|s~,a~)K^m_π(s ,a |s,a):= _ s, ap( s|s,a)\,π( a| s)\,K^m-1_π(s ,a | s, a) with Kπ0:=I×K_π^0:=I_S×A. The Bellman (optimality) operator is (Q)(s,a):=r(s,a)+s′∼p(⋅|s,a)[maxa′Q(s′,a′)]T(Q)(s,a):=r(s,a)+E_s p(·|s,a)[ _a Q(s ,a )]. The gain of a policy is ρπ(s):=lim infN→∞1Nπ[∑n=0N−1r(sn,an)|s0=s]ρ^π(s):= _N→∞ 1NE_π [ _n=0^N-1r(s_n,a_n)\, |\,s_0=s ], and the optimal gain is ρ∗(s):=supπρπ(s)ρ^*(s):= _πρ^π(s). Without discounting, T is not a contraction, and the relevant notion of fixed point is the optimality equation (Q∗)=Q∗+ρ∗T(Q^*)=Q^*+ρ^*1: for communicating MDPs, ρ∗ρ^* is constant and this equation has solutions (Schweitzer and Federgruen, 1978). Because the optimality equation is translation invariant, it is standard to use the span seminorm: Definition 1 (Span seminorms). For a finite set X and U∈ℝU ^X, the span seminorm is ‖U‖,sp:=maxx∈U(x)−minx∈U(x)\|U\|_X,sp:= _x U(x)- _x U(x). This is used on ℝR^S and ℝR^A, and for ℝ×R^S×A we write ‖Q‖sp:=‖Q‖×,sp\|Q\|_sp:=\|Q\|_S×A,sp. We also define the action-span seminorm ‖Q‖asp:=maxs∈‖Q(s,⋅)‖,sp\|Q\|_asp:= _s \|Q(s,·)\|_A,sp. Both vanish on constants, ∥⋅∥asp\|·\|_asp vanishes on every Q that depends only on the state, and ∥⋅∥asp≤∥⋅∥sp\|·\|_asp≤\|·\|_sp. 2.2 Regularized MDPs We use the regularized MDP framework of Geist et al. (2019). Let τ>0τ>0 be a temperature and, for each s∈s , let Ωs:Δ()→ℝ∪+∞ _s: (A) ∪\+∞\ be a strongly convex function. With the shorthands Qs(a):=Q(s,a)Q_s(a):=Q(s,a) and πs(a):=π(a|s) _s(a):=π(a|s), the regularized value function Ω∗(Q)∈ℝ ^*(Q) ^S and the greedy policy πΩ,Q:→Δ() _ ,Q:S→ (A) are Ω∗(Q)(s):=maxπs∈Δ()⟨πs,Qs⟩−τΩs(πs), ^*(Q)(s):= _ _s∈ (A) \ _s,Q_s -τ\, _s( _s) \, πΩ,Q(⋅|s):=argmaxπs∈Δ()⟨πs,Qs⟩−τΩs(πs), _ ,Q(·|s):= _ _s∈ (A) \ _s,Q_s -τ\, _s( _s) \, so that Ω∗(⋅)(s) ^*(·)(s) is the convex conjugate of τΩsτ _s, and the regularized Bellman operator is Ω(Q)(s,a):=r(s,a)+s′∼p(⋅|s,a)[Ω∗(Q)(s′)]T_ (Q)(s,a):=r(s,a)+E_s p(·|s,a)[ ^*(Q)(s )]. Under Assumption 2 below, the maximizing policy is unique and equals ∇Ω∗(Q)(s)∇ ^*(Q)(s). Examples include the α-divergences to a policy μ, such as the forward and reverse KL. 3 Mixing-Regularized MDPs We state our assumptions, define the objects that govern mixing regularization, and derive their effect on contraction, exploration and bias (Lemma 1); proofs are deferred to Subsection G.1. 3.1 Assumptions Assumption 1 (The reference policy μ is m-step Doeblin mixing). There exists a stationary full-support policy μ:→Δ()μ:S→ (A), an integer m≥1m≥ 1, a parameter βm∈(0,1] _m∈(0,1], and a full-support distribution ν∈Δ()ν∈ (S) such that, ∀(s,a,s′)∀(s,a,s ), ∑a′Kμm(s′,a′|s,a)≥βmν(s′) _a K^m_μ(s ,a |s,a)≥ _mν(s ). Furthermore, let η(s,a):=ν(s)μ(a|s)η(s,a):=ν(s)μ(a|s) and ηmin:=min(s,a)η(s,a) _ := _(s,a)η(s,a). The m-step Doeblin condition implies that the MDP is communicating so ρ∗ρ^* is constant and (Q∗)=Q∗+ρ∗T(Q^*)=Q^*+ρ^*1 has solutions. Assumption 2 (Convex regularity of Ω ). The temperature τ is positive and, for every s∈s , Ωs _s is strongly convex, closed and proper.11 1 Strict convexity suffices for all our results; strong convexity only provides the Lipschitz constant of the greedy map in Lemma F.1.(i) (Remark F.1). Assumption 3 (Admissibility of the mixing policy). For every s∈s we have μs∈dom(Ωs) _s ( _s), the effective domain of Ωs _s. 3.2 Mixing Properties Definition 2 (Mixing-regularization objects). Let Ω satisfy Assumption 2, let μ be the mixing policy of Assumption 1, and let R≥0R≥ 0. (i) Radius: for Q,Q′∈ℝ×Q,Q ^S×A, R[Q,Q′]:=max‖Q‖asp,‖Q′‖aspR[Q,Q ]:= \\|Q\|_asp,\|Q \|_asp\ is the largest action span on the segment [Q′,Q][Q ,Q] (by convexity of ∥⋅∥asp\|·\|_asp). (i) Domination profile: γΩ(R):=infπΩ,Q(a|s)/μ(a|s)∣(s,a)∈×,∥Q∥asp≤R _ (R):= \ _ ,Q(a|s)/μ(a|s) (s,a) ×A,\ \|Q\|_asp≤ R \ is the largest factor such that πΩ,Q≥γΩ(R)μ _ ,Q≥ _ (R)\,μ for every Q of action span at most R. It lies in [0,1][0,1] and is non-increasing in R. (i) Regularization modulus: ΛΩ(R):=sup‖gQ‖,sp∣‖Q‖asp≤R _ (R):= \\|g_Q\|_S,sp \|Q\|_asp≤ R \, where gQ(s):=Ω∗(Q)(s)−maxaQ(s,a)g_Q(s):= ^*(Q)(s)- _aQ(s,a) is the regularization gap. It is finite and non-decreasing in R. Our finite-time results also require that greedy policies are full support. This is equivalent to the following property: γΩ(R)>0,for every finite R≥0. _ (R)>0, every finite R≥ 0. (P1) Many regularizers satisfy (P1) by default, including the α-divergences to μ with α≤1α≤ 1. For those whose greedy policies can be sparse, such as α-divergences with α>1α>1, an ϵε-floor that mixes the greedy policy with μ restores it22 2 Subsection G.5. Under Assumptions 1, 2 and 3, the regularized optimality equation Ω(QΩ∗)=QΩ∗+ρΩ∗T_ (Q^*_ )=Q^*_ +ρ^*_ 1 has a solution, ρΩ∗ρ^*_ is unique, and under (P1) QΩ∗Q^*_ is unique up to an additive constant (Theorem G.1)33 3 Existence holds more generally, Subsection G.3. Lemma 1 (Mixing properties of Ω -regularization). Under Assumptions 1 and 2 and for any Q,Q′Q,Q : (i) Contraction: with ΩkT^k_ the k-th composition of the regularized Bellman operator, ‖Ωm(Q)−Ωm(Q′)‖sp≤(1−βm∏k=1m−1γΩ(R[Ωk(Q),Ωk(Q′)]))‖Q−Q′‖sp.\|T^m_ (Q)-T^m_ (Q )\|_sp≤ (1- _m _k=1^m-1 _ (R[T_ ^k(Q),T_ ^k(Q )]) )\|Q-Q \|_sp. (i) Coverage: let (sn,an)n≥0(s_n,a_n)_n≥ 0 be a trajectory with an∼πΩ,Qn(⋅|sn)a_n _ ,Q_n(·|s_n) and sn+1∼p(⋅|sn,an)s_n+1 p(·|s_n,a_n), where QnQ_n is ℱn−1F_n-1-measurable for ℱn:=σ(Q0,(sk,ak)k≤n)F_n:=σ(Q_0,(s_k,a_k)_k≤ n). If γ1,…,γm≥0 _1,…, _m≥ 0 are ℱtF_t-measurable and πΩ,Qt+k≥γkμ _ ,Q_t+k≥ _kμ almost surely, then almost surely ℙ((st+m,at+m)=(s,a)∣ℱt)≥βm(∏k=1mγk)η(s,a), for every (s,a).P ((s_t+m,a_t+m)=(s,a) _t )≥ _m ( _k=1^m _k )η(s,a), for every (s,a). (i) Bias: the one step operator gap is: ‖Ω(Q)−(Q)‖sp≤ΛΩ(‖Q‖asp)\|T_ (Q)-T(Q)\|_sp≤ _ (\|Q\|_asp). Moreover, if Assumption 3 and (P1) hold, then for every solution (Q∗,ρ∗)(Q^*,ρ^*) of (Q∗)=Q∗+ρ∗T(Q^*)=Q^*+ρ^*1, ‖QΩ∗−Q∗‖sp≤mΛΩ(R∗)βmγΩ(R∗)m−1,R∗:=max‖QΩ∗‖asp,max0≤k<m‖Ωk(Q∗)‖asp,\|Q^*_ -Q^*\|_sp≤ m\, _ (R^*) _m _ (R^*)^m-1, R^*:= \\|Q^*_ \|_asp, _0≤ k<m\|T^k_ (Q^*)\|_asp \, and, if ||≥2|S|≥ 2 and all the Ωs _s have the same minimum Ω¯ , as divergences to μ do with Ω¯=0 =0, −ΛΩ(‖Q∗‖asp)≤ρΩ∗−ρ∗+τΩ¯≤0.- _ (\|Q^*\|_asp)≤ρ^*_ -ρ^*+τ ≤ 0. The key observation behind the proof is that the difference of the operators applied to any two Q-functions can be written exactly as a single kernel applied to their difference: Ω(Q)−Ω(Q′)=Kπ¯(Q−Q′)T_ (Q)-T_ (Q )=K_ π(Q-Q ), where π¯ π is the average of the greedy policies along the segment [Q′,Q][Q ,Q]. Although existing work factorizes differences of hard maxima with a single policy kernel (Chen et al., 2026, Lemma B.6), ours makes this policy explicit and uses its properties: π¯ π inherits the domination of μ by the greedy policies, so composing m such kernels transfers the Doeblin condition of Assumption 1, which gives (i), and the same domination along a trajectory gives (i). The profile γΩ _ thus controls contraction and exploration at once. The last item exposes the trade-off between bias and convergence. The greedy policy only depends on Q/τQ/τ, so γΩ(R) _ (R) only depends on R/τR/τ, and a larger temperature raises it, hence the contraction and coverage rates; but it weakens the control of the modulus: for regularizers bounded on the simplex, ΛΩ≤τΔΩ _ ≤τ _ uniformly in the radius, with ΔΩ _ the range of Ω (Remark G.1), a bound that grows linearly in τ. The gain gap, in contrast, involves neither the effective horizon Heff:=m/(βmγΩm−1)H_eff:=m/( _m _ ^m-1) nor γΩ _ , only the modulus at the τ-independent radius ‖Q∗‖asp\|Q^*\|_asp: as τ→0τ→ 0, unlike the value bound, it vanishes linearly for bounded regularizers. 4 Algorithms and Finite-Time Analysis We analyze increasingly realistic algorithms, each isolating one mechanism of Lemma 1: Q-iteration the contraction, synchronous Q-learning the statistical error, epoch Q-learning the cost of exploration, and Q-planning the effect of localization. Missing pseudocodes are given in Appendix B and proofs in Appendices H and J. Throughout this section, let δ∈(0,1)δ∈(0,1) be the failure probability, T be the budget of the algorithm and e0:=‖Q0−QΩ∗‖sp>0e_0:=\|Q_0-Q^*_ \|_sp>0 be the initial error. The iterates stay, with high probability, within span distance 2e02e_0 of QΩ∗Q^*_ . On that ball, β~:=βmγ¯m−1 β:= _m γ^\,m-1 with γ¯:=γΩ(‖QΩ∗‖asp+2e0) γ:= _ (\|Q^*_ \|_asp+2e_0) is the contraction rate of Lemma 1.(i), w¯:=‖QΩ∗‖sp+2e0+‖Ω∗(0)‖,sp w:=\|Q^*_ \|_sp+2e_0+\| ^*(0)\|_S,sp bounds the noise, and ι:=log(6||||(T+1)/δ) := (6|S||A|(T+1)/δ) is the price of a union bound over pairs and time. 4.1 Synchronous Ω -Regularized Q-Iteration Synchronous Ω -regularized Q-iteration applies the update: Qt+1(s,a)=Π[Ω(Qt)](s,a), for all (s,a),Q_t+1(s,a)= [T_ (Q_t) ](s,a), for all (s,a), where Π is the mean-centering projection. This algorithm isolates the core convergence mechanism of regularization under m-step mixing, on which the model-free analyses below build. An immediate corollary of Lemma 1.(i) is the following. Corollary 1 (Synchronous Ω -regularized Q-iteration). Let Assumptions 1, 2 and 3 and (P1) hold. Then (‖Qt−QΩ∗‖sp)t≥0(\|Q_t-Q_ ^*\|_sp)_t≥ 0 is non-increasing and, for every t≥0t≥ 0, ‖Qt+m−QΩ∗‖sp≤(1−βmγΩ(‖QΩ∗‖asp+‖Qt−QΩ∗‖sp)m−1)‖Qt−QΩ∗‖sp.\|Q_t+m-Q^*_ \|_sp≤ (1- _m\, _ (\|Q^*_ \|_asp+\|Q_t-Q^*_ \|_sp )^m-1 )\|Q_t-Q^*_ \|_sp. (1) Consequently, for ε∈(0,e0) ∈(0,e_0), ‖QT−QΩ∗‖sp≤ε\|Q_T-Q_ ^*\|_sp≤ for every T≥m⌈β~−1log(e0/ε)⌉T≥ m β^-1 (e_0/ ) , i.e. after O(||2||mβ~−1log(e0/ε))O (|S|^2|A|\,m\, β^-1 (e_0/ ) ) model queries, one query being an evaluation of p(s′|s,a)p(s |s,a). For m=1m=1, the domination profile is not needed: β~=β1 β= _1 and the rate is 1−β11- _1 per sweep, i.e. O(||2||β1−1log(e0/ε))O(|S|^2|A|\, _1^-1 (e_0/ )) model queries, which recovers the rate of relative value iteration under a one-step minorization of the kernel (Puterman, 1994). Our result in (1) reveals a contraction rate that increases as the error shrinks: from β~ β to βmγΩ(‖QΩ∗‖asp)m−1 _m _ (\|Q^*_ \|_asp)^m-1. Although all of our analysis shares this feature, we present sample complexity results assuming the slowest convergence rate for simplicity. 4.2 Synchronous Ω -Regularized Q-Learning Synchronous Ω -regularized Q-learning applies the update: Qt+1(s,a)=(1−αt)Qt(s,a)+αt(r(s,a)+Ω∗(Qt)(st′(s,a))), for all (s,a),Q_t+1(s,a)=(1- _t)Q_t(s,a)+ _t(r(s,a)+ ^*(Q_t)(s_t (s,a))), for all (s,a), where st′(s,a)s_t (s,a) is sampled from p(⋅|s,a)p(·|s,a) for each (s,a)(s,a) at each iteration. This algorithm isolates the statistical cost of learning from data. Theorem 1 (Synchronous Ω -regularized Q-learning). Let Assumptions 1, 2 and 3 and (P1) hold, and set α(u):=(β~2u2)(9m(2+β~)2(1+9m)2w¯2ι)−1α(u):=( β^2u^2)(9m(2+ β)^2(1+9m)^2 w^2 )^-1 for u>0u>0. Run Algorithm 3 for T iterations with a constant step size α≤α(e0)α≤α(e_0). Then, with probability at least 1−δ1-δ, supt≤T‖Qt−QΩ∗‖sp≤2e0 _t≤ T\|Q_t-Q^*_ \|_sp≤ 2e_0 and ‖QT−QΩ∗‖sp≤(1−β~/2)2αT9m1−β~/2‖Q0−QΩ∗‖sp+3(2+β~)(1+9m)β~w¯mαι.\|Q_T-Q^*_ \|_sp≤ (1- β/2) 2α T9m1- β/2\,\|Q_0-Q_ ^*\|_sp+ 3(2+ β)(1+9m) β\, w mα . (2) In particular, for ε∈(0,e0) ∈(0,e_0), the choices α=α(ε/2)α=α( /2) and T≥9mαβ~log(2e0(1−β~/2)ε)T≥ 9mα β ( 2e_0(1- β/2) ) give ‖QT−QΩ∗‖sp≤ε\|Q_T-Q^*_ \|_sp≤ , i.e. O~(||||m4w¯2/(β~3ε2)) O (|S||A|\,m^4 w^2/( β^3 ^2) ) sampled transitions. Because this algorithm updates every state-action pair at every iteration, the effect of γΩ _ is limited to contraction, not exploration. The sample complexity’s dependence on contraction is β~−3 β^-3 or, equivalently, (βmγ¯m−1)−3( _m γ^m-1)^-3. In the comparable setting where m=1m=1, the dependence on γ disappears, and, for unit rewards, we match44 4 Corollary H.3 the rate of Zhang et al. (2021) i.e., O~(||||β1−5ε−2) O(|S||A| _1^-5 ^-2). 4.3 Epoch Ω -Regularized Q-Learning The pseudocode of epoch Ω -regularized Q-learning is given in Algorithm 1. It adds the effect of exploration because it learns from a single trajectory, rather than querying a generative model at arbitrary (s,a)(s,a) pairs. Algorithm 1 Epoch Ω -Regularized Q-Learning 1: Input: initial Q0Q_0, regularizer Ω , step sizes αj\ _j\, batch sizes Bj\B_j\, initial state s0s_0. 2: Set n←0n← 0 and j←0j← 0. 3: while true do 4: Nj←0N_j← 0 and ΩQj^←0 T_ Q_j← 0 on ×S×A. 5: while min(s,a)Nj(s,a)<Bj _(s,a)N_j(s,a)<B_j do 6: Sample an∼πΩ,Qj(⋅|sn)a_n _ ,Q_j(·|s_n) and sn+1∼p(⋅|sn,an)s_n+1 p(·|s_n,a_n). 7: if Nj(sn,an)<BjN_j(s_n,a_n)<B_j then 8: ΩQj^(sn,an)+=r(sn,an)+Ω∗(Qj)(sn+1) T_ Q_j(s_n,a_n) +=r(s_n,a_n)+ ^*(Q_j)(s_n+1) and Nj(sn,an)+=1N_j(s_n,a_n) +=1. 9: end if 10: n←n+1n← n+1. 11: end while 12: Qj+1←Π[(1−αj)Qj+αjΩQj^/Bj]Q_j+1← [(1- _j)Q_j+ _j T_ Q_j/B_j ] and j←j+1j← j+1. 13: end while Theorem 2 (Epoch Ω -regularized Q-learning with αj≡1 _j≡ 1). Let Assumptions 1, 2 and 3 and (P1) hold, and set B(u):=8m2w¯2ι(β~u)2B(u):= 8m^2 w^2 ( βu)^2 for u>0u>0. Run Algorithm 1 with αj≡1 _j≡ 1 and a constant batch size B≥B(e0)B≥ B(e_0), and let J(T)J(T) be the number of epochs completed within T transitions. Then, with probability at least 1−δ1-δ, supj≤J(T)‖Qj−QΩ∗‖sp≤2e0 _j≤ J(T)\|Q_j-Q^*_ \|_sp≤ 2e_0, each epoch completed within the budget lasts at most L:=m⌈(B+ι)2/(γ¯β~ηmin)⌉+1L:=m ( B+ )^2/( γ β _ ) +1 transitions, so that J(T)≥⌊T/L⌋J(T)≥ T/L , and ‖QJ(T)−QΩ∗‖sp≤(1−β~)⌊J(T)/m⌋‖Q0−QΩ∗‖sp+22mw¯β~ιB.\|Q_J(T)-Q^*_ \|_sp≤(1- β) J(T)/m \|Q_0-Q_ ^*\|_sp+ 2 2\,m\, w β B. (3) In particular, for ε∈(0,e0) ∈(0,e_0), the choices B=⌈B(ε/2)⌉B= B( /2) and T≥mL⌈β~−1log(2e0/ε)⌉T≥ mL β^-1 (2e_0/ ) give ‖QJ(T)−QΩ∗‖sp≤ε\|Q_J(T)-Q^*_ \|_sp≤ , i.e. O~(m4w¯2/(β~4γ¯ηminε2)) O (m^4 w^2/( β^4 γ _ ^2) ) sampled transitions. Compared to synchronous Q-learning, the sample complexity worsens by a factor of (γ¯β~ηmin)−1/||||( γ β _ )^-1/|S||A|. This is a natural quantity to pay in the case of exploration because γ¯β~ηmin γ β _ controls the per-pair visitation probability under greedy policy rollout, whereas a generative model can query arbitrary state-action pairs. The exploration price can be reduced from γ¯β~ηmin γ β _ to βmηmin _m _ by rolling out μ, but the mixing policy μ will not seek to optimize the reward55 5 Remark H.7. 4.4 Ω -Regularized Q-Planning We present theorems and proof sketches from Appendices J and I. From a reset distribution ξ∈Δ()ξ∈ (S), planning rolls out trajectories of length H and backpropagates a value boundary condition QBQ_B, similar to Monte Carlo Tree Search (Grill et al., 2020). Planning isolates the role of localization: instead of targeting QΩ∗Q^*_ , it targets the planning solution defined by the backwards recursion: QΩ,h∗:=Ω,h(QΩ,h+1∗)Q^*_ ,h:=T_ ,h(Q^*_ ,h+1) and QΩ,H∗:=QB|H×Q^*_ ,H:=Q_B|_S_H×A, where QhQ_h and Ω,hT_ ,h are the depth-dependent value function and Bellman operator, and hS_h is the set of h-step reachable states. In other words, the planning solution is the best update possible with the information available in the first H steps. Given this setting, we relax the mixing requirement from Assumption 1 to only consider the reachable states from ξ. Assumption 4 (Localized m-step Doeblin mixing). There exist integers m≥1m≥ 1 and h1≥0h_1≥ 0, a full-support policy μ, βm∈(0,1] _m∈(0,1] and ν∈Δ()ν∈ (S) with support ∞S_∞ such that (i) h=∞S_h=S_∞ for every h≥h1h≥ h_1, and (i) ∑a′Kμm(s′,a′|s,a)≥βmν(s′) _a K^m_μ(s ,a |s,a)≥ _mν(s ) for all (s,a)∈∞×(s,a) _∞×A and s′∈∞s _∞. The first planning algorithm is the single backward sweep: Qh=Ω,h(Qh+1)Q_h=T_ ,h(Q_h+1). This case terminates and returns the planning solution exactly after O(H(maxh|h|)2||)O(H( _h|S_h|)^2|A|) model queries. Its root level is ε -accurate for QΩ∗Q^*_ once H≥h1+m+Hefflog+(ΔB/ε)H≥ h_1+m+H_eff _+( _B/ )66 6 Proposition J.1. If H is too small, there is insufficient backup information, compared to the boundary mismatch ΔB _B. The last algorithm we consider estimates each depth operator from rollouts. Whereas the learning algorithms iterate the stationary update Qj+1=Ω(Qj)Q_j+1=T_ (Q_j), the planning algorithm applies, up to sampling noise, the depth-shifted update Qj+1,h=Ω,h(Qj,h+1)Q_j+1,h=T_ ,h(Q_j,h+1): level h at epoch j+1j+1 is computed from level h+1h+1 at epoch j, which is frozen during epoch j. In particular, choosing H≥h1+m+Hefflog+(2ΔB/ε)H≥ h_1+m+H_eff _+(2 _B/ ), gives ‖QJ,0−QΩ∗‖sp,0×≤ε\|Q_J,0-Q^*_ \|_sp,S_0×A≤ with high probability, after O~((h1+Heff)4w¯2/(qε2)) O ((h_1+H_eff)^4\, w^2/(q\, ^2) ) sampled transitions77 7 Theorem I.1, where q is the minimum visitation probability across depth. The root error is a sum of level residuals damped at rate β~ β every m levels, so noise does not accumulate over the horizon and only the h1+Heffh_1+H_eff levels closest to the root contribute. Merging the H tables into a single table learned from resets removes the extra factor 1/β~1/ β. 5 Instances and Experiments 5.1 Instances of Base Regularizers We instantiate our regularizers with the α-divergences from Amari (Cichocki and Amari, 2010): Dα(p∥q):=(∑apaαqa1−α−1)/(α(α−1)).D_α(p\|q):= ( _ap_a^αq_a^1-α-1 )/ (α(α-1) ). These include D1=KL(p∥q)D_1=KL(p\|q) and D0=KL(q∥p)D_0=KL(q\|p) by continuity, which have been used for KL-regularized reinforcement learning (Vieillard et al., 2020), the χ2χ^2-divergence underlying advantage-weighted updates (Belousov and Peters, 2019), and, for a uniform μ, the Tsallis entropies up to a rescaling of τ (Lee et al., 2018). We define Ωα,s:=Dα(⋅∥μs) _α,s:=D_α(·\| _s) and we compute the relevant quantities in Table 188 8 Notation and proofs in Appendix K.. In particular, on the window 0<α≤10<α≤ 1, the divergences exhibit two favorable properties: the modulus is bounded uniformly in R and the greedy policy is full support. Table 1: The α-divergence regularizers general α reverse KL forward KL Hellinger χ2χ^2 α=1α=1 α=0α=0 α=12α= 12 α=2α=2 Ωα,s(π) _α,s(π) Dα(π∥μs)D_α(π\| _s) KL(π∥μs)KL(π\| _s) KL(μs∥π)KL( _s\|π) 2‖π−μs‖222\| π- _s\|_2^2 12χ2(π∥μs) 12χ^2(π\| _s) strong convexity μmin(1−α)+ _ ^(1-α)_+ if α≤2α≤ 2 11 μmin _ μmin _ 11 πΩ,Q(a|s)/μa _ ,Q(a|s)/ _a [1+(α−1)(Qa−ζs)τ]+1α−1 [1+ (α-1)(Q_a- _s)τ ]_+ 1α-1 eQa/τ∑bμbeQb/τ e^Q_a/τ _b _be^Q_b/τ (1+ζs−Qaτ)−1 (1+ _s-Q_aτ )^-1 (1+ζs−Qa2τ)−2 (1+ _s-Q_a2τ )^-2 [1+Qa−ζsτ]+ [1+ Q_a- _sτ ]_+ γΩ(R)≥ _ (R)≥ exp2−α(−R/τ) _2-α(-R/τ) e−R/τe^-R/τ τ+R τ+R (1+R2τ)−2 (1+ R2τ )^-2 [1−Rτ]+ [1- Rτ ]_+ ΛΩ(R)≤ _ (R)≤ τμmin1−α−1α(α−1)τ _ ^1-α-1α(α-1) if α>0α>0 τlog1μminτ 1 _ τ(1+log+Rτ)τ (1+ _+ Rτ ) 4τ(1−μmin)4τ(1- _ ) τ2(1μmin−1) τ2 ( 1 _ -1 ) (P1) holds iff α≤1α≤ 1 yes yes yes no ΛΩ _ bounded in R iff α>0α>0 yes no yes yes 5.2 Experiments Does knowledge of μ improve convergence and exploration? We consider the Two Rooms MDP (Figure 1(a)): two 5×55× 5 gridworld rooms joined by a two-cell doorway, with the four cardinal moves as actions. Moves are deterministic, except entering the doorway, which succeeds with probability pcrossp_cross and otherwise leaves the agent in place. The agent starts in the top-left corner and receives a reward of 11 in the bottom-right corner of the right room. The reference policy μ is uniform except at the two doorway entrances, where the entering action has probability μ(DOOR)μ(DOOR) (0.250.25 is uniform): the doorway is a bottleneck, and μ can increase the flow through it. We use the ϵε-floored reverse KL (τ=2τ=2, ϵ=0.05ε=0.05) and sweep μ(DOOR)∈0.25,0.4,0.55,0.7,0.85,0.95μ(DOOR)∈\0.25,0.4,0.55,0.7,0.85,0.95\ and pcross∈0.05,0.3,1p_cross∈\0.05,0.3,1\; details and the bias are in Subsection C.1. Figure 1(b) measures convergence with synchronous regularized Q-iteration (Algorithm 2), which isolates contraction from sampling noise, through the number of iterations per e-fold, 1/ρ1/ρ, where ‖Qk−QΩ∗‖sp∝e−ρk\|Q_k-Q_ ^*\|_sp e^-ρ k once the rate has settled. It grows with the bottleneck severity and decreases with μ(DOOR)μ(DOOR) when the doorway is a bottleneck (×2.4× 2.4 at pcross=0.05p_cross=0.05), but is flat without one, consistent with the direction predicted by the contraction mechanism. Figure 1(c) measures exploration with epoch regularized Q-learning ( Algorithm 1), through the epoch length relative to the uniform μ: solid, data collected under μ; dotted, on-policy; dashed, the coverage constant minmm/(βmηmin) _mm/( _m _ ). Biasing μ towards the doorway shortens epochs by up to ×5× 5 at pcross=0.05p_cross=0.05 but triples them at pcross=1p_cross=1: knowledge of μ helps only when it targets an actual bottleneck. The coverage constant reproduces the shape of this dependence, though not its scale. (a) The Two-Rooms MDP. (b) μ’s effect on convergence. (c) μ’s effect on exploration. Figure 1: Empirical results on the Two-Rooms MDP. Biasing the reference policy μ towards the bottleneck improves both convergence (b) and exploration (c). Does theory predict regularizer ranking? We consider the RiverSwim MDP (Strehl and Littman, 2008), with the parameterization of Osband et al. (2013) (Figure 2(a)), where =0,…,n−1S=\0,…,n-1\ and =LEFT,RIGHTA=\LEFT,RIGHT\. LEFT deterministically moves the agent to s−1s-1, with a small reward r(0,LEFT)=0.005r(0,LEFT)=0.005 at the leftmost state. RIGHT swims against the current: it moves to s+1s+1 with probability 0.350.35, stays with probability 0.60.6 and moves to s−1s-1 with probability 0.050.05, with a large reward r(n−1,RIGHT)=1r(n-1,RIGHT)=1 at the rightmost state. The reference policy μ is uniform and n=6n=6. We compare regularizers from the α-divergence family Dα(π∥μ)D_α(π\|μ): χ2(μ∥π)χ^2(μ\|π) (α=−1α=-1), forward KL (α=0α=0), Hellinger (α=0.5α=0.5), reverse KL (α=1α=1), D1.5D_1.5 and χ2(π∥μ)χ^2(π\|μ) (α=2α=2). Figure 2(c) plots the bias ‖QΩ∗−Q∗‖sp\|Q^*_ -Q^*\|_sp against τ (solid: without floor; dashed: the α>1α>1 regularizers wrapped by the ϵε-floor, ϵ=0.05ε=0.05). At a common temperature, the bias decreases as α increases, consistent with the regularization modulus ΛΩ _ decreasing in α (Table 1), and all curves merge at large τ, where every regularized policy approaches μ. Comparing convergence at a common τ would therefore confound speed with bias. We thus choose, for each regularizer, the temperature giving the same bias ‖QΩ∗−Q∗‖sp=2\|Q_ ^*-Q^*\|_sp=2, and run epoch regularized Q-learning (B=64B=64, 2020 seeds). Since increasing α lowers the domination profile γΩ _ , our theory predicts that at equal bias convergence slows as α increases. Figure 2(b) confirms this ranking: it plots the span error of each method to its own target (QΩ∗Q^*_ for the regularized methods, Q∗Q^* for the baselines) against environment steps, and convergence slows as α increases. At equal bias, the choice of regularizer is thus not a trade-off but a ranking, in favour of small α. As baselines, the unregularized epoch algorithm never completes an epoch since the greedy policy does not cover the state-action space, and asynchronous RVI Q-learning only escapes the local reward trap after ∼108 10^8 steps (for asynchronous RVI Q-learning, we plot the span distance between the iterates and the hard fixed point). (a) River Swim MDP (n=3n=3). (b) Ω ’s effect on convergence. (c) Ω ’s effect on bias. Figure 2: Comparison of regularizers from the α-divergence family on the RiverSwim MDP. As predicted by the theory, convergence slows as α increases at equal bias level. 6 Related Work Table 2: Sample complexity of average-reward methods, in sampled transitions and up to logarithmic factors; for Q-iteration we count evaluations of p(s′|s,a)p(s |s,a). Let tmixt_mix be the mixing time, K the worst expected hitting time of a reference state, and θ<1θ<1 a span-contraction factor. Whereas other methods target Q∗Q^*, we target the regularized fixed point QΩ∗Q^*_ . Paper Assumptions Sample complexity Q-iteration (known transitions) Puterman (1994) one-step minorization p(⋅|s,a)≥βνp(·|s,a)≥βν ||2||β−1log(1/ε)|S|^2|A|\,β^-1 (1/ ) Ours, Corollary 1 m-step Doeblin policy μ; regularizer satisfying (P1) ||2||mβ~−1log(1/ε)|S|^2|A|\,m\, β^-1 (1/ ) Generative model, model-based Wang et al. (2024) all stationary policies uniformly mixing ||||tmixε−2|S||A|\,t_mix\, ^-2 Zurek and Chen (2024) weakly communicating ||||‖maxaQ∗‖spε−2|S||A|\,\| _aQ^*\|_sp\, ^-2 Generative model, model-free Lee et al. (2025) weakly communicating ||||‖maxaQ∗‖sp2ε−2|S||A|\,\| _aQ^*\|_sp^2\, ^-2 Zhang et al. (2021) unichain; J-step span contraction at rate θ ||||J3(1−θ)−5ε−2|S||A|\,J^3(1-θ)^-5\, ^-2 Chen et al. (2026) reachability, hitting time K |||| 26Kε−2|S||A|\,2^6K\, ^-2 Ours, Theorem 1 same as for Q-iteration ||||m4w¯2β~−3ε−2|S||A|\,m^4\, w^2\, β^-3\, ^-2 Single trajectory, model-free Chen (2025) (asynchronous) one-step span contraction with factor θ; fixed full-support behavior policy; irreducible aperiodic behavior chain ||3||3ηmin−2(1−θ)−5ε−2|S|^3|A|^3\, _ ^-2(1-θ)^-5\, ^-2 Chen et al. (2026) (asynchronous) reachability, hitting time K; fixed full-support behavior policy; aperiodic chain ||3||3 26Kηmin−2ε−2|S|^3|A|^3\,2^6K\, _ ^-2\, ^-2 Ours, Theorem 2 (epoch) same as for Q-iteration m4w¯2β~−3(γ¯β~ηmin)−1ε−2m^4 w^2 β^-3( γ β _ )^-1 ^-2 Planning (resets to ξ), model-free Ours, Theorem I.1 m-step Doeblin policy μ on the states reachable from ξ (Assumption 4); regularizer satisfying (P1) (h1+Heff)4w¯2q−1ε−2(h_1+H_eff)^4\, w^2\,q^-1\, ^-2 Where the contraction comes from. The average-reward Bellman operator is not a contraction, and finite-time analyses of Q-learning differ in how they deal with this. Zhang et al. (2021) and Chen (2025) assume that the Bellman operator itself is a span contraction. Chen et al. (2026) derive a contraction from a reachability assumption by running Q-learning on lazy dynamics, in which the system stays in its current state with a fixed probability. Jiao et al. (2026) analyze Q-learning in weakly communicating MDPs without any contraction, at the price of an ε−3 ^-3 dependence, and other works rely on discounted approximations (Jin et al., 2024) or Halpern iterations with a generative model (Lee et al., 2025). With a generative model, model-based methods are minimax optimal and set the target rates (Wang et al., 2024; Zurek and Chen, 2024). We take a different route: the user supplies a policy that mixes, and regularizing toward it yields a contraction whose rate is regularizer dependent and leads to an ε−2 ^-2 sample complexity, at the price of a bias that we bound (Table 2). The same policy also provides the coverage that asynchronous analyses usually pay through a covering or mixing time of the behavior policy (Even-Dar and Mansour, 2003; Li et al., 2020). Regularization toward a reference policy. Regularizing toward a default policy is widespread: KL regularization averages errors in approximate value iteration (Vieillard et al., 2020), behavior priors speed up exploration and transfer (Teh et al., 2017; Tirumala et al., 2022; Ouhamma and Kamgarpour, 2026), and KL penalties to a reference policy underpin offline RL and RLHF (Wu et al., 2019; Xiong et al., 2024). Closest to our instances, Labbi et al. (2026) regularize policy gradient toward a reference policy with the same α-divergences, in the discounted setting. Their convergence constants are driven by the inverse link (f′)−1(f )^-1 at large negative arguments, which decays exponentially for the KL and polynomially for α<1α<1; our domination profile is bounded below by this same quantity (Table 1). In the average-reward setting, Neu et al. (2017) and Adamczyk et al. (2025) study entropy-regularized objectives from the viewpoints of policy optimization and deep RL. These works use the reference policy for stability, prior knowledge, conservatism or conditioning; none uses it to make the dynamics mix, which is the mechanism we exploit. Planning depth. In discounted planning, the depth of the lookahead tree is set by the discount factor (Kearns et al., 2002). Regularized tree search enjoys fast convergence at the root (Xiao et al., 2019) and polynomial sample complexity (Grill et al., 2019), but it targets the regularized optimum, whose greedy actions can be suboptimal for the original objective (Painter et al., 2023). In average-reward problems, finite-horizon values minus their linear growth converge as the horizon grows (Schweitzer and Federgruen, 1977). We make the depth needed for a given accuracy explicit in the mixing and the regularizer, with Heff=m/β~H_eff=m/ β in the role of 1/(1−γ)1/(1-γ), and control the regularization bias separately. 7 Discussion and Limitations The Doeblin pair (m,βm)(m, _m) and the profile γΩ _ are uniform over states, so a bottleneck anywhere dictates the rate everywhere; state-dependent constants, weighted along the paths of μ, could yield much faster rates. Our criterion, uniform accuracy on all pairs, also favors exploring with μ and does not credit acting with the learned policy, which instance-dependent approaches may be able to capture. Future directions include relaxing Assumption 1 to allow transient states for single-trajectory learning, matching the (1−γ)−4ε−2(1-γ)^-4 ^-2 rate of discounted Q-learning (Li et al., 2024) with HeffH_eff in place of 1/(1−γ)1/(1-γ), an asynchronous analysis (Chen, 2025), function approximation as well as temperature schedules to remove the bias while keeping early contraction. AI use statement We used generative AI tools to identify relevant literature, polish writing, improve readability, and help implement the experiments. We have not used generative AI tools to develop the theoretical framework, to propose hypotheses, or for proving mathematical claims. We have reviewed all AI-assisted work. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. Acknowledgements This work was supported by the NYU Office of Global Science and Technology. References Adamczyk et al. (2025) Jacob Adamczyk, Volodymyr Makarenko, Stas Tiomkin, and Rahul V. Kulkarni. Average-reward soft actor-critic. Reinforcement Learning Journal, 6:412–430, 2025. Arapostathis et al. (1993) Aristotle Arapostathis, Vivek S. Borkar, Emmanuel Fernández-Gaucherand, Mrinal K. Ghosh, and Steven I. Marcus. Discrete-time controlled Markov processes with average cost criterion: A survey. SIAM Journal on Control and Optimization, 31(2):282–344, 1993. Bartlett and Tewari (2009) Peter L. Bartlett and Ambuj Tewari. REGAL: A regularization based algorithm for reinforcement learning in weakly communicating MDPs. In Conference on Uncertainty in Artificial Intelligence, 2009. Belousov and Peters (2019) Boris Belousov and Jan Peters. Entropic regularization of Markov decision processes. Entropy, 21(7):674, 2019. Bertsekas (2012) Dimitri P. Bertsekas. Dynamic Programming and Optimal Control, Vol. I. Athena Scientific, 4th edition, 2012. Chen (2025) Zaiwei Chen. Non-asymptotic guarantees for average-reward Q-learning with adaptive stepsizes. In Advances in Neural Information Processing Systems, 2025. Chen and Nanda (2026) Zaiwei Chen and Phalguni Nanda. From set convergence to pointwise convergence: Finite-time guarantees for average-reward Q-learning with adaptive stepsizes. arXiv preprint arXiv:2504.18743v2, 2026. Chen et al. (2026) Zijun Chen, Zaiwei Chen, Nian Si, and Shengbo Wang. Achieving ε−2 ^-2 dependence for average-reward Q-learning with a new contraction principle. arXiv preprint arXiv:2601.21301, 2026. Cichocki and Amari (2010) Andrzej Cichocki and Shun-ichi Amari. Families of alpha- beta- and gamma- divergences: Flexible and robust measures of similarities. Entropy, 12(6):1532–1568, 2010. Crandall and Tartar (1980) Michael G. Crandall and Luc Tartar. Some relations between nonexpansive and order preserving mappings. Proceedings of the American Mathematical Society, 78(3):385–390, 1980. De Pasquale et al. (2024) Giulia De Pasquale, Kevin D. Smith, Francesco Bullo, and Maria Elena Valcher. Dual seminorms, ergodic coefficients and semicontraction theory. IEEE Transactions on Automatic Control, 69(5):3040–3053, 2024. Dieudonné (1960) Jean Dieudonné. Foundations of Modern Analysis, volume 10 of Pure and Applied Mathematics. Academic Press, 1960. Dobrushin (1956) Roland L. Dobrushin. Central limit theorem for nonstationary Markov chains. I. Theory of Probability and Its Applications, 1(1):65–80, 1956. Even-Dar and Mansour (2003) Eyal Even-Dar and Yishay Mansour. Learning rates for Q-learning. Journal of Machine Learning Research, 5:1–25, 2003. Freedman (1975) David A. Freedman. On tail probabilities for martingales. The Annals of Probability, 3(1):100–118, 1975. Gaubert and Qu (2015) Stéphane Gaubert and Zheng Qu. Dobrushin’s ergodicity coefficient for Markov operators on cones. Integral Equations and Operator Theory, 81(1):127–150, 2015. Geist et al. (2019) Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In International Conference on Machine Learning, 2019. Grill et al. (2019) Jean-Bastien Grill, Omar Darwiche Domingues, Pierre Ménard, Rémi Munos, and Michal Valko. Planning in entropy-regularized Markov decision processes and games. In Advances in Neural Information Processing Systems, 2019. Grill et al. (2020) Jean-Bastien Grill, Florent Altché, Yunhao Tang, Thomas Hubert, Michal Valko, Ioannis Antonoglou, and Rémi Munos. Monte-Carlo tree search as regularized policy optimization. In International Conference on Machine Learning, 2020. Hernández-Lerma and Lasserre (1996) Onésimo Hernández-Lerma and Jean B. Lasserre. Discrete-Time Markov Control Processes: Basic Optimality Criteria, volume 30 of Applications of Mathematics. Springer, 1996. Hiriart-Urruty and Lemaréchal (2001) Jean-Baptiste Hiriart-Urruty and Claude Lemaréchal. Fundamentals of Convex Analysis. Grundlehren Text Editions. Springer, 2001. Jaksch et al. (2010) Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11:1563–1600, 2010. Jiao et al. (2026) Yuchen Jiao, Jiin Woo, Gen Li, Gauri Joshi, and Yuejie Chi. Sample complexity of average-reward Q-learning: From single-agent to federated reinforcement learning. arXiv preprint arXiv:2601.13642, 2026. Jin et al. (2024) Ying Jin, Ramki Gummadi, Zhengyuan Zhou, and Jose Blanchet. Feasible Q-learning for average reward reinforcement learning. In International Conference on Artificial Intelligence and Statistics, 2024. Kakade et al. (2012) Sham M. Kakade, Shai Shalev-Shwartz, and Ambuj Tewari. Regularization techniques for learning with matrices. Journal of Machine Learning Research, 13(59):1865–1890, 2012. Kearns et al. (2002) Michael Kearns, Yishay Mansour, and Andrew Y. Ng. A sparse sampling algorithm for near-optimal planning in large Markov decision processes. Machine Learning, 49(2–3):193–208, 2002. Kuszmaul and Qi (2021) William Kuszmaul and Qi Qi. The multiplicative version of Azuma’s inequality, with an application to contention analysis. arXiv preprint arXiv:2102.05077, 2021. Labbi et al. (2026) Safwan Labbi, Daniil Tiapkin, Paul Mangold, and Éric Moulines. Beyond softmax and entropy: Convergence rates of policy gradients with f-SoftArgmax parameterization & coupled regularization. In International Conference on Learning Representations, 2026. Lee et al. (2025) Jongmin Lee, Mario Bravo, and Roberto Cominetti. Near-optimal sample complexity for MDPs via anchoring. In International Conference on Machine Learning, 2025. Lee et al. (2018) Kyungjae Lee, Sungjoon Choi, and Songhwai Oh. Sparse Markov decision processes with causal sparse Tsallis entropy regularization for reinforcement learning. IEEE Robotics and Automation Letters, 3(3):1466–1473, 2018. Lemmens and Nussbaum (2012) Bas Lemmens and Roger Nussbaum. Nonlinear Perron–Frobenius Theory, volume 189 of Cambridge Tracts in Mathematics. Cambridge University Press, 2012. Li et al. (2020) Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Sample complexity of asynchronous Q-learning: Sharper analysis and variance reduction. In Advances in Neural Information Processing Systems, 2020. Li et al. (2024) Gen Li, Changxiao Cai, Yuxin Chen, Yuting Wei, and Yuejie Chi. Is Q-learning minimax optimal? a tight sample complexity analysis. Operations Research, 2024. Mitzenmacher and Upfal (2005) Michael Mitzenmacher and Eli Upfal. Probability and Computing: Randomized Algorithms and Probabilistic Analysis. Cambridge University Press, 2005. Nanda and Chen (2026) Phalguni Nanda and Zaiwei Chen. A minimal-assumption analysis of Q-learning with time-varying policies. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 10(2), 2026. Neu et al. (2017) Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized Markov decision processes. arXiv preprint arXiv:1705.07798, 2017. Osband et al. (2013) Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems, 2013. Ouhamma and Kamgarpour (2026) Reda Ouhamma and Maryam Kamgarpour. Learning in zero-sum Markov games: Relaxing strong reachability and mixing time assumptions. In AAAI Conference on Artificial Intelligence, 2026. Painter et al. (2023) Michael Painter, Mohamed Baioumy, Nick Hawes, and Bruno Lacerda. Monte Carlo tree search with Boltzmann exploration. In Advances in Neural Information Processing Systems, 2023. Peters et al. (2019) Ben Peters, Vlad Niculae, and André F. T. Martins. Sparse sequence-to-sequence models. In Annual Meeting of the Association for Computational Linguistics, 2019. Puterman (1994) Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 1994. Roch (2024) Sébastien Roch. Modern Discrete Probability: An Essential Toolkit. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2024. Rockafellar (1970) R. Tyrrell Rockafellar. Convex Analysis. Princeton University Press, 1970. Roulet et al. (2025) Vincent Roulet, Tianlin Liu, Nino Vieillard, Michael Eli Sander, and Mathieu Blondel. Loss functions and operators generated by f-divergences. In International Conference on Machine Learning, 2025. Schweitzer (1984) Paul J. Schweitzer. On the existence of relative values for undiscounted Markovian decision processes with a scalar gain rate. Journal of Mathematical Analysis and Applications, 104(1):67–78, 1984. Schweitzer (1985) Paul J. Schweitzer. On undiscounted Markovian decision processes with compact action spaces. RAIRO – Operations Research, 19(1):71–86, 1985. Schweitzer and Federgruen (1977) Paul J. Schweitzer and Awi Federgruen. The asymptotic behavior of undiscounted value iteration in Markov decision problems. Mathematics of Operations Research, 2(4):360–381, 1977. Schweitzer and Federgruen (1978) Paul J. Schweitzer and Awi Federgruen. The functional equations of undiscounted Markov renewal programming. Mathematics of Operations Research, 3(4):308–321, 1978. Seneta (2006) Eugene Seneta. Non-negative Matrices and Markov Chains. Springer Series in Statistics. Springer, 2006. Strehl and Littman (2008) Alexander L. Strehl and Michael L. Littman. An analysis of model-based interval estimation for Markov decision processes. Journal of Computer and System Sciences, 2008. Sutton and Barto (2018) Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018. Teh et al. (2017) Yee Whye Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, 2017. Tirumala et al. (2022) Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojciech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, and Nicolas Heess. Behavior priors for efficient reinforcement learning. Journal of Machine Learning Research, 23(221):1–68, 2022. Todorov (2007) Emanuel Todorov. Linearly-solvable Markov decision problems. In Advances in Neural Information Processing Systems, 2007. Tsitsiklis (2007) John N. Tsitsiklis. NP-hardness of checking the unichain condition in average cost MDPs. Operations Research Letters, 35(3):319–323, 2007. Vieillard et al. (2020) Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: An analysis of KL regularization in reinforcement learning. In Advances in Neural Information Processing Systems, 2020. Wan et al. (2024) Yi Wan, Huizhen Yu, and Richard S. Sutton. On convergence of average-reward Q-learning in weakly communicating Markov decision processes. arXiv preprint arXiv:2408.16262, 2024. Wang et al. (2024) Shengbo Wang, Jose Blanchet, and Peter Glynn. Optimal sample complexity for average reward Markov decision processes. In International Conference on Learning Representations, 2024. Wielandt (1950) Helmut Wielandt. Unzerlegbare, nicht negative Matrizen. Mathematische Zeitschrift, 52:642–648, 1950. Wu et al. (2019) Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019. Xiao et al. (2019) Chenjun Xiao, Ruitong Huang, Jincheng Mei, Dale Schuurmans, and Martin Müller. Maximum entropy Monte-Carlo planning. In Advances in Neural Information Processing Systems, 2019. Xiong et al. (2024) Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint. In International Conference on Machine Learning, 2024. Zhang et al. (2021) Sheng Zhang, Zhe Zhang, and Siva Theja Maguluri. Finite sample analysis of average-reward TD learning and Q-learning. In Advances in Neural Information Processing Systems, 2021. Zhu et al. (2023) Lingwei Zhu, Zheng Chen, Matthew Schlegel, and Martha White. General Munchausen reinforcement learning with Tsallis Kullback–Leibler divergence. In Advances in Neural Information Processing Systems, 2023. Zurek and Chen (2024) Matthew Zurek and Yudong Chen. Span-based optimal sample complexity for weakly communicating and general average reward MDPs. In Advances in Neural Information Processing Systems, 2024. Appendix Contents Appendix A Notation Summary Table 3: Notation used across sections. MDP and kernels S, A, p, r finite state and action sets, transition kernel, deterministic reward ℙπ(⋅∣s0=s)P_π(· s_0=s), π[⋅∣s0=s]E_π[· s_0=s] law and expectation of the chain started at s, with at∼π(⋅|st)a_t π(·|s_t) and st+1∼p(⋅|st,at)s_t+1 p(·|s_t,a_t) P rectangular kernel (s,a)↦p(⋅|s,a)(s,a) p(·|s,a), from ×S×A to S PπP_π state kernel Pπ(s′|s)=∑aπ(a|s)p(s′|s,a)P_π(s |s)= _aπ(a|s)p(s |s,a), acting on ℝR^S KπK_π state-action kernel Kπ(s′,a′|s,a)=p(s′|s,a)π(a′|s′)K_π(s ,a |s,a)=p(s |s,a)π(a |s ), acting on ℝ×R^S×A Kπ1:πmK_ _1: _m, KμmK^m_μ m-step state-action kernels, (G.2); Kμm(s′|s,a)K^m_μ(s |s,a) is the state marginal Π centering projection Π[Q]=Q−(⊤Q/||||) [Q]=Q- (1 Q/|S||A| )1 Norms and seminorms ∥⋅∥sp\|·\|_sp, ‖Q‖asp\|Q\|_asp span seminorm, action-span seminorm, Definition 1 ∥⋅∥sp,op\|·\|_sp,op operator span seminorm, Definition E.3 ∥⋅∥TV\|·\|_TV, α(P)α(P) total variation distance; Dobrushin coefficient, Definition E.2 Regularization τ, Ωs _s, dom(Ωs)dom( _s) temperature; regularizer at s; its effective domain Ω∗ ^*, πΩ,Q _ ,Q conjugate of τΩτ and greedy policy, Subsection 2.2 T, ΩT_ hard and regularized Bellman operators Q∗,ρ∗Q^*,ρ^*; QΩ∗,ρΩ∗Q^*_ ,ρ^*_ optimal and regularized relative value and gain r~π r_π regularized reward ⟨πs,rs⟩−τΩs(πs) _s,r_s -τ _s( _s) Ω¯s _s, Ω¯ , Ω¯ , ΔΩ _ minΩs _s; minsΩ¯s _s _s; maxs,aΩs(δa) _s,a _s( _a); Ω¯−Ω¯ - π¯Ω,[Q,Q′] π_ ,[Q,Q ], R[Q,Q′]R[Q,Q ] average-path policy and radius, Definition G.1.(i)–(i) γΩ(R) _ (R) domination profile, Definition 2.(i) gQg_Q, ΛΩ(R) _ (R), ΛΩpt(R) _ ^pt(R) regularization gap and moduli, Definition 2.(i), Definition G.2 Mixing and rates μ, m, βm _m, ν mixing policy and its m-step Doeblin constants, Assumption 1 η, ηmin _ η(s,a)=ν(s)μ(a|s)η(s,a)=ν(s)μ(a|s) and its minimum μmin _ , νmin _ mins,aμ(a|s) _s,aμ(a|s) and minsν(s) _sν(s) β~m(A) β_m(A) regularized m-step contraction rate βmγΩ(A)m−1 _m _ (A)^m-1 Learning analysis δ, T, e0e_0, ι failure probability, budget, initial error ‖Q0−QΩ∗‖sp\|Q_0-Q^*_ \|_sp, log(6||||(T+1)/δ) (6|S||A|(T+1)/δ), Section 4 γ¯ γ, β~ β, w¯ w domination constant γΩ(‖QΩ∗‖asp+2e0) _ (\|Q^*_ \|_asp+2e_0), contraction rate βmγ¯m−1 _m γ^\,m-1 and noise scale on the ball of radius 2e02e_0, Section 4 ARA_R, CRC_R, β¯(R) β(R), w¯(R) w(R) localized radii and constants, Definition H.2 pmp_m, κ0 _0 probability of m effective updates per block and sub-block budget, Lemma H.6 σR _R, ete_t exit time from the ball of radius R and error at time t, Definition H.2 ℬiB_i, bib_i, ℬresB_res, I(T)I(T) blocks, their first indices, last incomplete block, number of completed blocks, Definitions H.1 and H.2.2 J(T)J(T), ℓj _j, Lj(δ′)L_j(δ ), qRq_R completed epochs, epoch length and its bound, coverage constant, Lemma H.15 Planning analysis ξ, H, QBQ_B reset distribution, planning horizon, boundary value, Subsection 4.4 hS_h, hX_h, ′S , S¯H S_H reachable layers, h×S_h×A, reachable set ⋃h _hS_h, maxh≤H|h| _h≤ H|S_h|, Appendix I h1h_1, ∞S_∞ depth after which the layers no longer change, and their common value, Assumption 4 Ω,hT_ ,h, QΩ,h∗Q^*_ ,h depth-dependent operator from ℝh+1R^X_h+1 to ℝhR^X_h, planning solution, Subsection 4.4 ΔB _B, D0D_0 boundary mismatch ‖QB−QΩ∗‖sp,H\|Q_B-Q^*_ \|_sp,X_H, initialization error maxh<H‖Q0,h−QΩ,h∗‖sp _h<H\|Q_0,h-Q^*_ ,h\|_sp; in planning, e0:=D0+ΔBe_0:=D_0+ _B Heff(A)H_eff(A), M(h,A)M(h,A) effective horizon m/β~m(A)m/ β_m(A) and (h1−h)++Heff(A)(h_1-h)_++H_eff(A), (J.2); HeffH_eff and M denote their values at A=A2e0A=A_2e_0 and h=0h=0, Appendix I qμq_μ, qπ(A)q_π(A), q coverage constants under μ and under a behavior dominating γΩ(A)μ _ (A)μ, (I.1); q is the constant of Theorem I.1 ιH _H log(4JHS¯H||/δ) (4JH S_H|A|/δ), Theorem I.1 Appendix B Algorithm Pseudocodes B.1 Learning All the learning algorithms apply the centering projection Π[Q]:=Q−1||||⊤Q [Q]:=Q- 1|S||A|1 Q1, which only shifts the iterates by a constant and does not affect the analysis in span seminorm. Algorithm 2 Synchronous Ω -Regularized Q-Iteration 1: Input: initial Q0Q_0, regularizer Ω , transition kernel p. 2: for t=0,1,…t=0,1,… do 3: Ω(Qt)(s,a)←r(s,a)+∑s′p(s′|s,a)Ω∗(Qt)(s′)T_ (Q_t)(s,a)← r(s,a)+ _s p(s |s,a)\, ^*(Q_t)(s ) for every (s,a)∈×(s,a) ×A. 4: Qt+1←Π[Ω(Qt)]Q_t+1← [T_ (Q_t)]. 5: end for Algorithm 3 Synchronous Ω -Regularized Q-Learning 1: Input: initial Q0Q_0, regularizer Ω , step sizes αtt≥0\ _t\_t≥ 0, generative model of p. 2: for t=0,1,…t=0,1,… do 3: Sample st′(s,a)∼p(⋅|s,a)s _t(s,a) p(·|s,a) independently for every (s,a)∈×(s,a) ×A. 4: ΩQt^(s,a)←r(s,a)+Ω∗(Qt)(st′(s,a)) T_ Q_t(s,a)← r(s,a)+ ^*(Q_t) (s _t(s,a) ) for every (s,a)(s,a). 5: Qt+1←Π[(1−αt)Qt+αtΩQt^]Q_t+1← [(1- _t)Q_t+ _t T_ Q_t ]. 6: end for B.2 Planning The planning algorithms take the reachable layers 0,…,HS_0,…,S_H of Appendix I as input; they depend only on ξ and on the support of p, and the stopping rules of the model-free algorithms require them. Algorithm 4 Backward Synchronous Ω -Regularized Q-Planning 1: Input: boundary value QBQ_B, regularizer Ω , transition kernel p, layers 0,…,HS_0,…,S_H. 2: Set QH←QB|HQ_H← Q_B|_X_H. 3: for h=H−1,…,0h=H-1,…,0 do 4: Qh(s,a)←r(s,a)+∑s′∈h+1p(s′|s,a)Ω∗(Qh+1)(s′)Q_h(s,a)← r(s,a)+ _s _h+1p(s |s,a)\, ^*(Q_h+1)(s ) for every (s,a)∈h(s,a) _h. 5: end for 6: Return (Qh)h=0H(Q_h)_h=0^H. Algorithm 5 Epoch Ω -Regularized Q-Planning 1: Input: initial Q0Q_0, boundary value QBQ_B, regularizer Ω , batch sizes Bj\B_j\, reset distribution ξ, layers 0,…,HS_0,…,S_H. 2: Set Q0,h←Q0|hQ_0,h← Q_0|_X_h for h<Hh<H, Qj,H:=QB|HQ_j,H:=Q_B|_X_H for every j, and j←0j← 0. 3: while true do 4: Nj,h←0N_j,h← 0 and Ω,hQj,h+1^←0 T_ ,hQ_j,h+1← 0 on hX_h, for every h<Hh<H. 5: while minh<Hmin(s,a)∈hNj,h(s,a)<Bj _h<H _(s,a) _hN_j,h(s,a)<B_j do 6: Sample s0∼ξs_0 ξ. 7: for h=0,…,H−1h=0,…,H-1 do 8: Sample ah∼πΩ,Qj,h(⋅|sh)a_h _ ,Q_j,h(·|s_h) and sh+1∼p(⋅|sh,ah)s_h+1 p(·|s_h,a_h). 9: if Nj,h(sh,ah)<BjN_j,h(s_h,a_h)<B_j then 10: Ω,hQj,h+1^(sh,ah)+=r(sh,ah)+Ω∗(Qj,h+1)(sh+1) T_ ,hQ_j,h+1(s_h,a_h) +=r(s_h,a_h)+ ^*(Q_j,h+1)(s_h+1) and Nj,h(sh,ah)+=1N_j,h(s_h,a_h) +=1. 11: end if 12: end for 13: end while 14: Qj+1,h←Ω,hQj,h+1^/BjQ_j+1,h← T_ ,hQ_j,h+1/B_j for every h<Hh<H, and j←j+1j← j+1. 15: end while Algorithm 6 Epoch Ω -Regularized Q-Learning with Resets 1: Input: initial Q0∈ℝ∞×Q_0 ^S_∞×A, regularizer Ω , step sizes αj\ _j\, batch sizes Bj\B_j\, reset distribution ξ, burn-in depth h1h_1 and core ∞S_∞, rollout length H>h1H>h_1. 2: Set j←0j← 0. 3: while true do 4: Nj←0N_j← 0 and ΩQj^←0 T_ Q_j← 0 on ∞×S_∞×A. 5: while min(s,a)∈∞×Nj(s,a)<Bj _(s,a) _∞×AN_j(s,a)<B_j do 6: Sample s0∼ξs_0 ξ. 7: for h=0,…,H−1h=0,…,H-1 do 8: Sample ah∼μ(⋅|sh)a_h μ(·|s_h) if h<h1h<h_1 and ah∼πΩ,Qj(⋅|sh)a_h _ ,Q_j(·|s_h) otherwise, and sh+1∼p(⋅|sh,ah)s_h+1 p(·|s_h,a_h). 9: if h≥h1h≥ h_1 and Nj(sh,ah)<BjN_j(s_h,a_h)<B_j then 10: ΩQj^(sh,ah)+=r(sh,ah)+Ω∗(Qj)(sh+1) T_ Q_j(s_h,a_h) +=r(s_h,a_h)+ ^*(Q_j)(s_h+1) and Nj(sh,ah)+=1N_j(s_h,a_h) +=1. 11: end if 12: end for 13: end while 14: Qj+1←Π[(1−αj)Qj+αjΩQj^/Bj]Q_j+1← [(1- _j)Q_j+ _j T_ Q_j/B_j ] and j←j+1j← j+1. 15: end while Appendix C Additional Experiments C.1 Two Rooms: experimental details and bias Setting. At the two doorway entrances, μ puts mass μ(DOOR)μ(DOOR) on the action entering the doorway and (1−μ(DOOR))/3(1-μ(DOOR))/3 on each of the three others; everywhere else it is uniform. The regularizer is the ϵε-floored reverse KL with τ=2τ=2 and ϵ=0.05ε=0.05 in all panels. Convergence. Synchronous regularized Q-iteration is exact, so no seeds are involved. The rate is fitted on the part of the error curve with 10−9<‖Qk−QΩ∗‖sp/‖Q0−QΩ∗‖sp<10−310^-9<\|Q_k-Q^*_ \|_sp/\|Q_0-Q^*_ \|_sp<10^-3. The convergence accelerates before settling: early iterates have a larger action-span, hence a sharper policy, a smaller domination profile γΩ _ and a slower contraction. On the slowest configuration (pcross=0.05p_cross=0.05, uniform μ) the first e-fold takes about 170170 iterations against 54.554.5 once settled, so a rate fitted on a fixed initial window depends on the window; we report the settled rate, which does not. Exploration. Epoch lengths are for Bj=1B_j=1 accepted sample per state-action pair, with a cap of 4⋅1054· 10^5 transitions per epoch. Under μ: median over 1212 seeds. On-policy: 44 epochs, median over 55 seeds of the mean of the last 22 epochs, the first epochs being dominated by the initialization Q0=0Q_0=0. All curves are normalized by their value at the uniform μ. In absolute terms, one epoch under the uniform μ costs 12 30412\,304, 3 0963\,096 and 1 9011\,901 transitions for pcross=0.05,0.3,1p_cross=0.05,0.3,1, and the coverage constant sits 6969, 5252 and 3939 times above these values, which is why only its shape is compared. Figure 3: Bias ‖QΩ∗−Q∗‖sp\|Q^*_ -Q^*\|_sp on Two Rooms against μ(DOOR)μ(DOOR), exact. Dotted line: uniform μ. Bias. Figure 3 reports the bias ‖QΩ∗−Q∗‖sp\|Q^*_ -Q^*\|_sp, computed exactly. When the doorway is a bottleneck, biasing μ towards it reduces the bias from 36.836.8 to 4.74.7, but past μ(DOOR)≈0.7μ(DOOR)≈ 0.7 the bias rises again, likely because the same knob also pushes the agent from the goal room back through the doorway. The convergence and exploration panels reward a large μ(DOOR)μ(DOOR); the bias is what prevents pushing it indefinitely. C.2 Choice of the regularizer Figure 4: The objects of Definition 2 for the α-divergences to a uniform μ over two actions, and the resulting bias. (a) Domination profile γΩ _ and (b) regularization modulus ΛΩ/τ _ /τ, both functions of R/τR/τ only. (c) The pair (ΛΩ/τ,γΩ)( _ /τ, _ ) as R/τR/τ grows. Figure 4 computes the two objects that govern our rates and bias for the α-divergences to μ. Since the greedy policy only depends on Q/τQ/τ, γΩ(R) _ (R) and ΛΩ(R)/τ _ (R)/τ only depend on R/τR/τ, so panels (a) and (b) hold for every temperature. Panel (a) illustrates Theorem K.1.(iv): for α>1α>1, the profile vanishes at a finite radius (open circles), at R/τ=μmax1−α/(α−1)R/τ= _ ^1-α/(α-1), that is 22 for α=2α=2 and 222 2 for α=1.5α=1.5, so that (P1) fails; for α≤1α≤ 1, it stays positive, with an exponential decay for the reverse KL and a polynomial one for α<1α<1, slower for smaller α. Panel (b) illustrates Theorem K.1.(v): the modulus saturates for α>0α>0, at log2 2 for the reverse KL and 4(1−2−1/2)4(1-2^-1/2) for the Hellinger regularizer, and grows without bound for α≤0α≤ 0, as log(R/τ) (R/τ) for the forward KL and (R/τ)1/2(R/τ)^1/2 for α=−1α=-1. Panel (c) traces both objects together as the radius grows: a regularizer is preferable when its curve lies higher, for a faster contraction and coverage, and further left, for a smaller bias. No value of α dominates. Divergences with α≤0α≤ 0 keep a large profile at the price of an unbounded modulus, and those with α>1α>1 have a small modulus but lose (P1); only the window 0<α≤10<α≤ 1 keeps the modulus bounded and the profile positive. Within it, the Hellinger regularizer keeps a larger profile at large R/τR/τ, whereas the reverse KL has the smaller modulus for this balanced μ, as discussed in Remark K.1. The value bias on RiverSwim (Figure 2(c)) follows the ordering of the moduli in panel (b): at a given temperature, it decreases with α, and as τ→0τ→ 0 it vanishes fastest for the bounded regularizers α>0α>0, consistently with Lemma 1.(i) and Theorem K.1.(v). For large τ, all greedy policies approach μ and the biases coincide. The ϵε-floor restores (P1) for α>1α>1 at the price of a slightly larger bias at small temperature. Appendix D Discussion on Assumption 1 We characterize the MDPs that satisfy Assumption 1 and its constants. Throughout, μ has full support and Kμn(s′|s,a)=∑s1p(s1|s,a)Pμn−1(s′|s1)K^n_μ(s |s,a)= _s_1p(s_1|s,a)P^n-1_μ(s |s_1) (Appendix E). Canonical constants. Let V¯μn(s′):=min(s,a)Kμn(s′|s,a) V^n_μ(s ):= _(s,a)K^n_μ(s |s,a). Since the minorization of Assumption 1 is uniform in the starting pair, a pair (β,ν~)(β, ν) satisfies it at horizon m if and only if βν~≤V¯μmβ ν≤ V^m_μ pointwise. The assumption thus holds at horizon m if and only if V¯μm>0 V^m_μ>0, and the canonical pair βm:=‖V¯μm‖1 _m:=\| V^m_μ\|_1, ν:=V¯μm/βmν:= V^m_μ/ _m, which we always use, dominates every other one, both in the contraction constant β and in the coverage profile βν~⊗μβ ν μ. Averaging Kμn(s′|s1,a1)≥V¯μn(s′)K^n_μ(s |s_1,a_1)≥ V^n_μ(s ) over a1∼μ(⋅|s1)a_1 μ(·|s_1) and decomposing Kμn+1K^n+1_μ on its first step gives V¯μn+1≥V¯μn V^n+1_μ≥ V^n_μ: the assumption then holds at every horizon n≥mn≥ m, with canonical constants at least as good. In fact, the contraction and the exploration rely on two different properties of KμmK^m_μ. The contraction only requires a bound on its Dobrushin coefficient, which the minorization provides as α(Kμm)≤1−βmα(K^m_μ)≤ 1- _m, possibly strictly (Lemma E.3), whereas the exploration requires the minorization itself, which bounds below the probability of reaching each pair. We state both through the Doeblin pair (βm,ν)( _m,ν), so that the domination profile transfers them at once: the greedy policies contract at rate βmγΩm−1 _m _ ^m-1 and visit every pair with probability at least βmγΩmηmin _m _ ^m _ every m steps (Lemma 1). Definition D.1 (Union graph). The union graph G is the directed graph on S with an edge s→s′s→ s whenever p(s′|s,a)>0p(s |s,a)>0 for some a∈a . For X⊆X and k≥0k≥ 0, XGkXG^k is the set of states reached from X by a walk of length exactly k. G is primitive if it is strongly connected and aperiodic, and its exponent ex(G)ex(G) is then the smallest k≥1k≥ 1 such that sGk=\s\G^k=S for every s. Since μ has full support, Pμn(s′|s)>0P^n_μ(s |s)>0 if and only if s′∈sGns ∈\s\G^n: G is the support graph of PμP_μ for every full-support μ. Proposition D.1 (Which MDPs satisfy Assumption 1). For every full-support μ, Assumption 1 holds for some m if and only if PμP_μ is irreducible and aperiodic, if and only if G is primitive, if and only if the MDP is communicating and G is aperiodic. In particular, Assumption 1 holds with some full-support policy if and only if it holds with every full-support policy: whether it holds only depends on the support of p, and μ only affects its constants βm _m and ν. Proof. The second equivalence holds because G is the support graph of PμP_μ, and the third because G is strongly connected if and only if the MDP is communicating. If V¯μm>0 V^m_μ>0, then Pμm>0P^m_μ>0 and Pμm+1>0P^m+1_μ>0 entrywise, so PμP_μ is irreducible, with closed walks of lengths m and m+1m+1 through every state, hence aperiodic. Conversely, if G is primitive, then s1Gex(G)=\s_1\G^ex(G)=S for every s1s_1, so that, for m:=ex(G)+1m:=ex(G)+1 and any s1s_1 with p(s1|s,a)>0p(s_1|s,a)>0, Kμm(⋅|s,a)≥p(s1|s,a)Pμm−1(⋅|s1)>0K^m_μ(·|s,a)≥ p(s_1|s,a)P^m-1_μ(·|s_1)>0, i.e. V¯μm>0 V^m_μ>0. ∎ Proposition D.2 (The minimal horizon). Under Assumption 1, let e(X):=mink≥0∣XGk=e(X):= \k≥ 0 XG^k=S\ for nonempty X⊆X . The smallest horizon at which Assumption 1 holds is m∗=1+max(s,a)∈×e(suppp(⋅|s,a)),m^*=1+ _(s,a) ×Ae (supp\,p(·|s,a) ), and ex(G)≤m∗≤ex(G)+1≤(||−1)2+2ex(G)≤ m^* (G)+1≤(|S|-1)^2+2. Proof. By the first-step decomposition and the full support of μ, Kμm(⋅|s,a)>0K^m_μ(·|s,a)>0 if and only if Xs,aGm−1=X_s,aG^m-1=S, with Xs,a:=suppp(⋅|s,a)X_s,a:=supp\,p(·|s,a). As G is strongly connected, every state has a predecessor, so XGk=XG^k=S implies XGk+1=XG^k+1=S, and k∣XGk==k≥e(X)\k XG^k=S\=\k≥ e(X)\. Hence V¯μm>0 V^m_μ>0 if and only if m−1≥e(Xs,a)m-1≥ e(X_s,a) for every (s,a)(s,a), which gives the formula. Since e(X)≤ex(G)e(X) (G) for every nonempty X, m∗≤ex(G)+1m^* (G)+1; since Pμm∗(s′|s)=∑aμ(a|s)Kμm∗(s′|s,a)>0P^m^*_μ(s |s)= _aμ(a|s)K^m^*_μ(s |s,a)>0 for every (s,s′)(s,s ), m∗≥ex(G)m^* (G); and ex(G)≤(||−1)2+1ex(G)≤(|S|-1)^2+1 by Wielandt (1950). ∎ The threshold m∗m^* is thus a property of the transition graph alone, and it is small when the dynamics can idle: if every state stays in place with positive probability under some action, then ex(G)ex(G) is the diameter of G, since any path can be lengthened with self-loops, and m∗m^* is at most the diameter plus one. The horizon m of our rates, in contrast, is not a property of the MDP: any m≥m∗m≥ m^* is valid and, since the algorithms do not use it, our guarantees hold at every such m. Larger horizons can only improve βm _m and βmηmin _m _ , but they weaken the transfer factors γΩm−1 _ ^m-1 for the contraction and γΩm _ ^m for the exploration, so the best horizon to read our guarantees balances the mixing of μ against the domination of the greedy policies. Remark D.1 (Comparison with other assumptions). Under Assumption 1, the MDP is communicating (Corollary G.1) and every full-support policy induces an irreducible aperiodic chain (Proposition D.1). The assumption is thus equivalent to the standard one on the behavior policy of asynchronous Q-learning, a full-support policy inducing an irreducible aperiodic chain (Chen, 2025, Assumption 3.1). Chen (2025) additionally assumes that the Bellman operator is a span contraction (his Assumption 2.1), whereas we obtain the contraction from regularization toward μ, and the constants of μ enter the rates. It is incomparable with the unichain condition: two states swapped by a single action form a unichain MDP whose G is periodic, and adding a self-loop action makes G primitive but the MDP not unichain. Unlike the unichain condition, whose verification is NP-hard (Tsitsiklis, 2007), it is checked in polynomial time on G, and it is preserved by adding actions or transitions. Periodicity can be removed by laziness, i.e. replacing p(⋅|s,a)p(·|s,a) by 12(δs+p(⋅|s,a)) 12( _s+p(·|s,a)), which preserves the gain of every stationary policy. Finally, if PμP_μ is unichain and aperiodic but has transient states, then for m large enough βm _m is still positive but ν vanishes exactly on these states, which a trajectory following full-support policies visits finitely often: full support of ν is what makes single-trajectory learning possible on every pair. Appendix E Mathematical Background The following convention will be useful in the proofs. Convention E.1. Throughout, X, Y and Z denote finite sets, typically S or ×S×A, and P is a row-stochastic matrix from X to Y. It maps functions f∈ℝf ^Y to Pf∈ℝPf ^X and distributions ϕ∈Δ()φ∈ (X) to ϕP∈Δ()φ P∈ (Y): (Pf)(x)=∑y∈P(y|x)f(y),(ϕP)(y)=∑x∈ϕ(x)P(y|x).(Pf)(x)= _y P(y|x)f(y), (φ P)(y)= _x φ(x)P(y|x). The product of P1P_1 from X to Y and P2P_2 from Y to Z is (P1P2)(z|x)=∑y∈P1(y|x)P2(z|y)(P_1P_2)(z|x)= _y P_1(y|x)P_2(z|y), so that (P1P2)f=P1(P2f)(P_1P_2)f=P_1(P_2f) and ϕ(P1P2)=(ϕP1)P2φ(P_1P_2)=(φ P_1)P_2. In a product P1P2⋯PnP_1P_2·s P_n, the Markov chain therefore takes its first step with P1P_1 and its last with PnP_n, whereas a function is transformed first by PnP_n and last by P1P_1. For a policy π, we also write Pπ(s′|s):=∑aπ(a|s)p(s′|s,a)P_π(s |s):= _aπ(a|s)\,p(s |s,a) for its state kernel, and Kπm(s′|s,a):=∑a′Kπm(s′,a′|s,a)K^m_π(s |s,a):= _a K^m_π(s ,a |s,a) for the law of the state reached after m steps from (s,a)(s,a); in this notation, Assumption 1 reads Kμm(⋅|s,a)≥βmνK^m_μ(·|s,a)≥ _mν. E.1 Concentration inequalities The analysis of the model-free algorithms controls two different random quantities. The number of visits to a state-action pair is a sum of indicators whose conditional probabilities are only known to be bounded below, on an event, by a small constant such as qR=βmγΩ(AR)mηminq_R= _m _ (A_R)^m _ ; Lemma E.1 bounds its lower tail multiplicatively, which keeps the cover times of Lemma H.15 linear in 1/qR1/q_R. The noise in the Bellman targets is a martingale with bounded increments, whose running maximum is controlled by Lemma E.2. Lemma E.1 (Self-bounding lower tail). Let (Zi)i=1B(Z_i)_i=1^B be adapted to a filtration (i)(G_i) with Zi∈[0,b]Z_i∈[0,b] almost surely. Let S:=∑i=1BZiS:= _i=1^BZ_i and S¯:=∑i=1B[Zi∣i−1] S:= _i=1^BE[Z_i _i-1]. For any deterministic a>0a>0 and c∈(0,a)c∈(0,a), ℙ(S<c,S¯≥a)≤exp(−1b(a−c−clogac))P (S<c,\ S≥ a )≤ (- 1b (a-c-c ac ) ) (E.1) Proof. This is the lower tail of the multiplicative Azuma inequality (Kuszmaul and Qi, 2021, App. A), stated on the event S¯≥a\ S≥ a\ in the manner of Freedman (1975) rather than under an almost-sure lower bound on S¯ S; we recall the argument. Let θ>0θ>0 and pi:=[Zi/b∣i−1]p_i:=E[Z_i/b _i-1]. Convexity of y↦e−θy e^-θ y on [0,1][0,1] and 1−x≤e−x1-x≤ e^-x give [e−θZi/b∣i−1]≤1−pi(1−e−θ)≤e−pi(1−e−θ)E[e^-θ Z_i/b _i-1]≤ 1-p_i(1-e^-θ)≤ e^-p_i(1-e^-θ), so that Mk:=exp(−θb∑i≤kZi+(1−e−θ)∑i≤kpi)M_k:= (- θb _i≤ kZ_i+(1-e^-θ) _i≤ kp_i ) is a non-negative supermartingale with M0=1M_0=1. On the event S<c,S¯≥a\S<c,\ S≥ a\ we have MB≥exp(−θcb+(1−e−θ)ab)M_B≥ (- θ cb+(1-e^-θ) ab ), so Markov’s inequality bounds its probability by exp(1b(θc−(1−e−θ)a)) ( 1b(θ c-(1-e^-θ)a) ), which is minimized at θ=log(a/c)θ= (a/c); this is (E.1). ∎ Lemma E.2 (Maximal Azuma–Hoeffding inequality with predictable ranges). Let (Mk)0≤k≤n(M_k)_0≤ k≤ n be a martingale with M0=0M_0=0, adapted to (ℱk)(F_k), such that for each k the increment Mk−Mk−1M_k-M_k-1 lies a.s. in an interval [ℒk,ℒk+ck][L_k,L_k+c_k] with ℒkL_k ℱk−1F_k-1-measurable and ck>0c_k>0 deterministic. Then for every u>0u>0, ℙ(max0≤k≤n|Mk|≥u)≤2exp(−2u2∑k=1nck2).P ( _0≤ k≤ n|M_k|≥ u )≤ 2 (- 2u^2 _k=1^nc_k^2 ). Proof. Apply Roch (2024, Theorem 3.2.1) to (Mk)(M_k) with Ak=ℒkA_k=L_k and Bk=ℒk+ckB_k=L_k+c_k, and to (−Mk)(-M_k) with Ak=−ℒk−ckA_k=-L_k-c_k and Bk=−ℒkB_k=-L_k, then take a union bound. ∎ E.2 On span seminorms and the Dobrushin coefficient This section provides additional definitions and lemmas that are useful when working with the span seminorm. Definition E.1 (Total-variation distance). The total-variation distance between two probability distributions ϕ,ϕ′∈Δ()φ,φ ∈ (X) is defined as: ‖ϕ−ϕ′‖TV=12∑x∈|ϕ(x)−ϕ′(x)|.\|φ-φ \|_TV= 12 _x |φ(x)-φ (x)|. (E.2) Definition E.2 (Dobrushin coefficient). The Dobrushin coefficient of a row-stochastic matrix P∈ℝ×P ^X×Y is defined as: α(P)=supϕ≠ϕ′∈Δ()‖ϕP−ϕ′P‖TV‖ϕ−ϕ′‖TV=supx,x′∈∥P(⋅|x)−P(⋅|x′)∥TV,α(P)= _φ≠φ ∈ (X) \|φ P-φ P\|_TV\|φ-φ \|_TV= _x,x \|P(·|x)-P(·|x )\|_TV, (E.3) the equality of the two expressions being due to Dobrushin (1956). The following lemma turns a Doeblin minorization, such as Assumption 1, into a bound on the Dobrushin coefficient, hence into a contraction in span seminorm by Lemma E.6. Lemma E.3 (Doeblin and Dobrushin coefficients). Let X and Y be finite and let P be a row-stochastic matrix from X to Y. If there exist β∈(0,1]β∈(0,1] and ν∈Δ()ν∈ (Y) with P(y|x)≥βν(y)P(y|x)≥βν(y) for all (x,y)∈×(x,y) ×Y, then α(P)≤1−βα(P)≤ 1-β. Proof. This is classical (Seneta, 2006, §4.3); we recall the argument. Since |a−b|=a+b−2mina,b|a-b|=a+b-2 \a,b\ and both rows sum to one, ∥P(⋅|x)−P(⋅|x′)∥TV=1−∑yminP(y|x),P(y|x′)\|P(·|x)-P(·|x )\|_TV=1- _y \P(y|x),P(y|x )\ for all x,x′∈x,x . Each minimum is at least βν(y)βν(y) by the minorization, so the sum is at least β and every row distance is at most 1−β1-β. ∎ Lemma E.4 (The terminal action does not affect the contraction). Let n≥1n≥ 1, let π1,…,πn _1,…, _n be policies and K:=Kπ1⋯KπnK:=K_ _1·s K_ _n, the chain taking its first step with Kπ1K_ _1 (Convention E.1). If πk≥γkμ _k≥ _kμ for k∈ 1,n−1 ∈ 1,n-1 and Kμn(⋅|s,a)≥βνK^n_μ(·|s,a)≥βν for every (s,a)(s,a), then α(K)≤1−β∏k=1n−1γkα(K)≤ 1-β _k=1^n-1 _k. Proof. The rows of K factor as K(s′,a′|x)=A(s′|x)πn(a′|s′)K(s ,a |x)=A(s |x)\, _n(a |s ), where A(⋅|x)A(·|x) is the law of the n-th state. The terminal factor being common to all rows, 1−α(K)=min∑s′x,yminA(s′|x),A(s′|y)=1−α(A)1-α(K)= _x,y _s \A(s |x),A(s |y)\=1-α(A). Only the n−1n-1 intermediate actions enter A, so the transport inequality Pπk≥γkPμP_ _k≥ _kP_μ gives A(⋅|x)≥(∏k=1n−1γk)Kμn(⋅|x)≥β(∏k=1n−1γk)νA(·|x)≥ ( _k=1^n-1 _k )K^n_μ(·|x)≥β ( _k=1^n-1 _k )ν, and Lemma E.3 concludes. ∎ Definition E.3 (Operator span seminorm). The operator span seminorm of a linear operator A:ℝp→ℝqA:R^p ^q is defined as: ‖A‖sp,op=sup‖x‖sp≤1‖Ax‖sp∈ℝ+∪+∞.\|A\|_sp,op= _\|x\|_sp≤ 1\|Ax\|_sp ^+∪\+∞\. (E.4) Lemma E.5 (Operator span seminorm). Let A:ℝp→ℝqA:R^p ^q be linear. The following two conditions are equivalent: (i) ‖A‖sp,op<+∞\|A\|_sp,op<+∞; (i) A⋅Ker∥⋅∥sp⊆Ker∥⋅∥spA·Ker\|·\|_sp \|·\|_sp, i.e. A∈ℝA1 1, i.e. A has constant row sums. When they hold, for every f∈ℝpf ^p and every linear map B:ℝp′→ℝpB:R^p ^p satisfying (i), ‖Af‖sp≤‖A‖sp,op‖f‖sp,‖AB‖sp,op≤‖A‖sp,op‖B‖sp,op.\|Af\|_sp≤\|A\|_sp,op\|f\|_sp, \|AB\|_sp,op≤\|A\|_sp,op\|B\|_sp,op. (E.5) Proof. Assume A∉ℝA1 1, i.e. ‖A‖sp>0\|A1\|_sp>0. For every c>0c>0 the vector cc1 satisfies ‖c‖sp=0≤1\|c1\|_sp=0≤ 1, while ‖A(c)‖sp=c‖A‖sp→+∞\|A(c1)\|_sp=c\|A1\|_sp→+∞ as c→+∞c→+∞. Hence ‖A‖sp,op=+∞\|A\|_sp,op=+∞. Conversely, the span seminorm vanishes exactly on the constant vectors, so it induces a norm, still denoted ∥⋅∥sp\|·\|_sp, on the quotient spaces Ep:=ℝp/ℝE_p:=R^p/R1 and Eq:=ℝq/ℝE_q:=R^q/R1. Under (i), the factorization theorem gives that A induces a well-defined linear map A~:Ep→Eq A:E_p→ E_q, A~[f]=[Af] A[f]=[Af]. By construction ‖Af‖sp=‖A~[f]‖sp\|Af\|_sp=\| A[f]\|_sp and ‖f‖sp=‖[f]‖sp\|f\|_sp=\|[f]\|_sp, so ‖A‖sp,op\|A\|_sp,op is the operator norm of A~ A between the normed spaces (Ep,∥⋅∥sp)(E_p,\|·\|_sp) and (Eq,∥⋅∥sp)(E_q,\|·\|_sp), which is finite since EpE_p is finite-dimensional. Finally, the first inequality in (E.5) is the operator-norm inequality ‖A~[f]‖sp≤‖A~‖‖[f]‖sp\| A[f]\|_sp≤\| A\|\,\|[f]\|_sp. For the second, ABAB also satisfies (i) and induces the map A~B~ A B, and operator norms are submultiplicative. ∎ Lemma E.6 (Span seminorm of a row-stochastic matrix). Let P∈ℝq×pP ^q× p be row-stochastic. Then its operator span seminorm equals its Dobrushin coefficient, ‖P‖sp,op=α(P)\|P\|_sp,op=α(P), and consequently ‖Pf‖sp≤α(P)‖f‖sp\|Pf\|_sp≤α(P)\|f\|_sp for every f∈ℝpf ^p. Proof. Since P is row-stochastic, P=P1=1, so P satisfies condition (i) of Lemma E.5: ‖P‖sp,op\|P\|_sp,op is finite, and the last claim follows from (E.5) once the equality is proved. For f∈ℝpf ^p we have ‖Pf‖sp=maxi,j(Pf)i−(Pf)j=max∑ki,j(Pik−Pjk)fk\|Pf\|_sp= _i,j \(Pf)_i-(Pf)_j \= _i,j _k (P_ik-P_jk )f_k. Fix i,ji,j, let ck:=Pik−Pjkc_k:=P_ik-P_jk, and let I+I^+ (resp. I−I^-) be the set of indices k with ck≥0c_k≥ 0 (resp. ck<0c_k<0). Since P is row-stochastic, ∑kck=0 _kc_k=0, hence C:=∑k∈I+ck=−∑k∈I−ck=12∑k|ck|C:= _k∈ I^+c_k=- _k∈ I^-c_k= 12 _k|c_k|, and with f¯:=maxkfk f:= _kf_k and f¯:=minkfk f:= _kf_k, ∑kckfk=∑k∈I+ckfk+∑k∈I−ckfk≤Cf¯−Cf¯=C‖f‖sp _kc_kf_k= _k∈ I^+c_kf_k+ _k∈ I^-c_kf_k\ ≤\ C f-C f=C\|f\|_sp. Maximizing over (i,j)(i,j) and using Definitions E.1 and E.2 gives ‖Pf‖sp≤12max∑ki,j|Pik−Pjk|‖f‖sp=α(P)‖f‖sp\|Pf\|_sp≤ 12 _i,j _k|P_ik-P_jk|\,\|f\|_sp=α(P)\|f\|_sp, i.e. ‖P‖sp,op≤α(P)\|P\|_sp,op≤α(P). If α(P)=0α(P)=0 there is nothing to prove, so assume α(P)>0α(P)>0 and let (i∗,j∗)(i^*,j^*) attain the maximum defining α(P)α(P). The coefficients ck:=Pi∗k−Pj∗kc_k:=P_i^*k-P_j^*k are not all zero and sum to zero, so they take both signs, and the vector f∗f^* defined by fk∗:=1f^*_k:=1 if ck≥0c_k≥ 0 and fk∗:=0f^*_k:=0 otherwise is non-constant, with ‖f∗‖sp=1\|f^*\|_sp=1. Then ‖Pf∗‖sp≥(Pf∗)i∗−(Pf∗)j∗=∑k∈I+ck=12∑k|Pi∗k−Pj∗k|=α(P)\|Pf^*\|_sp\ ≥\ (Pf^*)_i^*-(Pf^*)_j^*= _k∈ I^+c_k= 12 _k|P_i^*k-P_j^*k|=α(P), hence ‖P‖sp,op≥α(P)\|P\|_sp,op≥α(P). ∎ E.3 A mean value inequality for the span seminorm Under Assumption 2, the regularized value function Ω∗ ^* is differentiable with a Lipschitz gradient (Lemma F.1.(i)), and the regularized Bellman operator ΩT_ is continuously differentiable with Jacobian KπΩ,QK_ _ ,Q (Corollary F.1). This Jacobian is row-stochastic, hence has constant row sums, which allows a mean value argument in span seminorm. Proposition E.1 is the classical mean value inequality for curves in a normed space (Dieudonné, 1960, §8.5), applied in the quotient ℝq/ℝR^q/R1, on which ∥⋅∥sp\|·\|_sp is a norm; condition (H2) is what makes the Jacobians act on that quotient, and we include a proof for completeness. For linear Markov operators, the contraction rate in this seminorm, known as Hopf’s oscillation seminorm, is the Dobrushin coefficient (Lemma E.6); see Gaubert and Qu (2015) for Markov operators on cones and De Pasquale et al. (2024) for the corresponding semicontraction theory. Throughout this section, for Q,Q′∈ℝpQ,Q ^p we write Q(t):=Q′+t(Q−Q′)Q(t):=Q +t(Q-Q ), t∈[0,1]t∈[0,1], for the segment [Q′,Q][Q ,Q] from Q′Q to Q, as in Definition G.1. Proposition E.1 (Mean value inequality in span seminorm). Let U⊆ℝpU ^p be open, let Φ:U→ℝq :U ^q, and let Q,Q′∈UQ,Q ∈ U be such that [Q′,Q]⊆U[Q ,Q] U. Assume: (H1) Φ is differentiable at every point of [Q′,Q][Q ,Q]; (H2) for every t∈[0,1]t∈[0,1], the Jacobian DΦ(Q(t))D (Q(t)) has constant row sums, i.e. DΦ(Q(t))∈ℝD (Q(t))1 1. Then ‖Φ(Q)−Φ(Q′)‖sp≤supt∈(0,1)‖DΦ(Q(t))‖sp,op‖Q−Q′‖sp, \| (Q)- (Q ) \|_sp\ ≤\ _t∈(0,1) \|D (Q(t)) \|_sp,op\,\|Q-Q \|_sp, (E.6) and in particular with the supremum taken over the closed segment. Proof. Let π:ℝq→Eq:=ℝq/ℝπ:R^q→ E_q:=R^q/R1 be the quotient map, EqE_q being normed by ∥⋅∥sp\|·\|_sp as in the proof of Lemma E.5, and let h:=Q−Q′h:=Q-Q . By (H1) the curve g(t):=π(Φ(Q(t)))g(t):=π( (Q(t))) is differentiable on [0,1][0,1] with g′(t)=π(DΦ(Q(t))h)g (t)=π(D (Q(t))h), and by (H2) and (E.5), ‖g′(t)‖sp=‖DΦ(Q(t))h‖sp≤‖DΦ(Q(t))‖sp,op‖h‖sp\|g (t)\|_sp=\|D (Q(t))h\|_sp≤\|D (Q(t))\|_sp,op\|h\|_sp. The mean value inequality for differentiable curves in a normed space (Dieudonné, 1960, §8.5) gives ‖g(1)−g(0)‖sp≤supt∈(0,1)‖g′(t)‖sp\|g(1)-g(0)\|_sp≤ _t∈(0,1)\|g (t)\|_sp, which is (E.6) since ‖g(1)−g(0)‖sp=‖Φ(Q)−Φ(Q′)‖sp\|g(1)-g(0)\|_sp=\| (Q)- (Q )\|_sp. ∎ Corollary E.1 (1C^1 case: an exact averaged kernel). If moreover Φ is 1C^1 on a neighborhood of [Q′,Q][Q ,Q], then Φ(Q)−Φ(Q′)=A¯(Q−Q′),A¯:=∫01DΦ(Q(t))t. (Q)- (Q )= A\,(Q-Q ), A:= _0^1D (Q(t))\,dt. A¯ A is the average of the Jacobians of Φ along the segment; it has constant row sums and ‖A¯‖sp,op≤supt∈(0,1)‖DΦ(Q(t))‖sp,op\| A\|_sp,op≤ _t∈(0,1)\|D (Q(t))\|_sp,op. Proof. The identity is the fundamental theorem of calculus applied coordinatewise to t↦Φ(Q(t))t (Q(t)), whose derivative DΦ(Q(t))(Q−Q′)D (Q(t))(Q-Q ) is continuous in t. Integrating (H2) gives A¯=∫01DΦ(Q(t))t∈ℝ A1= _0^1D (Q(t))1\,dt 1. Finally, for every f∈ℝpf ^p, the triangle inequality gives ‖A¯f‖sp≤∫01‖DΦ(Q(t))f‖spt≤supt∈(0,1)‖DΦ(Q(t))‖sp,op‖f‖sp\| Af\|_sp≤ _0^1\|D (Q(t))f\|_sp\,dt≤ _t∈(0,1)\|D (Q(t))\|_sp,op\,\|f\|_sp. ∎ Remark E.1. For Φ=Ω =T_ , which is continuously differentiable with Jacobian KπΩ,QK_ _ ,Q at Q (Corollary F.1), Corollary E.1 applies on every segment. The state-action kernel is linear in the policy, so A¯=∫01KπΩ,Q(t)t A= _0^1K_ _ ,Q(t)\,dt is the state-action kernel of the average-path policy π¯Ω,[Q,Q′] π_ ,[Q,Q ], and the identity is Lemma G.2.(i). Appendix F Proofs for Section 2 Lemma F.1 (Properties of Ω∗ ^*). For each s∈s , let Ωs:Δ()→ℝ∪+∞ _s: (A) ∪\+∞\ be proper, closed and strongly convex, and let σ>0σ>0 be a common modulus of strong convexity of the Ωs _s with respect to ∥⋅∥1\|·\|_1, which exists by equivalence of norms on ℝR^A and finiteness of S; let τ≥0τ≥ 0 and Q,Q′∈ℝ×Q,Q ^S×A. (i) Greedy policy: if τ>0τ>0, Ω∗(⋅)(s) ^*(·)(s) is differentiable, ∇Ω∗(Q)(s)=πΩ,Q(⋅|s)∇ ^*(Q)(s)= _ ,Q(·|s) is the unique maximizer in its definition, and ∥πΩ,Q(⋅|s)−πΩ,Q′(⋅|s)∥1≤12τσ∥Qs−Qs′∥,sp\| _ ,Q(·|s)- _ ,Q (·|s)\|_1≤ 12τσ\|Q_s-Q _s\|_A,sp. (i) Boundedness: if LΩs≤Ωs≤UΩsL_ _s≤ _s≤ U_ _s on dom(Ωs)dom( _s), then Ω∗(Q)(s)≤maxaQs(a)−τLΩs ^*(Q)(s)≤ _aQ_s(a)-τ L_ _s; if moreover δa∈dom(Ωs) _a ( _s) for every a∈a , then Ω∗(Q)(s)≥maxaQs(a)−τUΩs ^*(Q)(s)≥ _aQ_s(a)-τ U_ _s. (i) Translation equivariance: Ω∗(Q+c)=Ω∗(Q)+c ^*(Q+c1)= ^*(Q)+c1 for every c∈ℝc . (iv) Monotonicity: if Qs≤Qs′Q_s≤ Q _s, then Ω∗(Q)(s)≤Ω∗(Q′)(s) ^*(Q)(s)≤ ^*(Q )(s). (v) Non-expansiveness in ∥⋅∥∞\|·\|_∞: ‖Ω∗(Q)−Ω∗(Q′)‖,∞≤‖Q−Q′‖×,∞\| ^*(Q)- ^*(Q )\|_S,∞≤\|Q-Q \|_S×A,∞. (vi) Non-expansiveness in ∥⋅∥sp\|·\|_sp: ‖Ω∗(Q)−Ω∗(Q′)‖,sp≤‖Q−Q′‖sp\| ^*(Q)- ^*(Q )\|_S,sp≤\|Q-Q \|_sp. (vii) Sub-affinity in ∥⋅∥sp\|·\|_sp: ‖Ω∗(Q)‖,sp≤‖Q‖sp+‖Ω∗(0)‖,sp\| ^*(Q)\|_S,sp≤\|Q\|_sp+\| ^*(0)\|_S,sp. Proof of Lemma F.1. (i). Ω∗(⋅)(s) ^*(·)(s) is the convex conjugate of the closed, τστσ-strongly convex function τΩsτ _s, so it is differentiable, its gradient is the unique maximizer, and this gradient is 1τσ 1τσ-Lipschitz from ∥⋅∥∞\|·\|_∞ to ∥⋅∥1\|·\|_1 (Kakade et al., 2012); see Hiriart-Urruty and Lemaréchal (2001, §E.4) for the Euclidean case and Geist et al. (2019, Prop. 1) for Ωs _s finite on Δ() (A). Adding a constant to Qs′Q _s shifts the objective by that constant and leaves the maximizer unchanged, so the bound holds with ‖Qs−Qs′−c‖∞\|Q_s-Q _s-c1\|_∞ for every c∈ℝc , and minc∈ℝ‖Qs−Qs′−c‖∞=12‖Qs−Qs′‖,sp _c \|Q_s-Q _s-c1\|_∞= 12\|Q_s-Q _s\|_A,sp. (i). This refines Geist et al. (2019, Prop. 1), where Ωs _s is finite on the whole simplex. For every πs∈dom(Ωs) _s ( _s), ⟨πs,Qs⟩≤maxaQs(a) _s,Q_s ≤ _aQ_s(a) and τΩs(πs)≥τLΩsτ _s( _s)≥τ L_ _s; taking the supremum gives the upper bound. For the lower bound, let a⋆∈argmaxaQs(a)a ∈ _aQ_s(a): since δa⋆∈dom(Ωs) _a ( _s), Ω∗(Q)(s)≥maxaQs(a)−τΩs(δa⋆)≥maxaQs(a)−τUΩs ^*(Q)(s)≥ _aQ_s(a)-τ _s( _a )≥ _aQ_s(a)-τ U_ _s. (i)–(iv). For every πs∈Δ() _s∈ (A), ⟨πs,Qs+c⟩=⟨πs,Qs⟩+c _s,Q_s+c1 = _s,Q_s +c, and ⟨πs,Qs⟩≤⟨πs,Qs′⟩ _s,Q_s ≤ _s,Q _s when Qs≤Qs′Q_s≤ Q _s since πs≥0 _s≥ 0; take the supremum over πs _s. (v). This is the argument of Crandall and Tartar (1980). With C:=‖Q−Q′‖×,∞C:=\|Q-Q \|_S×A,∞, we have Q′−C≤Q≤Q′+CQ -C1≤ Q≤ Q +C1, and (i)–(iv) give Ω∗(Q′)−C≤Ω∗(Q)≤Ω∗(Q′)+C ^*(Q )-C1≤ ^*(Q)≤ ^*(Q )+C1. (vi). The same argument with c:=mins,a(Q−Q′)(s,a)c:= _s,a(Q-Q )(s,a) and C:=maxs,a(Q−Q′)(s,a)C:= _s,a(Q-Q )(s,a) gives Ω∗(Q′)+c≤Ω∗(Q)≤Ω∗(Q′)+C ^*(Q )+c1≤ ^*(Q)≤ ^*(Q )+C1, whence ‖Ω∗(Q)−Ω∗(Q′)‖,sp≤C−c=‖Q−Q′‖sp\| ^*(Q)- ^*(Q )\|_S,sp≤ C-c=\|Q-Q \|_sp. Maps that are monotone and translation equivariant are called topical, and are non-expansive for both norms (Lemmens and Nussbaum, 2012). (vii). By the triangle inequality and (vi) with Q′=0Q =0, ‖Ω∗(Q)‖,sp≤‖Ω∗(Q)−Ω∗(0)‖,sp+‖Ω∗(0)‖,sp≤‖Q‖sp+‖Ω∗(0)‖,sp\| ^*(Q)\|_S,sp≤\| ^*(Q)- ^*(0)\|_S,sp+\| ^*(0)\|_S,sp≤\|Q\|_sp+\| ^*(0)\|_S,sp. ∎ Remark F.1 (Strict convexity suffices). We keep strong convexity in Assumption 2, as is standard since Geist et al. (2019), but our results only use part of it. Uniqueness and continuity of the greedy map, and continuous differentiability of Ω∗ ^*, i.e. Lemma F.1.(i) without its Lipschitz constant, hold under strict convexity alone: τΩsτ _s is closed with bounded domain, so Ω∗(⋅)(s) ^*(·)(s) is finite and convex, its subdifferential, the set of maximizers, is a singleton (Hiriart-Urruty and Lemaréchal, 2001, §E.4), and a differentiable convex function is continuously differentiable (Rockafellar, 1970, Cor. 25.5.1). This is all that Lemma G.2, the mean value inequality (Propositions E.1 and E.1), the existence of the fixed point (Theorem G.1) and the finite-time bounds of Section 4 require, so they hold verbatim for strictly convex regularizers; the modulus only enters the Lipschitz constant of Lemma F.1.(i). For instance, the α-divergences with α>2α>2 are strictly but not strongly convex (Theorem K.1.(i)); they fail (P1), but their ϵε-floored versions satisfy it (Lemma G.10.(i) only needs a strictly convex base), and our results apply to them. Remark F.2 (Span convergence and centering). Adding a constant to QsQ_s shifts the objective defining Ω∗(Q)(s) ^*(Q)(s) by that constant, so πΩ,Q+c=πΩ,Q _ ,Q+c1= _ ,Q: as for the unregularized relative Q-function (Puterman, 1994), the greedy policy depends on Q only modulo constants, which is why the analysis is carried out in ∥⋅∥sp\|·\|_sp. The centering projection Π turns span convergence into convergence in ∥⋅∥∞\|·\|_∞ to the representative Π[QΩ∗] [Q^*_ ]: since Π[Q]−Π[Q′] [Q]- [Q ] has zero mean, ‖Π[Q]−Π[Q′]‖∞≤‖Π[Q]−Π[Q′]‖sp=‖Q−Q′‖sp\| [Q]- [Q ]\|_∞≤\| [Q]- [Q ]\|_sp=\|Q-Q \|_sp. A centering step plays the same role for asynchronous average-reward Q-learning in Chen and Nanda (2026), where it upgrades span-seminorm convergence to pointwise convergence at the same rate. Corollary F.1 (Properties of ΩT_ ). Under the assumptions of Lemma F.1, ΩT_ is monotone, translation equivariant, and non-expansive in both ∥⋅∥∞\|·\|_∞ and ∥⋅∥sp\|·\|_sp, and ‖Ω(Q)‖sp≤‖Q‖sp+‖Ω∗(0)‖,sp+‖r‖sp\|T_ (Q)\|_sp≤\|Q\|_sp+\| ^*(0)\|_S,sp+\|r\|_sp for every Q, where Ω∗(0)(s)=−τminΩs ^*(0)(s)=-τ _s; in particular ‖Ω∗(0)‖,sp=0\| ^*(0)\|_S,sp=0 when minΩs _s does not depend on s, as for divergences to μ. If τ>0τ>0, ΩT_ is continuously differentiable, with Jacobian DΩ(Q)=KπΩ,QDT_ (Q)=K_ _ ,Q. Proof. Ω(Q)=r+PΩ∗(Q)T_ (Q)=r+P\, ^*(Q), with P the transition kernel, row-stochastic from ×S×A to S. Since P≥0P≥ 0 and P=P1=1, monotonicity and translation equivariance follow from Lemma F.1.(i)–(iv), and non-expansiveness from (v)–(vi), P being non-expansive in ∥⋅∥∞\|·\|_∞ and, as α(P)≤1α(P)≤ 1 (Lemma E.6), in ∥⋅∥sp\|·\|_sp. The bound follows from (vii) and the triangle inequality, and Ω∗(0)(s)=supπs−τΩs(πs)=−τminΩs ^*(0)(s)= _ _s\-τ _s( _s)\=-τ _s, the minimum being attained since Ωs _s is closed and Δ() (A) is compact. For a divergence to μ, minΩs=Ωs(μs)=0 _s= _s( _s)=0 for every s. Finally, by Lemma F.1.(i), ∂Ω(Q)(s,a)/∂Q(s′,a′)=p(s′|s,a)πΩ,Q(a′|s′)=KπΩ,Q(s′,a′|s,a) _ (Q)(s,a)/∂ Q(s ,a )=p(s |s,a)\, _ ,Q(a |s )=K_ _ ,Q(s ,a |s,a), which is continuous in Q. ∎ Appendix G Proofs for Section 3 G.1 Proofs for Subsection 3.2 We recall Definition 2 together with the average-path policy used in the proofs. Definition G.1 (Mixing-regularization objects). Let Ω satisfy Assumption 2, let μ be the mixing policy of Assumption 1, let Q,Q′∈ℝ×Q,Q ^S×A, Q(t):=tQ+(1−t)Q′Q(t):=tQ+(1-t)Q for t∈[0,1]t∈[0,1], and R≥0R≥ 0. (i) Average-path policy: π¯Ω,[Q,Q′]:=∫01πΩ,Q(t)t π_ ,[Q,Q ]:= _0^1 _ ,Q(t)\,dt is the mean greedy policy along the segment [Q′,Q][Q ,Q]. (i) Average-path radius: R[Q,Q′]:=maxt∈[0,1]‖Q(t)‖aspR[Q,Q ]:= _t∈[0,1]\|Q(t)\|_asp is the largest action span along the segment; by convexity of ∥⋅∥asp\|·\|_asp, it equals max‖Q‖asp,‖Q′‖asp \\|Q\|_asp,\|Q \|_asp\. (i) Domination profile: γΩ(R):=infπΩ,Q(a|s)/μ(a|s)∣(s,a)∈×,∥Q∥asp≤R _ (R):= \ _ ,Q(a|s)/μ(a|s) (s,a) ×A,\ \|Q\|_asp≤ R \ is the largest factor such that πΩ,Q≥γΩ(R)μ _ ,Q≥ _ (R)\,μ for every Q of action span at most R. It lies in [0,1][0,1], is non-increasing in R, and governs contraction and exploration (Lemma G.2). (iv) Regularization modulus: ΛΩ(R):=sup‖gQ‖,sp:‖Q‖asp≤R _ (R):= \\|g_Q\|_S,sp:\ \|Q\|_asp≤ R \ is the largest variation across states of the regularization gap gQ(s):=Ω∗(Q)(s)−maxaQ(s,a)g_Q(s):= ^*(Q)(s)- _aQ(s,a) between the regularized and the hard maximum. It is finite, non-decreasing in R (Lemma G.1), and governs the bias (Proposition G.6). The following lemma gives some basic properties of the regularization modulus defined in Definition G.1.(iv). Lemma G.1 (Properties of ΛΩ _ ). Under Assumption 2, the supremum defining ΛΩ(R) _ (R) is attained and finite for every R≥0R≥ 0, ΛΩ _ is non-decreasing, and ΛΩ(0)=τ(maxsΩ¯s−Ω¯) _ (0)=τ ( _s _s- ), where Ω¯s:=minπs∈Δ()Ωs(πs) _s:= _ _s∈ (A) _s( _s) and Ω¯:=minsΩ¯s := _s _s. Proof. Since Ω∗(Q)(s) ^*(Q)(s) depends on Q only through QsQ_s, Lemma F.1.(i) applied state by state shows that Ω∗(Q)(s) ^*(Q)(s) and maxaQs(a) _aQ_s(a) both increase by c when QsQ_s is replaced by Qs+cQ_s+c1: gQg_Q is invariant under the addition of a constant to each QsQ_s separately. We may therefore restrict the supremum to the Q with maxaQs(a)=0 _aQ_s(a)=0 for every s, for which ‖Q‖asp≤R\|Q\|_asp≤ R reads Qs∈[−R,0]Q_s∈[-R,0]^A: a compact set. On it, Q↦‖gQ‖,spQ \|g_Q\|_S,sp is continuous, since Ω∗ ^* is non-expansive in ∥⋅∥∞\|·\|_∞ (Lemma F.1.(v)) and Q↦maxaQ(⋅,a)Q _aQ(·,a) is 11-Lipschitz; hence the supremum is attained and finite. It is non-decreasing in R because the feasible set grows with R. Finally, for R=0R=0 each QsQ_s is a constant csc_s1, so Ω∗(Q)(s)=cs+Ω∗(0)(s)=cs−τΩ¯s ^*(Q)(s)=c_s+ ^*(0)(s)=c_s-τ _s, the minimum defining Ω¯s _s being attained since Ωs _s is closed on the compact simplex. Thus gQ=−τ(Ω¯s)s∈g_Q=-τ( _s)_s , whose span is τ(maxsΩ¯s−Ω¯)τ( _s _s- ). ∎ Remark G.1 (Boundedness of the regularization modulus). Let Ω¯:=maxs∈maxa∈Ωs(δa) := _s _a _s( _a), Ω¯=minsΩ¯s = _s _s and ΔΩ:=Ω¯−Ω¯ _ := - . If Ω¯<+∞ <+∞, i.e. if every Dirac lies in the effective domain, then Lemma F.1.(i) applied state-wise with LΩs=Ω¯L_ _s= and UΩs=maxaΩs(δa)U_ _s= _a _s( _a) gives ΛΩ(R)≤τΔΩ _ (R)≤τ _ for every R: the modulus is bounded by a radius-independent constant and all the bounds below reduce to their constant counterparts. The α-divergences with α≤0α≤ 0, including the forward KL, and the floor-wrapped regularizers of Definition G.4 exclude every Dirac from their effective domain, so Ω¯=+∞ =+∞, while ΛΩ(R) _ (R) stays finite (Theorem K.1.(v) and Lemma G.10.(v)); this is what makes the radius-dependent form necessary. Lemma G.2 (Mixing properties of Ω -regularization). Let Assumptions 1 and 2 hold and let Q,Q′∈ℝ×Q,Q ^S×A. Then: (i) Average-path factorization of the conjugate and of the operator: for every s∈s , Ω∗(Q)(s)−Ω∗(Q′)(s)=⟨π¯Ω,[Q,Q′](⋅|s),Qs−Qs′⟩ ^*(Q)(s)- ^*(Q )(s)= π_ ,[Q,Q ](·|s),Q_s-Q _s and Ω(Q)−Ω(Q′)=Kπ¯Ω,[Q,Q′](Q−Q′)T_ (Q)-T_ (Q )=K_ π_ ,[Q,Q ](Q-Q ). (i) Domination of the average-path policy: π¯Ω,[Q,Q′](a|s)≥γΩ(R[Q,Q′])μ(a|s) π_ ,[Q,Q ](a|s)≥ _ (R[Q,Q ])\,μ(a|s) for every (s,a)∈×(s,a) ×A. (i) m-step minorization: if π1,…,πm _1,…, _m are policies with πk≥γkμ _k≥ _kμ for k∈ 1,m ∈ 1,m , the m-step kernel Kπ1:πm:=Kπ1⋯KπmK_ _1: _m:=K_ _1·s K_ _m satisfies Kπ1:πm(s′,a′|s,a)≥βm(∏k=1mγk)η(s′,a′),∀(s,a),(s′,a′)∈×.K_ _1: _m(s ,a |s,a)≥ _m ( _k=1^m _k )η(s ,a ), ∀(s,a),(s ,a ) ×A. (iv) Radius-dependent m-step contraction: ‖Ωm(Q)−Ωm(Q′)‖sp≤(1−βm∏k=1m−1γΩ(R[Ωk(Q),Ωk(Q′)]))‖Q−Q′‖sp. \|T^m_ (Q)-T^m_ (Q ) \|_sp≤ (1- _m _k=1^m-1 _ (R[T^k_ (Q),T^k_ (Q )] ) )\|Q-Q \|_sp. (v) State-action coverage: let (Qn)n≥0(Q_n)_n≥ 0 be Q-functions and (sn,an)n≥0(s_n,a_n)_n≥ 0 a trajectory with an∼πΩ,Qn(⋅|sn)a_n _ ,Q_n(·|s_n) and sn+1∼p(⋅|sn,an)s_n+1 p(·|s_n,a_n); let ℱn:=σ(Q0,(s0,a0),…,(sn,an))F_n:=σ(Q_0,(s_0,a_0),…,(s_n,a_n)) and assume that QnQ_n is ℱn−1F_n-1-measurable for n≥1n≥ 1. If t≥0t≥ 0 and γ1,…,γm _1,…, _m are non-negative ℱtF_t-measurable random variables with πΩ,Qt+k≥γkμ _ ,Q_t+k≥ _kμ almost surely for k∈ 1,m ∈ 1,m , then almost surely ℙ((st+m,at+m)=(s,a)∣ℱt)≥βm(∏k=1mγk)η(s,a),∀(s,a)∈×.P ((s_t+m,a_t+m)=(s,a) _t )≥ _m ( _k=1^m _k )η(s,a), ∀(s,a) ×A. (vi) One-step operator gap: for every Q∈ℝ×Q ^S×A, ‖Ω(Q)−(Q)‖sp≤α(P)ΛΩ(‖Q‖asp)≤ΛΩ(‖Q‖asp),\|T_ (Q)-T(Q)\|_sp≤α(P)\, _ (\|Q\|_asp )≤ _ (\|Q\|_asp ), where α(P)α(P) is the Dobrushin coefficient (Definition E.2) of the rectangular matrix (p(s′|s,a))(s,a),s′(p(s |s,a))_(s,a),s . In particular, under the assumptions of Remark G.1, ‖Ω(Q)−(Q)‖sp≤α(P)τΔΩ≤τΔΩ\|T_ (Q)-T(Q)\|_sp≤α(P)τ _ ≤τ _ uniformly in Q. Proof of Lemma G.2. Throughout the proof we write π¯=π¯Ω,[Q,Q′] π= π_ ,[Q,Q ], γk _k for the domination coefficients of each item and η(s′,a′)=ν(s′)μ(a′|s′)η(s ,a )=ν(s )μ(a |s ) for the minorizing measure of Assumption 1. Points (i), (i), (iv) and (v) use τ>0τ>0 through Lemma F.1.(i), whereas (i) requires no regularity at all and (vi) only uses Assumption 2 through the finiteness of ΛΩ _ (Lemma G.1). (i). Let g(t):=Ω∗(Q(t))(s)g(t):= ^*(Q(t))(s) for t∈[0,1]t∈[0,1]. Since Ω∗(Q)(s) ^*(Q)(s) depends on Q only through QsQ_s, Lemma F.1.(i) and the chain rule give that g is continuously differentiable, with g′(t)=⟨πΩ,Q(t)(⋅|s),Qs−Qs′⟩g (t)= _ ,Q(t)(·|s),Q_s-Q _s . The fundamental theorem of calculus and the linearity of the inner product then give Ω∗(Q)(s)−Ω∗(Q′)(s)=g(1)−g(0)=∫01⟨πΩ,Q(t)(⋅|s),Qs−Qs′⟩dt=⟨π¯(⋅|s),Qs−Qs′⟩, ^*(Q)(s)- ^*(Q )(s)=g(1)-g(0)= _0^1 _ ,Q(t)(·|s),Q_s-Q _s \,dt= π(·|s),Q_s-Q _s , (G.1) which is the first equality. Since Ω=r+PΩ∗T_ =r+P\, ^*, the reward terms cancel and applying (G.1) at each next state s′∈s gives, for every (s,a)∈×(s,a) ×A, Ω(Q)(s,a)−Ω(Q′)(s,a) _ (Q)(s,a)-T_ (Q )(s,a) =∑s′p(s′|s,a)(Ω∗(Q)(s′)−Ω∗(Q′)(s′)) = _s p(s |s,a) ( ^*(Q)(s )- ^*(Q )(s ) ) =∑s′,a′p(s′|s,a)π¯(a′|s′)(Q−Q′)(s′,a′)=(Kπ¯(Q−Q′))(s,a), = _s ,a p(s |s,a)\, π(a |s )\,(Q-Q )(s ,a )= (K_ π(Q-Q ) )(s,a), which is the second equality. (i). By Definition G.1.(i), ‖Q(t)‖asp≤R[Q,Q′]\|Q(t)\|_asp≤ R[Q,Q ] for all t∈[0,1]t∈[0,1], and Definition G.1.(i) applied with R=R[Q,Q′]R=R[Q,Q ] then gives πΩ,Q(t)(a|s)≥γΩ(R[Q,Q′])μ(a|s) _ ,Q(t)(a|s)≥ _ (R[Q,Q ])μ(a|s) for all (s,a)(s,a) and all t∈[0,1]t∈[0,1], uniformly in t. Integrating this inequality against the uniform measure on [0,1][0,1] and using Definition G.1.(i) yields π¯(a|s)=∫01πΩ,Q(t)(a|s)t≥γΩ(R[Q,Q′])μ(a|s) π(a|s)= _0^1 _ ,Q(t)(a|s)\,dt≥ _ (R[Q,Q ])μ(a|s), which is the claim. (i). Set Γ1:m=∏k=1mγk _1:m= _k=1^m _k. First, for any policy π with π(a|s)≥γμ(a|s)π(a|s)≥γμ(a|s) we have the transport inequality Pπ(s′|s)=∑aπ(a|s)p(s′|s,a)≥γ∑aμ(a|s)p(s′|s,a)=γPμ(s′|s)P_π(s |s)= _aπ(a|s)p(s |s,a)≥γ _aμ(a|s)p(s |s,a)=γ P_μ(s |s). By definition of Kπ1:πm=Kπ1⋯KπmK_ _1: _m=K_ _1·s K_ _m and Convention E.1, the first action is the one supplied by the conditioning, the intermediate actions are drawn from π1,…,πm−1 _1,…, _m-1 and the terminal action from πm _m: Kπ1:πm(s′,a′|s,a)=πm(a′|s′)∑s1,…,sm−1p(s1|s,a)∏k=1m−1Pπk(sk+1|sk),K_ _1: _m(s ,a |s,a)= _m(a |s ) _s_1,…,s_m-1p(s_1|s,a) _k=1^m-1P_ _k(s_k+1|s_k), (G.2) with sm=s′s_m=s , and since all factors are non-negative, the bounds multiply: applying the transport inequality to the m−1m-1 intermediate factors and the domination hypothesis to the terminal factor πm(a′|s′)≥γmμ(a′|s′) _m(a |s )≥ _mμ(a |s ) gives Kπ1:πm(s′,a′|s,a) K_ _1: _m(s ,a |s,a) ≥Γ1:mμ(a′|s′)∑s1,…,sm−1p(s1|s,a)∏k=1m−1Pμ(sk+1|sk) ≥ _1:m\,μ(a |s ) _s_1,…,s_m-1p(s_1|s,a) _k=1^m-1P_μ(s_k+1|s_k) =Γ1:mμ(a′|s′)∑s1p(s1|s,a)Pμm−1(s′|s1) = _1:m\,μ(a |s ) _s_1p(s_1|s,a)\,P_μ^m-1(s |s_1) =Γ1:mμ(a′|s′)Kμm(s′|s,a) = _1:m\,μ(a |s )\,K_μ^m(s |s,a) ≥Γ1:mβmν(s′)μ(a′|s′)=Γ1:mβmη(s′,a′), ≥ _1:m\, _m\,ν(s )μ(a |s )= _1:m\, _m\,η(s ,a ), the second-to-last line by the definition Kμm(s′|s,a)=∑s1p(s1|s,a)Pμm−1(s′|s1)K^m_μ(s |s,a)= _s_1p(s_1|s,a)P_μ^m-1(s |s_1) and the last by Assumption 1. (iv). For k∈ 0,m−1 ∈ 0,m-1 write Qk:=Ωk(Q)Q_k:=T_ ^k(Q), Qk′:=Ωk(Q′)Q_k :=T_ ^k(Q ), Rk:=R[Qk,Qk′]R_k:=R[Q_k,Q_k ], π¯(k)=π¯Ω,[Qk,Qk′] π^(k)= π_ ,[Q_k,Q_k ] and γk=γΩ(Rk) _k= _ (R_k). (i) applied at level k gives the one-step identity Qk+1−Qk+1′=Ω(Qk)−Ω(Qk′)=Kπ¯(k)(Qk−Qk′)Q_k+1-Q_k+1 =T_ (Q_k)-T_ (Q_k )=K_ π^(k)(Q_k-Q_k ), where Kπ¯(k)K_ π^(k) is a row-stochastic state-action kernel. Iterating this identity from k=m−1k=m-1 down to k=0k=0, we obtain Ωm(Q)−Ωm(Q′)=Qm−Qm′=KQ,Q′(m)(Q−Q′),KQ,Q′(m):=Kπ¯(m−1)Kπ¯(m−2)⋯Kπ¯(0).T^m_ (Q)-T^m_ (Q )=Q_m-Q_m =K^(m)_Q,Q (Q-Q ), K^(m)_Q,Q :=K_ π^(m-1)K_ π^(m-2)·s K_ π^(0). Following Convention E.1, KQ,Q′(m)K^(m)_Q,Q is exactly the m-step state-action kernel Kπ¯(m−1):π¯(0)K_ π^(m-1): π^(0) associated with the sequence of policies π¯(m−1),…,π¯(0) π^(m-1),…, π^(0). By (i), each of them satisfies π¯(k)≥γkμ π^(k)≥ _kμ, so (i) applies and gives the Doeblin condition KQ,Q′(m)(s′,a′|s,a)≥βm(∏k=0m−1γk)η(s′,a′)K^(m)_Q,Q (s ,a |s,a)≥ _m ( _k=0^m-1 _k )η(s ,a ). Lemma E.4, applied with π1=π¯(m−1),…,πm=π¯(0) _1= π^(m-1),…, _m= π^(0), gives α(KQ,Q′(m))≤1−βm∏k=1m−1γΩ(Rk)α (K^(m)_Q,Q )≤ 1- _m _k=1^m-1 _ (R_k), and by Lemma E.6, ‖KQ,Q′(m)‖sp,op=α(KQ,Q′(m))\|K^(m)_Q,Q \|_sp,op=α (K^(m)_Q,Q ). Therefore ‖Ωm(Q)−Ωm(Q′)‖sp=‖KQ,Q′(m)(Q−Q′)‖sp≤(1−βm∏k=1m−1γΩ(Rk))‖Q−Q′‖sp. \|T^m_ (Q)-T^m_ (Q ) \|_sp= \|K^(m)_Q,Q (Q-Q ) \|_sp≤ (1- _m _k=1^m-1 _ (R_k) )\|Q-Q \|_sp. (v). Fix t≥0t≥ 0 and (s,a)∈×(s,a) ×A. For j∈ 0,m ∈ 0,m , let Γj:=∏k=j+1mγk _j:= _k=j+1^m _k, which is non-negative and ℱtF_t-measurable, hence ℱt+jF_t+j-measurable. We prove by backward induction on j that, a.s., [(st+m,at+m)=(s,a)|ℱt+j]≥ΓjKμm−j(s,a∣st+j,at+j).E [1_\(s_t+m,a_t+m)=(s,a)\\ |\ F_t+j ]\ ≥\ _j\,K^m-j_μ (s,a s_t+j,a_t+j ). The case j=mj=m is an equality. Assume the claim at j+1j+1. Conditionally on ℱt+jF_t+j, st+j+1∼p(⋅|st+j,at+j)s_t+j+1 p(·|s_t+j,a_t+j) and at+j+1∼πΩ,Qt+j+1(⋅|st+j+1)a_t+j+1 _ ,Q_t+j+1(·|s_t+j+1), where Qt+j+1Q_t+j+1 is ℱt+jF_t+j-measurable. By the tower property, the induction hypothesis and the ℱt+jF_t+j-measurability of Γj+1 _j+1, the left-hand side at level j is therefore at least Γj+1∑s′p(s′|st+j,at+j)∑a′πΩ,Qt+j+1(a′|s′)Kμm−j−1(s,a∣s′,a′) _j+1 _s p(s |s_t+j,a_t+j) _a _ ,Q_t+j+1(a |s )\,K^m-j-1_μ (s,a s ,a ) ≥Γj+1γj+1∑s′,a′Kμ(s′,a′|st+j,at+j)Kμm−j−1(s,a∣s′,a′)=ΓjKμm−j(s,a∣st+j,at+j), ≥ _j+1 _j+1 _s ,a K_μ(s ,a |s_t+j,a_t+j)\,K^m-j-1_μ (s,a s ,a )= _j\,K^m-j_μ (s,a s_t+j,a_t+j ), the inequality by the domination hypothesis and the non-negativity of Γj+1 _j+1 and of Kμm−j−1K^m-j-1_μ. Taking j=0j=0 and using that the m-step state-action kernel of μ satisfies Kμm(s′,a′|s,a)=Kμm(s′|s,a)μ(a′|s′)≥βmν(s′)μ(a′|s′)=βmη(s′,a′)K^m_μ(s ,a |s,a)=K^m_μ(s |s,a)μ(a |s )≥ _mν(s )μ(a |s )= _mη(s ,a ) by Assumption 1 gives the claim. (vi). Definition G.1.(iv) applied with R=‖Q‖aspR=\|Q\|_asp immediately gives ‖gQ‖,sp≤ΛΩ(‖Q‖asp)\|g_Q\|_S,sp≤ _ (\|Q\|_asp). The two operators ΩT_ and T share the reward term and differ only through the state-value function, so for all (s,a)(s,a), Ω(Q)(s,a)−(Q)(s,a)=∑s′p(s′|s,a)(Ω∗(Q)(s′)−maxa′Q(s′,a′))=(PgQ)(s,a).T_ (Q)(s,a)-T(Q)(s,a)= _s p(s |s,a) ( ^*(Q)(s )- _a Q(s ,a ) )=(Pg_Q)(s,a). Lemma E.6 gives ‖PgQ‖sp≤α(P)‖gQ‖,sp\|Pg_Q\|_sp≤α(P)\|g_Q\|_S,sp, so ‖Ω(Q)−(Q)‖sp≤α(P)ΛΩ(‖Q‖asp)\|T_ (Q)-T(Q)\|_sp≤α(P) _ (\|Q\|_asp). The second inequality follows from α(P)≤1α(P)≤ 1, and the last claim from Remark G.1. ∎ See 1 Proof of Lemma 1. Items (i) and (i) are Lemma G.2.(iv) and (v). The first claim of (i) is Lemma G.2.(vi) with α(P)≤1α(P)≤ 1, and the second is Proposition G.6.(i), whose right-hand side is finite under (P1) and the gain bounds are Proposition G.6.(i), the last one using ||≥2|S|≥ 2 and a common minimum. ∎ Remark G.2 (Only m regularized steps are needed). Let π1,…,πM _1,…, _M with M≥mM≥ m be given, and assume that only the last m of them are known to be regularized, say πk≥γkμ _k≥ _kμ for k∈ M−m+1,M ∈ M-m+1,M , with ΓM−m+1:M:=∏k=M−m+1Mγk _M-m+1:M:= _k=M-m+1^M _k. Then Lemma G.2.(i) still holds for the full M-step kernel, with the same parameter ΓM−m+1:Mβm _M-m+1:M _m and the same minorizing measure η. Writing Khead:=Kπ1:πM−mK_head:=K_ _1: _M-m for the kernel of the first M−mM-m steps and Ktail:=KπM−m+1:πMK_tail:=K_ _M-m+1: _M for that of the last m ones, we have, consistently with Convention E.1, Kπ1:πM=KheadKtail,i.e.Kπ1:πM(s′,a′|s,a)=∑(s~,a~)Khead(s~,a~|s,a)Ktail(s′,a′|s~,a~).K_ _1: _M=K_headK_tail, .e. K_ _1: _M(s ,a |s,a)= _( s, a)K_head( s, a|s,a)K_tail(s ,a | s, a). Lemma G.2.(i) applies to KtailK_tail and gives Ktail(s′,a′|s~,a~)≥ΓM−m+1:Mβmη(s′,a′)K_tail(s ,a | s, a)≥ _M-m+1:M _m\,η(s ,a ) uniformly in the starting pair (s~,a~)( s, a); the head, which requires no regularization, only contributes a factor 11 by row-stochasticity: Kπ1:πM(s′,a′|s,a)≥ΓM−m+1:Mβmη(s′,a′)∑(s~,a~)Khead(s~,a~|s,a)=ΓM−m+1:Mβmη(s′,a′).K_ _1: _M(s ,a |s,a)≥ _M-m+1:M _mη(s ,a ) _( s, a)K_head( s, a|s,a)= _M-m+1:M _mη(s ,a ). The same holds for the coverage bound of Lemma G.2.(v): if its domination hypothesis holds only for k∈ M−m+1,M ∈ M-m+1,M , then, almost surely, ℙ((st+M,at+M)=(s,a)∣ℱt)≥ΓM−m+1:Mβmη(s,a).P ((s_t+M,a_t+M)=(s,a) _t )≥ _M-m+1:M _m\,η(s,a). Indeed, Lemma G.2.(v) applied at time t+M−mt+M-m gives this lower bound for the probability conditional on ℱt+M−mF_t+M-m, and the tower property transfers it to ℱtF_t, the bound being deterministic: the conditional expectation plays the role of the row-stochastic head. Since the proof of (v) only uses the steps after time t+M−mt+M-m, the actions of the head may moreover follow any adapted rule. G.2 Strict domination Lemma G.3 ((P1) is full support). Under Assumption 2, Ω satisfies (P1) if and only if πΩ,Q(a|s)>0 _ ,Q(a|s)>0 for every Q and every (s,a)(s,a). Proof. If (P1) holds, πΩ,Q≥γΩ(‖Q‖asp)μ>0 _ ,Q≥ _ (\|Q\|_asp)\,μ>0. Conversely, πΩ,Q(⋅|s)=∇Ωs∗(Qs) _ ,Q(·|s)=∇ ^*_s(Q_s) is continuous and invariant under Qs↦Qs+cQ_s Q_s+c1, so γΩ(R) _ (R) is the minimum of a positive continuous function over the compact set of QsQ_s with maxaQs(a)=0 _aQ_s(a)=0 and minaQs(a)≥−R _aQ_s(a)≥-R, s∈s . ∎ Property (P1) is pointwise in R: for the reverse KL, γΩ(R)=(μmin+(1−μmin)eR/τ)−1 _ (R)= ( _ +(1- _ )e^R/τ )^-1 vanishes as R→∞R→∞ (Remark K.1). It makes the contraction rate of Lemma G.2.(iv) positive on bounded sets, guarantees that epochs terminate (Proposition H.2) and that qR>0q_R>0, and is the full-support requirement of Proposition G.4. It holds for the α-divergences with α≤1α≤ 1 and fails for α>1α>1 (Theorem K.1.(iv)); the floor-wrapped regularizers satisfy it with γΩϵ≥ϵ _ _ε≥ε (Lemma G.10.(i)). G.3 Existence and uniqueness of the fixed point of ΩT_ Theorem G.1 (Existence of a fixed point for ΩT_ ). Under Assumptions 1, 2 and 3, there exist QΩ∗∈ℝ×Q_ ^* ^S×A and ρΩ∗∈ℝ _ ^* such that Ω(QΩ∗)=QΩ∗+ρΩ∗T_ (Q_ ^*)=Q_ ^*+ _ ^*1, with ‖QΩ∗‖sp≤‖r‖sp+4mRmaxβmνmin,|ρΩ∗|≤2Rmax, \|Q^*_ \|_sp≤\|r\|_sp+ 4mR_ _m _ , ρ^*_ ≤ 2R_ , where νmin=minsν(s)>0 _ = _sν(s)>0 and Rmax=‖r‖∞+τmaxsmax|Ω¯s|,|Ωs(μs)|R_ =\|r\|_∞+τ _s \ _s , _s( _s) \, both finite under Assumption 3. Moreover ρΩ∗ρ^*_ is unique and is the optimal regularized gain (Propositions G.3 and G.5), and under (P1) QΩ∗Q^*_ is unique up to an additive constant (Proposition G.4). This subsection proves Theorem G.1 as a corollary of a more general statement, Theorem G.2, which rests on a condition that is both necessary and sufficient and does not presuppose Assumption 1. The argument is the regularized counterpart of the vanishing discount approach. In the unregularized case, the equivalence it establishes is classical (Schweitzer, 1984), as is the bound on the span of the relative discounted values under weak communication, which underlies regret analysis (Jaksch et al., 2010; Bartlett and Tewari, 2009) (see also Puterman, 1994, §8.4.4; Bertsekas, 2012, §4.1–4.3; Hernández-Lerma and Lasserre, 1996, §5.5; and the survey of Arapostathis et al., 1993). We nevertheless give a self-contained argument, since the classical statements for compact action sets require the reward to be continuous in the action (Schweitzer, 1985), which fails for r~π r_π as soon as dom(Ωs)⊊Δ()dom( _s) (A), as for the α-divergences with α≤0α≤ 0. Special cases are known by other routes: entropy regularization by convex duality (Neu et al., 2017), and the reverse KL as a linearly solvable problem, whose average-cost equation is a Perron–Frobenius eigenproblem (Todorov, 2007). Throughout Ω satisfies Assumption 2 and we also assume the following. Assumption G.1 (Admissible policy). There exists a full-support policy π~ π such that π~s∈dom(Ωs) π_s ( _s) for every s∈s and maxs∈|Ωs(π~s)|<+∞ _s _s( π_s) <+∞. Notation and constants. Since Ωs _s is proper and closed and Δ() (A) is compact, Ω¯s:=minπs∈Δ()Ωs(πs) _s:= _ _s∈ (A) _s( _s) is finite. Under Assumption G.1, the following constants are well-defined and finite: δΩ:=maxs∈max|Ω¯s|,|Ωs(π~s)|<+∞,Rmax:=‖r‖∞+τδΩ. _ := _s \ _s ,\ _s( π_s) \<+∞, R_ :=\|r\|_∞+τ _ . (G.3) For an admissible policy π (i.e. πs∈dom(Ωs) _s ( _s) for every s) and discount factor λ∈(0,1)λ∈(0,1) define the regularized reward and the discounted evaluation and optimality operators on ℝR^S, r~π(s):=⟨πs,rs⟩−τΩs(πs),ℰπλV:=r~π+λPπV,ΩλV:=maxπ admissibleℰπλV, r_π(s):= _s,r_s -τ\, _s( _s), ^λ_πV:= r_π+λ P_πV, ^λ_ V:= _π admissibleE^λ_πV, (G.4) the maximum in ΩλT^λ_ being taken state-wise. By (G.3), r~π(s)≤‖r‖∞−τΩ¯s≤Rmax r_π(s)≤\|r\|_∞-τ _s≤ R_ for every admissible π, and |r~π~(s)|≤Rmax r_ π(s) ≤ R_ . Remark G.3 (Sufficient conditions for Assumption G.1). Under Assumption 1, Assumption 3 gives Assumption G.1 with π~=μ π=μ. More generally, if every Dirac lies in the domain, as in Remark G.1, convexity gives Ωs(πs)≤maxaΩs(δa)<+∞ _s( _s)≤ _a _s( _a)<+∞ on Δ() (A), so any full-support π~ π works; this covers the negative entropy, the Tsallis negentropies, the quadratic regularizer and the α-divergences to μ with α>0α>0 (Theorem K.1.(i)). For α≤0α≤ 0, including the forward KL, Ωs _s is unbounded on the relative interior of Δ() (A), but π~=μ π=μ still gives Ωs(μs)=0 _s( _s)=0; for the floor-wrapped regularizers, Ωϵ,s(μs)=(1−ϵ)Ω0,s(μs) _ε,s( _s)=(1-ε) _0,s( _s) (Lemma G.10). The assumption says nothing about the domination profile: regularizers with sparse greedy policies satisfy it while failing (P1) (Remark G.6). G.3.1 The discounted regularized problem Lemma G.4 (Discounted fixed point). Let λ∈(0,1)λ∈(0,1) and Ωλ(Q):=r+λPΩ∗(Q)T^λ_ (Q):=r+λ P\, ^*(Q). Then ΩλT^λ_ is a λ-contraction for ∥⋅∥∞\|·\|_∞ with unique fixed point QλQ^λ, and Vλ:=Ω∗(Qλ)V^λ:= ^*(Q^λ) satisfies Qλ(s,a)=r(s,a)+λ∑s′p(s′|s,a)Vλ(s′),Vλ=ΩλVλ.Q^λ(s,a)=r(s,a)+λ _s p(s |s,a)\,V^λ(s ), V^λ=T^λ_ V^λ. (G.5) The operator ΩλT^λ_ is monotone and λ-translation equivariant, and the maximum defining ΩλVλT^λ_ V^λ is attained at the unique πλ:=πΩ,Qλ=∇Ω∗(Qλ)π^λ:= _ ,Q^λ=∇ ^*(Q^λ); in particular Vλ=ℰπλVλ≥ℰπλVλV^λ=E^λ_π^λV^λ ^λ_πV^λ for every admissible π. Proof. We have Ωλ(Q)−Ωλ(Q′)=λP(Ω∗(Q)−Ω∗(Q′))T^λ_ (Q)-T^λ_ (Q )=λ P( ^*(Q)- ^*(Q )) with P row-stochastic and Ω∗ ^* non-expansive for ∥⋅∥∞\|·\|_∞ (Lemma F.1.(v)), so ΩλT^λ_ is a λ-contraction on the complete space (ℝ×,∥⋅∥∞)(R^S×A,\|·\|_∞) and Banach’s theorem applies; this is the regularized discounted operator of (Geist et al., 2019, Thm. 1). Substituting the first identity of (G.5) into the definition of Ω∗ ^* gives, for every s, Vλ(s)=maxπs∈dom(Ωs)r~π(s)+λ(PπVλ)(s)=(ΩλVλ)(s),V^λ(s)= _ _s ( _s) \ r_π(s)+λ (P_πV^λ )(s) \= (T^λ_ V^λ )(s), (G.6) the maximizer being unique and equal to ∇Ω∗(Qλ)∇ ^*(Q^λ) by Lemma F.1.(i), since τ>0τ>0. Monotonicity and translation equivariance of ΩλT^λ_ are immediate from (G.4), each ℰπλE^λ_π having them and the state-wise maximum preserving them. ∎ Lemma G.5 (Variational representation of the discounted values). For an admissible stationary policy π define the discounted regularized return Jπλ(s):=π[∑t≥0λtr~π(st)|s0=s]J^λ_π(s):=E_π [ _t≥ 0λ^t r_π(s_t)|s_0=s ], which is finite since r~π r_π is a finite vector. Then, for every λ∈(0,1)λ∈(0,1) and every s∈s , Vλ(s)=maxπ admissible stationaryJπλ(s),V^λ(s)= _π admissible stationaryJ^λ_π(s), (G.7) the maximum being attained at πλπ^λ. Proof. Let π be an admissible stationary policy. Iterating Vλ≥ℰπλVλV^λ ^λ_πV^λ (Lemma G.4) K times, the operator ℰπλE^λ_π being monotone, and using the Markov property we obtain Vλ(s)≥(ℰπλ)KVλ(s)=π[∑t<Kλtr~π(st)|s0=s]+λKPπKVλ(s)V^λ(s)≥ (E^λ_π )^KV^λ(s)=E_π [ _t<Kλ^t r_π(s_t)|s_0=s ]+λ^KP^K_πV^λ(s). Since VλV^λ is a finite vector and PπKP^K_π is row-stochastic, ‖λKPπKVλ‖,∞≤λK‖Vλ‖,∞→0 \|λ^KP^K_πV^λ \|_S,∞≤λ^K\|V^λ\|_S,∞→ 0 as K→+∞K→+∞ at fixed λ, so Vλ≥JπλV^λ≥ J^λ_π. The same computation with π=πλπ=π^λ, for which Vλ=ℰπλVλV^λ=E^λ_π^λV^λ holds with equality, gives Vλ=JπλV^λ=J^λ_π^λ. ∎ Lemma G.6 (Normalized values are uniformly bounded). For every λ∈(0,1)λ∈(0,1), (1−λ)‖Vλ‖,∞≤Rmax,(1−λ)‖Qλ‖∞≤2Rmax.(1-λ) \|V^λ \|_S,∞≤ R_ , (1-λ) \|Q^λ \|_∞≤ 2R_ . (G.8) Proof. By (G.3), every admissible π satisfies the one-sided bound r~π≤‖r‖∞−τΩ¯s≤Rmax r_π≤\|r\|_∞-τ _s≤ R_ , while the policy π~ π of Assumption G.1 satisfies the two-sided bound |r~π~(s)|≤‖r‖∞+τ|Ωs(π~s)|≤Rmax r_ π(s) ≤\|r\|_∞+τ _s( π_s) ≤ R_ . Bounding the summand of JπλJ^λ_π termwise and summing the geometric series therefore gives Jπλ≤Rmax1−λJ^λ_π≤ R_ 1-λ for every admissible π, and Jπ~λ≥−Rmax1−λJ^λ_ π≥- R_ 1-λ. Since VλV^λ is the maximum of JπλJ^λ_π over admissible π (Lemma G.5), the upper bound propagates to VλV^λ and the lower bound is inherited from the single policy π~ π, whence ‖Vλ‖,∞≤Rmax1−λ\|V^λ\|_S,∞≤ R_ 1-λ. The claim on QλQ^λ follows from (G.5): (1−λ)‖Qλ‖∞≤(1−λ)‖r‖∞+λ(1−λ)‖Vλ‖,∞≤2Rmax(1-λ)\|Q^λ\|_∞≤(1-λ)\|r\|_∞+λ(1-λ)\|V^λ\|_S,∞≤ 2R_ . ∎ G.3.2 A necessary and sufficient condition for the existence of the regularized fixed point Assumption G.2 (Uniformly bounded relative values). There exist λ0∈(0,1) _0∈(0,1) and CV<+∞C_V<+∞ such that ‖Vλ‖,sp≤CV \|V^λ \|_S,sp≤ C_V for every λ∈[λ0,1)λ∈[ _0,1). Theorem G.2 (Existence of a fixed point for ΩT_ ). Let Assumptions 2, G.1 and G.2 hold. Then there exist QΩ∗∈ℝ×Q^*_ ^S×A and ρΩ∗∈ℝρ^*_ such that Ω(QΩ∗)=QΩ∗+ρΩ∗,T_ (Q^*_ )=Q^*_ +ρ^*_ 1, (G.9) with ‖QΩ∗‖sp≤‖r‖sp+CV \|Q^*_ \|_sp≤\|r\|_sp+C_V and |ρΩ∗|≤2Rmax ρ^*_ ≤ 2R_ . Proof. Fix (s0,a0)∈×(s_0,a_0) ×A and set, for λ∈[λ0,1)λ∈[ _0,1), Q¯λ:=Qλ−Qλ(s0,a0) Q^λ:=Q^λ-Q^λ(s_0,a_0)1 and ρλ:=(1−λ)Qλ(s0,a0)ρ^λ:=(1-λ)Q^λ(s_0,a_0). By translation equivariance of Ω∗ ^* (Lemma F.1.(i)) and P=P1=1, substituting Qλ=Q¯λ+Qλ(s0,a0)Q^λ= Q^λ+Q^λ(s_0,a_0)1 into (G.5) and subtracting λQλ(s0,a0)λ Q^λ(s_0,a_0) gives Q¯λ(s,a)+ρλ=r(s,a)+λ∑s′p(s′|s,a)Ω∗(Q¯λ)(s′). Q^λ(s,a)+ρ^λ=r(s,a)+λ _s p(s |s,a)\, ^*\! ( Q^λ )(s ). (G.10) By Lemma G.6, |ρλ|≤(1−λ)‖Qλ‖∞≤2Rmax ρ^λ ≤(1-λ)\|Q^λ\|_∞≤ 2R_ . By (G.5) and Lemma E.6, ‖Qλ‖sp≤‖r‖sp+λ‖Vλ‖,sp≤‖r‖sp+CV\|Q^λ\|_sp≤\|r\|_sp+λ\|V^λ\|_S,sp≤\|r\|_sp+C_V under Assumption G.2, hence ‖Q¯λ‖∞=maxs,a|Qλ(s,a)−Qλ(s0,a0)|≤‖Qλ‖sp≤‖r‖sp+CV\| Q^λ\|_∞= _s,a Q^λ(s,a)-Q^λ(s_0,a_0) ≤\|Q^λ\|_sp≤\|r\|_sp+C_V. Both bounds are uniform in λ∈[λ0,1)λ∈[ _0,1). Let λn↑1 _n 1 with λn≥λ0 _n≥ _0. The pairs (Q¯λn,ρλn)( Q _n,ρ _n) lie in a compact subset of ℝ×ℝR^S×A×R, so by Bolzano–Weierstrass some subsequence λφ(n)↑1 _ (n) 1 satisfies Q¯λφ(n)→QΩ∗ Q _ (n)→ Q^*_ and ρλφ(n)→ρΩ∗ρ _ (n)→ρ^*_ . Since Ω∗ ^* is non-expansive hence continuous, letting n→+∞n→+∞ in (G.10) gives QΩ∗+ρΩ∗=r+PΩ∗(QΩ∗)=Ω(QΩ∗)Q^*_ +ρ^*_ 1=r+P\, ^*(Q^*_ )=T_ (Q^*_ ). The bounds are preserved in the limit. ∎ Proposition G.1 (Assumption G.2 is also necessary). Conversely, if (G.9) admits a solution (QΩ∗,ρΩ∗)(Q^*_ ,ρ^*_ ), then, with V∗:=Ω∗(QΩ∗)V^*:= ^*(Q^*_ ), ‖Vλ‖,sp≤ 2‖V∗‖,spfor every λ∈(0,1), \|V^λ \|_S,sp\ ≤\ 2 \|V^* \|_S,sp every λ∈(0,1), so Assumption G.2 holds. It therefore characterizes exactly the solvability of the regularized average-reward optimality equation, as in the unregularized case Schweitzer and Federgruen, 1978; Puterman, 1994, §8.10. Proof. Replacing QΩ∗Q^*_ by QΩ∗+cQ^*_ +c1 changes V∗V^* by c (Lemma F.1.(i)) and leaves (G.9) unchanged, so we may center V∗V^* so that ‖V∗‖,∞=12‖V∗‖,sp=:κ\|V^*\|_S,∞= 12\|V^*\|_S,sp=:κ. Taking the conjugate in (G.9) as in (G.6) gives the regularized average-reward optimality equation V∗+ρΩ∗=maxπ admissibler~π+PπV∗.V^*+ρ^*_ 1= _π admissible \ r_π+P_πV^* \. (G.11) Set V~:=V∗+ρΩ∗1−λ V:=V^*+ ρ^*_ 1-λ1. Using (G.11) and PπV∗≥−κP_πV^*≥- 1 termwise, ΩλV~ ^λ_ V =maxπr~π+PπV∗−(1−λ)PπV∗+λρΩ∗1−λ = _π \ r_π+P_πV^*-(1-λ)P_πV^* \+ λρ^*_ 1-λ1 ≤V∗+ρΩ∗+(1−λ)κ+λρΩ∗1−λ=V~+(1−λ)κ. ≤ V^*+ρ^*_ 1+(1-λ) 1+ λρ^*_ 1-λ1= V+(1-λ) 1. Hence Ωλ(V~+κ)≤V~+(1−λ)κ+λκ=V~+κT^λ_ ( V+ 1)≤ V+(1-λ) 1+λ 1= V+ 1 by λ-translation equivariance, and monotonicity of ΩλT^λ_ (Lemma G.4) makes the iterates from V~+κ V+ 1 non-increasing, so their limit VλV^λ satisfies Vλ≤V~+κV^λ≤ V+ 1. The symmetric argument gives Vλ≥V~−κV^λ≥ V- 1, whence ‖Vλ−V~‖,∞≤κ\|V^λ- V\|_S,∞≤κ and ‖Vλ‖,sp≤‖V∗‖,sp+2κ=2‖V∗‖,sp\|V^λ\|_S,sp≤\|V^*\|_S,sp+2κ=2\|V^*\|_S,sp. ∎ G.3.3 Sufficient structural conditions We now record the classical setting in which Assumption G.2 holds, and show that Assumption 1 is a special case of it with explicit constants. Assumption G.3 (Weakly communicating MDP). The state space splits as =⊔trS=C _tr, where (i) C is closed: s∈s , a∈a and p(s′|s,a)>0p(s |s,a)>0 imply s′∈s ; (i) C is communicating: for all s,s′∈s,s there exist a stationary policy π and k≥0k≥ 0 with Pπk(s′|s)>0P^k_π(s |s)>0; (i) every state of trS_tr is transient under every stationary policy. This is the standard definition (Puterman, 1994, §8.3.1); trS_tr may be empty, in which case the MDP is communicating. Lemma G.7 (Stopped evaluation). Let λ∈(0,1)λ∈(0,1), ℬ⊆B , σ:=inft≥0|st∈ℬ∈ℕ∪+∞σ:= \t≥ 0|s_t \ ∪\+∞\, and let π be an admissible stationary policy. For every K≥0K≥ 0 and s∈s , Vλ(s)≥π[∑t=0σ∧K−1λtr~π(st)+λσ∧KVλ(sσ∧K)∣s0=s],V^λ(s)\ ≥\ E_π [ _t=0^σ K-1λ^t r_π(s_t)+λ^σ KV^λ (s_σ K ) s_0=s ], (G.12) with equality when π=πλπ=π^λ. If moreover σ<+∞σ<+∞ ℙπ(⋅|s0=s)\,P_π(·|s_0=s)-almost surely, the same holds with σ∧Kσ K replaced by σ. Proof. Let GK(s)G_K(s) denote the right-hand side of (G.12), empty sums being 00. We show Vλ≥GKV^λ≥ G_K pointwise by induction on K, simultaneously over all states. For K=0K=0 we have σ∧0=0σ 0=0, so G0=VλG_0=V^λ for every π. Assume Vλ≥GKV^λ≥ G_K. If s∈ℬs then σ=0σ=0 and GK+1(s)=Vλ(s)G_K+1(s)=V^λ(s). If s∉ℬs then σ≥1σ≥ 1, so splitting off the first step and applying the Markov property at s1s_1 gives GK+1(s)=r~π(s)+λ(PπGK)(s)G_K+1(s)= r_π(s)+λ (P_πG_K )(s); the induction hypothesis, PπP_π being non-negative, and Lemma G.4 then give GK+1(s)≤r~π(s)+λ(PπVλ)(s)=(ℰπλVλ)(s)≤Vλ(s).G_K+1(s)\ ≤\ r_π(s)+λ (P_πV^λ )(s)\ =\ (E^λ_πV^λ )(s)\ ≤\ V^λ(s). Both inequalities are equalities at πλπ^λ, which attains the state-wise maximum of Lemma G.4, whence the equality case. For the last claim, S is finite and π is admissible, so maxs|r~π(s)| _s r_π(s) and ‖Vλ‖,∞\|V^λ\|_S,∞ are finite and the integrand of (G.12) is bounded, uniformly in K, by maxs|r~π(s)|1−λ+‖Vλ‖,∞ _s r_π(s) 1-λ+\|V^λ\|_S,∞. On σ<+∞\σ<+∞\ we have σ∧K=σ K=σ for K large enough, so the integrand converges almost surely to its value at σ; conclude by dominated convergence. ∎ Lemma G.8 (Uniformly bounded absorption time). Under Assumption G.3 there is L<+∞L<+∞ with π[σ|s0=s]≤LE_π[ _C|s_0=s]≤ L for every stationary policy π and every s∈s , where σ:=inft≥0|st∈ _C:= \t≥ 0|s_t \. Proof. This is the uniform-transience estimate underlying stochastic shortest path problems (Bertsekas, 2012, §3.1); we give the short argument. Let n:=|tr|n:= _tr , and fix a stationary π and s∈trs _tr. If ℙπ(∀t,st∈tr|s0=s)>0P_π(∀ t,s_t _tr|s_0=s)>0, then with positive probability some state of the finite set trS_tr is visited infinitely often, hence is recurrent for PπP_π, contradicting Assumption G.3.(i). So σ<+∞ _C<+∞ a.s., and deleting cycles from a path reaching C gives ℙπ(σ>n|s0=s)<1P_π( _C>n|s_0=s)<1. The map π↦ℙπ(σ>n|s0=s)π _π( _C>n|s_0=s) is polynomial, hence continuous on the compact ∏sΔ() _s (A), so θ:=maxs∈trmaxπℙπ(σ>n|s0=s)<1θ:= _s _tr _πP_π( _C>n|s_0=s)<1; the Markov property gives ℙπ(σ>kn|s0=s)≤θkP_π( _C>kn|s_0=s)≤θ^k and π[σ|s0=s]=∑j≥0ℙπ(σ>j|s0=s)≤n∑k≥0ℙπ(σ>kn|s0=s)≤n1−θ=:LE_π[ _C|s_0=s]= _j≥ 0P_π( _C>j|s_0=s)≤ n _k≥ 0P_π( _C>kn|s_0=s)≤ n1-θ=:L. If s∈s then σ=0 _C=0, so the bound holds for every s∈s . ∎ Proposition G.2 (Weakly communicating implies Assumption G.2). Let Assumptions G.3 and G.1 hold. Let K⋆≥1K ≥ 1 and p⋆>0p >0 be such that for every ordered pair (s,s′)(s,s ) of states of C there is k(s,s′)≤K⋆k(s,s )≤ K with Pπ~k(s,s′)(s′|s)≥p⋆P^k(s,s )_ π(s |s)≥ p . Such constants exist, with K⋆≤max1,||−1≤||K ≤ \1, -1\≤ , and they depend only on p and π~ π, not on λ. For any such pair, Assumption G.2 holds with λ0:=1−12K⋆,CV:=4K⋆Rmaxp⋆+4LRmax. _0:=1- 12K , C_V:= 4K R_ p +4LR_ . (G.13) Proof. Let s,s′∈s,s . By Assumption G.3.(i) there are π and k with Pπk(s′|s)>0P^k_π(s |s)>0, hence a path s=s0,…,sk=s′s=s_0,…,s_k=s with p(si+1|si,ai)>0p(s_i+1|s_i,a_i)>0 for some actions aia_i; it stays in C, which is closed, and deleting its cycles leaves a simple path, so k(s,s′)≤max1,||−1k(s,s )≤ \1, -1\. Since π~ π has full support, Pπ~(si+1|si)≥π~(ai|si)p(si+1|si,ai)>0P_ π(s_i+1|s_i)≥ π(a_i|s_i)p(s_i+1|s_i,a_i)>0 along that path, so Pπ~k(s,s′)(s′|s)>0P^k(s,s )_ π(s |s)>0; taking for K⋆K the largest of these lengths and for p⋆p the smallest of these finitely many positive numbers proves that admissible constants exist, and they involve only p and π~ π. Note also that π~ π is admissible by Assumption G.1 and πλ=∇Ω∗(Qλ)π^λ=∇ ^*(Q^λ) is admissible as a maximizer, so Lemma G.7 applies to both, with |r~π~|≤Rmax r_ π ≤ R_ and r~π≤Rmax r_π≤ R_ for every admissible π by (G.3). We first bound the span on C. Fix λ∈[λ0,1)λ∈[ _0,1) and let smaxs_ and smins_ maximize and minimize VλV^λ over C; if smax=smins_ =s_ that span vanishes and there is nothing to prove, so assume smax≠smins_ ≠ s_ and set K:=k(smin,smax)≥1K:=k(s_ ,s_ )≥ 1 and p:=Pπ~K(smax|smin)≥p⋆p:=P^K_ π(s_ |s_ )≥ p . Applying Lemma G.7 at smins_ with ℬ=∅B= , so that σ∧K=Kσ K=K, with π=π~π= π and ∑t<Kλt≤K _t<Kλ^t≤ K, gives Vλ(smin)≥−KRmax+λKπ~[Vλ(sK)∣s0=smin]V^λ(s_ )≥-KR_ +λ^KE_ π[V^λ(s_K) s_0=s_ ]. As C is closed, sK∈s_K almost surely, so Vλ(sK)≥Vλ(smin)V^λ(s_K)≥ V^λ(s_ ) pointwise and, isolating the event sK=smax\s_K=s_ \ of probability p, π~[Vλ(sK)∣s0=smin]≥pVλ(smax)+(1−p)Vλ(smin)E_ π[V^λ(s_K) s_0=s_ ]≥ pV^λ(s_ )+(1-p)V^λ(s_ ). Rearranging, the coefficient of Vλ(smin)V^λ(s_ ) being 1−λK1-λ^K, and using 1−λK≤K(1−λ)1-λ^K≤ K(1-λ) with Lemma G.6, pλK(Vλ(smax)−Vλ(smin))≤KRmax+(1−λK)Vλ(smin)≤ 2KRmax.p\,λ^K (V^λ(s_ )-V^λ(s_ ) )\ ≤\ KR_ + (1-λ^K )V^λ(s_ )\ ≤\ 2KR_ . Since K≤K⋆K≤ K and λK≥λ0K⋆=(1−12K⋆)K⋆≥12λ^K≥ _0^K = (1- 12K )^K ≥ 12, we get ‖Vλ|‖,sp≤4K⋆Rmax/p⋆=:C \|V^λ|_C \|_C,sp≤ 4K R_ /p =:C_C. It remains to pass from C to S. Write σ:=σ:= _C and fix s∈s . By Lemma G.8, π[σ∣s0=s]≤LE_π[σ s_0=s]≤ L for every stationary π, so σ<+∞σ<+∞ almost surely and Lemma G.7 applies with ℬ=B=C and the stopping time σ itself. Its two terms are each bounded by LRmaxLR_ : the reward term because ∑t<σλt≤σ _t<σλ^t≤σ, and the boundary term, once written as π[Vλ(sσ)∣s0=s]+π[(λσ−1)Vλ(sσ)∣s0=s]E_π[V^λ(s_σ) s_0=s]+E_π[(λ^σ-1)V^λ(s_σ) s_0=s], because 0≤1−λσ≤σ(1−λ)0≤ 1-λ^σ≤σ(1-λ) and (1−λ)‖Vλ‖,∞≤Rmax(1-λ)\|V^λ\|_S,∞≤ R_ by Lemma G.6. Since sσ∈s_σ , the previous paragraph gives 0≤Vλ(sσ)−Vλ(smin)≤C0≤ V^λ(s_σ)-V^λ(s_ )≤ C_C. Taking π=πλπ=π^λ, for which Lemma G.7 holds with equality, and r~πλ≤Rmax r_π^λ≤ R_ , yields Vλ(s)≤Vλ(smin)+C+2LRmaxV^λ(s)≤ V^λ(s_ )+C_C+2LR_ ; taking π=π~π= π and r~π~≥−Rmax r_ π≥-R_ yields Vλ(s)≥Vλ(smin)−2LRmaxV^λ(s)≥ V^λ(s_ )-2LR_ . Both hold for every s, so ‖Vλ‖,sp≤C+4LRmax=CV \|V^λ \|_S,sp≤ C_C+4LR_ =C_V. ∎ Corollary G.1 (Assumption 1 implies Assumption G.2, hence Theorem G.1). Let Assumptions 1, 2 and 3 hold. Then the MDP is communicating (=C=S, tr=∅S_tr= , L=0L=0), Proposition G.2 applies with π~=μ π=μ, K⋆=mK =m and p⋆=βmνminp = _m _ , where νmin:=minsν(s)>0 _ := _sν(s)>0, so Assumption G.2 holds with λ0=1−12m,CV=4mRmaxβmνmin, _0=1- 12m, C_V= 4mR_ _m _ , and Theorem G.2 gives Theorem G.1. Proof. Assumption 3 makes Assumption G.1 hold with π~=μ π=μ, which has full support, so that δΩ=maxsmax|Ω¯s|,|Ωs(μs)| _ = _s \ _s , _s( _s) \ and RmaxR_ are the constants of Theorem G.1. For all s,s′s,s , Pμm(s′|s)=∑aμ(a|s)Kμm(s′|s,a)≥βmν(s′)≥βmνmin>0P^m_μ(s |s)= _aμ(a|s)K^m_μ(s |s,a)≥ _mν(s )≥ _m _ >0, so every ordered pair of states is joined by a path of length m under μ: Assumption G.3 holds with =C=S, which is closed and communicating, and tr=∅S_tr= , so σ≡0 _C≡ 0 and L=0L=0 in Lemma G.8. Proposition G.2 then applies with K⋆=mK =m and p⋆=βmνminp = _m _ , its last paragraph being vacuous, and gives Assumption G.2 with the stated constants. Finally Theorem G.2 yields (G.9) with ‖QΩ∗‖sp≤‖r‖sp+CV\|Q^*_ \|_sp≤\|r\|_sp+C_V and |ρΩ∗|≤2Rmax ρ^*_ ≤ 2R_ . The gain ρΩ∗ρ^*_ is unique and optimal by Propositions G.3 and G.5, and under (P1) QΩ∗Q^*_ is unique up to an additive constant by Proposition G.4, whose Assumption G.4 holds since the MDP is communicating (Remark G.5). ∎ Remark G.4 (On the structural hypothesis). Among the classical ergodic classes of finite MDPs (Puterman, 1994, §8.3.1), weak communication is the weakest: both unichain models — in which every stationary policy induces a chain with a single recurrent class plus a possibly empty set of transient states — and communicating models are weakly communicating, while unichain and communicating are incomparable, neither class containing the other. Proposition G.2 therefore covers all these models at once. Assumption G.2 is nonetheless strictly weaker, being stated for the reward at hand rather than for p alone: two isolated states carrying the same reward do not form a weakly communicating MDP, yet VλV^λ is constant, its span vanishes and the fixed point exists. It is moreover necessary as well as sufficient (Proposition G.1), stated directly for the regularized model. G.3.4 Uniqueness and interpretation of the gain Proposition G.3 (Uniqueness of the gain). Let Assumption 2 hold. If Ω(Qi)=Qi+ρiT_ (Q_i)=Q_i+ _i1 for i=1,2i=1,2, then ρ1=ρ2 _1= _2. No structural assumption on the MDP is needed here, unlike for the uniqueness of QΩ∗Q_ ^* (Proposition G.4). Proof. Let c:=maxs,a(Q1−Q2)(s,a)c:= _s,a(Q_1-Q_2)(s,a), so Q1≤Q2+cQ_1≤ Q_2+c1. By monotonicity and translation equivariance of ΩT_ (Corollary F.1), Ω(Q1)≤Ω(Q2)+cT_ (Q_1) _ (Q_2)+c1, i.e. Q1−Q2≤(ρ2−ρ1+c)Q_1-Q_2≤( _2- _1+c)1. Evaluating at a maximizer of Q1−Q2Q_1-Q_2 gives c≤ρ2−ρ1+c≤ _2- _1+c, i.e. ρ1≤ρ2 _1≤ _2; exchanging the roles of Q1Q_1 and Q2Q_2 gives the reverse inequality. ∎ Assumption G.4 (Unique terminal class). The union graph G of Definition D.1 has a unique terminal strongly connected component, i.e. a unique strongly connected component C with no outgoing edge. Remark G.5. Assumption G.3 implies Assumption G.4: its set C is terminal by closedness and connectedness, and a second terminal component ′⊆trC _tr would be closed, so every stationary policy then has a recurrent state inside, contradicting (i). The converse fails: with =1,2,3S=\1,2,3\, a self-loop and an edge 1→21→ 2 at state 11, and 2→32→ 3, 3→33→ 3, the terminal component 3\3\ is unique, yet state 11 is recurrent under the self-loop policy, so the MDP is not weakly communicating. Proposition G.4 (Uniqueness of QΩ∗Q^*_ up to an additive constant). Let Assumption G.4 hold and assume (P1), i.e. that the greedy policies have full support (Lemma G.3). If Ω(Qi)=Qi+ρiT_ (Q_i)=Q_i+ _i1 for i=1,2i=1,2, then Q1−Q2∈ℝQ_1-Q_2 1. Conversely QΩ∗+cQ^*_ +c1 solves (G.9) with the same ρΩ∗ρ^*_ , for every c∈ℝc . Proof. The converse is translation equivariance (Corollary F.1). Directly, ρ1=ρ2 _1= _2 by Proposition G.3, so h:=Q1−Q2h:=Q_1-Q_2 satisfies h=Ω(Q1)−Ω(Q2)=Kπ¯h=T_ (Q_1)-T_ (Q_2)=K_ πh by Lemma G.2.(i), where π¯:=π¯Ω,[Q1,Q2] π:= π_ ,[Q_1,Q_2]: the function h is harmonic for the state-action kernel Kπ¯K_ π. By Definition G.1.(i), π¯ π is a mixture of greedy policies, hence has full support, so Pπ¯(s′|s)>0P_ π(s |s)>0 exactly when s→s′s→ s in G. The recurrent classes of the state chain are therefore the terminal strongly connected components of G, and there is only one, C, by Assumption G.4; since π¯(a|s)>0 π(a|s)>0 for every a, the state-action chain has ×C×A as its unique recurrent class. A bounded harmonic function of a finite Markov chain is constant on each recurrent class (Puterman, 1994, App. A.4), and on a transient s we have h(s,a)=[h(sn,an)|s0=s,a0=a]h(s,a)=E[h(s_n,a_n)|s_0=s,a_0=a] for every n, which converges by absorption to the value of h on that unique class. Hence h is constant. ∎ Remark G.6 (Where the full-support requirement bites). The full-support requirement of Proposition G.4 is exactly (P1) (Lemma G.3). It holds for the α-divergences with α≤1α≤ 1 (Theorem K.1.(iv)), hence for both KL divergences and for the Hellinger regularizer, for the negative entropy, and for every ϵε-floor-wrapped regularizer, where πΩϵ,Q≥ϵμ>0 _ _ε,Q≥εμ>0 by Lemma G.10.(i). It fails for regularizers whose greedy policy may be sparse, that is for α>1α>1 and in particular for the χ2χ^2: there ρΩ∗ρ^*_ is still unique by Proposition G.3, but QΩ∗Q^*_ need not be, exactly as at τ=0τ=0 (Puterman, 1994, §9.1). Proposition G.5 (ρΩ∗ρ^*_ is the optimal regularized gain). Let (QΩ∗,ρΩ∗)(Q^*_ ,ρ^*_ ) solve (G.9) and set π∗:=πΩ,QΩ∗π^*:= _ ,Q^*_ . For an admissible stationary policy π define the regularized gain ρ~π(s):=lim infn→+∞1nπ[∑t=0n−1r~π(st)|s0=s] ρ_π(s):= _n→+∞ 1nE_π [ _t=0^n-1 r_π(s_t)|s_0=s ]. Then ρ~π(s)≤ρΩ∗=ρ~π∗(s) ρ_π(s)≤ρ^*_ = ρ_π^*(s) for every s and every admissible stationary π: the gain is constant in s, ρΩ∗ρ^*_ is optimal, and π∗π^* is gain-optimal. Proof. Equation (G.11) holds with V∗=Ω∗(QΩ∗)V^*= ^*(Q^*_ ), the maximum being attained state-wise at π∗π^*. For any admissible π it gives r~π≤V∗+ρΩ∗−PπV∗ r_π≤ V^*+ρ^*_ 1-P_πV^*; applying PπtP^t_π and summing over t<nt<n telescopes to π[∑t<nr~π(st)|s0=s]≤nρΩ∗+V∗(s)−(PπnV∗)(s)E_π [ _t<n r_π(s_t)|s_0=s ]≤ nρ^*_ +V^*(s)-(P^n_πV^*)(s). Divide by n and let n→+∞n→+∞, V∗V^* being bounded. Equality holds throughout for π=π∗π=π^*. This is the standard verification argument Puterman, 1994, §8.4; Bertsekas, 2012, §4.2. ∎ G.4 Control of the bias We compare the regularized solution with the solutions (Q∗,ρ∗)(Q^*,ρ^*) of the average-reward optimality equation (Q∗)=Q∗+ρ∗T(Q^*)=Q^*+ρ^*1. Since Assumption 1 makes the MDP communicating, such solutions exist (Schweitzer and Federgruen, 1978, Theorem 3.1(b)) and ρ∗ρ^* is the optimal gain, but Q∗Q^* need not be unique up to an additive constant (Wan et al., 2024). The comparison uses two tools: a pointwise version of the regularization modulus, and the classical bounds on the gain given by an approximate fixed point. Definition G.2 (Pointwise regularization modulus). For a radius R≥0R≥ 0, with gQg_Q the regularization gap of Definition G.1.(iv), ΛΩpt(R):=maxQ:‖Q‖asp≤R‖gQ‖,∞. _ ^pt(R):= _Q\,:\,\|Q\|_asp≤ R \|g_Q \|_S,∞. The maximum is attained and finite by the argument of the proof of Lemma G.1, and ΛΩ(R)≤2ΛΩpt(R) _ (R)≤ 2 _ ^pt(R). Lemma G.9 (Gain bounds from an approximate fixed point). Assume that Ω(QΩ∗)=QΩ∗+ρΩ∗T_ (Q^*_ )=Q^*_ +ρ^*_ 1. If Q∈ℝ×Q ^S×A and c−,c+∈ℝc_-,c_+ satisfy Q+c−≤Ω(Q)≤Q+c+Q+c_-1 _ (Q)≤ Q+c_+1, then c−≤ρΩ∗≤c+c_-≤ρ^*_ ≤ c_+. Proof. This is the regularized form of the classical value-iteration bounds on the gain (Puterman, 1994, §8.5.5), (Bertsekas, 2012, §4.3). By monotonicity and translation equivariance of ΩT_ (Corollary F.1), Ωn+1(Q)≤Ωn(Q+c+)=Ωn(Q)+c+T^n+1_ (Q) ^n_ (Q+c_+1)=T^n_ (Q)+c_+1, so Ωn(Q)≤Q+nc+T^n_ (Q)≤ Q+nc_+1 by induction. Since ΩT_ is non-expansive for ∥⋅∥∞\|·\|_∞ (Corollary F.1), with d:=‖Q−QΩ∗‖∞d:=\|Q-Q^*_ \|_∞ we get QΩ∗+nρΩ∗=Ωn(QΩ∗)≤Ωn(Q)+d≤Q+nc++dQ^*_ +nρ^*_ 1=T^n_ (Q^*_ ) ^n_ (Q)+d1≤ Q+nc_+1+d1. Reading any coordinate, dividing by n and letting n→+∞n→+∞ gives ρΩ∗≤c+ρ^*_ ≤ c_+; the other inequality is symmetric. ∎ Proposition G.6 (Cost of Ω -regularization). Let Assumptions 1, 2 and 3 hold, and let (Q∗,ρ∗)(Q^*,ρ^*) solve (Q∗)=Q∗+ρ∗T(Q^*)=Q^*+ρ^*1. (i) Value gap: with R∗:=max‖QΩ∗‖asp,max0≤k<m‖Ωk(Q∗)‖aspR^*:= \\|Q^*_ \|_asp, _0≤ k<m\|T^k_ (Q^*)\|_asp \ and α(P)≤1α(P)≤ 1 the Dobrushin coefficient of the transition kernel P (Definition E.2), ‖QΩ∗−Q∗‖sp≤mα(P)ΛΩ(R∗)βmγΩ(R∗)m−1≤mΛΩ(R∗)βmγΩ(R∗)m−1,\|Q^*_ -Q^*\|_sp≤ m\,α(P)\, _ (R^*) _m _ (R^*)^m-1≤ m\, _ (R^*) _m _ (R^*)^m-1, the right-hand sides being +∞+∞ if γΩ(R∗)=0 _ (R^*)=0. (i) Gain gap: with Ω¯:=mins∈minπs∈Δ()Ωs(πs) := _s _ _s∈ (A) _s( _s), mins∈gQ∗(s)≤ρΩ∗−ρ∗≤maxs∈gQ∗(s)≤−τΩ¯, _s g_Q^*(s)\ ≤\ ρ^*_ -ρ^*\ ≤\ _s g_Q^*(s)\ ≤\ -τ , (G.14) and consequently |ρΩ∗−ρ∗|≤‖gQ∗‖,∞≤ΛΩpt(‖Q∗‖asp). ρ^*_ -ρ^* \ ≤\ \|g_Q^*\|_S,∞\ ≤\ _ ^pt(\|Q^*\|_asp). (G.15) If moreover ||≥2|S|≥ 2 and minΩs=Ω¯ _s= for every s, then −ΛΩ(‖Q∗‖asp)≤ρΩ∗−ρ∗+τΩ¯≤0- _ (\|Q^*\|_asp)≤ρ^*_ -ρ^*+τ ≤ 0. Proof. The regularized fixed point (QΩ∗,ρΩ∗)(Q^*_ ,ρ^*_ ) exists by Theorem G.1, and both T and ΩT_ are translation equivariant (Corollary F.1), so that k(Q∗)=Q∗+kρ∗T^k(Q^*)=Q^*+kρ^*1 and Ωk(QΩ∗)=QΩ∗+kρΩ∗T^k_ (Q^*_ )=Q^*_ +kρ^*_ 1 for every k≥0k≥ 0. (i) The span seminorm being invariant under constants, ‖QΩ∗−Q∗‖sp=‖Ωm(QΩ∗)−m(Q∗)‖sp≤‖Ωm(QΩ∗)−Ωm(Q∗)‖sp⏟(a)+‖Ωm(Q∗)−m(Q∗)‖sp⏟(b).\|Q^*_ -Q^*\|_sp=\|T^m_ (Q^*_ )-T^m(Q^*)\|_sp≤ \|T^m_ (Q^*_ )-T^m_ (Q^*)\|_sp_(a)+ \|T^m_ (Q^*)-T^m(Q^*)\|_sp_(b). (G.16) Term (a). Since ∥⋅∥asp\|·\|_asp is invariant under constants, ‖Ωk(QΩ∗)‖asp=‖QΩ∗‖asp\|T^k_ (Q^*_ )\|_asp=\|Q^*_ \|_asp, so that for every k∈ 1,m−1 ∈ 1,m-1 , R[Ωk(QΩ∗),Ωk(Q∗)]=max‖QΩ∗‖asp,‖Ωk(Q∗)‖asp≤R∗.R [T_ ^k(Q^*_ ),T_ ^k(Q^*) ]= \\|Q^*_ \|_asp,\|T^k_ (Q^*)\|_asp \≤ R^*. As γΩ _ is non-increasing (Definition G.1.(i)), every factor of the product in Lemma G.2.(iv), applied with Q=QΩ∗Q=Q^*_ and Q′=Q∗Q =Q^*, is at least γΩ(R∗) _ (R^*), hence (a) ≤(1−βmγΩ(R∗)m−1)‖QΩ∗−Q∗‖sp≤ (1- _m _ (R^*)^m-1 )\|Q^*_ -Q^*\|_sp. Term (b). Let Ak:=Ωk(m−k(Q∗))A_k:=T^k_ (T^m-k(Q^*) ) for k∈ 0,m ∈ 0,m , so that A0=m(Q∗)A_0=T^m(Q^*) and Am=Ωm(Q∗)A_m=T^m_ (Q^*). Since m−k−1(Q∗)=Q∗+(m−k−1)ρ∗T^m-k-1(Q^*)=Q^*+(m-k-1)ρ^*1, translation equivariance makes the constants cancel, and k applications of the non-expansiveness of ΩT_ in ∥⋅∥sp\|·\|_sp give ‖Ak+1−Ak‖sp \|A_k+1-A_k\|_sp =‖Ωk(Ω(Q∗))−Ωk((Q∗))‖sp≤‖Ω(Q∗)−(Q∗)‖sp = \|T^k_ (T_ (Q^*) )-T^k_ (T(Q^*) ) \|_sp≤\|T_ (Q^*)-T(Q^*)\|_sp ≤α(P)ΛΩ(‖Q∗‖asp)≤α(P)ΛΩ(R∗), ≤α(P) _ (\|Q^*\|_asp)≤α(P) _ (R^*), by Lemma G.2.(vi), ‖Q∗‖asp≤R∗\|Q^*\|_asp≤ R^* and the monotonicity of ΛΩ _ (Lemma G.1). Summing over k, (b) ≤mα(P)ΛΩ(R∗)≤ m\,α(P) _ (R^*). Injecting both bounds into (G.16) yields βmγΩ(R∗)m−1‖QΩ∗−Q∗‖sp≤mα(P)ΛΩ(R∗) _m _ (R^*)^m-1\|Q^*_ -Q^*\|_sp≤ m\,α(P) _ (R^*), which is the claim when γΩ(R∗)>0 _ (R^*)>0; otherwise the bound is trivial. (i) By Lemma G.2.(vi) and its proof, Ω(Q)−(Q)=PgQT_ (Q)-T(Q)=Pg_Q for every Q, so that Ω(Q∗)=Q∗+ρ∗+PgQ∗T_ (Q^*)=Q^*+ρ^*1+Pg_Q^*. Each entry of PgQ∗Pg_Q^* is a convex combination of the entries of gQ∗g_Q^*, and Lemma G.9 with c−:=ρ∗+minsgQ∗(s)c_-:=ρ^*+ _sg_Q^*(s) and c+:=ρ∗+maxsgQ∗(s)c_+:=ρ^*+ _sg_Q^*(s) gives the first two inequalities of (G.14). The last one is the upper part of Lemma F.1.(i), which needs no assumption on the Diracs: gQ∗(s)≤−τΩ¯s≤−τΩ¯g_Q^*(s)≤-τ _s≤-τ . Finally, (G.15) follows from (G.14) and Definition G.2. For the last claim, fix s and s0≠s_0≠ s, and let Q~ Q equal Q∗Q^* except at s0s_0, where it is constant in the action. Then ‖Q~‖asp≤‖Q∗‖asp\| Q\|_asp≤\|Q^*\|_asp, and since gQ(s)g_Q(s) only depends on Q(s,⋅)Q(s,·) and equals −τminΩs0-τ _s_0 at a constant, gQ~(s)=gQ∗(s)g_ Q(s)=g_Q^*(s) and gQ~(s0)=−τΩ¯g_ Q(s_0)=-τ ; hence −τΩ¯−gQ∗(s)≤‖gQ~‖,sp≤ΛΩ(‖Q∗‖asp)-τ -g_Q^*(s)≤\|g_ Q\|_S,sp≤ _ (\|Q^*\|_asp). Together with gQ∗≤−τΩ¯g_Q^*≤-τ and (G.14), this gives the claim. ∎ Item (i) bounds the gap by the regularization modulus times the effective horizon m/(βmγΩ(R∗)m−1)m/( _m _ (R^*)^m-1), the average-reward counterpart of the discounted bias bound τ(Ω¯−Ω¯)/(1−γ)τ( - )/(1-γ) of Geist et al. (2019), where Ω¯:=maxs,aΩs(δa) := _s,a _s( _a). It exhibits the trade-off that runs through the paper: a larger temperature raises γΩ _ , hence the contraction rates of Section 4, but also inflates ΛΩ _ . Unlike 1/(1−γ)1/(1-γ), this effective horizon depends on τ, so (i) is informative only when γΩ(R∗) _ (R^*) is not too small; for the reverse KL the bound grows as e(m−1)R∗/τe^(m-1)R^*/τ when τ→0τ→ 0. Item (i) has no such limitation, as the next corollary shows. Regularization also removes a degeneracy of the unregularized problem: under (P1), QΩ∗Q^*_ is unique up to an additive constant (Proposition G.4), and (i) bounds its distance to every solution Q∗Q^*. Corollary G.2 (The gain gap vanishes with τ). Under the assumptions of Remark G.1, −τΩ¯≤ρΩ∗−ρ∗≤−τΩ¯-τ ≤ρ^*_ -ρ^*≤-τ , so ρΩ∗→ρ∗ρ^*_ →ρ^* linearly as τ→0τ→ 0, with constants that are free of βm _m, of γΩ _ and of the radius. Proof. When every Dirac lies in the effective domain, Lemma F.1.(i) gives gQ∗(s)≥−τΩ¯g_Q^*(s)≥-τ for every s, and (G.14) concludes. ∎ Example G.1. For the reverse KL (Corollary K.1, α=1α=1), Ω¯s=0 _s=0 and Ω¯=log(1/μmin) = (1/ _ ) with μmin:=mins,aμ(a|s) _ := _s,aμ(a|s), so Corollary G.2 reads −τlog(1/μmin)≤ρΩ∗−ρ∗≤0-τ (1/ _ )≤ρ^*_ -ρ^*≤ 0. For the forward KL (Corollary K.1, α=0α=0), the Diracs lie outside the effective domain, but Ω¯s=0 _s=0 still gives ρΩ∗≤ρ∗ρ^*_ ≤ρ^*, and (G.15) and Theorem K.1.(v) give |ρΩ∗−ρ∗|≤τ(1+log+(‖Q∗‖asp/τ))=O(τlog(1/τ)) ρ^*_ -ρ^* ≤τ (1+ _+(\|Q^*\|_asp/τ) )=O (τ (1/τ) ) as τ→0τ→ 0. The gain gap also controls the performance of the policies returned by our algorithms: the greedy policy of any Q close to QΩ∗Q^*_ in span is near-optimal. Corollary G.3 (Performance of the greedy policy). Let Assumptions 1, 2 and 3 hold, let Q∈ℝ×Q ^S×A and π:=πΩ,Qπ:= _ ,Q. Then, for every s∈s , the regularized gain of Proposition G.5 satisfies ρΩ∗−‖Q−QΩ∗‖sp≤ρ~π(s)≤ρΩ∗ρ^*_ -\|Q-Q^*_ \|_sp≤ ρ_π(s)≤ρ^*_ , and the gain of Subsection 2.1 satisfies ρπ(s)≥ρ~π(s)+τΩ¯ρ^π(s)≥ ρ_π(s)+τ . In particular, under the assumptions of Remark G.1, ρ∗−ρπ(s)≤‖Q−QΩ∗‖sp+τΔΩρ^*-ρ^π(s)≤\|Q-Q^*_ \|_sp+τ _ . Proof. The upper bound is Proposition G.5. For the lower bound, let π(U)(s,a):=r(s,a)+∑s′p(s′|s,a)(⟨πs′,Us′⟩−τΩs′(πs′))T^π(U)(s,a):=r(s,a)+ _s p(s |s,a) ( _s ,U_s -τ _s ( _s ) ) be the regularized evaluation operator of π; it is monotone and translation equivariant, and π(Q)=Ω(Q)T^π(Q)=T_ (Q) since π is greedy for Q. With u:=Q−QΩ∗u:=Q-Q^*_ , Lemma G.2.(i) gives Ω(Q)−Q−ρΩ∗=Kπ¯u−u≥−‖u‖spT_ (Q)-Q-ρ^*_ 1=K_ πu-u≥-\|u\|_sp1, the entries of Kπ¯uK_ πu lying in [minu,maxu][ u, u]. Hence π(Q)≥Q+(ρΩ∗−‖u‖sp)T^π(Q)≥ Q+(ρ^*_ -\|u\|_sp)1 and, as in the proof of Lemma G.9, (π)n(Q)≥Q+n(ρΩ∗−‖u‖sp)(T^π)^n(Q)≥ Q+n(ρ^*_ -\|u\|_sp)1. Since (π)n(Q)(s,a)(T^π)^n(Q)(s,a) is r(s,a)r(s,a) plus the expected sum of r~π(st) r_π(s_t), 1≤t<n1≤ t<n, along π from (s,a)(s,a), up to a bounded term, dividing by n gives ρ~π(s)≥ρΩ∗−‖u‖sp ρ_π(s)≥ρ^*_ -\|u\|_sp. Next, ⟨πs,rs⟩=r~π(s)+τΩs(πs)≥r~π(s)+τΩ¯ _s,r_s = r_π(s)+τ _s( _s)≥ r_π(s)+τ , whence ρπ≥ρ~π+τΩ¯ρ^π≥ ρ_π+τ . Finally, Corollary G.2 gives ρ∗−ρΩ∗≤τΩ¯ρ^*-ρ^*_ ≤τ , and ρ∗−ρπ≤(ρ∗−ρΩ∗)+‖u‖sp−τΩ¯≤‖u‖sp+τΔΩρ^*-ρ^π≤(ρ^*-ρ^*_ )+\|u\|_sp-τ ≤\|u\|_sp+τ _ . ∎ G.5 ϵε-Floor-Wrapped Regularization Regularizers with sparse greedy policies, such as the α-divergences with α>1α>1, violate (P1), and even when (P1) holds the domination profile γΩ(R) _ (R) may go to zero as R grows, making contraction and exploration results local. We wrap the base regularization in a hard policy floor. This restores (P1) with γΩϵ≥ϵ _ _ε≥ε uniformly in R, which makes the results global (Lemma G.11), at the price of an additional bias of order ϵR∗ε R^* (Remark G.7). The resulting greedy policy ϵμ+(1−ϵ)πΩ0,Qεμ+(1-ε) _ _0,Q (Lemma G.10.(i)) is, for a uniform μ and the negative entropy as Ω0 _0, the ϵε-softmax learning policy of Nanda and Chen (2026), where the mixing weight ϵε plays the same role: a uniform lower bound on the action probabilities. Definition G.3 (ϵε-floored simplex). Let ϵ∈[0,1)ε∈[0,1) and let μ be the full-support mixing policy of Assumption 1. For s∈s define Δϵ(s):=πs∈Δ():πs≥ϵμs _ε(s):=\ _s∈ (A): _s≥ε _s\ and the affine bijection Φϵ,s:Δ()→Δϵ(s),Φϵ,s(p)=ϵμs+(1−ϵ)p _ε,s: (A)→ _ε(s), _ε,s(p)=ε _s+(1-ε)p, whose inverse is Φϵ,s−1(πs)=(πs−ϵμs)/(1−ϵ) _ε,s^-1( _s)=( _s-ε _s)/(1-ε). Definition G.4 (ϵε-floor-wrapped regularization). Let Ω0 _0 be a base regularization satisfying Assumption 2 and let ϵ∈[0,1)ε∈[0,1). The ϵε-floor-wrapped regularization Ωϵ _ε is defined state-wise by Ωϵ,s(πs)=(1−ϵ)Ω0,s(πs−ϵμs1−ϵ)if πs∈Δϵ(s),+∞otherwise. _ε,s( _s)= cases(1-ε)\, _0,s ( _s-ε _s1-ε )&if _s∈ _ε(s),\\[8.61108pt] +∞&otherwise. cases Equivalently, Ωϵ,s=(1−ϵ)(Ω0,s∘Φϵ,s−1)+ιΔϵ(s) _ε,s=(1-ε) ( _0,s _ε,s^-1 )+ _ _ε(s), where ιC _C denotes the convex indicator of the set C, equal to 00 on C and +∞+∞ elsewhere. For ϵ=0ε=0 we recover Ω0 _0. Lemma G.10 (Properties of Ωϵ _ε). Let Ω0 _0 satisfy Assumption 2 with strong convexity modulus σ, and let ϵ∈[0,1)ε∈[0,1). Then Ωϵ,s _ε,s is proper, closed, and σ1−ϵ σ1-ε-strongly convex, with effective domain Φϵ,s(dom(Ω0,s)) _ε,s(dom( _0,s)); in particular Ωϵ _ε satisfies Assumption 2, and it satisfies Assumption 3 whenever Ω0 _0 does. Moreover: (i) Conjugate: Ωϵ∗(Q)(s)=ϵ⟨μs,Qs⟩+(1−ϵ)Ω0∗(Q)(s) ^*_ε(Q)(s)=ε _s,Q_s +(1-ε) ^*_0(Q)(s); in particular Ωϵ∗ ^*_ε is differentiable with Lipschitz gradient. (i) Greedy policy: πΩϵ,Q(a|s)=ϵμ(a|s)+(1−ϵ)πΩ0,Q(a|s) _ _ε,Q(a|s)=εμ(a|s)+(1-ε) _ _0,Q(a|s). (i) Domination profile: γΩϵ(R)=ϵ+(1−ϵ)γΩ0(R)≥ϵ _ _ε(R)=ε+(1-ε) _ _0(R)≥ε for every R≥0R≥ 0. (iv) Span of the conjugate at the origin: ‖Ωϵ∗(0)‖,sp=(1−ϵ)‖Ω0∗(0)‖,sp\| ^*_ε(0)\|_S,sp=(1-ε)\| ^*_0(0)\|_S,sp. (v) Regularization modulus: ΛΩϵ(R)≤ϵR+(1−ϵ)ΛΩ0(R) _ _ε(R)≤ε R+(1-ε) _ _0(R) for every R≥0R≥ 0. In particular, if Ω0 _0 satisfies the assumptions of Remark G.1, then ΛΩϵ(R)≤ϵR+(1−ϵ)τΔΩ0 _ _ε(R)≤ε R+(1-ε)τ _ _0. Proof of Lemma G.10. Fix s∈s and write μs=μ(⋅|s) _s=μ(·|s), Φ=Φϵ,s = _ε,s for the affine bijection of Definition G.3. Effective domain and properness. Since Ωϵ,s(Φ(p))=(1−ϵ)Ω0,s(p) _ε,s( (p))=(1-ε) _0,s(p) for p∈Δ()p∈ (A) and Ωϵ,s=+∞ _ε,s=+∞ outside Δϵ(s)=Φ(Δ()) _ε(s)= ( (A)), the effective domain of Ωϵ,s _ε,s is Φ(dom(Ω0,s)) (dom( _0,s)). It is non-empty since Ω0,s _0,s is proper, and Ωϵ,s _ε,s never takes the value −∞-∞ since Ω0,s _0,s does not; hence Ωϵ,s _ε,s is proper. Moreover Φ(μs)=μs ( _s)= _s, so Ωϵ,s(μs)=(1−ϵ)Ω0,s(μs) _ε,s( _s)=(1-ε) _0,s( _s), which gives Assumption 3 for Ωϵ _ε whenever it holds for Ω0 _0. Closedness. Δϵ(s) _ε(s) is the intersection of the simplex with the closed half-spaces πs:πs(a)≥ϵμ(a|s)\ _s: _s(a)≥εμ(a|s)\, hence closed and convex. The map Ω0,s∘Φ−1 _0,s ^-1 is proper, convex and lower semicontinuous as the composition of a proper, closed convex function with a continuous affine map, hence closed, a proper convex function being closed if and only if it is lower semicontinuous (Rockafellar, 1970, Theorem 7.1); and ιΔϵ(s) _ _ε(s) is closed since Δϵ(s) _ε(s) is closed. A sum of closed functions is closed. Strong convexity. Let πs,πs′∈dom(Ωϵ,s) _s,π _s ( _ε,s), t∈[0,1]t∈[0,1], and set p=Φ−1(πs)p= ^-1( _s), p′=Φ−1(πs′)p = ^-1(π _s), so that πs−πs′=(1−ϵ)(p−p′) _s-π _s=(1-ε)(p-p ). Since Φ−1 ^-1 is affine, Φ−1(tπs+(1−t)πs′)=tp+(1−t)p′ ^-1(t _s+(1-t)π _s)=tp+(1-t)p , and the σ-strong convexity of Ω0,s _0,s gives Ωϵ,s(tπs+(1−t)πs′) _ε,s(t _s+(1-t)π _s) =(1−ϵ)Ω0,s(tp+(1−t)p′) =(1-ε) _0,s\! (tp+(1-t)p ) ≤(1−ϵ)[tΩ0,s(p)+(1−t)Ω0,s(p′)−σ2t(1−t)‖p−p′‖2] ≤(1-ε) [t\, _0,s(p)+(1-t)\, _0,s(p )- σ2t(1-t)\|p-p \|^2 ] =tΩϵ,s(πs)+(1−t)Ωϵ,s(πs′)−σ2(1−ϵ)t(1−t)‖πs−πs′‖2, =t\, _ε,s( _s)+(1-t)\, _ε,s(π _s)- σ2(1-ε)t(1-t)\| _s-π _s\|^2, using ‖p−p′‖2=‖πs−πs′‖2/(1−ϵ)2\|p-p \|^2=\| _s-π _s\|^2/(1-ε)^2 in the last line. Hence Ωϵ,s _ε,s is σ1−ϵ σ1-ε-strongly convex, and together with τ>0τ>0 this shows that Ωϵ _ε satisfies Assumption 2. (i). The constraint πs∈Δϵ(s) _s∈ _ε(s) is exactly the range of Φ , so the change of variable πs=Φ(p)=ϵμs+(1−ϵ)p _s= (p)=ε _s+(1-ε)p is a bijection from Δ() (A) onto Δϵ(s) _ε(s). Since Ωϵ,s(Φ(p))=(1−ϵ)Ω0,s(p) _ε,s( (p))=(1-ε) _0,s(p) and ⟨Φ(p),Qs⟩=ϵ⟨μs,Qs⟩+(1−ϵ)⟨p,Qs⟩ (p),Q_s =ε _s,Q_s +(1-ε) p,Q_s , we get Ωϵ∗(Q)(s) _ε^*(Q)(s) =maxπs∈Δ()⟨πs,Qs⟩−τΩϵ,s(πs) = _ _s∈ (A) \ _s,Q_s -τ _ε,s( _s) \ =maxπs∈Δϵ(s)⟨πs,Qs⟩−τΩϵ,s(πs) = _ _s∈ _ε(s) \ _s,Q_s -τ _ε,s( _s) \ =maxp∈Δ()ϵ⟨μs,Qs⟩+(1−ϵ)⟨p,Qs⟩−τ(1−ϵ)Ω0,s(p) = _p∈ (A) \ε _s,Q_s +(1-ε) p,Q_s -τ(1-ε)\, _0,s(p) \ =ϵ⟨μs,Qs⟩+(1−ϵ)maxp∈Δ()⟨p,Qs⟩−τΩ0,s(p) =ε _s,Q_s +(1-ε) _p∈ (A) \ p,Q_s -τ _0,s(p) \ =ϵ⟨μs,Qs⟩+(1−ϵ)Ω0∗(Q)(s), =ε _s,Q_s +(1-ε) _0^*(Q)(s), where the second equality holds because Ωϵ,s=+∞ _ε,s=+∞ outside Δϵ(s) _ε(s), the third is the change of variable, the fourth uses 1−ϵ>01-ε>0, and the last is the definition of Ω0∗ _0^*. The map Qs↦ϵ⟨μs,Qs⟩Q_s ε _s,Q_s is linear with constant gradient ϵμsε _s, so Ωϵ∗ _ε^* is differentiable as soon as Ω0∗ _0^* is (which is the case by Lemma F.1.(i) since τ>0τ>0), with ∇Ωϵ∗=ϵμs+(1−ϵ)∇Ω0∗∇ _ε^*=ε _s+(1-ε)∇ _0^*; if ∇Ω0∗∇ _0^* is L0L_0-Lipschitz then ∇Ωϵ∗∇ _ε^* is (1−ϵ)L0(1-ε)L_0-Lipschitz. (i). The maximum over p in the third line above is attained at p⋆=argmaxp∈Δ()⟨p,Qs⟩−τΩ0,s(p)=πΩ0,Q(⋅|s)p = _p∈ (A)\ p,Q_s -τ _0,s(p)\= _ _0,Q(·|s), which is unique by Lemma F.1.(i) applied to Ω0 _0. Since Φ is a bijection, the maximizer of the original problem is πΩϵ,Q(⋅|s)=Φ(p⋆)=ϵμs+(1−ϵ)πΩ0,Q(⋅|s) _ _ε,Q(·|s)= (p )=ε _s+(1-ε) _ _0,Q(·|s). This is consistent with (i) and Lemma F.1.(i), which identify the greedy policy with ∇Ωϵ∗(Q)(s)∇ _ε^*(Q)(s). (i). Let (s,a)∈×(s,a) ×A and Q with ‖Q‖asp≤R\|Q\|_asp≤ R. Dividing (i) by μ(a|s)>0μ(a|s)>0 gives πΩϵ,Q(a|s)μ(a|s)=ϵ+(1−ϵ)πΩ0,Q(a|s)μ(a|s). _ _ε,Q(a|s)μ(a|s)=ε+(1-ε) _ _0,Q(a|s)μ(a|s). Taking the infimum over (s,a)(s,a) and over Q:‖Q‖asp≤R\Q:\|Q\|_asp≤ R\, we obtain γΩϵ(R)=ϵ+(1−ϵ)γΩ0(R) _ _ε(R)=ε+(1-ε) _ _0(R) from Definition G.1.(i). Since γΩ0(R)≥0 _ _0(R)≥ 0 and ϵ<1ε<1, we get γΩϵ(R)≥ϵ _ _ε(R)≥ε for every R≥0R≥ 0. (iv). Trivial from (i). (v). Let Q satisfy ‖Q‖asp≤R\|Q\|_asp≤ R and write gQϵg^ε_Q and gQ0g^0_Q for the regularization gaps of Ωϵ _ε and Ω0 _0. Subtracting maxaQs(a) _aQ_s(a) from (i) gives, for every s∈s , gQϵ(s)=ϵhQ(s)+(1−ϵ)gQ0(s),hQ(s):=⟨μs,Qs⟩−maxaQs(a).g^ε_Q(s)=ε\,h_Q(s)+(1-ε)\,g^0_Q(s), h_Q(s):= _s,Q_s - _aQ_s(a). Since μs∈Δ() _s∈ (A), we have hQ(s)∈[−‖Qs‖,sp,0]⊆[−R,0]h_Q(s)∈ [-\|Q_s\|_A,sp,0 ] [-R,0] for every s, hence ‖hQ‖,sp≤R\|h_Q\|_S,sp≤ R. The triangle inequality and Definition G.1.(iv) applied to Ω0 _0 give ‖gQϵ‖,sp≤ϵ‖hQ‖,sp+(1−ϵ)‖gQ0‖,sp≤ϵR+(1−ϵ)ΛΩ0(R).\|g^ε_Q\|_S,sp≤ε\|h_Q\|_S,sp+(1-ε)\|g^0_Q\|_S,sp≤ε R+(1-ε) _ _0(R). Taking the maximum over Q:‖Q‖asp≤R\Q:\|Q\|_asp≤ R\ yields ΛΩϵ(R)≤ϵR+(1−ϵ)ΛΩ0(R) _ _ε(R)≤ε R+(1-ε) _ _0(R). The last claim follows from Remark G.1 applied to Ω0 _0. ∎ Lemma G.11 (Global properties of Ωϵ _ε). Let Assumption 1 hold, let Ω0 _0 satisfy Assumption 2, let ϵ∈(0,1)ε∈(0,1), and let Q,Q′Q,Q be two Q-value functions. Then: (i) Global m-step contraction: ‖Ωϵm(Q)−Ωϵm(Q′)‖sp \|T^m_ _ε(Q)-T^m_ _ε(Q )\|_sp ≤(1−βm∏k=1m−1γΩϵ(R[Ωϵk(Q),Ωϵk(Q′)]))‖Q−Q′‖sp ≤ (1- _m _k=1^m-1 _ _ε (R[T_ _ε^k(Q),T_ _ε^k(Q )] ) )\|Q-Q \|_sp ≤(1−βmϵm−1)‖Q−Q′‖sp. ≤ (1- _mε^m-1 )\|Q-Q \|_sp. (i) Global coverage: let (Qn)n≥0(Q_n)_n≥ 0 be Q-functions and let (sn,an)n≥0(s_n,a_n)_n≥ 0 be a trajectory with an∼πΩϵ,Qn(⋅|sn)a_n _ _ε,Q_n(·|s_n) and sn+1∼p(⋅|sn,an)s_n+1 p(·|s_n,a_n); let ℱn:=σ(Q0,(s0,a0),…,(sn,an))F_n:=σ(Q_0,(s_0,a_0),…,(s_n,a_n)) and assume that QnQ_n is ℱn−1F_n-1-measurable for n≥1n≥ 1. Then for all t≥0t≥ 0 and all (s,a)(s,a): ℙ((st+m,at+m)=(s,a)∣ℱt)≥βmϵmη(s,a)≥βmϵmηmin.P ((s_t+m,a_t+m)=(s,a) _t )≥ _mε^mη(s,a)≥ _mε^m _ . (G.17) (i) Fixed point up to a constant: there exists QΩϵ∗∈ℝ×Q^*_ _ε ^S×A, unique up to an additive constant, and a unique ρΩϵ∗∈ℝρ^*_ _ε , such that Ωϵ(QΩϵ∗)=QΩϵ∗+ρΩϵ∗T_ _ε(Q^*_ _ε)=Q^*_ _ε+ρ^*_ _ε1. Proof of Lemma G.11. By Lemma G.10, Ωϵ _ε satisfies Assumption 2, so Lemma G.2 applies to Ωϵ _ε. (i). The first inequality is Lemma G.2.(iv) applied to Ωϵ _ε. By Lemma G.10.(i), γΩϵ(R)≥ϵ _ _ε(R)≥ε for every R≥0R≥ 0, so ∏k=1m−1γΩϵ(R[Ωϵk(Q),Ωϵk(Q′)])≥ϵm−1 _k=1^m-1 _ _ε(R[T_ _ε^k(Q),T_ _ε^k(Q )])≥ε^m-1 regardless of the radii, and the second inequality follows. (i). By Lemma G.10.(i), πΩϵ,Qn≥ϵμ _ _ε,Q_n≥εμ for every n, so Lemma G.2.(v) applies to Ωϵ _ε with the deterministic constants γ1=⋯=γm=ϵ _1=…= _m=ε. (i). By translation equivariance (Corollary F.1), ΩϵT_ _ε induces a well-defined map ¯ T on the quotient space ℝ×/ℝR^S×A/R1, which is complete for the norm induced by ∥⋅∥sp\|·\|_sp, and (i) makes ¯m T^m a contraction with constant 1−βmϵm−1<11- _mε^m-1<1. Banach’s theorem gives a unique fixed point Q¯∗ Q^* of ¯m T^m; since ¯ T commutes with ¯m T^m, ¯(Q¯∗) T( Q^*) is also a fixed point of ¯m T^m, hence equals Q¯∗ Q^*. Any representative QΩϵ∗Q^*_ _ε of Q¯∗ Q^* therefore satisfies Ωϵ(QΩϵ∗)=QΩϵ∗+ρΩϵ∗T_ _ε(Q^*_ _ε)=Q^*_ _ε+ρ^*_ _ε1 for some ρΩϵ∗∈ℝρ^*_ _ε , unique by Proposition G.3, and uniqueness of QΩϵ∗Q^*_ _ε up to an additive constant is that of Q¯∗ Q^*. ∎ Remark G.7 (Bias of the floor). Applying Proposition G.6 to Ωϵ _ε, which satisfies its assumptions by Lemma G.10, and using Lemma G.10.(i) and (v) gives ‖QΩϵ∗−Q∗‖sp≤m(ϵR∗+(1−ϵ)ΛΩ0(R∗))βm(ϵ+(1−ϵ)γΩ0(R∗))m−1,\|Q^*_ _ε-Q^*\|_sp≤ m (ε R^*+(1-ε) _ _0(R^*) ) _m(ε+(1-ε) _ _0(R^*))^m-1, where R∗R^* is the radius of Proposition G.6 associated with Ωϵ _ε. The floor therefore trades a global contraction rate βmϵm−1 _mε^m-1 against a bias term of order ϵR∗ε R^* which does not vanish as τ→0τ→ 0: small ϵε keeps the fixed point close to Q∗Q^* but degrades the rate, large ϵε does the opposite. Appendix H Proofs for the learning algorithms of Section 4 Proof techniques. Three ingredients drive the analysis. First, since ∇Ω∗∇ ^* is the greedy policy, differences of regularized Bellman operators are kernels of average-path policies (Lemma G.2); unlike the single randomized policy that represents differences of hard maxima (Chen et al., 2026, Lemma B.6), this policy inherits the domination of the greedy policies, which turns the Doeblin condition on μ into a Doeblin condition on every product of m such kernels, much as lower bounds on learning policies transfer irreducibility in discounted on-policy Q-learning (Nanda and Chen, 2026). Second, we analyze the unmodified one-step update although ΩT_ only contracts over m steps: expanding the Jacobian of a block of relaxed updates as an expectation over Bernoulli selections shows that the step size acts as a lazy transformation of the state-action kernels, the counterpart of the aperiodicity transformation of Puterman (1994, §8.5.4), so that blocks with at least m selected updates inherit the m-step contraction. This avoids both the multi-step estimators of Zhang et al. (2021) and the lazy transformation of the dynamics of Chen et al. (2026). Third, since the contraction rate degrades with the radius, a stopping-time argument keeps the iterates, with high probability, in a ball on which the rate holds, with a self-consistent choice of the radius. Throughout this section, Assumptions 1, 2 and 3 and (P1) hold, and QΩ∗Q^*_ denotes a solution of (G.9), unique up to an additive constant (Theorem G.1); every quantity below is invariant under the choice of that constant. The Q-iteration bound is the regularized counterpart of the span-contraction analysis of relative value iteration (Puterman, 1994, §8.5). For Q-learning, we first analyze the noiseless relaxed flow, which contracts over blocks of updates (Subsection H.2.1), then track the stochastic iterates along this flow up to their exit time from a span ball (Subsection H.2.2); the epoch algorithm follows the same scheme, with batch averaging and a cover-time bound (Subsection H.3). The results for general step-size schedules use the following assumption. Assumption H.1 (Step sizes). The sequence of step sizes (αt)t≥0( _t)_t≥ 0 is non-increasing, satisfies 0<αt≤10< _t≤ 1 for all t≥0t≥ 0, and ∑t≥0αt=+∞ _t≥ 0 _t=+∞. H.1 Proof of Corollary 1 See 1 Proof. Let QΩ∗Q^*_ be given by Theorem G.1. By translation equivariance (Corollary F.1), Ωk(QΩ∗)=QΩ∗+kρΩ∗T^k_ (Q^*_ )=Q^*_ +kρ^*_ 1 for every k≥0k≥ 0 and, since the projection Π subtracts a constant, Qj+k=Ωk(Qj)−cQ_j+k=T^k_ (Q_j)-c1 for some c∈ℝc . Since ∥⋅∥sp\|·\|_sp annihilates constants, ej+k=‖Ωk(Qj)−Ωk(QΩ∗)‖spe_j+k=\|T^k_ (Q_j)-T^k_ (Q^*_ )\|_sp for every j,k≥0j,k≥ 0. With k=1k=1, the non-expansiveness of ΩT_ in ∥⋅∥sp\|·\|_sp (Corollary F.1) shows that (ej)j(e_j)_j is non-increasing. With k=mk=m, Lemma G.2.(iv) applied to (Qj,QΩ∗)(Q_j,Q^*_ ) gives (1), since γΩ _ is non-increasing and R[Ωk(Qj),Ωk(QΩ∗)]≤‖QΩ∗‖asp+ejR[T_ ^k(Q_j),T_ ^k(Q_ ^*)]≤\|Q^*_ \|_asp+e_j for k∈ 0,m−1 ∈ 0,m-1 . Indeed, ∥⋅∥asp\|·\|_asp is invariant under constants, so ‖Ωk(QΩ∗)‖asp=‖QΩ∗‖asp\|T^k_ (Q^*_ )\|_asp=\|Q^*_ \|_asp, and the triangle inequality together with ∥⋅∥asp≤∥⋅∥sp\|·\|_asp≤\|·\|_sp gives ‖Ωk(Qj)‖asp≤‖QΩ∗‖asp+ej+k≤‖QΩ∗‖asp+ej\|T^k_ (Q_j)\|_asp≤\|Q^*_ \|_asp+e_j+k≤\|Q^*_ \|_asp+e_j. Finally, ej≤e0e_j≤ e_0 for every j and γΩ _ is non-increasing, so each factor in (1) is at most 1−βmγΩ(‖QΩ∗‖asp+e0)m−1≤1−β~1- _m _ (\|Q^*_ \|_asp+e_0)^m-1≤ 1- β. Hence ekm≤(1−β~)ke0≤e−β~ke0e_km≤(1- β)^ke_0≤ e^- βke_0, which is at most ε for k≥β~−1log(e0/ε)k≥ β^-1 (e_0/ ), and monotonicity extends the bound to every j≥kmj≥ km. Each sweep evaluates p(s′|s,a)p(s |s,a) for all (s,a,s′)(s,a,s ), i.e. uses ||2|||S|^2|A| queries, which gives the announced count. ∎ H.2 Proofs for Subsection 4.2 (Synchronous Ω -regularized Q-learning) H.2.1 Synchronous full-knowledge updates We let (Qt)t≥0(Q_t)_t≥ 0 denote the sequence produced from an arbitrary initialization Q0Q_0 by the full-knowledge synchronous update Qt+1=Ft(Qt),where Ft(Q):=(1−αt)Q+αtΩ(Q),Q_t+1=F_t(Q_t), F_t(Q):=(1- _t)Q+ _tT_ (Q), (H.1) and we measure the error in span seminorm, et:=‖Qt−QΩ∗‖spe_t:=\|Q_t-Q^*_ \|_sp. Without noise, (H.1) is a damped value iteration, whose linearization (1−αt)I+αtKπ(1- _t)I+ _tK_π is a lazy version of the greedy state-action kernel: the step size plays the role of the aperiodicity transformation of Puterman (1994, §8.5.4), which acts on the state kernel and which Chen et al. (2026) impose on the dynamics through lazy sampling. The analysis below exploits this laziness at the level of the Jacobians (Lemma H.7): it turns the m-step minorization of Assumption 1 into a contraction over blocks, and it is this deterministic flow that the stochastic analysis of Subsection H.2.2 tracks. The fixed point is stationary only up to a constant, so it is not a fixed point of FtF_t; the next lemma records that it nonetheless generates a trajectory of (H.1) along which the error is exactly ete_t. This is what allows us to dispense with a second sequence. Lemma H.1 (Reference trajectory). Let Qt∗:=QΩ∗+ctQ^*_t:=Q^*_ +c_t1 with ct:=ρΩ∗∑k<tαkc_t:=ρ^*_ _k<t _k. Then Qt+1∗=Ft(Qt∗)Q^*_t+1=F_t(Q^*_t) for every t≥0t≥ 0, and ‖Qt−Qt∗‖sp=et\|Q_t-Q^*_t\|_sp=e_t. Proof. By translation equivariance of ΩT_ (Corollary F.1) and the fixed-point equation, Ω(QΩ∗+c)=QΩ∗+(c+ρΩ∗)T_ (Q^*_ +c1)=Q^*_ +(c+ρ^*_ )1, hence Ft(QΩ∗+c)=QΩ∗+(c+αtρΩ∗)F_t(Q^*_ +c1)=Q^*_ +(c+ _tρ^*_ )1, which is the announced recursion. The second claim follows since the span seminorm annihilates constants. ∎ Lemma H.2 (Monotonicity of the error). For any Q0Q_0, the sequence (et)t≥0(e_t)_t≥ 0 is non-increasing. Proof. Using Ω(QΩ∗)=QΩ∗+ρΩ∗T_ (Q^*_ )=Q^*_ +ρ^*_ 1, we have Qt+1−QΩ∗=(1−αt)(Qt−QΩ∗)+αt(Ω(Qt)−Ω(QΩ∗))+αtρΩ∗Q_t+1-Q^*_ =(1- _t)(Q_t-Q^*_ )+ _t (T_ (Q_t)-T_ (Q^*_ ) )+ _tρ^*_ 1, and the constant term vanishes in the span seminorm. The triangle inequality and the non-expansiveness of ΩT_ in ∥⋅∥sp\|·\|_sp (Corollary F.1) then give et+1≤(1−αt)et+αtet=ete_t+1≤(1- _t)e_t+ _te_t=e_t. ∎ Definition H.1 (Sub-blocks and global blocks). Let κ0>0 _0>0 be a fixed sub-block budget. We define recursively the sub-blocks Jn=tn,…,tn+1−1J_n=\t_n,…,t_n+1-1\ by t0=0t_0=0 and tn+1=mint>tn|∑k=tnt−1αk≥κ0,t_n+1= \t>t_n | _k=t_n^t-1 _k≥ _0 \, (H.2) and the i-th global block as the chronological concatenation of m consecutive sub-blocks, ℬi=⋃ℓ=0m−1Jim+ℓ,bi:=minℬi=tim.B_i= _ =0^m-1J_im+ , b_i:= _i=t_im. (H.3) Note that these definitions are valid under Assumption H.1. We write Φ(i):=Fbi+1−1∘…∘Fbi ^(i):=F_b_i+1-1 … F_b_i, so that Qbi+1=Φ(i)(Qbi)Q_b_i+1= ^(i)(Q_b_i) and, by Lemma H.1, Qbi+1∗=Φ(i)(Qbi∗)Q^*_b_i+1= ^(i)(Q^*_b_i). Lemma H.3. Under Assumption H.1, for any sub-block JnJ_n we have ∑k∈Jnαk<κ0+α0 _k∈ J_n _k< _0+ _0, and consequently, for any global block ℬiB_i, ∑k∈ℬiαk<m(κ0+α0) _k _i _k<m( _0+ _0). Proof. This follows from the minimality in the definition of a sub-block (H.2) and Assumption H.1 on the step sizes. A global block is the chronological concatenation of m consecutive sub-blocks, which gives the second result. ∎ Lemma H.4 (Boundedness along the frozen flow). Let R¯i:=‖QΩ∗‖asp+ebi R_i:=\|Q^*_ \|_asp+e_b_i. The sequence (R¯i)i≥0( R_i)_i≥ 0 is non-increasing, and for every Q~∈[Qbi,Qbi∗] Q∈[Q_b_i,Q^*_b_i] and every k∈ℬik _i, the frozen-flow iterate Q~k:=Fk−1∘…∘Fbi(Q~) Q_k:=F_k-1 … F_b_i( Q) (with Q~bi:=Q~ Q_b_i:= Q) satisfies ‖Q~k‖asp≤R¯i\| Q_k\|_asp≤ R_i. Proof. The monotonicity of (R¯i)i( R_i)_i is that of (et)t(e_t)_t (Lemma H.2). Let now Q~=(1−u)Qbi+uQbi∗ Q=(1-u)Q_b_i+uQ^*_b_i for some u∈[0,1]u∈[0,1]. Since Qbi∗=QΩ∗+cbiQ^*_b_i=Q^*_ +c_b_i1, we get Q~−QΩ∗=(1−u)(Qbi−QΩ∗)+ucbi Q-Q^*_ =(1-u)(Q_b_i-Q^*_ )+uc_b_i1, and the constant is annihilated by the span seminorm, so ‖Q~−QΩ∗‖sp=(1−u)ebi≤ebi\| Q-Q^*_ \|_sp=(1-u)e_b_i≤ e_b_i. The sequence (Q~k)k≥bi( Q_k)_k≥ b_i follows the update (H.1) started at Q~ Q, so Lemma H.2 applied to it, whose proof holds for any starting time, gives ‖Q~k−QΩ∗‖sp≤‖Q~−QΩ∗‖sp≤ebi\| Q_k-Q^*_ \|_sp≤\| Q-Q^*_ \|_sp≤ e_b_i, whence ‖Q~k‖asp≤‖QΩ∗‖asp+‖Q~k−QΩ∗‖sp≤R¯i\| Q_k\|_asp≤\|Q^*_ \|_asp+\| Q_k-Q^*_ \|_sp≤ R_i. ∎ Lemma H.5 (Contraction of products of at least m regularized greedy kernels). Let R>0R>0, let (Uk)k∈(U_k)_k be a finite family of Q-functions with ‖Uk‖asp≤R\|U_k\|_asp≤ R for every k, and let Kk:=KπΩ,UkK_k:=K_ _ ,U_k, i.e. Kk(s′,a′|s,a)=p(s′|s,a)πΩ,Uk(a′|s′)K_k(s ,a |s,a)=p(s |s,a)\, _ ,U_k(a |s ). For a subset =k1<…<kn⊆C=\k_1<…<k_n\ , let K:=Kkn⋯Kk1K_C:=K_k_n·s K_k_1 be the associated ordered product. Then K_C is row-stochastic and: (i) if n≥mn≥ m, then ‖K‖sp,op≤1−β~m(R)\|K_C\|_sp,op≤ 1- β_m(R), where β~m(R):=βmγΩ(R)m−1 β_m(R):= _m\, _ (R)^m-1; (i) if n<mn<m, then ‖K‖sp,op≤1\|K_C\|_sp,op≤ 1. Proof. Each KkK_k is row-stochastic, hence so is K_C, and (i) follows from Lemma E.6. Assume n≥mn≥ m. With Convention E.1, the Markov chain associated with K=Kkn⋯Kk1K_C=K_k_n·s K_k_1 applies the factors in the order Kkn,Kkn−1,…,Kk1K_k_n,K_k_n-1,…,K_k_1. We therefore split off its last m steps in chain order, i.e. the m oldest algorithmic indices: K=HBK_C=HB with H:=Kkn⋯Kkm+1H:=K_k_n·s K_k_m+1 and B:=Kkm⋯Kk1B:=K_k_m·s K_k_1, and H=IH=I when n=mn=m. Since ‖Ukℓ‖asp≤R\|U_k_ \|_asp≤ R for each ℓ , Definition G.1.(i) gives πΩ,Ukℓ≥γΩ(R)μ _ ,U_k_ ≥ _ (R)μ, so Lemma E.4, applied to B, whose m−1m-1 intermediate policies are πΩ,Ukm,…,πΩ,Uk2 _ ,U_k_m,…, _ ,U_k_2, gives α(B)≤1−βmγΩ(R)m−1=1−β~m(R)α(B)≤ 1- _m _ (R)^m-1=1- β_m(R). Finally, H is row-stochastic, so Lemma E.6 and the submultiplicativity (E.5) give ‖K‖sp,op≤‖H‖sp,op‖B‖sp,op≤α(B)\|K_C\|_sp,op≤\|H\|_sp,op\,\|B\|_sp,op≤α(B). ∎ It remains to count the effective updates within a global block. This is where the sub-block structure is used, and a plain independence argument suffices: no concentration inequality is required (a Chernoff bound gives a sharper estimate for large m, see Remark H.2). Lemma H.6 (Effective updates within a global block). Let ℬiB_i be the i-th global block of Definition H.1, let (Xk)k∈ℬi(X_k)_k _i be auxiliary independent Bernoulli random variables with Xk∼ℬ(αk)X_k ( _k) and let Ni:=∑k∈ℬiXkN_i:= _k _iX_k. Under Assumption H.1, ℙ(Ni≥m)≥pm:=(1−e−κ0)m.P(N_i≥ m)≥ p_m:= (1-e^- _0 )^m. (H.4) Proof. For each of the m sub-blocks Jn⊆ℬiJ_n _i, let n:=⋃k∈JnXk=1U_n:= _k∈ J_n\X_k=1\ be the event that at least one effective update occurs in JnJ_n. By independence of the XkX_k and 1−u≤e−u1-u≤ e^-u, ℙ(n)=1−∏k∈Jn(1−αk)≥1−exp(−∑k∈Jnαk)≥1−e−κ0,P(U_n)=1- _k∈ J_n(1- _k)≥ 1- (- _k∈ J_n _k )≥ 1-e^- _0, the last inequality by the definition (H.2) of the sub-block. The sub-blocks of ℬiB_i have disjoint index sets, so the events im,…,im+m−1U_im,…,U_im+m-1 are independent, and ⋂ℓ=0m−1im+ℓ⊆Ni≥m _ =0^m-1U_im+ \N_i≥ m\. Hence ℙ(Ni≥m)≥∏ℓ=0m−1ℙ(im+ℓ)≥(1−e−κ0)mP(N_i≥ m)≥ _ =0^m-1P(U_im+ )≥(1-e^- _0)^m. ∎ Lemma H.7 (Jacobian bound over a global block). Under Assumption H.1, for every global block ℬiB_i and every Q~∈[Qbi,Qbi∗] Q∈[Q_b_i,Q^*_b_i], ‖DΦ(i)(Q~)‖sp,op≤1−pmβ~m(R¯i).\|D ^(i)( Q)\|_sp,op≤ 1-p_m\, β_m( R_i). Proof. By the chain rule and Corollary F.1, the Jacobian of Φ(i) ^(i) at Q~ Q is DΦ(i)(Q~)=∏k∈ℬi((1−αk)I+αkKk),Kk:=KπΩ,Q~k,Q~k:=Fk−1∘…∘Fbi(Q~),D ^(i)( Q)= _k _i ((1- _k)I+ _kK_k ), K_k:=K_ _ , Q_k, Q_k:=F_k-1 … F_b_i( Q), the product being written in decreasing order of k. By Lemma H.4, ‖Q~k‖asp≤R¯i\| Q_k\|_asp≤ R_i for every k∈ℬik _i, so the family (Kk)k∈ℬi(K_k)_k _i satisfies the hypotheses of Lemma H.5 with Uk=Q~kU_k= Q_k and R=R¯iR= R_i. We introduce a probabilistic argument to bound the Jacobian. Let (Xk)k∈ℬi(X_k)_k _i be the auxiliary Bernoulli variables of Lemma H.6, realized as the coordinate maps of the finite product space 0,1ℬi\0,1\^B_i endowed with ⨂k∈ℬiℬ(αk) _k _iB( _k), which is well defined since αk∈(0,1] _k∈(0,1] by Assumption H.1. The flow (H.1) being deterministic, this finite space carries all the randomness of the argument. We define Yk=(1−Xk)I+XkKkY_k=(1-X_k)I+X_kK_k, so that (1−αk)I+αkKk=[Yk](1- _k)I+ _kK_k=E[Y_k]. By independence of the YkY_k, we have ∏k∈ℬi[Yk]=[∏k∈ℬiYk] _k _iE[Y_k]=E [ _k _iY_k ], and the triangle inequality gives ‖DΦ(i)(Q~)‖sp,op=‖[∏k∈ℬiYk]‖sp,op≤[‖∏k∈ℬiYk‖sp,op].\|D ^(i)( Q)\|_sp,op= \|E [ _k _iY_k ] \|_sp,op [ \| _k _iY_k \|_sp,op ]. Let :=k∈ℬi∣Xk=1C:=\k _i X_k=1\, so that Ni=||N_i=|C|. Since the factors equal to I can be removed from the product, we have the pathwise identity ∏k∈ℬiYk=K _k _iY_k=K_C with the notation of Lemma H.5. On the event Ni≥m\N_i≥ m\, point (i) of Lemma H.5 gives ‖K‖sp,op≤1−β~m(R¯i)\|K_C\|_sp,op≤ 1- β_m( R_i), and on the event Ni<m\N_i<m\, point (i) gives ‖K‖sp,op≤1\|K_C\|_sp,op≤ 1; hence ∥K∥sp,op≤1−β~m(R¯i)Ni≥m\|K_C\|_sp,op≤ 1- β_m( R_i)1_\N_i≥ m\ pointwise. Taking expectations, ‖DΦ(i)(Q~)‖sp,op≤1−ℙ(Ni≥m)β~m(R¯i)≤1−pmβ~m(R¯i),\|D ^(i)( Q)\|_sp,op≤ 1-P(N_i≥ m)\, β_m( R_i)≤ 1-p_m β_m( R_i), the last inequality by Lemma H.6. ∎ Lemma H.8 (Per-block contraction). Under Assumption H.1, for every global block ℬiB_i, ebi+1≤(1−pmβ~m(R¯i))ebi.e_b_i+1≤ (1-p_m\, β_m( R_i) )e_b_i. Proof. By Lemma H.1, ebi+1=‖Φ(i)(Qbi)−Φ(i)(Qbi∗)‖spe_b_i+1=\| ^(i)(Q_b_i)- ^(i)(Q^*_b_i)\|_sp. The map Φ(i) ^(i) is continuously differentiable (Corollary F.1) and all its Jacobians DΦ(i)(Q~)D ^(i)( Q), Q~∈[Qbi,Qbi∗] Q∈[Q_b_i,Q^*_b_i], are products of row-stochastic matrices, hence row-stochastic, so the mean value inequality in the span seminorm (Proposition E.1) applies and gives ebi+1≤supQ~∈[Qbi,Qbi∗]‖DΦ(i)(Q~)‖sp,op‖Qbi−Qbi∗‖spe_b_i+1≤ _ Q∈[Q_b_i,Q^*_b_i]\|D ^(i)( Q)\|_sp,op\,\|Q_b_i-Q^*_b_i\|_sp. Lemma H.7 bounds the supremum by 1−pmβ~m(R¯i)1-p_m β_m( R_i), while ‖Qbi−Qbi∗‖sp=ebi\|Q_b_i-Q^*_b_i\|_sp=e_b_i. ∎ Corollary H.1 (Full-knowledge updates for m=1m=1). Assume m=1m=1 and αt∈(0,1] _t∈(0,1] for every t. For any initialization Q0Q_0, the iterates of (H.1) satisfy eT≤∏k=0T−1(1−αkβ1)e0.e_T≤ _k=0^T-1(1- _k _1)\,e_0. In particular, Q-iteration (αt≡1 _t≡ 1) contracts at rate 1−β11- _1 per sweep, and reaches eT≤εe_T≤ after O(||2||β1−1log(e0/ε))O (|S|^2|A|\, _1^-1 (e_0/ ) ) model queries. Proof. Let ΨT:=FT−1∘…∘F0 _T:=F_T-1 … F_0 and Q~∈[Q0,Q0∗] Q∈[Q_0,Q^*_0]. As in Lemma H.7, DΨT(Q~)=∏k=0T−1((1−αk)I+αkPk)D _T( Q)= _k=0^T-1 ((1- _k)I+ _kP_k ). By Lemma E.4 with n=1n=1, each factor satisfies ((1−αk)I+αkKk)(s′,a′|s,a)≥αkKk(s′,a′|s,a)≥αkβ1ν(s′)πΩ,Q~k(a′|s′), ((1- _k)I+ _kK_k )(s ,a |s,a)≥ _kK_k(s ,a |s,a)≥ _k _1\,ν(s )\, _ , Q_k(a |s ), which is a Doeblin condition with parameter αkβ1 _k _1. Lemmas E.3 and E.6 give ‖(1−αk)I+αkPk‖sp,op≤1−αkβ1 \|(1- _k)I+ _kP_k \|_sp,op≤ 1- _k _1, and the submultiplicativity of the operator span seminorm together with Proposition E.1 and Lemma H.1 conclude. For αt≡1 _t≡ 1, the update (H.1) is Q-iteration without the centering projection, which is invisible to ∥⋅∥sp\|·\|_sp, so eT≤(1−β1)Te0≤εe_T≤(1- _1)^Te_0≤ for T≥β1−1log(e0/ε)T≥ _1^-1 (e_0/ ), each sweep using ||2|||S|^2|A| queries. ∎ Lemma H.9 (Number of completed blocks). Under Assumption H.1, let I(T):=maxi∣bi≤TI(T):= \i b_i≤ T\ be the number of global blocks completed within the first T updates. Then I(T)≥⌊∑t=0T−1αtm(κ0+α0)⌋I(T)≥ _t=0^T-1 _tm( _0+ _0) . Proof. By definition of I(T)I(T) we have bI(T)≤T<bI(T)+1b_I(T)≤ T<b_I(T)+1, and by Lemma H.3 each global block consumes at most m(κ0+α0)m( _0+ _0). The step sizes being non-negative, ∑k=0T−1αk≤∑k=0bI(T)+1−1αk<(I(T)+1)m(κ0+α0),i.e.I(T)>∑k=0T−1αkm(κ0+α0)−1, _k=0^T-1 _k≤ _k=0^b_I(T)+1-1 _k<(I(T)+1)\,m( _0+ _0), .e. I(T)> _k=0^T-1 _km( _0+ _0)-1, and I(T)I(T) being an integer gives the claim. ∎ Remark H.1 (Radius-adaptive rate). Chaining Lemma H.8 over the completed blocks gives the radius-adaptive bound ebI(T)≤∏i<I(T)(1−pmβmγΩ(R¯i)m−1)e0e_b_I(T)≤ _i<I(T) (1-p_m _m _ ( R_i)^m-1 )e_0, in which the per-block rate improves as the error decreases, since R¯i=‖QΩ∗‖asp+ebi R_i=\|Q^*_ \|_asp+e_b_i is non-increasing. The uniform constant β~m(R¯0) β_m( R_0) is its worst case, attained at the first block. Asymptotically, R¯i↓‖QΩ∗‖asp R_i \|Q^*_ \|_asp, so the rate approaches 1−pmβmγΩ(‖QΩ∗‖asp)m−11-p_m _m _ (\|Q^*_ \|_asp)^m-1: the local geometry of the fixed point, not the initialization, governs the asymptotic speed. The bounds of Theorems H.1 and H.2 use instead the worst-case constant β¯(R) β(R) of Definition H.2 on the whole ball of radius R; making them radius-adaptive would require shrinking the radius along the run. Remark H.2 (An alternative bound on ℙ(Ni≥m)P(N_i≥ m) through a Chernoff inequality). Each global block has a total budget α¯i:=∑k∈ℬiαk≥mκ0 α_i:= _k _i _k≥ m _0, so for κ0>1 _0>1 the multiplicative Chernoff inequality for the lower tail (Mitzenmacher and Upfal, 2005, Theorem 4.5) gives ℙ(Ni<m)≤exp(−(α¯i−m)22α¯i)≤exp(−(κ0−1)22κ0m),P(N_i<m)≤ (- ( α_i-m)^22 α_i )≤ (- ( _0-1)^22 _0m ), the last inequality because u↦(u−m)22u (u-m)^22u has derivative 12(1−m2u2)>0 12 (1- m^2u^2 )>0 on (m,∞)(m,∞) and is therefore minimized at u=mκ0u=m _0. This variant requires κ0>1 _0>1 but is much better for large m, since it tends to 11 whereas (1−e−κ0)m(1-e^- _0)^m decays exponentially in m. Both bounds being valid, Lemma H.6 holds with pm=max(1−e−κ0)m, 1−exp(−(κ0−1)22κ0m)κ0>1.p_m= \ (1-e^- _0 )^m,\ 1- (- ( _0-1)^22 _0m )1\ _0>1\ \. The proof of Theorem 1 uses κ0=4 _0=4, for which pm≥1−e−9/8≥12p_m≥ 1-e^-9/8≥ 12. H.2.2 Setting and localized constants We now study Algorithm 3, the synchronous Ω -regularized Q-learning algorithm, and write et:=‖Qt−QΩ∗‖spe_t:=\|Q_t-Q^*_ \|_sp for its error. We define the filtration ℱt:=σQ0,(sk′(s,a))(s,a)∈×:k<tF_t:=σ \Q_0,\ (s _k(s,a) )_(s,a) ×A:k<t \ and write the stochastic update as Qt+1=Qt+αt(Ω(Qt)−Qt+Wt+1),Q_t+1=Q_t+ _t (T_ (Q_t)-Q_t+W_t+1 ), (H.5) where for every pair (s,a)(s,a) the next state st′(s,a)∼p(⋅|s,a)s _t(s,a) p(·|s,a) is sampled independently and Wt+1(s,a):=Ω∗(Qt)(st′(s,a))−∑s′p(s′|s,a)Ω∗(Qt)(s′).W_t+1(s,a):= ^*(Q_t) (s _t(s,a) )- _s p(s |s,a)\, ^*(Q_t)(s ). (H.6) By construction Wt+1W_t+1 is ℱt+1F_t+1-measurable and [Wt+1∣ℱt]=0E[W_t+1 _t]=0. By translation equivariance of Ω∗ ^* (Lemma F.1.(i)), the centering projection of Algorithm 3 only shifts the iterates by a constant, which ∥⋅∥sp\|·\|_sp and ∥⋅∥asp\|·\|_asp annihilate, so we omit it; its only role is to upgrade span convergence to ∥⋅∥∞\|·\|_∞ convergence (Remark F.2). We reuse the sub-blocks JnJ_n and global blocks ℬiB_i of Definition H.1; the step sizes being deterministic, the block boundaries bib_i, the number I(T)I(T) of completed blocks, the bound Imax:=⌊1mκ0∑t<Tαt⌋I_ := 1m _0 _t<T _t and the last incomplete block ℬres:= bI(T),T−1 _res:= b_I(T),T-1 are all deterministic. Definition H.2 (Localized radii, exit time and localized constants). For a radius R≥e0R≥ e_0, define the exit time σR:=inft∈ℕ|et>R∈ℕ∪+∞, _R:= \t \ |\ e_t>R \ ∪\+∞\, (H.7) the domination radius and the noise radius AR:=‖QΩ∗‖asp+R,CR:=‖QΩ∗‖sp+R,A_R:=\|Q^*_ \|_asp+R, C_R:=\|Q^*_ \|_sp+R, and the localized constants β~m(AR):=βmγΩ(AR)m−1,β¯(R):=pmβ~m(AR)∈(0,1),w¯(R):=CR+‖Ω∗(0)‖,sp, β_m(A_R):= _m\, _ (A_R)^m-1, β(R):=p_m\, β_m(A_R)∈(0,1), w(R):=C_R+ \| ^*(0) \|_S,sp, (H.8) where pmp_m is the constant of Lemma H.6 (see also Remark H.2). Remark H.3. (i) σR _R is an (ℱt)(F_t)-stopping time and σR≥1 _R≥ 1 since e0≤Re_0≤ R. (i) By minimality, t<σRt< _R implies et≤Re_t≤ R, hence ‖Qt‖sp≤CR\|Q_t\|_sp≤ C_R and, since ∥⋅∥asp≤∥⋅∥sp\|·\|_asp≤\|·\|_sp, also ‖Qt‖asp≤‖QΩ∗‖asp+‖Qt−QΩ∗‖asp≤AR\|Q_t\|_asp≤\|Q^*_ \|_asp+\|Q_t-Q^*_ \|_asp≤ A_R. (i) γΩ _ is non-increasing (Definition G.1.(i)), so R↦β¯(R)R β(R) is non-increasing and any contraction estimate valid at a radius A≤ARA≤ A_R holds a fortiori with β¯(R) β(R); moreover β¯(R)>0 β(R)>0 by (P1). (iv) The localization serves two purposes: it bounds the noise scale by w¯(R) w(R), which grows linearly in R through the sub-affinity of Ω∗ ^* (Lemma F.1.(vii)), and, since γΩ _ may vanish at infinity, it keeps the contraction rate at least β¯(R) β(R). H.2.3 The noiseless flow before the exit time For each global block ℬiB_i we define the virtual sequence (Q~k)k≥bi( Q_k)_k≥ b_i by Q~bi:=Qbi,Q~k+1:=Q~k+αk(Ω(Q~k)−Q~k), Q_b_i:=Q_b_i, Q_k+1:= Q_k+ _k (T_ ( Q_k)- Q_k ), (H.9) i.e. the full-knowledge synchronous update (H.1) started at QbiQ_b_i; for i=I(T)i=I(T), it is defined on the last incomplete block ℬresB_res. It is a deterministic function of QbiQ_b_i, hence ℱbiF_b_i-measurable. Lemma H.10 (Contraction of the virtual flow at radius R). Let i∈ℕi with ebi≤Re_b_i≤ R. Then, pathwise: (i) ‖Q~k−QΩ∗‖sp≤ebi\| Q_k-Q^*_ \|_sp≤ e_b_i for every k≥bik≥ b_i; (i) ‖Q~bi+1−QΩ∗‖sp≤(1−β¯(R))ebi\| Q_b_i+1-Q^*_ \|_sp≤ (1- β(R) )e_b_i. Proof. The virtual sequence is exactly the full-knowledge update (H.1) started at QbiQ_b_i, so (i) is Lemma H.2 applied to that sequence, and (i) is Lemma H.8 applied within the single block ℬiB_i, which gives the factor 1−pmβ~m(R¯i)1-p_m β_m( R_i) where R¯i=‖QΩ∗‖asp+ebi R_i=\|Q^*_ \|_asp+e_b_i is the block radius of Lemma H.4. It remains to check R¯i≤AR R_i≤ A_R, which follows from ebi≤Re_b_i≤ R; γΩ _ being non-increasing, β~m(R¯i)≥β~m(AR) β_m( R_i)≥ β_m(A_R), i.e. 1−pmβ~m(R¯i)≤1−β¯(R)1-p_m β_m( R_i)≤ 1- β(R). ∎ H.2.4 Localized noise control Lemma H.11 (Stopped noise). Define W^t+1:=t<σRWt+1 W_t+1:=1_\t< _R\W_t+1. Then W^t+1 W_t+1 is ℱt+1F_t+1-measurable, [W^t+1∣ℱt]=0E [ W_t+1 _t ]=0, and each entry W^t+1(s,a) W_t+1(s,a) lies in an ℱtF_t-measurable interval containing 00 and of length at most w¯(R) w(R); in particular ‖W^t+1‖∞≤w¯(R) \| W_t+1 \|_∞≤ w(R). Moreover W^t+1=Wt+1 W_t+1=W_t+1 on t<σR\t< _R\. Proof. Since σR _R is a stopping time, t<σR=σR≤tc∈ℱt\t< _R\=\ _R≤ t\^c _t, so [W^t+1∣ℱt]=t<σR[Wt+1∣ℱt]=0E[ W_t+1 _t]=1_\t< _R\E[W_t+1 _t]=0. Conditionally on ℱtF_t, Ω∗(Qt) ^*(Q_t) is fixed, and Wt+1(s,a)W_t+1(s,a) is a value of Ω∗(Qt) ^*(Q_t) minus the ℱtF_t-measurable convex combination ∑s′p(s′|s,a)Ω∗(Qt)(s′) _s p(s |s,a)\, ^*(Q_t)(s ); it therefore lies in an ℱtF_t-measurable interval containing 00 and of length ‖Ω∗(Qt)‖,sp\| ^*(Q_t)\|_S,sp, and so does W^t+1(s,a) W_t+1(s,a), with length t<σR∥Ω∗(Qt)∥,sp1_\t< _R\\| ^*(Q_t)\|_S,sp. On t<σR\t< _R\ we have ‖Qt‖sp≤CR\|Q_t\|_sp≤ C_R by Remark H.3.(i), and Lemma F.1.(vii) gives ‖Ω∗(Qt)‖,sp≤‖Qt‖sp+‖Ω∗(0)‖,sp≤CR+‖Ω∗(0)‖,sp=w¯(R)\| ^*(Q_t)\|_S,sp≤\|Q_t\|_sp+\| ^*(0)\|_S,sp≤ C_R+\| ^*(0)\|_S,sp= w(R). Off the event, W^t+1=0 W_t+1=0. ∎ For an interval = ta,tb = t_a,t_b we define the stopped noise martingale U^ta():=0 U^(K)_t_a:=0, U^k+1():=U^k()+αkW^k+1 U^(K)_k+1:= U^(K)_k+ _k W_k+1 for k∈k , we write v:=∑k∈αk2v_K:= _k _k^2, and we set U¯(δ′) U_K(δ ) :=w¯(R)12vlog(2||||δ′), := w(R) 12v_K ( 2|S||A|δ ), (H.10) ε(δ′) _K(δ ) :=2(1+2∑k∈αk)U¯(δ′)≤ 2(1+2m(κ0+α0))U¯(δ′), :=2 (1+2 _k _k ) U_K(δ )\ ≤\ 2 (1+2m( _0+ _0) ) U_K(δ ), (H.11) the last inequality by Lemma H.3 when K is contained in a global block. Lemma H.12 (Good event). Let δ∈(0,1)δ∈(0,1) and δ′:=δImax+1δ := δI_ +1. The event GR(δ):=⋂i=0I(T)−1maxbi≤k≤bi+1‖U^k(ℬi)‖∞≤U¯ℬi(δ′)∩maxbI(T)≤k≤T‖U^k(ℬres)‖∞≤U¯ℬres(δ′)G_R(δ):= _i=0^I(T)-1 \ _b_i≤ k≤ b_i+1 \| U^(B_i)_k \|_∞≤ U_B_i(δ ) \∩ \ _b_I(T)≤ k≤ T \| U^(B_res)_k \|_∞≤ U_B_res(δ ) \ (H.12) satisfies ℙ(GR(δ))≥1−δP (G_R(δ) )≥ 1-δ. Proof. Fix an interval K among ℬ0,…,ℬI(T)−1,ℬresB_0,…,B_I(T)-1,B_res and a pair (s,a)(s,a). By Lemma H.11, the scalar process (U^k()(s,a))k ( U^(K)_k(s,a) )_k is an ℱkF_k-martingale whose increment αkW^k+1(s,a) _k W_k+1(s,a) lies in an ℱkF_k-measurable interval of length at most αkw¯(R) _k w(R), so ∑k∈(αkw¯(R))2=w¯(R)2v _k ( _k w(R) )^2= w(R)^2v_K and the maximal Azuma–Hoeffding inequality (Lemma E.2) bounds the failure probability of each brace, for each pair, by δ′|||| δ |S||A|. A union bound over the |||||S||A| pairs and the I(T)+1≤Imax+1I(T)+1≤ I_ +1 intervals concludes. ∎ H.2.5 Tracking before the exit time Lemma H.13 (Tracking up to the exit time). Work on the event GR(δ)G_R(δ). Let i≤I(T)i≤ I(T), let :=ℬiK:=B_i if i<I(T)i<I(T) and :=ℬresK:=B_res if i=I(T)i=I(T), write ta:=bit_a:=b_i and tb+1:=bi+1t_b+1:=b_i+1 (resp. T), and let Q~ Q be the virtual sequence started at QtaQ_t_a. Then, for every k with ta≤k≤mintb+1,σRt_a≤ k≤ \t_b+1, _R\, the tracking error Ek:=Qk−Q~kE_k:=Q_k- Q_k satisfies ‖Ek‖sp≤ε(δ′)\|E_k\|_sp≤ _K(δ ). Proof. If σR≤ta _R≤ t_a the range reduces to k=tak=t_a, where Eta=0E_t_a=0. Otherwise, for ta≤k<mintb+1,σRt_a≤ k< \t_b+1, _R\ the true update (H.5) occurs and Wk+1=W^k+1W_k+1= W_k+1 by Lemma H.11. Subtracting (H.9) from (H.5) and using the average-path identity of Lemma G.2.(i) at the pair (Qk,Q~k)(Q_k, Q_k), i.e. Ω(Qk)−Ω(Q~k)=K¯kEkT_ (Q_k)-T_ ( Q_k)= K_kE_k with K¯k:=Kπ¯Ω,[Qk,Q~k] K_k:=K_ π_ ,[Q_k, Q_k], Ek+1=(1−αk)Ek+αkK¯kEk+αkW^k+1=MkEk+αkW^k+1,E_k+1=(1- _k)E_k+ _k K_kE_k+ _k W_k+1=M_kE_k+ _k W_k+1, (H.13) where Mk:=(1−αk)I+αkK¯kM_k:=(1- _k)I+ _k K_k is row-stochastic as a convex combination of row-stochastic matrices. Decompose Ek=R^k+U^k()E_k= R_k+ U^(K)_k with R^ta=0 R_t_a=0. The noise terms in (H.13) cancel with the definition of U^k+1() U^(K)_k+1, so that R^k+1=MkR^k+αk(K¯k−I)U^k() R_k+1=M_k R_k+ _k ( K_k-I ) U^(K)_k. Since K¯k K_k is row-stochastic, every entry of K¯kU^k() K_k U^(K)_k is a convex combination of entries of U^k() U^(K)_k, hence lies in [minU^k(),maxU^k()] [ U^(K)_k, U^(K)_k ], and so does every entry of U^k() U^(K)_k; therefore every entry of (K¯k−I)U^k() ( K_k-I ) U^(K)_k lies in [−‖U^k()‖sp,‖U^k()‖sp] [-\| U^(K)_k\|_sp,\| U^(K)_k\|_sp ] and ‖αk(K¯k−I)U^k()‖sp≤2αk‖U^k()‖sp≤4αk‖U^k()‖∞ \| _k ( K_k-I ) U^(K)_k \|_sp≤ 2 _k \| U^(K)_k \|_sp≤ 4 _k \| U^(K)_k \|_∞. By Lemma E.6 the row-stochastic matrix MkM_k is span non-expansive, so ‖R^k+1‖sp≤‖R^k‖sp+4αk‖U^k()‖∞\| R_k+1\|_sp≤\| R_k\|_sp+4 _k\| U^(K)_k\|_∞. Summing from tat_a and using the definition of GR(δ)G_R(δ), for every k∈ ta,mintb+1,σR ∈ t_a, \t_b+1, _R\ , ‖R^k‖sp≤4(∑k′∈αk′)U¯(δ′)\| R_k\|_sp≤ 4 ( _k _k ) U_K(δ ) and ‖Ek‖sp≤‖R^k‖sp+2‖U^k()‖∞≤2(1+2∑k′∈αk′)U¯(δ′)=ε(δ′).\|E_k\|_sp≤\| R_k\|_sp+2 \| U^(K)_k \|_∞≤ 2 (1+2 _k _k ) U_K(δ )= _K(δ ). ∎ H.2.6 No escape before the horizon Proposition H.1 (No escape). Let δ∈(0,1)δ∈(0,1), R≥e0R≥ e_0, δ′=δImax+1δ = δI_ +1 and ε¯R(δ):=maxmax0≤i<I(T)εℬi(δ′),εℬres(δ′). _R(δ):= \ _0≤ i<I(T) _B_i(δ ),\ _B_res(δ ) \. Assume the radius admissibility condition ε¯R(δ)≤β¯(R)β¯(R)+1(R−e0). _R(δ)≤ β(R) β(R)+1 (R-e_0 ). (H.14) Then, on the event GR(δ)G_R(δ), we have σR>T _R>T, in particular sup0≤t≤Tet≤R _0≤ t≤ Te_t≤ R, and for every 0≤i≤I(T)0≤ i≤ I(T), ebi≤(1−β¯(R))ie0+∑ℓ=0i−1(1−β¯(R))i−1−ℓεℬℓ(δ′).e_b_i≤(1- β(R))^ie_0+ _ =0^i-1(1- β(R))^i-1- _B_ (δ ). (H.15) Proof. Work on GR(δ)G_R(δ). We first record that (H.15) implies, bounding the geometric sum and using (H.14), ebi≤e0+ε¯R(δ)β¯(R),henceebi+ε¯R(δ)≤e0+β¯(R)+1β¯(R)ε¯R(δ)≤R.e_b_i≤ e_0+ _R(δ) β(R), e_b_i+ _R(δ)≤ e_0+ β(R)+1 β(R) _R(δ)≤ R. (H.16) We prove by induction on i∈ 0,I(T) ∈ 0,I(T) that σR>bi _R>b_i and (H.15) holds at rank i. Base case. b0=0b_0=0 and σR≥1 _R≥ 1 by Remark H.3.(i); (H.15) at i=0i=0 reads e0≤e0e_0≤ e_0. Inductive step. Assume the claim at rank i<I(T)i<I(T) and let Q~ Q be the virtual sequence started at QbiQ_b_i; by (H.16), ebi≤Re_b_i≤ R, so Lemma H.10 applies to ℬiB_i. Suppose σR≤bi+1 _R≤ b_i+1; since σR>bi _R>b_i, we would have t⋆:=σR∈ bi+1,bi+1 := _R∈ b_i+1,b_i+1 , and as t⋆≤minbi+1,σRt ≤ \b_i+1, _R\, Lemma H.13 gives ‖Qt⋆−Q~t⋆‖sp≤εℬi(δ′)≤ε¯R(δ)\|Q_t - Q_t \|_sp≤ _B_i(δ )≤ _R(δ) while Lemma H.10.(i) gives ‖Q~t⋆−QΩ∗‖sp≤ebi\| Q_t -Q^*_ \|_sp≤ e_b_i. Hence, by (H.16), eσR≤ebi+ε¯R(δ)≤Re_ _R≤ e_b_i+ _R(δ)≤ R, contradicting the definition (H.7) of σR _R. Therefore σR>bi+1 _R>b_i+1; in particular bi+1≤minbi+1,σRb_i+1≤ \b_i+1, _R\, so Lemma H.13 at k=bi+1k=b_i+1 combined with Lemma H.10.(i) yields ebi+1≤‖Q~bi+1−QΩ∗‖sp+‖Qbi+1−Q~bi+1‖sp≤(1−β¯(R))ebi+εℬi(δ′)e_b_i+1≤\| Q_b_i+1-Q^*_ \|_sp+\|Q_b_i+1- Q_b_i+1\|_sp≤(1- β(R))e_b_i+ _B_i(δ ), which, injected into (H.15) at rank i, gives it at rank i+1i+1. Residual interval. The last step gives σR>bI(T) _R>b_I(T). If σR≤T _R≤ T, the same argument on ℬresB_res — Lemma H.13 with =ℬresK=B_res and Lemma H.10.(i) — gives eσR≤ebI(T)+ε¯R(δ)≤Re_ _R≤ e_b_I(T)+ _R(δ)≤ R, again a contradiction. Hence σR>T _R>T and the containment follows from the definition of σR _R. ∎ H.2.7 Main result Theorem H.1 (Finite-time bound for Ω -regularized synchronous Q-learning). Let δ∈(0,1)δ∈(0,1) and T≥1T≥ 1. Under Assumption H.1, let the sub-block budget κ0>0 _0>0 be arbitrary and let the radius R≥e0R≥ e_0 satisfy the admissibility condition (H.14). Then, with probability at least 1−δ1-δ, sup0≤t≤Tet≤R _0≤ t≤ Te_t≤ R and eT e_T ≤(1−β¯(R))I(T)e0+2(1+2m(κ0+α0))w¯(R)12log(2||||(Imax+1)δ) ≤(1- β(R))^I(T)e_0+2 (1+2m( _0+ _0) ) w(R) 12 ( 2|S||A|(I_ +1)δ ) ×(∑i=0I(T)−1(1−β¯(R))I(T)−1−i∑k∈ℬiαk2+∑k∈ℬresαk2), × ( _i=0^I(T)-1(1- β(R))^I(T)-1-i _k _i _k^2+ _k _res _k^2 ), where β¯(R) β(R) and w¯(R) w(R) are the localized constants of Definition H.2, I(T)≥⌊∑t<Tαtm(κ0+α0)⌋I(T)≥ _t<T _tm( _0+ _0) is the number of completed blocks and Imax≥I(T)I_ ≥ I(T). Proof. Work on GR(δ)G_R(δ), which holds with probability at least 1−δ1-δ by Lemma H.12. Proposition H.1 gives σR>T _R>T, so the ranges of Lemma H.13 cover every block entirely, as well as the residual block. Applying Lemma H.13 with =ℬresK=B_res at k=Tk=T, and Lemma H.10.(i) to the virtual sequence started at QbI(T)Q_b_I(T), the triangle inequality gives eT≤ebI(T)+εℬres(δ′)e_T≤ e_b_I(T)+ _B_res(δ ). Injecting (H.15) at i=I(T)i=I(T) and the explicit form (H.11) of the ε(δ′) _K(δ ) yields the announced bound; the lower bound on I(T)I(T) is Lemma H.9. ∎ Corollary H.2 (Constant step size). Under the assumptions of Theorem H.1, let the step size be constant, αt=α∈(0,1) _t=α∈(0,1), and set Cκ:=2(1+2m(κ0+α))C_κ:= 2 (1+2m( _0+α) ). If the radius R≥e0R≥ e_0 satisfies Cκw¯(R)m(κ0+α)αlog(2||||(αTmκ0+1)δ)≤β¯(R)β¯(R)+1(R−e0),C_κ\, w(R) m( _0+α)\,α\, ( 2|S||A| ( α Tm _0+1 )δ )≤ β(R) β(R)+1 (R-e_0 ), (H.17) then (H.14) holds and, with probability at least 1−δ1-δ, eT e_T ≤11−β¯(R)(1−β¯(R))αTm(κ0+α)e0 ≤ 11- β(R) (1- β(R) ) α Tm( _0+α)e_0 +(β¯(R)+1)Cκw¯(R)β¯(R)m(κ0+α)αlog(2||||(αTmκ0+1)δ). + ( β(R)+1)C_κ w(R) β(R) m( _0+α)\,α\, ( 2|S||A| ( α Tm _0+1 )δ ). Proof. With a constant step size, every sub-block has exactly ⌈κ0/α⌉ _0/α indices, so |ℬi|=m⌈κ0/α⌉|B_i|=m _0/α and, using ⌈κ0/α⌉<κ0+α _0/α < _0+α and ⌊x⌋>x−1 x >x-1, I(T)>αTm(κ0+α)−1I(T)> α Tm( _0+α)-1; moreover vℬi=|ℬi|α2≤m(κ0+α)αv_B_i=|B_i|α^2≤ m( _0+α)α for every i, and the same bound holds for vℬresv_B_res since ℬres⊆ℬI(T)B_res _I(T). Hence, for every interval K among the ℬiB_i and ℬresB_res, ε(δ′)≤2(1+2m(κ0+α))w¯(R)12m(κ0+α)αlog(2||||(αTmκ0+1)δ), _K(δ )≤ 2 (1+2m( _0+α) ) w(R) 12m( _0+α)α ( 2|S||A| ( α Tm _0+1 )δ ), which is the left-hand side of (H.17), so (H.17) implies (H.14). Theorem H.1 then applies, and the noise term is summed using ∑i=0I(T)−1(1−β¯(R))I(T)−1−i≤1β¯(R) _i=0^I(T)-1(1- β(R))^I(T)-1-i≤ 1 β(R), so that the bracket is at most (1β¯(R)+1)m(κ0+α)α ( 1 β(R)+1 ) m( _0+α)α. Finally 1−β¯(R)∈(0,1)1- β(R)∈(0,1) and I(T)>αTm(κ0+α)−1I(T)> α Tm( _0+α)-1 give the prefactor 11−β¯(R) 11- β(R) on the contraction term. ∎ H.2.8 Proof of Theorem 1 See 1 Proof of Theorem 1. Take κ0=4 _0=4 and the Chernoff bound of Remark H.2, so that pm≥1−e−9/8≥12p_m≥ 1-e^-9/8≥ 12; for α≤12α≤ 12, we also have m(κ0+α)≤92m( _0+α)≤ 92m and Cκ≤2(1+9m)C_κ≤ 2(1+9m), and α(e0)≤12α(e_0)≤ 12 since w¯≥e0 w≥ e_0. We apply Corollary H.2 with R=2e0R=2e_0: then AR=‖QΩ∗‖asp+2e0A_R=\|Q^*_ \|_asp+2e_0, so that β¯(R)=pmβ~m(AR)≥β~/2 β(R)=p_m β_m(A_R)≥ β/2 and w¯(R)=w¯ w(R)= w, and the logarithm in (H.17) is at most ι since αTmκ0+1≤T+1 α Tm _0+1≤ T+1. Admissibility. The left-hand side of (H.17) is at most 2(1+9m)w¯92mαι=3(1+9m)w¯mαι 2(1+9m) w 92mα =3(1+9m) w mα , which is at most β~2+β~e0 β2+ βe_0 when α≤α(e0)α≤α(e_0). Since x↦x1+x x1+x is increasing and β¯(R)≥β~/2 β(R)≥ β/2, this is at most β¯(R)1+β¯(R)(R−e0) β(R)1+ β(R)(R-e_0), which is (H.17). Bound. With probability at least 1−δ1-δ, Theorem H.1 thus gives supt≤Tet≤2e0 _t≤ Te_t≤ 2e_0. Since β¯(R)≥β~/2 β(R)≥ β/2, I(T)≥0I(T)≥ 0, and I(T)>αTm(κ0+α)−1≥2αT9m−1I(T)> α Tm( _0+α)-1≥ 2α T9m-1, its contraction term satisfies (1−β¯(R))I(T)e0≤(1−β~/2)I(T)e0≤(1−β~/2)2αT9m−1e0(1- β(R))^I(T)e_0≤(1- β/2)^I(T)e_0≤(1- β/2) 2α T9m-1e_0. Since x↦(1+x)/x (1+x)/x is non-increasing, the noise term of Corollary H.2 is at most 2+β~β~ 3(1+9m)w¯mαι 2+ β β\,3(1+9m) w mα . This is (2). Sample complexity. For ε∈(0,e0) ∈(0,e_0), the choice α=α(ε/2)≤α(e0)α=α( /2)≤α(e_0) makes the noise term equal to ε/2 /2, and (1−β~/2)x≤e−β~x/2(1- β/2)^x≤ e^- βx/2 makes the contraction term at most ε/2 /2 as soon as T≥9mαβ~log(2e0(1−β~/2)ε)T≥ 9mα β ( 2e_0(1- β/2) ), which is possible since ι depends on T only logarithmically. With α(ε/2)=Θ~(β~2ε2/(m3w¯2))α( /2)= ( β^2 ^2/(m^3 w^2) ), this gives O~(m4w¯2/(β~3ε2)) O (m^4 w^2/( β^3 ^2) ) iterations, each drawing |||||S||A| transitions. Since the radius is 2e02e_0, the constants β~ β and w¯ w do not depend on ε . ∎ Corollary H.3 (Synchronous Ω -regularized Q-learning for m=1m=1). Let m=1m=1. Then β~=β1 β= _1, and Theorem 1 holds even without (P1). Moreover ΩT_ is a (1−β1)(1- _1)-contraction for ∥⋅∥sp\|·\|_sp and ‖QΩ∗‖sp≤(‖r‖sp+‖Ω∗(0)‖,sp)/β1\|Q^*_ \|_sp≤(\|r\|_sp+\| ^*(0)\|_S,sp)/ _1; in particular, for Q0=0Q_0=0 and a regularizer whose minimal value does not depend on the state, w¯≤3‖r‖sp/β1 w≤ 3\|r\|_sp/ _1, and the sample complexity of Theorem 1 is O~(||||‖r‖sp2/(β15ε2)) O (|S||A|\,\|r\|_sp^2/( _1^5 ^2) ). Proof. For m=1m=1, β~1(R)=β1 β_1(R)= _1 for every R, so the localized constants of Definition H.2 do not depend on the radius and are positive without (P1), which the proof of Theorem 1 only uses to ensure β~>0 β>0; that proof then applies verbatim. For the second claim, Lemma G.2.(iv) with m=1m=1, whose product is empty, gives ‖Ω(Q)−Ω(Q′)‖sp≤(1−β1)‖Q−Q′‖sp\|T_ (Q)-T_ (Q )\|_sp≤(1- _1)\|Q-Q \|_sp. Since ∥⋅∥sp\|·\|_sp annihilates constants, ‖QΩ∗‖sp=‖Ω(QΩ∗)‖sp≤(1−β1)‖QΩ∗‖sp+‖Ω(0)‖sp\|Q^*_ \|_sp=\|T_ (Q^*_ )\|_sp≤(1- _1)\|Q^*_ \|_sp+\|T_ (0)\|_sp, and ‖Ω(0)‖sp=‖r+PΩ∗(0)‖sp≤‖r‖sp+‖Ω∗(0)‖,sp\|T_ (0)\|_sp=\|r+P\, ^*(0)\|_sp≤\|r\|_sp+\| ^*(0)\|_S,sp. Finally, for Q0=0Q_0=0, e0=‖QΩ∗‖spe_0=\|Q^*_ \|_sp, and for such a regularizer ‖Ω∗(0)‖,sp=0\| ^*(0)\|_S,sp=0 (Corollary F.1), so w¯=3‖QΩ∗‖sp≤3‖r‖sp/β1 w=3\|Q^*_ \|_sp≤ 3\|r\|_sp/ _1. ∎ H.3 Proofs for Subsection 4.3 (Epoch Ω -regularized Q-learning) Every constant below is a function of the domination profile γΩ _ (Definition G.1.(i)) evaluated at the radius actually reached by the iterates, which drives both the contraction rate and the exploration rate. Algorithm 1 has two noise handling mechanisms: the step sizes αj _j, which averages across epochs, and the batch size BjB_j, which averages within an epoch. We therefore control the noise in two ways, both valid at all parameters: • a batch bound (Lemma H.21.(i)), which bounds the per-epoch noise by O(w¯(R)/Bj)O( w(R)/ B_j) and sums it over a block by the triangle inequality; • a block-martingale bound (Lemma H.21.(i)), which concatenates all the transition-level increments of a block into a single martingale, as in Subsection H.2. Neither dominates: for constant parameters their ratio is (1+2Nα)/N(1+2Nα)/ N, with N the number of epochs of a block, so the batch bound is tighter at αj≡1 _j≡ 1 (Corollary H.4), the setting of Theorem 2, where the block-martingale bound loses a factor Θ(m) ( m) on the noise term, while the block-martingale bound wins for small step sizes and handles small batches, down to Bj≡1B_j≡ 1, where it recovers Theorem H.1 (Remark H.6). H.3.1 Setting, terminology and the epoch update We write ej:=‖Qj−QΩ∗‖spe_j:=\|Q_j-Q^*_ \|_sp for the error after j epochs and, for a radius R≥e0R≥ e_0, we use the localized constants ARA_R, CRC_R, β~m(AR) β_m(A_R), β¯(R) β(R) and w¯(R) w(R) of Definition H.2; under (P1), β~m(AR) β_m(A_R) and β¯(R) β(R) are positive. We reserve the index j and the notation EjE_j for algorithmic epochs, and the index i for the blocks ℬiB_i introduced by the analysis; BjB_j is always the batch size. Three time scales coexist: transitions (n), epochs (j) and blocks (i). The j-th epoch is Ej:=nj,…,nj+1−1E_j:=\n_j,…,n_j+1-1\ with n0:=0n_0:=0 and nj+1:=minn>nj|min(s,a)∑k=njn−1(sk,ak)=(s,a)≥Bj.n_j+1:= \n>n_j\ |\ _(s,a) _k=n_j^n-11_\(s_k,a_k)=(s,a)\≥ B_j \. We reuse the sub-blocks and blocks ℬiB_i of Definition H.1 on the epoch scale, i.e. with the iteration index replaced by the epoch index, so that mκ0≤∑j∈ℬiαj<m(κ0+α0)m _0≤ _j _i _j<m( _0+ _0) (Definitions H.1 and H.3), and we write bi:=minℬib_i:= _i. The step sizes and batch sizes being deterministic, so are the blocks and the bib_i. We write ℓj:=nj+1−nj∈ℕ∪+∞ _j:=n_j+1-n_j ∪\+∞\ for the (random) length of epoch j, with the convention Qj+1:=Qjon ℓj=+∞.Q_j+1:=Q_j \ _j=+∞\. (H.18) Let j:=σ(Q0,(s0,a0),…,(snj−1,anj−1),snj)G_j:=σ (Q_0,(s_0,a_0),…,(s_n_j-1,a_n_j-1),s_n_j ); then QjQ_j is jG_j-measurable and, by construction of Algorithm 1, the behavior policy πj:=πΩ,Qj _j:= _ ,Q_j is frozen throughout epoch j. Set ℋn(j):=σ(j,(snj,anj),…,(snj+n,anj+n))H^(j)_n:=σ (G_j,(s_n_j,a_n_j),…,(s_n_j+n,a_n_j+n) ). As in Subsection H.2.2, the centering projection Π only shifts Qj+1Q_j+1 by a constant, which leaves the greedy policies, hence the law of the trajectory, as well as ∥⋅∥sp\|·\|_sp and ∥⋅∥asp\|·\|_asp unchanged (Corollary F.1 and Lemma F.1.(i),(i)), so we omit it; its only role is to upgrade span convergence to ∥⋅∥∞\|·\|_∞ convergence (Remark F.2). Convention H.1 (Extended trajectory). Within epoch j we let the trajectory be defined for all n≥0n≥ 0 under πj _j: s~0(j):=snj s^(j)_0:=s_n_j, a~n(j)∼πj(⋅|s~n(j)) a^(j)_n _j(·| s^(j)_n), s~n+1(j)∼p(⋅|s~n(j),a~n(j)) s^(j)_n+1 p(·| s^(j)_n, a^(j)_n), coupled with the algorithm for n<ℓjn< _j. All visit times below refer to this extended trajectory, which conditionally on jG_j is a time-homogeneous Markov chain with kernel KπjK_ _j; its filtration ℋ~n(j) H^(j)_n coincides with ℋn(j)H^(j)_n on n<ℓj\n< _j\. The extension is a definition and requires no probabilistic input; Proposition H.2 shows a posteriori that it is almost surely never used. Lemma H.14 (Exact form of the epoch update). (i) On ℓj<+∞\ _j<+∞\ we have Nj(s,a)=BjN_j(s,a)=B_j for every pair at exit, and Qj+1=Fj(Qj)+αjWj+1,Fj(Q):=(1−αj)Q+αjΩ(Q),Q_j+1=F_j(Q_j)+ _jW_j+1, F_j(Q):=(1- _j)Q+ _jT_ (Q), (H.19) where, with t1(s,a)<⋯<tBj(s,a)t_1(s,a)<…<t_B_j(s,a) the first BjB_j visit times to (s,a)(s,a) in the extended trajectory of epoch j, counted from the beginning njn_j of the epoch, Wj+1(s,a) W_j+1(s,a) =1Bj∑l=1BjZl(j)(s,a), = 1B_j _l=1^B_jZ^(j)_l(s,a), (H.20) Zl(j)(s,a) Z^(j)_l(s,a) :=Ω∗(Qj)(s~(j)tl(s,a)+1)−s′∼p(⋅|s,a)[Ω∗(Qj)(s′)]. := ^*(Q_j) ( s^(j)_t_l(s,a)+1 )-E_s p(·|s,a) [ ^*(Q_j)(s ) ]. (i) Every QjQ_j is a well-defined jG_j-measurable random vector of ℝ×R^S×A, with ‖Qj‖∞<+∞\|Q_j\|_∞<+∞ everywhere. Consequently πj _j is a well-defined policy and the extended trajectory of Convention H.1 is a well-defined Markov chain, for every j and on every path. Proof. (i). The inner loop takes into account a transition at (s,a)(s,a) only while Nj(s,a)<BjN_j(s,a)<B_j, so it accumulates exactly the first BjB_j visits and exits when the last pair reaches BjB_j; hence Nj≡BjN_j≡ B_j at exit and tl(s,a)<ℓjt_l(s,a)< _j for l≤Bjl≤ B_j, so the extended trajectory coincides with the algorithm at every visit time used and at the next state. After normalization the estimate is ΩQj^(s,a)=r(s,a)+1Bj∑lΩ∗(Qj)(s~tl(s,a)+1(j)) T_ Q_j(s,a)=r(s,a)+ 1B_j _l ^*(Q_j)( s^(j)_t_l(s,a)+1), the reward being deterministic; subtracting Ω(Qj)(s,a)T_ (Q_j)(s,a) gives (H.20), and injecting into the update gives (H.19). (i). By induction on j; the argument is deterministic. The initialization Q0Q_0 is a fixed finite vector. Assume QjQ_j finite. On ℓj=+∞\ _j=+∞\, (H.18) gives Qj+1=QjQ_j+1=Q_j. On ℓj<+∞\ _j<+∞\, (i) gives Qj+1=Fj(Qj)+αjWj+1Q_j+1=F_j(Q_j)+ _jW_j+1, and each coordinate of Wj+1W_j+1 is a difference between a value of Ω∗(Qj) ^*(Q_j) and a convex combination of such values, so ‖Wj+1‖∞≤‖Ω∗(Qj)‖,sp≤‖Qj‖sp+‖Ω∗(0)‖,sp<+∞\|W_j+1\|_∞≤\| ^*(Q_j)\|_S,sp≤\|Q_j\|_sp+\| ^*(0)\|_S,sp<+∞ by Lemma F.1.(vii); hence Qj+1Q_j+1 is finite. Measurability is that of the update, and πj=∇Ω∗(Qj) _j=∇ ^*(Q_j) is well defined for any finite argument by Lemma F.1.(i). ∎ The length ℓj _j is random, but because of the cap the number of terms in (H.20) is deterministic, equal to BjB_j. The martingale arguments of Lemmas H.19 and H.20 rely on this, the second one through the deterministic length ∑j∈ℬiBj _j _iB_j of the concatenated martingale. H.3.2 Exit time, well-posedness and epoch length We define the exit time of the iterates in the epoch time scale, at fixed radius R≥e0R≥ e_0, by σR:=infj≥0|ej>R∈ℕ∪+∞. _R:= \j≥ 0\ |\ e_j>R \ ∪\+∞\. (H.21) Remark H.4 (Exit time). The iterates are frozen within an epoch, so σR _R is a stopping time for (j)j≥0(G_j)_j≥ 0 and j<σR∈j\j< _R\ _j. Since e0≤Re_0≤ R, σR≥1 _R≥ 1. As in Remark H.3.(i), j<σRj< _R implies ‖Qj‖sp≤CR\|Q_j\|_sp≤ C_R and ‖Qj‖asp≤AR\|Q_j\|_asp≤ A_R. As in Remark H.3.(iv), the localization bounds the noise and, since γΩ _ may vanish at infinity, keeps both the contraction and the coverage rates at least at their values at radius ARA_R. Lemma H.15 (Conditional cover time at a deterministic rate). Let j≥0j≥ 0, let u≥0u≥ 0 be a radius and let δ′∈(0,1)δ ∈(0,1). Set j(u):=∥Qj∥asp≤u∈j,q(u):=βmγΩ(u)mηmin,D_j(u):= \\|Q_j\|_asp≤ u \ _j, q(u):= _m\, _ (u)^m\, _ , (H.22) K(u,δ′):=⌈(Bj+ι(δ′))2q(u)⌉,L(u,δ′):=mK(u,δ′)+1,ι(δ′):=log(||||δ′).K(u,δ ):= ( B_j+ (δ ) )^2q(u) , L(u,δ ):=m\,K(u,δ )+1, (δ ):= ( |S||A|δ ). (H.23) Under (P1), q(u)>0q(u)>0 and, almost surely, ℙ(ℓj>L(u,δ′)∩j(u)|j)≤δ′ 1j(u).P ( \ _j>L(u,δ ) \ _j(u)\ |\ G_j )\ ≤\ δ \,1_D_j(u). (H.24) Moreover L(u,δ′)≥m||||BjL(u,δ )≥ m|S||A|B_j. In particular, for R≥e0R≥ e_0, the coverage constant and the localized length qR:=q(AR)=γΩ(AR)β~m(AR)ηmin,Lj(δ′):=L(AR,δ′)=m⌈(Bj+ι(δ′))2qR⌉+1q_R:=q(A_R)= _ (A_R)\, β_m(A_R)\, _ , L_j(δ ):=L(A_R,δ )=m ( B_j+ (δ ) )^2q_R +1 (H.25) satisfy ℙ(ℓj>Lj(δ′)∩j<σR∣j)≤δ′j<σRP ( \ _j>L_j(δ ) \∩\j< _R\ _j )≤δ 1_\j< _R\ almost surely. Proof. Since j(u)∈jD_j(u) _j, it suffices to prove ℙ(ℓj>L(u,δ′)∣j)≤δ′P( _j>L(u,δ ) _j)≤δ almost surely on j(u)D_j(u); we work under a regular conditional probability given jG_j and fix an outcome in j(u)D_j(u). All the constants q(u)>0q(u)>0, K:=K(u,δ′)K:=K(u,δ ) and L(u,δ′)L(u,δ ) are deterministic, u being deterministic; positivity of q(u)q(u) is (P1) at the deterministic radius u. By Lemma H.14.(i), the extended trajectory of epoch j is well defined and, conditionally on jG_j, is a time-homogeneous Markov chain with kernel KπjK_ _j; the conditioning is legitimate because j(u)∈j⊆ℋ~n(j)D_j(u) _j H^(j)_n. On j(u)D_j(u), Definition G.1.(i) applied at radius u gives πj(a|s)≥γΩ(u)μ(a|s) _j(a|s)≥ _ (u)μ(a|s) for all (s,a)(s,a), so Lemma G.2.(i) applied with π1=⋯=πm=πj _1=·s= _m= _j gives, for every n≥0n≥ 0 and every (s,a)∈×(s,a) ×A, ℙ((s~n+m(j),a~n+m(j))=(s,a)|ℋ~n(j))=Kπjm(s,a|s~n(j),a~n(j))≥βmγΩ(u)mη(s,a)≥q(u).P ( ( s^(j)_n+m, a^(j)_n+m )=(s,a)\ |\ H^(j)_n )=K_ _j^m (s,a\,|\, s^(j)_n, a^(j)_n )\ ≥\ _m\, _ (u)^m\,η(s,a)\ ≥\ q(u). (H.26) Fix (s,a)(s,a) and set Zl:=(s~(j)lm,a~(j)lm)=(s,a)∈[0,1]Z_l:=1_\( s^(j)_lm, a^(j)_lm)=(s,a)\∈[0,1] for l∈ 1,K ∈ 1,K , adapted to lsub:=ℋ~lm(j)G^sub_l:= H^(j)_lm. Consecutive sub-sampling instants being exactly m steps apart, (H.26) gives [Zl∣l−1sub]≥q(u)E[Z_l ^sub_l-1]≥ q(u), so the predictable mean satisfies S¯≥Kq(u) S≥ Kq(u) deterministically. The exact multiplicative-Chernoff form (E.1) of Lemma E.1, applied with increment bound b=1b=1, a=Kq(u)a=Kq(u) and c=Bjc=B_j, all deterministic, gives ℙ(∑l=1KZl<Bj|j)≤exp(−Bjh(θ)),θ:=Kq(u)Bj,h(θ):=θ−1−logθ,P ( _l=1^KZ_l<B_j\ |\ G_j )≤ (-B_j\,h(θ) ), θ:= Kq(u)B_j, h(θ):=θ-1- θ, (H.27) the hypothesis c<ac<a holding since Kq(u)≥(Bj+ι(δ′))2>BjKq(u)≥( B_j+ (δ ))^2>B_j. Now h(θ)≥(θ−1)2h(θ)≥( θ-1)^2 for θ≥1θ≥ 1: both sides vanish at θ=1θ=1 and h′(θ)=1−θ−1≥1−θ−1/2h (θ)=1-θ^-1≥ 1-θ^-1/2 there, the right-hand side being the derivative of θ↦(θ−1)2θ ( θ-1)^2. By the choice of K, θ≥(1+ι(δ′)/Bj)2θ≥(1+ (δ )/B_j)^2, hence Bjh(θ)≥Bj(θ−1)2≥ι(δ′)B_jh(θ)≥ B_j( θ-1)^2≥ (δ ) and the right-hand side of (H.27) is at most δ′/(||||)δ /(|S||A|). A union bound over the |||||S||A| pairs shows that, with conditional probability at least 1−δ′1-δ , every pair is visited at least BjB_j times among the instants m,2m,…,Km⊆ 0,mK \m,2m,…,Km\ 0,mK , so ℓj≤mK+1=L(u,δ′) _j≤ mK+1=L(u,δ ). This is (H.24). Finally q(u)≤ηmin≤(||||)−1q(u)≤ _ ≤(|S||A|)^-1, so K(u,δ′)≥||||BjK(u,δ )≥|S||A|B_j and L(u,δ′)≥m||||BjL(u,δ )≥ m|S||A|B_j. For the last claim, Remark H.4 gives j<σR⊆j(AR)\j< _R\ _j(A_R), both events belonging to jG_j; apply (H.24) with the deterministic radius u=ARu=A_R and intersect with j<σR\j< _R\. ∎ Proposition H.2 (Well-posedness). For every j≥0j≥ 0, ℓj<+∞ _j<+∞ almost surely. Consequently convention (H.18) is vacuous, the extension of Convention H.1 is almost surely never used, and Algorithm 1 produces the update (H.19) at every epoch almost surely. Proof. Fix j. For every deterministic u≥0u≥ 0 and every δ′∈(0,1)δ ∈(0,1), integrating (H.24) gives ℙ(ℓj=+∞)≤ℙ(ℓj>L(u,δ′)∩j(u))+ℙ(j(u)c)≤δ′+ℙ(∥Qj∥asp>u).P ( _j=+∞ )\ ≤\ P ( \ _j>L(u,δ ) \ _j(u) )+P (D_j(u)^c )\ ≤\ δ +P (\|Q_j\|_asp>u ). Letting δ′↓0δ 0 at fixed u gives ℙ(ℓj=+∞)≤ℙ(‖Qj‖asp>u)P( _j=+∞) (\|Q_j\|_asp>u) for every u, and letting u↑+∞u +∞ gives ℙ(ℓj=+∞)=0P( _j=+∞)=0, since ‖Qj‖asp≤2‖Qj‖∞<+∞\|Q_j\|_asp≤ 2\|Q_j\|_∞<+∞ everywhere by Lemma H.14.(i). ∎ Given a transition budget T≥0T≥ 0 we write J(T):=maxj∣nj≤T,I(T):=maxi∣bi≤J(T),J(T):= \j n_j≤ T\, I(T):= \i b_i≤ J(T)\, (H.28) and ℬres:= bI(T),J(T)−1 ⊆ℬI(T)B_res:= b_I(T),J(T)-1 _I(T) for the last incomplete block. Lemma H.16 (Epoch and block counts). Let T≥0T≥ 0, B¯:=minjBj≥1 B:= _jB_j≥ 1. Then: (i) Deterministic upper bounds: ℓj≥||||Bj _j≥|S||A|B_j, hence J(T)≤Jmax(T):=⌊T||||B¯⌋J(T)≤ J_ (T):= T|S||A| B and I(T)≤Imax(T):=⌊Jmax(T)m⌋I(T)≤ I_ (T):= J_ (T)m . (i) From epochs to blocks: I(T)>∑j<J(T)αjm(κ0+α0)−1I(T)> _j<J(T) _jm( _0+ _0)-1, i.e. I(T)≥⌊∑j<J(T)αjm(κ0+α0)⌋I(T)≥ _j<J(T) _jm( _0+ _0) . If αj≡α _j≡α, every block contains exactly m⌈κ0/α⌉m _0/α epochs and I(T)=⌊J(T)m⌈κ0/α⌉⌋I(T)= J(T)m _0/α . (i) From transitions to epochs: set J−(T):=maxJ∣∑j<JLj(δ′)≤TJ^-(T):= \J _j<JL_j(δ )≤ T \. If ℓj≤Lj(δ′) _j≤ L_j(δ ) for every j<minσR,Jmax(T)j< \ _R,J_ (T)\ and σR>J(T) _R>J(T), then J(T)≥J−(T)J(T)≥ J^-(T). In particular for αj≡α _j≡α, Bj≡B_j≡ B, writing L(δ′)L(δ ) for the common value, J(T)≥⌊TL(δ′)⌋,I(T)≥⌊αTm(κ0+α)L(δ′)−αm(κ0+α)⌋.J(T)≥ TL(δ ) , I(T)≥ α Tm( _0+α)L(δ )- αm( _0+α) . (H.29) Proof. (i). Epoch j terminates only when each of the |||||S||A| pairs has been visited at least BjB_j times, and each transition visits one pair, so ℓj≥||||Bj _j≥|S||A|B_j; summing, J(T)||||B¯≤nJ(T)≤TJ(T)|S||A| B≤ n_J(T)≤ T. Each sub-block is non-empty, so each block has at least m epochs. (i). By definition of I(T)I(T), bI(T)+1>J(T)b_I(T)+1>J(T), and each block consumes at most m(κ0+α0)m( _0+ _0) (Lemma H.3), so ∑j<J(T)αj≤∑j<bI(T)+1αj<(I(T)+1)m(κ0+α0) _j<J(T) _j≤ _j<b_I(T)+1 _j<(I(T)+1)m( _0+ _0); the floor form follows since I(T)I(T) is an integer. For αj≡α _j≡α the stopping rule fires after exactly ⌈κ0/α⌉ _0/α epochs. (i). If J(T)=Jmax(T)J(T)=J_ (T) there is nothing to prove, since J−(T)≤Jmax(T)J^-(T)≤ J_ (T) by the bound Lj(δ′)≥m||||BjL_j(δ )≥ m|S||A|B_j of Lemma H.15. Otherwise suppose J(T)<J−(T)J(T)<J^-(T). Every j≤J(T)j≤ J(T) satisfies j<σRj< _R and j<Jmax(T)j<J_ (T), hence ℓj≤Lj(δ′) _j≤ L_j(δ ), so nJ(T)+1=∑j≤J(T)ℓj≤∑j<J(T)+1Lj(δ′)≤Tn_J(T)+1= _j≤ J(T) _j≤ _j<J(T)+1L_j(δ )≤ T, the last inequality because J(T)+1≤J−(T)J(T)+1≤ J^-(T); this contradicts the maximality of J(T)J(T). The constant case follows by injecting J(T)≥⌊T/L(δ′)⌋J(T)≥ T/L(δ ) into the exact identity of (i) and using ⌊x⌋>x−1 x >x-1 along with ⌈κ0/α⌉<(κ0+α)/α _0/α <( _0+α)/α. ∎ Remark H.5. Only (i) is used for the union bounds, and it is deterministic: this is what lets δ′δ be fixed in advance although J(T)J(T) and I(T)I(T) are random. There is no circularity in (i): Proposition H.3 establishes σR>J(T) _R>J(T) using only the almost sure finiteness of Proposition H.2, and only then are the length bounds available. H.3.3 The noiseless flow before the exit time For each block ℬiB_i define the virtual sequence by Q~bi:=Qbi Q_b_i:=Q_b_i and Q~j+1:=Fj(Q~j) Q_j+1:=F_j( Q_j) for j≥bij≥ b_i, i.e. the full-knowledge synchronous update (H.1) started at QbiQ_b_i; it is biG_b_i-measurable. Lemma H.17 (Contraction of the virtual flow at radius R). Let i be such that ebi≤Re_b_i≤ R. Then, pathwise: (i) ‖Q~j−QΩ∗‖sp≤ebi\| Q_j-Q^*_ \|_sp≤ e_b_i for every j≥bij≥ b_i; (i) ‖Q~bi+1−QΩ∗‖sp≤(1−β¯(R))ebi\| Q_b_i+1-Q^*_ \|_sp≤(1- β(R))\,e_b_i. Proof. This is Lemma H.10 with the step index replaced by the epoch index: the statement concerns the full-knowledge update (H.1) only, and ℬiB_i is a concatenation of m sub-blocks in the sense of Definition H.1. ∎ H.3.4 Localized noise control The noise increments below lie in intervals whose endpoints are known one step ahead, which is the setting of the maximal Azuma–Hoeffding inequality of Lemma E.2. We write W^j+1:=j<σRWj+1 W_j+1:=1_\j< _R\W_j+1 and Z^(j)l:=j<σRZ(j)l Z^(j)_l:=1_\j< _R\Z^(j)_l for the stopped noise, so that W^j+1=1Bj∑lZ^l(j) W_j+1= 1B_j _l Z^(j)_l. Lemma H.18 (Epoch noise: martingale structure and range). Fix x=(s,a)x=(s,a) and let l(j):=σ(ℋ~tl(x)(j),s~tl(x)+1(j))K^(j)_l:=σ ( H^(j)_t_l(x), s^(j)_t_l(x)+1 ) for l∈ 1,Bj ∈ 1,B_j , 0(j):=jK^(j)_0:=G_j. Then: (i) 0(j)⊆⋯⊆Bj(j)⊆j+1⊆0(j+1)K^(j)_0 … ^(j)_B_j _j+1 ^(j+1)_0, so the concatenation over j is a filtration at fixed x; in particular Wj+1W_j+1 is j+1G_j+1-measurable; (i) [Z^l(j)(x)∣l−1(j)]=0E[ Z^(j)_l(x) ^(j)_l-1]=0; consequently [W^j+1∣j]=0E[ W_j+1 _j]=0; (i) Z^l(j)(x) Z^(j)_l(x) lies in a jG_j-measurable interval of length j<σR∥Ω∗(Qj)∥,sp≤w¯(R)1_\j< _R\\| ^*(Q_j)\|_S,sp≤ w(R). Proof. (i). The tl(x)t_l(x) are stopping times for (ℋ~n(j))n( H^(j)_n)_n, a.s. finite by Proposition H.2, and tl−1(x)+1≤tl(x)t_l-1(x)+1≤ t_l(x), so l−1(j)⊆ℋ~tl(x)(j)⊆l(j)K^(j)_l-1 H^(j)_t_l(x) ^(j)_l; and tBj(x)+1≤ℓjt_B_j(x)+1≤ _j, the extended trajectory coinciding with the algorithm before ℓj _j, gives Bj(j)⊆j+1K^(j)_B_j _j+1. (i). On tl(x)=n\t_l(x)=n\ we have (s~n(j),a~n(j))=x( s^(j)_n, a^(j)_n)=x and the conditional law of s~n+1(j) s^(j)_n+1 given ℋ~n(j) H^(j)_n is p(⋅∣x)p(· x), so [Zl(j)(x)∣ℋ~tl(x)(j)]=0E[Z^(j)_l(x) H^(j)_t_l(x)]=0. Since j<σR1_\j< _R\ is jG_j-, hence l−1(j)K^(j)_l-1-measurable and l−1(j)⊆ℋ~tl(x)(j)K^(j)_l-1 H^(j)_t_l(x), the tower property gives the claim. (i). Conditionally on jG_j, Ω∗(Qj) ^*(Q_j) is fixed, so Zl(j)(x)Z^(j)_l(x) lies in the jG_j-measurable interval of length exactly ‖Ω∗(Qj)‖,sp\| ^*(Q_j)\|_S,sp, translated by −(PΩ∗(Qj))(x)-(P ^*(Q_j))(x). On j<σR\j< _R\, Lemma F.1.(vii) and Remark H.4 give ‖Ω∗(Qj)‖,sp≤‖Qj‖sp+‖Ω∗(0)‖,sp≤CR+‖Ω∗(0)‖,sp=w¯(R)\| ^*(Q_j)\|_S,sp≤\|Q_j\|_sp+ \| ^*(0) \|_S,sp≤ C_R+ \| ^*(0) \|_S,sp= w(R); off the event Z^l(j)=0 Z^(j)_l=0. ∎ Batch bound. Lemma H.19 (Per-epoch noise). For every j and δ′∈(0,1)δ ∈(0,1), with probability at least 1−δ′1-δ , ‖W^j+1‖∞≤U¯j(δ′):=w¯(R)12Bjlog(2||||δ′),hence‖W^j+1‖sp≤2U¯j(δ′). \| W_j+1 \|_∞≤ U_j(δ ):= w(R) 12B_j ( 2|S||A|δ ), \| W_j+1 \|_sp≤ 2 U_j(δ ). (H.30) Proof. Fix x and apply Lemma E.2 to Mk:=∑l≤kZ^l(j)(x)M_k:= _l≤ k Z^(j)_l(x) with n=Bjn=B_j deterministic (Lemma H.14) and cl≡w¯(R)c_l≡ w(R), legitimate by Lemma H.18. With u=BjU¯j(δ′)u=B_j U_j(δ ) and a union bound over the pairs this is (H.30); the last inequality is ∥⋅∥sp≤2∥⋅∥∞\|·\|_sp≤ 2\|·\|_∞. ∎ Block-martingale bound. Lemma H.20 (Block martingale). Fix i and define U^bi(i):=0 U^(i)_b_i:=0, U^j+1(i):=U^j(i)+αjW^j+1 U^(i)_j+1:= U^(i)_j+ _j W_j+1. Then for every δ′∈(0,1)δ ∈(0,1), with probability at least 1−δ′1-δ , maxbi≤j≤bi+1‖U^j(i)‖∞≤V¯i(δ′):=w¯(R)12(∑j∈ℬiαj2Bj)log(2||||δ′). _b_i≤ j≤ b_i+1 \| U^(i)_j \|_∞≤ V_i(δ ):= w(R) 12 ( _j _i _j^2B_j ) ( 2|S||A|δ ). (H.31) The same holds with ℬiB_i replaced by any prefix, in particular by ℬresB_res. Proof. Fix x and concatenate chronologically the increments (αjBjZ^l(j)(x))j∈ℬi,l≤Bj ( _jB_j Z^(j)_l(x) )_j _i,\,l≤ B_j, adapted to the filtration of Lemma H.18.(i). They are martingale differences by (i), and by (i) the l-th increment of epoch j has predictable range of deterministic length αjBjw¯(R) _jB_j w(R). The total number of increments ∑j∈ℬiBj _j _iB_j is deterministic (Lemma H.14) and ∑j∈ℬi∑l=1Bj(αjBjw¯(R))2=w¯(R)2∑j∈ℬiαj2Bj. _j _i _l=1^B_j ( _jB_j w(R) )^2= w(R)^2 _j _i _j^2B_j. The partial sums at epoch boundaries are exactly U^j(i)(x) U^(i)_j(x), so Lemma E.2 with u=V¯i(δ′)u= V_i(δ ) and a union bound over the pairs give (H.31). ∎ The tracking lemma. Lemma H.21 (Tracking within a block). Let i be such that ebi≤Re_b_i≤ R, let Q~ Q be the virtual sequence started at QbiQ_b_i and Ej:=Qj−Q~jE_j:=Q_j- Q_j. Then for every j∈ℬi∪bi+1j _i∪\b_i+1\ with j≤σRj≤ _R: (i) on ⋂k∈ℬi∥W^k+1∥∞≤U¯k(δ′) _k _i\\| W_k+1\|_∞≤ U_k(δ )\: ‖Ej‖sp≤εibat(δ′):=2∑k∈ℬiαkU¯k(δ′)\ \|E_j\|_sp≤ ^bat_i(δ ):=2 _k _i _k U_k(δ ); (i) on maxk∥U^k(i)∥∞≤V¯i(δ′)\ _k\| U^(i)_k\|_∞≤ V_i(δ )\: ‖Ej‖sp≤εimg(δ′):=2(1+2∑k∈ℬiαk)V¯i(δ′)\ \|E_j\|_sp≤ ^mg_i(δ ):=2 (1+2 _k _i _k ) V_i(δ ). Both hold verbatim on any prefix, in particular on ℬresB_res. Proof. For k∈ℬik _i with k<σRk< _R, W^k+1=Wk+1 W_k+1=W_k+1 and ℓk<+∞ _k<+∞ (Proposition H.2), so (H.19) applies. Subtracting the virtual recursion and using the average-path identity of Lemma G.2.(i), i.e. Ω(Qk)−Ω(Q~k)=K¯kEkT_ (Q_k)-T_ ( Q_k)= K_kE_k with K¯k:=Kπ¯Ω,[Qk,Q~k] K_k:=K_ π_ ,[Q_k, Q_k], Ek+1=MkEk+αkW^k+1,Mk:=(1−αk)I+αkK¯k,E_k+1=M_kE_k+ _k W_k+1, M_k:=(1- _k)I+ _k K_k, (H.32) with MkM_k row-stochastic, hence span non-expansive (Lemma E.6). (i). ‖Ek+1‖sp≤‖Ek‖sp+2αkU¯k(δ′)\|E_k+1\|_sp≤\|E_k\|_sp+2 _k U_k(δ ) by (H.30), and an immediate induction from Ebi=0E_b_i=0 concludes. (i). The recursion (H.32) has the same form as (H.13), so the decomposition Ek=R^k+U^k(i)E_k= R_k+ U^(i)_k in the proof of Lemma H.13, with U^(i) U^(i) and V¯i(δ′) V_i(δ ) in place of U^() U^(K) and U¯(δ′) U_K(δ ), gives ‖Ej‖sp≤εimg(δ′)\|E_j\|_sp≤ ^mg_i(δ ). ∎ H.3.5 No escape before the horizon Let δ∈(0,1)δ∈(0,1), T≥0T≥ 0 and δ′:=δ3(Jmax(T)+1)δ := δ3(J_ (T)+1). We set εi(δ′):=minεibat(δ′),εimg(δ′),ε¯R(δ):=max0≤i≤Imax(T)εi(δ′), _i(δ ):= \ ^bat_i(δ ), ^mg_i(δ ) \, _R(δ):= _0≤ i≤ I_ (T) _i(δ ), (H.33) and write εres(δ′) _res(δ ) for the same quantity with ℬiB_i replaced by ℬresB_res. Both bounds of Lemma H.21 are non-decreasing when a prefix is extended, so εres(δ′)≤εI(T)(δ′)≤ε¯R(δ) _res(δ )≤ _I(T)(δ )≤ _R(δ). Lemma H.22 (Good event). The event ℰR(δ) _R(δ) :=⋂j<Jmax(T)‖W^j+1‖∞≤U¯j(δ′)∩⋂i≤Imax(T)maxbi≤j≤bi+1‖U^j(i)‖∞≤V¯i(δ′) := _j<J_ (T) \ \| W_j+1 \|_∞≤ U_j(δ ) \∩ _i≤ I_ (T) \ _b_i≤ j≤ b_i+1 \| U^(i)_j \|_∞≤ V_i(δ ) \ ∩⋂j<Jmax(T)(j≥σR∪ℓj≤Lj(δ′)) ∩ _j<J_ (T) (\j≥ _R\∪ \ _j≤ L_j(δ ) \ ) (H.34) satisfies ℙ(ℰR(δ))≥1−δP(E_R(δ))≥ 1-δ. Proof. The three families contain at most Jmax(T)J_ (T), Imax(T)+1≤Jmax(T)+1I_ (T)+1≤ J_ (T)+1 and Jmax(T)J_ (T) events, hence at most 3(Jmax(T)+1)3(J_ (T)+1) in total, each of failure probability at most δ′δ by Lemmas H.19 and H.20 and Lemma H.15, the latter applied conditionally on jG_j and integrated. ∎ Proposition H.3 (No escape). Let R≥e0R≥ e_0 satisfy the radius admissibility condition ε¯R(δ)≤β¯(R)β¯(R)+1(R−e0). _R(δ)≤ β(R) β(R)+1 (R-e_0 ). (H.35) Then, on ℰR(δ)E_R(δ), σR>J(T) _R>J(T); in particular sup0≤j≤J(T)ej≤R _0≤ j≤ J(T)e_j≤ R and ℓj≤Lj(δ′) _j≤ L_j(δ ) for every j<J(T)j<J(T), and for every 0≤i≤I(T)0≤ i≤ I(T), ebi≤(1−β¯(R))ie0+∑ℓ=0i−1(1−β¯(R))i−1−ℓεℓ(δ′).e_b_i≤(1- β(R))^ie_0+ _ =0^i-1(1- β(R))^i-1- _ (δ ). (H.36) Proof. Work on ℰR(δ)E_R(δ). The proof of Proposition H.1 applies verbatim on the epoch scale, with the exit time (H.21), Remark H.4 for σR≥1 _R≥ 1, Lemmas H.17 and H.21 in place of Lemmas H.10 and H.13, and εi(δ′) _i(δ ) and εres(δ′)≤ε¯R(δ) _res(δ )≤ _R(δ) in place of εℬi(δ′) _B_i(δ ) and εℬres(δ′) _B_res(δ ); it gives σR>J(T) _R>J(T) and (H.36). The length bound is then the third family of (H.34), since every j<J(T)j<J(T) satisfies j<σRj< _R and j<Jmax(T)j<J_ (T). ∎ H.3.6 Main result Theorem H.2 (Finite-time bound for Algorithm 1). Let δ∈(0,1)δ∈(0,1) and T≥0T≥ 0. Under Assumption H.1, let κ0>0 _0>0 be arbitrary and let R≥e0R≥ e_0 satisfy the admissibility condition (H.35). Then, with probability at least 1−δ1-δ, sup0≤j≤J(T)ej≤R _0≤ j≤ J(T)e_j≤ R and eJ(T)≤(1−β¯(R))I(T)e0+∑i=0I(T)−1(1−β¯(R))I(T)−1−iεi(δ′)+εres(δ′),e_J(T)≤(1- β(R))^I(T)e_0+ _i=0^I(T)-1(1- β(R))^I(T)-1-i _i(δ )+ _res(δ ), (H.37) where J(T)J(T) and I(T)I(T) are bounded below in Lemma H.16.(i). Explicitly, with ι1:=log(6||||(Jmax(T)+1)δ) _1:= ( 6|S||A|(J_ (T)+1)δ ), εi(δ′)=w¯(R)2ι1min∑j∈ℬiαjBj,(1+2∑j∈ℬiαj)∑j∈ℬiαj2Bj. _i(δ )= w(R) 2 _1\, \\ _j _i _j B_j,\ \ (1+2 _j _i _j ) _j _i _j^2B_j\ \. (H.38) Proof. Work on ℰR(δ)E_R(δ), of probability at least 1−δ1-δ (Lemma H.22). As in the proof of Theorem H.1, Proposition H.3, Lemma H.21 on ℬresB_res and Lemma H.17.(i) give the containment and eJ(T)≤ebI(T)+εres(δ′)e_J(T)≤ e_b_I(T)+ _res(δ ), and (H.36) at rank I(T)I(T) yields (H.37). For (H.38), 2U¯j(δ′)=w¯(R)Bj−1/22ι12 U_j(δ )= w(R)B_j^-1/2 2 _1 and 2V¯i(δ′)=w¯(R)2ι1∑j∈ℬiαj2/Bj2 V_i(δ )= w(R) 2 _1 _j _i _j^2/B_j. ∎ Remark H.6 (Recovering Theorem H.1). At Bj≡1B_j≡ 1, the block-martingale bound in (H.38) is exactly the quantity εℬi(δ′) _B_i(δ ) of (H.11), with a different value of δ′δ . Theorems H.2 and H.1 are then the same statement up to this value; the only difference is the cost model, |||||S||A| sampled transitions per iteration there against Lj(δ′)L_j(δ ) per epoch here. H.3.7 The case αj≡1 _j≡ 1 and proof of Theorem 2 Corollary H.4 (αj≡1 _j≡ 1: empirical Ω -regularized value iteration). Take αj≡1 _j≡ 1, Bj≡B_j≡ B, and define the blocks directly as ℬi:= im,(i+1)m−1 _i:= im,(i+1)m-1 , so that I(T)=⌊J(T)/m⌋I(T)= J(T)/m . Then Fj=ΩF_j=T_ , the block map is ΩmT^m_ , and Lemma H.17 holds with β¯(R) β(R) replaced by β~m(AR)=βmγΩ(AR)m−1 β_m(A_R)= _m _ (A_R)^m-1, with no sub-block budget κ0 _0 and no probability pmp_m. With U¯:=w¯(R)ι1/(2B) U:= w(R) _1/(2B), the batch bound gives εi(δ′)≤2mU¯ _i(δ )≤ 2m U and εres(δ′)≤2mU¯ _res(δ )≤ 2m U, condition (H.35) becomes 2mU¯≤β~m(AR)1+β~m(AR)(R−e0),2m U≤ β_m(A_R)1+ β_m(A_R) (R-e_0 ), (H.39) and with probability at least 1−δ1-δ, sup0≤j≤J(T)ej≤R,eJ(T)≤(1−β~m(AR))I(T)e0+(1+β~m(AR))2mU¯β~m(AR). _0≤ j≤ J(T)e_j≤ R, e_J(T)≤ (1- β_m(A_R) )^I(T)e_0+ (1+ β_m(A_R))2m U β_m(A_R). (H.40) Proof. With αj=1 _j=1, Q~j=Ωj−bi(Qbi) Q_j=T^j-b_i_ (Q_b_i). By Corollary F.1, Ωk(QΩ∗)=QΩ∗+kρΩ∗T^k_ (Q^*_ )=Q^*_ +kρ^*_ 1, of action-span ‖QΩ∗‖asp\|Q^*_ \|_asp. If ebi≤Re_b_i≤ R, span non-expansiveness gives ‖Q~j−QΩ∗‖sp≤ebi≤R\| Q_j-Q^*_ \|_sp≤ e_b_i≤ R, which is Lemma H.17.(i), hence ‖Q~j‖asp≤AR\| Q_j\|_asp≤ A_R, i.e. R[Ωk(Qbi),Ωk(QΩ∗)]≤ARR[T_ ^k(Q_b_i),T_ ^k(Q^*_ )]≤ A_R for k∈ 0,m−1 ∈ 0,m-1 (Definition G.1.(i)). Since γΩ _ is non-increasing, Lemma G.2.(iv) gives ‖Ωm(Qbi)−Ωm(QΩ∗)‖sp≤(1−βmγΩ(AR)m−1)ebi, \|T^m_ (Q_b_i)-T^m_ (Q^*_ ) \|_sp≤ (1- _m _ (A_R)^m-1 )e_b_i, which is Lemma H.17.(i) with β~m(AR) β_m(A_R) in place of β¯(R) β(R); neither Lemma H.6 nor the probabilistic argument of Lemma H.7 is needed. Proposition H.3 and Theorem H.2 then apply verbatim with β~m(AR) β_m(A_R) in place of β¯(R) β(R), the geometric sum being bounded by 1/β~m(AR)1/ β_m(A_R) and the residual block adding one copy of 2mU¯2m U. ∎ See 2 Proof of Theorem 2. We apply Corollary H.4 with R=2e0R=2e_0. Then AR=‖QΩ∗‖asp+2e0A_R=\|Q^*_ \|_asp+2e_0, so that β~m(AR)=β~ β_m(A_R)= β, w¯(R)=w¯ w(R)= w and qR=γ¯β~ηminq_R= γ β _ ; moreover ι(δ′)≤ι1≤ι (δ )≤ _1≤ since Jmax(T)≤TJ_ (T)≤ T. Admissibility. We have 2mU¯≤2mw¯ι/B2m U≤ 2\,m w /B and 1+β~≤21+ β≤ 2, so (H.39) holds as soon as 2mw¯ι/B≤β~e0/2 2\,m w /B≤ βe_0/2, i.e. B≥B(e0)B≥ B(e_0). Bound. With probability at least 1−δ1-δ, Corollary H.4 gives supj≤J(T)ej≤2e0 _j≤ J(T)e_j≤ 2e_0, and its residual term is at most 2⋅2mw¯ι/Bβ~ 2· 2\,m w /B β, which gives (3). On the same event, Proposition H.3 gives ℓj≤Lj(δ′)≤L _j≤ L_j(δ )≤ L for every j<J(T)j<J(T), since ι(δ′)≤ι (δ )≤ , so Lemma H.16.(i) gives J(T)≥⌊T/L⌋J(T)≥ T/L , and I(T)=⌊J(T)/m⌋I(T)= J(T)/m . Sample complexity. For ε∈(0,e0) ∈(0,e_0), the choice B=⌈B(ε/2)⌉≥B(e0)B= B( /2) ≥ B(e_0) makes the residual term at most ε/2 /2. For T≥mL⌈β~−1log(2e0/ε)⌉T≥ mL β^-1 (2e_0/ ) , which is possible since ι depends on T only logarithmically, we get I(T)≥β~−1log(2e0/ε)I(T)≥ β^-1 (2e_0/ ), and 1−u≤e−u1-u≤ e^-u makes the contraction term at most ε/2 /2. Since L=Θ~(mB/(γ¯β~ηmin))L= (mB/( γ β _ ) ) and B=Θ~(m2w¯2/(β~ε)2)B= (m^2 w^2/( β )^2 ), this gives O~(m4w¯2/(β~3⋅γ¯β~ηmin⋅ε2)) O (m^4 w^2/( β^3· γ β _ · ^2) ) sampled transitions. ∎ Corollary H.5 (Epoch Ω -regularized Q-learning for m=1m=1). Let m=1m=1. Then β~=β1 β= _1, so that in Theorem 2 the domination profile only enters through the coverage factor γ¯β1ηmin γ _1 _ of the epoch length L, and the sample complexity is O~(w¯2/(β13⋅γ¯β1ηmin⋅ε2)) O ( w^2/( _1^3· γ _1 _ · ^2) ). Proof. For m=1m=1, β~1(AR)=β1 β_1(A_R)= _1 for every R (Definition H.2), so the contraction constants of the proof of Theorem 2 do not depend on the radius, whereas the coverage constant qR=γΩ(AR)β~1(AR)ηminq_R= _ (A_R)\, β_1(A_R)\, _ of Lemma H.15 does. At R=2e0R=2e_0, it equals γ¯β1ηmin γ _1 _ , and the claim is Theorem 2 with m=1m=1. ∎ Remark H.7 (Acting with μ or with the greedy policies). The update of Algorithm 1 is off-policy: its target r(s,a)+Ω∗(Qj)(s′)r(s,a)+ ^*(Q_j)(s ) is an expectation under the greedy policy and does not sample the next action, so the noise analysis only uses that the successor of a visited pair (s,a)(s,a) is distributed as p(⋅|s,a)p(·|s,a) (Lemma H.18.(i)), whatever the law of the action. Running the algorithm with actions an∼μ(⋅|sn)a_n μ(·|s_n) is therefore covered by the same analysis, with γΩ _ replaced by 11 in Lemma H.15: qRq_R becomes βmηmin _m _ , L becomes m⌈(B+ι)2/(βmηmin)⌉+1m ( B+ )^2/( _m _ ) +1 in Theorem 2, and the sample complexity becomes O~(m4w¯2/(β~3⋅βmηmin⋅ε2)) O (m^4 w^2/( β^3· _m _ · ^2) ), the factor β~3 β^3 of the contraction being unchanged. Acting with the greedy policies thus costs at most a factor γ¯m γ^m in coverage. Our criterion, uniform accuracy on all pairs of a tabular MDP, favors a fixed well-mixing behavior, and cannot credit what acting with the learned policy brings: the reward collected during learning, data concentrated where a good policy goes and, beyond the tabular setting, the stability of on-policy data distributions under function approximation (Sutton and Barto, 2018, §11.3). What the analysis does show is that regularization makes this choice safe: with a hard maximum, the greedy policy is deterministic and needs an added exploration mechanism, whereas under (P1) it explores by itself, at a quantified rate. Appendix I Planning: Setting and Main Results This section sets up the objects of Subsection 4.4 and states the planning results, which are proved in Appendix J. From a reset distribution ξ∈Δ()ξ∈ (S), planning computes the regularized H-step solution bootstrapped at depth H by a boundary value QB∈ℝ′×Q_B ^S ×A, e.g. a value estimate or a heuristic, and uses its root level to approximate QΩ∗Q^*_ . It is learning localized in space, since only the states reachable from ξ are visited, and in time, since only the levels closest to the root matter. Depth-dependent objects. Let F(X):=s′∈∣∃(s,a)∈X×,p(s′|s,a)>0F(X):=\s ∃(s,a)∈ X×A,\ p(s |s,a)>0\ for X⊆X . The reachable layers are 0:=supp(ξ)S_0:=supp(ξ) and h+1:=F(h)S_h+1:=F(S_h); we write h:=h×X_h:=S_h×A, ′:=⋃h≥0hS := _h≥ 0S_h for the reachable set, which satisfies F(′)⊆′F(S ) , so that the MDP restricted to ′S is well defined and ΩT_ acts on ℝ′×R^S ×A, and S¯H:=maxh≤H|h| S_H:= _h≤ H|S_h|. A profile is a family (Qh)h=0H(Q_h)_h=0^H of depth-dependent Q-functions Qh∈ℝhQ_h ^X_h. Since the successors of hS_h lie in h+1S_h+1, the depth-dependent operator Ω,h:ℝh+1→ℝhT_ ,h:R^X_h+1 ^X_h, Ω,h(Q)(s,a):=r(s,a)+∑s′∈h+1p(s′|s,a)Ω∗(Q)(s′)T_ ,h(Q)(s,a):=r(s,a)+ _s _h+1p(s |s,a)\, ^*(Q)(s ), is well defined, and the planning solution is the unique profile defined backward by QΩ,H∗:=QB|HQ^*_ ,H:=Q_B|_X_H and QΩ,h∗:=Ω,h(QΩ,h+1∗)Q^*_ ,h:=T_ ,h(Q^*_ ,h+1). It solves this triangular system exactly, whereas QΩ∗Q^*_ only solves Ω(QΩ∗)=QΩ∗+ρΩ∗T_ (Q^*_ )=Q^*_ +ρ^*_ 1, so the planners need no centering projection. Mixing and target. Throughout the planning appendices, Assumptions 2, 3 and 4 and (P1) hold, Assumption 1 is not assumed, and μ, m, βm _m, ν denote the objects of Assumption 4, with respect to which Assumption 3 and (P1) are understood; we write η(s,a):=ν(s)μ(a|s)η(s,a):=ν(s)μ(a|s) and ηmin:=min∞×η _ := _S_∞×Aη. Remark I.1 (Reading Assumption 4). (i) The core ∞S_∞ is closed: by (i), ∞=h1+1=F(h1)=F(∞)S_∞=S_h_1+1=F(S_h_1)=F(S_∞). On the MDP restricted to ∞S_∞, (i) is Assumption 1 with a full-support ν, so Theorem G.1 gives (QΩ∗,ρΩ∗)(Q^*_ ,ρ^*_ ) on ∞×S_∞×A, with QΩ∗Q^*_ unique up to an additive constant under (P1). (i) Global case: if Assumption 1 holds, then Kμm(s′|s,a)>0K^m_μ(s |s,a)>0 for all (s,a,s′)(s,a,s ), so h=S_h=S for h≥mh≥ m and Assumption 4 holds with ∞=S_∞=S and h1=mh_1=m. The spatial localization is then vacuous, and only the temporal one remains. (i) Transient layers: nothing is assumed on 0,…,h1−1S_0,…,S_h_1-1, which enter the analysis only through S¯H S_H and the coverage of the shallow levels (Lemma J.9). (iv) When (i) holds: hS_h is the set of states reachable in exactly h steps from supp(ξ)supp(ξ) in the union graph G (Definition D.1), so (h)h(S_h)_h is eventually periodic, and (i) holds for some finite h1h_1 as soon as every strongly connected component of G that contains a cycle and is reachable from supp(ξ)supp(ξ) is aperiodic, e.g. contains a state s with p(s|s,a)>0p(s|s,a)>0 for some a: closed walks in such a component have all large enough lengths, so a state reached at arbitrarily large depths is reached at every large enough depth. By (i), QΩ∗Q^*_ is defined on ∞×S_∞×A; it extends uniquely to ′×S ×A through the fixed-point equation (Lemma J.2), and the norms of QΩ∗Q^*_ are taken over ′×S ×A. Errors and constants. We measure the quality of the boundary by ΔB:=‖QB−QΩ∗‖sp,H _B:=\|Q_B-Q^*_ \|_sp,X_H and that of the initial profile by D0:=maxh<H‖Q0,h−QΩ,h∗‖spD_0:= _h<H\|Q_0,h-Q^*_ ,h\|_sp, and use β~ β, w¯ w and HeffH_eff of Section 4 with e0:=D0+ΔBe_0:=D_0+ _B. We assume H≥h1+mH≥ h_1+m. With ηhres(s,a):=(ξPμh)(s)μ(a|s)η^res_h(s,a):=(ξ P^h_μ)(s)\,μ(a|s), the probability that a rollout started from ξ and following μ visits (s,a)(s,a) at depth h, which is positive exactly on hX_h (Lemma J.1), the coverage constants under μ and under a behavior dominating γΩ(A)μ _ (A)μ are qμ:=minminh<h1+mmin(s,a)∈hηhres(s,a),βmηmin,qπ(A):=γΩ(A)h1+mqμ(A≥0),q_μ:= \ _h<h_1+m\ _(s,a) _hη^res_h(s,a),\ _m _ \, q_π(A):= _ (A)^h_1+m\,q_μ (A≥ 0), (I.1) both positive, qπ(A)q_π(A) by (P1). Algorithms and results. With a model, the backward sweep Qh←Ω,h(Qh+1)Q_h _ ,h(Q_h+1) (Algorithm 4) returns the planning solution exactly, and Proposition J.1 bounds its horizon bias and its cost. Without a model, Algorithm 5 estimates each depth operator from rollouts started from ξ. Whereas the learning algorithms iterate the stationary update Qj+1=Ω(Qj)Q_j+1=T_ (Q_j), it applies, up to sampling noise, the depth-shifted update Qj+1,h=Ω,h(Qj,h+1)Q_j+1,h=T_ ,h(Q_j,h+1): level h at epoch j+1j+1 is computed from level h+1h+1 at epoch j, which is frozen during epoch j. Without noise, this recursion reaches the planning solution after H−hH-h epochs at level h (Lemma J.10), so the initialization is forgotten exactly rather than geometrically. As in Algorithm 1, a cap makes the number of samples per pair and per level deterministic. Theorem I.1 (Epoch Ω -regularized Q-planning). Let q:=qπ(‖QΩ∗‖asp+2e0)q:=q_π(\|Q^*_ \|_asp+2e_0), or q:=qμq:=q_μ if the actions are drawn from μ instead of the greedy policies (Remark J.1). Let J≥HJ≥ H, ιH:=log(4JHS¯H||/δ) _H:= (4JH S_H|A|/δ), M:=h1+HeffM:=h_1+H_eff and B(u):=2M2w¯2ιH/u2B(u):=2M^2 w^2 _H/u^2. Run Algorithm 5 for J epochs with a batch size B≥B(e0)B≥ B(e_0). Then, with probability at least 1−δ1-δ, each epoch lasts at most H⌈(B+ιH)2/q⌉H ( B+ _H)^2/q transitions and ‖QJ,0−QΩ∗‖sp,0≤(1−β~)⌊(H−h1)/m⌋ΔB+2Mw¯ιH2B. \|Q_J,0-Q^*_ \|_sp,X_0≤(1- β) (H-h_1)/m _B+2M w _H2B. (I.2) In particular, for ε∈(0,e0) ∈(0,e_0), the choices H≥h1+m+Hefflog+(2ΔB/ε)H≥ h_1+m+H_eff _+(2 _B/ ), J=HJ=H and B=⌈B(ε/2)⌉B= B( /2) give ‖QJ,0−QΩ∗‖sp,0≤ε\|Q_J,0-Q^*_ \|_sp,X_0≤ , i.e. O~((h1+Heff)4w¯2/(qε2)) O ((h_1+H_eff)^4\, w^2/(q\, ^2) ) sampled transitions. The bound (I.2) has the structure of (3), with the horizon bias in place of the initialization term. Finally, Algorithm 6 merges the H tables into a single table learned from resets, which removes the extra factor 1/β~1/ β (Theorem J.2). Appendix J Analysis of the Planning Algorithms This section proves the results of Appendix I and analyses Algorithm 6, which merges the depth-indexed tables of the epoch planner into a single one. The analysis rests on three facts. First, the planning solution solves its equations exactly (Appendix I), so the planners forget their initialization after H epochs. Second, a single lemma (Lemma J.8) bounds the error at depth h by the Bellman residuals of the deeper levels, damped at the rate of Lemma G.2.(iv) every m levels: it gives the planning depth and shows that noise does not accumulate across the horizon. Third, resets provide the exploration of the shallow levels, and the mixing of Assumption 4 that of the deep ones (Lemma J.9). The model-based sweep is treated in Subsection J.5, epoch planning in Subsection J.6 and epoch learning with resets in Subsection J.7. J.1 Reachability and the extended target Lemma J.1 (Reachability under μ). For every h≥0h≥ 0 and s∈s , (ξPμh)(s)>0(ξ P^h_μ)(s)>0 if and only if s∈hs _h. Proof. Induction on h, the case h=0h=0 being the definition of 0S_0. Since μ has full support, Pμ(s′|s)>0P_μ(s |s)>0 if and only if p(s′|s,a)>0p(s |s,a)>0 for some a, so (ξPμh+1)(s′)=∑s(ξPμh)(s)Pμ(s′|s)>0(ξ P^h+1_μ)(s )= _s(ξ P^h_μ)(s)P_μ(s |s)>0 if and only if s′∈F(h)=h+1s ∈ F(S_h)=S_h+1. ∎ Lemma J.2 (Extension of the target to the transient states). Let :=′∖∞V:=S _∞. The accessibility relation restricted to V has no directed cycle, and QΩ∗Q^*_ has a unique extension to ′×S ×A such that Ω(QΩ∗)=QΩ∗+ρΩ∗on ′×.T_ (Q^*_ )=Q^*_ +ρ^*_ 1 S ×A. (J.1) Shifting QΩ∗Q^*_ by a constant on ∞S_∞ shifts its extension by the same constant. Proof. If s∈∩js _j lay on a cycle of length ℓ≥1 ≥ 1 within V, then s∈j+ℓis _j+ i for every i≥0i≥ 0, hence s∈h=∞s _h=S_∞ for some h≥h1h≥ h_1, a contradiction. The subgraph induced on V is therefore acyclic, and its vertices admit a topological order in which every successor of s∈s lies either in ∞S_∞ or later in the order. Defining, along the reverse of that order, QΩ∗(s,a):=r(s,a)−ρΩ∗+∑s′p(s′|s,a)Ω∗(QΩ∗)(s′),s∈,Q^*_ (s,a):=r(s,a)-ρ^*_ + _s p(s |s,a)\, ^*(Q^*_ )(s ), s , each right-hand side only involves values already defined, which gives existence and uniqueness. The last claim follows from the translation equivariance of Ω∗ ^* (Lemma F.1.(i)). ∎ We keep the notation QΩ∗Q^*_ for the extension. By (J.1) and Corollary F.1, Ωk(QΩ∗)=QΩ∗+kρΩ∗T^k_ (Q^*_ )=Q^*_ +kρ^*_ 1 on ′×S ×A for every k≥0k≥ 0; this is the only property of the extension used below, and it allows us to compare a profile to QΩ∗Q^*_ at the depths h<h1h<h_1. J.2 Depth-dependent objects Lemma J.3 (Depth-indexed average-path representation). Let Q,Q′∈ℝh+1Q,Q ^X_h+1 and let π¯:=π¯Ω,[Q,Q′] π:= π_ ,[Q,Q ] be the average-path policy of Definition G.1.(i). Then Ω,h(Q)−Ω,h(Q′)=Kπ¯(h)(Q−Q′),Kπ¯(h)(s′,a′|s,a):=p(s′|s,a)π¯(a′|s′),T_ ,h(Q)-T_ ,h(Q )=K^(h)_ π (Q-Q ), K^(h)_ π(s ,a |s,a):=p(s |s,a)\, π(a |s ), where Kπ¯(h)K^(h)_ π, the restriction of the state-action kernel Kπ¯K_ π to h×h+1X_h×X_h+1, is row-stochastic from hX_h to h+1X_h+1, and π¯≥γΩ(R[Q,Q′])μ π≥ _ (R[Q,Q ])\,μ on h+1X_h+1. Proof. This is the computation of Lemma G.2.(i) restricted to (s,a)∈h(s,a) _h, all successors lying in h+1S_h+1; domination is Lemma G.2.(i), the greedy policy at s′s depending on Q only through Qs′Q_s . ∎ For policies π(h),…,π(h+k−1)π^(h),…,π^(h+k-1), we write Kh:h+k:=K(h)π(h)K(h+1)π(h+1)⋯K(h+k−1)π(h+k−1)K_h:h+k:=K^(h)_π^(h)K^(h+1)_π^(h+1)·s K^(h+k-1)_π^(h+k-1), with Kh:h:=IK_h:h:=I, for the product of depth kernels from hX_h to h+kX_h+k in the sense of Convention E.1; it is row-stochastic. Lemma J.4 (The planning solution is a restriction). For every Q∈ℝ′×Q ^S ×A and h∈ 0,H ∈ 0,H , (ΩH−hQ)|h=Ω,h∘⋯∘Ω,H−1(Q|H) (T^H-h_ Q ) |_X_h=T_ ,h ·s _ ,H-1 (Q|_X_H ). In particular, QΩ,h∗=(ΩH−hQB)|hQ^*_ ,h= (T^H-h_ Q_B ) |_X_h and (ΩH−hQΩ∗)|h=QΩ∗|h+(H−h)ρΩ∗ (T^H-h_ Q^*_ ) |_X_h=Q^*_ |_X_h+(H-h)ρ^*_ 1. Proof. Backward induction on h: at (s,a)∈h(s,a) _h, ΩT_ only involves the values of Q at the states s′∈h+1s _h+1, which coincide with the depth-(h+1)(h+1) restriction by the induction hypothesis. The last identity is (J.1) iterated. ∎ Constants. For a radius R≥0R≥ 0, we use the localized constants AR=‖QΩ∗‖asp+RA_R=\|Q^*_ \|_asp+R, β~m(AR)=βmγΩ(AR)m−1 β_m(A_R)= _m _ (A_R)^m-1 and w¯(R)=‖QΩ∗‖sp+R+‖Ω∗(0)‖,sp w(R)=\|Q^*_ \|_sp+R+\| ^*(0)\|_S,sp of Definition H.2; here R bounds the span distance to QΩ∗Q^*_ at every depth. With e0=D0+ΔBe_0=D_0+ _B as in Appendix I, the constants of the main text are β~=β~m(A2e0) β= β_m(A_2e_0) and w¯=w¯(2e0) w= w(2e_0). We also write Heff(A):=mβ~m(A),M(h,A):=(h1−h)++Heff(A),M(A):=M(0,A)=h1+Heff(A),H_eff(A):= m β_m(A), M(h,A):=(h_1-h)_++H_eff(A), M(A):=M(0,A)=h_1+H_eff(A), (J.2) so that Heff=Heff(A2e0)H_eff=H_eff(A_2e_0). By (P1), β~m(A)>0 β_m(A)>0 for every finite A. Lemma J.5 (Distance between the planning solution and the target). For every h∈ 0,H ∈ 0,H , ‖QΩ,h∗−QΩ∗‖sp,h≤ΔB\|Q^*_ ,h-Q^*_ \|_sp,X_h≤ _B. Consequently ‖QΩ,h∗‖asp≤AΔB\|Q^*_ ,h\|_asp≤ A_ _B and ‖QΩ,h∗‖sp≤‖QΩ∗‖sp+ΔB\|Q^*_ ,h\|_sp≤\|Q^*_ \|_sp+ _B, uniformly in h and H. Proof. By Lemma J.4, QΩ,l∗=Ω,l(QΩ,l+1∗)Q^*_ ,l=T_ ,l(Q^*_ ,l+1) and Vl:=QΩ∗|l+(H−l)ρΩ∗V_l:=Q^*_ |_X_l+(H-l)ρ^*_ 1 satisfies Vl=Ω,l(Vl+1)V_l=T_ ,l(V_l+1) for every l<Hl<H, with VH=QΩ∗|HV_H=Q^*_ |_X_H. Lemma J.3 gives QΩ,l∗−Vl=Kπ(l)(l)(QΩ,l+1∗−Vl+1)Q^*_ ,l-V_l=K^(l)_π^(l)(Q^*_ ,l+1-V_l+1) with π(l):=π¯Ω,[QΩ,l+1∗,QΩ∗]π^(l):= π_ ,[Q^*_ ,l+1,Q^*_ ], the greedy policies being invariant under constants. Unrolling from l=hl=h to l=Hl=H, QΩ,h∗−QΩ∗|h−(H−h)ρΩ∗=Kh:H(QB−QΩ∗)|H.Q^*_ ,h-Q^*_ |_X_h-(H-h)ρ^*_ 1=K_h:H (Q_B-Q^*_ ) |_X_H. Kh:HK_h:H is row-stochastic, hence non-expansive in span (Lemma E.6), and the span annihilates constants, whence the first claim. The other two follow from ∥⋅∥asp≤∥⋅∥sp\|·\|_asp≤\|·\|_sp, the invariance of ∥⋅∥asp\|·\|_asp under constants and the triangle inequality. ∎ J.3 The master lemma For 0≤h≤h+k≤H0≤ h≤ h+k≤ H, the contracting block count n(h,k):=⌊(h+k−maxh,h1)+m⌋n(h,k):= (h+k- \h,h_1\ )_+m (J.3) is the number of disjoint blocks of m consecutive depths inside maxh,h1,h+k−1 \h,h_1\,h+k-1 ; when h≥h1h≥ h_1, n(h,k)=⌊k/m⌋n(h,k)= k/m . Lemma J.6 (Contraction of a product of depth kernels). Let A≥0A≥ 0, 0≤h≤h+k≤H0≤ h≤ h+k≤ H, and let π(h),…,π(h+k−1)π^(h),…,π^(h+k-1) be policies with π(l)≥γΩ(A)μπ^(l)≥ _ (A)\,μ for every l. Then ∥Kh:h+k∥sp,op≤(1−β~m(A))n(h,k)\|K_h:h+k\|_sp,op≤(1- β_m(A))^n(h,k). Proof. Group the factors of depths l≥maxh,h1l≥ \h,h_1\ into n(h,k)n(h,k) consecutive blocks of exactly m factors, leaving ungrouped the factors of depths l<h1l<h_1 and the fewer than m remaining ones. A block starting at depth l≥h1l≥ h_1 is the product of m depth kernels from l=∞×X_l=S_∞×A to l+m=∞×X_l+m=S_∞×A, whose policies dominate γΩ(A)μ _ (A)μ; as ∞S_∞ is closed, these kernels coincide with the unrestricted ones on ∞×S_∞×A. The proof of Lemma E.4 only uses the minorization Kμm(⋅|s,a)≥βmνK^m_μ(·|s,a)≥ _mν at the starting pairs, which Assumption 4.(i) provides on ∞×S_∞×A, so it bounds the Dobrushin coefficient of each block by 1−βmγΩ(A)m−1=1−β~m(A)1- _m _ (A)^m-1=1- β_m(A), and Lemma E.6 bounds its operator span seminorm by the same quantity. The ungrouped factors are row-stochastic, hence of operator span seminorm at most 11, and we conclude by submultiplicativity (E.5). ∎ Lemma J.7 (Geometric sum). For every h≥0h≥ 0 and A≥0A≥ 0, ∑k≥0(1−β~m(A))n(h,k)≤M(h,A) _k≥ 0(1- β_m(A))^n(h,k)≤ M(h,A). Proof. For k<(h1−h)+k<(h_1-h)_+, n(h,k)=0n(h,k)=0, and these terms contribute (h1−h)+(h_1-h)_+. For k≥(h1−h)+k≥(h_1-h)_+, n(h,k)=⌊(k−(h1−h)+)/m⌋n(h,k)= (k-(h_1-h)_+)/m takes each value q≥0q≥ 0 exactly m times, and these terms contribute m∑q≥0(1−β~m(A))q=Heff(A)m _q≥ 0(1- β_m(A))^q=H_eff(A). ∎ The next lemma drives every planning bound. It compares a partial profile to the planning solution through its Bellman residuals: the model-based sweep has no residual, and the epoch planner is a profile whose residuals are the noise of successive epochs (Subsection J.6). Lemma J.8 (Master lemma). Let 0≤h0≤he≤H0≤ h_0≤ h_e≤ H, let Ql∈ℝlQ_l ^X_l for l∈ h0,he ∈ h_0,h_e , with residuals rl:=Ql−Ω,l(Ql+1),l∈ 0,he−1 ,r_l:=Q_l-T_ ,l(Q_l+1), l∈ h_0,h_e-1 , (J.4) and assume ‖Ql‖asp≤A\|Q_l\|_asp≤ A for l∈ h0+1,he ∈ h_0+1,h_e , with A≥AΔBA≥ A_ _B. Then, writing β~:=β~m(A) β:= β_m(A), ‖Qh0−QΩ,h0∗‖sp \|Q_h_0-Q^*_ ,h_0 \|_sp ≤(1−β~)n(h0,he−h0)‖Qhe−QΩ,he∗‖sp+∑l=h0he−1(1−β~)n(h0,l−h0)‖rl‖sp ≤(1- β)^n(h_0,h_e-h_0) \|Q_h_e-Q^*_ ,h_e \|_sp+ _l=h_0^h_e-1(1- β)^n(h_0,l-h_0)\|r_l\|_sp (J.5) ≤(1−β~)n(h0,he−h0)‖Qhe−QΩ,he∗‖sp+2M(h0,A)maxh0≤l<he‖rl‖∞. ≤(1- β)^n(h_0,h_e-h_0) \|Q_h_e-Q^*_ ,h_e \|_sp+2M(h_0,A) _h_0≤ l<h_e\|r_l\|_∞. Proof. Let el:=Ql−QΩ,l∗e_l:=Q_l-Q^*_ ,l. By (J.4), the definition of the planning solution and Lemma J.3, el=rl+Ω,l(Ql+1)−Ω,l(QΩ,l+1∗)=rl+Kπ(l)(l)el+1,π(l):=π¯Ω,[Ql+1,QΩ,l+1∗].e_l=r_l+T_ ,l(Q_l+1)-T_ ,l(Q^*_ ,l+1)=r_l+K^(l)_π^(l)e_l+1, π^(l):= π_ ,[Q_l+1,Q^*_ ,l+1]. Unrolling from h0h_0 to heh_e gives eh0=Kh0:heehe+∑l=h0he−1Kh0:lrle_h_0=K_h_0:h_ee_h_e+ _l=h_0^h_e-1K_h_0:l\,r_l. By Lemma J.5, R[Ql+1,QΩ,l+1∗]≤maxA,AΔB=AR[Q_l+1,Q^*_ ,l+1]≤ \A,A_ _B\=A, so every π(l)π^(l) dominates γΩ(A)μ _ (A)μ (Lemma J.3) and Lemma J.6 applies to each product. The triangle inequality gives the first bound, and ∥⋅∥sp≤2∥⋅∥∞\|·\|_sp≤ 2\|·\|_∞ together with Lemma J.7 gives the second. ∎ Corollary J.1 (Horizon bias and planning depth). For every h∈ 0,H ∈ 0,H , ‖QΩ,h∗−QΩ∗‖sp,h≤(1−β~m(AΔB))n(h,H−h)ΔB, \|Q^*_ ,h-Q^*_ \|_sp,X_h≤ (1- β_m(A_ _B) )^n(h,H-h) _B, (J.6) and the right-hand side is at most ε as soon as H≥maxh,h1+m+Heff(AΔB)log+(ΔBε).H≥ \h,h_1\+m+H_eff(A_ _B) _+ ( _B ). (J.7) Proof. By Lemma J.4, the profile Vl:=QΩ∗|l+(H−l)ρΩ∗V_l:=Q^*_ |_X_l+(H-l)ρ^*_ 1 satisfies Vl=Ω,l(Vl+1)V_l=T_ ,l(V_l+1) for l<Hl<H, and ‖Vl‖asp=‖QΩ∗‖asp≤AΔB\|V_l\|_asp=\|Q^*_ \|_asp≤ A_ _B. Lemma J.8 with h0=h_0=h, he=Hh_e=H, zero residuals and A=AΔBA=A_ _B gives (J.6), since ‖VH−QΩ,H∗‖sp=ΔB\|V_H-Q^*_ ,H\|_sp= _B and the span annihilates constants. Finally, (1−β~)n≤e−β~n(1- β)^n≤ e^- βn and n(h,H−h)≥(H−maxh,h1)/m−1n(h,H-h)≥(H- \h,h_1\)/m-1 give (J.7). ∎ Lemma J.8 has two consequences. First, noise does not accumulate across the horizon. Without contraction, the errors of the H levels of a backward induction add up. Here, the residual at depth l is damped by (1−β~)n(h0,l−h0)(1- β)^n(h_0,l-h_0) before it reaches depth h0h_0, so only the M(h0,A)M(h_0,A) levels closest to h0h_0 matter. Second, Heff(A)H_eff(A) plays the role of the discounted horizon 1/(1−γ)1/(1-γ) of discounted planning, but it is set by the regularizer through γΩ _ . The boundary value enters the required depth (J.7) only through log(ΔB/ε) ( _B/ ): a better bootstrap allows a shallower lookahead. J.4 Coverage Two mechanisms explore in planning: resets cover the shallow levels, and the mixing of Assumption 4.(i) covers the deep ones. The following lemma gives the coverage constants (I.1). Lemma J.9 (Level coverage). Let G be a σ-algebra and let (sh,ah)h<H(s_h,a_h)_h<H be a trajectory with s0∼ξs_0 ξ, ah∼πh(⋅|sh)a_h _h(·|s_h) and sh+1∼p(⋅|sh,ah)s_h+1 p(·|s_h,a_h), where the policies πh _h are G-measurable and the trajectory is otherwise independent of G. Let E∈E be an event on which πh≥γhμ _h≥ _hμ for every h, with deterministic γh∈[0,1] _h∈[0,1], and write ℱn:=σ(,s0,a0,…,sn,an)F_n:=σ(G,s_0,a_0,…,s_n,a_n). Then, almost surely on E, for every h<Hh<H and (s,a)∈h(s,a) _h: (i) if h≥h1+mh≥ h_1+m, ℙ((sh,ah)=(s,a)∣ℱh−m)≥(∏l=h−m+1hγl)βmη(s,a)P ((s_h,a_h)=(s,a) _h-m )≥ ( _l=h-m+1^h _l ) _m\,η(s,a); (i) if h<h1+mh<h_1+m, ℙ((sh,ah)=(s,a)∣)≥(∏l=0hγl)ηhres(s,a)P ((s_h,a_h)=(s,a) )≥ ( _l=0^h _l )η^res_h(s,a). In particular, almost surely on E, ℙ((sh,ah)=(s,a)∣)≥qμP((s_h,a_h)=(s,a) )≥ q_μ if γh≡1 _h≡ 1, and ≥qπ(A)≥ q_π(A) if γh≡γΩ(A) _h≡ _ (A). Proof. (i). Since h−m≥h1h-m≥ h_1, sh−m∈h−m=∞s_h-m _h-m=S_∞, so Kμm(⋅|sh−m,ah−m)≥βmνK^m_μ(·|s_h-m,a_h-m)≥ _mν by Assumption 4.(i). The proof of Lemma G.2.(v) only uses this minorization at the conditioning pair and the domination of the m following policies, and applies verbatim. (i). Conditionally on G, (sh)(s_h) is a Markov chain with kernels PπlP_ _l, and the transport inequality Pπl≥γlPμP_ _l≥ _lP_μ of the proof of Lemma G.2.(i) holds entrywise on E. Hence ℙ(sh=s∣)≥(∏l<hγl)(ξPμh)(s)P(s_h=s )≥ ( _l<h _l )(ξ P^h_μ)(s), and πh(a|s)≥γhμ(a|s) _h(a|s)≥ _hμ(a|s) concludes. For the last claim, integrate (i) and use γΩ(A)h+1≥γΩ(A)h1+m _ (A)^h+1≥ _ (A)^h_1+m in (i) and γΩ(A)m≥γΩ(A)h1+m _ (A)^m≥ _ (A)^h_1+m in (i). ∎ Remark J.1 (Which behavior to use). The sample r(s,a)+Ω∗(Q)(s′)r(s,a)+ ^*(Q)(s ) is unbiased for Ω,h(Q)(s,a)T_ ,h(Q)(s,a) whatever the law of the action a, so the behavior policy only affects which pairs are visited. The factor γΩm−1 _ ^m-1 of the contraction rate comes from the average-path policy of Lemma J.3, internal to the operator, whereas the factor γΩ(A)h1+m _ (A)^h_1+m of qπ(A)q_π(A) is the price of exploring with the greedy policies, and disappears under the behavior μ, exactly as on a single trajectory (Remark H.7). The shallow levels may however be covered slowly, since (ξPμh)(s)(ξ P^h_μ)(s) is small for states reachable only through a narrow path; if ξ and p are bounded below by ξmin _ and pminp_ on their supports, then ηhres≥ξmin(μminpmin)hμminη^res_h≥ _ ( _ p_ )^h _ on hX_h. J.5 Model-based planning Proposition J.1 (Horizon bias and model-based planning). For every h≤Hh≤ H, ‖QΩ,h∗−QΩ∗‖sp,h≤(1−β~)⌊(H−maxh,h1)/m⌋ΔB. \|Q^*_ ,h-Q^*_ \|_sp,X_h≤(1- β) (H- \h,h_1\)/m \, _B. (J.8) In particular, Algorithm 4 returns (QΩ,h∗)h=0H(Q^*_ ,h)_h=0^H exactly after ∑h<H|h‖h+1|≤HS¯H2|| _h<H|S_h||A||S_h+1|≤ H S_H^2|A| model queries, one query being an evaluation of p(s′|s,a)p(s |s,a), and QΩ,0∗Q^*_ ,0 is ε -accurate for QΩ∗Q^*_ on 0X_0 as soon as H≥h1+m+Hefflog+(ΔB/ε)H≥ h_1+m+H_eff _+( _B/ ). Proof. With a model, D0=0D_0=0, so e0=ΔBe_0= _B and β~=β~m(A2ΔB)≤β~m(AΔB) β= β_m(A_2 _B)≤ β_m(A_ _B), γΩ _ being non-increasing. Since n(h,H−h)=⌊(H−maxh,h1)/m⌋n(h,H-h)= (H- \h,h_1\)/m , (J.8) follows from Corollary J.1. The sweep computes Qh=Ω,h(Qh+1)Q_h=T_ ,h(Q_h+1) from QH=QB|HQ_H=Q_B|_X_H, which is the definition of the planning solution, and each value Ω,h(Qh+1)(s,a)T_ ,h(Q_h+1)(s,a) uses the |h+1||S_h+1| probabilities p(s′|s,a)p(s |s,a), s′∈h+1s _h+1, whence the count. Finally, Heff≥Heff(AΔB)H_eff≥ H_eff(A_ _B), so the condition on H implies (J.7) at h=0h=0. ∎ Compared with Corollary 1, model-based planning is Q-iteration restricted to the reachable set, with the boundary mismatch ΔB _B in place of the initial error: it saves a factor (S¯H/||)2( S_H/|S|)^2 in model queries, which is non-trivial only when ∞⊊S_∞ (Remark I.1.(i)). The same exactness holds for the depth-shifted recursion Qj+1,h=Ω,h(Qj,h+1)Q_j+1,h=T_ ,h(Q_j,h+1), which is the noiseless version of epoch planning (Algorithm 5): each level is computed from the previous iterate of the level below, and the initialization is forgotten after exactly H iterations. Lemma J.10 (Exact forgetting of the initialization). Let (Qj,h)j≥0,h≤H(Q_j,h)_j≥ 0,\,h≤ H satisfy Qj+1,h=Ω,h(Qj,h+1)Q_j+1,h=T_ ,h(Q_j,h+1) for h<Hh<H and Qj,H=QB|HQ_j,H=Q_B|_X_H for every j. Then Qj,h=QΩ,h∗Q_j,h=Q^*_ ,h for every h≤Hh≤ H and j≥H−hj≥ H-h. Proof. Backward induction on h. The case h=Hh=H holds by assumption. If Qj,h+1=QΩ,h+1∗Q_j,h+1=Q^*_ ,h+1 for every j≥H−h−1j≥ H-h-1, then Qj+1,h=Ω,h(QΩ,h+1∗)=QΩ,h∗Q_j+1,h=T_ ,h(Q^*_ ,h+1)=Q^*_ ,h for every such j, i.e. Qj,h=QΩ,h∗Q_j,h=Q^*_ ,h for every j≥H−hj≥ H-h. ∎ J.6 Epoch planning Algorithm 5 is the depth-indexed analogue of Corollary H.4: its step size is αj≡1 _j≡ 1, which makes the recursion over epochs nilpotent (Lemma J.10), and its batch sizes BjB_j are deterministic. Let jG_j be the σ-algebra generated by Q0Q_0 and the rollouts of epochs 0,…,j−10,…,j-1; the analysis covers Algorithm 5 as written, with actions drawn from πΩ,Qj,h _ ,Q_j,h at level h, and its variant with actions drawn from μ (Remark J.1); in both cases the behavior of epoch j is jG_j-measurable. As in Lemma H.14, on the event that epoch j terminates, for h<Hh<H, Qj+1,h Q_j+1,h =Ω,h(Qj,h+1)+wj,h, =T_ ,h(Q_j,h+1)+w_j,h, (J.9) wj,h(s,a) w_j,h(s,a) :=1Bj∑l=1Bj(Ω∗(Qj,h+1)(s′l)−s′∼p(⋅|s,a)[Ω∗(Qj,h+1)(s′)]), := 1B_j _l=1^B_j ( ^*(Q_j,h+1)(s _l)-E_s p(·|s,a) [ ^*(Q_j,h+1)(s ) ] ), where sl′s _l is the successor of the l-th level-h visit of (s,a)(s,a) during epoch j, and Qj,H=QB|HQ_j,H=Q_B|_X_H. We set Qj+1,⋅:=Qj,⋅Q_j+1,·:=Q_j,· if epoch j does not terminate, which happens with probability zero (Proposition J.2). Remark J.2 (The outer loop is sample splitting). Level h at epoch j+1j+1 is computed from level h+1h+1 at epoch j, which is jG_j-measurable, hence independent of the data of epoch j. A single backward sweep over stored transitions would instead evaluate Ω∗(Q^h+1) ^*( Q_h+1) at the level-(h+1)(h+1) states of the very rollouts on which Q^h+1 Q_h+1 depends, and lose the martingale structure of the noise. Restoring independence would require H disjoint groups of rollouts, one per level, which costs the same factor H as the H epochs of Lemma J.10. Proposition J.2 (Well-posedness). Every epoch terminates almost surely, and every Qj,hQ_j,h is a finite, jG_j-measurable random vector. Proof. Finiteness and measurability follow by induction on j as in Lemma H.14.(i), each coordinate of wj,hw_j,h being bounded by ‖Qj,h+1‖sp+‖Ω∗(0)‖,sp\|Q_j,h+1\|_sp+\| ^*(0)\|_S,sp (Lemma F.1.(vii)). For termination, fix j and a deterministic u≥0u≥ 0. Conditionally on jG_j, the rollouts of epoch j are i.i.d. and, on maxh<H∥Qj,h∥asp≤u∈j\ _h<H\|Q_j,h\|_asp≤ u\ _j, each of them visits a given (h,x)(h,x), x∈hx _h, with probability at least minqμ,qπ(u)>0 \q_μ,q_π(u)\>0 by Lemma J.9. Each pair is thus visited B times after finitely many rollouts, almost surely on that event, and letting u↑∞u ∞ concludes. ∎ For R≥e0R≥ e_0, let σR:=infj≥0|maxh<H‖Qj,h−QΩ∗‖sp,h>R, _R:= \j≥ 0\ |\ _h<H\|Q_j,h-Q^*_ \|_sp,X_h>R \, a (j)(G_j)-stopping time with σR≥1 _R≥ 1, since ‖Q0,h−QΩ∗‖sp,h≤D0+ΔB=e0\|Q_0,h-Q^*_ \|_sp,X_h≤ D_0+ _B=e_0 by Lemma J.5, and write w^j,h:=j<σRwj,h w_j,h:=1_\j< _R\w_j,h. On j<σR\j< _R\, ‖Qj,h‖asp≤AR\|Q_j,h\|_asp≤ A_R and ‖Ω∗(Qj,h)‖,sp≤‖Qj,h‖sp+‖Ω∗(0)‖,sp≤w¯(R)\| ^*(Q_j,h)\|_S,sp≤\|Q_j,h\|_sp+\| ^*(0)\|_S,sp≤ w(R) for every h≤Hh≤ H, the level H being covered by ΔB≤e0≤R _B≤ e_0≤ R. We write q:=qπ(AR)q:=q_π(A_R), or q:=qμq:=q_μ for the variant with actions drawn from μ, and, for J≥1J≥ 1 and δ∈(0,1)δ∈(0,1), ιH:=log(4JHS¯H||/δ) _H:= (4JH S_H|A|/δ). Lemma J.11 (Noise). With probability at least 1−δ/21-δ/2, ‖w^j,h‖∞≤u¯j(R):=w¯(R)ιH/(2Bj)\| w_j,h\|_∞≤ u_j(R):= w(R) _H/(2B_j) for all j<Jj<J and h<Hh<H. Proof. Fix (j,h,x)(j,h,x) with x∈hx _h, and order the B terms of w^j,h(x) w_j,h(x) by the successive level-h visits of x during epoch j, the rollouts being taken in chronological order. Given jG_j and the past of these visits, the successor of a visit is distributed as p(⋅|x)p(·|x), while Ω∗(Qj,h+1) ^*(Q_j,h+1) is jG_j-measurable. The terms are therefore martingale differences, lying in jG_j-measurable intervals of length at most w¯(R)/Bj w(R)/B_j on j<σR\j< _R\ and vanishing off it, and their number is BjB_j by the cap (Lemma H.14). Lemma E.2 with u=u¯j(R)u= u_j(R) gives a failure probability 2e−ιH2e^- _H, and a union bound over the at most JHS¯H||JH S_H|A| triples concludes. ∎ Lemma J.12 (Epoch length). With probability at least 1−δ/41-δ/4, every epoch j<Jj<J with j<σRj< _R lasts at most NjHN_jH transitions, where Nj:=⌈(Bj+ιH)2/q⌉N_j:= ( B_j+ _H)^2/q . Proof. Conditionally on jG_j, the rollouts of epoch j, extended beyond its end if needed, are i.i.d., and on j<σR∈j\j< _R\ _j, Lemma J.9 with =jG=G_j shows that each of them visits a given (h,x)(h,x), x∈hx _h, with probability at least q. As in the proof of Lemma H.15, the exact Chernoff bound (E.1) with b=1b=1, a=Njqa=N_jq and c=Bjc=B_j bounds by e−ιHe^- _H the conditional probability that fewer than BjB_j of the first NjN_j rollouts visit (h,x)(h,x), and a union bound over the at most JHS¯H||JH S_H|A| triples concludes. Independence across resets replaces the m-step sub-sampling of Lemma H.15. ∎ Theorem J.1 (Finite-time bound for epoch planning). Let R≥e0R≥ e_0 satisfy the radius admissibility condition e0+2M(AR)maxj<Ju¯j(R)≤R.e_0+2M(A_R) _j<J u_j(R)≤ R. (J.10) Then, with probability at least 1−δ1-δ, every epoch j<Jj<J lasts at most NjHN_jH transitions, maxh<H‖Qj,h−QΩ∗‖sp,h≤R _h<H\|Q_j,h-Q^*_ \|_sp,X_h≤ R for every j≤Jj≤ J, and, for every j≤Jj≤ J and h<Hh<H, ‖Qj,h−QΩ,h∗‖sp \|Q_j,h-Q^*_ ,h \|_sp ≤j<H−h(1−β~m(AR))n(h,j)D0 1_\j<H-h\ (1- β_m(A_R) )^n(h,j)D_0 (J.11) +2∑k=0minj,H−h−1(1−β~m(AR))n(h,k)u¯j−1−k(R). +2 _k=0 \j,H-h\-1 (1- β_m(A_R) )^n(h,k) u_j-1-k(R). In particular, the initialization has vanished exactly for j≥H−hj≥ H-h. Proof. Work on the intersection of the events of Lemmas J.11 and J.12, of probability at least 1−δ1-δ, on which all epochs terminate (Proposition J.2). We prove by induction on j≤Jj≤ J that σR>j _R>j and that (J.11) holds at rank j; the length bound then follows from Lemma J.12. At j=0j=0, both claims were checked above. Assume σR>j−1 _R>j-1, fix h<Hh<H, let ℓ:=minj,H−h := \j,H-h\ and consider the diagonal profile Pl:=Qj−(l−h),lP_l:=Q_j-(l-h),l, l∈ h,h+ℓ ∈ h,h+ . For l<h+ℓl<h+ , the epoch i:=j−(l−h)−1i:=j-(l-h)-1 satisfies 0≤i<j≤σR0≤ i<j≤ _R, so (J.9) gives Pl=Ω,l(Pl+1)+w^i,lP_l=T_ ,l(P_l+1)+ w_i,l: the residuals of the diagonal are the stopped noises, and Lemma J.11 and ∥⋅∥sp≤2∥⋅∥∞\|·\|_sp≤ 2\|·\|_∞ bound their span by 2u¯i(R)2 u_i(R). For l>hl>h, PlP_l is either computed at epoch j−(l−h)<j≤σRj-(l-h)<j≤ _R or equal to QB|HQ_B|_X_H, whose action span is at most AΔBA_ _B by Lemma J.5, so ‖Pl‖asp≤AR\|P_l\|_asp≤ A_R, and AR≥AΔBA_R≥ A_ _B. The last element is Ph+ℓ=QB|H=QΩ,H∗P_h+ =Q_B|_X_H=Q^*_ ,H if ℓ=H−h =H-h, and Ph+ℓ=Q0,h+jP_h+ =Q_0,h+j, within D0D_0 of QΩ,h+j∗Q^*_ ,h+j, if ℓ=j<H−h =j<H-h. The first inequality of Lemma J.8 with h0=h_0=h and he=h+ℓh_e=h+ gives (J.11) at rank j, since n(h,ℓ)=n(h,j)n(h, )=n(h,j) when ℓ=j =j. Finally, Lemma J.7 and M(h,AR)≤M(AR)M(h,A_R)≤ M(A_R) bound the sum in (J.11) by 2M(AR)maxi<Ju¯i(R)2M(A_R) _i<J u_i(R), so Lemma J.5 and (J.10) give ‖Qj,h−QΩ∗‖sp,h≤D0+2M(AR)maxi<Ju¯i(R)+ΔB≤R\|Q_j,h-Q^*_ \|_sp,X_h≤ D_0+2M(A_R) _i<J u_i(R)+ _B≤ R for every h<Hh<H, i.e. σR>j _R>j. ∎ Unlike (H.14) and (H.35), condition (J.10) has no factor β¯(R)/(β¯(R)+1) β(R)/( β(R)+1): the profile at epoch j is an explicit function of the data of epochs i<ji<j along finite diagonals, and the initialization is forgotten exactly, not geometrically. Proof of Theorem I.1. We apply Theorem J.1 with Bj≡B_j≡ B and R=2e0R=2e_0, for which (J.11) and Lemma J.7 give 2M(h,AR)u¯(R)2M(h,A_R) u(R). Then β~m(AR)=β~ β_m(A_R)= β, w¯(R)=w¯ w(R)= w, M(AR)=M(A_R)=M and qπ(AR)q_π(A_R) is the constant q of the greedy behavior, so that 2Mu¯(R)=2Mw¯ιH/(2B)2M u(R)=2M w _H/(2B), which is at most e0e_0 if and only if B≥B(e0)B≥ B(e_0): condition (J.10) holds. With probability at least 1−δ1-δ, every epoch lasts at most NH=H⌈(B+ιH)2/q⌉NH=H ( B+ _H)^2/q transitions and, since J≥HJ≥ H, (J.11) at h=0h=0 gives ‖QJ,0−QΩ,0∗‖sp≤2Mw¯ιH/(2B)\|Q_J,0-Q^*_ ,0\|_sp≤ 2M w _H/(2B). Corollary J.1 at h=0h=0, with n(0,H)=⌊(H−h1)/m⌋n(0,H)= (H-h_1)/m and β~≤β~m(AΔB) β≤ β_m(A_ _B), bounds ‖QΩ,0∗−QΩ∗‖sp,0\|Q^*_ ,0-Q^*_ \|_sp,X_0 by (1−β~)⌊(H−h1)/m⌋ΔB(1- β) (H-h_1)/m _B, and the triangle inequality gives (I.2). For ε∈(0,e0) ∈(0,e_0), the choice B=⌈B(ε/2)⌉≥B(e0)B= B( /2) ≥ B(e_0) makes the noise term at most ε/2 /2, and the choice of H gives ⌊(H−h1)/m⌋≥β~−1log(2ΔB/ε) (H-h_1)/m ≥ β^-1 (2 _B/ ), so that 1−u≤e−u1-u≤ e^-u makes the bias term at most ε/2 /2. The J=HJ=H epochs use at most H⋅NH· NH transitions, with N=Θ~(B/q)N= (B/q), B=Θ~(M2w¯2/ε2)B= (M^2 w^2/ ^2) and H=Θ~(h1+Heff)H= (h_1+H_eff), which gives O~((h1+Heff)4w¯2/(qε2)) O ((h_1+H_eff)^4 w^2/(q ^2) ); the dependence of ιH _H on J=HJ=H is logarithmic. ∎ Remark J.3 (Planning and learning). For h1=0h_1=0, Theorem I.1 costs O~(m4w¯2/(β~4qε2)) O(m^4 w^2/( β^4q ^2)) transitions, against O~(m4w¯2/(β~3qRε2)) O(m^4 w^2/( β^3q_R ^2)) for Theorem 2: at equal coverage constant, epoch planning is a factor 1/β~1/ β more expensive. Propagation is paid twice, once in depth, since a rollout gives one sample per level to H distinct tables, and once in epochs, since J=HJ=H iterations are needed to propagate the boundary to the root. Per table, planning is m times more efficient, the independence across resets removing the sub-sampling of Lemma H.15. Merging the tables removes the extra factor (Subsection J.7). J.7 Merging the tables: epoch Q-learning with resets The factor 1/β~1/ β of Remark J.3 is the price of maintaining H depth-indexed tables. By Corollary J.1, the planning solution is within ε of QΩ∗Q^*_ , up to constants, at every depth h1≤h≤H−m−Heff(AΔB)log+(ΔB/ε)h_1≤ h≤ H-m-H_eff(A_ _B) _+( _B/ ), so these tables are redundant on most of the horizon. Merging them gives Algorithm 6, which is Algorithm 1 run on a reset process: the transient prefix of each rollout is played with μ and discarded, and the transitions retained at depths h≥h1h≥ h_1 are transitions of the MDP restricted to ∞S_∞, since sh,sh+1∈∞s_h,s_h+1 _∞. This MDP satisfies Assumption 1 (Remark I.1.(i)). We keep the notation of Subsection H.3, every constant being computed on ∞S_∞: e0=‖Q0−QΩ∗‖spe_0=\|Q_0-Q^*_ \|_sp, the localized constants of Definition H.2, the exit time (H.21), the blocks ℬiB_i and the σ-algebras jG_j. Only the coverage constant and the epoch length change. Lemma J.13 (Cover time with resets). Assume H≥h1+m+1H≥ h_1+m+1 and let K0:=⌊(H−1−h1)/m⌋≥1K_0:= (H-1-h_1)/m ≥ 1. For j≥0j≥ 0, a deterministic radius u≥0u≥ 0 and δ′∈(0,1)δ ∈(0,1), let j(u):=∥Qj∥asp≤uD_j(u):=\\|Q_j\|_asp≤ u\, let qres(u):=βmγΩ(u)mηminq^res(u):= _m _ (u)^m _ , or qres(u):=βmηminq^res(u):= _m _ if the actions at depths h≥h1h≥ h_1 are drawn from μ instead of πΩ,Qj _ ,Q_j, let ι(δ′):=log(|∞|||/δ′) (δ ):= (|S_∞||A|/δ ) and Ljres(u,δ′):=H⌈1K0⌈(Bj+ι(δ′))2qres(u)⌉.L^res_j(u,δ ):=H 1K_0 ( B_j+ (δ ) )^2q^res(u) . (J.12) Then, almost surely, ℙ(ℓj>Ljres(u,δ′)∩j(u)∣j)≤δ′ 1j(u)P (\ _j>L^res_j(u,δ )\ _j(u) _j )≤δ \,1_D_j(u). If moreover H≥2(h1+m+1)H≥ 2(h_1+m+1), then Ljres(u,δ′)≤2m⌈(Bj+ι(δ′))2/qres(u)⌉+HL^res_j(u,δ )≤ 2m ( B_j+ (δ ))^2/q^res(u) +H. Proof. Conditionally on jG_j, the rollouts of epoch j, extended beyond its end if needed, are i.i.d. Consider the instants of depths h1+kmh_1+km, k∈ 1,K0 ∈ 1,K_0 , of the successive rollouts, in chronological order, fix (s,a)∈∞×(s,a) _∞×A, and let ZlZ_l indicate that the l-th instant visits (s,a)(s,a). On j(u)∈jD_j(u) _j, the actions at depths h≥h1h≥ h_1 are drawn from μ or from a policy dominating γΩ(u)μ _ (u)μ, and the m steps before each instant start at a depth h≥h1h≥ h_1, hence in ∞S_∞. Lemma J.9.(i) with =jG=G_j, conditional on the previous instant of the same rollout or integrated over a fresh reset, therefore gives [Zl∣Z1,…,Zl−1,j]≥qres(u)E[Z_l Z_1,…,Z_l-1,G_j]≥ q^res(u). The rest is the proof of Lemma H.15: the exact Chernoff bound (E.1) with b=1b=1, a=Kqres(u)a=Kq^res(u) and c=Bjc=B_j, where K:=⌈(Bj+ι(δ′))2/qres(u)⌉K:= ( B_j+ (δ ))^2/q^res(u) , and a union bound over ∞×S_∞×A show that, with conditional probability at least 1−δ′1-δ , every pair is visited BjB_j times within K instants, which ⌈K/K0⌉ K/K_0 rollouts of H transitions provide. Finally, K0≥(H−1−h1−m)/m≥H/(2m)K_0≥(H-1-h_1-m)/m≥ H/(2m) when H≥2(h1+m+1)H≥ 2(h_1+m+1). ∎ Theorem J.2 (Epoch Q-learning with resets). Let H≥h1+m+1H≥ h_1+m+1. Theorem H.2, under Assumption H.1, and Corollary H.4 hold for Algorithm 6, with every constant computed on the MDP restricted to ∞S_∞ and the epoch length Lj(δ′)L_j(δ ) of Lemma H.15 replaced by Ljres(AR,δ′)L^res_j(A_R,δ ). In particular, Theorem 2 holds with L:=H⌈1K0⌈(B+ι)2/qres⌉L:=H 1K_0 ( B+ )^2/q^res , where qres:=γ¯β~ηminq^res:= γ β _ , or qres:=βmηminq^res:= _m _ if the actions at depths h≥h1h≥ h_1 are drawn from μ. For H≥2(h1+m+1)H≥ 2(h_1+m+1), this gives the sample complexity O~(m4w¯2β~3qresε2+mHβ~), O ( m^4\, w^2 β^3\,q^res\, ^2+ mH β ), (J.13) whose first term is O~(m4w¯2/(β~3⋅βmηmin⋅ε2)) O (m^4 w^2/( β^3· _m _ · ^2) ) under the behavior μ and O~(m4w¯2/(β~3⋅γ¯β~ηmin⋅ε2)) O (m^4 w^2/( β^3· γ β _ · ^2) ) under the greedy behavior. Proof. The proofs of Subsection H.3 use the trajectory through three properties only: the behavior during epoch j is jG_j-measurable; given the past, the successor of a retained visit of (s,a)(s,a) is distributed as p(⋅|s,a)p(·|s,a) (Lemma H.18.(i)); and the cap makes the number of retained samples deterministic (Lemma H.14). The reset process satisfies all three, by the Markov property within each rollout and the independence of the rollouts given jG_j. The virtual flow, tracking and no-escape arguments (Lemmas H.17 and H.21, Proposition H.3) only concern the update (H.19), and the trajectory structure only enters through Lemma H.15, which Lemma J.13 replaces, and Proposition H.2, whose proof applies verbatim with it. The count follows as in the proof of Theorem 2, with L≤2m⌈(B+ι)2/qres⌉+HL≤ 2m ( B+ )^2/q^res +H by Lemma J.13. ∎ Remark J.4 (What the merge buys and what it loses). (i) What the resets bring. Relative to Theorem 2, (J.13) takes ηmin _ over ∞×S_∞×A, with the minorizing measure of Assumption 4, which can be much larger when ∞⊊S_∞ . The factor γΩm _ ^m removed under the behavior μ is not due to the resets: it is removed on a single trajectory as well (Remark H.7). Relative to epoch planning, merging the tables saves the factor 1/β~1/ β of Remark J.3, and the rollout length only enters through H≥2(h1+m+1)H≥ 2(h_1+m+1) and the second term of (J.13). (i) The boundary becomes the initialization. There is no horizon bias: the target is QΩ∗Q^*_ itself, and the boundary value QBQ_B of the planners plays the role of Q0Q_0. The horizon bias (1−β~)⌊(H−h1)/m⌋ΔB(1- β) (H-h_1)/m _B of Theorem I.1 and the initialization term (1−β~)⌊J(T)/m⌋e0(1- β) J(T)/m e_0 of Theorem 2 are the same term under two names. (i) The exact fixed point is lost. The single table only solves Ω(QΩ∗)=QΩ∗+ρΩ∗T_ (Q^*_ )=Q^*_ +ρ^*_ 1, which is why Algorithm 6 needs the centering projection Π , and the depth profile of Theorem J.1 has no counterpart. (iv) A single parameterized algorithm. Without resets, Algorithm 6 is Algorithm 1, and Theorems J.2 and 2 are one statement in which (H,ξ)(H,ξ) only enters through the coverage constant and the epoch length. Appendix K The α-divergence regularizers Table 1 summarizes Theorems K.1 and K.1 for Ωα,s:=Dα(⋅∥μs) _α,s:=D_α(·\| _s). In it, Qa:=Q(s,a)Q_a:=Q(s,a), μa:=μ(a|s) _a:=μ(a|s), expq(x):=[1+(1−q)x]+1/(1−q) _q(x):=[1+(1-q)x]_+^1/(1-q) with exp1=exp _1= , and ζs _s is the normalizer making πΩ,Q(⋅|s) _ ,Q(·|s) sum to one; strong convexity is with respect to ∥⋅∥1\|·\|_1 and fails for α>2α>2, and the profile bounds are tight as μmin→0 _ → 0 (Remark K.1). On the simplex Dα(p∥q)=D1−α(q∥p)D_α(p\|q)=D_1-α(q\|p) (Cichocki and Amari, 2010), so divergences to μ in either argument belong to the family. For a general reference μs _s, the greedy map of Ωα,s _α,s is the f-softargmax of Roulet et al. (2025) with reference measure μs _s, and Belousov and Peters (2019) give its closed form in MDPs. For μ uniform, Ωα,s _α,s is the Tsallis negentropy of index α up to the factor ||α−1|A|^α-1 on τ and an additive constant, and for α≥1α≥ 1 its greedy map is the α-entmax of Peters et al. (2019). Tsallis-KL regularization toward a reference policy, which is this family up to a reparametrization of the index and a rescaling of τ, is used in reinforcement learning by Zhu et al. (2023), and Labbi et al. (2026) use the same divergences for policy gradient. Parts (i)–(i) of Theorem K.1 collect known facts in our notation; our contribution is the domination profile (iv) and the regularization modulus (v). Since Ωα,s≥0 _α,s≥ 0 vanishes at μs _s, Lemma F.1.(i) gives gQ≤0g_Q≤ 0, while gQ≥−ΛΩαpt(R)g_Q≥- ^pt_ _α(R) for ‖Q‖asp≤R\|Q\|_asp≤ R by definition; hence ΛΩα≤ΛΩαpt _ _α≤ ^pt_ _α, and we bound the latter. Theorem K.1 (The α-divergence regularizer). Let α∈ℝα , τ>0τ>0 and μ have full support. (i) Domain. dom(Ωα,s)=Δ()dom( _α,s)= (A) for α>0α>0 and πs>0\ _s>0\ for α≤0α≤ 0; Ωα,s _α,s is proper and closed, and Assumption G.1 holds with π~=μ π=μ. (i) Strong convexity (Pinsker’s inequality at α=1α=1). Ωα,s _α,s is μmin(1−α)+ _ ^(1-α)_+-strongly convex with respect to ∥⋅∥1\|·\|_1 for α≤2α≤ 2, and strictly but not strongly convex for α>2α>2. (i) Greedy policy and conjugate (Roulet et al., 2025, Proposition 1). For α≠1α≠ 1, πΩα,Q(a|s) _ _α,Q(a|s) =μ(a|s)[1+(α−1)(Q(s,a)−ζs(Qs))τ]+1α−1, =μ(a|s) [1+ (α-1) (Q(s,a)- _s(Q_s) )τ ]_+ 1α-1, (K.1) Ωα∗(Q)(s) ^*_α(Q)(s) =(α−1)⟨πΩα,Q(⋅|s),Qs⟩+ζs(Qs)α(α≠0), = (α-1) _ _α,Q(·|s),Q_s + _s(Q_s)α (α≠ 0), (K.2) where ζs(Qs) _s(Q_s) is the unique normalizer making (K.1) sum to one; the truncation is active only for α>1α>1. For any a∗∈argmaxaQ(s,a)a^*∈ _aQ(s,a), maxaQ(s,a)−τμ(a∗|s)1−α−1α−1≤ζs(Qs)≤maxaQ(s,a), _aQ(s,a)-τ\, μ(a^*|s)^1-α-1α-1\ ≤\ _s(Q_s)\ ≤\ _aQ(s,a), so that ζs _s is computable by bisection. (iv) Domination profile. γΩα(R)≥exp2−α(−R/τ) _ _α(R)≥ _2-α(-R/τ). If α>1α>1 and ||≥2|A|≥ 2, then γΩα(R)=0 _ _α(R)=0 for R≥τμmax1−α/(α−1)R≥τ _ ^1-α/(α-1), with μmax:=maxs,aμ(a|s) _ := _s,aμ(a|s). Hence (P1) holds if and only if α≤1α≤ 1. (v) Regularization modulus. For α>0α>0, ΛΩαpt(R)≤τΔα ^pt_ _α(R)≤τ _α with Δα:=μmin1−α−1α(α−1) _α:= _ ^1-α-1α(α-1), and Δα≤1α(1−α) _α≤ 1α(1-α) for 0<α<10<α<1. For α≤0α≤ 0, ΛΩαpt(R)≤inf0<ε≤1εR+τ1−εα(1−α)=Θ(τ11−αR−α1−α)α<0,Θ(τlog(R/τ))α=0, ^pt_ _α(R)\ ≤\ _0< ≤ 1 \ R+τ\, 1- ^α(1-α) \= cases (τ 11-αR -α1-α )&α<0,\\ (τ (R/τ) )&α=0, cases (K.3) the fraction read as log(1/ε) (1/ ) at α=0α=0, and these rates are attained. Hence the modulus is bounded in R if and only if α>0α>0. Proof. Fix s and write πa _a, μa _a, QaQ_a for πs(a) _s(a), μ(a|s)μ(a|s), Q(s,a)Q(s,a). (i) The term πaαμa1−α _a^α _a^1-α is finite for every πa≥0 _a≥ 0 when α>0α>0, and infinite at πa=0 _a=0 when α<0α<0, as is log(μa/πa) ( _a/ _a) at α=0α=0. Ωα,s _α,s is convex and continuous on its domain and tends to +∞+∞ at the boundary when α≤0α≤ 0, hence proper and closed; Dα(μs∥μs)=0D_α( _s\| _s)=0 gives Assumption G.1 with π~=μ π=μ. (i) On the relative interior ∇2Ωα,s(π)=diag(πaα−2μa1−α)∇^2 _α,s(π)=diag( _a^α-2 _a^1-α), and Cauchy–Schwarz gives ⟨∇2Ωα,s(π)h,h⟩≥‖h‖12/∑aπa2−αμaα−1 ∇^2 _α,s(π)h,h ≥\|h\|_1^2 / _a _a^2-α _a^α-1. For 1≤α≤21≤α≤ 2, Hölder bounds the sum by (∑aπa)2−α(∑aμa)α−1=1( _a _a)^2-α( _a _a)^α-1=1; for α<1α<1, πa2−α≤πa _a^2-α≤ _a bounds it by μminα−1 _ ^α-1. Integrating along segments and closing by lower semicontinuity gives the modulus. For α>2α>2 the Hessian is positive definite on the interior but its entries vanish as πa→0 _a→ 0. (i) The form (K.1) of the maximizer, with a unique normalizer, is Roulet et al. (2025, Proposition 1) with reference measure μs _s and logits Qs/τQ_s/τ. For the bracket, S(ζ):=∑aμa[1+(α−1)(Qa−ζ)/τ]+1/(α−1)S(ζ):= _a _a[1+(α-1)(Q_a-ζ)/τ]_+^1/(α-1) is continuous and non-increasing on its domain, ℝR if α>1α>1 and (maxaQa−τ/(1−α),+∞)( _aQ_a-τ/(1-α),+∞) if α<1α<1; S(maxaQa)≤1S( _aQ_a)≤ 1, every ratio πa/μa _a/ _a being then at most 11; and S≥1S≥ 1 at ζ=maxaQa−τ(μa∗1−α−1)/(α−1)ζ= _aQ_a-τ( _a^*^1-α-1)/(α-1), which lies in the domain and where the term of a∗a^* alone equals 11. Finally, for α≠0α≠ 0, with ua:=1+(α−1)(Qa−ζs)/τu_a:=1+(α-1)(Q_a- _s)/τ, we have πaαμa1−α=πaua _a^α _a^1-α= _au_a and ∑aπaua=1+(α−1)(⟨π,Qs⟩−ζs)/τ _a _au_a=1+(α-1)( π,Q_s - _s)/τ, hence τΩα,s(π)=(⟨π,Qs⟩−ζs)/ατ _α,s(π)=( π,Q_s - _s)/α, and Ωα∗(Q)(s)=⟨π,Qs⟩−τΩα,s(π) ^*_α(Q)(s)= π,Q_s -τ _α,s(π) is (K.2). (iv) If ‖Q‖asp≤R\|Q\|_asp≤ R then Qa−ζs≥Qa−maxbQb≥−RQ_a- _s≥ Q_a- _bQ_b≥-R by (i), and x↦[1+(α−1)x/τ]+1/(α−1)x [1+(α-1)x/τ]_+^1/(α-1) is non-decreasing for every α≠1α≠ 1, which gives the bound; α=1α=1 is the limit. For α>1α>1, take a∗a^* with μa∗=μmax _a^*= _ , Qa∗=0Q_a^*=0 and Qb=−RQ_b=-R otherwise. The candidate π=δa∗π= _a^* has normalizer ζ=τ(1−μmax1−α)/(α−1)ζ=τ(1- _ ^1-α)/(α-1) and ub=μmax1−α−(α−1)R/τ≤0u_b= _ ^1-α-(α-1)R/τ≤ 0 for b≠a∗b≠ a^* as soon as R≥τμmax1−α/(α−1)R≥τ _ ^1-α/(α-1); by uniqueness it is the greedy policy, so γΩα(R)=0 _ _α(R)=0. That the greedy map can be sparse for α>1α>1 is known (Peters et al., 2019; Roulet et al., 2025); what (P1) requires, and what (iv) provides, is the uniform lower bound and the explicit threshold. (v) Let a∗∈argmaxaQaa^*∈ _aQ_a. For α>0α>0, Ωα,s _α,s is bounded on Δ() (A) by 0≤Ωα,s≤maxaΩα,s(δa)=Δα0≤ _α,s≤ _a _α,s( _a)= _α, the maximum of a convex function on the simplex being attained at a vertex, and the boundedness property of the conjugate (Geist et al., 2019, Proposition 1(i)) gives maxaQa−τΔα≤Ωα∗(Q)(s)≤maxaQa _aQ_a-τ _α≤ ^*_α(Q)(s)≤ _aQ_a; testing the maximum defining Ωα∗(Q)(s) ^*_α(Q)(s) against δa∗ _a^* gives the same bound with μa∗ _a^* in place of μmin _ . For 0<α<10<α<1, Δα=1−μmin1−α(1−α)≤1α(1−α) _α= 1- _ ^1-α(1-α)≤ 1α(1-α). For α≤0α≤ 0, Ωα,s _α,s is unbounded and we test instead πε:=(1−ε)δa∗+εμsπ :=(1- ) _a^*+ _s: ⟨πε,Qs⟩≥maxaQa−εR π ,Q_s ≥ _aQ_a- R, and πε≥εμsπ ≥ _s with x↦xαx x^α non-increasing gives Ωα,s(πε)≤εα−1α(α−1) _α,s(π )≤ ^α-1α(α-1), with limit log(1/ε) (1/ ) at α=0α=0. Minimizing over ε , at ε=min1,(τ/((1−α)R))1/(1−α) = \1,(τ/((1-α)R))^1/(1-α)\, gives (K.3). The rates are attained with Qa∗=RQ_a^*=R and Qb=0Q_b=0: writing πb=δ _b=δ, the gap is minδRδ+τΩα,s(π) _δ\Rδ+τ _α,s(π)\ with Ωα,s(π)∼μb1−αδα/(α(α−1)) _α,s(π) _b^1-αδ^α/(α(α-1)) for α<0α<0 and ∼μblog(1/δ) _b (1/δ) for α=0α=0, whose minimizer gives the same powers of R. ∎ Corollary K.1 (Four landmarks). Theorem K.1 at α∈1,0,12,2α∈\1,0, 12,2\ gives the columns of Table 1, and the conjugates: α Ω∗(Q)(s) ^*(Q)(s) ζs∈ _s∈ 11 τlog∑aμaeQa/τ _a _ae^Q_a/τ – 00 ζs−τ∑aμalog(1+ζs−Qaτ) _s-τ _a _a (1+ _s-Q_aτ ) (maxaQa−τ,maxaQa]( _aQ_a-τ, _aQ_a] 12 12 2ζs−⟨πΩ,Q(⋅|s),Qs⟩2 _s- _ ,Q(·|s),Q_s (maxaQa−2τ,maxaQa]( _aQ_a-2τ, _aQ_a] 22 12(⟨πΩ,Q(⋅|s),Qs⟩+ζs) 12 ( _ ,Q(·|s),Q_s + _s ) [minaQa,maxaQa][ _aQ_a, _aQ_a] Proof. At α=1α=1 the normalizer factors out of the exponential. At α=0α=0, (K.1) reads πa=τμa/(τ+ζs−Qa) _a=τ _a/(τ+ _s-Q_a), so ∑aπa(τ+ζs−Qa)=τ _a _a(τ+ _s-Q_a)=τ gives ⟨π,Qs⟩=ζs π,Q_s = _s, and Ω∗(Q)(s)=⟨π,Qs⟩−τKL(μs∥π) ^*(Q)(s)= π,Q_s - ( _s\|π) gives the conjugate; the modulus bound is (K.3) at ε=min1,τ/R = \1,τ/R\. At α=12α= 12 and α=2α=2 the conjugates are those of Theorem K.1.(i), and Ωs(δa)=4(1−μa) _s( _a)=4(1- _a) and 12(1/μa−1) 12(1/ _a-1) give the modulus bounds. The brackets at α∈0,12α∈\0, 12\ follow from Theorem K.1.(i); at α=2α=2, the sum in (K.1) is at least 11 at ζ=minaQaζ= _aQ_a and at most 11 at ζ=maxaQaζ= _aQ_a. ∎ Remark K.1 (Sharpness). For a fixed μ, ζs _s is non-decreasing in each QaQ_a while πb/μb _b/ _b is non-increasing in ζs−Qb _s-Q_b, so the infimum defining γΩ(R) _ (R) is attained with the least likely action at the bottom and every other one at +R+R; symmetrically, ΛΩpt(R) _ ^pt(R) is attained with the least likely action alone at the top and every other one at −R-R. Both depend on μ only through μmin _ , and the explicit normalizers give, for instance, γΩ(R)=(μmin+(1−μmin)eR/τ)−1 _ (R)= ( _ +(1- _ )e^R/τ )^-1 for the reverse KL and γΩ(R)=[1−(1−μmin)R/τ]+ _ (R)=[1-(1- _ )R/τ]_+ for the χ2χ^2, whose limits as μmin→0 _ → 0 are the bounds of Table 1. Inside the window 0<α≤10<α≤ 1 the comparison is quantitative and depends on how τ is matched across α. At fixed τ, the profile decays as (R/τ)−1/(1−α)(R/τ)^-1/(1-α), more slowly for smaller α, while the modulus constant Δα=(1−μmin1−α)/(α(1−α)) _α=(1- _ ^1-α)/(α(1-α)) tends, as μmin→0 _ → 0, to 1/(α(1−α))1/(α(1-α)) for α<1α<1, minimal at the Hellinger value 44, and to +∞+∞ at α=1α=1: the Hellinger regularizer is the most robust to a small μmin _ , whereas for a balanced μ the reverse KL can have the smaller constant: for a uniform μ, log||<4(1−||−1/2) |A|<4(1-|A|^-1/2) as long as ||≤24|A|≤ 24. The lower bound of Theorem K.1.(iv) equals (f′)−1(−R/τ)(f )^-1(-R/τ) for the generator f(u)=(uα−αu+α−1)/(α(α−1))f(u)=(u^α-α u+α-1)/(α(α-1)) of DαD_α, the inverse link that also drives the policy-gradient analysis of Labbi et al. (2026): their probability floor (Labbi et al., 2026, Theorem 3.5) is proportional to (f′)−1(f )^-1 evaluated at an argument of order −1/λ-1/λ, where λ is their temperature, up to factors depending on the discount and the initial distribution, and their exponential-versus-polynomial dichotomy between the KL and α<1α<1 is the one displayed by our profile bounds. They also find that the best α lies strictly inside (0,1)(0,1) (Labbi et al., 2026, Corollary 4.8), for a different criterion.